Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper shows that training a model with both cross-entropy loss and caption supervision are more robust than the model trained only on cross-entropy loss. It also releases CaptioNet, which is a 100-class subset of Imagenet with caption supervision.

strength_and_weaknesses: Strength: The paper is clear overall. It studies an important question in image classification with captioning supervision.

Weakness: The novelty of the paper is limited in my view. The authors conduct some experiments to show that caption supervisions are helpful to make the model more robust, but this method has been proposed before. The dataset is interesting, but its contribution may not be enough.

clarity,_quality,_novelty_and_reproducibility: Overall the paper is clear. I have a few suggestions:
 - It would be good to provide some examples explaining what are the supervisions for both the cross-entropy loss and caption supervisions.
 - It would be very helpful to provide the definition of “robustness” with concrete description of the metric. The authors cited Taori et.al, but it is not clear how reliable those metrics are.

summary_of_the_review: Based on the limited novelty and contribution, I think it does not reach the acceptance threshold marginally. I am happy to adjust my ratings after rebuttal.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper argues that using caption supervision in the way of using subset matching to label unsupervised data and use it to train cross-entropy loss would achieve a similar performance on robustness as CLIP model. Meanwhile, this paper proposes CaptionNet, a 100 classes Imagenet-based dataset with captions. 
strength_and_weaknesses: Strength:

1. The authors conduct comprehensive experiments and propose a dataset CaptionNet which is a sub-set of ImageNet and with newly added 50000 images, it is class-balanced and with caption information.

Weakness:

Overall the core contribution of the work is not clearly illustrated, and some key factors, such as subset matching are not well introduced even in the supplementary material. Also for the proposed dataset CaptionNet, it is not clear how this dataset can be further used for researchers as there are no clearly targetted research topics relating the CaptionNet.

1. The core statement of the work "caption supervision enables robust learners," seems under-illustrated. 
“CNNs trained on a standard cross-entropy loss also benefit from caption supervision." This expression in the abstract is not clear to me. Does it mean by subset matching labeling on unlabeled data, CE loss could be utilized and achieves similar or higher robustness compared with CLIP? After reading the paper, I am not sure in which section this point is addressed with deep discussion. Explanations are welcomed if there are misunderstandings. 

1. In this work, the authors consider comparing the standard supervised training with CLIP model from the perspective of cross-entropy loss and clip-style loss. I am confused that from this perspective, does it degrade the fundamental difference between supervised and unsupervised learning into a loss comparison? is it proper to just address this as a difference in loss function? 

1. Key factors are not well introduced in the paper. For subset matching, although it is used in another paper, it is still a new method that is not widely recognized by the community. Briefly introducing how it works and how it relates to your method is a necessity. However, in the supplementary section E, it is still not clear how it works and why the following evaluation in Sec.E matters. 

1. For CaptionNet, the introduction is not related to what the research topics it targets at. For example, the authors can not only bold/highlight the combination of sub-datasets but also highlight/emphasize the topic that the combination targets to explore. 

1. The "label noise" (sec.4.1) and "captioning strategy" (sec.4.2),  I feel like the cause for label noise or captioning strategy is the way (subset matching) used in CaptionNet to create caption labels. However, the authors use the drawback in dataset creation as the topic this dataset is to discuss. 

1. There are quite an amount of unclear abbreviations and references, which makes the paper not easy to follow. For example, "in100", ''in1000", "oi100" are not first introduced as abbreviations for each sub-dataset. In Table2. it is not introduced what "Eff. Rob. Acc. Ratio" is  , which I am afraid is also not discussed in the paper. In Table 4, all the abbreviations in the first row (evaluation metrics) and names of the evaluated models are not introduced, which makes the table really hard to read and conveys little useful information currently. 
clarity,_quality,_novelty_and_reproducibility: Clarity: this is the major part the paper could further work on, as illustrated in the weaknesses. 

Novelty: there is no proposed new methodology, and the major discovery is not clearly illustrated. 

Reproducibility: the authors provide the training details in the paper, but the details of subset matching seem missing. 
summary_of_the_review: Based on the weaknesses, I think the paper could be enhanced further by better illustrating their core contribution and better introducing CaptionNet by organizing it with the targeted research topics.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors try to understand why CLIP-like models can have great robustness on natural distribution shifts. To understand the difference, the authors collect a dataset CaptionNet and design a careful control experiment upon it. The authors show that standard classification cross-entropy loss can also be robust in some cases. The authors did many ablation studies to show how robustness and accuracy are affected by different factors.
strength_and_weaknesses: Strength:
- The authors provide discussions on how this paper relates to and differs from related works.
- The authors carefully designed a set of training datasets to study the effect of different factors in vision model training, including losses, label noises, caption quality, etc.
- The authors in the end highlighted a few aspects that can guide future researchers in studying vision language models.

Weaknesses:
- The paper presentation can be improved.
    - There is one missing section link and a few grammar errors.
    - The acronyms in100 and oi100 are not been officially introduced. Similarly, there are many acronyms, for example, in Table 4, are not described.
    - The experiment setup is not very clear at first sights, like how the evaluation works.
    - The authors use CLIP-loss sometimes and VL sometimes.
    - I also feel like the text is too narrative and not very essay-y.
- Although the authors show empirically that CE and VL-loss have different behaviors, there is no theoretical analysis nor any guideline on how to use the lesson to improve the loss. Can we modify the CE loss for image classification so that it is less prone to label noises, or can we modify the VL loss so that it can learn more from clean data?
- The authors mention in 4.1 that oi100 has 90% label noise; where does this number 90% come from?
    - Also for this section, the authors say that the noises are not equal while they are both around 90% accuracy. I wonder if the authors have looked at top-5. It is possible that the clip label noises are more reasonable while the subset matching label noises are more wild.
clarity,_quality,_novelty_and_reproducibility: The paper's clarity needs to be improved while the topic is interesting. If the dataset CaptionNet is released, the paper should be reproducible.
summary_of_the_review: Overall, this paper is investigating an interesting problem and conveys a clear message in the end. However, the experiments between the motivation and conclusion are hard to follow. The paper has great potential to be impactful but it is not ready for publishing yet.
I am not sure if ICLR allows submitting a newer version of the paper. If a better manuscript is not allowed to submit, I am inclined to reject it.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The authors construct extended datasets like ImageNet & OpenImages with additional image captions and class labels which allows a more fair comparison between training models on image captions vs. standard class labels. The authors experiment with increasing the training dataset in size but also varying different types of captions/labels with respect to their acquisition source and noise level.
strength_and_weaknesses: **Strengths:**

- The proposed experimental setup is potentially very interesting as it allows to compare different training strategies relatively fairly: using image captions vs. class labels
- The authors experiment with many different datasets also using combinations of them

**Weaknesses:**

- The paper is not written clearly, creating some ambiguity and possible misunderstandings, also is not self-contained. To give some examples:
    - (Section 2, first paragraph) The authors mention using some “subset matching” technique to train models on unlabeled images. That technique, however, is not properly explained - not even in the sense of how the method itself works but also just what does this technique supposed to do. The details in the appendix only mention that the method matches samples with classes, but nothing says what this matching is based on. Only looking at Feng et al. (2022) that was referred to I can guess that the authors mean “substring matching” (not “subset”) where caption strings are matched to label names. But as this “subset matching” is an important part to understand for interpreting the results and observations, it should be really clear what this is, limiting the risk of misunderstandings.
    - (Table 2) The authors argue that “labeling strategy” is important and choosing different terms for matching has a significant impact on robustness. However, they do not explain what’s their proposed terms for matching, how they are obtained, what are they based on, etc. This makes drawing any conclusions from the results difficult.
    - (Section 3, beginning) ImageNet-100 explained as a “superset of ImageNet”. I would suppose it’s a union of samples in ImageNet-Captions and ImageNet-100 + some extra captions for 50,000 samples? The need for extra captions are motivated for balancing classes - but are they perfectly balanced then? The number of samples in Table 3 is 124k - shouldn’t that be 130k then?
    - “we use SimCLR transformations rather than CLIP transformations for all model training on CaptionNet” - what type of transformations? Are these transformations for data augmentation?
    - Are the evaluation numbers reported in Table 3 computed on the listed datasets for each row or on ImageNet-100 (the last paragraph of the experimental setup seems to mention that)? This should be clear to the reader without any ambiguities, as it’s very important for interpreting the results.
    - (Section 4.1 + Table 3) VL models are referred to as more tolerant of label noise than CE models, where OpenImages-100 is given as an example of a dataset with noisy labels. However, looking at Table 3, second row (oi100) it seems that it’s VL models that get much lower accuracy and robustness than CE models, which appears to be the opposite of what the authors claim
- Some of the conclusions and observations are not explicitly justified or informative/clear:
    - (Section 5, point 1)  The authors say “losses matter“, “can have a substantial effect on model robustness” - however, the exact effect/relation observed is not summarized here, although some of the relations are discussed throughout the paper
    - (Section 5, point 3) “Length matters” - this is something that I don’t see explicitly discussed in the paper. It should be clear which specific results indicate that

Minor:
- The links to figures and tables often don’t work/point to wrong places in the paper
clarity,_quality,_novelty_and_reproducibility: Many details of created dataset and experimental setup are not discussed or are ambiguous (see Weaknesses). No code is available (yet?), and the proposed dataset is not accessible either (yet?).
summary_of_the_review: The authors study a potentially interesting and insightful experimental setup, however, the paper is not written clearly to the extent that it creates a significant amount of ambiguity and risk of misunderstandings, making experimental results, observations, and conclusions difficult to interpret with confidence.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors investigate the robustness of the model trained to distribution shifts by varying the loss functions, datasets, and the selection of targeted shifts. They found that (1) models trained by Vision-Language (VL) and Cross-Entropy (CE) loss have drastically different robustness even when the training sets are in similar sizes; (2) when trained with high label noise, VL models tend to be more robust in low accuracy regime; (3) VL models enjoy longer captions; (4) When caption quality is low, longer and more descriptive captions can still benefit the training; (5) CE models can be robust but require much longer training time; (6) CE models are more sensitive to label noise than VL models. Besides the findings, the authors also present a new combined dataset called CaptionNet by sourcing 4 existing datasets and augment with over 50k samples from Flickr. 
strength_and_weaknesses: Strength:
- Thorough study on the robustness of recent popular VL models vs conventional CE models. The findings of (2)(3)(4)(5) mentioned in the summary section above are especially interesting and can be helpful to the community when comparing new VL methods with existing methods.
- Many interesting robustness capabilities of VL models only emerge when trained with hundreds of millions of samples. With the proposed CaptionNet dataset, it is now possible to have a fairer comparison between VL models and CE models.
- Figure 1 is helpful in comprehending the effect of different loss functions vs robustness.

Weaknesses
- The writing and the paper organization can be improved. Sec 3.1 contains results on captionnet, Sec 4 also contains results on captionnet. It’s not easy to follow what the authors try to convey quickly at first glance. 
- The authors perform quite a lot of ablation studies. However, there are many numbers in different tables. It is also hard to navigate through all the information due to different ablation targets. The experiments cover loss functions, label noise, caption length, caption quality, VL vs CE etc. However, the text in the current form does not lead the readers well through different sections and contents.
- There are also many typos in the text, for example page 2: “Our results that when” -> “Our results show that when”, page 5: “CaptionNet subset can be found in Section ??” -> missing reference number.
- The captions are too small to read in Figure 2 & 3.

clarity,_quality,_novelty_and_reproducibility: The novelty and reproducibility are decent. However the clarity and writing quality can be largely improved.
summary_of_the_review: I enjoy reading the findings in this submission. However the text needs major revision.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper compares the effect of captions for training vision models, through the classic cross-entropy vs a vision-language loss (e.g. CLIP). It reports a number of findings, e.g. about robustness to noise, effect of different aspects of captions (length, descriptiveness), etc. While the evaluation is extensive and includes many datasets, several of the claims are not well justified, and reviewers raise many questions about clarity.
justification_for_why_not_higher_score: No reviewer argues for acceptance (except a single "marginally above threshold").
Decision: Reject