Is a Caption Worth a Thousand Images? A Study on Representation Learning
Authors: Shibani Santurkar
Yann Dubois
Rohan Taori
Percy Liang
Tatsunori Hashimoto
Venue: ICLR 2023 poster (2023)
Keywords: CLIP
transfer learning
contrastive learning
multi-modal

Abstract: The development of CLIP [Radford et al., 2021] has sparked a debate on whether adding language supervision can yield vision models with more transferable representations than traditional image-only methods. Our work studies this question through a carefully controlled comparison of two approaches, in terms of their ability to learn representations that generalize to downstream classification tasks. We find that when the pre-training data meets certain criteria---it is sufficiently large and contains descriptive captions with low variability----image-only methods do not match CLIP's  performance even when they are trained with more image data. However, contrary to what one might expect, there are practical settings in which these criteria are not met, wherein added supervision through captions is actually detrimental.
Motivated by our findings, we devise simple data and algorithmic interventions to improve the transfer performance of CLIP-style models.

Published as a conference paper at ICLR 2023
IS A CAPTION WORTH A THOUSAND IMAGES? A STUDY ON REPRESENTATION LEARNING
Shibani Santurkar, Yann Dubois, Rohan Taori, Percy Liang & Tatsunori Hashimoto Stanford University
(shibani, yanndubs, rtaori, paeerripllie iang, t edu
ABSTRACT
The development of CLIP (Radford et al. 2021) has sparked debate on whether adding language supervision can yield vision models with more transferable rep- resentations than traditional image-only methods. Our work studies this question through a carefuully controlled comparison of two approaches, in terms of their ability learn representations that generalize to downstream classification tasks. We find that when the pre-training data meets certain criteria- it is sufficiently large and contains descriptive captions with low -iaaiilttt--maa---ee ods do not match CLIP's performance even when they are trained with more image data. However, contrary to what one might expect, there are practical settings which these criteria are not met, wherein added supervision through captions is actually detrimental. Motivated by our findings, we devise simple data and algo- rithmic interventions to improve the transfer performance of CLIP-style models.
meth-
INTRODUCTION
Image-based contrastive learning approaches have shown promise building models that generalize beyond the data distributions they are trained on (Wu et al., 2018 He et al.) [020] Chen et al., 2020a Caron et al. 2020) Chen et al. 22020b] Caron et al. (021). l). By leveraging large (unlabelled) data sources via self-supervised training. these models learn representations that transfer to diverse
image classification tasks- more so than their supervised counterparts (Ericsson et al. 2021).
Recently, Radford Radford al. (2021) showed that different approach- contrastive learning with language supervision can yield models (CLIP) with remarkable transfer capabilities. This development has garnered significant interest in the vision and natural language processing communities alike, leading to a debate on the utility of multi-modality in visual representation learning (Zhai et al. 2022 Devillers et al. 2021: Fang et al. 2.22). Our work focuses on specific question within this debate:
Does added language supervision lead more transferable visual representations
than using images alone?
might seem like the answer to this question obvious. After all, CLIP utilized caption infor- mation unavailable to traditional image-based approaches and showed substantial gains over them (Radford et al.) 2021). However, CLIP is drastically different from these approaches in many ways, from training data to fine-grained implementation choices, which makes it difficult isolate the contribution of language supervision (see Section 5). Further, recent studies on CLIP's zero-shot classification and robustness properties cast doubt on whether adding language supervision is al- ways beneficial (Fang et al. 2022). Resolving the aforementioned debate thus requires carefully controlled comparison of the two approaches in which the only difference is the form of supervision.
Our contributions. We devise methodology to assess the utility of language supervision in CLIF from a visual representation learning standpoint. To do so, we recognize that CLIP pre- training and popular image-based methods share the same underlying primitive of contrastive learn- ing. Specifically, Radford et al. (202T)ss approach is strikingly similar to SimCLR (Chen et al.) 2020a). The only irreducible difference between them is whether supervision is provided to the
We use CLIP to refer to models trained with Radford et al. 2021 's approach, their pre-trained model.

---

Published as a conference paper at ICLR 2023
4 ~ 7(x)
X
Image-only supervision (e.g, SimCLR)
Objective: consistency
Image- laage-language supervision (o.g. CLIP)
"chonsfny pecking eadece tiie atzest
-duffel
"chu
Santa Mizze
parked on dtinns
packes on the aide the
marise day, Sante Maria catoelll sonceacy. Maronloca, spass,"
bexide amall
airelosit"
beick buzlding"
rall inthdlinge"
Figure 1: A conceptual view of contrastive image-only and image-language pre-training. Both methods rely on the same self-supervised objective: aligning the representations of positive exam- ples (z, +)) while distinguishing them from negative ones (æ,)) The transformation T(-) used to obtain x4 T(x) (augmented image or caption) encodes the equivalences the model must satisfy.
model via image augmentations or image-caption matching (see Figure J-whcch is precisely the quantity we want study. Thus, we can disentangle the effect of language supervision on visual representations by comparing matched versions of SimCLR and CLIP (trained from scratch). Our focus, in particular, is on how well the learned representations transfer to varied image classification tasks. We find that the picture is nuanced and depends three properties of the pre-training data:
When the scale of the dataset is sufficiently large, CLIP's visual representations indeed transfer better than their matched image-only SimCLR counterparts. In fact, this gap is not bridged by training SimCLR with more (image) data, suggesting that a caption can be worth more than any number of images. However, in the low-data regime, language supervision actually hurts model performance both in and out-of-iistribtion.
2. The descriptiveness (Kreiss al. 202T) of captions- -ieii.nnsiee...... the extent to which they refer to what is contained in Senage-direcee determines how well CLIP models transfer. In fact, we find that a single descriptive image-caption pair (e.g., from COCO (Lin et al. 2014)) worth five less descriptive. uncurated captions (c.g., from YFCC (Thoeee et al. 2016)).
3. The variability of captions (e.g. stylistic or lexical) within a dataset can impair CLIP's performance. We find that modification to standard CLIP mentations by sampling from pool of captions for each image can alleviate this drop.
aug-
can
These properties have inter-twined effects on CLIP's performance: e.g., dataset scale can, to some extent, compensate for less-descriptive and/or varied captions. Guided by our findings, we devise simple datasets interventions that can lead to more-transferrable CLIP models: (i) filtering out low- quality captions with text-based classifier, and (ii) applying data augmentation to captions by paraphrasing them using pre-trained language models.
AN APPLES-TO-APPLES COMPARISON
Prior works have studied image-only and image-language pre-training methods in isolation (Wul
et al. 2018 He et al. 2020 Chen et al. 2020a Caron et al., [020 Chen et al.) 2020b b Chen & He
2021 Caron al. 2021] Radford et al. 2021 and side-by-side (Desai & Johnson 2022 Devillers et al. 2021 Fang et al. 2022). Yet, they provide incomplete (and often contradictory) answers to our motivating question of the value of language supervision relative to using images alone (Section 5). Crucially, this is due to various confounders such as: (i) bespoke algorithmic optimizations within the two methods, and (ii) differing pre-trate pre-training datasets. In this section, we outline a series of steps that we take to mitigate these confounders and compare the two methods equal footing.
2.1 FINDING COMMON GROUND
Our approach for studying the value of language supervision guided by the following insight: CLIP pre-training is strikingly similar the popular image-only SimCLR method (Chen et al.
2

---

Published as a conference paper at ICLR 2023
2020a 2
Both methods rely on the same algorithmic primitive of contrastive learning, which illustrate in Figure ! Specifically, the (CLIP/SimCLR) model is trained cross-entroph based objec- tive, which for given pair (2,2+) of positive examples with with associated negatives N is: (2,s)m(2,2++/r)
== - -lo EmeNu(e4) (3p(sim(2,2))111
where = g(((x)) and 24/n = ('( (x((/a))) (1)
sim is cosine similarity, 0/ are encoders, and g/g' are projection heads. Positive examples 24 are obtained through a transformation of the image z, ie.. 24 ~ T(x)-such as image augmentations CLIP. Observe that this difference in T(.)
(e.g., rotations or crops) in SimCLR and captions between CLIP and SimCLR corresponds exactly to whether the model is trained with language, which is the quantity we want to study. Thus, to isolate the role of added language supervision, we can compare the downstream performance of matched CLIP and SimCLR models. To this end, must take some steps to make their implementatations consistent:
Datasets: Typically, CLIP and SimCLR are trained on different datasets, as the former requires image-caption pairs, while the latter can leverage any image data. To control for the effect of the data distribution, we pre-train both models from scratch on the same data.
Architecture: We use the ResNet-50 (He et al. 2016) architecture the image encoder for both methods, and a Transformer (Vaswani et al.) (017)) as the text encoder in CLIP. We also extensively tune hyperparameters for both methods (Appendix (A.3).
Augmentations:
Both methods apply data augmentations to the image itself at each train- ing step. However, the augmentations used in SimCLR (resi ze, crop, flip, jitter, blur, grayscale) are far more sophisticated than those in CLIP (resize ze and crop). We remove this confounder by using SimCLR augmentations unless otherwise specified.
Transformation stochasticity: The two methods differ in how they obtain 24, not just due to the choice of T(x) but also the generative process itself. In SimCLR x is a new random draw from T(x) every batch, while for CL.PP
single fixed caption. Perfectly matching them requires training CLIP by sampling a fresh caption J. for each image at each iteration. We will refer to this stochastic version of CLIP as CLIPs-
Mismatches. Despite our efforts match CLIP with SimCLR, some inconsistencies remain- partly due to their differing input modalities. In particular, CLIP (and CLIPs):
(i) Processes (x) using text transformer rather than SimCLR's ResNet-50.
(ii) Does not share weights between the encoders processing x and T(x) because they corre- spond to different modalities, unlike SimCLR.
(iii) Uses a linear projection head g/g' instead of SimCLR's MLP, which we allow as Radford et al. 2021) showed that this choice does not affect CLIP's performance.
(iv) Only uses other examples in the batch from the same modality as negatives. Thus CLIP has half the number of negatives compared to SimCLR, which also uses transformed versions of other examples in the batch (i.e. both î and 24) as negatives.
We now assess how the representations learned by our matched CLIP and SimCLR models compare. In particular, measure how well their representations transfer to the downstream tasks from Ko- mbblihh et al. (2019). Akin to (Radford ct al. 202I). 1) we focus on the fixed-feature setting, where we freeze the weights of a given model and then train linear probe using task data (see Appendix A)
2.2 A CASE STUDY
We begin by comparing CLIP and SimCLR models trained on the MS-COCO dataset (Lin et al. 2014) (henceforth referred as COCO), which contains ~ 120K images with multi-object labels. Each image has five human-provide chanical Turk. Annotators were given detailed instructions on how to caption an image such to describe only the important parts of the image and not to use proper names. We use COCO as our
captions, collected post-hoc by Chen et al. (015) using Me-
"Other based methods (He et al. 2020 Chen et al. 2020b Chen & He 2021 Caron et al. 2021
have optimizations that are not present in CL.P.
3

---

Published as a conference paper at ICLR 2023
Table I: Linear probe accuracy for COCO pre-trained models in-distribution and on the transfer
suite from Kornblith et al. 2019) (pT= denotes average transfer accuracy).
COCO
Aircraft
Birdsnap
Ctech101
Ctech256
CIFARIO
CIFARI00
Flowers
Food-101
SUN937
Cars
DID
Pets
PTX
Supervised
90.6
31.6
11.8
65.8 53.7 21.7 74.8 46.7 55.9 63.4 47.1 45.9 44.5 47.2 + 0.2
SimCLR
89.0 40.6
18.5
71.5 58.6 31.5 82.1 57.3 61.7 77.4 58.7 57.3
51.9
56.0 = 0.2
CLIP
88.4
41.4
17.6
73.2
60.4
35.8
83.6 60.8
65.7 80.5 60.9 57.0
50.8
57.5 1 0.1
CLIPs (5)
89.8 46.4 20.0 78.4 65.6 41.5 84.6 62.5 66.7 83.9 65.3 61.2 54.9 61.3 = 0.2
starting point for two reasons. First, we can assess the utility of language supervision in the ideal setting where the captions are of fairly high quality due to the careful curation process. Second, can approximate CLIP by sampling from the available set of five captions per image.
Captions (often) help COCO. In Table 1 compare various COCO pre-trate mre-trained models (supervised, SimCLR, CLIP/CLIPs) in terms of the accuracy of a linear probe on: (i) COCO classi- fication (in distribution), and (ii) transfer tasks. Note that contrast image-only and image-language supervision, the "right" comparison is between SimCLR and CLIPs: they are matched (to the best of our abilities) in terms of dataset, architecture, augmentations and stochasticity. We find that:
As expected, the supervised model outperforms self-supervised ones in distribution, but the trend flips when we consider transfer tasks.
Our matched stochastic variant of CLIP, CLIPs, outperforms SimCLR both in- (~ 1%) and oft-of-diitributiinn (~~ 6%).
Vanilla CLIP (which does not contain stochastic transformations like SimCLR/CLIPS) still performs better than SimCLR on downstream transfer tasks: albeit by a smaller margin (~ 2%). Further, CLIP is worse in-distribution than SimCLR. In Section 3.3 we take closer look at exactly why stochasticity in CLIP has such a drastic impact on performance.
Finally, matching image augmentations (applied to x) turns out to be crucial to assessing the merit of added language supervision. Using typical CLIP augmentations (only resi and crop) during training lowers CLIP's average transfer accuracy by 10% (Appendix Table and SimCLR's by 50%. This highlights the importance of controlling for potential confounders (Section 2.1 to fairly evaluate the utility of added language supervision.
3 THE IMPACT OF PRE-TRAINING DATA
Our analysis of COCO shows that language supervision can beneficial over using images alone. That being said, the datasets that CLIP is typically trained on differ, both in scale and quality, from COCO. For instance, COCO captions were collected post-hoc under controlled settings, which is markedly different from the automatic scraping procedure used to gather data at scale. Thus, we shift our focus to two frequently-usee (Ilharco et al., (02T) CLIP training datasets:
CanceptualCaptions (Sharma et al. 2018) (CC) contains 3.3M images harvested from web, with their AL.--tett attributes as captions. The data was filtered for text quality- e.g., well-formed cap- tions that mention at least one object found via the Google Cloud Vision API. Furthermore, all proper nouns in the captions were hypernymized (c.g., "Justin Timberlake" becomes "pop artist").
Yahoo Flickr Creative Commons (Thomee et al. 2016) (YFCC): This dataset has 99.2M images from Flickr, along with their posted titles as captions with no post-processing.
We henceforth overload notation and use CLIPss to denote: (i) the idealized stochastic version of CLJP. which samples from infinite captions per image, and (fi) our approximation of it with finite set of captions.
4

---

Published as a conference paper at ICLR 2023
transfer accuracy (96)
65
Mean transfer accuracy %
-0
20
55
SimCLR:
CLIP: COCO
SiMCLR: CC
YFCC
100K
80
SinCLR: YOOC CLIP: YFCC
35
COCO
500K
BLIP
2M
30
100 200
500
2000
7000
0.35
0.80 Caption descriptiveness
0.65
0.75
2.00
Pre-training datapoints (K)
Figure 2: (leff) Effect of pre-training dataset size on models' performance. While added language supervision consistently improves transfer accuracy in the medium large data regime over using images alone, is actually detrimental on small corpora. (Due to compute constraints, we train models for fewer epochs (100 instead of 200) on datasets of size 2M.) (right) Relationship between CLIP's transfer performance and the average descriptiveness of dataset captions, measured using pre-trained BLIP caption-scoring model (Li al., 2022). Additionally, we also visualize the performance of CLIP models trained on 100K YFCC/CC samples using "more descriptive" BLIP- generated captions: cf. the orange square/triangle outlined black.
Do captions still help? We start by comparing the transfer performance of CLIP and SimCLR on 100K subsets of COCO/CC/YFCC in Figure left). We observe that SimCLR's transfer capabili- ties do not vary much across pre-training datasets, while CLIP's performance is highly sensitive to them. With 100K samples from CC/YFCC, using CLIP is worse than image-only pre-training via SimCLR-unlin what we see for COCO.
The sensitivity CLIP to pre-training data. Inspecting dataset samples (Figure 3 yields pos- sible explanation for this sensitivity. The three datasets differ not just in scale and image diversity, also the extent to which captions: (i) describe visually salient aspects the image, and (ii) vary across images (e.g., in style and wording). For instance, COCO captions are homogenous and de- scriptive, while YFCC ones vary and are often complementary to the image. We now study effect these dataset properties scale, descriptiveness, and variabilityy have on CLIP's performance.
3.1 SCALE MATTERS
A major appeal of contrastive learning methods is that they can leverage the vast amounts of un- labeled data available on the Internet. Thus, it is natural to ask how different forms of contrastive supervision benefit from added pre-training data. We may expect image-only methods perform worse for smaller datasets as they are less likely to encounter (augmented) images which are similar. We might further expect Imagaelanguage models perform more favorably in this setting since they receive richer supervision. To test whether this is the case, we compare CLIP and SimCLR models trained on datasets of varying sizes: 10-100K samples for COCO, and 100K-2M for CC/YFCC.
Our results in Figure 2 leff) deviate from our earlier expectations. First, beyond a certain point, Sim- CLR's transfer performance improves only marginally with additional data. While surprising, simi- lar effects have been noted previously (Tian et al.) 2021;CC Cole et al.) (022), especially when the data uncurated (e.g.. YFCC) (Tian al., (021). Second, in the low-data regime /<50K/200K/500K for COCO/CC/YFCC), training with language actually hurts the models' transfer performance. In fact, (data) scale seems to be essential to benefit from language supervision. With sufficient data, CLIP outperforms SimCLR on all three datasets. This gap remains even if we train SimCLR with extra data, indicating that captions can be worth more than any number of images.
3.2 THE IMPORTANCE OF DESCRIPTIVE CAPTIONS
Prior work in linguistics and accessibility has drawn a distinction between image "descriptions" and "captions" [Berger & Dibb 2003; Chandler 2007 Hodosh et al.) 2013 Bernardi al. 2016 van
Miltenburg 2020; Kreiss et al.) 2 Dognin et al. [2022] Hutchinson et al.) 2022). In particulan
Bernardi et al. (2016) define descriptions texts that "verbalize what be seen in the image,

---

Published as a conference paper at ICLR 2023
COCO
ConceptualCaptions
YFCC
batkrose MOED tvo sinke under two MLCEOTR,"
"ariolec visa the ceoply efter vietory
"chunb Leco the anertyhere Black and white atesob, stmple drowtnng.
"shallned
"Lowse acood conterine the viav fue chickeard Is toway and coly pace nezzow ao you go sp.
glass vane 519 purtbosces"
22-24 2007 so 2oc Intecnecing Pouket. watch the local 2oad up the bosce befoce
sporta daciliay."
Figure 3: Random samples from the COCO, CC and YFCC datasets (also see Appendix Figure 6
i.e. they refer to the objects, actions, and attributes depicted, mention the scene type, etc.". In contrast, Panofsky 1939) suggest that typical caption "provides personal, cultural, or historical context for the image." " This line of work suggests that COCO captions are more descriptive due to the decontextualization of the image and strict instructions provided to the annotators during the caption generation process (Kreiss et al.] 202I). In contrast, Flickr captions (c.g., in CC/YFCC) tend to contain information that is complementary to the image Alikhani al. (2020) since people tend not to not restate what can already be observed in the photographs they post (Hodosh et al. (013).
Now perform well downstream classification tasks, ideally want model representations that encode salient image objects. Recall that in contrastively-tranned models, the learned representations determined by the transformation T Txx) (captions for CLIP). This suggests hypothesis: pre- training CLIP with descriptive captions will yield more transferrable (vision) representations.
To test this, we need to quantify the descriptiveness of caption. Since doing so precisely is in- feasible, we approximate descriptiveness using a pre-trained caption-scoring model. Specifically, leverage the BLIP model (Li al. 2022) which has shown state-of-the-art performance on image- based text retrieval. We then measure the average score assigned by BLIP to dataset captions matching their corresponding images- see Figure right). rright. As expected based on our earlier subjec- tive assessment well prior work (Hodosh et al.) 2013: Kreiss et al. 2021). we indeed find that the caption descriptiveness of COCO > CCS YFCC (see Appendix B. for a discussion of relevant noisy captions.) Furthermore, data directly correlates with CLIP's transfer performance. In fact, a CLIP model trained on 100K descriptive image-caption pairs from COCO attains performance comparable to one trained 2x and 5x more samples from CC and YFCC respectively.
see that the descriptiveness of captions in the pre-training
To further corroborate hypothesis, we train CLIP CC and YFCC with "more descriptive" cap- tions by re-captioning the images using BLIP (Li et al. 2022). Indeed, we find that CLIP trained on 100K CC/YFCC samples with BLIP captions longer performs worse than its COCO counterpart (see Figure (aannee((rght riihh)).. This indicates that CLIIP's sensitivity to the pre-training corpus is not just an artifact of differing image distributions, but due the presence (or absence) of descriptive captions.
3.3 THE EFFRCT OF INTRA-DATASET VARIATIONS IN CAPTIONS
Image captions (Figure seem to vary in how they describe object (e.g.. "duffel van" or "car") and the parts of the image they focus on (c.g., discussing the "strect" or "brick"). We now study how these lexical and focus variations in captions CLIP's ability to learn meaningful representations.
A simple setting, As a starting point, we investigate this effect on the COCO dataset using syn- thetic captions-construee constructed using the available multi-object labels-whereby we can precisely con- trol the intra-dataset captions variations. In an attempt to simulate the variations we observe in Fig- ure we design the captions to (not) be: (i) consistent: use fixed term or random synonyms to describe an object across the dataset (lexical variations): and (ii) complete: mention all or a random subset of image objects (focus variations). (See Appendix [A.6 for details and Appendix Figure S for examples.) Surprisingly, we find that a CLIP model trained with complete and consistent synthetic COCO captions outperforms a model trained on human- written captions (cf. row in Figure leftt to row 3 in Table Dp. However, dropping these two conditions causes the transfer performance of the model to drop significantly (cf. rows 1, 2, and 4 in Figure (eeft). These findings suggest that variability in dataset captions can have adverse effect on the resulting CLIP models.

---

Published as a conference paper at ICLR 2023
Consistency Completeness Model
COCO
PTx
#Unique I-grams
14-
tet
CLIP
88.8
59.2 +: 0.1
V X
CLIP
88.4
57.7 = 0.2
10C
CLIPs
89.1
59.3 - 0.2
X X
CLIP
88.4
56.6 + 0.2
X X CLIPs
89.3
58.9 + 0.2
COCO
CC
YFCC
Figure 4: (left) CLIP trained synthetic COCO captions (constructed using multi-object labels); effect of caption variability due to inconsistency (use a single term to describe a given object) and incompleteness (describes all image objects) transfer performance. (right) Quantifying the vari- ability of dataset captions based on the number of unique n-grams (n=1,2,3) (Li et al.) [015 Fung
et al. 2020 and the MTLD score (McCarthy & Jarvis 2010) in practical image-caption datasets.
now revisit our stochastic CLIP variant, CLIPs, this simple setting. Intuitively, might expect that sampling from set of diverse captions per image-which cover possibile lexical and stylistic variations- puriationsddurngg during training might alleviate the adverse effeets of caption variability. Indeed, find that for synthetic COCO captions, CLIPs is by caption inconsistency and/or incompleteness. The ~2% improvement of CLIPs over CLIP here mirrors the 3.6% gain for aunann-provide captions (ef. Table I These findings suggest that one of the reasons why stochasticity significantly boosts CLIP's performance is its role in (caption) variance reduction. We also find that CLIPs transfers 2% better when trained
The effect of stochasticity.
affected
numan-provide captions as opposed to synthetic ones (unlike CLIP). This indicates that hwman-wwitte captions do contain useful information that is not present in object labels alone. However, extracting this signal is not straightforward, and may require incorporating multiple captions into CLIP training.
Datasets in practice. We now attempt to characterize caption variability in real- world datasets. Inspired by prior work in natural language processing and linguistics (Appendix A.8). for set of dataset captions, we measure: (i) the total number of unique n-grams (N=1-3) (Li et al. 2015 Fung al., 2020) and (ii) measure of textual lexical diversity (MTLD) (McCarthy & Jarvis 2010). Along both these axes of variability, we see that COCO < CC < YFCC Figure44rightt Thus, aside from the lower descriptiveness of YFCC (and to lesser extent CC) captions, their variability could be the reason why the resulting CLIP models have worse transfer performance (Figure (eeff)). This also explains why scale is essential to benefit from language supervision CC and YFCC. After all, CLIP would need to be trained on more captions to even encounter the same words twice.
How many captions are enough? We saw above that "text data augmentations" via CLIPs could reduce the adverse impacts of caption variability. We now analyze how this effect scales with the number of available captions per image on the CC and YFCC datasets. Here, we use the BLIP captioning model to generate multiple captions per image via nucleus sampling (Holtzman et al.) 2020). This procedure intended to serve as a proxy for the manual caption annotation or automated scraping procedures that might be used for data collection in practice. We observe in Figure (eft, that CLIPs improves as the number of available captions per image increases (plateauing around 10). However, scaling up the overall number of image-caption pairs appears to be far more effective than incorporating more captions per image (at least those obtained via BLIP) from the perspective of improving transfer performance (see Figure right)) Note that the exact trade-offs and their costs are context dependent and vary based on the exact procedure used for caption collection.
4 MAKING EXISTING CAPTIONS WORK
So far, we identified three properties the pre-training data that influence CLIP's transfer perfor- mance: (i) scale, (ii) caption descriptiveness, and (ii) caption variability. Our analysis shows that one way to improve CLIP's performance, especially on uncurated data sources, is to simply pre-train with more data. Alternatively, for fixed data scale, we may be able to obtain better CLIP models we improve what captions describe and how they describe an image. We now focus on the latter and put forth simple dataset interventions to improve transfer capabilities of CLIP-style models.

---

Published as a conference paper at ICLR 2023
# Images(K]
Method Nc
Source
CC (100K)
YFCC (100K)
10
100
220
Mean transfer accuracy (%)
SimCLR 0
55.9 + 0.2
55.5 0.2
CLIP 1
Human
53.1 + 0.2
34.7 + 0.2
CLIP 1
BLIP
53.7 + 0.2
54.8 + 0.2
CLIPs 2
BLIP
56.1 + 0.2
56.9 + 0.2
CLIPs 5
BLIP
57.8 + 0.2
58.8 + 0.2
CLIPs
10
BLIP
59.1 + 0.2
Captions/image
Figure 5: A closer look at CLIPs. (left) Sensitivity of its transfer performance to the number of captions per image (Nc) used during training. For CC and YFCC, we use the BLIP captioning model to generate multiple diverse captions per image. Note: due to compute constraints, we run CC pre-training for a subset of Nc values. (right) Performance trade-offs between pre-training using more image-caption pairs more captions per image on the YFCC dataset.
Data pre-processing: Given the importance of caption descriptiveness, we might consider pre- processing scraped data to select for samples with this property. The CC data collection proce- dure (Sharma et al. 2018) partially demonstrates the effectiveness of this approach, as pre-training CLIP on CC samples leads better transfer performance than comparable number of "raw" YFCC ones. However, due to its reliance on the Google Vision API, this procedure can be quite expensive, with costs scaling with the size of the scraped data. Recent works have taken different approach, using pre- pre-trained image-language models (like CLIP) to filter data (Schuhmann et al. (2021). How- ever, since we are interested in building such models in the first place, we avoid taking this route.
Instead, we focus on understanding how far we can get by simply discarding low quality captions, agnostic to the images. We take inspiration from the filtering pipelines used to build large language models (Brown et al. 2220). Here, raw Internet data is cleaned by selecting samples that are "sim- ilar" known high-quality datasets (e.g., Wikipedia). Taking similar approach, we train linear classifier on bag-of-n-grant sentence embeddings (Joulin et al. 2017) to distinguish validation set CC/YFCC captions from COCO ones. This classifier then used filter CC/YFCC, only retaining samples that are predicted as being COCO-like. This simple procedure does end up selecting for captions that are more focused on objects and their descriptions, as opposed to describing contextual properties such dates or urls Appendix A.9 For given pre-training data budget, moderate gains (~~ 2%) from using this heuristic to filter datasets see Table (left).
Mitigating caption variability: As saw in Section 3.3 models trained with CLIPs less impacted by caption variability. However, typical image-captioning datasets (such as CC and YFCC) only have one caption per image. We thus devise methodology augment these captions by leveraging recent open-source large language models (Wang & Komatsuzaki 2021). Concretely, we provide GPT-J with 4 (caption, paraphrase) pairs in-context (Brown et al. (020)) examples. We then prompt it to paraphrase a given target caption. By sampling from GPT-J, we can obtain multiple (in our case, five) paraphrases for every such caption (examples in Appendix Figure In Table (right), we see that feeding these captions into CLIPs results in considerable performance boost over CLIP (trained with a single caption/image). For instance, for COCO, CLIPs trained on generated captions bridges more than half of the performance gap between vanilla CLIP and CLIPs trained with five numan-provided captions.
RELATED WORK
Representation learning. Building models with general representations that transfer to down- stream tasks has been long-standing goal in ML (Donahue et al. 2014 Razavian et al.) 2014 Chatfield et al. 2014; Agrawal et al.] 20144] Yosinski et al.) 2014). Our work is in line with prior studies aimed characterizing the effect of design choices made during training (Azizpour al.
2013 Huh et al. 2016 Chu et al.] 2016; Kornblith et al.) 2019 Zhai et al., 2011 Locatello et al. 2020 e.g. model architecture, datasets and loss functions, on learned representations.
8

---

Published as a conference paper at ICLR 2023
Table 2: Improving CLIP's transfer performance through simple datasets interventions. (left) Ap- plying simple bag-of-words classifier to identify data subsets with "high quality" captions (results with "descriptive" BLIP captions shown for comparison). (right) Using in-context learning with GPT-J to obtain (five) diverse captions for images (via paraphrasing) and then training CLIPs.
Dataset
Method
Preproc.
PTz
Dataset
Method
Caption
PTs
SimCLR
55.9 0.3
SimCLR
56.0 0.2
cc (100K)
CLIP
53.1 = 0.2
CLIP
Human
57.5 =
COCO (120K)
CLIP
Filter
54.2 0.2
CLIPs
Human
61.3 + 0.2
SimCLR
55.4 0.2
CLIPs
GPT-J
58.9 + 0.3
YFCC (500K)
CLIP
58.8 = 0.2
SimCLR
55.3 + 0.2
CLIP
BLIP
61.8 0.2
CC (200K)
CLIP
Human
57.0 0.3
CLIP
Filter
60.4 0.2
CLIPs
GPT-J
58.8 + 0.3
The utility of language in vision. There is long line of work on leveraging language im-
Srivastava & Salakhutdinov [002 Frome et al.) 2013 Baltrušaitis al., 2018: Guo al. 2019). Recent studies have sought to investigate how integral language is to the performance of such multi-modal models. Fang et al. (2022) study a different property of LLIP--zer-sshl robustness, rather than transfer learning- and show that it is compa- rable to that of a supervised classifier trained on the same YFCC images. Therefore, they conclude that data distribution is more important than language supervision. In concurrent work, (Nguyen et al., (2022) study the sensitivity of CLIP's zero-shot robustness to the pre-training dataset. How- ever, unlike our work, they do not: (i) contrast CLIP against image-only methods trained on the same corpora, and (ii) attempt to explain what properties of the data are responsible for CLIP's sensitivity. Ruan et al. (2022) argue theoretically that the robustness linear probes on CLIPP's representations stems from pretraining with large and diverse set of images and Nomain-agnostic augmentations T(x). Most similar to our work are the studies by (Desai & Johnson 2021 and Devillers et al. 20211. which study the role of language supervision on transfer performance in the context of Vir- Tex (a CLIP precursor) and CLIP respectively. Notably, the two works draw opposite conclusions as the utility of language compared purely image-based approaches. This difference stems from the fact that neither of the works attempt to directly control for a algorithmic, architectural, and data-related confounders. Our work performs a substantially more controlled study on the effect of language supervision, allowing us to make more direct claims than these works.
prove vision models (Quattoni et al.
2007:
6 DISCUSSION
Our work takes step towards resolving the debate as to whether multi-modality, and language in particular, can improve visual representation learning. A comparison of CLIP with matched image- only SimCLR model reveals that neither form of supervision (using images alone or coupled with language) is strictly better than the other. Indeed, there are practical regimes where CLIP's perfor- mance cannot be matched using SimCLR with any amount of image data and others where language supervision is harmful. This is direct consequence of CLIP's sensitivity its pre- pre-training data, especially its scale, descriptiveness, and variability of the captions. Through our analysis, we also discovered algorithmic improvements (CLIPs) and dataset modifications (filtering and augmenting captions) to better take advantage of language supervision.
Limitations. Our exploration allows us to quantify the utility of language supervision (over us- ing images alone) in a specific setting: transfer learning via probing certain object recognition tasks (Kornblith et al. 2019). We view expanding the scope of our analysis as a direction for fu- ture work. Further, despite the significant steps we took to control the differences between CLIP and SimCLR, there are still some inconsistencies that have not been accounted for (discussed in Section 2) Nevertheless, the differences between and previous results (c.g, Desai & Johnson, 2202 Devillers et al., (02T) suggest that we successfully pinned down some crucial confounders (architecture, augmentations, stochasticity, datasets, hyperparameters)

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
Below, we discuss certain ethical concerns pertaining to our work:
Although rely existing open source vision/muutiimodall datasets for analysis, prior work has raised concerns about some of these (or other similarly-soureed ones) being
biased (Stock & Cisse 20177 Yang et al.
2020
Birhane
2002 Paullada et al. 2021
and violating privacy (Prabhu & Birhane 2020
Yang cl al. 2022).
Our focus on understanding the extent to which CLIP's representations are influenced by what the captions they trained describe. However, sidestep whether not this is always desirable. After all, recent studies (Birhane et al. 2021) show that dision-linguisti datasets have various biases and stereotypes, which might not want our models to learn.
In Section 4 we use large language models (in particular, GPT-J) to augment dataset cap- tions via in-context learning. These models however are known to have their own limita- tions that might percolate into the generated captions.
REPRODUCIBILITY STATEMENT
Datasets: All the pre-traiing/transfer learning datasets we use are open-source. In the supplemen- tary material, we include certain variants of the COCO/CC/YFCC datasets created files: namely synthetic COCO captions, filtered CC/YFCC samples, and GPT-J paraphrased captions.
Code and hyperparameters: We discuss implementation details including hyperparameter set- tings in Appendix A We also include the code for training models in the supplementary material.
ACKNOWLEDGEMENTS
We are grateful to Niladri Chatterji, Elisa Kreiss, Nimit Sohoni and Dimitris Tsipras for helpful discussions. SS is supported by Open Philanthropy, YD by a Knights-Henness, Scholarship, and RT by the NSF GRFP under Grant No. DGE 1656518. We also thank Stanford HAI for Google Cloud credits grant.
REFERENCES
Agrawal, Girshick, and Malik. Analyzing the performance of multilayer neural networks for object recognition. In European Conference on Computer Vision (ECCV), 2014.
M. Alikhani, P. Sharma, S. Li, R. Soricut, and M. Stone. Clue: Cross-modal coherence modeling for caption generation. arXiv preprint arXiv:2005.00900 2020.
H. Azizpour, A. S. Razavian, J. Sullivan, A. Maki, and S. Carlsson. Factors of transferability for
generic convnet representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015.
M. M. Baese-Berk, Shiloh Drake, K. Foster, D. Lee, C. Staggs, and J. Wright. Lexical diversity, lexical sophistication, and predictability for speech in multiple listening conditions. Frontiers in psychology, 2021.
T. Baltrušaitis, C. Ahuja, and Morency. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018.
J. Berger and M. Dibb. Ways of seeing. 2003.
R. Bernardi, R. Cakici, D. Elliott, A. Erdem, F. Erdem, N. Ikizler-Cinbis, Keller, A. Muscat, and B. Plank. Automatic description generation from images: A survey of models, datasets, and evaluation measures. Journal of Artificial Intelligence Research, 2016.
A. Birhane, V. U. Prabhu, and E. Kahembwe. Multimodal datasets: misogyny, pornography, and malignant stereotypes. arXiv preprint arXiv:21 1rXiv:2110.01966 10.01963, 2021.
10

---

Published as a conference paper at ICLR 2023
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam,
G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, Clark, C. Berner, S. McCandlish, A. Radford, Sutskever, and D. Amodei. Language models few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Advances Neural Information Processing Systems (NeurIPS), 2020.
M. Caron, H. Touvron, Misra, H. J'egou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
D. Chandler. Semiotics: the basics. Routledge, 2007.
K. Chatfield, K. Simonyan, Vedaldi, and A. Zisserman. Return of the devil in the details: Delving deep into convolutional nets. arXiv preprint arXiv:1405.3531, 2014.
Chen, Kornblith, M. Norouzi, and G. Hinton. simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), 2020a.
Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. Hinton. Big self-supervised models are strong semi-supervised learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020b.
X. Chen and K. He. Exploring simple siamese representation learning. In Conference Computer Vision and Pattern Recognition (CVPR), 2021.
X. Chen, H. Fang, T. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.003325 2015.
B. Chu, V. Madhavan, O. Beijbom, J. Hoffman, and T. Darrell. Best practices for fine-tuning visual classifiers to new domains. In European Conference on Computer Vision (ECCV), 2016.
Elijah Cole, Xuan Yang, Kimberly Wilber, Oisin Mac Aodha, and Serge Belongie. When does contrastive visual representation learning work? In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
K. Desai and J. Johnson. Virtex: Learning visual representations from textual annotations. In Computer Vision and Pattern Recognition (CVPR), 2021.
B. Devillers, B. Choksi, R. Bielawski, and R. VanRullen. Does language help generalization in vision models? In Computational Natural Language Learning. 2021.
P. Dognin, I. Melnyk, Y. Mroueh, I. Padhi, M. Rigotti, J. Ross, Y. Schiff, R. A. Young, and B. Belgo- dere. Image captioning as an assistive technology: Lessons learned from vizwiz 2020 challenge. Journal of Artificial Intelligence Research, 2022.
Donahue, Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell. DeCAF: A deep convolutional activation feature for generic visual recognition. In International Conference Machine Learning (ICML), 2014.
L. Ericsson, H. Gouk, and T. M. Hospedales. How well do self-supervised models transfer? In Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
W. Falcon and the PyTorch Lightning team. Pytorch lightning. 2019. URL "https://githu" com/ Li Lightting-gnnnnnn AI/ Ah/Aiggt/titaaeeeeeee lightning
A. Fang, G. Ilharco, M. Wortsman, Y. Wan, V. Shankar, A. Dave, and L. Schmidt. Data deter- mines distributional robustness in contrastive language image pre-training (clip). arXiv preprint arXiv:2205.01399 2022.

---

Published as a conference paper at ICLR 2023
A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, M. Ranzato, and T. Mikolov. Devise: A deep visual-semantic embedding model. In Advances in Neural Information Processing Systems (NeurIPS), 2013.
Fung. Y. Chen, Z. Lin, and A. Madotto. Deeper conversational ai. In Advances in Neural Infor- mation Processing Systems (NeurIPS) Tutorial, 2020.
W. Guo, J. Wang, and S. Wang. Deep multimodal representation learning: A survey, IEEE Access, 2019.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Computer Vision and Pattern Recognition (CVPR), 2016.
K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual repre- sentation learning. In Conference Computer Vision and Pattern Recognition (CVPR), 2020.
M. Hodosh, P. Young, and J. Hockenmaier. Framing image description a ranking task: Data, models and evaluation metrics. Journal of Artificial Intelligence Research (JAIR), 2013.
A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi. The curious case of neural text degeneration. arXiv preprint arXiv: I 009009971, 2020.
M. Huh, P. Agrawal, and A. A. Efros. What makes imagenet good for transfer learning? arXiv preprint arXiv:1608 000.060088...444.. 2016.
B. Hutchinson, J. Baldridge, and V. Prabhakaran. Underspecification in scene description-to- depiction tasks. arXiv preprint arXiv:2210.05815, 2022.
G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt. Openclip, 2021. URL https:://oi..oo org/
5143773
Jarvis. Capturing the diversity in lexical diversity. Language Learning. 2013.
A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov. Bag tricks for efficient text classification. In European Association for Computational Linguistics (EACI.), 2017.
Korbblith, J. Shlens, and Q. V. Le. Do better imagenet models transfer better? In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
E. Kreiss, N. D. Goodman, and C. Potts. Concadia: Tackling image accessibility with descriptive texts and context. arXiv preprint wxXiv:2104.08.. 2021.
R. Krishna, Y.
Zhu, O. Groth, J. Johnson, Hata K. J. Kravitz, S. Chen, Y. Kalantidis, L. Li, D. Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 2017.
J. Li, M. Galley, C. Brockett, J. Gao, and B. Dolan. A diversity-promotiin objective function for neu- ral conversation models. North American Association for Computational Linguistics (NAACL), 2015.
Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping tanguage-image pre-training for unified vision-language understanding and generation. arXiv preprint arXiv:2201. 12086, 2022.
T. Lin, M. Maire. S. Belongie. J. Hays, P. Perona, D. Ramanan, P. Doll' ar, and C. L. Zitnick. Mi- crosoft coco: Common objects in context. In European Conference on Computer Vision (ECCV), 2014.
Locatello, S. Bauer, M. Lucic, G. R"atsch, S. Gelly, Sch" Sch"olkopf, and O. Bachem. A sober look at the unsupervised learning of disentangled representations and their evaluation. Journal of Machine Learning Research (JMLR), 2020.
McCarthy and S. Jarvis. MTLD, vocd-D, and HD-D: A validation study of sophisticated ap- proaches to lexical diversity assessment. Behavior research methods, 2010.
12

---

Published as a conference paper at ICLR 2023
T. Nguyen, G. Ilharco, M. Wortsman, S. Oh, and L. Schmidt. Quality not quantity: On the interaction between dataset design and robustness of clip. In Advances Neural Information Processing Systems (NeurIPS), 2022.
E. Panofsky. Studies in iconology: Humanistic themes in the study of history. 1939.
A. Paullada, D. Raji, E. M. Bender, E. Denton, and A. Hanna. Data and its (dis) contents: A survey of dataset development and use in machine learning research. Patterns, 2021.
V. U. Prabhu and A. Birhane. Large image datasets: A pyrrhic win for computer vision? arXiv preprint arXiv:2006. 16922,,,0...923, 2020.
A. Quattoni, M. Collins, and T. Darrell. Learning visual representations using images with captions.
In Conference on Computer Vision and Pattern Recognition (CVPR), 2007.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021.
A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson. CNN features off-the-shelf: an astounding baseline for recognition. arXiv preprint 1403: 6382, 2014.
Y. Ruan, Y. Dubois, and C. J. Maddison. Optimal representations for covariate shift. In International Conference on Learning Representations (ICLR), 2022.
C. Schuhmann, R. Vencu, Beaumont, R. Kaczmarczyk, Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki. LAION-400M: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 2021.
Sharma, N. Ding, S. Goodman, and R. Soricut. Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning. In Association for Computational Linguistics (ACL), 2018.
N. Srivastava and R. R. Salakhutdinov. Multimodal learning with deep boltzmann machines. Ad- vances Neural Information Processing Systems (NeurIPS), 2012.
Stock and M. Cisse. Convnets and imagenet beyond accuracy: Understanding mistakes and uncovering biases. arXiv preprint arXiv:1711.1144 2017.
B. Thomee, D. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L. Li. Yfcc100m: The new data in multimedia research. In Communications of the Association for Computing Machinery (ACM), 2016.
Y. Tian, O. J. Henaff, and A. van den Oord. Divide and contrast: Self-supervised learning from uncurated data. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
E. van Miltenburg. On the use of human reference data for evaluating automatic image descriptions. arXiv preprini urXiv:2006.08792. 2020.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. arXiv preprint arXiv:/706.03762, 2017.
B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 billion parameter autoregressive language model. www..o///nnnnooo/ooeeeeeeeeeeeee
jax, 2021.
Z. Wu, Y. Xiong, S. X. Yu, and D. Lin. Unsupervised feature learning via non-parametric instance discrimination. In Conference Computer Vision and Pattern Recognition (CVPR), 2018.
K. Yang, Yau, L. Fei-Fei, Deng, and O. Russakovsky, A study of face obfuscation in imagenet. In International Conference on Machine Learning (ICML), 2022.
L. Yang, K. Qinami, L. Fei-Fei, J. Deng, and O. Russakovsky. Towards fairer datasets: Filtering and balancing the distribution of the people subtree in the imagenet hierarchy. In Conference on fairness, accountability, and transparency (FaceT), 2020.
13

---

Published as a conference paper at ICLR 2023
J. Yosinski, J. Clune, Y. Bengio, and H. Lipson. How transferable features in deep neural networks? Advances in Neural Information Processing Systems (NeurIPS), 2014.
X. Zhai, J. Puigcerver, A. Kolesnikov, P. Ruyssen, C. Riquelme, M. Lucic, J. Djolonga, A. S. Pinto, M. Neumann, A. Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv: D.XX::910000467.,. 2019.
X. Zhai, X. Wang, B. Mustafa, A. Steiner, D. Keysers, A. Kolesnikov, and L. Beyer. LIT: Zero- shot transfer with locked-image text tuning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
14


Reviews:
[{"review_id": "xz2qV7Uu8ST", "reviewer": "Reviewer_G3mZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the transfer learning performance of vision language models (CLIP) in comparison with SimCLR. The primary motivation here is that SimCLR is essentially an image-image analogue of CLIP allowing for fair comparisons in performance while controlling for a variety of confounders. The paper claims three primary advantages of CLIP over VL. In terms of scaling, CLIP learns more transferrable representations when the data size increases. However, the required captions need to be descriptive. The authors leverage BLIP to score caption quality and show that datasets with higher caption quality lead to CLIP outperforming SimCLR as well as models trained with more low quality captions. The thirdly, the paper also provides experimental evidence that while variability in captions adversely affects VL training, this can be somewhat compensated by adding more data through text augmentations. ", "summary_of_the_review": "Overall, this paper is a positive contribution the community with a well structured study comparing the effect of language supervision to that of image based contrastive training. The authors control for several of the confounders that previous studies lack, and provide actionable insights that will be useful for further understanding vision-language models.", "strength_and_weaknesses": "**Strengths**\n1. The paper is well motivated. It is of great importance to study the exact effects of vision-language training versus pure image based training. The idea of using SimCLR as a controlled baseline is inspired.\n2. The experiments have clear hypotheses and support the conclusions presented. Specifically, the use of multiple captions per image as text augmentations shows improvements over standard CLIP training, highlighting the importance of both scale and the variability of captions.\n3. The presented approaches to improve VL training for more transferable representations are simple and intuitive.\n\n**Weaknesses/Queries**\n1. The overall transfer accuracy reported for SimCLR is much lower than that reported in Chen et al's original work. The difference primarily seems to be an effect of the training dataset chosen (Imagenet in the original v/s COCO here). Could the authors comment on this? One suggestion could be to use Imagenet and Imagenet-captions (Fang et al.) to better understand the difference.\n2. Another improvement would be to study the effect of scaling up the architectures with more parameters (Resnet-2x, Resnet-4x) and perhaps with Transformer based architectures to understand if the conclusions hold across a variety of architectures/parameters.\n\nHowever, I understand that these studies are time-consuming, and the paper in its current form is a valuable study in itself.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "tsBuh8RmuMa", "reviewer": "Reviewer_TB29", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a controlled empirical study that compares the capability of representational transfer between unsupervised image-only model (mostly SimCLR) and unsupervised vision and language models (mostly CLIP). The central question that the paper tries to answer is whether unsupervised vision-and-language models that exploit “language information” are richer than image only models.  Controls in the paper are related to experimental settings such as training data, training architectures, transformations. The primary observations emphasise that vision-and-language models are richer with caveats on the type of vision and language parallel data and scale. ", "summary_of_the_review": "The current draft of the paper lacks experimental rigour, for an empirical paper most details are currently hand-wavy. The transfer based claims are perhaps supported for vision only transfer learning benchmarks. ", "strength_and_weaknesses": "Strength: \nThe paper contains an interesting exposition between single modality model  v/s multimodal model and the results confirm that multimodal models are indeed better at representational transfer (in general).\n\nWeakness:\n- The claim in the paper about “transfer of representations” seems general (and tad too strong) however this is only evaluated over vision heavy transfer learning benchmarks. It is not clear if the behaviour would be similar for transfer learning on vision and language related benchmarks.   \n- The salient observations (relating to language - especially descriptiveness and variability) in the paper are perhaps fairly trivial considering the vast amount of similar work from the area of vision and language and semiotics [1, 2, 3, 4, 5 inter alia].\n- In general, the paper lacks empirical rigour, the paper contains a list of experimental interventions, but none are clear (expanding as the next point). \n\n[1] Automatic description generation from images: A survey of models, datasets, and evaluation measures. Bernardi et al. 2016\n\n[2] On the use of human reference data for evaluating automatic image descriptions. van Miltenburg. 2020\n\n[3] Ways of seeing. Berger. 2008 \n\n[4] Semiotics: the basics. Chandler. 2007\n\n[5] Underspecification in Scene Description-to-Depiction Tasks. Hutchinson et al. 2022\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WpSNaF7O28", "reviewer": "Reviewer_Bcho", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper performs a thorough investigation of CLIP-like image representation learning.\n\nFirst, they try to compare the CLIP-like image-language contrastive loss and SimCLR-like image-only contrastive loss in a setting where many confounders are controlled. They conclude that adding language supervision can enable the model to learn more transferable representations than their image-only self-supervision method.\n\nThen, they investigate the effect of data properties on CLIP by training CLIP on three different datasets. They find that language supervision can hurt the model transfer performance in low-data regimes but can be helpful in large-scale settings. Also, \"descriptive\" captions and low variability of captions can improve the model transfer performance.\n\nFinally, based on their findings, they propose to filter non-descriptive captions by training a classifier to detect if a caption is COCO-like and mitigate caption variability by using GPT-J and COCO captions to generate new captions for existing image-caption datasets. They demonstrate the effectiveness of their proposed approach to CLIP.\n \n", "summary_of_the_review": "The paper focuses on a specific question (i.e. does language supervision lead to more transferable representations than using images alone?) and presents a comprehensive analysis to answer this question. While many of their conclusions are interesting and insightful, I do find some issues that can be improved as stated in the weakness part. I am willing to discuss and change my score. ", "strength_and_weaknesses": "Strengths:\n1. The paper performs a relatively systematic study on how image-text contrastive loss is compared with an image-only contrastive loss and what properties of the training data can affect the performance of image-text contrastive loss. By drawing insights from their experiments, they can further demonstrate improvements.\n2. When comparing image-text contrastive and image-only contrastive losses, they do a good job of identifying potential confounders and carefully control them as much as they can, making their conclusions more convincing compared with several previous papers.\n3. Whether and to what extent language supervision can improve visual representation learning is an interesting and important topic. The conclusions of this paper can be helpful and inspire researchers in the future.\n\nWeaknesses:\n1. While image-text and image-only contrastive losses have some similarities, they are not mutually exclusive but can in fact be combined together [1]. Therefore, it would be better to see what are the fine-grained differences between the two paradigms and if combing them together can combine the best of them, instead of merely showing one is better than the other.\n2. They find that the \"descriptiveness\" of caption data can affect the model performance and define the term \"descriptiveness\" as the extent to which they refer to what is contained in an image, which is somewhat vague and it seems that \"descriptiveness\" is an antonym of the commonly used term \"noise\". They use a retrieval system to quantify the \"descriptiveness\" of a caption and find that COCO>CC>YFCC, which seems like they are just measuring the quality of the datasets. Because CC and YFCC are scraped from the web and the image-caption pairs are noisy and not well-aligned, it is normal and well-known that noisy data can lead to bad performance. A more precise definition and a more suitable quantification method are required so that the difference between their conclusion and the well-known fact that \"noisy data can be harmful\" is more clear.\n3. They choose to study the effect of caption variability on COCO, where the human-written captions are not quite diverse as noted in the paper. The Visual Genome dataset [2] can be used because it contains many more captions per image and each caption focuses on a specific part of its image, which can make their conclusions more convincing.\n\n\n\n[1] Mu, Norman, Alexander Kirillov, David Wagner, and Saining Xie. \"Slip: Self-supervision meets language-image pre-training.\" ECCV 2022.\n[2] Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A. and Bernstein, M.S., 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TE3RRdYIMb", "reviewer": "Reviewer_fwXk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the author investigates different popular datasets to explore the influence of the style, and scale of a dataset on vision-language representation pretraining task. The authors find that a dataset with sufficiently large and descriptive captions will be helpful in the vision-language representation learning and the representation learning can be negatively influenced by variability of captions. According to the findings, the author proposed data augmentation methods based on BLIP to boost the pretraining performance of the methods trained with different variants of datasets. ", "summary_of_the_review": "In my view, this paper investigates an interesting topic and provide some preliminary rules of dataset design with some quantitative experiments for the vision-language learning.\nHowever, some of experiments are not so convincing and some of them need to be completed to support the claim better, like more differentiation downstream tasks. \nTherefore, I currently consider it marginally below the acceptance threshold. ", "strength_and_weaknesses": "Strength:\nRaise an interesting topic that how many and what kind of captions are helpful for the vision-language pretraining.\n\nGive out some ideas of what kind of a vision-language dataset may help model to learn better representations.\n\nSufficient experiments on different kinds of setting and scale of image-caption training sets to show the influence of different aspects of captions on the representation learning.\n\nWeakness:\n\nSome phenomena can be further explained for the reader to get better understanding. For example,\nWhy cannot the “more descriptive” BLIP-generated captions beat the performance of COCO dataset?\n\nThe paper evaluated the performance of representation learning only with classification-based transfer performance, while some other tasks can also be used to evaluate the capability of representations, i.e. image-text matching, and image textual grounding. The experiments from multiple different kinds of tasks can better support your claim.\n\nSome experiments can be more completed. i.e. CLIP_s 2 and 10 in the left table of Fig. 5.\n\nThe proposed data augmentation methods are based on BLIP model which is also a supervised trained with large scale datasets. I doubt if the learned knowledge will be leaked during the data augmentation process especially in the image captioning process mentioned in Section 3.3. This can be view as a kind of knowledge distillation of BLIP model which makes it less supportive for the claim.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster