Improving multimodal datasets with image captioning
Authors: Thao Nguyen
Samir Yitzhak Gadre
Gabriel Ilharco
Sewoong Oh
Ludwig Schmidt
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: web datasets
image captioning
synthetic data
CLIP training
TL;DR: We demonstrate the effectiveness of generated captions in increasing the utility of web-scraped image-text pairs with nondescript text, in the context of CLIP pre-training

Abstract: Massive web datasets play a key role in the success of large vision-language models like CLIP and Flamingo. However, the raw web data is noisy, and existing filtering methods to reduce noise often come at the expense of data diversity. Our work focuses on caption quality as one major source of noise, and studies how generated captions can increase the utility of web-scraped datapoints with nondescript text. Through exploring different mixing strategies for raw and generated captions, we outperform the best filtering method proposed by the DataComp benchmark by 2% on ImageNet and 4% on average across 38 tasks, given a candidate pool of 128M image-text pairs. Our best approach is also 2x better at Flickr and MS-COCO retrieval. We then analyze what makes synthetic captions an effective source of text supervision. In experimenting with different image captioning models, we also demonstrate that the performance of a model on standard image captioning benchmarks (e.g., NoCaps CIDEr) is not a reliable indicator of the utility of the captions it generates for multimodal training. Finally, our experiments with using generated captions at DataComp's large scale (1.28B image-text pairs) offer insights into the limitations of synthetic text, as well as the importance of image curation with increasing training data quantity. The synthetic captions used in our experiments are now available on HuggingFace.

Improving multimodal datasets with image captioning
Thao Nguyen University of Washington thaottn cs. washington.edu
Samir Yitzhak Gadre Columbia University sy @y@c.columbia.edu
Gabriel Ilharco
University Washington
gamaga@es wew.@ess.wwaaaia washington.edu
Sewoong Oh University of Washington, Google Research sewoong cs. www.wsiington..d.e
Ludwig Schmidt
University of Washington,
Allen Institute for Artificial Intelligence schmidt cs. washington.edi
Abstract
Massive web datasets play key role in the success of large vision-language models like CLIP and Flamingo. However, the raw web data is noisy, and existing filtering methods to reduce noise often come at the expense of data diversity. Our work focuses on caption quality as one major source of noise, and studies how generated captions can increase the utility of web-scraped datapoints with nondescript text. Through exploring different mixing strategies for raw and generated captions, we outperform the best filtering method proposed by the DataComp benchmark by 2% on ImageNet and 4% on average across 38 tasks, given candidate pool of 128M image-text pairs. Our best approach is also 2x better at Flickr and MS-COCO retrieval. We then analyze what makes synthetic captions an effective source of text supervision. experimenting with different image captioning models, we also demonstrate that the performance of model on standard image captioning benchmarks (e.g., NoCaps CIDEr)
reliable indicator the utility of the captions it generates for multimodal training. Finally, our experiments with using generated captions at DataComp's large scale (1.28B image-text pairs) offer insights into the limitations of synthetic text, as well as the importance of image curation with increasing training data quantity. The synthetic captions used our experiments now available HuggingFacc
Introduction
Pre-training large multimodal networks image-text pairs sourced from the web has become
standard approach to obtaining high performance on vision tasks 37 24 40). However, raw web data can be noisy or uninformative (Figure Most existing data preprocessing efforts revolve around human-defined heuristics based on image and text content separately-e.. e.g., caption length, presence of nouns, sentence complexity, image aspect ratio, minimum image size 8
46
10
47
or the
reliability of the data source [14]. More complex filtering approaches target poorly aligned image-text
pairs, by using OpenAl's CLIP models [40] rank the cosine similarity score between image and text embeddings [46), or ensuring mentions of image objects in the captions
[47]. These approaches
discard between 60% to 90% of the initial data collected, regardless of whether the images themselves
are perfectly suitable for training.
In this work, seek to restore the utility of such discarded examples with the help of synthetic captions, and explore the impact on performance of expanding the training set this way. To do so, we leverage the DataComp benchmark [88, where initial data processing is kept to minimum
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Original caption: Image Not found
Original caption:
Therbeckestraat
Generated caption:
Generated caption:
green pand with mountains reflected in
large brick hame with driveway front
Original caption:
Original caption:
#00000000000000
Logo Barbazange Sarl plomberie installation
sanitaire Coméize 19
Generated caption:
Generated caption:
living room with nice looking sofa, coffee table, lamps and other items
a man
icon
Original caption:
Original caption:
Italien Lgurren
18
Generated caption:
Generated caption:
house in villa marina, villa with many rooms and palm trees
vintage photo of an old train
Figure I: Raw captions crawled from the web contain significant noise; cosine similarity filtering helps reduce noise but discards many images that are useful for training. Here we show some images that would be filtered out if only the top 30% examples from the candidate pool with highest image-texi cosine similarities are used for training. In these pairs, captions generated by BLIP2 tend to be more faithful to the respective images compared
captions obtained from the Internet. In Appendix we show 20 other samples drawn completely at random from the discarded pool.
(i.e. only filtering out NSFW examples and train-test overlap). This allows us to perform controlled experiments on the raw Common Crawl data directly and bypass subjective human-design choices that may be employed in the creation of other datasets (e.g., see LAION-5B [6)). We study several image captioning models and find that recent releases (e.g., BLIP2 30 and OpenCLIP-CoCa (38) can generate synthetic captions that improve CLIP training. and lead to a significant boost in zero- shot performance over existing data curation methods. In particular, at the medium scale (128M samples seen), training on the entire candidate pool with synthetic captions is sufficient to outperform common filtering baselines that were done on raw data (e.g., selecting top 30% examples with highest image-text cosine similarity based on OpenAl's CLIP-ViT/LAA Section [ describes our experiments with a variety of mixing strategies to combine signals from both raw and synthetic text.
To explain the performance benefits of synthetic captions, we measure caption noise and diversity in various training sets, and demonstrate the significance of both factors in achieving good performance. While existing data filtering methods are effective at reducing noise, they also hurt the diversity of the original training data in the process ge.g., by reducing concept coverage). Synthetic captions help alleviate this drop in diversity by increasing the number of useful captions available for training. In section we analyze various properties of caption data, as well as specific advantages of training with synthetic captions (e.g., improved retrieval capabilities).
Remarkably, our empirical investigation in Section 4 shows that choosing a captioning model to yield competitive downstream performance is non-trivial, as better performance on image captioning benchmarks does not necessarily mean better caption quality for CLIP training. We also note that while this work focuses on the the quality of captions used in multimodal training, image quality another equally important topic of study. As the size of the data pool we experiment with grows exponentially, we start to observe changes in the relative importance of text quality versus image quality in building a good pre-training dataset. We will comment on this Section 7
To summarize, our findings serve as a first step towards improving the quality of web-scale datasets via the use of synthetic captions. In the process, we offer insights on several research directions:
What are the considerations for choosing a captioning model? We find that specializing a pre- pre-trained network towards image captioning via fine-tuning, and optimizing for high CIDEr score standard benchmarks in general, end up producing captions that are less effective for multimodal training. Reference-free captioning metrics (e.g., CLIP-S (1) more reliably reflect the training quality of the generated captions.
How to combine signals from multiple sources of captions? We investigate different strategies for filtering and mixing raw and synthetic captions. This leads to performance gains on DataComp benchmark at small (12.8M pool size), medium (128M pool size) and large (1.28B pool size) scales, compared to existing approaches that utilize only raw data. On ImageNet, the performance benefits diminish with scale. On retrieval tasks, however, the gains significant across all scales.

---

What makes synthetic captions effective? Our analysis of text properties shows that on an individual level, synthetic captions are less noisy and contain more visual information. However, at the population level, synthetic captions are less diverse than raw captions. Consequently, using both sources of captions helps improve the overall caption quality, measured in terms of text diversity as well as image-text alignment.
How do benefits of synthetic captions scale? Unlike what found in the original DataComp experiments, given access to generated captions, the best filtering approach differs across scales. Experimenting with data quantities ranging from 12.8M 1.28B also allows us observe some limitations of synthetic captions. We posit that image-based filtering, as well as the diversity gap between model-generated and web-scraped captions, play an increasingly important role in large data regimes.
More broadly, our results have important implications for future work as additional progress (captured by the right metric) in image captioning can further enhance the quality of text used for vision- language pre-training. Moreover, the effectiveness of synthetic captions unlocks another massive source of training data: uncaptioned web images from Common Crawl. This can ultimately empower more large-scale multimodal training by improving the availability of properly aligned and sufficiently diverse image-text data.
2 Related work
Synthetic data. Previous work has explored using synthetic data to create new datasets or augment
19 12 inter alia]. Closer to our work, He et al. [20]. Azizi et al. (5). Bansal and Grover [6 use image generation models create synthetic images for classification tasks. In the context of CLIP, Santurkar et al. 44 show that a model trained on synthetic captions can outperform model trained on human-provided captions. The captions were generated procedurally for the 120K images in the MS-COCO training set [II] using multi-object image labels verified by Mechanical Turk workers, which would be difficult to obtain for web-scale datasets like LAION-SB (46) or CommonPool [18] that are about four orders of magnitude larger. Most similar to our work is the LAION-COCO dataset [45), containing 600M image-text pairs from LAION-5B [46] with synthetic captions generated using BLIP [29] and ranked using CLIP models 40 23 While 45
existing ones [T5]
41
36 25
56
heavily filters the raw data pool before generating captions, we work with uncurated web datasets.
Image captioning. Building models able to generate captions from images has been long-standing 31
subject of research (28 27
[22] inter alia]. More recently, models like BLIP2 29 33), Flamingo 3. and CoCa [55 38 have made significant progress on this task. Zhu et al. [57] couple
1
53
large language models with image captioning models to generate more enriched image descriptions. Our work builds existing image captioning systems to generate synthetic captions for web-crawled training images.
Improving image-text datasets. Given the importance of the pre-training data for multimodal networks [33 17, 18,18, several authors have proposed techniques for improving the quality of image-
text datasets. filter out samples that contain text regions in the image, and advocate for the benefits of increasing the number of samples given fixed compute budget. Abbas et al. [1] identify and remove samples that are semantically similar to each other. (39] propose a filtering technique called Complexity, Action, and Text-spotting (CAT), designed to select only informative image-text pairs. Many image-text datasets also have their own preprocessing techniques, often not fully disclosed 40 24 37 10 14, 46). All of these filtering approaches are complementary to the use of synthetic captions proposed by this work. Liu et al. (32) introduce TaiSu, Chinese image-text dataset where where an image captioning model is used to generate a supplementary description for the images, which are subsequently translated into Chinese with machine translation. Concurrent to our work, Fan et al. [16] present a form of data augmentation for training CLIP models where the captions are rewritten by a large language model. However, the rewriting process assumes access to some raw text and is not conditioned on the images, which may limit its effectiveness when the original captions are not descriptive (e.g., see Figure T]. In contrast, our work uses image captioning models, which are able to generate relevant captions for images regardless of the original text associated with them. We also work with raw Common Crawl data instead of preprocessed datasets, to study the trade-offs between raw and generated captions in systematic manner. Finally, Gadre et al. [18] introduces DataComp, a benchmark for designing better pre-trainn gre-training datasets for CLIP, which we use in experiments throughout the paper.
3

---

3
Experiment setup
Data. Most of our experiments involve the CommonPool provided by the DataComp benchmark
18. 'Tee data contains image-text pairs sourced from Common Crawl dumps between 2014 and 2022, deduplicated and randomly shuffled. The small, medi um and large scales of the benchmark contain
12.8M, 128M and 1.28B candidate pairs respectively. Data preprocessing kept minimum, involving only NSFW filtering, evaluation set deduplication and face blurring, to allow maximum flexibility for dataset design. We also experiment with LAION-COCO [45] and discuss in Appendix why it is not ideal for studying how to improve the quality of raw training data.
Captioning models. We experiment with BLIP and BLIP2 [30] using HuggingFace's Transformers framework. Both models were pre-trained on 129M image-text pairs from the web including MS- COCO 11 and LAION-4000 - [46. in addition to the bootstrapped version of the web data with captions generated by BLIP. We also look at OpenCLIP-CoCa [38 23], which was trained on 13B samples seen from LAION-2B [46). For each architecture, we experiment with both the pre-trained model and the one that has been fine-tuned on MS-COCO. Caption generation uses top-K sampling with K = 50, minimum caption length 5 and maximum caption length 40.
Training. Given CommonPool data of particular scale, generate synthetic captions for the images in the pool using the captioning models described above. Then we train CLIP model on the resulting image-text datasets, using ViT-B/32 as the image encoder for the small and medi um scales, and ViT-B/16 for the large scale. Following DataComp's is setup [18], the compute budget, architecture and hyperparameters for each scale fixed in order isolate data quality as the main factor influencing performance. Given a candidate pool of N image-text pairs, the CLIP model is then trained with N samples seen in total. Refer Appendix B for more details.
Evaluation. We adopt DataComp's zero-shot evaluation suite, and report both ImageNet accuracy and the average accuracy over 38 classification and retrieval tasks proposed by the benchmark (18). We also pay particular attention retrieval performance on Flickr30K (54) and MS-COCO [I]]. The retrieval score reported is the average of text-to-image Recall @ I and image-to-text Recall @
Unless specified otherwise, in the subsequent sections, "CLIP score filtering" or "top x%" refers to selecting top x% examples from the initial training set, based on the cosine similarity between image and text embeddings output by OpenAl's CLIP ViT-L/14 model (40). and "BLIP2" refers to captions generated by BLIP2, using top-K sampling with softmax temperature = 0.75, which we have found to yield the best downstream performance compared other sampling temperatures (see Appendix C
4 Impact of model specialization on captions generated for CLIP training
Captioning model
NoCaps CIDEr [51
CLIP-S
Cosine similarity
No. of unique
ImageNet accuracy
Flickr
21
trigrams
retrieval
BLIP, ViT-L/16 (finetuned)
113.2
0.698
0.231
2.82 x 10
0.207
0.498
BLIP2, ViT-g
80.6
0.737
0.251
2.72 10
0.281
0.507
BLIP2. ViT-g (finetuned)
119.7*
0.711
0.235
1.97 106
0.227
0.549
OpenCLIP-CoCa, ViT-L/14
0.354
0.752
0.260
4.45 x 106
0.321
0.395
OpenCLIP-CoCa, 106.5*
0.702
0.232
1.81 106
0.252
0.542
ViT-L/14 (finetuned)
Table CIDEr score does not reliably predict how effective a captioning model is at generating synthetic captions for multimodal pre-training: fine-tuning image captioning models leads to lower ImageNet accuracy when training CLIP on the generated captions. "indicates numbers obtained from the corresponding previous work and from contacting the authors. We fix the architecture and compare captioning models with and without fine-tuning on MS-COCO [T]] as sources text supervision for CLIP. Fine-tuning pre-trained networks on the task of image captioning ends up producing synthetic captions that are worse for training CLIP (see resulting ImageNet accuracy), possibly due to lower text diversity. On the contrary, retrieval performance is higher when using captions generated by fine-tuned models.
Given the abundance of image captioning models to choose from, a natural question to ask is: does performance on standard image captioning benchmarks correlate with how useful the generated captions are as text supervision for CLIP training?

---

In particular, CIDEr [T], together with other reference- based metrics like SPICE 4 and BLEU-4 [5], has been widely adopted as a yardstick for determining of on image captioning benchmarks [5
3
29
30
22]. Consequently, previous work [55] 29 30 also experiment with
fine-tuning captioning models on MS-COCO and obtain competitive CIDEr scores on common
evaluation sets like NoCaps 2
We compare the utility of synthetic captions produced by BLIP2 and OpenCLIP-CoCa with and without fine-tuning on MS-COCO, by training CLIP on the generated captions and evaluating the trained model on ImageNet classification and Flickr retrieval (Table TT. Fine-tuned captioning models produce captions that boost the retrieval capabilities of CLIP, but hurts its ImageNet classification performance. We hypothesize that fine-tuning on MS-COCO reduces the diversity of the generated text, as evidenced by the lower number of unique trigrams across 1M caption samples (Table [ Notably, captioning models that are not fine-tuned have very poor CIDEr scores; going with this metric would have suggested that these models are not suitable for caption generation at all.
While many image captioning metries like CIDEr, SPICE and BLEU-4 emphasize similarity be- tween generated captions and reference captions provided by humans, prior work has also proposed reference- free metrics-for example, CLIP-S (2I. which uses a trained CLIP model assess the compatibility between an image and the generated caption. We compute CLIP-S for the um candidate pool with different synthetic captions. and find that it more reflective of the ImageNet performance trend. Fine-tuned captioning models produce captions that have lower CLIP-S and image-text cosine similarity in general.
Since BIJP2 (no fine-tuning) produces sufficiently good text supervision for CLIP to do well on both IeageNet and Flickr, we it as the captioning model of choice in subsequent experiments that look at how to combine raw and synthetic captions.
5 Filtering raw and synthetic captions
Here explore in more detail different ways of filtering and combining raw and generated captions, at the medium scale of DataComp [8]
No filtering: we train on the entire, unmodified pool (i.e., 128M samples).
CLIP score filtering: we select the top x% of examples with highest image-text cosine similarity.
CLJP score intersect with ImageNetlk clustering: Gadre et al. 18] propose clustering image embeddings and only selecting images whose cluster center is a nearest neighbor one image from ImageNetlk. The authors then find the intersection between this set of images and those that are in the top x% based on CLIP score. This is the best baseline using raw captions on DataComp.
Combining raw and synthetic captions: we use raw captions for the top x% of examples based on CL.P score. For the remaining images (that would otherwise be filtered out), we generate corresponding BLIP2 captions and add them back to the training pool. We also experiment with filtering these additional image-text pairs with the same cosine similarity threshold set in the first step (i.e., BLIPP (x%, FILTERED) in Figure 2).
In Appendix D we describe other baselines we have tried and report how well each approach does with varying cosine similarity thresholds. Figure 2 (left) shows the relative performance of select baselines (the degree of CLIP score filtering has been tuned and only the best accuracy is plotted). We find that the best performance at medium scale, measured by either ImageNet or average accuracy, is achieved by mixing raw and synthetic captions, subject to cosine similarity threshold. Appendix Figure D shows the results for Flickr and MS-COCO retrieval, where including BLIP2 captions in the training pool also offers significant performance benefits.
In the right plot of Figure 2, we compare ImageNet performance at various filtering thresholds for methods that involve only one source of captions and those that involve both. We observe that given image-raw-ter pairs filtered with certain cosine similarity threshold (blue line), adding BLIP2 captions for some (red line) or all of the remaining images (green line) always helps. It is worth noting that as we lower the threshold and include more raw captions in the training mix, the performance starts to become lower than using just synthetic captions (orange line). Overall we find that filtering is still a necessary step even in the presence of synthetic captions that are supposedly more relevant to the training images.
5

---

38.0
BLIPZ (top B0%) -Raw (50 fitored)
32.5
30.0
37.0
BLIP2 50%)
Row (top 32%)
Average accuracy
36.5
BLIP2 (70%. fiboree)
ImsggeNet/Accurse Accuracy
27.55
36.0
DLIPP (70%)
0LIP2 (op 75%
25.0
OLIP2
intersect NN1
(itoring)
ImaggNell
22.5
54.5
20.0
34.0
Raw
17.5
Top rewnrmmaining BLIP2
Raw (top 30% Intersect IN/1))
15.0
Top rew-remaining BLIP2 (fttered)
3 270
27.5
28.0
28.5
20.0
20.5
30.0
30.5
31.0
31.5
32.0
60 CLIP score fine %
ImageNet accuracy
Figure 2: At the 128M scale of DataComp, obtain improvement on ImageNet and average accuracies compared to the best filtering method selecting only image-text pairs with cosine similarity above certain threshold. (L.fft) visualize how various data filtering strategies perform at medium scale, on ImageNet and across 38 tasks. Including BLIP2 captions in the training data significantly outperforms competitive baselines from DataComp trained on only text [88). (Right) As vary the percentage of top examples chosen from the pool (based CLIP score). we consistent benefits from using BLIP2 captions for samples that would he discarded otherwise, (ti) applying the same filtering threshold to new image-text pairs containing BLIP2 captions to keep noise level low. The exact accuracy numbers could be found in Appendix D
data, using mixture of and synthetic captions,
5,00
330
0.75
Frequency
5.125
Frequency
BLIP2
BLIP2 (top 30%)
2.100
0:25
Raw
Raw (top 30%)
1.500
Raw (top 30%)
(70%, filtered)
Number words
Grounding ratio
Figure 3: Individual synthetic captions can contain more information (especially visual one) compared to raw captions. We calculate the number of words and the fraction of those being visual tokens in each caption for different training Individual BLIP2 captions tend to yield higher numbers these metries compared individual web-crawled captions, suggesting that caption-por iittionneer-apttion basis, synthetic data may contain richer information.
What makes synthetic captions effective?
6.1 Defining caption quality
As seen from sample images in Figure web-scraped text may contain specific visual information (e.g., "Italien Ligurien") or may not reflect the content of the image (e.g., "Image Not Found"). We seek to understand how generated captions help overcome these problems.
Figure 4: Generated captions overall exhibit higher image-text alignment than raw captions; this indicates that the former may be less noisy as training source. We randomly sample 1% of the 128M candidate pool and given the same of images, compare the cosine simi- larity distrihution hetween raw caption data and BI BIIP2 caption data. We find that overall BLIP2 captions have much higher mmage-text cosine similarity (mean similarity 0.251 0.208).
BLIP2 Raw
0.06
0.04
0.02
0.00
0.1
0.0
0.1
0.2
0.3
0.4
0.5
CLIP VIT-L/14 1L/1//11 image-text cosine similerity

---

0.29
Figure 5: Combining raw and synthetic cap- tions subject to a cosine similarity threshold helps reduce noise level while boosting data di- versity, both which are essential for achiev- ing good performance. In this plot, circle size denotes the relative size of the resulting training set. While removing noisy image-tex, pairs, CLIP score filtering also lowers the diversity of the cap- tion set substantially, as measured by the number of unique trigrams in the pool. Adding more useful training data by using BLIP2 captions for filtered out images, while respecting the existing CLIP score threshold, helps overcome this limita- tion and improves the training data quality along both axes.
Average maage-tex similarity
0.28
800-27.3%
raw (top 30%)
more diverse
top SBLP2 (70%, (itred)
30
Acc=31 7%%
0.27
28
0.26
2eee accuras
26
0.25
Inco noise
BLIP (no Siler) 28.1-8.1%
0.24
24
0.23
22
0.22
raw (no (iler) a00=17.6%%
220
0.21
0.20
18
4 No. of unique trigrams in captions (log scale)1e6
To approximate the richness of information conveyed in the text data, for each sample in IM random subset, we measure the number of words, as well as the grounding ratio 49 (i.e., fraction of tokens that describe visual concepts, with the vocabulary defined by MS-COCO), in the corresponding captions. In Figure 3 we observe that synthetic captions and raw captions follow vastly different distributions, with the former generally containing more words (left pane) and more visual tokens (right pane) per sample. Performing CLIP score filtering on raw captions leads to improvements on both of these properties; so does mixing raw and synthetic captions. Regarding the issue of poor image-text alignment, we approximate the alignment using cosine similarity between image and text embeddings from CLIP, and find that web-crawled captions indeed have lower similarities overall compared to model-generated ones (Figure 4
The analyses above measure properties of individual captions. We next aim to capture a single diversity metric over all text in the training set. We again select a random subset, the size of which scales with the training set size, and calculate the number of unique trigrams across all captions in the subset. With this diversity metrie, we find that BLIP2 captions actually lag behind raw captions (Figure 55) Using only the top 30% raw captions (based on CLIP score) is even more detrimental.
We summarize these different aspects of caption quality in a noise versus diversity framework (Figure 5), which also offers some intuition for our best baseline uncovered in Section 5 CLIP score filtering that has been widely adopted in prior work [46] 18 is effective at improving performance on raw data by removing noisy examples (i.e., those with poor image-text alignment). However, this procedure also lowers diversity (note: Figure only provides a measure of text diversity, but image diversity is affected as well). By generating synthetic captions for the images that would be discarded otherwise, and subsequently only using pairs where the cosine similarities still meet the threshold, we manage to keep the overall noise level similarly low, while adding more diversity to the training pool. Progress along both axes enables further performance improvement compared to just filtering raw data.
6.2 Performance analysis
After diving deeper into properties of synthetic captions, we next analyze the training implications of these captions in more detail. We examine two models, one trained using only raw captions and the other using only BLIP2 captions, with both training sets having been filtered with CLIP score for top 30% pairs, and achieving similar performance on ImageNet (27.3% vs 27.5%). Averaged across 38 evaluation tasks, training on generated captions sees a 2.8% improvement. We break down performance difference between the two models on individual tasks (Figure(6) and observe that BLIP2 captions also perform better on ImageNet-derived distribution shifts and text recognition (e.g., MNIST, SVHN). Notably, among the tasks with the biggest performance gains are Flickr and MS-COCO retrieval. We provide similar analysis in Appendix Figure T0, where expanding a filtered training set with additional images and their BLIP2 captions improves performance on 30/38 tasks.
The two models compared above share similar ImageNet accuracy but may not be trained on the same images. Figure 7 we fix the set of training samples to be the top 30% with highest cosine similarity between image and raw text. Replacing the raw captions with BL.IP2 captions increases retrieval performance on Flickr and MS-COCO by more than 1.5x (first two columns of each task). We include retrieval performance of training on the entire pool with BLIP2 captions (generated using
7

---

rensient Urrsed craped deaenni
vondecs
countoor
croed
Hakadde dopeet pobisa dintacce
pmspecortd
amagercnco
diperteed
2014 Cabiian
interatiin 18,
videresie
man
dalitaert
Figure 6: Given similar ImageNet accuracy, training with synthetic captions improves performance on 23 out of 38 tasks compared to training with raw captions, including ImageNet distribution shifts, text recognition and retrieval tasks. We compare performance on each task between training with only BLIP2 captions and training with only raw captions; both datasets have heen filtered with CLIP score to select the top 30% examples. Even though the training yield similar ImageNet accuracy (~~27%), using generated captions leads 2.8% improvement on average accuracy, including minor gains on ImageNet distribution shifts and significant gains on MNIST, SVHN, Flickr and MS-COCO retrieval.
Figure 7: Synthetic captions display clear ad- vantage over captions retrieval tasks. We highlight the superior performance on Flickr and MS- COCO retrieval obtained from training on captions generated by BLIP2 (pre-trained model model that has been fine-tuned MS-COCO), compared to training on raw captions. particular, first two columns of each task represent two models trained on the same set of images (i.c., those whose cosine similarity between image and raw text embeddings in the top 30%), just with different captions. This suggests that substantial gains on retrieval tasks can be obtained solely by using better aligned captions.
0.5
Fow topp 30%0MSLIP2 100% (aaddd]
BLIP2 on Wwingg
BLIP2-imetured tering)
Retrieval score
0.3
0.2
0.1
Flickr
MS-COCO
either the pre-trained or the fine-tuned captioning model), as well as that of training on mixture of raw and BLIP2 captions, demonstrate the consistent gains that synthetic captions have offer.
7 Performance at scale
We next apply select baselines described in Section 5 to wide range of candidate pool sizes, ranging from 12.8M to 1.28B samples. In particular, we examine training on the entire pool with only raw captions only BLIP2 captions, CLIP score filtering, using the intersection of top CLIP score examples and examples that lie in clusters close to the ImageNet train set, as well as mixing raw and synthetic captions- our best baseline from the medium scale. The filtering percentage for each method is tuned on the medium scale candidate pool and then applied to experiments at other scales. Given starting pool of N samples, limit the training budget N steps. Note that the 400M and 1.28B scales use the large training settings from DataComp (see (8)).
We focus ImageNet classification and Flickr retrieval performance (note: MS-COCO training set was included in BLIP2's pre-training data so we have excluded MS-COCO retrieval from this comparison). At larger data quantity regimes, using synthetic captions continues to substantially outperform existing raw-text filtering baselines at retrieval (Figure 8, right plot). On ImageNet, however, adding BLIP2 captions the training mix sees diminishing returns: RAW (TOP 30% INTERSECT IN (N)K) BLIP2 (REMAINING 70%, FILTERED, INTERSECT IN INIK) outperforms existing wate-of-the-art baseline trained on raw data, RAW (TOP 30% INTERSECT IN by 2.5% at 400M scale and 1.2% at 1.28B scale (Figure g left plot).
To give some intuition for this result, offer two candidate hypotheses:

---

60
PMT1)
(1E5). dihwed)
(1095
60
(ne (Pwd
(0%) IP2 (78%) (itwed) intenect
60
01,P9 (rn Bile.
-B (T09S, flawwi interset Patk
ImageeWet wrz m1g)
DL.F2 (eeg 50%1
Flide rebieval boore
BLIP3 (mp-00%)
40
DLIP dop 75% interee INTC
BLIPS 73% Intermeet hrlk)
40
-00
20
20
12,80 40M
128M 400M
1,280
12. BM 40M
120M
400M
1,208
Figure 8: With access to synthetic captions, we find that the best data filtering method ImageNet classification varies with the scale of the candidate pool; when it comes to retrieval, however, using synthetic captions is highly beneficial across all scales, We apply select basclines from Section range of candidate pool sizes, and find that the best method on Flickr retrieval always involves synthetic captions (right plot). On ImageNet (left plot), selecting meaningful images (e.g., those that lie close to the ImageNet train set the embedding space) becomes increasingly important larger scales (see dotted versus striked columns). As the data pool size increases, using BLIP2 captions seems to yield diminishing returns, possibly due to the saturation of text diversity obtained from image captioning models.
As noted in Section 6 both noise level (i.e., image-text alignment) and text diversity are important for performance. Noise level stays about the same across all scales. In contrast, the diversity gap between model-generated text and web-scraped text may become more significant with increasing data quantities. We repeat the caption quality analyses from Section with varying random subset size, and find that when using the number of unique nouns and unique trigrams as proxies for text diversity, synthetic captions exhibit worse scaling trend than raw captions (Appendix Figure 12p
Image quality becomes increasingly important at larger scales:
(i) from 12.8M to 128M scale, training on the entire candidate pool with BLIP2 captions outperforms competitive filtering baselines done on raw data (e.g.. Raw (TOP 30%)). This not the case for larger scales.
(ii) starting from 128M scale, baselines that also curate image content (i.e., intersection of top CLIP score examples and those that lie in clusters close to the ImageNet train set) consistently outperform baselines that involve only CLIP score filtering, using either raw captions or BLIP2 captions.
Exact performance numbers can be found in Appendix Fad Table 3 Overall, we find that given a fixed training budget. making more datapoints useful by carefully replacing noisy raw captions with synthetic captions- i.c., RAW (TOP 30%) + BLIP2 (70%, FILTERED) versus RAw (TOP 30%) still offers significant classification and retrieval performance gains across all scales. However, for synthetic captions to perform competitively on ImageNet at larger data regimes, we need to start paying attention to image content, as well as enhancing the diversity of the generated text.
8 Conclusion
In this work, we demonstrate the effectiveness of synthetic captions in improving caption quality for multimodal training, well certain capabilities of the resulting model (e.g., retrieval). Notably, we find that fine-tuning general purpose models towards the task of image captioning actually makes them less effective producing good captions for CLIP training. Our experiments with various candidate pool sizes, ranging from 12.8M to 1.28B image-text pairs, show that including generated captions in the training data can be highly effective at small and medium scales. However, with larger data quantities, the diversity gap between model model-generated and web-scraped text begin to hinder performance gains, and it becomes increasingly harder to obtain state-of-the ImageNet accuracy by just improving text supervision alone.
Limitations. Our experiments do not involve exhaustive list of image captioning systems currently available. Given captioning model of sufficient capability-iee. it can generate captions for training CLIP to reach a good performance- major theme of our work is understanding how

---

to combine signals from both raw and synthetic captions, well as the differences between these two sources of text. We note that even with improved caption quality, multimodal web datasets may still contain harmful stereotypes, some of which have been extensively discussed in prior work [7]. In Appendix H. we conduct some preliminary investigation on the change in race and gender bias between training on synthetic captions and training on only web-crawled text. Besides, generated captions can introduce new biases that they inherit from the captioning models, and using these captions to train the next generation of models can amplify the biases. The risks from using model outputs to replace human annotations have been studied in a simplified settings in (50
Future work. Our findings motivate a number of interesting future directions. One concrete question is whether combining synthetic caption data from multiple image captioning systems can enhance text diversity. Another direction is proposing new algorithms to combine information from raw and generated captions, beyond what we already studied in Section 5 and Appendix [D] Future work could also explore using text-to-image generation [42 for concepts that underrepresented in existing captions, in order to boost data diversity and close knowledge gaps in the downstream model.
to create synthetic training images
Acknowledgments and Disclosure of Funding
We thank Stability AI for the generous assistance with compute resources. We are grateful Josh Gardner and Simon Kornblith for providing feedback on the manuscript. We also thank Maciej Kilian, Anas Awadalla, Alex Fang, Dhruba Ghosh and Jonathan Hayase for helpful discussions while working on this paper. SYG is supported by NSF Graduate Research Fellowship. This work is supported in part by Open Philanthropy. the Allen Institute for AI, and NSF grants DMS-213401 and CCF-20 CCF-2019844 as part of NSF Institute for Foundations of Machine Learning (IFML).
References
[1] A. Abbas, K. Tirumala, D. Simig, S. Ganguli, and A. S. Morcos. Semdedup: Data-efficient learning at web-scale through semantic deduplication. arXiv preprint arXiv:2303.09540 2023.
[2] H. Agrawal, K. Desai, Y. Wang, X. Chen, R. Jain, M. Johnson, D. Batra, D. Parikh, S. Lee, and P. Anderson. Nocaps: Novel object captioning at scale. In Proceedings of the IEEF/CVF international conference computer vision, pages 8948-8957 2019.
[3] J.BB. Alayrac, Donahue, P. Luc, A. Miech, Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 07223716227777 2022.
[4] P. Anderson, B. Fernando, M. Johnson, and S. Gould. Spice: Semantic propositional image caption evaluation. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Purt V 14, pages 382-398. Springer, 2016.
[5] S. Azizi, S. Korbblith, C. Saharia, M. Norouzi, and D. J. Fleet. Synthetic data from diffusion models improves imagenet classification. arXiv preprint arXiv:2304. 0084666,084 2023.
[6] H. Bansal and A. Grover. Leaving reality to imagination: Robust classification via generated datasets. arXiv preprint 1.xii::2302.0250000 2023.
[7] A. Birhane, V. U. Prabhu, and E. Kahembwe. Multimodal datasets: misogyny, pornography, and malignant stereotypes. arXiv preprint arXivv2110.0196, 2021.
[8] M. Byeon, B. Park, H. Kim, S. Lee, W. Back, and S. Kim. Coyo-700m: Image-text pair dataset. 2022.
https://gt thu.. www.eooooooaaaaaaae http://kaabbaaaooooooooooooo
[9] L. Cao, B. Zhang, C. Chen, Y. Yang. X. Du, W. Zhang, Z. Lu, and Y. Zheng. Less is more: Removing text-regions improves clip training efficiency and robustness. arXiv preprint arXiv:2305.05095 2023.
[10] S. Changpinyo, P. Sharma, N. Ding, and R. Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEFF/VV Conference Computer Vision and Pattern Recognition, pages 3558-3568, 2021.
10

---

[11] X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint 1504. 00300, 2015.
[12] C. M. de Melo, A. Torralba, L. Guibas, J. DiCarlo, R. Chellappa, and J. Hodgins. Next- generation deep learning based on simulators and synthetic data. Trends in cognitive sciences, 2021.
[13] K. Desai and J. Johnson. Virtex: Learning visual representations from textual annotations. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 11162-11173, 2021.
[14] K. Desai, G. Kaul, Z. Aysola, and J. Johnson. Redcaps: Web-curated image-text data created by the people, for the people. arXiv preprini arXiv:21 1.:::2/11./14333 11431, 2021.
[15] A. Dosovitskiy, P. Fischer, E. 1lg, P. Hausser, C. Hazirbas, V. Golkov, P. Van Der Smagt, D. Cremers, and Brox. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 2758-2766, 2015.
[16] L. Fan, D. Krishnan, P. Isola, D. Katabi, and Y. Tian. Improving clip training with language rewrites. arXiv preprint arXiv:2305.2000
2023.
[17] A. Fang. G. Ilharco, M. Wortsman, Y. Wan, V. Shankar, A. Dave, and L. Schmidt. Data determines distributional robustness in contrastive language image pre-training (clip). In International Conference on Machine Learning, pages 6216-6234, PMLR, 2022.
[18) S. Y. Gadre, G. Ilharco, A. Fang, J. Hayase, G. Smyrnis, T. Nguyen, R. Marten, M. Wortsman, D. Ghosh, Zhang, al. Datacomp: In search of the next generation of multimodal datasets.
arXiv preprint 2rXiv:2304.14100 2023.
[19] C. Gan, J. Schwartz, S. Alter, D. Mrowea, M. Schrimpf, J. Traer, J. De Freitas, Kubilius, A. Bhandwaldar, N. Haber, et al. Threedworld: A platform for interactive multi-modal physical simulation. arXiv preprint arXiv:2007.04955: 2020.
[20] R. He. Sun, X. Yu, C. Xue, W. Zhang. Torr. S. Bai. and X. Qi. Is synthetic data from generative models ready for image recognition? arXiv preprint arXiv:2210007574, 2022.
[21] J. Hessel, A. Holtzman, M. Forbes, R. L. Bras, and Y. Choi. Clipscore: A reference-
evaluation metric for image captioning. arXiv preprint arXiv:2104.0877 2021.
[22] X. Hu, Z. Gan, J. Wang, Z. Yang, Z. Liu, Y. Lu, and L. Wang- Scaling up vision-language pre-training for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17980-17989, 2022.
[23] G. Ilharco. M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar,
H. Namkoong. J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt. Openclip, July 2021. URL
https:///oot htt:///ii.or///////// 528155211/22 5228//eenoo.... 5143773
[24] C. Jia, Y. Yang. Y. Xia, Y.-T. Chen, Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, and T. Duerig. Scaling up visual and vision- vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4916 PMIR, 2021. 2021.
[25] J. Johnson, B. Haribaran, L. Van Der Maaten, L. Fei-Fei, C. Lawrence Zitnick, and R. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2901- 2910, 2017.
[26] K. Kärkkäinen and J. Joo. Fairface: Face attribute dataset for balanced race, gender, and age. arXiv preprint arXiv: 1908:0998899333 2019.
[27] A. Karpathy and I., Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
31 3128-3137, 2015.
[28] R. Kiros, R. Salakhutdinov, and R. Zemel. Multimodal neural language models. In International conference on machine learning, pages 595-603. PMLR, 2014.
[29] J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888-12900. PMLR, 2022.
[30] Li, D. Li, Savarese, and Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 2023.
11

---

[3] X. L., Yin, Li, Zhang, X. Hu, L. Zhang, L. Wang. H. Hu, L. Dong, F. Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Eision-ECC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16, pages 121-137. Springer, 2020.
[32] Y. Liu, G. Zhu, B. Zhu, Q. Song, G. Ge, H. Chen, G. Qiao, R. Peng, I.. Wu, and Wang. Taisu: A 166m large-scale high-quality dataset for chinese vision-language pre-training. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 16705-16717. Curran Associates, files
Inc., 2022. httpp:tttt///ttttt /ttttp::/pprrceeeinggss wwwp:innnnnnnnirncnncccoeeoeeeee httpp/errrreeeeepepreeeeeeeeeeee
http www.BBennchhhs.eee pdf
and
[33] T. Nguyen, G. Ilharco, M. Wortsman, S. Oh, and L. Schmidt. Quality not quantity: On the interaction between dataset design and robustness of clip. arXiv preprint arXiv:2208.05511 2022.
[34] A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint wrXXi:2212.107/44 2021.
[35] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318 2002.
[36] X. Peng. B. Usman, N. Kaushik, J. Hoffman, D. Wang, and K. Saenko. Visda: The visual domain adaptation challenge. arXiv preprint 17Xiv1710.069244 2017.
[37] H. Pham, Z. Dai, G. Ghiasi, K. Kawaguchi, H. Liu, A. W. Yu, Yu, Y.-T. Chen, M.-T. Luong,
Y. Wu, et al. Combined scaling for zero-shot transfer learning. arXiv preprint arXiv:2111.10050, 2021.
[38] G. Puccetti, M. Kilian, and R. Beaumont. Training contrastive captioners. URL https
//laion. alaion.ai/blg...... wttgat///aaooaaaaeaeaaeooootgaaa c
[39] F. Radenovic, A. Dubey, A. Kadian, T. Mihaylov, S. Vandenhende, Y. Patel, Y. Wen, V. Ra- manathan, and D. Mahajan. Filtering, distillation, and hard negatives for vision-language pre-training, arXiv preprint arXiv:2301.02280, 2023.
[40] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
[41] S. R. Richter, V. Vineet, S. Roth, and V. Koltun, Playing for data: Ground truth from com- puter games. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 1 11-14, 2016, Proceedings, Part II 14, pages 102-118. Springer, 2016.
[42] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[43] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, F. L.. Denton, K. Ghasemipour, R. Gon- tijo Lopes, B. Karagol Ayan, T. Salimans, al. Photorealistic text-to-image diffusion models with deep language understanding, Advances in Neural Information Processing Systems, 35: 36479-36494, 2022.
[44] S. Santurkar, Y. Dubois, R. Taori, P. Liang, and T. Hashimoto. Is a caption worth a thousand images? controlled study for representation learning. arXiv preprint arXiv:2207.07633, 2022.
[45] C. Schuhmann, A. Köpf, R. Vencu, T. Coombes, and R. Beaumont. Laion coco; 600m synthetic captions from laion2b-en. U http::://tainnnn..... httpa/a//a/l//l//.ccc///oo
[46] C. Schuhmann, R. Beaumont, Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:22 10. 10.08402, 2022.
[47] Sharma, N. Ding, S. Goodman, and R. Sorieut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 2556-2563 2018
12

---

[48] I. Shumailov, Z. Shumaylov, Y. Zhao, Y. Gal, N. Papernot, and R. Anderson. Model dementia: Generated data makes models forget. arXiv preprint arXiv:230S.17491 2023.
[49] H. Tan and M. Bansal. Vokenization: Improving language understanding with contextualized, visual-grounded supervision. arXiv preprini urXiv22010.06775, 2020.
[50] R. Taori and T. B. Hashimoto. Data feedback loops: Model-driven amplification of dataset biases. arXiv preprint w2Xiv:220.003444 2022.
[51] R. Vedantam, C. Lawrence Zitnick, and D. Parikh. Cider: Consensus-basee image description
evaluation. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 4566-4575, 2015.
[52] P. Wang, A. Yang, R. Men, J. Lin, S. Bai, Z. Li, J. Ma, C. Zhou, J. Zhou, and H. Yang. Ofa: Unifying architectures, tasks, and modalities through simple sequence-to-sequeene learning framework. In International Conference on Machine Learning, pages 23318-22222 18-44340. PMLR, 2022.
[53] Z. Wang, J. Yu, A. W. Yu. Z. Dai. Y. Tsvetkov, and Y. Cao. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint arXiv:2 10904, 2021.
[54] Young, A. Lai, M. Hodosh, and J. Hockenmaier. From image descriptions visual denota- tions: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
[55] Yu, Z. Wang, Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.0199 2022.
[56] J. Zheng, J. Zhang, J. Li, R. Tang, S. Gao, and Z. Zhou. Structured3dd: large photo-realistic dataset for structured 3d modeling. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX 16, pages 519-535. Springer, 2020.
[57] D. Zhu, J. Chen, K. Haydarov, X. Shen, W. Zhang, and M. Elhoseiny. Chatgpt asks, blip-
2 answers: Automatic questioning towards enriched visual descriptions. arXiv preprint WxXiv2333.06654 2023.
13

---

A More examples of image-text pairs (no cherry picking)
Raw: 2003 Mercedes-Bees Benzz C240 sedan, Leather, MUST BE
SEEN $$6199
BLIP (finetuned): a couple of cars parked in parking lot with Irees and
BLJP2: 2002 mercedex-benz c-class for sale
BLIP2 (finetuned): blue mercedes benz car parked in
parking lot next to yellow cars
OpenCLIP-CoCA. find used 0 mercedes benz 2 0 base sedan 4 door 2 5 for 2 0 I mercedes benz c
OpenCLIP-CoCa (finetuned): blue mercedes parked on the
side of road.
Raw: Gaziburma Ünal is one of Gespeicherte Orle von Can. BLIP (finetune)): dozens of trays of different types of treats at food stand
BLIP2: some trays of pastries and knife BLIP2 (finetuned): there are many trays filled with.food items
from the store OpenCLIP-CooCe baklava, sweets, pastries
OpenCLIP-CoCA (finetuned): there are trays full of different types of food.
Raw: Open Virgin of Kazan, Wooden Egg with Stand, Blue BLIP (finetuned): a gray and white logo with the words more info
MORE INFO
rectangular shape
BLIP2: a white button with the word more info
BLIP2 (finetuned): more information is shown on a white button with an orange background
OpenCLIP I-CCC- home page button. png
OpenCLIP-Code (finetuned): picture of a close up of a text message
Hello my name is
Raw: 2016.07.01 Nametags with Pronouns Avery 5392_non-
branded
BLIP (finetuned): there are no pictures here to provide caption for
BLIP2: hello, my name is name, my pronouns are pronouns
my pranouns
BLIP2 (finetuned): a blue and white label with a blue and white
text
OpenCLIP-CoCa: 1 15+ hello my name is names promunciations
and meanings
OpenCLIP-CoCs (finetuned): hello my name is, pronouns are.
Raw: Italien Ligurien
BLIP (finetuned): beige colored building with tan accents and palm trees on both sides of walkway
BLIP2: house villa marina, villa with many rooms and palm trees
BLIP2 (finetuned): a park with lots of trees and benches front of a large building
OpenCLIP-CoCa: residence villa maria di san giovanni near the sea in taormina
OpenCLIP-CoCa (finetuned): a picture of a large building with
bunch of palm trees..
14

---

Raw: 3 formas de pedir la mano de tu novia wikiHow
BLIP (finetuned): crates stacked up in a pile on top of each other
BLIP2: the building contains five floors of wooden planks
BLIP2 (finetuned): big pile of wooden planks stacked together
OpenCLIP-CoCS. the cost of wood pallets OpenCLIP-CoCA (finetuned): large pile of wooden pallets mounted to a wall.
Raw: lutz
BLIP (finetuned); blond haired man in black suit looking at camera BLIP2: man sitting on a chair with blank background BLIP2 (finetuned): man sitting in a chair with lapel button in front
OpenCLIP- actor tilda swinton is pictured during press
conference for the film a dangerous method at the 2 0 toronto
film festival
OpenCLIP- (finetuned): person sitting on a chair wearing suit and tie
Raw: Women Personality Creative Christmas Hat Face Expression
Gold Earring Funny Cartoon Ear Stud Jewelry Accessories Gift Hot
BLIP (finetuned): red and gold tone emojt earring
BLIP2: kawaii santa emoticuos en la cabeza
BLIP2 (finetuned): pair of emoji earrings with faces and hats OpenCLIP-CoCa. best christmas gift for her new arrivals emoji earrings christmas emoji earrings
OpenCLIP- (finetuned): a pair of gold earrings with a smiley
face and a christmas hat.
Raw: 10840 SW 126th St photo067
BLIP (finetuned): overview of a large backyard with a swimming pool and patio
BLIP2: 3344 sw 7th st, palm beach
BLIP2 (finetuned): a house with a pool from above, with yard
OpenCLIP-CoCa: home for sale in country club shores west palm beach florida
OpenCLIP-Code (finctuned): aerial image of a pool that has little bit of shade by the side
Raw: image8.JPG
BLIP (finetune)): members of a school play soccer a gymnasium
with crowd
BLIP2: a largg crowd of kids perform during a dance show
BLIP2 (finetuned): a group of young children standing on the basketball court
OpenCLIP-CoCa: kid dressed in white standing in a gym area
OpenCLIP-Codd (finetuned): a group of kids on the gym floor with fans the floor.
15

---

Raw: hair oil
BLIP (finetuned): smiling blonde woman blow drying hair in salon while getting mani
BLIP2: hair stylist using hair spray beauty salon
BLIP2 (finetuned): person is using hairdryer to blow dry
long blonde hair
OpenCLIP-Cord
female hairdresser styling a long blond hair with hairspray a beauty salon concept hair care hair straightening hair color correction
SpenCLIP-CoCA (finetuned): person is spraying a hair dryer on a long blonde hair.
Raw: Women long sleeve shirt 2015 Fashion shirts woman Full Comfortable leisure fashion womens long sleeve tops
Q Ooo10
BLIP (finetuned): the qaoo loading logo is shown above the quoo loading logo
BLIP2: qool0 loading logo on white
Loading
BLIP2 (finetuned): picture of image of phone screen showing a loading sign
OpenCLIP-CoCa, loading 1111120 01 png
OpenCLIP-CoCa (finetuned): light grey font and a dark grey font
with large white background
Raw: Nautica NAPTYRO05
BLIP (finetuned): navitta mens stainless steel bracelet watch with blue dial
BLIP2: nautica men't 's chronograph watch
BLIP2 (finetuned): nautica men's men's chronograph black dial stainless steel bracelet watch
OpenCLIP-CoCa: nautica newport chronograph
0 0
g
OpenCLIP-CoCa (finetuned): black watch is shown with red and blue accents
Raw: Greenberg Weathered Marble Plush Ivory Area Rug BLIP (finctuned): grey rug with a text home on by a ta- ble
BLIP2: a grey area rug on a wooden floor BLIP2 (finetuned): white coffee table with sign saying home on it. is sitting on a cream colored rug
OpenCLIP-CoCk. rugs and carpets in hyderabad buy online at best price in OLenCLIP-CoCa (finetuned): rug is shown in living room with a chair.
Raw: productivity, productivity, productivity
BLIP (finetuned): drivers guide the truck industry
BLIP2: buy and sell truck parts
BLIP2 (finetuned): a white truck with a cover on drives
The Trucking Industry
along highway
OpenCLIP-CoCa: how the trucking industry chang-
ing
OpenCLIP-CoCa (finetuned): there are some trucks on the road
16

---

Raw: Amigas
BLIP (finetund)) crowd of people outside a wedding ceremony near several trees
BLIP2: a wedding ceremony the middle of the street
BLIP2 (finetuned): black and white photograph of a number of women in prom dresses
OpenCLIP-Code CoC:: 20 1 0800 _wedding carlenan 003 3 OpenCLIP-CoCa (finetuned); a group of people hugging and
talking in group
Raw: Autozone
BLIP (finetuned): racing track with line of seats and sky background
BLIP2: photo of grand prix race track, under a blue sky BLIP2 (finetuned); the circuit track is empty, but the sun beams
into the sky
OpenCLIP-CoCa circuit of the americas
OpenCLIP-CoCe (finetuned): red and white pole next
racing track
Raw: Automne hiver enfants manteau et pantalon ensemble ca- puche veste de Ski pantalon garçon fille coupe-vent imperméable en plein air camping randonnée BLIP (finetuned); a man wearing a red and blue jacket and pair of pants and a pair of sneakers
+
BLIP2: the arctic light hooded jacket and pants set BLIP2 (finetuned): the colors of the jacket match the pant color of the jacket
OpenCLIP-CoCa: the arctic light 0 7 children 's clothing sets winter kids ski suits sets windproof waterproof warm jackets coats
pants boys set
OpenCLIP-CoCa (finetuned): child standing their ski wear and a jacket and pants
Raw: 1173x1500 Awesome Adult Coloring Pages Printable Zentangle Design
BLIP (finetuned): chinese dragon coloring pages dragon coloring pages for adults print coloring pages
BLIP2:
dragon coloring pages with large and large dragon
BLIP2 (finetuned): circle with dragon on in the cen- ter OpenCLIP-CoCa: the 5 best chinese dragon drawing ideas on pinterest chinese
OpenCLIP-Codd (finetuned): a chinese dragon looks like a dragon from the movie the karate kid
Raw: Der Lieferumfang
BLIP (finetuned): there several electronics laid out on the table ready to be used
BLIP2: samsung galaxy s10e review quick tour of the samsung galaxy s10e
BLIP2 (finetuned): wireless charging case and remole control, both packaged in the box
OpenCLIP-Codee best wireless chargers for samsung galaxy note 8 s-s 8 and iphone 8 OpenCLIP-CoCa (finetuned); set of various electronic items sitting on a table.
17

---

B Experiment details
Refer to Appendices M and N the DataComp benchmark [18] for training and evaluation details. To summarize, both small and mediun scales use ViT-B/32 as the image encoder for CLIP, in addition to fixing the hyperparameters used for training: learning rate 5e-4, 500 warmup steps, batch size 4096, AdamW optimizer B2 0.98. Large scale training uses the same hyperparameters, but with batch size 8192 and ViT-B/16 as the image encoder.
Using DataComp infrastructure and the AWS EC2 cloud. small model takes 4 A100 hours to train, while medium um requires 40 A100 hours and large utilizes 960 A100 hours. We additionally report both of these
CLIP ViT-L/14 and BLIP2 (OPT 2.7B backbone) inference costs. Recall that models on the DataComp's s large pool to curate the datasets used this paper. For the CLIP model, single A100. For BLIP2, get 75 samples
measure throughput 490 samples per second per second on the same hardware. Hence, for the large pool 1.28B samples, we spend 725 A100 hours computing CLIP features and 4,740 A A10 hours generating BLIP2 captions.
While the annotation cost (i.e., BLIP2 caption and CLIP score generation) is 6x larger than a single training proposed by the DataComp benchmark (which is equivalent going through the entire candidate pool for epoch), this additional cost can be easily amortized with more training epochs the final training set, well as with training different downstream models dataset. For reference, OpenAl trained various CLIP models on the same set of 400M curated image-text pairs; the best performing model was trained 256 GPUs for 2 weeks, totalling about 86,000 GPU hours This scale of training is common among existing large vision models. Future work could explore the option of adaptively allocating compute to CLIP training and synthetic caption annotation given a fixed compute budget.
the improved
C
Temperature ablations
Captioning model
Metric
T=0.5
T=0.75
T=1.0
T=1.5
ImageNet accuracy
0.207
0.212
BLIP (finetunee)
Average accuracy
0.303
0.312
ImageNet accuracy
0.212
0.281
0.280
0.251
BLIP2
Average accuracy
0.300
0.357
0.353
0.332
ImageNet accuracy
0.227
0.234
0.221
BLIP2 (finetuned)
Average accuracy
0.325
0.326
0.311
OpenCLIP-CoCa
ImageNet accuracy
0.306
0.321
0.314
Average accuracy
0.366
0.371
0.370
OpenCLIP-CoCa
ImageNet accuracy
0.252
0.264
0.262
(finetuned)
Average accuracy
0.364
0.374
0.364
Table 2: Performance on ImageNet and averaged across 38 tasks when training on the captions generated by captioning models in Table [ with different softmax temperatures. We find that 7' = 0.75 and T' = 1.0 generally lead good performance for CLIP training.
D More filtering baselines
18

---

50
BLPP2
+BLIP (70%))
Dvo +L (20%. Blwwd)
Howwiees 20%0
50%)
H00 2016)
HOLPSS (10%))
BLETS dep 75%
F2 2016
Rics retrival
40
30
17.5
(sp
Ubee ttop 30%
15.0
Rax 0op 10%)
Rew 2ip 10%
28
29
31
29
30
31
ImageNiel accurascy
ImageNell sccurscy
Figure 9: Retrieval performance on Flickr (left) and AS-COCO (right) versus ImageNet accuracy for select baselines. Similar the findings in Figure we find that using BI.IP2 captions or including them in the training data with raw captions significantly boosts performance.
Baseline
Training size
ImageNet accuracy
Average accuracy
small seale (12.8M candidate pool, 12.8M training steps)
Raw captions (no filtering)
12.8M
0.025
0.132*
BLIP2 captions (no filtering)
12.8M
0.076
0.200
Raw captions (top 30%)
3.8M
0.051
0.173*
BLIP2 captions (top 50%)
6.4M
0,080
0.203
Raw captions (intersect
top 30%)
1.4M*
0.039
0.144*
BLIP2 captions (intersect TNIk and top 75%)
2.4M
0.073
0.192
Raw captions (top 30%) + BLII2 captions (70%, filtered), intersect INIk
2.2M
0.045
0.153
Raw captions (top 30%) BLIP2 captions (70%, filtered)
8.4M
0.076
0.197
medium scale (128M candidate pool, 128M training steps)
Raw captions (no filtering)
128M
0.176
0.258*
BLIP2 captions (no filtering)
128M
0.281
0.357
Top BL.IP captions across temperatures (no filtering)
0.293
0.368
Raw captions (top 30%)
38M
0.273
0,328
BLIP2 captions (top 50%)
64.1M
0.302
0.370
Raw captions (intersect INIk top 30%)
14.0M
0.297
0.328
BLIP2 captions (intersect INIk and top 75%)
23.6M
0,306
0,360
Raw captions (top 30%) + BLIP2 captions (70%, filtered), intersect INIk
22.2M
0,281
Raw captions (top 30%) BLIP2 captions (70%. filtered)
83.6M
0.317
0.368
BLIP2 captions (top 50%) Raw captions (50%, filtered)
75.3M
0.310
0.376
large scale (1.28B candidate pool. 1.28B tranning steps)
Raw captions (no filtering)
1.28B
0.459*
0.437*
BLIP2 captions (no filtering)
1.28B
0.487
Raw captions (top 30%)
384M
0.578
0.529*
BLIP2 captions (lop 50%)
641M
0,526
0.522
Raw captions (intersect INIk and top 30%)
140M#
0.63
0,537*
BLIP2 captions (intersect INIk and top 75%)
237M
0.533
0.527
Raw captions (top 30%) BL.P2 captions (70%, filtered), intersect INIk
222M
0.643
0.549
Raw captions (top 30%) BL.IP2 captions (70%, filtered)
834M
0.598
0.551
Table 3: Performance for select baselines at small, medium and large scales of DataComp. indicates numbers obtained from the original paper [18). Underlined numbers are best pestfoeeformin baselines from the DataComp benchmark. trained only web-crawled captions. Bolded numbers the updated state-of-the- art figures after comparing with baselines involving synthetic captions. general, given fixed training budget, helpful to include more samples the training pool by carefully replacing noisy raw captions with synthetic captions (i.c., Raw (TOP 30%) BL.PP (70%, FILtERED) versus Raw (TOP 30%)). We experiment with many more filtering and mixing methods at the medium scale and report how the performance varies with CLIP filtering threshold, 4
19

---

CLIP score filtering
10%
20%
30%
50%
75%
90%
Cosine similarity threshold
Raw captions
0.295
0.266
0.243
0.203
0.160
0.129
BLIP2 captions
0.315
0.292
0.277
0.251
0.217
0.187
Only raw captions
Training set size
12.8M
2
38.4M
64
96. M
115M
ImageNet accuracy
0.198
0.260*
0.273
0.254*
0.212*
0.188
Average accuracy
0.277*
0.322*
0,328
0.315*
0.285*
0,266*
Only BLIP2 captions
Training set size
12.8M
25.6M
38.5M
64.1M
96.0M
115M
ImageNet accuracy
0.146
0.249
0.275
0.302
0.300
0.293
Average accuracy
0.254
0.333
0.356
0.370
0.365
0.366
Only BLIP2 captions, for top % based cosine similarity of image and text
Training size
12.8M
25.7M
38.4M
64.1M
96.1M
115M
ImageNet accuracy
0.192
0.245
0.261
0.266
0.267
0.276
Average accuracy
0.280
0.330
0.346
0.342
0.349
0.356
Raw captions for top % + BLIP2 captions for the remaining examples
Training set size
128M
128M
128M
128M
128M
128M
ImageNet accuracy
0.286
0.296
0.297
0.286
0.250
0,215
Average accuracy
0.360
0.357
0.365
0.349
0.323
0.293
Raw captions for top % + BLIP2 captions for the remaining examples,
subject the same cosine similarity threshold
Training size
30.5M
59.5M
83.6M
114M
127M
128M
ImageNet accuracy
0.267
0.310
0.317
0.296
0.251
0.212
Average accuracy
0.343
0.372
0.368
0.352
0.313
0.285
BLIP2 captions for top raw captions for the remaining examples,
subject the same cosine similarity threshold
Training set size
17.1M
32.8M
47.7M
75.3M
105M
121M
ImageNet accuracy
0,212
0.272
0.298
0.310
0.298
0,285
Average accuracy
0.305
0.353
0.367
0.376
0.375
0.355
Concatenate raw & BL.IP2 captions for top % BL.IP captions for the remaining examples, subject to the same cosine similarity threshold
Training set size
30.5M
59.5M
83.6M
114M
127M
128M
ImageNet accuracy
0.250
0.287
0.299
0.286
0.269
0.262
Average accuracy
0.336
0.368
0.367
0.359
0.340
0.337
Top % captions top % BI.IP2 captions
Training size
25.6M
51.3M
76.9M
128M
ImageNet accuracy
0.238
0.285
0.297
0.300
Average accuracy
0.318
0.358
0,366
0.356
BLIP2 captions top % intersect with examples from INlk clustering
Training set size
10.0M
16.4M
23.6M
27.
ImageNet accuracy
0.243
0.289
0.306
0.301
Average accuracy
0.310
0.343
0.360
0.344
Table 4: Summary of how various filtering and mixing strategics perform ImageNet and average across 38 evaluation tasks DataComp, given a 128M candidate pool (medium scale). indicates numbers obtained from Gadre et al. [8]. Note that all resulting training sets are trained for a fixed number of steps (128M samples seen) and other training variables (e.g., architecture, hyperparameters) are kept constant. Synthetic captions are generated using pre-trained BLIP2 model with top-K sampling (K 50) and softmax temperature 0.75. We find this scale, approaches that yield the best ImageNet and average accuracies leverage combination raw and synthetic captions.
20

---

E Synthetic caption analysis
124
11
castin Burterre
dndi denand offen pioterce
Interaieen rntoronitiie and
setteitci 2014 St feel
resia
taiow pastii
Figure 10: We find that expanding training set of filtered raw data by using BLIP2 captions for some of the discarded images improves performance on 30 out of evaluation tasks, addition boosting average accuracy by 4%. We compare performance on cach task between training on top 30% examples with raw captions (based on CLIP score) and training on the same set of examples but with the addition of BLIP2 captions for the remaining 70% images, filtered by the same CLIP score threshold. In Table we have shown that adding BLIP2 captions improves ImageNet accuracy by 4.4% and average accuracy by 4%. With this breakdown, we find that performance improvement applies most the tasks the evaluation set, especially retrieval.
We investigate whether there are systematic differences in training with raw and generated text when comes to recognizing certain object categories. To do so, we examine two CLIP models that perform similarly on ImageNet (i.e., +0.2%): one trained on only raw captions and one trained on only generated captions, both training sets have been filtered with CLIP score ranking to select the top 30% image-text pairs. In Figure IT we analyze performance on each ImageNet class, categorized as either 'living' or 'non-living" thing based on where the classname synset located in the WordNet hierarchy. We observe that dlsss classification performance is scattered evenly around the line, indicating that compared web-crawled captions, synthetic captions do not exhibit particular disadvantage either 'living' or 'non-living concepts.
Swing things nan-liing things
Rew expllions cleck
We break down per-class performance on ImageNet, between CLIP model trained on only raw captions and one trained on only synthetic captions with similar overall ImageNet accuracy. We find no systematic trends in the performance of either model when it comes to classifying 'living' or "non-living" things.
Figure 11:
21

---

Performance at Scale
1.8
BLIP2
BUP2
Raw
1.0
Number of unique nouns
Number umiqqe trigams
0.4
10a
10a
10
10'
Number of samples
Number samples
Our simple analyses text propertics suggest that the text diversity provided by synthetic captions may not scale well that of raw captions scraped from the Internet. We measure the number of unique nouns and unique trigrams for random subsets of BLIP2 and raw captions various sizes. We observe that on both metrics, the scaling trend for synthetic captions is worse than that of raw captions. This increasing gap in data diversity may impact the performance benefits can expect to obtain from using synthetic captions, when dealing with larger scale of training data.
Figure 12:
G Experiments with LAION-COCO
Our experiments with synthetic captions are partly inspired by the release of LAION-COCO dataset 445. which used BLIP [29] with various hyperparameter settings to caption LAION-5B data [46], and then selected the top synthetic caption for each image based on the cosine similarity output by OpenAl's CLIPs [40]. We pick random set of 100M samples from LAION-COCO and train this set using DataComp's 's medium scale configuration (i.e., 128M steps), with either only the raw captions only the top BLIP captions that come with the dataset. We find that training BLIP captions significantly lags behind training on raw captions, measured by both ImageNet and average accuracies (Figure B Consequently, natural question is how much of this gap be overcome with progress in image captioning models, e.g. the release of BLIP2.
45
SUF1 captions (no therng)
40
cactioe thwrrg)
ImageNet accuracy
Average accuracy
Dataset
caprions leeing)
35
Raw capdons Ofber (50shold-0225
captions (Rter [in
size
ImageNat Accuracy
300
BLIP
20.5
29.0
104M
BLIP2
29.3
34.6
104M
25
Raw
32.3
34.9
104M
20
BLIP2 (CLIP
26.8
32.4
41M
15
score 2 0.28)
10
Raw (CLIP
27.5
32.2
41M
score 0.28)
ImageNet
Average
Figure 13: BLIIP significantly closes the performance gap between BLIP captions raw captions on LAION-COCO; when controlled for noise level, the performance difference between using BLIP2 and using captions is actually negligible. We use BI_IP2 3D to generate captions for 100M random samples from the LAION-COCO dataset (45), which already come with corresponding BLIP 29 captions. We find that advances in the BLIP model family help synthetic captions close the gap with raw captions, measured by zero-shot performance of CLIP trained on the captions. After applying a cosine similarity threshold of 0.28 to the BLIP2 training pool, just like how LAION data was originally selected, we find that using either raw captions synthetic captions for the resulting of examples makes little difference (hatched columns).
22

---

We proceed to generating BLIP2 captions for the same set of 100M images, using only one configura- tion from the original hyperparameter grid in 445 due to compute constraints. Despite the lack of tuning, the new BLIP2 captions manage to close the previous ImageNet performance gap by 75% and come close the average accuracy obtained from training on raw captions (see table Figure 13). Since raw data in LAION was already filtered with CLIP score threshold of 0.28 during the dataset construction, we next experiment with applying the same filtering to BLIP2 captions, in order to control for noise quality in the caption data. On the resulting 41M images, using BLIP2 captions is about as effective as using raw captions (-0.7% ImageNet accuracy and +0.2% average accuracy).
We note that LAION is considered a curated web dataset, with heavy cosine similarity filtering being one of the preprocessing steps. This in turn leads approximately 90% of the raw data from Common Crawl to be discarded, according to Schuhmann et al. [46]. Since LAION only retains about 10% of the original candidate pool, similar experiments DataComp [8]] have shown that further CLIP these top examples will only hurt performance. In addition, given that the selected raw captions are already relatively clean (measured via image-text cosine similarity), and there is no record of datapoints that were filtered out for further experimentation, we find LAION-COCO to be an unsuitable benchmark for studying the utility of synthetic captions. Our experiments here mainly seek to demonstrate that progress in image captioning models (e.g.. the BLIP model family) can translate to better text supervision for CLIP training that rivals the effectiveness of using raw captions.
score filtering
H Fairness implications of using synthetic captions
We examine zero-shot classification accuracy of predicting and gender from face images in the Fairface dataset (26), for a model trained on only filtered raw captions, one trained on only filtered synthetic captions, and one trained on both. We acknowledge that there are limitations to these evaluations as race and gender should not be considered fixed categories.
With Fairface, we find that using synthetic captions improves the classification performance on the disadvantaged group (e.g. female) significantly, and reduces the performance gap between male and female groups while still boosting the overall performance on all race categories. We leave more extensive study of the fairness implications of using synthetic data (including and beyond gender biases) to future work.
Gender
Model
Race
Black
Latino/ Hispunic
South Asiun
Eastern
Asian
Raw (top 30%)
88.8
90.8
85.3
81.3
Male
BLIP2 (top 30%)
87.2
73.7
77.2
74.9
78.6
72.0
64.0
Raw (top 30%) BLIP2 (70%, filtered)
90.5
75.0
79.7
79.4
81.1
72.4
65.3
Raw (top 30%)
20.3
47.1
35.1
42.0
40.9
44.9
56.8
Female
BLIP2 (top 30%)
36.9
70.8
57.9
67.5
67.4
64.1
78.4
Raw (top 30%) + BLIP2 (70%, filtered)
32.9
74.8
56.5
66.3
67.9
67.8
81.9
Raw (lop 30%)
56.7
68.0
63.2
66.4
66.6
65.1
69.1
Overall
B1.IP2 (top 30%)
62.1
72.3
71.2
68.1
Raw (top 30%) BLIP2 (70%, filtered)
61.7
74.9
68.1
72.9
74.5
70.1
73.6
Table 5: Using synthetic captions in the training mix improves classification performance Fairface for the minority group (i.e. female) across all race categories.
23


Reviews:
[{"review_id": "DHfKxaUG2U", "reviewer": "Reviewer_fLcW", "summary": "", "questions": "", "limitations": "The authors discuss the limitations of their work in a sound manner.", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "This paper has several strengths:\n- This is a well written and clearly explained paper in all aspects\n- The work is a valuable contribution to the field of dataset creation\n- The ablations, analyses, and experiments are performed soundly and are of high quality. \n- Achieving SOTA performance on their chosen benchmarks proves the papers utility. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "X5cUZcbFZo", "reviewer": "Reviewer_cRge", "summary": "", "questions": "", "limitations": "For different tasks or different data sets, the effectiveness of filtering methods is inconsistent, which needs to be further considered and resolved.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. It serves as a first step towards improving the quality of web-scale datasets via the use of synthetic captions\n2. By using generated captions, the authors are able to increase image-text relations while reducing the noise from low-quality captions.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "81bzGuCz2B", "reviewer": "Reviewer_DaAH", "summary": "", "questions": "", "limitations": "See “Opportunities For Improvement”", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper offers a comprehensive analysis of the task from four perspectives:\n\n1. How to select a captioning model (based on which metric.)\n2. How to developing a strategy to combine signals from multiple caption sources.\n3. Investigating what makes synthetic captions effective.\n4. Understanding how the benefits of synthetic captions scale up.\n\nThrough this analysis, several conclusions are reached:\n\n1. Reference-free captioning metrics prove to be more reliable.\n2. A hybrid strategy that combines raw and synthetic caption data can lead to sota performance.\n3. Individual synthetic captions are typically less noisy and provide more visual information. However, at a population level, they tend to be less diverse than raw captions.\n4. The quality control of images, as well as the diversity gap between model-generated and web-scraped captions, become increasingly crucial in large data scenarios.\n\nThe result is insightful for the future work on utilizing synthetic data, especially on synthetic captions for training VLMs.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "ZJXb3eqkSY", "reviewer": "Reviewer_fUUj", "summary": "", "questions": "", "limitations": "The authors have adequately addressed the limitations and potential negative impacts of their work.", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Paper is clearly written and easy to understand.\n- Strong experiment results with a fresh angle. Synthetic captions have been around for a while, but this is one of the few works that attempt to formally study it, and it has done an elegant job.\n- Large-scale experiments that support the claim.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "G9Y9YQ1rWA", "reviewer": "Reviewer_DC33", "summary": "", "questions": "", "limitations": "The discussion of limitations exists, but is somewhat limited. It references a key detail: toxic information and harmful stereotypes in alt-text data, however this is not a core limitation of the approach itself, In fact, this approach may help to limit toxic information produced by pre-training models, as the captioning models are often less likely to produce toxic information (usually because of explicit training or supervision) than the models themselves. \n\nIt would be nice to see a more detailed discussion of the limitations of the method itself. Some potential discussion ideas are: \n- Is there a risk of reducing the dataset coverage by filtering? \n- Does CLIP filtering the data, or using synthetic captions induce (or reinforce) bias in the underlying dataset? (for example, it's well known that captioning models are often gender-impaired, does synthetic caption generation reinforce a gender biases, and cause imbalances in the downstream CLIP predictions?) \n- Does training on synthetic captions limit generalization ability of the pre-trained models, particularly to tasks that are not classification/captioning centric?\n- What does the computational overhead look like? Are there costs to using synthetic data?\n", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "This is an interesting paper which provides an interesting insight into how training CLIP-like models for pre-training can be improved with data filtering. The results are convincing and strong, and show significant benefits over baseline approaches. The paper has a lot of good analysis evaluating some of the hypotheses, and the included details are helpful and insightful. The approach, while lacking some novelty (see related work), is a valuable contribution to the field, as it clearly demonstrates the effect of pre-training on synthetic captions vs. alt-text (or a mix of synthetic and alt-text). The problem of building more efficient datasets from existing collected data is also an interesting area, which may be applicable outside of the field. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The paper has received a consistent set of review comments, including two strong acceptance reviews and three weak acceptance review. most reviewers have praised the substantial contributions made in terms of data collection, ablations, analysis and experiments. In light of these positive assessments, I believe the paper merits an  'accept' recommendation.
Decision: Accept (Poster)