SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Thao Nguyen
University of Washington  
thaottn@cs.washington.edu
Samir Yitzhak Gadre
Columbia University sy@cs.columbia.edu
Gabriel Ilharco
University of Washington gamaga@cs.washington.edu
Sewoong Oh
University of Washington, Google Research Ewoong@cs.washington.edu
University of Washington, Allen Institute for Artificial Intelligence schmidt@cs.washington.edu
Massive web datasets play a key role in the success of large vision-language models like CLIP and Flamingo. However, the raw web data is noisy, and existing filtering methods to reduce noise often come at the expense of data diversity. Our work focuses on caption quality as one major source of noise, and studies how generated captions can increase the utility of web-scraped datapoints with nondescript text. Through exploring different mixing strategies for raw and generated captions, we outperform the best filtering method proposed by the DataComp benchmark by  $2\%$  on ImageNet and  $4\%$  on average across 38 tasks, given a candidate pool of 128M image-text pairs. Our best approach is also  $2\times$  better at Flickr and MS-COCO retrieval. We then analyze what makes synthetic captions an effective source of text supervision. In experimenting with different image captioning models, we also demonstrate that the performance of a model on standard image captioning benchmarks (e.g., NoCaps CIDEr) is not a reliable indicator of the utility of the captions it generates for multimodal training. Finally, our experiments with using generated captions at DataComp's large scale (1.28B image-text pairs) offer insights into the limitations of synthetic text, as well as the importance of image curation with increasing training data quantity. The synthetic captions used in our experiments are now available on HuggingFace<sup>1</sup>.
Pre-training large multimodal networks on image-text pairs sourced from the web has become a standard approach to obtaining high performance on vision tasks [3, 37, 24, 40]. However, raw web data can be noisy or uninformative (Figure 1). Most existing data preprocessing efforts revolve around human-defined heuristics based on image and text content separately—e.g., caption length, presence of nouns, sentence complexity, image aspect ratio, minimum image size [8, 46, 10, 47]—or the reliability of the data source [14]. More complex filtering approaches target poorly aligned image-text pairs, by using OpenAI's CLIP models [40] to rank the cosine similarity score between image and text embeddings [46], or ensuring mentions of image objects in the captions [47]. These approaches discard between  $60\%$  to  $90\%$  of the initial data collected, regardless of whether the images themselves are perfectly suitable for training.
In this work, we seek to restore the utility of such discarded examples with the help of synthetic captions, and explore the impact on performance of expanding the training set this way. To do so, we leverage the DataComp benchmark [18], where initial data processing is kept to a minimum
<https://huggingface.co/datasets/thaottn/DataComp_medium_pool_BLIP2_captns, https://huggingface.co/datasets/thaottn/DataComp_large_pool_BLIP2_captns
Original caption: Image Not Found
Generated caption: a green pond with mountains reflected in it
Original caption: Thorbeckestraat
Generated caption: a large brick home with a driveway in front
Original caption: 东湖御院10200000003
Generated caption: living room with a nice looking sofa, coffee table, lamps and other items
Original caption:  
Logo Barbazange Sarl plomberie et installation  
sanitaire Correze 19
Generated caption: a man in a hard hat icon
Figure 1: Raw captions crawled from the web contain significant noise; cosine similarity filtering helps reduce noise but discards many images that are useful for training. Here we show some images that would be filtered out if only the top  $30\%$  examples from the candidate pool with highest image-text cosine similarities are used for training. In these pairs, captions generated by BLIP2 tend to be more faithful to the respective images compared to raw captions obtained from the Internet. In Appendix A, we show 20 other samples drawn completely at random from the discarded pool.
Original caption:  
Italien - Ligurien
Generated caption: house in villa marina, a villa with many rooms and palm trees
Original caption: PE 18
Generated caption: vintage photo of an old train
(i.e. only filtering out NSFW examples and train-test overlap). This allows us to perform controlled experiments on the raw Common Crawl data directly and bypass subjective human-design choices that may be employed in the creation of other datasets (e.g., see LAION-5B [46]). We study several image captioning models and find that recent releases (e.g., BLIP2 [30] and OpenCLIP-CoCa [38]) can generate synthetic captions that improve CLIP training, and lead to a significant boost in zero-shot performance over existing data curation methods. In particular, at the medium scale (128M samples seen), training on the entire candidate pool with synthetic captions is sufficient to outperform common filtering baselines that were done on raw data (e.g., selecting top  $30\%$  examples with highest image-text cosine similarity based on OpenAI's CLIP-ViT/L14). Section 5 describes our experiments with a variety of mixing strategies to combine signals from both raw and synthetic text.
To explain the performance benefits of synthetic captions, we measure caption noise and diversity in various training sets, and demonstrate the significance of both factors in achieving good performance. While existing data filtering methods are effective at reducing noise, they also hurt the diversity of the original training data in the process (e.g., by reducing concept coverage). Synthetic captions help alleviate this drop in diversity by increasing the number of useful captions available for training. In section 6, we analyze various properties of caption data, as well as specific advantages of training with synthetic captions (e.g., improved retrieval capabilities).
Remarkably, our empirical investigation in Section 4 shows that choosing a captioning model to yield competitive downstream performance is non-trivial, as better performance on image captioning benchmarks does not necessarily mean better caption quality for CLIP training. We also note that while this work focuses on the quality of captions used in multimodal training, image quality is another equally important topic of study. As the size of the data pool we experiment with grows exponentially, we start to observe changes in the relative importance of text quality versus image quality in building a good pre-training dataset. We will comment on this in Section 7.
To summarize, our findings serve as a first step towards improving the quality of web-scale datasets via the use of synthetic captions. In the process, we offer insights on several research directions:
- What are the considerations for choosing a captioning model? We find that specializing a pre-trained network towards image captioning via fine-tuning, and optimizing for high CIDEr score on standard benchmarks in general, end up producing captions that are less effective for multimodal training. Reference-free captioning metrics (e.g., CLIP-S [21]) more reliably reflect the training quality of the generated captions.
- How to combine signals from multiple sources of captions? We investigate different strategies for filtering and mixing raw and synthetic captions. This leads to performance gains on DataComp benchmark at small (12.8M pool size), medium (128M pool size) and large (1.28B pool size) scales, compared to existing approaches that utilize only raw data. On ImageNet, the performance benefits diminish with scale. On retrieval tasks, however, the gains are significant across all scales.
- What makes synthetic captions effective? Our analysis of text properties shows that on an individual level, synthetic captions are less noisy and contain more visual information. However, at the population level, synthetic captions are less diverse than raw captions. Consequently, using both sources of captions helps improve the overall caption quality, measured in terms of text diversity as well as image-text alignment.
- How do benefits of synthetic captions scale? Unlike what was found in the original DataComp experiments, given access to generated captions, the best filtering approach differs across scales. Experimenting with data quantities ranging from 12.8M to 1.28B also allows us to observe some limitations of synthetic captions. We posit that image-based filtering, as well as the diversity gap between model-generated and web-scraped captions, play an increasingly important role in large data regimes.
More broadly, our results have important implications for future work as additional progress (captured by the right metric) in image captioning can further enhance the quality of text used for vision-language pre-training. Moreover, the effectiveness of synthetic captions unlocks another massive source of training data: uncaptioned web images from Common Crawl. This can ultimately empower more large-scale multimodal training by improving the availability of properly aligned and sufficiently diverse image-text data.
Synthetic data. Previous work has explored using synthetic data to create new datasets or augment existing ones [15, 41, 36, 25, 56, 19, 12, inter alia]. Closer to our work, He et al. [20], Azizi et al. [5], Bansal and Grover [6] use image generation models to create synthetic images for classification tasks. In the context of CLIP, Santurkar et al. [44] show that a model trained on synthetic captions can outperform a model trained on human-provided captions. The captions were generated procedurally for the 120K images in the MS-COCO training set [11], using multi-object image labels verified by Mechanical Turk workers, which would be difficult to obtain for web-scale datasets like LAION-5B [46] or CommonPool [18] that are about four orders of magnitude larger. Most similar to our work is the LAION-COCO dataset [45], containing 600M image-text pairs from LAION-5B [46] with synthetic captions generated using BLIP [29] and ranked using CLIP models [40, 23]. While [45] heavily filters the raw data pool before generating captions, we work with uncurated web datasets.
Image captioning. Building models able to generate captions from images has been a long-standing subject of research [28, 27, 31, 13, 53, 52, inter alia]. More recently, models like BLIP2 [29, 30], Flamingo [3], and CoCa [55, 38] have made significant progress on this task. Zhu et al. [57] couple large language models with image captioning models to generate more enriched image descriptions. Our work builds on existing image captioning systems to generate synthetic captions for web-crawled training images.
Improving image-text datasets. Given the importance of the pre-training data for multimodal networks [33, 17, 18], several authors have proposed techniques for improving the quality of image-text datasets. [9] filter out samples that contain text regions in the image, and advocate for the benefits of increasing the number of samples given a fixed compute budget. Abbas et al. [1] identify and remove samples that are semantically similar to each other. [39] propose a filtering technique called Complexity, Action, and Text-spotting (CAT), designed to select only informative image-text pairs. Many image-text datasets also have their own preprocessing techniques, often not fully disclosed [40, 24, 37, 10, 14, 46]. All of these filtering approaches are complementary to the use of synthetic captions proposed by this work. Liu et al. [32] introduce TaiSu, a Chinese image-text dataset where an image captioning model is used to generate a supplementary description for the images, which are subsequently translated into Chinese with machine translation. Concurrent to our work, Fan et al. [16] present a form of data augmentation for training CLIP models where the captions are rewritten by a large language model. However, the rewriting process assumes access to some raw text and is not conditioned on the images, which may limit its effectiveness when the original captions are not descriptive (e.g., see Figure 1). In contrast, our work uses image captioning models, which are able to generate relevant captions for images regardless of the original text associated with them. We also work with raw Common Crawl data instead of preprocessed datasets, to study the trade-offs between raw and generated captions in a systematic manner. Finally, Gadre et al. [18] introduces DataComp, a benchmark for designing better pre-training datasets for CLIP, which we use in experiments throughout the paper.
3 Experiment setup
Data. Most of our experiments involve the CommonPool provided by the DataComp benchmark [18]. The data contains image-text pairs sourced from Common Crawl dumps between 2014 and 2022, deduplicated and randomly shuffled. The small, medium and large scales of the benchmark contain 12.8M, 128M and 1.28B candidate pairs respectively. Data preprocessing is kept to a minimum, involving only NSFW filtering, evaluation set dedduplication and face blurring, to allow maximum flexibility for dataset design. We also experiment with LAION-COCO [45] and discuss in Appendix G why it is not ideal for studying how to improve the quality of raw training data.
Captioning models. We experiment with BLIP and BLIP2 [30] using HuggingFace's Transformers framework. Both models were pre-trained on 129M image-text pairs from the web including MS-COCO [11] and LAION-400M [46], in addition to the bootstrapped version of the web data with captions generated by BLIP. We also look at OpenCLIP-CoCa [38, 23], which was trained on 13B samples seen from LAION-2B [46]. For each architecture, we experiment with both the pre-trained model and the one that has been fine-tuned on MS-COCO. Caption generation uses top-K sampling with  $\mathrm{K} = 50$ , minimum caption length 5 and maximum caption length 40.
Training. Given CommonPool data of a particular scale, we generate synthetic captions for the images in the pool using the captioning models described above. Then we train a CLIP model on the resulting image-text datasets, using ViT-B/32 as the image encoder for the small and medium scales, and ViT-B/16 for the large scale. Following DataComp's setup [18], the compute budget, architecture and hyperparameters for each scale are fixed in order to isolate data quality as the main factor influencing performance. Given a candidate pool of  $N$  image-text pairs, the CLIP model is then trained with  $N$  samples seen in total. Refer to Appendix B for more details.
Evaluation. We adopt DataComp's zero-shot evaluation suite, and report both ImageNet accuracy and the average accuracy over 38 classification and retrieval tasks proposed by the benchmark [18]. We also pay particular attention to retrieval performance on Flickr30K [54] and MS-COCO [11]. The retrieval score reported is the average of text-to-image Recall@1 and image-to-text Recall@1.
Unless specified otherwise, in the subsequent sections, "CLIP score filtering" or "top x%" refers to selecting top x% examples from the initial training set, based on the cosine similarity between image and text embeddings output by OpenAI's CLIP ViT-L/14 model [40], and "BLIP2" refers to captions generated by BLIP2, using top-K sampling with softmax temperature = 0.75, which we have found to yield the best downstream performance compared to other sampling temperatures (see Appendix C).
4 Impact of model specialization on captions generated for CLIP training
Captioning model	NoCaps CIDEr [51]	CLIP-S [21]	Cosine similarity	No. of unique trigrams	ImageNet accuracy	Flickr retrieval
BLIP, ViT-L/16 (finetuned)	113.2*	0.698	0.231	2.82 × 10^6	0.207	0.498
BLIP2, ViT-g	80.6	0.737	0.251	2.72 × 10^6	0.281	0.507
BLIP2, ViT-g (finetuned)	119.7*	0.711	0.235	1.97 × 10^6	0.227	0.549
OpenCLIP-CoCa, ViT-L/14	0.354*	0.752	0.260	4.45 × 10^6	0.321	0.395
OpenCLIP-CoCa, ViT-L/14 (finetuned)	106.5*	0.702	0.232	1.81 × 10^6	0.252	0.542
Table 1: CIDEr score does not reliably predict how effective a captioning model is at generating synthetic captions for multimodal pre-training; fine-tuning image captioning models leads to lower ImageNet accuracy when training CLIP on the generated captions. * indicates numbers obtained from the corresponding previous work and from contacting the authors. We fix the architecture and compare captioning models with and without fine-tuning on MS-COCO [11], as sources of text supervision for CLIP. Fine-tuning pre-trained networks on the task of image captioning ends up producing synthetic captions that are worse for training CLIP (see resulting ImageNet accuracy), possibly due to lower text diversity. On the contrary, retrieval performance is higher when using captions generated by fine-tuned models.
Given the abundance of image captioning models to choose from, a natural question to ask is: does performance on standard image captioning benchmarks correlate with how useful the generated captions are as text supervision for CLIP training?
In particular, CIDEr [51], together with other reference-based metrics like SPICE [4] and BLEU-4 [35], has been widely adopted as a yardstick for determining state-of-the-art on image captioning benchmarks [55, 3, 29, 30, 22]. Consequently, previous work [55, 29, 30] also experiment with fine-tuning captioning models on MS-COCO and obtain competitive CIDEr scores on common evaluation sets like NoCaps [2].
We compare the utility of synthetic captions produced by BLIP2 and OpenCLIP-CoCa with and without fine-tuning on MS-COCO, by training CLIP on the generated captions and evaluating the trained model on ImageNet classification and Flickr retrieval (Table 1). Fine-tuned captioning models produce captions that boost the retrieval capabilities of CLIP, but hurts its ImageNet classification performance. We hypothesize that fine-tuning on MS-COCO reduces the diversity of the generated text, as evidenced by the lower number of unique trigrams across 1M caption samples (Table 1). Notably, captioning models that are not fine-tuned have very poor CIDEr scores; going with this metric would have suggested that these models are not suitable for caption generation at all.
While many image captioning metrics like CIDEr, SPICE and BLEU-4 emphasize similarity between generated captions and reference captions provided by humans, prior work has also proposed reference-free metrics—for example, CLIP-S [21], which uses a trained CLIP model to assess the compatibility between an image and the generated caption. We compute CLIP-S for the medium candidate pool with different synthetic captions, and find that it is more reflective of the ImageNet performance trend. Fine-tuned captioning models produce captions that have lower CLIP-S and image-text cosine similarity in general.
Since BLIP2 (no fine-tuning) produces sufficiently good text supervision for CLIP to do well on both ImageNet and Flickr, we use it as the captioning model of choice in subsequent experiments that look at how to combine raw and synthetic captions.
5 Filtering raw and synthetic captions
Here we explore in more detail different ways of filtering and combining raw and generated captions, at the medium scale of DataComp [18]:
- No filtering: we train on the entire, unmodified pool (i.e., 128M samples).
- CLIP score filtering: we select the top  $\mathrm{x}\%$  of examples with highest image-text cosine similarity.
- CLIP score intersect with ImageNet1k clustering: Gadre et al. [18] propose clustering image embeddings and only selecting images whose cluster center is a nearest neighbor to one image from ImageNet1k. The authors then find the intersection between this set of images and those that are in the top  $x\%$  based on CLIP score. This is the best baseline using raw captions on DataComp.
- Combining raw and synthetic captions: we use raw captions for the top  $\mathrm{x}\%$  of examples based on CLIP score. For the remaining images (that would otherwise be filtered out), we generate corresponding BLIP2 captions and add them back to the training pool. We also experiment with filtering these additional image-text pairs with the same cosine similarity threshold set in the first step (i.e., BLIP2 ( $\mathrm{x}\%$ , FILTERED) in Figure 2).
In Appendix D, we describe other baselines we have tried and report how well each approach does with varying cosine similarity thresholds. Figure 2 (left) shows the relative performance of select baselines (the degree of CLIP score filtering has been tuned and only the best accuracy is plotted). We find that the best performance at medium scale, measured by either ImageNet or average accuracy, is achieved by mixing raw and synthetic captions, subject to a cosine similarity threshold. Appendix Figure 9 shows the results for Flickr and MS-COCO retrieval, where including BLIP2 captions in the training pool also offers significant performance benefits.
In the right plot of Figure 2, we compare ImageNet performance at various filtering thresholds for methods that involve only one source of captions and those that involve both. We observe that given image-text pairs filtered with certain cosine similarity threshold (blue line), adding BLIP2 captions for some (red line) or all of the remaining images (green line) always helps. It is worth noting that as we lower the threshold and include more raw captions in the training mix, the performance starts to become lower than using just synthetic captions (orange line). Overall we find that filtering is still a necessary step even in the presence of synthetic captions that are supposedly more relevant to the training images.
Figure 2: At the 128M scale of DataComp, we obtain improvement on ImageNet and average accuracies compared to the best filtering method on raw data, by using a mixture of raw and synthetic captions, selecting only image-text pairs with cosine similarity above a certain threshold. (Left) We visualize how various data filtering strategies perform at medium scale, on ImageNet and across 38 tasks. Including BLIP2 captions in the training data significantly outperforms competitive baselines from DataComp trained on only raw text [18]. (Right) As we vary the percentage of top examples chosen from the pool (based on CLIP score), we see consistent benefits from  $(i)$  using BLIP2 captions for samples that would be discarded otherwise,  $(ii)$  applying the same filtering threshold to new image-text pairs containing BLIP2 captions to keep noise level low. The exact accuracy numbers could be found in Appendix D.
Figure 3: Individual synthetic captions can contain more information (especially visual one) compared to raw captions. We calculate the number of words and the fraction of those being visual tokens in each caption for different training sets. Individual BLIP2 captions tend to yield higher numbers on these two metrics compared to individual web-crawled captions, suggesting that on a caption-per-caption basis, synthetic data may contain richer information.
6 What makes synthetic captions effective?
6.1 Defining caption quality
As seen from sample images in Figure 1, web-scraped text may not contain specific visual information (e.g., "Italien - Ligurien") or may not reflect the content of the image (e.g., "Image Not Found"). We seek to understand how generated captions can help overcome these problems.
Figure 4: Generated captions overall exhibit higher image-text alignment than raw captions; this indicates that the former may be less noisy as a training source. We randomly sample  $1\%$  of the 128M candidate pool and given the same set of images, compare the cosine similarity distribution between raw caption data and BLIP2 caption data. We find that overall BLIP2 captions have much higher image-text cosine similarity (mean similarity 0.251 vs 0.208).