SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 5: Combining raw and synthetic captions subject to a cosine similarity threshold helps reduce noise level while boosting data diversity, both of which are essential for achieving good performance. In this plot, circle size denotes the relative size of the resulting training set. While removing noisy image-text pairs, CLIP score filtering also lowers the diversity of the caption set substantially, as measured by the number of unique trigrams in the pool. Adding more useful training data by using BLIP2 captions for filtered out images, while respecting the existing CLIP score threshold, helps overcome this limitation and improves the training data quality along both axes.
To approximate the richness of information conveyed in the text data, for each sample in a 1M random subset, we measure the number of words, as well as the grounding ratio [49] (i.e., fraction of tokens that describe visual concepts, with the vocabulary defined by MS-COCO), in the corresponding captions. In Figure 3, we observe that synthetic captions and raw captions follow vastly different distributions, with the former generally containing more words (left pane) and more visual tokens (right pane) per sample. Performing CLIP score filtering on raw captions leads to improvements on both of these properties; so does mixing raw and synthetic captions. Regarding the issue of poor image-text alignment, we approximate the alignment using cosine similarity between image and text embeddings from CLIP, and find that web-crawled captions indeed have lower similarities overall compared to model-generated ones (Figure 4).
The analyses above measure properties of individual captions. We next aim to capture a single diversity metric over all text in the training set. We again select a random subset, the size of which scales with the training set size, and calculate the number of unique trigrams across all captions in the subset. With this diversity metric, we find that BLIP2 captions actually lag behind raw captions (Figure 5). Using only the top  $30\%$  raw captions (based on CLIP score) is even more detrimental.
We summarize these different aspects of caption quality in a noise versus diversity framework (Figure 5), which also offers some intuition for our best baseline uncovered in Section 5. CLIP score filtering that has been widely adopted in prior work [46, 18] is effective at improving performance on raw data by removing noisy examples (i.e., those with poor image-text alignment). However, this procedure also lowers diversity (note: Figure 5 only provides a measure of text diversity, but image diversity is affected as well). By generating synthetic captions for the images that would be discarded otherwise, and subsequently only using pairs where the cosine similarities still meet the threshold, we manage to keep the overall noise level similarly low, while adding more diversity to the training pool. Progress along both axes enables further performance improvement compared to just filtering raw data.
6.2 Performance analysis
After diving deeper into properties of synthetic captions, we next analyze the training implications of these captions in more detail. We examine two models, one trained using only raw captions and the other using only BLIP2 captions, with both training sets having been filtered with CLIP score for top  $30\%$  pairs, and achieving similar performance on ImageNet (27.3% vs 27.5%). Averaged across 38 evaluation tasks, training on generated captions sees a  $2.8\%$  improvement. We break down performance difference between the two models on individual tasks (Figure 6), and observe that BLIP2 captions also perform better on ImageNet-derived distribution shifts and text recognition (e.g., MNIST, SVHN). Notably, among the tasks with the biggest performance gains are Flickr and MS-COCO retrieval. We provide a similar analysis in Appendix Figure 10, where expanding a filtered training set with additional images and their BLIP2 captions improves performance on 30/38 tasks.
The two models compared above share similar ImageNet accuracy but may not be trained on the same images. In Figure 7, we fix the set of training samples to be the top  $30\%$  with highest cosine similarity between image and raw text. Replacing the raw captions with BLIP2 captions increases retrieval performance on Flickr and MS-COCO by more than  $1.5 \times$  (first two columns of each task). We include retrieval performance of training on the entire pool with BLIP2 captions (generated using
Figure 6: Given similar ImageNet accuracy, training with synthetic captions improves performance on 23 out of 38 tasks compared to training with raw captions, including ImageNet distribution shifts, text recognition and retrieval tasks. We compare performance on each task between training with only BLIP2 captions and training with only raw captions; both datasets have been filtered with CLIP score to select the top  $30\%$  examples. Even though the two training sets yield similar ImageNet accuracy  $(\sim 27\%)$ , using generated captions leads to  $2.8\%$  improvement on average accuracy, including minor gains on ImageNet distribution shifts and significant gains on MNIST, SVHN, Flickr and MS-COCO retrieval.
Figure 7: Synthetic captions display a clear advantage over raw captions on retrieval tasks. We highlight the superior performance on Flickr and MS-COCO retrieval obtained from training on captions generated by BLIP2 (pre-trained model or model that has been fine-tuned on MS-COCO), compared to training on raw captions. In particular, the first two columns of each task represent two models trained on the same set of images (i.e., those whose cosine similarity between image and raw text embeddings are in the top  $30\%$ ), just with different captions. This suggests that substantial gains on retrieval tasks can be obtained solely by using better aligned captions.
either the pre-trained or the fine-tuned captioning model), as well as that of training on a mixture of raw and BLIP2 captions, to demonstrate the consistent gains that synthetic captions have to offer.
7 Performance at scale
We next apply select baselines described in Section 5 to a wide range of candidate pool sizes, ranging from 12.8M to 1.28B samples. In particular, we examine training on the entire pool with only raw captions or only BLIP2 captions, CLIP score filtering, using the intersection of top CLIP score examples and examples that lie in clusters close to the ImageNet train set, as well as mixing raw and synthetic captions—our best baseline from the medium scale. The filtering percentage for each method is tuned on the medium scale candidate pool and then applied to experiments at other scales. Given a starting pool of  $N$  samples, we limit the training budget to  $N$  steps. Note that the 400M and 1.28B scales use the large training settings from DataComp (see [18]).
We focus on ImageNet classification and Flickr retrieval performance (note: MS-COCO training set was included in BLIP2's pre-training data so we have excluded MS-COCO retrieval from this comparison). At larger data quantity regimes, using synthetic captions continues to substantially outperform existing raw-text filtering baselines at retrieval (Figure 8, right plot). On ImageNet, however, adding BLIP2 captions to the training mix sees diminishing returns: RAW (TOP  $30\%$  INTERSECT IN1K) + BLIP2 (REMAINING  $70\%$ , FILTERED, INTERSECT IN1K) outperforms existing state-of-the-art baseline trained on raw data, RAW (TOP  $30\%$  INTERSECT IN1K), by  $2.5\%$  at 400M scale and  $1.2\%$  at 1.28B scale (Figure 8, left plot).
To give some intuition for this result, we offer two candidate hypotheses:
Figure 8: With access to synthetic captions, we find that the best data filtering method for ImageNet classification varies with the scale of the candidate pool; when it comes to retrieval, however, using synthetic captions is highly beneficial across all scales. We apply select baselines from Section 5 to a range of candidate pool sizes, and find that the best method on Flickr retrieval always involves synthetic captions (right plot). On ImageNet (left plot), selecting meaningful images (e.g., those that lie close to the ImageNet train set in the embedding space) becomes increasingly important at larger scales (see dotted versus striked columns). As the data pool size increases, using BLIP2 captions seems to yield diminishing returns, possibly due to the saturation of text diversity obtained from image captioning models.
- As noted in Section 6, both noise level (i.e., image-text alignment) and text diversity are important for performance. Noise level stays about the same across all scales. In contrast, the diversity gap between model-generated text and web-scraped text may become more significant with increasing data quantities. We repeat the caption quality analyses from Section 6 with varying random subset size, and find that when using the number of unique nouns and unique trigrams as proxies for text diversity, synthetic captions exhibit a worse scaling trend than raw captions (Appendix Figure 12).
- Image quality becomes increasingly important at larger scales:
(i) from 12.8M to 128M scale, training on the entire candidate pool with BLIP2 captions outperforms competitive filtering baselines done on raw data (e.g., RAW (TOP  $30\%$ )). This is not the case for larger scales.
(ii) starting from 128M scale, baselines that also curate image content (i.e., intersection of top CLIP score examples and those that lie in clusters close to the ImageNet train set) consistently outperform baselines that involve only CLIP score filtering, using either raw captions or BLIP2 captions.
Exact performance numbers can be found in Appendix F and Table 3. Overall, we find that given a fixed training budget, making more datapoints useful by carefully replacing noisy raw captions with synthetic captions—i.e., RAW (TOP  $30\%$  ) + BLIP2 (70%, FILTERED) versus RAW (TOP  $30\%$  )—still offers significant classification and retrieval performance gains across all scales. However, for synthetic captions to perform competitively on ImageNet at larger data regimes, we need to start paying attention to image content, as well as enhancing the diversity of the generated text.
In this work, we demonstrate the effectiveness of synthetic captions in improving caption quality for multimodal training, as well as certain capabilities of the resulting model (e.g., retrieval). Notably, we find that fine-tuning general purpose models towards the task of image captioning actually makes them less effective at producing good captions for CLIP training. Our experiments with various candidate pool sizes, ranging from 12.8M to 1.28B image-text pairs, show that including generated captions in the training data can be highly effective at small and medium scales. However, with larger data quantities, the diversity gap between model-generated and web-scraped text begins to hinder performance gains, and it becomes increasingly harder to obtain state-of-the-art ImageNet accuracy by just improving text supervision alone.
Limitations. Our experiments do not involve an exhaustive list of image captioning systems currently available. Given a captioning model of sufficient capability—i.e., it can generate captions for training CLIP to reach a good performance—a major theme of our work is understanding how
to combine signals from both raw and synthetic captions, as well as the differences between these two sources of text. We note that even with improved caption quality, multimodal web datasets may still contain harmful stereotypes, some of which have been extensively discussed in prior work [7]. In Appendix H, we conduct some preliminary investigation on the change in race and gender bias between training on synthetic captions and training on only web-crawled text. Besides, generated captions can introduce new biases that they inherit from the captioning models, and using these captions to train the next generation of models can amplify the biases. The risks from using model outputs to replace human annotations have been studied in a simplified settings in [50, 48].
Future work. Our findings motivate a number of interesting future directions. One concrete question is whether combining synthetic caption data from multiple image captioning systems can enhance text diversity. Another direction is proposing new algorithms to combine information from raw and generated captions, beyond what we already studied in Section 5 and Appendix D. Future work could also explore using text-to-image generation [42, 34, 43] to create synthetic training images for concepts that are underrepresented in existing captions, in order to boost data diversity and close knowledge gaps in the downstream model.
We thank Stability AI for the generous assistance with compute resources. We are grateful to Josh Gardner and Simon Kornblith for providing feedback on the manuscript. We also thank Maciej Kilian, Anas Awadalla, Alex Fang, Dhruba Ghosh and Jonathan Hayase for helpful discussions while working on this paper. SYG is supported by a NSF Graduate Research Fellowship. This work is supported in part by Open Philanthropy, the Allen Institute for AI, and NSF grants DMS-2134012 and CCF-2019844 as a part of NSF Institute for Foundations of Machine Learning (IFML).
[1] A. Abbas, K. Tirumala, D. Simig, S. Ganguli, and A. S. Morcos. Semdedup: Data-efficient learning at web-scale through semantic dedduplication. arXiv preprint arXiv:2303.09540, 2023.
[2] H. Agrawal, K. Desai, Y. Wang, X. Chen, R. Jain, M. Johnson, D. Batra, D. Parikh, S. Lee, and P. Anderson. Nocaps: Novel object captioning at scale. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8948-8957, 2019.
[3] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-23736, 2022.
[4] P. Anderson, B. Fernando, M. Johnson, and S. Gould. Spice: Semantic propositional image caption evaluation. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pages 382-398. Springer, 2016.
[5] S. Azizi, S. Kornblith, C. Sahara, M. Norouzi, and D. J. Fleet. Synthetic data from diffusion models improves imagenet classification. arXiv preprint arXiv:2304.08466, 2023.
[6] H. Bansal and A. Grover. Leaving reality to imagination: Robust classification via generated datasets. arXiv preprint arXiv:2302.02503, 2023.
[7] A. Birhane, V. U. Prabhu, and E. Kahembwe. Multimodal datasets: misogyny, pornography, and malignant stereotypes. arXiv preprint arXiv:2110.01963, 2021.
[8] M. Byeon, B. Park, H. Kim, S. Lee, W. Baek, and S. Kim. Coyo-700m: Image-text pair dataset. https://github.com/kakaobrain/coyo-dataset, 2022.
[9] L. Cao, B. Zhang, C. Chen, Y. Yang, X. Du, W. Zhang, Z. Lu, and Y. Zheng. Less is more: Removing text-regions improves clip training efficiency and robustness. arXiv preprint arXiv:2305.05095, 2023.
[10] S. Changpinyo, P. Sharma, N. Ding, and R. Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3558-3568, 2021.
[48] I. Shumailov, Z. Shumaylov, Y. Zhao, Y. Gal, N. Papernot, and R. Anderson. Model dementia: Generated data makes models forget. arXiv preprint arXiv:2305.17493, 2023.
[49] H. Tan and M. Bansal. Vokenization: Improving language understanding with contextualized, visual-grounded supervision. arXiv preprint arXiv:2010.06775, 2020.
[50] R. Taori and T. B. Hashimoto. Data feedback loops: Model-driven amplification of dataset biases. arXiv preprint arXiv:2209.03942, 2022.
[51] R. Vedantam, C. Lawrence Zitnick, and D. Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575, 2015.
[52] P. Wang, A. Yang, R. Men, J. Lin, S. Bai, Z. Li, J. Ma, C. Zhou, J. Zhou, and H. Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In International Conference on Machine Learning, pages 23318-23340. PMLR, 2022.
[53] Z. Wang, J. Yu, A. W. Yu, Z. Dai, Y. Tsvetkov, and Y. Cao. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108.10904, 2021.
[54] P. Young, A. Lai, M. Hodosh, and J. Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
[55] J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
[56] J. Zheng, J. Zhang, J. Li, R. Tang, S. Gao, and Z. Zhou. Structured3d: A large photo-realistic dataset for structured 3d modeling. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX 16, pages 519-535. Springer, 2020.
[57] D. Zhu, J. Chen, K. Haydarov, X. Shen, W. Zhang, and M. Elhoseiny. Chatgpt asks, blip-2 answers: Automatic questioning towards enriched visual descriptions. arXiv preprint arXiv:2303.06594, 2023.
A More examples of image-text pairs (no cherry picking)
side of a road.
Raw: 2003 Mercedes-Benz C240 sedan, Leather, MUST BE SEEN - $6199
BLIP (finetuned): a couple of cars parked in a parking lot with trees and cars
BLIP2: 2002 mercedes-benz c-class for sale
BLIP2 (finetuned): a blue mercedes benz car parked in a parking lot next to yellow cars
OpenCLIP-CoCa: find used 2001 mercedes benz c 240 base sedan 4 door 251 for 2001 mercedes benz c 2
OpenCLIP-CoCa (finetuned): a blue mercedes parked on the
Raw: Gaziburma Ünal is one of Gespeicherte Orte von Can.
BLIP (finetuned): dozens of trays of different types of treats at a food stand
BLIP2: some trays of pastries and a knife
BLIP2 (finetuned): there are many trays filled with food items
from the store
OpenCLIP-CoCa: baklava, sweets, pastries
OpenCLIP-CoCa (finetuned): there are trays full of different types of food.
MOREINFO
Raw: Open Virgin of Kazan, Wooden Egg with Stand, Blue
BLIP (finetuned): a gray and white logo with the words more info in a rectangular shape
BLIP2: a white button with the word more info
BLIP2 (finetuned): more information is shown on a white button with an orange background
OpenCLIP-CoCa: home - page - button . png
OpenCLIP-CoCa (finetuned): a picture of a close up of a text message
my pronouns are
Raw: 2016.07.01 Nametags with Pronouns - Avery 5392_non-branded
BLIP (finetuned): there are no pictures here to provide a caption for
BLIP2: hello, my name is name, my pronouns are pronouns
BLIP2 (finetuned): a blue and white label with a blue and white text
OpenCLIP-CoCa: 15 + hello my name is names pronunciations and meanings
OpenCLIP-CoCa (finetuned): hello my name is, my pronouns are.
Raw: Italien - Ligurien
BLIP (finetuned): beige colored building with tan accents and palm trees on both sides of walkway
BLIP2: house in villa marina, a villa with many rooms and palm trees
BLIP2 (finetuned): a park with lots of trees and benches in front of a large building
OpenCLIP-CoCa: residence - villa - maria - di - san - giovanni - near - the - sea - in - taormina
OpenCLIP-CoCa (finetuned): a picture of a large building with
a bunch of palm trees.
Raw: 3 formations dePEDIR la mano de tu novia - wikiHow
BLIP (finetuned): crates stacked up in a pile on top of each other
BLIP2: the building contains five floors of wooden planks
BLIP2 (finetuned): a big pile of wooden planks stacked together  
OpenCLIP-CoCa: the cost of wood pallets
OpenCLIP-CoCa (finetuned): a large pile of wooden pallets mounted to a wall.
Raw: lutz
BLIP (finetuned): blond-haired man in black suit looking at camera
BLIP2: a man sitting on a chair with a blank background
BLIP2 (finetuned): a man sitting in a chair with a lapel button in front
OpenCLIP-CoCa: actor tilda swinton is pictured during a press conference for the film ’ a dangerous method ’ at the 2011 toronto film festival
OpenCLIP-CoCa (finetuned): a person sitting on a chair wearing a suit and tie.
face and a Christmas hat.
Raw: Women Personality Creative Christmas Hat Face Expression Gold Earring Funny Cartoon Ear Stud Jewelry Accessories Gift Hot
BLIP (finetuned): red and gold tone emojt earring
BLIP2: kawaii Santa emoticuos en la cabeza
BLIP2 (finetuned): a pair of emoji earrings with faces and hats
OpenCLIP-CoCa: best Christmas gift for her new arrivals emoji earrings Christmas emoji earrings
OpenCLIP-CoCa (finetuned): a pair of gold earrings with a smiley
Raw: 10840 SW 126th St photo067
BLIP (finetuned): overview of a large backyard with a swimming pool and patio
BLIP2: 3344 sw 7th st, palm beach
BLIP2 (finetuned): a house with a pool from above, with a yard OpenCLIP-CoCa: home for sale in country club shores west palm beach florida
OpenCLIP-CoCa (finetuned): aerial image of a pool that has a little bit of shade by the side.
Raw: image8.JPG
BLIP (finetuned): members of a school play soccer in a gymnasium with a crowd
BLIP2: a large crowd of kids perform during a dance show
BLIP2 (finetuned): a group of young children standing on the basketball court
OpenCLIP-CoCa: kid dressed in white standing in a gym area
OpenCLIP-CoCa (finetuned): a group of kids on the gym floor with fans on the floor.
with a large white background
Raw: hair oil
BLIP (finetuned): smiling blonde woman blow drying hair in a salon while getting a mani
BLIP2: hair stylist using hair spray in beauty salon
BLIP2 (finetuned): a person is using a hairdryer to blow dry a long blonde hair
OpenCLIP-CoCa: – female hairdresser styling a longblond hair with hairspray in a beauty salon . concept: hair care , hair straightening , hair color correction
OpenCLIP-CoCa (finetuned): a person is spraying a hair dryer on a long blonde hair.
Raw: Women long sleeve t shirt 2015 Fashion shirts woman Full Comfortable leisure fashion womens long sleeve tops
BLIP (finetuned): the qaoo loading logo is shown above the qaoo loading logo
BLIP2: qoo10 loading logo on white
BLIP2 (finetuned): a picture of an image of a phone screen showing a loading sign
OpenCLIP-CoCa: loading_111120_01.png
OpenCLIP-CoCa (finetuned): a light grey font and a dark grey font
Raw: Nautica NAPTYR005
BLIP (finetuned): navitta mens stainless steel bracelet watch with blue dial
BLIP2: nautica men's chronograph watch
BLIP2 (finetuned): nautica men's men's chronograph black dial stainless steel bracelet watch
OpenCLIP-CoCa: nautica newport chronograph n 2 2 0 0 3 g
OpenCLIP-CoCa (finetuned): a mans black watch is shown with red and blue accents
Raw: Greenberg Weathered Marble Plush Ivory Area Rug
BLIP (finetuned): grey rug with a text home on it by a table
BLIP2: a grey area rug on a wooden floor
BLIP2 (finetuned): a white coffee table with a sign saying home on it. it is sitting on a cream colored rug
OpenCLIP-CoCa: rugs and carpets in hyderabad : buy online at best price in …
OpenCLIP-CoCa (finetuned): a rug is shown in a living room with a chair.
Raw: productivity, productivity, productivity
BLIP (finetuned): drivers guide to the truck industry  
BLIP2: buy and sell truck parts
BLIP2 (finetuned): a white truck with a cover on it drives along a highway
OpenCLIP-CoCa: how the trucking industry is changing
OpenCLIP-CoCa (finetuned): there are some trucks on the road.
Raw: Amigas
BLIP (finetuned): crowd of people outside a wedding ceremony near several trees
BLIP2: a wedding ceremony in the middle of the street
BLIP2 (finetuned): a black and white photograph of a number of women in prom dresses
OpenCLIP-CoCa: 20130805_wedding_carlenan_003
OpenCLIP-CoCa (finetuned): a group of people hugging and talking in a group
Raw: Autozone
BLIP (finetuned): racing track with a line of seats and a sky background
BLIP2: a photo of a grand prix race track, under a blue sky
BLIP2 (finetuned): the circuit track is empty, but the sun beams into the sky
OpenCLIP-CoCa: circuit of the americas
OpenCLIP-CoCa (finetuned): a red and white pole next to a
racing track
THE ARCTIC LIGHT