Improving Multimodal Datasets with Image Captioning
Authors: name: Thao Nguyen, name: Samir Yitzhak Gadre, name: Gabriel Ilharco, name: Sewoong Oh, name: Ludwig Schmidt

Massive web datasets play a key role in the success of large vision-language
models like CLIP and Flamingo. However, the raw web data is noisy, and existing
filtering methods to reduce noise often come at the expense of data diversity.
Our work focuses on caption quality as one major source of noise, and studies
how generated captions can increase the utility of web-scraped datapoints with
nondescript text. Through exploring different mixing strategies for raw and
generated captions, we outperform the best filtering method proposed by the
DataComp benchmark by 2% on ImageNet and 4% on average across 38 tasks, given a
candidate pool of 128M image-text pairs. Our best approach is also 2x better at
Flickr and MS-COCO retrieval. We then analyze what makes synthetic captions an
effective source of text supervision. In experimenting with different image
captioning models, we also demonstrate that the performance of a model on
standard image captioning benchmarks (e.g., NoCaps CIDEr) is not a reliable
indicator of the utility of the captions it generates for multimodal training.
Finally, our experiments with using generated captions at DataComp's large
scale (1.28B image-text pairs) offer insights into the limitations of synthetic
text, as well as the importance of image curation with increasing training data
quantity.