DataComp-VLM: Improved Open Datasets for Vision-Language Models
Authors: name: Matteo Farina, name: Vishaal Udandarao, name: Thao Nguyen, name: Selim Kuzucu, name: Maximilian Böther, name: Andreas Hochlehnert, name: Adhiraj Ghosh, name: Marianna Nezhurina, name: Karsten Roth, name: Joschka Struber, name: Yuhui Zhang, name: Sebastian Dziadzio, name: Elaine Sui, name: Soumya Jahagirdar, name: Dhruba Ghosh, name: Hasan Hammoud, name: Thomas De Min, name: Simone Caldarella, name: Jehanzeb Mirza, name: Sedrick Keh, name: Mehdi Cherti, name: Hilde Kuehne, name: Bernt Schiele, name: Serena Yeung-Levy, name: Muhammad Ferjad Naeem, name: Federico Tombari, name: Ana Klimovic, name: Elisa Ricci, name: Matthias Bethge, name: Sewoong Oh, name: Ameya Prabhu, name: Alessio Tonioni, name: Jenia Jitsev, name: Massimiliano Mancini, name: Ludwig Schmidt, name: Nikhil Parthasarathy

Building performant Vision-Language Models (VLMs) requires carefully curating large-scale training datasets, yet the community lacks systematic benchmarks for evaluating such curation strategies. We introduce DataComp for VLMs (DCVLM), a benchmark for controlled data-centric experiments to improve VLM training. As part of DCVLM, we collect 160 datasets spanning four data types -- image-caption pairs, multimodal interleaved documents, text-only, and instruction-tuning data -- into a corpus of 6T multimodal tokens. DCVLM allows participants to test curation strategies (filtering, mixing, formatting, sampling) across 1B-8B models and 6.25B-200B token budgets. Models are then evaluated on a carefully selected suite of up to 52 downstream benchmarks across 9 domains. We conduct extensive experiments on DCVLM and find that data mixing, not filtering, is key to a high-quality training dataset: instruction-heavy mixtures scale better than caption-heavy ones, with gains widening at larger scales. The resulting dataset, DCVLM-Baseline, enables training an 8B VLM to 63.6% accuracy on our 33-task core suite with 200B training tokens. Compared to FineVision, the state-of-the-art open VLM training dataset, this represents an improvement of +5.4pp. DCVLM and all accompanying artifacts will be made publicly available at https://www.datacomp.ai/dcvlm/.