ICML 2025

What If We Recaption Billions of Web Images
with LLaMA-3?
Xianhang Li * 1 Haoqin Tu * 1 Mude Hui * 1 Zeyu Wang * 1 Bingchen Zhao * 2 Junfei Xiao 3 Sucheng Ren 3
Jieru Mei 3 Qing Liu 4 Huangjie Zheng 5 Yuyin Zhou 1 Cihang Xie 1
Abstract
Web-crawled image-text pairs are inherently noisy.
Prior studies demonstrate that semantically align-
ing and enriching textual descriptions of these
pairs can significantly enhance model training
across various vision-language tasks, particularly
text-to-image generation. However, large-scale
investigations in this area remain predominantly
closed-source.
Our paper aims to bridge this
community effort, leveraging the powerful and
open-sourced LLaMA-3, a GPT-4 level LLM.
Our recaptioning pipeline is simple: first, we
fine-tune a LLaMA-3-8B powered LLaVA-1.5
and then employ it to recaption ∼1.3 billion im-
ages from the DataComp-1B dataset. Our em-
pirical results confirm that this enhanced dataset,
Recap-DataComp-1B, offers substantial benefits
in training advanced vision-language models. For
discriminative models like CLIP, we observe an
average of 3.1% enhanced zero-shot performance
cross four cross-modal retrieval tasks using a
mixed set of the original and our captions. For
generative models like text-to-image Diffusion
Transformers, the generated images exhibit a sig-
nificant improvement in alignment with users’
text instructions, especially in following complex
queries.
Our project page is https://www.
haqtu.me/Recap-Datacomp-1B/.
1. Introduction
The exponential growth in data availability is one of the most
paramount factors in driving the monumental successes of
deep learning over the past decade (Deng et al., 2009; Lin
et al., 2014; Changpinyo et al., 2021; Schuhmann et al.,
*Equal contribution
1University of California, Santa Cruz
2University of Edinburgh 3Johns Hopkins University 4Adobe Inc.
5University of Texas at Austin. Correspondence to: Cihang Xie
<cixie@ucsc.edu>.
Proceedings of the 42 nd International Conference on ...