What If We Recaption Billions of Web Images with LLaMA-3?
Authors: Xianhang Li
Haoqin Tu
Mude Hui
Zeyu Wang
Bingchen Zhao
Junfei Xiao
Sucheng Ren
Jieru Mei
Qing Liu
Huangjie Zheng
Yuyin Zhou
Cihang Xie
Venue: ICML 2025 poster (2025)
Keywords: image-text datasets; synthetic captions
Primary area: deep_learning->foundation_models
TL;DR: We propose Recap-DataComp-1B, the first publicly available image-text dataset with LLaMA-3–generated synthetic captions scaled to the billion level.

Abstract: Web-crawled image-text pairs are inherently noisy. Prior studies demonstrate that semantically aligning and enriching textual descriptions of these pairs can significantly enhance model training across various vision-language tasks, particularly text-to-image generation. However, large-scale investigations in this area remain predominantly closed-source. Our paper aims to bridge this community effort, leveraging the powerful and $\textit{open-sourced}$ LLaMA-3, a GPT-4 level LLM. Our recaptioning pipeline is simple: first, we fine-tune a LLaMA-3-8B powered LLaVA-1.5 and then employ it to recaption ~1.3 billion images from the DataComp-1B dataset. Our empirical results confirm that this enhanced dataset, Recap-DataComp-1B, offers substantial benefits in training advanced vision-language models. For discriminative models like CLIP, we observe an average of 3.1% enhanced zero-shot performance cross four cross-modal retrieval tasks using a mixed set of the original and our captions. For generative models like text-to-image Diffusion Transformers, the generated images exhibit a significant improvement in alignment with users' text instructions, especially in following complex queries. Our project page is https://www.haqtu.me/Recap-Datacomp-1B/.

What If We Recaption Billions of Web Images with LLaMA-3?
Xianhang Li Jieru Mei
Haoqin Tu 1
Mude Hui')
Zeyu Wang" 1 Bingchen Zhao
Junfei Xiao
Sucheng Ren
Qing Liu
Huangjie Zheng
Yuyin Zhou'
Cihang Xie
Abstract
2021; Gadre et al., 2023; Fang al., 2023). Typically, this data is sourced through web crawling with simple filtering mechanisms in place. While such approach has facili- tated large-scale data collection, exemplified by collections like LAION-400M (Schuhmann et al., 2021) and LAION- 5B (Schuhmann et al., 2021) with billions of image-text records, it has inadvertently compromised data quality. As illustrated in Figure 1, these internet-crawled image-text pairs frequently exhibit misalignments between images and their corresponding textual content, and often, the textual descriptions are brief and lack detailed information.
Web-crawled image-text pairs are inherently noisy. Prior studies demonstrate that semantically align- ing and enriching textual descriptions of these pairs can significantly enhance model training across various vision-language tasks, particularly text-to-image generation. However, large-scale investigations in this area remain predominantly closed-source. Our paper aims bridge this community effort, leveraging the powerful and open-sourced LLaMA-3, GPT-4 level LLM. Our recaptioning pipeline is simple: first, we fine-tune LLaMA-3-8P powered LLaVA-1.5 and then employ it to recaption 11. billion im- ages from the DataComp- IB dataset. Our em- pirical results confirm that this enhanced dataset, IB, offers substantial benefits in training advanced vision-language models. For discriminative models like CLIP, we observe an average of 3.1% % enhanced zero-shot performance cross four cross moosl retrieval tasks using a mixed set of the original and our captions. For generative models like text-to-image Diffusion Transformers. the generated images exhibit a sig- nificant improvement in alignment with users' text instructions, especially in following complex queries. Our project page is https:/ www. haqtu. me Recap- Datacomp-
To mitigate the noise present in web-crawled data, enhance- ments through post-processing- inaieessinn-immllmeentee via human- in ln-the-loop systems (Sun et al., 2023; Yu et al., 2023b) or autorntedd pipelines (Schuhmann al., 2021; Li et al., 2022; 2023a) crucial, which help train the advanced vision-language foundation models. Notably, both the close- sourced DALL-E (OpenAI, 2023) and SORA (OpenAl, 2024) incorporate advanced captioning techniques to re- label their training datasets, a crucial step highlighted in their technical reports. Despite various efforts to open- source and replicate these methodologies (Chen et al., 2023a; Li et al., 2022; 2023a; Liu et al., 2023b; Yu et al., 2023a; Fan et al., 2024; Rotstein et al., 2023), the commu- nity continues to face sigmificant challenges in accessing high-quality, well-aligned image-text data at scale (e.g., at the billion level) for training advanced asionnlanguage foun- dation models.
This paper endeavors contribute to this community initia- tive, inspired specifically by the release of LLaMA-3 (Meta LLaMA Team, 2024), a model demonstrating GPT-4-level capabilities across variety of linguistic tasks. Additionally, recent studies have shown that leveraging LLaMA-3 can sig- nificantly enhance model performance on vision-language tasks (Liu et al., 2024; Xu et al., 2024), comparable to those achieved by GPT-4V (Achiam al., 2023a). In response, we employ LLaMA-3 to develop our advanced captioner model. Our approach is straightforward: we first train L-aMA-3- powered LLaVA model to act as an image captioner, which is then utilized to recaption the entire DataComp- dataset. As depicted in Figure 1, the resulting dataset, dubbed Recap-
1. Introduction
The exponential growth in data availability is one of the most paramount factors in driving the monumental successes of deep learning over the past decade (Deng et al., 2009; Lin et al., 2014; Changpinyo et al., 2021; Schuhmann et al.,
Eqqual contribution Universsiy of California, Santa Cruz Universitt of Edinburgh 'Johns Hopkins University "Adobe Inc. A Uniierriity of Texas at Austin. Correspondence to: Cihang Xie <cixic@uese.cdux>
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

What If We Recaption Billions of Web Images with LLaMA-3?
Original: Cafe Van gooh stock image
Original: Moyle BC rallway
Original: Delure Twin Room
Ours: The first image chows the exterior of
Ours: A yellow train red and blue accent is trweeing slong milwwey track that nuna araliet a body of water, The train is surroun colors, yellow orange sallway track bor bordared by glassy some bushes and a dirt mound on the loft sidle, The wattr is callr,, offlecting the main an d'ehe trees.
Ours: A modern hotel room with a large bed
pare with a yellow and green facade, The care has large glass vindow with rod frames, and the
covered in white and red bedtling, nightttand with lamp, and a chair with small tablo in The purple curtains, facure, wallpaper. There are two colorful rugs the floor, and the room well-it with natural light coming througth the window.
ded aandssape
hanging iigtt outside
turing
celling patterned
light
tablestotns, plstes, and glasses. The chairs
are blackwit with wavan seat. The floor wooden, and share is small ported plant...
background
black
against
wearing
white
text
displayed
Recaption (92.86%)
top
right
placed
eaturing
4 Reclp
two
front
Kinghird smail, gray with black and black eyes perched on brown branch. The bird has fluffy appearance with mix gray and yellow teathers on its body. background green, suggesting natural
man
Daluse Room
each
Ours: A slice of cake with caramet
The frosting placed
garnished dollop whipped cream top and a sprinite of powdered sugar. The plate on a white surfaca, and there watermark the bottom right corner that reads 'Recipe Spain.com paain.comm
Word distributions of recaptions
out-of-
and the original captions.
The
Erite
Wix Wirotfert ooroorrttoon boooo
Blof Aome
Oriiiaa: siesta key florido picture
Original: Eidor Spredway (OH) 6/8
Oriiial Buckle geap The tnke Brothamood"
Durs: A bustting beach scene with numerous people enjoying the sun and sand. The beach
Oura gonup
is apeeding rracack. The lend car is red with the name 'Lu
ailver motoreycle charm
detaila is artached a red background with
ined with colorful umbrelias and sun
cas' the side, followed by blue car went
words The Tike Brotherhood and 'Blood
loungens, and the water vibrant turquaise, sky partly cloudy, atmosphere lively and
ha number 'Intt and black car with numb
Honor anggoeed cursive acript. The charm
matorcycle
background
Figure 1. Examples of the original caption and our recaption in DataComp IB, and word distributions. The word distribution compares the frequency of word usage in the re-captioned data to that in the original captions.
DataComp- IB, features enhanced textual descriptions and improved alignment with corresponding images, clearly surpassing its web-crawled counterparts. These quality en- hancements are further quantitatively verified in Section 4.
capabilities (e.g., 64.8% > 61.7% over four cross-modal re- trieval tasks). It also improves the alignment between gener- ated images and text instructions in text-to-image generative models pre-trained on our dataset, resulting in higher quality images and better relevance the input-demonstrated by 8.4 lower FID and 3.1% higher CLIP score. We hope that the release of Recap-DataComp ther developments in advanced vision-language foundation models, particularly encouraging the development within the open-source community.
Comprchensive evaluations highlight the significant im- provements that Decap-DataComp- contributes to the training of advanced vision-languse foundation models. Notably, this dataset enables CLIP models to achieve signif- icant enhancements their zero cross-modal retrieval
will catalyze fur-
2

---

What If We Recaption Billions of Web Images with LLaMA-3?
2. Related works
Better IP Medek
Tosks
Caption
Natrioaal Fbor Fosioo
Vision-Language Foundation Models. CLIP (Radford et al., 2021a) is one of the pioneering foundation models to connect image and text. By training on millions, and even billions, of image-text pairs (Changpinyo al., 2021; Desai et al., 2021; Fang et al., 2023; Gadre et al., 2023; Schuhmann et al., 2022b; 2021; Sharma et al.. 2018; Srinivasan et al., 2021), CLIP markedly showcases excessively strong zero- shot capacities, and furthermore, lays the coreerstone for building more advanced vision- language foundation models (Alayrac et al., 2022; Li et al., 2022; 2023a; Wang et al., 2022; Liu et al., 2023b; 2024; Chen et al., 2023b; Bai et al., 2023; Xu et al., 2024). Apart from discriminative vision- language understanding, text-to-image generation models (Ding et al., 2021; Nichol et al., 2021; OpenAl, 2023; Pee- bles & Xie, 2023; Ramesh et al., 2022: 2021; Rombach et al., 2021; Saharia et al., 2022: Yu et al., 2022a) have transformed the field of AI Al-generated content, facilitating the creation of high-quality images from natural language descriptions.
Recaption
DetxComp-18
Recapt anedd DataComp 1B
Goneeation
Better Diffusion Models
Figure 2. The illustration of recaptioning pipeline on DataComp- IB. We use LLaMA-3 p-oweere LLaVA to reception images, which enables us to train stronger CLIP models and Text- to-Image Diffusion models.
timodal models like LLaVA (Liu et al., 2023b) are employed for recaptioning purposes. However, our approach is distin- guished by two major aspects: 1) we strongly enhance the LLM module in LLaVA, i..., building with LLaMA-3; and 2) our recaptioning efforts are executed on a billion-scale dataset.
3. Recaptioning Pipeline
Our recaptioning pipeline is centered around the ad- vanced LLM LLaMA-3 (Meta LLaMA Team, 2024), which achieves exceptionally strong performance in language un- derstanding, reasoning, code generation, math problems, etc. (Chiang et al., 2024; Stevens, 2024). Specifically, we utilize the LLaVA framework (Liu et al., 2023b) to fully har- ness its capabilities for visual understanding. We describe the detailed training procedures below.
Enhancing Image-Text Data. Web-crawled image-text data (Schuhmann et al., 2021; Gadre et al., 2023; Fang et al., 2023) commonly face the problems of image-text misalignment and the low-quality of textual descriptions. Typically, there are two popular ways for improving the quality of these image-text pairs: 1) data filtering removes misaligned image-text pairs using various methods such as cleaning strategies (Schuhmann et al.. 2022b; Gadre et al., 2023; Xu et al., 2023), pretrained models (Li et al., 2022: Schuhmann et al., 2021: Gadre et al., 2023), and human- assisted systems (Sun et al., 2023; Yu et al., 2023b; Zhang et al., 2023): 2) data recaptioning improves the textual qual- ity of image-text pair via generating new captions, which is the focus of this paper. To recaption data, LaCLIP (Fan et al., 2024) utilizes large language models (LLMs) like ChatGPT to rewrite the original captions; Nguyen et al. (Nguyen et al., 2024) employ BLIP2 (Li et al., 2023a) to recaption images. More recently, advanced large multimodal models have been applied to further enhance the quality of image captioning. For example, ShareGPT4V (Chen et al., 2023a) employs GPT-4V to create highly descriptive captions from carefully crafted prompts and corresponding image inputs: the resulting dataset has significantly benefited the training of various models (Chen et al., 2024a; Zhang et al., 2024a; Chu et al., 2024; Lin et al., 2024; Fei et al., 2024; Awadalla et al., 2024). However, scaling such prompting with GPT- 4V to billions of records is less practical, as it will drastically increase the monetary cost (of intensively calling OpenAl APIs) by more than x0,000x.
3.1. Model details
Model Configuration. We follow the setup of LLaVA- 1.5 (Liu et al., 2023a) to build our captioner model, except that we use LLaMA-3-8B as the language decoder because of its superior performance. The visual branch of CLIP ViT-L/14 (Radford et al., 2021b) is used as the vision en- coder. Two trainable MLP layers are employed on top of the vision encoder to project visual features into the language embedding space.
2-Stage Training. We also follow LLa (Liu et al., 2023a) for model training- Essentially we conduct instruction-tuning on the pre-trained LLM with its origi- nal reto-regees training objective. In the first stage, only the projection MLP is trained; in the second stage, we fine- tune both the projection MLP and the language decoder. Note that the vision encoder remains frozen all the time. Following the protocols in LLa VA (Liu et al., 2023a), 558k image-text pairings filtered from LAION (Schuhmann et al., 2022b), CC (Changpinyo et al., 2021), and SBU (Ordonez et al., 2011) are used as training data in the first stage; then 665k instractions-followee
data from VL- (Liu et al., conversation, image de-
Our paper mostly follows the approach presented in (Chen et al., 2024b; Lu et al., 2023; Zhang et al., 2024a; Chu et al., 2024; Huang et al., 2024), where advanced open-source mul-
2023a), containing scriptions, and image-based complex reasoning tasks, are used for the second stage of training. To help our model
3

---

What If We Recaption Billions of Web Images with LLaMA-3?
generate higher-quality captions, we use the image-text pairs from HQ- Eddt dataset (Hui et al., 2024) for further tuning, which distinguishes our approach from concurrent work (Hinck et al.).
4. Analyzing Recap-Data Cecap-DataComp-1B
This section collects and presents a quantitative analysis of our generated captions on DataComp- We primarily focus on two aspects: 1) the inherent features of the captions, including word distributions and average lengths; and 2) the semantic quality of the captions, evaluated in terms of the matching similarity between images and captions and the inherent textual quality of the captions.
Table 1. Performance comparison of LLaVA.
Model
L.5VA-1.5-BB
LLaVA-1,5-13B 1LLWA-1--LLLMMM--- (ours)
GPT-IV
MMMU
33.6
36.4
37.5
56.8
MM-NeI
33,9
36.3
Evaluations. To probe the visual understanding and reason- ing ability of our 1-.AA-...--.LMM....... ---8888 model, we opt for two comprehensive multi-moda evaluation benchmarks, MMMU (Yue et al., 2024) and MM-Vet (Yu et al., 2024). These benchmarks assess a broad range of capabilities such as recognition, spatial awareness, OCR, knowledge, and language generation. As reported in Table 1, on both bench- marks, our LLaVA-1155 1--V----55--..1 1..5-155.aaM -LLLMM3-888 model surpasses the vanilla LLa VA- model by significant margin. These results also match, or even outperform, the considerably larger LLa 1.AA-..5-133 13B model, demonstrating the superior visual understanding and reasoning ability of our model.
4.1. Word & Length Distribution
We begin our analysis by comparing the word frequeney distributions hetween our recaptioned content and that from the original DataComp- IB, as illustrated in Figure 1, ana- lyzing a randomly sampled subset of approximately 0.35 billion instances. Our findings reveal that the recaptioned content displays a considerably richer vocabulary, captur- ing 82.86% tokens of the word collections from both ours and the original caption data. Additionally, there is a no- ticeable variety in the usage of nouns and adjectives in our captions (e.g.. "white" and "background"), We argue that this increased lexical diversity a direct consequence of the extended length of our data. We thus present the distribu- tion of instance caption lengths in Figure highlight this difference. On average, our recaptioned data demonstrates longer sequence length of 49.43, whereas the original DataComp captions have a much shorter length of 10.22. These observations validate that our Recap-DataCome DataComp- IB surpasses the original DataComp-1B version in terms of both caption length and diversity.
3.2. Recaptioning DataComp-1B
With this advanced LLaVA model, we next use it to gen- erate captions in a scalable and detailed manner, given the visual input, and the following text prompt: "Please generate a detailed caption of this image. Please be descriptive as possible." As for the dataset, we opt for DataComp-IB (Gadre et al., 2023), a widely accessible, large-scale vision-language dataset comprising 11. bil- lion web-crawled image-text pairs. To ensure its quality, DataComp-18 is already a curated subset from a much larger collection of 12.8 billion image-text pairs and has been subjected to rigorous preprocessing which includes safety checks, deduplication, CLIP score filtering, and image-based filtering. Despite these efforts, the quality of the original captions in DataComp-1B 1B still exhibits rela- tively low quality.
GPT-4V Evaluation Instruction:
[Image Caption] CAPTION
Rate whether the caption is of high-quality and fluent and correctly matches the given image. The rating should be 1-5, where is incorrect and not fluent at all, and 5 is correct and very fluent. Try to just give a numerical rating.
We apply our well-trained LLaVA-1.5-LLaMA33 model to recaption the entire DataComp-11 1B dataset. Specifically, captions are generated auto-regressively via greedy decod- ing, with the maximum output token length set to 128. De- tails on the computation cost are provided in Appendix B. We term this newly recaptioned dataset 1B.
Your response should be in the format: Rating: (int)
4.2. GPT-4V & CLIP Evaluations
Next, we evaluate the semantic quality of recaptioned con- tent using two models: 1) CLIP (Radford al., 202 which measures the semantic similarity between captions and images, and 2) GPT-4V (Achiam et al., 2023b), which assesses the fluency and alignment of captions with the given images.
Figure 3. Instance length distributions from 133 million sampled examples of both the original captions and our recaptioned data in DataComp-IB.
For the CLIP evaluation, we analyze a subset of 180,000 image-text pairs, Interestingly, we note that, when using the
4

---

What If We Recaption Billions of Web Images with LLaMA-3?
standard CLIP-large model with 428M parameters for this measurement, our recaptioned content performs just compa- rably to the original captions (49.57 vs. 50.43). We attribute this result primarily to the limitations of the standard CLIP model, which is trained on 'short' captions and may inade- quately capture the nuances in semantic similarity for longer captions. To probe deeper into semantic alignment between long captions and images, we utilize the LongCLIP-Larr model (Zhang et al., 2024a), which is specifically fine-tuned to handle longer captions. With this setup, the LongCLIP score of our newly generated caption impressively attains 89.91, nearly 9x higher than the LongCLIP score of the original DataComp captions (i.e., only 10.09).
including warm-up phase involving 51.2 million samples.
The batch size and base learning rate set to 32,768 and 8c-6, respectively, For the subsequent fine-tuning phase, we increase the image size to 224 and train the model on 128 million samples with a 25.6 million sample warm-up- We adjust the batch size to 16,384 and the learning rate to 4e-7.
Evaluation. The efficacy of Recap-CLIIP is gauged via several metrics. We evaluate zero-shot image classification on the ImageNet- I dataset (Russakovsky et al., 2015) and assess zero-shot cross-modaa retrieval performance using the validation set of MSCOCO 2014 (Lin et al., 2014) and the test set of Flickr30K (Young et al., 2014)', following the established practices (Radford et al., 2021a; Li et al., 2023b; Zhai et al., 2023; 2022).
In addition, to evaluate both the textual quality and the alignment of the captions with their corresponding images, we randomly select 10,000 instances for GPT-4V (Achiam et al., 2023b) evaluation, employing the prompting strat- egy outlined above (CAPTION is the textual input), as per (Padlewski et al., 2024; Lee et al.. 2024). We can observe that our recaptioned content achieves markedly superior rat- ings, registering an average rating increase of 0.43 (from 3.71 to 4.14.. Together with the findings from Section 4.1, this confirms the superior quality of our newly generated captions, in terms of length, vocabulary diversity, semantics, and image-text alignment.
We present our results from three aspects. First, we explore the impacts of differing mix ratios between original captions and our enhanced recaptions on CLIP performance. Next, we analyze the effects of enlarging the size of the CLIP text encoder. Lastly, we investigate the text understanding capa- bility of our Recap-CLIP, via testing on VG-Attribute (Yuk- sekgonul et al., 2022), which evaluates attributes understand- ing ability, and Urban (Zhang et al., 2024a), which tests the model's 's ability to handle long text.
5.2. Training with Mixed Captions
5. Training CLIPs with Recaptions
As pointed out by DALL-E 3 (OpenAI, 2023), blending both the briefgenuine captions and the long informative generated captions can effectively prevent the model from unwanted overfitting to recaption data. Therefore, we hereby first study the effect of varying mix ratios between the original captions and our recaptions on the training of the Recap- CLIP B/16 model, as detailed in Table 7. Specifically, for each sample in a training batch, we randomly sample the original caption with probability 0 S p S and our captions with probability 1
CLIP (Radford et al., 2021a) stands as a widely utilized vision-language model, where image encoder and text encoder are jointly trained to predict correct matches across entire batches of image-text pairs. In this section, we delve into the advantages of training CLIP models with our Recap- DataComp-11 B dataset. We anticipate that CLIP models trained on this dataset will exhibit superior zero-shot cross- modal retrieval capabilities and enhanced text understand- ing, especially with long and complex textual inputs, given the improved quality of our recaptions.
p, referring to the mixed ratio:
Original
with probability p with probability 1- p
Caption
Recaption
5.1. Experiment settings
This strategy ensures that each batch contains mixture of our recaption and the original captions controlled by probability p. The randomness allows each sample to en- counter different captions across training epochs, potentially enhancing the model's 's generalization.
Training. For reference, we term the CLIP model trained on our Recap-DataCCopp11B dataset as Recap-CLII Our training pipeline primarily follows CLIPA (Li et al., 2023b:c), which incorporates a two-state training, i.e., a pre-training process with a small image size followed by a fine-tuning stage incorporating a larger image resolution. We set the text token length to 128 to accommodate the learning of long captions presented in Recup-DataComp-II IB. We conduct experiments using three model scales: S/16, B/16, and L/16, with details listed in Appendix Table 7.
Main results. Our findings are summarized in Table 3. We observe that reducing the mixed ratio p (i.e., increasing the proportion of our recaption data) initially leads to an im- provement followed by a decline in cross-moda retrieval performance. This initial improvement suggests that high- quality recaptioned data effectively enhances contrastive
The Adam W (Loshchilov & Hutter, 2017) optimizer is used for training. In the pre-training phase, the model is trained with 2.56 billion samples with a reduced image size of 1 112,
We employ the widely used Karpathy split (Karpathy & Fei- Fei, 2015) of MSCOCO and Flickr30K.
5

---

What If We Recaption Billions of Web Images with LLaMA-3?
Table Train with larger text encoder.
We set p 0.8 for recaption based models. We report zero-shot top-1 accuracy on ImageNet- IK
and top-1 recall on COCO and Flickr30K.
ImageNettIIK
COCO RaI
F0 Rai
vision encoder
encoder
re-caption
Validation
1-T
T-+1
I-TT
T-=l
small
x
61.6
50.9
30.5
75.3
54.5
S/16
small
61.1 -0.3%
55.8 +4.9%
35.0 444
79.6 +4.3%
59.2 +4.7%
base
62.7
53.0
32.2
78.7
57.3
base
62.1 69.6%
56.7 +5.7%
36.0 +3.8%
80.4 +1.7%
60.3 +5.0%
hase
X
70.5
59.5
38.9
84.1
64.4
B/16
base
69.8
0.7%
62.8 13.3%
42,7
86.7
12.0%
67.1 12.7%
large
x
71.1
61.0
40.2
86.8
67.6
large
-0.69 6%
64.5 43.5%
43.4 +3.
88.3 4..55
68.9 +1.5%
large
76.5
62.5
44.2
89.0
71.1
1/16
large
76. -0.4%
66.8 +4.5%
47.8 +3.655
91.3 +2.0%
73.9 +2.88
huge
X
76.9
65.2
46.0
90.5
73.2
huge
76.3 -0.6%
68.5
43.5%
49.1
+3.4%
91.0
+0.5%
75.6 42.48
learning. However, the subsequent decrease indicates that the original captions from DataComp-IB provide necessary training regularization, preventing the model from overly adapting to the specific qualities of the recaption data. In- terestingly, we also observe that the performance of CLIP is relatively insensitive to certain variations in the mix ra- tio p. as evidenced by the consistent enhancement over the baseline (i.e. p=1.0) across all four different cross-modal retrieval metries when varying p from 0.2 (80% recaption data) to 0.9 (10% recaption data). For instance, setting p at 0.9 and 0.2 both yields a similar performance enhancement of ~2,77% with the peak performance occurring at p-0,, which delivers a substantial ~5% boost.
Table 3. Train with mixed captions. We choose Recap-CLIP. B/16 for this ablation. Larger p represents higher ratio of the original caption. We report top-1 zero-shot classification accuracy on ImageNet- K and top-1 recall for retrieval tasks.
ImageNet-1K
COCO R01
Flickr30K R:1
mixed ratio
Validaion
T-=1
I-+T
0.0
36.0
$3.0
34,1
74.1
$3.5
0.1
59.6
62.5
41.6
84,2
65.5
0.2
63.8
61.7
42.4
86.8
67.0
0.3
65.5
62.7
42.6
86.2
68.4
0.4
66.7
63.4
43.2
87.6
68.2
0,5
67.9
62.3
43.3
85.6
67.7
0.6
68.8
63.6
43.1
86.2
68.2
0.7
69.0
62.9
42.8
85.7
68, 1
86.7
0.9
70.3
62.8
41.8
$6,1
66.8
1.0
70.5
84.1
But meanwhile, we note that incorporating our recaptions (negatively) affects the zero-sho shot classification task, exem- plified by the consistent performance degradation across varying p values from 0 to 0.9. The phenomenon is similarly observed in the recent work (Zhang et al., 2024a) where they note directly fine-tuning on long text can significantly hurt the CLIP performance. Several works (Zheng et al., 2024; Liu et al., 2023d) propose novel techniques for enhancing learning with long texts. In this study, given our primary focus on assessing the quality of
parameters.
Main Results Our results, as shown in Table 2, first, demon- strate the effectiveness of using enhanced captions compared to original ones. On average, retrieval tasks show improve- ments of 4.6%, 3.1 %, and 3.3% for small, base, and large models, respectively. Second, enlarging the text encoder further boosts performance across all model scales. When the text encoder is enlarged, re-captioning consistently de- livers significant improvements, suggesting that enhanced captions provide benefits across models of varying scales.
we choose the ratio p 0.8 to strike promising balance between the classification performance (i.e., only marginally drops 0.7%) and the cross-modal retrieval performance (i.e., with a significant 3.1% % boost on average) for later ablations.
5.4. More evaluations on text understanding
Recent works demonstrate that CLIP models suffer from poor long context understanding and delicate attribute un- derstanding (Yuksekgonul et al., 2022; Zhang et al., 2024a). Given the long, enriched, and better-aligned captions, we expect Recap- ILIP to exhibit better text understanding ca- pability. Thus, we evaluate our Recap-CLIP model on two benchmarks: (1) Urban (Zhang et al., 2024a), a long- caption image-text retrieval benchmark that contains 1k urban images and corresponding GPT-4V captions; (2) VG- Attribution (Yuksekgonul et al., 2022), a modified version of
5.3. Training with Larger Text Encoder
We hereby investigate how the size of the text encoder af- fects models trained on a mixture of the original captions and our recaptions (with p = 0.8). Specifically, we keep the configuration of the vision branch as in appendix Table and only twitch the text encoder. For instance, in the case of the S/16 model, we change from smaller text encoder with 33M parameters to a larger, base-sized one with 53M
6

---

What If We Recaption Billions of Web Images with LLaMA-3?
Table Comparison with other CLIP models trained on public or private dataset. We report top-1 and recall of image and text retrieval on COCO and F0Kc
ImageNet- IK classification accuracy
IN-1K
Flickr30K R8
COCO R@1
method
model size
patches
dataset
public
val.
T-II
I-T
T-=I
I-T
CLIP (Radford et al., 202
Large
256
WIT-400M (Radford et al., 102
X
75.5
65.0
85.2
36.5
56.3
CLIP (Gadre et al., 2023)
Large
256
DataComp-I1 1B (Gadre et al., 2023)
79.2
73,4
89,0
45.7
63.3
OpenCLIP (lbarco et al., 2021)
Large
256
LAION-20 -B (Schuhmann al., 2022a)
75.5
75.5
89.5
46.5
63.4
SigLIP (Zhai et al,, 2023)
Large
256
WebLI-5B (Chen & Wang, 2022)
X
80.5
79.0
91.8
52.3
70.8
Recap-CLIP
Large
256
Recap-DataComp-l
79.3
79.5
94.1
53.7
72.0
CLIP (Fang et al., 2023)
Huge SO(400M)
729
DFN-5B (Fang et al., 2023)
X
84.4
82.0
94.0
55.6
71.9
SigLIP (Zhai et al., 2023)
729
WebLL-5B (Chen & Wang, 2022)
X
83.1
83.0
94.3
54.2
72.4
Recap-CLIP
Huge
256
Recap-DataComp-IIB
81.0
81.3
94.8
54.5
73.1
gin. Specifically, compared to the original DataComp- IB L/14 model, enhanced dataset and training result in sig- nificant performance improvements, with average gains of 5.6% and 8.4% on Flickr and COCO. Second, compared with the current state-of-the-art models trained on private datasets like WebLI-5B (Chen & Wang, 2022). our model demonstrates much higher training efficiency. For instance, SigLIP (Zhai et al., 2023) trained on 45 billion samples from a 5 billion-image in-house dataset, while our model, using only 12.8 billion training samples and significantly fewer model flops, achieves better retrieval performance. These results clearly highlight the quality of our proposed dataset and training method at scale.
Table 5. Comparison on the Urban- IK and -Grttbbibtte
method
rescuption
Urban-1K
VG
[-T
T-1
Amiluuic
OpenAT-CI JP-8216 (Radiced is oll, 2021a)
X
67.4
53.3
62.6
OpesCLIP-B/16 (Tlaaco et ol., 2021)
62.5
59.9
Kecap-CLIP-B/16
X
51.2
50.9
57.1
85.0
87.3
664 19
Recap CLIP L/16
69.8
64.6
60.1
89.0
m030
6631 18095
Visual Genome (Krishna et al., 2017) to test model abilities to attribute properties to objects, as shown in Tab. 5.
We observe consistent significant improvement if the model is trained on our Kecap-Datacomy B dataset. For both text- to-image and image-to-text retrieval on Urban- IK dataset, our Recap-CLLi CePP models surpass the vanilla baseline by at least 19% and sometimes up to an astonishingly high 36%. On the VG-attribution dataset, it is worth noting that our Recap-CLIP brings performance boost very close to that of the NegCLIP fine-tuning ( (Yuksekgonul et al., 2022) (e.g. ~9% vs. 10%), a lightweight downstream fine-tuning pro- cess designed to boost CLIP ability to understand attribute and order. Nonetheless, it is noteworthy that our Recap- CLIP naturally equipped with better text understanding ability, without any specific targeted fine-tuning, indicating the importance of better captions in web-scale data.
6. Training Text-to-Image Models with Recaptions
It has been known to the research community that train- ing with generated (high-quality) pseudo-captions improves text-to-image generative models in terms of generation qual- ity and prompt following ability (Chen et al., 2024b;a; Betker et al., 2023), primarily due to the low information and high noise density presented in the original web-crawled captions. Therefore, we evaluate the quality of our generated captions by training Text-to-Image Image (T2I) generative models on Recap-DataComp-11 for further justification. We expect the enriched information in the generated descriptions to better align the visual content in images, and thus improve the performance of the T2I models.
5.5. Scaling-up Recap-CLIP
To compare with current state-of-the-art CLIP models, by utilizing the training recipe discussed above (p = 0.8 and larger text encoder), we simply scale up the training sched- ule by 5 x. processing 12.8 billion samples, following com- mon practice in training CLIP (Radford et al., 2021 a; Li et al., 2023c; Ilharco et al., 2021). The hyperparameters like batch size learning rate mainly follow CLIPA (Li al., 2023b), and we train the L/14 and H/14 models. The results are shown in Table 4. First, when comparing advanced CLIP models trained on publicly available datasets like LAION- 2B (Schuhmann et al., 2022a) and DataComp- IB (Gadre et al., 2023), our Recap-CLII L/14 model achieves the best performance on zero-shot ImageNet- IK IK classification and COCO/Flickr30K retrieval tasks with a notable mar-
Training, We adopt Diffusion Transformers (DiT) (Pee- bles & Xie, 2023) as our T2I model, where the text condi- tion is extracted with a CLIP text encoder (Radford et al., a021a), and then injected into each DiT block with the cross- attention. We employ the original CLIP model to establish consistent evaluation baseline that aligns with commu- nity standards, enabling clearer analysis of the relation- ship between data modifications and performance changes. Specifically, we follow the image preprocessing pipeline in DiT (Peebles & Xie, 2023), where the images are pre- processed to have a square resolution of 256. The model
7

---

What If We Recaption Billions of Web Images with LLaMA-3?
Table Text-to-Image evaluation on COCO-30K results of DiT-BASE/4, trained with different mix ratios Rccap-DataComp- IB. Note for GPT-4V Score, we use subset of 3K for the evaluation.
Training
Evaluation
Raw
Our COCO-Recap
mixed ratio
FID
CLIP Scoret
FID
CLIP Scoret
Recap-Clip Scoret
GPT-4V Scoret
0.00
37.6
29.2
27.8 8
32.5 .3, 3.1%%
28.3 [88
2.53 +1.1
0.05
38.5
27.9
32.5
28.0
2.51
0.10
36.0
29.7
27.2
32.7
28.2
2.51
0.15
35.8
29.9
28.2
33.0
28.1
2.45
0,20
35.8
29.8
28.4
32.7
28.0
2,53
0.50
35.3
29.3
30.2
31.9
26.7
2.13
0.75
31.3
29.4
32.7
31.2
25.8
1.89
32.5
28.9
36.2
29.3
19.9
1.40
trained on visual latent extracted using a pretrained auto- encoder with downsampling ratio of 8 (Rombach al., 2021). Similar to the setup previous experiments, the training text consists of mixture of raw captions from Datacomp-1B, with a specified proportion and the rest of the captions replaced by refined captions from Recap- Datacomp-IB. Moreover, the training batch size is 2048, and the A optimizer (Loshchilov Hutter, 2017) is used with constant le-4 learning rate, without any warm- up schedule or weight decay. We name the resulting model Recap-DiT.
the raw COCO captions. Here, we observe consistent and significant improvements in both FID and CLIIP scores, par-
ticularly when more than half of the recaptioned data are integrated into the training dataset. Notably, models trained on Recap-Datacomp- IB (p = 0) surpass those trained on the vanilla Datacomp- 1B (p = 1) by a large margin, with improvements observed FID (-8.4), CLIP score (+3.1), Recap- score (+8.4), and GPT-4V score (+1.1). These observations justify that Recap-Datacomy IB better reveals the potential of text-to-image models in generating images with high visual quality and improved alignment with tex- tual conditions. The comparison between p 1.0 and p O is to illustrate that recap data can provide richer semantic information. Overall, p=0.1 I achieves the best performance across metrics. This means that mixing the original caption with the recaption gives the best results!
Evaluation. For sampling, we set the classifier-free guid- ance scale as 10 and use 250 DDPM steps to generate 30k images with captions from MSCOCO and our improved generated captions for zero-shot generation evaluation. Raw means evaluating with original coco captions, Our COCO- Recap means evaluating with recapted COCO captions. We calculate Fréchet Inception Distance (FID) (Heusel et al., 2017) with the reference images from MSCOCO (Lin et al., 2014) and CLIP score with both OpenAl ViT-B/32 model (Radford et al., 2021a) and our own Recap-CLIPP ViT-L/16 model, following the established pipeline in prior T2I works (Betker al., 2023; Yu al., 2022b; Sauer et al., 2023a; Kang al., 2023; Liu et al., 2023c; Zhou et al., 2024; Sauer et al., 2023b). Additionally, following the GPT-4V metric introduced in Section 4.2, we randomly select sub- set of 3,000 our generated images for GPT-4V evaluation.
Larger models. We further train larger model, DiT-L/2, for epoch with mixed ratio of = 0.0, while keeping other training parameters unchanged. The model achieves an FID of 25.1 14 and a CLIP Score of 34.82. In Figure 6 (see appendix), we visually compare the generated results of DiT- L/2 and DiT-B/4 at p = 0.0. is evident that although the quantitative scores may not show substantial improvement, scale up the model, there is noticeable enhance- ment in the alignment between the generated images and the corresponding text, i.e., this improved alignment results in higher-quality images that are able to capture and express more intricate details. These results confirm that DiT mod- els trained on our recaption DataComp- IB exhibit robust scalability for text-to-image generative tasks.
Main results. We report our observations in Tab. 6. Inter- estingly, when using raw COCO captions to generate 30,000 images for evaluation, the model trained with data integrated with our -atacopp (for p < 1) demonstrates better CLIP score, indicating improved vision-language alignment. However, there is no significant improvement observed in terms of FID. Our hypothesis is that the model adapts to the more informative and descriptive prompts, and could un- leash its full potential only when similar informative testing prompts are provided.
7. Conclusion
This paper introduces image dataset paired with detailed textual descriptions, gen- erated using the LLaMA-3-poweree LLaVA model. Our comprehensive analysis reveals that, compared to the origi- nal, web-crawled textual data, these generated descriptions align more accurately with their corresponding images and are more detailed. Utilizing Recap-DataComp-11 1B for train-
BB a large-scale
Therefore, in another setting, we evaluate images generated using our LLa 1,5A-1,5-LLAA 1,5-LLLMA3 ----8BBB 8B recaptioned version of
8

---

What If We Recaption Billions of Web Images with LLaMA-3?
ing resulted in consistent enhancements across various mod- els, notably CLIP, particularly in image-to-text and text-to- image retrieval tasks, and in text-to-image Diffusion models, specifically in their ability to follow more closely to user- provided text instructions. By providing this high-quality, publicly available, large-scale image-text dataset, we hope to inspire ongoing research and development that will push the boundaries of developing vision-language foundation models, more particularly in the open-source community.
M., et al. Flamingo: visual language model for few-shot learning. In NeurlPS, 2022.
Awadalla, A., Xue, L., Lo. O., Shu, M., Lee, H.,, Guha, E. K., Jordan, M., Shen, S., Awadalla, M., Savarese, S., et al. Mint- It: Scaling open-source multimodal data by 10x: A multimodal dataset with one trillion tokens. arXiv preprint arXiv:24066 11271, 2024.
Bai, J., Bai, S., Yang. S., Wang. S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, Qwen- VL: Versatile Vision- Language Model for Understanding, Localization, Text Reading, and Beyond. arXiv preprint arXiv:2308.12966 2023.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, one of it can be dataset bias. We selected Datacomp-1B 1B as the primary image source. Al- though Datacomp is derived from Common Crawl and rep- resents a snapshot of data in the public internet, it inevitably includes noisy and potentially unsafe content. The dataset's curation involves rigorous filtering, including NSFW filters on images and text, face blurring to reduce identity risks, and a take-down policy for disputed content. While these measures enhance safety, some offensive content may still persist due to the limitations of model-based filtering. We adhered to the Datacomp downloading and preprocessing procedures and only release the URLs for downloading. While using original captions may directly introduce offen- sive content, generating captions solely from images can also lead to unpredictable and potentially offensive model behavior, highlighting an ongoing concern that warrants further investigation.
Betker, J., Goh, G., Jing, L.., Brooks, T., Wang. J., Li, L., Ouyang, L.., Zhuang, J., Lee, J., Guo, Y., et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dallle: pdf, 2023.
Changpinyo, S., Sharma, P., Ding, N., and Soricut, R. Con- ceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail tail visual concepts. In CVPR, 2021.
Chen, J., Ge, C.,, Xie, E., Wu, Y., Yao, L., Ren, X., Wang, Z., Luo, P., Lu, H., and Li, Z. Pixart- fixart- Weak-to-strong training of diffusion transformer for 4k text-to-image generation. arXiv preprint 24X6x2240..0669, 2024a.
Chen, J., YU, GE, C.,, Yao, L.., Xie, E., Wang, Z., Kwok, J., Luo, P., Lu, H., and Li, Z. Pixart-$ allphas: Fast training of diffusion transformer for photorealistic text- to-image synthesis. In ICLR, 2024b.
Chen, I.., Li, J., Dong, X., Zhang, P., He, C., Wang. J., Zhao, F., and Lin, D. Sharegpt4v: Improving large multi-mode models with better captions. arXiv preprint 2rXiv:2311.1276) 2023a.
Acknowledge
This work is partially supported by a gift from Adobe, TPU Research Cloud (TRC) program, Google Cloud Research Credits program, AWS Cloud Credit for Research program, Edinburgh International Data Facility (EIDF) and the Data- Driven Innovation Programme at the University of Edin- burgh.
Chen, X. and Wang, X. Pali: Scaling language-image learn- ing in 100+ languages. In NeurlPS, 2022.
Chen, Z., WU, J., Wang, W., Su, W., Chen, G., Xing. S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., Li, B., Luo, P., Lu, T., Qiao, Y., and Dai, J. Internvl: Scaling up vision foundation models and aligning for generic visual- linguistic tasks. arXiv preprint arXiv:23 14238, 2023b.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023a.
Chiang, W.-L.., Zheng, L.,, Sheng, Y., Angelopoulos, A. N., Li, T., D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J. E., and Stoica, I. Chatbot arena: An open platform for evaluating 1lms by human preference. arXiv preprint arXiv:2403.0m 132,,041122200 2024.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.,, Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. GPT-oV(ision) system card. OpenAl Research Blog. 2023b.
Chu, X., Qiao, L., Zhang, X., Xu, S., Wei, F., Yang, Y., Sun, X., Hu, Y., Lin, X., Zhang, B., et al. Mobilevim v2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03766. 2024.
Alayrac J.-B., Donahue, J., Lue, P., Miech, A., Barr, I., Hasson, Y., Lene, K., Mensch, A., Millican, K., Reynolds,

---

What If We Recaption Billions of Web Images with LLaMA-3?
Deng, J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
10.5281//emmddo.514377. URLhttpa http::://doooooooo
10
Kang. M., Zhu, J.-Y., Zhang, R., Park, J., Shechtman, E., Paris, S., and Park, T. Scaling up GANs for text-to-image synthesis. In CVPR, 2023.
Desai, K., Kaul, G., Aysola, Z., and Johnson, J. Redcaps: Web-curated image-text data created by the people, for the people. arXiv preprint arXiv:2111.1142
2021.
Karpathy, A. and Fei-Fei, L. Deep visual- semantic align- ments for generating image descriptions. In CVPR, 2015,
Ding, M., Yang, Z., Hong, W., Zheng, W., Zhou, C., Yin, D., Lin, J., Zou, X., Shao, Z., Yang, H., et al. Cogview: Mastering text-to-image generation via transformers. In NeurlPS, 2021.
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J,, Shamma, D. A., al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. In IJC,, 2017.
Fan, L., Krishnan, D., Isola, P., Katabi, D., and Tian, Y. Im- proving clip training with language rewrites. In NewrIPS, 2024.
Lee, T., Mai, Y., Wong, C. H., Roberts, J. S., Yasunaga, M., Kaiyom, F., Bommasani, R., and Liang, P. The first steps holistic evaluation of vision-language models, May 2024.
Fang, A., Jose, A. M., Jain, A., Schmidt, L., Toshev, A., and Shankar, V. Data filtering networks. arXiv preprint arXiv:2309. 17425, 2023.
Fei, Z., Fan, M., Yu, C., Li, D., Zhang, Y., and Huang. Dimba: Transformer-mambe diffusion models. arXiv preprint arXiv:2406.0113: 01159, 2024.
Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language-iman image pre-training for unified vision- -anguale understanding and generation. In /CML, 2022.
Gadre, S. Y., Ilharco, G., Fang, A., Hayase, J., Smyriis, G., Nguyen, T., Marten, R., Wortsman, M., Ghosh, D., Zhang, J., et al. Datacomp: In search of the next generation of multimodal datasets. arXiv preprint arXiv:2304. /4 108, 2023.
Li, Li, D., Savarese, S., and Hoi, Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023a.
Li, X., Wang, Z., and Xie, C. An inverse scaling law for clip training. In NewrIPS, 2023b.
Gemma Team. Gemma 2: Improving open language models at practical size. ArXiv, abs/2408.0011 18, 2024.
Li, X., Wang, Z., and Xie, C. Clipa-v2: Scaling clip training with arxiv preprint arXiv:2306. 156588 2023c.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by two time-scale update rule converge to local nash equilibrium. In NeurIPS, 2017.
Lin, B., Tang, Z., Ye, Y., Cui, J., Zhu, B., Jin, P., Huang, J., Zhang. J., Ning, M., and Yuan, L. Moe-Ilava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401. 249..4199.977 2024.
Hinck, M., Olson, M. L., and Lal, V. Intel/lava-llama-3- 8b: Llava-v1.5 fine-tuned meta-llama-3-Sh multimodal model. URL tps: / //hugginggn ngfa cece co/ Intel/ 1lava alama- 3- 8b. Intel Research Use Licence. Accessed 2025-05-29.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Huang. H., Peng. S., Zhang, D., and Geiger, Renovat- ing names in vocn-vocabulary segmentation benchmarks. arXiv preprint arXiv:2403.09599, 2024.
Liu, H., Li, C.,, Li, Y., and Lee, Y. Improved baselines with visual instruction tuning. In NewrlPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023a
Hui, M., Yang, S., Zhao, B., Shi, Y., Wang, H., Wang, P., Zhou, Y., and Xie, C. Hq-edit: A high-quality dataset for instruction- based image editing. arXiv preprint a4Xiv:2404. 09990, 2024.
Liu, H., Li, C.. Wu, Q., and Lee, Y. Visual instruction tuning. arXiv preprint arXiv:2304. 08485, 2023b.
Ilharco, G., Wortsman, M. Wightman, R., Gordon, C. Carlini, N., Taori, R. Dave, A., Shankar, V., Namkoong, H., Miller, J., Hajishirzi, H., Farhadi, A., and Schmidt, L.. Openclip. github, July 2021. doi:
Liu, H., Li, Li, Y., Y., Li, B., Zhang, Y., Shen, S., and Lec, J. Llava-next: Improved reasoning, ocr, and world knowledge. https://laaa..giiihubb.... wtw.iihhbbbioblog/024000002 Ilava-next/, January 2024.
10

---

What If We Recaption Billions of Web Images with LLaMA-3?
Liu, X., Zhang, X., Ma, J., Peng, J., and Liu, Q. Instaflow: One step is enough for high-quality diffusion-based text- to-image generation. arXiv preprint arXiv:2309.06380, 2023c.
chine Learning Research, pp. 8748-8763. PMLR, 18- 24 Jul 2021a. URL htw.::ttp////t.tmm https::/proceeddnnggg .mlr. press/ /atss/vv139 rawford2aa...at..... h
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., al. Learning transferable visual models from natural language supervision. In ICML, 2021b.
Liu, Y. Wang, K., Shao, W., Luo, P., Qiao, Y., Shou, M. Z., Zhang, K., and You, Y. Milms-augmented visual- representation learning. arXiv preprint arXiv:2311. 2023d.
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C.,, Rad- ford, A., Chen, M., and Sutskever, Zero-shot text- to-image generation. In International Conference on Machine Learning, pp. 8821-8831, PMLR, 2021.
Loshchilov, I. and Hutter, E, Decoupled weight decay regu- larization. arXiv preprint arXiv: /71 1.05101, 2017.
Lu, G., Guo, Y., Han, J.. Niu, M., Zeng, Y., Xu, S., Huang. Z., Zhong, Z., Zhang, W., and Xu, H. Pangu- draw: Advancing resource-efficient text-to-image syn- thesis with dime-decoupled training and reusable coop- diffusion. arXiv preprint arXiv:2312. 16486, 2023.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents, 2022. URL https: //tp//aavvvv////////////////aaaa orgg /abs/ 2204 06125.
Meta LLLMMA Team. Introducing Meta Llama 3: The most capable openly available LLM to date, 2024.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models, 2021.
Nguyen, T., Gadre, Y., Ilharco, G., Oh, S., and Schmidt, L. Improving multimodal datasets with image captioning. In NeurlPS, 2024.
Rotstein, N., Bensaid, D., Brody, S., Ganz, R., and Kimmel, R. Fusecap: Leveraging large language models to fuse visual data into enriched image captions. arXiv preprint arXiv:2305.17718, 2023.
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I, and Chen, M. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741, 2021.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):11-255, 2015. doi:
OpenAI. Dall-e 3 system card. OpenAl Research Blog, 2023.
10. 10.1007/s1 0ttp:/1112260001--81818
OpenAI. Video generation models world simulators.
Saharia, C., Chan, W., Saxena, S., Li, L., Whang. J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to-image dif- fusion models with deep language understanding. In NeurIPS, 2022.
OpenAl Research Blog, 2024.
Ordonez, V., Kulkarni, G., and Berg, T. 1m2text: Describ- ing images using million captioned photographs. In NeurlPS, 2011.
Sauer, A., Karras, T., Laine, S., Geiger, A., and Aila, T. StyleGAN-T: Unlocking the power of GANs for fast large-scale text-to-image synthesis. arXiv preprint arXiv:2301.09515, 2023a.
Padlewski. P., Bain, M., Henderson, M., Zhu, Z., Relan, N., Pham, H., Ong, D., Aleksiev, K., Ormazabal, A., Phua, et al. Vibe-eval: A hard evaluation suite for measuring progress of multimodal language models. arXiv preprint 0tti:220550028877 2024.
Sauer, A., Lorenz, D., Blattmann, A., and Rombach, Adversarial diffusion distillation. ArXiv, abs/2311.1704 2023b.
Peebles. W. and Xie, S. Scalable diffusion models with transformers. In ICCV, 2023.
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk,
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language su- pervision. In Meila, M. and Zhang, T. (eds.), Pro- ceedings of the 38th International Conference on Ma- chine Learning, volume 139 of Proceedings of Ma-
R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A. Laion-400m: Open dataset of clip- filtered 400 million image-text pairs. arXiv preprint arXi::2111.02114 2021.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, Katta, A., Mullis,

---

What If We Recaption Billions of Web Images with LLaMA-3?
C., Wortsman, M., Schramowski, P., Kundurthy, S., Crow- son, K., Schmidt, L.., Kaczmarezyk, R., and Jitsev, J. Laion-5b: An open large-scale dataset for training next generation image-text models, 2022a.
Yu, Q. Sun, Q. Zhang, X., Cui, Y., Zhang, F, Wang, X., and Liu, J. Capsfusion: Rethinking image-text data scale. arXiv preprint 1rXiv:2310.205550 2023a.
Yu, T., Yao, Y., Zhang, H., He, T., Han, Y., Cui, G., Hu, J., Liu, Z., Zheng, H.-T., Sun, M., al. Rlhf-v: To- wards trustworthy milms via behavior alignment from fine- fine-grained correctional human feedback. arXiv preprint arXiv:2312.. 22.00999, 2023b.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35: 25278-25299 2022b.
Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., and Wang, L. Mm-vet: Evaluating large multimodal models for integrated capabilities. In ICML, 2024.
Sharma, P., Ding, N., Goodman, S., and Soricut, R. Con- ceptual captions: cleaned. hypernymed, image alt-text dataset for automatic image captioning. In ACI. 2018.
Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., Wei, C., Yu, B.. Yuan, R., Sun, R., Yin, M., Zheng, B., Yang, Z., Liu, Y., Huang, W., Sun, H., Su, Y., and Chen, W. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In CVPR, 2024.
Srinivasan, K., Raman, K., Chen, J., Bendersky, M., and Najork, M. Wit: skripedia-based image text dataset for multimodal multilingual machine learning. In SIGIR, 2021.
Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D., and Zou, J. When and why vision-language models behave like bags-of-words. and what to do about it? In ICLR, 2022.
Stevens, I. Llama 3's performance benchmark values ex- plained, 2024. Accessed: 2024-06-05.
Sun, Z., Shen, S., Cao, S., Liu, H., Li, C., Shen, Y., Gan, C., Gui, L.-Y., Wang, Y.-X., Yang. Y., al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint ar09 14525, 2023.
Zhai, X., Wang, X., Mustafa. B., Steiner, A., Keysers, D., Kolesnikov, A., and Beyer, L. Lit: Zero-shot transfer with locked-image text tuning. In CVPR, 2022.
Wang, Z. Yu, J., Yu, A. W., Dai, Z., Tsvetkov, Y., and Cao, Y. Simvlm: Simple visual language model pretraining with weak supervision. In ICIR, 2022.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. In ICCV, 2023.
Xu, H., Xie, S., Tan, X., Huang, P.-Y., Howes, R., Sharma, V., Li, S.-W., Ghosh, G., Zettlemoyer, L., and Feichten- hofer, C. Demystifying clip data. In ICLR, 2023.
Zhang, B., Zhang, P., Dong, X., Zang, Y., and Wang. J. Long-clip: Unlocking the long-text capability of clip. arXiv preprint arXiv:2403. 15378, 2024a.
Xu, R., Yao, Y., Guo, Z. Cui, J., Ni, Z., Ge, C., Chua, T.S., Liu, Z., and Huang, G. LLaVA-UHD: an Imm perceiving any aspect ratio and high-resolution images. arXiv preprint arXiv:2403.1 1/0.0 2024.
Zhang, L., Shu, F., Ren, S., Zhao, B., Jiang, H., and Xie, C. Compress & align: Curating image-text data with human knowledge. arXiv preprint 2rXiv:2312.06722 2023.
Zhang. Y., Unell, A., Wang, X., Ghosh, D., Su, Y., Schmidt, L., and Yeung-Levy, S. Why are visually-grounded lan- guage models bad at image classification? arXiv preprint arXiv:2405. 18415, 2024b.
Young, P., Lai, A., Hodosh, M., and Hockenmaier, J. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. In TACL, 2014.
Zheng, K., Zhang. Y., Wu, W., Lu, F., Ma, S., Jin, X., Chen, W., and Shen, Y. Dreamlip: Language-imas -mage pre-training with long captions. In ECCV, 2024.
Yu, J., Xu, Y., Koh, J. Y., Luong, T., Baid, G., Wang. Z., Vasudevan, V., Ku, A., Yang, Y., Ayan, B. K., Hutchinson, B., Han, W. Parekh, Z., Li, X., Zhang, H., Baldridge, J., and Wu. Y. Scaling autoregressive models for content- rich text-to-image generation, 2022a.
Zhou, M.. Wang. Z., Zheng, H., and Huang. H. Long and short guidance in score identity distillation for one-step texa-to-image generation. arXiv preprint arXiv:2406.01561, 2024.
Yu, J., Xu, Y., Koh, J. Y., Luong, T., Baid, G., Wang. Z., Vasudevan, V., Ku, A., Yang, Y., Ayan, B. K., et al. Scal- ing autoregressive models for content-rich text-to-image generation. In TMLR, 2022b.
12

---

What If We Recaption Billions of Web Images with LLaMA-3?
Original: Recap: 4
Original: Recap:
Michelle & Karl The Granary Newmarket 66
Christmas Ugly Golf Sweater Burgundy T- Shirt Front
Barns.
group of people gathered outside building with red roof, surrounded lush green lawn and variety plants. The sky is partly cloudy, and the building appears be mix of modern and traditional architecture.
maroon sweatshirt with festive holiday design featuring golfer in red sweater and white pants, holding golf club, surrounded by snowflakes, pine trees, and musical notes. The sweatshirt has ribbed collar and cuffs, and the design is centered on the chest.
Original: Recap: 4
Original: Recap:
1963 Chevrolet
Corvette Resto Mod
Ricoh SP 325SFNw, Mono Laser Printer
3 Print Image
sleek black sports car a shiny finish license plate that reads parked on a dirt road. car has low profile, pointed front, distinctive rear design with taillights. The road surrounded by desert landscape sparse vegetation and wooden fence background.
white and black modern laser printer with flatbed scanner on top, featuring control panel paper tray, is displayed against white background.
Original:
Original: Recap:
Recap:
BOD
ProSupps Mr. Hydc NitroX Pre- Workout Powder Energy Drink Intense Sustained Energy. Pumps & Focus with Beta Alanine, Creatine Nitrosigine
Valerij Ivanovich Sosna. Sunflowers
Servings
still life painting featuring bouquet of sunflowers in vase, with textured, brushstroke style that gives the image an impressionistic feel. The sunflowers are depicted in various stages of bloom, with some petals open and others closed. The vase is placed on a surface with a patterned cloth, and there are additional items such as a bowl, small green fruit, and red cloth. background a muted yellow, the overall color pulette is dominated by greens, greens, browns.
A close-up of red and silver canister with the brand name Candy Shoppe' and the product name 'NitroX', The canister has textured surface with a pattern of red and silver lines. The background gradient of to silver, there are no other objects text visible.
Original: I Recup:
Original: Recap:
Udemy 7 Steps to Perfectly Written Business Plan
Grunge texture Stock Image
with bald heads, wearing white t-shirt and the other dark i-shirt. They both smiling and looking directly at the viewer. Above them stylized image of notebook with pencil on it, surrounded by various icons such as lightbulb, magnifying glass, and speech bubble. The background is solid yellow color.
digital illustration
close-up of textured blue surface with orange and white speckles and patches. The surface has rough, almost peeling appearance, with some areas looking like they lifting off. The blue tone dominant, but there areas where the orange and white speckles are mure prominent.
Figure 4. Examples of the original caption and recaption in dataset. Blue box: reepation results. Orange box: human-eval ratings.
13

---

What If We Recaption Billions of Web Images with LLaMA-3?
Original: Recap:
Original:
Recap:
Thinlshe ISadd tstaluiier Scalling Regaiia 2018cbt (iicchr iotalo purchase ploto, see pricing infirmation for Prints and Dewnloods, click the blue 'Add Cart the
The King 1xc2x880x99
Sturring 2000
Rolls Royce 9r22 coming Soons
vintage car
design is parked on inside, one playing
of
The car's window reveals guitar and the other holding ukulele. The playing the guitar is wearing hat and glasses, while the with the ukulele wearing sunglasses. The car's body is light color with chrome bumper and white wall tire visible in the foreground.
person is rowing boat on calm lake. individual wearing red top and black shorts, and seated boat with in hand. The boat moving through the water, leaving trail behind it. The lake is surrounded by grassy area, and there is clear sky above.
Original: Recop:
Original: Recop: 3
BIGSTOCK
Devolpment Helmet voctor icon. Desekpmeet Helant syablel. Development image, Development
Predator 19.3 Firm Ground
Cleats Core Black / Core Black/ / Active Red D98003
picture. Development Heimet pictogramm Flat yellow developmcnt helnet icun.
close-up of black rubber sneaker with textured sole and white toe cap. The sneaker is positioned against white background, and the image is in portroit orientation.
A stylized yellow and blue illustration construction helmet with the word BIGSTOCK written in circular pattern around it. The helmet is centered on blue background, and the image has watermark with the text BIGSTOCK and the numbers "1234567890" in the bottom right corner.
Fuda brick machine
CE
Original: 3
Original:
Recap:
Recap:
Lego Clay Brick Molding Machine Interlocking Solid Block Machine pictures photos
Tour Aux Jambes by Jean Dubuffet
man operating a yellow brick-making setting. The machine large and has a conveyor There are several potted plants around the area, factory modern look with clear blue sky visible through windows.
a factory bricks
modern sculpture with textured surface variety of colors, including blue, white, displayed on shelf in with reflective floor and large window.
Original: Recop:
Original:
Recap:
Marwl Jaipuri Floral Print Pure Cotton Double Bed Quiit Jaipuri
large bathroom with tub
Razai
A white blanket with intricate blue and gold patterns is neatly folded on a white background.
A modern bathroom with white tiled walls and floor. There white sink with a chrome faucet to the left, white bathtub with chrome faucel and showerhead the center, and white toilet to the right. Above sink, there large, round, clear glass pendont light.
left of the sink, there is a basket on the floor. The bathroom has large window are various toiletries towels neatly arranged shelves above the bathtub.
curtains,
Figure 5, Examples of the original caption and recaption in dataset. Blue box: recpation results. Orange box: human-eval ratings.
14

---

What If We Recaption Billions of Web Images with LLaMA-3?
Table Recap- model configurations used in our paper.
Embed dim
Vision Transformer
Text Transformer width
# params (M)
model
layers
width
heads
layers
heads
vision
text
total
S/16
384
12
384
6
12
384
22
33
55
B/16
512
12
768
12
12
512
86
53
141
L/16
768
24
1024
16
12
768
12
303
109
414
H/14
1024
32
1280
16
24
1024
16
631
334
967
A. GPT-4V & Human Evaluation
We first provide two demos of detailed explanation our GPT-4V evaluation metrics shown in Sec 4.2 of the main text. The images are shown in Figure 4, first row. Following are GPT-4V's explanations. is clear that GPT-4V prefers Recap captions for their accuracy and detailed descriptions, capturing visual details the original captions miss. Recap excels in descriptive richness and contextual accuracy.
Example 1: Figure first row, left
Recap: A group of people are gathered outside a building with a red roof, surrounded by a lush green lawn and a variety of plants. The sky partly cloudy, and the building appears to mix of modern and traditional architecture.
GPT-4V Rating: 4
Explanation:
The given caption is mostly accurate and fluent, hence the high rating. It accurately describes group of people gathered outside building, which indeed has red roof as mentioned. The lawn is green and well-maintained, matching the description of a "lush green lawn." There are also plants visible, corresponding with the mention of "a variety of plants." The sky the image is partly cloudy, consistent with caption. The building exhibits blend of materials and design elements which could be seen as mix of modern and traditional architecture especially with the modern glass elements juxtaposed with traditional stone walls.
Original: Michelle & Karl The Granary Barns, Newmarket 66
GPT-4V Rating:
Explanation:
The image shows wide, outdoor view with several guests apparently an event, possibly a wedding or gathering, by large stone building with red-tiled roof. There do not appear to be individuals who are prominently featured as the main subjects that can be clearly identified as Michelle and Karl. Without distinguishing features or a focus on specific individuals that could correlate to names provided, caption fails to correctly match the image and does not provide an accurate description of the visible content. The setting does appear to be a barnlike venue which could align with "The Granary Barns," but without clearer context focus the individuals named, the overall caption quality is low for the purposes of image description.
Example 2: Figure 1, first row, right
Recap: maroon sweatshirt with a festive holiday design featuring golfer in a red sweater and white pants, holding golf club, surrounded by snowflakes, pine trees, and musical notes. The sweatshirt has ribbed collar and cuffs, and the design is centered the chest.
GPT-4V Rating: 5
15

---

What If We Recaption Billions of Web Images with LLaMA-3?
Explanation:
The caption accurately describes the image provided. The sweatshirt indeed maroon and features festive holiday design. The description of the golfer wearing red sweater and white pants matches the figure the sweatshirt, and the surrounding details of snowflakes, pine trees, and musical notes are all visibly present in the design. Additionally, the mention of ribbed collar and cuffs is confirmed by the image. The design is also correctly described as being centered on the chest. The caption is fluently written and provides a detailed and accurate portrayal of the sweatshirt's design.
Original:
Christmas Ugly Golf Sweater Burgundy T-Shirt Front
GPT-4V Rating: 2
Explanation:
The caption describes the item as "Christmas Ugly Golf Sweater Burgundy T-Shirt," which is partially correct context. The image shows a burgundy-colored Christmas sweater with golf theme, which fits part of the caption's s description. However, the description has inconsistencies: the garment is a sweater, not a T-shirt, and while the term "ugly" is subjective and commonly used to describe specific style of intentionally gaudy Christmas sweaters, it's difficult to assess the accuracy of that term without an understanding of personal taste. The significant error in identifying the type of garment warrants the lower rating.
To validate GPT-4V's ratings, we conducted human evaluation of our captions. We randomly selected 200 images from the Datacomp- IB dataset and asked human evaluators to rate both the recaptioned and original captions. To ensure objectivity, we followed a double-blind procedure, where annotators were unaware of the captions' origins. The human rating criteria are detailed below. Note that we emphasize the appearance of objects in the image as one our key criteria, allowing us to assess the level of hallucination in the response, as suggested in the review.
1. The caption does not make sense and is completely irrelevant to the given image.
2. The caption is readable but irrelevant to the given image.
3. The caption is readable and partially related the image, but it may contain contents that shown in the image.
4. The caption is fluent and related to the image, but it may contain contents that are not shown in the image.
5. The caption is fluent and exactly describes the image.
The average human evaluation results (4.3 vs. 3.1) closely align with GPT-4V's ratings, supporting our assessment of caption quality, particularly in terms of alignment.
B. Training cost
We benchmark the inference speed of LLaVA-L.-LLLMMAS on the TPU-V4 256 hardware, achieving throughput of 382 images per second. At this rate, generating captions for the entire Recap-DataCompy dataset ( billion images) would take approximately 29 days of continuous computation. Regarding CLIP training, training a ViT-L/16 model for two epochs 2.56 billion total samples) on decap-DataComa 1B requires 1 day on TPU-V3 256 infrastructure. For DiT training, training a base-sized DiT model with a batch size of 2048 for 650K steps takes approximately 7 days using TPU-V3 256 hardware.
C. Limitation
We selected Datacomp-18 1B as the primary image source to evaluate the captioning capabilities of multimodal large language models (MLLMs) like LLaVA (Liu et al., 2023b) and to explore the scalability of our synthetic captions. Although Datacomp is derived from Common Crawl and represents snapshot of data in the public internet, it inevitably includes noisy and potentially unsafe content. The dataset's curation involves rigorous filtering, including NSFW filters on images and text, face blurring to reduce identity risks, and a take-down policy for disputed content. While these measures enhance safety, some offensive content may still persist due the limitations of model-based filtering. We adhered the Datacomp downloading
16

---

What If We Recaption Billions of Web Images with LLaMA-3?
and preprocessing procedures and only release the URLs for downloading. While using original captions may directly introduce offensive content, generating captions solely from images can also lead to unpredictable and potentially offensive model behavior, highlighting an ongoing concern that warrants further investigation.
Second, our two-stage LLaVA training involves 558k image-text pairs from LAION (Schuhmann et al.. 2022b). CC (Chang- pinyo et al., 2021), and SBU (Ordonez et al., 2011), as well as 665k instruction-folloviii data from LLaVA-1.5 (Liu et al., 2023a) and HQ-Edit (Hui et al., 2024). Although the visual instructional data are filtered, the visual encoder and LLM are pre-trained on massive internet-soured data, which may contain biases related to gender, race, or culture. Consequently, our caption model might inherit and perpetuate these biases. Moreover, since the model generates text probabilistically. may produce information that is not grounded in the image, leading to potential hallucinations and impacting caption quality like other vision-language models.
Moreover, in our CLIP experiments, we found inferior performance when soiely train the model on generated captions. Initially, we suspect that the model we used inherently lacks strong classification capabilities and is primarily a text- generation model. According to previous research (Zhang et al., 2024b), the training data is one of the key factors that enable MLLMs to enhance their classification ability. In our work, we focused more on improving the alignment between image and text rather than enhancing the model's 's classification capabilities. The training data we used did not include examples that are targeted to the classification task, which we leave the future exploration.
Additionally, the prompts used to generate captions were designed to describe the images as accurately as possible based on the visual content, but the caption model may lack the knowledge to tell specific named entities. For example, in the second row of the teaser on the left, our captioner describe the environment and background vividly but cannot infer the specific species of the bird. Therefore, accurately describing some specific named objects within an image is one of our limitations.
Finally, due to limited computational resources and the high inference costs associated with large models, our choice of captioning model was constrained (e.g., size, type).
D. License
We have divided our licensing details into two parts: the captioner model and the generated captions. Additionally, we distribute the original Datacomp-Il IB images and URLs as well as original captions accordance with their respective licenses.
LLL model and generated caption license: Since our LLaVA model based on LLaMA 3, we must strictly adhere to the original licensing terms. First, we will include copy of the LLaMA license agreement with the model. Users must comply with all terms and conditions outlined in these original licenses. For example, the generated captions are limited for further training or improving other large language models that are not based on Llama 3 (Meta LLaMA Team, 2024).
Image-url and original caption license: the Datacomp- IB distributes the image URL-text samples and metadata under a standard Creative Common CC-BY-4.0 license. Our improvements can be considered as derivative work of Datacomp- IB. Therefore, we will continue to use the CC BY 4.0 license to release, retain the attribution to the original author, and clearly state that the work is based on the original DataComp- IB dataset. We will include a clear attribution such as "This dataset based on "DataComp- created by (Gadre et al., 2023), licensed under CC BY 4.0.'
E. DiT Qualitative Results
F. Ablations on Model and Prompt Selections
We conduct experiments for ablating the language model, vision-language model, and the input prompt for recaptioning experiments. In detail, we choose two different language models and two vision-language model types for ablation. The detailed model recipe is below:
Language Model: LLaMA3-8B (Meta LLaMA Team, 2024), Gemma2-27B (Gemma Team, 2024).
Vision- LasoanLLnggauge Framework: LLaVA-1.5 (Liu et al., 2023a), LLaVA-NeXT (Liu et al., 2024).
17

---

What If We Recaption Billions of Web Images with LLaMA-3?
DIF-RA p-10
DIT-B/4 p-00
DIF-L/2 p=00
group elephants grazing in grassy field with bress background.
A girafTe stands lush green field with tall grass and trues background.
A block dog standing in a sunny backy ard with variety potted plants and flowers around it. The dog is weuring a blue collar and looking lowards the right side the image.
Two brown bears arv playfully splashing water at each other in river.
Figure 6. Visual comparison of generate results from DīT-B/4 at = 1.0 and = 0.0, and DīT-L/2 at = 0.0. We can see that decreasing p enables models to comprehend texts better in image generation, increasing model size B/4 to L/2 improves overall quality of generated images. We mark entities in the instruction.
As for the input prompts, we select four different ways to probe their influences to our recaption framework.
baseline: The vanilla prompt input in the paper that asks the model to generate a descriptive caption of the given image.
w/ Brief Prompt: We the prompt instruct the model to generate short and concise captions.
w/ Diverse Prompt: We construct prompt candidate pool with 11 diverse input prompts. Then we randomly sample one prompt for each captioning instruction.
condition on Ori. Cap.: We use the vanilla instruction but also add the original captions from the DataComp- IB to enlighten the model with annotated knowledge.
We showcase the detailed pormpts below:
Original Prompt: Please generate detailed caption of this image. Please be as descriptive as possible.
Concise Prompt: Please generate short and clear explanation of the image. Please be as concise as possible.
Condition Prompt: Please generate detailed caption of this image. Please as descriptive as possible based on original caption: [ORIGINAL DATACOMP CAPTION).
Diverse Prompts Pool:
1. Describe the image concisely as short possible.
2. Provide brief description of the given image.
3. Offer succinct explanation of the picture presented.
4. Summarize the visual content of the image.
5. Give a short and clear explanation of the subsequent image.
6. Share a concise interpretation the image provided.
18

---

What If We Recaption Billions of Web Images with LLaMA-3?
Orignal: Rondreis Cappaclocié Eftalia Aytur
Orignal: užywany BMW 216 Inny 1.9dm3 190KM 006km
LLaVA1.5 (LaMM3-88B):A1 A panoramic view of Cappadocia's unique landscape with towering rock formations and vibrant sunset sky.
LLaVA1.5 [LLaMA3-895
BMW parked on gravel surface. The car has license plate with the letters 'AAA and 'AUTO' on it. The car positioned in front of a wall with logo that reads'spray' and'spray Polish, The car's headlights are are visible.
L (LLaMA3-88) 8B) w/ Brief Promet: The image features stunning landscape
off, the rearview
mountain range formations. The sky above the mountains filled with dlouds, and the sun is setting, casting a warm glow on the scene. The colors of the sky range from blue to orange, creating beautiful sunset.
variety rock
LLaVA1.5 [LLaMA3-B5 88) w/ Brief Prompt: A black BMW car is parked on gravel surface. The car has a license plate and
sticker on the front, The license
on the front the car, and sticker located on the side. car appears to be a sodan with a sleok design.
LLaVA1.5 (LLaMA3-88) (3)-8) w/ Diverse Prompt: The image features breathtaking landscape of mountain range with a beauttful filled with mix blue
5 (LLaMA3-88) w/ Diverse Prompt: image festures black BMW car parked on a gravel surface. The car is positioned the center of the framo, with Its front facing the viewer. The license plate visible on the front of the car, and the car appears to luxury model. The background is very detailed, but seerss be parking area driveway.
the background. The and orange hues, creating stunning backdrap the scene. The mountains are covered in rack formations, with some of them resembling a desert-like terrain. The overall atmosphere is serene and picturesque, capturing the essence of nature's beauty,
LLVA115 [LaaMA3 88) condition on Ori. Cop.: A black BMW 216 with license plate that reads 'AA AUTO' is parked surface. car d'spray' logo
LLaVA1.5 5 (LLaMA3-88) 88) condition on Oni, Cop.: A scenic view of Cappadocia's 's unique rock formations under sky with a gradient of warm colors from
is positioned
front a wall with a white sticker that reads spray'. The car has sunroof, fog lights, and a rear spoiler.
LLaVA-NeXT [LLaMA3-88]+ tranquil sunset bathes the rugged
landscape of desert canyon, where the warm hues of orange and pink from setting contrast with the coal blues evening sky, The canyon walls, sculpted from red sandstone, majestically, their undulating forms softened by the gentle light. The sparse vegetation at the base of the diifss adds a touch of life the otherwise barren landscape. This image captures the timeless beauty desert canyon sunset, natural splendor where earth and sky meet display colors and contrasts.
image a promational photograph o BMW car, likely taken for the purpose of advertising or showcasing the vehicle for sale. The car's design and features, such as the headlight shape and grille, suggest is model from the early 2010s. The presence of "SPRZDZONE" watermark indicates that the image may used for anline sales or promotional material, likely by an auto deslership or a car sales platform. The text "AAA AUTO" on the license plate could imply that the car being sold by dealership with that with that name.
LLaVA-NeXt NeXT (LlaMA3-88):
of
or part of fleet company
LLaVA-Nex Ne--eeTT (GemmaZ-278) The image captivating landscape photograph that captures essence of desort enviranment at sunset. The rock formations are the central focus, with their varied shapes and sizes creating sense of depth and texture. The color palette is dominated by warm tones, with the rocks ranging from light beige to deep orange, reflecting the golden hues the setting sun. The sky above is gradient of orange and blue, with wispy clouds adding a soft contrast. The absence of any visible human elements or structures suggests remate, untauched landscape. This image could be used to evoke sense natural beauty, solitude, and the grandeur of geological formations.
LLaVA-NeXT (Gemma2- (eemma2-27B): The image promotional photograph BMW Series Touring car, which station
wagon model. The car is presented in a showroom setting, with a clean and uncluttered background that emphasizes the vehicle's design. The car's color is dark shade, passibly black very dark blue, which vehicles as it conveys sense sophistication The BMW logo on the grille clearly visible, indicating the brand and model of the car. The presence of the "AAA Auto' logo and the 'Sprzedawca"
choice for loury elegance.
text.
Figure 7. Examples of image captions on DataComp- IB generated by different models with various textual prompts.
7. Present a compact description of the photo's key features.
8. Relay a brief, clear account of the picture shown.
9. Render a clear and concise summary of the photo.
10. Write terse but informative summary of the picture.
11. Create compact narrative representing the image presented.
We present several examples of captions generated by different recaptioning systems in Figures 7 and 8. While the original captions from DataComp-ID IB offer a basic understanding of the images, our recaptioning models consistently produce more descriptive, fluent, and detailed information, particularly when leveraging larger and more advanced base language models or vision-language frameworks (e.g.. Gemma2-22 LLaVA-NeXT), Notably, models conditioned on the original captions from DataComp often generate even richer image descriptions, packed with additional knowledge. For instance, as shown on the right side of Figure 8, only VLa with the original caption, identifies pancakes with salmon on the top in the image. This highlights the benefits of integrating intricate details from the original image descriptions to enhance recaptioning performance.
19

---

What If We Recaption Billions of Web Images with LLaMA-3?
CanstocPhotb.coom
Orignal: Raket &amp; Distels Wreath of dried flowers and grasses
Orignal: Pancakes With Salmon Cooking pancakes with salmon
large
on...
LLaVA1.5 (LLaMA3-88): rustic floral wreath with mix of dried flowers, leaves, and various decorative items such as seashells and pine cones is placed on a marble surface. The wreath is circular in shape and has a natural, earthy color palette. The marble surface has warm beige with visible veins, and there furry, light-colored object in the background.
LLaVA1.5 (LLaMA3-8B): variety of dishes are displayed on dining table, including round omelette with golden-brown crust and patter of orange and white filling, plate of sushi rolls, a bowl of pickled ginger, plate of wasabi, and small dish of say sauce.
LLaVA1.5 (LLaMA3-88) w/ Brief Prompt: The image displays a variety of food items dining table. There are two plates of food, one with quiche and the other with pizza. In addition, there are bowls containing salad and a plate with sushi. The table also has plate of shrimp and bowl of sauce. The food is arranged in visually appealing manner, with a focus on the quiche and pizza.
LLavA1.5 (LLaMA3-88) w/ Brief Prompt: A large, circular wreath made of various flowers and other natural elements is placed on a marble table. The wreath is adorned with a mix of flowers, leaves, and other decorative items, creating a rustic and charming appearance.
LLaVA1.5 (LLaMA3-88) w/ Diverse Prompt: The image displays table filled with variety of food items, including a large plate of sushi, plate of eggs, and a plate of meat. There are also bowls of food, such as bowl of sauce and bowl of salad. The table is set with multiple plates, bowls, and spoon, indicating meal is being prepared or served. The variety of dishes suggests a diverse and delicious meal is being enjoyed.
LLaVA1.5 (LLaMA3-8B) w/ Diverse Prompt: The image features large, circular wreath made of flowers and other natural elements, such as leaves and twigs, placed on a marble table, The wreath quite large, occupying significant portion of the table's surface. The table itself is white, and the wreath is positioned in the center, creating beautiful and eye-catching display.
LLavA1.5 (LLaMA3-88) condition on Ori. Cap.: A variety of dishes are displayed on a table, including a plate of pancakes with salmon top, a plate of sushi, plate of dumplings, plate of sushi rolls, plate of sushi nigiri, plate of sushi sashimi, plate of sushi maki, plate of sushi nigiri, a plate of sushi sashimi, a plate sushi maki, plate of sushi nigiri, a plate of sushi sashimi, plate of sushi maki, plate of sushi nigiri, a plate of sushi sashimi, plate of sushi maki.
LLaVA1.5 (LLaMA3-88) condition on Ori. Cap.: A rustic wreath made of dried flowers and grasses is placed on a marble surface. The wreath is adorned with various dried flowers, leaves, and twigs, creating natural and earthy appearance. The marble surface has warm tone, and the wreath positioned in the center, filling the frame.
LLaVA-NeXT (LLaMA3-8B): The image presents natural, rustic wreath that could be used for home decor or as seasonal centerpiece. The choice of materials suggests a focus on bustainability and the use of natural elements, The variety of colors and textures, from the softness of the flowers the roughness of the stones, adds to the organic aesthetic. The placement of the wreath on a marble surface could indicate a setting that values both natural beauty and refined design. The image could be used to illustrate concepts such as nature- inspired decor, the beauty of imperfection, or the art of floral arrangements.
LLaVA-NeXT (LLaMA3-8B): The image culinary display that showcases variety of dishes, likely intended for meal or buffet. The arrangement is symmetrical, with each dish placed a way that creates a balanced composition. The use of white plates and a white tablecloth suggests clean and minimalist aesthetic, which is often associated with fine dining or special occasion. The presence of both sweet and savory items indicates variety of options, catering to different tastes. The image could be used for culinary presentation, menu design, or as a visual reference for food styling. It's important to note that the image a representation food.
LLaVA-NeXT (Gemma2-27B)) The image captures wreath that appears to be crafted from natural materials, possibly foraged from the environment. The wreath's composition includes a variety of elements such as dried flowers, seed pods, and grasses, which arranged in circular pattern. The color palette is predominantly earthy tones, with the dried flowers adding pops of white and yellow. The wreath is placed on marble surface, which provides stark contrast to the organic materials of the wreath. The background is blurred, suggesting that the focus is solely on the wreath, which is the main subject of the image.
LLaVA-NeXT (Gemma2-278)) The image is a collection of various food items, likely from a buffet or a shared meal setting. The dishes include savory pie, quiche, and dessert, each with its own distinct characteristics. The pie has a golden-brown brown crust, suggesting it's been baked to a crisp, while the quiche is creamy and golden, indicating a rich egg and cheese filling. The dessert is light and airy pastry, likely filled with sweet cream or custard. The presentation is simple yet elegant, with the dishes arranged in a way that highlights the variety of textures and colors.
Figure 8, Examples of image captions on DataComp- 1B generated by different models with various textual prompts.
20


Reviews:
[{"review_id": "ZavRLZmClL", "reviewer": "Reviewer_Manh", "summary": "The paper introduces Recap-Datacomp-1B a large scale image-text data, where the text is \"synthetically generated\" using large multimodal models. The authors observed that compared to original web-crawled data, models trained on this generated synthetic texts are performing better for multimodal retrieval and  text-to-image generation tasks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing that our dataset significantly enhances performance in both multimodal retrieval and text-to-image generation tasks. We appreciate your insights into how advanced MLLMs can reduce reliance on costly human annotations and enable greater scalability.\nOur Recap data relies solely on LLaVA-based annotation, which **inherently incorporates commonsense knowledge** from a human perspective — since LLaVA is an aligned model trained on human preference data. In other words, our Recap approach not only reduces dependence on manual annotation but also opens up promising directions for future research.\nOur experiments further show that mixed-caption training — integrating both CLIP and T2I methods — offers a robust way to incorporate **external** commonsense knowledge, leading to consistent performance improvement.\n\nIn essence, our dataset serves as a valuable baseline and starting point for exploring these innovative strategies. We plan to expand on the broader impact of this work in the upcoming version.\n\n---\n\n### **Q1: Why not use earlier LLaVa-LLaMa3 models?**\n\n**A1:** Thank you for bringing up this work. First, we trained our own version to maintain full control over the training process while adhering to open-research practices and a broader licensing framework. Second, our initial investigation revealed that the original LLaVA-1.5 data consistently started its outputs with \"This image is a …\". To address this issue, we integrated a high-quality dataset (HQ-Edit) into our training process, which alleviated the problem. We will include this essential reference in our next version.\n\n---\n\n\n### **Q2: Technical innovation.**\n\n**A2:** We would like to stress that our primary focus is on creating **Recap-DataComp-1B**, which, to the best of our knowledge, is **the first publicly available image-text dataset with synthetic captions scaled to the billion level using LLaMA-3**. We believe this represents a novel and significant contribution to the multimodal research community. While the concept of recaptioning is not new, scaling it to this magnitude with advanced LLMs has not been seen before. More importantly, this large-scale dataset enables the first public, extensive, and fair investigations into training CLIP and T2I diffusion models with high-quality synthetic captions. For example, our results comprehensively demonstrate that Recap-DataComp-1B significantly enhances cross-modal tasks, long-context understanding, and text-to-image generation. Based on this evidence, we believe that Recap-DataComp-1B is a novel and important contribution to the community, with the strong potential to provide significant benefits to future multimodal research.\n\n\n---\n\n\n### **Q3: Complexity of creating and training models of our dataset.**\n\n**A3:** We benchmark the inference speed of LLaVA-1.5-LLaMA3-8B on the TPU-V4 256 hardware, achieving a throughput of 382 images per second. At this rate, generating captions for the entire Recap-DataComp-1B dataset (~1 billion images) would take approximately 29 days of continuous computation.\n\nRegarding CLIP training, training a ViT-L/16 model for two epochs (~2.56 billion total samples) on Recap-DataComp-1B requires ~ 1 day on TPU-V3 256 infrastructure. For DiT training, training a base-sized DiT model with a batch size of 2048 for 650K steps takes approximately 7 days using TPU-V3 256 hardware.\n\nWe will provide comprehensive details about computational complexity and runtime metrics in the next revision.", "claims_and_evidence": "Experimental results corroborate the claim that the recap dataset is better for training multimodal models.", "methods_and_evaluation_criteria": "Results are shown on standard tasks using acceptable metrics.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Experiments are rigorous,", "relation_to_broader_scientific_literature": "It's understood that the fine-grained aligned multimodal data is essential for learning good image-text models as per our current definition of goodness. That has been the motivation behind creating larger and larger datasets. As observed by the authors, the improvement especially about fine-grained textual input following in text-to-image diffusion models is notable. I'd argue that the research question on the other hand should be about how to incorporate common sense, or reducing dependencies on large scale annotations.", "essential_references_not_discussed": "I can see that others have trained LLaVa-LLaMa3 models earlier: https://huggingface.co/Intel/llava-llama-3-8b\nIs there any reason to train the model again?", "other_strengths_and_weaknesses": "It's understood that the fine-grained aligned multimodal data is essential for learning good image-text models as per our current definition of goodness. That has been the motivation behind creating larger and larger datasets. As observed by the authors, the improvement especially about fine-grained textual input following in text-to-image diffusion models is notable. I'd argue that the research question on the other hand should be about how to incorporate common sense, or reducing dependencies on large scale annotations.", "other_comments_or_suggestions": "Although it's a new dataset with improved fine-grained image-text data, I find its technical innovation is missing.", "questions_for_authors": "I encourage the authors to provide training time. Currently there has been no mention of the complexity associated with creating this Recap dataset; training models using this newly generated recap dataset.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "Thank you for providing the ethical statements about how the limitation of model-based filtering might have let some of the unsafe content to be used for training and specifying the copyright details."}}, {"review_id": "mpUEvOC3T7", "reviewer": "Reviewer_ogfc", "summary": "This paper explores the impact of recaptioning web-crawled image-text pairs using LLaMA-3. The authors identify that web-crawled datasets (like DataComp-1B) suffer from image-text misalignment and low-quality textual descriptions. Their approach is straightforward: they fine-tune a LLaMA-3-8B powered LLaVA-1.5 model and use it to generate detailed captions for approximately 1.3 billion images from DataComp-1B, creating a new dataset called Recap-DataComp-1B.\n\nThe paper demonstrates that these enhanced captions are longer, more diverse, and better aligned with their corresponding images. Through extensive experiments, they show that vision-language models trained on this enhanced dataset exhibit significant improvements: CLIP models achieve an average 3.1% boost in zero-shot cross-modal retrieval performance when trained on a mix of original and recaptioned data, while text-to-image Diffusion Transformers show better alignment with text instructions and produce higher-quality images (demonstrated by improved FID scores and CLIP scores).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Q1: Analysis of this limitation, insights, and technology to migrate classification performance issues.**\n\n**A1:**  Following your valuable suggestion,  we analyze the classification drop with synthetic captions and identify three likely causes: (1) CLIP's difficulty learning from long, rich captions [1], (2) captioners’ poor performance on classification tasks [2], and (3) limited evaluation of zero-shot only. We additionally evaluate performance on the standard ImageNet-1K classification task using both linear probing and full fine-tuning. We observe that introducing lightweight tuning significantly reduces the initial performance gap seen in zero-shot settings, indicating that models trained solely on synthetic captions still learn strong visual representations.\n\n| Model | Mix ratio | Zero-shot  | Linear Prob | Fully FT |\n|-------|-----------|-----------------|-------------|----------|\n| B/16  | p=0       | 33.8            | 76.1        | 83.6     |\n| B/16  | p=0.8     | 69.8           | 80.3        | 84.2     |\n| B/16  | p=1       | 70.5            | 80.4        | 84.1     |\n\n\nRegarding potential strategies, mixed training already achieves a preferable balance of classification and retrieval, and recent work [3] suggests that advanced CLIP training methods can further exploit synthetic captions.\n\n---\n\n\n\n### **Q2: Caption model and quantitative results.**\n \n**A2:** Thanks for bringing up the concern about ablation on caption models.  We evaluate four caption models on a 30M subset and the results are shown in the following Table.  We find that LLaVA-1.5-LLaMA3-8B consistently yields the best downstream CLIP performance. Larger models like LLaVA-NeXT-Gemma2-27B are significantly slower—requiring 201 days to recaption the full 1 billion samples on the same infrastructure—and also underperform on retrieval tasks.  Thus, we chose LLaVA-1.5-LLaMA3-8B for re-captioning the full dataset, balancing quality and efficiency.\n\n| Model | Caption Model | Caption Speed | Mix Ratio | IN1K | Flickr T2I | Flickr I2T | COCO T2I | COCO I2T |\n|-------|---------------------------|---------------|-----------|------|------------|------------|----------|----------|\n| L/16  | -                         | -             | 1.0       | 66.1 | 48.6       | 65.3       | 30.2     | 41.7     |\n| L/16  | LLaVA-1.5-LLaMA3-8B       | 382 img/s     | 0.6       | 67.5 | 61.1   |77.8   | 39.5 | 54.0 |\n| L/16  | LLaVA-next-Gemma2-27B     | 54 img/s      | 0.6       | 67.1 | 58.9       | 74.6       | 37.0     | 51.8     |\n\n\n---\n\n\n### **Q3:  Prompt engineering and quantitative results.**\n\n**A3:**  We ablate prompt engineering on a 30M subset and found prompt choice influences downstream tasks shown in the following table:\n\n| Model | Prompt Type      | Mix Ratio | IN1K  | Flickr T2I | Flickr I2T | COCO T2I | COCO I2T |\n|-------|------------------|-----------|-------|------------|------------|----------|----------|\n| L/16  | -                | 1.0       | 66.1  | 48.6       | 65.3       | 30.2     | 41.7     |\n| L/16  | Original-prompt  | 0.6       | 67.5  | 61.1       | 77.8       | 39.5     | 54.0     |\n| L/16  | Concise-prompt   | 0.6       | 67.8  | 61.5       | 80.6       | 40.5     | 57.1     |\n| L/16  | Diverse-prompt   | 0.6       | 68.2 | 63.7 | 81.4  | 42.3 | 57.3 |\n| L/16  | Condition-prompt | 0.6       | 68.2 | 62.0     | 78.7       | 40.1     | 55.5     |\n\nOur findings show that even simple prompts, such as the *Original* setup, already significantly enhance retrieval performance at this scale. As the first to release a billion-caption dataset, these promising results motivate us to further expand and refine our evaluations in future work.\n\n---\n\n\n### **Q4: Limited Human Evaluation.**\n\n**A4:**  We present human evaluation results in Appendix A and Figures 4 and 5, which align closely with GPT-4V assessments. We will further emphasize this consistency in future revisions.\n\n\n---\n\n### **Q5: Potential biases and failure cases in the generated captions.**\n\n**A5:**  We acknowledge the presence of biases and failure cases in generated captions inherited from the training data of LLaMA-3 and LLaVA. Addressing and mitigating these biases will be a key focus of our future work. Regarding presenting potential failure cases, one example of a current shortcoming is the pipeline’s inability to correctly identify certain objects—for instance, the “Western Kingbird” shown in Figure 1. We will include a more detailed analysis of failure cases in the next version.\n\n\n---\n\n\n\n### **Q6: Smaller, curated subset.**\n\n**A6:**  We will release the ablation subset to support further research.\n\n---\n\n[1] Zhang, Beichen, et al. \"Long-CLIP: Unlocking the long-text capability of CLIP.\" ECCV, 2024.  \n[2] Zhang, Yuhui, et al. \"Why are visually-grounded language models bad at image classification?” arXiv, 2024.\n[3] Liu, Yanqing, et al. \"CLIPS: An Enhanced CLIP Framework for Learning with Synthetic Captions.\" arXiv,  2024.", "claims_and_evidence": "Most claims in the paper are well-supported by clear and convincing evidence:\n\n1. The claim that web-crawled image-text pairs are inherently noisy and misaligned is effectively demonstrated through visual examples in Figure 1, comparing original captions with recaptions.\n2. The performance improvement of their LLaMA-3-powered LLaVA model over other LLaVA variants is adequately supported by comparisons on MMMU and MM-Vet benchmarks in Table 1.\n3. The claim that Recap-DataComp-1B contains higher-quality captions is validated through:\n   - Word and length distributions (Figures 1 & 3)\n   - GPT-4V evaluations showing an average rating increase from 3.71 to 4.14\n   - LongCLIP scores demonstrating better semantic alignment (89.91 vs. 10.09)\n4. The benefits of training vision-language models on recaptioned data are thoroughly evidenced in Tables 2-5 for CLIP models and Table 6 for DiT models.\n\nThe only claim that could use stronger evidence is the assertion that recaptioning enhances classification performance. The results in Table 3 show that using purely recaptioned data (p=0.0) actually hurts ImageNet classification. While the authors acknowledge this and pick a good compromise at p=0.8 for later ablations, more analysis of this limitation would strengthen the paper.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are appropriate and comprehensive:\n\n1. The evaluation of caption quality uses a multi-faceted approach:\n   - Automatic metrics (CLIP, LongCLIP)\n   - Human-in-the-loop evaluation (GPT-4V)\n   - Qualitative examples and word distribution analysis\n2. For CLIP evaluation, the authors perform extensive ablations:\n   - Testing different mixing ratios between original and recaptioned data\n   - Experimenting with various text encoder sizes\n   - Evaluating on standard benchmarks (ImageNet-1K, COCO, Flickr30K)\n   - Additional evaluations on challenging datasets (Urban1K, VG-Attribution)\n3. For DiT evaluation, they use standard metrics (FID, CLIP scores) as well as GPT-4V scoring, which provides a more holistic assessment of generation quality.\n\nThe experimental design includes appropriate baselines and ablations, making the conclusions reliable.", "theoretical_claims": "The paper is primarily empirical and does not make significant theoretical claims requiring formal proofs.", "experimental_designs_or_analyses": "The experimental designs are sound and well-executed:\n\n1. CLIP training experiments (Section 5) are comprehensive, exploring mixing ratios, text encoder sizes, and evaluating on multiple benchmarks. The decision to use p=0.8 as a compromise between classification and retrieval performance is well-justified.\n2. Text-to-image experiments (Section 6) properly evaluate how different mixing ratios affect generation quality, with both quantitative metrics and visual examples.\n3. Caption quality analyses (Section 4) use multiple complementary methods to assess semantic alignment and descriptiveness.\n\nA significant limitation, however, is that while the paper extensively discusses how recaptioned data affects downstream models, all experiments are conducted using a single recaptioning model configuration. The paper lacks systematic quantitative analysis of how different settings of the caption model itself (base LLM selection, image encoder variations, training data choices, or whether conditioning on original captions) affect caption quality and subsequent downstream performance. Although the authors provide qualitative examples in the supplementary materials (Figures 7 and 8), quantitative ablation studies on the captioning pipeline itself would significantly strengthen the work and provide insights into which components most strongly influence the quality of the recaptioned dataset.", "relation_to_broader_scientific_literature": "The paper is well-positioned within the broader scientific literature:\n\n1. It builds upon foundational work on vision-language models like CLIP (Radford et al., 2021) and text-to-image models (DiT, Diffusion Transformers).\n2. It clearly differentiates from previous recaptioning approaches like ShareGPT4V (Chen et al., 2023) and LaCLIP (Fan et al., 2024) by emphasizing:\n   - The use of open-source models (LLaMA-3) rather than closed APIs\n   - The billion-scale application, which is significantly larger than previous efforts\n   - The comprehensive evaluation across both discriminative and generative models\n3. The authors acknowledge the high costs associated with using API-based approaches like GPT-4V for billion-scale datasets, highlighting the practical significance of their approach.", "essential_references_not_discussed": "Not noticed.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper tackles an important problem (improving web-crawled image-text data) at an unprecedented scale (1.3 billion images).\n2. The approach is practical and accessible, using open-source models rather than closed APIs.\n3. The comprehensive evaluation across multiple model types (CLIP and DiT) and tasks demonstrates the broad applicability of the approach.\n4. The analysis of optimal mixing ratios provides practical insights for future researchers.\n\nWeaknesses:\n\n1. Limited exploration of prompt engineering: The paper uses a single prompt template for recaptioning. Testing multiple prompting strategies might yield more diverse captions.\n2. The degradation in classification performance when using only recaptioned data (p=0.0) is noted but not extensively analyzed.\n3. The human evaluation is limited to GPT-4V ratings rather than traditional human evaluation, though the large-scale nature of the dataset makes this a practical choice.", "other_comments_or_suggestions": "1. It would be valuable to analyze potential biases in the generated captions, as they may inherit biases present in the training data of LLaMA-3 and LLaVA.\n2. Showing some failure cases where the recaptioning model produces inaccurate descriptions would help understand the limitations of the approach.\n3. The appendix mentions experiments with different prompting strategies and conditioning on original captions. These findings could be highlighted more prominently in the main paper.\n4. Releasing smaller, curated subsets of the recaptioned dataset would benefit researchers with limited computational resources.", "questions_for_authors": "1. You found that training with purely recaptioned data (p=0.0) significantly hurts classification performance on ImageNet-1K. Do you have insights into why this happens, and have you explored techniques to mitigate this issue while preserving the benefits for retrieval tasks?\n2. In the appendix, you mention experimenting with conditioning the recaptioning model on original captions. Did you evaluate training vision-language models directly on these condition-based recaptions? Would this approach help with the classification performance drop observed with pure recaptions?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FsW0rScL9Y", "reviewer": "Reviewer_xErf", "summary": "This paper explores the impact of improving textual descriptions for large-scale web-crawled image-text datasets using LLaMA-3. The authors propose a recaptioning pipeline that fine-tunes a LLaMA-3-8B-powered LLaVA-1.5 model and applies it to ∼1.3 billion images from the DataComp-1B dataset. The resulting dataset, Recap-DataComp-1B, enhances training for vision-language models. Experiments demonstrate that for discriminative models like CLIP, the recaptioned dataset improves zero-shot performance on four cross-modal retrieval tasks. For generative models like Diffusion Transformers, the refined captions enable better alignment with text instructions, particularly in handling complex queries. In general, this paper is well organized.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Q1: Novel ideas,  insights, and conclusions.**\n\n**A1:** Thank you for raising concerns about novelty. Our primary focus is on developing **Recap-DataComp-1B**, which we believe to be **the first publicly available image-text dataset at the billion-scale generated by LLaMA-3**. Unlike previous works [1,2] that remain proprietary—both in terms of captioning models and datasets—our approach is fully open and demonstrably scalable to the billion level. We believe this openness and scale set a new milestone in multimodal research.\n\nWhile the idea of “recaptioning” is not entirely new, we have yet to see it executed at this scale using advanced large language models. By creating recaptions on such a massive level, our dataset enables the first public, large-scale experiments into training models like CLIP and T2I diffusion with high-quality synthetic captions. In our view, **Recap-DataComp-1B** offers a novel and significant contribution to the field, with considerable potential to advance future multimodal research.\n\n[1] Improving Image Generation with Better Captions. 2023\n[2] A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image. 2023\n\n---\n\n### **Q2: Marginal improvement on some tasks.**\n\n**A2:**  Thank you for the question. As shown in Table 1, the LLaVA-3 **8B** model offers slightly better performance than the **13B** model while being significantly faster. We selected it to balance captioning quality and efficiency.\nIn Table 4, Recap-CLIP clearly outperforms the previous DataComp-1B baseline trained on the same samples, highlighting the improved quality of our dataset. While our results slightly exceed those of SigLIP, it’s important to note that SigLIP requires four times more training samples  on a private dataset, which is five times larger—making our gains particularly noteworthy.\n\n---\n\n\n### **Q3: Weaker performance on ImageNet-1K. Are more tasks harmful?**\n\n**A3:**  Thanks for your constructive suggestion. We observe that training exclusively with synthetic captions degrades CLIP performance. However, introducing a small portion of original captions (e.g., 10%) effectively recovers performance, indicating that original captions remain crucial in preventing data collapse, as noted in prior work [1]. Additionally, we clarify that while recent works like LaCLIP [2] and VeCLIP [3] demonstrate synthetic captions can enhance CLIP training, to our knowledge, no prior work has trained models exclusively on synthetic captions. Our study represents the first public attempt, shedding light on the challenges of this approach.\n\n\nSecond, we further evaluate the visual representations learned from these models through linear probing and full fine-tuning on ImageNet-1K. While the CLIP ViT trained only on synthetic captions underperforms in zero-shot classification, We observe that lightweight tuning (e.g., linear probing or fine-tuning) greatly narrows the performance gap seen in zero-shot settings. Moreover, models trained on mixed captions perform on par with those trained on original data, striking a favorable balance between classification and retrieval performance.\n\n| Model | Mix ratio | Zero-shot | Linear Prob | Fully FT |\n|-------|-----------|-----------------|-------------|----------|\n| B/16  | p=0       | 33.8            | 76.1        | 83.6     |\n| B/16  | p=0.8     | 69.8            | 80.3        | 84.2     |\n| B/16  | p=1       | 70.5            | 80.4        | 84.1     |\n\nFollowing your advice, we also evaluated additional tasks to analyze the impact of recaption following the original DataComp paper across 22 datasets (VTAB contains 13, IN-1K-Shift contains 6, and Retrieval contains 3), shown in the table below. We found that purely synthetic captions negatively impacted zero-shot performance. However, our mix-training strategy still achieves superior performance in VTAB and retrieval tasks.\n\n| Model | Mix ratio | IN-1K | VTAB | IN-1K-Shift | Retrieval |\n|-------|-----------|-----------------|------|-------------|-----------|\n| B/16  | p=0       | 33.8            | 38.0 | 33.3        | 42.2      |\n| B/16  | p=0.8     | 69.8            | 57.5 | 55.5        | 56.6      |\n| B/16  | p=1       | 70.5            | 56.8 | 55.6        | 54.9      |\n\n\nThese results indicate that only synthetic captions may negatively impact zero-shot classification performance. Our mixed-training strategy effectively balances these competing factors. We acknowledge that further investigation into potential drawbacks is valuable. Given the demonstrated scalability and high-quality alignment of our proposed dataset, we believe it provides a strong baseline for future research.\n\n---\n\n[1] Seddik M E A, et al. How bad is training on synthetic data? a statistical analysis of language model collapse[J]. arXiv, 2024.\n[2] Fan, Lijie, et al. \"Improving CLIP training with language rewrites.\" NeurIPS. 2024.  \n[3] Lai, Zhengfeng, et al. \"VeCLIP: Improving CLIP training via visual-enriched captions.\" ECCV.  2024.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "yes", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths: \n+ The paper contributes an open-source, large-scale recaptioning pipeline, fostering community-wide research in vision-language model training.\n+ The study demonstrates quantitative gains for both discriminative and generative models, enhancing retrieval accuracy and text-to-image generation quality. \n+ By systematically refining web-crawled captions, the proposed method mitigates inherent noise, leading to more semantically rich and effective training data.\n\nWeaknesses: \n- Recaptioning has been explored and verified effective in prior work, like DALL-E 3 [a] and [b]. Compared with existing work, this work seems not to bring impressive discovery, insights , or conclusions. \n\n- The novelty of this work is weak and unclear. I understand this work may focus on empirical verification, but an empirical work also needs to provide some novel ideas or experience to contribute the development of the research community. \n\n- Performance improvement of using regenerated captions over that with raw captions on some tasks (such as the results in Tab 1 and Tab 4) is marginal. \n\n- Weaker performance w/ re-caption on ImageNet-1K in Tab 2 can be observed. Is there any other task that re-caption may be harmful? More analysis may be helpful, which should not weaken the quality of this work, but lets the community pay more attention to the potential disadvantages or risks of recaption. \n\n\n[a] Improving Image Generation with Better Captions. 2023\n[b] A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image. 2023", "other_comments_or_suggestions": "NA", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uNhLQLkZsK", "reviewer": "Reviewer_fVXe", "summary": "The authors finetune a better Llava-1.5 model using a more advanced Llama model. The authors then recaption the DataComp-1B dataset and show promising results in terms of ImageNet zero-shot accuracy and text-to-image retrieval benchmarks. The authors finally use the CLIP score to show that the recaptioned-caption is better correlated with the image than the initial web-scraped caption.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Q1: No algorithmic contributions.**\n\n**A1:**  While we don’t introduce new recaptioning algorithms, we highlight two core contributions:\nRecap-DataComp-1B, the first fully open, billion-scale image-text dataset with synthetic captions generated using LLaMA-3. Prior works [1,2] are proprietary and closed-source. Scaling recaptioning with advanced LLMs at this scale is unprecedented and enables reproducible, large-scale research in multimodal learning.\n\n\nThis dataset supports the first public large-scale evaluation of CLIP and T2I diffusion models trained on high-quality synthetic captions. Our experiments show notable gains in cross-modal tasks, long-context understanding, and image generation, making Recap-DataComp-1B a valuable resource for the community.\n\n\n[1] Improving Image Generation with Better Captions. 2023 \n[2] A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image. 2023\n\n---\n\n### **Q2: Misalignment between their dataset’s advantage and results.**\n\n**A2:** Our main goal is to improve noisy web captions by aligning them better with images. Hence, we focus on tasks like cross-modal retrieval that directly measure image-caption alignment. We also include human and GPT-based evaluations (Sec. 4.2 and Appendix A) to assess caption quality. In classification, training only on synthetic captions degrades CLIP performance. However, mixing just 10% of original captions substantially recovers accuracy, confirming that original captions still play a key role in avoiding data collapse. In contrast, T2I diffusion benefits from detailed, synthetic captions. Setting the mixing ratio to p=0.1 (i.e., 90% recaptions) leads to better generation results since richer captions produce better text latents.\n\n---\n\n### **Q3:  More complex benchmarks (such as image captioning and VQA). Table 6 is less comprehensive.**\n\n**A3:**  Thank you for the suggestion. Our goal is to provide a high-quality, large-scale dataset for both understanding and generation tasks. We conducted large-scale CLIP and T2I experiments to demonstrate its effectiveness and potential. Table 6 is limited by the high computational cost—for instance, training CLIP ViT-B/16 for 2 epochs only takes approximately 0.5 days, while DiT-B/4 requires around 7 days for a single epoch on TPU-v3-256. These constraints restricted the number of benchmarks we could include. Furthermore, the current results already demonstrate our dataset’s effectiveness, and we will leave further evaluations to future work.\n\n\n Following your advice, we adopted the LLaVA framework to evaluate more challenging tasks such as VQA and image captioning. We replaced its pretraining data (558K LAION/CC3M/SBU) with 558K Recap-DataComp-1B samples. Results show consistent gains, including +0.6% across four VQA tasks and +34 points on MME shown in the following table:\n\n| Pre-train Dataset                | Tunable Module | Training Steps | TextVQA | MME        | VQA-V2 | MM-Vet   |\n|----------------------------------|----------------|----------------|---------|------------|--------|----------|\n| LLaVA-LCS-558K                   | Projector      | 2K             | 59.1    | 1489/277 | 77.9   | 34.4     |\n| Recap-DataComp-1B (Recaption Only) | Projector      | 2K             | 60.1 | 1523/260 | 78.6 | 35.1 |\n\n\nLastly, we further evaluated the CLIP visual encoder trained with our mix-training strategy on our dataset. Our LLaVA with ViT trained only on synthetic captions (p=0) shows only a slight drop compared to training on the original data (p=1) — an average of 1.2% across four VQA benchmarks (TextVQA, GQA, MM-Vet, SEED). Notably, at p=0.8, LLaVA achieves the best overall performance, matching the p=1 model and surpassing it on MME by 35 absolute points.\n\n| Model | Mix ratio |  IN-1K | Text VQA | GQA  | MME  | MM-Vet | SEED |\n|-------|-----------|-----------------|----------|------|------|--------|------|\n| B/16  | p=0       | 33.8            | 50.0     | 58.9 | 1335 | 25.0   | 62.8 |\n| B/16  | p=0.8     | 69.8            | 52.0     | 60.2 | 1417 | 25.6   | 64.2 |\n| B/16  | p=1       | 70.5            | 51.8     | 60.0 | 1382 | 25.6   | 63.9 |\n\n\nThese results demonstrate that by evaluating more diverse and challenging tasks, our proposed dataset not only significantly enhances cross-modal retrieval performance but also reveals promising\n\n---\n\n\n\n### **Q4: Is this data DataComp still publicly available? Safety fixes of LAION.**\n\n**A4:** Yes, DataComp-1B remains public and is not derived from LAION-5B but from a different Common Crawl version. The  original dataset is accessible at Huggingface Dataset. More importantly, it applied strict NSFW filtering, and we discussed safety considerations in the Appendix. We believe releasing our recaptioned dataset poses no major safety risks.\n\n---\n\n**Other**:  on page 5, line 227, we cite the GPT-4V rating. We will clarify this in the revision.", "claims_and_evidence": "The claims are well supported.", "methods_and_evaluation_criteria": "Strengths:\n- The dataset itself is very useful to researchers. For example, it could be used to train better open-sourced CLIP models.\n\nWeaknesses:\n- There are no algorithmic contributions. The authors use off-the-shelf methods.\n- In my opinion, there is some serious misalignment between what the authors are trying to say about their dataset and what type of results the authors show in the tables. The authors show that their generated caption is more descriptive; consequently a CLIP model trained on the recaptioned dataset does not do well in zero-shot classification (see Table 3 p=0.0). Even for text-to-image retrieval and image-to-text retrieval tasks, the authors have to set p to 0.8 (a larger portion of original captions) to see some recall improvement. This paper could benefit from more complex benchmarks that test the model's understanding of the image (such as image captioning and VQA). \n- I know DataComp is derived from LAION, which was taken down a year ago. Is this data still publicly available?", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "As explained in the \"Methods And Evaluation Criteria\" section, I don't think the zero-shot and cross modal retrieval benchmarks are the best metrics to showcase the superiority of the new recaptioned dataset. (Tables 2,3,4,5).\n\nI do think Table 6's experiment on text-to-image generation is a step in the right direction. However, Table 6 is less comprehensive and not the focus of the paper.", "relation_to_broader_scientific_literature": "The proposed novel dataset is a significant contribution to the multimodal literature.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "See above.", "other_comments_or_suggestions": "Page 5 line 227: \"We can observe ...\" This sentence refers to some numbers that are not cited. Please cite the corresponding table.", "questions_for_authors": "Is this dataset derived from the LAION dataset with safety fixes (https://laion.ai/blog/relaion-5b/)? If not, the impact of this work is limited, because you can't release the dataset.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper examines the effects of re-captioning at scale web-crawled image-text pairs using LLaMA-3. The authors highlight that datasets like DataComp-1B, which are obtained through web crawling, often experience issues with image-text misalignment and low-quality textual descriptions. To address this, they propose using a fine-tuned LLaVA-1.5 with LLaMA-3-8B to generate detailed captions for approximately 1.3 billion images from DataComp-1B, resulting in a new dataset called Recap-DataComp-1B.

While reviewers had concerns about the impact of the recaption data on the ImageNet zero-shot classification tasks, the authors addressed this point during rebuttal by providing more empirical evidence (linear/finetuning protocol, VTAB evaluation...).

Overall, all reviewers highlighted the limited novelty of the approach, as recaptionning has been explored in prior work. However, reviewers also emphasized that releasing a large-scale recaption dataset would be very useful for the community and to foster more progress in multimodal research. I agree with the reviewer's assessment and support acceptance.
Decision: Accept (poster)