What If We Recaption Billions of Web Images with LLaMA-3?

## Abstract

Abstract Web-crawled image-text pairs are inherently noisy. Prior studies demonstrate that semantically aligning and enriching textual descriptions of these pairs can significantly enhance model training across various vision-language tasks, particularly text-to-image generation. However, large-scale investigations in this area remain predominantly closed-source.
Our paper aims to bridge this community effort, leveraging the powerful and open-sourced LLaMA-3, a GPT-4 level LLM.
Our recaptioning pipeline is simple: first, we fine-tune a LLaMA-3-8B powered LLaVA-1.5 and then employ it to recaption ∼ similar-to \scriptstyle\sim ∼ 1.3 billion images from the DataComp-1B dataset. Our empirical results confirm that this enhanced dataset, Recap-DataComp-1B, offers substantial benefits in training advanced vision-language models. For discriminative models like CLIP, we observe enhanced zero-shot performance in cross-modal retrieval tasks. For generative models like text-to-image Diffusion Transformers, the generated images exhibit a significant improvement in alignment with users’ text instructions, especially in following complex queries.
Our project page is https://www.haqtu.me/Recap-Datacomp-1B/ .

## 1 Introduction

The exponential growth in data availability is one of the most paramount factors in driving the monumental successes of deep learning over the past decade [, , , , , ]. Typically, this data is sourced through web crawling with simple filtering mechanisms in place. While such an approach has facilitated large-scale data collection, exemplified by collections like LAION-400M [] and LAION-5B [] with billions of image-text records, it has inadvertently compromised data quality. As illustrated in Figure 1, these internet-crawled image-text pairs frequently exhibit misalignments between images and their corresponding textual content, and often, the textual descriptions are brief and lack detailed information.

To mitigate the noise present in web-crawled data, enhancements through post-processing—implemented via human-in-the-loop systems [, ] or automated pipelines [, , ]—are crucial, which help to train the advanced vision-language foundation models. Notably, both the *close-sourced* DALL-E 3 [] and SORA [] incorporate advanced captioning techniques to re-label their training datasets, a crucial step highlighted in their technical reports. Despite various efforts to open-source and replicate these methodologies [, , , , , , ], the community continues to face significant challenges in accessing high-quality, well-aligned image-text data at scale (*e.g*., at the billion level) for training advanced vision-language foundation models.

This paper endeavors to contribute to this community initiative, inspired specifically by the release of LLaMA-3 [], a model demonstrating GPT-4-level capabilities across a variety of linguistic tasks. Additionally, recent studies have shown that leveraging LLaMA-3 can significantly enhance model performance on vision-language tasks [, ], comparable to those achieved by GPT-4V []. In response, we employ LLaMA-3 to develop our advanced captioner model. Our approach is straightforward: we first train a LLaMA-3-powered Llava model to act as an image captioner, which is then utilized to recaption the entire DataComp-1B dataset. As depicted in Figure 1, the resulting dataset, dubbed Recap-DataComp-1B, features enhanced textual descriptions and improved alignment with corresponding images, clearly surpassing its web-crawled counterparts. These quality enhancements are further quantitatively verified in Section 4.

Comprehensive evaluations highlight the significant improvements that Recap-DataComp-1B contributes to the training of advanced vision-language foundation models. Notably, this dataset enables CLIP models to achieve significant enhancements in their zero-shot cross-modal retrieval capabilities. It also enhances the alignment between generated images and text instructions in text-to-image generative models pre-trained on our dataset. We hope that the release of Recap-DataComp-1B will catalyze further developments in advanced vision-language foundation models, particularly encouraging the development within the open-source community.

## 2 Related works

#### Vision-Language Foundation Models.

CLIP [] is one of the pioneering foundation models to connect image and text. By training on millions, and even billions, of image-text pairs [, , , , , , , ], CLIP markedly showcases excessively strong zero-shot capacities, and furthermore, lays the cornerstone for building more advanced vision-language foundation models [, , , , , , , , ]. Apart from discriminative vision-language understanding,
text-to-image generation models [, , , , , , , , ] have transformed the field of AI-generated content, facilitating the creation of high-quality images from natural language descriptions.

#### Enhancing Image-Text Data.

Web-crawled image-text
data [, , ] commonly face the problems of image-text misalignment and the low-quality of textual descriptions.
Typically, there are two popular ways for improving the quality of these image-text pairs: 1) *data filtering* removes misaligned image-text pairs using various methods such as cleaning strategies [, , ], pretrained models [, , ], and human-assisted systems [, , ]; 2) *data recaptioning* improves the textual quality of image-text pair via generating new captions, which is the focus of this paper.
To recaption data, LaCLIP [] utilizes large language models (LLMs) like ChatGPT to rewrite the original captions; Nguyen *et al*. [] employ BLIP2 [] to recaption images. More recently, advanced large multimodal models have been applied to further enhance the quality of image captioning. For example, ShareGPT4V [] employs GPT-4V to create highly descriptive captions from carefully crafted prompts and corresponding image inputs; the resulting dataset has significantly benefited the training of various models [, , , , ]. However, scaling such prompting with GPT-4V to billions of records is less practical, as it will drastically increase the monetary cost (of intensively calling OpenAI APIs) by more than 10,000$\times$.

Our paper mostly follows the approach presented in [, , , ], where advanced open-source multimodal models like LLaVA [] are employed for recaptioning purposes. However, our approach is distinguished by two major aspects: 1) we strongly enhance the LLM module in LLaVA, *i.e*., building with LLaMA-3; and 2) our recaptioning efforts are executed on a billion-scale dataset.

## 3 Recaptioning Pipeline

Our recaptioning pipeline is centered around the advanced LLM
LLaMA-3 [],
which achieves exceptionally strong performance in language understanding, reasoning, code generation, math problems, *etc*. [, ]. Specifically, we utilize the LLaVA framework [] to fully harness its capabilities for
visual understanding. We describe the detailed training procedures below.

### 3.1 Model details

Figure: Figure 2: The illustration of our recaptioning pipeline on DataComp-1B. We use LLaMA-3-powered LLaVA to reception images, which enables us to train stronger CLIP models and Text-to-Image Diffusion models.
Refer to caption: x2.png

#### Model Configuration.

We follow the setup of LLaVA-1.5 [] to build our captioner model, except that we use LLaMA-3-8B as the language decoder because of its superior performance.
The visual branch of CLIP ViT-L/14 [] is used as the vision encoder. Two trainable MLP layers are employed on top of the vision encoder to project visual features into the language embedding space.

#### 2-Stage Training.

We also follow LLaVA-1.5 [] for model training. Essentially we conduct instruction-tuning on the pre-trained LLM with its original
auto-regressive training objective. In the first stage, only the projection MLP is trained;
in the second stage, we fine-tune both the projection MLP and the language decoder. Note that the vision encoder remains frozen all the time. Following the protocols in LLaVA [], 558k image-text pairings filtered from LAION [], CC [],and SBU [] are used as training data in the first stage; then 665k instructions-following data from
LLaVA-1.5 [], containing image-grounded conversation, image descriptions, and image-based complex reasoning tasks, are used for the second stage of training. *To help our model generate higher-quality captions, we use the image-text pairs from HQ-Edit dataset [] for further tuning*.

#### Evaluations.

To probe the visual understanding and reasoning ability of our LLaVA-1.5-LLaMA3-8B model, we opt for two comprehensive multi-modal evaluation benchmarks, MMMU [] and MM-Vet []. These benchmarks assess a broad range of capabilities such as recognition, spatial awareness, OCR, knowledge, and language generation.
As reported in Table 1, on both benchmarks, our LLaVA-1.5-LLaMA3-8B model surpasses the vanilla LLaVA-1.5-7B model by a significant margin. These results also substantially beat the considerably larger LLaVA-1.5-13B model, clearly demonstrating the superior visual understanding and reasoning ability of our model.

**Table 1: Performance comparison of LLaVA.**
| Model | LLaVA-1.5-7B | LLaVA-1.5-13B | LLaVA-1.5-LLaMA3-8B (ours) | GPT-4V |
| --- | --- | --- | --- | --- |
| MMMU | 33.6 | 36.4 | 45.2 | 56.8 |
| MM-Vet | 33.9 | 36.3 | 37.8 | 44.6 |

### 3.2 Recaptioning DataComp-1B

With this advanced LLaVA model, we next use it to generate captions in a scalable and detailed manner, given the visual input, and the following text prompt:

$$ "Please generate a detailed caption of this image. Please be as descriptive as possible." $$

As for the dataset, we opt for DataComp-1B  [], a widely accessible, large-scale vision-language dataset comprising $\scriptstyle\sim$1.3 billion web-crawled image-text pairs.
To ensure its quality, DataComp-1B is already a curated subset from a much larger collection of 12.8 billion image-text pairs and has been subjected to rigorous preprocessing which includes safety checks, deduplication, CLIP score filtering, and image-based filtering. Despite these efforts, the quality of the original captions in DataComp-1B still exhibits relatively low quality.

We apply our well-trained LLaVA-1.5-LLaMA3-8B model to recaption the entire DataComp-1B dataset. Specifically,
captions are generated auto-regressively via greedy decoding, with the maximum output token length set to 128.
We term this newly recaptioned dataset Recap-DataComp-1B.

## 4 Analyzing Recap-DataComp-1B

This section collects and presents a quantitative analysis of our generated captions on DataComp-1B. We primarily focus on two aspects: 1) the inherent features of the captions, including word distributions and average lengths; and 2) the semantic quality of the captions, evaluated in terms of the matching similarity between images and captions and the inherent textual quality of the captions.

### 4.1 Word & Length Distribution

We begin our analysis by comparing the word frequency distributions between our recaptioned content and that from the original DataComp-1B, as illustrated in Figure 1, analyzing a randomly sampled subset of approximately 0.35 billion instances.
Our findings reveal that the recaptioned content displays a considerably richer vocabulary, capturing 82.86% tokens of the word collections from both ours and the original caption data. Additionally, there is a noticeable variety in the usage of nouns and adjectives in our captions (*e.g*., “white” and “background”).
We argue that this increased lexical diversity is a direct consequence of the extended length of our data.
We thus present the distribution of caption lengths in Figure 3 to highlight this difference. On average, our recaptioned data demonstrates a longer sequence length of 49.43, whereas the original DataComp captions have a much shorter length of 10.22.
These observations validate that our Recap-DataComp-1B surpasses the original DataComp-1B version in terms of both caption length and diversity.

Figure: Figure 3: Average length distributions of both the original captions and our recaptioned data in DataComp-1B.
Refer to caption: x3.png

### 4.2 GPT-4V & CLIP Evaluations

Next, we evaluate the semantic quality of recaptioned content using two models: 1) CLIP [], which measures the semantic similarity between captions and images, and 2) GPT-4V [], which assesses the fluency and alignment of captions with the given images.

For the CLIP evaluation, we analyze a subset of 180,000 image-text pairs. Interestingly, we note that, when using the standard CLIP-large model with $\scriptstyle\sim$428M parameters for this measurement,
our recaptioned content performs just comparably to the original captions (49.57 *vs*. 50.43). We attribute this result primarily to the limitations of the standard CLIP model, which is trained on ‘short’ captions and may inadequately capture the nuances in semantic similarity for longer captions.
To probe deeper into semantic alignment between long captions and images, we utilize the LongCLIP-Large model [], which is specifically fine-tuned to handle longer captions. With this setup, the LongCLIP score of our newly generated caption impressively attains 89.91, nearly $9\times$ higher than the LongCLIP score of the original DataComp captions (*i.e*., only 10.09).

In addition, to evaluate both the textual quality and the alignment of the captions with their corresponding images, we randomly select 10,000 instances for GPT-4V [] evaluation, employing the prompting strategy outlined below (CAPTION is the textual input), as per [, ].

[backgroundcolor=gray!15]
GPT-4V Evaluation Instruction:
[Image Caption]
CAPTION
Rate whether the caption is of high-quality and fluent and correctly matches the given image. The rating should be 1-5, where 1 is incorrect and not fluent at all, and 5 is correct and very fluent. Try to just give a numerical rating.
Your response should be in the format:
Rating: (int)

We can observe that our recaptioned content achieves markedly superior ratings, registering an average rating increase of 0.43 (from 3.71 to 4.14). Together with the findings from Section 4.1, this confirms the superior quality of our newly generated captions, in terms of length, vocabulary diversity, semantics, and image-text alignment.

## 5 CLIP

CLIP [] stands as a widely utilized vision-language model, where an image encoder and a text encoder are jointly trained to predict correct matches across entire batches of image-text pairs.
In this section, we delve into the advantages of training CLIP models with our Recap-DataComp-1B dataset.
We anticipate that CLIP models trained on this dataset will exhibit superior zero-shot cross-modal retrieval capabilities and enhanced text understanding, especially with long and complex textual inputs, given the improved quality of our recaptions.

### 5.1 Experiment settings

#### Training.

For reference, we term the CLIP model trained on our Recap-DataComp-1B dataset as Recap-CLIP.
Our training pipeline primarily follows CLIPA [, ], which incorporates a two-state training, *i.e*., a pre-training process with a small image size followed by a fine-tuning stage incorporating a larger image resolution.
We set the text token length to 128 to accommodate the learning of long captions presented in Recap-DataComp-1B.
We conduct experiments using three model scales: S/16, B/16, and L/16, with details listed in Table 2. The AdamW [] optimizer is used for training.
In the pre-training phase, the model is trained with 2.56 billion samples with a reduced image size of 112, including a warm-up phase involving 51.2 million samples. The batch size and base learning rate are set to 32,768 and 8e-6, respectively.
For the subsequent fine-tuning phase, we increase the image size to 224 and train the model on 128 million samples with a 25.6 million sample warm-up. Here, we adjust the batch size to 16,384 and the learning rate to 4e-7.

#### Evaluation.

The efficacy of Recap-CLIP is gauged via several metrics. We evaluate zero-shot image classification on the ImageNet-1K dataset [] and assess zero-shot cross-modal retrieval performance using the validation set of MSCOCO 2014 [] and the test set of Flickr30K [](^1^11We employ the widely used Karpathy split [] of MSCOCO and Flickr30K.), following the established practices [, , , ].

We present our results from three aspects. First, we explore the impacts of differing mix ratios between original captions and our enhanced recaptions on CLIP performance. Next, we analyze the effects of enlarging the size of the CLIP text encoder.
Lastly, we investigate the text understanding capability of our Recap-CLIP, via testing on VG-Attribute [], which evaluates attributes understanding ability, and Urban1K [], which tests the model’s ability to handle long text.

**Table 2: Recap-CLIP model configurations used in our paper.**
|  | Embed | Vision Transformer | Text Transformer | # params (M) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| model | dim | layers | width | heads | layers | width | heads | vision | text | total |
| S/16 | 384 | 12 | 384 | 6 | 12 | 384 | 6 | 22 | 33 | 55 |
| B/16 | 512 | 12 | 768 | 12 | 12 | 512 | 8 | 86 | 53 | 141 |
| L/16 | 768 | 24 | 1024 | 16 | 12 | 768 | 12 | 303 | 109 | 414 |
| H/14 | 1024 | 32 | 1280 | 16 | 24 | 1024 | 16 | 631 | 334 | 967 |

**Table 3: Train with mixed captions. We choose Recap-CLIP-B/16 for this ablation. Larger $p$ represents a higher ratio of the original caption. We report top-1 zero-shot classification accuracy on ImageNet-1K and top-1 recall for retrieval tasks. *concat: Concat two types of captions.**
| mixed ratio $p$ | ImageNet-1K | COCO R$@$1 | Flickr30K R$@$1 |  |  |
| --- | --- | --- | --- | --- | --- |
| Validation | I$\rightarrow$T | T$\rightarrow$I | I$\rightarrow$T | T$\rightarrow$I |  |
| 0.0 | 36.0 | 53.0 | 34.1 | 74.1 | 53.5 |
| 0.1 | 58.4 | 60.9 | 40.5 | 83.9 | 65.5 |
| 0.2 | 62.5 | 61.7 | 41.4 | 85.8 | 65.7 |
| 0.3 | 65.1 | 62.7 | 42.6 | 86.3 | 67.0 |
| 0.4 | 66.7 | 62.6 | 42.5 | 87.4 | 67.7 |
| 0.5 | 67.2 | 61.9 | 42.7 | 85.9 | 66.7 |
| 0.6 | 68.0 | 62.2 | 42.4 | 86.0 | 67.4 |
| 0.7 | 68.4 | 60.7 | 42.3 | 86.3 | 66.9 |
| 0.8 | 69.2 | 61.5 | 42.2 | 85.2 | 66.9 |
| 0.9 | 69.2 | 60.6 | 41.1 | 86.0 | 65.7 |
| 1.0 | 69.7 | 57.3 | 37.7 | 84.2 | 63.0 |
| *concat | 43.3 | 57.8 | 35.6 | 80.2 | 56.4 |

### 5.2 Training with Mixed Captions

As pointed out by DALL-E 3 [], blending both the briefgenuine captions and the long informative generated captions can effectively prevent the model from unwanted overfitting to recaption data. Therefore, we hereby first study the effect of varying mix ratios between the original captions and our recaptions on the training of the Recap-CLIP B/16 model, as detailed in Table 2. Specifically, for each sample in a training batch, we randomly sample the original caption with probability $0\leq p\leq 1$ and our captions with probability $1-p$, referring to the mixed ratio:

$$ $\text{Caption}=\begin{cases}\text{Original}&\text{with probability }p\\ \text{Recaption}&\text{with probability }1-p\end{cases}$ $$

This strategy ensures that each batch contains a mixture of our recaption and the original captions controlled by probability $p$.
The randomness allows each sample to encounter different captions across training epochs, potentially enhancing the model’s generalization.

#### Main results.

Our findings are summarized in Table 3.
We observe that reducing the mixed ratio $p$ (*i.e*., increasing the proportion of our recaption data) initially leads to an improvement followed by a decline in cross-modal retrieval performance.
This initial improvement suggests that high-quality recaptioned data effectively enhances contrastive learning. However, the subsequent decrease indicates that the original captions from DataComp-1B provide necessary training regularization, preventing the model from overly adapting to the specific qualities of the recaption data.
Interestingly, we also observe that the performance of CLIP is relatively insensitive to certain variations in the mix ratio $p$, as evidenced by the consistent enhancement over the baseline (*i.e*. $p$=1.0) across all four different cross-modal retrieval metrics when varying $p$ from 0.2 (80% recaption data) to 0.9 (10% recaption data).
For instance, setting $p$ at 0.9 and 0.2 both yields a similar performance enhancement of $\sim$3.5%, with the peak performance occurring at $p$=0.5, which delivers a substantial $\scriptstyle\sim$5% boost.

But meanwhile, we note that incorporating our recaptions (negatively) affects the zero-shot classification task, exemplified by the consistent performance degradation across varying $p$ values from 0 to 0.9. The phenomenon is similarly observed in the
recent work [] where they note directly fine-tuning on long text can significantly hurt the CLIP performance and therefore propose several techniques for enhancing learning with long texts.
In this study, given our primary focus is on assessing the quality of Recap-DataComp-1B, we choose the ratio $p=0.8$ to strike a promising balance between the classification performance (*i.e*., only marginally drops 0.5%) and the cross-modal retrieval performance (*i.e*., with a significant 3.4% boost on average) for later ablations.

**Table 4: Train with larger text encoder. We set $p=0.8$ for recaption-based models. We report zero-shot top-1 accuracy on ImageNet-1K and top-1 recall on COCO and Flickr30K.**
| vision encoder | text encoder | re-caption | ImageNet-1K | COCO R$@$1 | Flickr30K R$@$1 |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Validation | I$\rightarrow$T | T$\rightarrow$I | I$\rightarrow$T | T$\rightarrow$I |  |  |  |
| S/16 | small | ✗ | 60.7 | 49.2 | 30.1 | 73.5 | 53.3 |
| small | ✔ | 60.2 | 53.7 | 34.3 | 78.6 | 57.9 |  |
| *base* | ✔ | 61.7 +1.5% | 56.4 +2.7% | 34.8 +0.5% | 79.7 +1.1% | 59.1 +1.2% |  |
| B/16 | base | ✗ | 69.7 | 57.3 | 37.7 | 84.2 | 63,0 |
| base | ✔ | 69.2 | 61.5 | 42.2 | 85.2 | 66.9 |  |
| *large* | ✔ | 69.8 +0.6% | 62.9 +1.4% | 42.8 +0.6% | 86.7 +1.5% | 67.3 +0.4% |  |
| L/16 | large | ✗ | 74.1 | 60.2 | 41.9 | 86.0 | 68.5 |
| large | ✔ | 73.8 | 64.3 | 46.1 | 88.3 | 70.5 |  |
| *huge* | ✔ | 74.2 +0.4% | 66.0 +1.7% | 46.6 +0.5% | 89.9 +1.6% | 72.7 +2.2% |  |

### 5.3 Training with Larger Text Encoder

We hereby investigate how the size of the text encoder affects models trained on a mixture of the original captions and our recaptions (with $p=0.8$).
Specifically, we keep the architectural configuration of the vision branch as in Table 2 and only twitch the text encoder.
For instance, in the case of the S/16 model, we change from a smaller text encoder with 33M parameters to a larger, base-sized one with 53M parameters.

#### Main Results

Our results, as shown in Table 4, highlight that
enlarging the text encoder can further enhance performance across all model scales.
The average improvement for adopting a larger text encoder in retrieval tasks is 1.4%, 1.0%, and 1.5% for small, base, and large models, respectively, suggesting that larger text encoders can help the CLIP model learn better from semantically rich captions.

Moreover, we re-assess the balanced ratio of recaption data using a larger text encoder. Specifically, we gradually increase the ratio of recaption data from 20% to 50%, utilizing the Recap-CLIP-B/16 model with the large text encoder. The results are presented in Table 5. Compared to the prior results where an optimal ratio is achieved at $p=0.8$, using a larger text encoder can further push this optimal ratio to $p=0.6$. In other words, this result concludes that, compared to the vanilla version, a stronger cross-modal retrieval performance can be achieved if 1) more recaptions are used and 2) a larger text encoder is used.

### 5.4 More evaluations on text understanding

Recent works demonstrate that CLIP models suffer from poor long context understanding and delicate attribute understanding [, ]. Given the long, enriched, and better-aligned captions, we expect Recap-CLIP to exhibit better text understanding capability. Thus, we evaluate our Recap-CLIP model on two benchmarks: (1) Urban1K [], a long-caption image-text retrieval benchmark that contains 1k urban images and corresponding GPT-4V captions; (2) VG-Attribution [], a modified version of Visual Genome [] to test model abilities to attribute properties to objects. The results are shown in Tab. 6.

We observe consistent significant improvement if the model is trained on our Recap-Datacomp-1B dataset. For both text-to-image and image-to-text retrieval on Urban-1K dataset, our Recap-CLIP models surpass the vanilla baseline by at least 19% and sometimes up to an astonishingly high 36%. On the VG-attribution dataset, it is worth noting that our Recap-CLIP brings a performance boost very close to that of the NegCLIP fine-tuning [] (*e.g*. $\sim$9% *vs*. 10%), a lightweight downstream fine-tuning process designed to boost CLIP ability to understand attribute and order. Nonetheless, it is noteworthy that our Recap-CLIP is naturally equipped with better text understanding ability, without any specific targeted fine-tuning, indicating the importance of better captions in web-scale data.

**Table 5: Larger text encoder with different mixed ratios. We choose Recap-CLIP-B/16 with large text encoder for this ablation.**
| mixed ratio $p$ | ImageNet-1K | COCO R$@$1 | Flickr30K R$@$1 |  |  |
| --- | --- | --- | --- | --- | --- |
| Validation | I$\rightarrow$T | T$\rightarrow$I | I$\rightarrow$T | T$\rightarrow$I |  |
| 0.5 | 68.5 | 64.3 | 43.4 | 86.8 | 67.8 |
| 0.6 | 69.2 | 64.4 | 43.2 | 87.5 | 68.8 |
| 0.7 | 69.3 | 63.2 | 42.7 | 88.0 | 68.2 |
| 0.8 | 69.8 | 62.9 | 42.8 | 86.7 | 67.3 |

**Table 6: Comparison on the Urban-1K and VG-Attribute benchmark.**
| method | re-caption | Urban-1K | VG |  |
| --- | --- | --- | --- | --- |
| I$\rightarrow$T | T$\rightarrow$I | Attribute |  |  |
| OpenAI-CLIP-B/16 [] | ✗ | 67.4 | 53.3 | 62.6 |
| OpenCLIP-B/16 [] | ✗ | 62.5 | 63.1 | 59.9 |
| Recap-CLIP-B/16 | ✗ | 53.2 | 50.9 | 57.1 |
| ✔ | 85.0 +31.8% | 87.3 +36.4% | 66.4 +9.1% |  |
| Recap-CLIP-L/16 | ✗ | 69.8 | 64.6 | 60.1 |
| ✔ | 89.0 +19.2% | 91.8 +27.2% | 66.8 +6.7% |  |

## 6 Text-to-Image Generation

It has been known to the research community that training with generated (high-quality) pseudo-captions improves text-to-image generative models in terms of generation quality and prompt following ability [, , ],
primarily due to the low information and high noise density presented in the original web-crawled captions. Therefore,
we evaluate the quality of our generated captions by training Text-to-Image (T2I) generative models on Recap-DataComp-1B for further justification. We expect the enriched information in the generated descriptions to better align the visual content in images, and thus improve the performance of the T2I models.

Training. We adopt Diffusion Transformers (DiT) [] as our T2I model, where the text condition is firstly extracted with a CLIP text encoder [], and then injected into each DiT block with the cross-attention design. Specifically, we follow the image preprocessing pipeline in DiT [], where the images are preprocessed to have a square resolution of 256. The model is trained on visual latent extracted using a pretrained auto-encoder with a downsampling ratio of 8 []. Similar to the setup in previous experiments, the training text consists of a mixture of raw captions from Datacomp-1B, with a specified proportion $p$, and the rest of the captions replaced by refined captions from Recap-Datacomp-1B. Moreover, the training batch size is 2048, and the AdamW optimizer [] is used with a constant 1e-4 learning rate, without any warm-up schedule or weight decay. We name the resulting model Recap-DiT.

Evaluation. For sampling, we set the classifier-free guidance scale as 10 and use 250 DDPM steps to generate 30k images with captions from MSCOCO and our improved generated captions for zero-shot generation evaluation. We calculate Fréchet Inception Distance (FID) [] with the reference images from MSCOCO [] and CLIP score with both OpenAI ViT-B/32 model [] and our own Recap-CLIP ViT-L/16 model, following the established pipeline in prior T2I works [, , , , , , ].
Additionally, following the GPT-4V metric introduced in Section 4.2, we randomly select a subset of 3,000 our generated images for GPT-4V evaluation.

Main results. We report our observations in Tab. 7. Interestingly, when using raw COCO captions to generate 30,000 images for evaluation, the model trained with data integrated with our Recap-Datacomp (for $p<1$) demonstrates a better CLIP score, indicating improved vision-language alignment. However, there is no significant improvement observed in terms of FID. Our hypothesis is that the model adapts to the more informative and descriptive prompts, and could unleash its full potential only when similar informative testing prompts are provided.

Therefore, in another setting, we evaluate images generated using our LLaVA-1.5-LLaMA3-8B recaptioned version of the raw COCO captions. Here, we observe consistent and significant improvements in both FID and CLIP scores, particularly when more than half of the recaptioned data are integrated into the training dataset. Notably, models trained on Recap-Datacomp-1B ($p=0$) surpass those trained on the vanilla Datacomp-1B ($p=1$) by a large margin, with improvements observed in FID (-8.4), CLIP score (+3.1), Recap-CLIP score (+8.4), and GPT-4V score (+1.1). These observations justify that Recap-Datacomp-1B better reveals the potential of text-to-image models in generating images with high visual quality and improved alignment with textual conditions.

Larger models.
We further train a larger model, DiT-L/2, for 1 epoch with a mixed ratio of $p=0.0$, while keeping other training parameters unchanged. The model achieves an FID of 25.14 and a CLIP Score of 34.82. In Figure 4, we visually compare the generated results of DiT-L/2 and DiT-B/4 at $p=0.0$. It is evident that although the quantitative scores may not show substantial improvement, as we scale up the model, there is a noticeable enhancement in the alignment between the generated images and the corresponding text, *i.e*., this improved alignment results in higher-quality images that are able to capture and express more intricate details. These results confirm that DiT models trained on our recaption DataComp-1B exhibit robust scalability for text-to-image generative tasks.

Figure: Figure 4: Visual comparison of generate results from DiT-L/2 and DiT-B/4 at $p=0.0$, DiT-L/2 has better text comprehension and image generation than DiT-B/4. We mark entities in the instruction.
Refer to caption: x4.png

**Table 7: Text-to-Image evaluation on COCO-30K results of DiT-BASE/4, trained with different mix ratios on Recap-DataComp-1B. Note for GPT-4V Score, we use a subset of 3K for the evaluation.**
| Training | Evaluation |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| mixed ratio $p$ | Raw | Our COCO-Recap |  |  |  |  |
| FID$\downarrow$ | CLIP Score$\uparrow$ | FID$\downarrow$ | CLIP Score$\uparrow$ | Recap-Clip Score$\uparrow$ | GPT-4V Score$\uparrow$ |  |
| 0.00 | 37.6 | 29.2 | 27.8-8.4 | 32.5+3.1% | 28.3+8.4% | 2.53+1.1 |
| 0.05 | 38.5 | 29.1 | 27.9 | 32.5 | 28.0 | 2.51 |
| 0.10 | 36.0 | 29.7 | 27.2 | 32.7 | 28.2 | 2.51 |
| 0.15 | 35.8 | 29.9 | 28.2 | 33.0 | 28.1 | 2.45 |
| 0.20 | 35.8 | 29.8 | 28.4 | 32.7 | 28.0 | 2.53 |
| 0.50 | 35.3 | 29.3 | 30.2 | 31.9 | 26.7 | 2.13 |
| 0.75 | 31.3 | 29.4 | 32.7 | 31.2 | 25.8 | 1.89 |
| 1.00 | 32.5 | 28.9 | 36.2 | 29.3 | 19.9 | 1.40 |

## 7 Conclusion

This paper introduces Recap-DataComp-1B, a large-scale image dataset paired with detailed textual descriptions, generated using the LLaMA-3-powered Llava model. Our comprehensive analysis reveals that, compared to the original, web-crawled textual data, these generated descriptions align more accurately with their corresponding images and are more detailed. Utilizing Recap-DataComp-1B for training resulted in consistent enhancements across various models, notably CLIP, particularly in image-to-text and text-to-image retrieval tasks, and in text-to-image Diffusion models, specifically in their ability to follow more closely to user-provided text instructions. By providing this high-quality, publicly available, large-scale image-text dataset, we hope to inspire ongoing research and development that will push the boundaries of developing vision-language foundation models, more particularly in the open-source community.

### Acknowledge

This work is partially supported by a gift from Adobe, TPU Research Cloud (TRC) program, Google Cloud Research Credits program, AWS Cloud Credit for Research program, Edinburgh International Data Facility (EIDF) and the Data-Driven Innovation Programme at the University of Edinburgh.

## Peer Reviews

confidence: 5
contribution: 4
initial_rating: 8
presentation: 4
questions: Please see Weaknesses. Questions are included their with the corresponding "Weakness".
soundness: 3
strengths: S1. Well written and clear.

S2. Well positioned in the literature.

S3. Extremely non-trivial engineering effort, which I consider a strength.

S4. Very useful dataset contribution (both in terms of quantity of data and scale), opening many future avenues in synthetic data and studying empirical phenomenon associated with training on this kind of data.

S5. Simple approach allowing the authors to study empirical phenomenon, which is the heart of the science in this paper.

S6. Further validation of the LLaVA recipe and it's usefulness for creating VLMs
summary: The paper proposes a version of DataComp-1B wherein images are re-captioned by a LLaVA fine-tune of Llama3-8B. The paper trains CLIP models on this dataset demonstrating improvements in image retrieval and regressions on classification. The authors also demonstrate applications of the dataset for text2image generation.
weaknesses: W1. What is "Word distributions of our re-captions and the original captions." in Figure 1? Could you clarify this in the caption. Might also be too much information for this figure and you can consider moving that.

W2. Consider including some numerical outcomes of the dataset in the abstract/intro. e.g., training a CLIP model on [ours] shows a XX percentage point improvement when compared to training on [baseline] on [benchmark].

W3. Missing citation. MINT-1T. Awadalla, et al. 2024.

W4. More evaluation. Could be helpful to add some more evals, for example the average score from the DataComp paper for the CLIP models: https://github.com/mlfoundations/datacomp?tab=readme-ov-file#evaluation. Are there any salient trends in specific datasets and domains? For example, I imagine there could be boosts or hits in non-ImageNet tasks, which seem important to document.

W5. Can the proposed dataset be used in the LLaVA training mix, replacing the LAION fraction, to get a better LLaVA model (conceptually, as a data flywheel).

W6. Given that this is a large download, it could be good to provide statistics for the community about how many images were actually downloaded––to give the community an idea about the link rot affecting DataComp. 

W7. The method itself is not so different from that of other papers (e.g., Nguyen et al. Improving multimodal datasets with image captioning. 2023). However, given that the contributions of this paper are framed properly as a dataset and empirical results, I consider this a very minor weakness. In fact, if other reviewers complain heavily about novelty, I am willing to go to bat for this paper as I feel this criticism is misguided and overlooks the core contributions.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: - How is the current prompt chosen? How will different prompts for captioning influence the data quality?
- Compared to other captioning models on the image captioning benchmark, how much better is the LLaVA-1.5-LLaMA3-8B? Compared to the original LLaVA recaptioned (or other model recaptioned) datacomp image-text pairs, what is the advantage of LLaVA-1.5-LLaMA3-8B?
- In Table 3, what are the results when training on exclusive recaptions (p=0)? If the performance is worse than p=1, what are the possible reasons?
soundness: 2
strengths: - The computation on LLaMA-3-powered LLaVA training and recaptioning on 1B data is tremendous. The data generation and evaluation are at scale.
- The modified and trained LLaVA-1.5-LLaMA3-8B model achieves comparable performance with the LLaVA-1.5-13B model across benchmarks for various tasks, including recognition, spatial awareness, OCR, and so on.
- Quantitative analyses regarding caption features and semantic quality are conducted. Recap-DataComp-1B has longer, more diverse, and more aligned captions than the original ones. 
- The recaptioned data helps models with different sizes of text encoders.
- The recaptioned data enhances the image generation quality.
summary: This paper presents image recaptioning on web-crawled data empowered by LLaMA-3. The proposed LLaVA-1.5-LLaMA3-8B replaces LLM in LLaVA with LLaMA-3 and is trained following LLaVA-1.5's setting. To enhance the caption quality in DataComp-1B, it adopts LLaVA-1.5-LLaMA3-8B for recaptioning. Extensive experiments and analyses are conducted, including the features of recaptions, the mixture usage of original and recaptions, the efficacy on recognition and generation, and the impact across different sizes of architectures.
weaknesses: - The main innovation lies in the recaptioning process, which has been adopted in data synthesis for better quality [1,2] and representation learning [3]. It is especially similar to [4], image captioning on DataComp, but provides fewer insights. Although the scale is at the billion level, the technical contribution is limited. 
- The prompt will impact the quality of image captioning. In particular, the paper's main contribution is image recaptioning. It would be better if they explored more than one prompt. These experiments do not have to be at the 1B scale. It could be on a reasonable-scale subset.
- Even though the proposed LLaVA-1.5-LLaMA3-8B is better than the previous LLaVA models on the MMMU and MM-Vet benchmarks (understanding/reasoning benchmarks), better image captioning ability and the necessity of LLaMA-3 are under verification. Additionally, there are no comparisons with recaptioned data from other captioning models. It is doubtful whether the data improvement comes from the captioning itself or it actually benefits from LLaMA-3. 
- It seems that the recaptioned dataset leads to much worse performance on ImageNet-1K and similar performance on the retrieval tasks. Table 3 illustrates model performance by training on different portions of original captions. Though the pure reception result (p=0) is not reported, inferring from the trend, it is likely to be worse than pure original caption-trained performance (p=1). The efficacy of recaptions is questionable. 

[1] Improving Image Generation with Better Captions.
[2] A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image Generation.
[3] LLaVA-NeXT: What Else Influences Visual Instruction Tuning Beyond Data?
[4] Improving multimodal datasets with image captioning.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Could the authors provide the total inference time on recaptioning the DataComp-1B?
soundness: 2
strengths: 1. The downstream tasks are comprehensive. The authors study the capabilities of recaptioning through different downstream tasks, including zero-shot classification, retrieval and text to image generation. In most of the benchmarks, recpationing procedure brings positive effects. 
2. The authors conduct experiments on different model sizes, showing the consistent improvements in retrieval tasks. 
3. The recaptioned DataComp-1B can potentially help the development of VLMs in the future.
summary: Web-crawled image-text pairs are noisy and usually are not descriptive enough. This paper studies recaptioning DataComp-1B to obtain detailed and descriptive for images and shows the recaptions can bring benefits to downstream vision-language tasks such as retrieval and text-to-image generation. The authors first fine-tune a LLaMA-3 based LLaVA-1.5, then use it as an image captioner to generate long, detailed captions for images in DataComp-1B. They analyze the lengths distribution before and after this recaptioning step in DataComp-1B, showing the recaptions are longer. They also use LongCLIP and GPT4V to validate the image-text alignment and caption fluency are increased by this recaptioning step. Then, they use the recaptioned DataComp-1B, named Recap-DataComp-1B, to train CLIP models (Recap-CLIP) from scratch. Specifically, they mix the original captions and recaptions in DataComp-1B to find the best way to use the recaptions. In the experimental section, they evaluate the Recap-CLIP on downstream tasks including zero-shot imagenet classification and the text-to-image/image-to-text retrieval tasks. They find the detailed captions can benefit the retrieval tasks but negatively influence the zero-shot classification task. They also conduct experiments on long-caption image-text retrieval benchmarks. To further validate the quality of the recaptions, they train T2I models using recaptions and find the detailed recaptions can help T2I generation.
weaknesses: 1. Table 3 shows the ImageNet-1K zero-shot classification results and COCO/Flickr30k retrieval results. The authors should also provide the results when mixed ratio p = 0, which means the performances from the model trained using only recaptions. Obviously, the recaptions bring negative effects in the zero-shot classification task. The authors discuss the possible reason may be the missing of specific named entities. For example, in Figure 1, the example "Western Kingbird" is a valid bird class for the image, but in the detailed caption, it is described as "A small, gray and yellow bird with a black beak and black eyes...". Have the authors considered to generate the detailed captions based on the original caption as well? If the original caption can also be provided in the inputs, with a proper prompt, maybe the specific named entities can also be kept in the recaptions. 
2. In Table 6, it would be better if the authors can compare with MetaCLIP as well.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - In Sec 6, why to use the raw CLIP text encoder instead of a encoder trained with recaptioned text? 

- Compared with retrieval, generation preferred recaptioning since the best performance happens in mixed ratio p = 0.00 in Tab 7. What is the reason? 

- Some experimental details are unclear. For instance, what does “re-caption” denote? Using re-captioned prompts for training or inference or both?
soundness: 3
strengths: + Effective recaptioning pipeline improves both discriminative and generative model performance.

+ It shows that the open-source model LLaMA-3 enables accessible advancements in dataset quality.

+ Demonstrated benefits in real-world applications, notably in zero-shot retrieval and text-to-image generation tasks.
summary: This paper addresses the challenge of noisy web-crawled image-text pairs, which can undermine the performance of vision-language models. The authors propose a straightforward pipeline that uses a fine-tuned, open-source LLaMA-3-8B language model to recaption around 1.3 billion images in the DataComp-1B dataset. By improving textual descriptions, their approach, termed Recap-DataComp-1B, provides a cleaner, more semantically aligned dataset that enhances vision-language model training. Empirical results show performance improvement: for discriminative models like CLIP, zero-shot performance in cross-modal retrieval tasks improves, while for generative models such as text-to-image Diffusion Transformers, image generation aligns more closely with user instructions.
weaknesses: - The work does not propose new ideas or methods and focuses on empirical verification, but it seems not provide in-depth or insightful experimental analysis, limitations (e.g., hallucination) and strengths of recaptioning, or sufficient valuable practices beneficial to the community. Most of conclusions (e.g., recaptioning is helpful for image generation in DALLE 3) have been known or have been explored in prior work. 

- Is the training with recaptioning harmful to other aspects, e.g., knowledge. Specifically, for example, “Western Kingbird” is recaptioned into a detailed appearance-oriented description. After trained with these recaptions, would the model know the concept “Western Kingbird”? Maybe some evaluations on knowledge-intensive VQA are needed. 

- Using LongCLIP-Large to evaluate semantic alignment between long captions and images may be improper since long sentences may be preferred by this model than short ones. 

- The cost of recaptioning need to be detailed. 

- The section names, e.g., CLIP in Section 5, are not informative enough. 

- Some challenging benchmarks should be considered, e.g., winoground [a] for retrieval and T2I-CompBench [b] for generation. 

[a] Thrush T, Jiang R, Bartolo M, Singh A, Williams A, Kiela D, Ross C. Winoground: Probing vision and language models for visio-linguistic compositionality. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2022 (pp. 5238-5248).

[b] Huang K, Sun K, Xie E, Li Z, Liu X. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems. 2023 Dec 15;36:78723-47.

Meta-review: This paper shows the empirical impact of the re-captioned DataComp-1B dataset with LLaVA-1.5 fine-tuned based on Llama3-8B. Using the re-captioned DataComp-1B, this paper achieves a better CLIP model in terms of ImageNet classification and cross-modal retrieval. In addition, this paper shows that by choosing the proper mixing ratio between the re-captioned captions and the original captions, the text-ot-image generation (DiT) shows better FID and CLIP scores.

This paper has mixed opinions, mostly negative ones. There were several concerns raised by the reviewers. I listed the main concerns of the reviewers and my opinions on each item.

- Lack of novelty (XLPB, mTNK):  I partially agree that the use of re-generated captions for training is not a novel contribution. However, the main contribution of this paper is more empirical; as the title, we do not know "What if We Recaption Billions of Web Images with LLaMA-3?" This paper investigates this question and if the re-captioned dataset and trained models on the dataset show important empirical findings, I think this paper has enough empirical contribution. However, as my following opinions, I think the empirical findings are somewhat weak to be published as an ICLR paper as is.
- Lack of comparison benchmarks (mTNK): I partially agree with this comment. The authors provided an additional result based on 38 tasks by DataComp. In the experiments, we can observe that the CLIP model trained on the re-captioned DataComp-1B performs worse in 14 VTAB tasks with a non-neglectable gap (67.4 vs. 65.9). This shows that the re-captioning does not perform better than the original caption, contradictory to the main argument of this submission.
- It is unclear whether the main contribution comes from LLaMA-3 or re-captioning (XLPB, mTNK): Similar to the first item, this could be handled if there exists enough empirical findings. I think comparing all the possible LLMs or captioners for this task is not feasible. However, it is also true that the current design choices (e.g., the captioning model and its base LLM and generation prompt) are not rigorously studied. I understand that this costs a lot (it already takes 29 days with TPU-V4 256), but we may need a more systematic approach for this. The additional results provided by the authors during the rebuttal may address this concern partially, but the revised paper does not include this result.

Overall, I think that although this paper tackles a very interesting question "What if We Recaption Billions of Web Images with LLaMA-3?", but the answer by this paper looks somewhat weak. Namely, it is unclear what we can expect from the re-captioned dataset and the trained model by the dataset. It is because (1) to get a better result, we need many design choices, especially for the mixing parameter $p$, (2) the newly trained model is not overwhelming the existing model (it shows a worse performance for VTAB), and (3) the overall re-captioning process is too expensive (~29 days with TPU-V4 256). Considering the expensive prediction cost of re-captioning and parameter search cost (different $p$ needs different CLIP training), the improvement is somewhat weak and the finding looks somewhat weak.

Decision: Reject