Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models

## Abstract

Abstract Recent advancements in multimodal models highlight the value of rewritten captions for improving performance, yet key challenges remain. For example, while synthetic captions often provide superior quality and image-text alignment, it is not clear whether they can fully replace AltTexts: the role of synthetic captions and their interaction with original web-crawled AltTexts in pre-training is still not well understood. Moreover, different multimodal foundation models may have unique preferences for specific caption formats, but efforts to identify the optimal captions for each model remain limited. In this work, we propose a novel, controllable, and scalable captioning pipeline designed to generate diverse caption formats tailored to various multimodal models. By examining Short Synthetic Captions (SSC) towards Dense Synthetic Captions (DSC+) as case studies, we systematically explore their effects and interactions with AltTexts across models such as CLIP, multimodal LLMs, and diffusion models. Our findings reveal that a hybrid approach that keeps both synthetic captions and AltTexts can outperform the use of synthetic captions alone, improving both alignment and performance, with each model demonstrating preferences for particular caption formats. This comprehensive analysis provides valuable insights into optimizing captioning strategies, thereby advancing the pre-training of multimodal foundation models.

## 1 Introduction

Figure: Figure 1: The role of image-text data in multimodal foundation models: a key component in training CLIP and Diffusion Model, and essential for multimodal LLM (MLLM) pre-training alongside text and interleaved image-text data. We propose a controllable captioning pipeline to synthesize different types of captions and explore optimal image-text data recipes for training these foundation models.
Refer to caption: extracted/5897046/Figure/Intro_v2.png

Large-scale image-text datasets have been crucial in advancing multimodal foundation models. For instance, CLIP (Radford et al., ) is pre-trained on 400 million image-text pairs collected from the Web. However, web-crawled data, particularly AltText, often suffer from insufficient visual details and noisy content, as illustrated in Fig. 1. Recent studies highlight the benefits of synthetic captions, which provide better image-text alignment and improved data quality. Research on LaCLIP (Fan et al., ) and ShareGPT4V (Chen et al., ) demonstrates that synthetic captions can improve the performance of CLIP and multimodal large language models (MLLMs), respectively. This raises a key question: if higher-quality synthetic captions can be generated, could they fully replace web-crawled AltText? Should we consider disregarding AltText altogether?

To investigate this question, we first adopt the approach of VeCLIP (Lai et al., ) and train CLIP using synthetic captions generated by LLaVA (Liu et al., ). Similar to the results discussed in Li et al. (), training CLIP fully on synthetic captions of higher quality degrades CLIP’s performance significantly: as shown in Fig. 2, when compared to using only AltText, the use of LLaVA captions results in a substantial drop on zero-shot ImageNet classification tasks. However, after combining original noisy AltText and LLaVA captions, we achieve the best results in both classification and retrieval tasks. This observation raises a critical question: what constitutes the optimal image-text data for multimodal foundation models? Despite its importance, research on the interplay between synthetic captions and AltText remains limited. Our findings suggest that while rewriting techniques can enhance image-text alignment, they may reduce data diversity due to dependence on a limited set of LLMs or MLLMs for caption generation. Specifically, since CLIP is a foundational vision model that benefits from learning diverse concepts, relying on synthetic captions can potentially hinder CLIP’s training due to lack of diversity in vocabulary and mentioned concepts (Fan et al., ).

In addition to the role of AltText, another open question concerns the optimal formats for synthetic captions. For instance, advanced multimodal models like LLaVA-NeXT (Li et al., ) indicate that recaptioned datasets are advantageous during stages focused on high-quality knowledge acquisition. DALL-E 3 (Betker et al., ) demonstrates that using 95% synthetic captions can yield superior results, particularly when the captions are highly descriptive. Similarly, MM1 (McKinzie et al., ) shows that even a small fraction (7%) of high-quality caption data can significantly boost few-shot performance. Given these insights, our work focuses on two key unresolved questions: 1) What is the role and value of synthetic captions, and how do they interact with the original AltText? 2) What types of synthetic captions are most effective for different foundation models? To address the first question, we revisit why prior works (Betker et al., ; Lai et al., ; McKinzie et al., ) continue using noisy web-crawled AltText, even when rewritten captions are available during training. Intuitively, since CLIP is a straightforward model pre-trained on image-text pairs, highly aligned captions should be advantageous. However, relying solely on synthetic captions may actually degrade CLIP’s performance, as shown in Fig. 2(a).
For the second question, we investigate the effects of Short Synthetic Captions (SSC) and Descriptive Synthetic Captions (DSC) on CLIP. As depicted in Fig. 2(b), surprisingly, more descriptive captions yield inferior results compared to shorter captions for CLIP training, despite their greater detail.

Figure: Figure 2: Zero-shot retrieval and classification performance of CLIP models. (a) The effect of synthetic captions (LLaVA recaptioned) and AltText: solely using LLaVA captions can improve retrieval tasks but significantly deteriorate the zero-shot classification performance. (b) The effect of different formats of synthetic captions on CLIP: Short Synthetic Captions (SSC) show superior results to Descriptive Synthetic Captions (DSC) and the combination of them achieves the best results.
Refer to caption: extracted/5897046/Figure/fig1a.png

To explore these insights further to address the two questions, we introduce a novel, controllable, and scalable captioning pipeline that enables the generation of diverse caption formats at scale, tailored to the specific needs of different multimodal foundation models. Our pipeline is designed to build large-scale image-text data for the pre-training stage in a scalable way.
With this pipeline, we use SSC and DSC as two main examples on how to customize the captioning format. Our pipeline can serve as a cost-effective alternative to GPT-4V for generating high-quality captions. To solve the second question, we conduct a comprehensive study on the effectiveness of different types of synthetic captions across a range of foundational models and downstream tasks. Our approach involves a systematic evaluation of various captioning strategies, including SSC, DSC, and mixed training methods that combine original AltText with synthetic data. We seek to determine the optimal captioning techniques for specific models, such as CLIP, diffusion models, and multimodal LLMs, and to assess their impact on both model performance and data diversity. Furthermore, we investigate the interaction between synthetic captions and original AltText, analyzing whether a hybrid approach can balance the need for diverse data with the benefits of enhanced image-text alignment.

Overall, our contributions are summarized as follows.

- •
We explore the MLLM as the image describer and present a controllable and human-aligned captioning pipeline to convert MLLM into an image captioner.
- •
We synthesize several formats of captions including Short Synthetic Captions (SSC) towards Dense Synthetic Captions (DSC+), then conduct extensive pre-training experiments to systematically study the role of synthetic captions and their intersection with original AltText across three multimodal foundation models.
- •
We verify the image-caption training recipe that 1) AltText provide data variety and synthetic captions provide better image-text alignment, 2) different foundation models have their own preferred formats, which highlights the necessity and importance of the controllable captioning pipeline in building multimodal foundation models.

## 2 Related Work

Multimodal Foundation Models. CLIP (Radford et al., ) is one of the pioneering multimodal foundation models connecting images and text. By training on 400 million image-text pairs, CLIP shows strong zero-shot image classification and retrieval capabilities. It lays the groundwork for the development of more advanced multimodal foundation models, such as multimodal large language models (MLLMs) (Liu et al., ; Wang et al., ; Chen et al., ; Tong et al., ; Zhang et al., ) for vision-language understanding and diffusion models (Rombach et al., ; Podell et al., ) for text-to-image generation. These advanced models often utilize CLIP’s vision tower as their vision encoder.

Improving Image-Text Data. Web-crawled image-text data often suffer from issues like image-text misalignment and poor-quality textual descriptions (Lai et al., ; Li et al., ). There are two common ways for improving image-text data: 1) data filtering based methods remove low-quality data such as misaligned image-text pairs by human-assisted systems (Yu et al., ; Sun et al., ) or pre-trained models (Li et al., ; Schuhmann et al., ; Gadre et al., ; Fang et al., ); 2) data recaptioning based methods usually leverage a LLM to rewrite the original caption or a MLLM to rewrite a caption for the image. For example, ShareGPT4V (Chen et al., ) uses GPT-4V to write highly descriptive captions for their images. LaCLIP (Fan et al., ) leverages several LLMs to rewrite captions with different writing styles for data diversity. Recap-DataComp-1B (Li et al., ) uses a LLaMA-3 based model to scale the captions. Different from the aforementioned works, we mainly focus on generating different types of captions and exploring 1) the format of ideal captions needed for each multimodal foundation model and 2) a systematic analysis of the intersection between AltText and synthetic captions.

## 3 Customized Re-captioning for Multimodal Foundation Models

Figure: Figure 3: Examples of controllable captions of diverse formats generated by our captioner: we can generate from brief to dense descriptions and fuse AltText into the caption (AFC).
Refer to caption: extracted/5897046/Figure/captioner_examples_v3.png

Image-text data are fundamental to the success of multimodal foundation models, serving as a bridge between visual and textual representations.
For example, CLIP (Radford et al., ) is pre-trained on 400M web-crawled image-text pairs, enabling it to learn rich, transferable representations that can be applied to various downstream tasks.
The importance of precise and detailed captioning is further highlighted in LLaVA-NeXT (Li et al., ), where re-captioned detailed descriptions are utilized for model training at Stage-1.5, enhancing the model’s ability to understand and generate nuanced content. Similarly, DALL-E 3 (Betker et al., ) shows that the prompt-following capabilities of text-to-image models can be significantly improved by training on highly-descriptive generated image captions. This shows the critical role of captions in shaping a model’s capacity to align visual and textual information, ultimately improving performance across a wide range of multimodal tasks. However, the optimal captioning strategy for different foundational models remains under-explored. To address this gap, we introduce a novel, controllable, and scalable captioning pipeline designed to generate diverse caption formats at scale, supported by evaluation metrics that ensure high CLIP scores and minimal hallucination.
We summarize the capability of our captioning model by generating the following formats of captions as shown in Fig. 3:

- •
Short Synthetic Caption (SSC): a concise sentence that describes the primary subject of the image.
- •
Descriptive Synthetic Caption (DSC): a description limited to 78 tokens, emphasizing the central subject and key visual elements.
- •
Dense Synthetic Caption (DSC+): a more comprehensive description detailing the main subject along with the background, setting, and any significant objects or actions.
- •
AltText Fusion Caption (AFC): a caption similar to DSC, but integrated with AltText where appropriate. This type of caption removes unnecessary details often found in AltText, offering a cleaner, more cohesive description than a simple concatenation of AltText and synthetic caption.

### 3.1 MLLM as An Image Describer

VeCLIP (Lai et al., ) employs LLaVA (Liu et al., ) for image captioning, while ShareGPT4V (Chen et al., ) utilizes GPT-4V for this task.
Compared to traditional image captioners like BLIP (Li et al., ), MLLMs offer several advantages and are natually good image describers. MLLMs can generate longer and more detailed captions, as demonstrated by LLaVA-NeXT (Li et al., ), where a 34B model was used to produce highly descriptive captions. This ability to generate more context-aware descriptions stems from the integration of a large language model (LLM) with a vision encoder. This combination allows MLLMs to capture fine-grained visual details and complex inter-object relationships. Additionally, MLLMs benefit from their multi-stage training process, which combines pre-training on large-scale datasets and supervised fine-tuning for downstream tasks. These characteristics make MLLMs a powerful tool for generating more descriptive captions.
However, directly using MLLM as an image describer may have two major limitations: 1) MLLM may not strictly follow the instruction to generate a specific format of caption (Liu et al., ); 2) these instruction fine-tuned MLLMs tend to hallucinate. As shown in Fig. 4, both LLaVA (Liu et al., ) and ShareGPT4V (Chen et al., ) fail to describe the image using only three sentences but generate hallucinated contents (highlighted in red).
Although GPT-4V shows stronger capability and many works use it for captioning, the scalability remains limited due to the cost. Therefore, in this work, we focus on building a cost-effective captioner instead of using GPT-4V.

Figure: Figure 4: Directly using MLLMs as image captioners may result in hallucinations and generate captions that do not align with specific instructions: both LLaVA (Liu et al., ) and ShareGPT4V (Chen et al., ) generate over three sentences and obvious hallucination.
Refer to caption: extracted/5897046/Figure/exist_captioner.png

To alleviate the above two limitations, we first investigate the origins of hallucination in MLLMs, hypothesizing two primary sources: 1) inherent limitations of the LLM, and 2) the quality of supervised fine-tuning (SFT) datasets, which are often themselves synthetically derived or processed. We focus on the latter, proposing that mitigating hallucinations at the dataset level is essential for converting an MLLM into an effective captioner. We also address the format-following issue by fine-tuning the MLLM on a curated captioning-specific dataset, transforming it into a purpose-built captioning model to generate diverse captions.

### 3.2 Two-Stage Human-Aligned Captioning

Stage 1: transforming MLLM into a customized captioner. To minimize hallucinations from MLLMs, we begin by constructing a clean and precise fine-tuning dataset. Instead of relying on GPT-4 generated data, we curate a high-quality dataset of human-annotated image-text pairs, named Stage-1-1M. This dataset contains short, human-curated captions, with five captions per image. Additionally, we integrate an OCR detection model to extract in-image text, which, alongside the captions, is fed into an LLM for summarization and controlled rewriting. By employing a strict prompt, we prevent the LLM from introducing extraneous information, while few-shot prompts guide the model in generating various caption formats, including both concise and descriptive styles. We further enhance the dataset through post-processing, using heuristic and model-based quality checks to improve its overall quality. This comprehensive fine-tuning on the Stage-1-1M dataset transforms the MLLM, specifically the 3B version of MM1 (McKinzie et al., ), into a customized captioner that aligns closely with the intended output characteristics.

Stage 2: Human-aligned further fine-tuning.
While the Stage-1-1M dataset effectively establishes a foundation, it lacks the depth required for more descriptive captioning tasks. In Stage 2, we address this by incorporating descriptive human-annotated data to enhance caption diversity and quality. We curate a new dataset, named Stage-2-HA, specifically designed for detailed captioning. This dataset is meticulously annotated by human experts to capture nuanced visual elements and complex scene descriptions. Post-annotation, we leverage an LLM to reformat these captions into multiple stylistic variations, including both concise and richly descriptive formats. By imposing strict constraints during the LLM processing, we ensure alignment with human-generated content, avoiding the pitfalls of hallucination. The captioner is then fine-tuned on the Stage-2-HA dataset, resulting in a highly refined and human-aligned captioning model capable of generating captions tailored to specific use cases. This dual-stage fine-tuning process not only enhances the model’s adaptability but also ensures a balance between brevity and descriptiveness, making it a versatile tool in controllable captioning. The overview of the complete captioning pipeline is shown in Fig. 5.

Figure: Figure 5: Overview of the controllable and human-aligned captioning pipeline. In Stage 1, we convert a pre-trained MLLM into a customized captioner that strictly follows the captioning instructions. In Stage 2, we leverage human-aligned captions to further fine-tune the captioner.
Refer to caption: extracted/5897046/Figure/captioner_v2.png

### 3.3 Caption Analysis

Figure: Figure 6: Distribution of token lengths of our generated captions in four formats: we caption COCO 2017 images and visualize their distributions.
Refer to caption: extracted/5897046/Figure/short_synthetic_caption_ssc.png

Figure: Figure 7: The number of unique entities in different synthetic captions (randomly sample 17.5k images) compared to AltText: AltText provides more unique entities as wider knowledge.
Refer to caption: extracted/5897046/Figure/ssc_vs_alt_text.png

Richness assessment: token length and average number of assertions.
Fig. 6 illustrates the distribution of the number of tokens of various caption types generated in this study. Specifically, SSC mainly ranges from 10 to 15 tokens, while DSC spans from 40 to 60 tokens, both fitting within the text encoder’s capacity in CLIP. In contrast, most DSC+ captions exceed 100 tokens. Besides that, we propose Average Number of Assertions (ANA) to quantify the richness of captions. We prompt an LLM to generate different assertions of a caption to analyze our different formats of captions in terms of the richness. More details of this approach is in Appendix. Note that the ANA for SSC is 2.49, DSC as 8.13 and DSC+ as 12.20, showing more visual contents.

Diversity assessment: number of unique entities in captions.
We hypothesize that the original, albeit noisy, AltText may carry a broader range of diverse information and knowledge, offering potential advantages for CLIP’s pre-training. To assess this diversity, we quantify the number of unique entities present in the captions and present a visualization in Fig. 7. Our analysis shows that AltText contains a higher number of unique entities, which could be beneficial in providing a wider knowledge base.

## 4 Image-Caption Data for Multimodal Foundation Models

In this section, we mainly discuss three foundation models: CLIP, multimodal LLM, and diffusion models. For both CLIP and diffusion models, since the text encoder is limited to 77 tokens (Zhang et al., ), we focus primarily on SSC and DSC. For multimodal LLM, we explore more detailed versions, including DSC+ and AFC. We summarize our key findings below:

- •
The tradeoff between the richness of captions and their accuracy needs to be balanced based on the multimodal tasks.
- •
Both AltText and synthetic captions are important for CLIP training, with shorter captions yielding better performance. Linear probing is an additional effective way to evaluate the representations.
- •
Pre-training and SFT benchmarks can behave differently in multimodal LLMs. On the SFT benchmarks, MM1 shows a preference for DSC+ alone.
- •
For diffusion models, DSC emerges as the most effective captioning strategy.

### 4.1 Image-Caption Data for CLIP

We use VeCap-300M (Lai et al., ), a web-crawled dataset with raw AltText as our main pre-training dataset for CLIP. Besides AltText, we generate several synthetic caption datasets for the study. Then, we use ViT-B/16 as the vision encoder. The training details can be found in Appendix.

**Table 1: Effect of different synthetic captions on CLIP with ViT-B/16 as the backbone. IN: ImageNet.**
| Table 1: Effect of different synthetic captions on CLIP with ViT-B/16 as the backbone. IN: ImageNet.<br>Pre-train Caption<br>COCO (R@1)<br>Flickr30k (R@1)<br>IN<br>INV2<br>I-T<br>T-I<br>I-T<br>T-I<br>AltText<br>54.24<br>36.98<br>81.30<br>65.80<br>65.70<br>58.58<br>DSC<br>52.28<br>29.00<br>80.90<br>54.75<br>27.30<br>21.91<br>SSC<br>57.00<br>35.15<br>84.67<br>63.35<br>49.10<br>43.31<br>AFC<br>54.82<br>34.84<br>84.00<br>62.18<br>38.98<br>35.11<br>DSC + AltText<br>65.84<br>46.08<br>90.26<br>73.94<br>66.18<br>58.74<br>SSC + AltText<br>66.67<br>48.13<br>91.81<br>76.54<br>66.63<br>59.57<br>AFC + AltText<br>63.98<br>43.76<br>89.10<br>73.32<br>66.47<br>58.84<br>All Synthetic + AltText<br>70.12<br>50.21<br>93.00<br>77.72<br>64.91<br>57.92 | Table 2: Evaluation with linear probing (LP) on ImageNet for CLIP.<br>Pre-train Caption<br>Zero-shot<br>LP<br>Gain<br>AltText<br>65.70<br>78.34<br>+12.64<br>DSC<br>27.30<br>75.72<br>+48.42<br>SSC<br>66.63<br>79.94<br>+13.31<br>AFC<br>38.98<br>75.53<br>+36.55<br>DSC + AltText<br>66.18<br>79.96<br>+13.78<br>SSC + AltText<br>66.63<br>79.94<br>+13.31<br>AFC + AltText<br>66.47<br>78.70<br>+12.23<br>DSC + SSC + AltText<br>65.16<br>80.01<br>+14.85<br>All Synthetic + AltText<br>64.91<br>79.55<br>+14.64 |
| --- | --- |

Effect of synthetic captions. We first study the effect of Short Synthetic Captions (SSC) and Descriptive Synthetic Captions (DSC).
Results are summarized in Table 2.
Interestingly, while SSC enhances retrieval performance, it leads to a substantial drop in zero-shot ImageNet accuracy. Moreover, despite demonstrating that synthetic captions have superior image-text alignment and reduced noise, the more descriptive captions (DSC) perform worse across all benchmarks. Based on these results, DSC appears suboptimal for CLIP training, leading to inferior performance. For example, despite DSC capturing more visual concepts within the captions, its zero-shot performance shows a significant degradation of 21.8% compared to SSC. We hypothesize this performance drop may be partially due to a distribution mismatch as the prompts in COCO/Flickr30k/ImageNet datasets are short (e.g., “a photo of {}”).

To further explore the performance gap between DSC and SSC on CLIP, we also use linear probing, which provides a direct measure of the quality and generalization capability of the representations learned by CLIP. Strong performance from a linear classifier on specific tasks indicates that the pre-trained model has effectively captured relevant and discriminative features, underscoring the robustness of its embeddings.
We summarize the results on linear probing in Table 2. Even though DSC and SSC show lower zero-shot performance, they achieve comparable results to AltText after linear probing, indicating similar pre-trained representations.
This indicates that the relatively poor zero-shot results of DSC can be significantly improved with linear probing, implying that representations learned from DSC are richer than initially presumed.

Intersection of synthetic captions and AltText. From Table 2, we find training CLIP solely on large-scale synthetic captions may get inferior results compared to the original AltText, even though synthetic captions have better image-text alignment. We hypothesize that synthetic captions rewritten from a MLLM may hurt the diversity and knowledge coverage of the original AltText. Therefore, we blend our synthetic captions and AltText, such as DSC + AltText and SSC + AltText: there is a significant boost in retrieval performance—e.g., over 10% improvement on COCO. Additionally, SSC + AltText also leads to gains in ImageNet accuracy. Utilizing a mixture of all synthetic caption formats yields the best performance in retrieval tasks, likely due to the wider entity-based knowledge, as visualized in Fig. 7.

Figure: Figure 8: The intersection of synthetic captions and AltText for CLIP. We gradually increase the proportion of SSC mixed with AltText during training. All experiments use ViT-B/16 as the backbone and VeCap-300M (Lai et al., ) as the pre-training dataset.
Refer to caption: extracted/5897046/Figure/clip_ratio_study.png

Optimal mixture ratio between synthetic captions and AltText.
Considering the wider knowledge of AltText (Fig. 7) and the better alignment of synthetic captions, we explore the optimal mixture ratio.
Specficially, we use SSC and AltText from VeCap-300M (Lai et al., ) as an example, with results shown in Fig. 8. A ratio of 0 corresponds to using only SSC, while 100 corresponds to using only AltText. We observe that CLIP achieves optimal performance across both retrieval and classification tasks when the ratio is tuned to around 40-50%. A lower proportion of AltText leads to a drop in retrieval performance, whereas a lower proportion of SSC results in decreased accuracy in ImageNet zero-shot classification.
This observation aligns with findings in Li et al. (). From this, we verify that AltText provides broader knowledge coverage and greater diversity, which benefits CLIP’s pre-training by enabling the model to grasp a wider range of concepts. This diversity may serve as a foundation for generalization, allowing CLIP to better represent varied contexts and domains in zero-shot classification.

Exploration of the optimal way of using AltText: AFC vs. simple mixture. As shown in Fig. 8 and Table 2, we verify that a simple mixture of our synthetic captions and AltText can achieve superior results.
In addition to this straightforward combination, we investigate alternative methods of incorporating AltText into the training process. One promising approach is to fuse the knowledge from AltText directly into the synthetic captions. To enable this, we fine-tune our captioner with instruction-following capabilities, generating AltText Fusion Captions (AFC). While AFC shows improvement over DSC due to the enriched AltText information, it underperforms when compared to the simple mixture of DSC and AltText.

### 4.2 Image-Caption Data for Multimodal LLM

**Table 3: The effect of synthetic captions on MM1 (1.2B model) pre-training.**
| Table 3: The effect of synthetic captions on MM1 (1.2B model) pre-training.<br>Pre-train Caption<br>TextCore<br>0-Shot<br>4-Shot<br>8-Shot<br>AltText<br>53.48<br>34.80<br>55.81<br>59.72<br>DSC<br>53.76<br>37.05<br>59.36<br>63.68<br>DSC + AltText<br>53.71<br>37.09<br>60.31<br>63.96<br>SSC<br>53.46<br>37.35<br>59.19<br>63.56<br>SSC + AltText<br>53.20<br>37.16<br>58.22<br>62.22 | Table 4: The ratio ablation on MM1 pre-training between synthetic captions and AltText.<br>Mixing Ratio<br>TextCore<br>0-Shot<br>4-Shot<br>8-Shot<br>33/66<br>53.86<br>35.16<br>59.69<br>63.97<br>50/50<br>53.86<br>36.40<br>60.17<br>64.13<br>66/33<br>54.24<br>37.02<br>60.26<br>64.71<br>80/20<br>53.96<br>35.74<br>60.09<br>64.69<br>100/0<br>54.19<br>15.54<br>60.24<br>64.02 |
| --- | --- |

As we study large-scale image-caption data for MLLMs, we use MM1 (McKinzie et al., ) as one example and focus on the pre-training stage. MM1 (McKinzie et al., ) claims that captioning data lift the zero-shot performance and synthetic captions are helpful for few-shot learning. Based on this insight, we further study the captioning data recipe on how to balance the use of original AltText and synthetic captions.
We follow the pre-training setup and the evaluation benchmark in MM1 (McKinzie et al., ) to report TextCore and 0/4/8-shot performance.
All of the experiments are conducted on the 1.2B model. We pre-train the model with 50K steps and the batch size is 512. More details are in Appendix.

**Table 5: SFT evaluation of models pre-trained with different types of captions. We use the same SFT evaluation benchmarks as in MM1 (McKinzie et al., ). (*): Concatenation of two captions.**
| Pre-Trained Data | VQAv2 | VQA${}^{\text{T}}$ | MMMU | MathV | MME${}^{\text{P}}$ | MME${}^{\text{C}}$ | SEED | POPE | LLaVA${}^{\text{W}}$ | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| AltText | 77.1 | 66.1 | 31.4 | 28.3 | 781.7 | 225.4 | 61.7 | 84.6 | 69.8 | 57.6 |
| LLaVA Caption | 78.0 | 65.4 | 30.0 | 27.6 | 773.4 | 200.4 | 63.5 | 83.7 | 63.6 | 56.5 |
| SSC | 78.8 | 65.8 | 33.6 | 28.2 | 760.2 | 216.4 | 63.1 | 84.4 | 69.0 | 57.9 |
| DSC | 77.1 | 61.8 | 30.8 | 27.9 | 596.6 | 213.6 | 64.6 | 83.9 | 70.8 | 56.3 |
| DSC+ | 79.0 | 65.4 | 32.6 | 29.4 | 727.2 | 224.3 | 66.8 | 85.1 | 71.5 | 58.7 |
| AltText + SSC (*) | 77.7 | 65.4 | 32.3 | 27.6 | 781.9 | 236.1 | 62.0 | 84.2 | 70.3 | 57.7 |
| AltText + DSC (*) | 78.2 | 66.9 | 31.9 | 30.7 | 686.4 | 216.4 | 63.8 | 84.3 | 72.0 | 58.2 |
| AltText + DSC+ (*) | 79.2 | 66.9 | 31.1 | 29.6 | 740.2 | 229.3 | 65.5 | 85.1 | 68.1 | 58.2 |
| AFC | 78.0 | 65.6 | 32.3 | 29.4 | 713.2 | 214.3 | 66.0 | 84.4 | 70.0 | 58.0 |
| SSC + DSC + DSC+ | 79.4 | 66.5 | 30.2 | 30.3 | 689.3 | 198.9 | 65.5 | 83.8 | 67.5 | 57.5 |
| AFC + SSC + DSC + DSC+ | 77.3 | 62.6 | 31.6 | 27.7 | 661.5 | 198.2 | 64.1 | 83.5 | 64.2 | 55.9 |

Effect of synthetic captions for pre-training benchmarks.
We generate DSC and SSC captions for VeCap-300M (Lai et al., ), as used in MM1 (McKinzie et al., ), and replace the captions during MM1 pre-training. The results, summarized in Table 4, show that our synthetic captions yield improved performance in image-text benchmarks across 0-shot to 8-shot settings. For example, SSC achieves a +1.3% performance gain in 0-shot evaluation compared to the original MM1. Unlike CLIP experiments, DSC outperforms SSC, with the combination of DSC and original AltText delivering the best results in this context. We also conduct an ablation study of data mixing ratios on MM1 pre-training to explore the optimal balance between synthetic captions and AltText, as summarized in Table 4. The results indicate that a 66/33 mixing ratio yields the best overall performance across all evaluation settings. Specifically, this ratio achieves the highest scores for TextCore (54.24), 0-shot (37.02), 4-shot (60.26), and 8-shot (64.71) performance. While increasing the proportion of synthetic captions generally improves performance, there is a significant drop in 0-shot performance when using only synthetic captions (100/0 ratio).

Effect of synthetic captions for SFT benchmarks.
Besides pre-training benchmarks, we also conduct SFT and then evaluate the model to analyze the profound effect of image-caption data in MLLMs. We use the same SFT recipe to have a fair comparison. As shown in Table 5, DSC+ and the concatenation of AltText with DSC+ deliver the best performance.
This strongly suggests the importance of detailed captions, despite them containing potentially the highest number of hallucinations among all the caption types we test. On the other hand, concatenating AltText with synthetic captions does not yield significant improvement in the SFT benchmarks, contrasting with the gains observed in pre-training benchmarks. We hypothesize that the primary role of image-caption data during the pre-training phase of MLLMs is to enhance image-text alignment. Consequently, more detailed captions, such as DSC+, deliver superior results after the SFT stage.

DSC+ alone can outperform diverse synthetic captions. Unlike CLIP, where mixing diverse synthetic captions leads to superior results, for MM1, detailed captions (DSC+) alone yield the best performance after the SFT stage.
As shown in Table 5, DSC+ achieves a 58.7% score, outperforming LLaVA captions (56.5%) by 2.2%. This suggests that providing richer and more specific information in captions helps multimodal LLMs like MM1 generalize better after the SFT stage. The combination of SSD, DSC, DSC+, and AFC does not lead to better results, suggesting that multimodal LLMs may benefit more from detailed captions during pre-training. These findings suggest that while combining synthetic captions proves beneficial in some contexts (e.g., CLIP), for multimodal LLMs like MM1, a single, detailed caption offers more effective guidance during pre-training.

### 4.3 Image-Caption Data for Diffusion Model

Inspired by DALLE-3 (Betker et al., ), detailed and short captions can improve the prompt following ability. In this work, our DSC not only covers the main objects within the scene, but also their relationships, attributes, and the broader context in which they are situated. We hypothesize that this level of detail allows the model to generate images that are not only visually accurate but also semantically aligned with the textual input. We implement Stable Diffusion 3 (Esser et al., ) and use this diffusion model as our studying example on text-to-image generation tasks. The backbone is based on the DiT architecture (Peebles & Xie, ) that focuses exclusively on class-conditional image generation and incorporates a modulation mechanism to condition the network based on both the diffusion process timestep and the class label. Different from DALLE-3 (Betker et al., ), we report results on more comprehensive benchmarks instead of only CLIP score, such as GenEval (Ghosh et al., ) and DSG (Cho et al., ).

Effect of synthetic captions.
Synthetic captions lead to significant improvements on the GenEval benchmark (Ghosh et al., ), as shown in Table 6, highlighting the advantage of enhanced prompt-following capabilities. Notably, incorporating SSC or DSC with AltText boosts the GenEval average score from 58.8 to 65.5. Additionally, synthetic captions yield over a 3.5% improvement on the DSG benchmark (Cho et al., ). However, SSC achieves better performance on the FID score (Jayasumana et al., ). Overall, the descriptive captions show better results among these benchmarks.

Ablation study on mixing ratio of synthetic captions and AltText. We examine the impact of varying the ratio between DSC and AltText in diffusion model training, evaluating performance across FID@COCO30k (Jayasumana et al., ), CLIP@COCO30k, DSG Average (Cho et al., ), GenEval Overall (Ghosh et al., ). Results are summarized in Fig. 9. The FID@COCO30k metric shows a gradual increase, suggesting that higher DSC ratios lead to improvements in generation quality. The DSG Average score exhibits improvements with a higher DSC ratio, indicating that DSC can enhance the model’s ability to handle complex tasks. However, the performance on the GenEval related metric peaks at 50% DSC, after which it begins to decline, highlighting the necessity of balancing synthetic and original captions to achieve optimal results across diverse evaluation merics. Besides DSC, we further conduct experiments using SSC as well. Results are summarized in Table 6. Overall, the use of SSC alsone also achieves competitive performance, but the use of DSC and AltText together appears to be a better captioning strategy.

**Table 6: The effect of synthetic captions on diffusion models.**
|  | GenEval (Ghosh et al., ) | FID<br>COCO30k | DSG<br>Average |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Single Obj | Two Obj | Counting | Colors | Position | Attribution | Average |  |  |  |
| AltText | 99.1 | 80.4 | 58.3 | 78.5 | 17.4 | 40.2 | 62.3 | 13.6 | 72.4 |
| SSC | 98.5 | 80.1 | 68.9 | 80.4 | 18.6 | 50.2 | 66.1 | 13.1 | 73.1 |
| SSC + AltText | 98.2 | 84.4 | 65.1 | 77.3 | 18.6 | 50.2 | 65.5 | 13.1 | 74.2 |
| DSC | 93.9 | 60.2 | 67.7 | 79.8 | 13.8 | 39.2 | 59.1 | 14.8 | 73.8 |
| DSC + AltText | 99.2 | 84.2 | 68.9 | 79.0 | 21.4 | 52.0 | 67.4 | 14.0 | 74.2 |

Figure: Figure 9: The intersection of synthetic captions and AltText for diffusion models. We gradually increase the proportion of DSC mixed with AltText during training.
Refer to caption: extracted/5897046/Figure/diffusion_ratio_study_v2.png

## 5 Discussion

In this study, we examine the role and value of image-text data in multimodal foundation models, including CLIP, multimodal LLMs, and diffusion models. Our research focuses on the intersection between synthetic image-aligned captions and the original web-crawled AltText. To identify the most effective captions for each foundation model, we develop a controllable and human-aligned captioning pipeline designed to minimize hallucinations and generate various types of captions as needed. Through extensive pre-training experiments, we derive the following key insights. 1) Both AltText and synthetic captions play crucial roles—AltText contributes to more diverse information, while synthetic captions offer improved image-text alignment. 2) CLIP tends to favor short synthetic captions, whereas MLLMs benefit from more descriptive captions. We also observe that the benchmarks in the pre-training and SFT stage of MLLMs may have different preferences of captions. 3) We verify the observation from DALLE-3 on text-to-image generation with more comprehensive benchmarks and show the benefits of synthetic captions.
For future work, we aim to further refine our captioning pipeline, enhancing its ability to generate task-specific captions across a wider range of multimodal applications.

## Appendix A Experimental Details

### A.1 CLIP

We summarize the training details in Table A1. For the pre-training stage, we pre-train models on up to 512 TPUs with JAX (Bradbury et al., ).

**Table A1: Pre-training hyper-parameters and settings for the in-house CLIP.**
| Batch size | $32768$ |
| --- | --- |
| Image size | $224\times 224$ (ViT-B/16) |
| Image pre-processing | long-side resizing with padding (i.e., tf.image.resize_with_pad) |
| Text tokenizer | T5 (Raffel et al., ), lowercase |
| Text maximum length | $77$ |
| Steps | $435,000$ (i.e., $\sim 14$B examples seen) |
| Optimizer | AdamW ($\beta_{1}=0.9,\beta_{2}=0.98$) |
| Peak learning rate (LR) | $0.0005$ |
| LR schedule | cosine decays with linear warm-up (first $2$k steps) |
| Weight decay | $0.2$ |
| Dropout rate | $0.0$ |

#### A.1.1 Additional Experiments

To further explore the performance gap between DSC and SSC on CLIP, we present two additional benchmarks to enhance the representativeness of CLIP’s existing evaluations: 1) linear probing and 2) transferability between CLIP pre-trained with different captions and LLaVA-style MLLMs. Linear probing provides a direct measure of the quality and generalization capability of the representations learned by CLIP. Strong performance from a linear classifier on specific tasks indicates that the pre-trained model has effectively captured relevant and discriminative features, underscoring the robustness of its embeddings. Additionally, we assess CLIP’s representation quality using LLaVA (Liu et al., ) as a case study, where the vision encoder remains frozen during both pre-training and SFT stages. This makes LLaVA an ideal benchmark for evaluating the transferability and integrity of CLIP’s learned representations.

We summarize the results on linear probing in Fig. 2: even though DSC and SSC shows lower zero-shot performance, they achieve comparable results to AltText after linear probing, indicating similar pre-trained representations. Furthermore, combining synthetic captions with AltText yields the best overall performance. Then we use these pre-trained vision encoders and insert them into LLaVA and complete the default pre-training and SFT stages in LLaVA (Liu et al., ). All of our pre-trained CLIPs use ViT-B/16 as the backbone. We use Vicuna-1.3 as the LLM for LLaVA training and report recent benchmarks in Table A2: POPE (Li et al., ), TextVQA (Singh et al., ), GQA (Hudson & Manning, ), SciQA (Lu et al., ), LLaVA-Bench (Liu et al., ), MME (Fu et al., ), and MM-Vet (Yu et al., ). In this case, the combination of SSC and AltText achieves the highest overall performance, leading in 5 out of 9 columns. This highlights the critical role of both synthetic captions and original AltTexts in CLIP’s pre-training: synthetic captions enhance image-text alignment, while AltTexts introduce valuable data diversity.

**Table A2: LLaVA as the benchmark for evaluating CLIP vision encoder pre-trained on different captions.**
| Pre-trained Caption | POPE | TextVQA | GQA | SciQA | LLaVA-Bench | MME | MM-Vet |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Avg. | IMG | Accuracy | COCO | Perception | Cognition |  |  |  |  |
| AltText | 84.4 | 50.2 | 59.1 | 64.3 | 64.8 | 76.4 | 1332.6 | 271.1 | 25.0 |
| DSC | 83.4 | 47.6 | 59.1 | 63.2 | 64.8 | 75.6 | 1307.8 | 312.9 | 23.9 |
| DSC + AltText | 84.6 | 49.8 | 59.9 | 63.6 | 65.3 | 77.2 | 1405.3 | 273.2 | 25.5 |
| SSC | 84.5 | 48.2 | 59.6 | 59.0 | 62.7 | 76.7 | 1343.6 | 248.6 | 22.4 |
| SSC + AltText | 84.7 | 49.4 | 60.1 | 65.3 | 65.8 | 77.9 | 1370.7 | 270.7 | 25.1 |

**Table A3: Compatibility between rewritten-based and filtering-based methods. We use DFN-2B (Fang et al., ) as the example and train CLIP with ViT/B-16 on different captions.**
| Table A3: Compatibility between rewritten-based and filtering-based methods. We use DFN-2B (Fang et al., ) as the example and train CLIP with ViT/B-16 on different captions.<br>Pre-train Caption<br>COCO (R@1)<br>Flickr30k (R@1)<br>ImageNet<br>ImageNetV2<br>I-T<br>T-I<br>I-T<br>T-I<br>AltText<br>61.22<br>49.27<br>86.20<br>68.94<br>76.12<br>68.49<br>DSC<br>51.47<br>30.99<br>77.28<br>55.68<br>27.97<br>23.67<br>SSC<br>59.06<br>31.91<br>87.41<br>62.49<br>53.96<br>46.39<br>DSC+SSC<br>60.68<br>38.46<br>89.60<br>68.68<br>56.10<br>49.15<br>AltText+DSC+SSC<br>70.56<br>50.74<br>92.40<br>76.92<br>72.45<br>64.98 | Figure A1: Effect of LiT (Zhai et al., ) on different captions after pre-training. |
| --- | --- |

Compatibility of rewritten-based methods and filtering-based methods. Besides rewritten-based datasets like web-crawled VeCap-300M (Lai et al., ), which leverage recaptioning techniques to improve image-text alignment, it is essential to evaluate the compatibility between rewriting-based and filtering-based methods that remove mismatched image-text pairs. We consider DFN-2B (Fang et al., ) as a representative example, where a pre-trained CLIP model filters the dataset to retain pairs that align well with CLIP’s capabilities. A key research direction is to explore whether synthetic captions can effectively replace the original CLIP-selected AltText. To investigate this, we apply our captioning pipeline to DFN-2B (Fang et al., ) to generate synthetic captions and pre-train CLIP models on these captions. The results are summarized in Table A3. In this CLIP-filtered dataset, neither rewritten DSC nor SSC, nor their combination, outperforms the original AltText. However, when combining all our synthetic captions with the original AltText (using uniform sampling during training), we observe significant improvements in retrieval tasks, such as +9.34% on COCO I-T and +7.98% on Flickr T-I tasks, demonstrating the enhanced image-text alignment provided by our synthetic captions. Nevertheless, incorporating synthetic captions results in a performance drop of around 4% on ImageNet, highlighting the crucial role of the diverse information contained in the original AltText for CLIP’s learning.

Effect of Locked-image text Tuning (LiT) (Zhai et al., ) after pre-training with different captions. LiT (Zhai et al., ) trains a text model to derive meaningful representations from a pre-trained image model by freezing the image encoder and fine-tuning only the text encoder. We investigate the impact of LiT after pre-training CLIP on different captions, with the results summarized in Fig. A1. During the LiT stage, we continue to train the text encoder using AltText. Our findings reveal that LiT with AltText consistently benefits both SSC and TSC across all evaluation sets, highlighting once again the critical role of AltText in CLIP training.

### A.2 Multimodal LLM

We summarize the training details in Table A4. For the pre-training stage, we pre-train models on up to 512 TPUs with JAX (Bradbury et al., ).

**Table A4: Pre-training hyper-parameters and settings for the Multimodal LLM experiments. We use the same configuration as the 1.2B model in MM1 (McKinzie et al., ).**
| General |  |
| --- | --- |
| Batch size | $512$ |
| Image encoder | $336\times 336$ ViT-L/14 |
| Visual-language connector | C-Abstractor with $144$ image tokens |
| Language model | $1.2$B transformer decoder-only language model |
| Steps | $50000$ |

For the SFT experiments, we follow the same datasets and configuration as in MM1 (McKinzie et al., ).

TextCore. For the pre-training benchmarks, TextCore is an average number of 8 benchmarks: ARC (Clark et al., ), PIQA (Bisk et al., ), LAMBADA (Paperno et al., ), WinoGrande (Sakaguchi et al., ),
HellaSWAG (Zellers et al., ), SciQ (Welbl et al., ), TriviaQA (Joshi et al., ), and WebQS (Berant et al., ).

SFT dataset.
For the SFT benchmarks, we summarize the details in Table A5: we mainly use MME (Fu et al., ), SEED (Li et al., ), POPE (Li et al., ), LLaVA-Bench (Wild) (Liu et al., ), MM-Vet (Yu et al., ), TextVQA (Singh et al., ), MMMU (Yue et al., ), MathVista (Lu et al., ), ScienceQA (Lu et al., ).

**Table A5: Details of benchmarks and their metrics used in MM1.**
| Benchmark | Metric |
| --- | --- |
| MME (Fu et al., ) | Normalized Accuracy |
| SEED (Li et al., ) | Seed-IMG |
| POPE (Li et al., ) | Average of random, popular and adversarial |
| LLaVA-Bench (Wild) (Liu et al., ) | GPT-assisted score |
| MM-Vet (Yu et al., ) | GPT-assisted score |
| TextVQA (Singh et al., ) | VQA Open Flamingo Accuracy |
| MMMU (Yue et al., ) | Accuracy |
| MathVista (Lu et al., ) | GPT-assisted score |
| ScienceQA (Lu et al., ) | Accuracy-IMG |

### A.3 Diffusion Model

We summarize the training details of our self-implemented diffusion model based on Stable Diffusion 3 (Esser et al., ) in Table A6.

**Table A6: Pre-training hyper-parameters for our diffusion model based on Stable Diffusion 3.**
| Batch size | $4096$ |
| --- | --- |
| Image size | $256\times 256$ |
| Step | $500,000$ |
| Text condition | in-house CLIP’s G/14 text encoder with T5 tokenizer  (Raffel et al., ) |
| Text maximum token length | $77$ |
| Optimizer | Adafactor ($\beta_{1}=0.9,\beta_{2}=0.999$) |
| Learning rate (LR) | $0.0001$ (constant with linear warm-up for $1$k steps) |
| Ema decay | $0.9999$ |
| Classifier free guidance | $7.5$ |

## Appendix B A Deeper Analysis of Generated Captions

Fig. 5 is an overview of our two-stage fine-tuning process: we first convert a MLLM into an image captioner, then we further fine-tune it to convert it into a human-aligned captioner.
Our smaller image captioning model (3B) efficiently generates large volumes of synthetic data for our experiments. Using this model, we re-captioned a dataset of 7 billion images across multiple iterations. Furthermore, our larger model, with 7 billion parameters, is designed to produce more detailed captions, surpassing the level of detail offered by our long caption format.

AFC fine-tuning dataset. To generate AltText Fusion Captions (AFC), we also prepare a fine-tuning dataset in this format. Specifically, given AltText and a DSC caption generated by our captioner, we ask LLM to fuse AltText information to the DSC. By this way, we construct a 20K training dataset for our captioner.

Less hallucinations in our DSC.
The Caption Hallucination Assessment with Image Relevance (CHAIR) metric (Rohrbach et al., ) is a custom-designed evaluation tool developed to identify and measure the extent of object hallucination in image captioning tasks. The metric determines the proportion of generated words that accurately correspond to objects present in the image, as verified by ground truth sentences and object segmentations. It includes two scores: one that measures the fraction of hallucinated object instances (referred to as $\text{CHAIR}_{i}$), and the other that calculates the fraction of sentences containing at least one hallucinated object (referred to as $\text{CHAIR}_{s}$):

$$ $\displaystyle\text{CHAIR}_{i}$ $\displaystyle=\frac{|\{\text{hallucinated objects}\}|}{|\{\text{all objects mentioned}\}|},$ $\displaystyle\text{CHAIR}_{s}$ $\displaystyle=\frac{|\{\text{sentences with hallucinated object}\}|}{|\{\text{ all sentences}\}|}.$ $$

**Table A7: Hallucination detection across different MLLMs and our captioner.**
| Model | $\text{CHAIR}_{i}(\downarrow)$ | $\text{CHAIR}_{s}(\downarrow)$ |
| --- | --- | --- |
| InstructBLIP (Dai et al., ) | 14.5 | 30.0 |
| MiniGPT-4 (Zhu et al., ) | 8.2 | 24.2 |
| Shikra (Chen et al., ) | 7.0 | 22.0 |
| LLaVA-1.5 (Liu et al., ) | 6.2 | 20.6 |
| Our | 5.9 | 19.6 |

As shown in Table A7, our model achieves a CHAIR${i}$ score of 5.9 and a CHAIR${s}$ score of 19.6, outperforming leading models such as LLaVA-1.5 (Liu et al., ), Shikra (Chen et al., ), and MiniGPT-4 (Zhu et al., ). The lower CHAIR scores indicate that our captioner produces fewer hallucinated objects per instance and fewer sentences containing hallucinated objects. This improvement shows the effectiveness of our two-stage fine-tuning process, which strategically reduces objects hallucination by leveraging human-aligned data and strict prompt constraints. Consequently, our model offers more reliable and accurate objects recognition capabilities, making it a robust tool for generating high-quality captions across various applications.

ANA: Average Number of Assertions, a metric for evaluating richness of captions. Besides hallucination, the richness of captions are also an important index to control the generated captions. We propose ANA to quantify the richness of captions. Inspired by GenEval (Ghosh et al., ) for evaluating text-to-image generation models, we reverse the process of text-to-image to image-to-text. As shown in Fig. A2, we prompt an LLM to generate different assertions of a caption. After that, we can also leverage a VQA model to check if these details are aligned with the visual contents.

CapScore: A metric for evaluating hallucinations in synthetic captions. Although the CHAIR metric is widely used, we find it insufficient for detecting hallucinations in object attributes, especially in highly descriptive captions. To overcome this limitation, we propose a new metric to evaluate both the hallucination and richness of captions. Inspired by GenEval (Ghosh et al., ) for evaluating text-to-image generation models, we reverse the process of text-to-image to image-to-text and propose CapScore.
CapScore measures the correctness of synthetic captions by evaluating the alignment between generated textual assertions and the actual content of the image. As shown in Fig. A2, CapScore has two key steps: 1) use an LLM to extract structured assertions from the captions; 2) use a MLLM to serve as a VQA (Visual Question Answering) model to verify the truthfulness of these assertions. Specifically, each assertion represents a distinct factual claim made within the caption. Then the VQA model determines whether the image supports each claim by answering questions based on the assertions. CapScore is then defined as the percentage of assertions validated as correct by the VQA model. A higher CapScore indicates fewer hallucinations and greater factual accuracy in the generated captions.

Figure: Figure A2: An overview of CapScore to evaluate the quality of captions: we use LLM to generate assertions based on the caption and then use a VQA model to check these assertions.
Refer to caption: extracted/5897046/Figure/capscore_v2.png

As shown in Table A8, there is a notable trade-off between the richness of captions and their accuracy. As captions become longer, the Average Number of Assertions (ANA) increases, reflecting the growing richness and complexity of the generated captions. However, CapScore drops with longer captions, which suggests that while the captions provide more content, they are more prone to hallucinations—where the captioning model introduces incorrect or irrelevant information not present in the image. For instance, while DSC+ produces the highest ANA, it also demonstrates the lowest CapScore, highlighting this balance. By contrast, SSC maintains a higher CapScore with fewer assertions, demonstrating better alignment with the image content but at the cost of less detailed descriptions.

This behavior highlights the importance of balancing richness and accuracy in multimodal tasks. Models aiming for high-precision applications (e.g., zero-shot classification with CLIP) may benefit from shorter captions (e.g., SSC), whereas scenarios requiring more detailed scene descriptions (e.g., mutlimodal LLMs) may prioritize longer captions like DSC+, accepting some decrease in factual accuracy.

**Table A8: CapScore and Average Number of Assertions (ANA) to evaluate the richness and accuracy of different captions.**
| Caption | $\textbf{CapScore}(\uparrow)$ | $\textbf{ANA}(\uparrow)$ |
| --- | --- | --- |
| LLaVA-1.5 (Liu et al., ) | 88.76 | 7.30 |
| SSC | 91.56 | 2.49 |
| DSC | 87.30 | 8.13 |
| DSC+ | 75.74 | 12.20 |

More detailed but noisy captions are helpful for MLLM pre-training. We examine the impact of hallucinations in image-text data used for MLLM pre-training, focusing on the type of hallucinations detected in our captions. For our primary comparison, we select the best-performing model, LLaVA-1.5 (Liu et al., ), as shown in Table A7. We first utilize LLaVA-1.5 to generate captions on the VeCap-300M dataset (Lai et al., ). Next, we apply our captioner to generate DSC+ for the same set of images. Our captions are more detailed but contain more hallucinations on object. We use MM1’s pre-training under a small scale setting as a case study to examine whether image-caption data with fewer hallucinations or with more details can offer advantages for MLLM pre-training. We evaluate the two models after applying fine-tuning using the same data recipe.

As shown in Table A9, the MLLM pre-trained with more detailed captions performs better, even though these data contain more hallucinations. When examining task-specific results, we observe that our DSC+ pre-trained model outperforms the LLaVA captions model on 7 out of 9 benchmarks. Specifically, DSC+ improves performance on VQAv2, MMMU, MathV, and SEED, among others, indicating that the added detail from DSC+ benefits these tasks. However, despite the slight degradation in MMEP results (727.2 vs. 773.4), the overall advantage of detailed captions with minor hallucinations demonstrates that a balance between information richness and accuracy can positively impact MLLM’s performance. The larger gains in multimodal understanding tasks, such as MMEC and LLaVAW, suggest that hallucination-tolerant MLLM pre-training may help with complex vision-language reasoning.

**Table A9: SFT evaluation of models pre-trained with captions generated by LLaVA-1.5 and our DSC+. We use the same SFT recipe and evaluation benchmarks as in MM1 (McKinzie et al., ).**
| Pre-Trained Data | VQAv2 | VQA${}^{\text{T}}$ | MMMU | MathV | MME${}^{\text{P}}$ | MME${}^{\text{C}}$ | SEED | POPE | LLaVA${}^{\text{W}}$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLaVA Captions | 78.0 | 65.4 | 30.0 | 27.6 | 773.4 | 200.4 | 63.5 | 83.7 | 63.6 |
| DSC+ | 79.0 | 65.4 | 32.6 | 29.4 | 727.2 | 224.3 | 66.8 | 85.1 | 71.5 |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: The human-annotated Stage-1-1M and Stage-2-HA datasets are valuable, and could be easily hosted on HuggingFace. Will these datasets be made available for future research?
soundness: 3
strengths: - The paper is well-motivated and grounded in a thorough analysis of various captioned datasets used for training multimodal models. This analysis offers valuable insights that effectively inform the proposed solution.
- The paper makes an important conclusion that AltText is valuable in training MLLMs, providing guidelines for future research.
- The experimental studies are comprehensive, encompassing three different multimodal models: CLIP-style models, ViT-LLM models, and diffusion models.
summary: This work studies the roles of (1) short description caption (2) dense caption, and (3) AltText in training multi-modal models.
weaknesses: - While the performance of models trained on different data ratios is presented, there is no guidance on how to determine the optimal ratio.
- There is no discussion regarding the release of the trained captioning models or the datasets, which could limit the impact of this work.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. What is the reason of using synthetic captions as a LLM re-write on top of the human captions? I notice the results in Table A7-9, and only LLM-generated captions are not far away from the approach taken in the paper. In fact, I am confident if better MLLMs such as Llava-1.6 and CogVLM are used, this gap in performance would be closer.
2.  The captioner is MM1 and the MLLM used is MM1. Is there an impact if another captioner was used?
3. zero-shot results of DSC can be significantly improved with linear probing -- what sort of linear probing is performed? If linear probing helps and is independent of the type of captions used, then do the zero-shot findings still hold?
4. Some information on the demographics of the human captioners leveraged would be helpful.
5. The reason why DSC is restricted to 78 tokens is because CLIP has the same token limit. If CLIP was not used, would both DSC and DSC+ be needed?
6. Are there any findings if Stage-1-1M and Stage-2-HA are reversed?
7. The Average Number of Assertions (ANA) pipeline is very similar to https://arxiv.org/pdf/2311.01477; are there any major differences?
soundness: 3
strengths: 1. The paper studies an important problem : understanding the impact of synthetic captions (a rapidly emerging paradigm) in conjunction with the baseline AltText captions.
2. For the models studied, such as CLIP, MM1 and SD3, the authors perform comprehensive evaluations on a wide-range of tasks. For CLIP, the authors study performance on retrieval and classification, for MM1, the authors study VQA and hallucination benchmarks and for diffusion the authors study object-centric and image fidelity metrics.
summary: The paper studies the impact of using synthetic captions in foundational vision-language models such as contrastive models, multimodal LLMs and diffusion models. The paper presents hierarchies of synthetic captions, SSC, DSC and DSC+ based on the density and length of captions and studies them across 3 broad categories of VL models. It presents results on a combination of baseline AltText and synthetic captions to understand ratios and setups that affect model performance.
weaknesses: 1. My biggest concern is that the findings are not generalizable enough. 
(i) For example, findings on CLIP will not be applicable to SigLIP [1], MM1 results will not translate to Qwen-VL [2], and findings on SD3 cannot be applied on the PixArt [3] family of models. Since the paper seeks to present insights into the impact of synthetic captions, the authors should perform, for each family of models, experiments on other models which vary widely in their architecture, loss functions etc. 
(ii) The synthetic captions adopted in this work is a function on top of human generated captions. Will all of the results still be valid if a different sampling of the human annotators is performed?

2. The findings are not necessary novel; for example, the fact that a mix of 50% synthetic+AltText captions for Diffusion models has been well reported in RECAP [4] and SPRIGHT [5]

[1] https://arxiv.org/abs/2303.15343
[2] https://arxiv.org/abs/2308.12966
[3] https://arxiv.org/abs/2310.00426
[4] https://arxiv.org/abs/2310.16656
[5] https://arxiv.org/abs/2404.01197

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: Similarly to your AFC, have you tried rewriting or fusing alt texts to form short synthetic captions?
soundness: 3
strengths: Overall the methodology appears sound and comprehensive. The array of tests provides insight into how to best combine caption data sources for multimodal training. I found particularly interesting the conclusions about the differing types of information provided by each – e.g. CLIP benefiting from the diversity of short alt texts while MLLMs may derive further benefit from longer, descriptive captions.
summary: This paper investigates the interplay between alt text and synthetic captions as training data for training multimodal foundation models. It finds that overall, a mixture between alt text, short, and long synthetic captions is advantageous, with different preferences between models and tasks.
weaknesses: The synthetic captions are derived from a custom model (Section 3) and thus it is not clear to what extent the results would generalize to synthetic data produced by other models or those used in other works.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: It seems like the following citations is missing? The paper explores the effect of synthetic versus AltText captions when training CLIP.

[1] Nguyen et al. Improving multimodal datasets with image captioning. 2023
soundness: 3
strengths: The paper addresses an interesting and important question by evaluating how captions with varying levels of visual detail impact the downstream performance of multimodal models. In addition, it approaches this evaluation more holistically than prior research by including a range of multimodal model types in the comparison.

Furthermore, the use of combined synthetic captions and original AltText represents an interesting direction that, to my knowledge, has not been extensively explored in previous studies.
summary: This paper investigates the impact of synthetic captions on training CLIP, MLLM and a Diffusion Model. Specifically, it compares the effects of training with short versus dense captions and examines the outcomes of combining synthetic captions with original AltTexts. Additionally, the authors introduce two custom human-annotated datasets aimed at reducing hallucinations in the captioner—one tailored for short captions and the other for dense captions.
weaknesses: While the paper addresses a relevant topic, several of the experiments appear similar to those in prior research, which limits the novelty of the findings. For example, as the authors note, Betker et al. [1] conducted a similar experiment evaluating the effects of long versus short synthetic captions on the downstream performance of diffusion models, reaching comparable conclusions. Similarly, Chen et al. [2] presented an experiment analogous to the MLLM analysis here, examining the impact of long versus short synthetic captions on downstream tasks. Although the models differ, the ideas and conclusions appear similar.


Additionally, while the authors justify training a custom captioner on a new dataset by claiming existing multimodal LLMs are prone to hallucinations, no experiments directly compare the effects of training with captions from, for example, LLaVA [3] or TinyLLaVA [4] (or newer and stronger models) versus the proposed captioner. Including such a comparison would enhance the reliability of the results. Moreover, the decision to create custom datasets is unclear, as similar datasets for short [3] and long captions [4] already exist. Further clarification on how the proposed captioning dataset differs from these existing resources would strengthen the paper.


[1] Betker et al. Improving Image Generation with Better Captions. 2023

[2] Chen et al. ShareGPT4V: Improving Large Multi-modal Models with Better Captions. 2024

[3] Chen et al. Microsoft coco captions: Data collection and evaluation server. 2014

[4] Onoe et al. DOCCI: Descriptions of Connected and Contrasting Images. 2024

Meta-review: The paper explores the contributions of synthetic and alt-text captions for pretraining MLLMs. There are some concerns about novelty (prior work by Betker) and generalizability (with respect to using different captioning models), with the main contribution lying in the benchmarking with these underexplored types of data (e..g synthetic data). These concerns are addressed sufficiently and all reviewers lean weakly (borderline) positive.

Decision: Accept (Poster)