Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models
Authors: Zhengfeng Lai
Vasileios Saveris
Chen Chen
Hong-You Chen
Haotian Zhang
Bowen Zhang
Wenze Hu
Juan Lao Tebar
Zhe Gan
Peter Grasch
Meng Cao
Yinfei Yang
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal
pre-training
image-text data
Primary area: foundation or frontier models, including LLMs
TL;DR: We systematically explore the role and value of image-caption data in pre-training multimodal foundation models and investigate the optimal format for each model.

Abstract: Recent advancements in multimodal models highlight the value of rewritten captions for improving performance, yet key challenges remain. For example, while synthetic captions often provide superior quality and image-text alignment, it is not clear whether they can fully replace AltTexts: the role of synthetic captions and their interaction with original web-crawled AltTexts in pre-training is still not well understood. Moreover, different multimodal foundation models may have unique preferences for specific caption formats, but efforts to identify the optimal captions for each model remain limited. In this work, we propose a novel, controllable, and scalable captioning pipeline designed to generate diverse caption formats tailored to various multimodal models. By examining short synthetic captions (SSC) and descriptive synthetic captions (DSC) as case studies, we systematically explore their effects and interactions with AltTexts across models such as CLIP, multimodal LLMs, and diffusion models. Our findings reveal that a hybrid approach that keeps both synthetic captions and AltTexts can outperform the use of synthetic captions alone, improving both alignment and performance, with each model demonstrating preferences for particular caption formats. This comprehensive analysis provides valuable insights into optimizing captioning strategies, thereby advancing the pre-training of multimodal foundation models.

Published as a conference paper at ICLR 2025
REVISIT LARGE-SCALE IMAGE-CAPTION DATA IN PRE- TRAINING MULTIMODAL FOUNDATION MODELS
Zhengfeng Lai; Vasileios Saveris; Chen Chen, Hong-You Chen, Haotian Zhang Bowen Zhang, Juan Lao Tebar, Wenze Hu, Zhe Gan, Peter Grasch Meng Cao, Yinfei Yang"
Apple
(jeff lail_aii,, saveris,, pgrasch, mengcao, wingeaa,,nnneeeaeeaoo..
ABSTRACT
Recent advancements in multimodal models highlight the value of rewritten cap- tions for improving performance, yet key challenges remain. For example, while synthetic captions often provide superior quality and image-text alignment, it is not clear whether they fully replace AltTexts: the role of synthetic captions and their interaction with original web-crawled AliTexts in pre-training is still not well understood. Moreover, different multimodal foundation models may have unique preferences for specific caption formats, but efforts to identify the optimal captions for each model remain limited. In this work, we propose a novel, con- trollable, and scalable captioning pipeline designed to generate diverse caption formats tailored various multimodal models. By examining Short Synthetic Captions (SSC) towards Dense Synthetic Captions (DSC+) as case studies, we systematically explore their effects and interactions with AltTexts across models such as CLIP, multimodal LLMs, and diffusion models. Our findings reveal that a hybrid approach that keeps both synthetic captions and AltTexts can outperform the use of synthetic captions alone, improving both alignment and performance. with each model demonstrating preferences for particular caption formats. This compre- hensive analysis provides valuable insights into optimizing captioning strategies, thereby advancing the pre-training of multimodal foundation models.
INTRODUCTION
Large-scale image-text datasets have been crucial in advancing multimodal foundation models. For instance, CLIP (Radford et al. (02T)) is pre-trained on 400 million image-text pairs collected from the Web. However, web-crawled data, particularly AltText, often suffer from insufficient visual details and noisy content, as illustrated in Fig. [ Recent studies highlight the benefits of synthetic captions, which provide better image-text alignment and improved data quality. Research on LaCLIP (Fan) et al., 2024) and ShareGPT4V (Chen et al.) 2024a) demonstrates that synthetic captions can improve performance of CLIP and multimodal large language models (MILMss, respectively. This raises key question: if higher-quality synthetic captions can be generated, could they fully replace web-crawled AltText? Should we consider disregarding AltText altogether?
To investigate this question, we first adopt the approach of VeCLIP (Lai al. 2024 and train CLIP using synthetic captions generated by LLaVA (Liu et al. 2023b). Similar to the results discussed in Li et al. 2024b), training CLIP fully synthetic captions of higher quality degrades CLIP's performance significantly: as shown in Fig. 2 when compared to using only AltText, the use of LLaVA captions results in substantial drop zero-shot ImageNet classification tasks. However, after combining original noisy AltText and LLaVA captions, we achieve the best results in both classification and retrieval tasks. This observation raises a critical question: what constitutes the optimal image-text data for multimodal foundation models? Despite its importance, research on the interplay between synthetic captions and AltText remains limited. Our findings suggest that while rewriting techniques can enhance image-text alignment, they may reduce data diversity due to
Equal contribution. Corresponding author.

---

Published as a conference paper at ICLR 2025
AltText
Lafayette Dr.
Synthetic Caption (SSC)
CLIP
Zero-shot Classlflaation Retrieval
+
brick house with a nod door and a chimsey.
Descriptive Synthetic Caption (DSC) image
doer chimnuy. The house has shingled roof is surrounded by trees. The frout yard is well maintained, with variety of plants and flowers including purple flower, adding inuch color
brick
Large Scale Image-Text Data
MLLM
Image Understanding
AltText
Misconssppione About Aloshul.
Synthetic Caption (SSC)
glass of beer is sitting table next to moother glass. Descriptive Synthetic Caption (DSC) image features
a glass placed on a uble. The beer glass is positioned in the center the scene, with snother glass lneated to the left of it. The tible appoars dining tobl,, and tbll, adding cozy ambiance to the setting.
up
Diffusion
Text noxt-to-lmage Generation
plased
The role of image-text data in multimodal foundation models: a key component in training CLIP and Diffusion Model, and essential for multimodal LLM (MLLM) pre-training alongside text and interleaved image-text data. We propose a controllable captioning pipeline to synthesize different types captions and explore optimal image-text data recipes for training these foundation models.
Figure I:
dependence limited set of LLMs or MLLMs for caption generation. Specifically, since CLIP foundational vision model that benefits from learning diverse concepts, relying on synthetic captions can potentially hinder CLIP's training due to lack of diversity in vocabulary and mentioned concepts (Fan et al. 2024).
In addition to the role of AltText, another open question concerns the optimal formats for synthetic captions. For instance, advanced multimodal models like LLaVA-NeXT (Li al. 2024a) indicate that recaptioned datasets are advantageous during stages focused high-quality knowledge acquisition. DALL-E (Betker et al.) 2023) demonstrates that using 95% synthetic captions can yield superior results, partieularly when the captions are highly descriptive. Similarly, MM1 (McKinzie al. 2024)
shows that even small fraction (7%) of high-quality caption data can significantly boost few-shot performance. Given these insights, our work focuses on two key unresolved questions: 1) What the role and value of synthetic captions, and how do they interact with the original AlrText? 2) What rypes of synthetic captions are most effective for different foundation models? To address the first question, we revisit why prior works (Betker et al. 2023 Lai et al. 2024, McKinzie et al.) 2024) continue using noisy web-crawled AltText, even when rewritten captions are available during training. Intuitively, since CLIP is a straightforward model pre-trained on image-text pairs, highly aligned captions should advantageous, However, relying solely on synthetic captions may actually degrade CLIP's performance. perf'mmaee shown in Fig. 2 a). For the second question, investigate the effects of Short Synthetic Captions (SSC) and Descriptive Synthetic Captions (DSC) on CLIP. As depicted in Fig. ()) surprisingly, more descriptive captions yield inferior results compared to shorter captions for CLIP training, despite their greater detail.
To explore these insights further to address the questions, we introduce novel, controllable, and scalable captioning pipeline that enables the generation of diverse caption formats at scale, tailored the specific needs of different multimodal foundation models. Our pipeline is designed build large-scale image-text data for the pre-training stage in a scalable way. With this pipeline, we use SSC and DSC as two main examples on how customize the captioning format. Our pipeline can serve as a cost-effective alternative to GPT-4V for generating high-quality captions. To solve the second question, we conduct a comprehensive study on the effectiveness of different types of synthetic captions across range of foundational models and downstream tasks. Our approach involves a systematic evaluation of various captioning strategies, including SSC, DSC, and mixed training methods that combine original AlrText with synthetic data. We seek to determine the optimal captioning techniques for specific models, such as CLIP, diffusion models, and multimodal LLMs, and to assess their impact on both model performance and data diversity. Furthermore, investigate the interaction between synthetic captions and original AltText, analyzing whether a hybrid approach balance the need for diverse data with the benefits of enhanced image-text alignment.
2

---

Published as a conference paper at ICLR 2025
90
AltTbxt
LLavA Capsions
DSC
60-
AltText LLNVA
80-
SSC DSC
Performence (%)
Performence (%)
70
70
60
60
30
30
20
COCO
Imageliet imageNetV2 Zera-shot RetrievaliCiasstiiaaan Taiks
0oCO Trl
Flickr 1-T
Flickr Tyl
20
COCO
0oco Trl
Flickrit
Flickr Txl Imageliet ImageltetVz
Zero-shat Retrieva/Chasssliaaton Tasks
Figure 2: Zero-shot retrieval and classification performance of CLIP models. (a) The effect of synthetic captions (LLaVA recaptioned) and AltText: solely using LLaVA captions can improve retrieval tasks but significantly deteriorate the zero-shot classification performance. (b) The effect of different formats of synthetic captions on CLIP: Short Synthetic Captions (SSC) show superior results Descriptive Synthetic Captions (DSC) and the combination of them achieves the best results.
Overall, our contributions are summarized as follows.
We explore the MILM as the image describer and present a controllable and human-aligned captioning pipeline to convert MI.LM into an image captioner.
We synthesize several formats of captions including Short Synthetic Captions (SSC) towards Dense Synthetic Captions (DSC+), then conduct extensive pre-training experiments to systematically study the role of synthetic captions and their intersection with original AltText across three multimodal foundation models.
We verify the image-capting training recipe that 1) AltText provide data variety and synthetic captions provide better image-text alignment, 2) different foundation models have their own preferred formats, which highlights the necessity and importance of the controllable captioning pipeline building multimodal foundation models.
2 RELATED WORK
Multimodal Foundation Models. CLIP (Radford et al. 1 is one of the pioneering multimodal foundation models connecting image and text. By training on 400 million image-text pairs, CLIP shows strong zero-shot capabilities. It lays the groundwork for the development of more advanced multimodal foundation models, such as multimodal large language models (MLLMs) (Liu et al.
2023b; Wang et al. 2023 Chen al. and diffusion models (Rombach et al.
2024b Tong al. 2024 for vision- understanding
2022 Podell et al.) 2023 for text-to-image generation. These
advanced models often utilize CLIP's vision lower their vision encoder.
Improving Image-Text Data. Web-crawled image-text data often suffer from issues like image-text
misalignment and poor-quality textual descriptions (Lai al. 2024 Li et al. 2024b). There are two
common ways for improving image-text data: 1) data filtering baannng--beeddd methods remove low-quality
data such as misaligned image-text pairs by numan-assiste systems (Yu al. [022a|] [Sun et al. (202))
or pre- mre-trinne models (Li et al. [2022b] [Schuhmann et al. [202]] Gadre et al. 2024 et al. 2023): 2) data recaptioning-bastt to recaption caption on the image. For example, ShareGPT4V (Chen al.] 2024a) uses GPT-4V to write highly descriptive captions for their images. LaCLIP (Fan et al. 2024) leverages several LLMs to rewrite captions with different writing styles for data diversity. Recap-DataCompg et al. 2024b) uses a LLaMA-3 based model to scale the captions. Different from the aforementioned works, mainly focus generating different types of captions and exploring 1) the format of ideal captions needed for each multimodal foundation model and 2) a systematic analysis of the intersection between AltTexts and synthetic captions.
methods usually leverage LLM rewrite the original caption MLLM
(Li
3

---

Published as a conference paper at ICLR 2025
Dense Synthetic Caption (DSC+):
Short Synthetic Caption (SSC):
A medium close-up front view of pronghorn antelope standing on ground floor made of tall green grass. The antelope is facing forward, it has a light brown body, a white underbelly, and black horns that curve up and stick out of its head. The antelope is surrounded by a blurry view of an open desert environment that consists of green plants, small bushes, tall grass. Towards the top right, a blurry view of a large rock hill can be seen.
A pronghom antelope standing field of tall grass.
AltText Fusion Caption (AFC):
A photograph of pronghorn standing in a field of grass. The pronghom is looking straight the camera. background is hazy sky and distant hill. pronghorn on Seedskadee National Wildlife Refuge Wyoming, USA, managed by USFWS Mountain Prairie.
Descriptive Synthetic Caption (DSC):
photograph of pronghorn antelope standing field of tall grass. The antelope is facing the camera and its horns background consists of hazy sky and field tall grass.
prominently featured. The
AltText: Pronghorn Seedskadee National Wildliie Refuge USFWS Mountain Prairie.
Figure 3: An example of controllable captions of diverse formats generated by our captioner: we can generate from brief to dense description and fuse AltText into the caption (AFC).
ShareGPT4V-7B
LLaVA-v1.5-11 Q: Describe the image with three sentances.
Q:: Describe the Image with three sentences.
A: The image features red brick house with black lush
A: The image features large brick house with a black roof, situsted on hill The yard filled with various plants and flowers. In adsition the house, there garage located on the right side of the property. The scena also includes bird, possiblly pigeon, perched on the roof of the house, acding touch of nature the soone.
a chimnay and red door. adding The house hes charm. The yard filld with various plants, including potted Tree. The scene also incldess oar perked front of the house, and fire hydrent nearby.
roof, surrounded
Figure 4: Directly using MILMs as image captioners may result in hallucinations and gener- ate captions that do not align with specific instructions: both LLaVA (Liu et al.] (023b) and ShareGPT4V (Chen et al. 2024a) generates over three sentences and obvious hallucination.
CUSTOMIZED RE-CAPTIONING FOR MULTIMODAL FOUNDATION MODELS
Image-text data are fundamental the success of multimodal foundation models, serving as bridge between visual and textual representations. For example, CLIP (Radford et al.) (021] is pre-trained on 400M web-crawled image-text pairs, enabling it to learn rich, transferable representations that can be applied to various downstream tasks. The importance of precise and detailed captioning is further highlighted in LLaVA-Next (Li al. 2024a), where re-captioned detailed descriptions
utilized in training Stage- 1.5, enhancing the model's ability to understand and generate nuanced content. Similarly, DALL-E (Betker al.) 2023) shows that the prompt-following capabilities of text-to-image models can be significantly improved by training on highly-descriptive generated image captions. This shows the critical role of captions in shaping model's capacity align visual and textual information, ultimately improving performance across wide range of multimodal tasks. However, the optimal captioning strategy for different foundational models remains under-explored. To address this gap, introduce novel, controllable. and scalable captioning pipeline designed to generate diverse caption formats at scale, supported by evaluation metrics that ensure high CLIP scores and minimal hallucination. We summarize the capability of our captioning model by generating the following formats captions as shown in Fig.
Short Synthetic Caption (SSC): concise sentence that describes the primary subject of the image.
Descriptive Synthetic Caption (DSC): a description limited to 78 tokens, emphasizing the central subject and key visual elements.
Dense Synthetic Caption (DSC+): a more comprehensive description detailing the main subject along with the background, setting, and any significant objects or actions.
4

---

Published as a conference paper at ICLR 2025
Short Synthetic Caption (SSC)
Descriptive Synthetic Caption (DSC)
20
Dense Synthetic Caption (DSC+)
AltText Fusion Caption (AFC)
100
122
250
200
225
250
275
200
150
Number of Takens
Number of Tokens
Figure Distribution of token lengths of our generated captions four formats: we caption COCO-2017 images and visualize their distributions.
AltText Fusion Caption (AFC): caption similar DSC, but integrated with AltText where appropriate. This type of caption removes unnecessary details often found in AltText, offering cleaner, more cohesive description than a simple concatenation of AltText and synthetic caption.
3.1
MLLM AS AN IMAGE DESCRIBER
VeCLIP (Lai al., 2024) employs LLaVA (Liu al. 2023b) for image captioning, while Shav (Chen et al. 2024a) utilizes GPT-4V for this task. Compared to traditional image captioners like BLIP (Li et al. 2022a). MLLMs offer several advantages and are natually good image describers. MLLMs can generate longer and more detailed captions, as demonstrated by LLaVA-Next (Li et al. 2024a), where 34B model was used to produce highly descriptive captions. However, directly using MLLM may not strictly follow the instruction to generate a specific format of caption (Liu et al. 2023a): 2)
image describer may have two major limitations: 1) MLLM
these instruction fine-tuned MILMM tends have hallucination. As shown in Fig. 4 both LLaVA (Liu et al. 2023b) and ShareGPT4V (Chen et al.) 2024a) fails to describe the image using only three sentences but generate hallucinated contents (highlighted in red). Although GPT-4V shows stronger capability and many works use it for captioning, the scalability remains limited due to the cost. Therefore, in this work, focus building cost-effective captioner instead of using GPT-4V.
To alleviate the above two limitations, we first investigate the origins of hallucination in MILMs, hypothesizing two primary sources: inherent limitations of the LLM and 2) the quality of supervised fine-tuning (SFT) datasets, which often are itself synthetically derived or processed. We focus on the latter, proposing that mitigating hallucinations the dataset level is essential for converting an MILM into an effective captioner. We also address the format-followin issue by fine-tuning the MLLM on a curated captioning-speciinn dataset, transforming into purpose-buill captioning model to generate diverse captions.
3.2 TWO-STAGE HUMAN- CAPTIONING
Stage 1: transforming MILM into customized captioner. To minimize hallucinations from MLLMs, we begin by constructing clean and precise fine-tuning dataset. Instead of relying on GPT-4 generated data, curate high-quality dataset of
image-text pairs, named Stage- M- This dataset contains short, human-curai curated captions, with five captions per image. Additionally, integrate OCR detection model extract in-image text, which, alongside the
5

---

Published as a conference paper at ICLR 2025
1699 7441
23708
4143
11224
19925
10788
14847
16302
5245
22514
8635
Figure 6: The number of unique entities in different synthetic captions (randomly sample 17.5k images) compared to AltText: AltText provides more unique entities as wider knowledge.
captions, is fed into a large language model (LLM) for summarization and controlled rewriting. By employing strict prompt, prevent the LLM from introducing extraneous information, while few-shot prompts guide the model in generating various caption formats, including both concise and descriptive styles. We further enhance the dataset through post-processing, using heuristic and model-based quality checks to improve its overall quality. This comprehensive fine-tuning on the Stage-I-IM dataset transforms the MILM, specifically the 3B version of MMI (McKinzie al. 2024.. into a customized captioner that aligns closely with the intended output characteristics.
Stage 2: Human-aligned further mun-tuning. While the Stage-I-IM dataset effectively establishes a foundation, it lacks the depth required for more descriptive captioning tasks. In Stage 2, we address this by incorporating descriptive muman-annotater data to enhance caption diversity and quality. We curate new dataset, named Stage-2-HA, specifically designed for detailed captioning. This dataset meticulously annotated by human experts to capture nuanced visual elements and complex scene descriptions. Post-annotation, we leverage an LLM to reformat these captions into multiple stylistic variations, including both concise and richly descriptive formats. By imposing strict constraints during the LLM processing, ensure alignment with human generated content, avoiding the pitfalls of hallucination. The captioner is then fine-tuned on the Stage-2-H 2-H dataset, resulting in highly refined and human-aligna captioning model capable of generating captions tailored to specific use cases. This dual-stage fine-tuning process not only enhances the model's adaptability but also ensures balance between brevity and descriptiveness, making it versatile tool in controllable captioning. The overview figure can be found in Appendix.
human-generated
3.3 CAPTION ANALYSIS
Richness assessment: token length and Average Number of Assertions. Fig. illustrates the distribution of the number of tokens of various caption types generated in this study. Specifically, SSC mainly ranges from 10 to 15 tokens, while DSC spans from 40 to 60 tokens, both fitting within the text encoder's 's capacity in CLIP. In contrast, most DSC+ captions exceed 100 tokens. Besides that, we propose Average Number of Assertions (ANA) to quantify the richness of captions. We prompt an LLM to generate different assertions of a caption to analyze our different formats of captions in terms of the richness. More details of this approach is in Appendix. Note that the ANA for SSC is 2.49, DSC as 8.13 and DSC+ as 12.20, showing more visual contents.
Diversity assessment: number of unique entities in captions. We hypothesize that the original, albeit noisy, AltText may carry a broader range diverse information and knowledge, offering potential advantages for CLIP's pre-training. To assess this diversity, quantify the number of unique entities present in the captions. As shown in Fig. [6, AltText contains a higher number of unique entities, which could be beneficial in providing wider knowledge base. Compared to SSC, more descriptive captions also contain more entities.
DATA FOR MULTIMODAL FOUNDATION MODELS
We mainly discuss three foundation models: CLIP, multimodal LLM, and diffusion models. For both CLIP and diffusion models, since the text encoder is limited to 77 tokens (Zhang et al. 2024). we focus primarily on SSC and DSC. For the multimodal LLM, we explore more detailed versions, including DSC+ and AFC. We summarize key findings below:

---

Published as a conference paper at ICLR 2025
Table 1: Effect of different synthetic captions on CLIP with VIT- Table 2: Evaluation with linear probing ImageNet for CLIP.
B/16 the backbonc.
traitrain Caption
0OCO (R@1)
Flickr30k (R@1)
ImageNet
ImageNNttV2
Pre-train Caption
Zem-slot
Linear Probing
Gain
T-1
AlfTest DSC SSC
65.70
78.34
+12.64
AirText
58.55
27.30
75.72
+48.42
DSC
29.00
80.90
54.75
27.30
21:01
96.63
79.94
+13.31
SSC
$7.00
35.15
84.67
63.35
49.10
43.31
AFC
38.95
75.53
+36.55
AFC
54.82
34.84
84,00
62.18
36.98
35.11
DSC Alffes
79.96
+15.79
All Symthetic AliTest
DSC Altieat
AFC AirFest
SSC AlrText
66.84
66.67
48.11
46.08
91.81
90.25
76.54
73.94
77.72
66.63
66.15
58.74
59.57
Synthetic Alffest
AFC AlfTest
SSC Altfext
SSC Alffed
16.62
79.54
50.01
78.70
+13.31
+12.23
+14.85
+34.64
The tradeoff between the richness of captions and their accuracy needs to be balanced based on the multimodal tasks.
Both AltText and synthetic captions are important for CLIP's training, with shorter captions yielding better performance. Linear probing is an additional effective way to evaluate the representations.
Pre-training and SFT benchmarks can behave differently in multimodal LLMs. On the SFT benchmark, MM1 shows a preference for DSC+ alone.
the diffusion model, DSC emerges as the most effective captioning strategy.
4.1
DATA FOR CLIP
We use VeCap-300M (Lai et al. 2024), web-crawled dataset with raw AltText our main pre- training dataset for CLIP. Besides Alteext, we generate several synthetic captions for the study. Then, use ViT-B/16 as the vision encoder. The training details can be found in Appendix.
Effect of synthetic captions. We first study the effect of Short Synthetic Captions (SSC) and Descriptive Synthetic Captions (DSC). We summarize the results in Table ! Interestingly, while SSC enhances retrieval performance, it leads to a substantial drop in zero-shot ImageNet accuracy. More- over, despite demonstrating that synthetic captions have superior image-text alignment and reduced noise, the more descriptive captions (DSC) perform worse across all benchmarks. Based on these results, the descriptive version appears suboptimal for CLIP training, leading to inferior performance. For example, despite DSC capturing more visual concepts within the captions, its zero-shot perfor- mance shows a significant degradation of 21.8% compared to SSC. We hypothesize this performance drop may be partially due to distribution mismatch as the prompts in COCO//llicr//magee short (e-g.. "a photo of ()"). To further explore the performance gap between DSC and SSC on CLIP, we also use linear probing, which provides direct measure of the quality and generalization capability of the representations learned by CLIP. Strong performance from a linear classifier on specific tasks indicates that the pre-trained model has effectively captured relevant and discriminative features, underscoring the robustness of its embeddings. We summarize the results on linear probing in Fig. [ even though DSC and SSC shows lower zero-shot performance, they achieve comparable results to AltText after linear probing, indicating similar pre-trained representations. This indicates that the relatively poor zero-shot results of DSC can be significantly improved with linear probing, implying that DSC's representations are richer than initially presumed.
are
Intersection of synthetic captions and AltText. Table | shows training CLIP solely on large-scale synthetic captions may get inferior results compared to the original AltText, even though synthetic captions have better image-text alignment. We hypothesize that synthetic captions rewritten from a MLLM may hurt the diversity and knowledge coverage of the original AltText. Therefore, we blend our synthetic captions and AltText, such as DSC AltText and SSC + AltText: there is a significant boost in retrieval performance-es g.g. over 10% improvement on COCO. Additionally, SSC + AltText also leads to gains ImageNet accuracy. Utilizing a mixture all synthetic caption formats yields the best performance in retrieval tasks, visualized in Fig. 6
Optimal mixture ratio between synthetic captions and AltText. Considering the wider knowledge of AltText (Fig. (6) and the better alignment of synthetic captions, we explore the optimal mixture ratio. Specficially, we use SSC and AltText from VeCap-300m 330M (Lai et al., 2024) as an example, with results shown in Fig 7 A ratio of 0 corresponds using only SSC, while 100 corresponds to using only AltText. We observe that CLIP achieves optimal performance across both retrieval and classification tasks when the ratio is tuned to around 40-50% A lower proportion of AltText leads to
7

---

Published as a conference paper at ICLR 2025
COCO F-T Retrieval
COCO T-I Retrieval
Flickr I-T Retrieval
67.5
Reformanc
65.0
Perfomance
62.5
60.0
66
57.5
100
Pre-training
Pre-training
Pre-training
Flickr T-1 Retrieval
ImageNet Classification
ImageNetV2 Classification
60
Perforance
Rerormance
Perfrmance
70
50
05
50
10
20
30
40
50
60
70
50 90
100
10
20
30
40
50
60
70
50
90
100
10
20
30
40
50
60
70
80 90 100
Ratio of SSC (%) used in Pre-training
Ratio of SSC (%) used in Pre-training
Retio of SSC (%) used in Pre-training
Figure 7: The intersection of synthetic captions and AltText for CLIP: we increase the proportion of SSC mixed with AlrText during training. All experiments use ViT-B/16 as the backbone and VeCap-300M (Lai et al. 2024) as the pre-training dataset.
drop in retrieval performance, whereas lower proportion of SSC results in decreased accuracy in ImageNet zero-shot classification. This observation aligns with findings in (Li et al.] 2024b). From this, verify that AltText provides broader knowledge coverage and greater diversity, which benefits CLIP's pre-training by enabling the model to grasp a wider range of concepts. This diversity may serve foundation for generalization, allowing CLIP better represent varied contexts and domains in zero-shot classification.
Exploration of the optimal way of using AltText: AFC simple mixture. As shown in Fig. and Table we verify that a simple mixture of our synthetic captions and AltText can achieve superior results. In addition to this straightforward combination, we investigate alternative methods of incorporating AltText into the training process. One promising approach to fuse the knowledge from AltText directly into the synthetic captions. To enable this, we fine-tune our captioner with instruction-followiin capabilities, generating Al(Text Fusion Captions (AFC). While AFC shows improvement over DSC due to the enriched AltText information, it underperforms when compared to the simple mixture of DSC and AltText.
4.2
DATA FOR MULTIMODAL LLM
Table 3: The effect of synthetic captions on MMI (1.2B model) pre-training.
Table 4: The ratio ablation on MM1 pre-training
between synthetic captions and AltText.
Pre-truin Caption
TextCore
0-Shot 4-Shot
8-Shot
Mixing Ratio
TexiCore
0-Shot
4-Shot
8-Shot
AlcText DSC
53.48
34,80
59.72
33/66
53.86
35.16
59.69
63.97
53.76
37.05
59.36
63.68
50/50
53.86
36.40
60.17
64.13
DSC + AlfText
53.71
37.09
60.31
63.96
66/33
54.24
37.02
60.26
64.71
SSC
53.46
37.35
63.56
80/20
53.96
60.09
SSC + AltText
53.20
37.16
58.22
62.22
100/0
54.19
15:54
60.24
64.02
As we study the large-scale image-caption data, we use MM1 McKinzie et al.) 2024) as one example
and focus on the pre-training stage. MMI (McKinzie et al., 2024) claims that captioning data lift the zero-shot performance and synthetic captions are helpful for few-shot learning. Based on this insight, we further study the captioning data recipe between original AltText and synthetic captions. We follow the pre-training setup and the evaluation benchmark in MM1 (McKinzie et al. 2024 to report TextCore and 0/4/8-shot performance. All of the experiments are conducted on the 1.2B model. We pre-train the model with 50K steps and the batch size is 512. More details are in Appendix.
Effect of synthetic captions for pre-training benchmark. We generate DSC and SSC captions for VeCap-300M (Lai al.] (024), used in MM1 (McKinzie et al. 2024), and replace the captions during MMI's pre-training. The results, summarized in Table 3] show that our synthetic captions yield improved performance in vision-text benchmarks across 0-shot to 8-shot settings. For example,
8

---

Published as a conference paper at ICLR 2025
Table 5: SFT evaluation of models pre- pre-trained with different types of captions. use the same SFT evaluation benchmarks as in MMI (McKinzie et al. 2024). ( Concatenation of two captions.)
Pre-Trained Data
VQA+2
VQAT
MMMU
MathY
MMEP
MMEC
SEED
POPE
I.aVAW
Average
AltTest
77.1
28.3
225
81.6
LLavA Caption
78.0
65.4
30.0
27.6
773.4
200.4
63.5
83.7
63.6
56.3
SSC
78.8
65.8
33.6
28.2
760.2
216.4
63.1
B4.4
69.0
57.9
DSC
77.1
61.8
30.8
27.9
596.6
213.6
64.6
83.9
70.8
56.3
DSC+
79.0
65.4
32.6
29.4
727-2
224.3
66.8
85.1
71.5
5%.7
ARTest + (S)
77.7
65.4
32.3
27.6
781.
236.1
62.0
B4.2
70.3
57.7
AltTest DSC
78.2
66.9
31.9
30.7
686.4
216.4
63.8
84.3
72.0
58.2
AlfText DSC+ (*)
79.2
66.9
31.1
29.6
74012
229.3
65.5
85.1
68.1
58.2
AFC SSC + DSC + DSC+
78.0
65.6
32,3
29.4
713.2
214.3
66.0
84.4
70.0
58.0
79.4
66.5
30.2
30.3
689.3
198.9
65.5
83.8
67.5
57.5
A DSC DSC+
77.3
62.6
27.7
661.5
198.2
83.5
64.2
55.9
SSC achieves a +1.3% % performance gain in O-shot evaluation compared to the original MMI. Unlike CLIP's experiments, the descriptive captions (DSC) outperform SSC, with the combination of DSC and original AltText delivering the best results in this context. We also conduct a ratio ablation study on MM1 pre-training to explore the optimal balance between synthetic captions and AlrText, as summarized in Table 4 The results indicate that 66/33 mixing ratio yields the best overall performance across all evaluation settings. Specifically, this ratio achieves the highest scores for TextCore (54.24). O-shot (37.02). 4-shot (60.26), and 8-shot (64.71) tasks. While increasing the proportion of synthetic captions generally improves performance, there is a significant drop in O-shot performance when using only synthetic captions (100/0 ratio).
Effect of synthetic captions for SFT benchmark. Besides pre-training benchmark, we also conduct SFT and then evaluate the model to analyze the profound effect of image-caption data in multimodal LLMs. We fix the SFT recipe to have a fair comparison. As shown in Table 5 DSC+ and the concatenation of AltText with DSC+ deliver the best performance. This strongly suggests the importance of detailed captions, despite them containing potentially the highest number of hallucinations among all the caption types we test. On the other hand, concatenating AltText with synthetic captions does yield significant improvement in the SFT benchmark, contrasting with the gains observed in pre-training benchmarks. We suggest that the primary role of image-caption data during the pre-training phase of multimodal LLMs is enhance image-text alignment. Consequently, more detailed captions, such as DSC+, deliver superior results after the SFT stage.
DSC+ alone can outperform diverse synthetic captions. Unlike CLIP, where mixing diverse synthetic captions leads to superior results, for MM1, detailed captions (DSC+) alone yield the best performance after the SFT stage. As shown in Table DSC+ achieves 58.7% score, outperforming LLaVA captions (56.5%) by 2.2%. This suggests that providing richer and more specific information in captions helps multimodal LLMs like MM1 generalize better after the SFT stage. The combination of SSD, DSC, DSC+, and AFC does not lead to better results, suggesting that multimodal LLMs may benefit more from detailed captions during pre-training. These findings suggest that while combining synthetic captions proves beneficial in some contexts (e.g.. CLIP), for multimodal LLMs like MM1, a single, detailed caption offers more effective guidance during pre- pre-training.
4.3 MAGE-CAPTION DATA FOR DIFFUSION MODEL
Inspired by DALLE-3 (Betker et al. 2023). detailed and short captions can improve the prompt following ability. In this work, our DSC not only covers the main objects within the scene, but also their relationships, attributes, and the broader context in which they are situated. We hypothesize that this level of detail allows the model to generate images that are not only visually accurate but also semantically aligned with the textual input. We implement Stable Diffusion (Esser et al., (024) and use this diffusion model as our studying example on text-to-image generation tasks. The backbone is based on the DiT architecture (Peebles & Xie 2023) that focuses exclusively on class-conditional image generation and incorporates modulation mechanism to condition the network based on both the diffusion process timestep and the class label. Different from DALLE-3 (Betker et al.
2023), we report results more comprehensive benchmarks instead of only CLIP score, such as GenEval (Ghosh et al. 2024) and DSG (Cho et al., 2024).
Effect of synthetic captions. Synthetic captions lead to significant improvements on the GenEval benchmark (Ghosh et al., (024). as shown in Table [ highlighting the advantage of enhanced prompt-following capabilities. Notably, incorporating SSC DSC with AltText boosts the GenEval

---

Published as a conference paper at ICLR 2025
FID@COCO3OK
Pick@COcO30K
DSG Average
75.0
21.75
Reformance
Reformance
Refomance
23.70
14.0
73.5
21.65
13.5
13.0
72.5
Training
(96) Training
Treining
GenEval Counting
GenEval Position
GenEval Overall
68
Peformence
Reffrmance
Neformance
62
64
62.5
16
60.0
10 20
30
40
50
60
70
80
90 100
10
20
30
40
50
60
20
100
10 20 30
60
70 80 50 100
Rabio of DSC (%) used in Training
Ratio of DSC (%5 used Training
Retio of DSC (%) used in Treining
Figure 8: The intersection of synthetic captions and AltText for the diffusion model: we increase the proportion of DSC mixed with AltText during training.
average score from 58.8 65.5. Additionally, synthetic captions yield over 3.5% improvement on DSG (Cho et al. 2024). However, SSC achieves better performance on the FID score (Jayasumana
et al. 2024).
Ablation study on mixing ratio of synthetic captions and AltText. We examine the impact of varying the ratio between DSC and AltText in diffusion model training, evaluating performance across TIDCCOC03OK (Jayasumana et al., 2024), CLIP CLIP@COCO30K COCO30k. DSG Average (Cho et al. 2024), GenEval Overall (Ghosh et al. 2024). Results are summarized in Fig. g The FID@COCO30K metric shows gradual increase, suggesting that higher DSC ratios lead to improvements in generation quality. The DSG Average score exhibits improvements with a higher DSC ratio, indicating that DSC enhance the model's ability handle complex tasks. However, the performance the GenEval related metric peaks at 50% DSC, after which it begins to decline, highlighting the necessity of balancing synthetic and original captions to achieve optimal results across diverse evaluation merics. Besides DSC, we further conduct experiments using SSC as well. Results are summarized in Table 6 Overall, the use of SSC alsone also achieves competitive performance, but the use of DSC and AltText together appears to be better captioning strategy.
Table 6:
The effect of synthetic captions on diffusion models.
GenEval Counting
Ghosh Coiors
2024
FID COCO30k
DSG
Single Obj
Two Obj
Position
Attribution
Avcrage
Average
AlfTest
99.1
80.4
58.3
78.5
17.4
40.2
62.3
13.6
72.4
SSC
98.5
80.1
68.9
80.4
18.6
50.2
66.1
13.1
73.1
SSC + AlrText
98.2
84.4
65.1
77.3
18.6
50.2
65.5
13.1
74.2
DSC
67.7
79.8
39.2
59.1
14.8
73.8
DSC + AltText
99.2
84.2
68.9
79.0
21.4
52.0
67.4
14.0
74.2
DISCUSSION
In this study, we examine the role and value of image-text data in multimodal foundation models, including CLIP, multimodal LLMs, and diffusion models. Our research focuses on the intersection between synthetic magge-aligned captions and the original web-crawled AltTexts. To identify the most effective captions for each foundation model, we develop a controllable and human- -alignd captioning pipeline designed to minimize hallucinations and generate various types of captions as needed. Through extensive pre-training experiments, we derive the following key insights: 1) both AltTexts and synthetic captions play crucial roles-Al(Texts contribute diverse information, while synthetic captions offer improved image-text alignment; 2) CLIP tends to favor short synthetic captions, whereas multimodal LLMs benefit from more descriptive captions. We also observe that the benchmarks in the pre- mre-training and SFT stage of MLLMs may have different preferences of captions; 3) we verify the observation from DALLE-3 on text-to-image generation with more comprehensive benchmarks and show the benefits of synthetic captions.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question -answrr pairs. In Proceedings of the 2013 conference on empirical methods natural language processing, pp. 1533-1544, 2013.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/paperstiallle-3. pdf. 2(3):8, 2023.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piga: Reasoning about physical commonsense natural language. Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs. Github, 2018. URL http://github. cwtp:/ogoooooooooooooooooooooooo le/ jax
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint arXiv:2306. 15/95, 2023.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang. Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. ECCV, 2024a.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing. Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer 2024b.
Vision and Pattrnn Recognition, pp.
Jaemin Cho, Yushi Hu, Jason Baldridge, Roopal Garg, Peter Anderson, Ranjay Krishna, Mohit Bansal, Jordi Pont-Tuset, and Su Wang. Davidsonian scene graph: Improving reliability fine-grained evaluation for text-to-image generation. In ICLR, 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 0803.88433557 2018
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- pprrpose vision-language models with instruction tuning, 2023.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning. 2024.
Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites. Advances in Neural Information Processing Systems, 36, 2024.
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. In The Twelfth International Conference on Learning Represen- tutions, 2023.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang. Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets, Advances in Neural Information Processing Systems, 36, 2024.
11

---

Published as a conference paper at ICLR 2025
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36, 2024.
Drew Hudson and Christopher Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 6700-6709, 2019.
Sadcep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar. Rethinking fid: Towards a better evaluation metric for image generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 9307-9315, 2024.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv: /tdv:/70500 170.003551, 2017.
Zhengfeng Lai, Haotian Zhang, Bowen Zhang, Wentao Wu, Haoping Bai, Alcksei Timofeev, Xianzhi Du, Zhe Gan, Jiulong Shan, Chen-Nee Chuah, Yinfei Yang, and Meng Cao. Veclip: Improving clip training via visual-enriched captions, 2024. httpst/::
http/axxiv.orr. www.org.///..eeoooooe/eo../eeeee aawaaaaa/2110 2310.0000000 076.9....77.
Bo Li, Hao Zhang. Kaichen Zhang. Dong Guo, Yuanhan Zhang, Renrui Zhang, Feng Li, Ziwei Liu, and Chunyuan Li. Llava-next: What else influences visual instruc- tion tuning beyond data?, May 2024a. URL
2024-05 05-20---2--00 llava next ablations/
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal Iims with generative comprehension. arXiv preprint arXiv:2307.16125, 2023a.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vasion-language understanding and generation. In International conference on machine learning, pp. 12888-12900. PMLR, 2022a.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping Lanuuge-image pre- training for unified vision-language understanding and generation. In International conference machine learning, pp. 12888-12900. PMLR, 2022b.
Xianhang Li, Haoqin Tu, Mude Hui, Zeyu Wang. Bingchen Zhao, Junfei Xiao, Sucheng Ren, Jieru Mei, Qing Liu, Huangjie Zheng, et al. What if we recaption billions of web images with llama-3? arXiv preprini arXiv:2406.0668 08478, 2024b.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji- Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In EMNLP, 2023b.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-moda models via robust instruction tuning. In The Twelfth International Conference on Learning Representations, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023b.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521 2022.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kei-Wei Chang. Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255. 2023.
Brandon McKinzie, Zhe Gan, Jean-piil Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mml: Methods, analysis & insights from multimodal Ilm pre-training. arXiv preprint arXiv:2403.06//1 2024.
12

---

Published as a conference paper at ICLR 2025
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context. arXiv preprint arXiv:/606.060)) 2016.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952. 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In ICML, pp. 8748-888 17788 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020. 2020.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. EMNLP, 2018.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recogmition, pp. 10684-10695, 2022.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-fittred 400 million image-text pairs. arXiv preprint arXiv:2111.022
2021.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Mareus Rohrbach. Towards vqa models that can read. In CVPR, 2019a.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019b.
Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu- Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv: 14525, 2023.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, al. Cambrian-1: fully open, vision-centric exploration of multimodal 1lms. arXiv preprini arXiv:2406. 16860, 2024.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvlm: Visual expert for pretrained language models, 2023.
Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. arXiv preprint arXiv: / 10000062099999 2017.
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 13807-13816. 2024a.
Weihao Yu, Zhengyuan Yang, Linjie Li. Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In Forty-first International Conference on Machine Learning, 2024b.
13

---

Published as a conference paper at ICLR 2025
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:3311.16502 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv: 1905.078300 2019.
Xiaohua Zhai, Xiao Wang. Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 18123-18133, 2022.
Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlocking the long-text capability of clip. ECCV, 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint 1xXiv:330411050) 2023.
14

---

Published as a conference paper at ICLR 2025
Appendices
We provide additional details for datasets, experimental settings, results, and analysis the supple- mentary material.
A EXPERIMENTAL DETAILS
A.1 CLIP
We summarize the training details in Al For the pre-training stage, we pre-train models on up
to 512 TPUs with JAX (Bradbury et al. 2018).
Table A1: Pre-training hyper-parameters and settings for the in-house CLIP.
Batch size
32768
Image size
221 x 224 (ViT-B/16)
Image pre-processing
long-side resizing with padding (i.e. tf.
Text tokenizer
T5 (Raffel et al. 2020), lowercase
Text maximum length
77
Steps
435, 000 (i.c., ~ 14B examples seen)
Optimizer
AdamW (8: 0.9, 92 0.98)
Peak learning rate (I.R)
0.0005
LR schedule
cosine decays with linear warm-up (first 2k steps)
Weight decay
0.2
Dropout rate
0.0
A.1.1 ADDITIONAL EXPERIMENTS
To further explore the performance gap between DSC and SSC on CLIP, we present two additional benchmarks to enhance the representativeness of CLIPP's existing evaluations: 1) linear probing and 2) transferability between CL.IP pre-trained with different captions and VLLaV-sttyl MIL.Ms. Linear probing provides direct measure of the quality and generalization capability of the representations learned by CLIP. Strong performance from a linear classifier on specific tasks indicates that the pre-trained model has effectively captured relevant and discriminative features, underscoring the robustness of its embeddings. Additionally, we assess CLIP's representation quality using LLaVA (Liu case study, where the vision encoder remains frozen during both trainiannin and
et al. 2023b) SFT stages. This makes LLaVA an ideal benchmark for evaluating the transferability and integrity of CL.PP's learned representations.
We summarize the results on linear probing in Fig. 2 even though DSC and SSC shows lower zero-shot performance. they achieve comparable results AltText after linear probing, indicating similar pre-trained representations. Furthermore, combining synthetic captions with AltText yields the best overall performance. Then use these pre-trained vision encoders and insert them into LLaVA and complete the default pre-training and SFT stages in LLaVA (Liu et al. 2023b). All of our pre-trained CLIPs use ViT-B/IG as the backbone. We use Vicuna- 1.3 as the LLM for LLaVA training and report recent benchmarks in Table A2 POPE (Li et al. 2023b). TexiVQA (Singh et al. (Liu 2003bb. MME Fuu et al. 20044, and MM- Vet (Yu al. 2024b). In this case, the combination of SSC and AltText achieves the highest overall performance, leading in 5 out of columns. This highlights the critical role of both synthetic captions and original AltTexts in CLIP's pre-training: synthetic captions enhance image-text alignment, while AltTexts introduce valuable data diversity.
2019b), GQA (Hudson & Manning 2019), SciQA et 2022). L.LaVA-
Compatibility of rewritten-based methods and bittering-based methods. Besides rewritten- based datasets like web-crawled VeCap-300M 300M (Lai et al.) 2024), which leverage recaptioning techniques to improve image-text alignment, it is essential to evaluate the compatibility between rewriting-based based
15

---

Published as a conference paper at ICLR 2025
Table A2: LLaVA as benchmark evaluating CLIP vision encoder pre-trained on different captions.
POPE Avg.
SciQA IMG
LLaVA Bench
MME Perception
Pre-trained Caption
TexVQQ
GQA
MM-Vet
Accuracy
COCO
Cognition
AltText
84.4
50.2
59.1
64.3
64.8
76.4
1332.6
271.1
25.0
83.4
47.6
59.1
63.2
1307.8
312.9
23.9
DSC + AirText
84.6
49.8
59.9
63.6
65.3
77.2
1405.3
273.2
25,5
84.5
48.2
59.6
59.0
1343.6
248.6
22.4
SSC AirText
84.7
49.4
60.1
65.3
65.8
77.9
1370.7
270.7
25.1
Table A3: Compatibility between rewritten-basee based and filtering- based methods. We DFN-2B (Pang al. (023) as the example and train CLIP with ViT/B-16 16 on different captions.
COCO (R@1)
Flickr300 (R@
Pre-train Caption
ImageNNtt
ImageNetV2
1-T
1-1
T-1
49.27
DSC
51.47
30.99
77.28
55.68
27.97
23.67
SSC
59.06
31:91
87.41
62.49
53.96
46.39
Sets Figure A1: Effect LiT (Zhii et al.,
ARTToN-DSC4SSC
70.56
30.74
92.40
76.92
72:45
64.98
2022)
different captions after pre- training.
and iltering-based methods that remove mismatched image-text pairs. We consider DFN-2B (Fang et al. 2023) as a representative example, where pre-trained CLIP model filters the dataset to retain pairs that align well with CLIP's capabilities. key research direction is to explore whether synthetic captions can effectively replace the original CLIP-sclected Al(Text. To investigate this, we apply our captioning pipeline to DFN-2B (Fang al., 2023) to generate synthetic captions and pre-train CLIP models on these captions. The results are summarized in Table A3 In this ELIP-filtered dataset, neither rewritten DSC nor SSC, nor their combination, outperforms the original AltText. However, when combining all our synthetic captions with the original AlrText (using uniform sampling during training). we observe significant improvements in retrieval tasks, such as 49.34% on COCO I-T and +7.98% % on Flickr T-I tasks, demonstrating the enhanced image-text alignment provided by synthetic captions. Nevertheless, incorporating synthetic captions results in a performance drop of around 4% ImageNet, highlighting the crucial role of the diverse information contained in the original AltText for CLIP's learning.
Effect of Locked-image text Tuning (LiT) (Zhai al. 2022) after pre-training with different captions. LiT (Zhai et al. 2022) trains a text model to derive meaningful representations from pre-trained image model by freezing the image encoder and fine-tuning only the text encoder. We investigate the impact of LiT after pre-training CLIP on different captions, with the results summarized Fig. [A] During the LiT stage, we continue train the text encoder using AltText. Our findings reveal that LiT with AltText consistently benefits both SSC and TSC across all evaluation sets, highlighting once again the critical role of AltText CLIP training.
A.2 MULTIMODAL LLM
We summarize the training details Table A4 For the pre-training stage, we pre-train models on up to 512 TPUs with JAX (Bradbury et al. 2018).
Table A4: Pre-training parameeas and settings for the Multimodal LLM experiments. We the same configuration as the 1.2B model in MM1 (McKinzie et al. 2024..
General
Batch size
512
Image encoder
336 x 336 VIT-L/14
Visual-language connector
C-Abstractor with image tokens
Language model
1.2B transformer decoder-only language model
Steps
50000
16

---

Published as a conference paper at ICLR 2025
For the SFT experiments, we follow the same datasets and configuration as in MM1 (MeKinzie et al. 2024).
TextCore. For the pre-training benchmarks, TextCore is an average number of 8 benchmarks:
ARC (Clark et al., 2018). PIQA (Bisk et al. 2020p. LAMBADA (Paperno et al. 2016). Wino-
Grande (Sakaguchi
al. 202T). HellaSWAG (Zellers et al. 2019), SciQ (Welbl et al. (017))
TriviaQA (Johh et al. 2017 and WebQS (Berant et 2013)
SFT dataset. For the SFT benchmarks,
mainly MME (Fu et al., 2024). SEED (Li et al. 2023a), POPE (Li et al. 2023b), LLa VA-Bench (Wild) (Li
summarize the details in Table A5
al. 2023b), MM- Yu et al. 2024b) TextVQA (Singh al. 2019a), MMMU (Yue et al. 2023
Math Vista (Lu et al. 2023). Science (Lu et al. 2022).
Benchmark
Metric
MME
Fu et al.
2024
Normalized Accuracy
SEED
Li et al.
202.a
Seed-IMG
POPE
L et
2023b
Average of random, popular and adversarial
LLaVA-Bench
Wild] Liu
2023b
GPT-assisted score
MM-Vet Yu et al.
2024b
GPT-assisted score
TextVQA
Singh
12019a
VQA Open Flamingo Accuracy
MMMU
Yuc
2023
Accuracy GPT-assisted score Accuracy-IMP -MMracyy-GGG-
Math Vista
[Lu ct al.
2023
ScienceQA
Lu et al,
2022
Table A5: Details of benchmarks and their metrics used in MM1.
A.3 DIFFUSION MODEL
We summarize the training details of our self- ceffeimpleeentdd diffusion model based on Stable Diffusion
3 (Esser et al. 2024) in Table A6
Table A6: Pre-training hyper-parameters for diffusion model based Stable Diffusion 3.
Batch size
4096
Image size
256 x 256
Step
500, 000
Text condition
in house CLIP's G/14 text encoder with T5 T5 Raffeel et al. 2020
Text maximum length
77
Steps
435, 000 (i.e., 14B examples seen)
Optimizer
Adafactor (3) = 0.9, 32 = 0.999)
Learning rate (LR)
0.0001 (constant with linear warm-up for 1k steps)
Ema decay
0.9999
Classifier free guidance
7.5
B A DEEPER ANALYSIS OF GENERATED CAPTIONS
Fig. A2 is an overview our two-stage fine-tuning process: we first convert a MILMM into an image captioner, then further fine-tune convert it into ilumnn-alinnee captioner. Our smaller image captioning model (3B) efficiently generates large volumes of synthetic data for our experiments. Using this model, re-captioned dataset of billion images across multiple iterations. Furthermore, larger model, with 7 billion parameters, is designed to produce more detailed captions, surpassing the level of detail offered by our long caption format.
AFC fine-tuning dataset. To generate AltText Fusion Captions (AFC), we also prepare a fine-tuning dataset in this format. Specifically, given AltText and DSC caption generated by captioner,
17

---

Published as a conference paper at ICLR 2025
Stage
Two-Stage Human-Aligned MLLM Fine-Tune
Describe
Fire Short
Stage-1-1M
Customized Short Descriptive
Pre-trained
X3 LLM
Detected
Test &8 OCR
Captioning-driven Fine-tune
Customized Captioner
Stage
Desortative Captioning
Descriptive
Rewntte X
Human- Alggned
Human-Alignment
Fine-tune
Short Descriptive
Stage-2-HA
LLM
Human-Allined Captioner
Figure A2: Overview of the controllable and human-alignees aligned captioning pipeline. In Stage 1, con- vert a pre-trained MLLM into a customized captioner that strictly follows the captioning instructions. In Stage 2, we leverage human-aligned captions to further fine-tune the captioner.
ask LLM fuse AltText information to the DSC. By this way, construct 20K training dataset for our captioner.
Less hallucinations in DSC. The Caption Hallucination Assessment with Image Relevance (CHAIR) metric (Rohrbach et al. 2018) is a custom-designed evaluation tool developed to identify and measure the extent of object ballucination image captioning tasks. The metric determines the proportion of generated words that accurately correspond to objects present in the image, verified by ground truth sentences and object segmentations. It includes two scores: one that measures the fraction of hallucinated object instances (referred to as CHAIR;), and the other that calculates the fraction of sentences containing least one hallucinated object (referred to CHAIRg):
CHAIR; [hallucinated objects]|
CHAIR, (sentences with hallucinated object]
[all objects mentioned]]
[all sentences]
As shown in Table A7 our model achieves CHAIRi score of 5.9 and CHAIRs score of 19.6, outperforming leading models such as
Table A7: Hallucination detection across different
MLLMs and captioner.
LLaVA-1.5 (Liu et al., 2023b). Shikra (Chen ct al. 2023), and MiniGPT-4 (Zhu et al., 2023). The lower CHAIR scores indicate that cap- tioner produces fewer hallucinated objects per instance and fewer sentences containing hallu- cinated objects. This improvement shows the effectiveness of our two-stage fine-tuning pro- cess, which strategically reduces objects hallucination by leveraging human- data and strict prompt constraints. Consequently, our model offers more reliable and accurate objects recognition capabilities, making it robust tool for generating high-quality captions across various applications.
CHAIR,(J)
CHAIR,
InstructBLIP Dai al. 2025
14.5
30.0
MiniGPT-4 20g at 2025
24.2
Shikma Clan e 20r
7.0
22.0
Our
5,9
19.6
ANA: Average Number of Assertions, metric for evaluating richness of captions. Besides hallucination, the richness of captions are also an important index to control the generated captions. We propose ANA to quantify the richness of captions. Inspired by GenEval (Ghosh al. (024) for evaluating text-to-image generation models, we reverse the process of text-to-image to image-to-text As shown in Fig. A3 we prompt an LIM to generate different assertions of a caption. After that, we can also leverage a VQA model to check if these details are aligned with the visual contents.
CapScore: A metric for evaluating hallucinations in synthetic captions. Although the CHAIR metric is widely used, we find it insufficient for detecting hallucinations in object attributes, especially in highly descriptive captions. To overcome this limitation, we propose a new metric to evaluate both the hallucination and richness of captions. Inspired by GenEval (Ghosh al. 2024) for evaluating toxt-to-image generation models, we reverse the process of text-to- text-to-image to image- toage-tt-texx and propose CapScore. CapScore measures the correctness of synthetic captions by evaluating the alignment
18

---

Published as a conference paper at ICLR 2025
between generated textual assertions and the actual content of the image. As shown in Fig. A3 CapScore has two key steps: 1) use an LLM to extract structured assertions from the captions; 2) use MLLM serve VQA (Visual Question Answering) model to verify the truthfulness of these assertions. Specifically, each assertion represents distinct factual claim made within the caption. Then the VQA model determines whether the image supports each claim by answering questions based on the assertions. CapScore is then defined as the percentage of assertions validated as correct by the VQA model. higher CapScore indicates fewer hallucinations and greater factual accuracy in the generated captions.
Caption: A man in a blue pola shirt; and blue jeans is performing a
Assertions:
skateboard trick at skate park. He is
man is performing a skateboard trick at skate park.
in the air, holding the skateboard with hand. The skateboard has multi-; colored design. The background is out focus, but there skate ramp cloudy sky.
The the
LLM
The man is holding the skateboard with one hand. The man is wearing blue polo shirt.
The man wearing blue jeans. The skateboard multi-colored design.
There is skate ramp in the background.
VQA
The background is out focus.
There are clouds sky.
CapScore: 0.89 AVG Assertions: 9
Figure A3: An overview of CapScore evaluate the quality captions: we use LLM to generate assertions based on the caption and then use VQA model to check these assertions.
Short Synthetic Caption (SSC):
Dense Synthetic Caption (DSC+):
beach with a crowd of people sunbathing and swimming in the ocean.
A wide shot of beach with numerous people on it. A group of white and blue umbrellas is on the sand on the right side. White sand and clear blue water in front of the beach. group of buildings are on a hill at the back of the beach, along with hill behind it. large crane is on top of one the buildings the right. is facing the right and slightly towards the front. Mountains are in the far background, behind the buildings. The sky is bright blue cloudless.
Alt Fusion Caption (AFC):
Descriptive Synthetic Caption (DSC):
wide shot photograph of beach in Nice, France the French Riviera. The beach is crowded with people and there many umbrellas and chairs. The ocean is deep blue and the is clear. the background, there buildings and hills.
wide shot photograph beach with crowd of people sunbathing and swimming the The
beach is surrounded palm trees and buildings. In background, there mountain and clear blue
sky.
AltText: Bird's view of the French Riviera in Nics, France.
Figure A4: Another example of controllable captions of diverse formats generated by our captioner.
As shown in Table A8 there is notable trade-off between the richness of captions and their accuracy, As captions become longer, the Average Number of Assertions (ANA) increases, reflecting the growing richness and complexity of the generated captions. However, CapScore drops with longer captions, which suggests that while the captions provide more content, they are more prone to hallucinations- the captioning model introduces incorrect or irrelevant information not present in the image. For instance, while DSC+ produces the highest ANA, also demonstrates the lowest CapScore, highlighting this balance. By contrast, SSC maintains higher CapScore with fewer assertions, demonstrating better alignment with the image content but at the cost of less detailed descriptions.
19

---

Published as a conference paper at ICLR 2025
This behavior highlights the importance of balancing richness and accuracy in multimodal tasks. Models aiming for high-precision applications (e.g., zero-shot classification with CLIP) may benefit from shorter captions (e.g., SSC), whereas scenarios requiring more detailed scene descriptions (e.g.. mutlimodal LLMs) may prioritize longer captions like DSC+, accepting some decrease in factual accuracy.
Table A8: CapScore and Average Number Assertions (ANA) to evaluate the richness accuracy of different captions.
Caption
CapScore(1)
ANA(f)
LLaVA-1.5 at. 20236
88.76
7.30
SSC
91.56
2.49
DSC
87.30
8.13
DSC+
75.74
12.20
More detailed but noisy captions are helpful for MLLM pre-training. We examine the impact of hallucinations in image-text data used for MLLM pre-training, focusing on the type hallucinations detected in our captions. For our primary comparison, we select the best-performing model, LA-VA- VA- We first utilize LLaVA-1.5 to generate captions on the VeCap-300k 300p-300MMMM dataset (Lai et al. 2024 Next, we apply captioner to generate DSC+ for the same set images. Our captions are more detailed but contain more hallucinations on object. We use MM1's 's pre-training under small scale setting as data with fewer hallucinations or with more details can offer advantages for MLLM pre-training, We evaluate the two models after applying fine-tuning using the same data recipe.
1.5 (Liu et al. 2023b), as shown Table A7
study examine whether image-caption
As shown Table A9 the MLLM pre-trained with more detailed captions performs better, even though these data contain more hallucinations. When examining task-specifie results, we observe that our DSC+ pre-trained model outperforms the LLaVA captions model on 7 out of 9 benchmarks. Specifically, DSC+ improves performance on VQAv2, MMMU, Math and SEED, among oth- ers, indicating that the added detail from DSC+ benefits these tasks. However, despite the slight degradation in MMEP results (727.2 vs. 773.4), the overall advantage of detailed captions with minor hallucinations demonstrates that balance between information richness and accuracy can positively impact MLLM's performance. The larger gains in multimodal understanding tasks, such as MMEC and LLaVAW, suggest that hallucination-toler tolerrant MLLM pre-training may help with complex vision-language reasoning.
Table A9: SFT evaluation of models pre-trained with captions generated by LLaVA-1.5 and our DSC+. We use the same SFT recipe and evaluation benchmarks as in MM1 (McKinzie et al. 2024).
Pre- Trained Data
VQAv2
VOAT
MMMU
Math
MMEP
MMEC
SEED
POPE
LLaVAW
LLaVA Captions
78.0
30.0
27.6
773.4
200.4
83.7
63.6
DSC+
79.0
65.4
32.6
29.4
727.2
224_3
85.1
71.5
20


Reviews:
[{"review_id": "9qAsTjf44n", "reviewer": "Reviewer_EoMe", "summary": "This work studies the roles of (1) short description caption (2) dense caption, and (3) AltText in training multi-modal models.", "questions": "The human-annotated Stage-1-1M and Stage-2-HA datasets are valuable, and could be easily hosted on HuggingFace. Will these datasets be made available for future research?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-motivated and grounded in a thorough analysis of various captioned datasets used for training multimodal models. This analysis offers valuable insights that effectively inform the proposed solution.\n- The paper makes an important conclusion that AltText is valuable in training MLLMs, providing guidelines for future research.\n- The experimental studies are comprehensive, encompassing three different multimodal models: CLIP-style models, ViT-LLM models, and diffusion models.", "weaknesses": "- While the performance of models trained on different data ratios is presented, there is no guidance on how to determine the optimal ratio.\n- There is no discussion regarding the release of the trained captioning models or the datasets, which could limit the impact of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ueMdGgUCoQ", "reviewer": "Reviewer_aQXA", "summary": "The paper studies the impact of using synthetic captions in foundational vision-language models such as contrastive models, multimodal LLMs and diffusion models. The paper presents hierarchies of synthetic captions, SSC, DSC and DSC+ based on the density and length of captions and studies them across 3 broad categories of VL models. It presents results on a combination of baseline AltText and synthetic captions to understand ratios and setups that affect model performance.", "questions": "1. What is the reason of using synthetic captions as a LLM re-write on top of the human captions? I notice the results in Table A7-9, and only LLM-generated captions are not far away from the approach taken in the paper. In fact, I am confident if better MLLMs such as Llava-1.6 and CogVLM are used, this gap in performance would be closer.\n2.  The captioner is MM1 and the MLLM used is MM1. Is there an impact if another captioner was used?\n3. zero-shot results of DSC can be significantly improved with linear probing -- what sort of linear probing is performed? If linear probing helps and is independent of the type of captions used, then do the zero-shot findings still hold?\n4. Some information on the demographics of the human captioners leveraged would be helpful.\n5. The reason why DSC is restricted to 78 tokens is because CLIP has the same token limit. If CLIP was not used, would both DSC and DSC+ be needed?\n6. Are there any findings if Stage-1-1M and Stage-2-HA are reversed?\n7. The Average Number of Assertions (ANA) pipeline is very similar to https://arxiv.org/pdf/2311.01477; are there any major differences?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper studies an important problem : understanding the impact of synthetic captions (a rapidly emerging paradigm) in conjunction with the baseline AltText captions.\n2. For the models studied, such as CLIP, MM1 and SD3, the authors perform comprehensive evaluations on a wide-range of tasks. For CLIP, the authors study performance on retrieval and classification, for MM1, the authors study VQA and hallucination benchmarks and for diffusion the authors study object-centric and image fidelity metrics.", "weaknesses": "1. My biggest concern is that the findings are not generalizable enough. \n(i) For example, findings on CLIP will not be applicable to SigLIP [1], MM1 results will not translate to Qwen-VL [2], and findings on SD3 cannot be applied on the PixArt [3] family of models. Since the paper seeks to present insights into the impact of synthetic captions, the authors should perform, for each family of models, experiments on other models which vary widely in their architecture, loss functions etc. \n(ii) The synthetic captions adopted in this work is a function on top of human generated captions. Will all of the results still be valid if a different sampling of the human annotators is performed?\n\n2. The findings are not necessary novel; for example, the fact that a mix of 50% synthetic+AltText captions for Diffusion models has been well reported in RECAP [4] and SPRIGHT [5]\n\n[1] https://arxiv.org/abs/2303.15343\n[2] https://arxiv.org/abs/2308.12966\n[3] https://arxiv.org/abs/2310.00426\n[4] https://arxiv.org/abs/2310.16656\n[5] https://arxiv.org/abs/2404.01197", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9ETyQpPzQf", "reviewer": "Reviewer_5ZLH", "summary": "This paper investigates the interplay between alt text and synthetic captions as training data for training multimodal foundation models. It finds that overall, a mixture between alt text, short, and long synthetic captions is advantageous, with different preferences between models and tasks.", "questions": "Similarly to your AFC, have you tried rewriting or fusing alt texts to form short synthetic captions?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Overall the methodology appears sound and comprehensive. The array of tests provides insight into how to best combine caption data sources for multimodal training. I found particularly interesting the conclusions about the differing types of information provided by each – e.g. CLIP benefiting from the diversity of short alt texts while MLLMs may derive further benefit from longer, descriptive captions.", "weaknesses": "The synthetic captions are derived from a custom model (Section 3) and thus it is not clear to what extent the results would generalize to synthetic data produced by other models or those used in other works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qZAcnnlhAM", "reviewer": "Reviewer_QQnt", "summary": "This paper investigates the impact of synthetic captions on training CLIP, MLLM and a Diffusion Model. Specifically, it compares the effects of training with short versus dense captions and examines the outcomes of combining synthetic captions with original AltTexts. Additionally, the authors introduce two custom human-annotated datasets aimed at reducing hallucinations in the captioner—one tailored for short captions and the other for dense captions.", "questions": "It seems like the following citations is missing? The paper explores the effect of synthetic versus AltText captions when training CLIP.\n\n[1] Nguyen et al. Improving multimodal datasets with image captioning. 2023", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper addresses an interesting and important question by evaluating how captions with varying levels of visual detail impact the downstream performance of multimodal models. In addition, it approaches this evaluation more holistically than prior research by including a range of multimodal model types in the comparison.\n\nFurthermore, the use of combined synthetic captions and original AltText represents an interesting direction that, to my knowledge, has not been extensively explored in previous studies.", "weaknesses": "While the paper addresses a relevant topic, several of the experiments appear similar to those in prior research, which limits the novelty of the findings. For example, as the authors note, Betker et al. [1] conducted a similar experiment evaluating the effects of long versus short synthetic captions on the downstream performance of diffusion models, reaching comparable conclusions. Similarly, Chen et al. [2] presented an experiment analogous to the MLLM analysis here, examining the impact of long versus short synthetic captions on downstream tasks. Although the models differ, the ideas and conclusions appear similar.\n\n\nAdditionally, while the authors justify training a custom captioner on a new dataset by claiming existing multimodal LLMs are prone to hallucinations, no experiments directly compare the effects of training with captions from, for example, LLaVA [3] or TinyLLaVA [4] (or newer and stronger models) versus the proposed captioner. Including such a comparison would enhance the reliability of the results. Moreover, the decision to create custom datasets is unclear, as similar datasets for short [3] and long captions [4] already exist. Further clarification on how the proposed captioning dataset differs from these existing resources would strengthen the paper.\n\n\n[1] Betker et al. Improving Image Generation with Better Captions. 2023\n\n[2] Chen et al. ShareGPT4V: Improving Large Multi-modal Models with Better Captions. 2024\n\n[3] Chen et al. Microsoft coco captions: Data collection and evaluation server. 2014\n\n[4] Onoe et al. DOCCI: Descriptions of Connected and Contrasting Images. 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)