Revisit Large-Scale Image-Caption Data in Pre-training Multimodal
  Foundation Models
Authors: name: Zhengfeng Lai; name: Vasileios Saveris; name: Chen Chen; name: Hong-You Chen; name: Haotian Zhang; name: Bowen Zhang; name: Juan Lao Tebar; name: Wenze Hu; name: Zhe Gan; name: Peter Grasch; name: Meng Cao; name: Yinfei Yang
Published: 2024-10-03 17:54:52

Abstract: Recent advancements in multimodal models highlight the value of rewritten
captions for improving performance, yet key challenges remain. For example,
while synthetic captions often provide superior quality and image-text
alignment, it is not clear whether they can fully replace AltTexts: the role of
synthetic captions and their interaction with original web-crawled AltTexts in
pre-training is still not well understood. Moreover, different multimodal
foundation models may have unique preferences for specific caption formats, but
efforts to identify the optimal captions for each model remain limited. In this
work, we propose a novel, controllable, and scalable captioning pipeline
designed to generate diverse caption formats tailored to various multimodal
models. By examining Short Synthetic Captions (SSC) towards Dense Synthetic
Captions (DSC+) as case studies, we systematically explore their effects and
interactions with AltTexts across models such as CLIP, multimodal LLMs, and
diffusion models. Our findings reveal that a hybrid approach that keeps both
synthetic captions and AltTexts can outperform the use of synthetic captions
alone, improving both alignment and performance, with each model demonstrating
preferences for particular caption formats. This comprehensive analysis
provides valuable insights into optimizing captioning strategies, thereby
advancing the pre-training of multimodal foundation models.