CapsFusion: Rethinking Image-Text Data at Scale
Authors: name: Qiying Yu, name: Quan Sun, name: Xiaosong Zhang, name: Yufeng Cui, name: Fan Zhang, name: Xinlong Wang, name: Jingjing Liu

Large multimodal models demonstrate remarkable generalist ability to perform
diverse multimodal tasks in a zero-shot manner. Large-scale web-based
image-text pairs contribute fundamentally to this success, but suffer from
excessive noise. Recent studies use alternative captions synthesized by
captioning models and have achieved notable benchmark performance. However, our
experiments reveal significant Scalability Deficiency and World Knowledge Loss
issues in models trained with synthetic captions, which have been largely
obscured by their initial benchmark success. Upon closer examination, we
identify the root cause as the overly-simplified language structure and lack of
knowledge details in existing synthetic captions. To provide higher-quality and
more scalable multimodal pretraining data, we propose CapsFusion, an advanced
framework that leverages large language models to consolidate and refine
information from both web-based image-text pairs and synthetic captions.
Extensive experiments show that CapsFusion captions exhibit remarkable
all-round superiority over existing captions in terms of model performance
(e.g., 18.8 and 18.3 improvements in CIDEr score on COCO and NoCaps), sample
efficiency (requiring 11-16 times less computation than baselines), world
knowledge depth, and scalability. These effectiveness, efficiency and
scalability advantages position CapsFusion as a promising candidate for future
scaling of LMM training.