Train a Unified Multimodal Data Quality Classifier with Synthetic Data
Authors: name: Weizhi Wang, name: Rongmei Lin, name: Shiyang Li, name: Colin Lockard, name: Ritesh Sarkhel, name: Sanket Lokegaonkar, name: Jingbo Shang, name: Xifeng Yan, name: Nasser Zalmout, name: Xian Li

The Multimodal Large Language Models (MLLMs) are continually pre-trained on a
mixture of image-text caption data and interleaved document data, while the
high-quality data filtering towards image-text interleaved document data is
under-explored. We propose to train an efficient MLLM as a Unified Mulitmodal
Data Quality Classifier to Filter both high-quality image-text caption and
interleaved data (UniFilter). To address the challenge of collecting diverse
labeled multimodal data, we introduce a semi-synthetic approach that leverages
readily available raw images and generates corresponding text across four
quality levels. This method enables efficient creation of sample-score pairs
for both caption and interleaved document data to train UniFilter. We apply
UniFilter to curate high-quality caption data from DataComp caption dataset and
interleaved data from the OBELICS image-text interleaved dataset. MLLMs
pre-trained on the filtered data demonstrate significantly enhanced
capabilities compared to those trained on baseline-filtered data, achieving
stronger zero-shot reasoning and in-context learning capabilities. After visual
supervised fine-tuning, these UniFilter-induced MLLMs achieve stronger
performance on various benchmarks, highlighting the downstream benefits of
high-quality multimodal pre-training. We release the synthetic training data
used for training UniFilter, the UniFilter model checkpoints, and the
high-quality interleaved document subset OBELICS-HQ, curated by UniFilter, to
the community for reproduction and further development.