MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning
Authors: name: Haotian Zhang; name: Mingfei Gao; name: Zhe Gan; name: Philipp Dufter; name: Nina Wenzel; name: Forrest Huang; name: Dhruti Shah; name: Xianzhi Du; name: Bowen Zhang; name: Yanghao Li; name: Sam Dodge; name: Keen You; name: Zhen Yang; name: Aleksei Timofeev; name: Mingze Xu; name: Hong-You Chen; name: Jean-Philippe Fauconnier; name: Zhengfeng Lai; name: Haoxuan You; name: Zirui Wang; name: Afshin Dehghan; name: Peter Grasch; name: Yinfei Yang
Published: 2024-09-30 17:59:34

Abstract: We present MM1.5, a new family of multimodal large language models (MLLMs)
designed to enhance capabilities in text-rich image understanding, visual
referring and grounding, and multi-image reasoning. Building upon the MM1
architecture, MM1.5 adopts a data-centric approach to model training,
systematically exploring the impact of diverse data mixtures across the entire
model training lifecycle. This includes high-quality OCR data and synthetic
captions for continual pre-training, as well as an optimized visual
instruction-tuning data mixture for supervised fine-tuning. Our models range
from 1B to 30B parameters, encompassing both dense and mixture-of-experts (MoE)
variants, and demonstrate that careful data curation and training strategies
can yield strong performance even at small scales (1B and 3B). Additionally, we
introduce two specialized variants: MM1.5-Video, designed for video
understanding, and MM1.5-UI, tailored for mobile UI understanding. Through
extensive empirical studies and ablations, we provide detailed insights into
the training processes and decisions that inform our final designs, offering
valuable guidance for future research in MLLM development.