MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at
  Scale
Authors: name: Jarvis Guo; name: Tuney Zheng; name: Yuelin Bai; name: Bo Li; name: Yubo Wang; name: King Zhu; name: Yizhi Li; name: Graham Neubig; name: Wenhu Chen; name: Xiang Yue
Published: 2024-12-06 18:14:24

Abstract: Open-source multimodal large language models (MLLMs) have shown significant
potential in a broad range of multimodal tasks. However, their reasoning
capabilities remain constrained by existing instruction-tuning datasets, which
were predominately repurposed from academic datasets such as VQA, AI2D, and
ChartQA. These datasets target simplistic tasks, and only provide phrase-level
answers without any intermediate rationales. To address these challenges, we
introduce a scalable and cost-effective method to construct a large-scale
multimodal instruction-tuning dataset with rich intermediate rationales
designed to elicit CoT reasoning. Using only open models, we create a dataset
containing 12M instruction-response pairs to cover diverse, reasoning-intensive
tasks with detailed and faithful rationales. Experiments demonstrate that
training MLLMs on this dataset significantly improves reasoning capabilities,
achieving state-of-the-art performance on benchmarks such as MathVerse (+8.1%),
MMMU-Pro (+7%), and MuirBench (+13.3%). Additionally, the model demonstrates
notable improvements of up to 4% on non-reasoning-based benchmarks. Ablation
studies further highlight the importance of key components, such as rewriting
and self-filtering, in the dataset construction process.