SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large
  Language Models
Authors: name: Peng Gao; name: Renrui Zhang; name: Chris Liu; name: Longtian Qiu; name: Siyuan Huang; name: Weifeng Lin; name: Shitian Zhao; name: Shijie Geng; name: Ziyi Lin; name: Peng Jin; name: Kaipeng Zhang; name: Wenqi Shao; name: Chao Xu; name: Conghui He; name: Junjun He; name: Hao Shao; name: Pan Lu; name: Hongsheng Li; name: Yu Qiao
Published: 2024-02-08 18:59:48

Abstract: We propose SPHINX-X, an extensive Multimodality Large Language Model (MLLM)
series developed upon SPHINX. To improve the architecture and training
efficiency, we modify the SPHINX framework by removing redundant visual
encoders, bypassing fully-padded sub-images with skip tokens, and simplifying
multi-stage training into a one-stage all-in-one paradigm. To fully unleash the
potential of MLLMs, we assemble a comprehensive multi-domain and multimodal
dataset covering publicly available resources in language, vision, and
vision-language tasks. We further enrich this collection with our curated OCR
intensive and Set-of-Mark datasets, extending the diversity and generality. By
training over different base LLMs including TinyLlama1.1B, InternLM2-7B,
LLaMA2-13B, and Mixtral8x7B, we obtain a spectrum of MLLMs that vary in
parameter size and multilingual capabilities. Comprehensive benchmarking
reveals a strong correlation between the multi-modal performance with the data
and parameter scales. Code and models are released at
https://github.com/Alpha-VLLM/LLaMA2-Accessory