SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large
  Language Models
Authors: name: Peng Gao, name: Renrui Zhang, name: Chris Liu, name: Longtian Qiu, name: Siyuan Huang, name: Weifeng Lin, name: Shitian Zhao, name: Shijie Geng, name: Ziyi Lin, name: Peng Jin, name: Kaipeng Zhang, name: Wenqi Shao, name: Chao Xu, name: Conghui He, name: Junjun He, name: Hao Shao, name: Pan Lu, name: Hongsheng Li, name: Yu Qiao

We propose SPHINX-X, an extensive Multimodality Large Language Model (MLLM)
series developed upon SPHINX. To improve the architecture and training
efficiency, we modify the SPHINX framework by removing redundant visual
encoders, bypassing fully-padded sub-images with skip tokens, and simplifying
multi-stage training into a one-stage all-in-one paradigm. To fully unleash the
potential of MLLMs, we assemble a comprehensive multi-domain and multimodal
dataset covering publicly available resources in language, vision, and
vision-language tasks. We further enrich this collection with our curated OCR
intensive and Set-of-Mark datasets, extending the diversity and generality. By
training over different base LLMs including TinyLlama1.1B, InternLM2-7B,
LLaMA2-13B, and Mixtral8x7B, we obtain a spectrum of MLLMs that vary in
parameter size and multilingual capabilities. Comprehensive benchmarking
reveals a strong correlation between the multi-modal performance with the data
and parameter scales. Code and models are released at
https://github.com/Alpha-VLLM/LLaMA2-Accessory