M^3IT: A Large-Scale Dataset towards Multi-Modal Multilingual
  Instruction Tuning
Authors: name: Lei Li; name: Yuwei Yin; name: Shicheng Li; name: Liang Chen; name: Peiyi Wang; name: Shuhuai Ren; name: Mukai Li; name: Yazheng Yang; name: Jingjing Xu; name: Xu Sun; name: Lingpeng Kong; name: Qi Liu
Published: 2023-06-07 12:35:37

Abstract: Instruction tuning has significantly advanced large language models (LLMs)
such as ChatGPT, enabling them to align with human instructions across diverse
tasks. However, progress in open vision-language models (VLMs) has been limited
due to the scarcity of high-quality instruction datasets. To tackle this
challenge and promote research in the vision-language field, we introduce the
Multi-Modal, Multilingual Instruction Tuning (M^3IT) dataset, designed to
optimize VLM alignment with human instructions. Our M^3IT dataset comprises
40 carefully curated datasets, including 2.4 million instances and 400 manually
written task instructions, reformatted into a vision-to-text structure. Key
tasks are translated into 80 languages with an advanced translation system,
ensuring broader accessibility. M^3IT surpasses previous datasets regarding
task coverage, instruction number and instance scale. Moreover, we develop
Ying-VLM, a VLM model trained on our M^3IT dataset, showcasing its potential
to answer complex questions requiring world knowledge, generalize to unseen
video tasks, and comprehend unseen instructions in Chinese. To encourage
further research, we have open-sourced both the dataset and trained models.