SVIT: Scaling up Visual Instruction Tuning
Authors: name: Bo Zhao; name: Boya Wu; name: Tiejun Huang
Published: 2023-07-09 03:25:14

Abstract: Thanks to the emerging of foundation models, the large language and vision
models are integrated to acquire the multimodal ability of visual captioning,
dialogue, question answering, etc. Although existing multimodal models present
impressive performance of visual understanding and reasoning, their limits are
still largely under-explored due to the scarcity of high-quality instruction
tuning data. To push the limits of multimodal capability, we Sale up Visual
Instruction Tuning (SVIT) by constructing a dataset of 3.2 million visual
instruction tuning data including 1.6M conversation question-answer (QA) pairs
and 1.6M complex reasoning QA pairs and 106K detailed image descriptions.
Besides the volume, the proposed dataset is also featured by the high quality
and rich diversity, which is generated by prompting GPT-4 with the abundant
manual annotations of images. We empirically verify that training multimodal
models on SVIT can significantly improve the multimodal performance in terms of
visual perception, reasoning and planing.