Video Instruction Tuning With Synthetic Data
Authors: name: Yuanhan Zhang; name: Jinming Wu; name: Wei Li; name: Bo Li; name: Zejun Ma; name: Ziwei Liu; name: Chunyuan Li
Published: 2024-10-03 17:36:49

Abstract: The development of video large multimodal models (LMMs) has been hindered by
the difficulty of curating large amounts of high-quality raw data from the web.
To address this, we propose an alternative approach by creating a high-quality
synthetic dataset specifically for video instruction-following, namely
LLaVA-Video-178K. This dataset includes key tasks such as detailed captioning,
open-ended question-answering (QA), and multiple-choice QA. By training on this
dataset, in combination with existing visual instruction tuning data, we
introduce LLaVA-Video, a new video LMM. Our experiments demonstrate that
LLaVA-Video achieves strong performance across various video benchmarks,
highlighting the effectiveness of our dataset. We plan to release the dataset,
its generation pipeline, and the model checkpoints.