Video Instruction Tuning With Synthetic Data
Authors: name: Yuanhan Zhang, name: Jinming Wu, name: Wei Li, name: Bo Li, name: Zejun Ma, name: Ziwei Liu, name: Chunyuan Li

The development of video large multimodal models (LMMs) has been hindered by
the difficulty of curating large amounts of high-quality raw data from the web.
To address this, we propose an alternative approach by creating a high-quality
synthetic dataset specifically for video instruction-following, namely
LLaVA-Video-178K. This dataset includes key tasks such as detailed captioning,
open-ended question-answering (QA), and multiple-choice QA. By training on this
dataset, in combination with existing visual instruction tuning data, we
introduce LLaVA-Video, a new video LMM. Our experiments demonstrate that
LLaVA-Video achieves strong performance across various video benchmarks,
highlighting the effectiveness of our dataset. We plan to release the dataset,
its generation pipeline, and the model checkpoints.