Video Instruction Tuning with Synthetic Data

## Abstract

Abstract The development of video large multimodal models (LMMs) has been hindered by the difficulty of curating large amounts of high-quality raw data from the web. To address this, we consider an alternative approach, creating a high-quality synthetic dataset specifically for video instruction-following, namely LLaVA-Video-178K . This dataset includes key tasks such as detailed captioning, open-ended question-answering (QA), and multiple-choice QA. By training on this proposed dataset, in combination with existing visual instruction tuning data, we introduce LLaVA-Video , a new video LMM. Our experiments demonstrate that LLaVA-Video achieves strong performance across various video benchmarks, highlighting the effectiveness of our dataset. We plan to release the dataset, its generation pipeline, and the model checkpoints.

## 1 Introduction

We are in an era where large-scale computing and data is crucial for multimodal learning (Li et al., ). A significant recent advancement was introduced by visual instruction tuning (Liu et al., ), which laid the foundation for building a general-purpose visual assistant. Notably, it proposed a data generation pipeline to create high-quality image-language instruction-following data. This pipeline has inspired subsequent researches (Li et al., ; ; ; Lin et al., ) aimed at generating diverse image-language instruction data across various visual domains, accelerating the development of visual instruction tuning techniques.

Compared to the construction of image-language instruction-following data, obtaining high-quality video-language instruction-following data is challenging (Zhang et al., ; Li et al., ). First, sourcing high-quality videos is difficult. We need to find videos with significant temporal changes that provide more knowledge than what image-language data can offer. However, we have found that most videos in current video-language instruction-following datasets (Chen et al., ; Zhang et al., ) are relatively static. Additionally, these videos are mostly trimmed based on scene changes, resulting in simplified plots. Such simplified video-language instruction-tuning data is inadequate for models to understand videos with complex narratives. Furthermore, current video-language instruction-following datasets often use a very sparse sampling rate for frame annotation. For instance, ShareGPT4Video (Chen et al., ) has an average sampling rate of 0.15, sometimes sampling only 2 frames from a 30-second video.
This sparse sampling rate is effective in describing overall scenes but fails to capture detailed movements or changes in the video, resulting in hallucination when detailed descriptions of the video are required.

To overcome these shortcomings, we introduce a comprehensive video instruction-tuning dataset named LLaVA-Video-178K, consisting of 178,510 videos ranging from 0 to 3 minutes. This dataset is enriched with detailed annotations, open-ended questions, and multiple-choice questions, developed through a combination of GPT-4o (OpenAI, ) and human efforts. It features four favorable properties: $(i)$ Extensive Video Source: We conduct a comprehensive survey on the video sources of exsiting video understanding datasets, and conclude 10 major video data sources, from which we start our video data collection by building a video pool.
Although there are over 40 video-language datasets, their video data are mainly sourced from 10 datasets (Zhou & Corso, ; Xue et al., ; Goyal et al., ; Caba Heilbron et al., ; Kay et al., ; Sigurdsson et al., ; Wang et al., ; Shang et al., ; Grauman et al., ; Zhu et al., ), covering a wide range of video domains, such as activities, cooking, TV shows, and egocentric views. $(ii)$ Dynamic Untrimmed Video Selection: From these sources, we use several filtering logic to select the most dynamic videos from the video data pool. Notably, we select original, untrimmed videos to ensure plot completeness. $(iii)$ Recurrent Detailed Caption Generation Pipeline with Dense Frame Sampling: We propose a detailed video caption pipeline that operates recurrently, enabling us to generate detailed captions for videos of any length. This pipeline has three levels, each level of description represents a different time-range: from 10 seconds to the entire video length. It is recurrent as the historical description from any level serves as the context for generating new descriptions at any level. Additionally, we adopted a dense sampling strategy of one frame per second to ensure the sampled frames are rich enough to represent the videos. $(iv)$ Diverse Tasks: Based on the detailed video descriptions, we can generate question-answer pairs. To ensure our questions cover a wide range of scenarios, by referring to the video question-answering dataset, we define 16 question types. We prompt GPT-4o to generate question-answer pairs by referring to these question types, covering open-ended and multi-choice questions.

Based upon the LLaVA-Video-178K dataset, we developed LLaVA-Video. Contrary to previous studies suggesting that training with single frames is sufficient for video-language understanding (Lei et al., ), our findings reveal a significant impact of frame count on LLaVA-Video’s performance, attributable to the detailed features of LLaVA-Video-178K. Observing this, we explored maximizing frame sampling within the constraints of limited GPU memory. We introduce LLaVA-Video${}_{\leavevmode\nobreak\ \mathtt{SlowFast}}$, a video representation technique that optimally distributes visual tokens across different frames. This approach allows for incorporating up to three times more frames than traditional methods, which allocate an equal number of visual tokens to each frame.

Our contributions are as follows:

- •
Video-language Instruction-Following Data: We present a high-quality dataset LLaVA-Video-178K tailored for video instruction-following. It consists of 178K video with 1.3M instruction samples, including detailed captions, free-form and multiple-choice question answering.
- •
Video Large Multimodal Models: We develop LLaVA-Video, a series of advanced large video-language models that expand the capabilities of open models in understanding video content.
- •
Open-Source: In an effort to support the development of general-purpose visual assistants, we release our multimodal instruction data, codebase, model checkpoints, and a visual chat demo to the public.

## 2 Related Work

In this work, our goal is to create a high-quality video-language dataset that goes beyond simple video captions. We aim to improve the ability to follow instructions, which includes detailed video descriptions, open-ended video question-answering, and multiple-choice video question-answering data. We discuss related datasets in Table 1. Previous video-language datasets (Miech et al., ) include manually annotated data for various tasks, such as video captions (Chen & Dolan, ; Xu et al., ; Rohrbach et al., ; Anne Hendricks et al., ; Caba Heilbron et al., ; Zhou & Corso, ), and video question-answering (Yu et al., ; Zadeh et al., ; Xiao et al., ). However, manual annotation is expensive and limits the size of such datasets. To address the shortage of data, studies like (Miech et al., ; Lee et al., ; Zellers et al., ; Xue et al., ) suggest automatically annotating data using subtitles created by ASR. While this method greatly expands the dataset size to 100 million samples, the subtitles often fail to accurately describe the main video content. Additionally, other studies (Xu et al., ; Grunde-McLaughlin et al., ; Wu et al., ) use language models (Xu et al., ) or question templates (Grunde-McLaughlin et al., ; Wu et al., ) to generate question-answer pairs. Although this approach can generate a large number of questions and answers, it often produces poor-quality questions that do not reflect real-world user inquiries. More recent research (Chen et al., ) has prompted video-language models such as BLIP-2 (Li et al., ), VideoChat (Li et al., ), Video-LLaMA (Zhang et al., ), and MiniGPT-4 (Zhu et al., ) to generate video captions. However, these models are limited in their ability to provide detailed descriptions.

The most related works to ours are the recent AI-generated synthetic video instruction tuning data, LLaVA-Hound (Zhang et al., ) and ShareGPT4Video (Chen et al., ), where they have used GPT-4 (OpenAI, ) to generate video captions and open-ended video question-answering. Although the quality of the captions and question-answer pairs has significantly improved, the video sources they use are too static to produce high-quality data for instruction-following scenarios. They also only use very sparse frames for prompting GPT-4V, which results in annotations that fail to capture nuanced actions and continuous plots in the videos. Additionally, Shot2Story (Han et al., ) and Vript (Han et al., ) also employ GPT-4V (OpenAI, ) for video captioning. Their outputs, however, include audio details, which are outside the scope of this study.

## 3 Video Instruction-Following Data Synthesis

Figure: Figure 1: Video sources in the proposed LLaVA-Video-178K. (Left) The relationship between 10 video sources we have utilized and other existing video-language datasets. (Right) Filtering logic for video sources. The detail of filtering logic: ① Sorted by Views, ② Number of scenes greater than 2, ③ Video duration between 5 seconds and 180 seconds, ④ Ratio of scenes to video duration less than or equal to 0.5, ⑤ Resolution greater than 480p, ⑥ 50 samples for each category.
Refer to caption: x1.png

A high-quality dataset for video instruction-tuning is crucial for developing effective video-language models. We identify a key factor in building such datasets: ensuring richness and diversity in both video content and its language annotations. We perform comprehensive survey on the existing video benchmarks, covering across various public video captioning and question-answering datasets, then identify ten unique video sources that contribute to over 40 video-language benchmarks. From each source, we select videos that exhibit significant temporal dynamics. To maintain diversity in the annotations, we establish a pipeline capable of generating detailed captions for videos of any length. Additionally, we define 16 types of questions that guide GPT-4o in creating question-answer pairs to assess the perceptual and reasoning skills of the video-language models.

### 3.1 Video source

One important starting point in building a high-quality video instruction-following dataset is to find a sufficiently diverse pool of video data. From this pool, we can select the qualified videos. In our study of public video-language datasets—including video captioning, video question answering, video summarization, and moment-wise captioning—we noticed that although different datasets focus on various video understanding tasks (e.g., AGQA (Grunde-McLaughlin et al., ) for spatial-temporal relations and STAR (Wu et al., ) for situational reasoning), most are sourced from ten main video sources. For instance, both AGQA and STAR use data from Charades (Sigurdsson et al., ). Specifically, these ten sources are HD-VILA-100M (Xue et al., ), InternVid-10M (Wang et al., ), VidOR (Shang et al., ), VIDAL (YouTube Shorts)(Zhu et al., ), YouCook2(Zhou & Corso, ), Charades (Sigurdsson et al., ), ActivityNet (Caba Heilbron et al., ), Kinetics-700 (Kay et al., ), Something-Something v2 (Goyal et al., ), and Ego4d (Grauman et al., ). These sources offer a wide range of video data from different websites, viewpoints, and domains. The relationship between these ten selected video datasets and others is shown in Fig. 1. The videos from this ten datsets build the video pool for the further video selection. Notably, we use untrimmed videos from each source except for YouCook2 and Kinetics-700. We believe that cutting videos into clips can break the plot continuity, which is essential for understanding the videos.

Based on the video pool, we aim to select dynamic videos. In Figure 1, we outline our criteria for selecting high-quality data. Our main method for identifying dynamic content involves using PySceneDetect, which calculates the number of scenes in a video
We found that the number of scenes is a good indicator of video dynamism. Additionally, we have designed a specific approach ④ to exclude videos that mainly contain “slides."

Figure: Figure 2: The video detail description creation pipeline. A three-level creation pipeline is considered, with each level developed via a recurrent approach. Note that $t$ is the index of time internal at its own level, and $T$ is the last time internal index. (a) To generate the caption for time internal $t$ at level-1, we condition on the current frames in this internal, the caption for time internal $t-1$, and the most recent description summary at level-2 if applicable. (b) To generate caption for time internal $t$ at level-2, we condtion on the previous caption at level-2, and captions from three most recent time internals at level-1. (c) To generate the overall caption at the last time internal $T$ at level-3, we condtion on the the most recent caption at level-2 and the current caption from level-1.
Refer to caption: x2.png

### 3.2 Video Detail Description

#### Automated Generation

For selected videos, we use GPT-4o (OpenAI, ) to systematically describe their content. We start by sampling video frames at one frame per second (fps). However, due to the input size constraints of GPT-4o, we cannot use all sampled frames. Instead, we describe the videos sequentially, as shown in Fig 2. We create descriptions at three distinct levels, detailed below.

- •
Level-1 Description: Every 10 seconds, we provide a level-1 description that outlines the events in that segment. This description considers: frames from the current clip and historical context, which includes all recent level-1 descriptions not yet summarized into a level-2 description and the latest level-2 description.
- •
Level-2 Description: Every 30 seconds, we creat a level-2 summary of the entire video plot up to that point. This is based on the last three level-1 descriptions, covering the most recent 30 seconds; and the latest level-2 description.
- •
Level-3 Description: At the video’s end, we generate a level-3 description to encapsulate the entire video. The inputs for this description are the recent level-1 descriptions not yet summarized, covering the last moments of the plot after the recent summary; and the latest level-2 description.

Figure: Figure 3: Question types for video question answering in data creation. For each type, we provide its name and an example question.
Refer to caption: x3.png

### 3.3 Video Question Answering

#### Question Type definition

In addition to detailed video descriptions, our dataset includes a variety of question-answer pairs designed for complex interactions. This setup improves the video understanding model’s ability to handle real-life queries. We refer to public video question-answering benchmarks (Xiao et al., ; Yu et al., ; khattak et al., ; Liu et al., ) to organize these questions into 16 specific categories, as shown in Fig. 3.

#### Automated Generation

Given a detailed video description, we use GPT-4o to generate at most one question-answer pair for each type of question. The prompts include: (1) The task definition for the current question type. (2) In-context examples for this type, which include three video descriptions and their three question-answer pairs of this specific type. (3) The detailed video description for the current video.
We instruct GPT-4o to return None if it cannot generate question-answer pairs for a specific question type.

#### Filtering.

To filter out the generated question-answer pairs, we apply the following strategy: (1) remove duplicates using the sentence-transformer (Reimers & Gurevych, ), (2) discard answers that begin with phrases like “does not specify,” “does not mention,” “does not specifically,” “does not depict,” or “does not show.”

Figure: Figure 4: One example to illustrate the video instruction-following data.
Refer to caption: x4.png

### 3.4 Dataset Statistics

#### Overview.

We carefully select from our collected data sources to form a balanced and comprehensive collection, resulting in a total of 178K videos and 1.3M instruction-following samples. This includes 178K captions, 960K open-ended QAs, and 196K multiple-choice QAs.

Figure: Figure 5: Distribution of data across different datasets and question types (Caption, Open-ended, and Multi-Choice).
Refer to caption: x5.png

Figure: Figure 6: (Left) Visualization of the video duration. (Middle) Visualization of the number of words in the video caption. (Right) Visualization of caption length versus video duration.
Refer to caption: x6.png

Figure: Figure 7: (Left) Display of YouTube Shorts across four video categories. (Right) Distribution of 5 uniformly chosen video categories.
Refer to caption: x7.png

We present the distribution in Figure 6. Our dataset shows a balanced mix across different video sources, providing a varied content selection. For each task type (caption, open-ended question, multiple-choice question), VIDAL (YouTube Shorts) has the highest share at 24.8%, 31.1%, and 30.1% respectively. It is followed by HD-VILA-100M (21.7%, 27.5%, 26.4%) and InternVid-10M (20.3%, 25.6%, 24.6%).

Figure 6 (Left) illustrates the distrubtion of the video duration. Video lengths range from 0s to 180s, with each length category containing at least 600 videos. Videos shorter than 50 seconds are numerous, mainly because all videos from VIDAL (24.8% of the dataset), which contains YouTube Shorts with lengths under 45 seconds.
Figure 6 (Middle) illustrates the distribution on the number of words for the synthetic captions.
Figure 6 (Right) shows how video length correlates with the length of captions. Generally, longer videos feature longer captions.

For each video in LLaVA-Video-178K, referencing InsTag (Lu et al., ), we employ an in-house tagging model to categorize the video content. Figure 7 displays the distribution of ten uniformly sampled video categories, showcasing examples from four of these categories. Among all videos, “comedy” predominates, primarily because YouTube Shorts is one of the most common sources in our dataset. Comedy is a typical genre that tends to attract high view counts—videos with large viewerships are more likely to be collected, as indicated in Table 1. Additionally, our dataset includes some domains less represented in current video-language datasets, such as computer games.

**Table 1: Comparison of LLaVA-Video-178K and other video-language datasets. Average FPS represents the average number of frames per second that are used to prompt GPT-4o/GPT-4V for annotation. ★ VIDAL, WebVid, ActivityNet. ◼ Panda-70M, Pexels, Pixabay, Mixkit, BDD100K, Ego4d. ✪ HD-VILA-100M, Kinetics-700M, Ego4D, VidOR, InternVid, YouCook2, ActivityNet, Sth-sthv2, VIDAL, Charades.**
|  | Text | Video | #Video | Total Video | Average | #Caption | #OE | #MC |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Source | Length | FPS | QA | QA |  |  |  |  |
| LLaVA-Hound | GPT-4V | ★ | 900K | 3Khr | 0.008 | 900K | 900K | 0 |
| ShareGPT4Video | GPT-4V | ◼ | 40K | 0.2Khr | 0.15 | 40K | 0 | 0 |
| LLaVA-Video-178K | GPT-4o | ✪ | 178K | 2Khr | 1 | 178K | 960K | 196K |

#### Dataset Comparison

We provide a comparison of high-quality instruction following video-language datasets, with a focus on synthetic data created with strong AI models, as shown in Table 1.
$(i)$ A broad collection of dynamic videos. In terms of video sources, although LLaVA-Hound (Zhang et al., ) contains the largest number of videos, 44% of its video data are sourced from WebVid (Bain et al., ), where most videos are static. ShareGPT4Video (Chen et al., ) includes 30% of its videos from Pexels, Pixabay, and Mixkit, which are aesthetically good but also mostly static. Additionally, the majority of its videos come from Panda-70M, which are short clips from longer videos—suggesting simpler plots. In contrast, we carefully select video sources that offer dynamic, untrimmed videos with complex plots, which are crucial for developing a powerful video understanding model.(^1^11Example videos: [WebVid](https://ak.picdn.net/shutterstock/videos/21179416/preview/stock-footage-aerial-shot-winter-forest.mp4),[Pixabay](https://pixabay.com/videos/plane-modelling-miniature-lockheed-134519/),[Pexels](https://www.pexels.com/video/a-bird-is-standing-on-the-beach-27916646/),[Mixkit](https://mixkit.co/free-stock-video/a-young-woman-clad-in-snugly-black-sportswear-doing-lunges-52112/).)
$(ii)$ High frames per second. Regarding frame sampling in language annotations, the proposed datasest considers 1 FPS, while other datasets consider much lower FPS. LLaVA-Hound uniformly samples 10 frames from videos of any length. The average FPS is 0.008, which may miss some fine details. ShareGPT4Video picks key frames using CLIP (Radford et al., ) based on frame uniqueness. This method might also miss subtle changes in the video because CLIP embeddings do not capture fine-grained dynamics well. Our method samples FPS=1 without using key frame selection algorithms, ensuring the detailed temproal information can be expressed in annotations and high coverage.
$(iii)$ Diverse tasks. The proposed dataset considers three common task types, including caption, free-form and closed-form QA, while existing datasets only consider a subset. Meanwhile, the quality and numbers of samples in our dataset is higher.

## 4 Experiments

We conducted evaluations for the LLaVA-Video models across all benchmarks using LMMs-Eval (Zhang et al., ) to ensure standardization and reproducibility. To fairly compare with other leading video LMMs, we primarily used results from original papers. When results were not available, we integrated the models into LMMs-Eval and assessed them under consistent settings. Following LLaVA-OneVision (Li et al., ), we employed SigLIP (Zhai et al., ) as our vision encoder, and Qwen2 (Yang et al., ) as the LLM. The LLaVA-Video model builds on the single-image (SI) stage checkpoint from the LLaVA-OneVision model (Li et al., ), which was trained using only image data.

#### Video Representations

Following the classic SlowFast idea in video representations (Feichtenhofer et al., ; Xu et al., ; Huang et al., ), we develop LLaVA-Video${}_{\leavevmode\nobreak\ \mathtt{SlowFast}}$ to optimize the balance between the number of frames and the count of visual tokens, within the budget of the limited context window in LLM and GPU memory for video representation. Please refer to Appendix B for detailed information. Specifically, we represent each video as a sequence with maximum $T$ frames. Each frame is represented in $M$ tokens. we categorize the frames into two groups, based on the a strike rate $s$, where the every $s$ frames are uniformly selected to form the slow frame group, and the rest of the frames are consdiered as the fast frame group. Note that a special case $s=1$ leads to only one group, reducing the SlowFast representation to the original simple representation. For each group, we apply different pooling rate using Pytorch function pooling $\mathtt{avg_pool2d}()$. $p\times p$ pooling and $2p\times 2p$ pooling for slow and fast frames, respectively. To summarize, we paramterize the video representation configuration as $\mathcal{V}=(T,M,s,p)$. The total number of tokens is $\#tokens=\left\lfloor T/s\right\rfloor\times\left\lfloor M/p^{2}\right\rfloor+
\left(T-\left\lfloor T/s\right\rfloor\right)\times\left\lfloor M/p^{2}\right\rfloor$

#### Evaluation Benchmarks.

For full evaluation, we consdier 11 video benchmarks. conducted tests across various video captioning , video open-ended question-answering and video multiple-choice question-answering benchmarks, including ActivityNet-QA (Yu et al., ), which features human-annotated action-related QA pairs from the ActivityNet dataset. We also utilized LongVideoBench (Wu et al., ), EgoSchema (Mangalam et al., ), and MLVU (Zhou et al., ) for long video understanding, PerceptionTest (Pătrăucean et al., ) for assessing fine-grained perception skills, and VideoMME (Fu et al., ) and NExT-QA (Xiao et al., ) for diverse video domains and durations. Additional tests included VideoDetailCaption (LMMs-Lab, ) Dream-1K (Wang et al., ) for detailed video descriptions and Video-ChatGPT (Maaz et al., ) for visual chat.

For ablation studies in . 4.2 and Sec. 4.3, we conduct evaluation across 4 datasets. NExT-QA (Xiao et al., ) and PerceptionTest (Pătrăucean et al., ), which use training data from the LLaVA-Video-178K, are treated as in-domain datasets. Conversely, VideoMME (Fu et al., ) and EgoSchema (Mangalam et al., ) are considegreen zero-shot datasets.

### 4.1 Overall Results

We fine-tune LLaVA-OneVision (SI) on the joint dataset of video and image data. Specifically, we added video data from the LLaVA-Video-178K dataset and four public datasets:
ActivityNet-QA (Yu et al., ), NExT-QA (Xiao et al., ), PerceptionTest (Pătrăucean et al., ), and LLaVA-Hound-255K (Zhang et al., ), focusing on videos shorter than three minutes. These datasets were selected to improve our model’s performance, contributing to a total of 1.6 million video-language samples, which include 193,510 video descriptions, 1,241,412 open-ended questions, and 215,625 multiple-choice questions. Remarkably, 92.2% of the video descriptions, 77.4% of the open-ended questions, and 90.9% of the multiple-choice questions were newly annotated. Additionally, we used 1.1 million image-language pairs from the LLaVA-OneVision model (Li et al., ).
We consider the same video representation configurations for the training and inference stages. On 128 NVIDIA H100 GPUs, the video representations for LLaVA-Video-7B and LLaVA-Video-72B are $\mathcal{V}=(64,679,1,2)$ and $\mathcal{V}=(64,679,3,2)$, respectively.

**Table 2: LLaVA-Video performance on video benchmarks. We report the score out of 5 for VideoDC, VideoChatGPT while other results are reported in accuracy. All results are reported as 0-shot accuracy. *indicates that the training set has been observed in our data mixture.**
|  | Caption | Open-Ended Q&A | Multi-Choice Q&A |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | VideoDC | Dream-1K | ActNet-QA | VideoChatGPT | EgoSchema | MLVU | MVBench | NExT-QA | PerceptionTest | LongVideoBench | VideoMME |
| test | test | test | test | test | m-avg | test | mc | val | val | wo/w-subs |  |
| Proprietary models |  |  |  |  |  |  |  |  |  |  |  |
| GPT-4V (OpenAI, ) | 4.00 | 34.4 | 57.0 | 4.06 | - | 49.2 | 43.5 | - | - | 61.3 | 59.9/63.3 |
| GPT-4o (OpenAI, ) | - | 39.2 | - | - | - | 64.6 | - | - | - | 66.7 | 71.9/77.2 |
| Gemini-1.5-Flash (Team et al., ) | - | 34.8 | 55.3 | - | 65.7 | - | - | - | - | 61.6 | 70.3/75.0 |
| Gemini-1.5-Pro (Team et al., ) | - | 36.2 | 57.5 | - | 72.2 | - | - | - | - | 64.0 | 75.0/81.3 |
| Open-source models |  |  |  |  |  |  |  |  |  |  |  |
| VILA-40B (Lin et al., ) | 3.37 | 33.2 | 58.0 | 3.36 | 58.0 | - | - | 67.9 | 54.0 | - | 60.1/61.1 |
| PLLaVA-34B (Xu et al., ) | - | 28.2 | 60.9 | 3.48 | - | - | 58.1 | - | - | 53.2 | - |
| LongVA-7B (Zhang et al., ) | 3.14 | - | 50.0 | 3.20 | - | 56.3 | - | 68.3 | - | - | 52.6/54.3 |
| IXC-2.5-7B (Zhang et al., ) | - | - | 52.8 | 3.46 | - | 37.3 | 69.1 | 71.0 | 34.4 | - | 55.8/58.8 |
| LLaVA-OV-7B (Li et al., ) | 3.75 | 31.7 | 56.6 | 3.51 | 60.1 | 64.7 | 56.7 | 79.4* | 57.1 | 56.5 | 58.2/61.5 |
| VideoLLaMA2-72B (Cheng et al., ) | - | 27.1 | 55.2 | 3.16 | 63.9 | 61.2 | 62.0 | - | - | - | 61.4/63.1 |
| LLaVA-OV-72B (Li et al., ) | 3.60 | 33.2 | 62.3 | 3.62 | 62.0 | 68.0 | 59.4 | 80.2* | 66.9 | 61.3 | 66.2/69.5 |
| LLaVA-Video-7B | 3.66 | 32.5 | 56.5* | 3.52 | 57.3 | 70.8 | 58.6 | 83.2* | 67.9* | 58.2 | 63.3/69.7 |
| LLaVA-Video-72B | 3.73 | 34.0 | 63.4* | 3.62 | 65.6 | 74.4 | 64.1 | 85.4* | 74.3* | 61.9 | 70.5/76.9 |

In Table 2, we compare the performance of different models on various video benchmarks. The 72B model performs as well as the commercial, closed-source model Gemini-1.5-Flash (Team et al., ), highlighting the effectiveness of open-source efforts in achieving comparable results. The LLaVA-Video-7B model outperforms the previous top model, LLaVA-OV-7B, in seven out of ten datasets. Analysis of individual datasets shows some noteworthy trends. For instance, on benchmarks like MLVU, LongVideoBench, and VideoMME, which primarily use video data from YouTube, this improvement may be due to the inclusion of extensive YouTube data in LLaVA-Video-178K, as illustrated in Fig. 5. Additionally, the improvement on ActivityNet-QA is small; this could be because many questions in ActivityNet-QA, such as “What’s the color of the ball?” can be answered by viewing a single frame. The visibility of the ball from the beginning to the end of the video means understanding the video sequence is unnecessary, so LLaVA-Video-178K offers little advantage in this context.
We find that LLaVA-Video-7B is notably weaker in the specialized task of EgoSchema, an ego-centric dataset. This weakness may be due to a significant reduction in the proportion of ego-centric data in the training dataset of LLaVA-Video. However, this impact is less pronounced in larger models, as demonstrated by the LLaVA-Video-72B model’s superior performance over LLaVA-OV-72B in EgoSchema.

### 4.2 Dataset Ablation

Note that the training set for LLaVA-Video includes six datasets: LLaVA-Video-178K, LLaVA-Hound (Zhang et al., ), NExT-QA (Xiao et al., ), ActivityNet-QA (Yu et al., ), PerceptionTest (Pătrăucean et al., ), and image data from LLaVA-OneVision (Li et al., ). In this section, we conduct ablation studies to assess the impact of each dataset. We separately fine-tune the LLaVA-OneVision (SI) model for each experimental setting, progressively adding datasets to the baseline. We use a video representation defined by $\mathcal{V}=(64,679,1,2)$

The results are presented in Table 3. Initially, we used a basic model trained solely on the LLaVA-Hound dataset as our baseline. Compared to this baseline, adding the LLaVA-Video-178K dataset significantly improved performance, enhancing scores in both in-domain and out-of-domain tasks. Specifically, we observed a 31.9-point increase in NExT-QA scores and a 9.1-point rise in VideoMME scores. Furthermore, including the PerceptionTest dataset significantly enhanced its associated task. Additionally, integrating high-quality image data provided modest benefits on EgoSchema.

**Table 3: Ablation study on the LLaVA-Video model with various configurations of training data. Three Q&A datasets indicate: NExT-QA, ActivityNet-QA and PerceptionTest.**
|  | in-domain | out-of-domain |  |  |
| --- | --- | --- | --- | --- |
| Method | NExT-QA | PerceptionTest | EgoSchema | VideoMME |
| mc | val | test | wo |  |
| LLaVA-Hound | 48.2 | 51.4 | 51.0 | 54.1 |
| +LLaVA-Video-178K | 80.1 | 57.1 | 56.5 | 63.2 |
| +Three Q&A datasets | 80.1 | 69.0 | 55.6 | 61.9 |
| +LLaVA-OV (images) | 83.2 | 67.9 | 57.3 | 63.4 |

### 4.3 Dataset Comparison

We conduct two ablation studies to further analyze our dataset and training strategy. As shown in Table 4, we compared three datasets where the language annotations are from GPT-4V/GPT-4o. For each experiment, we fine-tune the LLaVA-OneVision (SI) model separately on each specific dataset setting, utilizing a video representation defined by $\mathcal{V}=(64,679,1,2)$.

Two group of experiments are considered to assess the data quality of LLaVA-Video-178K compare to LLaVA-Hound and ShareGPT4Video. In the first group, to compare LLaVA-Video-178K with LLaVA-Hound, we randomly selected 900K open-ended questions to match the number in LLaVA-Hound. We included all captions and did not sample the multiple-choice questions. In the second group, comparing LLaVA-Video-178K to ShareGPT4Video, we randomly sampled 40K video captions to align with those in ShareGPT4Video. Since ShareGPT4Video lacks open-ended and multiple-choice questions, we supplemented with annotations from NExT-QA, PerceptionTest, and ActivityNet-QA.
In the first group of Table 4, we compare LLaVA-Video-178K with LLaVA-Hound. Although LLaVA-Hound has more captions than LLaVA-Video-178K, our results are still better. As shown in Table 1, despite LLaVA-Hound annotates more videos, its quality is limited due to two main issues: (1) Static video: Its primary video source is WebVid (Bain et al., ), which tends to have relatively static content. (2) Sparse sampling: Although it includes data sources with dynamic videos, its sampling rate of 10 frames per video leads to annotations that do not fully capture the complete plot of the video. This underscores that the quality of video instruction-following data is more important than its quantity. Additionally, the second experiment group in Table 4 shows that the model trained with LLaVA-Video-178K outperforms that of ShareGPT4Video, highlighting the superiority of our data’s quality.

**Table 4: Comparison of LLaVA-Video-178K and other video instruction-following datasets.**
|  |  |  |  | in-domain | out-of-domain |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | #Caption | #OE | #MC | NExT-QA | PerceptionTest | EgoSchema | VideoMME |
|  | mc | val | test | wo |  |  |  |
| LLaVA-Hound | 900K | 900k | 0 | 64.4 | 51.4 | 51.0 | 51.0 |
| LLaVA-Video-178K | 178K | 900k | 0 | 73.2 (+8.8) | 55.9 (+4.5) | 49.8 (-1.2) | 59.6 (+8.6) |
| ShareGPT4Video | 40K | 40K | 19K | 69.6 | 55.2 | 58.9 | 51.0 |
| LLaVA-Video-178K | 40K | 40K | 19K | 75.8 (+6.2) | 55.4 (+0.2) | 55.8 (-3.1) | 53.5 (+2.5) |

## 5 Conclusion

This study introduces the LLaVA-Video-178K dataset, a high-quality synthetic dataset for video-language instruction-following. It is favored for its dense frame sampling rate in longer, untrimmed videos, covering diverse tasks such as captioning, open-ended and multi-choice QA. By training on the joint dataset of LLaVA-Video-178K with existing visual instruction tuning data, we developed a new model family, LLaVA-Video, which also considers video representation to effectively use GPU resources. This allows us to include more frames in the training process. The experimental results have demonstrated the effectiveness of the proposed synthetic dataset, and LLaVA-Video models have achieved excellent performance on a wide range of video benchmarks.

## Appendix A Author Contributions

- Yuanhan Zhang contributed to the LLaVA-NeXT-Video series by developing the video training and inference codebase, creating the annotation pipeline, designing video representations, and conducting exploratory experiments.

- Jinming Wu contributed to the collecting and curating the video data, providing consistent technical support throughout the project, including helping improve codebase and conducting experiments.

- Wei Li engaged in designing the video data collection pipeline and experimental setups.

- Bo Li worked on providing LLaVA-OneVision codebase and training recipes as the initial foundation of current project, and offered help in integration to SGLang as well as demo service.

- Zejun Ma acquired a part of the data annotation and computational resources for the project.

- Ziwei Liu offered valuable suggestions throughout the project, contributing to discussions on the data collection pipeline design and experimental setups.

- Chunyuan Li initiated and led the project, designed the roadmap in the data collection, curation and expriments, and revised the paper.

## Appendix B Video Representations

### B.1 Efficient Video Representations in LMMs

Current designs of large multimodal models (LMM) typically connect a vision encoder (Radford et al., ; Zhai et al., ) to a large language model (Yang et al., ) through a lightweight projector (Liu et al., ) or a resampler (Li et al., ; Alayrac et al., ). These components transform a set of visual representations into “visual tokens” aligned with text embeddings. In contrast to image-based LMMs, which generate only a small number of visual tokens easily managed by a standard GPU, video LMMs face challenges due to a large number of visual tokens derived from multiple video frames. The LLaVA-NeXT-Video (Zhang et al., ) and PLLaVA (Xu et al., ) models address this by simly considering average pooling to reduce the number of tokens representing each frame.

Following the idea of SlowFast in the traditional video understanding (Feichtenhofer et al., ), adaptive reductions in visual tokens are demonstrated by recent video LMMs, LITA (Huang et al., ) and SlowFast-LLaVA (Xu et al., ). Initially, these methods represent all sampled frames with a minimal number of visual tokens (fast frame)— typically just one—by using a large pooling stride. They then switch to a smaller pooling stride for certain frames to retain more visual tokens (slow frame). Finally, they combine the visual tokens of fast frames with those of slow frames. However, this approach can lead to some frames being represented twice. In contrast, our method uses a larger pooling stride for sampled frames to maintain fewer visual tokens (fast frame) or a smaller stride for others to keep more (slow frame). We then arrange slow and fast frames in an interleaving pattern.

Figure: Figure 8: Video representations. A different number of tokens are utilized to represent frames.
Refer to caption: x8.png

### B.2 LLaVA-Video 𝚂𝚕𝚘𝚠𝙵𝚊𝚜𝚝 𝚂𝚕𝚘𝚠𝙵𝚊𝚜𝚝 {}_{\leavevmode\nobreak\ \mathtt{SlowFast}} start_FLOATSUBSCRIPT typewriter_SlowFast end_FLOATSUBSCRIPT

We represent each video as a sequence with maximum $T$ frames. Each frame is represented in $M$ tokens. FPS-based video representation can be considered in the future.
Specifically, each frame is encoded via an image encoder and a two-layer MLP for projection. These visual tokens are concatenated with word tokens and processed by a large language model (LLM).
Managing tokens for every frame can be computationally demanding.
For instance, employing the SigLIP (Zhai et al., ) encoder for a video with $T=100$ results in 67,600 tokens, assuming $M=729$ tokens per frame, which often exceeds GPU memory limits. This issue is exacerbated when using large-parameter LLMs; with the Qwen2-72B model, we could only process 8 frames before maxing out the memory on 128 NVIDIA H100 GPUs. Such a limited number of frames can introduce inconsistencies in language annotations, reducing model efficacy.
One strategy to incorporate more frames is by applying $p\times p$ spatial average pooling to reduce $M$ to $M/p^{2}$, thus lowering the token count per frame as suggested by recent studies (Xu et al., ; Zhang et al., ). However, the number of visual tokens is crucial for preserving the informational content of each frame, which is vital for video comprehension.

In our LLaVA-Video${}_{\leavevmode\nobreak\ \mathtt{SlowFast}}$, we categorize the frames into two groups, based on the a strike rate $s$, where the every $s$ frames are uniformly selected to form the slow frame group, and the rest of the frames are consdiered as the fast frame group. Note that a special case $s=1$ leads to only one group, reducing the SlowFast representation to the original simple representation. For each group, we apply different pooling rate using Pytorch function pooling $\mathtt{avg_pool2d}()$. $p\times p$ pooling and $2p\times 2p$ pooling for slow and fast frames, respectively. To summarize, we paramterize the video representation configuration as $\mathcal{V}=(T,M,s,p)$. The total number of tokens is $\#tokens=\left\lfloor T/s\right\rfloor\times\left\lfloor M/p^{2}\right\rfloor+
\left(T-\left\lfloor T/s\right\rfloor\right)\times\left\lfloor M/p^{2}\right\rfloor$

## Appendix C Data

### C.1 Video Detail Description

As discussed in Section 3.2, we show that generating level-1 description should consider historical context. Figure 9 illustrates the impact of excluding historical context on the quality of video descriptions. Specifically, including historical context helps accurately identify characters across different times as the same individual.

Figure: Figure 9: Generating video captions with or without historical context.
Refer to caption: x9.png

**Table 5: Question types for video question answering in data creation. For each type, we provide its name, description, and the proportion it represents in the LLaVA-Video-178K.**
| Question type | Description | Proportion |
| --- | --- | --- |
| Temporal | Designed to assess reasoning about temporal relationships between actions/events. Questions involve previous, present, or next actions. | 7.2% |
| Spatial | Tests ability to perceive spatial relationships between observed instances in a video scene. | 7.2% |
| Causal | Focuses on explaining actions/events, determining intentions of actions or causes for subsequent events. | 7.2% |
| Description-Scene | Assesses ability to describe the major scene of the video, like where it takes place and the overall environment. | 7.2% |
| Description-Human | Involves describing actions or attributes of people, such as their activities and appearances. | 6.7% |
| Description-Object | Assesses ability to describe attributes of objects, like their appearance and function. | 7.0% |
| Count | Tests ability to count instances of objects, people, actions, and to distinguish between old and new elements in a scene. | 7.1% |
| Binary | Involves yes or no questions related to the video content. | 7.2% |
| Fine Grained Action Understanding | Creates questions challenging comprehension of subtle actions. | 6.5% |
| Plot Understanding | Challenges ability to interpret the plot in the video. | 7.1% |
| Non-Existent Actions with Existent Scene Depictions | Assesses reasoning with introduced non-exist ent activities without changing physical details. | 6.6% |
| Time Order Understanding | Challenges recognition of temporal sequence of activities in videos. | 6.9% |
| Object Direction | Emphasizes perception of object movement direction. | 3.8% |
| Camera Direction | Focuses on the direction of camera movement. | 4.1% |
| Speed | Delves into discerning variations in speed, including absolute and relative speeds. | 3.6% |
| Attribute Change | Centers on how attributes of objects or the entire video change over time, like size, shape, color, and more. | 4.5% |

**Table 6: We explain the process of creating prompts for GPT-4O to gather question-answer pairs from each video description. tasks includes the definition of all question types along with examples of question-answer pairs. We instruct GPT-4O to generate questions that cover as many question types as possible.**
| tasks = “<br># Temporal: this task is designed to assess the capability of reasoning …$<$omitted$>$<br>## caption-1: The video features a child sitting in a baby chair at a dining table, creating…$<$omitted$>$<br>## question-1: What was the child doing as he sat on the baby chair?<br>## answer-1: The child was reading a book.<br>…<br>## caption-3: …$<$omitted$>$<br>## question-3: …$<$omitted$>$<br>## answer-3: …$<$omitted$>$<br># Spatial: this task involves creating questions that test a person’s ability…$<$omitted$>$<br>…$<$omitted$>$ ”<br>system_message = “<br>### Task:<br>Given a detailed description that summarizes the content of a video, generate question-answer pairs based on the description to help humans better understand the video.<br>The question-answer pairs should be faithful to the content of the video description and developed from different dimensions to promote comprehensive understanding of the video.<br>Here are some question dimensions and their explanations and exampled question-answer pairs for reference:<br>{task_definitions}<br>#### Guidelines For Question-Answer Pairs Generation:<br>- Read the video description provided carefully, paying attention to the content, such as the scene where the video takes place, the main characters and their behaviors, and the development of the events.<br>- Generate appropriate question-answer pairs based on the description. The question-answer pairs should cover as many question dimensions and not deviate from the content of the video description.<br>- Generate 1 question-answer pair for each dimension.<br>### Output Format:<br>1. Your output should be formed in a JSON file.<br>2. Only provide the Python dictionary string.<br>Your response should look like:<br>["Dimension": <dimension-1>, "Question": <question-1>, "Answer": <answer-1>,<br>"Dimension": <dimension-2>, "Question": <question-2>, "Answer": <answer-2>...] ”<br>user_message = “<br>Please generate question-answer pairs for the following video description:<br>Description: {caption} ”<br>for *cur_video in videos*: |
| --- |

### C.2 Video Question Answering

In Table 5, we list the names and descriptions of different question types and their corresponding proportions in the LLaVA-Video-178K dataset. The prompt used to generate video question-answer pairs from GPT-4O is shown in Table. 6. In Fig. 4, we show an example of a video along with its detailed description, an open-ended question, and a multiple-choice question.

### C.3 Dataset Comparison

**Table 7: Comparison of LLaVA-Video-178K and other video-language datasets. Average FPS represents the average number of frames per second that are used to prompt GPT-4o/GPT-4V for annotation.**
|  | Text | #Video | Total Video | Average | #Caption | #OE | #MC |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Length | FPS | QA | QA |  |  |  |  |
| HowTo100M (Miech et al., ) | ASR | 136M | 134.5Khr | - | 136M | 0 | 0 |
| ACAV (Lee et al., ) | ASR | 100M | 277.7Khr | - | 100M | 0 | 0 |
| YT-Temporal-180M (Zellers et al., ) | ASR | 180M | - | - | 180M | 0 | 0 |
| HD-VILA-100M (Xue et al., ) | ASR | 103M | 371.5Khr | - | 103M | 0 | 0 |
| MSVD (Chen & Dolan, ) | Manual | 1970 | 5.3h | - | 1K | 0 | 0 |
| LSMDC (Rohrbach et al., ) | Manual | 118K | 158h | - | 118K | 0 | 0 |
| MSR-VTT (Xu et al., ) | Manual | 10K | 40h | - | 10K | 0 | 0 |
| DiDeMo (Anne Hendricks et al., ) | Manual | 27K | 87h | - | 27K | 0 | 0 |
| ActivityNet (Caba Heilbron et al., ) | Manual | 100K | 849h | - | 100K | 0 | 0 |
| YouCook2 (Zhou & Corso, ) | Manual | 14K | 176h | - | 14K | 0 | 0 |
| TVQA (Lei et al., ) | Manual | 21K | 3.39Khr | - | 0 | 0 | 152K |
| ActivityNet-QA (Yu et al., ) | Manual | 5.8K | 290h | - | 0 | 58K | 0 |
| Social-IQ (Zadeh et al., ) | Manual | 1.2K | 20h | - | 0 | 0 | 7.5k |
| NExT-QA (Xiao et al., ) | Manual | 5.4K | 66h | - | 0 | 52K | 47K |
| MSVD-QA (Xu et al., ) | Open-source Model | 1.9K | 5.3h | - | 41K | 50K | 0 |
| MSRVTT-QA (Xu et al., ) | Open-source Model | 10K | 40h | - | 0 | 243K | 0 |
| Panda-70M (Chen et al., ) | Open-source Model | 70.8M | 166.8Khr | - | 70.8M | 0 | 0 |
| LLaVA-Hound (Zhang et al., ) | GPT-4V | 900K | 3Khr | 0.008 | 900K | 900K | 0 |
| ShareGPT4Video (Chen et al., ) | GPT-4V | 40K | 0.2Khr | 0.15 | 40K | 0 | 0 |
| LLaVA-Video-178K | GPT-4o | 178K | 2Khr | 1 | 178K | 960K | 196K |

We provide a more comprehensive comparison of LLaVA-Video-178K with other video-language datasets for the video caption task and video question answer task. Specifically, we organize the table into four groups, each characterized by its method of text annotation. As shown in Table 7, unlike other datasets, LLaVA-Video-178K uniquely includes all three types of annotations: captions, open-ended questions, and multiple-choice questions.

**Table 8: Visual Representation Configurations and Performance Correlation. $T^{\text{train}}$ and $T^{\text{test}}$ are the number of frames in the training and inference stage, respectively. $M/{p^{2}}$: number of visual tokens per frame.**
|  |  |  | in-domain | out-of-domain |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| $T^{\text{train}}$ | $T^{\text{test}}$ | $M/p^{2}$ | NExT-QA | PerceptionTest | EgoSchema | VideoMME |
| mc | val | test | wo |  |  |  |
| Training with more frames |  |  |  |  |  |  |
| 32 | 32 | 169 | 80.4 | 68.2 | 56.3 | 59.1 |
| 64 | 64 | 169 | 81.4 (+1.0) | 68.3 (+0.1) | 58.4 (+2.1) | 59.6 (+0.5) |
| 110 | 110 | 169 | 82.0 (+1.6) | 68.3 (+0.1) | 59.1 (+2.8) | 60.4 (+1.3) |
| Inference with more frames |  |  |  |  |  |  |
| 32 | 32 | 169 | 80.4 | 68.2 | 56.3 | 59.1 |
| 32 | 64 | 169 | 80.7 (+0.3) | 68.9 (+0.7) | 56.3 (+0.0) | 59.9 (+0.8) |
| 32 | 110 | 169 | 80.5 (+0.1) | 67.2 (-1.0) | 55.2 (-1.1) | 58.8 (-0.3) |
| Using more frames with fewer visual tokens per frame |  |  |  |  |  |  |
| 32 | 32 | 729 | 79.4 | 69.5 | 58.3 | 59.1 |
| 110 | 110 | 169 | 82.0 (+2.6) | 68.3 (-1.2) | 59.1 (+0.8) | 60.4 (+1.3) |
| 440 | 440 | 64 | 81.6 (+2.2) | 67.2 (-2.3) | 59.4 (+1.1) | 60.2 (+1.1) |

**Table 9: Comparison of different video representations. The video representation $\mathcal{V}$ is consistent in training and inference for all methods, except that SlowFast-LLaVA considers simple representation $\mathcal{V}$ in training and its specified $\mathcal{V}$ in inference.**
|  |  |  | in-domain | out-of-domain |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Method | $\mathcal{V}=(T,M,s,p)$ | #Visual | NExT-QA | PerceptionTest | EgoSchema | VideoMME |
| Tokens | mc | val | test | wo |  |  |
| Simple representation | (32, 729, 1, 2) | 5,408 | 80.4 | 68.2 | 56.3 | 59.1 |
| LLaVA-Video${}_{\leavevmode\nobreak\ \mathtt{SlowFast}}$ | (64, 729, 3, 2) | 5,396 | 81.1 | 67.7 | 57.1 | 59.8 |
| LITA | (42, 729, 2, 2) | 5,313 | 80.8 | 68.3 | 54.3 | 59.1 |
| SlowFast-LLaVA | (42, 729, 2, 2) | 5,313 | 79.4 | 68.2 | 56.2 | 58.9 |

## Appendix D Beyond Singularity: Extensive Sampling Matters

We perform experiments to explore how video representations affect the model’s performance. All experiments were carried out in a video-only setting, using video data with durations from 0 to 30 seconds as our training data. We focused on evaluating how the number of frames and the number of visual tokens per frame impact model performance. Regarding the frame count, it is noteworthy that observing the effects of a high number of frames—such as over 100—does not necessarily require long videos. Our results indicate that the dynamic properties of the data render even 100 frames insufficient to fully capture the condent of a 30-second video, which typically runs at 15 FPS.

In Table 8, the first group shows an increase in the number of frames from 32 to 110. We set 110 frames as the upper limit to avoid overloading the GPU. With more frames, we see significant improvements in all datasets. While it’s generally expected that using more frames boosts performance, previous studies (Luo et al., ; Lei et al., ; ) have noted that performance tends to plateau when training with more than 16 frames. We propose that the saturation observed in earlier studies arises due to the selection of training datasets such as MSVD (Chen & Dolan, ) and WebVid (Bain et al., ), where the video content is highly static, allowing a small number of frames to represent the entire video effectively. In contrast, the dynamic nature of the videos and the detailed nature of the annotations in LLaVA-Video-178K allow for continuous benefits from extensive sampling

The second group in Table 8 demonstrates the effects of varying the number of inference frames while keeping the number of training frames constant. A modest increase in the inference frames slightly enhances performance; however, excessively increasing the number of inference frames can degrade it.

In Table 8’s third group, we illustrates the trade-off between the number of frames and the number of tokens per frame. Configurations with fewer tokens per frame but more frames yield superior results, even with a lower total count of visual tokens (18,590 versus 21,632). This finding emphasizes that increasing the number of frames, rather than the tokens per frame or the total number of tokens, enhances performance. However, a balance is necessary; as the number of frames increases to 440 and the tokens per frame decreases to 64, performance drops. This observation led us to use LLaVA-Video${}_{\leavevmode\nobreak\ \mathtt{SlowFast}}$ for video representation.

## Appendix E Capabilities

Beyong achieve good benchmark performance, Our observations of LLaVA-Video reveal various capabilities in video
understanding. Specifically, it show a great abilities in the understanding video using real-world knowledge,including, but not limited to:

- •
Optical Illusion: As shown in Table 11, LLaVA-Video recognizes that the green dragon in the video is not a real 3D object. It appears three-dimensional due to an optical illusion that affects human perception.
- •
Special Domain: As indicated in Table 11, LLaVA-Video understands the content within special domains in the video, such as sketches and fights in video games.
- •
Unusual Action: As detailed in Table 12, LLaVA-Video identifies atypical actions in the video, such as "physical therapy" for pets, beyond ordinary activities.
- •
Physical Laws: As shown in Table 13, LLaVA-Video comprehends basic physical laws demonstrated in the video, like zero gravity in space stations, which allows objects to float without falling.

**Table 10: LLaVA-Video learns to understand the optical illusion in the video.**
|  |  |
| --- | --- |
| User | Please describe this video. |
| LLaVA-Video | The video features a person interacting with a large, cartoonish green dragon cutout in a well-lit, spacious room adorned with framed pictures on the walls. The person, dressed in a blue tank top and light blue jeans, is seen hugging and adjusting the dragon cutout, which has a wide, toothy grin and large, expressive eyes. The text overlay reads ’Reply to seloutria’s comment: Once you see it you can’t un see it,’ suggesting that the cutout is designed to be visually striking and memorable. The person moves around the cutout, showcasing different angles and perspectives. |
| User | Is this green dragon a real 3D structure? |
| LLaVA-Video | No, the green dragon is a 2D cutout that creates an optical illusion. |

**Table 11: LLaVA-Video learns to understand the the video in special domain.**
|  |  |
| --- | --- |
| User | Please describe this video. |
| LLaVA-Video | The video begins with a blank, lined sheet of paper. A hand holding a black pen starts drawing on the paper, creating the outline of a flower with five petals. The artist adds details to the petals, including lines and shading to give them a more realistic appearance. The stem and leaves are also drawn, completing the flower illustration. The artist adds final touches to the flower drawing, enhancing the details and shading to give it a more realistic appearance. The artist then switches to a yellow pen and begins coloring the center of the flower, adding depth and contrast to the drawing. The video concludes with the artist adding final touches to the coloring, enhancing the overall appearance of the flower illustration. |
| User | What could this flower be? |
| LLaVA-Video | This flower could be a magnolia or a similar species. |
|  |  |
| User | Please describe this video. |
| LLaVA-Video | The video showcases an intense battle scene set in a snowy, mountainous landscape. A character dressed in dark, heavy armor with glowing orange accents wields a sword and faces off against a large, menacing creature with glowing red eyes and antlers. The text ’EVERY BATTLE SCREAMS GOTY POTENTIAL!’ is prominently displayed at the top of each frame, emphasizing the high stakes and epic nature of the confrontation.<br>The battle is fierce, with the character and the creature exchanging powerful attacks. The creature emits blue energy and lightning, while the character retaliates with fiery and icy attacks. The snowy ground is littered with debris and frost, adding to the harsh and brutal environment. The background features a large body of water and a distant lighthouse, enhancing the sense of isolation and danger. |

**Table 12: LLaVA-Video learns to understand the unusual action in the video.**
|  |  |
| --- | --- |
| User | Please describe this video. |
| LLaVA-Video | The video begins with a person holding a tabby cat, with the text ’DOGS WITH BACK PAIN’ and a crying emoji displayed at the bottom of the screen. The person, who has tattoos on their arms and is wearing a light-colored shirt, is in a room with large windows and a white wall. The scene transitions to the person petting a large, brown and white dog, who is sitting on the floor. The person then stands up and gestures with their hands, while the dog looks up at them. The next scene shows the person kneeling and petting a large, fluffy dog with a black and white coat, who is sitting on a table. The person continues to pet the dog, who appears calm and relaxed. |
| User | What did this man do to his pets? |
| LLaVA-Video | This man provided physical therapy to his pets. |

Table: Table 13: LLaVA-Video learns to understand the physical laws in the video.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. In Table 3, what is the training data settings for +LLaVA-Video-178K, +Three Q&A datasets, and +LLaVA-OV (images)? From line 481 to line 485, it seems that LLaVA-Video-178K, three Q&A datasets and LLaVA-OV(images) are incrementally added for training. If so, what is the performance gain if LLaVA-Video-178K is the last dataset added for training? If datasets are trained separately in three settings, then why the performance of LLaVA-Video-178K is lower than LLaVA-OV (images)?
2. In Table 4, are there any insights on the out-of-domain performance loss of LLaVA-Video-178K compared to LLaVA-Hound on EgoSchema?
3. Figure 4 illustrates an interesting video. I am wondering whether the generated questions are only about the “facts” in the video such as “How many steps does “normal people” climb?”. I think people are more curious about whether the model understand the humor in the video. Whether the captions of the video can generate such questions?
soundness: 2
strengths: 1. The paper proposes a caption generation pipeline, which recurrently generates and refines the captions of video in three temporal levels.
2. To filter out the static videos that can be summarized by a single video frame, the authors construct the dataset using dynamic videos which are selected by detecting the number of scenes in the videos. 
3. The paper proposes 16 different question types on the video, which are more comprehensive compared to existing benchmarks.
summary: This paper proposed a synthetic dataset LLaVA-Video-178K, which consists of 178510 videos with detailed annotations, open-ended questions and multiple-choice questions. To build the dataset, the authors select the most dynamic videos from 10 major video data sources, and use a recurrent caption generation pipeline to generate video captions. The authors define 16 question types and generate question-answer pairs using GPT-4o. Based on LLaVA-Video-178K, the authors fine-tuned LLaVA-OneVision on the combination of LLaVA-Video-178K and other four public datasets to obtain the model called LLaVA-Video. Experiments show that the model trained with LLaVA-Video-178K will have a performance gain on a wide range of video benchmarks.
weaknesses: 1. The level-1 and level-2 description are generated on fixed time intervals (10s and 30s), which are not event-specific or scene-specific. 
2. Most of the videos in the dataset are 180 seconds long, which may hinder the dataset's effectiveness in the field of long video understanding. 
3. Several typos and errors exist in the paper. It seems that the paper is not carefully proofread and checked. For example:
line 184 & 186: condtion->condition
line 400: should be [M/(4*p^2)] for the fast frames
line 415: considegreen->considered as

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: - Could you elaborate on how your approach to using GPT-4o for generating video captions and QA pairs differs from prior work, such as ShareGPT4Video?
- Additionally, would you consider exploring or providing insights on how non-GPT-based annotations might influence the model’s performance or diversity in understanding?
- Your hierarchical captioning approach is similar to what has been previously proposed, such as in Video ReCap: Recursive Captioning of Hour-Long Videos (CVPR 2024). How does your method differ conceptually or practically from this prior work? Please clarify if I may have overlooked novel aspects in your captioning pipeline.
- EgoSchema contains the videos from Ego4D, so the training set or even test set (not sure) can be observed in your training data mixture.
soundness: 3
strengths: - The recurrent, multi-level annotation strategy for generating detailed captions and question-answer pairs is effective.
- The authors have conducted extensive experiments across multiple video benchmarks, demonstrating the effectiveness of the proposed LLaVA-Video models. The use of diverse datasets and thorough ablation studies supports the robustness of the results.
- The paper is well-structured, clearly explaining the problem, methodology, and experimental design.
- The significance of this work lies in its potential to advance video-language understanding models by providing a high-quality, open-source synthetic dataset. LLaVA-Video-178K has broad applicability in tasks such as video captioning and question-answering.
summary: This paper addresses the challenge of developing large multimodal models (LMMs) for video understanding, which has been limited by the scarcity of high-quality training data. To overcome this, the authors introduce LLaVA-Video-178K, a synthetic dataset designed for video instruction-following tasks such as detailed captioning, open-ended question-answering (QA), and multiple-choice QA. By training on this dataset alongside existing visual instruction tuning data, they develop LLaVA-Video, a new video LMM. Experiments demonstrate that LLaVA-Video performs strongly across various video benchmarks, underscoring the effectiveness of the synthetic dataset.
weaknesses: - The primary concern lies in the heavy reliance on GPT-4o for video captioning and question-answer (QA) pair generation. This approach essentially distills GPT-4o’s video capabilities into a structured format, raising questions about the originality of the contribution. Moreover, This has been done in previous work: ShareGPT4Video: improving Video Understanding and Generation with Better Captions, NeurIPS 2024 D&B Track. This work is an incremental improvement over prior work by scaling the data from 40K QA pairs to about 1.3M QA pairs. 
- The hierarchical captioning strategy employed in the paper is not new. The concept of recursive or hierarchical video captioning has been previously explored, for instance, in Video ReCap: Recursive Captioning of Hour-Long Videos (CVPR 2024). The failure to cite and differentiate from this work undermines the perceived novelty.
- The inclusion of multi-choice QA pairs appears to be tailored primarily for fitting into existing evaluation benchmarks rather than reflecting practical, real-world video understanding scenarios. This raises concerns about the broader utility of these QA pairs. 
- The model’s strong performance is largely due to fine-tuning from a powerful base model, LLaVA-OneVision. While the experimental results are compelling, the paper’s core contributions are somewhat overshadowed by the reliance on this pre-trained foundation.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: It appears there may be an error in the formula. line 400 & 871: ( T - t / s) * (M / 4p^2)
soundness: 3
strengths: 1. The proposed dataset has potential benefits for the video understanding community. The authors demonstrate through finetuning experiments that it complements existing datasets effectively.
2. The paper is well written and organized. The authors provide high-quality figures that illustrate various attributes of the proposed dataset from multiple perspectives.
summary: This paper introduces LLaVA-Video-178K, a video instruction-following dataset automatically annotated using GPT-4o. This dataset features dynamic untrimmed videos with dense frame sampling for annotation. The authors also present the LLaVA-Video model, which incorporates an optimized slow-fast video representation for multi-modal video understanding. They finetune LLaVA-Video on this new dataset, demonstrating that it complements existing datasets to improve video understanding performance further.
weaknesses: 1. Lack of Technical Novelty and Depth: The dataset lacks a clear motivation, making it unclear why it improves performance. While prior datasets have already demonstrated the feasibility of synthetic data, this dataset appears to simply expand upon them without significant innovation. Although the authors claim that dynamic scenes improve performance, there is no direct experimental evidence supporting this claim. There is no statistical comparison of dynamic scenes in the new and existing datasets, nor is there an ablation study on this claim.
2. Limited Insights in Video Representation: The proposed video representation method offers limited insights and seems more like an engineering trick than a conceptual advancement.
3. In Summary: While the dataset provides practical value, the scientific insights are relatively limited. I suggest the authors conduct further analyses to highlight the unique contributions of this dataset.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 1
questions: **Dataset Pipeline**:
- While this three-level approach is commendable, the description at Level-1 for time \(t\) heavily relies on the previous time \(t-1\), without leveraging longer-term dependencies, and the same applies to Level-2 and Level-3. This not only prevents it from capturing sufficiently long-range temporal information across the video but also makes the aggregated information less meaningful.
- The videos are all shorter than 3 minutes, and most scenes and content are singular without much dynamic transition information. Where is the significance of hierarchization, and what help can it provide for data quality?
- What is the motivation behind designing this architecture, and what evidence is there to prove that it generates higher-quality data? What is the essential difference between these QA datasets and previous ones, and can they provide more additional information and annotations?
- Are the 16 forced categories appropriate, what are the specific implementation details? If it was a manual review and summary, how many were reviewed, and are they representative? Or were they derived from the existing 40+ datasets' annotations for GPT-4o to summarize? Additionally, if QAs are generated for each video across the 16 categories, how are the unsuitable or low-quality QAs filtered out?

**Model Architecture**:
- The parameter \( s \) plays a crucial role in understanding the video; arbitrarily setting it obviously has no effect. How can events be dynamically recognized and videos segmented, employing differentiated sampling schemes at appropriate times? Are there potentially more suitable design schemes, along with experimental evidence and analysis?

**Ablation Study**:
- Why don't the experimental results in Table 3 present the individual results of LLaVA-Hound combined with the other three datasets, as well as the results of LLaVA-OV alone, and finally include the results of LLaVA-Video-178K? This makes it difficult to demonstrate that the significant improvement after adding LLaVA-Video-178K is not due to the disadvantages of LLaVA-Hound. Additionally, it cannot verify whether adding LLaVA-Video-178K after incorporating all datasets can still bring additional knowledge that is not present in the existing datasets to enhance performance sufficiently.
soundness: 2
strengths: - **Originality:** Introduces LLaVA-Video-178K, a synthetic dataset for video instruction tuning. And LLaVA-Video, a new video LMM leveraging the synthetic dataset.

- **Quality:** Offers a comprehensive dataset with detailed captions and diverse QA tasks, enhancing model training and evaluation, and develops a video representation technique that optimizes frame and token usage, improving model performance.

- **Clarity:** Articulates the significance of the dataset and model advancements in an understandable manner.

- **Significance:** Contributes to advancing video instruction tuning with synthetic data.
summary: The paper introduces LLaVA-Video-178K, a synthetic dataset for video instruction tuning, designed to address the challenge of curating large amounts of diverse and dynamic video data. The dataset contains 178,510 videos with detailed annotations, open-ended questions, and multiple-choice questions, created using a combination of GPT-4o and human efforts. The paper also presents LLaVA-Video, a new video large multimodal model (LMM) developed by training on LLaVA-Video-178K and existing visual instruction tuning data. LLaVA-Video outperforms previous models on various video benchmarks, demonstrating the effectiveness of the dataset. The authors plan to release the dataset, its generation pipeline, and model checkpoints to support the development of general-purpose visual assistants. The paper's contributions include the creation of a comprehensive video instruction-tuning dataset, the development of an advanced video LMM, and the commitment to open-source the resources to the public.
weaknesses: - **Dataset Pipeline**: The description of the pipeline is unclear, both in terms of intuitive understanding from the diagrams and the verbal explanation. Moreover, it appears that there is not much difference from previous methods of automatically constructing datasets.

- **Model Architecture**: The design approach of the model seems to have poor generalizability and robustness, and it cannot adaptively handle videos of diverse scenarios and lengths.