LLaVA-Video: Video Instruction Tuning With Synthetic Data
Authors: Yuanhan Zhang
Jinming Wu
Wei Li
Bo Li
Zejun MA
Ziwei Liu
Chunyuan Li
Venue: Accepted by TMLR ()

Abstract: The development of video large multimodal models (LMMs) has been hindered by the difficulty of curating large amounts of high-quality raw data from the web. To address this, we consider an alternative approach, creating a high-quality synthetic dataset specifically for video instruction-following, namely LLaVA-Video-178K. This dataset includes key tasks such as detailed captioning, open-ended question-answering (QA), and multiple-choice QA. By training on this proposed dataset, in combination with existing visual instruction tuning data, we introduce LLaVA-Video, a new video LMM. Our experiments demonstrate that LLaVA-Video achieves strong performance across various video benchmarks, highlighting the effectiveness of our dataset. We plan to release the dataset, its generation pipeline, and the model checkpoints.

Published in Transactions on Machine Learning Research (06/2025)
LLaVA-Video: Video Instruction Tuning With Synthetic Data
Yuanhan Zhang
yaanhan002@e..nuu.c edw......uuuuu.u.......
S-Lab, Nanyang Technological University
Jinming Wu
wjm ww@.upt.edu.ccommmm edu..c
BUPT
Wei Li ByteDance
lieel Sierisspecch@@bglcdaeeecccm
Bo Li
Stbo0013@e..tu.eduu..c
S-Lab, Nanyang Technological University
Zejun Ma Dance
mazejuun-bbglcdaanc..com com
Ziwei Liu S-Lob, Nangang Technological University
ziwei. liueiiiutrr.t...mm edu..su.ed....
Chunyuan Li ByneDance
lichumgann24 @grmail. com
Reviewed on OpenReview: https: openreview. net/ formm3 id= EEUFUu
Abstract
The development of video large multimodal models (LMMs) has been hindered by the difficulty of curating large amounts of high-quality raw data from the web. To address this, consider alternative approach, creating high-quality synthetic dataset specifically for video instruction-follwwin namely LLaVA-Video-178K This dataset includes key tasks such as detailed captioning, open-ended question-answering (QA), and multiple-choice QA. By training on this proposed dataset, in combination with existing visual instruction tuning data, we introduce VAAVA-Video, a new video LMM. Our experiments demonstrate that LaVA-Vido achieves strong performance across various video benchmarks, highlighting the effectiveness of our dataset. We plan to release the dataset, its generation pipeline, and the model checkpoints.
Introduction
We are an era where large-scale computing and data crucial for multimodal learning (Li et al., 2024d). A significant recent advancement. was introduced by visual instruction tuning (Liu et al., 2024a), which laid the foundation for building genoral-purposo visual assistant. Notably, it proposed data generation pipcline to create high-quality imaggelanguage instruction-folloiing data. This pipeline has inspired subsequent researches (Li et al., 2024c;b;a; Lin et al., 2024) aimed at generating diverse image-language instruction data across various visual domains, accelerating the development of visual instruction tuning techniques.
Compared to the construction of image-language mnstruetton-followiin data, obtaining high-quality video- language instruction-followin data is challenging (Zhang et al., 2023; Li et al., 2024e). First, sourcing high-quality vidcos is difficult. We need find videos with significant temporal changes that provide more knowledge than what image-language data can offer. However, we have found that most videos current. video-language instruction--Colloiing datasets (Chen et al., 2024a; Zhang al., 2024d) are relatively

---

Published in Transactions on Machine Learning Research (06/2025)
static. Additionally, these videos are mostly trimmed based scene changes, resulting in simplified plots. Such simplified video-language instruction-tuning data is inadequate for models to understand videos with complex narratives. Furthermore, current video-language instruction-folloving datasets often use a very sparse sampling rate for frame annotation. For instance, ShareGPT4Vidoo (Chen et al., 2024a) has an average sampling rale of 0.15, sometimes sampling only 2 frames from a 30-second video. This spanse sampling rate is effective in describing overall scenes but fails to capture detailed movements or changes in the video, resulting in hallucination when detailed descriptions of the video are required.
To overcome these shortcomings, named LLVAA-Vie--1788K, consisting of 178,510 videos ranging from 0 to 3 minutes. This dataset is enriched with detailed annotations, open-ended questions, and multiple-choice questions, developed through a combination of GPT-4o (OpenAI, 2024) and human efforts. It features four favorable properties: (i) Extensive Video Source: We conduct comprehensive survey on the video sources of exsiting video understanding datasets, and conclude 10 major video data sources, from which we start our video data collection by building video pool. Although there are over 40 video-language datasels, their video data are mainly sourced from 10 datasets (Zhou & Corso, 2017; Xue et al., 2022; Goyal et al., 2017; Caba Heilbron et al., 2015; Kay et al., 2017; Sigurdsson et al., 2016; Wang et al., 2023; Shang et al., 2019; Grauman et al., 2022; Zhu et al., 2023a), covering a wide range of video domains, such as activities, cooking, TV shows, and egocentric views. (ti) Dynamic Untrimmed Video Selection: From these sources, we use several filtering logic to select the most dynamic videos from the video data pool. Notably, we select original, untrimmed videos ensure plot completeness. (ii) Recurrent Detailed Caption Generation Pipcline with Dense Frame Sampling: We propose detailed video caption pipeline that operates recurrently, enabling to generate detailed captions for vidcos of any length. This pipeline has three levels, each level of description represents different time-range: from 10 seconds to the entire video length. It is recurrent as the historical description from any level serves as the context for generating new descriptions at any level. Additionally, we adopted a dense sampling strategy of one frame per second to ensure the sampled frames are rich enough to represent the videos. (iv) Diverse Tasks: Based on the detailed video descriptions, we can generate question-answer pairs. To ensure our questions cover a wide range of scenarios, by referring to the video dataset, we define 16 question types. We prompt GPT-40 to generate question-answer pairs by referring to these question types, covering open-ended and multi-choice questions.
introduce comprehensive video instruction-tuning dataset
Based upon the 178k dataset, we developed LLAVA-Video. Contrary to previous studies suggesting that training with single frames is sufficient for video-language understanding (Lei et al., 2022), our findings reveal a significant impact of frame count on LLaVA-Video's performance, attributable to the detailed features of LLa constraints of limited GPU memory. We introduce LLaVA-Video SlowPast, a video representation technique that optimally distributes visual tokens across different frames. This approach allows for incorporating up to three times more frames than traditional methods, which allocate an equal number of visual tokens to each frame.
Observing this, we explored maximizing frame sampling within the
Our contributions are as follows:
Following Data: We present a high-quality dataset VA- VA- Video-178 tailored for vidco mnstruction-folloiingg It consists of 178K vidoo with 1.3M instruction samples, including detailed captions, free-form and multiple-choice question answering.
Video-language
Video Large Multimodal Models: We develop LLa VA- Video, a series of advanced large video-language models that expand the capabilities of open models in understanding video content.
Open-Source: In effort to support the development of goneral-purposo visual assistants, release our multimodal instruction data, codebase, model checkpoints, and visual chat demo to the public.
2 Related Work
In this work, our goal is to create a high-quality vidco-language dataset that goes beyond simple vidco captions. We aim to improve the ability to follow instructions, which includes detailed video descriptions, open-ended video question-answering and multiple-choice video question-answering data. We discuss related datasets in
2

---

Published in Transactions on Machine Learning Research (06/2025)
Filtering Logic
Source
PandapoM
HD-VILA-100M
HD-VILA-100M
Kinetics-74 700
Ego4D
VidOR
DiDeMo
InterVVid
1
ShotaStory
VidLM
Vates
EgROA
VidSTR
ActivityNet
5
ShareG TTVideo
ViiueLLM
VIDAL
VIDAL
0
TextVR
ActivisNettAA
VideoXum
VideoChet
ActivityNet
LLaVA Hound
Charades
VideoChatt
InterViid
SharaGPT&O
TextVR
LLuud-Hound
ActhvityNet Caption
Sth-sthrz
Ego4D
5
VidOR
YouCook2
Propang
Hour
Sth-sthva
STAR
YouCooka
Charades
Kinetics-7oo
Charades STA
AGDA
Sth-sthvz
Figure 1: Video sources in the proposed LLa VA- Video-178K. (Left) The relationship between 10 video sources we have utilized and other existing video-language datasets. (Right) Filtering logic for video sources. The detail of filtering logic: D Sorted by Views, @ Number of scenes greater than 2, @ Video duration between 5 seconds and 180 seconds, @ Ratio of scenes to video duration less than or equaa to 0.5, $ Resolution greater than 480p, @ 50 samples for each category.
Table 1. Previous video-language datasets (Miech el al., 2019) include manually annotated data for various tasks, such as video captions (Chen & Dolan, 2011; Xu et al., 2016; Rohrbach et al., 2015; Anne Hendricks et al., 2017a; Caba Heilbron et al,, 2015; Zhou & Corso, 2017), and video question-answering (Yu al., 2019; Zadeh et al., 2019; Xiao et al., 2021). However, manual annotation is expensive and limits the size of such datasets. To address the shortage of data, studies like (Miech et al., 2019; Tee et al., 2021; Zellers et al., 2021; Xue et al., 2022) suggest automatically annotating data using subtitles created by ASR. While this method greatly expands the dataset size to 100 million samples, the subtitles often fail to accurately describe the main video content. Additionally, other studies (Xu et al., 2017; Grunde-McLautt 2024a) use language models (Xu al., 2017) or question templates (Grunde-MeLane McLaughlin et al., 2021; Wu et al., 2024a) generate question-answer pairs. Although this approach can generate a large number of questions and answers, it often produces poor-quality questions that do not reflect real-world user inquiries. More recent research (Chen et al., 2024b) has prompted video-language models such as BLIP-2 (Li et al., 2023), VideoChat (Li et al., 2024e), Video-LLaMA (Zhang et al., 2023), and MiniGPT-4 (Zhu et al., 2023b) to generate video captions. However, these models are limited in their ability to provide detailed descriptions.
al., 2021; Wu et al.,
The most related works to ours are the recent Al-generated synthetic video instruction tuning data, Islam et al. (2024) introduced Video ReCap, which recursively annotates video captions. Unlike Video ReCap, each clip-wise (level-1) description in our pipeline is generated with historical context. This ensures that connections from previous events in the video timeline are linked to the current event. ELaVA-Hound (Zhang et al., 2024d) and SharcGPT4Video (Chen et al., 2024a), where they have used GPT-4 (OpenAI, 2023) to generale video captions and open-ended video quussiin--ns..riing Although the quality of the captions and questtionannwer pairs has significantly improved, the video sources they use are too static to produce high-quality data for instruction-followe GPT-4V, which results in annotations that fail to capture nuanced actions and continuous plots in the videos. Additionally, Shot2Story (Han et al., 2023) and Vript (Han al., 2023) also employ GPT-4V (OpenAT, 2023) for video captioning. Their outputs, however, include audio details, which outside the scope of this study.
scenarios. They also only use very sparse frames for prompting
3 Video Instruction-Follovingg Data Synthesis
A high-quality dataset for video instruction-tuning is crucial for developing effective video-language models. We identify key factor in building such datasets: ensuring richness and diversity in both video content and
3

---

Published in Transactions on Machine Learning Research (06/2025)
its language annotations. We perform comprehensive survey the existing video benchmarks, covering across various public video captioning and question-answering datasets, then identify ten unique video sources that contribute to over 40 video-language benchmarks. From each source, we select videos that exhibit significant temporal dynamics. To maintain diversity in the annotations, we establish a pipeline capable of generating detailed captions for videos of any length. Additionally, we define 16 types of questions that guide GPT-4o creating question-answer pairs assess the perceptual and reasoning skills of the vidco-language models.
3.1 Video source
One important starting point in building a high-quality video instruction-followingg dataset is to find a sufficiently diverse pool of video data, From this pool, we can select the qualified videos. In our study of public video-language datasets includdinn video captioning, video question answering, vidco summarization, and moment-wise captioning we noticed that although different datasets focus on various video understanding tasks (e.g., AGQA (Grande-Me.aughlin 2024a) for situational reasoning), most are sourced from ten main video sources. For instance, both AGQA and STAR use data from Charades (Sigurdsson et al., 2016). Specifically, these ten sources are HD- HIDVILL-110M (Xue et al., 2022), InternVid-10M (Wang et al., 2023), VidOR (Shang et al., 2019), VIDAL (YouTube Shorts) et al., 2023a), YouCook2(Zhou & Corso, 2017), Charades (Sigurdsson et al., 2016), ActivityNet (Caba Heilbron et al., 2015), Kinetics- 700 (Kay et al., 2017), Srunethin-SSmmethine et al., 2017), and Ego4d (Grauman al., 2022). These sources offer a wide range of video data from different websites, viewpoints, and domains. The relationship between these ten selected video datasets and others is shown in Fig. 1. The videos from this ten datsets build the video pool for the further video selection. Notably, use untrimmed videos from each source except for YouCook2 and Kinetics-7000 We believe that cutting videos into clips can break the plot continuity, which is essential for understanding the videos.
al., 2021) for spatial-temporal relations and STAR (Wu et al.,
v2 (Goyal
Based on the video pool, we aim to select dynamic videos. In Figure 1, we outline our criteria for selecting high-quality data. Our main method for identifying dynamic content involves using PySceneDetect,, which calculates the number of scenes in video We found that the number of scenes good indicator of video dynamism. Additionally, we have designed specific approach + to exclude vidcos that mainly contain "slides."
3.2 Video Detail Description
Automated Generation For selected videos, we use GPT-4o (OpenAI, 2024) to systematically describe their content. We start by sampling vidco frames at one frame per second (fps). However, due the input size constraints of GPT-4o, we cannot use all sampled frames. Instead, we describe the videos sequentially, shown in Fig 2. We create descriptions at three distinct levels, detailed below.
Level-1 Description: Every 10 seconds, we provide a level-1 description that outlines the events in that segment. This description considers: frames from the current clip and historical context, which includes all recent level-1 descriptions not yet summarized into a level-2 description and the latest level-2 description.
Level-2 Description: Every 30 seconds, creat level-2 summary of the entire video plot up to that point. This is based on the last three level-1 descriptions, covering the most recent 30 seconds; and the latest. level-2 description.
Level-3 Description: At the video's end, we generate level-3 description to encapsulate the entire video. The inputs for this description are the recent level-1 descriptions not yet summarized, covering the last moments of the plot after the recent summary; and the latest level-2 description.
3.3 Video Question Answering
Question Type definition addition detailed video descriptions, our datasct includes a varioty of question-answer pairs designed for complex interactions. This setup improves the video understanding model's ability to handle real-life queries. We refer to public video question-answering benchmarks (Xiao et al., 2021;
4

---

Published in Transactions on Machine Learning Research (06/2025)
Video
Lisi o
X
Time
Os
5s
10s
15s
20s
25s
30s
35s
40s
45s
50s
55s
60s
Description
Level-1
Level-2
Level-3
Video clip
LMM
LLM
LLM
Time
interval
t-1
T-1
(a) Level-1 Description
(b) Level-2 Description
(c) Level-3 Description
Figure 2: The video detail description creation pipeline. A three-level creation pipeline is considered, with each level developed via recurrent approach. Note that t is the index of time internal at its own level, and 7' is the last time internal index. (a) To generate the caption for time internal t at level-1, we condition on the current frames in this internal, the caption for time internal / 1, and the most recent description summary at level-2 if applicable. (b) To generate caption for time internal at level-2, we condition on the previous caption at level-2, and captions from three most recent time internals at level-1. (c) To generate the overall caption at the last time internal T level-3, condition on the the most recent caption at level-2 and the current caption from level-1.
Temporal
pinata correctly?
Spatial
O-What is buhind man?
Causal
towards woman green
Speed
the bicycle?
faster,
Binary
the wear shoes while running on beach?
Count
G. How many times did the man put his right hand into his podket?
Plat
interaction between the monkey and the cat indicace?
Description Object
what colors are railings of stalicass?
Time
G-Whha actions did the person in the red hcodle
Fine-grain
Or Does undergo physical
video
Object
Go What the reaction the sudience when the keynete
Description
W-Waaa does the person on the right's expression sugget?
Order
in what ordar??
Action
transfommatios?
speaker delivers his speech??
Human
Attribute Change
G: How the cream
Camera Direction
O. the camera following the
Object
G. Which direction did man welk towerds befoore esiting
Description
Where did resoue openation the video tabe place?
change?
move>
Direction
scene relative to the camera?
Scene
Figure 3: Question types for video question answering in data creation. For each type, we provide its name and an example question.
Yu et al., 2019; khattak al., 2024; Liu al., 2024b) to organize these questions into 16 specific categories, shown in Fig. 3.
GPT-4o to generate at most question-answer pair for each type of question. The prompts include: (1) The task definition for the current. question type. (2) In-context examples for this type, which include three video descriptions and their three question-answer pairs this specific type. (3) The detailed video description for the current vidco. We instruct GPT-4o to return None cannot. generate question-answer pairs for a specific question type.
Automated Generation Given detailed video description,
Filtering. To filter out the generated question-answer pairs, we apply the following strategy: (1) remove duplicates using the sentence-transformee (Reimers & Gurevych, 2020), (2) discard answers that begin with
5

---

Published in Transactions on Machine Learning Research (06/2025)
Homell People
x
Os
5s 10s
15s
20s
25s
30s
35s
40s
45s
50s
55s
60s
Annotation type a: detailed description
Annotation type 2: open-ended question
video begins a black screen displaying the 'Normal People Ultra' in pink white letters, accompanied by two smiling face emojis. The scene transitions a modern building with a staircase. Three individuals, dressed in black stand
Question: many steps does "normal people"
climb?
Answer: "Normal people" climb steps in
white sneakers,
Annotation type 3: multi-choice question
'Normal' appears the top left corner. The individuals walking up the stairs synchronized manner... <omited>
the stairs.
box
Question: How many steps does "normal people" climb? A. 5 B. 6 C. D.8 Answer: C.7
Figure 4: One example to illustrate the video instruction-folllwin data.
Caption
Open-ended QA
Multi-choice QA
VIDAL
VIDAL
VIDAL
VidOR
VidOR
VidCO
YouCooks
YouCook?
YauCCola
Charades
Charades
Charades
Activityhie!
Kinetics-700 700
AnternVid 20M
HD- AD-VILA-100M
ActhvtlyNeet
Interniial interVii-zaa MieeniidzoMM
HD-VILA-100M
Actuity Actuity Net
Pawiiad Wor-
HD-VILA-100M
Sglatt
chil
Dataset
# Caption #Open- Ended #Multi-Choice Dataset
# Caption # #Open-Ende Ended #Multi-Choice
VidOR
4,018
19,875
4,773
Sthsth2
8,700
YouCook2
7,411
32,143
5,776
Ego4D
1,065
5,912
520
Charades
9,803
48, 187187
13,401
InternVid-10m
45,000
245,840
48,246
ActivityNet
7,953
44,100
12,771
HD-VILA-1008
48,260
263,652
51,743
Kinetics-700
34,998
0
VIDAL
55,000
300,472
58,968
Figure 5: Distribution of data across different datasets and question types (Caption, Open-ended, and Multi-Choice).
phrases like "does not specify," "does not mention," " "does not specifically," "does not depict," or "does not show."
3.4 Dataset Statistics
Overview. We carefully select from our collected data sources to form a balanced and comprehensive collection, resulting in total 178K videos and 1.3M instrrecionnllllowiingg samples. This includes 178K captions, 960K open-ended QAs, and 196K multiple-choice QAs
Dataset Comparison We provide a comparison of high-quality instruction following video-language datascts, with a focus on synthetic data created with strong AT models, as shown Table 1. (1) A broad collection of dynamic videos. terms of video sources, although LLAVA-Hound (Zhang et al., 2024d) contains the largest number of videos, 44% of its video data sourced from WebVid (Bain et al., 2021), where most.

---

Published in Transactions on Machine Learning Research (06/2025)
Histogram of video Duration
Histogram Video Caption Length
Video Duration vs. Video Caption Length
25K
Number of Videosm600
1.7K
# video Count
20K
video Count
8K
Wordd Count
1.5K
15K
6K
1.0K
4K
0.8K
0.5K
5K
2K
0.3K
0
0
0
50
100
150
180
0
500
1000
1500
0
20
40
60
80
100 120 140 160
Video Duration (s)
Word Count
Video Duration (s)
Figure 6: (Left) Visualization of the video duration. (Middle) Visualization of the number of words in the video caption. (Right) Visualization of caption length versus video duration.
Distribution of Uniformly Sampled Video Categories
Beauty & Style
Compupter Games
261
Travel
S23
Beauty & style
Camedy
Sports
4500
Comedy
15633
20000
20000
Figure 7: (Left) Display of YouTube Shorts across four video categories. (Right) Distribution of 5 uniformly chosen video categories.
videos are static. ShareGPT4Vides (Chen et al., 2024a) includes 30% of its videos from Pexels, Pixabay, and Mixkit, which are aesthetically good but also mostly static. Additionally, the majority of its videos come from Panda-70M, which are short clips from longer rideos-suggeste carefully select video sources that offer dynamic, untrimmed videos with complex plots, which are crucial for developing a powerful video understanding model. (ii) High frumes per second. Regarding frame sampling in language annotations, the proposed datasest considers 1 FPS, while other dalasets consider much lower FPS. LLAVA-Hound uniformly samples 10 frames from videos of any length. The average FPS is 0.008, which may miss some fine details. ShareGPT4Vides picks key frames using CLIP (Radford et al., 2021) based frame uniqueness. This method might also miss subtle changes in the video because CLIP embeddings do not capture fine-graiee sre-grained dynamics well. Our method samples FPS=1 without using key frame selection algorithms, ensuring the detailed temproal information can be expressed in annotations and high coverage. (iit) Diverse tasks. The proposed datasct considers three common task types, including caption, free-form and closed-form QA, while existing datasets only consider a subset. Meanwhile, the quality and numbers of samples in our dataset is higher.
simpler plots. In contrast, we
4 Experiments
We conducted ovaluations for the LLaVA-Video models across all benchmarks using LMMs-Eval (Zhang et al., 2024a) to ensure standardization and reproducibility. To fairly compare with other leading video LMMs, we primarily used results from original papers. When results were not available, we integrated the models into LMMs-Eval and assessed them under consistent settings. Following LLaVA-OneVision (Li et al., 2024c), we employed SigLIP (Zhai et al., 2023) as our vision encoder, and Qwen2 (Yang et al., 2024) as the LLM. The
Example videos: WebVid, Webaid,,ixabay, Pixabay, Pexels, Mixkit..

---

Published in Transactions on Machine Learning Research (06/2025)
Table 1: Comparison of LLaVA- and other video-language datasets. Average FPS represents the average number of frames per second that are used to prompt GPT-40/GPT-4 for annotation. VIDAL, WebVid, ActivityNet. Panda-70M, Pexels, Pixabay, Mixkit, BDD100K, Ego4d. @ HD-VILA- 100M, Kinctics-700M,, Ego4D, VidOR, InternVid, YouCook2, ActivityNet, Sth-sthv2, VIDAL, Charades.
Video Source
Total Video Average FPS
#OE
#MC
Text
Video
#Caption
Length
QA
QA
LLa ShTVVGPT
GPT-4V
*
900K
3Khr
0,008
900K
900K
0
GPT-4V
10K
0.2Khr
0.15
40K
0
GPT-4o
178K
2Khr
1
178K
960K
196K
LLa VA-Vioid model builds on the single-image (ST) stage checkpoint from the L.LVA-OneVision model (Li et al., 2024c), which was trained using only image data.
Vidco Representations Following the classic SlowFast idea in video representations (Feichtenhofer et al,, 2019; Xu et al., 2024b; Huang et al., 2024), we develop LLa VA-Video stawPast to optimize the balance between the number of frames and the count of visual tokens, within the budget of the limited context window in LLM and GPU memory for video representation. Please refer to Appendix for detailed information. Specifically, we represent each video as sequence with maximum T frames. Each frame is represented in M tokens. we categorize the frames into two groups, based on the a strike rate 8, where the every frames are uniformly selected to form the slow frame group, and the rest of the frames are consdiered as the fast frame group. Note that a special case s= 1 lcads to only one group, reducing the SlowFast representation to the original simple representation. For each group, we apply different pooling rate using Pytorch function pooling avg pooo_oo))(( p x p pooling and 2p x 2p pooling for slow and fast frames, respectively. To summarize, we paramterize the video representation configuration as V (T, The total number of tokens is #tokens = [T/s] M/p²] + -T- (T/s))s)) x M/4p"]
Evaluation Benchmarks. For full evaluation, we consdier 11 video benchmarks. conducted Lests across various video captioning, , video open-ended question-answering and video multiple-choice question-answering benchmarks, including ActivityNet-QA (Yu et al., 2019), which features human-annotated action-related QA pairs from the ActivityNet dataset. We also utilized LongVideoBench (Wu et al., 2024b), EgoSchema (Man- galam et al., 2024), and MLVU (Zhou et al., 2024) for long video understanding, Perception Test (Pätráncean et al., 2023) for assessing fine-grained perception skills, and VideoMME (Fu et al., 2024) and NExT-QA (Xiao el al., 2021) for diverse video domains and durations. Additional tests included VideoDetailCaption (LMMs- Lab, 2024), Dream-1K (Wang et al., 2024), Video- Chat-CC (Maaz et al., 2024) for detailed video descriptions, TemporalBench Cai ct al. (2024) for fine-grained temporal understanding.
For ablation studies 4.2 and Sec. 4.3, we conduct evaluation across datasets. NExT-QA (Xiao et al., 2021) and Perception'Test (Päträucean et al., 2023), which use training data from the LLa V-VV---o---78. are treated as in-domain datascts. Conversely, VidcoMME (Fu et al., 2024) and EgoSchema (Mangalam et al., 2024) are consider as zero-shot datasets.
4.1 Overall Results
We fine-tune LLa VA-OneVision (SI) on the joint dataset of video and image data. Specifically, we added video data from the LLa -ide dataset and four public datasets: ActivityNet-QA (Yu et al., 2019), NExT-QA (Xiao et al., 2021), PerceptionTesst (Pätráucean et al., 2023), and LLaVAA-ooudd2555/ (Zhang et al., 2024d), focusing on videos shorter than three minutes. These datascts were selected to improve our model's performance, contributing to a total of 1.6 million video-language samples, which include 193,510 video descriptions, 1,241,412 open-ended questions, and 215,625 multiple-choice questions. Remarkably, 92.2% of the video descriptions, 77.4% of the open-ended questions, and 90.9% of the multiple-choice questions were model
newly annotated. Additionally, we used 1.1 million image-language pairs from the (Ti et al., 2024c). We consider the same video representation configurations for the training and inference
8

---

Published in Transactions on Machine Learning Research (06/2025)
Table 2: ViieeoVideo performance on video benchmarks. We report the score out of 5 for VideoDC, VidcoChatGPT while other results are reported in accuracy. All results are reported as O-shot accuracy. "indicates that the training set has been observed in our data mixture.
Caption Open-Ended Q&A
Multi-Choice Q&A
Long VongVideeBench
VideoChatGPI
Parcoption
TomporalBentt
VideoDC
Dream-1K
ActNot-QA
EgoSchema
MVBench
NEXT-QA
VideoMME
MLVU
Model
test test
test
test
test m-avg test mc
val
val m-ace wo/w-subs
Proprictary models
CPT-4o (OpenAI, 2024)
39.2
64.6
66.7
35.3
71.9/77.2
Gemini-1.5-PP (Team et al., 2023)
36.2
57.5
72.2
64.0
25.6
75.0/81.3
Open--soure models
VILA-40B (Lin et al., 2024)
3.37
33.2
58.0
3.36
58.0
67.9
54.0
60.1 6011/
PLIaVA-3dB (Xu et al., 2024a)
28.2
60.9
3.48
58.1
53.2
LongVA-7B (Zhang et al., 2024c)
3.14
50.0
3.20
56.3
68.3
52.6/54.3
IXC-2.5-7B (Zhang et al., 2024b)
52.8
3.46
37.3
69.1
71.0
34.4
16.7
55.8/58.8
LLaVA-OV-7B (Li et al., 2024c)
3.75 31.7
56.6
3,51
60.1
64.7
56.7
79.4*
57.1
56.5
18.7
58.2/61.5
Vide2TLaMA2-72B (Cheng et al., 2024)
-
27.1
55.2
3.16
63.9
61.2
62.0
61.4/63.1
LLaVA-OV-72B (Li al., 2024c)
3.60 33.2
62.3
3.62
62.0
68.0
59.4 80.2*
66.9
61.3
26.6
66.2/69.5
LLaVA-Video-7B LLVVA-Vide--22B
3.66 32.5
56.5*
3.52
57.3
70.8
58.6
67.9*
58.2
22.9
63.3/69.7
3.73 34.0
63.4*
3.62
65.6
74.4
64.1
85.4*
74.3* 61.9
33.7
70.5/76.9
stages. On 128 NVIDIA H100 GPUs, the video representations for LLaVA- and LLaVA-Video-77 are (64, 1,9, 1,2) and V (64, 679, 3, 2), respectively.
In Table 2, we compare the performance of different models on various video benchmarks. The 72B model performs as well as the commercial, closed-source model Gemini-1.5-Plash (Team et al., 2023), highlighting the effectiveness of open-source efforts in achieving comparable results. The LLaVA-Video-- model outperforms the previous top model, LLAVA-OV-7B, in seven out of ten datasets. Analysis of individual datascts shows some noleworthy trends. For instance, on benchmarks like MLVU, LongViae VideoBench, and VideoMME, which primarily use video data from YouTube, this improvement may be due to the inclusion of extensive YouTube data in LLaVA-Video-1788 Video-178K, as illustrated in Fig. Additionally, the improvement on ActivityNet-QU is small; this could be because many questions in ActivityNet- -A,,,, such as "What's the color of the ball?" be answered by viewing single frame. The visibility of the ball from the beginning to the end of the video means understanding the video sequence is unnecessary, LLa LLVV-Video--788 offers little advantage in this context. We find that LLaVA-Video-7B is notably weaker in the specialized task of EgoSchema, an ego-centric dataset. This weakness may be due to significant reduction in the proportion of cen-centric data the training datasct of LLAVA-Vidoo. However, this impact is less pronounced in larger models, as demonstrated by the LLaVA-Video-700 model's superior performance over LLaVA-OV-72B EgoSchema.
Video-7B
4.2 Dataset Ablation
Note that the training set for LLa includes six datasets: LLaVA- et al., 2024d), NExT-QA (Xiao et al., 2021), ActivityNet-QA (Yu et al., 2019), Perception Test (Pätráuccan et al., 2023), and image data from LLaVAs VaaV---eeVision (Li et al., 2024c). In this section, we conduct ablation studies to assess the impact of each dataset. We separately fine-tune the L.aVA-OneVisioo (ST) model for each experimental setting, progressively adding datasets to the baseline.
LLa VA-Hound (Zhang
The results are presented in Table 3. Initially, we used a basic model trained solely on the LLaVA- dataset as our baseline. Compared to this baseline, adding the L.aVA--Vieoo 178K dataset significantly
9

---

Published in Transactions on Machine Learning Research (06/2025)
Table Ablation study on the LLaVA-Video model with various configurations training data. Three Q&A
datasets indicate: NExT-QA, ActivityNet-QA and Perception
in-domain
out-of-domain
EgoSchema
NEXT-QA
Porcep
VideoMMa
Method
me
val
test
WO
LLaVA-Hound
64.4
51.4
51.0
54.1
80.1
57.1
56.5
63.2
Three Q&A datasets
80.1
69.0
55.6
61.9
LLaVA-OV (images)
83.2
67.9
57.3
63.4
Table 4: Comparison of LLa VA-Video-117 and other video instruction-following datasets.
in-domain out-of-domain
EgoSchema
VideoMME
NEXT-QA
Porcep
#Cap #OE #MC
val
test wo
LLa V.-
900K
900k
0
39.8 53.1
25.8 55.2
LLa V-18-K
178K
900k
0
73.2 55.9
49.8 59.6
ShareGPT4Video
40K
40K
19K
69.6 55.2
58.9 51.0
LLaVA LLaVA-V-178A 40K
40K
19K
75.8 55.4
55.8 53.5
improved performance, enhancing scores in both in-domain and out-of-domain tasks. Specifically, observed 31.9-point increase in NEx T-QA scores and 1.1-no rise in VideoMME scores. Furthermore, including the PerceptionTest dataset enhanced its associated task. Additionally, integrating high-quality image data provided modest benefits on EgoSchema.
4.3 Dataset Comparison
We conduct two ablation studies to analyze our datasel and training stralegy. In Table 4, we compared three datasets where the language annotations are from SOPT-4V/GPT-A For each experiment, we fine-tune the LLaVA-OneVision (SI) model separately each specific dataset setting.
Two group of experiments are considered to assess the data quality of T.LaVA-Video-720 compare to LL- Hound and ShareGPT4Viddeo. In the first group, to compare LLa Va-aaAA Video-178K with LLaVA-Hound, we randomly selected 900K open-ended questions to match the number in LLaVA-Hound We included all captions and did not sample the multiple-choice questions. In the second group, comparing Vi-Veooe 178K to ShareGPT4Video, we randomly sampled 40K video captions to align with those ShareGPT4Video. Since ShareGPT4Videe lacks open-ended and multiple-choice questions, we supplemented with annotations from NEXT-QA, PerceptionTest. and ActivityNet-QA.. In the first group of Table with LLa VA-HHond... Although LLaVA. LLLVA-Hour has more captions than LLaVA- -ideo--78K, our results are still better. The quality of VLaVA-Hounn is limited duc to two main issucs: (1) Static vidco: Its primary video source is WebVid (Bain et al., 2021), which tends to have relatively static content. (2) Sparse sampling: its sampling rate of 10 frames per video leads to annotations that do not fully capture the complete plot of the video. This underscores that the quality of vidco instruction-following data more important than its quantity. Additionally, the second experiment group in Table 4 shows that the model trained with LLaVA- VA-Vieee 178K outperforms that of ShareGPT4Vided highlighting the superiority of data's quality.
compare
10

---

Published in Transactions on Machine Learning Research (06/2025)
5 Conclusion
This study introduces the LLVVA-Vido--78K datasct, high-quality synthetic datasct for video-language instruction-folloiing It is favored for its dense frame sampling rate in longer, untrimmed videos, cover- ing diverse tasks such as captioning, open-ended and multi-choice QA. By training on the joint dataset. of LLaVA-Video-177 178K with existing visual instruction tuning data, we developed a new model family, PLaVA-Vidco which considers video representation effectively use GPU resources. This allows us include more frames in the training process. The experimental results have demonstrated the effectiveness of the proposed synthetic dataset, and LLaVA-Vides models have achieved excellent performance wide range of video benchmarks.
6 Limitations
The videos in LLaVA-Video-172 are sourced from various platforms. This diversity introduces potential biases inherent in these sources. Furthermore, there is a concern regarding the potential skew in the question-answer pairs, possibly influenced by the annotators' perspectives.
Acknowledgement
This study is supported by the Ministry of Education, Singapore, under its MOE AcRF Tier 2 (MOE- T2EP20221-0012, MOF--2EF22023-002) and under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (LAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).
11

---

Published in Transactions on Machine Learning Research (06/2025)
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Tain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina. Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning, 2022. URL https://arxix: arggaba//204/ 14198. 17
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in vidco with natural languago. In Proceedings of the IEEE international conference on computer vision, pp. 5803 5812, 2017a. 3
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josel Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In Proceedings of the IEEE international conference computer vision, pp. 5803 5812, 2017b. 21
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision, 2021. 6, 10, 19
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 961-970, 2015. 2, 3, 4, 21
Mu Cai, Reuben Tan, Jianrui Zhang, Bocheng Zou, Kai Zhang, Feng Yao, Fangrui Zhu, Jing Gu, Yiwu Zhong, Yuzhang Shang, Yao Dou, Jaden Park, Jianfeng Gao, Yong Jac Lee, and Jianwei Yang. Temporalbench: Towards fine- finegrrined temporal understanding for multimodal video models. arXiv preprint vrXiv:2410.10811 2024.
David Chen and William B Dolan. Collecting highly parallel data, for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguistics; human language technologies, pp. 190-200, 2011. 3, 19, 21
Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, Li Yuan, Yu Qiao, Dahua Lin, Feng Zhao, and Jiaqi Wang. Sharegpt4video: Improving video understanding and generation with better captions. arXiv preprint arxiv:2006. 04325, 2024a. 1, 2, 3, 7, 21
Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying 70m videos with multiple cross-modality teachers. arXiv preprinl Xiv:2402.19479, 2024b. 21
Ying Leo, Jian Ren, Ming- Hsuan Yang, and Sergey Tulyakov. Panda-70m: Captioning
Ming-Hsuan
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms, 2024. URL http:///xxvvvv.o/oaab/ 07t///222000776..... 9
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. faow networks for video recognition. In Proceedings of the IEEE/CVP international conference on computer vision, pp. 2202-6211, 2019. 8, 17
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal ilms in video analysis. arXiv preprint arXiv:2405.21077 2024. 8
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, al. The" something something' video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference computer vision, pp. 5842-58 5842-5850, 2017. 2,
12

---

Published in Transactions on Machine Learning Research (06/2025)
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Läu, et al. EgoAd: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18995-19012, 2022. 2, 4
Madcleine Grunde-MLLaughinn Ranjay Krishna, and Mancesh Agrawala. Agqa: A benchmark for composi- tional spalio-temporal reasoning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Paltern Recognition, pp. 11287-11297, 2021. 3,
Minglei Han, Linjie Yang, Xiaojun Chang, and Heng Wang. Shot2story20k: A new benchmark for compre- hensive understanding of multi-shot videos. arXiv preprint arXie:2311.17443, 2023. 3
De-An Huang, Shijia Liao, Subhashree Radhakrishhan, Hongxu Yin, Pavlo Molchanov, Zhiding Yu, and Jan Kautz. Lita: Language instructed temporal-loclization assistant. arXiv preprint (1f::400.119440 2024.
8, 17
Md Mohaiminul Islam, Ngan Ho, Xitong Yang, Tushar Nagarajan, Lorenzo Torresani, and Gedas Bertasius. Video recap: Recursive captioning of hour-long videos. arXiv preprint arXiv:2402.1980 13250, 2024. 3
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprinl ar Xiv: 170. 06950, 2017. 2, 4
Muhammad Uzair khattak, Muhammad Ferjad Naeem, Jameel Hassan, Naseer Muzzamal, Federcio Tombari, Fahad Shahbaz Khan, and Salman Khan. How good is my video Imm? complex video reasoning and robustness evaluation suite for video-Imms. arXiv:2405.00600 2024. 5
Sangho Lee, Jiwan Chung, Youngjae Yu, Gunhee Kim, Thomas Breuel, Gal Chechik, and Yale Song. Acav100m: Automatic curation of large-scale datasets for audio-visual video representation learning. In Proceedings of the CREE/CVE International Conference on Computer Vision, pp. 10274-10284, 2021. 3, 21
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. Tvqa: Localized, compositional video question answering. Xiviv preprint arXiv:1809.00 06666, 2018. 21
Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for vidleoaandllanguage learning via sparse sampling. In Proceedings of the IEEE/CVF conference computer vision and pottern recognition, pp. 7331-7344 2021. 19
Jie Lei, Tamara L Berg, and Mohit Bansal. Revealing single frame bias for jideo-and-language learning.
arXiv preprint arXiv:2206003. 034228 2022. 2, 19
Bo Li, Hao Zhang, Kaichen Zhang, Dong Guo, Yuanhan Zhang, Renrui Zhang, Feng Li, Ziwei Liu, and Chunyuan Li. Llava-next: What else influences visual instruction tuning beyond data?, May 2024a. URL
https://1lava-w// gi wwtttt....boo///eeee 1ot.:o/////222000000--00- -aw-000---------------a--------- deext-ablations
Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger 1mms supercharge multimodal capabilitics the wild, May 2024b. URL https://llava- gi thub. ho/bloog/024005---1
100-1llaa-neee seiva-next-stronee stronttsttoggrrlmee//
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint Xrxi:220080038 2024c. 1, 7, 8,9
Chunyuan Lí, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, Jianfeng Gao, et al. Multimodal foundation models: From specialists to general-purpose assistants. Foundations and Trends in Computer Grophics and Vision, 2024d.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping languago-image pre-training with frozen image encoders and large language models, 2023. URL https: httpss:///adddooo///////00.
wrtp.bs///011.11111/ 12597.
3, 17
13

---

Published in Transactions on Machine Learning Research (06/2025)
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang. Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding, 2024e. URI. https://arxiv. org/ abs/ wbw.aa/............. 06355. 1,
Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 26689 26699, 2024. 1,
Haotian Liu, Chunyuan Li, Qingvang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024a. 1, 17
Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishno Chen, Xu Sun, and Lu Hou. Tempcompass: Do video 1lms really understand videos? arXiv preprinl 2xXi:2403.004700 2024b.
LMMs-Lab. Video detail caption, 2024.
URL
datasets/Imms http://fwwc..attttttee//////aaaa
co/
VideoDeetaiCaaption.
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval. arXiv preprint arXiv:2104 088660, 2021. 19
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards
detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lánguistics (ACL 2024), 2024. 8
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Eggssheea: diagnostic benchmark for very long-form video language understanding, Advances in Neural Information Processing Systems, 36, 2024. 8
Antoine Miech, Dimitri Zhukov, Jean-Bapliste Alayrac, Makarand Tapaswi, Ivan Laplev, and Josef Sivie. HowTo100M: Learning a TTxt-Vides Embedding by Watching Hundred Million Narrated Video Clips. In ICCV, 2019. 3, 21
OpenAl. Gpt-4v.
2023. 3
OpenAl. Hello gpt-4o.
2024. 2, 4, 9
Viorica Päträucean, Lucas Smaira, Ankush Gupta, Adrià Recasens Continente, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Hcyward, Matcusz Malinowski, Yi Yang, Carl Docrsch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alex Frechette, Hanna Klimczak, Raphael Koster, Junlin Zhang, Stephanie Winkler, Yusuf Aytar, Simon Osindero, Dima Damen, Andrew Zisserman, and João Carreira. Perception test: A diagnostic benchmark for multimodal video models. In Advances in Neural Information 8, 9
Processing Systems, 2023. URL https: /twp://veeeeeeeeeee//eeeeeeeeeee
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), pp. 8748-8763. PMLR, 2021. 7, 17
Nils Reimers and Iryna Gurevych. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2020. URL
http:/aaxxvv.roo/aaaaa org/abs/2004 09813.
Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele. A dataset for movie description. In Proceedings of the TEEF conference on computer vision and pattern recognation, pp. 3202-3212, 2015. 3,
Xindi Shang, Donglin Di, Junbin Xiao, Yu Cao, Xun Yang, and Tat-Seng Chua. Annotating objects and relations user-generan seer-generated videos. In Proceedings of the 2019 on International Conference on Multimedia Retrieval, pp. 279 287. ACM, 2019. 2, 4
14

---

Published in Transactions on Machine Learning Research (06/2025)
Gunnar A Sigurdsson, Gil Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In Computer 14th European Conference, Amsterdam, The Netherlands, October 11 14, 2016, Proceedings, Port 14, pp- 510-520 526. Springer, 2016. 2,
2016:
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, -ran-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint orXiv:2312. 1 11805, 2023. 9
Jiawei Wang, Liping Yuan, and Yuchen Zhang. Tarsier: Recipes for training and evaluating large video description models, 2024, URL
.rgg hbbs/2407000623 8
Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. In The Twelfth International Conference on Learning Representations, 2023. 2, 4
Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reasoning in real-world vidcos. arXiv preprint 2rXiu:2405.09711, 2024a. 3, 4
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context.
interleaved video-language understanding, 2024b. URL httppp:://aaiiv.og/aaa////////// 15754. 8
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 9777-9786, 2021. 3, 4, 8, 9, 21
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yucting Zhuang. Video question answering via gradually refined attention over appearance and motion. In ACM Multimedia, 2017. 3, 21
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proccedings of the IEEE conference computer vision and pattern recognition, pp. -298- 2016. 3, 21
Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:444411999, 2024a. 9, 17, 18
Mingze Xu, Mingfei Gao, Zhe Gan, Hong- You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, and Afshin Dehghan. Slowfast-llava: A strong training-free baseline for video large language models. arXie preprint arXiu:2407.18411 2024b. 8
Mingze Xu, Mingfei Gao, Zhe Gan, Hong- You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, and Afshin Dehghan. Slowfast-llava: A strong training-free bascline for video large language models, 2024c. URL wwt.ooog//a/aaeaeeoeeeee /207. 15841. 17
https://arxiis /ttrs://aarii..or//b////0000
Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo. Advancing high-resolution video-language representation with large-scale video transcriptions. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2, 3, 4, 21
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.1000 10671, 2024. 7, 17
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In AAAI, pp. 9127 9127-9134, 2019. 3, 5, 8, 9, 21
Amir Zadch, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency. Social-iq: question answering benchmark for artificial social intelligence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 8807-8817, 2019. 3, 21
15

---

Published in Transactions on Machine Learning Research (06/2025)
Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script. knowledge models. Aduances in neural informalion processing systems, 34:23634 23651, 2021. 3, 21
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- training. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 11975-11986, 2023. 7, 17
Hang Zhang, Xin Li, and Lidong Bing. Vidco-llama: An instruction-tuned audio-visual language model for video understanding. Xiviv preprint or Xiv:2330 Xix:2306.02858, 2023. URL https://arrdv..og/abbaaa///////a 02858.
1, 3
Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, al. Lmms-oval: Reality check on the evaluation of large multimodal models. arXiv preprind arXiv:2477. 12772, 2024a. 7
Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, et al. Internim-xoon ciaiin-xoooooe-e222 2.5. versatile large vision language model supporting long-contextual input and output. Xix preprint wrXiv:2407.03322 2024b.
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16880 16852, 2024c.
Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, and Yiming Yang. Direct preference optimization of video large multimodal models from language model reward, 2024d. 1, 3, 6, 8, 9, 21
Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024e. URL https: //llavavvl.. gi github. ho/blog//024-0000011//11// 100000-11000 wwllllaeett-taaeeeeeeeaaaaaaaeee next-video/. 17, 18
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-lask long video understanding. ar preprinl arXiv:2406. 04264, 2024. 8
Luowei Zhou and Jason J. Corso. Youcookii dataset. 2017. URL https://api. http://ppp..emmntissccchllaaoo
CorpusID: 19774151. 2, 3, 4, 21
Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, Wang HongFa, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, Cai Wan Zhang, Zhifeng Li, Wei Liu, and Li Yuan. Languagebind: Extending video-language pretraining to n-modality by language-basee based semantic aligument, 2023a. 2,
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint 20le:2344..1992. 2023b.
16

---

Published in Transactions on Machine Learning Research (06/2025)
7 Video Representations
7.1 Efficient Video Representations LMMs
Current designs of large multimodal models (LMM) typically connect a vision encoder (Radford et al., 2021; Zhai et al., 2023) to a large language model (Yang et al., 2024) through a lightweight projector (Liu et al., 2024a) or a resampler (Li et al., 2023; Alayrac al., 2022). These components transform a set of visual representations into "visual lokens" aligned with text. embeddings. In contrast image-based LMMs, which generate only small number of visual tokens easily managed by standard GPU, video LMMs face challenges due to large nunber of visual tokens derived from multiple video frames. The LLa VA-NeXT-Videe (Zhang et al., 2024e) and PLLaVA (Xu et al., 2024a) models address this by simly considering average pooling to reduce the number of tokens representing each frame.
Following the idea of FlowFast in the traditional video understanding (Reichtenhofer al., 2019), adaptive reductions in visual tokens are demonstrated by recent video LMMs, LITA (Huang et al., 2024) and SlowFast- LLaVA (Xu et al., 2024c). Initially, these methods represent all sampled frames with minimal number of visual tokens (fast frame) typically just one-by using a large pooling stride. They then switch to smaller pooling stride for certain frames to retain more visual tokens (slow frame). Finally, they combine the visual tokens of fast frames with those of slow frames. However, this approach can lead to some frames being represented twice. In contrast, our method uses a larger pooling stride for sampled frames to maintain fewer visual tokens (fast frame) or a smaller stride for others to keep more (slow frame). We then arrange slow and fast frames an interleaving pattern.
= 2
2
3
p
2
Pooling
2
4
5
5
5
5
M tokens
Frames
Faast are assigned fewer visual tokens
Fslow are assigned more visual tokens
Figure 8: Vidco representations. A different number of tokens are utilized to represent frames.
7.2 LLaVA-Video SlowFast
We represent each video as a sequence with maximum frames. Each frame represented in M tokens. FPS-based video representation can be considered in the future. Specifically, each frame encoded via an image encoder and a two-layer MLP for projection. These visual tokens are concatenated with word tokens and processed by large language model (LLM). Managing tokens for every frame be computationally demanding. For instance, employing the SigLIP (Zhai et al., 2023) encoder for a vidco with T = 100 results in 67,600 lokens, assuming M = 729 Lokens per frame, which often exceeds GPU memory limits. This issue is exacerbated when using Large-paranee before maxing out the memory on 128 NVIDIA H100 GPUs. Such a limited number of frames can introduce inconsistencies in language annotations, reducing model efficacy. One strategy incorporate more frames is by applying p X p spatial average pooling to reduce M to M/p², thus lowering the token count per frame
LLMs; with the Qwen2-72B model, we could only process 8 frames
17

---

Published in Transactions on Machine Learning Research (06/2025)
suggested by recent studies (Xu et al, 2024a; Zhang et al., 2024c). However, the number of visual tokens is crucial for preserving the informational content of each frame, which is vital for video comprehension.
In our LLaVA-Video slodFaat, we categorize the frames into two groups, based on the a strike rate where the every 8 frames are uniformly selected to form the slow frame group, and the rest the frames are consdiered as the fast frame group. Note that special case & = leads to only one group; reducing the SlowFast representation to the original simple representation. For each group; we apply different pooling rate using Pytorch function pooling avg wwg_pool2d() p x p pooling and 2p 2p pooling for slow and fast frames, respectively. To summarize, paramterize the video representation configuration V (T,M, s, p). The total number of tokens is #tokens [T/s] x M/p² + (T [T/s]) x MM/4p²
8
Data
8.1
Video Detail Description
As discussed in Section 3.2, we show that generating level-1 description should consider historical context. Figure illustrates the impact of excluding historical context the quality of video descriptions. Specifically, including historical context helps accurately identify characters across different times as the same individual.
Generating Video Captions with Historical Context
0-10s
10-20s
The video begins with person standing balcony, holding large water balloon one hand and knife in the other,.... The camera then shifts to show woman below the person on the balcony, near pool, wearing yellow top, pink leggings, ... The person on the balcony moves closer to the edge, positioning the water balloon over the woman below....
Generating Video Captions without Historical Context
The video shows with person standing balcony, holding large red water balloon in one hand and knife in the other..... Then woman in yellow shirt and pink pants walking on sidewalk near pool. Finally, one person stand on the high platform cuts the balloon, causing water to splash out and woman.
Figure Generating video captions with or without historical context.
8.2 Video Question Answering
In Table 5, we list the names and descriptions of different question types and their corresponding proportions in the LLaVA- Video-178K dataset. The prompt used to generate video question-answer pairs from GPT-40 is shown in Table. 6. In Fig. 4, we show an example of video along with its detailed description, an open-ended question, and multiplo-choice question.
8.3 Dataset Comparison
We provide a more comprehensive comparison of LLaVA-Video-178K with other video-language datasets for the video caption task and video question answer task. Specifically, we organize the table into four groups, each characterized by its method of text annotation. As shown in Table 7, unlike other datasets, LLaVA-Video-1788 V78K uniquely includes all three types of annotations: captions, open-ended questions, and multiple-choice questions.
9 Beyond Singularity: Extensive Sampling Matters
We perform experiments explore how vidco representations affoct the model's performance. All experiments were carried out video-only setting, using video data with durations from seconds as our training data. We focused evaluating how the number of frames and the number of visual tokens per frame impact.
18

---

Published in Transactions on Machine Learning Research (06/2025)
Table 5: Question types for video question answering data creation. For each type, we provide its name, description, and the proportion it represents in the
Question type
Description
Proportion
Designed assess reasoning about temporal relationships between actions/events. Questions involve previous, present, next actions.
Temporal
7.2%
Tests ability to perceive spatial relationships between observed instances in video scene.
Spatial
7.2%
Focuses on explaining actions//ventts determining intentions actions or causes for subsequent events.
Causal
7.2%
Assesses ability describe the major scene of the viden, like where takes place and the overall enviromment.
Description-Seenc
7.2%
Involves describing actions attributes of people, such as their activities and appearances.
Description Human
6.7%
Assesses ability to describe attributes of objects, like their appearance and function.
Description-Obye
7.0%
Tests ability to count instances objects, people, actions, and distinguish between old and new elements in scene.
Count
7.1%
Binary
Involves ycs questions related to the vidco content.
7.2%
Fine Grained Action Understanding
Creates questions challenging comprehension subtle actions.
6.3%
Plot Understanding
Challenges ability to interpret the plot in the video.
7.1%
Mon-Existent Actions with Existent Scene Depictions
Assesses reasoning with introduced non-exist activities without changing physical details.
6.6%
Time Order Understanding
Challenges recognition of temporal sequence of activities in videos.
6.9%
Object Direction
Emphasizes perception of object movement direction.
3.8%
Camera Direction
Focuses on direction of camera movement.
4.1%
Delves into discerning variations in speed, including absolute and
Speed
3.6%
relative speeds.
Centers on how attributes of objects or the entire video change over time, like size, shape, color, and more.
Attribute Change
4.5%
model performance. Regarding the frame count, it noteworthy that observing the effects of a high number of frames such as over 100 does not necessarily require long videos. Our results indicate that the dynamic properties of the data render even 100 frames insufficient to fully capture the condent of 30-second video, which typically runs at. 15 FPS.
In Table 8, the first group shows an increase in the number of frames from 32 to 110. We set 110 frames as the upper limit to avoid overloading the GPU. With more frames, we see significant improvements in all datasets. While it's generally expected that using more frames boosts performance, previous studies (Luo et al., 2021; Lei al., 2021; 2022) have noted that performance tends plateau when training with more than 16 frames. We propose that the saturation observed in earlier studies arises due the selection of training datasets such MSVD (Chen & Dolan, 2011) and WebVid (Bain et al., 2021), where the video content is highly static, allowing a small number of frames to represent the entire video effectively. In contrast, the dynamic nature of the videos and the detailed nature of the annotations in LLa Viie---iie--188 allow for continuous benefits from extensive sampling
The second group Table demonstrates the effects of varying the number of inference frames while keeping the number of training frames constant. modest increase in the inference frames slightly enhances performance; however, excessively increasing the number of inference frames degrade it.
19

---

Published in Transactions on Machine Learning Research (06/2025)
tasks
# Tempprrl: this task is designed to assess the capability of reasoning ... <.mmiitted>>
## caption-1: The video features child sitting in baby chair at dining table, creating <omittedd>
## question-1: What was the child doing as he sat on the baby chair?
## answer-1: The child was reading book.
## caption-3: ...omitted>>
## # question-3: ...asiion <..oniteed>>>
## answer-3: ...omiited>>
# Spatial: this task involves creating questions that test person's comiity...mmittee <..omitted>> "
system_messagge
### Task:
Given detailed description that summarizes the content of video, generate question-answer pairs based on the description to help humans better understand the video. The question-answer pairs should be faithful the content of the video description and developed from different dimensions promote comprehensive understanding of the video.
Here are some question dimensions and their explanations and exampled question-answer pairs for reference: [lask definitions)
# # # Guidelines For Ouestion-Answer Pairs Generation:
Read the video description provided carefully, paying attention to the content, such as the scene where the video takes place, the main characters and their behaviors, and the development of events.
Generate appropriate question-answer pairs based on the description. The question-answer pairs should cover as many question dimensions and not deviate from the content of the video description. Generate 1 question-answer pair for each dimension.
## ## Output Format:
1. Your output should be formed in JSON file.
2. Only provide the Python dictionary string.
Your response should look like:
["Dimension": <dimension-1>, "Queation": <question-1>, "Answer": <anawer-1>,
"Dimension": <dimension-2>, "Question"; <question-22, "Answer"; <answer-2>...
user issermessage
Please generale question-answer pairs for the following video description:
Description: [caption]
for cur wi_dddee in videos:
sys_mag = system_ setteem_mesaaes.co format(task, (tt.aaaa.eeeiiiiiiioootssssaaa
usr usr_msg user _asr_messageseeo . format (caption=cur_ video)
response (eys msg, usr m_sg)
Table 6: We explain the process of creating prompts for GPT-40 to gather question-answer pairs from each video description. tasks includes the definition of all question types along with examples of question-ansen pairs. We instruct GPT-4O to generate questions that cover as many question types as possible.
In Table 8's third group, we illustrates the tradc-off between the number of frames and the number of tokens per frame. Configurations with fewer tokens per frame but more frames yield superior results, even with a lower total count of visual tokens (18,590 versus 21,632). This finding emphasizes that increasing the number of frames, rather than the tokens per frame or the total number of tokens, enhances performance. However, balance necessary: as the number of frames increases 440 and the tokens per frame decreases to 64, performance drops. This observation led to VA-Vieedee SlexFast for video representation.
20

---

Published in Transactions on Machine Learning Research (06/2025)
Table 7: Comparison of LLaVA-Video-1788 and other video-language datasets. Average FPS
represents the average number of frames per second that are used prompt GPT-40/GPT-4V for annotation.
Total Video Average #Caption
#OE #MC
Text
#Video
Length
FPS
QA
QA
How0o100M (Miech al., 2019)
ASR
136M
134.5Khr
136M
0
ACAV (Lee et al., 2021)
ASR
100M
277.7Khr
100M
0
YT-Temporal-180M. (Zellers et al,, 2021)
ASR
180M
180M
0
HD-VITAA-10M (Xue et al., 2022)
ASR
103M
371.5Khr
103M
0
0
MSVD (Chen & Dolan, 2011)
Manual
1970
5.3h
1K
0
0
LSMDC (Rohrbach al., 2015)
Manual
118K
158h
118K
0
ASR-VTT (Xu al., 2016)
Manual
10K
40h
10K
0
DiDeMo (Anne Hendricks et al., 2017b)
Manual
27K
87h
27K
0
Net (Caba Heilbron et al., 2015)
Manual
100K
849h
100K
0
YouCook2 (Zhou Corso, 2017)
Manual
14K
176h
14K
0
TVQA (Lei et al., 2018)
Manual
21K
3.39Khr
O
152K
ActivityNet-QQA (Yu et al., 2019)
Manual
5.8K
290h
0
58K
0
Social-TQ (Zadeh al., 2019)
Manual
1.2K
20h
O
7.5k
NEXT-QQA (Xiao et al,, 2021)
Manual
5.4K
66h
O
52K
47K
MSVD-QA (Xu et al., 2017)
Open-source Model
1.9K
5.3h
41K
50K
0
MSRVTT-QA (Xu al., 2017)
Open-source Model
10K
10h
243K
0
Panda-70M (Chen et al., 2024b)
Open-source Model
70.8M
166.8Khr
70.8M
0
LLaVA-Hound (Zhang al., 2024d)
GPT-4V
900K
3Khr
0.008
900K
900K
0
ShareGPT4Video (Chen et al., 2024a)
GPT-4V
40K
0.2Khr
0.15
40K
0
0
LLaVA-Video-177 Video-178K
GPT-4o
178K
2Khr
178K
960K
196K
Table Visual Representation Configurations and Performance Correlation. Ttrain and T are the number of frames in the training and inference stage, respectively. M/r²: number of visual tokens per frame.
in-domain
out-of-domain
NEXT-QA Perception/est
EgoSchema VideoMME
Tetrann
Trest
M/p²
val
test wo
Training with more frumes
32
32
169
80.4
68.2
56.3
59.1
64
81.4 (+1.0)
(+0.)
58.4 (#2.1)
59.6 (+0.5)
110
110
169
82.0 (+1.6)
68.3 (+0.1)
59.1 (+2.8)
60.4 (+1.3)
Inference with frames
32
32
169
80.4
68.2
56.3
59.1
64
169
80.7 (+0.3)
68.9 (+0.7)
56.3 (+0.0)
59.9 (+0.8)
32
110
169
80.5 (+0.1)
67.2 (-1.0)
55.2 (-1.1)
58.8 (-0.3)
Using more
frames with fewer visual tokens frame
32
729
79.4
58.3
59.1
110
110
169
82.0 (42.6)
68.3 (-1.2)
59.1 (+0.8)
60.4 (+1.3)
440
440
64
81.6 (42.2)
67.2 (-2.3)
59.4 (+1J))
60.2 (+1.1)
10
Capabilities
Beyong achieve good benchmark performance, Our observations of LLaVA-Video reveal various capabilities in vidco understanding. Specifically, it show great abilities the understanding video using real-world inowledge,includdngg but not. limited to:
21

---

Published in Transactions on Machine Learning Research (06/2025)
Table 9: Comparison of different video representations. The video representation V is consistent training and inference for all methods, except that SlowFast-LLan LLaVA considers simple representation V in training and its specified V in inference.
in-domain
out-of-domain
Visual
NEx T QA PerceptionTest EgoSchema VideoMME
Method
V (T, M, s; p)
Tokens
mc
val
test
Simple representation (32, 729, 1, 2)
5,408
80.4
68.2
56.3
59.1
LLa Video SlouFaat
(64, 729, 3, 2)
5,396
81.1
67.7
57.1
59.8
LITA
(12, 729, 2, 2)
5,313
80.8
68.3
54.3
59.1
SlowFast-LLLVa.
(12, 729, 2, 2)
5,313
79.4
68.2
56.2
58.9
Optical Nlusion: As shown in Table 11, LLaVA-Video recognizes that the green dragon in the video is not real 3D object. It appears threo-dimensional due to optical illusion that affects human perception.
Special Domain: As indicated Table 11, LLa understands the content within special domains in the video, such as sketches and fights in video games.
Unusual Action: As detailed in Table 12, LLaVA- Video identifies atypical actions in the video, such as 'physical therapy' for pets, beyond ordinary activities.
comprehends basic physical laws demonstrated in the video, like zero gravity in space stations, which allows objects to float without falling.
Physical Laws: As shown in Table 13,
Real-world Knowledge: Illusion
User
Please describe this vidro.
Vidoo-Vido
The video features person interacting with large, cartoonish green dragon cutout in a well-lit, spacions room adorned with framed pictures on the walls. The person, dressed in blue tank top and light blue jeans, is seen hugging and adjusting the dragon cutout, which wide, toothy grin and large, expressive eyes. The text overlay reads 'Reply seloutria's comment: Once you see you can't l see suggesting that the designed to be visually striking and memorable. The person around the eutout, showensing different angles and perspertives.
Is this green dragon real 3D structure?
MaVA-Vidoo
No, the green dragon 2D that creates optical illusion.
Table 10: LLa VL-VA-V Video learns to understand the optical illusion in the video.
22

---

Published in Transactions on Machine Learning Research (06/2025)
Real-world Knowledge: Special Domain
User
Please describe this video.
Video
The video begins with blank, lined sheet of paper. A hand holding black pen drawing the paper, creating the outline of flower with five petals. The artist adds details the petals, including lines and shading give them more realistic appearance. The stem and leaves are also drawn, completing the flower illustration. The artist adds final touches to the flower drawing, eubancing the details and shading to give it a more realistic appearance. The artist then switches to a yellow pen and begins coloring the center of the flower, adding depth and contrast the drawing. The video concludes with artist adding final touches to coloring, euhancing the overall appearance the flower illustration.
User
What could this flower be?
MLvVA-Vido
This flower could be a magnolia or a similar species.
User
Please drscribe this vidro.
LLaeo-Vide
The video showeases an intense battle character dresed in dark, heavy armor with glowing orange accents wields sword and faces off against large, menacing creature with glowing red eyes and autlers. The text 'EVERY BATTLE SCREAMS GOTY POTENTIAN TENTIAL"" is prominently displayed at the top of each frame, emphasising the high stakes and epic nature of the confrontation. The battle is fierce, with the character and the creature exchanging powerful attacks. The creature emits blue energy and lightning, while the character retaliates with fiery and iey attacks. The snowy ground littered with debris and frost, adding to harsh and brutal environment. The background features large body of water distant lighthouse, euhancing the sense of isolation and danger.
suowy, mountainous landscape. A
Table 11: LLa Video learns to understand the the video in special domain.
23

---

Published in Transactions on Machine Learning Research (06/2025)
Real-world Knowledge: Unusual Action
EXITVE
User
Please describe this video
MLVA--Vido
The video begins with person holding tabby cat, with the text DDGS WITH BACK PAIN' and a crying emoji displayed at the bottom of screen. The person, who has tattoos on their arms is wearing light-colored shirt, in room with large windows white
white person petting large, brown dog, who sitting on floor. The person then stauds up gestures with their hands, while the dog looks up them. The next large, thuffy dog with black and white cost, who is sitting table. The person continues to pet the dog, who appears calm and relaxed.
scene transitions
shows the person kneeling and petting
User
What did this man do his pets?
MiVA-Video
This provided physical therapy to his pets.
Table 12: VA-V Video learns to understand the unusual action in the video.
Real-world Knowledge: Physical Laws
describe
LLVV Video
The video features person in space station environment, wearing dark gray shirt wristwatch. They are holding containers filled with thick, yellowish substance. person tilts the containers demonstrate the fluid's behavior, showing low it stretches and moves between the containers. The background includes various equipment and panels typical of space station, with labels such 'GALLEY" and 'FUSION. The person continues manipulste the containers, emphasizing the fluid's properties. The video concludes with the person holding one container and twisting the lid to close it, further demonstrating the fluid's properties.
Use
Why doesn't 's this yellowish substance fall?
Video
This yellowish substance doesn't 't fall because it is in zero gravity.
Table 13: LLaVA-Viden learns to understand the physical laws in the video.
24


Reviews:
[{"review_id": "HvWLgr2AQh", "reviewer": "Reviewer_4xoL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. Comprehensive and Diverse Synthetic Dataset:\nThe paper presents LLaVA-Video-178K, a large-scale synthetic dataset featuring motion-intensive videos, rich three-level captions, and diverse, benchmark-aligned question types—greatly enhancing resources for video-based large language model (LLM) training.\n\n2. Representation Learning Framework:\nIt introduces a slow-fast representation learning approach that effectively captures high frame rate video features, improving the model’s ability to understand dynamic video content.\n\n3. Strong Practical Relevance:\nBoth the dataset and the learning framework are designed to closely match real-world evaluation benchmarks, making them highly practical for advancing downstream video-language tasks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces LLaVA-Video-178K, a synthetic dataset designed for training video-based large language models. The dataset features motion-intensive video samples accompanied by comprehensive three-level caption data. Additionally, LLaVA-Video-178K incorporates diverse question types that align well with downstream video benchmarks, enhancing its practical utility for model training. The work also presents a slow-fast representation learning framework intended to better capture high frame rate video features.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The evaluation relies on a single model architecture for training, which limits the conclusiveness of the dataset's effectiveness claims. I recommend incorporating recent state-of-the-art models (e.g., Qwen2.5-VL) for finetuning experiments to strengthen the generalizability of the findings.\n\n2. According to the results presented in Table 5 of the Supplementary Material, the proposed video representation scheme achieves only marginal improvements compared to the simple representation approach, and underperforms on the NExT-QA validation set. This raises questions about the practical value of the slow-fast representation learning framework.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WOGMPuxOZA", "reviewer": "Reviewer_iu6v", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths:*\n-  LLaVA-Video-178K addresses key issues in prior datasets with dynamic, untrimmed videos and dense frame sampling.\n- The paper uses a three-level GPT-4o pipeline to capture temporal context effectively.\n- The SlowFast design helps handle more frames under GPU limits.\n\n*Weakness:*\n- The whole dataset relies on GPT-4o to generate captions and QA pairs, but can we fully trust its outputs? There's no mention of any human rechecking to ensure the quality or correctness of the data. It would be better if the authors included at least some manual validation to check the automatic pipeline.\n- The model struggles on egocentric benchmarks since the training data seems unbalanced.\n- In Table 3, the order of adding datasets is a bit confusing. If the goal is to show the contribution of LLaVA-Video-178K, why not add it last?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces *LLaVA-Video-178K*, a large synthetic dataset for training video-based LMMs. It fixes issues in current datasets like static scenes, short clips, and low frame sampling. The dataset has 178K untrimmed videos from 10 sources like Kinetics-700 and Ego4D. They use a 3-level GPT-4o pipeline to generate detailed captions and summaries, sampling at 1 FPS to keep details. It also includes over 1M QA pairs covering different reasoning types. \nBased on this, they build *LLaVA-Video*, which uses a SlowFast method to handle more frames efficiently.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Since comedy is the most sampled category (Figure 7), I’m curious—can the model actually understand humor? Humor is often subtle and context-dependent, so this could be an interesting test of the model's deeper reasoning abilities. It would strengthen the paper to include some analysis or examples around this.\n\n2. The author claims the dataset is more dynamic by using PySceneDetect, but it would be much more convincing if they provided quantitative stats, like the average number of scene changes per video. This would help readers better understand how dynamic the videos really are compared to other datasets.\n\n3. There are several missing citations throughout the paper—for example, some GPT-related references are mentioned without proper citations. Please make sure all references are complete to maintain academic rigor.\n\n4. In Appendix 2.2, the figure reference is missing. It’s a minor issue but should be fixed for clarity", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3o6iG9Mp9M", "reviewer": "Reviewer_swz3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. High-Quality Synthetic Dataset (LLaVA-Video-178K): The authors create a large-scale dataset with 178K videos and 1.3M instruction samples including dense captions, open-ended QA, and multiple-choice QA. This fills a major gap in current video-language training data which is often sparse, static, or overly trimmed. The dataset is derived from ten diverse and well-known video sources with careful filtering for dynamism and scene complexity.\n\n2. Three-Level Recurrent Captioning Pipeline: The proposed captioning system introduces temporal coherence using recurrent generation across three hierarchical levels. This addresses a core challenge in video-language modeling: maintaining narrative consistency across time. It balances fine-grained details with high-level summaries, which is a notable improvement over single-frame or flat captioning systems.\n\n3. The authors introduce 16 detailed question categories, which go beyond generic question-answering and test various aspects of spatiotemporal and causal reasoning. This enhances the training signal quality for instruction-following models.\n\n4. Inspired by SlowFast networks, the paper proposes a token-efficient video representation method that allows sampling up to 3× more frames per video by adjusting token allocation and pooling across frames.\nThis leads to better use of GPU memory and longer video context modeling.\n\n5. The LLaVA-Video models achieve strong zero-shot performance on 11 diverse video understanding benchmarks, including LongVideoBench, VideoMME, and PerceptionTest.\nThe 72B variant matches or surpasses proprietary models like Gemini-1.5-Pro on several tasks, which is a notable achievement for open-source models.\n\n6. The promise to release the dataset, code, model checkpoints, and visual chat demo reflects excellent reproducibility and commitment to the research community.\n\n\nWeaknesses:\n\n\n1. Despite using GPT-4o and curated heuristics, the dataset remains entirely synthetic. There is a potential gap in real-world QA alignment and model behavior under human-style supervision or dialog.\n\n2. The paper focuses solely on visual modality and ignores audio, which is often essential for comprehensive video understanding. Competing methods like VideoLLaMA2 include audio support.\n\n3. The dataset draws heavily from a few dominant benchmarks (e.g., YouTube Shorts, ActivityNet), and prompt templates are fixed. This could lead to distributional bias or repetitive patterns in both videos and language annotations.\n\n4. Although the dataset is evaluated quantitatively on standard benchmarks, the paper lacks human qualitative analysis or annotation consistency studies, which would strengthen the case for dataset quality.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces LLaVA-Video, a new large video-language model (LVM) trained on a novel synthetic dataset, LLaVA-Video-178K, specifically designed for video instruction-following tasks. \n\nIt introduces LLaVA-Video-178K: A High-Quality Synthetic Dataset, a large-scale, synthetic dataset comprising 178,510 videos and 1.3 million instruction-following samples, including detailed multi-level captions, 960K open-ended QA pairs, and 196K multiple-choice QA items. The dataset is sourced from ten well-known video collections and emphasizes untrimmed, dynamic videos to preserve narrative continuity and realism.\n\nThe paper proposes a three-level recurrent captioning system that describes short segments (Level-1), summarizes every 30 seconds (Level-2), and produces a final global description (Level-3). Captions are generated using GPT-4o, with each level conditioned on historical context, allowing coherent long-range video narration. LLaVA-Video, a fine-tuned video-language model built on the LLaVA-OneVision architecture which develops a SlowFast-style video representation, optimizing token allocation across frames to allow more frames per video within GPU/memory constraints.\n\nIt defines 16 diverse QA categories, including temporal, spatial, causal, object-level, and fine-grained perception questions.\n\nThe 72B variant achieves performance comparable to proprietary models like Gemini-1.5-Pro, highlighting the effectiveness of high-quality synthetic data and open-source model design. The importance of LLaVA-Video-178K in performance gains over previous datasets like LLaVA-Hound and ShareGPT4Video", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Include a small-scale human rating of caption and QA relevance, coherence, and answer correctness. This will provide important validation for the synthetic data pipeline's efficacy beyond automated filtering.\n\n2. Since many videos are derived from existing benchmarks or public platforms (e.g., YouTube Shorts), explicitly mention how licensing, fair use, and redistribution are handled for LLaVA-Video-178K.\n\n3. Even if audio is out of scope, consider adding a discussion about why audio was excluded, and future directions for incorporating multi-modal audio-visual understanding.\n\n4. Provide per-question-type breakdown of model performance to show which types of reasoning (e.g., causal, spatial) are", "broader_impact_concerns": "1. The dataset is entirely synthetic, generated using GPT-4o and filtered heuristics. This raises concerns about:\n   1.a. Bias replication from the underlying LLMs (e.g., cultural stereotypes, gender norms, Western-centric viewpoints).\n   1.b. Lack of diverse annotation styles that come from human raters across cultures or age groups.\n   1.c. Reinforcement of model hallucinations or surface-level reasoning if synthetic data lacks real-world nuance.\n\n2. The dataset aggregates videos from public sources (YouTube Shorts, ActivityNet, etc.) but does not clearly describe:\n   2.a. Whether these videos are permissively licensed for redistribution.\n   2.b. If derived annotations (e.g., captions and QA) fall under fair use or transform the original content enough to be redistributed.\n\n3. The system could be applied to automated surveillance, behavioral profiling, or emotion detection in video data without user consent, especially in sensitive settings (e.g., schools, workplaces, public spaces). The authors should note potential misuse cases and recommend safeguards, such as requiring responsible deployment practices or limiting commercial use.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "B6uniRmZWw", "reviewer": "Reviewer_TMFo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well written and easy to follow.\n- The dataset meticulously collected and the methods used in collection and creation are clearly explained\n- The paper open sources the data, codebase and checkpoint which will be helpful for future use.\n\nWeaknesses:\n- The paper has some unclear points about the effectiveness of adding LLaVA-Video-178K in the training data.\n- The paper does not mention the source of the testing data and if there are any overlaps with the sources of LLaVA-Video-178K.\n- The final ablation study in section 4.3 lacks clarity.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a video dataset for enhancing training of video LLMs. The paper introduces:\n\n- A new video instruction tuning dataset LLaVA-Video-178K. This dataset was created with improvements in the following aspects:\n  - Extensive video source: The authors reviewed all data sources and narrowed the sources down to 10 sources for all datasets\n  - Dynamic untrimmed videos: Prior datasets included a single scene and trimmed videos which restricted video LLMs capabilities. To overcome this, LLaVA-Video-178K contains dynamic untrimmed videos.\n  - Recurrent Detailed Caption Generation Pipeline with Dense Frame Sampling: Prior datasets sampled at lower rates of 2 frames per 30 seconds, thus missing details. LLaVA-Video-178K contains detailed captions at various levels from every 5 seconds to full video.\n  - Diverse Tasks: Captioning, Open ended QA and MCQs.\n- A new open source Video LLM trained with LLaVA-Video-178K.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The authors mention to refer to Appendix 7 for more details on SlowFast idea but the appendix is missing. The authors need to include the appendix.\n- The authors claim an improvement of 31.9 points on NeXT-QA dataset on adding LLaVA-Video-178K. However from table 3 we see an improvement of 15.7%. Can the authors clarify this?\n- The authors show the improvement on adding LLaVA-Video-178K and then adding subsequent datasets to it. To show the effectiveness of LLaVA-Video-178K the authors should also show the performance on all training datasets - LLaVA-Video-178K.\n- The authors should comment on how adding LLaVA-Video-178K to the training affects 72B models vs 7B models. That is if the 72B models show more improvement or the 7B models on adding LLaVA-Video-178K to the training data.\n- The authors should clarify the sources of the testing data and if there are any overlaps with LLaVA-Video-178K.\n- In section 4.3:\n  - The authors should clarify what they mean by in domain and out of domain in table 4.\n  - The authors should clarify why they choose to generate MCQ questions and open ended questions for ShareGPT4Video but not do the same for MCQ questions for LLaVA-Hound.\n  - The authors should explain why they think ShareGPT4Video outperforms LLaVA-Video-178K on one dataset and matches the performance on another.\n  - The authors should clarify if they are using the 7B model or 72B model for this experiment. And how the other model performs in this ablation.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)