Unleashing Hour-Scale Video Training for Long Video-Language Understanding
Authors: Jingyang Lin
Jialian Wu
Ximeng Sun
Ze Wang
Jiang Liu
Yusheng Su
Xiaodong Yu
Hao Chen
Jiebo Luo
Zicheng Liu
Emad Barsoum
Venue: NeurIPS 2025 spotlight ()
Keywords: Long Video-Language Understanding
Synthetic Video Instruction-following Dataset
Efficient Video-Language Modeling
Primary area: deep_learning
TL;DR: We present VideoMarathon, a large-scale synthetic video instruction-following dataset for hour-long video-language understanding, and propose Hour-LLaVA, a Video-LLMs that enables efficient video-language modeling.

Abstract: Recent long-form video-language understanding benchmarks have driven progress in video large multimodal models (Video-LMMs). However, the scarcity of well-annotated long videos has left the training of hour-long Video-LMMs underexplored. To close this gap, we present VideoMarathon, a large-scale hour-long video instruction-following dataset. This dataset includes around 9,700 hours of long videos sourced from diverse domains, ranging from 3 to 60 minutes per video. Specifically, it contains 3.3M high-quality QA pairs, spanning six fundamental topics: temporality, spatiality, object, action, scene, and event. Compared to existing video instruction datasets, VideoMarathon significantly extends training video durations up to 1 hour, and supports 22 diverse tasks requiring both short- and long-term video comprehension. Building on VideoMarathon, we propose Hour-LLaVA, a powerful and efficient Video-LMM for hour-scale video-language modeling. It enables hour-long video training and inference at 1-FPS sampling by leveraging a memory augmentation module, which adaptively integrates question-relevant and spatiotemporally informative semantics from the cached full video context. In our experiments, Hour-LLaVA achieves the best performance on multiple representative long video-language benchmarks, demonstrating the high quality of the VideoMarathon dataset and the superiority of the Hour-LLaVA model.

Unleashing Hour-Scale Video Training for Long Video-Language Understanding
Jingyang Lin
Jialian Wu
Ximeng Sun'
Ze Wang'
Jiang Liu'
Yusheng Su'
Xiaodong Yu²
Hao Chen
Jiebo Luo²
Zicheng Liu
Emad Barsoum'
AMD
University of Rochester
https://videemmrrchon .it.iii.i.ottuoooo/o.ooo
Abstract
Recent long-form video-language understanding benchmarks have driven progress in video large multimodal models (Video-LMMs). However, the scarcity well- annotated long videos has left the training of hour-long Video-LMMs underex- plored. To close this gap, we present VideoMarathon, a large-scale hour-long video instruction- following dataset. This dataset includes around 9,700 hours of long videos sourced from diverse domains, ranging from 3 to 60 minutes per video. Specifically; it contains 3.3M high-quality QA pairs, spanning six fundamental top- ics: temporality, spatiality, object, action, scene, and event. Compared to existing video instruction datasets, VideoMarathon significantly extends training video dura- tions up to hour, and supports 22 diverse tasks requiring both short- and long-term video comprehension. Building on VideoMarathon, we propose Hour-LLa VA, a powerful and efficient Video-LMM for hour-scale video-language modeling, It enables hour-long video training and inference 1-FPS sampling by leveraging a memory augmentation module, which adaptively integrates question-relevant relevann and spatiotemporally informative semantics from the cached full video context. our experiments. Hour-LLavV achieves the best performance on multiple repre- sentative long video- Indo-languagee benchmarks, demonstrating the high quality of the VideoMarathon dataset and the superiority of the Hour-LLaVe VA model.
Introduction
Leveraging strong foundation models [44, 67] and high-quality video instruction-followingg data [7, 26, 70], recent Video-LMMs [7, 28, 51, 70] have achieved promising performance on basic video- language tasks, such video question-answerte 22,31, 39, 48]. As attention shifts toward longer video modeling, these models face new challenges in capturing long-term dependencies. To measure progress in this direction, several benchmarks have been introduced for hour-scale video understanding, where the goal is to assess model performance on tasks involving long-form video content over hour [5, 14, 55, 58]. Early attempts long-form video-language modeling [45, 46, 50, 68, 74] have shown promising results on these benchmarks.
(QA) [60, 61, 66) and video summarization [20,
However, significant gap remains between the length of videos used during training and those encountered at inference time. While testing videos often exceed one hour [5. 14, 55], most existing training datasets [7, 28, 70] consist of vidcos shorter than ten minutes, summarized in Table 1. This mismatch limits the models' ability to explicitly learn long-term dependencies, highlighting the necessity for long-form video instruction-follooing datasets to bridge this gap.
To this end, we introduce VideoMarathon, a large-scale video instruction- follooiing dataset specifically designed for long-form video-language modeling. VideoMarathon contains around 9, 7,00 hours of
"Work done during the internship AMD. S Corresponding author: Jialian. Wu @amd.com.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Table I: Comparison between VideoMarathon and other existing video instroicioonfollooinng datasets. OE and MC denote open-ended and multiple-choike choice, respectively.
Total Video
Average Video
Duration
#OE
#MC
Dataset
Captioner
Summarizer
Time
Length
Range
QA
QA
MlaVA-Houng [69]
GPT-4V
GPT-4
3K hrs
0.2 mins
0-8 mins
900K
0
ShareGP44Video [7]
GPT-4V
GPT-4
0.2K hrs
0.3 mins
< 2 mins
0
LLaVA- -LaVA-Video- 178K [70]
GPT-4o
GPT-4o
2K hrs
0.6 mins
< 3mins
960K
196K
VidenMarathon (ours)
Qwen2VI-77
DeepSeek-VS
9.7K hrs
20.9 mins
3-60 mins
1.73M
1.57M
long videos (3-60 min per video) sourced from diverse domains, such as activities [4], egocentric [17], movie [47], cooking [72], and open-world [8] scenarios. Inspired by the recent advances [5, 14, 28, 43], we carefully establish comprehensive task taxonomy covering six essential topics-temporality, spatiality, object, action, scene, and event-across 22 tasks requiring both short- and long-term comprehension. To generate high-quality video instruction-followigg QA samples, we develop a hierarchical video captioning pipeline: clip-level descriptions are first produced using Qwen2VL- 7B [54] across the six topics, then aggregated into the event- and global-level summaries via DeepSeek- V3 [33]. Based on these hierarchical video captions, we synthesize 3.3M high-quality QA pairs by DeepSeek-V3. V3, guided by task-specific prompts incorporating explicit task definitions and exemplars from prior benchmarks [5, 14, 28, 43]. The resulting QA samples cover both open-ended (OE) and multiple-choice (MC) formats. As shown in Table I, VideoMarathon substantially surpasses existing video ceatrructtonfoolloinnn datasets in average video length, duration range, and scale of QA pairs.
Due to the limited availability of long video data, most existing Video-LMMs [7, 29, 35, 70] are trained on short videos, typically less than minute on average (see Table 1). Uniform temporal sampling [27, 70], which selects a fixed number of frames (8-256 per video), works well for short videos but causes substantial information loss when applied to hour-long videos, resulting in severe performance degradation [5, 14, 55]. Moreover, even with large-scale hour-long data available, training existing Video-LMMSs on such long videos remains non-trivial, the quadratic complexity of attention mechanisms [53] causes token budgets to explode. To this end, we propose Hour-L. .a VA, a Video-LMM specifically designed for efficient long-form video-language modeling and capable of directly ingesting hour-long and densely sampled videos. In particular, Hour-LLaVA stores the full video context sampled at 1 FPS in a memory repository. To accommodate GPU memory constraints, only small subset (approximately 6%) of the full video context, referred to as decayed video tokens, used as input to the language model for efficient training and inference. These decayed tokens are then augmented through a memory augmentation (MemAug) module, which adaptively retrieves and integrates spatiotemporally informative and question-relevan semantics from the cached full context. The MemAug module effectively preserves the contextual fidelity of long videos. In summary, Hour-LLaVe La VA achieves a trade-off between computational efficiency and context fidelity, enabling effective learning from hour-scale video instruction- instuuctionfoolloiing data.
Our main contributions are three-fold: (1) we introduce VideoMarathon, the first large-scale hour- long video instruction-folloinng dataset that contains videos ranging from 3 minutes to hour with a total duration of around 9,700 hours and 3.3M diverse QA instructions across 22 tasks; (2) We propose Houv-LLaVA, Video-LMM capable of processing hour-long videos at 1 FPS in both training and inference, supported by the proposed MemAug mechanism: (3) Training Hour- LLaVA on VideoMarathon, we achieve the best performance among open-source Video-LMMs on four well-known video-language benchmarks, including LVBench [55] (average duration: 4037s), Video-MME [14] (1021s), Long VideoBench [58] (459s), and TempCompass [36] (11s).
2 Related Work
Video Instruction-followin Data. Human annotation is costly and even limits the scale of video cntructtionfoolowiing datasets. Early attempts [25, 40] address the high cost of human annotation by using AASR-generaae subtitles, while later methods [18, 27] employ template-based QA generation on existing captioning datasets. However, these approaches often suffer from noisy, low-quality labels. Recent pipelines [7, 69, 70] leverage powerful LMMs (e.g.. GPT-4V [41]/4o [42]) to produce high-quality video captions and diverse QA pairs. Despite these advancements, most training videos remain short, limiting the models' ability to learn long-term dependencies explicitly. To this end, this work introduces a high-quality, synthetic video instruction-followinn dataset built on long-form videos, providing foundation for future research on capturing effective patterns in long videos.

---

VideoMarathon
Temporality
Spatiality
Object
Action
Scene
Event
Tempora Porepiio
Temport Duraton Percspia Spiin
Temporal Orler
Spatial Porestioe
Spaaal Resoning
Objeet Recopation
Object Armiout
Objoct Interation
Object Rossoiigg
Objert Existenoe
Object Pemtancees
Action Recogitton
Action actiivatoon
Action Sequince
Scase Recignisso
Key Event Extrscioon
Reasning
(a) Task Taxonomy of VideoMarathon Action 20.3%
MovieChar-IK
You
2.7%
8,%%
ArtistyNa K2%
10090
4000
21.3%
5091-
VVideo
6.7%
Oblect
3000
4090
Temporality 25.3%
1000
2003
Panda NMM
63.7%
Spatiality
11.1%
Duration (mia)
46
90
Event Counts
m
(b) Data Source
(c) Question Types
(d) Video Duration
Event Counting
Figure I: VideoMarathon: A diverse long video instruction-followingg dataset. (a) The dataset contains 22 diverse tasks, covering both short-form (yellow tag) and long-form (red tag) comprehension.
(b) The dataset spans diverse video source domains. (c) The dataset features a wide range of question types for long-foo form video-language modeling. (d) The dataset consists of long videos ranging from three minutes to one hour. (e) The dataset includes complex video content reflected by the number of events per video.
Long 'ideo-Languall Modeling with Language Language Models. Recent advanced Video- LMMs have shown significant promise in long video-language modeling. Two main approaches facili- tate long-form video understanding: Compression-bet based and extension-based methods. Compression- based methods compress input video tokens across temporal, spatial, and question-guiat dimensions, allowing models to process fewer but more informative video tokens. Temporal compression tech- niques include keyframe selection [45, 50] and slow-fast sampling [13]. Spatial compression involves
average pooling [70] and token merging [24]. Other methods [46, 57] apply joint temporal-spatial
compression. Question-guided methods [1 [2, 45] retain tokens based on question relevance. Following the success of long-context L.LMs [15, 32, 34, 49], extension-basee methods expand context windows for long video modeling. LongVA VA [68] processes over 200K visual tokens, and Long VILA [63] scales 2M using multimodal sequence parallelism. However, existing approaches confront a fundamental methods prioritize computation affordable and training efficiency, but inevitably suffer from information loss caused by aggressive compression; extension based methods preserve high-fidelle fideliiyity video context by sequence parallel techniques at the cost of significantly in- creased training time and GPU usage (i.e., linear growth with video length), particularly for hour-scale videos. To resolve this dilemma, we propose the Hour-LLaVa LLaVA framework, which intends to prioritize computational efficiency while keeping high-fidelity video context.
dilemma: compression-bast
Video.Marathnn A Synthetic Long Video Instruction-following Dataset
This section elaborates on details of constructing VideoMarathon, long video instruction-folloving dataset with total duration of around 9,700 hours, consisting of 3.3M QA pairs across 22 tasks.
Task Taxonomy. A comprehensive long video fostroctionnfollloiing dataset requires a well-defined and comprehensive task taxonomy. Inspired by existing lideu-language benchmarks [5. 14, 28, 43] that cover over 100 tasks, we introduce a comprehensive task taxonomy designed for long video-language understanding. The task taxonomy consists of 22 QA tasks summarized from prior benchmarks and organized into six fundamental topics (see Figure la), including temporality. spatiality, object, action, scene, and event. In particular, the temporal topic captures long-term dependencies, while the spatial topic enhances localization. Object and action topics are essential for fundamental video perception, and the scene topic offers contextual grounding and transition cues. Event topic supports long-range understanding of key video content. This task taxonomy encourages Video-LMMSs to develop both long-form and short-form video comprehension. Please refer to Figure lI for detailed descriptions of the above 22 QA tasks in Appendix A.
3

---

Video Sources. High-quality and diverse video content is essential for effective video-language modeling. To ensure a sufficient amount of long videos while maintaining quality and diversity, VideoMarathon integrates five representative public video datasets: Panda-70M [8], Ego4D [17], NctivityNet [4], YouCook2 [72], and MovieChat-IK [47]. Figure 1b presents the proportions of these video sources. Panda-70M constitutes the majority of VideoMarathon, covering diverse topics such as daily activities, sports, science, arts, transportation, travel, education, wildlife, entertainment, and industry. Domain-specifer datasets further enrich the collection: Ego4D captures real-world activities from a first-person view; ActivityNet covers a broad range of human actions; YouCook2 focuses on cooking scenarios; and MovieChat-IP 1K contributes content from movies and TV series. To enhance temporal complexity within videos, we select videos with at least three distinct events. Figure le shows the distribution of event counts per video. In total, VideoMarathon comprises 28K videos across diverse domains, supporting long-form video-language modeling.
Hierarchical Video Captioning. In contrast to short-form video QA tasks that typically rely on concise captions, long-form video QA demands hierarchical contextual understanding, including detailed clip-wise content, event-level structures, and global narratives. To meet this need, we introduce a hierarchical video captioning pipeline for capturing both fine-grained and high-level semantics over extended temporal ranges. The resulting captions serve as essential inputs for generating reliable, diverse, and context-rich QA training samples tailored to long-form video- language understanding.
Clip-Level Video Captioning. Instead of generating a single brief summary for each video clip [7, 63, 70J, VideoMarathon provides detailed descriptions (see Figure 12a) for each video clip from six perspectives: temporality, spatiality, object, action, scene, and overall summary. These chronologically ordered descriptions serve reliable contexts for the subsequent diverse QA generation. Technically, we leverage Qwen2VL-7B [54], a powerful yet lightweight LMM, as the clip-level captioner, following the prior practice [74].
Event-Level Video Captioning. We first identify event boundaries (i.e., start and end times- tamps) based on the chronologically ordered clip-level captions. Once the event boundaries are determined, all detailed captions within each event are then aggregated into a comprehensive event-level summary. Specifically, we use DeepSeek- V3 [33] to summarize the six six-perspective clip-level captions along with adjacent event descriptions into a cohesive event-level summaries. As shown in Figure 12c, this process results in detailed event-level video captions, each associated with its corresponding event boundaries.
V3
Global-Level Video Captioning. This step focuses on generating a natural-toned and global- level description for the entire video. Specifically, we feed the event-level captions along with their boundaries into DeepSeck-V3, obtaining cohesive and comprehensive video summary. Figure 12d shows that these captions provide global context for the subsequent QA generation.
Appendix A provides the detailed prompts used for the hierarchical video captioning, including Figure for clip-level video captioning, Figure for event splitting, Figure 7 for event-level video captioning, and Figure 8 for global-level video captioning.
Diverse QA Generation for Long Videos. Leveraging the multi-level video captions, we generate high-quality and diverse QA pairs across 22 tasks, spanning six major topics in both open-ended and multiple-choice formats. To achieve this, we design topic-specific prompts that incorporate (I) detailed instructions for QA generation, (2) topic-specific task descriptions (e.g., scene recognition and transitions within the scene topic), (3) clip-level captions of the corresponding topic following the chronological order, (4) the comprehensive global-level captions, and (5) high-quality QA demo examples sourced from established video benchmarks [5. 14, 28, 43]. These structured prompts facilitate the creation of contextually grounded and diverse QA pairs that reflect the unique challenges of long-form video understanding. Detailed prompt templates for open-ended (Figure 9) and multiple- choice (Figure 10) QA generation are provided in Appendix A.
Comparison. Table presents a comparison between our proposed VideoMarathon dataset and exist- ing video eestruction-folloiing datasets [7, 70]. The most significant distinction is Video.Marathon' 's substantially longer total video time and average video length. Moreover, VideoMarathon covers a broader duration range from 3 to 60 minutes, effectively bridging the gap in long-form video instruc- tion data. Also, the diverse QA tasks and the balanced distribution of open-ended and multiple-choice QA pairs enable Video-LMMs to better handle wide range of challenging real-world questions.

---

Input Video Features I-FPS Sampling
Memory Reposilory
Forgetting Mechanism
MemAug Module
ILM LLMoDeddee
User Question
11
Full Video Token
Decayed Video Token
Question Token
Memory-Augmentel Video Tokm
Final Response
Figure 2: Overview of the Hour-LLaVA Framework, Input video features H. encoded from I-FPS sampled frames are selectively decayed spatially and temporally through forgetting mechanism, producing decayed video tokens H, for efficient video modeling. Mcanwhile, video features H, are stored in a memory repository. Given the decayed tokens H. and with full video context and user question-relevant details from the memory repository, obtaining memory- augmented video tokens H,. These augmented tokens then passed with the original user question tokens Hç into the L.IM decoder generate the final response X,.
question tokens Hq. the MemAug module enhances them
Hour-LLaVa. An Efficient Hour-scale Video-LMM
Problem Formulation. Standard video QA tasks require video-language models to generate an answer X., based given input video X., and user question X, by modeling the conditional probability of pa( X,, Xx, Xg). where is the parameter of video-language models. More specifically, the input video is first fed into vision encoder (+)(-) parameterized by 0v to produce the visual feature Z. = fe, (X,). Then, projector W converts the visual feature Z. to visual embedding H. W Z. the language embedding space. Mcanwhile, the imput user question Xg projected to language embeddings H,, by word embedding model. Finally, the language model fo. (-) predicts the answer X, fo. H,, H.,) where 0g is the parameter of the language model.
Memory Augmentation for Long Video Modeling. Modeling hour-long videos poses a challenge, densely processing all frames with LLMs is impractical under GPU memory constraints. To address this challenge, we draw inspiration from the human memory system, which selectively retains and recalls essential past experiences while discarding irrelevant redundant information, striking a balance between efficiency and comprehensiveness [2, 11]. Motivated by this principle, we design memory augmentation mechanism enable hour-scale video-language understanding. This mechanism consists of three main components: memory repository, forgetting mechanism, and MemAug module. Together, they allow the model to operate on compressed representations while maintaining access to the full video context.
Memory Repository. High-fidelit video features H, extracted FPS are stored in memory repository serving as long-term memory, enabling the model to retain complete video context without requiring the LLM decoder process every frame.
Forgetting Mechanism. Due to the GPU memory constraints, we employ forgetting mechanism
Mforget to compresses the full video tokens H, into reduced set of decayed video tokens H, Mforget H.) by selectively discarding the tokens in both spatial and temporal dimensions. This prevents the training and inference cost from growing linearly with video length. In practice, the forgetting mechanism Morggt can implemented by various token compression strategies [23, 45, 46, 70]. Section 5.4 provides comprehensive comparison among different forgetting mechanisms.
MemAug Module. The MemAug module dynamically recovers the informative video tokens discarded during the forgetting stage from the memory repository. Without the MemAug module, forgetting mechanism alone would inevitably ineur irreversible context loss and lead sub-optimal performance. Technically, the MemAug module is implemented using standard transformer blocks [53]. In cross-attention, the decayed video tokens H, and user question tokens H, (as queries) attend retrieve relevant information from the memory repository H, (as keys and values), In sclf-attention, question-relevant video information flows from the user question tokens
5

---

Hg to the decayed video tokens H,, Through this process, the decayed video tokens H, are enriched with both full video context and question-speciife details, resulting memory- ronmory-augmented video tokens Hy that support long -ongdeependence modeling for video understanding:
H, frn (H,. Hy),,
(1)
where BM denotes the parameters of the MemAug module (.)
Hour-LLaVA, Powered by memory augmentation. propose Hour-LLaVA, efficient video- language model capable of modeling hour-long videos at FPS. It comprises three key modules: video encoder, memory augmentation module (i.e., MemAug), and an LLM decoder. Figure shows the Hour-LLaVa framework, with the video encoder omitted for simplicity.
Video Encoding. Following existing Video-LMMs [70, 74], we adopt SigLIP [67] followed by vision-language projector for video encoding. Specifically, we sample video frames at 1 FPS and extract visual features Z., which are then average-pooless to a fixed spatial resolution of 8 x 8 per frame. We then project the vverage-pooless visual features into the language embedding space via two-layer MLP with GELU [21] activation, obtaining the full video tokens Hy-
MemAug. The decayed video tokens H. are obtained by "forgetting" ~94% of the full video tokens with an overall compression ratio of 1/16 through the forgetting mechanism, achieved by discarding tokens with a 1/2 compression ratio along both spatial and temporal dimensions. These decayed tokens are then enriched with the full video tokens via the MemAug module, implemented with N transformer blocks. To associate the spatial-temporal correspondence between the full video tokens in the memory repository and the decayed tokens, we first flatten the full video tokens into 1D embeddings and apply 1D structured RoPE, following prior works [16, 49]. Based on their hemporltempooal alignment, we then construct corresponding ID RoPE for the decayed tokens, ensuring that each decayed token (query) carries positional information consistent with its related full tokens (keys and values) during cross-attention.
video tokens H, and the user question tokens Hgg LLM decoder generates final response Xg. We employ -wven..5-3B-Instie [65] and Qwen2-7B- Instruct [64] as the LLM decoders for Hour-LLaVA-33 and Hour-LLaVA,
LLM Decoder. Given the remory-auggmnnt
respectively.
Training Schedules. The training schedules of Hour-LLaVA follow three stages: image-language pretraining, video-language adaptation, and video instruction tuning. Table 8 presents more details of the training schedules.
Image-Language Pretraining. We perform image-language pretraining with 3B image-text pairs from the single-image subset of VLVVA-OV [26]. Full-resolution image tokens are stored in the memory repository, from which 1/4 tokens are retained decayed visual tokens via the spatial forgetting mechanism. Only the Transformer blocks of MemAug are trained for one epoch.
Video-Language Adaptation. Following the data composition that adheres to established prac- tices [70, 74], the model is adapted to video-language inputs using small amount of mixture of 0.12M single-image, 0.05M multi-image, and 0.09M text data from VL-VV. plus 0.3M short video samples from and temporal dimensions via the forgetting mechanism. All model parameters are trained for one epoch. This stage intends to slightly adapt all model parameters to the video domain, thereby constructing a strong baseline for the subsequent long-video training stage.
VIAA-Vide--111
[70]. A 1/1 token compression is applied in both spatial
Video Instruction Tuning. This stage uses content. For training efficiency, up to five QA pairs from the same VideoMarathon video are grouped as a multi-turn conversation. The corpus includes 1.14M single-image, 0.5M multi-image, and 0.8 MM text samples from LL 1.3M short video samples from LLaVV- ---V-ido----- 178K, and 0.7M long video samples from VireoMarathon. The same forgetting mechanism is applied as the previous stage. The vision encoder is frozen, while the remaining modules are fine-tuned.
data involving long video
5 Experiments
5.1 Experimental Setting
Evaluation Benchmarks. We evaluate our models on four mainstream video-language benchmarks: TempCompass [36], Long VideoBench [58], Video-MME [14], and LVBench [55]. TempCompass fo- cuses assessing the temporal reasoning ability of Video-LMMs for short videos. Long VidhoBench

---

Table 2: Performance comparison of existing LMMs on TempCompass, Long VideoBench, VideoMME, and LVBench datasets. M-Avg denotes the average performance of multiple-chooee chhicce tasks. Blue boxes denote average durations of benchmarks. Red highlights that LVBench's 's average video length exceeds the maximum length in the training stage. The symbol marks our reimplemented results. Bold font denotes the best performance among models the same scale, while underline indicates the second-best.
TempCompass
LongVideoBench
VideoMME (wlo subtitles)
LVBench
Method
Input
M-Avg
M-Avg
Overall
Medium
Long
Avg
Purams
Video
Proprietary LMM
1411
frames
61.3
59.9/63.3
55.8/59.7
53.5/56.9
GPT-4o [+]]]
384 frames Fps 0.5/1 ips
70.9
66.7
71.9/77.2
70.3/76.6
65.3/72.1
48.9
Gemini 1.5 Flash [52]
61.6
20.3/55.0
68.8/74.7
61.176888
Gemini 1.5 Pro [52]
69.3
64.0
75.0081.3
74.3/11.0
67.4/77.4
33.1
Open-source LMM (<7B)
VILAI.5-3B [30]
3B
8 Frames
56.1
42.9
42,2/44.2
Phi.3.5.Vision-44 [I]
frames
LongVU-3.2B [45]
3.20
I lps
/51.5
/47.2
IntemmL12.522 [9]
2B
framcs
$1.9/54.1
Apollo-1.58 [7-]
1,58
2 fps
60.8
$4.1
$3.0/54.6
Apollo-3B [74]
3B
2
62.5
55.1
58.4/60.6
LLaVA-OV-SI-3B [26]
3B
32 frames
55.6
49.6
$1.1/54,5
49.2/52.1
44.1/46.4
35.4
LLaVA-Videe-38 [70]
3B
framcs
55.2
58.7/60.7
55.2/57.3
47.0449.9
41.7
Hour-LLaVA-3B (owrs)
3B
fps
63.6
57.8
60.6/6677
59.0/65.4
52,1/60.4
44.7
Open-source 1.M (7-8B)
Video-LLaVA (29)
7B
8 frames
37.9
39.1
39.9/116
38.0/40.7
36,2/38.1
VideoChai2 128]
7B
frames
51.1
39.3
39.3/43.8
37.039.4
33.2/39.2
SharoGPT-Viddo [7]
8B
frames
59.4
41.8
39.9/43.6
3633/39.3
35.0/37.9
7B
47.9/50.3
33.2/39.2
Vidco-XL. [46]
7B
I fps
50.7
$5.5/61.0
Kangaroo [35]
8B
frames
54.8
66.0
55,3155.4
46/7/49.3
39.4
LongVA [68]
7B
128 frumes
52.0/54.3
50.4/5366
46.2/47.6
LongVILA
7B
256 frumes
60.1/66.1
58.164.9
53.0/7.4
LongVU [45]
7B
1 Ips
/60.9
/59.5
Apollo-78 [74]
7B
2 fps
64.9
58.5
61.3/6333
1LaVA-OV-SI-78 [26]
7B
frames
57.2
52.11
58.2/61.5
52.6' (55.4)
47.91
36.01
LLaVA- [70]
7B
frames
64.31
58.2
63.3/69.7
58.962299
53.0/55.0"
42.21
Hour-LLaVA-78 (ours)
7B
fps
68.1
60.4
63.6/70.2
63.8/70.0
55.0/65.1
45.6
consists of varying-length web-collected videos up to one hour with their subtitles across diverse themes, evaluating the abilities in retrieving and reasoning over detailed information from long videos. Video-MME is comprehensive multimodal benchmark designed to evaluate long video understand- ing across diverse video types and temporal spans. LVBench challenges models to demonstrate long-term memory and extended comprehension across multimodal inputs.
Implementation Details. Following existing practices [45, 68, 70], we initialize our Hour-LLaVA models with pretrained Image-LMMs. Specifically, the vision encoder and the LLM decoder of Hour-LLaV-77B are initialized from
[26]. which is trained solely image data. Due to the absence of LLaVA-OV-SI-3B. we pretrain a 3B version of LLaVA-OV. SV-SI using Qwen2.5- 3B-Instruct model, and then initialize LLar--L from it. For rideo-languel training, set the global batch sizes to 128 and 256 for the 3B and 7B models, respectively. A learning rate of 2e-5 is used with 0.03 warmup ratio under cosine annealing schedule. The models optimized using the AdamW W [37] optimizer with a cross-entropy loss. We train Hour-L.LaVA-3B with 64 AMD MI250 GPUs and Hour-LLLaVA-7B with 64 AMD MI300X GPUs, respectively. We conduct all the ablation studies using Hour-LLaVA-333 in Section 5.3 and 5.4. Please refer to Appendix A.3 for more implementation details.
5.2 Main Results
Overview. As shown in Table 2, Hour-LLaVA consistently achieves the best performance on four well-established video benchmarks in both the 3B and 7-8B model size categories. Notably, Hour-LLaVA-3B even surpasses more than half of the current 7-8B Video-LMMs.
TempCompass. The results TempCompass show that -Lar- maintains strong performance on short-form video-language tasks, even after introducing long langual training samples.
Long Vingeedeeench Hour-LLaVA demonstrates state-of-the-ar performance among open-source models across both the 3B and 7B parameter scales on Long VideoBench. Specifically. Hour-LLaVA, VA- 3B and Mour-LLaVA-71 outperform the second-best model by 2.6 and 1.9 points, respectively.

---

66.5
Moted: Heve LaVe
600
535
235
530
Dia Ste fann I5 VAL..
525
565
380
360
(a) TempCompass (Short)
(h) Long VidcoBench (Long)
(c) LVBench (Long)
Figure 3: Dataset ablation and methodology comparison. The analysis is evaluated three benchmarks: (a) TempCompass, (b) Long VideoBench, and (c) LVBench. It presents performance of Hour-LLLVA- and LLaVA- Video-3B models. The x-axis represents different training data mixture configurations, with exact ratios indicated in the legend. V.M. and L.V. refer to the VideoMarathon and LIVA--ideo-1788 datasets.
VideoMME. Hour-LLaVA consistently outperforms other open-source models at both the 3B and 7B scales on VideoMME. Remarkably, Hour-LLaVA achieves significantly higher performance both medium- and long-length video settings, highlighting its strong capability in long video understanding
LVBench. The maximum video length of LVBench is more than two hours, and its average video length (67 minutes) already exceeds the maximum video length in our training (60 minutes). As such, this benchmark serves as a pressure test for long video understanding and highlights the model's ability to generalize beyond its training distribution. As shown in Table 2, Hour-LLaVA achieves leading performance, outperforming the second- best models by 3.0 and 3.4 points under the 3B and 7B settings, respectively.
Comparison with Baseline Models. LLaVA-OV-SI [26] serves the image-language pretrained model for Hour-LLaVA, while both ILLVA- [70] and Hour-LLat LL adopt the same L.MM decoder. The results show that Hour-LLavA considerably enhances the performance of its pretrained model across both 3B and 7B model scales. Furthermore. Hour-LLaVA consistently outperforms its strong peer, VLLVA-Vided especially in medium- and long-video scenarios.
5.3 Analysis on VideoMarathon Dataset
Comparison between VideoMarathon and LLaVA-Video-11755 We compare our proposed Video- Marathon dataset with LLaVA ViV--iidoo--788 [70], the largest publicly available video instruction- tuning dataset date. For a fair comparison, we construct two subsets from VideoMarathon (V.M.) and VaaaVA-Vieeoo
(L. matched in both the number of videos and the number of video- language instruction samples. Specifically, we randomly select 70K inslruction-follovingg samples over 10K videos from each dataset. To support multimodal learning, both datasets are further combined with the same 30K single-image, 10K multi-image, and 20K text-only samples from LLaVA-OV [26].
Figure illustrates how performance varies with different mixtures of V.M. (long-video) and L.V. (short-video) training data. On the far left of the x-axis, the model is trained solely on 70K short-video samples from L.V.; moving right, short- video samples are gradually replaced by long-video samples from V.M., reaching 100% V.M. on the far right. The pie charts visualize the data mix ratios for each setting. As shown in Figures 3b and 3c, Houv-LLaVA (-+) achieves progressively better performance as the proportion of long-video data increases, peaking when the long-to-short ratio reaches 3:1, and then declining once L.V. data is entirely absent. These results highlight the crucial role of VideoMarathon in enhancing long video-language understanding. They also suggest that mixing different datasets can be beneficial even when the training and testing video lengths differ, possibly due to the increased diversity introduced by heterogencous data sources. In addition, Figure 3a shows that incorporating long videos from VideoMarathon in training does not considerably affect performance on short video-language understanding. In the following ablation experiments, use the same data recipe, along with mixture of long- and short-video at a 3:1 ratio.
Sparse Sampling Limits Long Video-Language Learning. Using the VideoMarathon dataset, we also train LLaVA- -Vdeo [70], a representative Video-LMM that adopts sparse temporal sampling by uniformly selecting 64-frame video features as input to the LLM decoder. As shown in Figure 3, -LAVAViideo (-a-) fails to benefit from an increased proportion long-video samples, with its performance even declining after long-video training. This result indicates that sparse sampling

---

Table 3: Comparison spatial forgetting strategies.
Table Comparison of temporal forgetting stralegies.
Spatial Forgetting
Tokens/ing MMStar SciQA R.W.QA
Temporal Forgetting
Long VideoBenchh
LVBench
LLLVA OV-S1-3B (Base)
729
52.8
84.7
58.8
Uniform
Keyframe
53.5
54.0
40.3
40.6
+ Random
196
51.9
84.5
59.6
Question-uuided
53.4
Uniform
196
83.7
Random
53.2
38.8
Score oo L.Boch
Score on LNBrasch
Lia ILaN Vileo
#Tokems (K)
38
Samplling
26-
2000 Video Duration (9)
Tem poral Compresiio Ratio
Memory Repasitory Scale
Figure 4: Impact of compression ratio for temporal forgetting (left), and memory repository scale (middle). Comparison of the number of visual tokens input to the LLM decoder (right).
insufficient to capture the effective patterns required for long video-language understanding. In contrast, Hour-LLaVA (-+)) effectively achieves long-video language modeling by leveraging access to a memory repository that stores full dense video content at FPS. These results highlight the urgent demands for new training paradigms tailored to long video-language modeling, and position VideoMarathon as a practical platform for developing them.
5.4 Analysis on Key Components of Hour-LLaVA
Forgetting Mechanisms. The forgetting mechanism is proposed to compress full video tokens into decayed video tokens reduce the number of tokens processed by the LLM decoder. The decayed tokens will then be augmented by the MemAug module to retain or recall the informative semantics. The forgetting mechanism
be implemented with different token compression strategies [23, 45, 46, 70] in both spatial and temporal domains. We ablate different forgetting mechanisms guided by the MemAug module as follows:
Spatial Forgetting (SF). We compare two straightforward methods: uniform SF (discarding
tokens regular intervals over the 2D image) and random SF (discarding tokens randomly) with a compression ratio of 1/4. We conduct this ablation during the image-language pretraining stage and evaluate on three popular image-language benchmarks: MMStar [6], ScienceQA [38] (Sci.QA), and RealdOoIA [59] (R...QA). Notably, despite using only 1/4 tokens, both SF strategies achieve comparable performance to the base model without token compression shown in Table 3. It highlights the success of our MemAug module for token compression in the image domain, suggesting that MemAug also be applied to Image-LMMS for token compression. For Hour-LLaVA, we adopt random SF with 1/4 compression ratio for its simplicity and effectiveness.
Temporal Forgetting (TF). We evaluate four temporal compression approaches: random, uniform [23, 35, 62, 70], keyframe- based [45, 50], and dnesttio-guidee temporal compres- sion [45, 46]. All strategies are evaluated using a compression ratio of 1/4- As shown in Table 4, uniform TF yields the best overall performance, suggesting that the MemAug module is effective without relying on manually designed compression strategies. Furthermore, we analyze the effect of different compression ratios for temporal forgetring LVBench, ranging from 1 down to 1/1 as shown Figure 4 (left). We adopt the uniform TF with a ratio of 1/4 for Hour-LLaVA, balancing the trade-off between performance and efficiency introduced by temporal compression.
Token Control for Videos with Extreme Lengths. Although forgetting mechanisms significantly reduce the tokens processed by the LLM decoder, they remain insufficient for extremely long videos (e.g., 24-hour videos). We set the maximum number of retained frames 512 so to constrain the computational cost. Conversely, for extremely short videos, we enforce minimum of 32 retained frames to preserve sufficient contextual information. Note that these thresholds apply to decayed tokens only, which means Hour-LLaVA still accesses the full video context at I-FPS sampling via memory repository. Figure (right) compares the number of visual tokens fed into the LLM decoder across LL,,, LLa Videoo, and vanilla I-FPS sampling methods. It highlights that Hour- Hour-LLaVA consistently uses fewer tokens than others.

---

Table 5: Impact of MemAug module on different token Table 6: Impact the number of MemAug blocks. compression strategies.
M.A. refers to MemAug.
Token Compression
MemAug
LongVideoBench
LVBench
M.A.. Blocks TempCompass
Long Vide LVBeuch
(+1.9)
(+2.))
Uniform
52.1
38.3
59.4
53.2
40.0
Keyfrume
/
(+1.5)
40.3 (+..4)
59.8
53.5
40.3
X
52.0
38,9
59.7
54.0
40.6
Question Question-uided
X
53.4 (+2.4)
$1.0
37,5
(+2.2)
53.4
40.5
MemAug Module. The MemAug module is designed to dynamically recover the informative video tokens from the memory repository. To retain informative tokens from full videos, previous studies have explored various video token compression approaches [12, 45, 50]. However, these approaches typically rely on hand-crafted rules (e.g., predefined thresholds) and inevitably suffer from irreversible information loss. In contrast, the MemAug module performs video context compression through learnable process supervised by video instruction data.
Impact of MemAug under Different Compression Strategies. To validate the effectiveness of MemAug, we apply MemAug module on several representative video token compression methods [12, 45, 50], including uniform [23, 35, 62, 70], keyframe-baset [45, 50], and question- guided [45, 46] temporal compression. For fairness, all methods feed the same total number of apply random spatial compression with ratio of 1/4 and various temporal compression techniques with ratio of 1/4 on 1-FPS input videos. As shown in Table 5, models equipped with MemAug consistently achieve 1.4-2.4% empirical gains over those without across all token compression strategies, demonstrating that MemAug effectively re-injects informative cues. Without MemAug, conventional token compression approaches incur irreversible context loss, resulting sub-optimal performance. As a result, by coupling MemAug with forgetting mechanism, Hour-LLaVA delivers compression-level efficiency while still preserving the sig-fiielitty video content.
compressed tokens to the LLM decoder. Specifically,
Robustness of MemAug Module. Notably, Table 5 presents that the performance with MemAug remains comparable regardless of the underlying token compression approaches, indicating that MemAug is robust and largely insensitive to the choice of token forgetting mechanism. This consistency demonstrates that MemAug effectively adapts to distinct temporal compression paradigms, ensuring reliable information recovery even under different token compression settings.
Effect of the number of MemAug blocks. We evaluate 1, 2, 4, and 8 MemAug blocks on three benchmarks. Table 6 demonstrates that performance improves up to 2 or 4 blocks and then decreases slightly at More specifically, for short videos (TempCompass), the sweet spot lies in the range from 2 to 4 blocks, while for medium- and long-form videos (Long VideoBench and LVBench), blocks provides the optimal results. Consequently, we adopt MemAug blocks as the default.
Memory Repository. We investigate the impact of the memory repository scale on performance on LVBench. Figure 4 (middle) shows that the performance of Hour-LLaVA decreases as the memory repository scale becomes smaller. This trend indicates that information loss in the memory repository limits the capacity of long video understanding. Therefore, to preserve as much temporal information as possible, we retain video tokens sampled FPS in the memory repository.
Please refer to the Appendix A.3.2 for more details of the above experiments.
6 Conclusion
In this study, we introduce VideoMarathon, a large-scale video Destruction-folloiing dataset. Com- prising long videos with a total duration of around 9,700 hours and 3.3M QA pairs, VideoMarathon covers 22 challenging tasks that require both short- and long-term video understanding. Building on VideoMarathon, we propose Hour-LLaVA an efficient and powerful Video-LMM optimized for hour-scale video modeling. By leveraging a memory augmentation (MemAug) mechanism, Hour- LLaVA effectively integrates information from the full video context while maintaining efficiency through 1-FPS sampling. Extensive evaluations on several video-languan benchmarks validate the high quality of the VideoMarathon dataset and the superiority of the Hour-LLa VA model.
10

---

References
[1] Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jiammin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone, 2024, arXiv:2404. 14219, 2024.
[2] Alan Baddeley. Working memory, Memory, 2020.
[3] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv:2502. /3923, 2025.
[4] Fabian Caba Heilbron, Victor Escorcia, Berard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR, 2015.
[5] Keshigeyan Chandrasegaran, Agrim Gupta, Lea M Hadzic, Taran Kota, Jimming He, Cristóbal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Li Fei-Fei. Hourvideo: 1-hour video-language understanding. In NeurlPS, 2024.
[6] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang. Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large ssiion-language models? In NeurlPS, 2024.
[7] Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understanding and generation with better captions. In NeurIPS, 2024.
[8] Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon. Yuwei Fang, Hsin- Ying Lee, Jian Ren, Ming- Asuan Yang, al. Panda-70m: Captioning 70m videos with multiple cross- modslity teachers. In CVPR, 2024.
[9] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundarics of open-source multimodal models with model, data, and test-time scaling. arXiv:2412.05271, 2024.
[10] Zesen Cheng, Sicong I Leng, Hang Zhang. Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-Ilms. arXiv:2406.0744 072440000777... 2024.
[11] Nelson Cowan. What are the differences between long-term, short-lerm, and working memory? Progress in brain research, 2008
[12] Shangzhe Di. Zhelun Yu, Guanghao Zhang. Haoyuan Li, TaoZhong, Hao Cheng, Bolin Li, Wanggui Hc, Fangxun Shu, and Hao Jiang. Streaming vidco questio--answrrigg with in-context video KV-cache retrieval. In ICLR, 2025.
[13] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In CVPR, 2019.
[14] Chanyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In CVPR, 2025.
[15] Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context. 27Xiv:2402.10171, 2024.
[16] Mingze Gao, Jingyu Liu, Mingda Li, Jiangtao Xie, Qingbin Liu, Bo Zhao, Xi Chen, and Hui Xiong. Te-llava: Rethinking the transfer from image to video understanding with temporal considcrations. 2025.
[17] Kristen Grauman, Andrew Westhury, Eugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, al. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, 2022.
[18] Madeleine Meande-MLLaughhin Ranjay Krishna, and Maneesh Agrawala. Agga: benchmark for compositional spatio-temporal reasoning. In CVPR, 2021.
[19] Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang. Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large rision-languag models. In CVPR, 2024.
11

---

[20] Michael Gygli, Helmut Grabner, Hayko Riemenschneider, and Luc Van Gool. Creating summaries from videos. In ECCV, 2014.
[21] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv:1606.08411 084110 2016.
[22] Hang Hua, Yunlong Tang, Chenliang Xu, and Jiebo Luo. V2xum-Ilm: Cross-modal video summarization with temporal prompt instruction tuning. In AAAI, 2025.
[23] Jindong Jiang, Xiuyu Li, Zhijian Liu, Muyang Li, Guo Chen, Zhiqi Li, De-An Huang, Guilin Liu, Zhiding Yu, Kurt Keutzer, et al. Token-efficient long video understanding for multimodal llms, arXiv:2503.04I: 2025.
[24] Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In CVPR, 2024.
[25] Sangho Lee, Jiwan Chung, Youngjae Yu, Gunhee Kim, Thomas Breuel, Gal Chechik, and Yale Song. Acav100m: Automatic curation of large-scale datasets for audio-visus video representation learning. In ICCV, 2021.
[26] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.0332 2024.
[27] KunChang Li, Yinan He, Yi Wang. Yizhuo Li, Wenhai Wang. Ping Luo, Yali Wang. Limin Wang. and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint 2rXiv:3305.06355. 2023.
2023.
[28] Kunchang Li, Yali Wang. Yinan Hc, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mybench: A comprchensive multi-modal video understanding benchmark. In CVPR, 2024.
[29] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning. Peng Jin, and Li Yuan. Video-ILLaA: Learning united visual representation by alignment hefore projection. In EMNIP, 2024.
[30] Ji Lin, Hongxu Yin, Wei Ping. Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In CVPR, 2024.
[31] Jingyang Lin. Hang Hua, Ming Chen, Yikang Li. Jenhao Hsiao, Chiuman Ho, and Jiebo Luo. Videoxum: Cooss- visual and textural summarization of videos. IEFE Transactions on Multinedia, 2023.
[32] Jingyang Lin, Andy Wong, Tian Xia, Shenghua He, Hui Wei, Mei Han, and Jicbo Luo, Facilitating long context understanding via supervised chain-of-thought reasoning. arXiv:2502. 13127, 2025.
[33] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv:24/2.19437. 2024.
[34] Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context. 2rxiv223...0188, 2023.
[35] Jiajun I Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xiaoqi Ma, Xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. Kangaroo: A powerful video-language model supporting long-context video input. arXiv:2408. 15542, 2024.
[36] Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang. Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, Hou, Andre Martins, and Vivek Srikumar. TempCompass: Do video LLMs really understand videos? In ACL Findings, 2024.
[37] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[38] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chur Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NewrlPS, 2022.
[39] Alexander Martin, Reno Kriz, William Gantt Walden, Kate Sanders, Hannah Recknor, Eugene Yang, Francis Ferraro, and Benjamin Van Durme. Wikivideo: Article generation from multiple videos. M0KV:22504000339,200
[40] Antoine Miech, Dimitri Zhukov, Jean- Bapt-sBtttteeeee Alayrac Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howtol 100: Learning a text-video embedding by watching hundred million narrtedd video clips. In CVPR, 2019.
12

---

[41] OpenAL. Gpt-4v,
2023.
[42] OpenAL. Hello 4o, https://ppenat..omminnexx//////
2024.
[43] Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse. Skanda Koppula, Mateusz Malinowski, Yi Yang. Carl Doersch, et al. Perception test: A diagnostic benchmark for multimodal video models. In NeurlPS, 2023.
[44] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML., 2021.
[45] Xiaoqian Shen, Yunyang Xiong. Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu. Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al. Longvu: Spatiolemporal adaptive compression for long video-language understanding. arXiv:2410. 17434, 2024.
[46] Yan Shu, Peitian Zhang, Zheng Liu, Minghao Qin, Junjie Zhou, Tiejun Huang, and Bo Zhao. Video-xl: Extra-long vision language model for hour-scale video understanding. In CVPR, 2025.
[47] Enxin Song. Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In CVPR, 2024.
[48] Yale Song. Jordi Vallmitjana, Amanda Stent, and Alejandro Jaimes. Tysum: Summarizing web videos using titles. In CVPR, 2015.
[49] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 2024.
[50] Reuben Tan, Ximeng Sun, Ping Hu, Jui-hsien Wang, Hanieh Deilamsalehy, Bryan A Plummer, Bryan Russell, and Kate Saenko. Koala: Key frame-conditioned long video-Ilm. CVPR, 2024.
[51] Yunlong Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Video understanding with large language models: survey. IEEE Transactions on Circuits and Systems for Video Technology, 2025.
Lin, Rongyi Zhu,
[52] Gemini Team, Rohan Anil, Sebastian Borgeaud, Caan-Bapiiste Alayrac Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katic Millican, et al. Gemini: a family of highly capable multimodal models. arXiv:2312.1 /805, 2023.
[53] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurIPS, 2017.
[54] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing laiiunla model's perception the world any resolution. arXiv:2409. 12/91, 2024.
[55] Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang. Ji Qi. Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, et al. Lybench: An extreme long video understanding henchmark. 2rXiv:2406.08.33 2024.
[56] Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma. Haian Huang, Jianfei Gao, al. Internvideo2. 5: Empowering video milmss with long and rich context modeling, arXiv preprint arXiv:2501. 12386, 2025.
[57] Yuetian Weng. Mingfei Han, Haoyu He, Xiaojun Chang, and Bohan Zhuang. Longylm: Efficient long video understanding via large language models. ECCV, 2024.
[58] Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. In booktitle, 2025.
[59] XAI. Realworldga. httppp///uggggnggeeeeeesaaasaaaa
2024. [60] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- to explaining temporal actions. In CVPR, 2021.
[61] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In ACM Multimedia, 2017.
13

---

[62] Mingze Xu, Mingfei Gao, Zhe Gan, Hong-Yor You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, and Afshin Dehghan. Slowfast-llava: strong training-free baseline for video large language models. MxXiv:2407.1884 2024.
[63] Fuzhao Xue, Yukang Chen, Dacheng Li. Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang. Haotian Tang. Shang Yang. Zhijian Liu, Longvila: Scaling cong-context visual language models for long videos. arXiv:2408. 10188, 2024.
[64] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. 1XXi:2440.116711 2024.
[65] Yang, Baosong Yang. Beichen Zhang, Binyuan Hui. Bo Zheng. Bowen Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv:2412.15113 2024.
[66] Zhou Yu, Dejing Xu, Jun Yu. Ting Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: dataset for understanding complex web videos via question answering. In AAAI, 2019.
[67] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023.
[68] Peiyuan Zhang, Kaichen Zhang. Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang. Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv:2406. 2024.
[69] Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li. Alexander Hauptmann, Yonatan Bisk, et al. Direct preference optimization of video large multimodal models from language model reward. 011522404.01258, 2024.
[70] Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data, arXiv:2410.02713. 2024.
[71] Junjie Zhou, Yan Shu. Bo Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin, Xi Yang. Yongping Xiong. Bo Zhang, et al. Mlvu: Benchmarking multi-task long video understanding, In CVPR. 2025.
[72] Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. AAAI, 2018.
[73] Jinguo Zhu, Weiyun Wang. Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes open-source multimodal models. arXiv preprint arXiv:2504. 104999 2025.
[74] Orr Zohar, Xiaohan Wang, Yann Dubois. Nikhil Mehta, Tong Xiao, Philippe Gansen-Estruce Licheng Yu. Xiaofang Wang. Felix Jue Ning Zhang, et al. Apollo: An exploration of video understanding in large multimodal models. In CVPR, 2025.
14

---

A Appendix
This document provides more implementation details of the VideoMarathon dataset construction, additional experimental details, limitations, and broader impacts, organized as follows:
Details of VideoMarathon Construction (Section A.I). We provide the exact prompts and examples for hierarchical video captioning and topic-specific QA generation.
Human Evaluation on VideoMarathon (Section A.2), We conduct the human evaluation on the
proposed VideoMarathon dataset.
More Experimental Details (Section A.3). We present additional experimental details, including detailed training schedules for Hour-LLaVA and experimental settings of ablation studies in Sections 5.3 and 5.4.
Additional Empirical Analysis (Section A.4). We provide additional empirical analysis to support the superiority of the proposed Hour-LLaVA framework.
Limitations (Section A.5). We discuss several limitations of this study.
Broader Impacts(Section A.6). We analyze both the potential positive societal impacts and the negative societal impacts of this work.
A.1 Details of VideoMarathon Construction
A.1.1 Details of Hierarchical Video Captioning
Prompts. We provide the exact prompts used for hierarchical video captioning, including clip-level captioning (Figure 5), event-level captioning (Figure 7), and global-level captioning (Figure 8). Additionally, the prompt used for event splitting is presented in Figure 6.
Examples. Figure 12 presents several examples of hierarchical video captions, including clip-level, event-level, global-level video captions, and event splits. In addition, Figure 13 presents the word cloud of all the global-level video descriptions in the VideoMarathon dataset.
A.1.2 Details of QA Generation
Prompts. We present the exact prompts used for topic-specific QA generation, including the open- ended (OE) QA prompt in Figure 9 and the multiple-choiee (MC) QA prompt in Figure 10. Also, the detailed descriptions of the 22 sub-tasks across six core topics are provided in Figure 11. Please refer to the data file for exact QA examples from the VideoMarathon dataset.
A.2 Human Evaluation on VideoMarathon
To quantify the data quality, we manually check 330 randomly selected VideoMarathon QA pairs, including 80 multiple-choice and 250 open-ended samples.
For the Multiple-Choice (MC) QA samples, we assess the accuracy of the synthetic samples by assigning score of if there is exactly one correct answer. A score of is given if the question irrelevant to the video. if no correct option exists, if multiple options are correct. The human evaluation on 80 samples achieves an accuracy of 78.7%, which demonstrates the reliability of the synthetic multiple-choice QA samples.
For the Open-Ended (OE) QA samples, we validate the data quality from two perspectives:
Yes/No Bias: Yes/No Bias evaluation [19] validates the extent to which the models tend respond with "yes". particular, the evaluation consists of "Yes Percentage Difference" (Pet. Diff) and "False Positive Ratio" (FP Ratio) metrics. Smaller Pct. Diff (closer to 0.0) reveals less bias, which means that the predicted number of "Yes" is closer to the ground truth. FP Ratio measures how likely the model responds with "Yes" out of all incorrect responses, where value closer to 0.5 indicates less bias. We randomly select 200 Yes/No QA samples. As a result, Pet. Diff reaches 0.04 and FP Ratio achieves 0.40, which reflects limited yes/no bias in the VideoMarathon dataset.
Hallucination: 20 volunteers evaluate the quality of 50 randomly selected OE QA samples. Each sample is rated on scale from 10 based on the following criteria. Score denotes a totally irrelevant response, while Score 10 indicates almost perfect answer with no hallucinations.
15

---

Table 7: Human evaluation of hallucination open-ended QA samples in VideoMarathon dataset.
Score
2
3
5
6
7
8
10
Ratio (%) 2.22
2.44
1.78
3.56
4.67
4.67
13.33
22.67
18.00
26.67
Higher scores correspond to fewer hallucinations. As show in Table 7, human evaluation achieves an average score 7.81 0.59 (based on ratings from volunteers over 50 OE QA samples), validating the quality of OE QA samples in the VideoMarathon dataset (Scores above 7.5 are considered largely correct). Notably, the score distribution shows that over 80% of the samples receive scores 7. To further support the claim "Scores above 7.5 are considered largely correct", we present examples with averaged scores between 7-8 below:
Exemple
Question: Where is the vonan positioned relative to the man in the scene vhere they are sitting on the
Syathetic Ansver: The voman sitting next the the couch.
Ground Truth: The wonan is eitting to the right of the man on the couch.
Averaged Score: 8.0
Example
Question: After the boy lights the candle in Clip, what does the woman do? 1it. Ground Truth: She starts to and holds the candle.
Synthatic Ansver: Sha holda
Averaged Score: 7.3
Example
sewing nachine Syathstic Ansver: She is soving the cuffed han onto the pants.
Question:
voman doing
Ground Truth: sewing the han of the panta.
Averaged Score: 7.5
A.3 Additional Experimental Details
A.3.1 Detailed Training Schedules for Hour-LLLaVA
In Section 4, briefly introduce the training schedules of Hour-LLaVA, Furthermore, Table presents the detailed training schedules for each training stage of Hour- LLaVA, containing compres- sion details, data usage, and training hyperparameters.
Table 8: Detailed training schedule for each training stage of Hour-LLa including compression details, data usage, and training hyperparameters. In Data part, the following abbreviations: T for text, SI for inngle- MI for multi-image, SV for short-video, LV for long-video, OV-SI for LLaVA-OV-SI [26], L.V. for LLaVA-Vided 178K [70], and V.M. for VideoMarathon.
Image- Language Pretraining
Language Adaptation
Tuning
3B
7B
3B
7B
3B
7B
Compresion
Spatial Forgetting (SF)
SP Mechanism
Random
Random
Random
Random
Random
Random
Temporal Forgetting
TF Mechanism
Uniform
Uniform
Uniferm
Unifoom
Type
SV
T OV-S1 L.N.
SE
Data Sonrees
OW-S1
CW-S1
OV-I L.V.
L. VM.
OVS + VM.
#Samples
AB
0.6M
0.6M
4.M
4.4M
Ratch
128
Traning
LR of Vision Encoder
0
x000-
x10-=
Projector
x100
LR of MemAug
xx10
x10
xx10
x100
1x10
1x10
Epoch
A.3.2 Experimental Settings of Ablation Studies.
Due to page limitations, we unable to include all experimental settings for the ablation studies in Section 5. Below, we provide the experimental settings for each ablation study. We conduct all the ablation studies using the LLaV--L model.
16

---

Ablation Study for VideoMarathon (Section 5.3). In this section, we investigate how performance changes with different mixtures of VideoMarathon (long-video) and LLaVA- Video- 178K (short- video) training data. We construct two subsets from VideoMarathon and I.L. VA-Vieo- matched in both the number of videos and the number of video-language instruction training samples. Each sub-dataset consists of 70K video-language instruction samples over 10K videos. We then train the Hour-LLaVA-333 model on different mixtures of these two subsets (as shown in Figure 3), while also incorporating the same 60K multimodal samples from LLaVA-OV [26] (i.e., 20K text-only, 30K single-image, and 10K multi-image samples) to support multimodal learning. In addition, the Hour-LLaVA-3B model is initialized from the checkpoint obtained after image-languas pretraining (Stage in Table 8). All training hyperparameters follow the settings used for video instruction tuning of the 3B model, as detailed in Table 8.
Analysis on Key Components of Hour-LLavA (Section 5.4). In this section, we primarily conduct analysis on three key components in Hour-LLa VA: forgetting mechanisms, MemAug module, and memory repository.
We ablate forgetting mechanisms from both spatial and temporal perspectives.
Spatial Forgetting. In this setting, we adopt 3B VLaVA-OV-SI data for training. The vision encoder, projector, and LLM decoder of the Hour-LLaVA, LLVVA33B model are initialized by a pretrained LLaVA-OV-SI-3B model as mentioned in Section 5.1. During training, only the parameters of the MemAug module are tuned, while the rest of the model remains frozen. Training hyperparameters follow the settings used for image-language pretraining of the 3B model in Table 8.
Temporal Forgetting. We compare several representative video token compression techniques, including uniform| [23, 62], keyframe [45, 50], and question-guided [12, temporal compression. Also, we implement random temporal compression as reference baseline. In particular, the implementation details of different temporal compression methods are shown below:
Random. We randomly select 1/4 frames from a video sampled at 1 FPS.
Uniform. We uniformly sample 1/4 frames across the temporal dimension from a FPS video. Keyframe. Following [45], for each frame, we compute the average cosine similarity with its K nearest temporal neighbors (with K 8). We then retain the 1/4 frames that are least similar to their neighbors, removing the 3/4 most redundant frames. In addition, video frame features are extracted by SigLIP vision encoder [67].
Question-guided. Following [12], we calculate the cosine similarity between the frame embedding and question embedding q. Next, we select 1/4 frames with the highest similarity scores. In particular, the frame embedding v is computed as v = 1 E.N, vs where N, is the number of tokens per frames and v; refers to the i-th visual token vector. The question embedding is calculated as q = N. Σi
N.
- q: where N, is the number of tokens in the given question and q, denotes the i-th query token vector. Additionally, v, is obtained using the SigLIP vision encoder followed by the projector, while q, is derived from the Qwen2.5-3B word embedding model.
For a fair comparison, all the above temporal compression techniques apply different temporal compression strategies with the same compression ratio of 1/4 on I-FPS input videos, along with a random spatial compression with compression ratio of 1/4. Note that for training, we use dataset composed of 70K langu-ln instruction samples with a 3:1 mixture of VideoMarathon and LLa Video- along with 60K multimodal samples from LLa The model is initialized from the checkpoint obtained after image-language pretraining. All training hyperparameters follow the video instruction luning setup for the 3B model.
For MemAug module, we adopt the same training data and recipe as mentioned for ablation studies on forgetting mechanisms. All training hyperparameters are consistent with the video instruction tuning configuration of the 3B model in Table 8.
For memory repository, we mainly analyze the impact of the memory repository scale in Figure 4 (middle). The 100% scale refers to storing full video tokens sampled at FPS. For the 50% and 25% scales, we uniformly retain 50% and 25% of the frame-level features along the temporal dimension, respectively. The <10% scale denotes lightweight configuration in which only decayed video tokens are replaced in the memory repository, resulting in fewer than 10% of the full video token count. For the training, we follow the same setup as mentioned above. All training hyperparameters consistent with the video instruction tuning configuration of the 3B model in Table 8.
17

---

A.4 Additional Empirical Analysis
Impact of ID Structured RoPE. In MemAug module, we adopt ID structured RoPE to associate the spatial-temporal correspondence between the full video tokens in the memory repository and the decayed tokens, following prior works [16, 49]. We conduct ablation study on the ID structured RoPE on temporal reasoning subset of VideoMME dataset, shown in Table 9. Results show substantial performance drop on the temporal reasoning task when this structural RoPE is removed. demonstrates that constructing the spatial-temporal association between the full video tokens and decayed video tokens is necessary.
Table 9: Effect of ID structure RoPE within the MemAug module.
Method
ID structure RoPE
VideoMME (Temporal Reasoning)
Hour-LLaVA-3B
46,3
X
35.6 (-10.)
Hau--LJaVA-7B
56.5
38.4 (-12.1)
Performance on Free-form Generation Tasks. We evaluate Hour-LLaVA-33 and Hour-LLaVA-7B on MLVU-Dev [71] with free-form generation tasks, including sub-science captioning (SSC) and video summarization (SUM). The generated results are measured by GPT-4-Turbo with carefully designed instructions, where the rating score is on 2-10 scale. We also report results from other state-of-the-ard Video-LMMs for reference. In Table 10, the results demonstrate the strong capability of LLa on free-form generation tasks.
Table 10: Performance on free-form generation tasks (MLVU-Dev) evaluated by GPT-4-Turbo. SSC: Sub-Scene Captioning: SUM: Video Summarization.
Method
Overall
SSC
SUM
MovieChat
2.78
3.23
2.33
LLaVA-1.6
3.23
4.35
2.11
ShareGPT4Vidoo-78
3.77
5.02
2.52
Long VA-7B
4.33
5.26
3,39
Video-XL-7BB
4.50
Hour Hour-LLaVA-33 LLaVA-3B
4.48
5.04
3.91
Hour LLB7
4.60
5.22
3.97
Comparison with recent advanced Video-LMMs. In Table 11, we present comparison with the
most recent advanced Video-LMMSs, including Qwen2 Vee [54), Qwen2.5-VI. [3], InternVI.3 [73], and Intern Video2.5 [56]. Compared among 3B-LMMs, Hour-LLaVA-3B substantially enhances its pretrained model LLLLVA----S---- and is generally comparable to its strong peer Qwen2.5- VL-3B. Compared with the 7B-LMMs. Hour-LLaVA- VA-7B markedly enhances its pretrained model (LLaVA-OV-SI-7B) and surpasses Qwen2-VL-PP VL-7B. also performs comparably with, and sometimes outperforms, state-of-the-art models built on the stronger Qwen-2.5-7B LLM decoder.
A.5
Limitations
Despite the promising results of Hour-LLaVA, several limitations remain. First, due to the lack of comprehensive evaluation metrics for hour-long video-language understanding, multiple-choice QA remains the most practical task for evaluating long video-language models. However, this evaluation format limited in scope and fails to assess the broader capabilities of Video-LMMs. The development of more diverse and holistic benchmarks is therefore essential for advancing this field. data, which inevitably
Second, Hour-LLavA is trained on large-scale synthetic cnatructionfoollwweee contains noise. The current training pipeline does not explicitly consider this issue, and future work can further explore noise-robust training strategies. Third, the current framework is limited to video and language modalities, neglecting audio, crucial component in many long-form videos such as lectures, interviews, and documentaries. Incorporating audio or additional modalities could further enhance the model's capacity for comprehensive multimodal understanding.
18

---

Table 11: Performance comparison of existing LMMs LongVideoBench, VideoMME, and LVBench datasets. M-Avg denotes the average performance multiple-chool choice tasks. symbol marks our plemented results. Bold font denotes the best performance among models at the same scale, while underline indicates the second best.
LongVideoleench
VideoMME (w/o & w/ subtitles)
LVBench
Method
M-Avg
Overall
Medium
Long
Avg
LLM Decoder: Qwen2.5-3B
LLaVA-OV-SI-3B [26]
49.6
51.1/54.5
49.2/52.1
44.1/46.4
35.4
Owen2.5-VL-3B [3]
54.2
61.5/67.6
58.7 /66.3'
51.9' /61.6'
43.3
Hour-LLaVA-3B
57.8
60.6/66.7
59.0/65.4
52.1/60.4
44.7
LLM Decoder: Qwen2-7B
LLaVA-OV-SI-7B [26]
52.1
58,2/61.5
52.6 /55.4
47.9' /50.2
36,0
Qwen2: VB-7B [54]
54.6
63.3/69.0 n
60.6 /67.8'
50.2 /62,7'
40.0
Hour-LLaVA-7B (owns)
60.4
63.6/70.2
63.8/70.0
55.0/65.1
LLM Decoder: Qwen2.5-7B
Owen2_5.VI.. [3]
56.0
63.3/69.0
62.7 62.7'/70.7 /70.71
53.2 164.41
45.3
IntemVL3-88 [73]
58.8
66.3/68.9
40.0
IntemWidee02.5888 [56]
60.6
65.1/-
+/-
46.4
A.6 Broader Impacts
This study presents significant advancements long-form video-language modeling through the in- troduction of the Marahhon dataset and the Hour-LLaVA model. Positively, the VideoMarathon dataset paves the way for more sophisticated Al systems capable of handling realistic, long-duration scenarios, which are essential for practical applications in education, security, autonomous driving, and augmented or virtual reality. However, potential negative impacts include the risk of misuse in surveillance, such continuous monitoring and profiling of individuals in public private set- tings without consent, and the possibility of misinterpreting nuanced or sensitive content in long videos, which might lead to harmful decisions in critical domains like healthcare security. These implications highlight the importance of responsible development practices, including robust privacy safeguards and clear ethical guidelines for the deployment of long-form form video-language models.
19

---

interaction and Clip-level Video Captioning
You given 30-second video clip. Your lask generate detailed, structured description of the video based on six specific perspectives.
### Instructions:
Please ensure your descriptions are vivid, precise, and rich in detail. For each of the six topics below, please provide comprchensive caption. Your goal is to clearly convey the video's visual and contextual content from multiple dimensions:
Temporality: Describe how the scene evolves over time. Highlight key transitions, unfolding actions, changes from the beginning the end of the video.
Spatiality: Describe the spatial layout of the scene. Explain how key elements are positioned and oriented, and how they relate one another within the frame.
Object: Identify key objects the video, including inanimate items and humans. Describe their appearance, clothing, physical traits, materials, and possible roles.
Action: Describe the main actions taking place. Specify what actions occur, who what performs them, and the manner or sequence in which they unfold.
Scene: Provide a high-level overview of the setting. Describe the environment, background, and
general activity or context presented the video.
Summary: Offer brief yct comprehensive summary that contains the core event or purpose of the given video clip.
### Output Format (JSON):
Your response should be formatted JSON object, where each key corresponds topic and each value is the description associated with that topic.
#### Example Output:
jjoon
"The video begins with the group standing idle under the tree and progresses to active converastion and movement, auggesting preperation for an activity.".
"Temporality":
"Spatiality": "The individuals are grouped under large tree, spaced out in semictrcle. surrounding area open, grassy with scattered background.",
obbject:: "The scene includes several casually dressed men wearing t-shirts, shorts, hats, sunglassea,
medtun-stzed cardboard features natural objects Like trees, grass, and park benches.",
The background
"Action": "The men engage in conversation, gesturing with their hands, and preparing for activity. One walks away, possibly retrieve additional items.", "Scene": 'The video captures casual outdoor setting in a park. A small group of men gather under shady tree, seeningly preparing for social recreational activity in relaxed, natural environment.",
"Summary": group of men gather under a tree a public park, casually conversing captures
preparing for sctlvity. coordination in relaxed outdoor setting."
Figure The prompt for clip-level video captioning.
20

---

Event Splitting
video clips, where each clip is defined by a start time, time, video content. Your task is to merge related clips into small number of coherent events, each represented by merged time span and an event title.
will be given
##FF Instructions:
Consider the context for each video clip and ignore some outliers unreasonable video content, the given video clips are already an event, do not merge.
Merge Related Clips: Combine consecutive clips that logically belong to the same event based content continuity.
Preserve Context: Each merged event should represent a self-contained and contextually consistent unit of action activity.
Filter Outliers: Igmore clips that are irrelevant, inconsistent with the surrounding context, or clearly of place.
video. Aim minimize the number of events while ensuring that each one captures distinct scene, set of actions, or objects.
Balance Granularity: Avoid over-segmenting
Respect Standalone Clips: If single clip already represents complete, meaningful event, retain it
without merging.
#HH Output Format (JSON):
Return a JSON object where:
Each key is string denoting the merged time span of an event (e.g., "0-40s")..
Each value a concise event tille summarizes main content or activily.
##HH Example Output:
'Jsson
"0-40s": "Introduction to Holiday Desserts",
"Preparing ing Ingredients Tools",
1900-990s;
"Nixing Ingredients",
280--290s":
"Placing Ingredients into Tools",
380-350m"; "Baking of Desserts",
360-410s": "Presentation and Enjoyment of Desserts",
Figure 6: The prompt for event splitting.
21

---

Event-level Video Captioning
You will be provided with sequence of video clips depicting specific event. Fach clip includes metadata (start time, end time) structured descriptions from six perspectives. Your task to generate a cohesive, natural-language narrative that summarizes the entire event in chronological order. To support your understanding of the broader context, you will also receive brief descriptions of the events immediately before and after the current one. Use this surrounding context to enhance the flow and interpretation, but do not include those descriptions in your output.
#*# Instructions:
#**# 1. Maintain Chronological Flow
The clip-level video captions are already chronologically ordered. Your summary must preserve
this timeline and describe the event as continuous progression.
Avoid explicitly refereneing individual clips (e.g., "The first clip shows..." " "As the clip
progrexsev...") Instead, describe the event as seamless and continuous narrative.
Do not assume the first last frame of any clip represents the beginning or end of the entire event. Instead, focus on how the event unfolds as a whole.
#### 2. Use Adjacent Event Context Thoughtfully
To improve coherence and contextual understanding, you will receive brief descriptions of the events that occur immediately before and after the current event.
Use this information better understand current event.
IMPORTANT: Do not include these descriptions in your summary. They are provided solely to help you maintain context and continuity.
#### 3. Preserve Key Details, Filter Outliers
Retain all relevant details from the clip descriptions
accuracy and completeness. current event, ignore outliers or clips with inconsistent,
Based on the brief description
irrelevant, contradictory content that do nol contribule meaningfully to the event-level narrative.
#### Write in Natural, Engaging Tone
Your summary should read like natural video description, if you directly describing the event rather than summarizing segmented clips.
Avoid mechanical phrases often found in clip descriptions, such as "The clip begins...", "As the clip progresses..." " "The clip concludes..." "The first/last frame of this clip...", "The second clip shows...". Instead, focus on crafting a flowing narrative. The goal is to help a reader visualize the full event as if watching it unfold.
### Output Format (JSON):
The output should be structured JSON object.
#*## Example Output:
'Jsson
"Event-Level Description": "YOUR DESCRIPTION HERE.
### Input:
Brief description current event: <EVENT TITLE>
The event before the current event: <PREV EVENT DESCRIPTION
The event after the current event: <NEXT EVENT DESCRIPTION>
Detailed descriptions of all clips in the current event:
<CLIP-LEVEL DESCRIPTION 1>
CLIP-LEVEL DESCRIPTION 2>
CLIP-LEVEL DESCRIPTION N
Figure The prompt for event-level video captioning.
22

---

Global-level Video Captioning
You will be provided with a chronologically ordered sequence video events, cach accompanisd by both brief and detailed description. Your task to synthesize these descriptions into single, cohesive, and vivid narrative summarizes entire sequence of events as unified whole, without breaking the flow into separate segments or referencing individual clips mechanically.
#*# Instructions:
#### 1. Event-level Video Descriptions
Each event described with two levels granularity:
Brief Description: A concise overview of the event.
Detailed Description: A more in-depth description, including finer details.
#### 2. Generate a Unified Narrative
Write single, flowing narrative that describes the full sequence of events from beginning end. Maintain chronological order without explicitly mentioning timestamps or referencing individual
events (e.g., "In the second event...") Ensure the narrative reads if you are describing continuous experience, series of separate parts.
#### Focus Relevance and Consistency
Incorporate all meaningful and consistent details across the event descriptions. If a detail appears inconsistent, irrelevant, clearly unrelated, omit it based your understanding of the overall narrative.
#### 4. Use Natural and Engaging Tone
Write in a fluent, deseriptive style, suitable for someone reading or hearing a natural summary of the video.
Avaid mechanical phrases like: event begins...", "As the event progresses..." "The first/last
event shows...".
Instead, immerse the reader the experience, emphasizing continuity, clarity, and engagement.
### Output Format (JSON):
The output should be structured as JSON object.
#### Example Output:
''json
"Globst-Level Descr ipscription": "YOUR DESCRIPTION HERE."
### Input:
Event from <START 1> <END TIME 1>:
Brief description: <EVENT TITLE 1>
Detailed description: <EVENT-LEVEL DESCRIPTION
Event from <START TIMH TIME < TIME 2>;
Brief description: 2HVENT-ITTLE
Detailed description: <EVENT-LEVEI DESCRIPTION 2>
Event from <START TIME N <END TIME N>:
Brief deseription: <EVENT TITLE N
Detailed description: <EVENT-LEVEL DESCRIPTION N
Figure 8: The prompt for global-level video captioning.
23

---

Open-Ended QA Generation
You intelligent assistant specializing in open-ended question-answer generation for video understanding. Please follow instructions precisely and use only information provided input. Do not introduce any content unrelated to the described video clips.
You will be provided with:
chronologically ordered sequence of -TOPPC>>based video clip descriptions (each 30 seconds long, with and end timestamps).
global-Ieve description summarizing overall content the vidco.
Your task is to generate open-ended Question-Answer (QA) pairs from the perspective of <TOPIC>,
These QA pairs should promote deep understanding and must be strictly grounded in the given descriptions. No hallucination fabrication allowed.
Instructions:
#### 1. wwwPPCTT-----Stte. Sub-Tasks
The <TOPIC=-basc sub-tasks can be categorized into the following sub-tasks:
<SUB-TASK 1>; <TASK DESCRIPTION 1>
<SU--AAK 2>; <TASK DESCRIPTION 2> (..odditional sub-tasks i as needed)
#### 2. QA Examples for Each Sub-Task
guide your generation, here example QA pairs for each sub-task: Examples -SUB-TASK 1>:
"question": <DEMO Q1-1>; "answer": <DEMO A1-1>],
"question": <DEMO Q1-2>,
"answer": :DISW <DMO A1-2>),
Examples of <SUB-TASK 2>:
""question": <DEMO Q2-1>; |>, "answer": <DEMO A2-1>),
"question": <DEMO Q2-2>, "answer": <DEMO A2-2>},
#### Guidelines for Question-Answer Generation:
Focus on <TOPIC>-Releaan Information: Carefully analyze the deseriptions to identify patterns
relevant to the <TOPIC>.
Relevance and Context: The questions and answers must align with the content and context of the
video clips. The generated question-answer pairs should not introduce information that is not present the description.
Balance Diversity and Clarity: Create a variety of questions that collectively capture a full
understanding of the topic.
Quantity: Generate exactly three QA pairs for each sub-task.
Output Format (JSON):
The output should be structured as JSON object.
####### Example Output:
jsoon {
1>"; [(qquestion": "sQuestion 1-12", "answer": "cAnswer 1-1²"}, ...],
<Sub-Task 2>": [[qquestion": "equestton 2-12", "answer": "cAnswer 2-1>"}, ...l>
Input:
<TOPIC>-based Descriptions:
Clip from <START TIME 1> <END TIME 1>; <TOPIC-SPECIFIC CLIP DESCRIPTION 1>
Clip from <START TIME 20 <END TIME 2>; STOPIC-SPECIFIC CLIP DESCRIPTION 2>
Overall Description: <GLOBAL-LEVEL DESCRIPTION
Figure 9: The prompt for open-ended (OE) question generation.

---

Multiple-Choice QA Generation
You are intelligent assistant specializing in multiple-choice question-answer generation for video understanding. Please follow the instructions precisely and only the information provided in the input. Do not introduce any content unrelated to the described video clips. You will be provided with:
A chronologically ordered sequence of -TOPIC>~buseee video clip descriptions (each 30 seconds long, with start and end timestamps).
global-level description summarizing overall content the vidco.
Your task is to generate multiple-choice Duestion-Option-A Answer triplets from the perspective of <TOPIC>. These triplets should promote deep understanding and must be strictly grounded in the given descriptions. No ballucination or fabrication is allowed.
### Instructions:
#**# 1. <w.PTTOO---BBBB Based Sub-Tasks
The =TOPITT--bsbaseee sub-tasks can be categorized into the following sub-tasks:
<SUB-TASK >>; -TASK DESCRIPTION 1>
-SUB-TASK 2>; <TASK DESCRIPTION 2
(..additional sub-tasks as needed)
#### 2. QA Examples for Each Sub-Task
To guide your generation, here example QA pairs for each sub-task: Examples of =SUB-TASK 1>:
"question": <DEMO Q1-1>; "options": <DEMO OP1-1>,
"unswer": <1-1)
("question": <DEMO Q1-2>, "options": <DEMO OPI-2>,
"answer": <DEMO A1-2>),
...
Examples <SUB-TASK 2
"question": <DEMO Q2-1>; 1>, "options": <DEMO OP2-1>, "answer": <DEMO A2-1>),
"question": <DEMO Q2-2>, "options": <DEMO OP2-2>,
"answer": <DEMO A2-2>),
##hh# 3. Guidelines for Question-Answer Generation:
Focus <TOPiC>-Relevat Information: Carefully analyze the descriptions to identify patterns relevant to the <TOPIC>,
Relevance and Context: The questions, options, and answers must align with the content and context of the video clips. The generated question-option-amswe triplets should not introduce present the description.
information that
Clarity: Create variety questions that collectively capture a understanding of the topic.
Balance Diversity
Quantity: Generate exactly three QA pairs for each sub-task.
### Output Format (JSON):
The output should structured as JSON object.
#### Example Output:
"jsan {
"eSub-Task 1>": (Cauestton": "sQuestion 1-12", "opttons's [...). "answer": "chnswer 1-19"), ...l.
"eSub-Takk 2>": [C" ("question": "equesti wowsttion 2- 12". "aptions's [...], "answer" "chnswwr 2-32 ...].
### Input:
<TOPIC><based Descriptions:
Clip from <START TIME 1> <END TIME 1>: <TPPICSPECIIIC CLIP DESCRIPTION 1>
Clip from <START TIME 2 <END TIME 2>; -TPPIC-SPCIIICC CLIP DESCRIPTION 2>
Overall Description: <GLOBAL-LEVEI DESCRIPTION>
Figure 10: The prompt for multiple-choice (MC) question-answer generation.
25

---

Temporality
Focuses on rooognizing and interpreting temporal flow of events within video, including their sequencing. relative timing. and transitions between scenes. The task emppasszees understanding the local context of temporal configurations.
Temporal Perception:
Duration Perception: lnvolves estimating the length of individuull events and comparing the durations of multiple actions. objective is to deternine which activities are longer, shorter, or equal duration. The task emphasizes understanding the global contert of temporal configurations.
Temporal Reasoning: Requires logical inference hased temporal structure. This includes understanding causal relationshiis, predicting upcoming events, and reasoning about dependencies between tomporally related actions. The task emphasizes understunding the glohal contess temporal configurations.
inler the most accurate time gup bascd on contextual clues within the vidco. The task emphasizes understanding the global conteat of temporal configurations. Temporal Order: Focuses on identifying the correct chronological sequence of events. task assesses the ability deternine order which actions or visual elements appear throughout the video. The task emphusizes understanding the global contestr of lemporal configurations.
Temporal Gup Estimution: Entails estimating the elapsed time between events. The
Spatiality
Spatial Perception: Focuses recognizing and interpreting the spatial relationships among objects, people, and movements within scene. This includes identifying directions, positions, orientations, and local arrangement.. The task emphasizes understunding the
context spatial configurations.
Spatial Reasoning: Involves higher-level inforence about spatial stractures, such as deducing the relative positions, movements, and locations of entities within broader envirunments. This task requires interpreting inteructions acroes scenes and relies on understanding the global context spatial relationships.
Object
Object Recoguition: Involves ideatifying naming specifle objects, products, or items appear in a video. The task
includes providing contextual information about the recognized objects. This task primarily relies understanding the local context.
Object Interaction: Assesse the ability describe inteructions between objects within a scene how they are used, manipulated, or emphasizes
local contest object dynamics. Object Artribute: Focuses on describing the visual or functional chamactrissicc of objects, such as appearancs, material, color, size. This task also deperdss on local content understanding.
influence cach other.
Object Reasoning: lnvolves higher-level inference about objects in the scene. such their roles, functions, contextual significance. This roquires global conteat understanding across the video.
Object Existence: Evaluates the ability determine the presence or absence specifii objects at various points the
lask requires reusoning global context.
Object Permanence: Assesses the understanding of an objoct's continuity over time, including its movement, transformation, or disappearance. This task the model's ability track
about objects in global
Action
Action Recognition: Focuses on identifying and deseribing specific actions performed by subjects at given moments in the video. requires detailed interpretation movements and activities
the local context.
Action Localization: Aims to determine the temporal and spatial boundaries of actions. It involves identifying when and where
actions occur, including their start and end times, and comparing durations. This task relies on understanding the global context.
requires recognizing the sequence events, including what occuns before or after a given action, and understanding the progression over time, This task requires glohal context reasoning.
Action Sequence: Involves analyzing the chronological order actions.
Action Reasoning: Requires inferring the mutivations causes behind observed actions. This includes identiffing and effect relationships and interpreting contextual cues to explain why an action was taken. This task depends on global context comprehension.
Scene
Scene Recoguition: Assesses the ability to identify and deseribe the primary sctting or caviromment which the vidco takes place (e.g, indoor vs. outiloor kitehen park). This task involves understanding where the events and geneml atmosphere surnounding the churacters. relies on glohal context comprehension.
Scene Transition: lnvolves detecting and analyzing changes in scene or setting throughout the video. The task requires recognizzig shifts environmentt lighting, background, thematic focus, and describing these transitions accuntely, This task also depends global conrext understanding.
Event
Event Sequencing: Involves identifying the correct chronological order of events. This task requires recognizing the sequence in which actions unfold and determining preroquisite stops that lead to specific outcomes. It depends on a cobereat understanding of the globa! context.
Key Events Extraction: Foccuee on summarizing the most important actions and interactions within a video. The tusk entails ideutifying pivotal moments, evaluating their significance, and extracting overarching themes that define the eveat flow. This also requires comprehensive global context comprehension.
Figure 11: Detailed descriptions of 22 sub-tasks over six fundamental topics (except "overall summarization" task). For the "overall summarization" task, we directly use global-level video descriptions as the corresponding answers, so there is no need to design an additional prompt.
26

---

:20-50s":
"Temporality": "The video progresses from the woman introducing the recipe showing the ingredients and possibly demenstrating the steps involved." ..,
"Spattallty"; "The woman is seated in front of the table, which ts posttioned against plain wall. The ingredients "Object": "The woman wears a black tap and a chaker necklace with a pendant. On the table, there are three bowls contatning augar, flour, and butter, an egg, and glass bowl.",
arranged neatly the table.",
"Action": "The woman speaks to the camera, gestures with her hands, and points to the ingredients on table. She appears to be explaining the recipe.",
"Scene": "The video features woman sitting (n chatr, discussing rectpe while showing various ingredients latd out on a table.", "Summary": :
"This video showcases woman presenting homemade chocolate salted caramel tart recipe. She explains the ingredients and their arrangement on the table, providing step-by-step gutde for viewers to follow along."
(a) Examples of clip-level video captioning.
"0-70s": "Introduction and Overv of Chocolate Salted Caramel Tart Rectpe",
"60-130s": "Preparing and Kneading the Dough",
"120-190s": "Rolling and Shaping
Tart Crust",
"180-270s" "Creating the Caramel Sauce and Assembling the Tart",
"280-350s"; "Finalizing and Presenting the Chocolate Salted Caramel Tart"
(b) Examples of event splitting
"120-190s":
"Brtef Description": "Rolling and Shaping the Tart Crust", "Detailed Description": "In a cozy kitchen setting, persen meticulously prepares tart crust. The process begins with ball of dough on wooden surface, surrounded by generous aprinkling of flour. The indtvidual, dressed in a dark-colored shtrt, kneads the dough by hand, ensuring it is wall-mixed and pliable. Once the dough is ready, they use traditional rolling pin to flatten it, moving the pin back and farth to achieve an even thickness. The rolling pin's rhythmic motton ts a testament to the baker's a skill and expertence. As the dough ta rolled out, it is carefully transterred to a round tart pan, which is lined with parchment paper to prevent sticking. The person then shapes the dough
sides and botton to create smooth and even crust. Excess dough ts trinmed sway, and the final touches are added to ensure the tart base is perfect. The entire process is carried out with focused and meticulous approach, highlighting the importance of precisian in baking. The scene is a blend of traditional techniques and modern kitchen tools, creating a harmontous and engaging visual of the tart crust preparation."
pan, pressing
gently into
(c) Examples of event-level video captioning.
The vtdeo begtns with save and toutring Introtuction to hraraata chocal are satred carseel tart ractpe. Aoaar, snaased to black top and strectly coeata, pesturing with her hande io friandly
The coare transtrtine har ctstureing out the ingredd tn baid out us the uuster sigir, ftour, butter, un ont, and a giurs cont. The seiting ing usserrs i be u kilshon or custngg uru.. vuzy siectaterre the womm atands and ruthodicelly cly.. sumomatruting coch ausp clsurly
recton, potnnting
in tural, day ligtight, cembress the flzar, Uhe tart, Up rollow vling. As the vodeo pro progresses. the toous ahat ia to the gresure tion und knoucing of the doug". The momen non in stighcly vosur ielicter anv poriramant, norts an wooden suncu, covered in lour. demoutrute prict tend lostmuge tightiht her utbemiton to cleant insor and prec
unn.. aa shure
for begins to mis ungreed encorragin viomery
wet will the mucrasrry ingres tenta. starty by micing
Foour und bo bet
fore, anserin trg utemny. the soore
knouts the dough os the wooder uprt atus ing sutling the stagle the nost step the reckow.
the procsss of rotting und Huasing the turt crust tot lows, the soean, dreaase surfuce, the uses tradictoral re pin to taston the duigh, noving ond it bask and torth to sun uoso an oven thicknese. ho dough ts tron carefully
cur curkerutored shirl, narka with the dough on Lourred nouden
Eacms dough trumer parcheatt the fime toushes are udoed to pertout the lurt, conut testn waes und sodomn kecomen tos enporizatn the uppor lanos of proc unonn in baking.
and scame
sterd
vect, the video tranatetors and adds rice to weigh it. down escurtng an aver disteibution of nnight, fftar reasving the parcosent papar, ahe spplits ena with the pra- outter added and sextume, setting the stage for the dtxal steps io the recipe.
craation of the cararal sauce and the asserbly
The voean places parcheent into the tart
vith,
abtch to spread
she'l. Bank choestate chunke added for flavor
setie pourted Lits the tart shell; whter ta placed on a black slate basrd. The chocolate to spread eventy unting spoon, and the sart ta then neved to woodan cutring board for presentactor. Sax aste esrtakted on toe. addting touch contrast and antancingg the sart' slaver. The hightiigrta stapenoe of ctocotate tee coreeat tort, from Via proporation to Lta Tural presentecion, leoving vieeorr inspired to try the reotse thonse ivze.
(d) Examples of global-level video captioning.
Figure 12: Examples of hierarchical video captioning.
27

---

dres rressedd
toward
interaction
K
T
approach
Chen
activities
actiict Pac
car
Light
paint point
S
audience
work
movement
structure
me
cal
techn Laue
visser by
near
pr
rrviide
product
S
hourney
Limple
one
men
Teatur stage
Cark
aather
game
arranged
showcas ing
store
casua
exploration
Lvve
skilicuu
Food
rourine
blend
deliberate
Vibrant
bendling
wooden
throughout
adding
reanlessh
Genall
activity
metal
Tiiinnludi
Hllinn Tecldnng
smarr
uuggr
tance
Jocket
Propi
press
ingredient
cleaning
red
tive
impor
capturing
pink
engaging
stardturtet
paper
white
a
incerter
color
hl
dince
g
ensuring
ing
wandow
family
Tone
arimna
ani
Indoor
intri
mie
demoris
sma
touch
office
hamii
conclude
denonstrating
action
bui Loingse
se
space
een
cutting
lively
pr
var-oou
prepeet
Ceriwing
surrounded
reated
www
ensure
around
dinning
set
sert proper
clean
book
Lassing
cooking
cozy
tree
ready
speak
eplleplant:
holding
attire
Mrancing
task
mildet
brrwn
take
Anviti aetian
conover
Satineesa
na
naturelle
highlight
people
e
navi gaata
ConErAUs
LifE
e
t
yet
backdrop
gesture
cand
elenet naterial
Greatingon
connect wwncrratteeceeeoommmmoom
b.0
beauty
playful
O
norkspage
sense
Full
fot esenn Conttatiio
O
reflectinn
per
per
U.s SII
exper ence
detail
kist
Ineedvee
everyday
e
S
the
final
segment
Pccision
sioon
Session
fresh
lagg
oup
moment
front
fonntfettee
feature
eemphas
tze
e
mointing
bl uee
filled
E
@display
jar
n
argaan ed
system
og
sar den
ambi
ancce
cage eess
BALL
relaxed
individu
duor
rocused
nng
Clear
care
engaged
iccli
ous
Capture e
Beoight
akllihilyy
step
natura
1
Badorned
surr
P
progress
progressegio
sistrtting
scorescoree
vehicle
t
next
cention
studio
bu
estiimentt
providing
blending
noving
add
open
surface
Clathire
pan
large ge
seréne
interact
lightin
evident
blackemp
gr
een
plate
diffrres
enngaee
i'em
disobayane www.uo
wew.poommsstteetooooo Seatedaweinng
MExtore
irrbnn more
pr erea ataion
eddication
Figure 13: Word cloud of global-level video descriptions in VideoMarathon dataset.
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please check the abstract and Section
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See supplementary materials.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include theoretical results.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Implementation details are described in Section 5. Additional experimental details are shown in Appendix A. URLs to data and code are attached in the supplementary materials.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: URLs to data and code are attached in the supplementary materials.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Section 5. Additional details shown in Appendix A.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to the substantial training cost of Video-LLMs, do not report error bars this paper.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: See Section 5,
29

---

9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: httplp::///euris.... http://wwp.p...p..a..acaaaaaaaaa
Answer: [Yes]
Justification: This paper conforms, in every respect, with the NeurIPS Code of Ethics.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix A.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: See Section 5 and Reference.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: See supplementary materials.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMMs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: This paper describe the usage of LLMs for synthetic data generation for hideo-language understanding in Section 3. In addition, more usage details are described in the supplementary materials.
30


Reviews:
[{"review_id": "pO5GnxCEpg", "reviewer": "Reviewer_6t1N", "summary": "This paper introduces a new framework for training video-language models on hour-long videos instead of the usual short clips. They address the scalability and memory issues that come with processing such long content.\n\nKey Contributions:\n\n1. Present **VideoMarathon**, a large-scale hour-long video instruction-following dataset (sources are Panda-70M [7], Ego4D [15], ActivityNet [3], YouCook2 [67], and MovieChat-1K [44]).\n    - They achieve this through hierarchical video captioning (captioning segments and incrementally fusing dense captions together),\n        1. CLIP level: leverages Qwen2VL-7B LMM as the video clip captioner.\n        1. Fuses captions into \"event level\" (using deepseekv3).\n        1. Fuses captions into global level (using deepseekv3).\n2. **HourLLAVA:** The manuscript also introduces HourLLAVA , a Video-LMM capable of processing hour-long videos at 1 FPS in both training and inference.\n    - They design a memory augmentation mechanism to enable hour-scale video-language understanding. This mechanism consists of three main components:\n    - a memory repository (the high-fidelity video features Hv, extracted at 1 FPS w/ SigLIP)\n    - a forgetting mechanism (compresses the full video tokens by discarding tokens). \n        - Randomly drops 3/4 of the tokens spatially, and uniformly drops 3/4 of the frames temporally.\n    - a MemAug module (cross attention to recover information from the full memory repository - ie the uncompressed video features).\n\nResults show that HourLLAVA trained on VideoMarathon outperforms models like Llava-Video.", "questions": "Have the authors validated the LLM-synthesised QA pairs for correctness or bias, or measured hallucination rates?", "limitations": "Yes, in supplement.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "VideoMarathon contributes almost 10k hours of videos with captions and QA pairs. Most existing long video datasets are small and useful mostly for evaluation (eg EgoSchema, HourVideo), so this contribution will help close the training–inference length gap.\n\nArchitecture achieves good results with 1/16th~6% of the tokens.\n\nWeaknesses:\n- Missing human validation of the questions and answers generated (or at least some analysis of what the hallucination rate is).\n- The contribution is minor in that they mostly just adapt existing MLLMs by using heuristics for dropping frames. The only learnable component is the cross attention based MemAug module. This module by itself is not novel and resembles most of the literature on long video understanding - cross attention is a standard technique to avoid the quadratic explosion of self attention layers when inputs are long video sequences.  Furthermore, these references are missing (neither cited nor results referenced). Eg. Vamba combines CA and mamba layers and outperforms HourLlava on LVBench.", "was_revised": true, "final_justification": "I switched my score to Accept after seeing the rebuttals.\nMy original review criticized the missing evaluation of the quality of the generated samples. Their added evaluation does show promise, although more could be done (eg more than 80 MC questions evaluated).\n\nFurthermore, the added comparisons and results do strengthen their architecture contribution, despite being simple (this simplicity might even be a strength then). And the good results are achieved with 1/16th~6% of the tokens.", "rebuttal": "We sincerely appreciate your constructive suggestions, and we take these reviews seriously. We hope this response letter can help relieve your concerns.\n\n**W1 & Q1**: Human Evaluation on the Synthetic VideoMarathon Dataset.\n\n**A1**: Thank you for highlighting the need to validate the quality of synthetic data. We agree that the human evaluation on the VideoMarathon dataset is necessary. To quantify the data quality, we manually check 330 randomly selected VideoMarathon QA pairs, including 80 multiple-choice and 250 open-ended samples. \n\nFor the *Multiple-Choice (MC) QA* samples, we assess the **accuracy** of the synthetic samples by assigning a score of 1 if there is exactly one correct answer. A score of 0 is given if the question is irrelevant to the video, if no correct option exists, or if multiple options are correct. The human evaluation on 80 samples achieves an accuracy of **78.7%**, which demonstrates **the reliability of the synthetic multiple-choice QA samples**.\n\nFor the *open-ended (OE) QA* samples, we validate the data quality from two perspectives:\n\n- **Yes/No Bias**: Yes/No Bias evaluation [1] validates the extent to which the models tend to respond with ''yes''. In particular, the evaluation consists of ''Yes Percentage Difference'' (Pct. Diff) and ''False Positive Ratio'' (FP Ratio) metrics. Smaller Pct. Diff (**closer to 0.0**) reveals less bias, which means that the predicted number of ''Yes'' is closer to the ground truth. FP Ratio measures how likely the model responds with “Yes” out of all incorrect responses, where a value **closer to 0.5** indicates less bias. We randomly select 200 Yes/No QA samples. As a result, Pct. Diff reaches **0.04** and FP Ratio achieves **0.40**, which reflects **limited yes/no bias in the VideoMarathon dataset**.\n- **Hallucination**: 10 volunteers evaluate the quality of 50 randomly selected OE QA samples. Each sample is rated on a scale from 1 to 4 based on the following criteria: **Score 1 **- Totally irrelevant; key entities missing; mostly hallucinated; **Score 2** - Key entities present; contains major hallucinations; **Score 3** - Key entities present; contains minor hallucinations; **Score 4** - Almost identical to ground truth; no hallucinations. Scores range from 1 (low quality) to 4 (high quality). Scores above **3.0** are considered *largely correct*. Human evaluation achieves an average score of **3.29 ± 0.25**, indicating that the **OE QA samples in the VideoMarathon dataset are consistently reliable**.\n\n**W2.1**: Limited novelty of the proposed method.\n\n**A2.1:** Thanks for raising concerns about the novelty of the proposed method. As cross-attention and self-attention modules are fundamental building blocks of modern LLMs, their presence is not sufficient to determine whether the overall framework is novel or not. To clarify the **novelty of our Hour-LLaVA framework**, we analyze the limitations of existing long video large multimodal models (Video-LMMs), explain the necessity of the Hour-LLaVA framework, and **provide additional empirical results** to support our claims below.\n\n- **Limitations of existing long Video-LMMs**: Existing long Video-LMMs confront a fundamental dilemma:\n  - *Compression*-based methods prioritize computation affordable and training efficiency, but inevitably suffer from **information loss** caused by aggressive compression (e.g., uniform sampling 64 frames from a one-hour video);\n  - *Extension*-based methods preserve high-fidelity video context by using sequence parallel (SP) at the cost of **significantly increased training time and GPU usage** (linear growth with video length), particularly for hour-scale videos. \n  - As video length becomes longer (e.g., introducing VideoMarathon with hour-scale videos), such a dilemma will be intensified. Compression-based methods must incur greater information loss, while *extension*-based methods should tolerate longer training/inference time. Therefore, a new framework that effectively balances **video context** with **training efficiency** is urgently needed.\n\n- **Necessity of Hour-LLaVA framework:** To break this dilemma, we propose the Hour-LLaVA framework, which intends to keep **high-fidelity video context** while prioritizing **training efficiency.** Beyond the extension-based methods, Hour-LLaVA introduces the **forgetting mechanism** (implemented by spatial-temporal token dropping) that decouples computation from the input sequence length, so training and inference costs no longer rise linearly with video length. **MemAug module** (implemented by a cross-attention mechanism) is designed to dynamically recover lost video tokens from the **memory repository**. Without the MemAug module, purely compression‑based approaches inevitably incur irreversible context loss, which leads to sub-optimal performance. As a result, by coupling the forgetting mechanism with MemAug, Hour‑LLaVA delivers compression‑level efficiency while still preserving the full video context sought by extension methods.\n\n- **Hour‑LLaVA is more than a cross-attention module**: The novelty of Hour-LLaVA lies in the *interaction* of the two tightly coupled components: (1) the forgetting mechanism guarantees training efficiency, and (2) MemAug adaptively re-injects video tokens from the memory repository to recover information loss caused by the forgetting mechanism. However, MemAug (implemented by cross attention) is necessary but not sufficient. Without the forgetting mechanism, the trade‑off between contextual fidelity and training efficiency collapses.\n\n- **Hour‑LLaVA does more than frames/tokens dropping**: its learnable **MemAug** module *re‑injects* informative cues that token dropping would otherwise ignore. Moreover, the ablation study below shows that the MemAug module consistently improves the performance across different token dropping strategies, which highlights that MemAug effectively recovers information lost during compression and enhances the overall video-language modeling.\n\n  | Token Dropping Strategy | MemAug | LongVideoBench  | LVBench         |\n  | ----------------------- | ------ | --------------- | --------------- |\n  | Uniform                 | w/     | 54.0 (**+1.9**) | 40.6 (**+2.3**) |\n  |                         | w/o    | 52.1            | 38.3            |\n  | Random                  | w/     | 52.2 (**+0.5**) | 38.8 (**+0.7**) |\n  |                         | w/o    | 51.7            | 38.1            |\n  | Keyframe                | w/     | 53.5 (**+1.5**) | 40.3 (**+1.4**) |\n  |                         | w/o    | 52.0            | 38.9            |\n  | Question-Guided         | w/     | 53.4 (**+2.4**) | 39.7 (**+2.2**) |\n  |                         | w/o    | 51.0            | 37.5            |\n\n**W2.2**: Lack of comparison with the latest related work (Vamba).\n\n**A2.2:** Thanks for pointing out the latest related work, Vamba []. We will cite this work and include a direct comparison in the revision. The comparison with Vamba shows that both Hour‑LLaVA variants (3B and 7B) consistently outperform the considerably larger Vamba‑10B across all three long‑video benchmarks.\n\n| Method        | LVBench | LongVideoBench | VideoMME (w/o subtitles) |\n| ------------- | ------- | -------------- | ------------------------ |\n| Hour-LLaVA-3B | 44.7    | 57.8           | 60.6                     |\n| Hour-LLaVA-7B | **45.6**   | **60.4**           | **63.6**                     |\n| Vamba-10B     | 42.1    | 55.9           | 57.8                     |\n\n**Reference**: \n\n[1] HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models. CVPR 2024.\n\n[2] VAMBA: Understanding Hour-Long Videos with Hybrid Mamba-Transformers. ICCV 2025."}, {"review_id": "oMDGn4RiTk", "reviewer": "Reviewer_K5fQ", "summary": "This paper presents VideoMarathon, a large-scale hour-long video instruction-following dataset containing approximately 9,700 hours of videos (each 3–60 minutes long) with 3.3M QA pairs across 22 tasks spanning six core topics. Building on this dataset, the paper proposes Hour-LLaVA, a Video-LMM designed for hour-scale video understanding that employs a memory augmentation (MemAug) mechanism to process long videos at 1 FPS. The approach addresses a key gap between short training videos (typically <10 minutes) and long inference videos (>1 hour) in existing models. The proposed method achieves state-of-the-art performance on four video-language benchmarks while maintaining efficiency through learnable compression.", "questions": "1. How do you ensure the quality of the 3.3M synthetic QA pairs? What measures were taken to filter hallucinated or inconsistent content from the LMM-generated captions and questions? Was a human evaluation conducted on a subset to validate data quality?\n2. The 1/16 compression ratio discards 94% of visual tokens. What information is lost, and have you analyzed how performance degrades with higher compression? What is the theoretical lower bound?\n3. For videos exceeding 1 hour, how does the memory repository scale? What is the memory footprint compared to baselines? Can you provide a computational complexity analysis?\n4. Given that training videos are up to 60 minutes and LVBench averages 67 minutes (max >2h), how do you explain the generalization? Is this solely due to MemAug, or are there other contributing factors?\n5. Why does uniform temporal forgetting outperform more sophisticated strategies like keyframe-based or question-guided selection? Does this suggest that MemAug is doing most of the heavy lifting, making the forgetting mechanism less critical?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The work addresses an important problem in video-language understanding — the mismatch between training and inference video lengths.\n2. The hierarchical captioning pipeline (clip → event → global) is well-designed and leverages appropriate models (Qwen2VL-7B for captioning, DeepSeek-V3 for summarization).\n3. The MemAug mechanism is technically sound, using transformer blocks with cross-attention to recover information from compressed representations.\n4. The paper is well-written with clear motivation and methodology. Figures effectively illustrate the architecture and dataset pipeline.\n5. Experiments are comprehensive, covering multiple benchmarks with diverse video lengths.\n\nWeaknesses:\n1. The core components are mostly adaptations of existing techniques:\n(a) MemAug is essentially standard transformer blocks with cross-attention, similar to retrieval-augmented architectures.\n(b) The forgetting mechanism is a standard spatial/temporal token reduction at a 1/4 ratio.\n(c) Hierarchical captioning follows standard LMM practices across multiple granularities.\n2. Heavy reliance on synthetic data generation using LMMs may introduce systematic biases and quality inconsistencies.\n3. The paper lacks analysis of noise in the 3.3M QA pairs or quality control measures beyond prompt design.\n4. The 22-task taxonomy, while comprehensive, appears somewhat arbitrary without theoretical justification for these particular categories.", "was_revised": true, "final_justification": "Several of my concerns have been addressed. The rebuttal clarifies the importance of MemAug beyond standard cross-attention and token compression in enabling standard Video-LMMs to process hour-scale videos; its integration with the forgetting mechanism is well-motivated and shows consistent gains across token-dropping strategies. The fully synthetic VideoMarathon dataset still carries quality and bias risks, but the human evaluation suggests generally reliable QA pairs, and the paper is transparent about its limitations. The framework is explicitly designed for hour-scale video–language understanding, and the LVBench results demonstrate its effectiveness on long videos. Overall, the contribution is valuable for advancing efficient modeling of long videos, and I lean toward a borderline accept.", "rebuttal": "Thanks for the constructive comments. We hope this response letter can help relieve your concerns.\n\n**W1**: Methodology is mostly adaptations of existing techniques, such as the cross-attention module and standard spatial/temporal token reduction. Hierarchical captioning follows standard LMM practices.\n\n**A1**: Thanks for raising this concern. As cross-attention and self-attention modules are fundamental building blocks of modern LLMs, their presence is not sufficient to determine whether the overall framework is novel or not. To clarify the **novelty of our Hour-LLaVA framework**, we analyze the limitations of existing long video large multimodal models (Video-LMMs), explain the necessity of the Hour-LLaVA framework, and **provide additional empirical results** to support our claims below.\n\n- **Limitations of existing long Video-LMMs**: Existing long Video-LMMs confront a fundamental dilemma:\n  - *Compression*-based methods prioritize computation affordable and training efficiency, but inevitably suffer from **information loss** caused by aggressive compression (e.g., uniform sampling 64 frames from a one-hour video);\n  - *Extension*-based methods preserve high-fidelity video context by using sequence parallel (SP) at the cost of **significantly increased training time and GPU usage** (linear growth with video length), particularly for hour-scale videos. \n\n- **Necessity of Hour-LLaVA framework:** To break this dilemma, we propose the Hour-LLaVA framework, which intends to keep **high-fidelity video context** while prioritizing **training efficiency.** Beyond the extension-based methods, Hour-LLaVA introduces the **forgetting mechanism** (implemented by spatial-temporal token dropping) that decouples computation from the input sequence length, so training and inference costs no longer rise linearly with video length. **MemAug module** (implemented by a cross-attention mechanism) is designed to dynamically recover lost video tokens from the **memory repository**. Without the MemAug module, purely compression‑based approaches inevitably incur irreversible context loss, which leads to sub-optimal performance. As a result, by coupling the forgetting mechanism with MemAug, Hour‑LLaVA delivers compression‑level efficiency while still preserving the full video context sought by extension methods.\n\n- **Hour‑LLaVA is more than a cross-attention module**: The novelty of Hour-LLaVA lies in the *interaction* of the two tightly coupled components: (1) the forgetting mechanism guarantees training efficiency, and (2) MemAug adaptively re-injects video tokens from the memory repository to recover information loss caused by the forgetting mechanism. However, MemAug (implemented by cross attention) is necessary but not sufficient. Without the forgetting mechanism, the trade‑off between contextual fidelity and training efficiency collapses.\n\n- **Hour‑LLaVA does more than frames/tokens dropping**: its learnable **MemAug** module *re‑injects* informative cues that token dropping would otherwise ignore. Moreover, the ablation study below shows that the MemAug module consistently improves the performance across different token dropping strategies, which highlights that MemAug effectively recovers information lost during compression and enhances the overall video-language modeling.\n\n  | Token Dropping Strategy | MemAug | LongVideoBench  | LVBench         |\n  | ----------------------- | ------ | --------------- | --------------- |\n  | Uniform                 | w/     | 54.0  | 40.6  |\n  |                         | w/o    | 52.1            | 38.3            |\n  | Random                  | w/     | 52.2  | 38.8  |\n  |                         | w/o    | 51.7            | 38.1            |\n  | Keyframe                | w/     | 53.5 | 40.3|\n  |                         | w/o    | 52.0            | 38.9            |\n  | Question-Guided         | w/     | 53.4  | 39.7  |\n  |                         | w/o    | 51.0            | 37.5            |\n\n**W2**: Heavy reliance on synthetic data generation is unreliable.\n\n**A3**: We agree that heavily relying on synthetic data is risky. However, human annotation is too expensive to keep pace with the rapidly expanding video data, especially for long videos. Recent advances in LMMs (e.g., GPT-4o and DeepSeek-V3)  allow synthetic data to approach or even surpass human quality. In this way, a virtuous cycle is created, which means better LMMs produce higher-quality synthetic data, which in turn further improves subsequent models. Moreover, empirical studies consistently show that incorporating carefully constructed synthetic data improves the performance of video-LMMs, highlighting the practical value of synthetic data.\n\n**W3 & Q1:** Human evaluation is required.\n\n**A4**: We agree that the human evaluation on the VideoMarathon dataset is necessary. To quantify the data quality, we manually check 330 randomly selected VideoMarathon QA pairs, including 80 multiple-choice and 250 open-ended samples. \n\nFor the *Multiple-Choice (MC) QA* samples, we assess the **accuracy** of the synthetic samples by assigning a score of 1 if there is exactly one correct answer. A score of 0 is given if the question is irrelevant to the video, if no correct option exists, or if multiple options are correct. The human evaluation on 80 samples achieves an accuracy of **78.7%**, which demonstrates **the reliability of the synthetic multiple-choice QA samples**.\n\nFor the *open-ended (OE) QA* samples, we validate the data quality from two perspectives:\n\n- **Yes/No Bias**: Yes/No Bias evaluation [1] validates the extent to which the models tend to respond with ''yes''. In particular, the evaluation consists of ''Yes Percentage Difference'' (Pct. Diff) and ''False Positive Ratio'' (FP Ratio) metrics. Smaller Pct. Diff (**closer to 0.0**) reveals less bias, which means that the predicted number of ''Yes'' is closer to the ground truth. FP Ratio measures how likely the model responds with “Yes” out of all incorrect responses, where a value **closer to 0.5** indicates less bias. We randomly select 200 Yes/No QA samples. As a result, Pct. Diff reaches **0.04** and FP Ratio achieves **0.40**, which reflects **limited yes/no bias in the VideoMarathon dataset**.\n- **Hallucination**: 10 volunteers evaluate the quality of 50 randomly selected OE QA samples. Each sample is rated on a scale from 1 to 4 based on the following criteria. Scores range from 1 (low quality) to 4 (high quality). Scores above **3.0** are considered *largely correct*. Human evaluation achieves an average score of **3.29 ± 0.25**, indicating that the OE QA samples in the VideoMarathon dataset are consistently reliable.\n\n**W4:** The 22-task taxonomy is comprehensive but somewhat arbitrary. \n\n**A5**: We need to clarify that the 22-task taxonomy in the VideoMarathon dataset is not arbitrary but carefully designed based on an extensive survey of existing video-language datasets (Line 96). These datasets collectively cover over 100 distinct tasks. The current 22-task taxonomy is a distilled summary of those tasks across a wide range of benchmarks, aiming to provide a comprehensive and structured categorization of video-language understanding tasks.\n\n**Q2**: Analysis on **higher compression ratios.** What is the theoretical lower bound?\n\n**A6**: Thanks for the comment. It is challenging to provide a theoretical lower bound for the video token compression ratio due to the complexity of video modeling. To address this concern, we provide an empirical analysis to identify a suitable compression range. As shown in **Figure 4 (left)**, we evaluate the performance under higher temporal compression ratios. The results indicate that a ratio of **1/2 to 1/4** achieves a great balance between computational efficiency and performance. With a compression ratio beyond 1/4, performance begins to degrade significantly.\n\n**Q3.1**: How does the memory repository scale? What is the memory footprint compared to baselines?\n\n**A7**: MemAug with memory repository is a light-weight module. We compare the memory usage of Hour-LLaVA with its baseline when processing different numbers of frames during inference. We report the additional memory usage below, which demonstrates the efficiency of Hour-LLaVA.\n\n| # Frames | Additional Memory Usage |\n| -------- | -------------------------- |\n| 1,800    | +8%                       |\n| 3,600    | +13%                       |\n| 5,400    | +23%                       |\n| 7,200    | +40%                      |\n\n**Q3.2**: A complexity comparison between Hour-LLaVA and its base model is required.\n\n**A9**: **Figure 4 (right)** compares the number of visual tokens fed into the LLM decoder across Hour-LLaVA, LLaVA-Video, and vanilla 1-FPS sampling methods. The larger token number indicates the higher computational complexity, highlighting the low complexity of Hour-LLaVA.\n\n**Q4**: How to explain the generalization of Hour-LLaVA for videos longer than the training period? Are there other contributing factors?\n\n**A10**: The light-weight MemAug module allows Hour-LLaVA to process the longer videos (more than 2 hours). The proposed VideoMarathon dataset serves as another key contributing factor to the generalization of Hour-LLaVA.\n\n**Q5**: Explanation of the superiority of uniform temporal forgetting. Is the MemAug module responsible for most of the heavy lifting?\n\n**A11**: The table in **A1** shows that uniform temporal forgetting performs comparably to keyframe-based forgetting. Although we report different token compression strategies, developing a more advanced token compression strategy with clear superiority is not the focus of this work.\nThe table in **A1** also presents that when combined with MemAug, most token compression methods achieve a performance gain of approximately **1.5%–2.5%**, suggesting that the MemAug module plays a major role in boosting overall performance. This finding may serve as supporting evidence for your question."}, {"review_id": "d36XnegOKH", "reviewer": "Reviewer_6Aid", "summary": "The paper introduces VideoMarathon, a large-scale long-video instruction dataset, and Hour-LLaVA, an efficient video-language framework with a memory-augmented module (MemAug) for hour-scale video understanding. VideoMarathon addresses the scarcity of long-video training data by providing 3.3M QA pairs spanning 22 tasks across six themes. The model achieves SOTA results on four long-video benchmarks, demonstrating significant gains over existing video-LMMs.", "questions": "1. Could the authors evaluate Hour-LLaVA on generative long-video tasks to validate open-ended reasoning beyond MC-QA?\n2. What steps were taken to quantify noise in VideoMarathon’s synthetic QA pairs?\n3. Why was N=4 chosen for MemAug blocks?  Would performance improve with deeper architectures, and how does N impact training latency?\n4. Can the author add comparisons to Qwen2.5-VL-3B/7B, InternVL3-8B, InternVideo2.5-8B, and LLaVA-OV-SI-3B/7B?", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths \n1. VideoMarathon represents a significant advance in long-video instruction data, with 9.7K hours of videos and 3.3M task-oriented QA pairs spanning six thematic categories. The hierarchical captioning pipeline and task-specific QA generation mitigate data scarcity for hour-scale modeling.\n2. Extensive experiments across four benchmarks with varying video lengths demonstrate consistent SOTA performance. Ablations validate MemAug’s superiority over heuristic compression methods and VideoMarathon’s necessity for long-context learning .\n\nWeaknesses.\n1. Benchmarks exclusively assess multiple-choice QA (Table 2).  Generative capabilities (summarization, causal reasoning)—critical for long-video understanding—remain unevaluated.  Without testing open-ended tasks, claims of holistic \"long video-language understanding\" are unsubstantiated.\n2. All 3.3M QA pairs are synthetically generated via Qwen2VL-7B/DeepSeek-V3.  While prompts include task definitions and examples, no human evaluation or error analysis of QA quality is provided.  Potential noise/ambiguity in generated data may impact model reliability.\n3. The paper states MemAug uses N=4 transformer blocks (Sec. 4) but omits ablations on block depth/configurations. The impact of varying N on performance or efficiency remains unverified.\n4. Results (Table 2) omit critical baselines like Qwen2.5-VL-3B/7B[1], InternVL3-8B[2], InternVideo2.5-8B[3], and LLaVA-OV-SI-3B/7B. These exclusions weaken claims of superiority, especially given Qwen’s relevance as the backbone for Hour-LLaVA. \n\n\n\n[1] Bai S, Chen K, Liu X, et al. Qwen2. 5-vl technical report[J]. arXiv preprint arXiv:2502.13923, 2025.\n\n[2] Chen Z, Wang W, Cao Y, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling[J]. arXiv preprint arXiv:2412.05271, 2024.\n\n[3] Wang Y, Li X, Yan Z, et al. InternVideo2. 5: Empowering Video MLLMs with Long and Rich Context Modeling[J]. arXiv preprint arXiv:2501.12386, 2025.", "was_revised": true, "final_justification": "Most of my concerns have been addressed. I am raising my rating to borderline accept.", "rebuttal": "We sincerely appreciate your constructive suggestions, and we take these reviews seriously. We hope this response letter can help relieve your concerns.\n\n**W1 & Q1**: Evaluation on open-ended QA benchmarks is needed.\n\n**Ans**: Following your suggestion, we evaluate Hour-LLaVA-3B and Hour-LLaVA-7B on MLVU-Dev [1] with free-form generation tasks, including sub-science captioning (**SSC**) and video summarization (**SUM**). The generated results are measured by GPT-4-Turbo with carefully designed instructions, where the rating score is on a 2–10 scale. We also report results from other state-of-the-art Video-LMMs for reference. The results demonstrate the strong capability of Hour-LLaVA on open-ended tasks.\n\n| Method           | Overall  | SSC  | SUM  |\n| ---------------- | -------- | ---- | ---- |\n| Shargpt4Video-7B | 3.77     | -    | -    |\n| LongVA-7B        | 4.33     | -    | -    |\n| Video-XL-7B      | 4.50     | -    | -    |\n| Hour-LLaVA-3B    | 4.45     | 5.04 | 3.91 |\n| Hour-LLaVA-7B    | **4.57** | 5.22 | 3.97 |\n\n**W2 & Q2**: Human Evaluation is needed.\n\n**Ans**: We agree that the human evaluation on the VideoMarathon dataset is necessary. To quantify the data quality, we manually check 330 randomly selected VideoMarathon QA pairs, including 80 multiple-choice and 250 open-ended samples.\n\nFor the *Multiple-Choice (MC) QA* samples, we assess the **Accuracy** of the synthetic samples.\n\n- **Accuracy**: This evaluation assigns a score of 1 if there is exactly one correct answer. A score of 0 is given if the question is irrelevant to the video, if no correct option exists, or if multiple options are correct. The human evaluation on 80 samples achieves an accuracy of **78.7%**, which demonstrates **the reliability of the synthetic multiple-choice QA samples**.\n\nFor the *open-ended (OE) QA* samples, we validate the data quality from two perspectives:\n\n- **Yes/No Bias**: Yes/No Bias evaluation [2] validates the extent to which the models tend to respond with ''yes''. In particular, the evaluation consists of ''Yes Percentage Difference'' (Pct. Diff) and ''False Positive Ratio'' (FP Ratio) metrics. Smaller Pct. Diff (**closer to 0.0**) reveals less bias, which means that the predicted number of ''Yes'' is closer to the ground truth. FP Ratio measures how likely the model responds with “Yes” out of all incorrect responses, where a value **closer to 0.5** indicates less bias. We randomly select 200 Yes/No QA samples. As a result, Pct. Diff reaches **0.04** and FP Ratio achieves **0.40**, which reflects **limited yes/no bias in the VideoMarathon dataset**.\n\n- **Hallucination**: 10 volunteers evaluate the quality of 50 randomly selected OE QA samples. Each sample is rated on a scale from 1 to 4 based on the following criteria: **Score 1** - Totally irrelevant; key entities missing; mostly hallucinated; **Score 2** - Key entities present; contains major hallucinations; **Score 3** - Key entities present; contains minor hallucinations; **Score 4** - Almost identical to ground truth; no hallucinations. Scores range from 1 (low quality) to 4 (high quality). Scores above **3.0** are considered *largely correct*. Human evaluation achieves an average score of **3.29 ± 0.27**, indicating that the OE QA samples in the VideoMarathon dataset are consistently reliable.\n\n**W3 & Q3**: Ablation study on the number of MemAug blocks is missing.\n\n**Ans**: Thanks for raising this point. We agree that an ablation study on the number of MemAug blocks is required. Using the experimental setup in Section 5.5, we train four Hour‑LLaVA variants with 1, 2, 4, and 8 MemAug blocks. Across three video benchmarks, performance improves up to 2 or 4 blocks and then decreases slightly at 8. More specifically, for short videos (TempCompass), the sweet spot lies in **the range from 2 to 4 blocks**, while for medium‑ and long‑form videos (LongVideoBench and LVBench), **4 blocks** provides the optimal results.\n\n| # MemAug Blocks | TempCompass | LongVideoBench | LVBench  |\n| --------------- | ----------- | -------------- | -------- |\n| 1               | 59.4        | 53.2           | 40.0     |\n| 2               | **59.8**    | 53.5           | 40.3     |\n| 4               | 59.7        | **54.0**       | **40.6** |\n| 8               | 59.2        | 53.4           | 40.5     |\n\n**W4 & Q4**: Comparisons to advanced Video-LMM are required.\n\n**Ans**: We have listed all the mentioned Video-LMM below. * indicates our re-implementation.\n\n**Comparison among 3B-LMMs**: In this setting, we compare three highly relevant 3B LMMs. LLaVA-OV-SI serves as the pretrained model of Hour-LLaVA, while Qwen-VL and Hour-LLaVA both employ Qwen as their LLM decoder. The results demonstrate that Hour-LLaVA-3B significantly improves the long video-language modeling capability of its pretrained model (LLaVA-OV-SI-3B). Moreover, Hour-LLaVA generally outperforms its strong peer (Qwen2.5-VL-3B), especially in medium- and long-video settings.\n\n| Method         | LLM        | LVBench         | LongVideoBench  | VideoMME (overall) | VideoMME (Medium) | VideoMME (Long) |\n| -------------- | ---------- | --------------- | --------------- | ------------------ | ----------------- | --------------- |\n| LLaVA-OV-SI-3B | Qwen2.5-3B | 35.4            | 49.6            | 51.1               | 49.2              | 44.1            |\n| Hour-LLaVA-3B  | Qwen2.5-3B | **44.7** (+9.3) | **57.8** (+8.2) | 60.6 (+9.5)        | **59.0** (+9.8)   | **52.1** (+8.0) |\n| Qwen2.5-VL-3B  | Qwen2.5-3B | 43.3            | 54.2            | **61.5**           | 58.7*             | 51.9*           |\n\n**Comparison among ~7B-LMMs**: Following prior works [4, 5], we initialize Hour-LLaVA with LLaVA-OV-SI-7B [3], as Qwen-2.5-VL had not yet been released during our training period. Consequently, Hour-LLaVA-7B inherits the **Qwen2-7B** as LLM decoder from its pre-training backbone. For a fair comparison, we also add the results of Qwen2-VL-7B below. Similar to the 3B setting, Hour-LLaVA-7B significantly improves its pretrained model (LLaVA-OV-SI-7B) and surpasses its strong counterpart (Qwen2-VL-7B). Moreover, compared with other state-of-the-art LMMs (Qwen2.5-VL and Intern series) built on the stronger Qwen-2.5-7B backbone, Hour-LLaVA-7B performs comparably on most of the following benchmarks. It even outperforms these models on certain benchmarks, such as the medium and long subsets of VideoMME.\n\n| Method            | LLM        | LVBench     | LongVideoBench | VideoMME (overall) | VideoMME (Medium) | VideoMME (Long) |\n| ----------------- | ---------- | ----------- | -------------- | ------------------ | ----------------- | --------------- |\n| LLaVA-OV-SI-7B    | Qwen2-7B   | 36.0*       | 52.1*          | 58.2               | 52.6*             | 47.9*           |\n| Hour-LLaVA-7B     | Qwen2-7B   | 45.6 (+9.6) | 60.4 (+8.3)    | 63.6 (+5.4)        | **63.8** (+10.2)  | **55.0** (+7.1) |\n| Qwen2-VL-7B       | Qwen2-7B   | 40.0        | 54.6           | 63.3               | 60.6*             | 50.2*           |\n| Qwen2.5-VL-7B     | Qwen2.5-7B | 45.3        | 56.0           | 65.1               | 62.7*             | 53.2*           |\n| InternVL3-8B      | Qwen2.5-7B | -           | 58.8           | **66.3**           | -                 | -               |\n| InternVideo2.5-8B | Qwen2.5-7B | **46.4**    | **60.6**       | 65.1               | -                 | -               |\n\n**Reference**:\n\n[1] MLVU: Benchmarking Multi-task Long Video Understanding. In CVPR 2025.\n\n[2] HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models. CVPR 2024.\n\n[3] Llava-onevision: Easy visual task transfer. ArXiv 2024.\n\n[4] Longvu: Spatiotemporal adaptive compression for long video-language understanding. ICML 2025.\n\n[5] Long context transfer from language to vision. ArXiv 2024."}, {"review_id": "F2RYezxc4n", "reviewer": "Reviewer_ozeC", "summary": "This paper introduces VideoMarathon, a large-scale, hour-long video instruction-following dataset. It also presents a novel model, Hour-LLaVA, which features three key innovations: a memory repository, a forgetting mechanism, and a MemAug module. Experimental results demonstrate that the combination of Hour-LLaVA and VideoMarathon achieves superior performance on long video benchmarks.", "questions": "1. First of all, it would be very interesting if the authors could show some attention heatmaps on the memory repository for a few examples. This would help people understand whether the model learns to retrieve the relevant content.\n2. For Weakness 1, could the authors report some performance results on temporal reasoning questions, if possible?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper is very well-written and easy to follow.\n2. The proposed VideoMarathon dataset is a big contribution to the community.\n3. The proposed memory repository, forgetting mechanism, and MemAug module seem novel. As lots of works try to compress the video tokens, I strongly agree that we need a memory repository and cross-attention which behaves like retrieval to fetch detailed information if needed.\n4. The experimental setup and ablation studies are comprehensive.\n\n**Weakness:**\n\n1. The method uses SigLIP (an image-text model) to extract image features at 1 FPS, then stores these features (tokens) in a memory repository. But these stored features are not temporally processed. The cross-attention would just search for the most relevant content at the frame level, so the model is likely **order-agnostic**, and **temporal relationships may be lost**. I am concerned, especially for tasks that require **temporal reasoning**, such as:\n    - “What happened before event X?”\n    - “Describe the sequence of actions.”\n    - “What changed between minute 5 and minute 10?”", "was_revised": true, "final_justification": "After reading the reviews, rebuttal, and the authors' discussion, I believe the proposed VideoMarathon dataset is a good contribution to the community, and the proposed Hour-LLaVA demonstrates strong results in the comprehensive experiments. Therefore, I will maintain my positive rating.", "rebuttal": "Thanks for acknowledging our key contributions. We sincerely appreciate your constructive suggestions, and we hope this response letter can help relieve your remaining concerns.\n\n**W1**: **Temporal relationships may be lost.**\n\n**Ans:** Thanks for raising this concern. In MemAug, we do consider the spatial-temporal relationship between the *full* video tokens in the memory repository and *decayed* video tokens. In particular, we first directly flatten the *full* video tokens into 1D embeddings and apply the 1D structure RoPE, which is commonly used by the prior works [1, 2]. Based on their spatial-temporal relationship, we then construct a corresponding 1D RoPE for *decayed* video tokens. In this way, each *decayed* token (query) is encoded with positional information that matches the similar spatial-temporal positions of its corresponding *full* tokens (keys and values) in the cross-attention module.\n\nAblation studies show a substantial performance drop on the temporal reasoning task when this structural RoPE is removed. It demonstrates that constructing the spatial-temporal relationship between the *full* video tokens and *decayed* video tokens is necessary.\n\n| Method        | w/ 1D structure RoPE | VideoMME (Temporal Reasoning) |\n| ------------- | -------------------- | ----------------------------- |\n| Hour-LLaVA-3B | Yes                  | 46.3                          |\n|               | No                   | 35.6 (-10.7)                  |\n| Hour-LLaVA-7B | Yes                  | 56.5                          |\n|               | No                   | 38.4 (-12.1)                  |\n\n**Q1**: Attention heatmaps on the memory repository for visualizing whether the model learns to retrieve the relevant content.\n\n**A2**: Thanks for the insightful suggestion. Attention heatmaps over the memory repository indeed offer a way to visualize which frames or regions the model focuses on during retrieval. While we are currently unable to include these visualizations due to policy constraints, we plan to incorporate them in the revised version to better illustrate the model’s behavior.\n\n**Q2**: Results on temporal reasoning tasks.\n\n**Ans**: We evaluate the Hour-LLaVA on VideoMME (Temporal Reasoning subset). We also provide the results of LLaVA-OV-SI and Qwen-VL models as references (* indicates our re-implementation). Notably, Hour-LLaVA achieves considerable improvements in the temporal reasoning task over its pretrained model (LLaVA-OV-SI) with gains of **13.5%** for the 3B model and **20.9%** for the 7B model. Moreover, Hour-LLaVA consistently outperforms the Qwen-VL models on this task, demonstrating its superior temporal reasoning capabilities.\n\n| Method               | VideoMME (Temporal Reasoning) |\n| -------------------- | ----------------------------- |\n| LLaVA-OV-SI-3B       | 32.8                          |\n| Hour-LLaVA-3B (ours) | 46.3 (+13.5)                  |\n| Qwen2.5-VL-3B        | 41.8*                         |\n| LLaVA-OV-SI-7B       | 35.6                          |\n| Hour-LLaVA-7B (ours) | 56.5 (+20.9)                  |\n| Qwen2-VL-7B          | 42.4*                         |\n| Qwen2.5-VL-7B        | 48.5*                         |\n\n**Reference**:\n\n[1] RoFormer: Enhanced transformer with rotary position embedding. Neurocomputing 2024.\n\n[2] TC-LLaVA: Rethinking the transfer from image to video understanding with temporal considerations. ArXiv 2024."}]

Author rebuttal:

Meta-review: The paper deals with the problem of hour-long Video-LLMs and proposes a new dataset for instruction fine-tuning, VideoMarathon (3.3M  QA pairs. To address the challenge, the paper features a new framework, Hour-LLaVA, based on memory augmentation, which achieves SOTA performance on multiple long video-language benchmarks.

The paper was considered by four reviewers. The final ratings were: BA - BA - BA - A

All reviewers highlight the contribution of the dataset as well as of the method. 
The AC follows the consensus of the reviewer voting and recommends accepting the paper. The AC would encourage the authors to integrate the findings of the rebuttal in the CR version of the paper.
Decision: Accept (spotlight)