Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understanding
Authors: Xiaoqian Shen
Wenxuan Zhang
Jun Chen
Mohamed Elhoseiny
Venue: NeurIPS 2025 spotlight ()
Keywords: Long Video Understanding; Retrieval-augmented generation; GraphRAG
Primary area: applications

Abstract: Understanding and reasoning over long videos pose significant challenges for large video language models (LVLMs) due to the difficulty in processing intensive video tokens beyond context window and retaining long-term sequential information. Retrieval-Augmented Generation (RAG) has demonstrated effectiveness in processing long context for Large Language Models (LLMs); however, applying RAG to long video faces challenges such as disrupted temporal dependencies and inclusion of irrelevant information that can hinder accurate reasoning. To address these limitations, we propose Vgent, a novel \textbf{graph-based retrieval-reasoning-augmented generation framework} to enhance LVLMs for long video understanding. Our approach introduces two key innovations: (i) It represents videos by structured graphs with  semantic relationships across video clips preserved to improve retrieval effectiveness. (ii) It introduces an intermediate reasoning step to mitigate the reasoning limitation of LVLMs, which leverages structured verification to reduce retrieval noise and facilitate the explicit aggregation of relevant information across clips, resulting in more accurate and context-aware responses. We comprehensively evaluate our framework with various open-source LVLMs on three long-video understanding benchmarks. Our approach yielded an overall performance improvement of $3.0\%\sim 5.4\%$ over base models on MLVU, and outperformed state-of-the-art video RAG methods by $8.6\%$. Our code is publicly available at https://xiaoqian-shen.github.io/Vgent.

Vgent: Graph-based Retrivaa-Reasonnn-AAgmnened Generation For Long Video Understanding
Xiaoqian Shen', Wenxuan Zhang', Jun Chen 12, Mohamed Elhoseiny'
King Abdullah University of Science and Technology. Meta Al (xiaoqian. wenu,,
wenxuan. wenxuan.zhang, jun. chen, waan.chee,,mmmmeecm mohamed, 2lhoseenn))kauuee edu. edu.
Abstract
Understanding and reasoning over long videos pose significant challenges for large video language models (LVLMs) due to the difficulty in processing intensive video tokens beyond context window and retaining long-term sequential information. Generation (RAG) has demonstrated effectiveness in process- ing long context for Large Language Models (LLMs); however, applying RAG to long video faces challenges such disrupted temporal dependencies and inclusion irrelevant information
can hinder accurate reasoning. To address these limi- tations, we propose Vgent, novel graph-based generation framework to enhance LVLMs for long video understanding. Our approach introduces two key innovations: (i) It represents videos by structured graphs with semantic relationships across video clips preserved to improve re- trieval effectiveness. (ii) It introduces intermediate reasoning step to mitigate the reasoning limitation LVLMs, which leverages structured verification to reduce retrieval noise and facilitate the explicit aggregation of relevant informa- tion across clips, resulting in more accurate and context-aware responses. We comprehensively evaluate our framework with various open-source LVLMs three long-video understanding benchmarks. Our approach yielded an overall performance improvement of 3.0% 5.4% over base models on MLVU, and outperformed state-of-the-ar video RAG methods by 8.6%. Our code is publicly available https: xiaoqian shen. github. ig/Vtt...
Introduction
Multi-Modal Large Language Models (MILMs) [6, 22, 30, 39, 40, 58] have demonstrated remarkable visual understanding and reasoning capabilities, paving the way for advancements in video tasks. Recently, numerous studies have showcased impressive progress building Large Video Language Models (LVLMs) for video understanding [9, 7, 24, 35, 55]. Moreover, long-video understanding is particularly crucial for applications in web content, life-logging, and streaming media, where intricate narratives and evolving contexts span extended durations.
However, processing and reasoning existing LVLMs, as representing video frames requires an extensive number of tokens-for example, 30-minute video can exceed 200K tokens [4, 22], beyond most models' context limits. To handle longer videos, existing methods resort to sparse frame sampling [55, 22] or token compression [43], but these approaches inevitably lead visual information loss, weakening fine- imn--graindd temporal understanding and coherent reasoning.
long-contex: videos remain formidable challenge for
Recent studies [1, 2, 12, 34, 52, 53] utilize Petrieval-Augmentie Generation (RAG) [21] to enhance long-form video understanding by retrieving relevant information. However, they encounter certain limitations. First, some works segment lengthy videos into shorter clips, treating each as individual document for retrieval [2], which disrupts the continuity of entities and temporal dependencies,
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Cipp
Cipp
Graph Conseruction
Graph based Rettleval
Structured Relinement
The plate vas initially in the Link, then possibly moved to
Where was the plate before I putting bread
2
A. cabinet B. refrigerator C. sink D. floor
Query
Reasoning
Fittered Video
Entrieval-Reassoning Augmented Generation
Large Video Language Model
Figure I: Overview of our graph-based retifevaalleesoninnnnaaageeeee generation framework. Each video clip is represented as a node within a graph, interconnected through shared entities. This graph representation enables effective retrieval of relevant clips based on node connections, followed by an intermediate reasoning step to refine retrievals and aggregate over multimodal context for accurate generation.
leading to retrieval inaccuracies. Second, some methods [47, 34] rely on proprictary LLMs like GPT-4 [37] for multi-turn interactions, planning, and reasoning. making them costly and less flexible. Lastly, several approaches [47, 33] extract information from sparse key frames, neglecting temporal coherence in long videos.
Graph representation for enhanced retrieval: To address the limitation of existing RAG methods, we propose a structured graph-based representation, where video clips are modeled as nodes inter- connected by recurring subjects scenes. This graph representation not only enables effectively retrieving nodes associated with specific entities, but also facilitates capturing temporal dependencies spanning over lengthy videos. Another advantage is that the graph construction is performed offline and is query-inde indeeednneee. Once the graph is built, it can reused for multiple questions on the same video, allowing retrieval to operate directly on the graph without reprocessing the video. However, feeding all retrieved clips into LLMs can cause information overload, where key details are diluted by irrelevant content [14]. In videos, the issue further amplified, each frame consumes hundreds of tokens, with irrelevant information overshadowing the critical content.
Structured post-retrieval reasoning: To address the aforementioned issue and fully harness the benefits from our GraphRAG, we introduce the structured reasoning step the post-retrieval stage. Instead of generating answers directly from the retrieved clips, it decomposes the question and systematically verifies the relevance of each clip. As shown in Figure 1, this process refines the retrieved set by identifying clips that mention critical clements-suce bread-and and then aggregates information across them for temporal reasoning (c.g.. the plate moving from the sink the cabinet). This process mitigates noise, facilitates information aggregation across refined clips, and thus creates a more reliable pathway for producing accurate responses.
the plate. sink, cabinet, and
We evaluate our framework upon seven different LVLMs with sizes ranging from 2B to 7B across three long-video benchmarks: MLVU [57], VideoMME [13] and Long VideoBench [50]. Experimental results demonstrate that our framework consistently improves the performance of existing LVLMs by 3.0%-5.4%. We further show that our framework surpasses existing RAG-based video understanding works by 8.6%.
Contribution. Our contribution summarized as follows:
We developed a novel graph-based RAG framework for long-video understanding, where video clips are represented as nodes within a graph, interconnected through shared enti- ties, thereby preserving semantic relationships and temporal dependencies across clips. facilitating more effective retrieval.
We propose structured reasoning to tackle the limited reasoning ability LVLMs, which can be distracted by hard negative retrieved samples. Our approach introduces an intermediate reasoning step for retrieval verification and aggregates information across verified clips to enhance generation accuracy.

---

Our graph-base based provements over various LVLMs ranging from 2B to 7B and surpasses existing RAG-based video understanding works by 8.6% in long-video understanding tasks.
augmentted framework demonstrates 3.0%-5.4% im-
2 Related Work
2.1 Large Video Language Models
Multimodal large language models (MLLMs) [6. 58, 31, 30, 45] have demonstrated remarkable progress in vision- language tasks. Recent advancements have further extended their capabilities to video understanding tasks [2, 9, 24, 25, 28, 32]. Large Video Language Models (LVLMs)) process videos by extracting and encoding frames, and then rearranging them into final video representations. Some approaches [24, 25, 9] leverage the Q-Former module from BLIP-2 [23] to integrate visual and textual features, while others [2. 28, 32] directly concatenate frame features. However, these models struggle with processing hour-long videos in a single pass as the number of video tokens exceeds their training context size. To address these limitations, most existing works train on sparsely sampled frames no matter how long the video is [24, 9, 55, 22], while others try to handle long videos by token pooling [35, 27, 44], token compression [43], or memory aggregation [19]. However, they struggle to effectively capture and reason about temporal dependencies spanning hour-long videos.
2.2 Agent-based Video Understanding
A dominant trend in long-context video question-answeriing involves equipping Large Language Models (LLMs) with tools that heavily rely on proprietary models to process queries and handle video clips. MM-VID [29] uses a video-to-script generation with GPT-4V [37] to transcribe multimodal elements into a long textual script. VideoAgent [47] integrates diverse foundation models through unified memory architecture. DrVideo [34]. VideoTTre [49], VideoAgent [12] and OmAgent [53] dynamically invoke tools enhance query processing and accuracy. Such methods consequently suffer from high operational costs and critical reliance on external, closed-sourg systems, limiting their adaptability. In contrast, our work targets the development of a self-contained pipeline designed
for flexible deployment with open-source LVLMs.
2.3 Video Retrieval-Augmentee Generation
Retrieval-Augmentcc Generation (RAG) enhances large language models (LLMs) by retrieving relevant information to improve long context memory, factual accuracy, and reduce ballucinations. The process involves three stages: (i) indexing, which organizes raw data into a knowledge base; (ii) retrieval, which searches for relevant information based on user queries; and (iii) generation, where the model takes the retrieved context to generate the final response. Recent advancements of RAG in LLM mainly follow two directions, i.e. chunk-based methods [15, 5] and graph based methods [11, 17, 26, 18], both of which have been applied video understanding tasks. Goldfish [3] chunks long videos into shorter clips, processes each clip independently, and retrieves the most relevant clip in response to user queries. Wang et al. [48] applied graph structures for action recognition in short clips and Husscin et al. [20], Luo al. [33] employ scene graphs for video understanding. However, constructing graphs for long videos and effectively retrieving information from the noisy and complex graph remains challenge. Only recently. concurrett work [42] constructs graphs for long-context video understanding, but they heavily relies on external proprietary LLM for graph construction, while graph-bact based video RAG with open-sourced LVLM itself remains unexplored, which our work aims to address.
3 Method
We introduce novel, training-fret free framework, Vgent, for long-conters video understanding. Un- like conventional Retrieval-Augmente Augmented Generation (RAG), our pipeline proposes graph-based generation paradigm, specifically designed to address complex video scenarios with improved contextual comprehension and structured reasoning. As illustrated in Figure 2, our proposed pipeline contains four stages: (1) Offline video graph construction (Sec- tion 3.1): Builds video graph offline by extracting knowledge from long videos. (2) Graph-based

---

Offline Graph Construction
Retrieval-Reasoning Augmented Generation
Question: Plaase identify the option that romespondh the order soents they Rumba, dean and jark, milking cow, playing tromison)]
Subtitles
Query Keywords Extraction Izuubba", y "cleen", "milking "playing ing trasbone")
Strutured Query Refinement "rutba" shown the video?
01:
video?
LVLM
Goph-tared Opp Betieeal
IeMentity:: "*, "description: ""),
Ciip
G1
QR
Information Aggregation
X
("entity"s "description": nr]]
Y
Milking is shown first, followed by zumba, clean, and finally playing trombone.
Entiyy Marging and Node Connection
11
Retined Clai
LVLM
Viduo Clip
Enttlee
C
Arower: miking row zumba -2 dean and jerk playing trombone
Figure 2: Pipeline of Vgent, novel framework for long- long-context video understanding in the proposed graph-based seaorvalrrasoonnnaaaaeeeeee
generation paradigm. It consists of four key stages: (I) Offline video graph construction (Section 3.1): Builds video graph by extracting knowledge from long videos. (2) Graph-based retrieval (Section 3.2): Retrieves relevant clips based on keywords extracted from the query. (3) Structured reasoning (Section 3.3): Refines clips using structured queries and aggregates information. (4) Multimodal augmented generation (Section 3.4); Combines refined clips and reasoning results to generate the final response.
retrieval (Section 3.2): Retrieves relevant video clips from graph based on the user query. (3) Struc- tured Reasoning (Section 3.3): Refines the retrieved clips using structured queries and aggregates information across the filtered clips. (4) Multimodal Augmented Generation (Section 3.4); Combines refined clips and intermediate reasoning results to generate the final response.
3.1 Video Graph Construction
To better capture the complex relationships and dependencies in long-context videos, we propose a graph-base based representation to store video content and enhance semantic connections. Specif- ically, given video V with F frames, we first partition it into sequence of short video clips where each video clip Vi consists of K frames. We then dynamically construct the graph by a series of structured steps, as detailed below.
(V1, ......,
Visual Entity Extraction. For each video clip, we leverage the LVLM extract the key semantic entities (i.e., the primary subjects, actions, or scenes) from both the spoken content (subtitles) C, and video clip V.
((e) f1), (e t2),,.. + LVLM (C,, Vi),
where the set of entity denoted E, ef..e?.... ... ............. } and its corresponding description set is denoted as T; = {t], (2, ..... J. In this step, the LVLM captures subjects, actions, and scene dynamics, seamlessly linking visual entities with spoken content to extract meaningful knowledge. Please refer to Appendix B.1 for illustrative examples.
Graph Construction. Based on extracted information, we construct video knowledge graph g = (V,E), where V denotes the nodes set representing video clips, and edges in E represents connectivity between nodes. Additionally, we define global set unique prototype entities u = {u E,, i 1, ... [1 that spans across all nodes. As more video clips are processed, we will dynamically add newly extracted unique entity u to the set link to an existing entity. We define t as the description of each entity E U.

---

Entity Merging and Node Connection. Since LVLMs process video clips independently, it is essential identify and unify semantically equivalent entities across clips. Given newly extracted intity-descriptoon pair (e,, f.) from video clip V, we determine whether it belongs to an existing entity in the global entity set U. Specifically, we compute the similarity score between the textual entities
based on their respective text embeddings. the similarity score > т, the entity ej is considered semantically equivalent to an existing entity and these two merged into single entity representation. Otherwise, eý is treated as distinct entity and added to U. This process is formulated as follows:
descriptions 1j and descriptions
fu,, max sim(ly,.)) (), u² = arg max sim(l), ("), ej + {ur (u+uuees, otherwise NEl
if 2
wEM
Once entity is merged, we then build edges from the node vs associated with the video clip Vi to all the nodes that have the same entity u", denoted as D(u").
+ E {(v,, v) D(y))
As new video clips are processed, the graph is dynamically updated such that nodes containing the same entity are connected, which preserves semantic relationships and contextual dependencies. This forms structured representation that facilitates effective video retrieval in subsequent processing stages.
3.2 Graph-based Retrieval
Keywords Extraction. Direct retrieval based on the original query may not provide sufficient context, especially when reasoning across multiple temporal clips required. To address this, we extract keywords from the query for effective retrieval. Specifically, we prompt the LVLM to identify key semantic elements, denoted as K, from the query Q. The detailed prompt is provided in Appendix B.2.
Graph-based Clip Retrieval. Next, we leverage these extracted keywords for graph- based retrieval. Specifically, for each keyword & K. and each entity U, compute similarity score sim(k, (")) to determine whether the entity matches the keyword. If sim(k, t") ) >> 8, we include all nodes associated with entity the target retrieval node set R:
R =
{uEVIC PREVIUEUTE (uE(()) sim(k, > 0} KEU,KEK
U
(4)
After obtaining the retrieval node set R, refine the results by re-ranking the nodes based on the similarity between the query's 's keywords and the extracted information of each node, including entities, corresponding textual deseriptions, and subtitles if available. Finally, we select the Top-N nodes with the highest average similarity scores across all associated information of each video clip.
3.3 Structured Reasoning
Feeding all relevant clips directly into LLMs can lead information overload, diluting the focus on key details with irrelevant content [14]. Our empirical analysis also reveals that in roughly 40% of failure cases, the correct clip is successfully retrieved, yet the model still generates incorrect responses-even though it can answer correctly when provided with that clip alone. We then introduce structured reasoning in the post-retrieval stage that refines the retrieved clips and aggregates useful information towards final generation.
Structured Query Refinement. We introduce the dividd-andd-onquee strategy to refine the retrieval through structured query verification. Specifically, we prompt the LVLM to generate structured subqueries, denoted as Q, based on the original query Q and extracted keywords K. These subqueries focus on verifying the presence of relevant entities quantifying their occurrences, whose answers are expected to be binary (yes/no) or numerical value. Please refer to Appendix B.3 for the detailed prompt and Figure 3 for example of generated subqueries.
5

---

After generating the subqueries, we process the Top-N retrieved video clips using the LVLM, producing either binary (yes/no) or numerical responses for each subquery. As shown in Figure this structured verification systematically assesses the relevance of each clip to the original query. filtering out irrelevant clips that were wrongly retrieved based on semantic embedding similarity. Denoting 1 to yes and 0 to no in binary questions, this refined clip set R' can be formulated as:
R' = {u1 E R Eg E 2, [(2 93) > 0}
(5)
where [w;; qj) denotes the response of retrieved clip v to subquery 45- We keep at most r clips after refinement. This refinement step ensures that only video clips satisfying the structured queries retained, effectively eliminating hard negatives from the initial retrieval.
Information Aggregation. As shown in Figure all useful information from structured queries and their corresponding results for each video clip, providing an enriched auxiliary context that enhances the final inference.
then let LVLM aggregate and summarize
3.4 Multimodal Augmented Generation.
We incorporate both the intermediate reasoning results and the filtered video clips as multimodal context inputs the LVLM for the final response. This enriched input allows the model leverage both structured reasoning and relevant visual information, enabling it to generate a more accurate and contextually grounded final response to the original question.
4 Experiments
4.1 Experimental Setups
Baselines. We apply our framework Vgent on open-sourced LVLMS including Intern VI.2.5 [8], Qwen2 [46], Qwen2.5-VI. [4], Lung [43] and LLaVA-Video [56] as base video understanding model. We further compare Vgent against otate-of-the-art RAG baselines as follows: NaïveRAG [3],
Video-RAG [33], and proprietary LLM-based methods including VideoAgent [47]. LLoVi [52],
DrVideo [34] and VideoTree [49]. More details can be found in Appendix A.3.
Benchmarks. We evaluate the performances of each model across three video benchmarks. Video-MME [13] is a widely used benchmark designed to evaluate LVLMs' capability to process detailed, real-world videos. It comprises three subsets categorized by video length, ranging from 11 seconds to hour. MLVU [57] is a long-video understanding benchmark with videos ranging from minutes 2 hours, with an average length of about 12 minutes. Long VideoBench (LVB) [50] focuses on referred reasoning tasks that require models to analyze long frame sequences. These questions depend on extensive temporal context and cannot be effectively addressed using a single frame or small set of sparsely sampled frames.
Implementation Details. During the offline video graph construction, we sample videos at 1.0 FPS, segmenting the long video into clips, each containing K 64 frames. We use the larnge--arreee-----
[51] embedding for similarity calculation. The entity merging threshold is set to r = 0.7. In the online retrieval stage, we use BAAI/bge http:/beeearggee..e en- to retrieve the top N 20 clips based on extracted keywords (maximum 20 discard low-relevance, with a similarity threshold 0 = 0.5). After structured query refinement, we retain a maximum of r 5 clips. Thresh- olds are set as the same for all three benchmarks, with hyper-parameter selection details provided in the supplementary. For MLVU [57], we extract spoken content using openai wwwaaaiwwwsieereeeee large, while for VideoMME [13] and Long VideoBench [50], we use the provided subtitles from benchmark. All experiments are conducted on A A00 80G GPUs.
4.2 Main Results
Comparison with LVLMs. In Table 1 and 6, we present the performance of our proposed Vgent framework on the MLVU [57] benchmark, where we consistently observe substantial im- provements across all models. Specifically, our framework enhances LongVU [43] by 5.4% and boosts Qwen2.5VL [4] (7B) by 3.3%. Notably, when applied to Qwen2.5VL (3B), Vgent achieves

---

Table 1: Performance comparison with LVLMs. Vgent consistently improves all models on MLVU [57], enhancing LongVU by 5.4% and Qwen2.5VL (7B) by 3.3%. Notably, Vgent achieves 70.4% accuracy on Qwen2.5VL (3B), surpassing its 7B counterpart and improving the base model by 4.2%. Vgent outperforms base models across all video lengths on VideoMME [13] achieving improvement of 3.2% overall.
VideoMME
Models
Size
MLVU
LVB
w/o sub.
w/ sub.
Proprietary LVLMS
Gemini 1,5 Pro [16]
75.0
81.3
64.0
GPT-4o [38]
64.6
71.9
77.2
66.7
Open-Source LVLMS
VL2.5 [8]
2B
56.7
49.5
55.2
52.0
InternVL2.. Vgent (Ours)
2B
61.1 14.4
50.9*14
56.8+1.6
54.8+2.8
Qwen2.5-VI. LL [4]
3B
66.2
61.4
67.6
54.2
Qwen2.5-VI. + Vgent (Ours)
3B
70.4
63.0+1.6
69.6+2.
57.8+3.6
LongVU [54]
7B
65.4
55.2
60.9
50.2
LongVU Vgent (Ours)
7B
70.8+5.4
57.3+2.1
63,7+28
52.7+2.5
Qwen2-VL. [46]
7B
65.7
62.7
68.1
55.6
Qwen2-VL Vgent (Ours)
7B
70.3+4.6
63.5+0.8
70.1 +2.0
58,4+2.8
LLaVA-Video [56]
7B
69.5
64.3
69.2
59.5
LLaVA-Video Vgent (Ours)
7B
72.5+3.0
66.7+2.4
71.1+1.9
62,4+2.9
Qwen2.5-VL [4]
7B
68.8
65.1
71.1
56.0
Qwen2.5-VL + Vgent (Ours)
7B
72. +3.3
68.9+3.8
74.3+3.2
59.7+3.7
accuracy of 70.4%, surpassing its larger 7B counterpart and improving the base model by 4.2%. This result underscores the effectiveness of our approach in bridging the performance gap between small models and their larger counterparts. At the category level (Table 6), our framework notably improves Count and Order tasks, which demand event-level understanding and multi-clips reasoning.
In Table showcase the results of V gent the VideoMME [13] benchmark, where it consistently outperforms base models across all video lengths, achieving an average performance gain of 4.2%. Notably, framework excels in oong-video scenarios, surpassing the best baseline by 5.4%. These findings highlight the strength of our structured graph-basee retrieval and reasoning approach, demonstrating its ability enhance long-video comprehension by effectively capturing cross-segment dependencies and refining information retrieval for improved reasoning and final response generation.
Comparison with SoTA RAG Methods. In Table 2, we provide comprehensive comparison of Vgent against state-of-the-art RAG methods on MLVU [57] and VideoMME [13] benchmarks.
(1) Our framework consistently outperforms the RAG baseline, Video-RAG [33], across three different LVLM base models. Unlike Video-RAG [33], which relies on CLIP [4 --sseedd keyframe selection and external tools such as object detection and OCR for frame-level information extraction, Vgent eliminates these dependencies by leveraging LVLMs themselves for graph construction, verification, and intermediate reasoning. This structured approach significantly enhances retrieval precision and reasoning accuracy, leading to more reliable final responses.
(2) Our framework also surpasses proprietary RAG-based methods for long-video understanding. Compared to closed-source API-dependent methods which heavily rely on closed-source APIs, our framework is more flexible and effective solution for long-video understanding.

---

Table 2: RAG methods comparison. i denotes results are sourced from [34]. Proprietary LVLMS refer to methods that rely on closed-source APIs. We include them here for reference only, as our primary focus is on building self-contained pipeline to improve open-source LVLMs.
VideoMME
Models
Size MLVU
w/o sub. w/ sub.
Proprietary LVLMs
VideoAgent' [47]
44.4
LLoVit [52]
67.7
DrVideo [34]
71.7
Open-Source LVLMs
Qwen2.5-VL + Video-RAG [33]
3B
62.2
60.3
65.1
Qwen2.5-VL Vgent (Ours)
3B
70.4
63.0
69.6
+ Video-RAG [33]
7B
71.3
64.8
70.0
LLaVA-Video+ Vgent (Ours)
7B
72.5
66.7
71.1
Qwen2.5-VL+ Video-RAG [33]
7B
63.4
60.5
65.7
Qwen2.5-VL. + Vgent (Ours)
7B
72.1
68.9
74.3
Table Ablation study results of the performance improvement contributed by each component of our proposed pipeline. SR denotes our proposed structured reasoning.
Models
MLVU
VideoMME
Long
Qwen2.5-VL [4]
68.8
71.1
56.0
Qwen2.5-VL + NaīveRAG
65.4
68.3
56.2
Qwen2.5-VL, GraphRAG
69.5
72.7
57.1
Qwen2.5-VL NaīveRAG + SR
68.6
69.8
57.3
Qwen2.5-VL + GraphRAG + SR (default)
72.1
74.3
59.7
4.3 Ablation Studies
NaïveRAG vs GraphRAG. As shown in Table 3, integrating GraphRAG yields an average im- provement of 2.9% over NaīveRAG, with particularly notable 4.1% gain MLVU [57]. This is because NaiveRAG's difficulty in handling complex queries that requires temporal reasoning across multiple clips, it treats each video clip effectively preserves semantic relationships between clips, enabling more accurate retrieval and reasoning. By structuring video content into graph representation, approach addresses retrieval inconsistencies inherent in traditional RAG methods.
independent document. In contrast, GraphRAG
However, the improvement remains marginal compared to the base models. Upon checking failure cases in MLVU, we observe that in 44% of the failures, the correct clip is actually present within the model's retrieved set, which indicates that while the retrieval was successful, irrelevant retrievals still distract the model, hindering accurate responses. Consequently, post-retrieval stage is necessary to amplify the potential of our GraphRAG by refining the retrieved nodes and improving reasoning towards more precise answers.
Structured Reasoning (SR). By refining retrieved nodes through intermediate reasoning with structured queries, we achieve an additional 2.6% improvement on MIVU [57] and 1.6% on VideoMME [13], resulting in overall 3,4% average gain over the base model. This interme-

---

diate reasoning step decomposes complex queries into targeted sub- sub-questions and generates binary or numerical answers. These structured response are then used to systematically filter out irrelevant clips and aggregate relevant information across clips, guiding the model toward the correct final answer. Our findings also indicate that the final improvement is contingent upon Graph-base haphh RAG. Specifically, if SR is applied to NaïveRAG, the inherent inaccuracy of NaiveRAG's retrievals restricts the potential for significant improvement through refinement alone.
Number of retrieval We conduct ablation study examine the impact of the number of retrieved clips after structured query refinement. Table presents both the overall performance and results across several MLVU [57] subcategories. Among these, Count and Order are two tasks that heavily require reasoning across multiple video clips. Count involves identifying the number of events or actions throughout an entire video, while Order requires the model arrange multiple events in chronological sequence. represents the maximum number of video clips retained after refinement. Our findings indicate that increasing the number of retrieved clips consistently improves performance, particularly for tasks demanding multi-clip reasoning, with the highest performance observed at 5.
Table 4: The number of retrieved clips impacts performance on MLVU [57].
#Retrieval
Count
Ego
Needle
Order
PlotQA
Anomaly
Topic
Overall
25.7
54.2
75.7
51.7
67.4
71.0
84.3
63.2
40.2
55.6
78.0
57.1
69.1
73.5
87.0
66.9
47.5
57.1
78.0
61.0
70.0
71.5
87.2
68.4
58.7
56.6
78.8
65.2
73.6
72.5
87.6
71.0
(default)
58.7
59.5
79.7
67.1
74.6
74.0
88.0
72.1
58.7
58.4
78.8
67.2
73.9
73.5
87.4
71.9
Further details, including category-level performance on MLVU (A.1), limitations (D), ablation studies on the number of retrievals N (A.5), confidence-bact baased refinement (A.2), retrieval threshold (A.6) are provided in the Appendix.
Table 5: Inference time analysis. Since processing time depends on the video duration, we report the normalized time required to process each minute of video.
Query Independent (offline)
Query Dependent (online)
Model Time (sec)
#Proprietary LVLMs VideoAgent [47]
N/A
67.25
#Open-Source LVLMs
Qwen2.5VL-7B [4]
N/A
2.95
+ +Video-RAG [33]
N/A
20.81
+ Vgent (Ours)
20.13
3.93
4.4 Inference Time Analysis
We analyze the computational trade-off's and report the processing times in Table 5 for the API- based method VideoAgent [47]. Video-RAG [33] well
framework built Qwen2.5VL [4]. VideoAgent [47] leverages a proprictary LLM (GPT-4 [36]) to iteratively perform self-reflection for frame selection and aggregating key information from the video. Video-RAG [33] relies on query- dependent key frame selection and per-frame object detection, introducing online computational overhead. In contrast, which takes 20.13 seconds. Once the graph is built, the online retrieval, reasoning and generation process requires only 3.93 seconds per minute-video.
framework can offline constructs query-indepee seery-iddeeedeent graph from the video,
Our offline graph construction further improves efficiency in multi-question scenarios (e.g., three questions per video in VideoMME [13]). Unlike query-dependent methods that reprocess the entire

---

Clip
Clip
Clip
Clip 7
Clip 0
11
Video Graph
Question: Did open the laptop? A. Maybe B. No C. don't know Yes
Construction
Graph-based Clip Retrieval
GropeRtG w/o reasoningg
Ansucrt B. No
9,,6,7,9,13]
GraphttA with reasening
Strutured Queries
Reasoning
Laptop is video opened, interacting with Then, laptop is shown
zomeone visible interacting
laptop the video?
03: Does the video show the laptop
opened?
Answer:
Figure 3: A qualitative example illustrates our graph-based retrieval-reasoning approach, which connects relevant video clips via shared entities. While the model initially fails to correctly identify the action of opening the laptop, misled by hallucinations from hard negatives. However, adding an intermediate reasoning step that validates each retrieved node through structured subqueries enables the model to generate the accurate response.
video for each question, our approach constructs the graph once, allowing the model to retrieve relevant clips based entity Descriptions--iiheee the need to rewatch the entire video. As result, our method achieves a 1.73x speedup over Video-RAG [33] when performing inference on VideoMME [13].
4.5
Qualitative Examples
We show a qualitative example Figure and 6. Our graph construction effectively connects relevant video clips through shared entities. In Figure 3 the graph-based retrieval system can identify relevant nodes that contains a laptop, with Clip providing crucial evidence answer the query. However, the model incorrectly responded "No" to the question "Did I open the laptop?", presumably due hard negatives from multiple clips featuring opened laptop, hallucinating the model to overlook the closed laptop and the action of opening it.
In contrast, with intermediate reasoning step, validate each retrieved node with structured subqueries (e.g.. "Is there a laptop open?" "Is someone interacting with the laptop?"). This verified information is aggregated to form an enhanced reasoning chain, allowing the model correctly infer that the laptop was opened, overcoming the distraction from hard negatives.
Conclusion
In this work, we introduced a novel gaseh Retrieval-Augmented Generation (RAG) framework designed for long-video understanding. Our approach represents video clips as nodes in a graph and leverages entities maintain semantic relationships, thereby enhancing retrieval effectiveness. To address retrieval noise, proposed a structured query refinement strategy that systematically filters out irrelevant clips, ensuring a more precise selection relevant video content. Additionally, we introduced an intermediate reasoning step that summarizes the response to the structured query, using the filtered retrieved clips as multimodal context to significantly improve the accuracy of the final answer generation. Our framework outperforms state-of-the-arr video RAG methods by 8.6%, demonstrating effectiveness in enhancing long-video understanding tasks. This work paves the way for more accurate and context-aware long-form video retrieval and reasoning systems.
10

---

References
[1] Md Adnan Arefeen, Biplob Debnath, Md Yusuf Sarwar Uddin, and Srimat Chakradhar, irag: Advancing rag for videos with incremental approach. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 4341-4348, 2024.
[2] Kirolos Ataallah, Xiaoqian Shen, Eslam Ahdelrahman, Essam Sleiman, Deyao Zhu, Jian Ding, and Mohamed Elhoseiny, Minigpt4-video: Advancing multimodal Ilms for video understanding with interleaved visual-textual tokens. arXiv preprint arXiv:2404.0344 2024.
[3] Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Mingchen Zhuge, Jian Ding, Deyao Zhu, Jürgen Schmidhuber, and Mohamed Elhoseiny. Goldfish: Vision-language understanding of arbitrarily long videos. arXiv preprint 2rXiv:2407.12677 2024.
[4] Shuai Bai, Keqia Chen, Xuejing Liu, Jialin Wang, Weabin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-v1 technical report, arXiv preprint arXiv:2502. 13923, 2025.
[5] Chi-Min Chan, Chunpu Xu, Ruibin Yuan. Hongyin Luo, Wei Xue, Yike Guo, and Jie Fu. Rq-rag: Learning to refine queries for retrieval augmented generation. arXiv preprint arXiv:2404.0000 00610, 2024.
[6] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krish- namoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: v2: large language model as unified interface for risionllanguage multi-task learning. arXiv preprint arXiv:2310.09478, 2023.
[7] Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang. Zehui Chen. Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understanding and generation with better captions. arXiv preprint arXiv:2406.04333 04325, 2024.
[8] Zhe Chen, Weiyun Wang. Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:4412.05271, 2024.
[9] Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint www.iv:200.074444 2024.
[10] Jacob Devlin, Ming-
Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume
1 flong and short papers), pages 4171 4186-448,.220 2019.
[11] Darren Edge, Ha Trinh. Newman Cheng. Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. From local to global: A graph rag approach query- summarization. arXiv preprint arXiv:2404. 16130, 2024.
[12] Yue Fan, Xiaojian Ma. Rujie Wu, Yuntao Du Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory- augmented multimodal agent for video understanding. arXiv preprint arXiv:2403. / /1481, 2024.
[13] Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mongdan Zhang, et al. Video-mme: The first-ever comprchensive evaluation benchmark of multi-modal Ilms in video analysis. arXiv preprint arXiv:2405. 2024.
[14] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, W. Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Jiao Qiao. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304. 15010, 2023.
[15] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Haofen Wang. and Haofen Wang. Retrieval-augmente generation for large language models: A survey. arXiv preprint arXiv:2312. 10997. 2. 2023.
[16] Gemini Team Google. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint WrXiv:2403.05530 2024.
[17] Zirui Guo, Lianghao Xia. Yanhua Yu, Tu Ao, and Chao Huang. Lightrag: Simple and fast retrieval- augmented generation. arXiv preprint arXiv:2410.05779, 2024.
[18] Haoyu Han, Yu Wang. Harry Shomer, Kai Guo, Jiayuan Ding, Yongjia Lei, Mahantesh Halappanavar, Ryan A Rossi, Subhabrata Mukherjee, Xianfeng Tang, et al. Retrieval auggentte generation with graphs (graphrag). arXiv preprint arXiv:2501.03300 003,0 2024.

---

[19] Bo He. Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao. Ashish Shah. Abhinav Shrivastava, and Ser-Nam Lim. Ma-Imm: auumented large multimodal model for long-term video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13504- 13514, 2024.
[20] Noureldien Hussein, Eistratios Gavves, and Amold WM Smeulders. Videograph: Recognizing minutes- long human activities in videos. arXiv preprint wwtXiv://55.05550 2019.
[21] Patrick Lewis, Ethan Perez, Aleksandra Pikus,, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmenttd generation for knowledge- intensive nlp tasks. Advances in neural information processing systems, 33:9459-9474, 2020.
[22] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang. Feng Li, Hao Zhang, Kaichen Zhang. Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 033366 2024.
[23] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2 Bootstrapping language-iman pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[24] KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang. Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint 2.X.x:2333000335 2023,
[25] Kunchang Li, Yali Wang. Yinan He, Yizhuo Li, Yi Wang. Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mybench: A comprehensive multi-modal video understanding benchmark. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195-22206, 2024.
[26] Mufei Li, Siqi Miao, and Pan Li. Simple is effective: The roles of graphs and large language models in knowledge-grapp Bariigg--graa--bbaseee retrieval-augmente generation. arXiv preprint arXiv:2410.2077 2024.
[27] Yanwei Li, Chengyao Wang. and Jiaya Jia, Llama-vid: An image worth tokens in large language models. arXiv preprint arXiv:231..1704. 2023.
[28] Bin Lin, Bin Zhu, Yang Ye. Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint 1rX1v:2311.1012,, 2023.
[29] Kevin Lin, Faisal Ahmed, Linjie Li, Chung- Shing Lin, Ehsan Azarnasab, Zhengyuan Yang, Jianfeng Wang. Lin Liang. Zicheng Liu, Yumao Lu, et al. Mm-vid: Advancing video understanding with gpt-4v (ision). arXiv preprint 23 197773377,. 2023.
[30] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024.
[31] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[32] Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Minghui Qiu, Pengeheng Lu, Tao Wang, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability. arXiv preprint 2r36v: 07207, 2023.
[33] Yongdong Luo, Xiawu Zheng, Xiao Yang, Guilin Li, Haojia Lin, Jinfa Huang, Jiayi Ji, Fei Chao, Jiebo Luo, and Rongrong Ji. Video-rag: Visually-aligned retrieval-augmeeneed long video comprehension. arXiv preprint arXiv:2411.1308 2024.
[34] Ziyu Ma, Chenhui Gou, Hengean Shi, Bin Sun, Shutao Li, Hamid Rezatofighi, and Jianfei Cai. Drvideo: Document retrieval based long vidco understanding. arXiv preprint arXiv:2406. 12846, 2024.
[35] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arX44:2306.05424,
2023.
[36] OpenAL. Gpt-4 technical report, 2023.
[37] OpenAL. Gpt-4v(ision) system card, 2023.
[38] OpenAL. Gpt-4o system card, 2024.
[39] Team LLb. Internv12: Better than the best expanding performance boundaries of open-souree multimodal models with the progressive scaling strategy. 2024.
12

---

[40] Team Qwen. Qwen2 technical report, 2024.
[41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-58888 18888866, 2021.
[42] Xubin Ren, Lingrui Xu, Long Xia, Shuaiqiang Wang, Dawei Yin, and Chao Huang. Videorag: Retrieval- augmented generation with extreme long-context videos. arXiv preprint arXiv:2502.01549, 2025.
[43] Xiaoqian Shen, Yunyang Xiong, Changsheng 7hao, Lemeng Wu, Jun Chen, Chenchen Zhu, 7echun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, al. Longvu: Spatiotemporal adaptive compression for long video-language understanding, arXiv preprint arXiv:2440. 2024.
[44] Enxin Song, Wenhao Chai, Guanhong Wang. Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tian Ye, Yan Lu, Jenq-Neng Hwang, et al. Moviechat: From dense token to sparse memory for long video understanding. arXiv preprint arXiv:2307. 2023.
[45] Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang. Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: I: A fully open, vision-centric exploration of multimodal Nmss. arXiv preprint arXiv:2406. 16860, 2024.
[46] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge. et al. Qwen2-vl: Enhancing vision-languan model's perception of the world at any resolution. Xiv preprint Xr0: /2/91, 2024.
[47] Xiaohan Wang. Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy Videoagent: Long- form video under- standing with large language model as agent. arXiv preprint arXiv:2403. 10517, 2024.
[48] Yang Wang, Gedas Bertasius, Tye- Oh, Abhinav Gupta, Minh Hoai, and Lorenzo Torresani. Super- voxel attention graphs for long-range video modeling. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages
2021.
[49] Ziyang Wang. Shoubin Yu, Elias Stengel-Eskin, Jachong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Videotree: Adaptive tree-based video representation for Ilm reasoning on long videos. arXiv preprint or Xiv:2400. 19209, 2024.
[50] Haoning Wu. Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved lineo-language understanding. Advances in Neural Information Processing Systems, 37: 28828-28857, 2025.
[51] Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embodding, 2023.
[52] Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu. Mohit Bansal, and Gedas Bertasius.
A simple Ilm framework for long-range video question-unswering. arXiv preprint arXiv: 17235, 2023.
[53] Lu Zhang, Tiancheng Zhao, Heting Ying, Yibo Ma, and Kyusong Lee. Omagent: A multi-modal agent framework for complex video understanding with task swvid--and-conuue arXiv preprint arXiv:2406. 16620, 2024.
[54] Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang. Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406. 16852, 2024.
[55] Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, 2024.
[56] Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma. Ziwei Liu, and Chunyuan Li. Vidco instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713. 2024.
[57] Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang. Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint ar Xiv:22406.0422
2024.
[58] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint 2r3: 10592, 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper discussed the limitations of the work in Appendix D.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
The paper does not contain theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper disclosed all the information needed to reproduce the main experi- mental results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification: The paper provided open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/ public/guides/ wwtpe//ooSSSSeiiiiinneneeeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https: CCodeSubmiis
httpps.cccpubiic//eecc / ssi asionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specified all the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: While we fixed the random seed to ensure experimental stability and repro-
ducibility, we did not report error bars or conduct statistical significance testing.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provided sufficient information on the computer resources needed to reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: httt:::://rrrpps.cc/u/l/////e/e
Answer: [Yes]
Justification: The research is conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The models used in the paper are properly credited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https htt////eerpppp.pe http://aaap..c///oeeeeeeeeeeeeee 22/ps///ee.oeeeeeeee/22///////// for what should or should not be described.
20

---

A Experiments
A.1 Category-level performance on MLVU
Table shows performance on the multiple-choice task of MLVU [57]. Our framework consis- tently improves all models, enhancing VU by 5.4% and Qwen2.5VL (7B) by 3.3%. Notably, Vgent achieves 70.4% accuracy on Qwen2.5VL. (3B), surpassing its 7B counterpart and improving the base model by 4.2%. Significant gains are observed in Count and Order tasks, highlighting the effectiveness of our approach in srome-tegment reasoning and long- long-video understanding.
Table 6: Category-level performance MI.U [57]. Our framework consistently improves all models, enhancing LongVU by 5.4% and Qwen2.5VL (7B) by 3.3%. Notably, Vgent achieves 70.4% accuracy on Qwen2.5VL (3B), surpassing 7B counterpart and improving the base model by 4.2%. Significant gains are observed in Count and Order tasks, highlighting the effectiveness of our approach cross-segment reasoning and long-vides understanding.
Model
Size Count
Ego
Needle Order
PlotQA
Anomaly
Topic
Overall
Proprietary LVLMS
GPT-4o
46.3
57.1
64.8
56.7
65.1
74.5
87.4
Open-Source LVLMs
Intern [8]
2B
34.9
50.4
61.6
34.7
62.8
61.5
81.5
56.7
VL2.5 Vgent (Ours)
2B
59.2
53.1
66.7
38.2
63.9
62.0
61.1+14
Qwen2-VL [46]
2B
30.1
56.0
72.3
32.8
65.3
55.5
80.4
58.6
Qwen2-VL Vgent (Ours)
2B
58.7
57.6
76.9
34.3
63.8
59.5
80.3
62.5+3.
Qwen2.5-VL [4]
3B
36.4
53.0
77.7
55.5
70.1
75.5
86.4
66.2
Qwen2.5-VL Vgent (Ours)
3
60.1
58.1
78.5
61.7
70.5
74.5
87.3
70.4
LU [54]
7B
28.9
59.3
76.3
58.3
71.6
76.0
LongVU Vgent (Ours)
7B
60.0
62.3
76.5
60.1
71.6
76.4
87.8
70.8+5.4
Qwen2-VI. [46]
7B
32.5
62.0
79.1
53.2
69.6
63.0
85.3
65.7
Qwen2-V1. Vgent (Ours)
7B
60.2
65.8
80.2
60.2
70.6
63.5
70.3
LLaVA-Video [56]
7B
42.2
61.5
76.3
61.0
75.8
72.0
85.3
69.5
LLa VA-Video Vgent (Ours)
7B
58.7
63.0
76.9
67.1
76.4
72.5
86.9
72.5+3.0
Qwen2.5-V1.
7B
41.7
58.1
78.0
61.0
73.6
72.5
87.4
Qwen2.5-V1. Vgent (Ours)
7B
58.7
59.5
79.7
67.1
74.6
74.0
88.1
72.1 43.3
A.2 Confidence-based Refinement
A straightforward solution is filter out the hard negative retrievals by their relevance scores. Initially, we experimented with confidence-ber based refinement, as used in VideoAgent [47], where the model self-reflect the relevance of retrieved nodes. However, this approach proved ineffective in case, as the confidence score failed to reliably reflect video clip relevance, leading to an average improvement of only 0.2%, as shown in Table 7.
Table 7: Ablation study results the performance improvement contributed by each component of proposed pipeline. CR denotes confidence-buutt reasoning and SR is proposed structured reasoning.
Models
MLVU VideoMME Long
Qwen2.5-V1.
68.8
71.1
56.0
Qwen2.5-VI. GraphRAG +
69.5
72.9
57.5
Qwen2.5-VL- GraphRAG SR (default)
72.1
74.3
59.7
21

---

A.3 Baseline Details
Following GoldFish [3], we construct NaiveRAG baseline for video understanding by representing each video clip as plain text and retrieving relevant clips based on similarity to the query.
NaïveRAG:
This method selects keyframes by evaluating CLIP similarity between each frame's features and the text embeddings of keywords extracted from the question. Additionally, an object detection model and an Optical Character Recognition (OCR) model are applied to each keyframe extract detailed information.
Video-RAG:
[33]:
VideoAgent [47], LLoVi [52], DrVideo [34] and VideoTree [49] utilizes interactive reasoning and planning of proprietary L.M APIs to enhance long-video understanding.
Proprictary LLM-based:
A.4
Retrieval Embedding
We explore different types of retrieval embeddings. i.e., CLIP [41], BERT [10] and BGE [51] on VideoMME [13] benchmark with Qwen2.5-VI. [4] backbone, as shown Figure (left).
A.5 Number of Retrieval N
We conduct ablation on the number of retrieval N before structured reasoning (SR) VideoMME [13] benchmark with Qwen2.5-VI. [4] backbone, as shown Figure (middle). We set N 20 by default.
A.6 Retrieval Threshold
We investigate retrieval threshold r on VideoMME [13] benchmark with Qwen2.5-V2 VL [4] backbone, as shown in Figure (right). As the value of increases, less video clips are retrieved based on similarity scores, potentially leading to the loss of relevant information. We set = 0.5 by default.
A.7 Qualitative Results
We show a qualitative example Figure 3, and 6, Our graph construction effectively connects relevant video clips through shared entities. In Figure 3 the graph-based retrieval system identify relevant nodes that contains a laptop, with Clip providing crucial evidence to answer the query. However, the model incorrectly responded "No" to the question "Did I open the laptop?", presumably due to hard negatives from multiple clips featuring a opened laptop, hallucinating the model to overlook the closed laptop and the action of opening it.
In contrast, with an intermediate reasoning step, we validate each retrieved node with structured subqueries (c.g., "Is there a laptop open?" "Is someone interacting with the laptop?"). This verified information is aggregated to form an enhanced reasoning chain, allowing the model to correctly infer that the laptop was opened, overcoming the distraction from hard negatives.
Stert
Metion
Madken
70
2EPT
CUP
802
15
28
044
5.5
0.5
Retrieval Embedding
TopN Retrieval
Retrieval Threshold
Figure 4: Ablation studies. Left: retrieval embedding. Middle: number of retrieval N before SR. Right: ablation on retrieval threshold T.
22

---

Clip
Clip
Video Graph
Queeium: Amazge following coente
acrss clecnological onder man altempo
rope;
ocean;
filss
ton: (6) mun acmoss rope
may the sttzchal
Graph-based Clip Retrieval
GraphttGG w/o reasonáng
S Answer: 2-94-91-93
[8,2,3,4,6]
Strutured
GrapeRAG with reasontigg
Reasoning
01: Does the nan atsppts to walk
seal sitting by the ocean is an initial scene. There is nan atterpting to walk across the rope, who then falls, holds onto it and arrives the rock. Then the man filming tron beach clift.
across the roge but falls and holds onto the repe?
02: Is there seal sitting on rock
near ocean" shown in video7
03: Boes the nan fila from beach
cliff
tent?
the
Answer: 2-91-94-03
way
Figure 5: A qualitative example illustrates our graph-based retrieval-rcasoning approach.
Clip
Clip
Clip
Clip 4
Clip 5
Clip g
CLip
Clip 11
Video Graph Construction
In this video, how many instaaces are there of the 'cooking stusages' action scene total?
Question:
tras
Graph-based CLip
GraphRAG w/o reaconing
0,4,9,10]
Strutured Queries
GraphRAG with reasoning
Reasoning
The video shows instances of caoking sausages in two segnents.
01: Is there cooking sausages in the video?
Answer:
Figure 6: A qualitative example illustrates our graph-based retrieval-rcasonin approach.
B Prompts
B.1 Visual Entity Extraction
Figure illustrates the prompts used to describe entities, actions, and scenes given a video segment for the LVLM.
B.2 Keyword Extraction
Figure 8 presents the prompt designed for the LVLM to perform task identification and extract keywords from the original question to facilitate retrieval.
B.3 Subqueries Generation
Figure presents the prompt designed for the LVLM to generate structured subqueries for retrieved nodes refinement.
23

---

Prompt:
Describe entities, actions, scenes.
Please analyze the given video and extract key information in structured JSON format in English. Identify and describe:
List all distinct objects, people, animals, or other significant elements present in the video.
Entities:
If the entities are interacting, describe their actions and relationships in a structured manner.
Actions:
Scenes: Identify and describe the locations, environments, or contexts where the events occur. If the video is filmed from first-person point of view, please also describe "subject" as "me" and actions or interactions from this person.
Ensure the output strictly follows the JSON format below:
"entittes": ("entity name": "description": ""], "actions": ["entity name": "action description"), "scenes": ["location": "n]
The "entity name" in actions should belong to "entity name" entities.
Each section should be detailed but concise, capturing all relevant interactions and contextual elements from the video. Avoid unnecessary text outside the JSON output.
Figure 7: Prompt for video segment description.
Prompt: Task Identification and Keyword Extraction.
Given a question of a long video and potential candidates:
Question: {query} Candidates: {candidates}
You need to retrieve the relevant video segments to answer the question. Note that you do not need to see the video. But based on the question please think step by step what are the important things for retrieval.
[keywords] Please identify the information, like entities, scene, action from the question that is important to retrieve the segments for further answer the question. Do not include the candidates in the keywords.
[candidates_nne] _ Do you think the information in the candidates is necessary for retrieval? Answer yes or no.
[multiple] Do you think it needs to aggregate the information from multiple segments to answer the question? ONLY answer yes or no.
[time] Please identify if it can tell the question is asking which part of the video. Answer begin, end or none.
[tool] Do you think it needs additional step for answering the question, please select from [object counting, action counting, order, none]. [global] Can this question be answered based on the overall understanding of the whole video? (e.g., "What is the main topic of the video?" or "What is the main content of the video?")
Please output the final answer in json format, for example:
|"multiple": "no", "keywords": ["man in black"], "time": "begin", "tool": none, "candi- kates_necessoe "yes", "global": "yes"]
Figure 8: Prompt for task identification and keywords extraction.
24

---

Prompt: Subqueries Generation.
Given question of long video and potential candidates:
Question: {query}
Candidates: {candidates)
Given multiple-choice question about a video, break it down into several sub-questions that analyze the key elements required answer it step by step.
First, identify the key subject or event in the question (e.g., an object, an animal, an action, or location). Form yes/no or counting questions verify the presence of the subject event in the video (c.g., "Does the video show [subject/event]?"). Ensure the sub-questions cover all necessary aspects to reach the correct answer.
==important== Please give me the answer in JSON format. Do not include references to specific time positions the video when generating questions (e.g., "at the beginning. "in the middle," or "at the end") Do not go through all the numbers in the candidates for counting quesitons.
Figure 9: Prompt for subqueries generation.
Model Output Examples
C.1 Visual Entity Extraction
"idx": 0, "info": "entities";
"entity name": "sailboat", "description": "A classic sailboat with white sails and wooden rigging"
"entity name": "man", "description": "A man wearing dark sweater"
"entity name": "ocean",
"description": "A calm ocean under partly cloudy sky
].
"actions":
entitty name": "sailboat", "description": "sailing smoothly on the water"
"entity name": "man" "description": "steering the sailboat" "
>,
{
"entity name": "man" "description": "looking around"
{
"entity name": "man", "description": "adjusting his hair"
].
"scenes": [
25

---

llccation;
"open sea"
D Limitation
In this work, represent video content using textual deseriptions- such entities and their associated details-as a lightweight and efficient alternative to raw visual features. However, we do not incorporate visual embeddings frame-level features into graph. While computing similarity across frames can be computationally intensive, it remains a promising direction for future improvement.
Additionally, our framework is model-agnostic and compatible with any LVLM, meaning its per- formance is inherently bounded by the capabilities of the base LVLMM As more powerful LVLMMs emerge, our pipeline can be readily adapted to take advantage of their enhanced video understanding and reasoning abilities.
26


Reviews:
[{"review_id": "z85mitIxuP", "reviewer": "Reviewer_3ccS", "summary": "It introduces the Vgent, a graph-based Retrieval-Augmented Generation (RAG) framework tailored for long video understanding. The method addresses the inherent challenges of applying RAG in the video domain, such as disrupted temporal dependencies and noise from irrelevant content. This paper proposes a novel pipeline that includes (i) offline video graph construction to preserve semantic and temporal relationships across clips, and (ii) a structured post-retrieval reasoning step that filters retrieved clips and aggregates relevant information for more accurate responses. The Vagent framework demonstrates consistent improvements (3.0%–5.4%) over base open-source LVLMs and outperforms state-of-the-art RAG-based methods by 8.6% across three long-video benchmarks.", "questions": "1. This work selects the clips according to the embedding similarity, which might have problems of false-positive and true-negative. The method introduces structured query refinement to try to filter the false-positive question. So how the framework resists the true-negative problem if some key information is lost in the early stage of the pipeline?\n\n2. Have you considered non-entity-based or multi-modal graph construction (e.g., action sequences, visual similarity)? Would incorporating motion features (e.g., optical flow) improve temporal coherence?\n\n3. In Section 4.3, the Ablation Studies section, you show that adding Structured Reasoning (SR) shows more improvement than the GraphRAG module, showing the importance of the SR approach, but the source of this improvement remains somewhat abstract. Have you conducted a finer-grained analysis on where SR contributes the most? This is what i am very curious.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n1. This paper provides a quite solid and interesting method. It combines graph-based representation and middle-stage reasoning into one pipeline, which doesn’t need extra training. The idea is fresh and practical.\n\n2. The method is introduced clearly, and the diagrams are easy to understand. The whole paper has good structure and is not difficult to read.\n\n3. The problem of long video understanding is very meaningful. Especially now, large vision-language models have some trouble when facing long input. The results in experiments show that Vgent improves both retrieval accuracy and reasoning ability a lot.\n\n4. Although this work is based on some previous ideas like RAG and scene graph, the usage of offline entity video graph and structured sub-query checking is new and not so simple. It shows the authors did deep thinking.\n\nWeaknesses:\n\n1. It uses multiple calls for LVM tools, and there is accumulated errors in the graph construction and filtering process.\n\n2. The novelty is not limited compared to the existing scene-graph based VLLM works. the innovation lies it introduces a problem decomposition and verification step it is good but not so novel.\n\n3. Limited ablation on graph construction strategy: The authors assume the effectiveness of entity-level similarity and merging, but do not explore alternative graph construction techniques, like, motion cues, which is very important in some queries where dynamic information in videos are needed.\n\n4. The method relies heavily on accuracy improvements, but lacks deeper analysis on error types, qualitative improvements, or user studies for interpretability gains.", "was_revised": true, "final_justification": "I am overall satisfied with the author rebuttal.", "rebuttal": "**W1 (1): It uses multiple calls for LVM tools**\n\nThe utilization of multiple LVM calls is inherent to the design of sophisticated agent-based approaches, a common practice, not a weakness, in the field. For example, VideoTree [1] called LLM for relevance scoring by breadth and depth expansion. DrVideo [2] iteratively searches videos with an agent-based loop to converge on final predictions. Multiple calling should not be our weakness. Furthermore, our refinement and answer stages operate on a significantly reduced set of videos, N=20 after retrieval and r=5 after filtering, which significantly reduces computations.\n\n[1] VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos, CVPR 2025\n\n[2] DrVideo: Document Retrieval Based Long Video Understanding, CVPR 2025\n\n**W1 (2): There is accumulated errors in the graph construction and filtering process.**\n\nWe respectfully disagree with the claim that errors accumulate during graph construction and filtering. \nNaïve RAG can indeed suffer from information loss by summarizing video segments into textual description, however, we try to preserve information in a structured manner through graph construction. By saving entities along with corresponding attributes and actions, we alleviate the information loss and error accumulation issue compared to naïve text-based retrieval. To validate this, we evaluated our method on the temporal grounding benchmark ReXTime [1], measuring both retrieval performance (Recall@1 with IoU > 0.3) and final answer accuracy. As shown in the table below, our method (Vgent) outperforms the NaïveRAG baseline by a significant margin in both metrics:\n\n| ReXTime | R@1 (IoU>0.3) | Acc |\n| ----- | ----- | ----- |\n| NaïveRAG | 23.6 | 30.1 |\n| Vgent | 27.8 | 35.6 |\n\nMoreover, our filtering process reduces retrieval noise by eliminating irrelevant information during the retrieval stage, retaining only the most relevant video segments necessary for the final answer. This effectively alleviates potential error accumulation introduced during retrieval. Consequently, the integration of Structured Reasoning (SR) yields an average improvement of 2.2% across three benchmarks.\n\n| Methods | MLVU | VideoMME | LongVideoBench |\n| ----- | ----- | ----- | ----- |\n| Qwen2.5-VL + GraphRAG | 69.5 | 72.7 | 57.1 |\n| Qwen2.5-VL + GraphRAG + SR | 72.1 | 74.3 | 59.7 |\n\n[1] ReXTime: A Benchmark Suite for Reasoning-Across-Time in Videos, NeurIPS 2024\n\n**W2: The novelty is not limited compared to the existing scene-graph based VLLM works. the innovation lies it introduces a problem decomposition and verification step it is good but not so novel.**\n\nWe respectfully disagree with the comment on limited novelty. Our work introduces two distinct and significant innovations: \n+ First, we've introduced the idea of cross-clip graph-based retrieval for video understanding. This is a novel approach that differentiates us from existing scene-graph based VLLM methods. For instance, approaches like VideoRAG are limited to frame-level scene graphs, neglecting the crucial relationships that span across frames or clips. However, we innovate by connecting distinct video segments via shared entities, which preserves semantic relationships and contextual dependencies.\n+ Second, it is not just simple verification. One naive way is just to let the model verify whether each retrieved video segments are relevant or not. We designed the structured reasoning as a post-retrieval refinement step. This mechanism, designed to enhance the accuracy and robustness of retrieved information, is both novel and empirically effective, which has not been explored in existing literature.\n\n**W3 & Q2 (1): Limited ablation on graph construction strategy: The authors assume the effectiveness of entity-level similarity and merging, but do not explore alternative graph construction techniques, like, motion cues, which is very important in some queries where dynamic information in videos are needed.**\n\nWe appreciate the reviewer's excellent point regarding the exploration of alternative graph construction techniques and the incorporation of multimodal information. Our approach is not exclusively limited to static entity names. As detailed in Appendix C.1, our visual entity extraction process also includes actions, providing a level of dynamic information. \n\n**W3 & Q2 (2): Have you considered non-entity-based or multi-modal graph construction (e.g., action sequences, visual similarity)?**\n \nWe have explored incorporating visual features of entities into the graph construction process. Specifically, we employed the Florence-2-large model to generate detailed descriptions and bounding boxes for each detected entity. Using these, we cropped the corresponding regions and extracted visual features for integration into the graph. However, this approach introduced substantial computational overhead during both feature extraction and node merging. Moreover, our preliminary experiments showed only marginal performance improvements, i.e., 0.6% on MLVU. As a result, we did not include this design in the final version of our pipeline. We consider multimodal graph construction a promising direction for future work.\n\n**W3 & Q2 (3): Would incorporating motion features (e.g., optical flow) improve temporal coherence?**\n\nRegarding more dense and low-level motion cues like optical flow, we recognize their potential. However, computing optical flow is very computationally intensive, especially for long videos, which is not a primary focus of our work. We agree that temporal motions derived from optical flow could be beneficial in specific, action-focused benchmarks that demand fine-grained dynamic understanding. This remains a promising avenue for future work.\n\n**W4: The method relies heavily on accuracy improvements, but lacks deeper analysis on error types, qualitative improvements, or user studies for interpretability gains.**\n\nWe've conducted significant qualitative analysis of the enhancements derived from both our graph construction and structured reasoning components. Specifically, we've investigated the qualitative improvements resulting from both our graph construction and structured reasoning components. Detailed examples and visual evidence illustrating these gains can be found in A.8 and Figures 1, 6, and 7 of our supplementary material (PDF in the zip file). Furthermore, Table 6 presents category-level performance on MLVU, investigating how the proposed method gains across specific tasks. Notably, we observe substantial gains in action counting and ordering tasks, highlighting the effectiveness of our approach in cross-segment temporal reasoning.\n\n| Model              | Count | Order |\n| ------------------ | ----- | ----- |\n| Qwen2.5-VL         | 41.7  | 61.0  |\n| Qwen2.5-VL + Vgent | 58.7  | 67.1  |\n\n**Q1: This work selects the clips according to the embedding similarity, which might have problems of false-positive and true-negative. The method introduces structured query refinement to try to filter the false-positive question. So how the framework resists the true-negative problem if some key information is lost in the early stage of the pipeline?**\n\nWe assume that the reviewer’s concern about the “true-negative” problem is a question of whether key information might be lost during the retrieval stage prior to structured reasoning.\n\nOur ranking-based retrieval approach is designed to tolerate retrieval errors (true-negative problem). And by adjusting retrieval threshold ($\\tau$) and the number of retrieved candidates (N), we effectively minimize information loss. We conduct ablation studies on the $\\tau$ and $N$, as shown in Figure 5 (in the supplementary PDF within the zip file). The results show that increasing the retrieval size ($N = 20$) leads to optimal performance, particularly on the long-video split. This improvement stems from a larger candidate set to retain more relevant information. Beyond $N=20$, further increases in $N$ yield only minimal additional enhancement.\n\n| N    | Short | Medium | Long |\n| ---- | ----- | ------ | ---- |\n| 5    | 76.5  | 74.2   | 61   |\n| 10   | 76.8  | 75.3   | 64.1 |\n| 15   | 77.1  | 77.3   | 66.5 |\n| 20   | 77.2  | 78.0  | 67.7 |\n| 25   | 77.2  | 78.1   | 67.9 |\n\nSubsequently, our structured refinement module effectively addresses the false-positive problem by filtering out irrelevant or noisy retrievals.\n\n**Q3: Why adding Structured Reasoning (SR) shows more improvement than the GraphRAG module? Finer-grained analysis on where SR contributes the most?**\n\nSince direct ground truth for retrieval accuracy is unavailable, we manually examined both the retrieved nodes and the model's responses for some examples. We observed, in many cases, the correct video segment needed for a question was indeed retrieved, yet the model produced an incorrect answer. However, when the model was given only this accurate segment, it often yielded the correct response, indicating that retrieval noise (ranking-based Top N retrieval) significantly contributes to prediction errors.\n\nThis explains why relying solely on graph-based retrieval offered only marginal performance gains, and why structured reasoning played a more critical role.\n\nIn A.8 of our supplementary materials (pdf in zip file), we present several fine-grained cases that demonstrate how GraphRAG, when combined with structured queries and reasoning, leads to accurate answers. For instance, Figure 6 in the supplementary materials shows that, w/o SR, the model failed to correctly answer an action ordering question based on the retrieved segments. However, by verifying and filtering out irrelevant segments and incorporating an intermediate reasoning step to summarize the sequence of events, the model successfully produces the correct answer."}, {"review_id": "QxT6OxxY9q", "reviewer": "Reviewer_FuXV", "summary": "This paper proposes Vgent, a novel graph-based retrieval-reasoning-augmented generation framework that enhances open-source Long Video Language Models (LVLMs) in long video understanding tasks. Vgent constructs a Video Knowledge Graph with semantic relationships across video clips preserved to improve retrieval effectiveness and introduces Structured Query Refinement (SQR) to perform intermediate reasoning and information aggregation. Evaluations on three popular benchmarks (MLVU, VideoMME, LongVideoBench) according to Table 1 demonstrate consistent performance improvements across various LVLM backbones.", "questions": "- Were the same subtitle/transcript settings applied to all baseline models in Table 1 and 2? How would Vgent perform under *pure visual settings* (e.g., VideoMME-w/o subs or visual-only MLVU)?\n- Could the proposed cross-clip graph reasoning approach be further validated on benchmarks focusing specifically on **cross-segment temporal reasoning**?\n- What is the upper limit of video frame length that Vgent can handle with its graph-based retrieval structure? Can it scale to hour-long videos? What are the average processing times on VideoMME-long and LVBench?\n- Appendix Figure 5 suggests that Top-N selection has limited impact on VideoMME performance—does the same trend hold on other datasets like MLVU and LongVideoBench?", "limitations": "The authors discuss limitations in detail in Appendix D.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### **Strengths:**\n\n1. The proposed training-free graph-based retrieval and reasoning framework eliminates dependency on proprietary models or large external backbones, providing a flexible and self-contained solution.\n2. Vgent achieves consistent improvements on three widely-used long video understanding benchmarks (MLVU, VideoMME, LongVideoBench), and an experimental comparison is made with recent work on similar graph structure methods such as Video-RAG.\n\n### **Weaknesses:**\n\n- **Concerns about fair comparison.** As stated in lines 236–238, the authors used `openai/whisper-large` for spoken content extraction in MLVU and used benchmark-provided subtitles for VideoMME and LongVideoBench. It is unclear whether the same preprocessing pipeline was applied in comparison baselines  in Table 1 and 2(especially Table 2, e.g., “Qwen2.5-VL + Video-RAG” and “LLaVA-Video + Video-RAG”). Moreover, Table 1 reports results for LongVU on MLVU that align with the original paper, while the LLaVA-Video results differ. Many existing long video models (e.g., LongVA, LongVILA, VideoChat-flash, Apollo) do not include audio-derived transcripts during evaluation, as such textual tokens may heavily influence model performance. A comparison under *pure visual settings* with SOTA methods would better validate the visual reasoning capability of Vgent.\n- The proposed **Structured Query Refinement (SQR)** involves decomposing a query into multiple sub-questions and validating against multiple retrieved clips with VLM. While this increases accuracy, it may introduce significant **inference latency**, especially when scaling to longer videos or larger candidate sets.\n- The paper lacks ablation studies on critical hyperparameters such as **Top-N retrieval size**. The choice of N directly impacts whether the reasoning module can access necessary clips for answering decomposed binary questions. More discussions are needed on how N influences model performance and efficiency trade-offs.", "was_revised": true, "final_justification": "The authors’ rebuttal satisfactorily addresses my concerns, particularly the comparison of results with and without subtitles. I recommend incorporating these findings into the revision, and I am inclined to raise my score.", "rebuttal": "**W1 (1) & Q1: Concerns about fair comparison for using video subtitles.**\n\nFor evaluations on both VideoMME and LongVideoBench, all models utilize the subtitles provided by the respective benchmarks, ensuring a fair and consistent comparison across all approaches. For MLVU, we extracted subtitles to ensure consistent, fair comparisons with other agent-based or tool-using methods like Video-RAG.\n\n**W1 (2) : Moreover, Table 1 reports results for LongVU on MLVU that align with the original paper, while the LLaVA-Video results differ.**\n\nOur reproduced result for LLaVA-Video matched their reported result in the original paper.\n\n| Models | MLVU | VideoMME |\n| -------- | ------- | ------- |\n| LLaVA-Video-7B (reported in original paper)  | 70.8 | 69.7 |\n| LLaVA-Video-7B (reproduced) | 69.5 | 69.2 |\n\n**W1 (3): Many existing long video models (e.g., LongVA, LongVILA, VideoChat-flash, Apollo) do not include audio-derived transcripts during evaluation, as such textual tokens may heavily influence model performance.**\n\nFor VideoMME, most of the models mentioned above (LongVILA, VideoChat-Flash, Apollo) reported the performance with subtitles. Since questions in VideoMME require the model to understand both visual and spoken content, those questions could not be answered solely on visual information. Unlike VideoMME, MLVU does not contain dense conversational or spoken content. The questions in MLVU are primarily focused on visual understanding and are independent from audio and spoken information.\n\n**W1 (4): A comparison under pure visual settings with SOTA methods would better validate the visual reasoning capability of Vgent.**\n\nWe appreciate the reviewer's suggestion for a comparison under pure visual settings to validate visual reasoning capabilities. We provided the result comparing performance with and without transcriptions/subtitles below.\n\n| Models | Transcriptions / Subtitles | MLVU | VideoMME |\n| ------ | -------------------------- | ---- | -------- |\n| Qwen2.5-VL | No | 68.8 | 65.1 |\n| Qwen2.5-VL + Video-RAG | No | 63.1 | 61.2 |\n| Qwen2.5-VL + Vgent | No | 71.8 | 69.9 |\n| Qwen2.5-VL | Yes | 69.0 | 71.1 |\n| Qwen2.5-VL + Video-RAG | Yes | 63.4 | 65.7 |\n| Qwen2.5-VL + Vgent | Yes | 72.1 | 74.3 |\n\n**W2: While SR increases accuracy, it may introduce significant inference latency, especially when scaling to longer videos or larger candidate sets.**\n\nWe respectfully disagree with this statement. Structured Reasoning (SR) is applied only after the Top-$N$ retrieval, meaning it processes at most $N$ segments. As a result, its computational cost does not scale with the length of the video and does not introduce additional latency for longer inputs.\n\n**W3: The paper lacks ablation studies on critical hyperparameters such as Top-N retrieval size. The choice of N directly impacts whether the reasoning module can access necessary clips for answering decomposed binary questions. More discussions are needed on how N influences model performance and efficiency trade-offs.**\n\nWe appreciate the reviewer's comment regarding ablation studies on critical hyperparameters. We have indeed conducted a detailed ablation study on the Top-$N$ retrieval size, as presented in A.6 of our supplementary material (zip file). The results, also summarized below, illustrate the impact of $N$:\n\n| N    | Short | Medium | Long |\n| ---- | ----- | ------ | ---- |\n| 5    | 76.5  | 74.2   | 61   |\n| 10   | 76.8  | 75.3   | 64.1 |\n| 15   | 77.1  | 77.3   | 66.5 |\n| 20 (default)   | 77.2  | 78.0  | 67.7 |\n| 25   | 77.2  | 78.1   | 67.9 |\n\nAs shown, increasing the retrieval size from $N=5$ to $N=20$ consistently leads to more optimal performance, particularly noticeable on long videos. This improvement stems from providing greater tolerance to embedding search errors, ensuring the reasoning module can access necessary clips. Furthermore, potential errors in the Top-$N$ retrieved segments are mitigated through structured reasoning, which effectively filters out retrieval noise. Beyond $N=20$, further increases in size introduce only marginal gains while raising computational costs, indicating that most relevant segments are already retrieved within this range.\n\n**Q2: Could the proposed cross-clip graph reasoning approach be further validated on benchmarks focusing specifically on cross-segment temporal reasoning?**\n\nTable 2 in the supplementary material presents category-level performance on MLVU. The Order category represents a cross-segment temporal reasoning task, involving questions that require understanding the order of different actions across multiple video segments. *(We post one example here: \"Arrange the following events from the video in the correct chronological order: (1) Woman tapes her hands with white tape; (2) Woman starts boxing in the ring with a guy; (3) Woman does sit ups on a towel on the beach; (4) Pictures of woman in her bikini are shown.\")*. In this challenging category, our pipeline achieves a 6.1% improvement over the base model Qwen2.5-VL, demonstrating its effectiveness in temporal reasoning.\n\n**Q3 (1): What is the upper limit of video frame length that Vgent can handle with its graph-based retrieval structure? Can it scale to hour-long videos?**\n\nVgent's design, utilizing a graph-based retrieval structure, offers unlimited scalability regarding video frame length, making it fully capable of handling hour-long videos and beyond. The core strength lies in its ranking-based retrieval mechanism, which efficiently identifies and prioritizes the most relevant video segments irrespective of the overall video's duration. This capability is empirically validated by our results on the VideoMME benchmark's long split, where video durations range from 30 to 60 minutes. As presented in Table 1 of our main paper, Vgent significantly outperforms the base model in answer accuracy for long video scenarios.\n\n**Q3 (2): What are the average processing times on VideoMME-long and LVBench?**\n\nFor VideoMME's long split (videos ranging from 30 to 60 minutes), the average processing time is 297.4 seconds. It's important to note that our graph construction is query-independent. Since each VideoMME video typically has three associated questions, the graph is built only once per video, significantly amortizing this cost across multiple queries. For LongVideoBench, the average processing time is 182.8 seconds.\n\n**Q4: Appendix Figure 5 suggests that Top-N selection has limited impact on VideoMME performance—does the same trend hold on other datasets like MLVU and LongVideoBench?**\n\nWe appreciate the reviewer’s observation. The perceived limited impact in Figure 5 may be due to scaling issues in the visualization. As shown in the table above, Top-N retrieval actually has a significant influence on VideoMME performance. Specifically, increasing the number of retrieved candidates from $N=5$ to $N=20$ improves the accuracy from 61.1% to 67.7% on VideoMME long split, representing a notable gain of 6.6%. We observe a similar trend on other datasets such as MLVU and LongVideoBench, where larger $N$ values generally lead to improved retrieval quality and downstream performance.\n\n| N  | MLVU | LongVideoBench | VideoMME (long) |\n| ---- | ----- | ------ | ---- |\n| 5    | 70.2  | 57.3   | 61 |\n| 20   | 72.1 | 59.7 | 67.7 |"}, {"review_id": "GFRFosd7zu", "reviewer": "Reviewer_6nEZ", "summary": "The paper proposes Vgent, a novel graph-based RAG framework designed to improve long video understanding in Large Video Language Models. By representing videos as structured graphs and incorporating an intermediate reasoning step with information aggregation, Vgent addresses key challenges in long-context retrieval and reasoning. Experimental results on three main long-video benchmarks demonstrate consistent performance gains over base models and improvement over prior RAG-based methods.", "questions": "1.\tIn both the entity merging and clip retrieval stages, the similarity is computed based solely on the text embeddings of entity names. How does the method address issues of coreference and ambiguity (e.g., when two different people are both referred to as “a man”)? A more detailed discussion on this would be helpful.\n\n2.\tBuilding an accurate graph from video content is inherently challenging. Has there been any deeper analysis of the quality of the constructed graphs, including ablation studies on parameters like τ?\n\n3.\tSince graph construction is likely to be computationally intensive, a time analysis of this component would be valuable for understanding the overall efficiency.\n\n4.\tSome implementation details could be clarified. In Table 1, for the different LVLM+Vgent settings, was the graph constructed using the corresponding LVLM in each case? Also, what were the inference-time model settings used in these experiments? It would be good to report them.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n⦁\tThe proposed method of representing video segments as graph nodes is inspiring, enabling the construction of structured connection between video clips.\n\n⦁\tThe divide-and-conquer strategy employed in the Retrieval-Reasoning Augmented Generation effectively refines the retrieved video clips, which alleviates the limitations of LVLMs in graph construction and reasoning. This approach is intuitive and makes sense.\n\n⦁\tThe paper conducts extensive experiments on three widely used long-video benchmarks, with very detailed discussion on the results. The proposed method consistently improves performance across major LVLMs and outperforms existing approaches, demonstrating both effectiveness and generalizability.\n\nWeaknesses:\n\n⦁\tThe paper lacks a more in-depth analysis of the Offline Graph, including its construction efficiency, accuracy, and both qualitative and quantitative evaluations.", "was_revised": true, "final_justification": "All my conerns have been addressed. So  I am inclined to accept the paper.", "rebuttal": "**W1 & Q2: The paper lacks a more in-depth analysis of the Offline Graph, including its construction efficiency, accuracy, and both qualitative and quantitative evaluations.**\n\nWe have conducted a comprehensive analysis of the Offline Graph's impact and performance through extensive ablations and qualitative examples in our supplementary material. Specifically, we provide detailed quantitative evaluations in the supplementary material covering:\n\n+ The impact of different retrieval embedding choices (A.5).\n+ The optimal number of retrieved items ($N$) (A.6).\n+ The effectiveness of the retrieval threshold ($\\tau$) (A.7).\n+ Furthermore, our analysis includes significant qualitative evaluations, with several illustrative examples presented in A.8 and Figures 1, 6, and 7.\n\n**Q1: In both the entity merging and clip retrieval stages, the similarity is computed based solely on the text embeddings of entity names. How does the method address issues of coreference and ambiguity (e.g., when two different people are both referred to as “a man”)? A more detailed discussion on this would be helpful.**\n\nWe appreciate the reviewer's insightful question regarding coreference and ambiguity in entity merging and clip retrieval. It is important to clarify that our method does not solely rely on entity names for similarity computation.\n\nAs shown in Equation 2, we maintain an entity-description pair $(e_j^i​,t_j^i​)$ for each entity $j$ within video clip $V_i$​. The similarity score for merging is then computed between the textual description $t_j^i$​ and the descriptions of existing entities in our global entity set $U$, based on their respective text embeddings. We prompt the model to generate detailed descriptions for each entity, thereby mitigating coreference and ambiguity issues by enriching the textual context with more distinctive and informative cues.\n\nWe also explored incorporating visual features of entities into the graph construction process. Specifically, we employed the Florence-2-large model to generate detailed descriptions and bounding boxes for each detected entity. Using these, we cropped the corresponding regions and extracted visual features for integration into the graph. However, this approach introduced substantial computational overhead during both feature extraction and node merging. Moreover, our preliminary experiments showed only marginal performance improvements, i.e., 0.6% on MLVU. As a result, we did not include this design in the final version of our pipeline. We consider multimodal graph construction a promising direction for future work.\n\n**Q3: Time analysis of graph construction.**\n\nWe show the inference time analysis of graph construction in Table 4 in the main paper. The offline, query-independent graph construction requires 20.13 seconds to process every minute of video on average.\n\n| Model / Time(sec) | Graph Construction (offline) | Retrieval, Reasoning (online) |\n| ----------------- | ---------------------------- | ----------------------------- |\n| VideoAgent        | N/A                          | 67.25                         |\n| Qwen2.5VL-7B      | N/A                          | 2.95                          |\n| Video-RAG         | N/A                          | 20.81                         |\n| Vgent (Ours)      | 20.13                        | 3.93                          |\n\n**Q4: Was the graph constructed using the corresponding LVLM in each case? What were the inference-time model settings used in these experiments?**\n\nYes, we use the same model for graph construction and question answering. Regarding the inference-time model settings, we adhered to the configurations (context size, number of frames) specified by each corresponding base model, and reproduced the matched results as reported in their original papers."}, {"review_id": "CuQAS7bgBE", "reviewer": "Reviewer_wVMm", "summary": "This paper presents Vgent, a comprehensive RAG-based framework for long-form video understanding. The key idea is to build the GraphRAG that models video clips as graph nodes, followed by a structured reasoning module, where subqueries are automatically generated to verify and filter noisy content. Extensive experiments on benchmarks like MLVU, LongVideoBench, and VideoMME show that Vgent consistently improves the performance of open-source LVLMs and outperforms prior RAG baselines.", "questions": "Some additional questions:\n\nHow does Vgent perform without subtitle input? \n\nIs there any special reason not including other datasets such as LVBench and MVBench in the testing?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nThe graph-based retrieval module is query-independent and constructed offline, making it efficient and scalable across multi-turn or multi-query QA tasks. Although this feature isn’t particularly novel, it remains essential.\n\nThe proposed approach achieves consistent performance improvements over baselines. Ablation studies have also been well done.\n\nThe paper is also well structured and clearly presented.\n\nWeaknesses:\n\nThe proposed framework improves existing training-free, RAG-based long-form video understanding methods. Although the proposed modules are solid and effective, the overall framework may not have sufficient novelty to be considered a top paper. \n\nThe potential and future work of the proposed method are unclear. I may raise my rating if the authors can well address this part.", "was_revised": true, "final_justification": "The authors have well addressed my concerns. The explanation of future work sounds good, especially the incorporation of visual features. The new results in terms of with and without subtitle look reasonable.", "rebuttal": "**W1: Although the proposed modules are solid and effective, the overall framework may not have sufficient novelty.**\n\nWe respectfully assert the significant novelty of our work, grounded in two key innovations:\nPioneering cross-clip graph-based retrieval for video understanding: Unlike existing frame-level scene graph methods (e.g., VideoRAG), our approach uniquely connects distinct video segments via shared entities, preserving crucial semantic and contextual relationships across clips.\n\nThen we proposed structured reasoning as a post-retrieval refinement step, a novel, empirically effective mechanism that enhances the accuracy and robustness of retrieved information.\n\n**W2: The potential and future work of the proposed method are unclear.**\nWe appreciate the reviewer's query regarding the potential and future work of our proposed method. Our pipeline offers significant advancements across several aspects:\n\n+ **Efficient for streaming or long-form videos.** First, we propose an offline graph construction mechanism that provides a structured representation of video segments. This process is performed in a query-independent manner and only once per video. As a result, in multi-round video question answering settings, subsequent queries for the same video can directly retrieve relevant segments without reprocessing the entire video. This significantly improves efficiency, especially for streaming or long-form videos. \n+ **Enhanced reasoning in post-retrieval stage.** Second, our proposed structured reasoning further enhances the model reasoning ability. LVLMs often suffer from limited reasoning ability and hallucinations, particularly when faced with hard negative retrievals. Our method addresses this by introducing an intermediate reasoning step to verify retrievals and aggregate information across verified segments, thereby improving the robustness and accuracy of the final response.\n\nIn future work, our pipeline can be extended by integrating richer multimodal information into the graph construction process. Due to current computational limitations, we are unable to incorporate visual feature similarity when building the graph. However, with more efficient computation and feature indexing techniques, future versions of the system can leverage visual cues to construct more informative graphs. Additionally, advancements in indexing strategies can further enhance the efficiency and scalability of the retrieval stage, enabling faster and more accurate candidate selection.\n\n**Q1: How does Vgent perform without subtitle input?**\n\nWe present below the result for Vgent without subtitle/transcription as input in comparison to base model Qwen2.5-VL and one agent-based baseline Video-RAG.\n\n| Models | Transcriptions / Subtitles | MLVU | VideoMME |\n| ------ | -------------------------- | ---- | -------- |\n| Qwen2.5-VL | No | 68.8 | 65.1 |\n| Qwen2.5-VL + Video-RAG | No | 63.1 | 61.2 |\n| Qwen2.5-VL + Vgent | No | 71.8 | 69.9 |\n| Qwen2.5-VL | Yes | 69.0 | 71.1 |\n| Qwen2.5-VL + Video-RAG | Yes | 63.4 | 65.7 |\n| Qwen2.5-VL + Vgent | Yes | 72.1 | 74.3 |\n\nFor VideoMME, questions are designed to necessitate an understanding of both visual and spoken content; consequently, they cannot be answered solely based on visual information. In contrast, MLVU datasets lack dense conversational or spoken content, with their questions primarily focused on visual understanding, independent of audio and spoken information.\n\n**Q2: Is there any special reason not including other datasets such as LVBench and MVBench in the testing?**\n\nMVBench is a short video benchmark, only 16 seconds on average. Our method is specifically designed for long video understanding, operating by segmenting videos into 32-second chunks for graph construction, retrieval, and reasoning. Applying our pipeline to MVBench would effectively reduce it to a naive baseline, as such short videos do not require retrieval or long-range reasoning. Our pipeline is more beneficial in scenarios that require understanding and reasoning over extended video content. As for LongVideoBench, we have presented our model's performance in comparison to the base model Qwen2.5-VL in Table 3 of the main paper."}]

Author rebuttal:

Meta-review: The paper presents Vgent, a graph-based retrieval-reasoning framework for long video understanding, which builds offline video graphs to preserve semantic/temporal relations and introduces structured query refinement for noise reduction and better reasoning. It is training-free, flexible, and reproducible, and demonstrates consistent and notable gains (3–5% over LVLM baselines, ~8% over prior RAG methods) across three standard benchmarks. The strengths lie in tackling an important and timely challenge, providing a practical and well-structured solution, and showing robust improvements across multiple LVLMs. While some concerns remain about incremental novelty, reliance on subtitles, and limited graph analysis, the rebuttal clarified comparisons and scalability, convincing reviewers that the method is technically solid and impactful. Given its clear contributions to a high-priority problem area, strong empirical evidence, and broad applicability, I recommend acceptance as a spotlight, recognizing its significance in advancing long video reasoning in LVLMs.
Decision: Accept (spotlight)