Title: AdaVideoRAG: Omni-Contextual Adaptive Retrieval-Augmented Efficient Long Video Understanding
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video Understanding

Multimodal Large Language Models

RAG
Abstract: Multimodal Large Language Models (MLLMs) have demonstrated excellent performance in video understanding but suffer from degraded effectiveness when processing long videos due to fixed-length contexts and weaknesses in modeling long-term dependencies. Retrieval-Augmented Generation (RAG) technology can mitigate these limitations through dynamic knowledge expansion, but existing RAG schemes for video understanding employ fixed retrieval paradigms that use uniform structures regardless of input query difficulty. This introduces redundant computational overhead and latency (*e.g.*, complex graph traversal operations) for simple queries (*e.g.*, frame-level object recognition) while potentially causing critical information loss due to insufficient retrieval granularity for multi-hop reasoning. Such single-step retrieval mechanisms severely constrain the model's balance between resource efficiency and cognitive depth. 
To address this, we first propose a novel AdaVideoRAG framework for long-video understanding, which uses a lightweight intent classifier to dynamically and adaptively allocate appropriate retrieval schemes, ranging from the simplest to the most sophisticated, for different video understanding tasks based on query complexity. We introduce an Omni-Knowledge Indexing module to extract valuable information from multi-modal signals for context modeling and build corresponding databases, *i.e.*, a text base from clip captions, ASR, and OCR; a visual base; and a graph for deep semantic understanding. This enables hierarchical knowledge access, integration, and generation from naive retrieval to graph retrieval, achieving an optimal balance between resource consumption and video understanding capabilities.  
Finally, we construct the HiVU benchmark for deep understanding evaluation. Extensive experiments show that our framework enhances the overall efficiency and accuracy of Video-QA for long videos and can be seamlessly integrated with existing MLLMs via lightweight API calls, establishing a new paradigm for adaptive retrieval augmentation in video analysis.

Omni-Contextual Adaptive Retrieval-Augmented

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated excellent performance in video understanding but suffer from degraded effectiveness when processing long videos due to fixed-length contexts and weaknesses in modeling long-term dependencies. Retrieval-Augmented Generation (RAG) technology can mitigate these limitations through dynamic knowledge expansion, but existing RAG schemes for video understanding employ fixed retrieval paradigms that use uniform structures regardless of input query difficulty. This introduces redundant computational overhead and latency ( e.g. , complex graph traversal operations) for simple queries ( e.g. , frame-level object recognition) while potentially causing critical information loss due to insufficient retrieval granularity for multi-hop reasoning. Such single-step retrieval mechanisms severely constrain the model's balance between resource efficiency and cognitive depth. To address this, we first propose a novel AdaVideoRAG framework for long-video understanding, which uses a lightweight intent classifier to dynamically and adaptively allocate appropriate retrieval schemes, ranging from the simplest to the most sophisticated, for different video understanding tasks based on query complexity. We introduce an Omni-Knowledge Indexing module to extract valuable information from multi-modal signals for context modeling and build corresponding databases, i.e. , a text base from clip captions, ASR, and OCR; a visual base; and a graph for deep semantic understanding. This enables hierarchical knowledge access, integration, and generation from naive retrieval to graph retrieval, achieving an optimal balance between resource consumption and video understanding capabilities. Finally, we construct the HiVU benchmark for deep understanding evaluation. Extensive experiments show that our framework enhances the overall efficiency and accuracy of Video-QA for long videos and can be seamlessly integrated with existing MLLMs via lightweight API calls, establishing a new paradigm for adaptive retrieval augmentation in video analysis.

1 Introduction

With its powerful multimodal perception and generalization capabilities, the Multimodal Large Language Model (MLLM) has become a universal technical paradigm for addressing diverse scenarios and has demonstrated strong generative capabilities in video understanding [31, 48, 29, 1]. However,

Flexibility

Efficiency

MLLM

AdaVideoRAG(Ours)

A1:White

Flexibility

Efficiency

Query

A2:5

A1:White

NoneRetrieval

A3:Incompleteinherentknowledge

A2:6

Any MLLM

MLLM

A3:Comprehensiveresponsethat

incorporates the video content.

Flexibility

Efficiency

VideoRAG

A1:White

Luo, Yongdong, et al.

Straightforward

Intent

Query

A2:6

Query

Classification

NavieRetrieval

A3:Lackofconnectionbetweenthei

Query

Navie Retrieval

MLLM

frontandbackpartsofthevideo.

SimpleQuery

AnyMLLM

Hard Query

Flexibility

Efficiency

VideoRAG

A1:White

Ren, Xubin, et al.

Query

A2:6

A3:Comprehensiveresponse that

GraphRetrieval

Graph Retrieval

MLLM

incorporates the video content.

Any MLLM

Q1:StraightforwardQuery

Q2:StraightforwardQuery

Whatcoloristhebear?--

Howmanytimesdoesthewolfappearthroughoutthevideo?

04:01

Q3:Hard Query

05:17

08:10

What istheimpactof climatechangeongrassland animals?

when applied to specific domains, it is constrained by challenges such as knowledge solidification (inability to dynamically update the latest knowledge), uncontrollable reasoning (risk of hallucinations), and weak generalization (requiring additional fine-tuning costs and time costs), making it difficult to handle multi-hop question and cross-modal association requirements (especially in long video scenarios), which leads to performance degradation [36, 32]. Retrieval-Augmented Generation (RAG), by integrating the collaborative reasoning of external knowledge bases and generative models without being confined to pre-trained knowledge, can easily adapt to private domain data scenarios and has become a core paradigm for improving the factual accuracy and domain adaptability of large language models.

Current RAG research mostly focuses on text modality [25, 10, 19], static images [7], and tabular forms [6], overlooking the unique value of video as a multimodal knowledge carrier. The increasingly popular long-video understanding has put forward new demands for RAG models supporting video modality input. Most existing RAG studies on long videos attempt to enhance question-answering generation by constructing and retrieving knowledge bases from multimodal information derived from videos. For example, Luo et al . [32] incorporates visually-aligned auxiliary text features from optical character recognition (OCR), automatic speech recognition (ASR), and object detection to create video knowledge bases, enabling question-answering for long videos. However, this method does not support sensemaking queries or multi-hop questions, which require global understanding of the entire database as shown in Fig. 1. Recent VideoRAG [36] significantly improves the accuracy of long-video contextual information by constructing a graph database, but it requires maintaining a hierarchical graph database that demands substantial computational and time resources, and incurs higher costs when migrating to new scenarios. We believe that a practical RAG for video understanding needs to flexibly allocate appropriate processing methods for different videos and query difficulties, which both maintains accuracy and improves efficiency.

Considering that real-world video understanding tasks involve content comprehension needs of varying complexity, the problem-solving strategies for questions of different difficulty levels will have distinct priorities. Short-video QA involving simple common sense does not require retrieval and can directly obtain correct answers by querying the MLLM, while complex long-video questions rely on RAG for retrieval to filter effective information. For more complex questions, such as those requiring multi-step reasoning or relying on multiple types of knowledge graph based RAG is necessary to derive correct answers. Therefore, a one-size-fits-all approach of retrieving and then returning results

is not optimal. To address this, this paper proposes an adaptive-RAG-based video understanding scheme termed AdaVideoRAG, as shown in Fig. 1. It first classifies user queries into difficulty levels and then adaptively assigns the most reasonable retrieval strategy based on the difficulty. Additionally, we further integrate visual features, clip captions, ASR, and scene text composite information flows contained in videos, and use relevant text information obtained from external retrieval for data augmentation. According to the difficulty of questions, queries are routed to different levels of database retrieval modes ( i.e. , naive and graph retrieval). These multimodal knowledge inputs and retrieval strategies can more effectively provide fine-grained contextual representation capabilities, ultimately further enhancing the upper limit of MLLM's processing capabilities for long videos and complex question-answering tasks.

To demonstrate the effectiveness of the proposed AdaVideoRAG framework, we officially release HiVU, the first open benchmark dataset for full-stack capability evaluation in video understanding. This dataset groundbreakingly integrates 120 video samples covering a continuous duration spectrum from short clips (1 minute) to extra-long videos (106 minutes), spanning high-frequency scene categories across three major themes: knowledge education (lectures, finance, law, psychology, documentaries), information (news, interviews), and entertainment (sports, cooking, makeup, fitness, TV dramas, animations). In terms of question design, we innovatively develop a three-level difficulty quantification system: 1) Basic Level-1 (L1) focuses on frame-level content perception (e.g., "Which objects appear at the 5th second of the video?"). 2) Advanced Level-2 (L2) requires temporal logic reasoning (e.g., "When does the speaker start explaining graph neural networks?"). 3) Expert Level-3 (L3) challenges cross-modal causal inference (e.g., "How would deleting the narration at the 15th minute affect the plot development?"). Compared with traditional datasets such as ActivityNet [2] (single action recognition) and MovieQA [37] (open-ended QA), this benchmark achieves, for the first time, cognitive complexity evaluation at different levels, providing a hierarchical evaluation framework for video understanding research. It supports systematic optimization of models in longvideo modeling, complex reasoning tasks, and real-world scenario generalization. In summary, our contributions are as follows:

1) We propose a novel AdaVideoRAG framework that dynamically and adaptively routes appropriate retrieval schemes, from the simplest to the most sophisticated, depending on the query complexity. This approach aims to achieve an optimal balance between resource consumption and video understanding capabilities for different video understanding tasks.

2) We introduce an Omni-Knowledge Indexing module to extract valuable information from multi-modal signals for context modeling and establish corresponding databases. A lightweight intent classification model is used to determine the difficulty level of input queries, enabling hierarchical knowledge access, integration, and generation from naive retrieval to graph retrieval, while balancing resource consumption and video understanding capabilities.

3) We publicly release the hierarchical video understanding benchmark HiVU for the first time, which evaluates the multi-level reasoning capabilities of video understanding models. Extensive comparative experiments and ablation studies demonstrate the advantages of AdaVideoRAG in deep understanding of long videos.

2 Method

We introduce an MLLM-centric adaptive RAG framework for long-video understanding termed AdaVideoRAG, which can significantly improve efficiency while ensuring accuracy. As shown in Fig. 2, our method includes four parts: 1) Query Intent Classification (Sec. 2.1). 2) Omni-Knowledge Indexing (Sec. 2.2). 3) Adaptive Retrieval Paradigm (Sec. 2.3). 4) Integration and Generation (Sec. 2.4).

2.1 Query Intent Classification

Not all user requests have the same level of complexity. For simple user requests, we can use a straightforward solution to reduce computing power consumption and users' perception of latency. For complex questions, we rely on complex multi-model, multi-modal, and multi-step queries to achieve higher accuracy. To achieve the above goals, we propose to use a lightweight intent classification model to perform the classification of the difficulty level of the query at the input end. Specifically, we have defined and established a fine-grained evaluation system for the difficulty level of video understanding:

AuxiliaryTextExtraction

Text-Base

AdaptiveRetrievalSec.3.3

Vision-Base

VLM

口

TC2

CN

L2|L3

L2L3

F

FN

Enc.

{Ccao}

{C}

ASR

{Tcao}

{T}

UserQuery(Q)

Integrationand

HowdidtheNumber

GenerationSec.3.4

OCR

目

Tor

To2

TON

IntentLLM

Sec.3.1

30playerperform?

Response (R)

TheNumber30playeris

CN

Omni-KnowledgeIndexing

[3{C}{T}

StephenCurry...Heregained

thegamepointwithathree-

Sec.3.2

Knowledge Graph

MLLM

point shotin thisgame...

Level-1: Straightforward reasoning. There are basically few logical relationships involved in the questions, and the knowledge required for answering questions is directly provided in the video content. For example, "What color of clothes is the woman who appears at the fifth second wearing?" For such questions, the existing MLLMs models are already very mature in solving them. If complex processing is still applied to such simple queries, it will result in unnecessary computational overhead.

Level-2: Simple reasoning. It involves single-step reasoning about basic spatio-temporal/causal relationships, requiring the model to establish logical associations between local events. For example, "Why did the woman cry before the rainy scene started?" requires two-stage reasoning: 1) Determine the starting time point of "rain" through temporal positioning; 2) Retrieve the character behaviors (such as the audio of an argumentative conversation) and scene changes (such as weather forecast subtitles) before this time point, and construct a causal chain to explain the motivation. Such tasks expose the integration flaws of existing MLLMS methods regarding cross-modal temporal clues, and are prone to the lack of key intermediate evidence due to the mismatch in retrieval granularity.

Level-3: Hard reasoning. The video understanding at the highest difficulty level requires extracting different subjects and relationships from the long-context, and constructing a knowledge graph that maps entities and relationships across temporal and semantic dimensions, and combining it with powerful MLLM reasoning capabilities to make judgments. For example, "What life lessons does this movie convey?" Questions of this kind require the model to mine the deep semantic relationships provided in the video and conduct multi-hop reasoning to obtain the correct answers.

Intent classification model. Given the basic definitions and examples from level 1 to level 3, we use a large language model with Chain-of-Thought (CoT) reasoning to classify the query Q . This can be integrated into a RAG (Retrieval-Augmented Generation) architecture as a plug-and-play API, providing intent classification results through appropriate prompts without the need for finetuning. Based on the classification results, it can automatically trigger a progressive knowledge retrieval strategy, ranging from none retrieval to simple naive retrieval, and further to complex graph retrieval. The calculation of the intent classification result L can be formulated as: L = LLM intent ( Q,prompt intent ) . where the LLM is a lightweight CoT model. In this paper, we adopt Qwen2.5-7B [18, 45], whose time-consuming proportion is extremely small (averagely ≤ 5%) compared to the entire process.

2.2 Omni-Knowledge Indexing for Long-Context Understanding

When performing video understanding tasks, MLLMs equipped with RAG can achieve context modeling through dynamic activation of external knowledge bases, which alleviates the window length limitation of long contexts to some extent and enhances the semantic understanding of global videos. To this end, we propose the Omni-Knowledge Indexing module, which extracts valuable information from multiple modal signals for context modeling and establishes corresponding databases, enabling the RAG system to more accurately retrieve the most relevant information and perform high-quality generation.

2.2.1 Omni-Knowledge Text-Base Establishment

In long video understanding tasks, due to the context window size limitations of MLLMs, we need to perform frame sampling and resizing on videos under hardware constraints. However, this inevitably leads to the loss of rich visual information in the videos, as well as unused audio and text multimodal information. Therefore, we utilize an external normalization module to extract multimodal information from videos and construct our private text base.

Auxiliary text extraction and database construction. The input long video V is divided into N consecutive and semantically complete clips V = ( C 1 , C 2 , . . . , C N ) = { C n } at fixed time intervals (30 seconds per clip in the paper). For each clip C n , uniform frame sampling is performed to extract key frames. In this paper, we select 5 frames as the multimodal representation primitive F n , as more frames do not significantly improve performance but increase computational power and model complexity. Specifically, auxiliary text extraction includes three categories: 1) The quantized MiniCPM-V [46] (used as the VLM model) generates fine-grained text descriptions T C for the sampled frames, including semantic elements such as character attributes and spatio-temporal actions, ultimately constructing a caption database D C ; 2) Audio is the most direct information carrier in videos, driving story development, conveying plot clues, and revealing character relationships through language, providing information that cannot be mined from visual features alone. Therefore, we use FastWhisper [33] as the audio extractor to convert the audio in each clip into text format T A , which is stored as vectors via an embedding model to generate an ASR database D A ; 3) Characters T O in each frame are extracted through EASYOCR [22], and an OCR database D O is constructed to compensate for the insufficient recognition ability of MLLMs.

Knowledge graph construction. To address Level-3 complex reasoning queries, we construct a knowledge graph based on clip captions ( T C ), ASR ( T A ), and OCR ( T O ). Specifically, BGE-M3 [5] extracts entities and relationships from text chunks: 1) Entity represents the minimal domainspecific semantic interpretation unit in the video, characterized by a triple <entity type, entity name, spatio-temporal attribute>. 2) Relationship encompasses various semantic associations between entities, including spatio-temporal relationships, causal relationships, functional relationships, etc., to systematically structure video text information.

2.2.2 Vision-Base Establishment

Simply relying on text information extracted from clip captions, ASR, and OCR makes it difficult to construct an optimal Knowledge Indexing. As a typical carrier of multimodal data, videos contain visual features with abundant details that are hard to describe precisely in text, such as object appearance changes, scene spatial layouts, and human facial expressions and movements. These visual information play an indispensable role in complex knowledge reasoning and retrieval tasks. Therefore, we introduce the ImageBind [16] image encoder (Enc. in Fig. 2) to extract features from key frames and concatenate them as the final features, because this model is based on advanced cross-modal alignment algorithms that can map heterogeneous modal data such as images, text, and audio into the same high-dimensional semantic space.

2.3 Adaptive Retrieval Paradigm

After intent classification (Sec. 2.1), the user query (Q) is routed to different retrieval paths according to its difficulty level, so as to improve comprehensive efficiency on the premise of ensuring effectiveness.

None retrieval with direct MLLM. For Level-1 scenarios, the model directly feeds the query (Q) and the entire video { C n } into the MLLM to obtain a direct response. This approach leverages the inherent knowledge and reasoning capabilities of the MLLM without introducing external knowledge bases, significantly enhancing overall efficiency for simple questions.

Naive retrieval with simple reasoning. For Level-2 retrieval scenarios, this study proposes a multimodal collaborative grounding framework that significantly enhances the retrieval efficiency and accuracy of long videos in handling simple logical questions by jointly optimizing the semantic alignment between auxiliary texts (clip captions, ASR, OCR) and visual modalities. Specifically, we first decouple the original query into sub-queries adapted to different modal characteristics: 1) For clip caption retrieval, we rewrite the query into declarative sentences, remove option interference, and add scene-appropriate descriptive content. 2) For ASR-recognized text, we extract colloquial

expressions from the query, retain core actions and events, and add contextual modifiers to match fragmented speech segments. 3) For discrete OCR text, we extract specific entity information from the query. A typical example: when the input query is "How did the Number 30 player perform?" , the rewritten outputs are: i) "clip caption": " The performance of Number 30 player. "; ii) "ASR text": " How's the number 30 player doing. "; iii) "OCR text": " Number 30 player ". Query rewriting effectively mitigates distribution shifts between different semantics. Through cross-modal similarity calculation, we can then quickly locate query-relevant candidate content and the corresponding video clips for each text block.

This study further locates and queries the semantically most relevant video content from the visual feature database D V . Specifically, our model reuses the rewritten results of clip captions as semantic anchors for visual retrieval. The pre-trained cross-modal semantic alignment encoder ImageBind [16] is employed to map videos into the text embedding space { F n } . By calculating the cosine similarity between text and visual embeddings, candidate segments with similarity scores exceeding a threshold (set to 0.5 in this paper) are filtered out. These segments are then ranked to retain the top-K visual evidence with the highest confidence. This approach significantly reduces the modality gap in visualtext alignment by leveraging a unified semantic embedding space, effectively alleviating the problem of local detail loss in long videos. Finally, the videos { C v } retrieved through visual feature-text space alignment are merged with the video chunks { C c,a,o } located via auxiliary text retrieval to construct a retrieval evidence pool for simple reasoning at Level-2.

Graph retrieval in hard reasoning. Relying solely on information obtained from auxiliary text and visual feature retrieval falls short in enabling MLLMs to tackle more complex sensemaking query scenarios. Therefore, we require more abundant and semantically precise auxiliary information capable of modeling multiple events and temporal nodes to facilitate MLLM reasoning. To address this challenge, we adopt a deeper retrieval approach based on Light-RAG [19] to handle hard queries, replacing the naive retrieval method used for simple queries. Specifically, considering resource constraints, we reuse auxiliary text embeddings to construct a graph. We then compute similarity scores between rewritten clip captions and entity/relationship descriptions, returning the most relevant entities and relationships. Within the graph map, we gather other information associated with the retrieved entities and relationships, which can be combined into a query-centered thinking map. This retrieved graph map assists MLLMs in considering global and multi-layered information, aiding in better modeling spatio-temporal and causal relationships within events. Furthermore, we employ a unified semantic embedding space to represent visual evidence obtained from grounding, enhancing retrieval accuracy. We overlay the retrieved videos { C v } with graph retrieval results { C g } to construct a multi-level retrieval evidence pool for hard reasoning under Level-3.

Filtering then sorting evidences. After obtaining the preliminary retrieval results, we perform coarse-to-fine information purification on the search results. First, we filter out duplicate video information blocks retrieved from different databases. Then, the content description of the video blocks (including clip captions, ASR, and OCR texts) and the query are simultaneously input into a small-scale LLM (Qwen2.5-7B [45, 18] in the paper) for fine-grained filtering to exclude some irrelevant search results. Finally, we rerank the selected video clips based on the order of original video time to preserve temporal causal relationship information.

2.4 Multimodal Information Integration and Generation

To provide MLLMs with more comprehensive information for enhancing query accuracy, we acquire auxiliary text information (denoted as { T c,a,o } for simple reasoning and { T g } for hard reasoning) derived from clip captions, ASR, and OCR contexts, along with visual information { C v } from visualto-text grounding. After integrating the retrieved context and corresponding video clips { C c,a,o } , the combined inputs are fed into MLLMs for reasoning and generation to produce the final output R :

2.5 HiVU: Hierarchical Video Understanding Benchmark

Existing video understanding datasets either have insufficient duration [14] or lack engaging content [43, 53], failing to generate queries that require deep comprehension. To support robust reasoning

tasks on long videos and evaluate different methods, we constructed the Hierarchical Video Understanding (HiVU) Benchmark. For this purpose, we selected three genres: knowledge-education (lectures, finance, law, psychology, documentaries), information (news, interviews), and entertainment (sports, cooking, makeup, fitness, TV dramas, animations). We manually collected 120 long-video datasets rich in knowledge content from YouTube, totaling 60 hours, with distributions shown in Fig. 3. Additionally, we designed three tiers of query reasoning from straightforward to hard, as described in Sec. 2.1. This hierarchical query design enables comprehensive and detailed evaluation of models' reasoning capabilities across varying difficulty levels.

Evaluation metrics. For the open-ended questionanswering tasks on the HiVU dataset, we draw inspiration from the Win-Rate metric system widely used in the RAG field to evaluate model capabilities [10, 19]. Specifically, we use large language models (LLMs) as the judgment basis, quantify the comparative results of the two schemes through model outputs, and finally present their competitive scores in percentage form. The Win-Rate Comparison comprehensively considers queries from five dimensions: 1) Comprehensiveness: This dimension focuses on whether the model's response fully covers the query, avoiding missing critical information or providing

one-sided answers. 2) Empowerment: It primarily examines whether the model's response can provide practical value and inspiration to users. 3) Trustworthiness: This dimension emphasizes the reliability and authenticity of the model's output content. 4) Depth: It assesses whether the model can go beyond surface phenomena, uncover the essential issues behind the query, and conduct in-depth analysis and discussion. 5) Density: It focuses on the information content and compactness of the model's response, avoiding verbose, empty, or redundant expressions.

3 Experiments

3.1 Experimental Setup

We conduct comprehensive evaluations of the proposed AdaVideoRAG method and the effectiveness of each module primarily on the newly proposed HiVU benchmark Sec. 2.5, and also introduce public video understanding benchmarks for further thorough assessment. Specifically: 1) HiVU includes over 10 sub-genres across 3 domains, comprising 120 knowledge-rich long-video datasets totaling 60 hours. 2) Video-MME [14] is a full-spectrum multi-modal evaluation benchmark for MLLMs in video analysis, featuring diverse videos and multi-modal data. It contains 900 videos (ranging from 11 seconds to 1 hour, categorized into short, medium, and long), with 2,700 multiplechoice questions covering 6 major visual domains (e.g., knowledge, film, sports) and 30 subdomains, focusing on evaluating the perception, reasoning, and summarization capabilities of multimodal large language models (MLLMs) in video analysis. 3) MLVU [53] is a multi-task benchmark for evaluating long-video understanding with diverse genres and extended durations. Centered on long videos ranging from 3 minutes to over 2 hours (average 12 minutes), it sets 9 multi-tasks (e.g., single/multi-detail understanding) across diverse video types (films, surveillance, games, etc.), aiming to comprehensively assess long-video understanding capabilities.

3.2 Experimental Results

Improving open-sourced MLLMs with AdaVideoRAG on MLVU_test [53] benchmark. The overall evaluation results of all the investigated multi-modal large language models in the MLVU test set are shown in Tab. 1. These results cover the baseline model, Video-LLaV A [29], along with two highly regarded open-source models released recently: Qwen2.5-VL series [1] and VideoLLaMA3 [47]. The evaluation results clearly demonstrate that the AdaVideoRAG strategy we proposed significantly improves the question-answering accuracy of each MLLM. And it particularly stands out in two key types of tasks. Firstly, in tasks such as Topic Reasoning (TR) that require multi-hop reasoning about videos, and secondly, in tasks like Action Count (AC) that involve holistic reasoning. This indicates that AdaVideoRAG can not only strengthen the basic question-answering ability but also

effectively assist the MLLMs in achieving breakthroughs within complex reasoning and multi-detail processing tasks. It is worth noting that although the Qwen2.5-VL-7B model that performs relatively weakly on the MLVU dataset, it exhibits more pronounced accuracy improvements after adopting our AdaVideoRAG, increasing nearly by 40% and even reaching the accuracy of large-parameter models like Qwen2.5-VL-32B. What's more, the open-source model VideoLLaMA equipped with AdaVideoRAG, even though it has fewer parameters than Qwen2.5-VL-32B, shows better performance on long videos, and its performance can even be comparable to that of GPT-4o. These experimental results fully verify the universality and effectiveness of AdaVideoRAG in enhancing the reasoning ability of MLLMs.

Comparison with state-of-the-art VideoRAG [32] on Video-MME [14] dataset. Given that the experimental results in Tab. 1 have fully verified that AdaVideoRAG can effectively enhance the reasoning performance of MLLMs, we select the VideoLLaMA3 and the Qwen2.5-VL-7B model as the basic model for subsequent control experiments, which with the same number of parameters. In Tab. 2, we conduct a horizontal comparative test between our AdaVideoRAG and Video-RAG [32] on the Video-MME dataset. The experimental results show that both RAG methods can significantly enhance the video understanding ability of the base MLLMs. However, in tasks involving the processing of long videos, our AdaVideoRAG demonstrates a more distinct advantage. This is mainly due to the fact that AdaVideoRAG is capable of constructing a more complex and reasonable knowledge map during the retrieval of long videos, thus enabling precise understanding and efficient reasoning of video.

Impact of LLM arbiters. To explore the performance of the retrieval strategies in sensemaking tasks of varying difficulties, we conducts comparative experiments based on the proposed hierarchical video understanding benchmark(HiVU), and an LLM is then used as the evaluation referee to assess the quality of the final answers. Regarding the selection of specific LLMs, we carry out two sets of control experiments: Deepseek-R1-7B [45, 18] and Deepseek-R1-32B [45, 18], Qwen2.5-32B [45] and QwQ-32B [39], which represent the models with different parameters and reasoning capabilities respectively, as illustrated in Tab. 3. The experimental results demonstrate that models with larger parameters and equipped with the Chain-of-Thought (CoT) reasoning mechanism exhibit stronger discriminatory abilities when evaluating the performance of other models. Based on these findings, we choose DeepSeek-32B model as the evaluation arbiter for HiVU benchmark evaluation to ensure the accuracy and reliability of the evaluation results.

Comparison with state-of-the-art VideoRAG [32] on HiVU dataset. In our HiVU data benchmark, there are three tasks classified according to the difficulty of reasoning: straightforward (L1), simple (L2), and hard (L3). For different levels, AdaVideoRAG employs different retrieval strategies: from without retrieval, naive retrieval, to graph retrieval, which forms a hierarchical enhancement mechanism. And the following series of experiments are designed to verify the improvement of

reasoning ability, as shown in Tab. 4, in the hard-level video understanding task, the multi-modal large language model integrated with AdaVideoRAG demonstrates more significant advantages compared to its original model, and the gap between the two becomes more evident as the task difficulty increases. This result not only confirms the effectiveness of AdaVideoRAG in complex reasoning scenarios but also indirectly validates the rationality and scientific nature of the three-level difficulty division in the HiVU benchmark, providing a reliable basis for quantitatively evaluating the reasoning ability of models.

Meanwhile, we conducted a horizontal comparison with VideoRAG [32] in the HiVU benchmark, as shown in Tab. 4. Consistent with our expectations, AdaVideoRAG is on par with VideoRAG [32] at the Level-1 and Level-2 levels. However, our method exhibits more prominent advantages at the Level 3 which need global and multi-hop reasoning.

Ablation Study In the following analysis, we perform three ablation studies to precisely assess the key components of our proposed method. They are as follows: 1) Without Graph: We cancel the retrieval of entities and relationships in the graph map; 2) Without vision retrieval: We remove the feature retrieval in vision-to-text grounding; 3) Without naive text retrieval: We cancel the the retrieval from the caption, OCR, and ASR databases, as shown in Tab. 5. It can be seen that the design of each module is effective and can improve the understanding ability of the model.

Impact of Query Classification. We conducted two experiments to examine how query classification affects retrieval and reasoning performance. First, we compared different classifier models in terms of classification precision and downstream results. Second, we tested the effect of removing the classifier by routing all queries through the same difficulty level.

(1) Classifier Comparison. We evaluated several models on HiVU (with ground-truth Level labels) and the VideoMME benchmark. As shown in Table 6, Qwen2.5-7B achieved the best balance between classification accuracy and overall performance.

(2) Effect of Removing the Classifier. We further examined the role of query classification by disabling the classifier and forcing all queries to follow a single retrieval path. When all queries were treated as Level-1, Level-2, or Level-3, the overall scores on the VideoMME benchmark dropped to 64.2, 67.5, and 67.1, respectively. In contrast, our adaptive approach ( AdaVideoRAG ) achieved the

highest score of 68.5 , demonstrating that dynamic query routing effectively balances reasoning depth and efficiency.

3.3 Efficiency Analysis

To evaluate the efficiency of different retrieval paths, we conducted time-cost experiments on 100 randomly selected videos from the MLVU dataset under three settings: no retrieval (Level-1), simple retrieval (Level-2), and hard retrieval (Level-3).

(1) Database Construction. Level-1 performs direct inference without database construction. The average construction time for Level-2 and Level-3 is 351 s and 412 s, respectively, with the additional cost in Level-3 attributed to graph construction. The adaptive scheduling strategy of AdaVideoRAG effectively reduces such high-cost operations by prioritizing simpler retrieval paths when applicable.

(2) Single-Process Inference. On a single H20 GPU, the average response times are 8 s (Level-1), 26 s (Level-2), and 27 s (Level-3). AdaVideoRAG achieves a balanced trade-off between accuracy and efficiency, with an average response time of 20 s.

(3) Parallelization. To further improve deployment efficiency, we introduced multi-process and multi-GPU parallelism. Using dual processes on a single H20 GPU (96 GB, batch size = 2), database construction for Level-2 and Level-3 achieved ∼ 2 × acceleration, reducing the time to 176 s and 210 s. Scaling to 8 GPUs yielded near-linear speedup ( ∼ 8 × ), cutting construction time to 22 s and 26 s. For multi-process retrieval on a single GPU, Level-2 and Level-3 achieved ∼ 2 × acceleration, with total processing times of 15 s and 16 s, compared to 20 s for AdaVideoRAG.

4 Conclusion

Our AdaVideoRAG demonstrates outstanding performance advantages when dealing with difficult video understanding tasks that require multi-hop thinking and deep reasoning. Meanwhile, AdaVideoRAG can efficiently integrate omni-information, fully leverage the value of multi-source data such as images, videos, and texts. It can also flexibly switch between basic question-answering and high-order semantic understanding tasks, effectively balancing efficiency and accuracy, and greatly enhancing the generalization ability and application universality of multi-modal large language models.

Limitations, broader impact and social impact. Due to the limited computational resources for local deployment, this study only evaluates models up to 32 billion parameters. Moreover, the research only implemented three levels of routing, while real-world applications may require more detailed classification. From a social impact perspective, this technology could pose new risks such as spreading false information, for example, using generated fake videos to manipulate public opinion. This highlights the urgent need to establish a collaborative governance system that includes technical ethics, legal regulations, and industry standards.

Acknowledgments and Disclosure of Funding. This work was supported by the "Leading Goose" Key R&D Program of Zhejiang Province (2025C01069).

References

S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923 , 2025. 1, 7, 21

F. Caba Heilbron, V. Escorcia, B. Ghanem, and J. Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition , 2015. 3

D. Caffagni, F. Cocchi, N. Moratelli, S. Sarto, M. Cornia, L. Baraldi, and R. Cucchiara. Wikillava: Hierarchical retrieval-augmented generation for multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1818-1826, 2024. 21

Y. Cai, J. Zhang, H. He, X. He, A. Tong, Z. Gan, C. Wang, and X. Bai. Llava-kd: A framework of distilling multimodal large language models. arXiv preprint arXiv:2410.16236 , 2024. 21

J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, and Z. Liu. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216 , 2024. 5

S.-A. Chen, L. Miculicich, J. Eisenschlos, Z. Wang, Z. Wang, Y. Chen, Y. Fujii, H.-T. Lin, C.-Y. Lee, and T. Pfister. Tablerag: Million-token table understanding with language models. Advances in Neural Information Processing Systems , 2024. 2, 22

W. Chen, H. Hu, X. Chen, P. Verga, and W. W. Cohen. Murag: Multimodal retrieval-augmented generator for open question answering over images and text. arXiv preprint arXiv:2210.02928 , 2022. 2, 22

Y. Chen, S. Ji, H. Wang, Z. Wang, S. Chen, J. He, J. Xu, and Z. Zhao. Wavrag: Audio-integrated retrieval augmented generation for spoken dialogue models. arXiv preprint arXiv:2502.14727 , 2025. 22

Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR , 2024. 21

D. Edge, H. Trinh, N. Cheng, J. Bradley, A. Chao, A. Mody, S. Truitt, D. Metropolitansky, R. O. Ness, and J. Larson. From local to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404.16130 , 2024. 2, 7, 21

T. Fan, J. Wang, X. Ren, and C. Huang. Minirag: Towards extremely simple retrieval-augmented generation. arXiv preprint arXiv:2501.06713 , 2025. 22

W. Fan, Y. Ding, L. Ning, S. Wang, H. Li, D. Yin, T.-S. Chua, and Q. Li. A survey on rag meeting llms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 6491-6501, 2024. 21

Y. Fan, X. Ma, R. Wu, Y. Du, J. Li, Z. Gao, and Q. Li. Videoagent: A memory-augmented multimodal agent for video understanding. In European Conference on Computer Vision , pages 75-92. Springer, 2024. 21

C. Fu, Y. Dai, Y. Luo, L. Li, S. Ren, R. Zhang, Z. Wang, C. Zhou, Y. Shen, M. Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075 , 2024. 6, 7, 8

Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y . Bi, Y . Dai, J. Sun, H. Wang, and H. Wang. Retrievalaugmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997 , 2:1, 2023. 21

R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, and I. Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 15180-15190, 2023. 5, 6

A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024. 21

D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 , 2025. 4, 6, 8

Z. Guo, L. Xia, Y. Yu, T. Ao, and C. Huang. Lightrag: Simple and fast retrieval-augmented generation. arXiv preprint arXiv:2410.05779 , 2024. 2, 6, 7, 22

Y. Hu and Y. Lu. Rag and rau: A survey on retrieval-augmented language model in natural language processing. arXiv preprint arXiv:2404.19543 , 2024. 22

A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 , 2024. 21

JaidedAI. Easyocr: Ready-to-use ocr with 80+ languages. https://github.com/JaidedAI/ EasyOCR , 2024. 5

S. Jeong, J. Baek, S. Cho, S. J. Hwang, and J. Park. Adaptive-RAG: Learning to adapt retrieval-augmented large language models through question complexity. In NAACL , 2024. 21

D. Kim, B. Kim, D. Han, and M. Eibich. Autorag: automated framework for optimization of retrieval augmented generation pipeline. arXiv preprint arXiv:2410.20878 , 2024. 22

P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems , 2020. 2, 21

F. Li, R. Zhang, H. Zhang, Y. Zhang, B. Li, W. Li, Z. Ma, and C. Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895 , 2024. 21

X. Li, Z. Yan, D. Meng, L. Dong, X. Zeng, Y. He, Y. Wang, Y. Qiao, Y. Wang, and L. Wang. Videochat-r1: Enhancing spatio-temporal perception via reinforcement fine-tuning. arXiv preprint arXiv:2504.06958 , 2025. 21

Z. Li, X. Chen, H. Yu, H. Lin, Y. Lu, Q. Tang, F. Huang, X. Han, L. Sun, and Y. Li. Structrag: Boosting knowledge intensive reasoning of llms via inference-time hybrid information structurization. arXiv preprint arXiv:2410.08815 , 2024. 21

B. Lin, Y. Ye, B. Zhu, J. Cui, M. Ning, P. Jin, and L. Yuan. Video-LLaVA: Learning united visual representation by alignment before projection. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , 2024. 1, 7

K. Lin, F. Ahmed, L. Li, C.-C. Lin, E. Azarnasab, Z. Yang, J. Wang, L. Liang, Z. Liu, Y. Lu, et al. Mm-vid: Advancing video understanding with gpt-4v (ision). arXiv preprint arXiv:2310.19773 , 2023. 21

H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. In Advances in neural information processing systems , 2023. 1, 21

Y. Luo, X. Zheng, X. Yang, G. Li, H. Lin, J. Huang, J. Ji, F. Chao, J. Luo, and R. Ji. Video-rag: Visually-aligned retrieval-augmented long video comprehension. arXiv preprint arXiv:2411.13093 , 2024. 2, 8, 9, 22

A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning , pages 28492-28518. PMLR, 2023. 5

O. Ram, Y. Levine, I. Dalmedigos, D. Muhlgay, A. Shashua, K. Leyton-Brown, and Y. Shoham. In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics , 11:1316-1331, 2023. 21

W. Ren, W. Ma, H. Yang, C. Wei, G. Zhang, and W. Chen. Vamba: Understanding hour-long videos with hybrid mamba-transformers. arXiv preprint arXiv:2503.11579 , 2025. 21

X. Ren, L. Xu, L. Xia, S. Wang, D. Yin, and C. Huang. Videorag: Retrieval-augmented generation with extreme long-context videos. arXiv preprint arXiv:2502.01549 , 2025. 2, 22

M. Tapaswi, Y. Zhu, R. Stiefelhagen, A. Torralba, R. Urtasun, and S. Fidler. Movieqa: Understanding stories in movies through question-answering. In Proceedings of the IEEE conference on computer vision and pattern recognition , 2016. 3

G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 , 2023. 21

Q. Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https://qwenlm.github.io/blog/qwq-32b/ . 8

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023. 21

H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023. 21

Z. Z. Wang, A. Asai, X. V. Yu, F. F. Xu, Y. Xie, G. Neubig, and D. Fried. Coderag-bench: Can retrieval augment code generation? 2024. 22

H. Wu, D. Li, B. Chen, and J. Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. Advances in Neural Information Processing Systems , 37:2882828857, 2024. 6

S.-Q. Yan, J.-C. Gu, Y. Zhu, and Z.-H. Ling. Corrective retrieval augmented generation. arXiv preprint arXiv:2401.15884 , 2024. 22

A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115 , 2024. 4, 6, 8, 21

Y. Yao, T. Yu, A. Zhang, C. Wang, J. Cui, H. Zhu, T. Cai, H. Li, W. Zhao, Z. He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800 , 2024. 5

B. Zhang, K. Li, Z. Cheng, Z. Hu, Y. Yuan, G. Chen, S. Leng, Y. Jiang, H. Zhang, X. Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13106 , 2025. 7, 21

H. Zhang, X. Li, and L. Bing. Video-LLaMA: An instruction-tuned audio-visual language model for video understanding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , 2023. 1, 21

L. Zhang, T. Zhao, H. Ying, Y. Ma, and K. Lee. Omagent: A multi-modal agent framework for complex video understanding with task divide-and-conquer. arXiv preprint arXiv:2406.16620 , 2024. 21

Y. Zhang, B. Li, h. Liu, Y . j. Lee, L. Gui, D. Fu, J. Feng, Z. Liu, and C. Li. Llava-next: A strong zero-shot video understanding model, April 2024. URL https://llava-vl.github.io/ blog/2024-04-30-llava-next-video/ . 21

X. Zheng, Z. Weng, Y. Lyu, L. Jiang, H. Xue, B. Ren, D. Paudel, N. Sebe, L. Van Gool, and X. Hu. Retrieval augmented generation and understanding in vision: A survey and new outlook. arXiv preprint arXiv:2503.18016 , 2025. 22

B. Zhou, Y. Hu, X. Weng, J. Jia, J. Luo, X. Liu, J. Wu, and L. Huang. Tinyllava: A framework of small-scale large multimodal models. arXiv preprint arXiv:2402.14289 , 2024. 21

J. Zhou, Y. Shu, B. Zhao, B. Wu, S. Xiao, X. Yang, Y. Xiong, B. Zhang, T. Huang, and Z. Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264 , 2024. 6, 7

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction have clearly included the motivations, important assumptions, and contributions made in the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The authors have discussed the limitations of the work at Ablation Study.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The authors have clearly provided the full set of assumptions and complete proofs in the Method section.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The authors have presented all the experimental details in the Implementation Details section.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The authors have presented all the experimental details in Appendix which provided sufficient instructions to faithfully reproduce the main experimental results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The authors have presented all the training and test details in the Implementation Details section.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The authors have presented all the experimental details in Appendix with provided sufficient instructions.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The authors have included the sufficient information on the computer resources in the Implementation Details.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The authors have explained the broader impacts of the work in Appendix.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The authors have cited the original paper that produced the code package or dataset.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The authors have submitted the details of the code/model which includes details about training, license, limitations, etc.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The paper does not involve LLM usage in manuscript preparation.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Appendix

The appendix presents the following sections to strengthen the main manuscript:

-Sec. A shows the Related Work part of the paper.

-Sec. B shows the dynamic sampling strategies when building the database. We set different sampling frequencies to evaluate the performance of our AdaVideoRAG on the MLVU dataset.

-Sec. C shows the instructions for level classification. We develop hierarchical prompting templates that enable LLMs to classify query complexity into three tiers (L1-L3), triggering adaptive retrieval strategies from direct lookup to sensemaking reasoning.

-Sec. D shows the more detailed results, including multiple-choice QA and summary video understanding, that visually demonstrate our AdaVideoRAG can enhance the reasoning ability of MLLMs.

A Related Work

A.1 Multimodal Large Language Models

With the successful application experience of Large Language Models (LLMs) [40, 41, 17, 45], Multimodal Large Language Models (MLLMs) have also made significant breakthroughs in the field of visual language understanding. LLaVA [31], by first conducting instruction fine-tuning training on a dataset carefully selected by GPT-4, has become one of the most popular methods for constructing MLLMs and has been followed by subsequent works [52, 4, 3]. Benefiting from the rapid development in the fields of short video applications and video generation models, video understanding has also become increasingly popular recently [48, 50, 26]. In contrast, due to the complexity of spatio-temporal joint modeling, video understanding poses higher requirements for the fine-grained spatial understanding of the model and multi-hop prompts, so some works targeting the long-video setting have emerged recently. Vamba [35] has constructed a hybrid Mamba-Transformer model that encodes video tokens with linear complexity, and recent VideoChat-R1 [27] has explored the use of GRPO for the reinforcement fine-tuning of video MLLMs. Recently, there are also some works on video understanding based on agents [30, 13, 49]. Closed-source commercial models such as GPT-4o [21] and Gemini [38] have leading video understanding capabilities, while open-source model series such as the typical Qwen-VL [1], InternVL [9], and VideoLLaMA [47] have received widespread attention and academic research. However, when applied to specific fields, MLLMs have lower accuracy in answers or larger hallucinations due to their inability to dynamically update the latest knowledge. To improve the accuracy, additional fine-tuning costs and time costs are required, and it is difficult to deal with multi-hop questions in long-video cross-modal understanding. In order to alleviate the above problems, this paper studies Retrieval-Augmented Generation (RAG) for long-video, and improves the video understanding ability of the model by integrating the collaborative reasoning between an external knowledge base and the generation model.

A.2 Retrieval-Augmented Understanding

Retrieval-Augmented Generation (RAG) optimizes large language models (LLMs) by integrating external knowledge retrieval with generative model capabilities [25, 34, 12]. It enables low-cost knowledge expansion without retraining the model through dynamic updates to external knowledge bases, effectively mitigating traditional LLMs' issues of hallucinations, outdated knowledge, and data security risks. RAG [28, 15] converts user queries into vectors, retrieves the most relevant information from external databases, and integrates retrieval results as context into the generative model's prompt to deliver more accurate, reliable, and fact-based responses. However, these methods often overlook complex relationships between documents or contexts-such as entity connections, hierarchical structures, or causal relationships-that are critical for contextual understanding. Consequently, graph-based RAG methods [10] have gained traction, exploring structured knowledge representations to enhance retrieval efficiency and precision while excelling in query-focused summarization tasks. Additionally, Adaptive-RAG [23] significantly improves traditional RAG systems' performance and efficiency in complex scenarios by dynamically adjusting retrieval strategies and generative logic.

Recent research has deeply investigated efficiency challenges [19, 11], dataset-specific optimizations [24], and hallucination correction [44].

Recent RAG approaches have also integrated multimodal information to meet growing application demands [20, 51], such as images [7], code [42], tables [6], and audio [8]. However, constrained by video data's complex modal information and requirements for spatio-temporal modeling, RAG has seen limited adoption in the video understanding field-particularly for long-video understanding, which poses significant challenges for long-context information modeling. Luo *et al.* [32] incorporates visually-aligned auxiliary text features from OCR, ASR, and object detection to construct video knowledge bases, while VideoRAG [36] significantly improves the accuracy of long-video contextual information by building graph databases. However, current solutions either struggle to address multi-hop questions effectively or face substantial computational resource and time costs when maintaining hierarchical graph databases. To tackle these issues, we propose a query intent classification strategy to adaptively route queries to different retrieval paths based on their difficulty, achieving a robust balance between performance and efficiency.

B Dynamic Sampling Strategies

During the database construction phase, whether it is naive vector retrieval or graph retrieval, both of them are constructed upon the extracted auxiliary texts, including the caption, ASR, and OCR. Therefore, dynamic sampling strategies directly determine the semantic coverage density. Generally speaking, the faster the sampling frequency, the more effective information can be obtained from the video, which is more conducive to subsequent retrieval.

To systematically validate the impact of sampling strategies, we conduct ablation studies comparing frame sampling density (varying from 5 to 30 frames/clip), measuring their effects on MLVU test dataset, as shown below Tab. A1. As the sampling frequency increases, the evaluation metrics will become higher. Since the final results of 30 frames and 5 frames only differ by about 1 point, we adopt a sampling frequency of 5 frames per 30 seconds for AdaVideoRAG, considering both the improvement of accuracy and the efficiency of resource utilization.

C Instructions for Level Classification

In this paper, we use prompts to classify the difficulty level of queries in Fig. A2, including the detailed description of each level: L1 (Direct Factual) queries requiring single-modality pattern matching (e.g., "Identify the object at 0:05"), L2 (Contextual) needing temporal/causal reasoning across clips (e.g., "Why did X occur before Y?"), and L3 (Multi-Hop) demanding cross-modal hypothesis validation with external knowledge (e.g., "How would Z change if scene A were removed?")

D More Detailed Results.

To further validate the enhancement of AdaVideoRAG on the video understanding capabilities of MLLMs, we provide more detailed and specific instance demonstrations, including multiple-choice questions and sensemaking questions, as shown as Fig. A1.

We selected two specific multiple-choice cases and compared them with the MLLM baseline model (VideoLLaMA) and VideoRAG [32] respectively. The results show that our AdaVideoRAG pays more attention to details in terms of video understanding capabilities and thus makes more accurate and confident responses to user queries. Meanwhile, we also analyze the sensemaking problems. As shown at the bottom of Fig. A1, our AdaVideoRAG can provide more fine-grained information. All the answers are based on the real video content, which significantly enhances its ability to reduce visual hallucinations and enables it to output more complete and logical answers.

multiple-choice questions

Query:Howmanypeoplejump into thewaterfirst at thebeginning of thevideo? (A)7 (B)3 (C)8 (D)2 (E)5 (F)4

MLLM(VideoLLaMA): F

Query: What is the genre of this video?

(A) It is a news report that introduces the history behind Christmas decorations.

(C) It is a travel vlog exploring Christmas markets around the world

(D) It is a tutorial on DlY Christmas ornament crafting

MLLM(VideoLLaMA)withVideoRAG:C

MLLM(VideoLLaMA)withAdaVideoRAG:A

Sensemaking questions

Query: How does Jean's makeup routine reflect her personal style and preferences?

MLLM(VideoLLaMA):Jean'smakeuproutinereflectsherpersonalstyleandpreferencesthroughherchoiceof drugstore and high-end products. Her preference for a minimalist approach is evident in her simple eye makeup and the use of a single lip color.

MLLM(VideoLLaMA) with AdaVideoRAG : Jean's makeup routine reflects her personal style and preferences through her emphasis on a 'Clean Girl' look, which focuses on creating a dewy, hydrated appearance without heavy makeup. Sheprefers usingproducts liketheMaclipstick and theGiorgioArmaniluminoussilkinthe shade575fora natural yet polished finish. Her approach to skincare and makeup suggests she values a fresh and radiant look, often appearingcakeyoroverdone.

MLLM(VideoLLaMA) with AdaVideoRAG: C

Goal

1. Given a query, Classify queries into exclusively one difficulty level (A/B/C).

2. When you are confused about which level to choose, choose a higher level.

#### Classification Criteria ###

Level 1 (Local Perception or question is very easy that we can put all of video as input)

Level 2 (Global Search) : It contains simple logical relationships. It can be used for global awareness, which requires global search and positioning of video segments related to questions, and then understanding of the searched video segments.

Level 3 (Complex Reasoning) : Requires external knowledge/abstract interpretation. It may be necessary to make use of knowledge graphs of multi-layer or complex relationships, or semantic questions with high generalizations

####Examples ####

Query1: What color is the dog that appears in the 5s part of the video?

Output: ###Level A###

Query2: What happened before the rain?

Output: ###Level B###

Query7: Identify the position relationships of the

characters in the meeting room scene?

Output: ###Level C###

Prompts for Level Classification

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides a novel approach to video retrieval in the context of long context video understanding. The authors propose specific query-based video preprocessing based on query complexity. The queries are divided into three levels of complexity, based on which the visual, text descriptions, and graph relationships are explicitly provided. The authors also contribute a new dataset, HiVU, with 120 long-form videos and questions. The dataset is used to test their hierarchical approach to video retrieval.\n\nThe paper defines straightforward reasoning as queries requiring very few logical steps with information completely present in the video provided, suggesting a naive video sampling for the same.\nSimple reasoning refers to queries which require some kind of temporal or causal reasoning, with parts of the videos requiring text-based guidance for the frames.\nHard queries refer to queries requiring global relationship context, which use a knowledge graph to infer the same along with text-based information.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The works show significant improvement in long-context video reasoning while being model agnostic and providing a scalable solution for real-world applications.\n2. The work runs extensive experimentation and ablation studies and different models on their introduced dataset and provides insightful results.\n3. The paper is written clearly, explaining all parts of their work in a detailed manner. \n\n\nWeaknesses:\n1. While the paper shows extensive results on different models, the experiments across different datasets seem inconsistent, for example, the MMLVU dataset does not have the SOTA VideoRAG method, but it is present in VideoMME.\n2. The authors themselves highlight the inconsistencies in LLM as a judge as a benchmarking metric and proceed to use the same, which makes it difficult to judge the actual performance of the models. While they do show strong improvement on the VideoMME and MLVU datasets, it would be nice to see some human-annotated gold answers in the HiVU dataset in the future.\n\nQUESTIONS:\n1. While the ablation studies show the impact of each part of the video retrieved, is it possible to see the effect of the same without the classification of queries? Precisely, how much performance do we gain/lose if we treat each question as the hardest query, giving it all the information provided? If there is no loss in performance, what is the gain in terms of compute efficiency that we can see?\n2. Please fix table 2 as the highlighting of numbers in the short column is wrong, and the gain column notation is inconsistent.\n\nI would love to increase the rating if some of the above questions and weaknesses are answered.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel AdaVideoRAG framework to dynamically and adaptively route appropriate retrieval schemes and introduces an Omni-Knowledge Indexing module to extract valuable information from multi-modal signals for context modeling and establish corresponding databases. It also releases the hierarchical video understanding benchmark HiVU for the first time, which evaluates the multi-level reasoning capabilities of video understanding models.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n- This paper basically came up with a system that can pick the right retrieval strategy depending on how hard the question is.This way,it stays fast while still giving accurate answers.\n- The paper puts together a big database using video captions, speech-to-text, OCR, and visual features, so the model has richer context to pull from when answering questions.\n\nWeakness \n- The paper lacks detailed analysis on how it specifically improves efficiency.\n- It lacks a detailed comparison analyzing how different retrieval strategies (no retrieval, naive retrieval, graph retrieval) contribute to efficiency improvements across varying query types.\n- The paper says it uses clip captions, ASR, OCR, and visual features together, but it doesn’t clearly explain how these different types of data are actually used in practice.\n- The paper uses a fixed threshold of 0.5 when matching text and visual embeddings, but it doesn’t analyze how using different thresholds would affect retrieval.\n- The paper does not investigate the impact of using different LLM backbones(it just uses Qwen2,video-llama,video-llava) on performance.\n- The effectiveness of the intent classification module is not ablated separately, so it is unclear how much it contributes to overall performance gains.\n\nQUESTIONS:\n- How important the intent classification accuracy is for overall performance?\n- How much each modality(captions, ASR, OCR) contributes to the final performance? \n- How do different retrieval strategies (none, naive, graph) specifically contribute to efficiency improvements across different query types?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work tackles the problem of long video understanding with large language models. The authors claim that existing VideoLLM + RAG techniques are sub-optimal due to their fixed strategy for retrieval, which uses uniform structures regardless of input query difficulty, bringing redundant computational overhead for simple queries and potential information loss for complex queries. Therefore, they propose AdaVideoRAG, an adaptive VideoLLM + RAG technique that leverages a lightweight intent classifier to determine the difficulty of the query, and enables dynamic routing to different retrieval methods according to the classification results. The authors also introduced an Omni-Knowledge Indexing module to support better multi-modal information extraction, and the HiVU benchmark for evaluation. Experiments have been conducted on public benchmarks and the proposed evaluation HiVU benchmark to verify the effectiveness of AdaVideoRAG.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Overall, the paper is easy to follow. The motivation for utilizing a lightweight classifier to determine whether to use complex retrieval methods is clear -- allocating more compute to complex queries and less to simple queries.\n2. The effectiveness of the proposed methods is well-demonstrated by ablation studies in Table 1 and Table 2.\n\nWeaknesses:\n\n1. Why adopting an extra Qwen2.5-7B as the intent classifier, instead of using the MLLM itself instead? Ablation studies on the choice of intent classifiers shall be included in the study. Besides, I believe a 7B-size model cannot be regarded as a \"lightweight\" classifier, given that the MLLM is generally also 7B.\n2. Although it is clear, the motivation seems to be incremental. Leveraging another 7B LLM + prompting to act as a classifier is not a smart idea for routing. Authors are encouraged to provide strong justifications to demonstrate the significance of the proposed method.\n3. The authors mentioned \"efficiency\" of the proposed method throughout the paper, but it seems that detailed analysis/discussions/experiments on such effect are not provided. It would be better to conduct an in-depth study to see how much latency is introduced by adopting VideoRAG/AdaVideoRAG (including database construction and knowledge retrieval) compared to VideoLLM baselines.\n4. The VideoLLM + RAG technique seems to be closely relevant to test-time scaling methods on video understanding. It would be interesting to incorporate a detailed discussion/comparison between the proposed method (RAG-based) and recent TTS methods (e.g., CoT-based [1-3] and agent-based [4, 5]) on long video understanding.\n\nMinor Issues:\n\n1. Typo: In Figure 2, Sec. 3.1, 3.2, 3.3 shall be 2.1, 2.2, 2.3\n\n[1] Chain-of-Frames: Advancing Video Understanding in Multimodal LLMs via Frame-Aware Reasoning. arXiv 2025.\n[2] Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition. ICML 2024.\n[3] VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection. CVPR 2025.\n[4] VideoMind: A Chain-of-LoRA Agent for Long Video Reasoning. arXiv 2025.\n[5] VideoAgent: Long-form Video Understanding with Large Language Model as Agent. ECCV 2024.\n\nQUESTIONS:\nPlease refer to the weaknesses section for my concerns. May major concern is the novelty of the proposed method. Authors are encouraged to provide in-depth analysis to justify 1. the proposed method is indeed novel, rather than a simple combination of two video RAG methods plus a LLM-prompting-based classifier, and 2. how much efficiency can the proposed method bring.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents AdaVideoRAG, an adaptive retrieval-augmented generation framework designed to improve long-video understanding in multimodal large language models (MLLMs). Existing VideoRAG systems apply fixed retrieval strategies regardless of query complexity, leading to inefficiencies for simple tasks and insufficient depth for complex reasoning. AdaVideoRAG addresses this limitation by employing a lightweight intent classifier to dynamically select appropriate retrieval pathways—from simple text-based lookups to more sophisticated graph-based retrieval—based on the complexity of the user query. The framework incorporates an Omni-Knowledge Indexing module that constructs multi-modal knowledge bases (text, vision, and graph) from video inputs, enabling hierarchical and context-sensitive retrieval. Additionally, the authors introduce HiVU, a new benchmark for evaluating deep video understanding. Experimental results demonstrate that AdaVideoRAG achieves improved accuracy and efficiency in long-video QA tasks, while being compatible with existing MLLMs through lightweight API integration.\n\nSTRENGTHS AND WEAKNESSES:\n# Strength\n### Novel adaptive retrieval mechanism:  \nThe paper introduces a compelling and novel approach to video retrieval-augmented generation by leveraging a lightweight intent classifier that dynamically adjusts the retrieval strategy based on the complexity of the input query. This design allows the system to balance efficiency and reasoning depth more effectively than existing fixed-retrieval paradigms, addressing key limitations in current long-video understanding methods.\n\n### Comprehensive benchmark for full-stack video understanding:  \nThe authors propose HiVU, a well-designed benchmark that enables systematic evaluation of video-language models across a spectrum of reasoning complexity. By incorporating a three-level difficulty quantification system and including diverse question types—from fine-grained object recognition to multi-hop temporal reasoning—HiVU provides a comprehensive assessment of full-stack video understanding capabilities. This benchmark fills a crucial gap in evaluating both the breadth and depth of model comprehension in realistic long-video scenarios.\n\n# Weaknesses\n\n### Limited comparison with prior Video-RAG baselines: \nWhile the proposed AdaVideoRAG framework is thoroughly evaluated on several benchmarks, the comparison to prior Video-RAG approaches appears somewhat limited. In Tables 2 and 4, only VideoRAG [36] is used as a baseline, whereas other relevant methods such as VideoRAG [32] are omitted. Although the performance of [32] may be lower than [36], including it would improve the completeness and consistency of the comparisons—particularly on the newly proposed HiVU benchmark, where the set of comparable baselines is already narrow. Additionally, other recent VideoRAG-style works such as Jeong et al. [1] could also be cited and considered for reference.\n\n### Limited baseline diversity on the HiVU benchmark: \nThe HiVU benchmark in Table 4 includes only a small set of evaluated models, limiting our understanding of the generalizability of AdaVideoRAG. It would strengthen the empirical claims if more diverse baselines—such as LLaVA-Video [2] and InternVL 2.5 [3]—were also evaluated under the same framework. Applying AdaVideoRAG to these varied architectures and model scales would not only demonstrate its adaptability, but also provide a valuable point of comparison for future research using HiVU.\n\n---\n[1] Jeong et al. VideoRAG: Retrieval-Augmented Generation over Video Corpus. Arxiv 2025.  \n[2] Zhang et al. Llava-next: A strong zero-shot video understanding model. Arxiv 2024.  \n[3] Chen et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. Arxiv 2024.\n\nQUESTIONS:\n1. In Table 4, it is unclear why the level-1 results are missing from the left side of the table, while AdaVideoRAG results for all levels (including level-1) are shown on the right. Furthermore, the values for AdaVideoRAG appear to differ between the left and right sections of the table, which creates confusion about what each column represents. Could the authors clarify the intended structure of Table 4 and explain why these inconsistencies appear?\n\n2. The paper highlights AdaVideoRAG’s ability to reduce computational overhead by selecting simpler retrieval paths when appropriate. However, the actual latency or cost reduction (e.g., in terms of FLOPs or runtime) is not quantified. Could the authors provide more concrete measurements of retrieval cost across different query types?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}