Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension
Authors: Yongdong Luo
Xiawu Zheng
Guilin Li
Shukang Yin
Haojia Lin
Chaoyou Fu
Jinfa Huang
Jiayi Ji
Fei Chao
Jiebo Luo
Rongrong Ji
Venue: NeurIPS 2025 poster ()
Keywords: retrieval-augmented generation
video large language model
long video understanding
Primary area: deep_learning

Abstract: Existing large video-language models (LVLMs) struggle to comprehend long videos correctly due to limited context. To address this problem, fine-tuning long-context LVLMs and employing GPT-based agents have emerged as promising solutions. However, fine-tuning LVLMs would require extensive high-quality data and substantial GPU resources, while GPT-based agents would rely on proprietary models (e.g., GPT-4o). In this paper, we propose Video Retrieval-Augmented Generation (Video-RAG), a training-free and cost-effective pipeline that employs visually-aligned auxiliary texts to help facilitate cross-modality alignment while providing additional information beyond the visual content. Specifically, we leverage open-source external tools to extract visually-aligned information from pure video data (e.g., audio, optical character, and object detection), and incorporate the extracted information into an existing LVLM as auxiliary texts, alongside video frames and queries, in a plug-and-play manner. Our Video-RAG offers several key advantages: (i) lightweight with low computing overhead due to single-turn retrieval; (ii) easy implementation and compatibility with any LVLM; and (iii) significant, consistent performance gains across long video understanding benchmarks, including Video-MME, MLVU, and LongVideoBench. Notably, our model demonstrates superior performance over proprietary models like Gemini-1.5-Pro and GPT-4o when utilized with a 72B model.

Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension
Yongdong Luo'
Xiawu Zheng'* Guilin Li' Jiayi Ji Fei Chao
Shukang Yin Jiebo Luo
Haojia Lin' Rongrong Ji
Chaoyou Fu
Jinfa Huang*
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China 2 Nanjing University
University of Rochester
Abstract
Existing large video-language models (LVLMs) struggle to comprehend long videos correctly due to limited context. To address this problem, fine-tuning long- context LVLMM and employing GPT-based agents have emerged as promising solutions. However, fine-tuning LVLMS would require extensive high-quality data and substantial GPU resources, while GPT-based agents would rely on proprietary models (c.g., GPT-4o). In this paper, we propose Video Retrieval-Augmentee Generation Wideo-RAG) training-free and cost-effective pipeline that employs visuall auxiliary texts help facilitate cross-modalins alignment while providing additional information beyond the visual content. Specifically, we leverage open-source external tools extract visually- information from pure video data (e.g.. audio, optical character, and object detection). and incorporate the extracted information into an existing LVLLM as auxiliary texts, alongside video frames and queries, in a plug-and-play manner. Our Video-RAG offers several key advantages: (i) lightweight with low computing overhead due to single- turn retrieval; (ii) easy implementation and compatibility with any LVLM; and (iii) significant, consistent performance gains across long video understanding benchmarks, including Video-MME, MLVU, and Long VideoBench. Notably, our model demonstrates superior performance over proprietary models like Gemini- 1.5-Pro and GPT-4o when utilized with 72B model. Codes are available https://githir http://ttub..mmmLooo////// com/Leon1207/vaai
master.
Introduction
With the advancements in Large Language Models (LLMs), numerous studies have been conducted enhance their ability to comprehend and process videos [12, 16, 18. 51, 24, 47, 19, 3, 2, 50, 23, 25, 17]. collectively termed Large Video-Language Models (LVLMs). Although current LVLMs have demonstrated promising performance understanding short videos, effective comprehension of extremely long videos continues to be major challenge.
To address this challenge, recent studies [49, 45, 35, 42, 55] have sought to extend the reasoning context length of LVLMs, essentially finetuning long-context LVLMs for long video understanding, Long VA [49] first introduces increasing the token capacity an LLM and transferring its long-context comprehension capabilities to video data. However, training such model requires pre-training on an extended corpus, and often there are distribution shifts between deployment videos and finetuning videos. As demonstrated in Video-MME [6], LongVA VA declines when increasing the video frame sampling rate from 128 384 (52.6% + 51.8%). This outcome suggests that simply increasing the
"Correspondngg author: zhengxiawu @ xmu.edu.cn
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) Long context LVLMs
Training- feeee
Resource:
Flexbility:
ViceoMie Theenning
Video
Spatial Reasoning
Extend Visual Tokens
Answer
D
Arsslant Peroeplaz
Specially Training LVLM
Action
(b) GPT-based Agent
Training-free:
Resource:
Flexbility:
Video
Query
66
50 30
ViteMMM Statial
Visual Agents
Documents
Interaction
Proprietary (GPT-po)
VideaMMEE Beserning
(c)
Training-free:
Resource:
Hexbility:
Auxiliary texts
Owrall
RerrievllAugmennedd
Q
VipasMil Syoopss
Query
Video
Viduo Da daases
Longsleeeenchh Overal
External
Tools
11
Gerinn-11 1.5mhp
GPT4c
LLahkA-aaeee722 725
Vakeu JSAG
Figure I: (Left) Two common approaches for understanding long videos, alongside our Video-RAG. Video-RAG provides resource-efficien training-frees pipeline compatible with any LVLM. By leveraging RAG, it retrieves auxiliary texts for input, leading to notable performance enhancement. (Right) Comparison of the performance of Video-RAG with LLaVA-Video-72B [52], Gemini-1.5-PP [32], and GPT-4o [29] across various benchmarks, including the sub-tasks from Video-MME [6] (we focus only those that outperform Gemini-1.5-Prot LongVideoBal [43], and MLVU [54].
number of sampled frames not only leads to information redundancy but also imposes additional challenges for the model to handle complex reasoning. Retrieval-Augmentte Generation [14] (RAG) is a technique that enhances generative tasks by retrieving relevant documents from an external corpus, thus improving response quality in LLMs. Recent studies have begun exploring the integration of RAG with video-based tasks [1, 27, 48, 33], employing tools to process videos in long contexts and sending them to proprietary model for generation, which is known as the GPT-based Agent method. However, they come with serval limitations. First, most of them process long video content as plain text, subsequently utilizing the RAG mechanisms to retrieve relevant documents for LLMs. Therefore, they lack alignment with the visual context of the video, resulting in a loss of critical visual information. Second, they are often eesource-intesiing require powerful LLMs to function as the driving force, thus limiting their flexibility and generative capabilities. Executing the whole Video-MME [6] using VideoAgent [4] requires approximately 20 days and incurs a substantial consumption of GPT-4o API tokens.
multi-turn interactions and typically
In this study, we propose Video-RAG, an effective RAG pipeline that can be seamlessly integrated with any LVLM. Specifically, instead of simply increasing the number of sampled video frames, we propose replace the corresponding extended visual tokens with auxiliary texts extracted from pure video data by invoking open-source foundation models, such as optical character recognition (OCR), automatic speech recognition (ASR), and object detection. These auxiliary texts are more aligned with the visual context while providing additional information beyond the visual data, as demonstrated in [20, 4]. Besides dealing with the context windows limit of LVLMs, we employ RAG in Video-RAG to filter auxiliary texts, ensuring their relevance to the user's query in the text embedding space. As sampled visual context often lacks explicit alignment with the instructions, the auxiliary texts can facilitate mross-modality alignment while reducing the modality divide. As illustrated in Figure 5, with Video-RAG, the retrieved auxiliary texts help guide the LVLM to pay more attention to the query-relevant keyframes, while simultancously facilitating cross mods-mt alignment between query and keyframes. this framework, an LVLM serves as the central component Video-RAG, processing visual tokens to preserve detailed visual context and minimize potential information loss. Moreover, the retrieval process is parallelly executed a single operation, ensuring efficiency.
We evaluate Video-RAG across several long video benchmarks, including Video-MME [6], MLVU [54], and Long VideoBench [43]. By applying the Video-RAS to seven distinctive open-source IVLMs, we achieve an average performance improvement of 2.8% on Video-MME with only 2.0K text tokens addition (equal to 14 frames most configuration) per case, while beating the proprietary LVLM when integrated with the 72B model, shown in the right part of Figure 1. Applying Video-

---

RAG to 7B LVLM only requires an additional 8GB of inference GPU memory and approximately 5 seconds of inference time per case (details in the Supplemental Material). In summary, our contributions are as follows:
We integrate RAG into open-source LVLMs: Video RAG incorporates three types of visually-aligned auxiliary texts (OCR, ASR, and object detection) processed by external tools and retrieved via RAG, enhancing the LVLM. It's implemented using completely open-source tools, without the need for any commercial APIs
We design a versatile plug-and-play RAG-based pipeline for any LVLM: Video-RAG offers training-free solution for wide range of LVLMs in plug-and-play manner, delivering performance improvements with minimal additional resource requirements.
We achieve proprietary- performance with open-source models: Applying Video- RAG to 72B open-source model yields proprictary-level performance, surpassing models such as GPT-4o and Gemini-1.5 1.5-ri-
Related Work
2.1
Large Video-Language Models
With the rapid advancement of large language models (LLMs), there has been increasing interest in developing generalist video models capable of handling video-related tasks. Video-ChatGPT [28] extracts features from individual frames and aggregates them through both spatial and temporal pooling operations. VideoChat [16] encodes videos by generating both textual descriptions and video appearance embeddings. Video-LLaVA [18] aligning image and video encoders during pre-processing phase, using a shared projector to map the encoded representations into a common language space.
[51] I] extends LLaVA-NeXT [22] by fine-tuning the model specifically on video data. Despite their contributions, these approaches face challenges when processing long videos, primarily due to the limited number of frames sampled for analysis.
2.2 Long-context Large Video-Language Models
Recent approaches have sought to expand the context window size to enhance long video understand- ing. LongVA VA [49] and Long-LLaVA [45] address this by continuously training LLMs extended textual data, to transfer their long-text comprehension capabilities to video processing. INTP [35] introduces video token rearrangement technique while proposing training-free method for extend- ing the LLM context window, allowing LVLMs to process increased visual tokens. However, these methods face challenges in striking balance between the high computational costs associated with sampling video frames and the limited performance improvements achieved. Due to the inherent redundancy in video content and constraints on model capacity, performance degradation may occur when the number of sampled frames surpasses a certain threshold.
2.3 GPT-based Agent Video Understanding
Initial efforts [46, 41, 8. 38, 33] have employed LLMs interact with tools to process visual information as structured long context for
MM-VID [20] enhances long video understanding by aligning video frames with corresponding text descriptions. VLog [21] leverages multimodel pre-trained models to capture and interpret visual and audio information, summarizing into documents for video comprehension. VideoAgent [4]. DrVideo [27], and OmAgent [48] integrate multimodal inputs and enable dynamic querying of video segments to support long video reasoning tasks. VG [33] and VideoRAG [11] achieve tighter integration between the RAG framework and proprietary models. However, these methods take long time to process videos while relying on proprietary models (e.g., GPT-4o), thus limiting their efficiency and adaptability to other open-source frameworks.
3

---

① Query Decouple
Retrieval Request
3 Integration and Generation
Decouple Prompe
3508
Fromz Sompler
request
"ASI": "Make is ready to the 1strary" ("ook",, "table"), mYWe": ("hue", "loc"]
According the sidea, to the liboary, how many are still placed on the square table the middle of room?
Auy
Any LYLM
Final Answer
Maye and Rormnng
City Location
OCR Teat
Obj Kelatton
Query
2 Auxiliary Text Generation and Retrieval
Q Retriever
Auxillarrl Texts
Q Dequestt"Tveer
(request["AsR")
Audio
Frome Sumpler
request[ ["DET]]
(Mikel f'm mady pato
cerral timw
simtlesity showiskk
Visual Endoed
CLIP Similasity
Scene Graph
(request("
bryffame
Inxenc chresteld
OCR
Princpler of Tertbokk
Grounding
Datahase
Sampled Video Frames
DET Database
Figure 2: The framework of our Video-RAG. In the query decouple phase, the LVLM is prompted to generate a retrieval request for auxiliary texts. Next, in the auxiliary text generation and retrieval phase, the video processed in parallel extract three types of textual information (OCR, ASR, and object detection), and the relevant text is retrieved as the auxiliary text. Finally, in the integration and generation phase, auxiliary texts are combined with the query and the video generate the response.
3 Method
We propose novel, training-free pipeline for large video-language models (LVLMs), named Video- RAG, which can be integrated into any LVLM. As illustrated in Figure 2, our pipeline comprises three key phases: (i) Query Decouple: In this phase, the user's query is decomposed into retrieval request aimed at extracting auxiliary texts from the target video. (fi) Auxiliary Text Generation & Retrieval: Multiple auxiliary texts generated from the queried video in parallel. Then, the retrieval request is used to obtain relevant external information. (iii) Integration and Generation: This phase integrates the retrieved auxiliary texts with the user's query, feeding this combined input into the LVLMs to generate the final response.
3.1 Large Video-Language Model
Given video V. frame sampler first sample N frames F. Most existing methods uniformly sample frames from video for both effectiveness and simplicity. Then, video features are extracted as F, VisualEnc(F)) where VisualEnc is an image-based visual encoder, such as CLIP-L [30]. Finally, the video features F, and the user's 'e query Q are fed into the LVLM to generate an output O:
0 = LVLM(F Q)
3.2 Query Decouple
In this phase, upon receiving a user's query about the video, the LVLM begins by decoupling the query and generating retrieval requests, denoted as R. for auxiliary texts. During this phase, the LVLM processes only textual information, without access to video frames, and the output requests are formatted in JSON. We prompt the LVLM using decoupling prompt P to generate the following retrieval requests as necessary:
Rasr: Requests about automatic speech recognition, to extract audio information from the video that may pertain to the query. (ii) Rae:: Requests for identifying physical entities within the video that may assist in answering the query. (iii) Reype: Requests for details about the location, quantity, and relationships of the identified physical entities. These requests, which may be NULL (the corresponding information is not required), are then parsed and forwarded to the auxiliary text retrieval phase. The entire process be described as:

---

R LVL Q), R (Rasr: Rutets Reype
3.3 Auxiliary Text Generation
In this phase, we first generate the auxiliary texts from the video and then retrieve them to assist the LVLMs according the retrieval requests R. As the length of the video increases, the number of tokens generated from the processed data also grows, leading to an increase in redundant information. Additionally, current open-source models are constrained by the limited length of their context windows, which may prevent them from fully processing all auxiliary texts. To address this issue, we draw inspiration from Setrivval-Aumentter Generation (RAG) [14], retrieving only the auxiliary texts relevant to the user's s query. Before retrieval, we construct the necessary databases from the given video in parallel. Specifically, we implement three distinct databases: the Optical Character Recognition (OCR) database, denoted as DB the Automatic Speech Recognition (ASR) database, denoted as Darri and the Object Detection (DET) database, denoted as D.
OCR database. Current LVLM are still illusory in their ability to accurately recognize characters, and their performance often falls short compared proprietary models. To better leverage the information contained in video frames and reduce hallucinations, we employ a proprietary OCR model to extract text from each video frame with the same frame-sampled strategy as LVLMs. Specifically, EasyOCR [10] as our text recognition model and segmented the recognized texts on a per- frame basis, denoted T Tocr. Subsequently, we implemented RAG by utilizing the advanced text encoding model Contriever [9] to encode the fetched OCR texts into text embeddings E These embeddings are then stored in a database with the FAISS index [13], a library designed for efficient similarity search and clustering of dense vectors. The entire building process can be formally described as:
use
T Tocr EasyDCR(F)
DBar
FAISS
E Contriever(T.rr ocr)
(4)
ASR database. Audio information (e.g., subtitles) plays crucial role in video comprehension, often providing additional context that may not be available through visual cues alone. To incorporate them, we first extract the raw audio U from the video and then transcribe them into texts Tasr. Specifically, we use Whisper [31] as our audio transcription model. Since the recognized texts can be quite extensive, we chunk and encode them into vector database, following the same procedure used to construct the OCR database. The building process can be formally described as:
Tasr (hhisper(U)
(5)
DB
FAISS
Easr Contriever(Tasr)
(6)
DET database. While LVLMs demonstrate strong performance in object recognition, they continue to face challenges such as object counting, precise object localization, and understanding relative relationships between objects. To mitigate the issue of hallucination, which can stem from these challenges, we incorporate object detection information as auxiliary texts. We leverage visual grounding model to extract both the object categories and their corresponding positions from sampled video frames. This approach helps provide more accurate and context-aware object detection. To enhance processing efficiency, we limit object detection to keyframes only. Specifically, we compute the CLIP similarity [30] between the object retrieval request Riget and the sampled video frames F and select relevant keyframes F Key based on threshold t
F Fkey = CLIP ULIP_similarin (Rdet, F} >
Once the keyframes are identified, we utilize APE [36], an efficient open-vocabulary object detection model that accepts object descriptions as prompts to detect relevant objects within frames based on retrieval queries. The capability of APE makes it particularly well-suited to our requirements for on-demand object retrieval. Finally, the detected objects' categories and their corresponding positional information are stored in the DET database using natural language representations:
DBat Taet APE( key. R.ter)
(8)
5

---

3,4 Auxiliary Text Retrieval
During the retrieve phase, we employ Contriever [9] encode the user's query and the parsed requests for OCR and ASR into text embeddings, then concatenating to form the final query request Ereg Contriever(Concatt/
Q)), R. (Roer: Resr). Then we retrieve the auxiliary texts from DB E BDr Da J by the FAISS tool, which computes the vector similarity between the query and text chunks stored in the database. Text chunks with a FAISS similarity score greater than threshold are indexed as the retrieval results A E [Aoer: Aasr): J. The process can be formulated as:
A Index SwisS_similariie
Ereq) >
Since the text generated by the detection model is in a raw format ("category: [x_min, y_min, length, width]"), it challenges LVLMs to understand the relative relationships between objects. We preprocess the object information using a scene graph, which helps to represent spatial and relational information more explicitly. This preprocessing allows construct more coherent and semantically meaningful texts, denoted as Ales which are more readily interpretable by LVLMs. We incorporate three types of object information for each video keyframe: (i) Object Location Atoci This refines the positional information of the object, formatted as: "Object (node ID) } is a (object category) ) located at coordinates [x. y] with dimensions (length width)" (ii) Object Counting Acni This counts the number of objects and generates text in the following format: "Object counting: {object category): (number)" (iii) Relative Positional Relationships Arot This captures the relative spatial relationships between objects using the format: "Object (node ID] ([object category) )) is <positional description> Object (node ID) ([object category))* By combining this information, we construct detailed representation of the objects in the frame, denoted as A = {Alocr Acnat A
A = SceneGraph( (Bed))))
(10)
Finally, we acquire the object auxiliary texts based on the object information type retrieval requests Reype. which selects and finalizes the object auxiliary information Adet. Ade: is one of the elements of the power set P of A det selected by Riype. and the retrieve process can be formulated as:
Aett (P (P(A))))) E (PA
(11)
3.5 Integration and Generation
After obtaining different types of auxiliary texts, organize them chronologically using natural language to create a unified auxiliary input, denoted as Am = Concat (ancat((occr... Aesr: Adet). These merged auxiliary inputs, along with the user's query and the sampled video frames, are then fed into the LVLM to produce the final result. The overall process can be formulated as:
0 LVLM(F,, Concat(A,,, Q))
(12)
4 Experiments
4.1 Datasets
Video-MME [6] is widely used benchmark for assessing the ability of LVLMs handle detailed videos in real-world scenarios. It is divided into three subsets based on video length, with durations ranging from 11 seconds to 1 hour. MLVU [54] long video understanding benchmark with a large wide of 9 distinct tasks. It is created based on long videos of diversified lengths, ranging from minutes 2 hours with about 12 minutes average video length. Long VedeoBench [43] is benchmark designed to accurately retrieve and reason over detailed multimodal information from human-annotated
long videos, with 6,678 human annotated multiple-choice questions in 17 fine-grainel categories.
4.2 Implementation Details
We performed all experiments on NVIDIA A100 80G GPUs. During the auxiliary text generation phase, we first restrict the detection requests Riter generated by LVLMs in decouple prompt then

---

Table 1: Performance the Video-MME [6] benchmark in without subtitles (w/o S), with subtitles (w/ S) and equipped with our Video-RAG (Ours), Frames and Gain means the input frame number and performance gain by applying Video-RAG compared to the baseline with subtitles. By applying Video-RAG to seven LVLMs, we observed an average performance improvement of 2.8% by adding only average of 220K auxiliary texts compared to ~3.0K full-subtitled tokens per sample. In particular, we perform better when applying Video-RAG with 72B LLaVA- Video [52] than the proprietary method GPT-4o [29] (77.4 4% vs. 77.2%). All results are republication.
Short w/o S Ours S Ours w/o S w/ Ours w/o w/ S Ours
Medium
Long
Overall
Model
Params Frames
Gain
Proprietary LVLMs
GPT-4o [29]
384
80.0
82.8
70.3 76.6
65.3
72.1
71.9
77.2
Gemini-1.5-- P.o-Pro [32]
0.5 fps
81.7
84.5
74.3 81.0
67.4
77.4
75.0
81.3
Open-Sone Source LVLMs
Video-LLavA [18]
7B
8
45.3
46.1
49.5
38.0
40.7
43.0
36.2
38.1
42.5
39.9
41.6
45.0
+3.4
I.aVA-NeXT-Vider [51]
7B
16
49.4
51.8
56.6
43.0
46.4 47.4
36.7
44.9
46.0
43.0
47.7 50.0
+2.3
VITA-1.5 [7]
7B
16
67.0
69.9
71.0
54.2
55.7 55.4
47.1
50.4
52.4
56.1
58.7
59.6
+0.9
LongVA [49]
7B
128
61.1
61.2
66.1
50.4
53.8 60.4
46.2
52.9
59.4
52.6
56.0
62.0
+6.0
Long-LLaVA [45]
7B
64
61.9
62.4
67.1
51.4
56.2
60.4
45.4
54.7
60.1
52.9
57.8 62.6
+4.8
Qwen2-Ve VL [40]
72B
32
75.0
76.7
77.4
63.3
69.9
70.2
56.3
69.2
71.0
64.9
71.9
72.9
+1.0
LLaVA-Videe [52]
72B
64
80.7
81.8
82.8
68.7
73.8
76.3
62.1
72.2
73.
70.3
75.9
77.4
+1.5
further filter them using spaCy, ensuring they correspond to CeIP-sensitive physical entities, avoiding the inclusion of abstract concepts. In the auxiliary text retrieval phase, we set both the CLIP similarity threshold and the FAISS similarity threshold to 0.3. We employ the IndexFlatIP as the similarity calculating method of FAISS [13]. Note that we don't include the GPT-based Agent methods for comparison due to their resource-intensive nature (complete execution of Video-MME [6] costs around $2000 for API purchasing when using VideoAgent [4]). Still, we include mini-experiment of VideoAgent the Supplemental Material that compares the overall performance, inference time, and GPU requirements with two common long-context LVLMs and Video-RAG.
Table 2:
The overall performance in the multiple- choice lask of the ML.U [54] benchmark. # do- nates the results of our replication.
Table 3: The overall performance on the validation set of Long VideoBench [43]. donates the results of our replication.
Model
#Params Frames Overall
Model
#Params Frames Overall
Proprictary LVLMs
Proprietary IVLMs
Gemini-1.55 1.5-Pr [32]
256
64.0
GPT-4o [29]
0.5 fps
64.6
GPT-4u [29]
256
66.7
Open-Source LVLMs
Open-Source LVLMs
VITA-1.5 1.5
7B
16
60.4
AieeChhat-MMistr [16]
7B
39.3
Video-CAA [5]
14B
96
63.1
Video-XL [37]
7B
256
64.9
ShareGPT44idae [2]
7B
39.7
LLaVA-Naa -Next-Misttal [22]
49.1
Aria [15]
25.3B
256
70.6
LLaVA-Video [52]
7B
64
70.8
PLLaVA [44]
53.2
VITA-115 [7]
53.6
Oryx-1.5 [26]
32B
128
72.3
LLaVA-Video [52]
72B
64
73.1
LLaVA- Videe [52]
7B
64
56.6
LLaVA- [52]
72B
64
61.9
LLaVA-Video Video-RAG
7B
64
72.4
LLVA--Viee Viden-RAG
7B
64
58.7
Video-RAG
72B
64
73.8
LaVA-Video + Video-RAG
72B
64
65.4
4.3 Main Results
Video-MME. We evaluate our Video-RAG in five 7B open-source LVLMs, including Video-LLaVA [18], LLaVA-NeXT-Vides [51], Long [49], Long-LLaVA [45], and two 72B LVLM Qwen2-Vi VL [40] and LLaVA-Video [52]. Constraining by computational resources, we evaluate the LVLMs with their official frame rate setting in Video-MME except for 72B Qwen2- VL, which requires about 3K GPU memory with 768 video frame input (~~38 A100 GPUs). Results are shown in Table 1. Specifically, after applying our Video-RAGi in 72B VA- [52], we perform better than proprietary model GPT-4o [29] (77.4% vs. 77.2%). Across the seven LVLMS used in our experiments, we gained an average performance boost of 2.8% compared to results with subtitles, especially significant gain on long vidcos, demonstrating its effectiveness. This performance improvement is
7

---

achieved by incorporating token counts from approximately 14 additional video frames (equivalent to 2.0K tokens). each contributing around 144 tokens under most LVLM configurations. We obtain such a large performance enhancement because most LVLMs are pre-trained primarily within the text space and aligned with visual information, often lacking explicit alignment between embedding spaces. Auxiliary texts can serve as semantic supplements sensitive to LVLMs, facilitating model activation and easing the understanding of complex videos.
MLVU. We evaluate Video-RAG when integrating into the 7B and 72B LLaVA-Vided [52] of MLVU [54], a benchmark that close to performance saturation. As shown in Table Video-RAG's 1.6% improvement at 7B-scale is substantial, considering that it outperforms the 32B Qryx-1.5 [26] by 0.1%, while recent 7B-scale models average only 1.3% gain (across 15 approaches in MLVU's leaderboard). Additionally. the 72B LLaVA- LLaVA-Video Video also has performance gain of 0.7%, which sets
new state-of-the-arr.
Long BongVideeBench. We evaluate Video-RAG when applied in the 7B and 72B I.LVA--Vded [52] of Bong [43]. We omit the interleaved input format introduced in Long VideoBench when applying Video-RAG The evaluation results in Table demonstrate that 72B LLaVA- with our Video-RAG achieves an overall performance of 65.4% on the validation set. This result surpasses the proprietary LVLM Gemini-1.554 [32] by 1.4%, securing the second place, just 1.3% behind GPT-4o [29]. Meanwhile, the 7B LLaVA- -Vdeo also has performance enhancement of 2. % when equipped with our Video-RAG.
4.4 Ablation Studies
Effect of different sampling frame number. To explore the effect of the number of sampling frames
on Video-RAG, experience sampling frames number of 8, 16, 32, 64, 128, and 256 in 7B model LongVA [49], results are shown in Figure 3. As demonstrated, Video-RAG consistently delivers performance improvements across all frame rates, especially in long videos. The experimental results also indicate that Video-RAG can achieve higher performance gains with fewer frames, demonstrating its potential for applications under resource-constrained conditions.
Effect of different components of Video-RA RAGG. To explore the effective- ness of auxiliary texts, we add DET, OCR, and ASR as auxiliary texts be- fore and after retrieving by the RAG to evaluate Long-LLaVA, 32-frame setting in the Video-MME [6] benchmark. As shown in Table the performance of Long-LLaye pro-
Performance in Mile-MMM: bome benchmark
70
60
317
[45] with
50
40
gressively improves as auxiliary texts after retrieving by the RAG system are incrementally added (52.0% + 52.9% + 55.7% + 62. 1%). Among these components, ASR auxiliary texts con- tribute to a general improvement for different video durations, especially for long videos. When all components are integrated, we obtain an optimal performance, as shown in the last row Table Meanwhile, the experiment shows 2.3% improvement (59.8% vs 62.1%) 1%) in performance after incorporating RAG for retrieval, demonstrating that auxiliary texts after retrieving by the RAG system are guire-aligned, which helps cross- modsltty alignment. We also evaluate across sub-tasks within Video-MME [6] and other video benchmarks like MLVU [54], BinnViideBee)h431)] and VNBench [53], more details are shown in the Supplemental Material.
20
Frames
16 Frames
Frames
64 Frames 128 Frames 256 Frames
Figure 3: Performance gain with different sampling frames rate on Video-MME [6] when implement Long VA-7B [45].
Effect of different thresholds of RAG processing. When retrieving, we specify similarity threshold t as a criterion for information selection. In the retrieval for OCR and ASR texts, information is selected if its FAISS similarity exceeds t. For object detection, frames are selected as keyframes based on their CLIP similarity surpassing I, and the relevant information then extracted. Setting too high may hinder the retrieval of relevant information while setting it too low can result in information redundancy and increased reasoning complexity. To investigate this trade-off, we conduct ablation experiments evaluate the impact of different threshold values. The results are shown in Table 5.
8

---

Table 4: Results on combinations of different aux- iliary texts in Video-MME [6] when using Long- LLaVA-7B [45] as the L.I.M.
Table 5: Performance with different thresholds of retrieval on Video-MME [6] when using Long- LLaVA-7B [45] as the LVLM. #Token and Time denote the total token number of the auxiliary texts and the average inference time per question, respec-
RAG DET OCR ASR Short Medium Long Overall
60.3
51.4
44.1
52.0
tively.
62.2
55.4
54.4
57.4
61.0
56.2
55.0
58.4
#Token Time
Short Medium Long Overall
63.0
57.3
56.4
58.9
64.3
58,8
56.3
59.8
0.0
3.6K
36s
67.6
59.4
59.1
62.0
0.1
3.4K
30s
67.0
59.7
59.1
61.9
61.4
51.9
45.2
52.9
0.2
2.7K
18s
66.0
60,2
59.2
61.8
63.2
53.2
46.3
54,3
0.3
1.9K
11s
66.4
60.2
59.8
62.1
65.1
59.1
60.7
61.6
0.4
0.8K
8s
65.6
58.0
58.3
60.6
64.1
54.6
48.4
55.7
0.5
0.3K
7s
63.1
54,9
50.2
56.1
64.9
60.7
61.5
1.0
0.0K
6s
60.3
51.4
44.1
52.0
66.3
60.3
59.3
62.0
V
66.4
60.2
59.8
62.1
rdd
1.9K
11s
65.7
55.8
56.0
59.1
demonstrulingg Germany Christmas tree initially which kind of the decoration has the largest number? A. Apples. B. Candles, Besiesi D. The three kinds are of the same number.
apples,
may B.
Buminng EVLAI
X
request (
"ARRT' aull, ("sppees,, "candles", "Berries)), *TyPp"s ("haaber")
Ob Contting
auplns:
Old Detecttion
Vidlen JEAG
Paradia
Eivyy FINInIO
Figure 4: Qualitative result on Video-MME [6] when applying Video-RAG with LLaVA- [52].
w/j Ametitary Texts
with Austilary Tests
NEW
CCD LVVMM
NEW AT 6
Video-Ri0
(quespitreeecaat Prvsses with attention]
[Key Trome with move attention]
[More altgned crose reaat-modallty featured
Figure 5: Grad visualizations of the last hidden state heatmap along with t-SNE visualizations of the user's query and keyframe features of the example shown in Figure 4. The retrieved auxiliary texts help cross- mods-mydallty alignment by assisting the model to pay more attention query-relevant keyframes and thus generate more robust and accurate answers the user's query.
Notably, f = 0 and f = correspond to all auxiliary texts input into the model and no auxiliary texts input, respectively. To balance performance with information density and processing time (especially APE [36] detection in keyframes), we selected threshold of 0.3 for our implementation. More details about similarity scores are shown in the Supplemental Material. Under this configuration, the additional text length of approximately 1.9K tokens typically remains within the context window limits of open-source LVLMs. For models with more stringent context window limitations, a threshold of 0.4 may also be a viable option. We also randomly sample an equivalent token number of auxiliary texts to serve as inputs for assessing the effectiveness of RAG retrieval, as shown in the last row of Table 5.
4.5 Qualitative Evaluation
We present qualitative results in the case of Video-MME [6] in Figure 4 and Figure 5. As illustrated, augmenting LLLVA-Vides with external tools to process and retrieve auxiliary texts from videos significantly enhances its ability to reduce visual hallucinations, thereby enabling more accurate responses to user queries. Grad-CAM [34] and t-SNE [39] visualization results also show that applying RiGco-RAG helps the LVLM's cross-modality alignment.

---

Conclusion
In this paper, we present Video-RAG for effective long video understanding through integrating retrieved auxiliary texts with LVLMs, achieving proprietary-level performance with 72B open-source LVLM. Unlike traditional methods that are resource-intensive with limited gains, Video-RAG offers resource-efficient, alug-and-play solution leveraging only open-source tools to extract visually- aligned auxiliary texts from video data. However, Video-RAG may be limited by the visual tools we choose and their performance, which lacks adaptation. In the future, will explore how more efficiently integrate auxiliary texts and provide an adaptive frame selection strategy for LVLMs.
6 Acknowledge
This work was supported by the National Science Fund for Distinguished Young Scholars (No.62025603) the National Natural Science Foundation of China (No. U21B2037, No. U22B2051, No. U23A20383, No. U21A20472. No. 02126 No. 62176223, No. 62176226, No. 62072386, No. 62072387, No. 62072389, No. 62002305 and No. 62272401), the Natural Science Foundation of Fujian Province of China (No. 2021J06003, No.2022J06001), and the Fundamental Research Funds for the Central Universities.
References
[1] Md Adnan Arefeen, Biplob Debnath, Md Yusuf Sarwar Uddin, and Srimat Chakradhar. irag: An incremental retrieval augmented generation system for videos. arXiv preprini arXiv:2404. /2309, 2024.
[2] Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang. Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understanding and generation with better captions. arXiv preprini 2rXiv:2406.043255 2024.
[3] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang. Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistie tasks. In Proceedings of the IEEE/CVF Conference on Computer 2024.
Vision and Pattern Recognition, pages
[4] Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Da, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory-augmente multimodal agent for video understanding. European Conference on Computer Vision, pages 92-92. Springer, 2025.
[5] Jiajun Fei, Dian Li, Zhidong Deng, Zekun Wang, Gang Liu, and Hui Wang. Video-ccam: Enhancing video-language understanding with causal cross-attention masks for short and long videos. arXiv preprint arXiv:2408. 14023, 2024.
[6] Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis. arXiv preprint arXiv:2405.21075, 2024.
[7] Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Haoyu Cao, Zuwei Long, Heting Gao, Ke Li, al. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction. arXiv preprint arXiv:2501.0199 2025.
[8] Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 14953-14962. 2023.
[9] Gautier Izacard, Mathilde Caron, Lucas Hosscini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2 /2.09/ 18, 2021.
[10] JaidedAI. Easyocr. https: htaps:///iiee //github..ttttt/// ctw Comm/JaiiedA//BasyyOo 2023.
10

---

[11] Soyeong Jeong, Kangsan Kim, Jinheon Baek, and Sung Ju Hwang. Videorag: Retrieval- augmented generation over video corpus, arXiv preprint arXiv:2501.05874, 2025.
[12] Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. arXiv preprini arXiv:23 1.08066, 2023.
[13] Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Trunsuctions on Big Data, 7(3):335-547, 2019.
[14] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et Retrieval-augmentt generation for wewwledge-ittessiie nlp tasks. Advances in Neural Information Processing Systems, 1199459-94744 2020.
[15] Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Guoyin Wang, Bei Chen, and Jumnan Li. Aria: An open multimodal native mixture-of--et of-peerrtt-eessseeeeeeeeeee model. arXiv preprini urXiv:2410.05992 2024.
[16] KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang. and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.0633 06355, 2023.
[17] Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pages 323-340. Springer, 2025,
[18] Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311. 2023.
[19] Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IÉEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16689-266699 2024.
[20] Kevin Lin, Faisal Ahmed, Linjie Li, Chung-Ching Lin, Ehsan Azamasab, Zhengyuan Yang, Jian- feng Wang, Lin Liang, Zicheng Liu, Yumao Lu, et al. Mm-vid: Advancing video understanding with gpt-4v (ision). arXiv preprint 2rXiv:2310.1977: 2023.
[21] Qinghong Lin. Vlog: Transform video as a document with chatgpt, clip, blip2, grit, whisper, langchain. https /ttp//gtttttt//t///////ee com/ showlab/ ab/LLo, 2023.
[22] Haotian Liu, Chunyuan Li, Yuheng Li. Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Leava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https: //1lava-vl, github. ho/blog//02444 200000/2222-0--30000 11t-------------a--eeeeeeee nextt
[23] Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xiaoqi Ma, xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. Kangaroo: A powerful video-language model supporting long-context video input. arXiv preprint arXiv:2408. /5542, 2024.
[24] Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. St-lim: Large language models are effective temporal learners. In European Conference on Computer Vision, pages 1-18. Springer, 2025.
[25] Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Oryx mllm: On- demand 2024.
understanding at arbitrary resolution. arXiv preprint arXiv:2409.12961
[26] Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Oryx mllm: On- demand spatial-temporal understanding at arbitrary resolution. arXiv preprint arXiv:2409. 122.61. 2024.
[27] Ziyu Ma, Chenhui Gou, Hengcan Shi, Bin Sun, Shutao Li, Hamid Rezatofighi, and Jianfei Cai. Drvideo: Document retrieval based long video understanding. arXiv preprint arXiv:2406. /2846, 2024.
11

---

[28] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424 2023.
[29] OpenAI. Gpt-4o system card. https://openal:: www.coooicnneeeeeee hawmee///g-----ssstee---aaa 2024.
[30] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748-8763. PMLR, 2021.
[31] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and llya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492-28518, PMLR, 2023.
[32] Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap. Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou. Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint
2024.
[33] Xubin Ren, Lingrui Xu, Long Xia, Shuaiqiang Wang, Dawei Yin, and Chao Huang. Vide- orag: Retrieval-augmentee generation with extreme long-context videos. arXiv preprint arXiv:2502.01549 2025.
[34] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- localization. In Proceedings of the IEEE international conference computer vision, pages 618-626, 2017.
[35] Yuzhang Shang, Bingxin Xu, Weitai Kang. Mu Cai, Yuheng Li, Zehao Wen, Zhen Dong, Kurt Keutzer, Yong Jae Lee, and Yan Yan. Interpolating video-llms: Toward snger-sequence Imms in training-free manner. arXiv preprint arXiv:2409. 12963, 2024.
[36] Yunhang Shen, Chaoyou Fu, Peixian Chen, Mengdan Zhang, Ke Li, Xing Sun, Yunsheng Wu, Shaohui Lin, and Rongrong Ji. Aligning and prompting everything all once for universal visual perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13193-13203, 2024.
[37] Yan Shu, Peitian Zhang. Zheng Liu, Minghao Qin, Junjie Zhou, Tiejun Huang, and Bo Zhao. Video-xl: Extra-long vision language model for hour-scale video understanding. arXiv preprint htti:2400914488. 2024.
[38] Didac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11888-11888 1189,, 2023.
[39] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
[40] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, al. Qwen2-vl: Enhancing rision-languaa model's perception of the world at any resolution. arXiv preprint 121111409.122111, 2024.
[41] Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video understanding with large language model as agent. arXiv preprint arXiv:2403.1055 2024.
[42] Xidong Wang, Dingjie Song, Shunian Chen, Chen Zhang, and Benyou Wang. Longllava: Scaling multi-modal llms to 1000 images efficiently via hybrid architecture. arXiv preprint arXiv:2409.02889 2024.
[43] Haoning WU, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. arXiv preprint arXiv:2407.15754 2024.
12

---

[44] Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404. 16994, 2024.
[45] Yin Song and Chen Wu and Eden Duthie. ww--ppoarriinggnooglllaaaee--n2- 2024. URL petps://hugginge wuggingface....mooooooooooooog.. cw/ww/ wwt.-oooooooooooogggo--ooo 1owaaaoonnnaaaa--rrr----------
[46] Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A simple Ilm framework for long-range video question-answening arXiv preprint arXiv:2312.17255 2023.
[47] Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tunce audio-visual language model for video understanding. arXiv preprint arXiv:2306. 02858, 2023.
[48] Lu Zhang, Tiancheng Zhao, Heting Ying. Yibo Ma, and Kyusong Lee. Omagent: A multi-modal agent framework for complex video understanding with task divide-and-conquee arXiv preprint arXiv:2406. 66200 2024.
[49] Peiyuan Zhang, Kaichen Zhang. Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang. Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language vision. arXiv preprint arXiv:2406. 16852, 2024.
[50] Yi-Fan Zhang, Qingsong Wen, Chaoyou Fu, Xue Wang, Zhang Zhang, Liang Wang, and Rong Jin. Beyond Ilava-hd: Diving into high-resolution large multimodal models. arXiv preprini UrXiv:2406.08887, 2024.
[51] Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: strong zero- shot video understanding model, April 2024. URL https://llava- vl.
20opbloog//22--//////////// 20-2004-22200000000000000000000- 30-30- 111330111ll -ww--nieeetto....
[52] Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data, 2024. URL https : / /ttxxvxxvvvv///////////////a prrg/abs/2410 027113.
[53] Zijia Zhao, Haoyu Lu, Yuqi Huo, Yifan Du, Tongtian Yue, Longteng Guo, Bingning Wang, Weipeng Chen, and Jing Liu. Needle in video haystack: A scalable synthetic framework for benchmarking video milms. arXiv preprint, 2024.
[54] Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406. 042644, 2024.
[55] Yongshuo Zong, Ismail Elezi, Yongxin Yang, Jiankang Deng, and Timothy Hospedales. Long- context vision large language models: Empirical insights and a baseline. In Workshop on Long Context Foundation Models, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer the abstract and Section I, also the experimental results can be found at Section 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: In the Section 5, we discuss the future work of our proposed method, and also discuss limitations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: In Section 4, we describe the implementation details of our experimental results, and also open-source our code anonymously, which can be found our supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We open-source our code with an anonymous link in our supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //niss httppsccccpbccccccceececcccccccc httpe/guuudeeeeeeuubiuieeeeeee /Cooeeeodiissseeeeeee cessionnoocyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We describe the implementation details in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Because the experiment requires the use of large number of resources, only provide the data from one experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Table 5, we provide the time of execution of our method, and we also include relative contents in supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: //tner/in...pp/e http/uublc//tBhhee htte//hhiiiiiiiiieeeeeeeee csGuidelines"
Answer: [Yes]
Justification:
We have ensured that the paper complies with the guidelines.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of our paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our paper poses no such risks since it is based on open-source tools or models without providing any new model.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the datasets and models used in this paper all from open-source collections like Github and HuggingFace, with the license of CC-BY 4.0.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
The LLM models involved the study are all soureed from open-source communities, Additionally, any extra LLM tools are used solely to assist with writing and do not affect the core methods, scientific rigor, or originality of the research.
Justification:
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standare components.
Please refer to our LLM policy (https: //tpur/ees../////ee cttp.ccccoocooooe / for what should or should be described.
20

---

Supplemental Material
A Decouple Query
In the initial phase of the proposed Video-RAG, we employ a decouple prompt, denoted as P, to guide the LVLM in generating retrieval requests. In this section, we present one example of prompt designed for multiple-choice questions, as illustrated Figure 8.
B Sub-set of Video-MME
As outlined in the implementation details, we randomly sampled subset of the Video-MME [6] dataset to evaluate a computationally resource-intensive, agent-hased method with long-context LVLMs. Specifically, we selected 10% of the full dataset, comprising short, medium-length, and 30 long videos. Each video contains three multiple-choice questions. Importantly, ensured that the performance ranking of the methods on the subset mirrored that the full dataset. As shown in Tables 6 and 7, we evaluated four distinct 7B models Chat-Univi-v1.5 1.5 [12]. LLaVA-NeXT: Video [51], LongVA [49], and Long-LLaVe [45] using a frame sampling rate of 16 for both the subset and the full set. Our results indicate that the performance rankings remained consistent across both evaluations.
Table Performance of Video-MME sub-set.
Method
Short Medium Long Overall
Chat-Univi-v1.55 [12]
50.0
33.3
17.8
33.7
LLaVA-NeeT-Vicde [51]
54.4
33.3
23.3
37.0
Long [49]
56.7
50.0
38.9
48.5
Long-LLLaVA [45]
58.9
52.2
40.0
50.4
Table 7: Performance of Video-MME full-set.
Method
Short Medium Long Overall
Chat-Univi-vl.. 1.5 [12]
45.7
39.0
35.7
40.1
-eaVA-NeXT-VVieo [51]
51.1
41.8
36.8
43.2
Long VA [49]
60.8
45.2
41.4
49.1
Long-LLaVA [45]
59.3
49.3
44.4
51.0
C Results on Video-MME Sub-Set
We examine Video-RAC against two representative methods in terms of inference time, GPU resource requirements, and overall performance. Given that GPT-based Agent methods are resource-intensive. we randomly sampled a sub-set of the Video-MME [6] for evaluation, as described in Section B. As demonstrated in Figure 6, VideoAgent [4], typically GPT-based Agent method, requires significant time to process video and deliver suboptimal performance. Meanwhile, Long [49], representative long-context LVLM, shows limited improvement from increasing the frame rate and even experiences performance degradation. Integrating our Video-RAG into the 16-frame LA results in substantial performance improvements while reducing GPU resource consumption. Specifically, with only increasing 8GB GPU memory compared to the base (16-frames Long VA). we achieve 11.5% overall performance improvement, while outperforming another long- long-context LVLM Long-LLa VA-7B [45] in 16-frames setting by 9.6% with less GPU memory requirements and compatible total inference time. These results demonstrated that our Video-RAG is lightweight with lower computing overhead than the other typical methods. Moreover, we provide detailed time consuming to construct three types of databases (which can be built in parallel) and inference per query, shown in Table 8.
21

---

Table 8: Overall performance, databases construct and average inference time (include building databases) per
query (#Time) in Video-MMIE-mini.
w/o subs
w/ Video-RAG
Model
ASR
OCR
DET
Total Time
#Time Overall
#Time
Overall
VideoAgent
14min
47.7
LongVA-166
21min 2min 3min max(21, 2, 3m=21min
1s
48.5
5s 5s
60.0
LongVA 128ss 21min 16min 16min max(21, 16, 16)=21 min
8s
54.1
8s + 58
63.3
LongVA
LongV 384fs
42min 48min 24min max(42, 48, 24)=48min
20s
53.7
20s + 11s
63.6
Ovcrall Paformexein Vaho-MME subrest (%)
LongVA (16 frames) + Video-RAG
Saving 150G GPU Memory
14% better
I LongVA (128 frames)
Long-LLaVA (16 frames)
50
LongVA (381 frames)
LongVA (16 frames)
GPT-Basee Content LVLM
Contert
VideoA gent fps)
Our Video- RAG
60
140
160
G GPU Resource R Requlrements (G
Figure 6: The comparison of our Video-RAG with two common approaches. The size of the bubbles represents the total time consumed for completing inference on the Video-MME [6] sub-set.
D Details of Similarity Score Calculation
In the process of using the RAG system retrieve auxiliary texts extracted from videos, define similarity threshold / to ensure the selection of relevant texts. Specifically, we employ FAISS- bassed [13] similarity select OCR and ASR texts, while CLIP [30] similarity is used for keyframe selection. In our implementation, the similarity threshold is set to 0.3. As for OCR and ASR selection, For any given list of the retrieve request R and auxiliary texts A, the Contriever [9] framework maps the text to a text embedding as:
Ea, Contriever A+), i = 1.2...., E,, Contriever(R.). i 1.2.....
The average embedding of the retrieve request is then computed as:
E,
E.
n
After that, the embedding of the request and the list of auxiliary texts is normalized:
Ex; Ea,
E,
Eas
E,
E,
The similarity between the query embedding E, and the document vector E, is computed using the inner product, the FAISS library employed to efficiently perform this search and return the indices of the auxiliary texts meeting the criterion:
S(E,, E.,) E, E., >t
As for object detection, we use CLIP to select the video keyframe. During this process, we first filter the object detection request Rete to ensure they correspond CL.P-sensitive physical entities, avoiding the inclusion of abstract concepts. Specifically, if it is single word, direct ofrt-of-speech
22

---

filtering is applied: if is a compound word, certain rules are followed to check for compliance, such as whether it an adjective plus noun, or noun plus a noun. We use the Spacy library achieve this. After this, we put the text "A picture of" before each object detection request.
Then, extracting embedding from both the video frames F and the detection request Rudet!
EF, CLI
1.........
ER,- = CLaee
i= 1,2.......
The similarity between each video frame and the detection retrieve requests is computed using the dot product between the image and text feature embeddings. For each frame Fj. and for each retrieve request ER,, the similarity score given by:
S;j = EF, ER,
where denotes the dot product. The final similarity score for each frame is the average similarity across all requests:
5, = 71
S
This computes the mean similarity for each frame across all text descriptions, resulting in a similarity The similarity scores are adjusted by scaling factor which is computed based on the number of frames m and base frame number (which is 16 and 4.0,
vector S = [S1, ..
S.
respectively) to adapted different video sampling rate of LVLMs:
B
where B is predefined scaling parameter. Next, the similarity scores are scaled and normalized to ensure that they sum to
S
Syor
M k= Sp
where Syor represents the normalized similarity score for frame Fj-
The final step is to select the keyframes based on the normalized similarity scores. A threshold / is applied to the normalized similarities, such that frames with similarity scores above the threshold are selected as keyframes:
Keyframe: F, if Sper > t
Thus, the set of selected keyframes is given by:
F Fkey = (F, Sporm > t, j = 11,2...,, mm)
E More Ablation Studies
Effect different components of Video-RAG. We evaluate the performance across sub-tasks within Video-MME [6], as shown in Figure 7. The results reveal that object detection auxiliary texts significantly enhance spatial perception and object counting, while OCR auxiliary texts specifically improve performance text recognition tasks. Additionally, ASR auxiliary texts contribute to general improvement inference tasks, underscoring the critical role of audio transcription in video understanding. Given that audio transcription is considerably more time-consuming than character recognition or object detection, these texts should be selected based on the requirements of the application.
Besides studying the inference of different components of Video-RAG in the Video-MME [6] benchmark, we also experiment with different type of video benchmark. We first evaluate LLaVA- Video in MLVU [54] and Long VideoBench [43] in both and 72B scale with the 64-frame setting, results are shown in Table 9, As demonstrated, when all components are combined, we get optimal
23

---

Action Recognition
Object Recognition
Attribute Perception
OCR Problems
Spatial Perception
Counting Problems
80 70 60 50 4 30 20 100
Temporal Perception
Temporal
Information Synopsis
Reasoning
Spatial Reasoning
Object Reasoning
Action Reasoning
Long-LLavA
Long-LLaV++DET. OCR+
Long-LLavA DET OCT+
Figure Performance on 12 sub-tasks in Video-MME [6] benchmark after applying different components in Long-LLaVA.
Table 9: Ablation study in MLVU and Long VideoBench.
7B
72B
RAG DET OCR ASR
MLVU
LVB
MLVU
LVB
70.8
56.6
73.1
61.9
71.0
56.5
73.4
63.2
71.3
56.8
73.5
63.4
72.4
58.7
73.8
65.4
70.3
58.3
72.9
64.0
performance in both datasets, including 7B and 72B scales. Specifically, the performance in MLVU [54] even declined when the RAG system was not implemented.
Then, to better point out the role of DET and OCR, we evaluate Video-RAG in VNBench [53] with Long-LLaVA-7B [45]. VNBench a synthetic benchmark designed evaluate models' long-context abilities, covering tasks such as retrieval, ordering, and counting. VNBench randomly inserts stickers or text into the video that has nothing to do with the original content of the video, thus typically challenging the model's needle-in-the the----hhhhystcck capability. As shown in Table 10, we find that applying DET and OCR as auxiliary texts can significantly improve the performance in retrieval, ordering, and counting tasks. However, the ASR component will decline the performance due to the subtitles are not ancillary to this partieular task. These results demonstrated that our proposed distinct types of auxiliary texts can be selected according to the application needs to meet the requirements better.
More Qualitative Results
In this section, show more results of LLaVA-Vdieo-P7 when applying Video-RAG in different examples in Figure 9. The figure highlights several representative cases involving detailed video comprehension from Video-MME [6]. As illustrated, augmenting LLaVA-Video with external tools to process and retrieve auxiliary texts from videos significantly enhances its ability to reduce visual hallucinations, thereby enabling more accurate and confident responses user queries.
24

---

Decouple Prompt of the Multiple-choice Question
To answer the question step by step, list all the physical entities related to the question you want to retrieve, you can provide your retrieve request to you by following
"ASR": Optional[str]. The subtitles of the video that may relavent to the question you want to retrieve, in two sentences. If you no need this information, please
null.
"DET": Optional[list]. (The output must include only physical entities, not abstract concepts, less than five entities) All the physical entities and their concepts.
question you need for this information, please return null.
related
retrieve,
"TYPE": Optional[list]. (The output must be specified as null list containing only one or more the following strings: 'location', 'number', valid
'relation'. No other to obtain about the detected objects. If you need the object location in the video frame, output "location"; if you need the number of specific object, output "number"; you need the positional relationship between objects, output "relation".
field) information you
## Example
Question: many blue balloons room at the end of this video? B. 2.
long table the middle of
Your retrieve can be:
"ASR": "The location and the color of balloons, the number of the blue balloons.",
"DET": ["blue ballons", "long table"),
"TYPE": ["relation", "number"]
## Example 2:
Question: the lower left corner of video, what color the woman wearing the right side of the man in black clothes? A. Blue. B. White. Red. D. Yellow.
Your retrieve can be:
"ASR": null,
"DET": ["the man in black", "woman"],
"TYPE": ["location", "relation"]
## Example 3:
Question: In which country is the comedy featured the video recognized worldwide? China. B. Germany. D. United States.
Your retrieve can be:
"ASR": country recognized worldwide conedy.",
"DET": null,
"TYPE": null
}
Note that you don't need infomation about the video of image. You only need to provide retrieve request (it's optional), and will help you retrieve the infomation you want. Please provide the json format.
the question in this step, you don't 't need
Figure 8: Decouple prompt of the multiple-choice question for LVLMs.
25

---

Table 10: Results combinations of different auxiliary texts in VNBench [53] with 1-try setting
when applying 7B Long-LLaVA [45] as LVLM under the 32-frames setting. Ret, Ord, and Cnt represent retrieval, ordering, and counting tasks, respectively.
RAG DET OCR ASR
Ret
Ord
Cnt
Overall
65.1
25.6
24.2
38.3
66.9
28.4
23.8
39.7
v
68.2
31.3
28.9
42.8
66.7
31.3
29.6
42.5
When and where did the athlette and student first the video? A. 12:30 PM the
B. PM in the contoon.
La:go PM the classroom:
D. 12:00 PM the classroom.
Aul talentak
X
EVEM
request
the AsS buildinng class guz Sust get
MUSHT: "The athlete first neet the student,", ("athlete", "student"),
ATILETES
STUDENT: 12: PM]
The CEETETI
TYPE": ["relation")
Video RAG
OCR ASR Dalabasss
peetty
Video RAG
2C3TM
1100 SEREERA
[OCI Dirame =]
[OCR Framey]
How attractions shown the video?
A.
B.10)
Lt.
aud Viles
Finetuning LYLM
X
request
"WSR":
"The namter of the attractions.,
Death Valleys
duath valliy
ctop is
"DET": ("uttactooss),
Frace 18: Cathedral Rock
netional park
The answer B!
["nasber"]
Viddoo BAAG
Dotebasus
OUR FASR
aodona
Video RAG
LOURE Pramne
[OCR Framee)
[OCR Tiemm vol
Where is the disappeuring voin gone in the fifth magic? throan under the hridden behind playing
hidkden
hiddeen
magician's
Dut Viza Liz
X
Finetuning EYEM
roquest
"Asin's
gens. "DEt": Poon" "cable", "slewve",
Ale
"playing card", "pxln!"). ["relation"]
dropping the
(coin) aoove doject (pala)
The unswer
Videoo
DET
Figure 9: Qualitative results of LLaVA- when applying Video-RAG.
26


Reviews:
[{"review_id": "olFiTMxt0L", "reviewer": "Reviewer_aLCV", "summary": "This paper proposes Video-RAG, a training-free pipeline to enhance long video understanding for large multimodal models. This approach firstly leverages open-source tools to extract visually-aligned auxiliary texts (e.g., OCR, ASR, object detection) from videos, integrates them via Retrieval-Augmented Generation (RAG) to align cross-modal information, and feeds them into VLMs alongside video frames and queries. Evaluations on benchmarks including Video-MME, MLVU, and LongVideoBench show that Video-RAG improves average performance by 3.2% across six open-source LVLMs, with a 72B model even outperforming proprietary models like GPT-4o. Key contributions include integrating RAG with open-source LVLMs, a plug-and-play design, and achieving proprietary-level performance with open-source models.", "questions": "1. How does Video-RAG perform on videos with poor audio quality (e.g., noisy environments) where ASR accuracy drops significantly?\n2. What is the impact of using lower-quality open-source tools (e.g., a less accurate OCR than EasyOCR) on Video-RAG’s performance?\n3. Are there cases where auxiliary texts introduce noise, and how does the pipeline mitigate such negative effects?", "limitations": "1. The novelty of this paper is somewhat limited and this paper focuses on the engineering techniques for video understanding.\n2. The proposed method is training-free but this paper lacks further research on how to train video agents with tools.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The proposed method is training-free and plug-and-play, which also supports some proprietary models.\n2. The proposed method obtains performance improvements on several long video benchmarks, such as VideoMME, MLVU, and LongVideoBench.\n3. The proposed method is easy to follow and might perform on downstream applications.\n\nWeaknesses:\n1. The performance of Video-RAG is determined by the quality of the external tools (EasyOCR, Whisper, APE). Errors or inaccuracies in transcription, OCR, or object detection will inevitably propagate and potentially mislead the LVLM. The paper does not sufficiently analyze the impact of such errors.\n2. The inference latency is higher compared to end-to-end models. The process involves several tools to extract information from long videos which will slow down the inference speed for practical applications.\n3. This paper focuses on single-turn process which limits the performance for iterative reasoning or multi-turn conversations.\n4. This paper lacks the experimental / qualitative comparisons with previous VideoAgent-like methods, such as VideoAgent[1,2, 3], and other open-source well-established baselines, such as OmAgent for video understanding, which also adopts similar tools, such as ASR.\n\n\nReferences\\\n[1] Wang et al. VideoAgent: Long-form Video Understanding with Large Language Model as Agent. ECCV 2024.\\\n[2] Fan et al. VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding. ECCV 2024.\\\n[3] Jeong et al. VideoRAG: Retrieval-Augmented Generation over Video Corpus. ACL 2025.", "was_revised": true, "final_justification": "I have carefully read the author's response, which has addressed most of my concerns. I believe this work is practical for long video understanding, and I have no further questions. I tend to accept this paper.", "rebuttal": "Thank you for the time, thorough comments, and nice suggestions. We are pleased to clarify your questions step-by-step.\n\n> **Q1**: The errors or inaccuracies in transcription, OCR, or object detection that will inevitably propagate and potentially mislead the LVLM.\n\n**A1**: Follow your suggestion, we replaced the components of Video-RAG with alternative components that demonstrated lower performance, and conducted ablation studies on the Video-MME, revealing **only a negligible drop in performance, demonstrating the robustness of Video-RAG to component variability**.\n\nSpecifically, we have substituted EasyOCR in the OCR component with Tesseract OCR[1] and the OCRAD API. For the DET component, we have replaced APE with Owl ViT[2] and GRiT[3]. Additionally, in the ASR component, we have replaced Whisper with the SpeechRecognition library and the Vosk API.\n\n| Method                  | Short | Medium | Long | Overall |\n| :---------------------- | :----: | :----: | :----: | :----: |\n| Video-RAG (Base) | 77.0 | **69.3** | 63.8 | **70.0** |\n| replace with Tesseract OCR [1]       | 77.0 | 68.3 | 63.2 | 69.5 |\n| replace with OCRAD API              | **77.6** | 67.8 | 63.9 | 69.7 |\n| replace with Owl ViT [2]             | 76.7 | **69.3** | 63.6 | 69.9 |\n| replace with GRiT [3]                | 76.4 | 68.4 | **64.0** | 69.6 | \n| replace with SpeechRecognition library  | 77.4 | 67.8 | 63.3 | 69.5 |\n| replace with Vosk API               | 76.3 | 68.2 | 63.7 | 69.4 |\n*All models adopt LLaVA-Video-7B with 64 frames for inference.*\n\n> **Q2**: Are there cases where auxiliary texts introduce noise, and how does the pipeline mitigate such negative effects?\n\n**A2**: To investigate this, we injected controlled noise into the text outputs of external tools and evaluated LLaVA-Video-7B-64frames equipped with Video-RAG in Video-MME. Results indicate that **Video-RAG demonstrates robustness in handling noise**. The underlying mechanism is that **the key contextual information retrieved through the RAG process has a high information density**, which helps effectively mitigate performance degradation due to noise and redundant content by filtering out such interference.\n\n| Noise Ratio   | 0% | 10% | 20% | 30% | 40% | 50% |\n| :-------------       | :--: | :--: | :--: | :--: | :--: | :--: |\n| Overall Acc. w/ RAG  | 70.0 | 69.8 | 69.6 | 69.4 | 69.6 | 69.7 |\n| Overall Acc. w/o RAG | 68.6 | 68.3 | 68.2 | 68.0 | 67.9 | 67.6 |\n\n> **Q3**: The inference latency is higher compared to end-to-end models, which slows down the inference speed for practical applications.\n\n**A3**: We appreciate your observation regarding the additional latency introduced by the tools associated with Video-RAG. It’s important to note that our analysis suggests that **a significant portion of the inference latency for long videos is primarily due to the LVLM itself**. The contribution of these tools to the overall inference time decreases notably with the increase in the number of sampled frames. As highlighted in Supplementary Table 3, when processing videos with 384 frames, we draw the following conclusions:\n\n- The base end-to-end LVLM (LongVA-7B [4]) incurs an approximate inference time of 20 seconds per sample on Video-MME.\n- With the cost of additional ~11 seconds inference time increasing, our Video-RAG can achieve a substantial performance improvement of 9.9%.\n\n> **Q4**: This paper focuses on single-turn process which limits the performance for iterative reasoning or multi-turn conversations.\n\n**A4**: Following your suggestion, we have conducted experiments involving Video-MME with IRCoT[5], demonstrating that employing a multi-turn scheme in Video-RAG resulted in a 1.3% enhancement in performance for long videos. \n\nConsequently, we assert that the examination of multi-round question answering warrants further investigation, and we intend to pursue additional research in this domain in the future. Notably, we focused our experiments on long videos (exceeding 30 minutes) because multi-turn question answering is particularly effective for long videos that require complex reasoning. \n    \n| Method                                                |  Long Video Acc. |\n| :--------------------------------------               | :------: | \n| Baseline            | 44.1 | \n| Baseline + Video-RAG (1-turn)    | 59.8 | \n| Baseline + Video-RAG (3-turn, IRCoT) | **61.1** | \n*Baseline is the  Long-LLaVA-7B with 32 frames for inference.*\n\n> **Q5**: This paper lacks experimental/qualitative comparisons with previous VideoAgent-like methods.\n\n**A5**: Following your suggestion, we have conducted further comparative experiments involving the VideoAgent-like methods VideoAgent[6], VideoAgent[7], and LLoVi[8]. The results indicate that **Video-RAG consistently outperforms other leading VideoAgent-like methodologies**. Given the considerable API resources required by agent-based approaches, these experiments were executed using our Video-MME-mini benchmark, as detailed in Supplementary Section 3. We would like to clarify that VideoRAG[9] was not initially designed for video understanding tasks, and OmAgent[10] incorporates a highly complex architecture specifically tailored to leverage GPT-4o. We attempted to substitute GPT-4o in OmAgent with LLaVA-Video-72B, but this resulted in inaccurate model outputs. As a result, we were unable to perform comparative experiments on VideoRAG and OmAgent. We intend to incorporate these experimental findings into the main text in future work.\n\n| Method                                    | Short | Medium | Long | Overall |\n| :--------------------------------------   | :-: | :----: | :----: | :----: |\n| Video-RAG (Ours)                     | **85.6** | **74.4** | **68.9** | **76.3** | \n| VideoAgent[6]                        | 50.0 | 46.8 | 42.1 | 46.3 | \n| VideoAgent[7]                        | 64.8 | 56.6 | 44.2 | 55.2 | \n| LLoVi[8]                             | 79.0 | 69.0 | 65.9 | 71.3 | \n\nWe use LLaVA-Video-72B in 64 frames as the LVLM in Video-RAG, while other Agent-like methods are all driven by GPT-4. \n\n> **Q6**: The novelty of this paper is somewhat limited and this paper focuses on the engineering techniques for video understanding. The proposed method is training-free but this paper lacks further research on how to train video agents with tools.\n\n**A6**: We appreciate the reviewer's insightful observation. While Video-RAG incorporates some established components, we wish to elucidate its principal contributions as follows:\n\n- **Essence of Video-RAG**: Video-RAG fundamentally establishes a straightforward yet innovative workflow. By leveraging RAG, it assimilates outputs from a variety of multimodal tools, thereby equipping LVLMs with a more comprehensive and information-rich context. This integration significantly ameliorates performance in video understanding tasks.\n\n- **Value proposition of Video-RAG**: The primary significance of this work stems from our recognition that the potential of LVLMs remains underutilized. Video-RAG illustrates that supplying a sufficiently dense and relevant context can more effectively orient the LVLM’s attention towards crucial visual elements. This enhancement promotes improved multimodal interaction and fusion. Future research could expand on Video-RAG by integrating additional external tools, facilitating adaptations to diverse application scenarios.\n\n- **Broader research relevance**: It is important to highlight that the design of such workflows that integrate tool outputs for targeted tasks or the distillation of high-quality contextual inputs represents a significant research avenue within the broader LLM domain. Recent methodologies, including Agentic-like approaches, context compression techniques, and context state management strategies, fall within this thematic area. Our analysis reveals that papers fitting this category comprise approximately 10%–12% of the accepted submissions at NeurIPS 2024, emphasizing the critical importance of this area of inquiry.\n\n- **Future directions regarding training**: We acknowledge the reviewer’s pertinent observations concerning the significance of training methodologies, such as recent works (e.g., ToolRL[11]) that utilize reinforcement learning strategies to enhance the tool-calling capabilities of LLMs. Your suggestion provides a valuable trajectory for future optimization: investigating how to further fine-tune LVLMs so they can more adeptly incorporate high-quality contextual information sourced from external tools.\n\n[1] R. Smith. An Overview of the Tesseract OCR Engine. Ninth International Conference on Document Analysis and Recognition. 2007.\n\n[2] M Minderer, et al. Simple Open-Vocabulary Object Detection with Vision Transformers. ECCV 2022.\n\n[3] Jialian Wu, et al. GRiT: A Generative Region-to-text Transformer for Object Understanding. ECCV 2024.\n\n[4] Peiyuan Zhang, et al. Long Context Transfer from Language to Vision.\n\n[5] Trivedi H, et al. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. ACL 2023.\n\n[6] Fan, et al. VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding. ECCV 2024.\n\n[7] Wang, et al. VideoAgent: Long-form Video Understanding with Large Language Model as Agent. ECCV 2024.\n\n[8] Ce Zhang, et al. A Simple LLM Framework for Long-Range Video Question-Answering. EMNLP 2024.\n\n[9] Jeong, et al. VideoRAG: Retrieval-Augmented Generation over Video Corpus. ACL 2025.\n\n[10] Lu Zhang, et al. OmAgent: A Multi-modal Agent Framework for Complex Video Understanding with Task Divide-and-Conquer.\n\n[11] Cheng Qian, et al. ToolRL: Reward is All Tool Learning Needs."}, {"review_id": "q01U6PzHsA", "reviewer": "Reviewer_BTw3", "summary": "This paper proposes Video-RAG, a novel training-free and visually-aligned retrieval-augmented pipeline for long video comprehension. Instead of directly increasing the number of sampled frames or relying on proprietary LLMs, the authors leverage open-source tools (e.g., Whisper, EasyOCR, APE) to extract textual information from videos (ASR, OCR, DET). These auxiliary texts are then filtered using RAG mechanisms to select query-relevant information, which is finally integrated with sampled video frames and the user query into a large video-language model (LVLM). The proposed method achieves consistent performance gains across multiple long-video benchmarks (Video-MME, MLVU, LongVideoBench) and even outperforms GPT-4o when integrated with a 72B LVLM.", "questions": "1. How robust is Video-RAG to frame sampling errors, especially when key visual cues are missed due to uniform sampling in ultra-long videos?\n  2. Could the authors elaborate on how spatial relations (e.g., “on top of”) are computed in the scene graph from raw object coordinates? Why is “Obj Relation” missing in Figure 2 despite an apparent object interaction?\n  3. Has the method been evaluated or extended to handle temporally structured queries (e.g., \"after X happens, what does Y do\")? How would Video-RAG handle such temporal dependencies given its current frame-wise processing?", "limitations": "The method depends on the existing models.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n1. Video-RAG introduces a lightweight framework that requires no model training or fine-tuning. It can be directly applied to any LVLM with minimal computation overhead, making it highly practical and easy to adopt in real-world or low-resource scenarios.\n2. Unlike prior RAG approaches that treat videos as plain text, this work extracts and retrieves auxiliary texts (OCR, ASR, object detection) that are semantically aligned with the visual content. This alignment significantly improves cross-modality grounding and reduces hallucinations.\n3. The method demonstrates solid and consistent improvements across various long video QA benchmarks. When paired with LLaVA-Video 72B, Video-RAG achieves performance surpassing GPT-4o, with only ~2K additional tokens and marginal GPU overhead.\n\n# Weaknesses\n1. While Video-RAG avoids increasing the frame count directly, its overall performance remains fundamentally constrained by the initial uniform sampling strategy. If a query-relevant moment (e.g., a keyframe containing a crucial visual cue) is not sampled, the downstream RAG process cannot recover that information, limiting its robustness to extremely long videos.\n2. The paper introduces a scene-graph-based preprocessing step to convert raw object detection outputs into structured descriptions, including spatial relations. However, it lacks a clear explanation of how these relations are determined (e.g., what geometric rules define “on top of” or “next to”). Moreover, Figure 2 shows the \"Obj Relation\" as NULL, even though a visible spatial relation (\"books on table\") should likely be present, raising concerns about completeness and reliability.\n3. Although the method enhances static visual-textual alignment, it does not model temporal dynamics or sequence-level reasoning, which are crucial for many long video tasks (e.g., procedural understanding, action tracking, cause-effect). All auxiliary texts are retrieved independently per frame, without leveraging inter-frame continuity or temporal structure.", "was_revised": true, "final_justification": "The proposed method for long video understanding represents a promising direction in lightweight agentic design, eliminating the need for large-scale model training.", "rebuttal": "We sincerely thank you for the constructive and valuable comments. The concerns are addressed as follows.\n\n> **Q1**: Video-RAG's overall performance remains fundamentally constrained by the initial uniform sampling strategy. How robust is Video-RAG to frame sampling errors especially in ultra-long videos?\n\n**A1**: Thanks for your constructive suggestions. We employed **uniform sampling** for two primary reasons:\n- To ensure that our frame sampling strategy aligns with the majority of LVLMs, thereby **demonstrating the plug-and-play capability of our method without reliance on key frames selection**.\n- To facilitate a fair comparison. Empirical evidence indicates that uniform sampling provides the most computationally balanced and generalized solution for video understanding, as corroborated by state-of-the-art open-source models such as Qwen2.5-VL and LLaVA-Video.\n\nIn response to your question about how robust Video-RAG is to frame sampling errors, we conducted additional experiments incorporating the ASK[1] (CVPR 2025) keyframe selection method alongside Video-RAG. The results from Video-MME reveal **Video-RAG demonstrates robustness against the loss of keyframe information, particularly in the context of long videos**. This finding underscores two important implications:\n- Negligible orthogonal effects between keyframe selection and Video-RAG.\n- Video-RAG highlights the system's ability to compensate for the absence of critical visual cues through auxiliary text support.\n\n| Method                                          | Short | Medium | Long | Overall| \n| :--------------------------------------         | :-: | :----: | :----: | :----: | \n| Baseline    | 75.4 | 62.6 | 51.8 | 63.3 | \n| Baseline + ASK[1]   | 76.1 | 62.9 | 54.4 | 64.5 | \n| Baselin + Video-RAG   | 77.0 | **69.3** | 63.8 | 70.0 |\n| Baseline + ASK[1] + Video-RAG  | **78.1** | **69.3** | **63.9** | **70.4** |\n*Baseline is the LLaVA-Video-7B with 64 frames for inference.*\t\n\n> **Q2**: Could the authors elaborate on how spatial relations are computed in the scene graph from raw object coordinates?\n\n**A2**: The spatial relationships are determined by  **calculating the positions of bounding boxes generated through object detection**. The computational workflow is as follows:\n\n---\n\n**Input**: Accepts a list of objects, each characterized by an ID, class label, and bounding box represented as [xmin, ymin, width, height].\n\n**Processing**:\n1. Constructs a directed graph using NetworkX, where nodes represent objects and edges represent spatial relationships.\n2. Performs pure geometric computations (without external dependencies) to determine pairwise relationships between objects:\n- Overlap (determined by bounding box intersection)\n- Relative positioning (including left/right and above/below relationships, where one bounding box is completely situated in the designated direction relative to another without overlap)\n- Prevents duplicate edges by enforcing a node1 < node2 ordering\n\n**Output**: Produces natural language descriptions based on spatial relationships, formatted as follows: \"Object 3 (dog) is to the left of Object 7 (cat).\"\n\n---\n\n> **Q3**: Why is “Obj Relation” missing in Figure 2 despite an apparent object interaction?\n\n**A3**: This is because the figure illustrates a reasoning process where **the LVLM determined that addressing this particular question required only positional and number information in the query decouple phase, making object relations unnecessary (hence NULL)**.\n\nWe would like to further clarify:\n\n- **Different models may utilize varied reasoning paradigms**. For example, Long-LLaVA-7B would consider object relations to answering the same question.\n\n- **Object relations are fundamentally derived from the positions of objects for better understanding by LVLM**. More advanced models generally depend less on explicit relation annotations, favoring instead a direct analysis that utilizes raw bounding box data.\n\n> **Q4**: Questions about temporal reasoning of Video-RAG.\n\n> **Q4.a**: Although the method enhances static visual-textual alignment, it does not model temporal dynamics or sequence-level reasoning, which are crucial for many long video tasks.\n\n**A4.a**: In long video understanding, modeling temporal dynamics or sequence-level information is indeed crucial. However, we deliberately avoided explicit modeling of these aspects to ensure **improve performance in video understanding across diverse durations while keeping modeling complexity to a minimum**. The temporal modeling techniques you referred to are indeed complementary to our work, and we sincerely appreciate your suggestions for pursuing this avenue in future research.\n\n> **Q4.b**: Has the method been evaluated or extended to handle temporally structured queries? \n\n**A4.b**: An example can be found in Video-MME, which includes temporal reasoning tasks such as \"What does Y do after X occurs?\" as well as tasks involving the ordering and reasoning of temporal events. In these contexts, **Video-RAG exhibits significant enhancements in both temporal perception and reasoning tasks**.\n\n| Method                                  |  Temporal Perception | Temporal Reasoning |\n| :-------------------------------------- | :----: | :----: | \n| Baseline        | 50.9   | 40.7   |\n| Baseline + Video-RAG   | 69.1   | 45.8   |\n| Performance Gain                    | (+18.2)  | (+5.1)   |  \n*Baseline is the Long-LLaVA-7B with 32 frames for inference.*\t\n\n> **Q4.c**: How would Video-RAG handle such temporal dependencies given its current frame-wise processing?\n\n**A4.c**: The enhancement of temporal perception and reasoning task of Video-RAG arises from the use of temporally structured text encoding, which involves the **chronological integration of auxiliary texts** (such as frame-wise detection transitioning to tracking, incremental ASR facilitating semantic coherence, and sequential OCR reflecting text dynamics). This approach effectively captures video dynamics without requiring explicit temporal modeling.\n\n[1] Tang X, et al. Adaptive keyframe sampling for long video understanding. CVPR 2025."}, {"review_id": "pIyynzwArf", "reviewer": "Reviewer_YJ27", "summary": "This paper presents a RAG-based Video QA technique, which extracts speech, OCR results, and visual objects related to a query, and feeds it to a LLM with video frames. The use of JSON-structured retrieval requests is logical, and the method outperforms existing VideoQA methods on three datasets: Video-MME, MLVU, and LongVideoBench.", "questions": "Are there any synergistic effects resulting from these components? Table 4 shows a detailed comparison of each condition, indicating additive rather than synergistic contributions. Data showing such synergistic effects would motivate reevaluation.", "limitations": "The authors show mainly positive cases in its main part and supplementary (Fig. 4). An error analysis would increase the contribution of this paper to the community.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 1, "originality": 1, "strengths_and_weaknesses": "## Strength \n1. The implementation is reasonable and the result is reliable.\n2. The presentation is very clear.\n3. The proposed method performs better than previous methods.\n4. The experiments are thorough.\n\n## Weakness\n\nThe technological contribution is considered insufficient for the NeurIPS standard. Using ASR for extracting video events combined with RAG is a standard technique in video analysis, as indicated by the following sources [a, b].\n\n- [a] MultiModal RAG for Advanced Video Processing with LlamaIndex & LanceDB, https://www.llamaindex.ai/blog/multimodal-rag-for-advanced-video-processing-with-llamaindex-lancedb-33be4804822e, 2024-02-17\n- [b] Do Jung et al., \"Speech Retrieval-Augmented Generation without Automatic Speech Recognition,\" arxiv, 2024.12\n\nTherefore, the novelty lies potentially in using OCR and object detection with video frames. However, each component is an existing method, and this reviewer recognizes less significance than other average NeurIPS-accepted studies.", "was_revised": true, "final_justification": "Even after reviewing the rebuttal and discussion, I did not find a strong reason to consider the technical novelty is significant enough as a NeurIPS paper. Hence, I maintain the rating. A blog-post has already introduced a similar technique. Hence, the novelty lies in the multimodal extension but the modality selection is somewhat ad-hoc and one of the modalities, RAG with ASR, is also another known technique. More analysis on the modality selection will be required to clarify the significance of the paper to be a NeurIPS level study. If ignoring [a] and [b], the novelty of the paper can be misrecognized, and adding [a] and [b] requires an amount of rewriting.", "rebuttal": "Thanks for your constructive suggestions! Here are our clarifications.\n\n> **Q1**: Using ASR for extracting video events combined with RAG is a standard technique in video analysis, the novelty lies potentially in using OCR and object detection with video frames.\n\n**A1**: It is worth noting that our Video-RAG introduces a straightforward yet innovative workflow by leveraging RAG to consolidate outputs from various multimodal tools. This approach **equips LVLMs with a comprehensive and information-rich context, significantly enhancing performance in video understanding tasks**.\n\nAlthough all models employ ASR and utilize RAG to fetch relevant textual information to improve video understanding,  our Video-RAG has the following highlight:\n\n**Differences between Video-RAG and the referenced works:**\n\n- While these efforts focus on combining agents for video comprehension, Video-RAG specifically emphasizes how to further enhance the capabilities of LVLMs through information-rich auxiliary context inputs.\n\n- Many of these methods are complex and rely heavily on GPT, utilizing tools to convert video content into text for video QA. In contrast, Video-RAG prioritizes simplicity and flexibility, centering on LVLMs for visual content processing, allowing for plug-and-play integration.\n\n- Video-RAG maximizes existing LVLM performance with minimal resource increase compared to the referenced works that demand higher resource consumption.\n\n**Core Contributions of Video-RAG:**\n\n- Video-RAG retrieves high-density auxiliary texts, directing the LVLM’s focus to relevant visual cues, thus promoting cross-modality alignment.\n\n- Video-RAG's thoughtful design enables effective connections between LVLMs and external tools through RAG, making it adaptable to various scales of LVLMs while enhancing performance.\n\n- Video-RAG with open-source LVLMs demands minimal additional GPU memory and inference latency, often outperforming proprietary models, especially notable with a 72B model.\n\nIn summary, the integration of tool outputs for targeted tasks and the generation of high-quality contextual inputs represent a **significant research avenue within the LLM domain**. Recent methodologies addressing these themes, including Agentic-like approaches and context management strategies, have seen notable interest, with approximately 10%–12% of accepted NeurIPS 2024 submissions aligning with this critical area of inquiry.\n\n> **Q2**: Are there any synergistic effects resulting from the components of Video-RAG?\n\n**A2**: Based on our literature review, we define a [synergistic effect] as **[Gain from Component A + Gain from Component B] > [Gain from Component A alone] + [Gain from Component B alone]**. \n\nAn analysis of Long-LLaVA-7B-32frames with Video-RAG in Video-MME indicates that:\n\n- Table 4 in the manuscript indicates that **OCR and DET demonstrate synergistic effects under RAG conditions in Video-RAG**, with a gain of +3.7 exceeding the expected gain of +(0.9+2.3=3.2), resulting in a synergy of Δ=+0.5. This suggested that RAG forces OCR and DET to spatially align (e.g., only reading text on detected objects), eliminating noise from irrelevant regions. \n\n- Our additional experiments further demonstrate **synergistic effects between ASR and DET without RAG conditions in Video-RAG**, with a gain of +6.9 exceeding the expected gain of +(0.9+5.4=6.3), resulting in a synergy of Δ=+0.6. The results reveal that ASR provides temporal action context (e.g., \"opening the box\") while DET locates the corresponding object, creating grounded semantics. \n\nWe carefully considered your insightful comments and designed rigorous experiments to more effectively demonstrate the synergistic interactions between components, thereby highlighting their interdependence as following:\n\n| Method | Baseline | DET | OCR | ASR | DET + OCR | DET + ASR\n| :------ | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: |\n| Overall Acc. | 52.0 | 52.9(+0.9) | 54.3(+2.3) | 57.4(+5.4) | 55.7(+3.7) |  58.9(+6.9) | \n\n> **Q3**: The authors show mainly positive cases in its main part and supplementary (Fig. 4). An error analysis would increase the contribution of this paper to the community.\n\n**A3**: Thank you for your valuable feedback! In response to your suggestions, **we performed an error analysis on approximately 100 samples from Video-MME**. This analysis has led us to identify two primary challenges that contribute to the errors and bottlenecks experienced in Video-RAG:\n\n- Bottleneck 1: **Key Frame Selection Errors in Object Detection**. The selection process for key frames sometimes fails, resulting in either the omission of relevant objects or incorrect identifications. This leads to the loss of essential object information or the introduction of erroneous data into subsequent processing stages.\n\n- Bottleneck 2: **Challenges in Complex Spatiotemporal Reasoning**. The current approach exhibits limitations when addressing questions that necessitate complex spatiotemporal reasoning, particularly those requiring iterative referencing and the synthesis of information from diverse segments of the video.\n\nTo address these limitations in future work, we propose the following directions:\n\n- For Bottleneck 1: The existing key frame selection process within the DET component relies on a CLIP-based similarity retrieval approach. While this method provides speed and versatility, it exhibits limited sensitivity to high-level semantic textual information. To enhance this process, **we propose substituting CLIP with a smaller LVLM, such as InternVL3-1B**, which would evaluate frame relevance through a binary choice format (e.g., \"Is this frame relevant to the query? A. Yes / B. No\"). The model's confidence score (e.g., the logit associated with \"A\") could then serve as the criterion for key frame selection. Preliminary experiments suggest that this adjustment leads to performance improvements, which is worth future exploration:\n\n| Method                              |  Overall Acc. |\n| :---------------------------------- | :------: | \n| Baseline                         | 52.0 | \n| Baseline + CLIP-based DET    | 52.9 | \n| Baseline + InternVL3-1B-based DET  | 53.3 | \n*Baseline is the Long-LLaVA-7B with 32 frames for inference.*\n\n- For Bottleneck 2: The existing single-round question-answering paradigm achieves a particular balance between performance and inference latency. However, our experiments reveal that **multi-turn QA significantly outperforms single-round approaches in long-video comprehension**. Future research will aim to enhance the complex reasoning capabilities of LVLMs by implementing designs that incorporate Chain-of-Thought strategies, more sophisticated processing pipelines, and expanded utilization of analytical tools.\n\n| Method                                    |  Long Video Acc. |\n| :--------------------------------------   | :------: | \n| Baseline            | 44.1 | \n| Baseline + Video-RAG (1-turn)   | 59.8 | \n| Baseline + Video-RAG (3-turn, IRCoT[1]) | **61.1** | \n*Baseline is the Long-LLaVA-7B with 32 frames for inference.*\n\n[1] Trivedi H, et al. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. ACL 2023."}, {"review_id": "YPv5NIJSMH", "reviewer": "Reviewer_LHvP", "summary": "The article first decouples the question into Retrieval Requests, then employs various tools (e.g., OCR) to retrieve relevant audio and text within Video from the video. Finally, these two modalities are converted into text and input into the LVLM to obtain the Final Answer.", "questions": "1. Would using a larger model for Query Decouple improve pipeline performance?\n\n2. For the raw video, what sampling method did the article use—uniform sampling, similarity-based sampling, or another method?", "limitations": "Authors discuss the limitations of the work in the   Conclusion.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nBeyond commonly used video frmaes and subtitles, the article utilizes multiple tools to retrieve relevant audio clips and text within video frames, using them as auxiliary text to improve performance. The method achieves a good trade-off between performance and effectiveness. The experiments are sufficient and validate the effectiveness of the proposed approach.\n\nWeakness:\n1. Could Query Decouple become a performance bottleneck? If the Retrieval Request generated by the LVLM is inaccurate, then all subsequent steps will be inaccurate.\n2. The article uses multiple different external tools and sets various judgment conditions. This makes the method potentially sensitive to the choice of different tools (e.g., using different OCR or Grounding models) and hyperparameters. The article should discuss the impact of different external tools and hyperparameters on the final method.\n3. For long-video understanding, another class of methods calculates the relevance between frames and the question, then samples frames based on relevance to answer user queries, such as [1], [2]. The article should discuss and compare these in related work.\n\nMinor Issues:\n\n1. The code repository link is expired.\n\n2. In line 39, \"most of them process long...\", please add citations to clarify which specific methods \"them\" refers to.\n\n[1] Re-thinking Temporal Search for Long-Form Video Understanding\n\n[2] BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding", "was_revised": true, "final_justification": "The authors have addressed my concerns through detailed experiments and acknowledged certain limitations of their method. While some explanations, such as the use of the Information Bottleneck principle, remain unconvincing and overly generic, I recognize that the authors are moving in a promising direction. Therefore, I have raised my score to accept.", "rebuttal": "We sincerely thank you for the valuable comments. \n\n> **Q1**: Could Query Decouple become a performance bottleneck?\n\n**A1**: Empirically, Query Decouple resulting in minimal impact on overall performance as it's a straightforward task for LVLMs. This conclusion is supported by two key observations:\n\n- Consistent improvements across model capacities: As illustrated in Table 1 of the paper, Video-RAG provides stable performance enhancements when implemented with various 7B models, with less powerful models showing proportionately greater gains. Notably, on the Video-MME-mini benchmark (as shown in Supplementary Section 3), even the lowest-performing baseline model, Video-LLaVA-7B, achieved approximately 90% query decoupling accuracy across 270 queries.\n\n- Deliberate noise robustness testing: We conducted experiments involving random text noise injections to the decoupled outputs. The results revealed that LLaVA-Video-7B-64frames, when paired with Video-RAG, maintained performance within acceptable limits even at significant levels of noise, as shown on the Video-MME benchmark. These findings collectively demonstrate that Query Decouple does not limit Video-RAG's effectiveness.\n\n| Noise Ratio  | 0% | 10% | 20% | 30% |\n| :------      | :--: | :--: | :--: | :--: |\n| Overall Acc. | 70.0 | 69.7 | 69.7 | 69.4 |\n\n> **Q2.a**: The article should discuss the impact of different external tools.\n\n**A2.a**: As suggested, we conducted ablation studies on various components of Video-RAG on Video-MME, and we observed **only minimal declines in performance**. For each component, we assessed two alternative implementations. \n\nSpecifically, we have substituted EasyOCR in the OCR component with Tesseract OCR[1] and the OCRAD API. In the DET component, we have replaced APE with Owl ViT[2] and GRiT[3]. Additionally, in the ASR component, we have replaced Whisper with the SpeechRecognition library and the Vosk API. The sustained performance demonstrates the robustness of Video-RAG to component variability.\n\n| Method                  | Short | Medium | Long | Overall |\n| :---------------------- | :----: | :----: | :----: | :----: |\n| Video-RAG (Base) | 77.0 | **69.3** | 63.8 | **70.0** |\n| replace with Tesseract OCR [1]       | 77.0 | 68.3 | 63.2 | 69.5 |\n| replace with OCRAD API              | **77.6** | 67.8 | 63.9 | 69.7 |\n| replace with Owl ViT [2]             | 76.7 | **69.3** | 63.6 | 69.9 |\n| replace with GRiT [3]                | 76.4 | 68.4 | **64.0** | 69.6 | \n| replace with SpeechRecognition library  | 77.4 | 67.8 | 63.3 | 69.5 |\n| replace with Vosk API               | 76.3 | 68.2 | 63.7 | 69.4 |\n*All models adopt LLaVA-Video-7B with 64 frames for inference.*\n\n> **Q2.b**: The article should discuss the impact of different hyperparameters of the external tools.\n\n**A2.b**: Regarding hyperparameters, due to time constraints, we limited our evaluation to different confidence thresholds in the APE of the DET component and EasyOCR of the OCR component in Video-RAG. The results demonstrated that Video-RAG can maintain strong performance across a range of component hyperparameter combinations, indicating its robustness to the settings of these hyperparameters.\n\n| Method                  | Short | Medium | Long | Overall |\n| :---------------------- | :----: | :----: | :----: | :----: |\n| OCR threshold=0.3            | **77.0** | 68.8 | **63.9** | 69.9 |\n| OCR threshold=0.5 (Original) | **77.0** | **69.3** | 63.8 | **70.0** |\n| OCR threshold=0.7            | 76.7 | 68.9 | 63.6 | 69.7 |\n| DET threshold=0.1 (Original) | **77.0** | **69.3** | 63.8 | **70.0** |\n| DET threshold=0.2            | 76.3 | 69.4 | 63.7 | 69.8 |\n| DET threshold=0.3            | 76.2 | 68.9 | 63.7 | 69.6 |\n*All models adopt LLaVA-Video-7B with 64 frames for inference.*\n\n> **Q3**: For long-video understanding, another class of methods calculates the relevance between frames and the question, then samples frames based on relevance to answer user queries. The article should discuss and compare these in related work.\n\n**A3**: Thank you for your constructive suggestions. Our research identifies a total of six recent studies that can be categorized in this domain (e.g., ASK[4], Nar-KFC[5], etc.). Below, we outline the key distinctions between Video-RAG and these existing methods:\n\n- **Motivation**: The primary objective of these studies is to identify and select frames that are pertinent to the query for LVLM. In contrast, Video-RAG aims to enhance the capabilities of the LVLM by providing contextually relevant textual input that directly relates to the query.\n\n- **Methodology**: The studies above utilize algorithms to calculate image-text similarity to extract frames. Conversely, Video-RAG employs a refinement process that consolidates outputs from external tools through the RAG approach.\n\n- **Effectiveness**: The performance improvements in these studies are relatively modest. In comparison, Video-RAG showcases a substantial enhancement in performance, achieving a  +6.7% boost with Video-RAG in contrast to a +1.2% boost with ASK[4] in the Video-MME using LLaVA-Video-7B with 64 frames.\n\nFurthermore, we have conducted experiments on Video-MME by integrating the frame selection method ASK[4] (CVPR 2025) with Video-RAG. The limited performance gain suggests that Video-RAG does not significantly depend on keyframe selection. This system demonstrates resilience to the absence of keyframes by utilizing complementary modalities to support the reasoning process of the LVLM.\n\n| Method                                       | Short | Medium | Long | Overall | \n| :--------------------------------------      | :-: | :----: | :----: | :----: | \n| Baseline               | 75.4 | 62.6 | 51.8 | 63.3 | \n| Baseline + ASK[4]   | 76.1 | 62.9 | 54.4 | 64.5 | \n| Baseline + Video-RAG    | 77.0 | **69.3** | 63.8 | 70.0 |\n| Baseline + ASK[4] + Video-RAG  | **78.1** | **69.3** | **63.9** | **70.4** |\n*Baseline model adopts LLaVA-Video-7B with 64 frames for inference.*\n\n> **Q4**: Would using a larger model for Query Decouple improve pipeline performance?\n\n**A4**: Utilizing larger models for query decoupling does result in performance enhancements, but the gains are marginal. Specifically, we conducted experiments by substituting Video-RAG's Query Decouple model with more advanced 72B LVLMs, and we observed only incremental improvements:\n \n| Method                                  | Short | Medium | Long | Overall | \n| :-------------------------------------- | :-: | :----: | :----: | :----: | \n| Video-RAG (Based)             | 77.0 | 69.3 | 63.8 | 70.0 |\n| replace Query Decouple model with LLaVA-Video-72B        | 76.9 | 69.7 | 63.9 | 70.1 | \n| replace Query Decouple model with Qwen2-VL-72B           | 77.1 | 69.8 | 63.8 | 70.1 |\n*All models adopt LLaVA-Video-7B with 64 frames for inference.*\n \n> **Q5**: For the raw video, what sampling method did the article use—uniform sampling, similarity-based sampling, or another method?\n\n**A5**: We employed uniform sampling for two primary reasons:\n- To ensure alignment of our frame sampling strategy with most LVLMs, thus demonstrating the plug-and-play capability of our method without reliance on key frames.\n- To facilitate a fair comparison. Empirical evidence supports that uniform sampling provides the most computationally balanced and generalized approach for understanding long videos.\n\n> **Q6**: Minor issues about the code repository link and lack of citations to clarify which specific methods.\n\n**A6**: Thanks for your reminder. We have updated the valid code repository link and added the relevant citations to clarify the specific methods in line 39.\n\n[1] R. Smith. An Overview of the Tesseract OCR Engine. Ninth International Conference on Document Analysis and Recognition. 2007.\n\n[2] M Minderer, et al. Simple Open-Vocabulary Object Detection with Vision Transformers. ECCV 2022.\n\n[3] Jialian Wu, et al. GRiT: A Generative Region-to-text Transformer for Object Understanding. ECCV 2024.\n\n[4] Tang X, et al. Adaptive keyframe sampling for long video understanding. CVPR 2025.\n\n[5] Bo Fang, et al. Threading Keyframe with Narratives: MLLMs as Strong Long Video Comprehenders."}]

Author rebuttal:

Meta-review: The paper discusses a training-free pipeline that augments LVLMs for long-video QA by extracting visually aligned auxiliary texts, filtering them via RAG, and concatenating them with frames and the query. It reports gains on several benchmarks. 

The reviewers found the method practical, clearly presented, and broadly effective across benchmarks. The ablations support robustness to weaker tools and show the auxiliary-text pathway is necessary. The rebuttal clarified spatial relations handling and temporal reasoning improvements.

Some reviews saw the novelty/significance as limited. Further concerns were that the approach uses uniform frame sampling and adds latency and that comparisons to other systems were not fully integrated in the main paper.

During the discussion some of the reviewers' concerns were addressed, e.g. through added analyses and multi-turn comparisons.

Overall, I recommend to **accept** this paper to NeurIPS. Addressing a current problem with a practical method, the paper is very suitable for presentation in my opinion.

---
Request for the final version: It looks like some concurrent work has appeared since the manuscript was created. For example, the below two works seem relevant from a first look, and there may be more. (I do not have any relation to any of the authors or their institutions, this is just the based on an online search.) I would like to ask the authors of this paper to update their related work and references to include more current references for the camera ready version, where appropriate.
- [VideoRAG: Retrieval-Augmented Generation over Video Corpus](https://arxiv.org/abs/2501.05874)
- [VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videos](https://arxiv.org/abs/2502.01549)
Decision: Accept (poster)