Title: Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension
Venue: NeurIPS 2025
Accepted: yes
Keywords: retrieval-augmented generation

video large language model

long video understanding
Abstract: Existing large video-language models (LVLMs) struggle to comprehend long videos correctly due to limited context. To address this problem, fine-tuning long-context LVLMs and employing GPT-based agents have emerged as promising solutions. However, fine-tuning LVLMs would require extensive high-quality data and substantial GPU resources, while GPT-based agents would rely on proprietary models (e.g., GPT-4o). In this paper, we propose Video Retrieval-Augmented Generation (Video-RAG), a training-free and cost-effective pipeline that employs visually-aligned auxiliary texts to help facilitate cross-modality alignment while providing additional information beyond the visual content. Specifically, we leverage open-source external tools to extract visually-aligned information from pure video data (e.g., audio, optical character, and object detection), and incorporate the extracted information into an existing LVLM as auxiliary texts, alongside video frames and queries, in a plug-and-play manner. Our Video-RAG offers several key advantages: (i) lightweight with low computing overhead due to single-turn retrieval; (ii) easy implementation and compatibility with any LVLM; and (iii) significant, consistent performance gains across long video understanding benchmarks, including Video-MME, MLVU, and LongVideoBench. Notably, our model demonstrates superior performance over proprietary models like Gemini-1.5-Pro and GPT-4o when utilized with a 72B model.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe article first decouples the question into Retrieval Requests, then employs various tools (e.g., OCR) to retrieve relevant audio and text within Video from the video. Finally, these two modalities are converted into text and input into the LVLM to obtain the Final Answer.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nBeyond commonly used video frmaes and subtitles, the article utilizes multiple tools to retrieve relevant audio clips and text within video frames, using them as auxiliary text to improve performance. The method achieves a good trade-off between performance and effectiveness. The experiments are sufficient and validate the effectiveness of the proposed approach.\n\nWeakness:\n1. Could Query Decouple become a performance bottleneck? If the Retrieval Request generated by the LVLM is inaccurate, then all subsequent steps will be inaccurate.\n2. The article uses multiple different external tools and sets various judgment conditions. This makes the method potentially sensitive to the choice of different tools (e.g., using different OCR or Grounding models) and hyperparameters. The article should discuss the impact of different external tools and hyperparameters on the final method.\n3. For long-video understanding, another class of methods calculates the relevance between frames and the question, then samples frames based on relevance to answer user queries, such as [1], [2]. The article should discuss and compare these in related work.\n\nMinor Issues:\n\n1. The code repository link is expired.\n\n2. In line 39, \"most of them process long...\", please add citations to clarify which specific methods \"them\" refers to.\n\n[1] Re-thinking Temporal Search for Long-Form Video Understanding\n\n[2] BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding\n\nQUESTIONS:\n1. Would using a larger model for Query Decouple improve pipeline performance?\n\n2. For the raw video, what sampling method did the article use—uniform sampling, similarity-based sampling, or another method?\n\nLIMITATIONS:\nAuthors discuss the limitations of the work in the   Conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a RAG-based Video QA technique, which extracts speech, OCR results, and visual objects related to a query, and feeds it to a LLM with video frames. The use of JSON-structured retrieval requests is logical, and the method outperforms existing VideoQA methods on three datasets: Video-MME, MLVU, and LongVideoBench.\n\nSTRENGTHS AND WEAKNESSES:\n## Strength \n1. The implementation is reasonable and the result is reliable.\n2. The presentation is very clear.\n3. The proposed method performs better than previous methods.\n4. The experiments are thorough.\n\n## Weakness\n\nThe technological contribution is considered insufficient for the NeurIPS standard. Using ASR for extracting video events combined with RAG is a standard technique in video analysis, as indicated by the following sources [a, b].\n\n- [a] MultiModal RAG for Advanced Video Processing with LlamaIndex & LanceDB, https://www.llamaindex.ai/blog/multimodal-rag-for-advanced-video-processing-with-llamaindex-lancedb-33be4804822e, 2024-02-17\n- [b] Do Jung et al., \"Speech Retrieval-Augmented Generation without Automatic Speech Recognition,\" arxiv, 2024.12\n\nTherefore, the novelty lies potentially in using OCR and object detection with video frames. However, each component is an existing method, and this reviewer recognizes less significance than other average NeurIPS-accepted studies.\n\nQUESTIONS:\nAre there any synergistic effects resulting from these components? Table 4 shows a detailed comparison of each condition, indicating additive rather than synergistic contributions. Data showing such synergistic effects would motivate reevaluation.\n\nLIMITATIONS:\nThe authors show mainly positive cases in its main part and supplementary (Fig. 4). An error analysis would increase the contribution of this paper to the community.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 1, "ORIGINALITY": 1, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Video-RAG, a novel training-free and visually-aligned retrieval-augmented pipeline for long video comprehension. Instead of directly increasing the number of sampled frames or relying on proprietary LLMs, the authors leverage open-source tools (e.g., Whisper, EasyOCR, APE) to extract textual information from videos (ASR, OCR, DET). These auxiliary texts are then filtered using RAG mechanisms to select query-relevant information, which is finally integrated with sampled video frames and the user query into a large video-language model (LVLM). The proposed method achieves consistent performance gains across multiple long-video benchmarks (Video-MME, MLVU, LongVideoBench) and even outperforms GPT-4o when integrated with a 72B LVLM.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n1. Video-RAG introduces a lightweight framework that requires no model training or fine-tuning. It can be directly applied to any LVLM with minimal computation overhead, making it highly practical and easy to adopt in real-world or low-resource scenarios.\n2. Unlike prior RAG approaches that treat videos as plain text, this work extracts and retrieves auxiliary texts (OCR, ASR, object detection) that are semantically aligned with the visual content. This alignment significantly improves cross-modality grounding and reduces hallucinations.\n3. The method demonstrates solid and consistent improvements across various long video QA benchmarks. When paired with LLaVA-Video 72B, Video-RAG achieves performance surpassing GPT-4o, with only ~2K additional tokens and marginal GPU overhead.\n\n# Weaknesses\n1. While Video-RAG avoids increasing the frame count directly, its overall performance remains fundamentally constrained by the initial uniform sampling strategy. If a query-relevant moment (e.g., a keyframe containing a crucial visual cue) is not sampled, the downstream RAG process cannot recover that information, limiting its robustness to extremely long videos.\n2. The paper introduces a scene-graph-based preprocessing step to convert raw object detection outputs into structured descriptions, including spatial relations. However, it lacks a clear explanation of how these relations are determined (e.g., what geometric rules define “on top of” or “next to”). Moreover, Figure 2 shows the \"Obj Relation\" as NULL, even though a visible spatial relation (\"books on table\") should likely be present, raising concerns about completeness and reliability.\n3. Although the method enhances static visual-textual alignment, it does not model temporal dynamics or sequence-level reasoning, which are crucial for many long video tasks (e.g., procedural understanding, action tracking, cause-effect). All auxiliary texts are retrieved independently per frame, without leveraging inter-frame continuity or temporal structure.\n\nQUESTIONS:\n1. How robust is Video-RAG to frame sampling errors, especially when key visual cues are missed due to uniform sampling in ultra-long videos?\n  2. Could the authors elaborate on how spatial relations (e.g., “on top of”) are computed in the scene graph from raw object coordinates? Why is “Obj Relation” missing in Figure 2 despite an apparent object interaction?\n  3. Has the method been evaluated or extended to handle temporally structured queries (e.g., \"after X happens, what does Y do\")? How would Video-RAG handle such temporal dependencies given its current frame-wise processing?\n\nLIMITATIONS:\nThe method depends on the existing models.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Video-RAG, a training-free pipeline to enhance long video understanding for large multimodal models. This approach firstly leverages open-source tools to extract visually-aligned auxiliary texts (e.g., OCR, ASR, object detection) from videos, integrates them via Retrieval-Augmented Generation (RAG) to align cross-modal information, and feeds them into VLMs alongside video frames and queries. Evaluations on benchmarks including Video-MME, MLVU, and LongVideoBench show that Video-RAG improves average performance by 3.2% across six open-source LVLMs, with a 72B model even outperforming proprietary models like GPT-4o. Key contributions include integrating RAG with open-source LVLMs, a plug-and-play design, and achieving proprietary-level performance with open-source models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The proposed method is training-free and plug-and-play, which also supports some proprietary models.\n2. The proposed method obtains performance improvements on several long video benchmarks, such as VideoMME, MLVU, and LongVideoBench.\n3. The proposed method is easy to follow and might perform on downstream applications.\n\nWeaknesses:\n1. The performance of Video-RAG is determined by the quality of the external tools (EasyOCR, Whisper, APE). Errors or inaccuracies in transcription, OCR, or object detection will inevitably propagate and potentially mislead the LVLM. The paper does not sufficiently analyze the impact of such errors.\n2. The inference latency is higher compared to end-to-end models. The process involves several tools to extract information from long videos which will slow down the inference speed for practical applications.\n3. This paper focuses on single-turn process which limits the performance for iterative reasoning or multi-turn conversations.\n4. This paper lacks the experimental / qualitative comparisons with previous VideoAgent-like methods, such as VideoAgent[1,2, 3], and other open-source well-established baselines, such as OmAgent for video understanding, which also adopts similar tools, such as ASR.\n\n\nReferences\\\n[1] Wang et al. VideoAgent: Long-form Video Understanding with Large Language Model as Agent. ECCV 2024.\\\n[2] Fan et al. VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding. ECCV 2024.\\\n[3] Jeong et al. VideoRAG: Retrieval-Augmented Generation over Video Corpus. ACL 2025.\n\nQUESTIONS:\n1. How does Video-RAG perform on videos with poor audio quality (e.g., noisy environments) where ASR accuracy drops significantly?\n2. What is the impact of using lower-quality open-source tools (e.g., a less accurate OCR than EasyOCR) on Video-RAG’s performance?\n3. Are there cases where auxiliary texts introduce noise, and how does the pipeline mitigate such negative effects?\n\nLIMITATIONS:\n1. The novelty of this paper is somewhat limited and this paper focuses on the engineering techniques for video understanding.\n2. The proposed method is training-free but this paper lacks further research on how to train video agents with tools.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}