Title: Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understanding
Venue: NeurIPS 2025
Accepted: yes
Keywords: agent

computer vision

long video understanding
Abstract: Long-form video understanding presents significant challenges due to extensive temporal-spatial complexity and the difficulty of question answering under such extended contexts. 
While Large Language Models (LLMs) have demonstrated considerable advancements in video analysis capabilities and long context handling, they continue to exhibit limitations when processing information-dense hour-long videos. 
To overcome such limitations, we propose the $\textbf{D}eep \ \textbf{V}ideo \ \textbf{D}iscovery \ (\textbf{DVD})$ agent to leverage an $\textit{agentic search}$ strategy over segmented video clips. Different from previous video agents manually designing a rigid workflow, our approach emphasizes the autonomous nature of agents.
By providing a set of search-centric tools on multi-granular video database,
our DVD agent leverages the advanced reasoning capability of LLM to plan on its current observation state, strategically selects tools to orchestrate adaptive workflow for different queries in light of the gathered information.
We perform comprehensive evaluation on multiple long video understanding benchmarks that demonstrates our advantage.
Our DVD agent achieves state-of-the-art performance on the challenging LVBench dataset, reaching an accuracy of $\textbf{74.2\%}$, which substantially surpasses all prior works, and further improves to $\textbf{76.0\%}$ with transcripts.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel approach for comprehending long-form videos, which is a complex and challenging task due to the vast temporal-spatial complexity and dense information. The proposed solution DVD, leverages an autonomous, agentic search strategy across segmented video clips. Unlike previous systems that rely on predefined workflows, DVD uses modular search tools at multiple granular levels to enhance video understanding.\n\nThis paper demonstrates the effectiveness of DVD through extensive evaluations on multiple long-form video benchmarks, achieving state-of-the-art results. Compared to existing methods, DVD shows significant performance improvements, particularly on the LVBench dataset, surpassing previous best approaches by a large margin.\n\nThe Key contributions of this paper include: (1) An agentic search mechanism that adapts based on the query and context. (2) The design and integration of search-centric tools for efficient video understanding. (3) State-of-the-art performance on long video benchmarks. (4) Insights into reasoning model behavior and tool usage, aiding the development of future video agents.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n(1) Long-form video understanding is crucial for a variety of applications, including media analysis, surveillance, and educational content. By addressing the problem of reasoning over large video datasets, this paper has the potential to impact numerous industries where video analysis is vital.\n\n(2) This paper offers experimental validation of its approach. It presents comprehensive evaluations on long-form video benchmarks, such as LVBench, LongVideoBench, and EgoSchema. DVD demonstrates substantial improvements over previous methods, showing its efficacy in real-world tasks.\n\nWeaknesses:\n\n(1) The DVD system significantly outperforms existing methods, but it may introduce higher computational costs due to the iterative reasoning process and multiple tool invocations. \n\n(2) While the system is flexible, providing case studies or examples of diverse queries would help illustrate the effectiveness and versatility of the integrated tools.\n\n(3) The video processing itself (i.e., segmentation into clips, captioning) relies on existing methods and may not push the boundaries of video processing techniques.\n\nQUESTIONS:\nQuestion 1: This paper primarily focuses on benchmark datasets, but it would be helpful to see more discussion on how well the approach generalizes to other types of video content or less structured environments. How well does the system adapt to videos with different types of content (e.g., documentaries vs. user-generated videos)? Please show some results.\n\nQuestion 2: Please provide detailed use cases or examples where these tools are iteratively employed to answer complex queries on long video. How does the system balance between global context and fine-grained details when addressing a single query?\n\nQuestion 3: The paper mentions that the DVD system introduces higher computational overhead due to the iterative reasoning process and multiple tool invocations. Please provide some results of computational overhead compared to other agent-based approaches.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Deep Video Discovery (DVD), an agentic framework designed for long-form video understanding. The key idea is to leverage a multi-granular video database and a set of search-centric tools (Global Browse, Clip Search, Frame Inspect) to iteratively retrieve and reason over video content using an LLM as the central orchestrator. The agent autonomously plans, refines queries, and synthesizes answers through an observe-reason-act loop.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths:\n\n1. Strong Empirical Results\n\nThe method achieves SOTA on multiple benchmarks, demonstrating clear improvements over existing approaches.\n\n2. Comprehensive Ablations\n\nThe paper includes detailed ablation studies on tool contributions and model choices, offering insights into system behavior.\n\n### Weaknesses:\n\n1. Limited Novelty in Video Retrieval & Search\n\nThe core idea of segmenting videos into clips and using an LLM for iterative search closely resembles prior works like VideoAgent. While DVD improves performance, the fundamental approach (clip-level retrieval + LLM reasoning) is not significantly differentiated. The tools (Global Browse, Clip Search, Frame Inspect) are intuitive extensions of existing retrieval strategies. A deeper discussion on how DVD uniquely advances agentic video search is needed.\n\n2. Inefficient Model Inference\n\nConstructing the multi-granular database requires processing the entire video and generating captions for each clip, leading to substantial computational costs, especially for long-form videos. In contrast, VideoAgent processes a much smaller subset of frames, making it more efficient.\n\n3. Narrow Model Evaluation\n\nThe experiments only test GPT-4.1 and o3 as the backbone models. Given the rapid progress in VLMs, the paper should evaluate proprietary models (Gemini 1.5/2.5) to assess generalization and open-source models (Qwen/Qwen VL, LLaVA, InternVL, DeepSeek R1) to ensure reproducibility and accessibility. Without broader model comparisons, it’s unclear whether DVD’s gains stem from the framework or simply from using a stronger base LLM (o3).\n\n4. Heavy Reliance on Zero-Shot LLM Capabilities\n\nThe system currently relies on the in-context learning and reasoning capabilities of an LLM. It does not fine-tune the LLM to tailor it for video understanding. This raises concerns about scalability (performance may degrade with weaker LLMs) and adaptability (task-specific fine-tuning could further improve results). A comparison with a fine-tuned variant would better demonstrate the framework’s robustness.\n\n5. Lack of Qualitative Analysis\n\nThe reviewer noticed that the authors include a few text-only results in the appendix. It is recommended that the authors include visualizations of success/failure cases in the main paper to make it easier to assess where DVD excels (e.g., complex temporal reasoning) and where it struggles (e.g., fine-grained spatial details, ambiguous queries).\n\nQUESTIONS:\nPlease refer to the weakness section.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

comments: SUMMARY:
The paper introduces DVD agents addressing a set of search-centric tools on video database. The core idea is in devising a modular search tool operating at different granularities, global, clip and frame-level. Leveraging this strategy the agent performs autonomous reasoning and iterative decision-making.

STRENGTHS AND WEAKNESSES:
Multigranular database construction is intuitive. Across all 4 datasets, their proposed method outperforms the base VLM. Ablation study shows the value of tool design.

QUESTIONS:
The difference between the proposed method and other video-llms are that the proposed method utilizes iterative reasoning. What's the trade off between the boost in accuracy vs additional cost? What's the value of step limit N? Which is the smallest N>1 value that shows better performance than the baseline? 
Why longer reasoning chain makes the results within the same model even worse?

LIMITATIONS:
The difference between the proposed method and other video-llms are that the proposed method utilizes iterative reasoning. What's the trade off between the boost in accuracy vs additional cost?
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01