Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understanding
Authors: Xiaoyi Zhang
Zhaoyang Jia
Zongyu Guo
Jiahao Li
Bin Li
Houqiang Li
Yan Lu
Venue: NeurIPS 2025 poster ()
Keywords: agent
computer vision
long video understanding
Primary area: applications
TL;DR: To resolve long video understanding task, we propose Deep Video Discovery agent to iterative reasoning and gather information from video content via an agentic search and tool use strategy..

Abstract: Long-form video understanding presents significant challenges due to extensive temporal-spatial complexity and the difficulty of question answering under such extended contexts. 
While Large Language Models (LLMs) have demonstrated considerable advancements in video analysis capabilities and long context handling, they continue to exhibit limitations when processing information-dense hour-long videos. 
To overcome such limitations, we propose the $\textbf{D}eep \ \textbf{V}ideo \ \textbf{D}iscovery \ (\textbf{DVD})$ agent to leverage an $\textit{agentic search}$ strategy over segmented video clips. Different from previous video agents manually designing a rigid workflow, our approach emphasizes the autonomous nature of agents.
By providing a set of search-centric tools on multi-granular video database,
our DVD agent leverages the advanced reasoning capability of LLM to plan on its current observation state, strategically selects tools to orchestrate adaptive workflow for different queries in light of the gathered information.
We perform comprehensive evaluation on multiple long video understanding benchmarks that demonstrates our advantage.
Our DVD agent achieves state-of-the-art performance on the challenging LVBench dataset, reaching an accuracy of $\textbf{74.2\%}$, which substantially surpasses all prior works, and further improves to $\textbf{76.0\%}$ with transcripts.

Deep Video Discovery: : Agentic Search with Tool Use for Long-form Video Understanding
Xiaoyi Zhang*! Zhaoyang Jia Zongyu Guo
Jiahao Li-
Bin Li Hougiang Li² Yan Lu
MMiCroooft Research Asia
University of Science and Technology of China
(xiaoyizhang, zongyuguo, jiahaoli, binli, hallu//microooft.comm (jzy_uste, @isq) edu...educcc
Abstract
Long-form video understanding presents significant challenges due extensive temporal-spatial complexity and the difficulty of question answering under such ex- tended contexts. While Large Language Models (LLLMs) have demonstrated consid- erable advancements in video analysis capabilities and long context handling, they continue exhibit limitations when processing information-dense hour-long vidcos. To overcome such limitations, we propose the Deep Video Discovery (DVD) agent leverage an agentic search strategy over segmented video clips. Unlike previous video agents that rely on predefined workflows applied uniformly across different queries, our approach emphasizes the autonomous and adaptive nature of agents. By providing a set of search-centre tools on multi multi-granular video database, our DVD agent leverages the advanced reasoning capability of I.L.M to plan on its current observation state, strategically selects tools to orchestrate adaptive workflow for different queries in light of the gathered information. We perform comprehensive evaluation multiple long video understanding benchmarks that demonstrates our advantage. Our DVD agent achieves sate-of-thhear performance on the challeng- ing LV Benchh dataset, reaching accuracy of 74.2 %, which substantially surpasses all prior works, and further improves to 76.0% % with transcripts. The code has been released wtthttps:///
Key Information Retrieval
LLM with Reasoning
Search Tooleet
67B
Video Database
$
Temporal Grounding
gets lollipop lasi?"
Thought:
Event Understanding
subtaskit) need to first...
gatharing Info
"yppSeerch""lollpppp"
Observation:
Reasoning
Relevant captions::
Thought:
Ourt
aubdask/21 1 need
Entity Recognition
MR Video
GlobalBrowse))
Tool use:
gatherlng Info
Gemini2 2-
OpenAi
8PT.46
"The zombie gets at last accorting to the observation that..."
Summarizatiinn
Overall
Figure 1: Left: Illustration of our Deep Video Discovery agent, which autonomously reasons on user query, iterative tools to obtain the final answer. Right: Performance comparison on LVBench.
Equual contribution.
This work was done during the internship at Microsoft Research Asia as an open-source project.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

1 Introduction
Long-form videos are ubiquitous in everyday life, spanning diverse domains such as movies, meeting recordings, sports games, and variety shows. Accurately comprehending and interpreting content within these extensive videos remains an intrinsically challenging task [8] 26 [1], demanding an ability to simultaneously integrate and reason about intricate spatiotemporal details across broad global contexts. Effective retrieval of relevant information from hour-long or even longer sequences not only necessitates attending to fine-grained local details but also simultaneously interpreting subtle semantic relations distributed throughout extended temporal intervals. Recent advancements in Large Language Models (LLMs) and Large Vision-language Models (VLMs) have notably improved capabilities in video understanding [7 28] and increased context length handling more than one
million tokens [17 23 33) However, even this extended context length remains insufficient for comprehending the information density typically found in long-form videos of hour-long duration. Empirical observations [17] also suggest decline in the model's effective costruction-oolowinng ability and reasoning clarity as the temporal dimension and information density increase.
g on reasoning capability of LLMs have facilitated advances in agentic systems capable of complex information gathering tasks, such as Deep Re- search [16] 10, [0] or Deep Search 2 These agentic approaches demonstrate how decomposing difficult tasks into modular sub-tasks enables iterative reasoning, information searching, and content synthesis. Inspired by these successes, we conceptualize the challenge of understanding extremely long videos as a multi-step information search problem, where the video serves as the exploration environment and is segmented into multiple shorter video clips serving as manageable units of information. We name our approach as Deep Video Discovery (Fig. [ left).
Concurrently, recent breakthroughs
incorporate searching processes in their designs, they manually design the search process with their human prior. For instance, both VideoTree [30] and VCA (34] employ tree-based search strategies that navigate from root nodes to leaf nodes. This approach alleviates the context length limitations of VLMs but is inefficient for fine-grained queries which might benefit more from direct retrieving among leaf nodes. Additionally, semantically relevant entities may not exhibit temporal proximity, potentially diminishing the efficiency of backdate mechanism in tree-based search methods. Most critically, these existing frameworks preseribe fixed workflows that are uniformly applied across all query types, failing to adapt to the diverse information needs and optimal search strategies required by different questions.
While existing video agent frameworks [34 19 30
In contrast to existing systems that typically rely on manually defined, rigid workflows, our approach is distinctly designed around an autonomous and adaptive agentic search paradigm for different queries. Instead of explicitly prescribing task workflows or search behaviors, we develop modular search tools that operate multiple granularities, including (1) Global Browse, (2) Clip Search, and (3) Frame Inspect. Global Browse enables global summarization and indexing of subjects and global contexts across the entire video. Clip Search implements efficient semantic retrieval of relevant events within segmented clips. Specifically, Frame Inspection empowers the agent to extract fine-graine -raiied details directly from pixel-level information in a specified temporal range.
Equipped with this cerrch-centric toolkit and multi guattl-r video database, our agent inherently capable of autonomous reasoning, dynamic strategy orchestration, and iterative decision-making to proactively discover and extract crucial evidence. By leveraging the sophisticated reasoning capabilities intrinsic in the latest LLM, our agent does not merely use these tools independently. but adaptively combines their complementary strengths into a chain of thoughts and tool uses, effectively addressing diverse temporal-spatial and complex questions for long video. In the end, Deep Video Discovery can autonomously reason, plan, and retrieve pertinent information through adaptive search strategies tailored to specific video understanding queries.
We conduct comprehensive evaluations on long video benchmarks, demonstrating the efficiency and strong performance of our agent. In particular, on the challenging LV Bench, we push forward the state-of-the-ar performance by large margin to 74.2% (as shown in Fig. [ right), further achieving 76.0% with auxiliary transcripts. We also set a series of ablation studies that show the effectiveness of our tool design. In addition, we analyze the behavior patterns of different reasoning models tool use sequences, providing future insight of developing agents for long video understanding tasks.

---

Stage 1: Multt-granular Video Database Construction
Stage 2: Agentic Search and Answer with Tool Use
g
Reasoning
Video Clips
User Query
Answer
Long Video
abservation thinking
plan & action
Search-centric Toolset
Large VLM
indexed frames
Global Browse
Clip Search
Frame Inspect
subjects
captions
Multi-grained Video Database
Multi-granular Video Database
Global Subjects
Clip Captions
Frames
Figure 2: Deep Video Discovery consists of two stages: 1) Multi granular Video Database Construc- tion. We extract video information from different levels to enable comprehensive understanding. efficient retrieval, and preservation original content. Agentic Search and Answer. The agent iteratively reasons on user query and leverage the tailored toolset to gather information to answer.
2
Related Work
Long Video Understanding. Long video understanding remains a formidable challenge due to the intricate demands of temporal and spatial reasoning over extended durations and the complexity of information retrieval [31, [26]. Recent efforts in VLM for long video understanding primarily
tackle challenges of limited input frame number by extending the context length of models [25] 6
or minimizing video redundancy to reduce visual token numbers [12] 14 28 Approaches such as AdaRETAKE (88 dynamically compress visual token by allocating adaptive compression ratios across time and model layers, thus significantly expanding the effective input frame number. However, token compression inherently introduces uncertainty regarding information loss, and models continue to face difficulties when answering complex queries under elongated context windows. In parallel, given the sparsity of key information about the given query, some works (2 34 30
32 propose
explore the video content by agentic system. But they usually manually guide the agent about the search workflow by their priors search (27). which cannot make full use of the reasoning capability of LLMss, resulting in suboptimal search efficiency and a lack of holistic, global comprehension of the long video content.
34 30 or only allow the agent at simplex frame granularity to
Agent and tool use. Recent advancements in large language models (LLMss, particularly their enhanced reasoning and planning capabilities, have significantly accelerated the development of autonomous agents [35 39 38 The ability to leverage external tools (23 22 21 further narrows the gap between general-purposs. LLMs and real-world applications, enabling LLMs acquire information, perform planning, and execute actions in complex environments. Our work extends this line of research long video understanding, contributing to the broader investigation of solving complex video understanding tasks by integrating the advanced reasoning capabilities of LLMs with sophisticated tool use. We introduce a suite of search-centric tools that allow LLMs autonomously gather information at varying levels of granularity. By dynamically composing these tools, the agent can construct multi-step tool-use chains to improve the ability to answer complex queries effectively.
3 Deep Video Discovery
Overview. To solve the long-form video understanding problem in an agentic search way, we first build multi- muatie structured database from the long-form video. The database then serves for search-centric tools that work at different granularities. Specifically, our Deep Video Discovery agent consists of three main components: the multi-grained video database D, search-centric toolset T, and the LLM M as the agent's 's orchestrator. Given the user query Q, the agent reasons iteratively to choose an action A, T (ANSWER) with parameters to gather information for the video database D or make decision to answer the query by referring to the accumulated information in this
3

---

process. In the following subsections, we sequentially introduce the multi multi-grained video database construction and Agentic Search and Answer with Tool Use.
3.1 Multi-granular Video Database Construction
Given an ultra long input video V. our goal is to transform it into a database that can provide efficient fast retrieval and also provide the original pixels of video for detailed information when necessary. Hence, we design in a multiple granularity style which can provide different levels of video information for corresponding search tools. Specifically, we first segment the video into clips as the basic information unit then make the database include global summarized information to cover the whole video, clip-based caption corpus and indexed frames from the clip. Fig. (left) provides an overview. We introduce these components sequentially.
Temporal segmentation. We start by uniformly partitioning the input video V into a temporal
sequence of non-overlapping short clips {vz } where the total segments N = [ : 1. [ set 5 seconds to provide an adequate balance between computing cost and semantic and action completeness. Then all the video clips are decoded into frames {ft ( under 2 frames per second for further process.
Empirically,
Multi-granular information extraction. Our multi- graluiigrauuaar video information is designed as three levels: global video level, clip level and frame level. Specifically, at the global level we summarize the video content into compact, subject-centric representation. At the clip level, we leverage textual captions to facilitate efficient information retrieval, while at the frame level we preserve original decoded frames indexed according to their corresponding clips, enabling precise reference and detailed analysis when required.
To derive the subject-centric global representation while minimizing redundancy in caption generation, we maintain an progressive structured subject registry S throughout the clip captioning process. Specifically, given a video clip 2 and decoded frame fi, we prompt large VLM to generate the captioning e, and evolve registry whenever new subjects appear. The process is denoted as Si,G VLM (Ji, where So is initialized empty, and at the conclusion of the captioning process, the final subject registry denoted by S SN- Each subject within the registry is represented by comprehensive set of attributes, including name, physical appearance, identity descriptors, associated actions, and corresponding temporal spans in the video. The obtained caption is subsequently embedded into dense semantic vector e; Rd using a language embedding model, facilitating fast retrieval downstream applications. Despite careful design choices, perceptual compression inherent in caption generation inevitably entails some information loss. To mitigate this when necessary, we explicitly retain the decoded frames f; alongside their corresponding textual captions and embeddings.
Outcome. The finalized database therefore encapsulates the decoded frames, captions and cor- responding embedding triples, thus forming a structured database D {S, (fr, C₁ e, c This offline construction procedure transforms a lengthy raw video into a structured set of textually searchable embeddings with associated clips, while simultaneously preserving the complete visual content at pixel resolution. The resulting database becomes the basis for adaptive tool usage, enabling global information browsing, efficient semantic retrieval at the video-clip scale, and comprehensive grounding of generated outputs back to their source frames.
3.2
Agentic Search and Answer with Tool Use
With the built multi- granular video database, we design set of search-centric tools that can enable global information understanding, efficient clip retrieval by semantic query, and details exploration on original video content. By equipping reasoning large language model with this toolset, we build our DVD that can address complex user query on long video though autonomous planning and strategical search tool combination, as shown in Fig. (right). We refer to this stage as Agentic Search and Answer with Tool Use (ASA). We introduce this stage through two subsections: Search-centric Tool Preparation and Agent Design.

---

Table 1:
Action space overview of our DVD. The first three actions are from our toolset and the final ANSWER action is de- signed as stop criterion.
Algorithm 1: Agentic Search and Answer.
Input :Initial query Q, max step N, LLM M, tool
set T, action space T [ANSWER]
Output I Answer Q
Initialize history Hn <- {Q.A)
Action
Parameter
for i 4- to N do
R: M.reason H.
GLOBAL
video database D user query Q
A;, P; M.call R, H;-1) where At
BROWSE
if A; ANSWER then
video database '
break end
CLIP SEARCH
agent synthesized query Q relurn top-k captions
O; - A(P))
Hi H,-1U A., AR,
video dalabase D
ift N then P. & Monnswer (H:)
FRAME INSPECT
agent synthesized query Q temporal range [fs,
end
end
ANSWER
the answer to user query
return ANSWER P.)
3.2.1 Search-centric Tool Preparation
Leveraging the established video database, we have developed suite of tools designed efficiently gather information from video data at varying levels of granularity. Specifically, we divide long videos into three distinct hierarchical levels and introduce corresponding specialized tools: (1) Global Browse, (2) Clip Search, and (3) Frame Inspect. Given the significant computational cost associated with processing lengthy videos using VLMs, our tool design carefully balances efficiency and performance. Central to our approach is an agentic search paradigm, wherein the agent decomposes the user query and strategically chains up tools with synthesized parameters, enabling iterative reasoning and information collection to resolve the task. Through the effective integration and coordinated use of these tools, the agent progressively enhances its understanding user intent and precisely locates relevant information within extensive video content. We introduce the three tools sequentially in the following paragraphs.
Tool: Global Browse. The Global Browse tool takes the video database and the original user query input, and returns global summaries capturing high-level contextual information. We construct two distinct types of global information: subject-centri, and event-centrie summaries. For subject-centric summarization, we pre-construct it when building the multi-granular video dataset as mentioned in Section 3.J since it is query-irrelevan
For event-centric summarization, we uniformly sample frames across the entire video and feed these sampled frames into the VLM. We instruct the VLM to describe noteworthy events explicitly related to the original user query. Upon invocation by the agent, the Global Browse tool efficiently retrieves and returns these global representations, providing the agent immediate access to high-level global context information.
Tool: Clip Search. Clip Search provides a mid-level granularity retrieval capability, enabling fast and efficient exploration of video content via caption embedding. Given a query Q synthesized based on the agent's 's current internal reasoning context, this module retrieves a ranked list of top-k relevant video clips along with their captions. Specifically, the tool computes the cosine similarity between the embedding of the provided query and the pre-computed embeddings of all video clip captions, returning the clips corresponding to the highest-ranked caption matches. Each retrieved observation contains both the corresponding caption and the time ranges of the associated video clip. To achieve an accurate and detailed understanding, the agent can iteratively invoke this tool, progressively refining temporal constraints reformulating its queries based newly acquired contextual knowledge. This iterative chain-of of-ieryf--uuer approach effectively guides the agent toward precise temporal segments relevant to the original high-level query.
Tool: Frame Inspect. Frame Inspect receives a temporal range [la, 4n] within the video and an sub-query freely defined by the agent as input, returning open format sutual-uuettionanswrriii (VQA) response. The agent can invoke this tool whenever explicit frame-level details such as subtle attributes, object counting, or fine- fine-grained spatial relationships, are required but not clearly depicted
5

---

in captions or global summaries. The open- geen-ended query format allows significant freedom for the agent leverage its reasoning capability, enabling highly adaptable visual inspection. Specifically, the Frame Inspect tool loads raw frames from the requested interval and prompts a VLM with these frames and agent- synthssytteeieeed query. To ensure computational efficiency, we limit processing to a maximum of 50 frames, uniformly sampling from frames exceeding this limit. The resulting response thus equips the agent with accurate, iisuallygrounded evidence essential for detailed reasoning tasks.
3.2.2 Agentic Design
To maximally leverage the reasoning and planning capacity intrinsic to modern LLMs, we intention- ally abstain from manually instructing explicit seaching workflow or tool utilization patterns. Instead, enable the agent to reason, plan, and take actions through streamlined iterative observe-reason- act loop, similar to ReAct[35]. For a given query, the agent reasons about its current observation state, strategically selects search tools, formulates appropriate parameters for actions, and dynamically refines its internal reasoning in light of the gathered evidence. Within ASA, the LLM acts as a sophisticated cognitive driver, taking actions each iteration based on cumulative knowledge and reasoned evidence, thereby reinforcing its pivotal role in adaptively navigating the discovery process.
Specifically, illustrated in Algorithm D given initial user query Q, a predefined action space A = TT U [ANSWER), and a maximum allowable step count N, our agent performs iter- ative reasoning to strategically navigate the available actions. The agent leverages an LLM M to reason upon the current dialogue history, plan its immediate action, interact with the toolset T= [GLOBALBROWSSE CLIPSEARCH, FRAMEINSPECT) and collect observations O, More con- cretely, at each step t. the agent maintains a historical context H,, reflects to generate a reasoning step R, selects action A; E TU [ANSWER] } accompanied by relevant parameters P.. and receives subsequent observation outcomes O; from the environment. These components, reasoning, action, and obtained outcomes, are successively appended to the interaction history Hi, enriching the context for subsequent iterations of inference. The iterative process terminates either when the agent explicitly selects the ANSWER action, or upon reaching the step limit N, at which prompts the agent directly generates a final answer prediction. The agent then outputs the final answer to the original user query,
By positioning the LLM's sophisticated reasoning at the core of this iterative loop, this approach endows the agent with an inherently autonomous, evidence wiidence-guided, and flexible action-takiin -aaiittakggggg mecha- nism. This autonomous and iterative paradigm fosters strategic and thereby enabling the agent to effectively leverage the available tools to iteratively decompose the original query into progressively refined sub-queries, updating and improving the query representation as it receives new observations. Through iterative reasoning and interaction cycles, guided by deeper and increasingly comprehensive observations collected from prior tool usage, the agent systematically enhances its understanding and interpretation of the task context, ultimately leading to more accurate and informed answers to the given question.
inquiry cycle,
4 Experiment
4.1 Benchmarks
We assess the long-form video understanding capabilities of Deep Video Discovery using several es- tablished long video benchmarks. Our primary evaluation benchmark, LVBench [26), includes 1,549 multiple-choice questions across 103 hour-long videos. It stands as one of the most comprehensive and challenging benchmarks for extreme long-form video understanding. Long VedeeBenchh (31) features 6,678 questions from 3,763 videos, ranging in duration from a few seconds to an hour. We emphasize the longest subset with durations in (900s, 3600s] (denoted as the Long subset), compris- ing 564 questions from 188 videos. Video MME (8] is segmented by video duration; we concentrate on the Long subset without subtitles to isolate long-video comprehension, covering 300 videos of 30 to 60 minutes with 900 questions. Finally, EgoSchema [15] serves as a diagnostic benchmark for long-video understanding, where we evaluate on its validation split of 500 videos with 500 questions.
4.2 Implementation Details
Baselines. We compare Deep Video Discovery with a range of long-video understanding systems, including both VLM-based (24
18
9
36
29
37
4 [3 28 and agent-based approaches [30 7

---

Table Comparison on LVBench under different question categories. The metric shown accuracy
(%).
Methods
ER
EU
KIR
TG
Rea
Sum
Overall
Commercial VL.Ms
Gemini-1.5-Pry
24
32.1
30.9
39.3
31.8
27.0
32.8
33.1
Gemini-2.0-Finn
24
47.4
48.5
56.8
39.3
44.4
41.4
48.6
GI-M-V-Pllus
47.8
54.1
42.7
46.5
37.9
48.7
GPT-4o
48.9
49.5
48.1
40.9
50.3
50.0
48.9
OpenAl o3
18
57.6
56.4
62.9
46.8
50.8
67.2
57.1
Open-Source VLMs
Intern
29
43.8
42.0
42.1
36.8
51.0
37.9
43.6
VideoLLaMA3-7B
45.8
42.4
47.8
35.9
45.8
36.2
45.3
Qwen2.5-VI-72B
47.7
VideoChat-Flash
51.1
46.0
49.0
38.9
48.5
34.5
48.2
A 28
53.0
50.7
62.2
45.5
54.7
37.9
53.3
Video Agents and Others
VideoTree 30
30.3
25.1
26.5
27.7
31.9
25.5
28.8
VideoAgent 22
28.0
30.3
28.0
29.3
28.0
36.4
29.3
VCA 34
43.7
40.7
37.8
38.0
46.2
27.3
41.3
MR. Video
19
59.8
57.4
71.4
58.8
57.7
50.0
60.8
Deep Video Discovery (Ours)
73.4
73.3
80.4
72.3
70.7
74.1
74,2
+ Auxiliary transcripts
75.5
77.1
79.0
72.7
68.7
84.5
76.0
19]. Most baseline results taken from official leaderboards published reports, except for the recently released OpenAl o3 [18), which has not yet been evaluated on these benchmarks. Following uniformly sample 256 frames per video to evaluate OpenAl o3.
34
19
Deep Video Discovery flexibly integrates different models depending on the needs of each component. For the VLM in video database construction, we use GPT-4.1 [17] to produce high-quality captions on LVBench, and GPT-4.1 1-mii for other benchmarks reduce cost. During agentic search and answering, we employ OpenAl o3 as LLM M for its strong reasoning ability, including in the Frame Inspect module for fine-grained VQA. All frames are resized to 720p maintain visual details. In Clip Search, we set 16 as the default value of top-k while leaving the flexibility for L.L. to change it. Maximum reasoning step is set to N 15. To explore the upper bound of understanding ability, we additionally evaluate LVBench using auxiliary transcripts. Audio is transcribed with WhisperX(5) and transcripts are used to guide video segmentation and enrich captions. This audio-visual fusion enhances understanding of long, complex content, leading stronger results.
API Content filtering. We use LLM API via Azure OpenAI Service. We observe that the safety content filtering mechanism of the service misjudges a small part of data from the benchmark as offensive and block the request, which leads to the reduced performance both OpenAl baseline and our DVD agent. We provide more details and mitigation strategies in Appendix A.3.2
4.3 Main Results
Table 2
the comparison results on LVBench. DVD significantly outperforms all baselines, surpassing the previous SOTA MR. Video by 13.4%. Compared to the prior leading video agent VCA. our method achieves a remarkable 32.9% gain. Against our base VLM, OpenAI o3, our full system delivers substantial 17.1% 1% gain, highlighting the importance of agentic reasoning. Incorporating transcript information provides an additional 1.8% boost. These results highlight the effectiveness of search-centric agentic reasoning framework in handling ultra-long video understanding tasks.
presents prbsenpresents
Table 3 provides a comprehensive evaluation across several long-video benchmarks. On Long VideoBencc,, DVD outperforms the previous SOTA by 4.1% overall and 7.0% on the longest-

---

Table 3: Comparison on long video benchmarks.
LVBench
Long VongeBeeen (Val)
Video MME Long (w/o sub)
EgoSchema Val
Methods
Overall
Overall
Long
Commercial VLMs
Gemini-1.5-Pro 24
33.1
64.0
58.6
67.4
Gemini-2.0-Flash 24
48.3
45.7
63.0
71.2
GPT-4o |
48.9
66.7
60.9
65.3
70.4
OpenAI o3
18
57.1
67.5
60.6
64.7
63.2
Open-Source VLMs
mPLUG-OW13 36
43.5
59.8
50.1
InternVL..5-11 29
43.6
63.6
62.6
Qwen2.5-VL-777 4
47.7
60.7
63.9
Ada 28
53.3
67.0
65.0
Video Agents and Others
VideoTrree 300
28.8
67.0
VidcoAgent
29.3
63.2
VCA 3
41.3
73.6
MR. Video
60.8
61.6
61.8
73.0
Deep Video Discovery (Ours)
74.2
71.6
68.6
67.3
76.6
Table 4: Ablation used models. Mäntabase for cap- tioning database construction, M Mreasoning for reason- ing in ASA, Miool Frame Inspect.
Table 5: Ablation on the search-centeric tools T. Note that the anchor uses 4.1 1i-min for Mástobuses and both Mreessoning Mtool-
Adopted models
LVBench transcripts
Search-centeric Tools
IVBench
Maatabase
Mreasoning
Moool
Global Clip Browse
Frame
Search
Inspect
w/ transcripts
4.1 4.1-mini
72.3
4.1
04-mini
o3
70.2
69.0
4.1
4o
o3
62.3
59.6
4.1-mini
o3
o3
71.9
63.5
4.1
76.0
V
71.9
duration subset. On the Video MME Long subset, it beats the best open-source VLM, AdaRETAKE, by 2.3%, and MR. Video by 5.5%, approaching the performance of Geminni-15-PPro On EgoSchema, our method exceeds previous best by 3.0%. Notably, exceeds reported human-level accuracy of 766 this benchmark. Across all datasets, system consistently outperforms the base VLM OpenAl o3, confirming the effectiveness and generalizability of our agentic reasoning framework.
4.4 Ablation Study
We evaluate the impact of different model choices across system components. By default, GPT-4.1 is used for captioning and subject extraction during Multi-granular Video Database Construction and OpenAI o3 serves as the reasoming model in the Agentic search and Answer with tool use process while Frame Inspect tool also leverages OpenAl o3 to query the fine-grained details on the frame pixels. We denote the three models Maatabases Mreasoning and Mtool in Table Replacing GPT-4.1 with GPT-4.1 1-mini for database construction or Frame Inspect tool results in moderate drops of 4.1% and 3.7%, respectively, indicating relatively minor impact. For reasoning model in agentic search, switching to OpenAl o4-mini [18) leads to 5.8% drop, while GPT-4o causes a substantial 13.7% decline. It highlights the reasoning model as the most critical component in our agentic system because our system is designed surrounding and to make full use of the reasoning capability of I.M.. The lack of reasoning ability leads the collapse of agent behavior, analyzed further the subsequent subsection.

---

Table 6: Comparison of DVD agent with different LLMs as reasoning models on LVBench. denotes
Table 7: Comparison DVD and its variation on different max step number.
Qwen3-32B- -2nnnkinn model.
Max Step Limit
12
15
Category
Model
Accuracy (%)
Ours
DVD w/ closed-sourced
OpenAl o3
76.0
Avg. Actual Steps
6.7
7.2
7.3
GPT-4o
62.3
Accuracy (%)
72.3
73.8
74.2
DeepSeek-R1
68.5
Ours w/ VideoAgent
27
workflow
DVD w/ open-soureed
DeepSeek-V2
57.5
Avg. Actual Steps
5.0
8.3
11.1
Qwen3-32B*
57.3
Accuracy (%)
48.4
66.3
70.2
Behavior Type (Ratio)
GBO Global Browse Only
SA Sipple
CST
Stegs Score
herasive Bearch
Frame Inspect tma
Clpp
GBO (2.8%)
GBO (2.9%)
(1.6%)
CST
3.0/95
CST (16.0%)
3.0/31
7.8/36
7.2/45
GBO (5.5%)
CST (27.4%%
IS (0.7%)
3 78
6.3/36
11.6/50
FIT (3.4%)
IS (5.8%)
FIT (10.4%)
SA (53.3%)
10.7/63
5.2/04
IS (8.1% 8.0/71
SA (71.9%)
SA (91.4%)
4.6/56
OpenAl o3
OpenAl o4-mini
GPT-4o Steps Score 4.8/62.3
Steps/ Score - 7.8/78.0
Steps / Score 70.2
Figure 3: Analysis of the behavior of Deep Video Discovery using different reasoning models. We categorize tool-calling behavior into five types. For each type, we report its proportion (Ratio, sector angels), average reasoning steps (Steps, sector radius) and score (Score, dashed lines). A clear correlation emerges among behavior patterns, reasoning depth, and score (see Section for details).
We next examine how the choice of reasoning model affects performance when pairing DVD with various open-source L.LMs on LVBench. reported in Table 6 combining DVD with the open- source DeepSeek-R1 reasoning model yields 68.5% accuracy, surpassing all prior methods. Notably, even when restricted to 32B model Qwen3-32B, DVD attains 57.3% accuracy, outperforming existing VLM approaches, including GPT-4o and OpenAl o3.
Finally we assess the contribution of each tool in the agentic search and answer phase (Table (5) Removing Global Browse which is responsible for global summarization and long-range event linking leads to a 2.9% drop. Disabling Frame Inspect with the fine-gra ane-grained VQA results in 8.4% decline, highlighting its role in fine- iiee-graied understanding, Removing Clip Search causes the largest drop of 12.3%, as breaks the searching ability for iteratively refine reasoning. These results underscore the importance of tool integration in our search-centric framework.
4.5 Analysis on Efficiency
To understand the efficiency of our designed agent, we compare DVD's performance with different maximum step limits on LVBench without transcripts. Since the core strength of DVD lies in its adaptive workflow orchestrated by the reasoning model, we also evaluate variant that replaces this adaptive workflow with the hand-crafted workflow from VideoAgent [27] while keeping all other settings identical. Specifically, the VideoAgent workflow follows a fixed sequence: first predicting answers and confidence scores, then identifying missing information through self-reflection, searching video clips for the missing information, merging found information into current observations, and finally re- re-predicting answers and confidence scores. As shown in Table 7 the hand-crafter approach

---

results in a significant 7.5% absolute accuracy drop while requiring more steps. Even when scaled 11.1 average steps (52% more than our DVD), the hand-crafted workflow achieves only 70.2% accuracy, still 4.0% below our adaptive approach. This comparison highlights the superior search efficiency of our adaptive workflow design.
4.6 Analysis Agentic Reasoning Behavior
The reasoning model is the most critical component in DVD. During the observe-reason--e loop, the agent autonomously integrates current context and flexibly decides the next tool to invoke. To better understand this, we analyze the tool-calling behavior during the agentic search and answer phase and category it into five types for analysis (see Fig. At the same time, provide more detailed qualitative case study in Appendix B.2
Global Browse Only. The agent confidently answers after single Global Browse call. This pattern, though rare, correlates with very high accuracy, suggesting the initial global context was sufficient.
Simple Action. The agent follows a direct search-inspect cnsspect-answerr path, typically involving one or two Clip Search and Frame Inspect calls. This the most common and effective strategy, employed in over half of the queries and achieving high accuracy.
Iterative Search. The agent alternates between Clip Search and Frame Inspect multiple times. This indicates an initial difficulty in locating sufficient information, requiring a more exploratory approach. These trajectories are longer (e.g.. 8.0 vs. 5.2 steps for OpenAI o3) and yield slightly lower accuracy than Simple Action.
Frame Inspect Trap. The agent becomes stuck in loop of fine-graine analysis, invoking Frame Inspect more than three consecutive times without reaching a conclusion. This behavior leads to prolonged reasoning chains and significantly lower accuracy.
Clip Search Trap. The agent repeatedly calls Clip Search more than three times consecutively, often when key information is absent from the video database. The agent fails to adapt, resulting in unproductive loops. This pattern is primary failure mode for the OpenAl o3 model.
This analysis yields two key insights into designing effective autonomous video agents:
Insight 1: The dual nature of reasoning length. For single model, longer reasoning trajectories often signal uncertainty and correlate with lower accuracy (e.g., Simple Action vs. Iterative Search). However, when comparing across models, superior performance is often achieved by the model capable of conducting more thorough, and thus longer, reasoning processes.
Insight 2: Overconfidence leads to behavioral collapse. The GPT-4o model significantly un- derperforms, collapsing into the Simple Action pattern for 91.4% of queries. With an average of only 4.6 reasoning steps, it frequently concludes prematurely, rarely exploring alternative strategies. This "overconfidence" and lack of flexibility appear to be the primary cause of its poor performance, highlighting the need for agents that can balance exploitation with exploration.
5 Conclusion
We introduces the proposed Deep Video Discovery agent for long-form video understanding, utilizing multi-granular search tools constructed database for iterative search and reasoning extensive video content. Our approach outperforms prior methods by adaptively integrating global browsing. clip search, and frame inspection, as demonstrated by state-of-the the-art results on multiple benchmarks. Ablation studies confirm the effectiveness of our tool design, while analyses of reasoning model behavior provide insight into model reasoning patterns. Overall, our framework offers scalable and flexible solution for comprehensive analysis of long videos.
Limitations. While our agent significantly improves long video understanding, the iterative reasoning introduces higher computational overhead. In future work, we will explore more effective database construction and searching to reduce reasoning difficulty and thereby lower computational costs.
10

---

References
[1] Achiam, Adler, S. Agarwal, L. Ahmad, Akkaya, F. L. Aleman, D. Almeida,
J. Altenschmidt, S. Altman, S. Anadkat, al. GPT-4 technical report. arXiv preprint
arXiv:2303.08874, 2023.
[2] J. AI. DeepSearch - Jina AI.
2025.
[3] X. AL. Grok 3 Beta The Age of Reasoning Agents. https: / www.aii//wwww//gg. grooww/ggoo-- 2025.
[4] S. Bai, K. Chen, X. Liu, J. Wang. W. Ge, S. Song, K. Dang. P. Wang. S. Wang, J. Tang, et al. Qwen2.5-VL technical report. arXiv preprint arXiv:2502./392. 33933, 2025.
[5] M. Bain, Huh, T. Han, and A. Zisserman. Whisperx: Ticuraccurate speech transcription of long-form audio. INTERSPEECH 2023, 2023.
[6] Y. Chen, F. Xue, D. Li, Q. Hu, L. Zhu, X. Li, Y. Fang, H. Tang, S. Yang, Z. Liu, al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408. 10188, 2024.
[7] Y. Fan, X. Ma, R. Wu, Du, Li, Z. Gao, and Q. Li. VideoAgent: A memory-augmented multimodal agent for video understanding. In ECCV, 2024.
[8] C. Fu, Y. Dai, Y. Luo, L. Li, S. Ren, R. Zhang, Z. Wang, C. Zhou, Y. Shen, M. Zhang, et al. Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis. arXiv preprini arXiv: 1075, 2024.
[9] т. GLM, A. Zeng, B. Xu, B. Wang, C. Zhang, D. Yin, D. Zhang, D. Rojas, G. Feng, H. Zhao, et al. ChatGLM: A family of large language models from GLM-130B to GLM-4 all tools. arXiv preprint arXiv:2406. 12793, 2024.
[10] Google. Gemini Deep Research your personal research assistant. https://gemini.gooll.... wwerview/deep-ree desearr--eeeeeeeheeeeeeeeec 2025.
[11] D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseck-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501. /2948, 2025.
[12) Y. Han, Q Guo, L. Pan, L. Liu, Y. Guan, and M. Yang. Dynfocus: Dynamic cooperative network empowers ilms with video understanding. arXiv preprint (tt.:2411.123555 2024.
[13] X. Li, Y. Wang, J. Yu, X. Zeng. Y. Zhu, H. Huang. J. Gao, K. Li, Y. He, C. Wang, Y. Qiao, Y. Wang, and L. Wang. VideoChat-Flase Hierarchical compression for long-context video modeling. arXiv preprint arXiv:2501.00574 2024.
[14] X. Liu, Y. Shu, Z. Liu, A. Li, Y. Tian, and B. Zhao. Video-xl-pro: compression for extremely long video understanding. arXiv preprint MXXi::2503.1847 2025.
Reconstructive token
[15] K. Mangalam, R. Akshulakov, and J. Malik. EgoSchema: A diagnostic benchmark for very long-form video language understanding. In NeurlPS, 2023.
[16) OpenAI.
Introducing deep research.
https://openai.com/nnndd
-
2025.
[17] OpenAl. Introducing GPT-4.1 in the API. Accessed: 2025-04-14
2025.
[18] OpenAI. introoducing-oo--ad----------
Introducing OpenAl o3 and 04-mini. https://openat.comm/ndee 2025. Accessed: 2025-05-15.
[19] Z. Pang and Y.-X. Wang. Mr. video:" " mapreduce" is the principle for long video understanding. arXiv preprint arXiv:2504. 2025.
[20] Perplexity. Introducing Perplexity Deep Research.
research 2025.

---

[21] Y. Qin, S. Hu, Y. Lin, W. Chen, N. Ding, G. Cui. Z. Zeng. X. Zhou, Y. Huang, C. Xiao, et al. Tool learning with foundation models. ACM Computing Surveys, 57(():1-40, 2024.
[22] C. Qu, S. Dai, X. Wei, H. Cai, S. Wang, D. Yin, J. Xu, and J.-R. Wen. Tool learning with large language models: A survey. Frontiers of Computer Science, 19(8):198343, 2025.
[23] T.
Schick, J. Dwivedi- Yu, R. Dessi, R. Raileanu, M. Lomeli, E. Hambro, Zettlemoyer N. Cancedda, and T. Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36:68539-68551, 2023.
[24] G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:23/2.11805, 2023.
[25] G. Team, P. Georgiev, I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055300 2024.
[26] W. Wang, Z. He, W. Hong. Y. Cheng, X. Zhang, J. Qi, X. Gu, S. Huang, B. Xu, Y. Dong, et al. I.Vennch: An extreme long video understanding benchmark. arXiv preprint arXiv:2406.08033. 2024.
[27] X. Wang, Y. Zhang, O. Zohar, and S. Yeung-Levy, VideoAgent: Long-form video understanding with large language model agent. In ECCV, 2024.
[28] X. Wang, Q. Si, J. Wu, S. Zhu, L. Cao, and L. Nic. Adaretake: Adaptive redundancy reduction to perceive longer for video-language understanding. arXiv preprint urXiv:2503.12559, 2025.
[29] Y. Wang, X. Li, Z. Yan, He, J. Yu, X. Zeng, C. Wang. C. Ma, H. Huang, Gao, al. Intern Video2.5: Empowering video MLLMs with long and rich context modeling. arXiv preprint arXiv:2501. /2386, 2025.
[30] Z. Wang, S. Yu, E. Stengel- J. Yoon, F. Cheng, G. Bertasius, and M. Bansal. VideoTree: Adaptive tree-based video representation for LLM reasoning on long videos. arXiv preprins arXiv:2405. 19209 2024.
[31] H. Wu, D. Li, B. Chen, and J. Li. Longvideobench: A benchmark for long-context interleaved vidco-language understanding. NeurlPS, 2024.
[32] Y. Yan, S. Jiang, T. Cao, Y. Yang, Q. Yang, Y. Shu, Y. Yang, and L. Qiu. Empowering agentic video analytics systems with video language models. arXiv preprint arXiv:2505.00254. 2025.
[33] A. Yang, B. Yu, C. Li, D. Liu, F. Huang, H. Huang. J. Jiang, J. Tu, J. Zhang, J. Zhou, et al. Qwen2. 5-1m technical report. arXiv preprint arXiv:2501.1538. 2025.
[34] Z. Yang, D. Chen, X. Yu, M. Shen, and C. Gan. VCA: Video curious agent for long video understanding. arXiv preprini urXiv:24/2.1047 2024.
[35] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. ReAct: Synergizing reasoning and acting in language models. In ICLR, 2023.
[36] Ye, H. Xu, H. Liu, A. Hu, M. Yan, Q. Qian, Zhang, F. Huang, and J. Zhou. mPLUG-OWL.3: Towards long image-sequence understanding in multi-modal large language models. In ICLR, 2024.
[37] B. Zhang, K. Li, Z. Cheng, Z. Hu, Y. Yuan, G. Chen, S. Leng, Y. Jiang, H. Zhang, X. Li, P. Jin, W. Zhang, F. Wang, L. Bing, and D. Zhao. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13106, 2025.
[38] K. Zhang, J. Li, G. Li, X. Shi, and Z. Jin. Codeagent: Enhancing code generation with tool- integrated agent systems for real-world repo-level coding challenges. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13643-13658, 2024.
[39] 7. Zhang, X. Zhang, W. Xie, and Y. Lu. Responsible task automation: Empowering large language models responsible task automators. arXiv preprint urXiv:2306.01242, 2023.
12

---

1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
13

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We believe have clearly described our algorithm in Section and provide corresponding prompts and artifacts in our supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model limited in some way (e.g., registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [NA]
14

---

Justification:
We are committed to open-source our code as soon as possible.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
cctipp.cc/pbliic/giiicoo gwwp:// Coddeeubmisiiiiiicc. wwwpiimiiisiooooooooo.ooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See in Section 4.2 and our supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
See in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
15

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: wwt.rripsccppuublii http:/is....puublccc/ttccccualaa
Answer:
[Yes]
Justification:
We have reviewed the NeurIPS Code of Ethies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See in supplemental material.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
16

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: All the input and output of our system is guarded by Azure Al Content Safety service.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided relcasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: See in Section 4
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwpsspipetattteeeeee thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
17

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact core methodology, scientific rigorousness, originality of the research, declaration is not required.
18

---

Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not non Please refer to our LLM policy fattps://neune cetpouriip.c.oooofeeeeeeeeeeee /cc/oonferencce/22222 for what should or should not be described.
involve LLMs as any important, original, non non-standard components.
19

---

A Evaluation Details
A.1 Prompts
We describe the prompts used by different models within the proposed DVD, including: (1) Captioning and subject extraction model Maaaaass used in
Video Database Construction (Tab. 9)
(2) Reasoning model Meesoning used Agentic Search and Answer (Tab. [Tbb (3) Fine-grained detail querying model Mocl used in the Frame Inspect tool (Tab. [T); (4) event-centric global information in the Glohal Browse tool(Tab. 12
The ISON schema for tool use of OpenAl model are also provided
in Tab. 13 and Tab. 14
A.2 Cost
We measure the average API cost of our agent when calling OpenAl o3 on LVBench. On average, DVD incurs cost of $2.05 with 20803 tokens per question.
A.3 Azure OpenAl Service
We utilize the LLM API through Azure OpenAl Service evaluate both our proposed DVD agent and the OpenAl o3 model [88. However, we note two key limitations in the service that may affect performance: (1) restriction on the maximum number of input images, and (2) safety content
filtering mechanism. This section discusses their impact and our mitigation strategies.
A.3.1 Maximum Image Count Limitation
Azure OpenAI Service imposes limit of 50 images per request, which is significantly lower than the token limit allowed by Large VLMs we used. This restriction poses notable challenges in tasks involving long-form video understanding, as the model's 's comprehension capability strongly depends on the number of input frames provided. To overcome this limitation and deliver richer visual context within the imposed constraints, we adopt strategy of combining multiple frames into single composite image:
For DVD, we horizontally splice up to 5 frames into a single image. This allows us to input up to 250 frames (50 images 5 frames). Note that we only use the 250 frames setting in event-centric global information gathering in the Global Browse tool while maximum 50 frames are used in the Frame Inspect tool.
For OpenAI o3, we use 2x4 format format--plicing up to 4 frames horizontally and stacking such groups vertically per image. This enables our test condition of 256 frames by using 32 spliced images. We find that this configuration yields better performance than the Ix5 format used in DVD.
A.3.2 API Content Filtering
Azure OpenAl's safety filtering mechanism occasionally misclassifies benign inputs as offensive, blocking some requests. This affects the performance of both OpenAl o3 and DVD agent. For such filtered questions, we adopt different mitigation strategies to reduce its influence.
For OpenAl o3, when query is blocked, we randomly select an answer from the remaining candidates. Across benchmarks, approximately 10% to 25% of queries are filtered, as shown Tab. To better assess true performance, we also report scores on the subset of unfiltered responses, where performance improves significantly.
For DVD, when VLM returns none due to the content filtering, we do not implement additional error handling for the agent. Instead, we the agent make its own decisions to handle error. It may invoke alternative tools or attempt to answer based on available information. If captioning query is blocked, we leave the corresponding entry blank in the database. While this affects database completeness, the agent remains operational due to its adaptive behavior. Tab, 8
20

---

Table 8: Statistics for content filtering in Azure OpenAI API.
LVBench Overall
LongVideoBench (Val)
Video MME
EgoSchema Val
Methods
Overall
Long
Long (w/o sub)
OpenAI 03
57.1
66.7
59.6
64.7
63.2
Unfiltered Ratio
83%
90%
85%
83%
75%
Unfiltered Subset Score
63.3
71.5
72.6
73.2
77.5
Deep Video Discovery (Ours)
71.9
70.5
68.4
66.8
76.6
Unfiltered Ratio
24%
61%
40%
24%
54%
Unfiltered Subset Score
70.5
71.6
70.5
69.7
77.0
B
More Results
B.1 Statistical Significance
To assess the statistical significance of DVD, we evaluate it on LVBench (using auxiliary transcrip- tions) three times. It achieves an average score of 74.0 with variance of 0.15, demonstrating the consistency and robustness the proposed method.
B.2 Case Study
We provide illustrative examples of the tool-calling process for different agent behaviors described in Section 4.5 of the paper. All examples are from LVBench [26]. Each example highlights how the agent selects actions and invokes tools to gather information for answering the question, and how each tool call retrieves relevant data from database. In the tool responses, we highlight useful and correct information in orange and useless or incorrect information in purple.
Global Browse Only is shown in Fig. 4 When asked about the type of video, the agent directly invokes Global Browse to retrieve global context. The subject-level browsing reveals many technical subjects, while the event-level browsing summarizes the key themes and categorizes the video as technology documentary. Based on this information, the agent correctly classifies the video into technology documentary.
Simple Action is shown in Fig. 5 In this case, Global Browse incorrectly indicates that the mirror fragments thrown away. To verify, the agent uses Clip Search locate the time range of the mirror breaking event and then calls Frame Inspect for finer-grained analysis. It discovers that the fragments actually fly into the sky and thus selects the correct answer.
Iterative Search is shown in Fig. This question involves multiple subjects (boy, dog, cat, bird, and man), making it more complex. Global Browse initially suggests that the man is chasing the dog. However, the first round of Clip Search and Frame Inspect indicates that he is chasing a bird. To resolve the ambiguity, the agent issues more Clip Search queries for events involving "man chasing dog" and "man chasing bird". The search yields relevant and useful results only for the later, suggesting that the man is actually chasing the bird. To confirm the subject's 's identity, the agent checks for the presence of a mustache. Convinced that the mustached man is chasing the bird, the agent selects the correct option. Notably, when Frame Inspect call fails find relevant results in step 6, the agent can automatically ask another question to obtain an useful response.
Frame Inspect Trap is shown in Fig. 7 This question targets event at specific timestamp. Around this time, the agent asks three consecutive questions. While Frame Inspect sometimes captures accurate information, it occasionally fails to interpret frames correctly, providing misleading responses. As a result, the agent cannot discern which information is reliable and ultimately answers incorrectly.
Clip Search Trap shown Fig. g This example relies on identifying the correct time range using key terms like "protagonist" and "utility room." However, the database contains few directly relevant captions or subjects. Despite the agent's attempts to rephrase and refine its Clip Search queries, the tool fails to return relevant results, leading to incorrect answer.
21

---

// system proupt
You helpful assistant.
// prompt
There are consecutive frames from video. BENCHMARK, CUSTOM, with given corresponding tranacript and hiatorical subject information then output following J80X rormat with filled content:
COWCCCOSSLSSEEIIIIE PLACEHOLDER Please understand the video clip
Transcript of the current video clip:
[TRANSCRIPTS]
Historical subject information:
[SUBJECTS]
Output template:
"clip_atart_t:an :ime": CLIP CLIP_STARS START_TIME. "elip_end_tine side; CLIP END, TIME,
jev_eubjjet_rrgisttr" { add subjects any in the historical subject information else leave it ampty
<nee_ pus_aabject__id>>
"name": <nek, suex_subject.nane if the name is unknown please use 'unknown'>> "eppearance"; list subject appearance description phase, including hair, clothes>, "identity"; 1iat of aubject identity description shown in current video clip>, "firet_eeenn; <timestamp>
delipddeccrppiinn:: {
"subjecte_present" [<eubject_id_ lists], "captione"; <a smooth and very detailed nerration about the whole video clip>
Table Prompts for captioning and subject extraction model Mastame in video database construction.
C Broader Impacts
C.1 Positive Impacts
DVD facilitates natural language interaction with long-form video content, improving accessibility for users with visual or cognitive impairments. It can also enhance efficiency in media summarization and analysis, significantly reducing the need for manual review.
C.2 Negative Impacts
As DVD leverages large models (e.g., L.MMs and VLMMs) for captioning and reasoning, it inherits common concerns such potential bias from training data, which may result in inaccurate or unfair interpretations of video content. Additionally, the agentic search process demands substantial computational resources, raising sustainability and accessibility concerns.
22

---

system proapt
You are helpful assistant sho can ansver miltistap questiona by sequentially calling functions. Follou pattern of THOUGHT (reason step-by-atep about which Tunction to call next), ACTIOK (call fuaction as tovards the final ansver), OBSERVATION (output the function). Reasen actions to take to get to the anaver.
step which about file content codebase structure pertaining the user's request, tools to read files and gather the relevant information: do XOT guess or make up an ansver.
Only call functions with argunents coning verbatin from the user the output of other functions, Please user's query completely resolved,
keep going user. Unly terninate your turn when are sure that the problen is solved.
anding your turn yiolding
If you
You NUST plan extensively before each function call, and reflect extensively on the outcomes of the previous function
entire process by making function calls only, ability to solve the problen and think inaightrully.
inpair
The timestamp be formatted SS: "W!:SS"
prompt
Carefully read timestanp and narration in the folloving script. Pay attention to the cause and sequence of events, the datail movament of objecta and action and peae of persons.
You will be given subject registry (vhich could inaccurate) this video and video-level coarse understanding the question. Flease not diractly find anszar fron the given information.
your think procens and correspanéing functions
CLIP_SEARCH WITH SIMME_LAAEEOODEES If nead to find details without knoving the specific timespot, + CLIP_ SEARCH
please use 'C1 'ClipSearch'.
If the observed searched information does not include accurate questien-relevatt content, e.g. 'unkoun' Inspactt ask fine-grained question within list the specified time ranges (list [tuple [hhmmas, hhnnsa]]).
other
If can not make aure the anaver by searched content, should use 'Frama IrapaInspact' to related content in specifled tine ranges.
If you have found ansver in a specific script, you should use IPrpaet" to ask a fine-grained question **confirae*
can use miltiple
prgressivaly anszer.
Based on your obeervations and returned result from functions, select the best option that accurately addresses the question.
Queation: [queation]
given choices directly and only give option. // history nessages the reasoning chain
option's
[history nessages]
Table 10: Prompts for reasoning model Mressoing-
// system proupt
You helpful assistant questions.
// prompt
Carefully vatch the video franes. Pay attention to the cause and sequence of evente, the detail and movement objects the and pose persons.
Based on your observations, if you find content that can answer the question, select the best option that accurately the question. If relevant content found within the time range, return: Error: Cannot find corresponding result. in the given time range.'.
Question: [question]
Table 11: Prompts for fine-grained detail query Mice in Frame Inspect tool.
23

---

// prempt
You are helpful assistant to anaver questiiona.
// user prompt Carefully vatch the video franes. Pay attention to the cause and sequence of evente, the detail and novenent of objects and the action and of persons.
Basad your observations, provide detailed insights help intarprat the question and video. rind content that ansvers the question, suggest possible ansver. If no relevant content is found in the Cannot corresponding
sampled frames, returns into single paragraph.
sampled frames.
Question: [question]
Table 12: Prompts for query Global Browse tool.
24

---

"function": {
"nans"; gilobalbbroves
information query shole video. \a'nReturns: An str: fornatted subject registry includes multiple identical attributes of subjects and narration about relevant information about the query, ln', "paranstars":
"description": "Obtain subject registry
"proparties": {
"database": {
"$ref": "n/Baels/lamuVectooDD""
"description":
object supports querying enbaddings."
"query": "description": textual description of the question search for.",
"type": "string"
"required": "database",
"query"
"type": "object"
"type":
"function": {
"nane": "clip_search", "description": "Searches for evente
clip database containing concatenated captiona of
given event description
and retrieves the top-k mest relerant video clip captions, \nlnReturns: in str: formatted string
video clip scripta.An/nNotee Anw.nnnnnottta This function utilizes the OpenAI Enbedding Service to generate enbeddinga for the input text. An Usa default valuas for "top_k' to limit the nunber results raturnad.", "paraneters": {
"properties"; {
"database":
"$ref": #t/Bdets/lamnVecttrr0""
"description": "The database object that supporta querying with enbeddings."
"event description" {
"description"; textual deecription "type": "string"
search for.",
}. "top_k": x
"default"; 16,
"description": "The maximan nunber results retrieve. Juet the defsult value." "type": "integer"
"rquiired": [
"database", devent_descripttinn"
type": "object"
}.
"type": "fucction"
Table 13: Function JSON schema for OpenAl model's tool use. Part 1/2.
25

---

"function": {
"nans": "Irane. siee_eensse'', "description": "Crop
based question about the cropped video clips. VnBeturne: 'a stx: The model's reeponse to the question. no relevant content is found within the time range, la returns an error message: \'Eroor Cannot find correspending result in the given time range. \"", "aranatars":
ranger
detailed
"properties";
"database"; {
"$ref": "a//Beits/lamooocttrrD""
"description": "The database containing video metadsta. Must instance
ManoVectorDa.""
"queation": {
"description"; specific detailed question the specified time ranges. No need to add time ranges in the question.", "type": "string"
during
"time ranges hhmnes": {
"description": 1iat of tuples cantaining and and times in HH: fornat. If the time range is longer than 50 seconda, the function samples 50 evenly distributed fremes. Othervise, "itens":
vithin specified range.',
"itens": O.
"tye"" "array"
}, "type": array"
'required": [ "database", "question", aann_ranges__hhaee
"tpp"": "object"
b.
"type": "function"
"function"; {
"name" "ansver", "description":
confirming ansver user's question,
finish the conversation.", "paraneters":
"properties"; {
"answer"; "description";
question.",
"type": "string"
"rquiired:: [ "answer"
"tpe": "object"
}
}.
"type": "fucction"
Table 14: Function JSON schema for OpenAl model's tool use. Part 2/2.
26

---

Question
Question: What type of video this? (A) A technology documentary
(B) A news report
(C) A travel documentary
(D) A travel vlog
Agent (Step !)
Calling tool Global Browse.
Global Browse
Subjects Information
"subject_
"name": "industrial
"appearance":
"identity":
"Industrial automation robot, performing repetitive manufacturing assembly
line tasks"
J.
"subject_10"
"name": "digital face__scan_dieeeeee displals "appeurance";
"identity":1
"Computer interface for facial recognition and identification, used for matching and unalyzing faces in security or surveillance contexts"
1.
"subject__" 21"
"name": "Pedro Domingos",
"appearance" ....
"identity": ::
"Academic expert and commentator, providing context on the Go match and public expectations"
1.
Event Information
The montage shows extended, polished coverage iiffiial-intelligenne themes: the AlphaGo challenge match and Google DecpMind branding, autonomous trucks desert highways... Therefore, the video is best classified technology documentary.
Agent (Step 2)
Executing Asnwer: (A) technology documentary. + Right
Figure 4: Case study of Global Browse Only behavior.
27

---

Question
Question: What huppens when the villain's mirror breaks?
(A) The mirror fragments automatically fuse the ground into whole mirror
again
(B) The mirror fragments the villain's 's hand
(C) The mirror fragments thrown away by the villains
(D) The mirror fragments fly into the sky
Agent (Step I)
Calling tool Global Browse.
Global Browse
Subjects Information:
"subject__ 18%
"name": "Baron Badblood",
"appearance":
"identity": [
"Villainous character. shown invoking magic reveal
J.
Event Information
The scene shows the villain ... next shots focus on villain ungrily flinging the pieces aside that the multicolored shards clatter the ground and lie seattered around his lair; therefore the mirror fragments are thrown away by the villains.
Agent (Step 2)
Calling tool Clip Search with query "villain mirror breaks shards thrown away."
Clip Search
Searched Video Clip Scripts
[00:03:7.0, 00:03:20.0] Baron Badblood stands before a large, ornate mirror..
strikes mirror sword, causing shatter. shards explode outward, tumbling and spinning through the they scatter...
100:09:46.0, (0:10:6.00
[00:23:52.0, 00:24:4.0] ... The mirror shimmers with swirling pink light,
Agent (Step 3)
Calling tool Frame Inspect ["00:09:46", "00: 10:06"]: "What do the broken pieces of the mirror do after Baron Badblood smashes it?"
Frame Inspect
Fine-grained Response
The fragments come to life: they quiver, gather into a swirling flock and fly out room, sailing away the sky.
Agent (Step
Executing Asnwer: (D) The mirror fragments fly into the sky. Right
Figure 5: Case study Simple Action behavior.
28

---

Question
Question: What does the with the mustache do when buy, cat, and dog currying
treasure chest. shovel and pickuxe back?
(A) He chases cat
(B) He chases
(C) He chases dog
(D) He chuses bird
Agent (Step 1)
Calling tool Global Browse.
Global Browse
Subjects Information
Event Information
Among the night-time forest shots the boy, cut and dog are shown tramping back along path laden. The black mousttchet postman suddenly rushes out runs after Sharik dog, trying grub the dog, while the cat and boy keep hold of the chest and tools.
Agent (Step 2)
Calling tool Clip Search with query "man with mustache chases dog while boy cat carrying treasure chest".
Clip Search
Searched Video Clip Scripts:
[00:08:0.0, 00:08:20.0] the boy pushes cart carrying large, padlocked chest, followed closely by the cat and dog, each carrying a pickaxe and shovel Overhead, black crow across the sky ... Mcanwhile, the tall thin man rushes after the group. ...
Agent (Step 3)
Calling tool Frame Inspect at ("00:08:00", "00:08:24"): "Describe exactly what the tall thin adult man does as the boy, cat, and dog carry the treasure chest, shovel, and pickaxe back along the path. Is he chasing anyone specifically?"
Frame Inspect
Fine-grained Response:
As the boy. and dog come back. the man appears on the same path.
1. He is sprinting, coat scarf flying, eyes fixed on sky.
2. His arms reach up he runs, grahhing the air.
3. He never even looks at the buy, the animuls or the chest; instead he is clearly trying to reach black crow.
man leaps
air, following
flight.
So he is not chasing the hoy, the cat, the dog the treasure chest at all; he is chasing only crow that is carrying the white object.
Agent (Step 4)
Calling tool Clip Search with query "mustached chases dog".
29

---

Clip Search
Searched Video Clip Scripts:
[00:33:12.0, 00:33:32.0] ... The cat darts away down the sandy road, desperate to escape, while dog quickly recovers resumes the chase, bounding after
Agent (Step S)
Calling tool Clip Search with query "man with mustache chases crow".
Clip Search
Searched Video Clip Scripts:
[00:08:28.0, 28.0, 00:08:48.00 The man points accusatorily the boy and the crow, prompting the boy to hand over shiny white object which the crow stolen earlier
Agent (Step 6)
Calling tool Frame Inspect ("00:08:06", "00:08:14"]: "Does the man who is running after the crow have a visible mustache? Describe his facial hair."
Frame Inspect
Cannot find corresponding result in the given time range.
Agent (Step 7)
Calling tool Frame Inspect at ("00:08:06", "00:08:14") "Does the tall man chasing the crow have a mustache, heard, is clean-shaven?"
Frame Inspect
Fine-grained Response
He has mustache.
Agent (Step 8)
Executing Asnwer: (D) chases a bird.
Right
Figure 6: Case study of Iterative Search behavior.
30

---

Question
Question: Why does the tall off the chains 21:49?
To open the gate
(B) To blow out the candles
(C) To trigger trap
(D) To help the character
Agent (Step !)
Calling tool Global Browse.
Global Browse
Subjects Information
Event Information
The tall mun hunging length of chain that holds the barred door of a cell shut, raises sword and slices clean through the links; the chain clatters free and the cell door can pushed open, allowing the wounded protagonist inside supported out.
Agent (Step 2)
Calling tool Frame Inspect at ["00:21:40", "00:21:58"]: "Why does the tall man cut off the chains?"
Frame Inspect
Fine-grained Response:
He severs the ceiling chains so the chandleea comes crashing down, plunging the corridor into darkness, giving his companion fighting chance to survive the uttack.
Agent (Step 3)
Calling tool Frame Inspect ["00:21:45", "00:21:55"|: "What happens immedi- ately after the tall man slices through the chains hanging from the ceiling?"
Frame Inspect
Fine-grained Response:
The chain holds candle- chandelier; soon severed, chandelier comes crashing down from the ceiling onto the floor.
Agent (Step 4)
Calling tool Frame Inspect at :"00:21:40", "00:22:10"]: "Docs the tall man ever cut chain to open gate or door around 21:49?"
Frame Inspect
Fine-grained Response:
No.
Agent (Step 5)
Executing Asnwer: (C) To trigger trap. + Wrong
Figure 7: Case study of Frame Inspect Trap behavior.
31

---

Question
Question: What does the protagonist through the window after she is taken the utility room?
(A) A group of monks sitting lroge- in the
(B) A group of citizens chatting together
(C) A group of warriors practicing swords
(D) A group of samuruis cating
Agent (Step !)
Calling tool Global Browse.
Global Browse
Subjects Information
Information
Inside the cramped utility room the heroine peers out through the lattice window and her eyes settle on the snow-covered courtyard where several monks sit cross legged and hare- chested despite the freezing weather, engaged silent meditation.
Agent (Step 2)
Calling tool Clip Search with query "utility room window sees group monks cross- legged snow".
Clip Search
Searched Video Clip Scripts:
[00:19:31.00 .0. 00:19:42.0] As soft snowflakes drift through the pale morning air, the solitary traveler appears on frosted hill overlooking vast, truditional Japanese town...
Agent (Step 3)
Calling tool Clip Search with query "utility room protagonist looks through window sees".
Clip Search
Searched Video Clip Scripts:
[00:02:2.0, 00:02:7.0] Dust motes drift lazily through heams of sunlight streaming from paper window, highlighting the wooden shelves lined with bowls...
Agent (Step 4- Step 14)
Calling tool Clip Search with queries "utility room small storage taken escorted feed him utility", "utility room window look out",
"storeroom small room snow see out window"
Clip Search
Searched Video Clip Scripts:
32

---

Agent (Step 15)
Calling Clip Search with query "utility room".
Clip Search
Searched Video Clip Scripts:
(0:317.3..0, 00:17:49.0] The setting dim somber. likely a shelter or workshop with stone flooring and faint light filtering in from the outside
Agent (Step 16)
Executing Asnwer: (A) group of monks sitting cross-legged the snow. Wrong
Figure 8: Case study of Clip Search Trap behavior.
33


Reviews:
[{"review_id": "gdH2zOnTyF", "reviewer": "Reviewer_vnLA", "summary": "The paper introduces DVD agents addressing a set of search-centric tools on video database. The core idea is in devising a modular search tool operating at different granularities, global, clip and frame-level. Leveraging this strategy the agent performs autonomous reasoning and iterative decision-making.", "questions": "The difference between the proposed method and other video-llms are that the proposed method utilizes iterative reasoning. What's the trade off between the boost in accuracy vs additional cost? What's the value of step limit N? Which is the smallest N>1 value that shows better performance than the baseline? \nWhy longer reasoning chain makes the results within the same model even worse?", "limitations": "The difference between the proposed method and other video-llms are that the proposed method utilizes iterative reasoning. What's the trade off between the boost in accuracy vs additional cost?", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Multigranular database construction is intuitive. Across all 4 datasets, their proposed method outperforms the base VLM. Ablation study shows the value of tool design.", "was_revised": true, "final_justification": "I read all the responses to all the questions from all the reviewers. Speed/memory comparison with other agentic systems such as Mr. Video and Video Agent helped put things into perspective better. I keep my original rating \"accept\".", "rebuttal": "**Q1: The trade-off between the boost in accuracy vs. additional cost.**\n\n**A1:**\nThanks for the valuable comment. Here we provide our evaluation results under different max step numbers with OpenAI o3 on LVBench without transcripts. \n\n| Max Step Limit | 4 | 6 | 7 | 8 | 12 | 15 | - |\n| :- | :- | :- | :- | :- | :- | :- |  :- |\n| Ours AVG. step | 3.9 |  4.7 | 5.7 | 6.7      | 7.2          | 7.3 |  - |\n| Ours accuracy (%) | 57.6 |  63.1 |68.4 | 72.3      | 73.8           | 74.2 | - |\n| OpenAI o3 (VLM Baseline) accuracy (%) | - |  - | - | - | - | -|  57.1 |\n\nWe are committed to add these results with further analysis in our revision.\n\n------\n\n**Q2: Detailed setting of max step number.**\n\n**A2**:\nIn our paper, the max step (iteration) number is set as 15 for the best performance via test-time scaling. As shown in the table of response 1, the smallest iteration with better performance than our baseline Vision-Language Model (VLM) is 4. The accuracy scales effectively, reaching 74.2% at 15 max steps, a 17.1% absolute improvement over the strong baseline.\n\n------\n\n**Q3: Why longer reasoning chain makes the results within the same model even worse?**\n\n**A3:**\nThanks for the valuable question. For the same query, the growth of iteration steps generally improves accuracy, as shown in the table of response 1. However, across different queries, the agent may follow different reasoning modes depending on the video and query content, where we observe that queries involving longer reasoning chains often correspond to lower accuracy. \n\nSupported by our case study (Appendix B.2) and quantitative data (Figure 3), we identified two typical failure modes that result in longer iterations with a decrease in accuracy, including:\n\n+ Frame Inspect Trap: The agent repeatedly utilizes the Frame Inspect Tool to pose various queries about the same temporal segment of the video. This behavior suggests a cognitive fixation, wherein the agent presumes that crucial information must exist within that specific time range. This fixation prevents it from exploring alternative video content, resulting in a 140.7\\% increase in iteration steps and an 11.1 percentage point absolute drop in accuracy.\n\n+ Clip Search Trap: The agents repeatedly employ the Clip Search Tool to find information with similar queries. This pattern typically occurs when there is little information in the video database corresponding to the user question. This behavior leads to a 152.6% increase in iteration steps and a 16.1 percentage point absolute decline in accuracy."}, {"review_id": "ghIffdHAzB", "reviewer": "Reviewer_JPc5", "summary": "The paper introduces Deep Video Discovery (DVD), an agentic framework designed for long-form video understanding. The key idea is to leverage a multi-granular video database and a set of search-centric tools (Global Browse, Clip Search, Frame Inspect) to iteratively retrieve and reason over video content using an LLM as the central orchestrator. The agent autonomously plans, refines queries, and synthesizes answers through an observe-reason-act loop.", "questions": "Please refer to the weakness section.", "limitations": "Yes.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths:\n\n1. Strong Empirical Results\n\nThe method achieves SOTA on multiple benchmarks, demonstrating clear improvements over existing approaches.\n\n2. Comprehensive Ablations\n\nThe paper includes detailed ablation studies on tool contributions and model choices, offering insights into system behavior.\n\n### Weaknesses:\n\n1. Limited Novelty in Video Retrieval & Search\n\nThe core idea of segmenting videos into clips and using an LLM for iterative search closely resembles prior works like VideoAgent. While DVD improves performance, the fundamental approach (clip-level retrieval + LLM reasoning) is not significantly differentiated. The tools (Global Browse, Clip Search, Frame Inspect) are intuitive extensions of existing retrieval strategies. A deeper discussion on how DVD uniquely advances agentic video search is needed.\n\n2. Inefficient Model Inference\n\nConstructing the multi-granular database requires processing the entire video and generating captions for each clip, leading to substantial computational costs, especially for long-form videos. In contrast, VideoAgent processes a much smaller subset of frames, making it more efficient.\n\n3. Narrow Model Evaluation\n\nThe experiments only test GPT-4.1 and o3 as the backbone models. Given the rapid progress in VLMs, the paper should evaluate proprietary models (Gemini 1.5/2.5) to assess generalization and open-source models (Qwen/Qwen VL, LLaVA, InternVL, DeepSeek R1) to ensure reproducibility and accessibility. Without broader model comparisons, it’s unclear whether DVD’s gains stem from the framework or simply from using a stronger base LLM (o3).\n\n4. Heavy Reliance on Zero-Shot LLM Capabilities\n\nThe system currently relies on the in-context learning and reasoning capabilities of an LLM. It does not fine-tune the LLM to tailor it for video understanding. This raises concerns about scalability (performance may degrade with weaker LLMs) and adaptability (task-specific fine-tuning could further improve results). A comparison with a fine-tuned variant would better demonstrate the framework’s robustness.\n\n5. Lack of Qualitative Analysis\n\nThe reviewer noticed that the authors include a few text-only results in the appendix. It is recommended that the authors include visualizations of success/failure cases in the main paper to make it easier to assess where DVD excels (e.g., complex temporal reasoning) and where it struggles (e.g., fine-grained spatial details, ambiguous queries).", "was_revised": true, "final_justification": "I have read the rebuttal and would like to keep my score.", "rebuttal": "**Q1: Limited Novelty in Video Retrieval & Search**\n\n**A1:**\nThanks for the comment. We want to clarify that our key innovation distinguishing our work from previous approaches like VideoAgent [1] lies in our **agentic search workflow orchestration** through selective combination of different search tools. \n\nUnlike previous hand-crafted, fixed workflows, our framework empowers the reasoning model to dynamically compose workflows tailored to each specific user query, resulting in flexible and query-adaptive search strategies.\n\n| Method | Workflow | Adaptive Workflow | \n| :- | :- |  :- | \n| Our DVD       |  Loop[ ToolSelect( Answer \\| GlobalBrowse \\| ClipSearch \\| FrameInspect ) ] | ✓ |\n| VideoAgent[1] | Loop[ PredictAnswer → SelfReflect → FindMissInfo → Search → Caption -> Merge ] | ✗ |\n\nTo empirically demonstrate the effectiveness of our agentic search workflow orchestration, we conducted an ablation study replacing our adaptive workflow with a hand-crafted fixed workflow. \n\nAs shown in the table below, the hand-crafted approach results in a significant 7.5% absolute accuracy drop while requiring more steps. Even when scaling up to 11.1 average steps (52% more than our DVD), the hand-crafted workflow achieves only 70.2% accuracy, still 4.0% below our adaptive approach. This comparison highlights the superior search efficiency of our dynamic workflow design.\n\n| Max Step Limit | 8 | 12  | 15 |\n| :- | :- | :- | :- |\n|**Ours**|\n| Avg. Steps | 6.7  | 7.2  | 7.3   |\n| Accuracy (%)  | 72.3 | 73.8 | 74.2  |\n|**Ours w/ hand-crafted workflow**|\n| Avg. Steps | 5.0  | 8.3  | 11.1 (↑52%)   |\n| Accuracy (%)  | 48.4 | 66.3 | 70.2 (-4.0%)  |\n\n-----\n\n**Q2: Inefficient Model Inference**\n\n**A2:**\nWe appreciate the efficiency concern regarding our framework. We discuss the efficiency of our DVD from two perspectives: (1) the trade-off between *pre-building* and *online-generating* captions, and (2) overall efficiency comparison with VideoAgent.\n\n**(1) Trade-off between *pre-building* and *online-generating* captions:**\n\nThe comparison with VideoAgent reflects an efficiency trade-off between two database construction strategies: *pre-building* captions versus *online-generating* captions. This trade-off is not specific to any single video understanding agent but rather represents a setting that can be applied to various frameworks, including our DVD. These strategies serve different scenarios with online/offline efficiency profiles:\n\n**Offline preprocessing efficiency**: VideoAgent precomputes CLIP embeddings for all frames, while DVD generates captions and embeddings using video clips as basic units. During this phase, *pre-building* captions incurs higher computational cost compared to *online-generating* while similar time cost benifited from parallel processing across clips.\n\n**Online inference-time efficiency**: Without pre-generated captions, they must be generated during inference. Here, *online-generating* becomes substantially more expensive than *pre-building*, with latency accumulating across iterative reasoning steps.\n\nThe choice between these approaches depends on usage patterns:\n- For videos queried once, *online-generating* may be more efficient.\n- For videos queried by multiple times, *pre-building* significantly reduces cumulative inference cost. For example, a video uploaded to YouTube may be queried by thousands of users with different questions, where *pre-building* can significantly reduce inference cost and improve overall efficiency.\n\nWhile DVD employs *pre-building* in this submission, it can also operate in *online-generating* mode. To evaluate this alternative, we conducted an experiment replacing our original *pre-building* database with an *online-generating* approach. The results show that this replacement leads to a slight performance drop. This further clarifies that the two strategies serve different purposes: *online-generating* is well-suited for scenarios where videos are queried once with relatively simple queries, while *pre-building* is better suited for scenarios where videos receive multiple queries and achieving optimal performance is prioritized.\n\n| DVD Database Construction | AVG. Steps | Accuracy (%) |\n| :- | :- | :- |\n| Pre-building      |  7.3   |  74.2 |\n| Online-generating |  7.5   |  71.7 |\n\n**(2) Overall efficiency comparison with VideoAgent:**\n\nWe reproduce VideoAgent using the same LLM and VLM in our codebase (denoted as VideoAgent-fashion). Our DVD framework achieves superior performance with fewer iterations, demonstrating our agent design's efficiency:\n\n| Max Iteration | 8 | 12  | 15 |\n| :- | :- | :- | :- |\n|**Our DVD**|\n| AVG. step | 6.7  | 7.2  | **7.3**   |\n| Accuracy  (%)   | 72.3 | 73.8 | **74.2**  |\n|**VideoAgent-fashion**|\n| AVG. step | 6.5  | 9.7  | **11.3**   |\n| Accuracy  (%)  | 60.0 | 62.0 | **64.1**  |\n\nUnder 15 max steps, DVD achieves 74.2% accuracy with only 7.3 average steps, while VideoAgent-fashion requires 11.3 average steps to reach 64.1% accuracy—demonstrating both higher accuracy and better efficiency.\n\n-----\n   \n**Q3: Narrow Model Evaluation on models**\n\n**A3:**\nThanks for the suggestion. Due to limited access to proprietary models, we evaluated our framework across three open-source models with varying architectures and sizes to demonstrate generalization capability. We tested DeepSeek-V3 (671B MoE without reasoning), DeepSeek-R1 (671B MoE with reasoning), and Qwen3-32B-thinking (32B dense model with reasoning) as orchestrator models on LVBench without transcripts. \n\nAll three models significantly outperform the OpenAI o3 Vision-Language Model baseline, validating our framework's effectiveness across different model architectures and scales. **Notably, our DVD framework with Qwen3-32B-thinking is already on par with original OpenAI o3**, which is the top-tier commercial reasoning vision-language model. When we equip our DVD with OpenAI o3 as the orchestrator model, we achieve a **17.0% absolute accuracy improvement** over original OpenAI o3. The consistent improvement demonstrate that DVD's gains stem from the framework design itself rather than simply using stronger models. We will include these results with further analysis in our revised paper.\n\n| Methods  | Accuracy  (%)  | \n|----|----|\n| ours w/ OpenAI o3 |74.1|\n| ours w/ DeepSeek-R1 |68.5  |\n| ours w/ GPT-4o |62.3|\n| ours w/ DeepSeek-V3 |57.5|\n| ours w/ Qwen3-32B-thinking |57.3|\n| OpenAI o3 (Commercial VLM) | 57.1 |\n\n-----\n\n**Q4: Heavy Reliance on Zero-Shot LLM Capabilities and Generalization on Small Model**\n\n**A4:**\nThanks for the suggestion. We agree that fine-tuning LLMs for video understanding agent is a promising and effective approach to improve performance. However, it is non-trivial to collect training data of long video understanding or build up reinforcement learning environment to fine-tune a smaller LLM. We plan to explore this direction in future work. \n\nHere we focus on evaluating the generalization ablity of our DVD on smaller LLMs. We conduct experiment with latest Qwen3-32B as the orchester model in our DVD. It shows even with 4.7% parameter compared to the DeepSeek-R1, DVD can still show impressive performance surpassing the Vision-Language Model OpenAI o3. It demonsrates our framework works well even with smaller LLM's zero-shot capability.\n\n\n| Methods  | Parameter | Accuracy  (%)  | \n|----|----|----|\n| Ours w/ OpenAI o3 | - |74.1|\n| Ours w/ Qwen3-32B-thinking | 32B |57.3|\n| OpenAI o3 (Commercial VLM) | - | 57.1 |\n\n-----\n\n**Q5: Lack of Qualitative Analysis**\n\n**A5:**\nThanks for the suggestion. We will add more qualitative visualization for better agent behavior analysis in our revised version to supplement the textual case study in our Append B.2. As quantitative evidence, we want to highlight Figure 3 of our main body, which shows common error patterns including:\n\n+ Repeated tool calls (\"Clip-Search Trap\" and \"Frame-Inspect Trap\"):  The agent loops on the same tool, wasting budget and lowering accuracy by 8.0 percentage.\n\n+ Behavioral collapse: when replacing OpenAI o3 with GPT-4o, the agent adapts the same simple action strategy for most of queries, resulting accuray lower than average.\nthe policy converges to a short “search-query-answer” routine covering 54 % of queries but with sub-average accuracy.\n\n------\n\n**Reference:**\n\n[1] VideoAgent: Long-form Video Understanding with Large Language Model as Agent, ECCV 2024"}, {"review_id": "Am80z8vsAI", "reviewer": "Reviewer_dHsw", "summary": "This paper presents a novel approach for comprehending long-form videos, which is a complex and challenging task due to the vast temporal-spatial complexity and dense information. The proposed solution DVD, leverages an autonomous, agentic search strategy across segmented video clips. Unlike previous systems that rely on predefined workflows, DVD uses modular search tools at multiple granular levels to enhance video understanding.\n\nThis paper demonstrates the effectiveness of DVD through extensive evaluations on multiple long-form video benchmarks, achieving state-of-the-art results. Compared to existing methods, DVD shows significant performance improvements, particularly on the LVBench dataset, surpassing previous best approaches by a large margin.\n\nThe Key contributions of this paper include: (1) An agentic search mechanism that adapts based on the query and context. (2) The design and integration of search-centric tools for efficient video understanding. (3) State-of-the-art performance on long video benchmarks. (4) Insights into reasoning model behavior and tool usage, aiding the development of future video agents.", "questions": "Question 1: This paper primarily focuses on benchmark datasets, but it would be helpful to see more discussion on how well the approach generalizes to other types of video content or less structured environments. How well does the system adapt to videos with different types of content (e.g., documentaries vs. user-generated videos)? Please show some results.\n\nQuestion 2: Please provide detailed use cases or examples where these tools are iteratively employed to answer complex queries on long video. How does the system balance between global context and fine-grained details when addressing a single query?\n\nQuestion 3: The paper mentions that the DVD system introduces higher computational overhead due to the iterative reasoning process and multiple tool invocations. Please provide some results of computational overhead compared to other agent-based approaches.", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n(1) Long-form video understanding is crucial for a variety of applications, including media analysis, surveillance, and educational content. By addressing the problem of reasoning over large video datasets, this paper has the potential to impact numerous industries where video analysis is vital.\n\n(2) This paper offers experimental validation of its approach. It presents comprehensive evaluations on long-form video benchmarks, such as LVBench, LongVideoBench, and EgoSchema. DVD demonstrates substantial improvements over previous methods, showing its efficacy in real-world tasks.\n\nWeaknesses:\n\n(1) The DVD system significantly outperforms existing methods, but it may introduce higher computational costs due to the iterative reasoning process and multiple tool invocations. \n\n(2) While the system is flexible, providing case studies or examples of diverse queries would help illustrate the effectiveness and versatility of the integrated tools.\n\n(3) The video processing itself (i.e., segmentation into clips, captioning) relies on existing methods and may not push the boundaries of video processing techniques.", "was_revised": true, "final_justification": "This paper demonstrates the effectiveness of DVD through extensive evaluations on multiple long-form video benchmarks, but I think that conducting comparisons with inconsistent numbers of model calls is unfair. Considering the existence of this issue, I will maintain my initial score.", "rebuttal": "**W1: Higher computational costs and computational overhead comparison with other agent-based approaches.** \n\n**A1:**\nThanks for the comment. Though this work focuses on advancing the boundaries of long-form video understanding capabilities by test-time scaling, we acknowledge the importance of computational efficiency. We compare our DVD agent with representive VideoAgent [1] in terms of efficiency.\nTo fair comparison, we reproduced VideoAgent[1] in our codebase with all same models, denoted as VideoAgent-fashion. Then we conducted experiments on LVBench without transcripts. The results demonstrate that our DVD achieves significantly less search steps while superior accuracy, compared to VideoAgent-fashion.\n\nUnder 15 max steps, DVD achieves 74.2% accuracy with only 7.3 average steps, while VideoAgent-fashion requires 11.3 average steps to reach 64.1% accuracy, demonstrating both higher accuracy and better efficiency.\n\n| Max Step Limit | 8 | 12  | 15 |\n| :- | :- | :- | :- |\n|**Our DVD**|\n| AVG. step | 6.7  | 7.2  | **7.3**   |\n| Accuracy  | 72.3 | 73.8 | **74.2**  |\n|**VideoAgent-fashion**|\n| AVG. step | 6.5  | 9.7  | **11.3**   |\n| Accuracy  | 60.0 | 62.0 | **64.1**  |\n\n-------\n\n**W2: Lack of cases studies to illustrate the effectiveness and versatility of the integrated tools.**\n\n**A2:**\nThank you for your suggestion. We have included the case study in Appendix B.2, wherein we present representative examples for five distinct behavioral patterns: (1) Global Browse Only, (2) Simple Action, (3) Iterative Search, (4) Frame Inspect Trap, and (5) Clip Search Trap. The Iterative Search case demonstrates the agent’s use of various tools and query formulations in pursuit of the correct answer. In contrast, the Frame Inspect Trap and Clip Search Trap illustrate scenarios in which the agent encounters difficulties. Specifically, when initial clip searches yield limited relevant information, prompting the agent to repeatedly invoke the same tools and ultimately leading to incorrect answers.\n\nWe are committed to incorporate more qualitative visualization to facilitate a more nuanced analysis of the agent's behavior in our revised version. \n\n-------\n\n**W3: The video processing itself relies on existing methods and may not push the boundaries of video processing techniques.**\n\n**A3:**\nWe appreciate your feedback and would like to clarify our core contribution. While our framework indeed employs standard video processing techniques (e.g., clip segmentation, caption generation), our primary innovation lies in the **agentic search workflow orchestration** that fundamentally changes how the search strategy is utilized for video understanding.\n\nUnlike previous video agent that rely on fixed, pre-defined workflows, our framework empowers the agent to **dynamically construct query-adaptive workflows**. The agent intelligently selects and composes different search tools based on each query's unique requirements, resulting in more efficient exploration of video content and significantly improved understanding accuracy.\n\nTo illustrate this conceptual difference:\n\n| Method | Workflow | Adaptive Workflow | \n| :- | :- |  :- | \n| Our DVD       |  Loop[ ToolSelect( Answer \\| GlobalBrowse \\| ClipSearch \\| FrameInspect ) ] | ✓ |\n| VideoAgent[1] | Loop[ PredictAnswer → SelfReflect → FindMissInfo → Search → Caption → Merge ] | ✗ |\n\nThis adaptive approach represents a paradigm shift from traditional hand-crafted, fixed video processing workflow. Rather than pushing the boundaries through novel low-level video processing techniques, we demonstrate that agentic orchestration of existing tools can achieve substantial performance gains, a 10.1% absolute improvement over VideoAgent-fashion as shown in Table of A1.\n\n-------\n\n**Q1: Generalization on different types of video content.**\n\n**A1_:**\nThanks for the suggestion. We are commited to add more analysis on video content into the revised paper. Here we provide the performance on different video content categories of LVBench. It shows that our DVD agent achieves strong performance across all categories, including ego-centric self-media, cartoon with massive scene changes, documentary with long shot and carefully edited TV show. This demonstrates the generalization capability of our framework on diverse video content types.\n\n| Category     | #Queries | OpenAI o3 (VLM)   |  Our DVD agent |\n|-------------|------|------|--------------------|\n| Cartoon     | 283  | 56.9 | 80.2               |\n| Documentary | 197  | 44.7 | 73.6               |\n| Live        | 262  | 61.5 | 75.2               |\n| Self Media  | 317  | 55.5 | 78.9               |\n| Sport       | 240  | 65.4 | 74.2               |\n| TV show     | 250  | 56.8 | 72.0               |\n| **Average** | 1549 | 57.1 | 76.0           |\n\n-------\n\n**Q2: Detailed use cases of tool use.**\n\n**A2_:**\nThanks for the question. Please refer to our response of W2. We provide the specific case study of search workflow orchestrated by our agent in Appendix B.2. For quantative analysis, as shown in Figure 3 of our main body, with OpenAI o3 as the orchestrator model, for 2.8% of queries the agent only uses the Global Browse Tool then draw the final answer. For most of the queries, the agent uses the Global Browse Tool to find relevant event and subjects, then uses the Clip Search Tool to localize specific time ranges, and finally uses the Frame Inspect Tool to inspect the frames to find more fine-grained information to confirm the answer. We are committed to add more qualitative visualization to facilitate a more nuanced analysis of the agent's behavior in our revised version.\n\n-------\n\n**Q3: Computational overhead comparison with other agent-based approaches.**\n\n**A3_:**\nThanks for the question. Please refer to our response of W1. \n\n**Reference:**\n\n[1] VideoAgent: Long-form Video Understanding with Large Language Model as Agent, ECCV 2024"}]

Author rebuttal:

Meta-review: # Summary
This paper proposes Deep Video Discovery (DVD), an agentic framework designed for long-form video understanding.

# Reasons to accept the paper
* Strong performance on various benchmarks
* Comprehensive ablations

# Reason to reject the paper
* Limited novelty in video retrieval & search.
* Higher computational costs / inefficient model inference.

# Discussion
During rebuttal, the author(s) provide an answer about the novelty: being a more adaptive framework compared with existing ones. The author(s) also provide additional comparisons on efficiency and additional based models. AC founds the response are reasonable and believes the current merits (strong empirical results) can outweigh the concerns, thus AC recommends to accept the paper. The paper had been discussed with SAC and SAC agrees with the decision.
Decision: Accept (poster)