Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Task
Authors: Sunqi Fan
Jiashuo Cui
Meng-Hao Guo
Shuojin Yang
Venue: NeurIPS 2025 poster ()
Keywords: Video Question Answering
Tool Learning
Visual Reasoning
Primary area: applications

Abstract: Video Question Answering (VideoQA) task serves as a critical playground for evaluating whether foundation models can effectively perceive, understand, and reason about dynamic real-world scenarios. However, existing Multimodal Large Language Models (MLLMs) struggle with simultaneously ensuring the ability to model spatial relationships between video frames and to understand the causal dynamics of temporal evolution on complex and reasoning-intensive VideoQA. In this work, we equip MLLM with a comprehensive and extensible Video Toolkit, to enhance MLLM’s spatiotemporal reasoning capabilities as well as guarantee the harmony between the quantity and diversity of tools. To better control the tool invocation sequence and avoid toolchain shortcut issues, we propose a Spatiotemporal Reasoning Framework (STAR) that strategically schedules temporal and spatial tools, thereby progressively localizing the key area in the video. Our STAR framework enhances GPT-4o using lightweight tools, achieving an 8.2% gain on VideoMME and 4.6% on LongVideoBench. We believe that our proposed Video Toolkit and STAR framework make an important step towards building autonomous and intelligent video analysis assistants. The code is publicly available at https://github.com/fansunqi/VideoTool.

Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Task
Sunqi Fan, Jiashuo Cui, Meng-Hao Guo, Shuojin Yang" BNRist, Department of Computer Science and Technology, Tsinghua University fansq206mails. tsinghua. edu. en, {gmh, yangshuojin Stssinggua edu, en
Abstract
Video Question Answering (VideoQA) task serves as critical playground for evaluating whether foundation models can effectively perceive, understand, and reason about dynamic real-wo world scenarios. However, existing Multimodal Large Language Models (MLLMs) struggle with simultaneously modeling spatial rela- tionships within video frames and understanding the causal dynamics of tempo- ral evolution on complex and seasoning-intensiin VideoQA task. this work, we equip MLLM with comprehensive and extensible Video Toolkit, hance MLLM's spatiotemporal reasoning capabilities and ensure the harmony between the quantity and diversity of tools. To better control the tool invoca- tion sequence and avoid toolchain shortcut issues, we propose a Spatiotemporal Reasoning Framework (STAR) that strategically schedules temporal and spatial tools, thereby progressively localizing the key area in the video. Our STAR framework enhances GPT-4o using lightweight tools, achieving on VideoMME and 4.6% on Long VideoBench. We believe that our proposed Video Toolkit and STAR framework make an important step towards building autonomous and intelligent video analysis assistants. The code is publicly available at /ttttps://gtthubb com f /as///aauunn//iioooollll
8.2% gain
Introduction
"Man is a tool-using animal. Without tools he is nothing, with tools he is all."
-Thomas Carlyle
The key to the VideoQA task lies the model's capacity to accurately perceive and reason over both the temporal logic and spatial layout of video content. This renders VideoQA compelling arena for assessing whether foundation models can comprehend dynamic, multidimensional real-world scenarios in a manner akin human cognition. With the rapid development of Large Language Models (LL.Ms), existing approaches to VideoQA can be broadly classified into two categories: Video-centric Large Language Models (Video-LLMss) [2 50
88 93 and Tool-augmented Large
Language Models (Tool-augmented LLMs) 10 82 85
Video-LLMs, represented by Qwen-V VL 22) are typically required to process large number of video frames, resulting in a redundant and inefficient pipeline. augmeeeedd LLMs, represented by DoreamonGPT 82 have several fundamental limitations: (1) Unidimensional tool utilization. Existing tool-augeented large language models primarily focus on the use of tools along a single dimension dimension-either either spatial or temporal, failing to simultancously ensure the ability to model spatial relationships within video frames and to understand the causal dynamics of temporal evolution. This limitation hinders the model's 'ss capacity for deep understanding and reasoning- number and diversity tools. As we know, increasing the number of tools can help compensate for
(2) Unbalanced
Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Spatial Tool
(3 place) Toolchain Shorteut
Temporal Tool
Toolkit
General Tool
(a) types of tools
(2ac)) Santiittmporal-Diientaaggg Toolchain
3D Region of Interest (3D Rol) determined by spatial and temporal tools
(1# place) Spatiotemporal- Interleaved Toolchain
(b) 3D Rol
(c) 3 types of toolchains
Figure I: We categorize our toolkit into 3 types: spatial tools, temporal tools, and general tools. We also compare 3 toolchain strategies: the toolchain shorteut, which shows the lowest efficiency in toolchain, which lacks mutual feedback between
tool utilization; the conntttmoorrlddientanaeee spatial and temporal reasoning; and the apatiotemmporal-nntreeeatn toolchain, which achieves the best accuracy and frame efficiency. We attribute this to its progressive localization of the 3D Region of Interest (3D Rol). See Section 4.2 for details.
many of the limitations present in LLMs. However, simply increasing the number of tools introduces various issues during execution, largely due to the inherent uncertainty of LLMs. (3) Insufficient tool scheduling strategy. The lack of an effective scheduling mechanism may lead disordered tool invocation, which negatively affects the efficiency and accuracy of model reasoming.
To address the aforementioned limitations of Tool-augmented LLMs in previous works, propose new Video Toolkit to enhance MILMS for VideoQA task. For example, by introducing tools with lightweight object detection model (7), and combining
spatial localization capabilities- them with basic operations like zooming and image cropping, we can effectively compensate for the shortcomings of MLLMs in spatial localization, thereby improving the accuracy of the VideoQA task. Additionally, tools can help filter out unimportant regions in both temporal and spatial dimensions, reducing the computational cost of image processing significantly.
Building on this video toolkit, we propose an iterative Spatiotemporal Reasoning Framework (STAR) to address the issues of insufficient strategy scheduling and avoid toolchain shorteut. The idea of this framework is to alternately narrow the scope along the temporal and spatial dimensions to locate the key regions that support the answer the question. We refer these regions 3D Regions of toolchains,
Interest (3D) Rols). As shown in Figure STAR generates spatiotemmporrlinnterleaaaa which facilitate the progressive localization of the 3D Rol to support question answering. STAR framework has the following features: (I) Autonomy. The LLM autonomously invokes tools within the framework. (2) Adaptivity. The framework dynamically adjust the tool invocation strategy based on the video's length, content, and the characteristics of the question. (3) Progressiveness. The framework progressively processes image frames, starting with small number and gradually expanding as needed.
Overall, our contributions follows:
We build comprehensive toolkit specifically designed for video analysis to enhance spatiotemporal reasoning capabilities well ensure the harmony between the quantity and diversity of tools,

---

Qwen VL 2.5 7B
74
70
3-8B
72-
66.3
GPT-4o 50
(32 frames)
70-
Apcuracy (%)
65
65.1
16.2
STAR (ours)
68-
61.8
66
60
Accuracy
64-
57.2
62
60
53
$2.6
58
LLVi
56-
VidenAgen
50
9
54
Video
AKeyS g
STAR (ours)
50
VidcoMME
Long VideoBench
Number of frames processed
(a) Augmenting GPT-4o with Video Toolkit achieve 8.2% gain VideoMME [12 and 5% Long VideoBench 77
(b) As the number of input frames increases.
STAR consistently achieves the highest accuracy
EgoSchema
which includes 22 different tools covering a wide range of functionalities. All tools are plug-and-play, making this Video Toolkit highly extensible and easy to expand.
We propose a novel reasoning framework named STAR that enables LLM to alternately invoke temporal and spatial tools. STAR facilitates stepwise visual reasoning by adapting to intermediate tool results, progressively narrowing down essential information to solve the problem.
Through extensive experiments across multiple VideoQA benchmarks, we demonstrate that aug- menting GPT-4o with lightweight Video Toolkit, results an 8.2% gain on VideoMME [22] and 4.6% on Long VideoBench [77). It also outperforms the current leading 7B Video-LLMs shown in Figure (aa)
We also show that STAR has stronger scalability shown in Figure 2(b)
Related Work
2.1
Video Question Answering
Video Question Answering (VideoQA) requires machines to answer questions based on videos.
Driven by growing demands from real-world applications, the research community has progressively developed more sophisticated benchmarks [12 19
30 44
[54
57
(4
77
78 (92.. featuring longer
videos and richer spatial and temporal dependencies. The emergence transformer architectures has advanced video question answering through cross-modal pretraining and spatiotemporal modeling [4]
26
39,
59
69] [79 9T]. Recent advances in Video-LLMs [2]
32
68
72
73
93 have
enabled open-ended video question answering by integrating LLMs with video encoders. Additionally,
LLMs are used to reason across the temporal dimension and select keyframes adaptively for video understanding 220 23 24 62 7 74, 75 8I 85, [8]. Some other works [36] [1] 6] 86] improve
computational efficiency by first using lightweight network modules to extract keyframes. builds scenes event graphs to enable stepwise video understanding.
2.2 Tool Learning
Tool learning has emerged as a promising direction to enhance (M)LLMs by equipping them with the ability invoke external tools. Early works focus on augmenting LLMs with API calling capabilities to perform tasks beyond their parametric knowledge, such as computation, web search, and code exccution 45
[5
83 33 34. Integrating tools with LLMs also boosts their reasoning abilities,
especially when multiple tools are combined in sequence to solve complex tasks [13] 27 43) Some works fine-tune LLMs for tool use, while others integrate tools in a alug-and-plas manner without extra training. The order of tool invocation is crucial, and researchers have built large-scale tool network datasets [52 58 to study how to better orchestrate sequential tool calls. Tool-augmented LLMSs have also been applied to solve various tasks in computer vision by invoking specialized visual tools
65 While most existing studies focus on image-based applications, growing number of works
25
10
29
60
82 90 have explored video understanding through tool and

---

visual program augmentation. DoraemonGPT 82 relies on text-to-SQL queries over an intermediate database containing tools' outputs, which often fail and thus hurt both accuracy and efficiency. In contrast, our STAR framework circumvents this issue with a more reliable tool execution pipeline.
3 Method
3.1 Video Toolkit Construction
In this section, we first describe the design principles and construction process of the proposed Video Toolkit. The design of the Video Toolkit is guided by the following principles. (1) Video processing tasks naturally decompose into temporal and spatial dimensions. Accordingly. the Video Toolkit incorporates three types of tools: temporal, spatial, and general-purpose. (2) Video Toolkit should integrate wide range of computer vision results through natural language interface. For example, how can outputs such as bounding boxes from object detection be integrated into natural language to assist in question answering? (3) In the temporal dimension, the Video Toolkit should support both segment-level and frame-level processing, enabling analysis of video segments as well as individual frames. Following these three design goals. we developed comprehensive Video Toolkit consisting of about 22 video analysis tools. The full list of tools and implementation details are provided in Appendix Section A Some of these tools are based on lightweight and specialized models, while others are simple image and video manipulations (c.g.. the Patch Zoomer, which can zoom in or out on image regions). Additionally, fully exploit the potential of MLLMs, we encapsulate several effective prompting pipelines as tools (c.g. iterative prompting MLLM for open-vocabulary action localization 67]) Below, we present two representative tools.
Object Detection and Bounding Box Utilization. Spatially, one of the most commonly used tools is object detection tool. To support different usage scenarios, we implemented two versions of object detection tool based on YOLO (7] and Grounding DINO (38, respectively. A typical tool chain involving object detection tool proceeds as follows: the L.M analyzes the question to determine the target object, which is then detected in the image using YOLO or Grounding DINO, yielding set of Bounding Boxes (bboxes). We further implement three ways to utilize the resulting bboxes: (1) Converting the bboxes into textual descriptions and feeding them back to the LLM. (2) Using the Patch Zoomer tool to enlarge the regions within the bboxes. (3) Following Set-of-Mark Prompting [80]. we overlay the bbox regions with visual markers to highlight them before passing the image back to the MLLM for further visual reasoning.
Frame Selector for Temporal Navigation. Temporally, the core tool is the Frame Selector, which is based on an LLM. It selects informative frames while discarding irrelevant ones based on the question and accumulated information. Initially, the Frame Selector collects sparsely and uniformly sampled frames. As more information is gathered through the use of other tools, we instruct the LLM to imagine the full video and reason about which temporal segments are most relevant to answering the question. The Frame Selector can output a single key frame, in which case the tool chain typically proceeds with spatial tools designed for image-level processing; it can also call video trimming tool to extract segment-level video clips for subdivision. We also integrate several efficient (M)L.LM-based keyframe selection pipelines. For example, AKeyS (9] considers both the question and scene transitions, and selects keyframes using an A based search strategy. Another example is 7 [85), which arranges multiple frames into an image grid and feeds into the MLLM for selection.
Following Octotools 422 our Video Toolkit employs standardized tool cards to encapsulate each tool's 's functionality. All tools are designed to be glug-and-play ensuring high extensibility and ease of integration.
3.2 Spatiotemporal Reasoning Framework
By augmenting (M)LLM with Video Toolkit, we propose STAR, a training-free, user-friendly, and extensible agentic and reasoning framework for streamlining VideoQA task across diverse domains.
Toolchain Shortcut. Our initial design allows the core LLM Planner to autonomously select tools from the entire toolkit and fully control the sequence of tool invocations (i.c., the toolchain [94)).

---

+1200
-100
-2400
-30000
-3500
04200
<4800
-5400
10000
00000
this video, by using a tractor, how many times fäster is the cultivation of the same land compared manual labor?
Calling temporal- hatporraa-gouudi--ttoo tool to ground the tractor.
Temporaa-grounding-oo tool: appears
Calling imago-caption-toll for the 20 frames from =1 20- <190s
Q
Imago-caption-tool 766:; image is compased
pictures.
Video Toolkit
picture, two people are working together to plow the field, while the second picture, a single person operating tractor. texts the image indicate time complese the task...
Description
Calling frame-selector select frames.
Input Types
Frame-selector: Look into current frimes, 766 selected.
Output Types
Motadata
Tool Card
Frame <176
Visible Frames
It seems thene are some texts in the image. Calling OCR-tool for frame <1 <176
Frame 1....
OCR-too): "45 min" and "45 sce" frame <176>,
Frame 178:..
Calling summarizer summarize all the information.
Summarizer: Using tracter is 10 times faster than manual labor.
Visible Frame Dictionary
Figure 2: Visualization of our video toolkit, tool cards, and visible frame dictionary. Demonstration of the STAR pipeline. In this case, the LLM planner sequentially invokes five tempora grounding, image captioning, frame selection, OCR, and summarization to solve the problem.
Following the ReAct [33 agent framework, the LLM Planner makes an observation based on each tool invocation's 's results. The LLM Planner then generate a thought based on the observation, which decides the selection of the next tool invoke whether enough information has been gathered to answer the question. However, this loosely constrained and overly flexible design can sometimes lead to Toolchain Shortcut problem. We define Toolchain Shortcut a phenomenon where the LLM Planner takes shortcuts by favoring single-step tools to directly answer the question, rather than progressively decomposing the problem and constructing longer, step-by-step toolchain for reasoning. In our VideoQA task, toolchain shortcut manifests when the LLM Planner directly invokes general- purepol tool (e.g., video-language model) answer the question, bypassing our (4. Jf22effers offers a detailed
intended strategy of progressively breaking down the problem. Our ablation study/4.2 examination of how Toolchain Shortcut affects both the accuracy and computational efficiency of the system.
STAR Algorithm To address Toolchain Shortcut problem, we impose a constraint on the toolchain: temporal and spatial tools must be invoked in an alternating manner, and general tools can only be called as a last resort when temporal and spatial tools alone are insufficient to answer the question. The initial tool can be either temporal or spatial, depending on the task. We also maintain Visible Frame Dictionary V. where the keys are the indices visible frames and the values are information gathered by various tools for each frame. At initialization, set of uniformly sampled video frames, which initially contain no additional information. We then start the tool calling process, where temporal tools are responsible for selecting the frame indices to be processed- which can be a single frame, a continuous segment, or a set of discrete frames and updating the visible frame dictionary by adding removing frame indices needed.
populate this dictionary with sparse
Spatial tools are responsible for processing the frames specified by the temporal tools and updating the corresponding values in the Visible Frame Dictionary V for the respective frame indices. At each step, the core LLM Planner decides whether the information in the Visible Frame Dictionary is sufficient to answer the question. If not, it determines which tool to invoke next. If the first tool
5

---

Algorithm Spatiotemporal Reasoning (STAR)
Require: Video v, question LLM Planner P. iteration num 7, Visible Frame Dictionary
7.9
D, spatial toolset T., temporal toolset Tt
general toolset T
Ensure: Answer '
14
1: D < UnfformSpaaseSampleey
2: + SelectTool P, D, g. T, U Ti)
3: < ExcuteTTool ()
4: D < UpdateDict(D,,,
102
5: for 1 do
6: Text - T if E T3
1.4
5.6
ift E Te T,
1319
2.3
7:
t + SelectTool D. D.D.q.1. mext)
Firnc Sdocar
Google
Vitos Ok
8:
r ExcuteTTool v. t)
Imqua Eale
Dinge
Action Rooggabee
Vikeo
9:
D <- Updateliict D,
Lacibratee
Teas Sende
10: end for
Samardd Seprccatiio
Oiject Tlacee
11: 4- SelectTool( P. q.
Figure 3: Diverse tool usage distribution on VideoMME,, showcasing wide range of tools being actively utilized.
12: 9 f ExxcuteTool
13: return y
invoked a temporal tool, the LLM Planner will select tools from the temporal toolset at every odd step, and from the spatial toolset at every even step; the process works in reverse if the first tool is spatial. If the LLM Planner determines that neither temporal nor spatial tools can resolve the issue, will resort general tools to solve the problem. We conclude this algorithmic procedure in Figure
and Algorithm]] I We also visualize the percentage distribution tool usage on the VideoMME 12 dataset in Figure 3] showing that tools are utilized in well-balanced way, with tool being excessively favored or underused.
Why spatiotemporal interleave? Spatiotemporal tool interleaving ensures that, after temporal tools narrow the time range, spatial tools can further refine the spatial range. The results from spatial tools, in turn, influence the subsequent use of temporal tools, ensuring that the understanding of time and space is complementary. If time and space were decoupled, with time being progressively narrowed first and then space, without information transfer between the two, it would lead to a decline in both the accuracy and efficiency of video understanding. demonstrated in our ablation study 4.2
How STAR mitigates the limitations of MLLMs in spatiotemporal reasoning? STAR frame- work mitigates the spatiotemporal reasoning limitations of MLLMs mainly in two ways. (1) Spe- cialized tools can generate more accurate outputs than MLLMs for specific tasks. For example, in video with heavy pedestrian traffic, Video-LLM often struggles with estimating the total number of unique individuals. In contrast, when the LLM invokes object tracking and person re-identification tools, the accuracy can be significantly improved.
(2) Even when tools cannot directly produce definitive answers, the STAR framework progressively narrows down the temporal and spatial scope localize the question-relevant 3D Region of Interest (3D Rol). In video analysis, a 3D Rol refers to specific subset of the video defined across both spatial (height and width) and temporal (time) dimensions. By narrowing down and focusing on 3D Rol, STAR reduces irrelevant content and minimizes interference. Similar to how
prompting [76] elicits deliberate, system II reasoning [1 in complex reasoning tasks, our tool-augmented spatiotemporal reasoning framework also encourages such deliberate visual thinking, leading to improved accuracy and computational efficiency in complex video question answering scenarios.
Experiments
We evaluate our method on four widely used VideoQA datasets: VideoMME 12),
NEXT-QA 78
VideeBench [77], and EgoSchema [44. These datasets cover diverse video lengths (ranging

---

Table Results on VideoMME. We highlight the best results for each method category in bold, and mark our improvements over GPT-4o in blue.
Model
Size
Frames
4
Runtime
Short
Medium
Long
All
Proprietary Image-hased Ms
GPT-4o 50
[ps / 384
> 10 min
80.0
70.3
65.3
71.9
GPT-4o
32
< 30 s
68.3
60.7
56.3
61.8
GPT--4 T
85
32
30 1 min
69.5
63.5
59.3
64.1
4v 49
10
< <0
70.5
55.8
53.5
60.0
Gemini 1.5 Pro 15
0.5 fps 20
10 min
81.7
74.3
67.4
75.0
Claude 3.5 Sonnet
< 30
71.0
57.4
51.2
60.9
Open-soure Video LLMS
Qwen2-V2 VL 70
72B
2 [ps 768
6 min min
80.1
71.3
62.2
71.2
Qwen2-VL VL
7B
63.3
Qwen2.5-VL P
7B
65.1
Intern V1.2.5 6
8B
64
<30
64.2
Intern V1.3 993
8B
64
<30
66.3
VideoLLaMA3 888
180
30
60
54.9
mPLUG-OwB
128
30 60
70.0
50.1
STAR (ours)
30.2
15.8
78.9
68.3
62.9
70.0 (8.2 1)
Table 2: Results on Bing Validation Set. We highlight the best results for each method category bold, and mark our improvements over GPT-4o blue.
Model
Params
Frames
Runtime
8s-
15s-
180s-
900s- 3600s
All 1
15s 1
60s f
600s
Proprietary Image-basee based MLLMs
GPT-4o 50
256
> 10 min
71.6
76.8
66.7
61.6
66.7
GPT-4o
32
<0
60.7
62.4
50.1
49.0
52.6
Gemini 1.5 Pro
256
10 min
64.0
Open-souree Video-LLM, (-7B)
Qwen2.5-VL
fps 512
min min
59.0
65.6
48.8
53.7
IL3
256
min 3 min
54.7
66.9
44.0
48.9
STAR
29.6
15.3
63.6
68.0
56.8
52.3
57.2 (4.6
from few seconds to several hours), video types and question types, enabling a comprehensive comparison between our approach and baseline methods. Introductions these datasets provided in Appendix B We primarily compare our method against the following four categories of baselines: (1) image-based MILMs, (2) Video-LLMs, (3) (M)LLM-basee frame selection methods, and (4) LLM-driven tool learning methods. Detailed introductions of these baselines can be found in Appendix We use multiple-chote choice QA accuracy measure performance, and use the number of processed frames and runtime (the average time answer question under identical conditions) to measure computational efficiency.
C
To accommodate different computational budgets and compare fairly, we design two variants of our framework depending on whether the full video toolkit is used: STAR and STAR-MINL. STAR is the full version, employing tools based on open-source models with up to 3B parameters (c.g.. VLe2.L-2255333 2 and using EPT-4o-2024-088
[50] APIs. Following previous baseline ()]
It uses the same version GPT-4o its LLM Planner. STAR-MING excludes any tools larger than 500M parameters; its largest tool is 500M
500M-parameter paramettr BLIP (28. also avoids any tools that require GPT-4o APIs. Following previous baseline [82] It uses the same version P-T-3.5-ttrbb--e -025 [48 as its LLM Planner. Our STAR framework can run on one NVIDIA RTX 4090 GPU, while variant STAR-MINT framework can run personal computer like MAC. For videos longer than 16 seconds, we extract 16 initial frames uniformly; for videos with a duration 16 seconds or less, initial frames are extracted at rate of I fps.
7

---

Table 3: Results on NEXT-QA Test Set. We highlight the best results for each method category in bold, and mark our improvements over the best baseline T in blue.
Method
Base Model
Frames
Cau. 1
Tem. 1
Des. 1
All
Open-source Video-LLMs (-7B)
Intern VL3-8B 93
8
77.5
72.1
77.1
75.7
Qwen2.5-VI-7B
2 fps
80.6
79.3
85.5
80.9
(M)L..M-busee based Frame Selection Methods
VideoAgent 71
GPT-4
8.2
64.5
72.7
81.1
71.3
Video'Trre 75
GPT-4
63.2
70.6
76.5
83.9
75.6
LVNet [51
GPT-4o
12
65.5
75.0
81.5
72.9
VidF4 [3
8
69.6
74.2
83.3
74.1
AKeyS
GPT-4o
7.6
72.9
79.0
86.1
78.1
Detector-based 7
85
GPT-4o
8
80.4
STAR (ours)
GPT-4o
7.2
81.1
81.5
86.3
82.1 (1.2 1)
Table 4: Results on NExT-QA Validation Set. We highlight the best results for each method category
in bold, and mark our improvements over the best baseline DoraemonGPT [2 in blue.
Method
Base Model
Frames 4
#LLM calls
Cau.
Tem.
Des.
All f
LLM-driven Tool Learning Methods
ViperGPT 60
GPT-3 Codex
43.2
41.0
62.3
45.5
VideoChat
50.2
47.0
65.7
51.8
DomemonGiP
GPT-3.5-turb
28.7 fps 1144.4
8.5
54.7
50.4
70.3
55.7
STAR-MINI (ours)
GPT-3.5-turb
0.6 fps 22.6
5,4 (3.1
62.8
55.1
73.7
62.0 (6.3 f)
4.1
Main Results
4.1.1 Results on VideoMME
As shown in Table I. with the nearly same number of input frames, our method achieves an 8.2% improvement over GPT-4o [50]. Meanwhile, our method significantly outperforms all open-source Video-IIIMs around 7B scale (achieving 3.7% improvement over the best-performing Intern 8B), and approaches the performance of open-source Video-LL.Ms L.MMs around 72B scale. Moreover, in terms of efficiency, our method substantially reduces the number of video frames required for processing. Compared to the 72B Qwen2-VL. VL, our method achieves a substantial speedup, reducing runtime from 6-8 minutes to 15.8 seconds. We reproduce GPT-4o with 32 input frames and other baseline results in Table I come from the official leaderboard and technical reports.
4.1.2 Results Long VingeBebeen
As shown in Table 2 with the nearly same number of input frames, method achieves 4.6% improvement over GPT-4o. Meanwhile, our STAR framework outperforms both Qwen2.5-VL-7B and VL3--8 s) parts. This demonstrates that our method is more effective than the current 7B Video-LL.Ms identifying key information from long videos. Moreover, the STAR framework substantially reduces both the number of processed frames and runtime, leading to lower computational cost. We reproduce GPT-4o's 's results with 32 input frames using no subtitle information. We also reproduce the performance of Qwen2.5-VL L-- and InternVL3-88 on the validation set without using subtitle information for comparison. Results of Gemini and GPT-4o with 256 input frames taken from the official leaderboard. Their use of subtitle information is one of the key reasons why they outperform the STAR framework. STAR currently focuses solely on the video content: we consider better integration of subtitle and audio information as future work.
large margin, particularly in the long (180-600 s) and extra-long (900-3600
4.1.3 Results on NEXT-QA
As shown Table STAR achieves the highest accuracy while using the fewest frames, outper- forming both 7B Vidco-LLMs and all (M)LLM-basee frame selection methods. It attains over 80%

---

Table 5: Ablation on Different Methods Generate Toolchain
Methods
Accuracy 1
Num of Frames
Toolchain Length f
Num of Tools f
No Constraints
61.2
112.6
2.9
1.3
Prompting
60.4
98.7
3.6
1.9
In-Context Learning
63.2
50.1
5,4
3.2
Spatiotemporal Disentanglement
68.6
40.6
5.6
3.4
STAR
70.0 (1.4
30.2 (10.4 D)
8.7 (3.1 ()
6.3 (2.9 f)
accuracy across all three question categories- temporal, and effectiveness across diverse question types. The performance of open-source Video-LLMs is our own reproduction, and the results of (M)LLM- -baded frame selection methods are reported from their original papers. And in Table on the NExT-QA validation set, STAR STAR-MINI outperforms Dorae- monGPT [2] by 6.3% in accuracy, demonstrating the superiority of the spatiotemporal interleaving tool invocation strategy. All the results of LLMM-drieen tool learning methods reported from their original papers.
its
4.2 Ablation Studies
A Ablation on Toolchain Besides the STAR framework, we also explore the following strategies to control the tool invocation order. (1) No Constraints: The LLM Planner has no restrictions; all tool information provided in tool cards, and the planner autonomously selects tools according to the question and the tool cards. In practice, this often leads to the problem of Toolchain Shortcut. (2) Prompting: Inspired by chain-of-thought prompting [76), we instruct the LLM Planner to invoke longer and more complex toolchains, solving the problem step by step. (3) In- Contectt Learning: We provide manually annotated examples with long toolchains, leveraging the LLM's in-context learning ability to imitate and generate more complex tool sequences. (4) Spatiotemporal Disentanglement: The LLM Planner guided to first invoke one or more temporal tools to narrow the temporal scope, followed by one more spatial tools to refine the spatial scope, and finally either directly generate the answer or optionally call general tools before answering.
We conducted experiments on VideoMME 12 to compare these strategies in Table 5
The No Constraints method results in overly short toolchains and repetitive tool usage, leading to exces- sive frame processing and high computational costs. The Prompting method has only marginal improvements over the No Constraints method, indicating that prompt- control alone is in- sufficient. The In-Context Learning method provides stronger control compared to the Prompting method, significantly reducing the number of frames processed and increasing toolchain length. However, it still falls short in terms of accuracy. Both Spatiotemporal Disentanglement and STAR framework impose explicit constraints on the LLM Planner by directly controlling the toolchain structure. Among them, the spatiotemporal-intee -tttemmmppraann-eeraeeeeedd -nterreeaeed STAR achieves the best accuracy and frame efficiency, benefiting from longer, more diverse toolchains. We mark STAR's 's improvements over the becond-best Spatiotemporal Disentanglement method in blue.
More Ablations and Analysis We also conducted ablation study on each tool, showing that each tool contributes positively to the overall performance Appendix Section D Additionally, Appendix cxamions the impact of varying frame numbers on the performance of the STAR framework, while Appendix Section [ investiiaate the effects of diffferent LLM Planners on its performance. For more in-depth analysis, we verified whether the STAR framework promotes a balanced and comprehensive use of tools, as presented in Appendix Section G in Appendix Section summarize the failure cases in Appendix Section
We also conduct more case studies
H
5 Conclusion
In this work, we equip (M)LLMs with a carefully designed and comprehensive video toolkit to compensate for the temporal and spatial reasoning limitations of MILMs, streamlining the VideoQA task. To better control tool invocation, we propose Spatiotemporal Reasoning framework (STAR), which alternately calls temporal and spatial tools to progressively localize the 3D Rol, thereby improving accuracy and reducing computational cost. We conduct extensive experiments four

---

widely used VideoQA datasets. Our STAR framework, built with lightweight models of fewer than 3B parameters, significantly boosts GPT-4o's ability in video understanding and reasoning, and consistently outperforms related baselines in accuracy and efficiency.
Our main limitation lies in that STAR framework still relies the capabilities of OpenAl GPT models, which can lead to certain API costs due to repeated invocations. In future work, we plan explore replacing GPT-4o planner with more lightweight models and investigate techniques to better integrate tool usage into the reasoning process of the planner. Nevertheless, we believe that our proposed Video Toolkit and STAR framework make an important step towards building autonomous and intelligent video analysis assistants.
Acknowledgement This research was supported by the National Natural Science Foundation of China (project No. 62495060, 623B2057), the Research Grant of Tsinghua-Tenee Tencent Joint Laboratory for Internet Innovation Technology.
10

---

References
[1] Anthropic. Introducing claude 3.5 sonnet. http:://www.aaatnprrrccc.ooooeee 2024.
[2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025.
[3] Ziyi Bai, Ruiping Wang, Difei Gao, and Xilin Chen. Event graph guided compositional spa- tial-temporal reasoning for video question answering. IEEE Transactions on Image Processing, 33:1109-1121, 2024.
[4] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding?. 2021.
[5] Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang. Zhijian Liu, Ethan He, Hongxu Yin, Pavlo Molchanov, Jan Kautz, Linxi Fan, Yuke Zhu, Yao Lu, and Song Han. Longvila: Scaling long-context visual language models for long videos, 2024.
[6] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang. Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng, Jiaye Ge, Kai Chen, Kaipeng Zhang, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, 2025.
[7] Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real-time open- vecabuacayllary object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16901-16911, 2024.
[8] Rohan Choudhury, Koichiro Niiuuma, Kris M. Kitani, and László A. Jeni. Zero-shot video question answering with procedural programs, 2023.
[9] Sunqi Fan, Meng-Hao Guo, and Shuojin Yang. Agentic keyframe search for video question answering, 2025.
[10] Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory--aumennted multimodal agent for video understanding, 2024.
[11] Hao Fei, Shengqiong Wu. Wei Ji, Hanwang Zhang, Meishan Zhang, Mong-Li Lee, and Wynne Hsu. Video-of-thought: Step-by-step video reasoning from perception to cognition, 2024.
[12] Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li. Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. Vidco-mme: The first-ever comprehensive evaluation benchmark of multi-modal lims in video analysis, 2024.
[13] Silin Gao, Jane Dwivedi- Yu, Ping Yu, Xiaoqing Ellen Tan, Ramakanth Pasunuru, Olga Golovneva, Koustuv Sinha, Asli Celikyilmaz, Antoine Bosselut, and Tianlu Wang. Efficient reasoning, 2025.
tool use with
[14] Zhi Gao, Yuntao Du, Xintong Zhang, Xiaojian Ma, Wenjuan Han, Song-Chun Zhu, and Qing Li. Clova: A closed-loop visual assistant with tool usage and update, 2024.
[15] Team Gemini, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055300 2024.
11

---

[16] Google. Gemini 2.5: Our most intelligent ai model. https://boogggoggleeecceneet wooggee/icccnnooog//oogeeee deepmind/gemini-noone modeeemiii--moll-iieee moddo-odtiiiie thiii-innnknnggg sapds--ddddaaaaaee 20ttt/2005//eemii22---------
2025.
[17] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501 129488 2025.
[18] Meng-Hao Guo, Xuanyu Chu, Qianrui Yang, Zhe- Mo, Yiqing Shen, Pei-lin Li, Xinjie Lin, Jinnian Zhang. Xin- Xin-Sheng Chen, Yi Zhang, et al. Rbench-v: A primary assessment for visual reasoning models with multi-modal outputs. arXiv preprint arXiv:2505. 16770, 2025.
[19] Meng-Hao Guo, Jiajun Xu, Yi Zhang, Jiaxi Song, Haoyang Peng, Yi-Xuan Deng, Xinzhi Dong. Kiyohiro Nakayama, Zhengyang Geng, Chen Wang. Bolin Ni, Guo-Wei Yang, Yongming Rao, Houwen Peng, Han Hu, Gordon Wetzstein, and Shi-Min Hu. RBench: Graduate-level multi- disciplinary benchmarks for LLM & MLLM complex reasoning evaluation. In Proceedings of the 42nd International Conference Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 21 242011 PMLR, 13-19 Jul 2025.
[20] Kai Hu, Feng Gao, Xiaohan Nie, Peng Zhou, Son Tran, Tal Neiman, Lingyun Wang, Mubarak Shah, Raffay Hamid, Bing Yin, and Trishul Chilimbi. M-llm based video frame selection for efficient video understanding, 2025.
[21] Yushi Hu, Otilia Streteu, Chun-Ti: Ta Lu, Krishnamurthy Viswanathan, Kenji Hata, Enming Luo, Ranjay Krishna, and Ariel Fuxman. Visual program distillation: Distilling tools and programmatic reasoning into vision-language models, 2024.
[22] Xinyu Huang, Youcai Zhang, Jinyu Ma, Weiwei Tian, Rui Feng, Yuejie Zhang, Yaqian Li, Yandong Guo, and Lei Zhang. Tag2text: Guiding vision-language model via image tagging, 2024.
[23] Sullam Jeoung, Goeric Huybrechts, Bhavana Ganesh, Aram Galstyan, and Sravan Bodapati, Adaptive video understanding agent: Enhancing efficiency with dynamic frame sampling and feedback-driven reasoning. 2024.
[24] Kumara Kahatapitiya, Kanchana Ranasinghe, Jongwoo Park, and Michael S. Ryoo. Language repository for long video understanding, 2024.
[25] Minkuk Kim, Hyeon Bae Kim, Jinyoung Moon, Jinwoo Choi, and Scong Tae Kim. Do you remember? dense video captioning with cross-modal memory retrieval, 2024.
[26] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L. Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for aido-and--anguue learning via sparse sampling, 2021.
[27] Chengpeng Li, Mingfeng Xue, Zhenru Zhang, Jiaxi Yang, Beichen Zhang, Xiang Wang. Bowen Yu, Binyuan Hui, Junyang Lin, and Dayiheng Liu. Start: Self-taught reasoner with tools, 2025.
[28] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision lisgun-language understanding and generation, 2022.
[29] KunChang Li. Yinan He, Yi Wang, Yizhuo Li. Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding, 2024.
[30] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mybench: A comprehensive multi-modal video understanding benchmark, 2024.
[31] Rui Li, Xiaohan Wang, Yuhui Zhang, Zeyu Wang, and Serena Yeung-Levy. Temporal preference optimization for long-form video understanding, 2025.
[32] Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan He, Chenting Wang, Yu Qiao, Yali Wang, and Limin Wang. Videochat- flash: Hicrarchical compression for long-contex video modeling, 2025.
12

---

[33] Zhenyu Li, Sunqi Fan, Yu Gu, Xiuxing Li, Zhichao Duan, Bowen Dong, Ning Liu, and Jianyong Wang. Flexkbqu: A flexible Ilm-pon powered framework for few-shot knowledge base question answering. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 18608-18616, 2024.
[34] Zhenyu Li, Xiuxing Li, Sunqi Fan, and Jianyong Wang. Optimization techniques for unsuper- vised complex table reasoning via self-training framework. IEEE Transactions on Knowledge and Data Engineering, 2024.
[35] Zhuowan Li, Bhavan Jasani, Peng Tang, and Shabnam Ghadar. Synthesize step-by-step: Tools, templates and llms data generators for reasoning-based chart vqa, 2024.
[36] Jianxin Liang, Xiaojun Meng, Yueqian Wang, Chang Liu, Qun Liu, and Dongyan Zhao. End-to- end video question answering with frame scoring mechanisms and adaptive sampling, 2024.
[37] Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, et al. Llava-plus: Learning use tools for creating multimodal agents. In European Conference on Computer Vision, pages 126-142. Springer, 2024.
[38] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang. Jie Yang, Qing Jiang. Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision, pages 38-55. Springer, 2024.
[39] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu, Video swin transformer, 2021.
[40] Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Oryx milm: On-demand spatial-temporal understanding arbitrary resolution, 2025.
[41] Scott C. Lowe. System 2 reasoning capabilities are nigh, 2024.
[42] Pan Lu, Bowen Chen, Sheng Liu, Rahul Thapa, Joseph Boen, and James Zou. Octotools: An agentic framework with extensible tools for complex reasoning, 2025.
[43] Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: loug-and-pll compositional reasoning with large language models. Advances in Neural Information Processing Systems, 36:43447-4344 2023.
[44] Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. 2023.
[45] Reiichiro Nakano, Jacob Hilton, Suchir Balaji. Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browscr-assisted question-answerigg with human feedback, 2022.
[46] Elliot Nelson, Georgios Kollias, Payel Das, Subhajit Chaudhury, and Soham Dan. Needle in the haystack for memory based large language models, 2024.
[47] Open-MMLab. Mmaction document. http://mmacciin222eaaeeeeeeeosoo
2023.
[48] OpenAl. Gpt-3.5-turbo document. http://pwfaom..pp.m....cooooo/// 2023.
[49] OpenAl. Gpt-4v(ision) system card.
2023.
[50] OpenAl. Gpt-4o system card. htts://openai..co//// http:://wwn..ccommmudeeggggeeeee 2024.
[51] Jongwoo Park, Kanchana Ranasinghe, Kumara Kahatapitiya, Wonjeong Ryu, Donghyun Kim, and Michael S. Ryoo. Too many frames, not useful: Efficient strategies for long-form video qa, 2025.
13

---

[52] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis, 2023.
[53] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision, 2022.
[54] Ruchit Rawal, Khalid Saifullah, Miquel Farré, Ronen Basri, David Jacobs, Gowthami Somepalli, and Tom Goldstein. Cinepile: A long video question answering dataset and benchmark, 2024.
[55] Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.1415% 2024.
[56] Timo Schick, Jane Dwivedi-Yuu Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools, 2023.
[57] Enxin Song, Wenhao Chai, Weili Xu, Jianwen Xie, Yuxuan Liu, and Gaoang Wang. Video- mmlu: A massive misti-discipline lecture understanding benchmark, 2025.
[58] Yifan Song, Weimin Xiong, Dawei Zhu, Wenhao Wu, Han Qian, Mingbo Song, Hailiang Huang, Cheng Li, Ke Wang. Rong Yao, Ye Tian, and Sujian Li. Restgpt: Connecting large language models with real-world restful apis. 2023.
[59] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning, 2019.
[60] Didac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: for reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11888-11899, 2023.
Visual inference via python execution
[61] Xi Tang, Jihao Qiu, Lingxi Xie, Yunjie Tian, Jianbin Jiao, and Qixiang Ye. Adaptive keyframe sampling for long video understanding, 2025.
[62] Yunlong Tang. Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin. Rongyi Zhu, Ali Vosoughi, Chao Huang, Zeliang Zhang, Pinxin Liu, Mingqian Feng, Feng Zheng, Jianguo Zhang, Ping Luo. Jiebo Luo, and Chenliang Xu. Video understanding with large language models: A survey, 2024.
[63] Ultralytics Team. Ultralytics document.
http:///dos..uulaaaale.e...o///s 2025.
[64] Xi Tian, Yong-Liang Yang, and Qi Wu. Script-to-storyboardd:: A new contextual retrieval dataset and benchmark. Computational Visual Media, 11(1):103-122, 2025.
[65] Imad Eddine Toubal, Aditya Avinash, Neil Gordon Alldrin, Jan Dlabal, Wenlei Zhou, Enming Luo, Otilia Stretcu, Hao Xiong, Chun-Ta Lu, Howard Zhou, Ranjay Krishna, Ariel Fuxman, and Tom Duerig. Modeling collaborator: Enabling subjective vision classification with minimal human effort via IIm tool-use, 2024.
[66] Naoki Wake, Atsushi Kanehira, Kazuhiro Sasabuchi, Jun Takamatsu, and Katsushi Ikeuchi. Opea-vocabulary temporal action localization using vims. arXiv e-prints, pages sxXiv22408 2024.
[67] Naoki Wake, Atsushi Kanehira, Kazuhiro Sasabuchi, Jun Takamatsu, and Katsushi Ikeuchi. vocabulayy action localization with iterative visual prompting. IEEE Access, 13:56908-566 -58888556177-11 2025.
[68] Haibo Wang. Zhiyang Xu, Yu Cheng, Shizhe Diao, Yufan Zhou, Yixin Cao, Qifan Wang, Weifeng Ge, and Lifu Huang, Grounded-videollm: Sharpening fine-graines temporal grounding in video large language models, 2024.
[69] Wii Wang and Li- Yong Shen. Spatiotemporal fusion transformer for video demoiréing. Computational Visual Media, 11(4):849-86) 2025.
14

---

[70] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu. Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision- language model's perception of the world at any resolution, 2024.
[71] Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video understanding with large language model as agent, 2024.
[72] Xijun Wang, Junbang Liang, Chun-Kai Wang, Kenan Deng, Yu Lou, Ming Lin, and Shan Yang. Vila: Efficient video-language alignment for video question answering, 2024.
[73] Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang. Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. Internvideo: General video foundation models via generative and discriminative learning, 2022.
[74] Ying Wang. Yanlai Yang, and Mengye Ren. Lifelongmemory: Leveraging Ilms for answering queries in long-form egocentric videos, 2024.
[75] Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Videotree: Adaptive tree-based video representation for Ilm reasoning on long videos, 2025.
[76] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thouul prompting elicits reasoning in large language models, 2023.
[77] Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. 2024.
[78] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa:next phase of question- answering explaining temporal actions, 2021.
[79] Haomin Yan, Ruize Han, Wei Feng, Jiewen Zhao, and Song Wang. Weakly supervised instance action recognition. Computational Visual Media, 11(3):603-60 618, 2025.
[80] Jianwei Yang. Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, 2023.
[81] Zeyuan Yang. Delin Chen, Xueyang Yu, Maohao Shen, and Chuang Gan. Vea: Video curious agent for long video understanding, 2025.
[82] Zongxin Yang, Guikun Chen, Xiaodi Li, Wenguan Wang, and Yi Yang. Doraemongpt: Toward understanding dynamic scenes with large language models (exemplified as a video agent). In ICML, 2024.
[83] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023.
[84] Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. moll: Towards long image-sequence understanding in multi-modal large language models, 2024.
[85] Jinhui Ye, Zihan Wang, Haosen Sun, Keshigeyan Chandrasegaran, Zane Durante, Cristobal Eyzaguirre, Yonatan Bisk, Juan Carlos Niebles. Ehsan Adeli, Li Fei-Fei, Jiajun Wu, and Manling Li. Re-thinking temporal search for long-form video understanding, 2025.
[86] Shoubin Yu, Jaemin Cho, Prateck Yadav, and Mohit Bansal. Self-chained image- languala model for video localization and question answering. 2023.
[87] Yuqian Yuan, Hang Zhang, Wentong Li, Zesen Cheng, Boqiang Zhang, Long Li, Xin Li, Deli Zhao, Wenqiao Zhang, Yueting Zhuang, Jianke Zhu, and Lidong Bing. Videorefer suite: Advancing spatial-temport object understanding with video 11m, 2025.
15

---

[88] Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng. Yuming Jiang. Hang Zhang, Xin Li, Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing. and Deli Zhao. Videollama 3: Frontier multimodal foundation models for image and video understanding, 2025.
[89] Ce Zhang, Taixi Lu, Md Mohaiminul Islam. Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A simple Ilm framework for long-range video question question-answerne answering. 2024.
[90] Lu Zhang, Tiancheng Zhao, Heting Ying, Yibo Ma, and Kyusong Lee. Omagent: A multi-modal agent framework for complex video understanding with task divide-and-conquer, 2024.
[91] Pengyu Zhang, Dong Wang, and Huchuan Lu. Multi-modal visual tracking: Review and experimental comparison. Computational Visual Media, 1022)193-214, 2024.
[92] Yilun Zhao, Lujing Xie, Haowei Zhang, Guo Gan, Yitao Long, Zhiyuan Hu, Tongyan Hu, Weiyuan Chen, Chuhan Li, Junyang Song, Zhijian Xu, Chengye Wang, Weifeng Pan, Ziyao Shangguan, Xiangru Tang. Zhenwen Liang, Yixin Liu, Chen Zhao, and Arman Cohan. Mmvu: Measuring expert-level multi-discipline video understanding, 2025.
[93] Jinguo Zhu, Weiyun Wang. Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuchui Wang. Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, Hao Jiahao Wang, Nianchen Deng, Songze Li, Yinan He, Tan Jiang, Jiapeng Luo, Yi Wang, Conghui He, Botian Shi, Xingcheng Zhang, Wenqi Shao, Junjun He, Yingtong Xiong, Wenwen Qu, Peng Sun, Penglong Jiao, Han Lv, Lijun Wu, Kaipeng Zhang, Huipeng Deng, Jiaye Ge. Kai Chen, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models, 2025.
[94] Yuchen Zhuang, Xiang Chen, Tong Yu, Saayan Mitra, Victor Bursztyn, Ryan A. Rossi, Somdeb Sarkhel, and Chao Zhang. Toolchain*: Efficient action space navigation in large language models with a search, 2023.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We believe the main claims the abstract and introduction (e.g. our work represents an important step towrds building autonomous and intelligent video analysis assistants) accurately reflect the paper's 's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of our work in the Conclusion part (Section 5 "Our main limitation lies in the fact that the full STAR framework still partially relies on the visual capabilities of GPT-4o, which may incur certain API costs."
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
17

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have provided all necessary details in the paper to reproduce the main ex- perimental results, including descriptions of the algorithm, framework design, datasets used, and evaluation protocols. Additionally, we will release our complete method implementation along with baseline reproduction scripts after publication, ensuring full reproducibility of our results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
18

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will release the code in compliance with the requirements, along with sufficient instructions to reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
poplic/guides. StaiLi//uuides/oedeuuuisssss
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https Sotteeeese//SSeSemmiaeee
wtt.submmmmssssooooocoo.... for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The core experimental settings are presented in the main text, while additional implementation details are included in the appendix and will be made fully available through the released code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
19

---

Justification: In our experiments, we fixed three arbitrary random seeds and reported the averaged results. We observed that randomness had negligible impact on the outcomes. Additionally, the responses from the proprietary (M)LLM were stored and reused. Therefore, we consider our experiments be stable and reproducible, with no significant sources of variability to report.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етгог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the main text, we specify the type of compute resources used: "Our STAR framework can run the NVIDIA RTX 4090 GPU, while the STAR-MINI framework can run on personal computers". The amount of compute required for each individual experiment varies depending the number of videos and the available resources. No compute resources beyond those reported the paper were used for the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpps:/neurrpp: ct/ httpppiiii/tiiicsiiiiieeeneeneee
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
20

---

The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work not directly tied any specific application or deployment, As such, there no immediate socictal impacts addressed in this paper. We acknowledge that while our methods may have potential future applications, the current research does not involve any direct societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Our work does not involve the release of models or datasets with high risks of misuse. However, we acknowledge that future work might involve the release of models datasets, and we would ensure necessary safeguards put in place, such as requiring usage guidelines, access restrictions, or safety filters.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided relcasing unsafe images.
21

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: In our paper, we have properly cited the original papers that provided the code packages and datasets used in our work. We have also ensured that the licenses and terms of use for these assets are fully respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, papprssithcoom thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
22

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Our paper does not involve the use of large language models (LLMs) in any important, original, or non-standard component of the core methods.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy httt:p://eeuripsccc/ooofftt for what should or should not be described.
23

---

Appendix
A
Tool Detail
A.1 Temporal Tools
Frame Selector Refer Section 3.1 of the main text. Specifically, we implement variants of the
frame selector for different computational resources.
Vanilla version: The frame selector is an LLM that takes as input the IDs and textual descriptions of all visible frames, such as captions generated by an image captioner. outputs the selected frame IDs. The prompt is directly instructing the LLM to select relevant frames.
AKeyS-inspired version: This variant shares the input and output format the Vanilla version but employs specially designed prompts [9] to encourage the LLM to consider task goals and scene transition.
T*-inspired version: In this case, the frame selector is an MILM, and its input additionally includes the actual images of visible frames. These images are further arranged into an image grid to facilitate selection [85]. The output remains the same.
The selected frame IDs produced by the frame selector are subsequently added the Visible Frame Dictionary.
Temporal Grounding Tool We adopt the most lightweight version of Grounded- Videol (with approximately 4B parameters), which augments the language model with temporal tokens and trained on large corpus of temporally annotated video-text pairs. Given an event description, the tool predicts its temporal span in the video.
Temporal Referring Tool For temporal referring, we use the same model as in temporal grounding, but with different input-output format. Instead of grounding described event in time, the tool takes a temporal span as input and generates a textual description the visual content within that interval.
Video Trimmer Video Trimmer is Python-based tool to trim the video along the temporal axis based on a temporal span.
Action Localization Tool We implement action localization tool based on the method proposed in [66]. The core idea is to organize the video into an image grid and feed it into GPT-4o [50], which then identifies the frame indices corresponding to specified action.
A.2 Spatial Tools
Object Detector
Refer to Section 3.1 of the main text.
Bbox Marker Following Set-of-Mark [8), we annotate the 2D Regions Interest (RoIs) on corresponding frames, which enhances the effectiveness of visual question answering by MLLMs such as GPT-4o.
Image Captioner We employ three Vision-Languee Models (VLMs) of different scales to perform the image captioning task: BLIP2 (about IB parameters), LLaVA (about 7B parameters), and GPT-4o. The input this tool is individual video frames, and the output is textual description of each frame. These descriptions are stored and maintained in the Visible Frame Dictionary.
Image QA Tool We use the same models as those the image captioning tool, but the input now consists of images paired with corresponding questions. and the output is textual answers. These answers are stored and maintained in the Visible Frame Dictionary.
Text Detector
We an off- the-hhh-shlff OCR tool recognize text in video frames.

---

Following Octotools 42]. this tool is used to enlarge the Rol in an image and crop irrelevant areas before further processing.
Relevant Patch Zoomer
Semantic Segmentation Tool We use Grounded-SAl
55 model for semantic segmentation.
A.3 Tools that can be regarded as both temporal and spatial tools.
When faced with questions that exceed internal world knowledge, the L.L Planner can invoke the Google Search API to retrieve relevant external information, thereby enabling knowledge-enhanced VideoQA.
Google Search
Object Identifier Inspired by T 85). this tool is designed identify key objects mentioned in the question, serving as a preparatory step for subsequent object detection and tracking operations.
Action Recognition Tool Action recognition involves understanding both the temporal dynamics and spatial cues of a video. We implement our action recognition tool based on MMAction [47) enabling the identification of target actions within the video.
we organize the visible frames of the video into an image grid, which is then treated as a single image for subsequent image question answering.
Image Grid QA Tool Following (85],
Multiple Image QA Tool Instead of creating an image grid, we feed the visible video frames sequentially as individual images into the MI.LM model for question answering.
Python Code Generator This tool can generate and excute any Python code to manipulate videos.
Object Tracker
The object tracker can track specified objects in videos and includes person re-identification function. Our implementation is based on Ultralytics (63
Video Summarizer This tool employ a video-language model (Qwen2.5-VL-3B (2) to perform captioning or summarization the entire video, generating textual descriptions or summarizations.
A.4 Gerprel-Purpose Tools
Text Summarizer
The textual outputs from each tool are maintained in the Visible Frame Dictionary for the corresponding frames. This tool is responsible for summarizing the information and attempting to answer the question.
Video QA Tool This tool leverages a video-language model (Qwen2.5-VL-33 ( to directly perform video question answering, generating answers based on the input question and video content.
B Datasets
VideoMME VideoMME 12] is the most widely adopted benchmark for evaluating video anal- ysis capabilities. It is test-only dataset, containing 2,700 multiple-choice VideoQA examples. VideoMME is known for its diversity and comprehensive coverage across video lengths, video do- mains, and question types. In terms of length, the dataset categorizes videos into short (<2 minutes), medium (4-15 minutes), and long (30-60 minutes) segments. Regarding video types, VideoMME spans nearly all categories commonly found on the internet, grouped into six domains: Knowledge, Film & Television, Sports Competition, Artistic Performance, Life Record, and Multilingual. Due to its broad coverage, VideoMME serves an effective benchmark assess whether VidcoQA system can perform robustly across different video lengths and domains, thus providing solid measure of the system's 's generality and transferability.
NEXT-QA NEXT-QA 78 dataset comprises 5,440 videos and approximately 52,000 human- annotated QA pairs. It is specifically curated benchmark models' capabilities understanding the causal structures and temporal dependencies inherent in complex video events. In this work,
25

---

adopt the multiple-choiee choice question answering subset of NEXT-QA, which includes 34K training, 5K validation, and 9K testing samples. A notable characteristic NEXT-QA its diverse question formulations, spanning various interrogatives including how, who, when, where, what, and why, thereby providing a comprehensive testhed for evaluating a model's adaptability to different query types. The questions are further categorized into three classes: causal questions that probe reasoning over cause-effect relations, temporal questions that assess understanding of event chronology. and descriptive questions that focus on visual and contextual details. Additionally, the videos in NEXT-QA are relatively short, with an average duration of approximately 0.7 minutes.
Long Becc Long VideoBench 77 is specifically designed to evaluate long-form video understanding, with videos averaging around minutes in duration. Analogous to the "needle-in-a haystack" challenge [46], Long VideoBench introduces referring reasoning questions, where each query references specific video contexts guide the answering process. This formulation imposes greater demands on models' temporal reasoning capabilities. In this work, we evaluate our method on the validation split of LongVine VideoBench, which contains 1,337 questions. For fair comparison, we exclude the use of subtitle information provided by Long VideoBench and focus solely on video content understanding.
EgoSchema
The EgoSchema 4
dataset includes over 5,000 carefully curated multiple-choice
question-answer pairs, making it prominent benchmark for long-form video question answering. EgoSchema stands out for its challenging nature- nature-human performance reaches only 76% accu- racy, while current Video-LLMs fall below 70%. The dataset's 's extended video durations and high complexity highlight the critical need for key information retrieval.
C Baselines
Our evaluation focuses comparison between our method and the following four baseline approaches.
Image-based MLLMs Image-based MLLMs address VideoQA tasks by converting videos into image sequences and guiding the models with carefully designed prompts. Thanks to their strong general capabilities, models like GPT-4o (50) and Gemini-2.55PPr [16] have achieved impressive results on video question answering benchmarks such as VideoMME 12. However, this approach has several limitations: (1) the best-performing
mmage-based MILMS are mostly proprietary and expensive to access via APIs, especially when handling long videos with extended context; (2) as discussed above, MLLMs exhibit limited spatiotemporal reasoning abilities, which require specialized lightweight tools for effective augmentation; and (3) their performance tends to degrade when the number of input frames is insufficient.
LLMs Video-LLMs are specialized class of MLLMs designed for video-language under- standing tasks. These models either incorporate video-specific architectural designs 32 87 are
trained on large-scale video-text datasets (72), or include temporal modeling optimizations B1 68)
In this study, we primarily examine the following representative Video- Qwen- Owen-VL 2 In-
VI [93] and VideoI.LaMA [8)) Similar
general I.LMs, Video-LLMMs come in various parameter scales; in our experiments, we focus on models around 7B and 72B parameters. Our goal for the STAR framework to outperform the 7B models and approach the performance of the 72B models. The performance of Video-LLMMs is also influenced by the number of input frames provided.
(M)LLM-basse Frame Selection Methods Compared with this line of methods, our primary goal highlight the complementary role of our video toolkit in augmenting MLLMs. We include several representative works in this category. VideoAgent (71) first performs sparse and uniform sampling over the video and then leverages an LLM dive deeper into important segments. VideoTree [75] builds a video tree structure guided by CLIP based on the given question, followed by tree-base based search using an LLM. LVNet (5I) similarly constructs a hierarchical keyframe selector with CLIP, selecting key frames before feeding them into an MILM for processing. VidF4 (36] proposes differentiable adaptive frame sampling mechanism to enable end-to-end training of the frame selector.
T (85) and AKeyS 9 directly integrated as tools within our video toolkit, as described earlier in
26

---

the Method section. We mainly compare our approach against these methods on the NExT-QA 78 dataset.
LLM-driven Tool Learning Methods
Several recent works have begun integrating tool learning
60 analyzes images and videos by generating and exe- incorporates various perception tools, such Intern- Video [73), Whisper [53], Tag2Texx 12 to perform multi-modal video analysis. Among them, DoraemonGPT [82 serves
into video understanding. ViperGPT
cuting Python programs. VideoChat
22
most important reference. It leverages tools like object trackers and BLIP (8 to pre-process videos, constructing space-dominant and time-dominant memory and then employs text-to-SQI. querying tool retrieve answers from the memory. It uses Monte Carlo Tree Search (MCTS) to search for the best toolchain. To ensure fair comparison, we built STAR-MINI using tools of comparable scale, and conducted experiments on the NExT-QA dataset against these methods.
D Ablation on each tool
We conducted ablation studies on each individual tool, demonstrating that every tool contributes positively to the overall performance. Keeping all other conditions unchanged, we remove a specific tool from the Video Toolkit and evaluate how its removal affects the accuracy of our method and the number of video frames processed. We conduct experiments on VideoMME [[2] and I Long Bech [77], recording the accuracy drop and frame increase caused by removing each individual tool, as shown Table We highlight the largest change within each tool category in bold, indicating the relative importance of that tool within the toolkit.
We observe that, in most cases, removing tool leads drop in accuracy and an increase in the number of processed frames. In a few cases, tools that require many frames for computation result in reduced accuracy but fewer frames when removed. This indicates that nearly all tools contribute positively to the overall performance of our method.
E
Scalability with more frames
We conducted experiments to evaluate the impact of increased frame sampling rates. Due to API budget and time constraints, we randomly sampled 1,000 examples from the Video-MME test set for this evaluation. The accuracy results are show in Table 7 They demonstrates that the STAR framework continues to improve performance as the number of sampled frames increases. For example, under a denser sampling setting (1 fps, up to 384 frames), the framework achieves 5.2% accuracy gain.
Generalizability with different base LLMs
We also examined the performance the STAR framework when using different LLMs as the LLM Planner. The results in Table 8 show that, compared to their tool-free counterparts, these models generally demonstrate 7% improvement in accuracy, highlighting STAR effectiveness across model types.
G Analysis on Tool Balance
The STAR framework addresses the issue of unbalanced tool quantity and diversity primarily by removing the tools that the LLM Planner tends to over-rely on under the no-constraints setting (e.g.. the Video QA Tool and Image QA Tool). With these tools longer "monopolizing" the calls, the distribution tool usage becomes more balanced both across and within tool categories.
In Table we calculated the percentage of tool usage frequency on the Video-MME test set for both the No Constraints setting and the STAR framework. Based on Table we compute the variance of tool usage counts within each tool category in Table [Q] It can be observed that after adopting the STAR framework, the variance of tool usage counts within each category generally decreases, indicating more balanced utilization of tools within each class.
27

---

Table Performance Change on VideoMME and Long BideeBeenn after Removing Each Tool
Tool Removed
VideoMME
Long VideoBenhn
Acc Drop
Frames Increase
Ace Drop
Frames Increase
Temporal Tools
Frame Selector
4.6
14.3
3.4
15.6
Temporal Grounding Tool
0,9
3.1
0,6
2.3
Temporal Referring Tool
0.8
2.9
0.3
1.2
Video Trimmer
0.8
11.5
0.9
7.2
Action Localization Tool
0.6
1.2
0.2
0.7
Spatial Tools
Object Detector
1.4
3.5
1.5
4.4
Bbox Marker
0.8
2.1
0.2
1.0
Image Captioner
1.3
2.4
1.6
3.5
Image QA Tool
1.5
5.5
1.2
3.5
Text Detector
0.4
0.3
1.1
2.1
Relevant Patch Zoomer
1.0
2.3
0.8
2.8
Semantic Segmentation Tool
0.3
0.7
0.1
0.4
Both Spatial and Temporal Tools
Google Search
0.3
0.5
0.2
0.4
Object Identifier
0.5
1.1
0.6
1.3
Action Recognition Tool
0.3
0.9
0.2
0.5
Image Grid QA Tool
1.2
6.8
1.5
7.8
Multiple Image QA Tool
0.9
-0.1
1.0
-0.4
Python Code Generator
0.2
0.0
0.0
0.3
Object Tracker
0.3
0.1
0.2
0.4
General Tools
Text Summarizer
0.9
0.2
1.3
1.1
Video Summarizer
1.0
-0.3
0.7
-0.2
Video QA Tool
2.3
-0.2
1.1
-0.4
Table 7: Performance with Different Frame Sampling Rates
Model
LLM Planner
Avg. Frames
Short (%)
Medium (%)
Long (%)
All (%)
GPT-40
32
68.6
60.6
56.0
61.5
GPT-4o
100
72.8
63.2
59.5
64.9
GPT-4o
fps 384
79.2
70.4
66.5
71.8
STAR
GPT-4o
31.3
78.2
68.7
62.7
69.6
(+8.1% %)
STAR
GPT-4o
100.2
80.1
71.0
66.4
72.4
(+7.5%)
STAR
GPT-4o
0.98 fps 384
85.3
78.0
68.5
77.0 (+5.2%
H
Failure Cases
We analyzed the failure cases of the STAR framework and categorized the common ones into three major types:
Missing or ambiguous visual information. In some cases, the video alone does not provide sufficient clarity and must be supplemented with subtitles audio Video-MME test set question 302-1, an arrow points to a character labeled "Victor Garber". However, this label does not imply that the character is Victor Garber; instead, it means that the actor Victor Garber plays this character. Understanding this requires reference to the narration. We plan address such issues by incorporating subtitle inputs and developing tools for audio understanding in the future work.
For example, in
Incomplete understanding of the video's main theme. Our STAR framework sometimes fails to fully capture the primary focus of a video due to sparse frame sampling. For instance,
28

---

Table 8: Performance Comparison across Different LLM Planners
Model
LLM Planner
Avg. Frames
Short (%)
Medium (%)
Long (%)
All (%)
GPT-4o 50
32
68.6
60.6
56.0
61.5
Gemini- 2.5-pro
57.1
65.4
Qwen2.5-V1.7225
68.9
58.8
55.4
60.8
STAR
GPT-4o
78.2
69.6 (+8.1 f)
STAR
Gemini-2.5-pro
31.0
79.8
70.7
69.1
72.9 (+7.5 f)
STAR
Qwen2.5-VLL-
31.5
77.9
66.1
62.4
68.5 (+7.7 f)
STAR
DeepSeek-RI R1
17
31.2
77.2
67.2
63.0
68.9
Table 9: Tool Usage Frequency Comparison between No Constraints and STAR Framework
Tool Type Name
No Constraints (%)
STAR Framework (%)
Temporal Tools All
32.1
35.7
Frame Selector
27.1
21.3
Temporal Grounding
2.1
8.2
Temporal Referring
0.0
1.4
Video Trimmer
1.3
2.6
Action Localization
1.6
2.2
Spatial Tools All
23.6
33.1
Object Detector
2.3
10.2
Bbox Marker
0.2
1.3
Image Captioner
2.1
7.9
Image QA
17.8
5,6
Text Detector
1.0
2.5
Relevant Patch Zoomer
0.2
3.7
Semantic Segmentation
0.0
1.9
Both (Temporal and Spatial Tools) All
5,4
16.1
Google Scarch
0.3
1.3
Object Identifier
1.0
3.6
Action Recognition
0.5
1.4
Image Grid QA
0.1
5.7
Multiple Image QA
2.8
1.2
Python_Code Code_ Generator
0.0
1.4
Object Tracker
0.7
1.5
General Tools All
38.9
15.1
Text Summarizer
2.2
8.3
Video Summarizer
3.5
2.1
Video QA
33.2
4.7
in Video-MME test set question 303-3, the question is "What is the primary focus this video for the audience?" but the sampled frames did not sufficiently reflect the video's theme. We aim to improve performance on such global reasoning tasks by encouraging denser sampling in future versions.
Difficulty in inferring underlying motivations behind human actions. STAR struggles with questions that require understanding the deeper intent behind actions. For example, in Video-MME test set question 312-2, the question "Why does Michael Bierut mention religious symbols?" reveals challenges in modeling causal and temporal dependencies. This suggests that capturing fine-grained causal reasoning remains fundamental difficulty in video understanding.
I Case Study
4 and 6 present STAR's cases of diverse question types tackled by leveraging different tools from our Video Toolkit.
In Figures
29

---

Table 10: Variance Comparison of Tool Usage between No Constraints and STAR Framework
Tool Type
Var. of No Constraints
Var. of STAR Framework
Temporal Tools
134.26
69.90 (64.36
Spatial Tools
41.34
11.09 (30.25 ()
Both
0.92
2.95 (2.03
General Tools
307.45
9.69 (297.76 4)
Paradise
frees
this video, how red socks above the fireplace at the end of this video
Calling frame-selector seloct frames the end of this vidco.
Frame-telector:
7660;
Calling object-detector detect red socks for frame -66>
Q
Object-detector: red socks detected above the fireplace.
Figure 4: A case of counting problem from VidcoMME, mainly solved by object detector.
<1>
<155
<22>
<200
<36>
6430
<00>
6645
In this video, how is the smoke generated the man depicted in the video? Select from the below options: A. burning piece smoking. By lighting
By lighting
Calling temporal-gronee wrwaal-ggoonnnnooooio to ground
Temporul-groue chapuruulgrouudinggtoo.. Smoke appeurs from frame <8 <15>
Calling retion-reooaaition-oo to recognize the 's action from frame +8> to =1.-
Action-recopee recognition-toor Frame < No action recognized. Frame <12>:The man lighting fire. Frame <15s; No action recognized.
Calling frame-selector re-select keyframes.
Frame-sclector: Frame <1 selected.
Frame <12
Calling image-Q A-ttool
Question:
Image-QA-tool:: Answer: The man set fire to heap of wood.
Calling Text Summarizer.
Summurizer: lights
Figure A case of action recognition problem from VideoMME.
30

---

100 WORD wot
+12
+315
<815
What's 's the trigger that set off the war mentioned in the video? Select from the below options: Milirarisim. assasination
King England's
D. Germany's is blitzkrieg against Poland.
Calling Imuge-grid QA-tool.
image sequence?
X
Image-grid-OA. A-tool: Use trame [1><11>=21>>> Jooof 1 1 1-d6 (--011><1-08813 to form a 3*3 image grid.
Answer: image sequence
World
Calling goegle-search for the trigger World War
Google-search:
trigger World
Archduke Fredinand.
The answer is B.
Figure 6: A case of action reasoning problem from VideoMME.
31


Reviews:
[{"review_id": "WNlLrm6aXO", "reviewer": "Reviewer_QhNP", "summary": "This paper presents STAR, a tool-augmented spatiotemporal reasoning framework for video understanding. The authors construct a diverse set of spatial tools, temporal tools and general tools, then use the STAR algorithm to coordinate all tools for video understanding with an LLM as the controller. To prevent toolchain shortcut, the authors also design a spatiotemporal interleave tool calling strategy. Extensive experiments on Video-MME and LongVideoBench show strong performance of the proposed method.", "questions": "Please see the ‘Weaknesses’ section. Overall, the reviewer thinks that the method is intuitive, interesting, simple and works well. However, it would be good if the authors could clarify the implementation details and show that the proposed method also performs well with more frames. The reviewer will consider raising the score if the above weaknesses are solved.", "limitations": "It would be good to provide some failure case analysis to help people understand more about the limitations of the proposed method.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- Overall, the flow of paper writing is clear and easy-to-follow.\n- The proposed method is simple, intuitive, training-free, yet works well. Experiments on VideoMME and LongVideoBench show that the proposed method outperforms multiple baselines with a smaller runtime.\n- The qualitative examples are clear, intuitive, and can help people understand the method.\n\nWeaknesses:\n\n- Some important implementation details are not clearly presented.\n    - What is the controller LLM?\n    - How many frames are sampled initially?\n    - In Table 1, how is the runtime calculated? What GPUs did the authors use? Is the comparison with other methods apple-to-apple? Why are some runtimes very vague? For example, the runtime of GPT-4o is presented as <30s, but the runtime of the proposed method which uses GPT-4o is clearly presented as 15.8s.  Without standardized evaluation conditions and transparent reporting, it is challenging to assess the actual efficiency advantage of the proposed method over the baselines.\n    - The introduction to the frame selector tool is not clear. Is the frame selector tool an LLM taking captions as input and output frame IDs, or video trimming, or frame selection methods as AKeyS and T*?\n- It is good that the proposed method outperforms GPT-4o, with both methods taking ~30 frames as input. However, the toolset of the proposed method includes GPT-4o itself. This raises the question: how does the proposed method perform when provided with more frames, such as in GPT-4o’s 1 fps / 384 setting? It would significantly strengthen the paper if the authors could show that their method continues to outperform GPT-4o even when both are using the same number of frames (e.g., 1 fps / 384).\n- As the authors mention, one of the motivation is to solve the ‘Unbalanced number and diversity of tools’ issue. However, it is not clear how exactly the authors solve this issue. From the reviewer’s understanding, the spatiotemporal interleave tool calling strategy can balance the spatial tools and temporal tools, but what about the tools within the same categories?\n- It would be good to provide some failure case analysis to help people understand more about the limitations of the proposed method.", "was_revised": true, "final_justification": "Thanks for the rebuttal. My main concerns are well addressed, especially for the missing implementation details and the performance of the proposed method with more frames. The method is interesting, well-motivated and works well on many datasets. Therefore, I will raise my score to 5. I hope that the authors add the missing implementation details and the new experimental results to the final paper.", "rebuttal": "We sincerely thank you for your constructive feedback, which greatly helps us improve the quality of this work. Below, we provide point-by-point responses for your concerns:\n\n##  Weakness 1\n\n> Some missing implementation details\n\n+ For STAR framework，we use **gpt-4o-2024-08-06** as the controller LLM. For STAR-mini, we use **gpt-3.5-turbo-0125** as the controller LLM to ensure a fair comparison with baselines. \n\n+ For videos longer than 16 seconds, we extract 16 initial frames uniformly; for videos with a duration of 16 seconds or less, initial frames are extracted at a rate of 1 fps.\n\n+ For proprietary models such as GPT-4o, we use their official APIs. For open-source models such as Qwen, we benchmark models up to 8B on 8 NVIDIA RTX 4090 GPUs with vLLM, and benchmark 72B models on 8 NVIDIA H100 GPUs with vLLM. For both prior work such as T* and our proposed STAR framework, experiments are conducted on 8 NVIDIA RTX 4090 GPUs using the same official APIs, ensuring a relatively fair comparison.\n\n  **Why are some runtimes very vague?** This is primarily due to three reasons: \n  1. The speed of API is affected by network conditions; \n  2. Even with the same GPU, different compute clusters may exhibit variations in speed due to overall hardware configuration; \n  3. Processing time is highly dependent on the number of frames.\n\n  Therefore, we report a performance range obtained from multiple runs of the baseline methods, rather than a specific runtime. For our own method, we conduct more extensive testing (about 10 runs) under different environments to obtain a more accurate runtime reported in the table. Our intention is to show: \n    1. STAR (averaging 15.8 seconds per video with 30.2 sampled frames) operates in the same time scale as models like GPT-4o (0–30 seconds for 32 sampled frames). \n    2. STAR achieves comparable accuracy to GPT-4o (>10 minutes for 384 sample frames) but completes processing much faster. \n\n+ Specifically, we implement 3 variants of the frame selector for different computational resources.\n\n  1. **Vanilla version**: The frame selector is an LLM that takes as input the IDs and textual descriptions of all visible frames, such as captions generated by an image captioner. It outputs the selected frame IDs. The prompt is directly instructing the LLM to select relevant frames.\n  2. **AKeyS-inspired version**: This variant shares the same input and output format as the Vanilla version but employs specially designed prompts to encourage the LLM to consider task goals and scene transition.\n  3. **T\\*-inspired version**: In this case, the frame selector is an MLLM, and its input additionally includes the actual images of visible frames. These images are further arranged into an image grid to facilitate selection. The output remains the same.\n\n  The selected frame IDs produced by the frame selector are subsequently added to the *Visible Frame Dictionary*. It is important to note that the *frame selector* and the *video trimmer* are two distinct and independent temporal tools, both scheduled by the LLM planner. We will revise and highlight this point in the later version.\n\n## Weakness 2\n\n> The performance with more frames \n\nWe conducted experiments to evaluate the impact of increased frame sampling rates. Due to API budget and time constraints, we randomly sampled 1,000 examples from the Video-MME test set for this evaluation. The accuracy results are as follows. They demonstrates that the STAR framework continues to improve performance as the number of sampled frames increases. For example, under a denser sampling setting (1 fps, up to 384 frames), the framework achieves a **5.2% accuracy gain**.\n\n| Model  | LLM Planner | Average Frames | Short (%) | Medium (%) | Long (%) | All (%)           |\n| ------ | ----------- | -------------- | --------- | ---------- | -------- | ----------------- |\n| GPT-4o | -           | 32             | 68.6      | 60.6       | 56.0     | 61.5              |\n| GPT-4o | -           | 100            | 72.8      | 63.2       | 59.5     | 64.9              |\n| GPT-4o | -           | 1 fps / 384    | 79.2      | 70.4       | 66.5     | 71.8              |\n| STAR   | GPT-4o      | 31.3           | 78.2      | 68.7       | 62.7     | 69.6 (8.1% ↑)     |\n| STAR   | GPT-4o      | 100.2          | 80.1      | 71.0       | 66.4     | 72.4 (7.5% ↑)     |\n| STAR   | GPT-4o      | 0.98 fps / 384 | **85.3**  | **78.0**   | **68.5** | **77.0** (5.2% ↑) |\n\n## Weakness 3\n\n> How STAR solves the \"unbalanced number and diversity of tools\" issue? How to balance tools within category?\n\n The STAR framework addresses the issue of unbalanced tool quantity and diversity primarily by **removing the tools that the LLM Planner tends to over-rely on** under the no-constraints setting (e.g., the Video QA Tool and Image QA Tool). With these tools no longer “monopolizing” the calls, the distribution of tool usage becomes more balanced both across and within tool categories.\n\n**Below are some supporting data:**\n\nBelow is the distribution of tool usage frequencies on the Video-MME test set for both the **No Constraints setting** and the **STAR framework**. The **No Constraints setting** setting is described in line 253-255 in the paper.\n\n| Tool Type & Name                          | No Constraints (%) | STAR framework (%) |\n| ----------------------------------------- | -------------- | -------------- |\n| **Temporal Tools All**                    | **32.1**       | **35.7**       |\n| Frame Selector                            | 27.1           | 21.3           |\n| Temporal Grounding                        | 2.1            | 8.2            |\n| Temporal Referring                        | 0.0            | 1.4            |\n| Video Trimmer                             | 1.3            | 2.6            |\n| Action Localization                       | 1.6            | 2.2            |\n| **Spatial Tools All**                     | **23.6**       | **33.1**       |\n| Object Detector                           | 2.3            | 10.2           |\n| Bbox Marker                               | 0.2            | 1.3            |\n| Image Captioner                           | 2.1            | 7.9            |\n| Image QA                                  | 17.8           | 5.6            |\n| Text Detector                             | 1.0            | 2.5            |\n| Relevant Patch Zoomer                     | 0.2            | 3.7            |\n| Semantic Segmentation                     | 0.0            | 1.9            |\n| **Both (Temporal and Spatial Tools) All** | **5.4**        | **16.1**       |\n| Google Search                             | 0.3            | 1.3            |\n| Object Identifier                         | 1.0            | 3.6            |\n| Action Recognition                        | 0.5            | 1.4            |\n| Image Grid QA                             | 0.1            | 5.7            |\n| Multiple Image QA                         | 2.8            | 1.2            |\n| Python_Code_Generator                     | 0.0            | 1.4            |\n| Object Tracker                            | 0.7            | 1.5            |\n| **General Tools All**                     | **38.9**       | **15.1**       |\n| Text Summarizer                           | 2.2            | 8.3            |\n| Video Summarizer                          | 3.5            | 2.1            |\n| Video QA                                  | 33.2           | 4.7            |\n\nFrom this raw data, we can compute the **variance of tool usage counts within each tool category**：\n\n| Tool Type      | Var. *of* No Constraints | Var. *of* STAR framework |\n| -------------- | ------------------------ | ------------------------ |\n| Temporal Tools | 134.26                   | 69.90 (64.36 ↓)                   |\n| Spatial Tools  | 41.34                    | 11.09 (30.25 ↓)                   |\n| Both           | 0.92                     | 2.95 (2.03 ↑)                    |\n| General Tools  | 307.45                   | 9.69 (297.76 ↓)                    |\n\n**It can be observed that after adopting the STAR framework, the variance of tool usage counts within each category generally decreases, indicating a more balanced utilization of tools within each class.**\n\nIn future work, we plan to further explore how to achieve balanced tool usage within the same category, potentially by manually annotating the ground-truth toolchains for each problem. This would allow us to analyze the ideal distribution of tool usage within categories and design appropriate constraints or incentives to guide the LLM planner toward balanced usage.\n\n## Weakness 4\n\n> Missing failure cases\n\n**We thank the reviewer for pointing out this limitation. In future versions, we will include representative failure cases along with an analysis of their underlying causes.**\n\nThe main types of failure cases are as follows:\n\n1. **Missing or ambiguous visual information.**\n   In some cases, the video alone does not provide sufficient information and must be supplemented with subtitles or audio cues, e.g., Video-MME test set 302-1. We plan to address such issues by incorporating subtitle inputs and developing tools for audio understanding.\n2. **Incomplete understanding of the video's main theme.**\n   Our STAR framework sometimes fails to fully capture the main theme of a video due to sparse frame sampling. We aim to improve performance on such global reasoning tasks by encouraging denser sampling in future versions.\n3. **Difficulty in inferring underlying motivations behind human actions.**\n   STAR struggles with questions that require understanding the deeper intent behind actions. For example, Video-MME test set 312-2: “Why does Michael Bierut mention religious symbols?” This suggests that capturing fine-grained causal reasoning remains a fundamental difficulty in video understanding."}, {"review_id": "uLfAHwBgja", "reviewer": "Reviewer_JAXG", "summary": "This paper introduces a plug-and-play Spatiotemporal Reasoning Framework (STAR), which addresses a critical limitation in existing Video Toolkit-augmented MLLMs: their inability to effectively integrate spatiotemporal information for complex, reasoning-intensive VideoQA. The proposed framework incorporates a novel tool scheduling strategy that optimizes the fusion of complementary spatiotemporal cues, enabling more robust understanding of 3D Regions of Interest (3D RoIs). Through extensive experiments, STAR demonstrates significant improvements over existing approaches, achieving an 8.2% gain on VideoMME [12] and a 4.6% gain on LongVideoBench [62] when augmenting GPT-4o with its lightweight toolkit.\nThe paper is well-structured, with clear and intuitive figures that enhance readability, reflecting the authors’ experience in rigorous scientific writing. The experimental validation is thorough, encompassing multiple video understanding benchmarks (eg.VideoMME) and diverse state-of-the-art models (eg. Open-source Video-LLMs), which collectively substantiate the framework’s effectiveness and generalizability. These results validate STAR’s capability to bridge the gap between spatial and temporal reasoning, offering a versatile and extensible solution for advanced video understanding.", "questions": "1. Experimental Comparisons (Time & Frame Processing)\n\n1)Tool Call Time Reduction: The results show that using tools is faster than raw GPT-4o processing. How is this possible?\nIs the tool execution time negligible (e.g., due to parallelization or caching)?\nWere any specific acceleration techniques applied (e.g., pre-filtering frames, early termination)?\n\n2)Reduction in Processed Frames: The decrease in processed frames is attributed to STAR’s selective reasoning, but:\nCould this also be influenced by additional textual information (e.g., metadata or captions) reducing the need for full-frame analysis?\nIf so, is the frame reduction truly due to spatiotemporal reasoning, or partly due to auxiliary inputs?\n\n2. Tool Quality & Harmony Assurance (Abstract vs. Implementation)\n\nThe abstract claims \"ensuring tool quality and harmony\", but the paper lacks details on how this is achieved.\nIs this solely reliant on the Planner + carefully crafted prompts, or are there additional mechanisms (e.g., tool validation, fallback strategies)? If prompt engineering is the key, how is robustness guaranteed across diverse queries?\nAre there quantitative metrics (e.g., success rate, error recovery cases) to support this claim?", "limitations": "This work appears to have minimal potential for negative societal impact.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\n\n1. The paper introduces a novel tool scheduling strategy that effectively optimizes the fusion of complementary spatiotemporal cues via interleaved spatial and temporal tool utilization.\n2. The proposed approach successfully mitigates the Toolchain Shortcut issue and enables more robust 3D Regions of Interest (3D RoIs) understanding, as validated by extensive benchmarking.\n\nWeaknesses：\n\nTypo：\n\n1. The text in Lines 40-46 appears somewhat abrupt. Please reorganize the logical flow for better coherence.\n2. Regarding \"(3rd place) Toolchain Shortcut\" in Figure 1: Could you clarify what this specifically refers to?\n3. For the comparison with DoraemonGPT in Line 95: Please provide more detailed explanations to substantiate the claims of superior reliability and efficiency.\n4. In Line 106: Should \"In the temporal dimension\" perhaps be \"In the spatiotemporal dimension\" instead? This would better reflect the comprehensive analysis being presented.\n\nCore Concerns and Questions:\n\n1. Toolkit Design:\n\nDoes the tool include a custom-designed toolkit (e.g., integrating multiple existing tools/models to form a specialized solution for a specific problem class), rather than relying solely on individual off-the-shelf models/tools?\n\n2. Frame Index Adjustment (Line 166): The statement “by adding or removing frame indices as needed” raises questions: When would frames be added? (Line 55 also mentions expansion—under what conditions?) How is it determined which frames to add and how many? (Removal is straightforward via index deletion, but addition logic is unclear.)\n\n3. Toolchain Shortcut Definition (Line 149): The definition “We define Toolchain Shortcut as a phenomenon…” prompts a concern:\nCould cases exist where a general tool alone suffices, but this method unnecessarily complicates the process? Is there a mechanism to distinguish between simple (single-step solvable) and complex (step-by-step toolchain required) problems?\n4. Spatiotemporal Interleaving (Line 178 & Fig. 2): The example in Fig. 2 shows temporal → spatial tool order. However，if a spatial tool (e.g., frame captioning) is called first, followed by temporal localization, would the system re-invoke spatial tools afterward? Is the temporal-first preference inherent to most problem types, or is the order flexible?\n5. Relation to Prior Work (Lines 184–189): How does the proposed STAR reasoning compare to prior methods that decompose queries stepwise to identify required tools/information? Are there key differences or improvements over existing query decomposition + tool retrieval approaches?\n6. Experiments (Toolkit Generalizability): The claim “we equip (M)LLMs with a carefully designed and comprehensive video toolkit” suggests broad applicability: Beyond GPT-4o, were other MLLMs tested with this toolkit? Could additional experiments (e.g., applying the toolkit to other MLLMs like LLaMA or Gemini) strengthen the generalizability argument?", "was_revised": true, "final_justification": "They have addressed my concern. I will maintain my score.", "rebuttal": "We sincerely thank you for your constructive feedback. We are especially grateful for your recognition. Below, we provide point-by-point responses for your concerns:\n\n## Weaknesses\n\n### Typos\n\n1. Thank you for your feedback. We will revise this paragraph to ensure greater logical coherence.\n2. The \"(3rd place) Toolchain Shortcut\" means that the *Toolchain Shortcut* method performs worst among the three approaches in figure 1. For a more detailed description and corresponding results, please refer to Section 4.2 and Table 5 in the main paper.\n3. DoraemonGPT use tools to processes all frames of a video to obtain information, which are then stored in a SQLite database. An LLM subsequently translates natural language queries into SQL statements to retrieve answers from the database. However, since DoraemonGPT operates on all video frames—unlike the STAR framework, which temporally narrows the search scope—**it is less efficient**. Moreover, due to limitations in text-to-SQL accuracy, **the generated SQL queries in DoraemonGPT sometimes fail to execute, compromising reliability**. In contrast, STAR avoids such failure-prone intermediate query steps.\n4. The original intent of line 106 is to emphasize that the Video Toolkit should be capable of handling both individual frames and video sub-clips—this is considerations along the **temporal dimension**. Correspondingly, along the **spatial dimension**, Video Toolkit should support operations on both full images and localized image patches. In the revised version, we will explicitly discuss both the temporal and spatial dimensions to reflect comprehensive analysis.\n\n### Core Concerns and Questions\n\n> 1. About custom-designed tools\n\n**Our current Video Toolkit does not yet include custom-designed tools. However, when develop the Video Toolkit, we consider various classic problem classes** such as counting, action reasoning, text reasoning problems, and so on. The Video Toolkit support the full solution pipeline needed to address these representative problems. In future work, we plan to develop custom-designed toolkits and solution pipelines, and integrate them with the LLM Planner’s autonomous tool-calling mechanism. \n\n> 2. When and How to add frames?\n\nThe following temporal tools, once invoked and executed, each apply their own selection logic to expand the set of visible frames. \n\n   + **Frame Selector** is an (M)LLM-based tool that, guided by single-turn or multi-turn prompting, analyzes the current Visible Frame Dictionary and identifies additional unseen frames that are likely to be informative for answering the question. These selected frames are then incorporated into the visible set to support further reasoning.\n\n   + **Temporal Grounding Tool** and **Action Localization Tool**: When invoked by the LLM Planner, the tool takes a textual description as input and grounds it to a corresponding temporal segment in the video. Frames from this segment are then sampled at 1 fps and added to the set of visible frames.\n\n> 3. Toolchain shortcut definition \n\nMore rigorously, *toolchain shortcut* refers to cases where **the toolchain invoked by the LLM Planner is shorter than the ground-truth toolchain.** The *ground-truth toolchain* represents the most computation-efficient sequence of tool calls required to solve the task. Cases where “a general tool alone suffices” should not be categorized as toolchain shortcuts. In the upcoming revision of this paper, we will manually annotate the ground-truth toolchains for the dataset and compare them with those generated by the LLM Planner, thereby enabling a more precise characterization of the toolchain shortcut phenomenon and examining whether there are cases that unnecessarily complicate the process.\n\n> 4. Spatialtemporal interleaving detail\n\nThe older is flexible as we mention in line 172. In other words, the first tool in the chain can be either a temporal or a spatial tool, but the subsequent tools are required to alternate between temporal and spatial tools.\n\n> 5. Comparison with \"query decomposition\" works\n\nThere are 2 main differences: \n\n   + Prior works [1, 2] about query decomposition is better suited for **complex multi-step reasoning tasks where each step requires a different tool**. For example, Chain-of-Action [1] focuses on retrieving real-time information from heterogeneous sources. In contrast, our approach focuses on **decomposing the process of locating key regions into a step-by-step procedure**, which is better suited for real-world long-form video QA tasks depending on accurate localization of key frames and regions. \n   + STAR determines which tool to invoke and what action to take step by step during the exploration of the video environment, rather than predefining the entire reasoning trajectory based solely on the query. This design makes the framework more flexible and dynamic.\n\n   In future work, we will further investigate this comparison and aim to validate it experimentally.\n  \n   References: \n\n   [1] Chain-of-action: Faithful and multimodal question answering through large language models.\n   \n   [2] ToolACE-DEV: Self-Improving Tool Learning via Decomposition and EVolution.\n\n> 6. Performance with other MLLMs\n\nDue to API budget and time constraints, we randomly sampled 1,000 examples from the Video-MME test set for this evaluation. The results show that, compared to their tool-free counterparts, **these models demonstrate a 7%–8% improvement in accuracy, highlighting the effectiveness across model types.**\n\n\n| Model          | LLM Planner    | Average Frames | Short (%)   | Medium (%)   | Long (%)    | All (%)               |\n| -------------- | -------------- | -------------- | -------- | -------- | -------- | ------------------ |\n| GPT-4o         | -              | 32             | 68.6     | 60.6     | 56.0     | 61.5               |\n| Gemini-2.5-pro | -              | 32             | 77.6     | 62.6     | 57.1     | 65.4               |\n| Qwen2.5-VL-72B | -              | 32             | 68.9     | 58.8     | 55.4     | 60.8               |\n| STAR           | GPT-4o         | 31.3           | 78.2     | 68.7     | 62.7     | 69.6 ( 8.1 ↑ ) |\n| STAR           | Gemini-2.5-pro | **31.0**       | **79.8** | **70.7** | **69.1** | **72.9** ( 7.5 ↑ ) |\n| STAR           | Qwen2.5-VL-72B | 31.5           | 77.9     | 66.1     | 62.4     | 68.5 ( 7.7 ↑ )     |\n| STAR           | DeepSeek-R1    | 31.2           | 77.2     | 67.2     | 63.0     | 68.9               |\n\n## Questions\n\n### Experimental Comparisons\n\n> 1. Why STAR faster than raw GPT-4o?\n\nThe main reason is that in our work, the video is not directly fed into GPT-4o. Instead, GPT-4o—serving as the LLM Planner—analyzes the question along with intermediate textual results stored in the Visible Frame Dictionary to decide which tool to invoke next. The LLM Planner does not process video directly. **Therefore, In our work avoids processing a large number of image tokens.**\n\n> 2. What cause the reduction in processed frames?\n\nWe believe the key lies in **selective reasoning**.\n\n- **Regarding metadata**: Our framework does not introduce any additional metadata—the input remains a single video and a question.\n- **Regarding captions**: If the LLM Planner invokes an tool (e.g., an image captioner) on a particular frame, that frame is counted as a *sampled/visible/processed* frame. Therefore, captions are not considered auxiliary inputs but rather *intermediate tool results*.\n\nAs such, the system **does not rely on auxiliary inputs**. However, it can be interpreted as using tools to extract *different kinds of information* from selected frames. This spatiotemporal reasoning approach enables the QA system to selectively fuse and utilize diverse information sources, allowing it to reduce the number of processed frames while maintaining high accuracy.\n\n### Tool Quality & Harmony Assurance\n\n> How to improve tool quality and harmony? How to measure?\n\nWe use the following methods to improve **tool quality** and ensure robust tool execution:\n\n- **Structured tool invocation**: We leverage LLM's (e.g., GPT-4o)  capability for structured output by requiring the LLM Planner to generate tool parameters in JSON format. These parameters are then parsed to ensure syntactic correctness.\n- **Time range check**: We ensures that temporal manipulations such as frame sampling or trimming are accurately bounded within the valid time range of the video, preventing out-of-bounds errors.\n- **Resilient workflow design**: The workflow is designed to be fault-tolerant. A single failed or ineffective tool invocation does not halt the process. For instance, if a spatial tool like an object detector fails to detect the target object, the planner may subsequently invoke a temporal tool to adjust frame sampling and continue exploration. \n- **Fallback General Tools**: We include general-purpose tools, such as the Video QA Tool and Video Summarizer based on video-language models, as a fallback mechanism. These tools serve as a safety net when specialized tools fail to yield sufficient information.\n\nIn terms of **metrics**, we adopt the following approaches to evaluate tool execution and usage quality:\n\n- **Tool Success Rate**: We measure the proportion of successful tool executions—defined as invocations where input parameters are correct and the tool returns valid outputs. On the Video-MME test set, the success rate is **97.2%**, and on the LongVideoBench test set, it is **96.5%**, indicating that tools are generally executed reliably.\n- **Tool Usage Distribution**: To assess the **balance and harmony** of tool usage, we analyze the distribution of tool calls across different tools and tool categories. This helps us evaluate whether certain tools are disproportionately relied upon, and whether the LLM Planner effectively utilizes the full range of available tools. The results of this metric can be found in Figure 3 of the main paper, as well as in our rebuttal to Reviewer QhNP, Weakness 3."}, {"review_id": "OL397UIbaB", "reviewer": "Reviewer_ah2Q", "summary": "When MLLMs do VideoQA, it is hard for the model to comprehend spatial relationships and temporal evolution. The author equips an extensible video toolkit that contains three types of tools: spatial tools, temporal tools, and general tools, on MLLMs to help MLLMs perform better on spatiotemporal reasoning. In addition, the author proposes the Spatiotemporal Reasoning Framework(STAR), which could arrange the use of “spatial tools” and “temporal tools” and controls the tool invocation sequence, called spatiotemporal-interleaved toolchain and uses tools with spatial localization capabilities, to compensate for the shortcomings of MLLMs in spatial localization. Moreover, STAR could also help avoid toolchain shortcut issues and find the key area in the video. This method performs well in multiple video question answering tasks(VideoQA).", "questions": "1. The paper mentioned that there are 30 tools in the toolkit, but only 22 are described in the appendix. Do the remaining tools appear to be omitted or described elsewhere?\n\n2. The method seems to be portable, but it seems tha the authors only tested it on GPT-4o. I'm interested to know how well it performs with other backbone models, especially open-source ones like Qwen or Llama. \n\n3. What if the model chooses the wrong time at the beginning? Can the model correct the error?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n- This paper is well written, with clear figures that help readers understand.\n- STAR is very portable, as the author designed the STAR mini that could run on a MAC. \n- The author compared STAR to various other methods, including no constraints, prompting, in-context learning, and spatiotemporal disentanglement, showing that STAR has the best accuracy and frame efficiency.\n- The authors tested STAR on various datasets, including VideoMME, NExT-QA, LongVideoBench, and EgoSchema. \n\nWeakness\n- All the tools are plug-and-play, which could lead to the model not being able to study from previous experiments. \n- For VideoMME, STAR gets a 70% accuracy; however, according to Zero-Shot Video Question Answer on Video-MME (https://paperswithcode.com/sota/zero-shot-video-question-answer-on-video-mme-1), there is already a model that gets an 81.3% accuracy, which is higher than the results claimed in the paper.", "was_revised": true, "final_justification": "Thanks for the responses. The responses have successfully addressed my concerns. Although the paper is not particularly interesting, it is solid and well-written, so I will raise my score to 5.", "rebuttal": "Thank you for your thoughtful and encouraging feedback. We are especially grateful for your recognition of STAR’s portability, superior performance across multiple methods, and strong results on diverse benchmarks with clear visualizations. The followings are points-to-points respones to your concerns.\n\n## Weaknesses\n\n### W1\n\n> Model not being able to learn from previous experiments\n\nWe thank the reviewer for pointing out this limitation. We acknowledge that **tool-integrated post-training** is an important direction and will leave it as **future work** to be explored in follow-up studies. \n\n   This paper focuses on a **training-free, plug-and-play** paradigm, where tools are dynamically selected and invoked without additional fine-tuning. In practice, we have found that current LLM such as GPT-4o already exhibit strong capabilities in instruction-following and zero-shot tool use. As a result, **training-free approaches** can achieve competitive performance without the need for additional fine-tuning. We believe this paradigm offers a complementary alternative to **tool-integrated post-training**, with each approach having its own trade-offs.\n\n### W2\n\n> Why some results on Video-MME is higher than 80%? Is STAR's 70% accuracy a good performance? Details about Video-MME evaluation\n\nThis is primarily due to two reasons:\n\n   (1) **Video-MME provides two evaluation modes: *with subtitles* and *without subtitles***. Models generally perform better in the *with subtitles* setting. For example, on the official Video-MME leaderboard, Gemini 1.5 Pro achieves 81.3% accuracy with subtitles, compared to 75.0% without subtitles. Our work currently focuses exclusively on pure video understanding, without leveraging subtitle information. **Therefore, in Table 1, we report results under the *without subtitles* setting**. In contrast, many entries on the PapersWithCode leaderboard either report *with subtitles* results or do not specify the evaluation setting, making direct comparison less reliable.\n\n   (2) **The accuracy of video question answering is closely related to the number of sampled frames**—higher sampling rates generally lead to better performance. Therefore, fair comparisons require evaluations under the same sampling rate. Some results reported on PapersWithCode may be obtained under higher sampling rates. \n\nTo ensure fairness, we also report our performance under increased sampling rates as follows. Due to API budget and time constraints, we randomly sampled 1,000 examples from the Video-MME test set for this evaluation. The following results demonstrate that the STAR framework continues to improve performance as the number of sampled frames increases and reach 77% performance under the *without subtitles, 1fps sampling, up to 384 frames* condition.\n\n| Model  | LLM Planner | Average Frames | Short (%) | Medium (%) | Long (%) | All (%)           |\n| ------ | ----------- | -------------- | --------- | ---------- | -------- | ----------------- |\n| GPT-4o | -           | 32             | 68.6      | 60.6       | 56.0     | 61.5              |\n| GPT-4o |             | 100            | 72.8      | 63.2       | 59.5     | 64.9              |\n| GPT-4o | -           | 1 fps / 384    | 79.2      | 70.4       | 66.5     | 71.8              |\n| STAR   | GPT-4o      | 31.3           | 78.2      | 68.7       | 62.7     | 69.6 (8.1% ↑)     |\n| STAR   | GPT-4o      | 100.2          | 80.1      | 71.0       | 66.4     | 72.4 (7.5% ↑)     |\n| STAR   | GPT-4o      | 0.98 fps / 384 | **85.3**  | **78.0**   | **68.5** | **77.0** (5.2% ↑) |\n\n## Questions\n\n### Q1\n\n> Tool number discrenpency between the main paper and appendix\n\nThe number *22* is the number of tool types, while *30* is the number of tool implementations. A tool may have multiple implementations to accommodate different computational resources or deployment environments. For example, the **Frame Selector** has 3 variants: \n\n   + **Vanilla version**: The frame selector is an LLM that takes as input the IDs and textual descriptions of all visible frames (i.e., the *Visible Frame Dictionary* described in the paper), such as captions generated by an image captioner. It outputs the selected frame IDs. The prompt is directly instructing the LLM to select relevant frames.\n\n   + **AKeyS-inspired version**: This variant shares the same input and output format as the Vanilla version but employs specially designed prompts to encourage the LLM to consider task goals and scene segmentation.\n\n   + **T\\*-inspired version**: In this case, the frame selector is an MLLM, and its input additionally includes the actual images of visible frames. These images are further arranged into an image grid to facilitate selection. The output remains the same.\n\n   Similarly, the **Object Detector** has both YOLO- and GroundingDINO-based versions tailored for low- and high-resource settings, respectively. The **Image Captioner** and **Image QA Tool** have different implementations based on BLIP2 (\\~1B parameters), LLaVA (\\~7B), and GPT-4o.\n\n   The number *30* mentioned in the paper refers to the total number of tool implementations. **We acknowledge that this may cause confusion and will revise in later version to clarify this distinction and eliminate ambiguity.**\n\n### Q2\n\n> Performance with other backbone models\n\nWe also evaluated our framework with other (M)LLMs, and the results are as follows. Due to API budget and time constraints, we randomly sampled 1,000 examples from the Video-MME test set for this evaluation.\n\n   | Model          | LLM Planner    | Average Frames | Short (%)    | Medium (%)   | Long (%)     | All (%)            |\n   | -------------- | -------------- | -------------- | -------- | -------- | -------- | ------------------ |\n   | GPT-4o         | -              | 32             | 68.6     | 60.6     | 56.0     | 61.5               |\n   | Gemini-2.5-pro | -              | 32             | 77.6     | 62.6     | 57.1     | 65.4               |\n   | Qwen2.5-VL-72B | -              | 32             | 68.9     | 58.8     | 55.4     | 60.8               |\n   | STAR           | GPT-4o         | 31.3           | 78.2     | 68.7     | 62.7     | 69.6 ( 8.1 ↑ ) |\n   | STAR           | Gemini-2.5-pro | **31.0**       | **79.8** | **70.7** | **69.1** | **72.9** ( 7.5 ↑ ) |\n   | STAR           | Qwen2.5-VL-72B | 31.5           | 77.9     | 66.1     | 62.4     | 68.5 ( 7.7 ↑ )     |\n   | STAR           | DeepSeek-R1    | 31.2           | 77.2     | 67.2     | 63.0     | 68.9               |\n\n   The results show that both closed-source models (e.g., GPT-4o, Gemini 2.5 Pro) and open-source models (e.g., Qwen2.5-VL-72B, DeepSeek-R1) achieve strong performance. Compared to their tool-free counterparts, these models demonstrate a **7%–8% improvement in accuracy**, highlighting the effectiveness of our work across model types.\n\n### Q3\n\n> Error correction after initial temporal misalignment\n\nIt is normal for our STAR framework to initially focus on an incorrect video segment. However, our STAR framework is equipped with backtracking ability that enables correction and prevents cascading errors. When the LLM Planner first invokes the frame selector and the Visible Frame Dictionary is still empty, the selector performs a coarse, uniform sampling over the entire video, dividing it into several segments. It then chooses to explore one segment in depth. **If, after comparing the question with the retrieved information, it turns out that the current segment is not relevant, the frame selector is able to shift its focus to other segments.** The exploration process terminates only when the LLM Planner has gathered sufficient information to answer the question accurately.\n\nFor example, in a case study from the Video-MME test set (`question_id=301-2`), the question is: *“Why can Cusco and Tenochtitlan be easily found while Amazonian settlements cannot?”*. The associated video is a YouTube video titled *“How the ‘lost cities’ of the Amazon were finally found”*. Initially, STAR selected frames from the segment between 1:20 and 1:40, which only mentioned Cusco and Tenochtitlan, but did not address why Amazonian settlements were hard to find. Detecting the lack of relevant information, STAR then shifted to another segment and ultimately identified the correct time span (6:10–6:20), which states: *“Because Amazonian settlements were built with wood and earth.”*"}, {"review_id": "TYsffnUZRi", "reviewer": "Reviewer_Fvz6", "summary": "To better control the tool invocation sequence and avoid toolchain shortcut issues, the authors propose a Spatiotemporal Reasoning Framework (STAR) that strategically schedules temporal and spatial tools, thereby progressively localizing the key area in the video. The STAR framework enhances GPT-4o using lightweight tools, achieving an 8.2% gain on VideoMME and 4.6% on LongVideoBench.", "questions": "See weakness.", "limitations": "See weakness.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "W1. Baseline performance concern: The reported VideoMME performance of Qwen2.5VL is 71.9. However, the result marked by the authors is 61.8, which shows a discrepancy of nearly 10 points. Could the authors please explain the reason for this difference, provide the exact prompt used, or alternatively test their method on the open-sourced Qwen or InternVideo models? Doing so would make the results more convincing.\n\nW2. In Figure 3, the categorization of the Video Summarizer differs from that in the supplementary materials. Could the authors clarify the reason for this inconsistency?\n\nW3. Regarding efficiency, the authors are encouraged to provide more detailed information about the missing Runtime values in the table, as well as clarify the rather vague description of \"<30s\".\n\nS: The use of tools is reasonable, with a well-defined and detailed categorization of tool types. Moreover, the performance improvement appears to be significant.", "was_revised": true, "final_justification": "Thanks for the rebuttal. I will maintain the original score.", "rebuttal": "We sincerely thank you for your selfless efforts and constructive feedback, which greatly helps us improve the quality of this work. We are especially grateful for the recognition of our categorization of tool types, as well as the performance improvement. Below, we provide point-by-point responses for your concerns.\n\n## W1\n\n> Discrepancy in reported Video-MME score and lack of prompt transparency\n\nIn Table 1, the score of 71.9 corresponds to GPT-4o evaluated with a sampling rate of 1 fps, with up to 384 frames sampled per video. The score of 61.8 corresponds to GPT-4o evaluated with a fixed sampling of 32 frames per video.\n\nThe accuracy of video question answering is closely tied to the number of sampled frames—higher sampling rates generally yield better performance. Therefore, fair comparisons should be conducted under comparable sampling conditions. Since our method samples an average of 30.2 frames per video, we compare against GPT-4o’s 32-frame setting (61.8) and mark 61.8 accordingly.\n\nThe sources and evaluation method of the results reported in Table 1 are as follows, all under the *without subtitles* setting:\n\n| Model             | Sample Rate / Num of sampled Frames | Accuracy (%) | Source or Eval Method                                        |\n| ----------------- | ----------------------------------- | -------- | ------------------------------------------------------------ |\n| GPT-4o            | 1 fps / 384                         | 71.9     | Video-MME official leaderboard                               |\n| GPT-4o            | 32                                  | 61.8     | self-eval by Video-MME recommended way (using `lmms-eval` Github repository) |\n| GPT-4o + T*       | 32                                  | 64.1     | T* paper Table 4                                             |\n| GPT-4v            | 10                                  | 60.0     | Video-MME official leaderboard                               |\n| Gemini 1.5 Pro    | 0.5 / 1 fps                         | 75.0     | Video-MME official leaderboard                               |\n| Claude 3.5 Sonnet | 20                                  | 60.9     | Video-MME official leaderboard                               |\n| Qwen2-VL-72B      | 2 fps / 768                         | 71.2     | Video-MME official leaderboard                               |\n| Qwen2-VL-7B       | -                                   | 63.3     | InternVL3 tech report Table 8                                |\n| Qwen2.5-VL-7B     | -                                   | 65.1     | InternVL3 tech report Table 8                                |\n| InternVL2.5-8B    | 64                                  | 64.2     | InternVL3 tech report Table 8                                |\n| InternVL3-8B      | 64                                  | 66.3     | InternVL3 tech report Table 8                                |\n| VideoLLaMA3       | 180                                 | 66.2     | Video-MME official leaderboard                               |\n| mPLUG-Owl3        | 128                                 | 59.3     | Video-MME official leaderboard                               |\n\nAll time measurements are conducted using either official APIs or open-source models, following the Video-MME recommended protocol (via the `lmms-eval` GitHub repository). Each measurement is repeated multiple times (at least 3) on 8 NVIDIA GeForce RTX 4090 GPUs.\n\n## W2\n\n> Categorization inconsistency of Video Summarizer between the main paper and the supplementary materials\n\n**There is a classification error in Figure 3: Video Summarizer should be categorized under *General Tool* rather than the *Both* category. The information provided in the appendix is correct.**\n\n*General Tools* are only used at the end of the toolchain, when neither temporal nor spatial tools can resolve the question. In contrast, *Both* tools can function as either temporal or spatial tools within the toolchain. The rationale for classifying **Video Summarizer** as a *General Tool* is that it is implemented using a 3B MLLM and is intended for use only as a last resort. In typical cases, we prioritize lightweight tools to improve computational efficiency. We will correct this classification in the later version.\n\n## W3\n\n> Lack of clarity in runtime reporting and missing efficiency details\n\n### 1. **About the missing Runtime values**\n\n  There is a strong positive correlation between runtime and the number of sampled frames. **The runtime value is missing for that particular data point in the table because it was taken from another paper that did not report the number of sampled frames, making it impossible to accurately estimate the runtime.**\n\n  This includes two data points in Table 1—**Qwen2-VL-7B** and **Qwen2.5-VL-7B**—which are sourced from Table 8 of the InternVL3 technical report, as well as two data points in Table 4—**ViperGPT** and **VideoChat**—which are sourced from Table 2 of the DoraemonGPT paper. \n\n  We thank the reviewer for pointing out this limitation. In the later version, we plan to replace these externally reported results with those obtained from our own reproduction, thereby eliminating the missing runtime values.\n\n### 2. **About the vague description of \"<30s\"**\n\n  **These runtime measurements are inherently difficult to obtain accurately, as they are sensitive to various environmental factors**: (1) The speed of API is affected by network conditions; (2) Even with the same GPU model, different compute clusters may exhibit variations in video processing speed due to factors such as CPU memory, system age, and overall hardware configuration. \n\n  Moreover, runtime is highly dependent on the number of frames, making fine-grained time comparisons meaningless when the number of frames differs slightly.\n\n  Therefore, we run each baseline experiment multiple times to obtain a rough performance range, e.g., <30s. Regardless of the interference factors, the timing of these methods generally falls within this range. For our own method, STAR, we conduct more extensive testing (around 10 runs) under different environments to obtain a more accurate average reported in the table, e.g. 15.8s. \n\n  **Our intention is to determine approximate performance ranges**—such as *<30s, 30s–60s, or >10min*—in scenarios where precise runtime measurement is challenging. Models falling within the same range can be considered to have comparable computational efficiency, as small differences within a range are unlikely to impact practical usage. \n\n  **Our intention is also not to underrepresent baseline performance**, but rather to show:\n  (1) STAR (averaging 15.8 seconds per video with 30.2 sampled frames) operates in the same time scale as models like GPT-4o (0–30 seconds for 32 sampled frames). \n  (2) STAR achieves comparable accuracy to GPT-4o (>10 minutes for 384 sample frames) but completes processing much faster. \n\n  **In future revisions, we will conduct additional experiments across diverse environments to obtain more precise runtime measurements wherever possible.**"}]

Author rebuttal:

Meta-review: The introduces the Spatiotemporal Reasoning Framework (STAR) and an accompanying Video Toolkit to enhance the video question answering capabilities of Multimodal Large Language Models (MLLMs), demonstrating strong performance gains. 

The initial reviews were positive but with notable concerns regarding implementation details, generalizability, and the definition of a "Toolchain Shortcut". In a comprehensive rebuttal, the authors successfully addressed these issues by providing detailed explanations of their methodology, presenting new experimental results showing the framework's scalability with more frames and its portability to other MLLMs like Gemini and Qwen. The AC agrees with the reviewers' opinions (4/5/4/5) and recommends acceptance.
Decision: Accept (poster)