Title: Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Task
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video Question Answering

Tool Learning

Visual Reasoning
Abstract: Video Question Answering (VideoQA) task serves as a critical playground for evaluating whether foundation models can effectively perceive, understand, and reason about dynamic real-world scenarios. However, existing Multimodal Large Language Models (MLLMs) struggle with simultaneously ensuring the ability to model spatial relationships between video frames and to understand the causal dynamics of temporal evolution on complex and reasoning-intensive VideoQA. In this work, we equip MLLM with a comprehensive and extensible Video Toolkit, to enhance MLLM’s spatiotemporal reasoning capabilities as well as guarantee the harmony between the quantity and diversity of tools. To better control the tool invocation sequence and avoid toolchain shortcut issues, we propose a Spatiotemporal Reasoning Framework (STAR) that strategically schedules temporal and spatial tools, thereby progressively localizing the key area in the video. Our STAR framework enhances GPT-4o using lightweight tools, achieving an 8.2% gain on VideoMME and 4.6% on LongVideoBench. We believe that our proposed Video Toolkit and STAR framework make an important step towards building autonomous and intelligent video analysis assistants. The code is publicly available at https://github.com/fansunqi/VideoTool.


Peer reviews:
comments: SUMMARY:
To better control the tool invocation sequence and avoid toolchain shortcut issues, the authors propose a Spatiotemporal Reasoning Framework (STAR) that strategically schedules temporal and spatial tools, thereby progressively localizing the key area in the video. The STAR framework enhances GPT-4o using lightweight tools, achieving an 8.2% gain on VideoMME and 4.6% on LongVideoBench.

STRENGTHS AND WEAKNESSES:
W1. Baseline performance concern: The reported VideoMME performance of Qwen2.5VL is 71.9. However, the result marked by the authors is 61.8, which shows a discrepancy of nearly 10 points. Could the authors please explain the reason for this difference, provide the exact prompt used, or alternatively test their method on the open-sourced Qwen or InternVideo models? Doing so would make the results more convincing.

W2. In Figure 3, the categorization of the Video Summarizer differs from that in the supplementary materials. Could the authors clarify the reason for this inconsistency?

W3. Regarding efficiency, the authors are encouraged to provide more detailed information about the missing Runtime values in the table, as well as clarify the rather vague description of "<30s".

S: The use of tools is reasonable, with a well-defined and detailed categorization of tool types. Moreover, the performance improvement appears to be significant.

QUESTIONS:
See weakness.

LIMITATIONS:
See weakness.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nWhen MLLMs do VideoQA, it is hard for the model to comprehend spatial relationships and temporal evolution. The author equips an extensible video toolkit that contains three types of tools: spatial tools, temporal tools, and general tools, on MLLMs to help MLLMs perform better on spatiotemporal reasoning. In addition, the author proposes the Spatiotemporal Reasoning Framework(STAR), which could arrange the use of “spatial tools” and “temporal tools” and controls the tool invocation sequence, called spatiotemporal-interleaved toolchain and uses tools with spatial localization capabilities, to compensate for the shortcomings of MLLMs in spatial localization. Moreover, STAR could also help avoid toolchain shortcut issues and find the key area in the video. This method performs well in multiple video question answering tasks(VideoQA).\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n- This paper is well written, with clear figures that help readers understand.\n- STAR is very portable, as the author designed the STAR mini that could run on a MAC. \n- The author compared STAR to various other methods, including no constraints, prompting, in-context learning, and spatiotemporal disentanglement, showing that STAR has the best accuracy and frame efficiency.\n- The authors tested STAR on various datasets, including VideoMME, NExT-QA, LongVideoBench, and EgoSchema. \n\nWeakness\n- All the tools are plug-and-play, which could lead to the model not being able to study from previous experiments. \n- For VideoMME, STAR gets a 70% accuracy; however, according to Zero-Shot Video Question Answer on Video-MME (https://paperswithcode.com/sota/zero-shot-video-question-answer-on-video-mme-1), there is already a model that gets an 81.3% accuracy, which is higher than the results claimed in the paper.\n\nQUESTIONS:\n1. The paper mentioned that there are 30 tools in the toolkit, but only 22 are described in the appendix. Do the remaining tools appear to be omitted or described elsewhere?\n\n2. The method seems to be portable, but it seems tha the authors only tested it on GPT-4o. I'm interested to know how well it performs with other backbone models, especially open-source ones like Qwen or Llama. \n\n3. What if the model chooses the wrong time at the beginning? Can the model correct the error?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a plug-and-play Spatiotemporal Reasoning Framework (STAR), which addresses a critical limitation in existing Video Toolkit-augmented MLLMs: their inability to effectively integrate spatiotemporal information for complex, reasoning-intensive VideoQA. The proposed framework incorporates a novel tool scheduling strategy that optimizes the fusion of complementary spatiotemporal cues, enabling more robust understanding of 3D Regions of Interest (3D RoIs). Through extensive experiments, STAR demonstrates significant improvements over existing approaches, achieving an 8.2% gain on VideoMME [12] and a 4.6% gain on LongVideoBench [62] when augmenting GPT-4o with its lightweight toolkit.\nThe paper is well-structured, with clear and intuitive figures that enhance readability, reflecting the authors’ experience in rigorous scientific writing. The experimental validation is thorough, encompassing multiple video understanding benchmarks (eg.VideoMME) and diverse state-of-the-art models (eg. Open-source Video-LLMs), which collectively substantiate the framework’s effectiveness and generalizability. These results validate STAR’s capability to bridge the gap between spatial and temporal reasoning, offering a versatile and extensible solution for advanced video understanding.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n\n1. The paper introduces a novel tool scheduling strategy that effectively optimizes the fusion of complementary spatiotemporal cues via interleaved spatial and temporal tool utilization.\n2. The proposed approach successfully mitigates the Toolchain Shortcut issue and enables more robust 3D Regions of Interest (3D RoIs) understanding, as validated by extensive benchmarking.\n\nWeaknesses：\n\nTypo：\n\n1. The text in Lines 40-46 appears somewhat abrupt. Please reorganize the logical flow for better coherence.\n2. Regarding \"(3rd place) Toolchain Shortcut\" in Figure 1: Could you clarify what this specifically refers to?\n3. For the comparison with DoraemonGPT in Line 95: Please provide more detailed explanations to substantiate the claims of superior reliability and efficiency.\n4. In Line 106: Should \"In the temporal dimension\" perhaps be \"In the spatiotemporal dimension\" instead? This would better reflect the comprehensive analysis being presented.\n\nCore Concerns and Questions:\n\n1. Toolkit Design:\n\nDoes the tool include a custom-designed toolkit (e.g., integrating multiple existing tools/models to form a specialized solution for a specific problem class), rather than relying solely on individual off-the-shelf models/tools?\n\n2. Frame Index Adjustment (Line 166): The statement “by adding or removing frame indices as needed” raises questions: When would frames be added? (Line 55 also mentions expansion—under what conditions?) How is it determined which frames to add and how many? (Removal is straightforward via index deletion, but addition logic is unclear.)\n\n3. Toolchain Shortcut Definition (Line 149): The definition “We define Toolchain Shortcut as a phenomenon…” prompts a concern:\nCould cases exist where a general tool alone suffices, but this method unnecessarily complicates the process? Is there a mechanism to distinguish between simple (single-step solvable) and complex (step-by-step toolchain required) problems?\n4. Spatiotemporal Interleaving (Line 178 & Fig. 2): The example in Fig. 2 shows temporal → spatial tool order. However，if a spatial tool (e.g., frame captioning) is called first, followed by temporal localization, would the system re-invoke spatial tools afterward? Is the temporal-first preference inherent to most problem types, or is the order flexible?\n5. Relation to Prior Work (Lines 184–189): How does the proposed STAR reasoning compare to prior methods that decompose queries stepwise to identify required tools/information? Are there key differences or improvements over existing query decomposition + tool retrieval approaches?\n6. Experiments (Toolkit Generalizability): The claim “we equip (M)LLMs with a carefully designed and comprehensive video toolkit” suggests broad applicability: Beyond GPT-4o, were other MLLMs tested with this toolkit? Could additional experiments (e.g., applying the toolkit to other MLLMs like LLaMA or Gemini) strengthen the generalizability argument?\n\nQUESTIONS:\n1. Experimental Comparisons (Time & Frame Processing)\n\n1)Tool Call Time Reduction: The results show that using tools is faster than raw GPT-4o processing. How is this possible?\nIs the tool execution time negligible (e.g., due to parallelization or caching)?\nWere any specific acceleration techniques applied (e.g., pre-filtering frames, early termination)?\n\n2)Reduction in Processed Frames: The decrease in processed frames is attributed to STAR’s selective reasoning, but:\nCould this also be influenced by additional textual information (e.g., metadata or captions) reducing the need for full-frame analysis?\nIf so, is the frame reduction truly due to spatiotemporal reasoning, or partly due to auxiliary inputs?\n\n2. Tool Quality & Harmony Assurance (Abstract vs. Implementation)\n\nThe abstract claims \"ensuring tool quality and harmony\", but the paper lacks details on how this is achieved.\nIs this solely reliant on the Planner + carefully crafted prompts, or are there additional mechanisms (e.g., tool validation, fallback strategies)? If prompt engineering is the key, how is robustness guaranteed across diverse queries?\nAre there quantitative metrics (e.g., success rate, error recovery cases) to support this claim?\n\nLIMITATIONS:\nThis work appears to have minimal potential for negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents STAR, a tool-augmented spatiotemporal reasoning framework for video understanding. The authors construct a diverse set of spatial tools, temporal tools and general tools, then use the STAR algorithm to coordinate all tools for video understanding with an LLM as the controller. To prevent toolchain shortcut, the authors also design a spatiotemporal interleave tool calling strategy. Extensive experiments on Video-MME and LongVideoBench show strong performance of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- Overall, the flow of paper writing is clear and easy-to-follow.\n- The proposed method is simple, intuitive, training-free, yet works well. Experiments on VideoMME and LongVideoBench show that the proposed method outperforms multiple baselines with a smaller runtime.\n- The qualitative examples are clear, intuitive, and can help people understand the method.\n\nWeaknesses:\n\n- Some important implementation details are not clearly presented.\n    - What is the controller LLM?\n    - How many frames are sampled initially?\n    - In Table 1, how is the runtime calculated? What GPUs did the authors use? Is the comparison with other methods apple-to-apple? Why are some runtimes very vague? For example, the runtime of GPT-4o is presented as <30s, but the runtime of the proposed method which uses GPT-4o is clearly presented as 15.8s.  Without standardized evaluation conditions and transparent reporting, it is challenging to assess the actual efficiency advantage of the proposed method over the baselines.\n    - The introduction to the frame selector tool is not clear. Is the frame selector tool an LLM taking captions as input and output frame IDs, or video trimming, or frame selection methods as AKeyS and T*?\n- It is good that the proposed method outperforms GPT-4o, with both methods taking ~30 frames as input. However, the toolset of the proposed method includes GPT-4o itself. This raises the question: how does the proposed method perform when provided with more frames, such as in GPT-4o’s 1 fps / 384 setting? It would significantly strengthen the paper if the authors could show that their method continues to outperform GPT-4o even when both are using the same number of frames (e.g., 1 fps / 384).\n- As the authors mention, one of the motivation is to solve the ‘Unbalanced number and diversity of tools’ issue. However, it is not clear how exactly the authors solve this issue. From the reviewer’s understanding, the spatiotemporal interleave tool calling strategy can balance the spatial tools and temporal tools, but what about the tools within the same categories?\n- It would be good to provide some failure case analysis to help people understand more about the limitations of the proposed method.\n\nQUESTIONS:\nPlease see the ‘Weaknesses’ section. Overall, the reviewer thinks that the method is intuitive, interesting, simple and works well. However, it would be good if the authors could clarify the implementation details and show that the proposed method also performs well with more frames. The reviewer will consider raising the score if the above weaknesses are solved.\n\nLIMITATIONS:\nIt would be good to provide some failure case analysis to help people understand more about the limitations of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}