Video-STAR: Reinforcing Open-Vocabulary Action Recognition with Tools

## Abstract

Abstract Multimodal large language models (MLLMs) have demonstrated remarkable potential in bridging visual and textual reasoning, yet their reliance on text-centric priors often limits their ability to disentangle semantically similar actions in open-vocabulary scenarios. To address this, we propose Video-STAR, a framework that harmonizes contextual sub-motion decomposition with tool-augmented reinforcement learning for open-vocabulary action recognition (OVAR). Unlike prior methods that treat actions as monolithic entities, our approach innovatively decomposes actions into discriminative sub-motions for fine-grained matching while dynamically invoking domain-specific tools for cross-modal interleaving, thereby enabling category-specific reasoning capacity and reducing cross-modal hallucination. Moreover, by designing a hierarchical reward that balances tool-usage efficiency, sub-motion relevance, and structural coherence in reasoning, our method autonomously leverages external tools to prioritize sub-motion patterns without explicit supervision, transmitting from text-centric reasoning to visually grounded inference. Extensive evaluations on HMDB-51, UCF-101, SSv2, Kinetics-400, and Kinetics-600 datasets demonstrate our state-of-the-art performance, outperforming existing methods in distinguishing fine-grained actions and handling cross-modal hallucination, validating our excellent robustness and generalization.

## 1 Introduction

Figure: Figure 1: Key insight of Video-STAR. (a) MLLMs + CoT is prone to hallucinations due to over-reliance on text-centric reasoning while ignoring visual cues. (b) MLLMs + Tool-Augmented CoT mitigates hallucinations by integrating domain-specific tools to extract visual information. However, both (a) and (b) lack category-specific reasoning capabilities and struggle to distinguish semantically similar or complex actions. (c) Video-STAR enhances reasoning capacity by introducing contextual sub-motion decomposition, which disentangles actions into discriminative motion primitives. This enables fine-grained action discrimination and robust performance in open-vocabulary scenarios.
Refer to caption: x2.png

Open-vocabulary action recognition (OVAR) aims to identify action classes not present during training, extending beyond the closed-vocabulary constraints of traditional action recognition systems (Yao et al., 2019; He et al., 2019). This capability is crucial for real-world applications where novel actions frequently emerge, such as surveillance systems, human-robot interaction, and content analysis platforms. While early approaches (Feichtenhofer et al., 2019; Lin et al., 2019) relied on hand-crafted features and limited class taxonomies, recent advances in vision-language pre-training (Radford et al., 2021; Jia et al., 2021) have opened new possibilities for zero-shot action understanding. However, OVAR remains challenging due to the inherent complexity of human actions, which often involve subtle temporal dynamics, spatial relationships, and semantic similarities that are difficult to distinguish through conventional approaches.

Recent multimodal large language models (MLLMs) have shown strong visual reasoning and cross-modal understanding (Shen et al., 2025; Feng et al., 2025). Through a unified architecture that tokenizes visual signals and fuses them with text via cross-modal attention, they achieve fine-grained grounding and stepwise reasoning. Systems such as Qwen-VL (Bai et al., 2025a) and GPT-4 (OpenAI et al., 2024) consequently deliver strong performance on image captioning (Zhao et al., 2025a), visual grounding (Bai et al., 2025b), and video understanding (Zhang et al., 2024b).

Despite recent advances, these models still encounter two critical limitations when applied to OVAR tasks: ❶ Existing approaches rely heavily on text-based chain-of-thought (CoT) reasoning for visual understanding, which leads to misalignment between temporal visual features with discrete textual representations. As depicted in Fig. [1](https://arxiv.org/html/2510.08480v1#S1.F1) (a), such misalignment makes the models prone to cross-modal hallucinations during reasoning. ❷ Current methods typically struggle to handle semantically similar actions in open-vocabulary scenarios.
The unpredictability of actions in open-vocabulary scenarios prevents models from learning discriminative patterns directly.
As shown in Fig. [1](https://arxiv.org/html/2510.08480v1#S1.F1) (b), such inability in distinguishing semantically ambiguous actions may lead to erroneous classification.

To solve the above-mentioned challenges of *similar-action misclassification and cross-modal hallucinations*, we argue that effective OVAR requires two key capabilities:
❶ Enabling Fine-grained action discrimination through contextual sub-motion decomposition, which enables the model to disentangle hierarchical actions into discriminative sub-motion primitives for fine-grained classification. (e.g., "shoot basketball": "leg bending" → "torso jumping" → "arm extending" → "hand-ball releasing").
❷ Enhancing cross-modal interleaving through multimodal CoT reasoning, which guides the model to autonomously invoke domain-specific tools for enhanced visual-semantic representation.
While recent MLLMs (Zhang et al., 2025b; Zheng et al., 2025) have demonstrated the potential of tool-augmentation in open-world video understanding (Qian et al., 2025c; Su et al., 2025a), these approaches typically operate on frame-level manipulations (e.g., zooming, cropping) that lack explicit modeling of motion continuity. In contrast, OVAR requires capturing both fine-grained temporal dynamics and hierarchical sub-motion dependencies.
Consequently, a unified framework that harmonizes sub-motion decomposition with tool-mediated perception to ensure both spatial precision and semantic coherence in OVAR is urgently needed.

Therefore, we introduce VIDEO-STAR, a framework that synergistically integrates contextual sub-motion decomposition with tool-augmented reinforcement learning (RL) for open-vocabulary action recognition. Specifically, during the inference process, our method dynamically invokes domain-specific tools like pose estimation, human detection, and online retrieval for cross-modal interleaving, thereby effectively resolving visual-semantic ambiguities and reducing hallucination.
Moreover, to enable category-specific reasoning capacity, we formulate action recognition as a sequential decision-making process at the following stages: ❶ decomposing actions into discriminative sub-motion primitives, ❷ matching these primitives to several candidate actions, ❸ scoring candidates based on hierarchical relevance.
Furthermore, we propose a reward mechanism that jointly optimizes for tool-usage efficiency, sub-motion relevance, and structural coherence in reasoning. This ensures that tools are activated only when they contribute meaningful insights, while sub-motion hierarchies are weighted to prioritize semantically salient components.
Our contributions are threefold:

- •
Tool-Augmented Agentic RL. VIDEO-STAR introduces a novel framework that unifies sub-motion decomposition with tool-augmented reinforcement learning, thereby enabling robust open-vocabulary action recognition through dynamic, visually grounded reasoning.
- •
Multi-Tool Integration. We design a multimodal tool library integrating pose estimation, human detection, and online retrieval, which dynamically augments reasoning with domain-specific knowledge to resolve cross-modal hallucinations during reasoning.
- •
Emprical Effectiveness. Experimental results on HMDB-51, UCF-101, SSv2, Kinetics-400, and Kinetics-600 datasets demonstrate that our method achieves state-of-the-art performance, outperforming existing methods by significant margins while maintaining computational efficiency.

## 2 Related Work (Extended Ver. in Appx. B )

Open-Vocabulary Action Recognition.
Recent methods for open-vocabulary action recognition (OVAR) primarily leverage CLIP’s cross-modal alignment. Early approaches like ActionCLIP (Wang et al., 2021) and ViFi-CLIP (Rasheed et al., 2023) adapt CLIP via full fine-tuning or joint encoder optimization, but often overfit to static features. Parameter-efficient strategies, such as Adaptformer (Chen et al., 2022) and ST-Adapter (Pan et al., 2022), use compact modules to distill temporal knowledge, while prompt engineering (AIM (Yang et al., 2023), VPT (Ju et al., 2022)) biases models toward action semantics. However, these methods struggle with cross-architecture generalization due to limited capacity or rigid designs. Recent works like Open-VCLIP (Weng et al., 2023) and FROSTER (Huang et al., 2024b) improve generalization via weight interpolation and residual distillation, yet remain constrained by static assumptions. In contrast, Video-STAR introduces sub-motion decomposition with tool-augmented RL for robust open-vocabulary inference.

Multimodal LLMs Reasoning.
Recent advances in large language models (LLMs) have demonstrated that RL-based post-training can significantly enhance reasoning capabilities, as exemplified by OpenAI-o1 (Jaech et al., 2024) and DeepSeek-R1 (Guo et al., 2025). These paradigms have been extended to multimodal language models (MLLMs) for tasks like mathematical VQA (Peng et al., 2025), image segmentation (Liu et al., 2025), and video understanding (Feng et al., 2025). However, existing methods struggle with long-sequence hallucination (Chen et al., 2025a) and limited cross-modal interaction. To address this, we propose multimodal CoT reasoning via tool-augmented RL, which explicitly reduces hallucination through dynamic tool integration.

Tool-Augmented Agentic System.
Recent advancements in LLMs have shown that external tools can enhance multimodal reasoning. Early works like FAST (Sun et al., 2025) and MVoT (Li et al., 2025a) introduce visual evidence into reasoning, forming multimodal CoT for image tasks. LLaVa-Plus (Liu et al., 2023b) pioneered training strategies for tool use, while VPD (Hu et al., 2024) leveraged program-derived data to transfer tool skills. Recent methods like TACO (Ma et al., 2024) and PyVision (Zhao et al., 2025b) expand tool use with RL. However, existing approaches rely on static tool pipelines, limiting their ability to adapt to complex, open-vocabulary actions. Our framework addresses this by explicitly modeling the contextual sub-motions and dynamically balancing tool usage efficiency with hierarchical motion relevance through a structured reward mechanism.

## 3 Methodology

Figure: Figure 2: Pipeline of Video-STAR. (i) Introduce a three-stage sub-motion logic chain to construct tool-augmented reasoning data that decomposes actions into discriminative sub-motions. (ii) Pre-train the MLLMs on structured reasoning chains and fine-tune it for domain-specific adaptation. (iii) Adopt the GRPO algorithm for reinforcement learning, which optimizes a hierarchical reward function considering both tool-usage and sub-motion to ensure robust and consistent inference.
Refer to caption: x3.png

Overview.
We propose Video-STAR, a unified framework that synergizes contextual sub-motion decomposition with tool-augmented reinforcement learning, as illustrated in Fig. [2](https://arxiv.org/html/2510.08480v1#S3.F2).
Sec [3.1](https://arxiv.org/html/2510.08480v1#S3.SS1) first details the problem formulation of the inference process and the tool library.
Sec [3.2](https://arxiv.org/html/2510.08480v1#S3.SS2) details the Training Data Construction through multimodal CoT generation, synthesizing video-query pairs with predefined prompts.
Sec [3.3](https://arxiv.org/html/2510.08480v1#S3.SS3) presents Agentic Supervised Fine-tuning, where the Qwen2.5-VL base model is pretrained on the reasoning data for domain-specific adaptation.
Sec [3.4](https://arxiv.org/html/2510.08480v1#S3.SS4) outlines Agentic Reinforcement Learning, which introduces a hierarchical reward function that balances tool-usage, sub-motion, and structural coherence for inference.
The above-mentioned three stages are tightly coupled: the training data provides the foundation for SFT, which in turn initializes the RL policy, ensuring end-to-end alignment between perception, reasoning, and action inference.

### 3.1 Problem Formulation.

In open-vocabulary action recognition (OVAR), the task is to identify action classes not seen during training. The input consists of a video $V$ and a query $Q$, while the output is a predicted action $A$ that may belong to unseen categories during training. To achieve this goal, we design the following two-stage framework that dynamically integrates domain-specific tools:
❶ The first stage selects the most relevant tool from an augmented toolset $T=\{T_{\text{p}},T_{\text{d}},T_{\text{r}}\}$. ❷ The second stage combines tool outputs with raw inputs for final classification. This architecture ensures contextual consistency between visual observations and semantic reasoning by explicitly modeling the interaction between perceptual features and task-specific knowledge. The implementation details are as follows:

Stage 1: Tool Selection.
Given a video $V$ and query $Q$, the model analyzes contextual cues to determine which tool is most relevant for resolving our task. The toolset contains four components: $T_{\text{p}}$ for pose estimation, $T_{\text{d}}$ for human detection, $T_{\text{a}}$ for action explanation, $T_{\text{v}}$ for video description.
The first-stage output $y^{\prime}\sim\pi_{\theta}(\cdot|V,Q;T)$ produces a selected tool call $C\subseteq T$ along with intermediate results $R$, where $\pi_{\theta}$ represents the policy network of MLLMs parameterized by weights $\theta$.

Stage 2: Result Integration & Prediction
The second stage refines action recognition by integrating tool outputs $R$ with original inputs. For the visual tools $T_{\text{p}}$ and $T_{\text{d}}$, extracted features $F$ are concatenated with raw video frames $V$ to enrich spatial-temporal representations. For the semantic reasoning tool $T_{\text{r}}$, the generated explanation $E$ is appended to the original query $Q$ to provide contextual grounding. The final prediction $A$ is computed as:

$$ $A\sim\pi_{\theta}(\cdot|V\oplus F,Q\oplus E;T),$ (1) $$

where $\oplus$ denotes feature concatenation for visual modalities and text appending for reasoning, while $\pi_{\theta}$ maintains the same policy network of MLLMs parameterized by weights $\theta$.

Figure: Figure 3: Tool Libirary. Given the input video, Video-STAR respectively adopts the YOLO 11 for human detection & pose estimation, and the Qwen API for action explanation & video description.
Refer to caption: x4.png

Tool Library
As shown in Fig. [3](https://arxiv.org/html/2510.08480v1#S3.F3), our toolkit library contains four sub-tools:
❶ Human Detection: we adopt the YOLO 11 (Khanam & Hussain, 2024), which employs a hybrid spatial attention mechanism for anchor-free human detection.
❷ Pose Estimation: we introduce YOLO 11’s 17-keypoint skeletonization capability (e.g., COCO format) to capture sub-pixel joint positions for fine-grained motion analysis.
❸ Action Explanation: we leverage the Qwen API for *online retrieval augmentation (RAG)*, then redefine actions through transitional phases (e.g., "stand" → "transit from sit or lay to stand").
❹ Video Description: we extract temporally salient frames, then adopt the Qwen API for online RAG to disambiguate multi-phase action sequences (e.g., "sit-stand-walk") via spatial-temporal descriptions.
The detailed descriptions are shown in Appendix [C](https://arxiv.org/html/2510.08480v1#A3).

### 3.2 Training Data Construction

Existing MLLMs methods typically suffer from two critical limitations when applied to OVAR tasks: ❶ The over-reliance on textual priors that neglects domain-specific visual cues, and ❷ The inability to distinguish ambiguous actions in open-vocabulary scenarios. To address these issues, we construct a novel training dataset by integrating multimodal CoT reasoning with tool-augmented decomposition, thereby ensuring both spatial-temporal alignment and semantic accuracy.

Data Collection. As shown in the cyan part of Fig. [2](https://arxiv.org/html/2510.08480v1#S3.F2), the dataset is structured through a three-stage logic chain. First, we systematically curate 5,000 video-query pairs from HMDB-51 (Kuehne et al., 2011) dataset, then augmented with synthetic reasoning chains generated by Qwen2.5-VL-72B (Bai et al., 2025a). For each pair, the input video and ground-truth action label are processed through a predefined dual-phase CoT prompt, which explicitly instructs the model to first invoke relevant tools and then integrate their outputs for action classification. Such a design effectively ensures that the generated reasoning chains dynamically incorporate both visual features and domain knowledge, thereby bridging the gap between text-centric reasoning and visually grounded inference.

Generating Pipeline. The core innovation lies in our three-stage framework for action recognition:

- •
Sub-motion Decomposition first breaks down the holistic action into discriminative sub-motion primitives by identifying key body-part interactions. For "shoot ball" is decomposed into "body bending → leg extending → feet-ball interaction". This decomposition addresses the inherent ambiguity of holistic perception by isolating perceptually distinct motion patterns.
- •
Candidate Selection then maps each sub-motion to pre-defined action definitions, generating 2-3 semantically relevant candidates. In our example, "body bending" suggests "run", "leg extending" implies "hurdle", and "feet-ball interaction" aligns with "shoot". This step narrows the search space while preserving semantic diversity, ensuring robust open-vocabulary generalization.
- •
Matching Scoring finalizes recognition by comparing all sub-motions against each candidate’s detailed definition. The system evaluates spatial precision and description alignment, ultimately selecting the highest-scoring candidate. In our case, the combination of precise spatial interactions and semantic coherence with the definition "shoot ball" would lead to the correct classification.

This approach enhances generalization across open-vocabulary scenarios by balancing perceptual accuracy with semantic consistency, enabling robust discrimination in ambiguous conditions.

Data Assessment
To ensure the reliability of the constructed data, we introduce a specialized LLM to evaluate its factual accuracy and logical coherence. This model systematically removes entries containing inconsistent reasoning steps or unverified claims, resulting in a refined dataset that combines explicit tool usage with logically coherent, evidence-supported reasoning.

### 3.3 Agentic Supervised Fine-Tuning

After constructing the training data, we introduce a cold-start phase that prioritizes supervised fine-tuning (SFT) before applying reinforcement learning (RL).
Inspiring from R1-Zero (Guo et al., 2025), we initially attempt direct RL optimization to train our method. However, preliminary experiments reveal a progressive decline in the frequency of tool invocation during policy rollouts.
This behavior likely arises from a distributional discrepancy in target domain between tool-enhanced visual features and model’s pretraining data. Therefore, we opt to perform SFT before RL.

Specifically, We formalize each training instance as $\mathcal{T}=(\mathcal{X},\mathcal{I},\mathcal{S},\mathcal{Y})$, where $\mathcal{X}$ denotes the input modality, $\mathcal{I}$ represents task instructions, $\mathcal{S}=\{s_{1},\dots,s_{T}\}$ captures reasoning steps, and $\mathcal{Y}$ is the target output. The objective minimizes the negative log-likelihood of the reasoning process:

$$ $\mathcal{L}_{\text{SFT}}=-\mathbb{E}_{\mathcal{T}\sim\mathcal{D}}\left[\sum_{t=1}^{T}\log p_{\theta}(s_{t}\mid\mathcal{X},\mathcal{I},s_{<t})\right],$ (2) $$

where $p_{\theta}(\cdot)$ is the model’s conditional probability distribution.
This formulation enables model to generate reasoning chains that align with both task instructions and ground-truth during pre-training.

### 3.4 Agentic Reinforcement Learning

Group Relative Policy Optimization (GRPO).
We adopt the GRPO algorithm (Jiang et al., 2025) for policy optimization.
GRPO innovatively employs a groupwise comparison framework to evaluate candidate responses.
Specifically, for each query $q$ paired with its ground-truth solution $a$ from dataset $D$, the algorithm generates a set of rollout trajectories $\{o_{1},o_{2},\dots,o_{G}\}$ based on the previous policy $\pi_{\theta_{\text{old}}}$. The policy $\pi_{\theta}$ is then refined through the optimization of this objective function:

$$ $\begin{split}&\mathcal{L}_{GRPO}(\theta)=-\mathbb{E}[q\text{\textasciitilde}P(Q),\{o_{i}\}_{i=1}^{G}{\pi_{\theta}}_{old}(O|q)]\frac{1}{G}\sum_{i=1}^{G}\\ &\Bigg(min\bigg({\frac{\pi_{\theta}(o_{i}|q)}{{\pi_{\theta}}_{old}(o_{i}|q)}}*{Adv}_{i},clip\Big({\frac{\pi_{\theta}(o_{i}|q)}{{\pi_{\theta}}_{old}(o_{i}|q)}},1-{\epsilon},1+{\epsilon}\Big)*{Adv}_{i}\bigg)-\beta\mathbb{D}_{KL}(\pi_{\theta}||\pi_{ref})\Bigg),\end{split}$ (3) $$

$$ $\mathbb{D}_{KL}(\pi_{\theta}||\pi_{ref})={\frac{\pi_{ref}(o_{i}|q)}{{\pi_{\theta}}(o_{i}|q)}}-log{\frac{\pi_{ref}(o_{i}|q)}{{\pi_{\theta}}(o_{i}|q)}}-1,$ (4) $$

where $\beta$ is adopted to balance the trade-off between exploration and stability during optimization. Then the advantage estimator $A_{i}$ is calculated using normalized rewards from the trajectory group:

$$ $A_{i}=\frac{r_{i}-\text{mean}(\{r_{1},r_{2},\dots,r_{G}\})}{\text{std}(\{r_{1},r_{2},\dots,r_{G}\})}.$ (5) $$

Each trajectory $o_{i}$ receives a binary reward $r_{i}\in\{0,1\}$ through a rule-based verification system designed to mitigate reward manipulation risks.

Reward Design.
Effective reward functions should balance accuracy, structural coherence, tool-use efficiency, and sub‑motion relevance to support the training of agentic systems for OVAR tasks.
While traditional reward design primarily prioritizes the correctness of the answer and format, we extend this by incorporating hierarchical sub-motion relevance and tool-mediated perception to guide structured reasoning over sequential sub-motions.

The total reward integrates four components: accuracy reward $R_{\text{acc}}$, format reward $R_{\text{format}}$, tool-usage reward $R_{\text{tool}}$, and sub-motion reward $R_{\text{sub}}$.
The accuracy reward $R_{\text{acc}}$ evaluates the correctness of the final action classification, while the formatting reward $R_{\text{format}}$ penalizes unstructured or incomplete reasoning chains.
The tool-usage reward $R_{\text{tool}}$ is activated only when correct answers are produced alongside valid tool invocations.
The sub-motion reward $R_{\text{sub}}$ employs a hierarchical weighting mechanism to prioritize semantically salient sub-motions.
Specifically, the model first generates $n$ sub-motions through contextual analysis, ordered by their relevance to the action definition.
For the $k$-th ranked candidate, its weight is assigned as:
$w_{k}=n-k+1$.
When the predicted classification matches a subset of $m$ sub-motions $\{k_{1},\dots,k_{m}\}$, we define:
$R_{\text{sub}}=\sum_{i=1}^{m}w_{k_{i}}/\sum_{i=1}^{n}w_{k_{i}}.$
This formulation ensures higher-priority sub-motions receive exponentially greater weighting, while lower-priority ones contribute proportionally less. Formally, the total reward is defined as:

$$ $R(\tau)=R_{\text{acc}}(\tau)+R_{\text{format}}(\tau)+\mathbb{I}_{R_{\text{acc}}(\tau)>0}\cdot\left(R_{\text{tool}}(\tau)+R_{\text{sub}}(\tau)\right),$ (6) $$

where $\mathbb{I}_{R_{\text{acc}}(\tau)>0}$ is the indicator function which assigns a binary value of 1 when $R_{\text{acc}}(\tau)>0$ and 0 otherwise. This conditional structure enforces two key principles: ❶ perception-aware reasoning requires simultaneous activation of tool usage and sub-motion prioritization only when the final answer is correct, and ❷ redundant tool invocations or irrelevant sub-motions receive no reward. By coupling tool-mediated perception with hierarchical sub-motion weighting, the framework ensures that agents invoke tools meaningfully while maintaining coherent reasoning chains.

## 4 Experiment

Experimental Settings.
We adopt two experimental settings: ❶ base-to-novel: For each dataset, the label space is partitioned into two disjoint subsets, i.e., the base classes $Y_{B}$ and the novel classes $\mathcal{Y}_{N}$, where $\mathcal{Y}_{B}\cap\mathcal{Y}_{N}=\emptyset$. We train the models on data from $\mathcal{Y}_{B}$ and evaluate on testing samples from $\mathcal{Y}_{B}\cup\mathcal{Y}_{N}$.
❷ cross-dataset: In this setting, the model is trained on a source dataset with the label set as $\mathcal{Y}_{S}$ and evaluated on a target dataset with another label set as $\mathcal{Y}_{T}$, where $|\mathcal{Y}_{S}\cup\mathcal{Y}_{T}|\geq|\mathcal{Y}_{S}\cap\mathcal{Y}_{T}|$.

Dataset and Metrics.
We evaluate our method on five standard action recognition benchmarks: UCF-101 (Soomro et al., 2012), HMDB-51 (Kuehne et al., 2011), Kinetics-400 (K-400) (Carreira & Zisserman, 2017), Kinetics-600 (K-600) (Carreira et al., 2018), and Something-to-Something V2 (SSv2). UCF-101 has 13,320 video clips from 101 classes, while HMDB-51 includes 6,849 videos across 51 classes. For large-scale evaluation, we use K-400 and K-600, which contain 400 and 600 classes, respectively. SSv2 contains 174 fine-grained classes.
Following prior work (Rasheed et al., 2023; Ni et al., 2022; Weng et al., 2023), we report average top-1 accuracy under the two settings.

Implementation Details.
We implement experiments on both Qwen2.5-VL-3B and Qwen2.5-VL-7B models.
Learning rate is set to 5e-7 with an accumulated total batch size of 8 in both stages.
The GRPO adopts a maximum completion length of 4,096 tokens and samples 4 responses per input.
In stage one, we select 5,000 training samples constructed from Sec [3.2](https://arxiv.org/html/2510.08480v1#S3.SS2) for SFT. These same training samples are then reused for subsequent RL optimization during the stage two.
The RL process was executed for 600 iterations and 1 epoch.
More experimental details are shown in Appendix [A.1](https://arxiv.org/html/2510.08480v1#A1.SS1).

### 4.1 Main Results

**Table 1: Performance comparison (Top1-Acc (%)) with the CLIP-based methods using ViT-B/16 under the base-to-novel setting. "HM" denotes the harmonic mean of the accuracy from the base and novel sets. Note that the best and second best performances are highlighted.**
| Method | K-400 | HMDB-51 | UCF-101 | SSv2 |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Base | Novel | HM | Base | Novel | HM | Base | Novel | HM | Base | Novel | HM |  |
| ActionCLIP | 61.0 | 46.2 | 52.6 | 69.1 | 37.3 | 48.5 | 90.1 | 58.1 | 70.7 | 13.3 | 10.1 | 11.5 |
| X-CLIP | 74.1 | 56.4 | 64.0 | 69.4 | 45.5 | 55.0 | 89.9 | 58.9 | 71.2 | 8.5 | 6.6 | 7.4 |
| VPT | 69.7 | 37.6 | 48.8 | 46.2 | 16.0 | 23.8 | 90.5 | 40.4 | 55.8 | 8.3 | 5.3 | 6.4 |
| ST-Adapter | 73.6 | 62.0 | 67.3 | 65.3 | 48.9 | 55.9 | 85.5 | 76.8 | 80.9 | 9.3 | 8.4 | 8.8 |
| ViFi-CLIP | 76.4 | 61.1 | 67.9 | 73.8 | 53.3 | 61.9 | 92.9 | 67.7 | 78.3 | 16.2 | 12.1 | 13.9 |
| Open-VCLIP | 76.5 | 62.6 | 68.9 | 70.3 | 50.4 | 58.7 | 94.8 | 77.5 | 85.3 | 16.0 | 11.0 | 13.0 |
| FROSTER | 77.8 | 64.3 | 70.4 | 74.1 | 58.0 | 65.1 | 95.3 | 80.0 | 87.0 | 18.3 | 12.2 | 14.6 |
| Open-MeDe | 77.2 | 63.8 | 69.9 | 73.6 | 56.4 | 63.9 | 94.9 | 78.5 | 85.9 | 17.1 | 12.3 | 14.3 |
| Qwen2.5-VL-3B | 48.3 | 40.5 | 44.1 | 54.0 | 40.8 | 46.5 | 71.4 | 62.4 | 66.6 | 3.5 | 3.2 | 3.3 |
| Qwen2.5-VL-7B | 87.8 | 84.8 | 86.3 | 41.7 | 50.3 | 45.6 | 85.0 | 82.4 | 83.7 | 14.0 | 9.9 | 11.6 |
| Video-STAR-3B | 86.0 | 86.4 | 86.2 | 92.1 | 91.7 | 91.9 | 96.9 | 98.9 | 97.9 | 13.5 | 11.3 | 12.3 |
| Video-STAR-7B | 96.3 | 97.2 | 96.7 | 92.3 | 91.9 | 92.1 | 99.6 | 99.8 | 99.7 | 19.2 | 13.0 | 15.5 |

Base-to-novel.
In Tab. [1](https://arxiv.org/html/2510.08480v1#S4.T1), we compare our Video-STAR with the state-of-the-art (SOTA) methods under the base-to-novel setting. Notably, our model is fine-tuned only on the base set of HMDB-51 and evaluated in a zero-shot manner on K-400, UCF-101, and SSv2. In contrast, all other methods are fine-tuned on the base set of each respective dataset, making our evaluation more challenging. From the results, we highlight four findings:
❶ Our Video-STAR established a new SOTA, consistently outperforming previous methods across all datasets. Specifically, Video-STAR-7B achieves 26.3% in K-400 and 27.0% in HMDB-51 absolute improvement over the previous SOTA.
❷ Video-STAR demonstrates strong generalization to novel categories. We see the performance gap of Video-STAR between the base and novel sets is nearly closed. Remarkably, on K-400 and UCF-101, the accuracy of novel classes exceeds that of base ones.
❸ Scaling helps the complex dataset. We observe a 10.5% and 3.2% absolute increase in HM on K-400 and SSv2 datasets when scaling from 3B to 7B. However, on HMDB-51 and UCF-101, performance is already near-perfect.
❹ Generic MLLMs are insufficient. Although Qwen2.5-VL-7B is competitive on K-400 (HM 86.3), it collapses on HMDB-51 (HM 45.6) and SSv2 (HM 11.6), whereas Video-STAR maintains uniformly high performance, suggesting that our specific alignment is crucial for robustness across datasets.

**Table 2: Performance comparison (Top1-Acc (%)) with the CLIP-based methods under the cross-dataset setting. "Full" denotes the full validation set, while "Split" denotes evaluating across three validation splits. The results of most other papers are taken from Open-VCLIP and ViFi-CLIP. Note that the best and second best performances are highlighted.**
| Method | UCF-101 | HMDB-51 | K-600 |  |  |
| --- | --- | --- | --- | --- | --- |
| Full $\uparrow$ | Split $\uparrow$ | Full $\uparrow$ | Split $\uparrow$ | Split $\uparrow$ |  |
| ActionCLIP (Wang et al., 2021) | 77.4 | 77.5$\pm$0.8 | 48.0 | 48.2$\pm$1.5 | 62.5$\pm$1.2 |
| X-CLIP (Ni et al., 2022) | - | 72.0$\pm$2.3 | - | 44.6$\pm$5.2 | 65.2$\pm$0.4 |
| ST-Adapter (Pan et al., 2022) | 77.9 | 77.6$\pm$0.7 | 50.3 | 51.1$\pm$0.6 | 60.2$\pm$1.8 |
| Vita-CLIP (Wasim et al., 2023) | - | 75.0$\pm$0.6 | - | 48.6$\pm$0.6 | 67.4$\pm$0.5 |
| ViFi-CLIP (Rasheed et al., 2023) | - | 76.8$\pm$0.7 | - | 51.3$\pm$0.6 | 71.2$\pm$1.0 |
| Open-VCLIP (Weng et al., 2023) | 83.5 | 83.4$\pm$1.2 | 53.2 | 53.9$\pm$1.2 | 73.0$\pm$0.8 |
| FROSTER (Huang et al., 2024b) | 85.0 | 84.8$\pm$1.1 | 54.5 | 54.8$\pm$1.3 | 74.8$\pm$0.9 |
| Open-MeDe (Yu et al., 2025) | - | 83.7$\pm$1.3 | - | 54.6$\pm$1.1 | 73.7$\pm$0.9 |
| Qwen2.5-VL-3B | 58.3 | 58.1$\pm$0.2 | 39.2 | 42.1$\pm$0.3 | 37.7$\pm$1.8 |
| Qwen2.5-VL-7B | 77.5 | 77.2$\pm$0.6 | 50.6 | 53.1$\pm$0.1 | 68.0$\pm$1.5 |
| Video-STAR-3B | 96.8 | 96.7$\pm$0.3 | 83.5 | 86.2$\pm$0.2 | 90.5$\pm$0.7 |
| Video-STAR-7B | 99.4 | 99.4$\pm$0.2 | 90.1 | 92.5$\pm$0.1 | 98.2$\pm$0.1 |

Cross-dataset.
In Tab. [2](https://arxiv.org/html/2510.08480v1#S4.T2), we compare our method with the previous methods under the cross-dataset setting. We see two key observations. ❶ Video-STAR exhibits remarkable stability, achieving comparable results on the full validation sets (UCF-101*, HMDB-51*) and the splits (UCF-101, HMDB-51). This indicates that our model achieves robustness through tool usage and sub-motion decomposition. ❷ Our smaller 3B model outperforms previous state-of-the-art methods, as well as the larger Qwen2.5-VL-7B. This further demonstrates the effectiveness of our Video-STAR. Moreover, Fig. [4](https://arxiv.org/html/2510.08480v1#S4.F4) presents a comparative case study. More case studies are provided in Appendix [A.3](https://arxiv.org/html/2510.08480v1#A1.SS3).

Figure: Figure 4: Case Study between Qwen2.5-VL-3B and our Video-STAR-3B. Qwen2.5-VL-3B misclassifies action "turn" as "smile", while our Video-STAR-3B accurately identifies the correct action.
Refer to caption: x5.png

**Table 3: Ablation of key components in Video-STAR under cross-dataset setting. "Split" denotes evaluating across three validation splits. The best and second best performances are highlighted.**
| Method | UCF-101 | HMDB-51 | K-600 | *RL* | UCF-101 | HMDB-51 | K-600 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Split$\uparrow$ | Split$\uparrow$ | Split$\uparrow$ | Split$\uparrow$ | Split$\uparrow$ | Split$\uparrow$ |  |  |
| *(a)* Qwen2.5-VL-3B | 58.1$\pm$0.2 | 42.1$\pm$0.3 | 37.7$\pm$1.8 | *(g)* w/. *Base* | 93.8$\pm$0.4 | 81.7$\pm$0.2 | 84.7$\pm$1.2 |
| *(b)* Video-STAR-3B | 96.7$\pm$0.3 | 86.2$\pm$0.2 | 90.5$\pm$0.7 | *(h)* w/o. *Tol* | 95.1$\pm$0.3 | 83.2$\pm$0.2 | 86.7$\pm$0.9 |
| *(c)* w/o. *RL* | 76.8$\pm$0.8 | 63.5$\pm$0.2 | 61.3$\pm$2.0 | *(i)* w/o. *Sub* | 95.7$\pm$0.3 | 84.1$\pm$0.2 | 87.9$\pm$0.8 |
| *(d)* w/o. *SFT* | 71.4$\pm$0.9 | 57.6$\pm$0.2 | 54.8$\pm$2.1 | *(j)* *Num = 2* | 96.4$\pm$0.2 | 85.0$\pm$0.2 | 89.3$\pm$0.3 |
| *(e)* w/o. *TOL* | 87.1$\pm$0.5 | 74.9$\pm$0.3 | 78.4$\pm$1.5 | *(k)* *Num = 4* | 96.7$\pm$0.3 | 86.2$\pm$0.2 | 90.5$\pm$0.7 |
| *(f)* w/o. *SUB* | 88.5$\pm$0.5 | 76.8$\pm$0.3 | 81.2$\pm$1.3 | *(l)* *Num = 6* | 96.9$\pm$0.3 | 86.9$\pm$0.2 | 91.0$\pm$0.9 |

### 4.2 Ablation Studies

Training Stages.
In rows *(a)-(d)* of Tab. [3](https://arxiv.org/html/2510.08480v1#S4.T3), we compare zero-shot, full model, SFT-only, and RL-only training paradigms, respectively. The zero-shot baseline in row *(a)* yields the lowest results among all methods. SFT-only training in row *(c)* significantly improves accuracy, outperforming RL-only training in row *(d)*. We attribute the gap to the difficulty of RL in directly navigating the high-dimensional reasoning space without prior supervised guidance. These results support our two-stage strategy, where SFT provides a structured reasoning foundation before RL optimization.

Tool-Usage & Sub-Motion.
In rows *(e)-(f)* of Tab. [3](https://arxiv.org/html/2510.08480v1#S4.T3), we show the effect of removing tool-usage and sub-motion decomposition from our full framework. Removing tool usage in row *(e)* causes a notable performance degradation with at least 9.1%. This confirms the critical role of external tools in resolving cross-modal ambiguities and reducing hallucination. Similarly, removing sub-motion decomposition in row *(f)* reduces accuracy 8.2%, 9.4%, 9,3% in UCF-101, HMDB-51, and K-600, respectively. This indicates the importance of breaking complex actions into discriminative primitives and enabling category-specific reasoning. The full model (row *(b)* achieves the highest scores, demonstrating that tool-augmented reasoning and sub-motion hierarchy are complementary and jointly essential for robust open-vocabulary action recognition.

Reinforcement Learning.
In rows *(g)-(i)* of Tab. [3](https://arxiv.org/html/2510.08480v1#S4.T3), we analyze the effect of different reward configurations in RL training. Removing both tool-usage and sub-motion rewards *(g)* produces the lowest performance, showing that accuracy and format rewards alone are insufficient to guide reasoning. Including sub-motion reward in row *(h)* improves results at least 1.3%, illustrating that hierarchical motion decomposition provides strong semantic cues even without tool-mediated perception. Conversely, including tool-usage reward *(i)* also yields better performance than *(g)* with at least 1.9%, confirming that tool integration helps resolve visual-semantic ambiguities. The full model in row *(b)*, which utilizes the complete reward function, achieves the highest accuracy. This demonstrates that our reward design is crucial for effective RL optimization in our framework.

Number of Generation.
In rows *(j)-(l)* of Tab. [3](https://arxiv.org/html/2510.08480v1#S4.T3), we analyze the impact of different numbers of generations during the RL training (GRPO). Increasing the number of candidate responses from 2 to 6 consistently improves performance across all datasets. Specifically, when generating six responses per input *Num = 6*, the model achieves the highest performance with average accuracy scores of 96.9%, 86.9%, and 91.0% on UCF-101, HMDB-51, and K-600, respectively, outperforming the configurations with *Num = 2* and *Num = 4*. However, the marginal gains diminish beyond *Num = 6*, indicating a trade-off between computational cost and performance improvement. Therefore, we opt for *Num = 4* to balance accuracy and efficiency. More ablation study is shown in Appendix [A.2](https://arxiv.org/html/2510.08480v1#A1.SS2).

## 5 Conclusion

In this work, we propose Video-STAR, a novel framework that integrates contextual sub-motion decomposition with tool-augmented reinforcement learning. By decomposing complex actions into discriminative sub-motions and leveraging domain-specific tools for spatial-temporal alignment, our approach bridges the gap between text-centric reasoning and visually grounded inference. The hierarchical reward mechanism ensures efficient tool usage while prioritizing semantically salient sub-motion patterns, enabling robust performance in various scenarios. Extensive experiments demonstrate our state-of-the-art performance, outperforming existing methods by significant margins. Future work will explore cross-modal generalization to broader video understanding applications.

## Ethics Statement

Our work centers on advancing open-vocabulary action recognition through the Video-STAR framework, which leverages publicly available video datasets (e.g., HMDB-51, UCF-101, Kinetics) and domain-specific tools (e.g., YOLO for pose estimation, Qwen API for semantic reasoning). All training and evaluation data were sourced from existing benchmarks with established ethical guidelines, ensuring compliance with data usage policies and minimizing risks of privacy violations. The framework processes visual and textual inputs without extracting or retaining personally identifiable information (PII), focusing solely on abstract action patterns and contextual sub-motions to avoid unintended individual tracking. We emphasize the responsible deployment of our model, advocating for its use in applications that prioritize human well-being, such as assistive technologies or sports analytics, while explicitly discouraging misuse in surveillance or discriminatory practices. To promote transparency, we commit to open-sourcing our code, detailed methodologies, and anonymized training data, enabling independent verification and fostering community-driven ethical oversight. Our experiments adhered to rigorous academic standards, and we remain dedicated to addressing potential biases in tool integration or sub-motion decomposition to ensure equitable performance across diverse populations and scenarios.

## Reproducibility Statement

To ensure the full reproducibility of our findings, we provide comprehensive implementation details in the paper and its appendices. The construction of the training data, including the three-stage sub-motion logic chain and multimodal chain-of-thought (CoT) generation process, is detailed in Sec [3.2](https://arxiv.org/html/2510.08480v1#S3.SS2) and Appendix [D](https://arxiv.org/html/2510.08480v1#A4). The architecture of the Video-STAR framework, encompassing contextual sub-motion decomposition, tool selection mechanisms, and the hierarchical reward function, is described in Sec [3.3](https://arxiv.org/html/2510.08480v1#S3.SS3) and Sec [3.4](https://arxiv.org/html/2510.08480v1#S3.SS4). Implementation specifics for the GRPO algorithm, including reward function components (accuracy, format, tool-usage, and sub-motion rewards) and training hyperparameters (e.g., learning rate, batch size, and generation settings), are presented in Sec [3.4](https://arxiv.org/html/2510.08480v1#S3.SS4) and Appendix [A.1](https://arxiv.org/html/2510.08480v1#A1.SS1). The tool library, including YOLO 11 for human detection and pose estimation, as well as the Qwen API for action explanation and video description, is fully documented in Appendix [C](https://arxiv.org/html/2510.08480v1#A3). In line with our commitment to open science, the source code, pre-trained models, and detailed documentation will be made publicly available.

## References

- Bai et al. (2025a)
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin.
Qwen2.5-vl technical report, 2025a.
URL [https://arxiv.org/abs/2502.13923](https://arxiv.org/abs/2502.13923).
- Bai et al. (2025b)
Sule Bai, Mingxing Li, Yong Liu, Jing Tang, Haoji Zhang, Lei Sun, Xiangxiang Chu, and Yansong Tang.
Univg-r1: Reasoning guided universal visual grounding with reinforcement learning.
*arXiv preprint arXiv:2505.14231*, 2025b.
- Carreira & Zisserman (2017)
Joao Carreira and Andrew Zisserman.
Quo vadis, action recognition? a new model and the kinetics dataset.
In *CVPR*, 2017.
- Carreira et al. (2018)
Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman.
A short note about kinetics-600.
*Arxiv e-prints*, 2018.
- Chen et al. (2022)
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo.
Adaptformer: Adapting vision transformers for scalable visual recognition.
In *NeurIPS*, 2022.
- Chen et al. (2023)
Yinda Chen, Che Liu, Wei Huang, Xiaoyu Liu, Sibo Cheng, Rossella Arcucci, and Zhiwei Xiong.
Generative text-guided 3d vision-language pretraining for unified medical image segmentation.
*arXiv preprint arXiv:2306.04811*, 2023.
- Chen et al. (2024)
Yinda Chen, Haoyuan Shi, Xiaoyu Liu, Te Shi, Ruobing Zhang, Dong Liu, Zhiwei Xiong, and Feng Wu.
Tokenunify: Scalable autoregressive visual pre-training with mixture token prediction.
*arXiv preprint arXiv:2405.16847*, 2024.
- Chen et al. (2025a)
Yukang Chen, Wei Huang, Baifeng Shi, Qinghao Hu, Hanrong Ye, Ligeng Zhu, Zhijian Liu, Pavlo Molchanov, Jan Kautz, Xiaojuan Qi, et al.
Scaling rl to long videos.
*arXiv preprint arXiv:2507.07966*, 2025a.
- Chen et al. (2025b)
Zhiwei Chen, Yupeng Hu, Zixu Li, Zhiheng Fu, Xuemeng Song, and Liqiang Nie.
Offset: Segmentation-based focus shift revision for composed image retrieval.
*arXiv preprint arXiv:2507.05631*, 2025b.
- Cheng et al. (2025)
Junhao Cheng, Yuying Ge, Teng Wang, Yixiao Ge, Jing Liao, and Ying Shan.
Video-holmes: Can mllm think like holmes for complex video reasoning?
*arXiv preprint arXiv:2505.21374*, 2025.
- Chu et al. (2025)
Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang.
Gpg: A simple and strong reinforcement learning baseline for model reasoning.
*arXiv preprint arXiv:2504.02546*, 2025.
- Feichtenhofer et al. (2019)
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.
Slowfast networks for video recognition.
In *ICCV*, 2019.
- Feng et al. (2025)
Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Junfei Wu, Xiaoying Zhang, Benyou Wang, and Xiangyu Yue.
Video-r1: Reinforcing video reasoning in mllms.
*arXiv preprint arXiv:2503.21776*, 2025.
- Ge et al. (2025)
Yuying Ge, Yixiao Ge, Chen Li, Teng Wang, Junfu Pu, Yizhuo Li, Lu Qiu, Jin Ma, Lisheng Duan, Xinyu Zuo, et al.
Arc-hunyuan-video-7b: Structured video comprehension of real-world shorts.
*arXiv preprint arXiv:2507.20939*, 2025.
- Goyal et al. (2017)
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al.
The" something something" video database for learning and evaluating visual common sense.
In *ICCV*, 2017.
- Guo et al. (2025)
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, and et al.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025.
URL [https://arxiv.org/abs/2501.12948](https://arxiv.org/abs/2501.12948).
- He et al. (2019)
Dongliang He, Zhichao Zhou, Chuang Gan, Fu Li, Xiao Liu, Yandong Li, Limin Wang, and Shilei Wen.
Stnet: Local and global spatial-temporal modeling for action recognition.
In *Proceedings of the AAAI conference on artificial intelligence*, volume 33, 2019.
- Hu et al. (2024)
Yushi Hu, Otilia Stretcu, Chun-Ta Lu, Krishnamurthy Viswanathan, Kenji Hata, Enming Luo, Ranjay Krishna, and Ariel Fuxman.
Visual program distillation: Distilling tools and programmatic reasoning into vision-language models.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 9590–9601, 2024.
- Huang et al. (2025)
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin.
Vision-r1: Incentivizing reasoning capability in multimodal large language models.
*arXiv preprint arXiv:2503.06749*, 2025.
- Huang et al. (2024a)
Xiaohu Huang, Hao Zhou, Kun Yao, and Kai Han.
FROSTER: frozen CLIP is A strong teacher for open-vocabulary action recognition.
In *ICLR*, 2024a.
- Huang et al. (2024b)
Xiaohu Huang, Hao Zhou, Kun Yao, and Kai Han.
Froster: Frozen clip is a strong teacher for open-vocabulary action recognition.
In *International Conference on Learning Representations*, 2024b.
- Jaech et al. (2024)
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al.
Openai o1 system card.
*arXiv preprint arXiv:2412.16720*, 2024.
- Ji et al. (2025)
Yuxiang Ji, Ziyu Ma, Yong Wang, Guanhua Chen, Xiangxiang Chu, and Liaoni Wu.
Tree search for llm agent reinforcement learning.
*arXiv preprint arXiv:2509.21240*, 2025.
- Jia et al. (2021)
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.
Scaling up visual and vision-language representation learning with noisy text supervision.
In *International conference on machine learning*, pp. 4904–4916. PMLR, 2021.
- Jiang et al. (2025)
Bo Jiang, Shaoyu Chen, Qian Zhang, Wenyu Liu, and Xinggang Wang.
Alphadrive: Unleashing the power of vlms in autonomous driving via reinforcement learning and reasoning, 2025.
URL [https://arxiv.org/abs/2503.07608](https://arxiv.org/abs/2503.07608).
- Ju et al. (2022)
Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie.
Prompting visual-language models for efficient video understanding.
In *ECCV*, 2022.
- Khanam & Hussain (2024)
Rahima Khanam and Muhammad Hussain.
Yolov11: An overview of the key architectural enhancements, 2024.
URL [https://arxiv.org/abs/2410.17725](https://arxiv.org/abs/2410.17725).
- Kuehne et al. (2011)
Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre.
Hmdb: a large video database for human motion recognition.
In *ICCV*, 2011.
- Li et al. (2025a)
Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vulić, and Furu Wei.
Imagine while reasoning in space: Multimodal visualization-of-thought.
*arXiv preprint arXiv:2501.07542*, 2025a.
- Li et al. (2025b)
Zixu Li, Zhiwei Chen, Haokun Wen, Zhiheng Fu, Yupeng Hu, and Weili Guan.
Encoder: Entity mining and modification relation binding for composed image retrieval.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 39, pp. 5101–5109, 2025b.
- Li et al. (2025c)
Zixu Li, Zhiheng Fu, Yupeng Hu, Zhiwei Chen, Haokun Wen, and Liqiang Nie.
Finecir: Explicit parsing of fine-grained modification semantics for composed image retrieval.
*https://arxiv.org/abs/2503.21309*, 2025c.
- Lin et al. (2019)
Ji Lin, Chuang Gan, and Song Han.
Tsm: Temporal shift module for efficient video understanding.
In *ICCV*, 2019.
- Liu et al. (2023a)
Ruyang Liu, Jingjia Huang, Ge Li, Jiashi Feng, Xinglong Wu, and Thomas H Li.
Revisiting temporal modeling for clip-based image-to-video knowledge transferring.
In *CVPR*, 2023a.
- Liu et al. (2023b)
Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, Lei Zhang, Jianfeng Gao, and Chunyuan Li.
Llava-plus: Learning to use tools for creating multimodal agents, 2023b.
URL [https://arxiv.org/abs/2311.05437](https://arxiv.org/abs/2311.05437).
- Liu et al. (2025)
Yuqi Liu, Bohao Peng, Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, and Jiaya Jia.
Seg-zero: Reasoning-chain guided segmentation via cognitive reinforcement.
*arXiv preprint arXiv:2503.06520*, 2025.
- Lu et al. (2024)
Chunhao Lu, Qiang Lu, and Jake Luo.
An explainable vision question answer model via diffusion chain-of-thought.
In *European Conference on Computer Vision*, pp. 146–162. Springer, 2024.
- Ma et al. (2024)
Zixian Ma, Jianguo Zhang, Zhiwei Liu, Jieyu Zhang, Juntao Tan, Manli Shu, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, Caiming Xiong, et al.
Taco: Learning multi-modal action models with synthetic chains-of-thought-and-action.
*arXiv preprint arXiv:2412.05479*, 2024.
- Ni et al. (2022)
Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling.
Expanding language-image pretrained models for general video recognition.
In *ECCV*, 2022.
- OpenAI et al. (2024)
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, and etc. Florencia Leoni Aleman.
Gpt-4 technical report, 2024.
URL [https://arxiv.org/abs/2303.08774](https://arxiv.org/abs/2303.08774).
- Pan et al. (2022)
Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li.
St-adapter: Parameter-efficient image-to-video transfer learning.
In *NeurIPS*, 2022.
- Park et al. (2025)
Jinyoung Park, Jeehye Na, Jinyoung Kim, and Hyunwoo J Kim.
Deepvideo-r1: Video reinforcement fine-tuning via difficulty-aware regressive grpo.
*arXiv preprint arXiv:2506.07464*, 2025.
- Park et al. (2023)
Jungin Park, Jiyoung Lee, and Kwanghoon Sohn.
Dual-path adaptation from image to video transformers.
In *CVPR*, 2023.
- Peng et al. (2025)
Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang.
Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl.
*arXiv preprint arXiv:2503.07536*, 2025.
- Qi et al. (2024)
Ji Qi, Ming Ding, Weihan Wang, Yushi Bai, Qingsong Lv, Wenyi Hong, Bin Xu, Lei Hou, Juanzi Li, Yuxiao Dong, and Jie Tang.
Cogcom: Train large vision-language models diving into details through chain of manipulations, 2024.
URL [https://arxiv.org/abs/2402.04236](https://arxiv.org/abs/2402.04236).
- Qian et al. (2025a)
Chengxuan Qian, Kai Han, Jingchao Wang, Zhenlong Yuan, Chongwen Lyu, Jun Chen, and Zhe Liu.
Dyncim: Dynamic curriculum for imbalanced multimodal learning.
*arXiv preprint arXiv:2503.06456*, 2025a.
- Qian et al. (2025b)
Chengxuan Qian, Shuo Xing, Shawn Li, Yue Zhao, and Zhengzhong Tu.
Decalign: Hierarchical cross-modal alignment for decoupled multimodal representation learning.
*arXiv preprint arXiv:2503.11892*, 2025b.
- Qian et al. (2024)
Haotian Qian, YD Chen, Shengtao Lou, Fahad Shahbaz Khan, Xiaogang Jin, and Deng-Ping Fan.
Maskfactory: Towards high-quality synthetic data generation for dichotomous image segmentation.
*NeurIPS 24*, 2024.
- Qian et al. (2025c)
Kangan Qian, Sicong Jiang, Yang Zhong, Ziang Luo, Zilin Huang, Tianze Zhu, Kun Jiang, Mengmeng Yang, Zheng Fu, Jinyu Miao, Yining Shi, He Zhe Lim, Li Liu, Tianbao Zhou, Huang Yu, Yifei Hu, Guang Li, Guang Chen, Hao Ye, Lijun Sun, and Diange Yang.
Agentthink: A unified framework for tool-augmented chain-of-thought reasoning in vision-language models for autonomous driving, 2025c.
- Qu et al. (2025)
Xiangyan Qu, Jing Yu, Jiamin Zhuang, Gaopeng Gou, Gang Xiong, and Qi Wu.
MADS: multi-attribute document supervision for zero-shot image classification.
*CoRR*, abs/2503.06847, 2025.
- Radford et al. (2021)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.
Learning transferable visual models from natural language supervision.
In *International conference on machine learning*, pp. 8748–8763. PMLR, 2021.
- Rasheed et al. (2023)
Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan.
Fine-tuned clip models are efficient video learners.
In *CVPR*, 2023.
- Shao et al. (2024)
Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li.
Visual cot: Unleashing chain-of-thought reasoning in multi-modal language models, 2024.
- Shen et al. (2025)
Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al.
Vlm-r1: A stable and generalizable r1-style large vision-language model.
*arXiv preprint arXiv:2504.07615*, 2025.
- Song et al. (2025)
Zijian Song, Xiaoxin Lin, Qiuming Huang, Guangrun Wang, and Liang Lin.
Siri-bench: Challenging vlms’ spatial intelligence through complex reasoning tasks.
*arXiv preprint arXiv:2506.14512*, 2025.
- Soomro et al. (2012)
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.
Ucf101: A dataset of 101 human actions classes from videos in the wild.
*Arxiv e-prints*, 2012.
- Su et al. (2025a)
Alex Su, Haozhe Wang, Weiming Ren, Fangzhen Lin, and Wenhu Chen.
Pixel reasoner: Incentivizing pixel-space reasoning with curiosity-driven reinforcement learning.
*arXiv preprint arXiv:2505.15966*, 2025a.
- Su et al. (2025b)
Zhaochen Su, Linjie Li, Mingyang Song, Yunzhuo Hao, Zhengyuan Yang, Jun Zhang, Guanjie Chen, Jiawei Gu, Juntao Li, Xiaoye Qu, et al.
Openthinkimg: Learning to think with images via visual tool reinforcement learning.
*arXiv preprint arXiv:2505.08617*, 2025b.
- Sun et al. (2025)
Guangyan Sun, Mingyu Jin, Zhenting Wang, Cheng-Long Wang, Siqi Ma, Qifan Wang, Tong Geng, Ying Nian Wu, Yongfeng Zhang, and Dongfang Liu.
Visual agents as fast and slow thinkers.
In *The Thirteenth International Conference on Learning Representations*, 2025.
- Wang et al. (2025a)
Jiacong Wang, Zijian Kang, Haochen Wang, Haiyong Jiang, Jiawen Li, Bohong Wu, Ya Wang, Jiao Ran, Xiao Liang, Chao Feng, et al.
Vgr: Visual grounded reasoning.
*arXiv preprint arXiv:2506.11991*, 2025a.
- Wang et al. (2021)
Mengmeng Wang, Jiazheng Xing, and Yong Liu.
Actionclip: A new paradigm for video action recognition.
*Arxiv e-prints*, 2021.
- Wang et al. (2025b)
Qi Wang, Yanrui Yu, Ye Yuan, Rui Mao, and Tianfei Zhou.
Videorft: Incentivizing video reasoning capability in mllms via reinforced fine-tuning.
*arXiv preprint arXiv:2505.12434*, 2025b.
- Wang et al. (2025c)
Ye Wang, Qianglong Chen, Zejun Li, Siyuan Wang, Shijie Guo, Zhirui Zhang, and Zhongyu Wei.
Simple o3: Towards interleaved vision-language reasoning.
*arXiv preprint arXiv:2508.12109*, 2025c.
- Wang et al. (2025d)
Ye Wang, Ziheng Wang, Boshen Xu, Yang Du, Kejun Lin, Zihan Xiao, Zihao Yue, Jianzhong Ju, Liang Zhang, Dingyi Yang, et al.
Time-r1: Post-training large vision language model for temporal video grounding.
*arXiv preprint arXiv:2503.13377*, 2025d.
- Wasim et al. (2023)
Syed Talal Wasim, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah.
Vita-clip: Video and text adaptive clip via multimodal prompting.
In *CVPR*, 2023.
- Weng et al. (2023)
Zejia Weng, Xitong Yang, Ang Li, Zuxuan Wu, and Yu-Gang Jiang.
Open-vclip: Transforming clip to an open-vocabulary video model via interpolated weight optimization.
In *ICML*, 2023.
- Xiong et al. (2025)
Feng Xiong, Hongling Xu, Yifei Wang, Runxi Cheng, Yong Wang, and Xiangxiang Chu.
Hs-star: Hierarchical sampling for self-taught reasoners via difficulty estimation and budget reallocation.
*arXiv preprint arXiv:2505.19866*, 2025.
- Yang et al. (2023)
Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li.
Aim: Adapting image models for efficient video action recognition.
*Arxiv e-prints*, 2023.
- Yang et al. (2021)
Zhidong Yang, Fa Zhang, and Renmin Han.
Self-supervised cryo-electron tomography volumetric image restoration from single noisy volume with sparsity constraint.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, pp. 4056–4065, October 2021.
- Yang et al. (2024)
Zhidong Yang, Dawei Zang, Hongjia Li, Zhao Zhang, Fa Zhang, and Renmin Han.
Self-supervised noise modeling and sparsity guided electron tomography volumetric image denoising.
*Ultramicroscopy*, 255:113860, 2024.
ISSN 0304-3991.
- Yao et al. (2019)
Guangle Yao, Tao Lei, and Jiandan Zhong.
A review of convolutional-neural-network-based action recognition.
*Pattern Recognition Letters*, 118, 2019.
- Yu et al. (2025)
Yating Yu, Congqi Cao, Yifan Zhang, and Yan-Ning Zhang.
Learning to generalize without bias for open-vocabulary action recognition.
*CoRR*, abs/2502.20158, 2025.
- Yuan et al. (2025)
Zhenlong Yuan, Jing Tang, Jinguo Luo, Rui Chen, Chengxuan Qian, Lei Sun, Xiangxiang Chu, Yujun Cai, Dapeng Zhang, and Shuo Li.
Autodrive-r2: Incentivizing reasoning and self-reflection capacity for vla model in autonomous driving.
*arXiv preprint arXiv:2509.01944*, 2025.
- Zhang et al. (2024a)
Dapeng Zhang, Dayu Chen, Peng Zhi, Yinda Chen, Zhenlong Yuan, Chenyang Li, Sunjing, Rui Zhou, and Qingguo Zhou.
Mapexpert: Online hd map construction with simple and efficient sparse map element expert, 2024a.
URL [https://arxiv.org/abs/2412.12704](https://arxiv.org/abs/2412.12704).
- Zhang et al. (2025a)
Dapeng Zhang, Jing Sun, Chenghui Hu, Xiaoyan Wu, Zhenlong Yuan, Rui Zhou, Fei Shen, and Qingguo Zhou.
Pure vision language action (vla) models: A comprehensive survey, 2025a.
URL [https://arxiv.org/abs/2509.19012](https://arxiv.org/abs/2509.19012).
- Zhang et al. (2025b)
Haoji Zhang, Xin Gu, Jiawen Li, Chixiang Ma, Sule Bai, Chubin Zhang, Bowen Zhang, Zhichao Zhou, Dongliang He, and Yansong Tang.
Thinking with videos: Multimodal tool-augmented reinforcement learning for long video reasoning, 2025b.
URL [https://arxiv.org/abs/2508.04416](https://arxiv.org/abs/2508.04416).
- Zhang et al. (2024b)
Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang, Qingsong Wen, Zhang Zhang, et al.
Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans?
*arXiv preprint arXiv:2408.13257*, 2024b.
- Zhang et al. (2025c)
Yi-Fan Zhang, Xingyu Lu, Shukang Yin, Chaoyou Fu, Wei Chen, Xiao Hu, Bin Wen, Kaiyu Jiang, Changyi Liu, Tianke Zhang, et al.
Thyme: Think beyond images.
*arXiv preprint arXiv:2508.11630*, 2025c.
- Zhao et al. (2025a)
Fei Zhao, Chengcui Zhang, Runlin Zhang, Tianyang Wang, and Xi Li.
Mitigating image captioning hallucinations in vision-language models, 2025a.
URL [https://arxiv.org/abs/2505.03420](https://arxiv.org/abs/2505.03420).
- Zhao et al. (2025b)
Shitian Zhao, Haoquan Zhang, Shaoheng Lin, Ming Li, Qilong Wu, Kaipeng Zhang, and Chen Wei.
Pyvision: Agentic vision with dynamic tooling.
*arXiv preprint arXiv:2507.07998*, 2025b.
- Zheng et al. (2025)
Ziwei Zheng, Michael Yang, Jack Hong, Chenxiao Zhao, Guohai Xu, Le Yang, Chao Shen, and Xing Yu.
Deepeyes: Incentivizing" thinking with images" via reinforcement learning.
*arXiv preprint arXiv:2505.14362*, 2025.

## Appendix

In this appendix, we provide more experimental details, related work, tool library, and discussions for a comprehensive evaluation and understanding of our method.
Detailed contents are as follows:

## Appendix A Experiment

### A.1 More Experimental Details

Datasets.
Our experiments are conducted on a comprehensive set of benchmark datasets designed for action recognition tasks, including UCF-101 (Soomro et al., 2012), HMDB-51 (Kuehne et al., 2011), Kinetics-400 (K-400) (Carreira & Zisserman, 2017), Kinetics-600 (K-600) (Carreira et al., 2018), and Something-to-Something V2 (SSv2) (Goyal et al., 2017). The Kinetics series represents large-scale datasets with 400 and 600 distinct action classes in K-400 and K-600, respectively, where the latter expands upon the former by incorporating an additional 220 non-overlapping categories. Specifically, K-400 contains 240k training samples and 20k validation samples, while K-600 provides 410k training samples and 29k validation samples, all sourced from YouTube clips depicting human actions. Transitioning to smaller but widely adopted benchmarks, UCF-101 comprises 13,320 video samples distributed across 101 action classes, with three official train/test splits that allocate 9,537 training and 3,783 testing samples. This dataset captures diverse daily activities through short-duration videos, serving as a standard evaluation protocol in action recognition research. HMDB-51 follows a similar structure with 6,849 videos spanning 51 action categories, maintaining over 101 samples per class and three canonical splits. Notably, its content integrates clips from cinematic productions, public archives, and online platforms, offering rich contextual variations. For fine-grained interaction analysis, we evaluate on SSv2, which focuses on object-manipulation actions involving 174 semantically distinct categories. This dataset features 168,913 training samples and 24,777 testing samples, emphasizing temporally complex interactions between humans and objects that require precise spatiotemporal modeling.

Evaluation metrics.
Building upon prior work in vision-language pre-training (Huang et al., 2024a; Yu et al., 2025; Weng et al., 2023), we adopt two complementary evaluation frameworks to benchmark performance. For the base-to-novel setting, other methods train models exclusively on high-frequency action categories while testing their generalization to low-frequency counterparts. Differently, our method is fine-tuned only on the high-frequency action categories of HMDB-51 and evaluated in a zero-shot manner on K-400, UCF-101, and SSv2, making our evaluation more challenging.
For this configuration, we apply the methodology across four benchmark datasets, including UCF-101, HMDB-51, K-400, and SSv2, systematically partitioning each into base and novel class subsets. Final results represent averages across these configurations, computed using the standard validation splits: for HMDB-51 and UCF-101, only the first of three available splits is used for novel class evaluation, while K-400 and SSv2 leverage their single validation split in full.

For the cross-dateset setting, we establish cross-domain evaluation protocols by strategically pairing training and testing datasets. Specifically, models trained on HMDB-51 (Kuehne et al., 2011) are evaluated on UCF-101 (Soomro et al., 2012) and K-600 (Carreira et al., 2018), while models trained on UCF-101 are tested on HMDB-51. This bidirectional transfer design enables rigorous analysis of domain adaptation capabilities. Performance characterization for HMDB-51 and UCF-101 incorporates their native three-validation-split structure, reporting both mean top-1 accuracy and standard deviation across splits. For the K-600 evaluation, we focus on 220 exclusive categories absent in Kinetics-400, adopting the standardized three-split protocol from prior work (Ni et al., 2022; Rasheed et al., 2023). Each split contains 160 novel categories, with final results representing their average performance. This multi-faceted evaluation strategy provides comprehensive insights into both within-dataset generalization and cross-dataset transfer capabilities.

More Hyperparameter Configurations.
The proposed method was implemented on a system equipped with an Intel(R) Xeon(R) Platinum 8480+ CPU and eight NVIDIA H20 GPUs (90 GB memory each). For training, the reinforcement learning (RL) pipeline utilized the TRL framework, with supervised fine-tuning (SFT) and RL stages taking 1 hour and 20 hours, respectively.

### A.2 More Ablation Studies

**Table 4: More ablations of Video-STAR under cross-dataset setting. "Split" denote evaluating across three validation splits. The best and second best performances are highlighted.**
| Method | UCF-101 | HMDB-51 | K-600 | Tool Library | UCF-101 | HMDB-51 | K-600 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Split$\uparrow$ | Split$\uparrow$ | Split$\uparrow$ | Split$\uparrow$ | Split$\uparrow$ | Split$\uparrow$ |  |  |
| *(a)* Qwen2.5-VL-3B | 58.1$\pm$0.2 | 42.1$\pm$0.3 | 37.7$\pm$1.8 | *(g)* *w/o.* *Human* | 95.3$\pm$0.3 | 84.3$\pm$0.2 | 88.3$\pm$0.8 |
| *(b)* Video-STAR-3B | 96.7$\pm$0.3 | 86.2$\pm$0.2 | 90.5$\pm$0.7 | *(h)* *w/o.* *Pose* | 92.8$\pm$0.4 | 81.2$\pm$0.2 | 83.7$\pm$1.1 |
| *(c)* *SFT: w/o. Val* | 94.8$\pm$0.3 | 83.2$\pm$0.2 | 86.6$\pm$0.7 | *(i)* *w/o.* *Action* | 93.6$\pm$0.3 | 82.3$\pm$0.2 | 85.5$\pm$0.9 |
| *(d)* *RL: w/. Same $w_{k}$* | 94.2$\pm$0.3 | 82.4$\pm$0.3 | 85.7$\pm$1.0 | *(j)* *w/o.* *Video* | 94.7$\pm$0.3 | 83.7$\pm$0.2 | 87.4$\pm$0.7 |

Data Assessment.
In row *(c)* of Tab. [4](https://arxiv.org/html/2510.08480v1#A1.T4), we remove the data assessment stage from our SFT training. Accuracy drops 1.9%, 3.0%, and 3.9% from the full model in *(b)* on UCF-101, HMDB-51, and K-600, respectively. This indicates that filtering out inconsistent reasoning is important for maintaining data reliability and supporting tool-augmented logical coherence. Without data assessment, errors and factual inconsistencies in training data degrade cross-dataset generalization.

Weighting $w_{k}$ in Sub-Motion Reward $R_{sub}$.
Row *(d)* evaluates the effect of assigning equal weights to all sub-motions. Compared to the full model in *(b)*, accuracy decreases 2.5%, 3.8%, and 4.8% in *(d)*, showing that sub-motion-specific weighting helps prioritize semantically discriminative primitives. This confirms that our strategy of assigning dynamic importance to different sub-motions is crucial for fine-grained action reasoning.

Tool Library.
In rows *(g)–(j)*, we analyze the contribution of each tool in our framework. Removing the human detection tool *(g)* reduces accuracy 1.4%, 1.9%, and 2.2%, illustrating its role in isolating action-relevant regions and avoiding visual-semantic confusion. Without pose estimation *(h)*, performance declines 3.9%, 5.0%, and 6.8%, confirming that structured kinematic representation is essential for capturing subtle motion cues. Excluding the action-retrieval tool *(i)* yields a decrease 3.1%, 3.9%, and 5.0%, showing that high-level category-specific definitions help bridge label–motion gaps. Removing the video-frame description tool *(j)* lowers accuracy 2.0%, 2.5%, and 3.1%, indicating that detailed temporal-segment descriptions contribute to disambiguating multi-phase actions. The full model *(b)*, which integrates the entire tool library, achieves the highest scores, proving that these tools are complementary and essential for robust performance.

### A.3 More Case Studies

As shown in Fig. [5](https://arxiv.org/html/2510.08480v1#A1.F5) and Fig. [6](https://arxiv.org/html/2510.08480v1#A1.F6), we present more case studies between Qwen2.5-VL-3B and our Video-STAR-3B model on the HMDB-51 dataset.
Our method can effectively leverage domain-specific tools combined with sub-action decomposition to achieve fine-grained action recognition.

Figure: Figure 5: More Case Study between Qwen2.5-VL-3B and our Video-STAR-3B. Qwen2.5-VL-3B misclassifies "push" as "play", while our Video-STAR-3B accurately identifies the correct action.
Refer to caption: x6.png

Figure: Figure 6: More Case Study between Qwen2.5-VL-3B and our Video-STAR-3B. Our Video-STAR-3B can accurately identify the correct action.
Refer to caption: x7.png

## Appendix B Related Work

### B.1 Open-Vocabulary Action Recognition.

Building on the robust feature extraction capabilities of CLIP’s pre-training, researchers have devised numerous video recognition methods leveraging its cross-modal alignment strengths.
Initial approaches primarily focused on adapting CLIP via full fine-tuning or parameter-efficient strategies. For instance, ActionCLIP (Wang et al., 2021) pioneered the adaptation of CLIP for video recognition by fine-tuning and augmenting the entire model to capture motion. Similarly, STAN (Liu et al., 2023a) introduced auxiliary networks to extract temporal features from CLIP’s frozen representations, while ViFi-CLIP (Rasheed et al., 2023) simultaneously fine-tuned both visual and text encoders to achieve better performance. These methods, though effective in closed-set scenarios, often overfit to static cues due to extensive parameter updates.

There are two primary paradigms for parameter-efficient paradigms: architecture adaptation and prompt engineering.
Adaptation-based methods like Adaptformer (Chen et al., 2022), X-CLIP (Ni et al., 2022), and ST-Adapter (Pan et al., 2022) employ compact neural modules in CLIP to distill temporal knowledge from static image representations. Prompt engineering approaches like AIM (Yang et al., 2023) and DualTrans (Park et al., 2023), which inject learnable prefix tokens into CLIP’s text encoder to bias the model towards action-specific semantics, while VPT (Ju et al., 2022) and Vita-CLIP (Wasim et al., 2023) explore both prefix and suffix prompting strategies to align video features. Despite these, their performance is limited by adapters’ limited capacity, prompt-based modulations, and architectural rigidity, which hinders cross-architecture generalization.

Recent efforts explicitly targeted open-vocabulary generalization. Open-VCLIP (Weng et al., 2023) introduced weight interpolation to regularize fine-tuning, but this approach restricted compatibility to architectures with identical weight dimensions. FROSTER (Huang et al., 2024b) further advanced this by leveraging residual feature distillation.
Open-MeDe (Yu et al., 2025) adopts a meta-learning framework that explicitly mitigates static bias via cross-batch meta-optimization, yet their dependency on CLIP’s static generalization hindered robustness to motion-centric tasks.
Differently, our work diverges from these paradigms by proposing a framework that integrates sub-motion decomposition with tool-augmented RL, thereby enabling robust open-vocabulary inference.

### B.2 Multimodal LLMs Reasoning.

The advancement of deep learning algorithm has led to the development of numerous self-supervised learning methods (Yang et al., 2021; 2024), vision-language model (Song et al., 2025; Chen et al., 2023), multimodal balance (Qian et al., 2025b; a), image retrieval (Li et al., 2025b; c; Chen et al., 2025b), autonomous driving  (Zhang et al., 2024a; 2025a; Yuan et al., 2025), and image segmentation (Chen et al., 2024; Qian et al., 2024; Qu et al., 2025).
Recent advances in large language models (LLMs) have demonstrated that RL-based post-training (Chu et al., 2025; Ji et al., 2025; Xiong et al., 2025) can significantly enhance reasoning capabilities, as exemplified by OpenAI-o1 (Jaech et al., 2024) and DeepSeek-R1 (Guo et al., 2025).
Several works have extended these paradigms to multimodal language models (MLLMs) for tasks such as mathematical and scientific image VQA (Peng et al., 2025; Huang et al., 2025; Lu et al., 2024), image segmentation and grounding (Liu et al., 2025; Bai et al., 2025b; Shen et al., 2025), video spatial or temporal grounding (Wang et al., 2025d; Ge et al., 2025; Park et al., 2025) and video understanding VQA (Feng et al., 2025; Wang et al., 2025b; Cheng et al., 2025).
Despite these, existing methods remain limited in cross-modal interaction and hallucination, especially for long video scenarios (Chen et al., 2025a). To address this, we propose multimodal CoT reasoning through tool-augmented RL, which explicitly reduces hallucination through structured perception.

### B.3 Tool-Augmented Agentic System.

Recent advancements in LLMs have demonstrated that integrating external tools can significantly enhance multimodal reasoning capabilities. Early works like FAST (Sun et al., 2025) and MVoT (Li et al., 2025a) established foundational approaches by incorporating visual evidence into reasoning processes, forming multimodal CoT for image tasks. Concurrently, LLaVa-Plus (Liu et al., 2023b) pioneered training strategies for tool use in visual reasoning, while Visual Program Distillation (Hu et al., 2024) leveraged program-derived CoT data to transfer tool-use skills. Visual CoT (Shao et al., 2024) further enriched this domain by creating synthetic datasets to boost reasoning. CogCoM (Qi et al., 2024) introduced six manipulation strategies through synthetic chain-of-manipulation (CoM) data, and TACO (Ma et al., 2024) provided 273K multimodal reasoning traces from 15 visual tools. Recent methods like Simple (Wang et al., 2025c), THYME (Zhang et al., 2025c), and PyVision (Zhao et al., 2025b) extended tool use with RL, thus enhancing general reasoning capacity. Visual search capabilities were advanced by DeepEyes (Zheng et al., 2025), VGR (Wang et al., 2025a), and OpenThinkImg (Su et al., 2025b), which integrated tools such as image zoom-in and sketching to improve visual analysis. However, partial methods often rely on static tool pipelines, limiting their ability to disentangle complex actions in open-vocabulary settings. Our framework addresses this by explicitly modeling contextual sub-motions and dynamically balancing tool usage efficiency with hierarchical motion relevance.

## Appendix C Tool Library Details

This section provides a detailed specification of tools used by our Video-STAR framework, including human detection tools, pose estimation tools, and online retrieval-augmentation tools.

### C.1 Human detection Tools

Our framework employs the Ultralytics YOLO 11 (Weng et al., 2023) detector as a human detection model. We chose this model for its optimization for real-time performance and accuracy. Its primary function is to locate people in video frames. The model employs an anchor-free detection architecture combined with spatial attention mechanisms. This design enables it to handle common challenges such as occlusions, varying human scales, and crowded scenarios. The tool’s ability to ensure precise localization of human bounding boxes, even in cluttered environments, is critical for our framework. It allows our model to isolate action-relevant regions from background distractions, which is a key step in reducing visual-semantic ambiguity and preventing model hallucination during the reasoning process.

### C.2 Pose estimation Tools

For pose estimation, we utilize the skeletonization capability of Ultralytics YOLO 11, which translates a detected person into a structured kinematic representation. The model outputs a skeleton with 17 keypoints that follow the COCO format, including joints for the elbows, wrists, and knees. It achieves this with sub-pixel precision, which is essential for detailed motion analysis. The model infers these joint positions using its knowledge of hierarchical spatial relationships between body parts. For instance, it understands the natural correlation between hip and knee angles during locomotion. This underlying knowledge allows it to generate coherent skeletons even when body parts are in fast motion or partially occluded. This fine-grained representation enables our system to capture subtle motion patterns. For example, we can analyze the precise wrist rotation in a "handshake" or the exact knee flexion during a "jump".

### C.3 Online Retrieval-Augmentation Tools

We use the Qwen API as our online retrieval-augmentation tool. This tool provides our framework with essential contextual information. It has two primary functions,
❶ The first function is to resolve the ambiguity of high-level action definitions. We use this tool to get a detailed, category-specific explanation for a given action label. It helps redefine action words through their transitional phases (e.g., "stand" is described as "transit from sit or lay to stand"). This process provides our model with a deeper semantic understanding, closing the gap between a simple label and its complex physical meaning. ❷ The second function is to describe the motion within specific key frames, which helps the model understand complex, multi-phase movements. After our framework extracts temporally salient frames (e.g., frames with maximum joint displacement), we send these frames to the tool. The tool then generates a concise textual description of the action happening at that moment. This description emphasizes both spatial configurations, like "rapid arm extension," and temporal dynamics, like "followed by torso rotation". This maps descriptive text precisely to informative video segments, allowing the model to disambiguate sequential actions by understanding each component.

## Appendix D Prompts

During the supervised fine-tuning (SFT) stage of Video-STAR, we designed a structured input prompt to generate high-quality chain-of-thought (CoT) data. The prompt template is as follows:

First Round Prompts:

Second Round Prompts:

This structured prompt ensures our dataset contains diverse and causally plausible reasoning processes, which are critical for cultivating the model’s foundational perception and planning capabilities before RL fine-tuning.

## Appendix E LLM clarification

We clarify the role of Large Language Models (LLMs) in the development of this manuscript. During the drafting process, LLMs were utilized primarily for two purposes: translating initial technical content from Chinese to English and refining the language of the final draft. This included correcting grammatical inconsistencies, optimizing sentence structure, and improving the clarity and coherence of the narrative. It is essential to emphasize that all conceptual contributions, including the formulation of open-vocabulary action recognition challenges, the design of the Video-STAR framework (e.g., contextual sub-motion decomposition and hierarchical reward mechanisms), the integration of domain-specific tools (e.g., YOLO 11 for pose estimation and Qwen API for explanations), and the experimental design across HMDB-51, UCF-101, and Kinetics-400/600 datasets, are the original work of the human authors. The LLM was employed strictly as a linguistic tool to enhance the presentation of ideas, with no involvement in hypothesis generation, algorithmic innovation, or data analysis. The core scientific claims, methodological innovations, and empirical validation presented in this paper is entirely attributable to the research team.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. The paper claims to mitigate “text-centric reasoning,” but the explanation of how this is achieved is somewhat abstract.
Please explicitly describe the mechanism by which visual grounding is enforced during training and inference.
soundness: 3
strengths: 1.The paper is motivated by a well-defined and practically significant problem—bridging the gap between text-centric reasoning and visually grounded inference.

2.The use of a multimodal CoT to train MLLMs for structured, tool-guided reasoning is a valuable direction.

3.The framework integrates detection, pose estimation, and semantic reasoning tools with a hierarchical reward design, which is novel and effectively demonstrated across multiple benchmarks.
summary: This paper addresses two fundamental limitations of current multimodal large language models (MLLMs) for open-vocabulary action recognition:
(1) the over-reliance on textual priors that neglect domain-specific visual cues, and
(2) the inability to distinguish semantically ambiguous actions in open-vocabulary settings.
These issues are indeed of great importance and widely exist across multiple MLLM-based video understanding tasks. The proposed Video-STAR framework attempts to mitigate these problems by constructing multimodal Chain-of-Thought (CoT) data and introducing tool-augmented reasoning with reinforcement learning optimization.
weaknesses: Major Comments

1.During data construction, the model is trained using decomposed motion sequences (sub-actions).
How does the framework behave when encountering previously unseen actions in the open-vocabulary test set?
Is there any mechanism ensuring compositional generalization beyond the motion patterns observed during training?
Without such a mechanism, the model might overfit to the seen sub-motion combinations.


2.How exactly is each action decomposed into sub-actions?
Is the decomposition manually annotated, automatically generated, or derived from an existing motion ontology?
Moreover, how do the authors ensure that the set of sub-actions can comprehensively cover unseen action categories during testing?
Since this decomposition is central to the model’s reasoning ability, more transparency on this process is necessary for reproducibility and understanding its generalization scope.

3.Video-STAR performs task-specific supervised fine-tuning (SFT) and reinforcement learning (RL), while most baselines (e.g., Qwen2.5-VL,) are evaluated without any fine-tuning.
This introduces a fairness issue: the superior performance of Video-STAR may partly result from extra supervision rather than the proposed method itself.
A fairer comparison would include a fine-tuned Qwen2.5 baseline trained on the same dataset but without tool usage and sub-motion decomposition, to isolate the true contribution of the proposed framework.

4.The ablation studies only compare the presence vs. absence of tool usage but do not analyze which tool or combination contributes most.
How is the tool selected in practice?
What is the performance when all tools are used simultaneously (pose, detection, action explanation, and video description)?
A more detailed comparison of tool selection strategies would clarify whether the proposed policy is optimal or if simpler combinations yield similar gains.

5.Since Video-STAR relies on multiple external tools, the inference pipeline likely introduces additional computational overhead.
The paper should report both the overall inference latency and the module-wise cost (e.g., tool invocation vs. model reasoning).
Comparing the efficiency of Video-STAR with standard MLLMs would help quantify the trade-off between accuracy improvement and computational expense.

6. Line 213: The symbol T_r appears for the first time without explicit definition. Please clarify its meaning and source.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: The questions are the same as weakness.
soundness: 3
strengths: 1. The paper names two real OVAR pain points, cross-modal hallucination and similar-action confusion, and every design choice (tools, sub-motions, hierarchical reward) points back to those, making a coherent motivation.
2. Treating “shoot ball” as “bend → jump → arm extend → release” matches how actions are actually separable in video; it’s more plausible than pure text-CoT on top of global video tokens.
3. Tools aren’t a fixed pipeline — the model decides whether to call pose / human / RAG /video description, and the reward penalizes useless tool calls. That’s better than many “agentic VLM” papers that just always run pose.
summary: The paper discusses this idea: don’t let the model guess in one shot. Make it first break the action into sub-motions (arms, torso, legs, contact), then match those to candidate actions, and finally score them — and let the model call tools (YOLO human detection, pose estimation, Qwen-based action/video explanation) when it thinks vision cues are not enough.
weaknesses: 1. They say open-vocab, but the system leans on online RAG / Qwen API to pull category-specific definitions at inference. That narrows the search space. It’s closer to “recognition with external label dictionary + video grounding” than to “truly open” recognition.
2. YOLO 11 for human + pose, Qwen API for explanation / video description — that’s a very specific tool stack.
3. First round: “which tool(s)?” Second round: “do sub-motion reasoning.” Plus GRPO sampling 4–6 responses. That might not be cheap for real-time video, and the paper doesn’t talk about latency / streaming.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: See Weaknesses.
soundness: 3
strengths: A novel Tool Augmented CoT framework and implementation approach for open-vocabulary action recognition (OVAR), concrete evaluations of 5 benchmarks and new SOTA performance.
summary: This paper proposes, implements and investigates a novel strategy and framework, i.e., Tools Augmented CoT reasoning for zero-shot fine-grained human action recognition. The framework employs VFMs, VLMs and Visual RAG to extract sub-concept vision representations such as human, pose, and video explanation, and generated a combined prompt on proposed format for CoT to MLLM for final stage prediction. The core innovations are the format definition and implementation of sub-action decomposition, candidate selection, and matching scoring for CoT and reinforcement learning. Concrete evaluations are performed on five formal benchmarks and the results shown significant improvements over the SOTA.
weaknesses: There are still some uncertain issues. (1) the experiments on only one base MLLM model, Qwen2.5-VL, are reported, where the training prompts of reasoning chain for CoT are generated by Qwen2.5-VL-72B, and then fine-tune the small-size model Qwen2.5-VL-3B and Qwen2.5-VL-7B for experiments, is teacher-student knowledge distillation on the proposed reasoning chain format able to achieve similar effectiveness? May be better to add more results on other leading frontier MLLMs such as InternVL2.5, Gemini-2.5-Pro, Llama, etc. (2) is it applicable to professional actions such as FineGym, Diving? Where expertise sub-action concepts might not well be learned for AGI models. (3) On lines 356-360, are the protocols defined for previous benchmarks? Please cited them. If novel classes Y_N are completed unknown in MLLM, how to generate the novel nouns of the new classes? So that the base VLM and MLLM have been trained on related concepts, and may be not strictly zero-shot performance. (4) As the training sub-concepts and reasoning chains are generated by VFM, VLM, and MLLM, are there hallucinated chains which lead to final correct answers on ground truth? Maybe the discussion on the Visual Grounded CoT is helpful.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see the weakness.
soundness: 3
strengths: The combination of sub-motion decomposition and tool-augmented RL constitutes a meaningful departure from static cross-modal alignment pipelines commonly used in OVAR.

The method delivers consistent and large improvements across diverse benchmarks and evaluation settings.
summary: The paper proposes Video-STAR, a framework for open-vocabulary action recognition that integrates contextual sub-motion decomposition with tool-augmented reinforcement learning on top of MLLMs. Actions are no longer treated as atomic labels; instead, they are decomposed into discriminative motion primitives, while external tools are invoked to reduce cross-modal hallucinations and enable category-specific reasoning. A hierarchical reward is designed to jointly optimize tool-usage efficiency, structural coherence, and sub-motion relevance. Extensive experiments demonstrate substantial gains over CLIP-based baselines and vanilla MLLMs across multiple settings.
weaknesses: 1. While the integration is well-designed, the core building blocks (tool-augmented CoT, RL-based post-training, and sub-action decomposition) are all known paradigms; the novelty is primarily at the system-level composition rather than at the level of a fundamentally new principle.

2. The approach depends on specific external tools, yet the paper does not analyze robustness to tool inaccuracies or the portability of the method under alternative tool choices.

3. The computational overhead of repeated tool invocation and multi-round RL inference is not reported nor compared against CLIP-based or purely SFT-based OVAR pipelines. Furthermore, the training cost of GRPO fine-tuning is also not quantified or compared with prior methods.

4. Although standard CLIP-based OVAR methods are included, the paper does not benchmark against recent LLM-augmented CLIP paradigms that explicitly incorporate generative priors for action understanding, such as [1–3].

5. The evaluation is conducted primarily against mid-scale or earlier-generation MLLMs (e.g., Qwen2.5-VL), without comparison to state-of-the-art frontier models (e.g., Qwen3-VL, GPT-5, Claude, Gemini), many of which already demonstrate strong video reasoning capabilities.

[1] Building a Multi-modal Spatiotemporal Expert for Zero-shot Action Recognition with CLIP

[2] Generating Action-Conditioned Prompts for Open-Vocabulary Video Action Recognition

[3] VTD-CLIP: Video-to-Text Discretization via Prompting CLIP


Decision: Accept (Poster)