Video-STAR: Reinforcing Open-Vocabulary Action Recognition with Tools
Authors: Zhenlong Yuan
Xiangyan Qu
Chengxuan Qian
Rui Chen
Jing Tang
Lei Sun
Xiangxiang Chu
Dapeng Zhang
Yiwei Wang
Yujun Cai
Shuo Li
Venue: ICLR 2026 Poster (2026)
Keywords: multimodal reasoning
vision-language model
action recognition
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Multimodal large language models (MLLMs) have demonstrated remarkable potential in bridging visual and textual reasoning, yet their reliance on text-centric priors often limits their ability to disentangle semantically similar actions in open-vocabulary scenarios. To address this, we propose Video-STAR, a framework that harmonizes contextual sub-motion decomposition with tool-augmented reinforcement learning for open-vocabulary action recognition (OVAR). Unlike prior methods that treat actions as monolithic entities, our approach innovatively decomposes actions into discriminative sub-motions for fine-grained matching while dynamically invokes domain-specific tools for cross-modal interleaving, thereby enabling category-specific reasoning capacity and reducing cross-modal hallucination. Moreover, by designing a hierarchical reward that balances tool-usage efficiency, sub-motion relevance, and structural coherence in reasoning, our method autonomously leverages external tools to prioritize sub-motion patterns without explicit supervision, transmitting from text-centric reasoning to visually grounded inference. Extensive evaluations on HMDB-51, UCF-101, SSv2, Kinetics-400, and Kinetics-600 datasets demonstrate our state-of-the-art performance, outperforming existing methods in distinguishing fine-grained actions and handling cross-modal hallucination, while maintaining computational efficiency.

Published as a conference paper at ICLR 2026
VIDEO-STAR: REINFORCING OPEN-VOCABULARY ACTION RECOGNITION WITH TOOLS
Zhenlong Yuan Xiangyan Qu' Chhngxuan Qian 2 Rui Chen', Jing Tang', Lei Sun't, Xiangxiang Chu', Dapeng Zhang', Yiwei Wang", Yujun Cai Shuo Li*
AMAP, Alibaba Group, University of California at Mereed, 3 University of Queensland, Case Westemn Reserve University
Equal contribution
Project Lead Comesponning Author
ABSTRACT
Multimodal large language models (MLLMs) have demonstrated remarkable po- tential in bridging visual and textual reasoning, yet their reliance on text-centric priors often limits their ability disentangle semantically similar actions in open- vocabulary scenarios. To address this, we propose STdeo-STAR, a framework that harmonizes contextual sub-motion decomposition with tool-augmented rein- forcement learning for open-vocabulary action recognition (OVAR). Unlike prior methods that treat actions monolithic entities, our approach innovatively de- composes actions into discriminative sub-motions for fine-grained matching while dynamically invoking Komain-speciin tools for cross moosll interleaving, thereby enabling category-speciig reasoning capacity and reducing cross- cross-modal halluci- nation. Moreover, by designing a hierarchical reward that balances tool-usage ef- ficiency. sub-motion relevance, and structural coherence in reasoning, our method autonomously leverages external tools prioritize sub-motion patterns without explicit supervision, transmitting from text-centric reasoning to visually grounded inference. Extensive evaluations on HMDB-51, UCF-101, SSv2, Kinetics-400, and Kinetics-600 datasets demonstrate our state-of-the-ee performance, outper- forming existing methods in distinguishing men-ggrined actions and handling cross-modal hallucination, validating our excellent robustness and generalization.
Inference Example
2VL-33
POSTER
Question
Video-G
What kind human action
Video STAR
21.13
Action: Throw, Golf,
Thinko Determine which visual and reasoning fools should be adapted T/Tninoo
saction> pose, estimation </action>
37 75
74.8%
90.3%
Training Data Construction
Training
<action> ses.iineeppaaatiinn/etti
YOLO
Online RAG
Supervised Fine- Tuning
Reinforcement
Threw Hand: Grips and releases the object..
Pase Estimation
Action Explonation
Learning
Gelf: Arms: Twists back and forth
Punch: Arm: Extends ferward swiftly.
Tools
Vision Encoder
Accuracy Reword
Format Reward
becompase key body parts
Humon Detection
Video Description
Arms:
slightly with
Vision- Language Model
Toal-usage Reward
Sub-AAtton Reward
Contact: Hands
Matching Candidate: Gelf, Hit, Punch.
[Score: 6/10], Punch: (Scone: 3/10).. Thirk
Scoring Candidete: Galf: [Score: 9/10). Hit:
Conaidate Generation
Sub-Motion Decomposition
Maschinn
Evaluation
koower belf sonswer
Work done during the internship at AMAP, Alibaha Group.

---

Published as a conference paper at ICLR 2026
(a) VLM + CoT Reasoning
(b) VLM Tool-Augmented CoT
(c) VLM Toal-Augmented & Sub-motion ToT
Outpal
Tools
Output
VLMG
Complicated Action
Sub-mottion becompasihon
Couestion fant
GQuestians
Hand Arms
Tomno
Lage
What of action shown in the
What kind oction is shown the vides? Grean Act Thros: A person is propeling on object through or. person hittiing a golf ball
(Question
What kind of oction is shows in the vides? Stean Action:
Sheen Action Throw, Self, Purch...
Golfi Armu Twests bock and forth, Hand Gripa club rightly.....
Outpur Format:
Teree Ratate qu dickyy Leg: keep on bround
<trinlo ing process c/hiinio
cenove roccionn type-/enswer>
inhrrisRcosooing procese//tirlo Preserracticnc/eeeee
Thros: Handi Grias laases object, Armi Extend outward rapedly
cthisto Key bedy parts, and novement:
thislo Observing the video the person
Thislo Step by step reaseting process The person in center of the video hitting something into There also
1. Hende:: in connect with gelf chin.
litting somathing with a pole. This baseball, the oction ThTiisoo
2. Arres Extended forward the swing bock
slighlly
Lage: Keep
their
Motchinng Condidatee Golf. Punch.
cutward, which simier the defirstionn of
Scoring Condidete: Self: [Score: 9/103: Match descriptional of
thross A porson is propelling or object through the or, so the action type is hitt </think corswer Throw sansverr
golf swing. with the arma owing ng the club and the lage in staree
Cross-
Cross-Modal Modal Hallucinations
Hit: [Scome: 6/10): Context of gol swing make this less likely.
Punchi [Score: 3/10) indicat on of fast or strilee </Thivio
Lock Category-specife Reasoning
Lock Category-
Less Hallucinations
specific Reasoning
Less Hollucinations
Category specific Reasoning
Key insight of Video-STAR. (a) MLLMs CoT is prone hallucinations due over- reliance on text-centric reasoning while ignoring visual cues, (b) MLLMs- + Tool-Augmente CoT mitigates hallucinations by integrating comainn-specife tools to extract visual information. However, both (a) and (b) lack category-specciff reasoning capabilities and struggle to distinguish semantically similar or complex actions. (c) Video-STAR enhances reasoning capacity by introducing contextual sub-motion decomposition, which disentangles actions into discriminative motion primitives. This enables fine-gres -rained action discrimination and robust performance in open- vocabbocayy scenarios.
Figure 1:
INTRODUCTION
Open-vocabulary action recognition (OVAR) aims to identify action classes not present during training, extending beyond the closed-vocabulary constraints of traditional action recognition sys- tems (Yao al., 2019; He et al., 2019). This capability is crucial for real-world applications where novel actions frequently emerge, such as surveillance systems, tent analysis platforms. While early approaches (Feichtenhofer al., 2019; Lin et al., 2019) re- lied on hand-crafted features and limited class taxonomies, recent advances in vision-language pre- training (Radford et al., 2021; Jia et al., 2021) have opened new possibilities for zero-shot action understanding. However, OVAR remains challenging due to the inherent complexity of human tions, which often involve subtle temporal dynamics, spatial relationships, and semantic similaritics that are difficult to distinguish through conventional approaches.
interaction, and con-
Recent multimodal large language models (MLLMs) have shown strong visual reasoning and cross- modal understanding (Shen et al., 2025; Feng et al., 2025). Through unified architecture that to- kenizes visual signals and fuses them with text via cross-modal attention, they achieve fine-grained grounding and stepwise reasoning. Systems such as Qwen VL. (Bai al., 2025a) and GPT-4 (Ope- nAI et al., 2024) consequently deliver strong performance on image captioning (Zhao et al., 2025a), visual grounding (Bai al., 2025b), and video understanding (Zhang et al., 2024).
Despite recent advances, these models still encounter two critical limitations when applied to OVAR tasks: O Existing approaches rely heavily on text-based chain-of-thoughh (CoT) reasoning for visual understanding, which leads to misalignment between temporal visual features with discrete textual representations. As depicted in Fig. 1 (a), such misalignment makes the models prone to cross- modal hallucinations during reasoning. @ Current methods typically struggle to handle semantically similar actions in open-vocabulary scenarios. The unpredictability of actions scenarios prevents models from learning discriminative patterns directly. As shown in Fig. 1 (b), such inability in distinguishing semantically ambiguous actions may lead to erroneous classification.
open-vocabulary
To solve the above-mentioned challenges of similar-action misclassification and cross-modal hallu- cinations, we argue that effective OVAR requires two key capabilities: 1 Enabling Fine-grained action discrimination through contextual sub-motion decomposition, which enables the model to disentangle hierarchical actions into discriminative sub-motion primitives for fine- mne-grained classifi- cation. (e.g., "shoot baskethall": "leg bending" - "torso jumping" - "arm extending" - "hand-ball
2

---

Published as a conference paper at ICLR 2026
releasing"). @ EEhanciing cross-modal interleaving through multimodal CoT reasoning, which guides the model to autonomously invoke domain-speciffe tools for enhanced risual-semante repre- sentation. While recent MLLMs (Zhang et al., 2025a; Zheng et al., 2025) have demonstrated the po- tential of tool-augmentation in open-world video understanding (Qian et al., 2025; Su et al., 2025a), these approaches typically operate on frame-level manipulations (e-g., zooming, cropping) that lack explicit modeling of motion continuity. In contrast, OVAR requires capturing both fine-grained tem- poral dynamics and hierarchical sub-motion dependencies. Consequently, unified framework that harmonizes sub-motion decomposition with tool-mediated perception to ensure both spatial preci- sion and semantic coherence in OVAR is urgently needed.
Therefore, we introduce VIDEO-STAR, a framework that synergistically integrates contextual sub- motion decomposition with tool-augmented reinforcement learning (RL) for open-vocabulary action recognition. Specifically, during the inference process, our method dynamically invokes domain- specific tools like pose estimation, human detection, and online retrieval for cross-modal interleav- ing, thereby effectively resolving visual-semantic ambiguities and reducing hallucination. More- over, to enable category-specific reasoning capacity, we formulate action recognition as a sequential decision-making process at the following stages: 1 decomposing actions into discriminative sub- motion primitives, matching these primitives to several candidate actions, @ scoring candidates based on hierarchical relevance. Furthermore, we propose reward mechanism that jointl optimizes for tool-usage efficiency, sub-motion relevance, and structural coherence in reasoning. This ensures that tools are activated only when they contribute meaningful insights, while sub-motion hierarchies are weighted to prioritize semantically salient components. Our contributions are threefold:
Augmented Agentic RL. VIDEO-STAR introduces novel framework that unifies sub- motion decomposition with tool-augmented reinforcement learning, thereby enabling robust open- vocabulary action recognition through dynamic, visually grounded reasoning.
Multi- Tool Integration. We design a multimodal tool library integrating pose estimation, hu- man detection, and online retrieval, which dynamically augments reasoning with domain-specifer knowledge to resolve cross-modal hallucinations during reasoning.
Emprical Effectiveness. Experimental results on HMDB-5I, UCF-101, SSv2, Kinetics-400, and Kinetics-60k datasets demonstrate that our method achieves state-of-the-a art performance, outper- forming existing methods by significant margins while maintaining computational efficiency.
2 RELATED WORK (EXTENDED VER. IN APPX. B)
Open-Vocabulary Action Recognition. Recent methods for open-vocabulary action recogni- tion (OVAR) primarily leverage CLIP's modal aligmment. Early approaches like Action- CLIP (Wang et al., 2021) and ViFi-CLIP (Rasheed et al., 2023) adapt CLIP via full fine-tuning or joint encoder optimization, but often overfit to static features. Parameter-efffiient strategies, such as Adaptformer (Chen et al., 2022) and ST-Adapter (Pan et al., 2022), use compact modules to dis- till temporal knowledge, while prompt engineering (AIM (Yang al., 2023). VPT (Ju et al., 2022)) biases models toward action semantics. However, these methods struggle with cross-architecture generalization due to limited capacity or rigid designs. Recent works like Open-VCLII (Weng et al., 2023) and FROSTER (Huang et al., 2024b) improve generalization via weight interpolation and residual distillation, yet remain constrained by static assumptions. In contrast, Video-STA in- troduces sub-motion decomposition with aug-enede RL for robust open-vocabulary inference.
Multimodal LLMS Reasoning. Recent advances in large language models (LLMs) have demon- strated that RL-based post-training can significantly enhance reasoning capabilities, as exemplified by OpenAl-ol (Jaech et al., 2024) and DeepSeek-RR (Guo et al., 2025). These paradigms have been extended to multimodal language models (MLLMs) for tasks like mathematical VQA (Peng et al., 2025), image segmentation (Liu et al., 2025a), and video understanding (Feng et al., 2025). However, existing methods struggle with long- cross-modal interaction. To address this, we propose multimodal CoT reasoning via nool-augmented RL, which explicitly reduces hallucination through dynamic tool integration.
hallucination (Chen et al., 2025) and limited
Toul-Augmented Agentic System. Recent advancements in LLMs have shown that external tools can enhance multimodal reasoning. Early works like FAST (Sun et al., 2025) and MVoT (Li et al., 2025a) introduce visual evidence into reasoning, forming multimodal CoT for image tasks. LLa Va- Plus (Liu et al., 2023b) pioneered training strategies for tool use, while VPD (Hu et al., 2024)
3

---

Published as a conference paper at ICLR 2026
CoT Data Construction
Supervised Fine- Tuning
Reinforcement Learning
Evaluation
Supervised Fine-Ture Tuning
Video
Video
Accunaay &
Teal alace
Video
adk geenstee CalT recoening
Eddlaniing Compied
Vision Encoder
Guemte What kid hurar ac Aroven Frow
Cluestion
Answer
Question
Somple
H
Quastion
VOLO 11
Online RAG
Vision-Longuage Model
Sample
0.5
Determing Tool-usage
Pace Hlmmos Distcction
Achio Victo
Explonation
Tools
Defermine
Tool -usage
Semple
Determine Tool-ssage
Sub-Motion Logic Chain
Hori Vilne cutrett and
rectinn hopi- vlkchan Hene
Generating
Secand Roand
meeloo Dacempers Jay body parts motaer
Matcchng
O Mit--Mttinn
Fine- Tuned
Decompedtion
Sencration
Candidate
Motching
Scoring
Data
Wetoliig by Gendikiee
lel Type chatsert
VLM
Teem: (Scred 9790) ./tkuro
Reasoning Answer
Figure 2: Pipeline of Video-STAR. (i) Introduce three-stage sub-motion logic chain to construct tool-augmented reasoning data that decomposes actions into discriminative sub-motions. (ii) Pre- train the MLLMs structured reasoning chains and fine-tune it for domain-speciffe adaptation. (iii) Adopt the GRPO algorithm for reinforcement learning, which optimizes a hicrarchical reward function considering both tool-usage and sub-motion to ensure robust and consistent inference.
leveraged program derivved data to transfer tool skills. Recent methods like TACO (Ma et al., 2024) and Py Vision (Zhao et al., 2025b) expand tool use with RL. However, existing approaches rely on static tool pipelines, limiting their ability to adapt to complex, open-vocabulary actions. Our frame- work addresses this by explicitly modeling the contextual sub-motions and dynamically balancing tool usage efficiency with hierarchical motion relevance through a structured reward mechanism.
METHODOLOGY
Overview. We propose Video-STAR, a unified framework that synergizes contextual sub-motion decomposition with tool-augmented reinforcement learning, illustrated in Fig. 2. Sec 3.1 first details the problem formulation of the inference process and the tool library. See 3.2 details Training Data Construction through multimodal CoT generation, synthesizing video-query pairs with predefined prompts. See 3.3 presents Agentic Supervised Fine-tuning, where the Qwen2.55 VL base model is pretrained on the reasoning data for domain-specine adaptation. Sec 3.4 outlines Agentic Reinforcement Learning. which introduces a hierarchical reward function that balances tool-usage, sub-motion, and structural coherence for inference. The above-mentioned three stages are tightly coupled: the training data provides the foundation for SFT, which in turn initializes the RL policy, ensuring end-to-end alignment between perception, reasoning, and action inference.
3.1 PROBLEM FORMULATION.
In open-vocabulary action recognition (OVAR), the task is to identify action classes not seen during training. The input consists of video V and a query Q, while the output is predicted action A that may belong to unseen categories during training. To achieve this goal. we design the following two-stage framework that dynamically integrates domain domain-specific tools: The first stage selects the
most relevant tool from an augmented toolset 7 = (Tp. Ta, T,, T.). @ The second stage combines tool outputs with raw inputs for final classification. This architecture ensures contextual consistency between visual observations and semantic reasoning by explicitly modeling the interaction between perceptual features and task-specific knowledge. The implementation details are as follows:
Tool Selection. Given video V and query Q, the model analyzes contextual cues to de- termine which tool is most relevant for resolving our task. The toolset contains four components: Tp for pose estimation, Tg for human detection, T2 for action explanation, T, for video description. The first-stage output y ~ To( Q) produces a selected tool call C C T along with intermediate results R. where my represents the policy network of MI.LMs parameterized by weights 0.
Stage 1:
4

---

Published as a conference paper at ICLR 2026
ultralyytcc
Qwen API
YOLO
"Shoot ball: person throwing towards torget, such o bosketholl hoop or scccer godl
Leg Bend for jump, then quickly become straight. Arm: Extend forward, then stroightening outword. Torso: Jump up to release the power. Interaction. Ball relenses from hands."
3. Action Explonation
Defection
video shows o men playing baskettoll on an outdoor court.
This action involves the use of the baskettall and the oct of shooting, which is common basketball- relsted action.
Input Video
Pose
4 Video Description
Estination
Figure 3: Tool Libirary. Given the input video, Video-STAR respectively adopts the YOLO 11 for human detection & pose estimation, and the Qwen API for action explanation & video description.
Result Integration & Prediction The second stage refines action recognition by integrat- ing tool outputs R with original inputs. For the visual tools Tp and Ta, extracted features F are concatenated with raw video frames V to enrich spatial-temporal representations. For the semantic reasoning tool Ta and Tv, the generated explanation E' is appended to the original query Q to provide contextual grounding. The final prediction A is computed as:
Stage 2:
A ~ ma (-1 V + F,Q @ E;T),
(I)
where @ denotes feature concatenation for visual modalities and text appending for reasoning, while To maintains the same policy network of MILMS parameterized by weights 0.
Tool Library As shown in Fig. 3, our toolkit library contains four sub-tools: 2 Human Detection:
adopt the YOLO 11 (Khanam & Hussain, 2024), which employs hybrid spatial attention mech- anism for anchor-free human detection. Pose Estimation: we introduce YOLO 11's 17-keypoint skeletonization capability (e.g.. COCO format) capture sub-pixel joint positions for fine-grained motion analysis. @ Action Explanation: we leverage the Qwen API for online retrieval augmenta- tion (RAG), then redefine actions through transitional phases (e.g.. "stand" - "transit from sit or lay to stand"). O Video Description: we extract temporally salient frames, then adopt the Qwen API for online RAG to disambiguate multi-phase action sequences (e.g., statsstdddwwlk")) via spatial- temporal descriptions. The detailed descriptions are shown in Appendix D.
3.2 TRAINING DATA CONSTRUCTION
Existing MILMS methods typically suffer from two critical limitations when applied to OVAR tasks: O The over-reliance textual priors that neglects domain-specifet visual cues, and @ The inability to distinguish ambiguous actions in open- ppen-vocabulary scenarios. To address these issues, construct novel training dataset by integrating multimodal CoT reasoning with tool-augmented decomposition, thereby ensuring both spatial-
alignment and semantic accuracy.
Data Collection. As shown in the cyan part of Fig. our dataset is constructed via three- stage logic chain. First, we curated an initial pool of approximately 7,000 video-query pairs from the HMDB-51 dataset (Kuehne et al., 201 I). We then leveraged Dwen2.5-VVL-721 (Bai al., 2025a) to synthesize initial of-innoof-though (CoT) reasoning sequences for these pairs. Crucially, to ensure data quality, we introduced a rigorous filtering stage: each generated reasoning chain was scored and reviewed by the closed-source Qwen-VL-Max VL-Max model as an expert validator. Chains containing factual errors or logical inconsistencieste. (e.g., "hallucinated" or incoherent reasoning) were system- atically discarded. This malidation-and-paa sigh-fidelity reasoning chains, specifically curated for supervised fine-tuning (SFT). These chains are designed to dynamically integrate visual features and domain-specifet knowledge, thereby bridg- ing the gap between text-centric reasoning and visually grounded inference.
procedure yielded a final set of approximately 5 5,000
Generating Pipeline. The core innovation lies in our three-stage framework for action recognition:
Sub-motion Decomposition first breaks down the holistic action into discriminative sub-motion primitives by identifying key body-part interactions. For "shoot ball" is decomposed into "body bending + leg extending + feet-ball interaction". This decomposition addresses the inherent ambiguity of holistic perception by isolating perceptually distinct motion patterns.
Candidate Selection then maps each sub-motion to pre-defined action definitions, generating 2-3 semantically relevant candidates. In our example, "body bending" suggests "run". "leg extending"
5

---

Published as a conference paper at ICLR 2026
implies "hurdle", and "feet-ball interaction" aligns with "shoot". This step narrows the search space while preserving semantic diversity, ensuring robust open-vocabulary generalization.
Matching Scoring finalizes recognition by comparing all sub-motions against each candidate's detailed definition. The system evaluates spatial precision and description alignment, ultimately selecting the highest-scoring candidate. In our case, the combination of precise spatial interactions and semantic coherence with the definition "shoot ball" would lead to the correct classification.
Data Assessment To ensure the reliability of the constructed data, introduce a specialized LLM to evaluate its factual accuracy and logical coherence. This model systematically removes entries containing inconsistent reasoning steps or unverified claims, resulting in a refined dataset that com- bines explicit tool usage with logically coherent, evidence-supported reasoning.
3.3 AGENTIC SUPERVISED FINE- TUNING
After constructing the training data, we introduce a cold-start phase that prioritizes supervised fine-tuning (SFT) before applying reinforcement learning (RL). Inspiring from R1-Zero (Guo al., 2025), we initially attempt direct RI. optimization to train our method. However, preliminary exper- iments reveal progressive decline in the frequeney of tool invocation during policy rollouts. This behavior likely arises from a distributional discrepancy in target domain between tool-enhanced visual features and model's pretraining data. Therefore, we opt perform SFT before RL.
Specifically, We formalize each training instance as T IX, I,,, where X' denotes the input modality, represents task instructions, S {si..... $r } captures reasoning steps, and y is the target output. The objective minimizes the negative log-like likelihood of the reasoning process:
SSFT ETND
Σ log lo (s | s<t)
where pa is the model's conditional probability distribution. This formulation enables model to generate reasoning chains that align with both task instructions and ground-truth during pre-training.
3.4 AGENTIC REINFORCEMENT LEARNING
Group Relative Policy Optimization (GRPO). We adopt the GRPO algorithm (Jiang et al., 2025) for policy optimization. GRPO innovatively employs groupwise comparison framework to evalu- ate candidate responses. Specifically, for each query q paired with its ground-trut solution a from dataset D, the algorithm generates set of rollout trajectories {01, 02 ..., og) based the previous policy Tew- The policy ny is then refined through the optimization of this objective function:
(oRPO( -00--
(3)
mán To(oxl0) (o)9)(
To(0:14) (oola()ilg) c, c,1+c) Advs) -DDu(TollErer))
Adv,, clip
D K (TellTTTTrr Tres(Oilg) log Tre (o,19) 1, (e(0))))
(4)
(e(0))
where B is adopted to balance the trade-off between exploration and stability during optimization. Then the advantage estimator A; is calculated using normalized rewards from the trajectory group:
mean (f1,, rr.),,
A std (ri, (r1, (.,...,.
Each trajectory o; receives binary reward r, E {0, 1} through rule-based verification system designed to mitigate reward manipulation risks.
Reward Design. Effective reward functions should balance accuracy, structural coherence, tool-use efficiency, and sub-motion relevance to support the training of agentic systems for OVAR tasks. While traditional reward design primarily prioritizes the correctness of the answer and format,

---

Published as a conference paper at ICLR 2026
extend this by incorporating hierarchical sub-motion relevance and tool-mediated perception to guide structured reasoning over sequential sub-motions.
The total reward integrates four components: accuracy reward Race format reward Rfoeaat- tool- usage reward Ryol and sub-motion reward R The accuracy reward Race evaluates the correctness of the final action classification, while the formatting reward Rivemat penalizes unstructured incom- plete reasoning chains. The tool-usage reward Roool is activated only when correct answers are pro- duced alongside valid tool invocations. The sub-motion reward Rsah employs a hierarchical weight- ing mechanism to prioritize semantically salient sub-motions. Specifically, the model first generates n sub-motions through contextual analysis, ordered by their relevance the action definition. For the k-th ranked candidate, its weight is assigned as: wy - k + 1. When the predicted classifi- cation matches a subset of m sub-motions (ki,.... hm) we define: Rash Ei= Wity/ Σ Wk;,- This formulation ensures higher-pon eeeiorrprririty sub-motions receive exponentially greater weighting, while lower-priority ones contribute proportionally less. Formally, the total reward is defined as:
R(T)= = Rase (T) + Riormat (T) + ()))))) (Rioot + Rauh(T))
where (r)so is the indicator function which assigns a binary value of 1 when (T) > 0 and 0 otherwise. This conditional structure enforces two key principles: perception-aware reasoning requires simultaneous activation of tool usage and sub-motion prioritization only when the final answer is correct, and @ redundant tool invocations or irrelevant sub-motions receive no reward. By coupling tool-mediated perception with hierarchical sub-motion weighting, the framework ensures that agents invoke tools meaningfully while maintaining coherent reasoning chains.
4 EXPERIMENT
Experimental Settings. We adopt two experimental settings: base-to-novel: For each dataset, the label space is partitioned into two disjoint subsets, i.e., the base classes YB and the novel classes VN. where V'8 n VN 0. We train the models on data from V/ and evaluate on testing samples from VB U VNN cross cross-dataser: In this setting, the model is trained on source dataset with the label set as Vss and evaluated on a target dataset with another label set as Vr, where D/ss U UOrl 2 [D's n n-rl--
Dataset and Metries. We evaluate our method on five standard action recognition benchmarks: UCF-101 (Soomro et al., 2012), HMDB-51 (Kuehne et al., 20, Kinetics-400 (K-400) (Carreira & Zisserman, 2017), Kinetics-600 (K-600) (Carreira et al., 2018), 8), and (SSv2). UCF-101 has 13,320 video clips from 101 classes, while HMDB-51 includes 6,849 videos across 51 classes. For large-scale evaluation, we use K-400 and K-600, which contain 400 and 600 classes, respectively. SSv2 contains 174 fine-grained classes. Following prior work (Rasheed al., 2023; Ni et al., 2022: Weng et al., 2023), we report average top-1 accuracy under the two settings.
V2
Implementation Details. We implement experiments on both Owwn2.55VL-33 and Qwen2.5-VL- 7B models. In stage one, we select 5,000 training samples constructed from Sec 3.2 for SFT. These same training samples training was conducted on eight NVIDIA H20 GPUs (90 GB memory each) using a 5k sample dataset with batch size of 8. The process took 20 hours for 600 iterations (I epoch), with 4 rollouts per sample and learning rate of 5e-7. More experimental details are shown in Appendix A.
then reused for subsequent RL optimization during the stage two. The
4.1 MAIN RESULTS
Base-to-novel. In Tab. 1, we compare our Video-STAR with the state-of-the-art (SOTA) methods under the base-to-novel setting. Notably, our model is fine-tuned only on the base set of HMDB-51 and evaluated in a zero-shot manner on K-400, UCF-101, and SSv2. In contrast, all other methods are fine-tuned on the base set of each respective dataset, making our evaluation more challenging- From the results, we highlight four findings: ① Our Video-STAR established a new SOTA, consis- tently outperforming previous methods across all datasets. Specifically, Video-STAR-77 achieves 26.3% in K-400 and 27.0% in HMDB-51 absolute improvement over the previous SOTA. @ Video- STAR demonstrates strong generalization to novel categories. We see the performance gap of Video- STAR between the base and novel sets is nearly closed. Remarkably, on K-400 and UCF-101, the accuracy of novel classes exceeds that of base ones. @ Scaling helps the complex dataset. We ob- serve 10.5 % and 3.2% absolute increase in HM on K-400 and SSv2 datasets when scaling from
7

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison (Topl-Ace (%)) with the CLiP-based methods using ViT-B/16 under the base-to-novel sefting. "HM" denotes the harmonic mean of the accuracy from the base and novel sets. Note that the best and second best performances are highlighted.
K-400
HMDB-51
UCF-10 101
SSv2
Method
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
Basc
Novel
HM
ActionCLIP
61.0
46.2
52.6
69.1
37.3
48.5
90.1
58.1
70.7
13.3
10.1
11.5
X-CLIP
74.1
56.4
64.0
69.4
45.5
55.0
89.9
58.9
71.2
8.5
6.6
7.4
VPT
69.7
37.6
48.8
46.2
16.0
23.8
90.5
40.4
55.8
8.3
5.3
6.4
ST-Adupter
73.6
62.0
67.3
65.3
48.9
55.9
85.5
76.8
80.9
9.3
8.4
8.8
ViFi-CLIP
76,4
61.1
67.9
73.8
53.3
61.9
92.9
67.7
78.3
16.2
12.1
13,9
FROSTER
77.8
64.3
70.4
74.1
58.0
65.1
95.3
80.0
87.0
18.3
12.2
14.6
Open-MeDe
77.2
63.8
69.9
73.6
56.4
63.9
94.9
78.5
85.9
17.1
12.3
14.3
VTD-CLIP
78.5
63.5
70.1
78.4
63.5
70.0
95.5
73.7
83.2
17.8
13.9
15.4
AP-CLIP
77.2
64.1
70.0
74.6
55.9
63.9
94.8
77.0
84.8
16.3
12.9
14.4
Owen2.5-V1.-33
48.3
40.5
44.1
54.0
40.8
46.5
71.4
62.4
66.6
3.5
3.2
3.3
Qwen2.5-V VL
87.8
84.8
86.3
41.7
50.3
45.6
85.0
82.4
83.7
14.0
9.9
11.6
Adeo-STAAR-3B
86.0
86.4
86.2
92.1
91.7
91.9
13.5
11.3
12.3
Vidco-STAR-7B
96.3
97.2
96.7
92.3
91.9
92.1
99.6
99.7
19.2
13.0
15.5
Table 2: Performance comparison (Topl-Acc (%)) with the CLIP-based methods under the cross- dataset setting. "Full" denotes the full validation set, while "Split" denotes evaluating across three validation splits, Note that the best and second best performances highlighted.
UCF-101
HMDB-51
K-600
Method
Full f
Split f
Full f
Split
Split f
ActionCIIP (Wang al., 2021)
77.4
77.5+-0.8
48.0
62.5+ 4..
X-CLIP (Ni et al., 2022)
72.0.12.3
44.61.5.2
65.2.10.4
ST-Adapter (Pan al., 2022)
77.9
77.6++0.7
50.3
51.1+0.6
60.2+1.8
Vita-CLIP (Wasim et al., 2023)
75.0100.6
48.610.6
67.410.5
ViFi-CIIP (Rasheed et al., 2023)
76.8+0077
$1.3++0.6
71.2+
Open-VCLIP (Weng ct al., 2023)
83.5
83.4+1.2
53.2
53.9+1.2
73.0=0088
FROSTER (Huang et al., 2024b)
85.0
84.8.11.1
$4.5
54.811.3
74.8.10.9
Open-MeDe (Yu al., 2025a)
83.7+1.3
54.6+1.1
73.7+0.9
STDD (Yu et al., 2025b)
85.211.2
55.940.2
75.1=0.7
Qwen2.5-VL-33
58.3
58.1.10.2
39.2
42.110.3
37.711.8
Qwen2.5- VL-7BB
77.5
77.2110.6
50.6
53.110.1
68.0.1..5
96.8
96.7
83.5
86.2.10.2
90.5
Vidco-STAR-7B
99.4
99.4++0.2
90.1
92.5+0.1
98.2+00.1
3B 7B. However, on HMDB-51 and UCF-101, performance is already near-perfect. 1 Generic MLLMS are insufficient. Although Qwen2.5-VI--7B is competitive on K-400 (HM 86.3), col- lapses on HMDB-51 (HM 45.6) and SSv2 (HM 11.6), whereas Video-STAR maintains uniformly high performance, suggesting that our specific alignment is crucial for robustness across datasets.
Cross-dataset. In Tab. 2, compare our method with the previous methods under the cross- dataset setting. We see two key observations. ! Video-STAR exhibits remarkable stability, achiev- ing comparable results on the full validation sets (UCF-101* HMDB-51 and the splits (UCF-10) HMDB-51). This indicates that our model achieves robustness through tool usage and sub-motion decomposition. Our smaller 3B model outperforms previous wate-of-the-an methods, as well as the larger Qwen2.5-VL-788 This further demonstrates the effectiveness of our Video-STAR, More- over, Fig. presents a comparative case study. More case studies are provided in Appendix A.3.
4.2 ABLATION STUDIES
Training Stages. In rows (a)-(d) of Tab. 3, we compare zero-shot, full model, SFT-only, and RL- only training paradigms, respectively. The zero-shot baseline in (a) yields the lowest results among all methods. SFT-only training in row (c) significantly improves accuracy, outperforming RL-only training in row (d). We attribute the gap to the difficulty of RL. in directly navigating
8

---

Published as a conference paper at ICLR 2026
What kind of action shown in the video?
Question:
Given shoot__boll, Punch, Throw, Kick...
ethinloReasoning </thinlo censeer>Action//nsserr
Tool Augmentation
Qwen2. ---n--
[
Observing video, the two individuals stonding living room. They appear be engaged conversation, as indicated by their body language and facial expressions. The woman is smillng and looking of the mon, while the man is also looking ot her. This Suggests that they having friendly casual interaction.</hink <answer> Smile </onswer>
Sub Motion Decomposition
Video-STAR-33-
<thinkx[1] Observed bedy parts and movement characteristics:
Torso: The torso rotating around fixed point. The person facing one direction and turs face opposite direction.
Head: The head follows the rotation of the torso, turning in sync with the body't 's movement.
relatively stathonary. with significant movement. Legs: Stay ground, providing stable base. for rotation [2] Matching candidate actions: Turn, Walk, Run.
Arns: The
[3] Pattern comparison for each condidate:
Turn: [Scone: 9/10] The torso head are rotating around fixed point, which the defining characteristic of The
and legs are in stable position, supporting the rotation.
Walk: [Score: 2/10] The movement is not forword and backward, and the body is not rotating around a fixed point.
forward and backword,, and body rotating around fixed point. The legs contact with the ground, but the movement is not consistent with rurning.</hirie <arswer> Turn </onswer>
[Score: 3/10]- The movement
Figure 4: Case Study between Qwen2.5-VL-3B and our Video-STAR-3B. Qwen2.5-VL-3B mis- classifies action "turn" as "smile", while our Video-STAR-3B accurately identifies the correct action.
Table 3: Ablation of key components in Vidco-STAR under cross-dataset setting. "Split" denotes evaluating across three validation splits. The best and second best performances are highlighted.
UCF-101
HMDB-51
K-600
UCF-101
HMDB-51
K-600
Method
RL
Splitt
Splitt
Splitt
Splitt
Splitt
Splitt
Qwen2.5-VI-33
58.1+0.2
42.1+t0.3
37.7+
(8) w/. Base
93.8=0.4
81.7=002
84.7+112
Video-STAR-3B
96.7-1033
86.2-10.2
90.5100.7
(h) w/o. Tol
95.1
83.2-0.2
86.7110.9
(c) w/o. RL
76.8-+0.88
63.5++0.2
61.3++2.0
(i) w/o. Sub
95.7-10.3
84.1-:0.2
87.9:+0.88
(d) w/o. SFT
71.4=0.9
57.6+0.2
54.8:+2.1
(j) Num
96.4+2022
85.0=0.2
89.3++0.3
(e) w/o. TOL
87.1+005
74.9+0.3
78.4+11.5
(k) Num = 4
96.7=0.3
86.2+0.2
90.5+:0.7
(f) wlo. SUB
88.5110.5
76.840.33
81,21113
(1) Num
96.910.3
86.9-0.2
91.011099
the high-dimensional reasoning space without prior supervised guidance. These results support our two-stage strategy, where SFT provides a structured reasoning foundation before RL optimization.
Tool-Usage & Sub-Motion. In rows (e)-(f of Tab. 3, we show the effect of removing tool-usage and sub-motion decomposition from our full framework. Removing tool usage in row (e) causes notable performance degradation with at least 9.1%. This confirms the critical role of external tools in resolving cross-modal ambiguities and reducing hallucination. Similarly, removing sub- motion decomposition in row (f) reduces accuracy 8.2%. 9.4%, 9,3% in UCF-101, HMDB-51 51, and K-600, respectively. This indicates the importance of breaking complex actions into discriminative primitives and enabling category-specino scores, demonstrating that tool- augmenedd reasoning and sub-motion hierarchy are complementary and jointly essential for robust open- vocabulary action recognition.
reasoning. The full model (row (b) achieves the highest
Reinforcement Learning. In rows (g)-(i) of Tab. 3, we analyze the effect of different reward config- urations in RL training. Removing both tool-usage and sub-motion rewards (g) produces the lowest performance, showing that accuracy and format rewards alone are insufficient to guide reasoning. Including sub-motion reward in row (h) improves results at least 1.3%, illustrating that hierarchical motion decomposition provides strong semantic cues even without tool-mediated perception. Con- versely, including tool-usage reward (i) also yields better performance than (g) with least 1.9%, confirming that tool integration helps resolve visual-semantic ambiguities. The full model in row (b), which utilizes the complete reward function, achieves the highest accuracy. This demonstrates that our reward design is crucial for effective RL optimization in framework.
Number of Generation. In rows (j)-(1) of Tab. 3, we analyze the impact of different numbers of generations during the RL training (GRPO). Increasing the number of candidate responses from 2 to 6 consistently improves performance across all datasets. Specifically, when generating six responses per input Num 6, the model achieves the highest performance with average accuracy scores of

---

Published as a conference paper at ICLR 2026
96.9%, 86,9% and 91.0% UCF-101, HMDB-51, and K-600, respectively, outperforming the configurations with Num = and Num = 4. However, the marginal gains diminish beyond Num = 6, indicating trade-off between computational cost and performance improvement. Therefore, opt for Num = to balance accuracy and efficiency. More ablation study is shown in Appendix A.2.
Table 4: Comparison between our agentic system and static pipeline that adopts all tools.
Method
UCF Acct
Infer Timel
Tool Time)
Total Time.
All Tools (Static)
97.2% 96.7%
1,86s 1.75s
2.24s
4.10s
Video-STAR 3B
1.43s
3.18s
Agentic System vs. Static Pipeline. To validate the effectiveness of our dynamic tool-selection policy, compare our agentic system against static pipeline that invariably invokes all tools. As shown in Table 4, our agentic model achieves an excellent trade-off between performance and efficiency. For marginal 0.5% drop accuracy on UCF-101, our approach reduces total inference time by approximately 22% (from 4. 10s to 3s1 This result demonstrates that the learned policy is adept at pruning redundant tool calls, securing substantial efficiency gains while maintaining top-tier accuracy, thus confirming the superiority of an agentic design over rigid, static pipeline.
Table 5: Computational overhead analysis. Video-STAR achieves a massive accuracy boost with a moderate increase in latency and GFLOPs, highlighting favorable performance-cos. trade-off.
Method
Memory
Latency
UCF Acct
GFLOPs.
BSZ
Qwen-2.5V1-3B
60.03 GB
0.79s Os
58.1%
18,296
50
Video-STAR-3B
60.89 GB
1.75s 1.43s
96.7%
25,773
50
Computational Overhead. We analyzed the computational cost of Video-STAR compared to its base model, Qwen-2.5VL-3B. As reported in Table 5, the majority of the added latency comes from tool invocation (1.43s), which is spent executing external models and APIs. Despite this overhead, our framework boosts accuracy on UCF-101 from mere 58. 1% to an exceptional 96.7%. This massive +38.6% absolute improvement for a moderate increase in computational cost highlights a highly favorable trade-off and underscores the value of our tool- -uggeneddd reasoning approach.
Table 6: Ablation on tool selection, demonstrating the framework's modularity. Performance re- mains high when swapping core tools. proving the agentic logic the key contribution.
Tool Stack (3B)
UCF Acct
Latency (per sample)
YOLO Qwen (Ours)
96.9%
18s
YOLO - OpenPose
96.1%
4.61s
Qwen + Gemini-1.5
97.4%
3.47s
Robustness to Tool Selection. To demonstrate the framework's 's modularity and robustness, we con- ducted experiments by swapping our default tools with alternatives. As detailed in Table 6, replacing YOLO 11 with OpenPose for pose estimation or substituting the Qwen API with Gemini- 1.5--i for semantic reasoning results in consistently high performance. Although latency varies with the tool choice (c.g., OpenPose is slower, Gemini- 1.5imi--1.5-Pra 1.5-Pro is more powerful but more expensive), the ac- curacy remains robust. This proves that primary innovation lies in the underlying agentic logic, which can flexibly orchestrate a variety of tools, rather than on a single, specific tool stack.
CONCLUSION
In this work, we propose Video-STAR, novel framework that integrates contextual sub-motion decomposition with tool-augmented reinforcement learning. By decomposing complex actions into discriminative sub-motions and leveraging domain-specific tools for spatial-temporal alignment, our approach bridges the gap between text-centric reasoning and visually grounded inference. The hier- archical reward mechanism ensures efficient tool usage while prioritizing semantically salient sub- motion patterns, enabling robust performance in various scenarios. Extensive experiments demon- strate our state-of-the-ar performance, outperforming existing methods by significant margins. Fu- ture work will explore cross-modal generalization to broader video understanding applications.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
Our work centers on advancing open-vocabulary action recognition through the Video-STAR frame- work, which leverages publicly available video datasets (e.g., HMDB-51, UCF-101, Kineties) and domain-specific tools (e.g., YOLO for pose estimation, Qwen API for semantic reasoning). All training and evaluation data were sourced from existing benchmarks with established ethical guide- lines, ensuring compliance with data usage policies and minimizing risks of privacy violations. The framework processes visual and textual inputs without extracting retaining personally identifiable information (PII), focusing solely on abstract action patterns and contextual sub-motions to avoid unintended individual tracking. We emphasize the responsible deployment of our model, advocating for its use in applications that prioritize human well-being, such as assistive technologies sports analytics, while explicitly discouraging misuse in surveillance or discriminatory practices. To pro- mote transparency, commit to open-sourcing training data, enabling independent verification and fostering community-drin driven ethical oversight. Our experiments adhered to rigorous academic standards, and remain dedicated addressing potential biases in tool integration or sub-motion decomposition to ensure equitable performance across diverse populations and scenarios.
code, detailed methodologies, and anonymized
REPRODUCIBILITY STATEMENT
To ensure the full reproducibility of our findings, provide comprehensive implementation details in the paper and its appendices. The construction of the training data, including the three-stage sub- motion logic chain and multimodal chain-of-thought (CoT) generation process, is detailed in See 3.2 and Appendix E. The architecture of the Video-STAR framework, encompassing contextual sub- motion decomposition, tool selection mechanisms, and the hierarchical reward function, is described in Sec 3.3 and Sec 3.4. Implementation specifics for the GRPO algorithm, including reward function components (accuracy, format, tool-usage, and sub-motion rewards) and training hyperparameters (e.g.. learning rate, batch size, and generation settings), The tool library, including YOLO 11 for human detection and pose estimation, as well as the Qwen API for action explanation and video description, is fully documented in Appendix D. In line with our commitment to open science, the source code, pre-trained models, and detailed documentation will be made publicly available.
presented in See 3.4 and Appendix A.
11

---

Published as a conference paper at ICLR 2026
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang. Jun Tang. Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025a. URL https:tt////mm /aaxiviiii../////aa/aaaaa/aaatv. org/ abs/2502 13923.
Sule Bai, Mingxing Li, Yong Liu, Jing Tang, Haoji Zhang, Lei Sun, Xiangxiang Chu, and Yansong Tang. Univg-rl: Reasoning guided universal visual grounding with reinforcement learning. arXiv preprint arXiv:2505. 14231, 2025b.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, 2017.
Joao Carreira, Eric Noland, Andras Banki-Horvati Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. Arxiv e-prints, 2018.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang. Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. In NeurIPS, 2022.
Yukang Chen, Wei Huang, Baifeng Shi, Qinghao Hu, Hanrong Ye, Ligeng Zhu, Zhijian Liu, Pavlo Molchanov, Jan Kautz, Xiaojuan Qi, et al. Scaling rl to long videos. arXiv preprint arXiv:2507.07966, 2025.
Junhao Cheng, Yuying Ge, Teng Wang, Yixiao Ge, Jing Liao, and Ying Shan. Video-holmes: Can miim think like holmes for complex video reasoning? arXiv preprint arXiv:2505.21374. 2025.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, 2019.
Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng. Junfei Wu, Xiaoying Zhang, Benyou Wang. and Xiangyu Yue. Video-rl: Reinforeing video reasoning in milms, arXiv preprint arXiv:2503.2
2025.
Yuying Ge, Yixiao Ge, Chen Li, Teng Wang, Junfu Pu, Yizhuo Li, Lu Qiu, Jin Ma, Lisheng Duan, Xinyu Zuo, et al. Arc-hungyuan-videe-7bb Structured video comprehension of real-world shorts. arXiv preprint 1XXi::2007.2099, 2025.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In ICCV, 2017.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, and al. Deepseek-rl: Incentivizing reasoning capability in llm via reinforcement learning, 2025. URL https:///// / /aax/avvvvvvoo.o org. aabs////////// 2501. 12948.
Dongliang He, Zhichao Zhou, Chuang Gan, Fu Li, Xiao Liu, Yandong Li, Limin Wang, and Shilei Wen. Stnet: Local and global spatial-temporal modeling for action recognition. In Proceedings of the AAAI conference artificial intelligence, volume 33, 2019.
Yushi Hu, Otilia Stretcu, Chun-Ti Ta Lu, Krishnamurthy Viswanathan, Kenji Hata, Enming Luo, Ran- jay Krishna, and Ariel Fuxman. Visual program distillation: Distilling tools and programmatic reasoning into lasion-l models. In Proceedings of the IEEF/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 9590 1590-9601, 2024.
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao. Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.067. 066449 2025.
Xiaohu Huang, Hao Zhou, Kun Yao, and Kai Han. FROSTER: frozen CLIP is A strong teacher for open-vocabulary action recognition. In ICLR, 2024a.
12

---

Published as a conference paper at ICLR 2026
Xiaohu Huang, Hao Zhou, Kun Yao, and Kai Han. Froster: Frozen clip is strong teacher for open- vecn-oocyyllay action recognition. In International Conference on Learning Representations, 2024b.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky. Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Careey, et al. Openai ol system card. arXiv preprint arXiv:2412. /6720, 2024.
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference machine learning, pp. 4904 4916. PMLR, 2021.
Bo Jiang, Shaoyu Chen, Qian Zhang, Wenyu Liu, and Xinggang Wang. Alphadrive: Unleashing the power of vlms in autonomous driving via reinforcement learning and reasoning, 2025, URL https: /aarivv//aavvva//a org/ atcp/aas//0/ /2003. 07608.
Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual-language models for efficient video understanding. In ECCV, 2022.
Rahima Khanam and Muhammad Hussain. Yolov 11: An overview of the key architectural enhance- ments, 2024. URL httppss//axxvv..regeeos http://aaab221111722222
Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: large video database for human motion recognition. In ICCV, 2011.
Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vulić, and Furu Wei. Imagine while reasoning space: Multimodal ciffiiaatiiooo-ttuulllee arXiv preprint arXiv:2501.07542, 2025a.
Hongyu Li, Songhao Han, Yue Liao, Junfeng Luo, Jialin Gao, Shuicheng Yan, and Si Liu. Re- inforcement learning tuning for videollms: Reward design and data efficiency. arXiv preprint arXiv:2506.01 000006 2025b.
Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng. Yinan He, Yali Wang, Yu Qiao, Yi Wang, and Limin Wang. Videochat-rl: Enhaneing spatio-temporal perception via reinforce- ment fine-tuning. arXiv preprint arXiv:2504.0695: 2025c.
Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In ICCV, 2019.
Ruyang Liu, Jingjia Huang, Ge Li, Jiashi Feng, Xinglong Wu, and Thomas H Li. Revisiting temporal modeling for clip-based image-to-video knowledge transferring. In CVPR, 2023a.
Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang. Hang Su, Jun Zhu, Lei Zhang, Jianfeng Gao, and Chunyuan Li. Llava-plus: Learning to use tools for creating multimodal agents, 2023b. URL htttthtss///aaatee http://awwww...or/a/////aaaa orggabs//3111/1/oo/ 05437.
Yuqi Liu, Bohao Peng, Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, and Jiaya Jia. Seg- zero: Reasoning-chain guided segmentation via cognitive reinforcement. arXiv preprint arXiv:2503.06520, 2025a.
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforeement finetuniig, arXiv preprint arXiv:2503.0123 01785 2025b.
Zixian Ma, Jianguo Zhang, Zhiwei Liu, Jieyu Zhang, Juntao Tan, Manli Shu, Juan Carlos Niebles, Shelby Heinecke, Huan Wang. Caiming Xiong, et al. Taco: Learning multi-modal action models with synthetic chains-of of sf--thought-and and-tiio... arXiv preprint arXiv:2412.05479, 2024.
Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng. Jianlong Fu, Shiming Xiang, and Haibin Ling. Expanding ianguage-image pretrained models for general video recog- nition. In ECCV, 2022.
13

---

Published as a conference paper at ICLR 2026
OpenAl, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, and etc. Florencia Leoni Aleman. Gpt-4 technical report, 2024. URL 2303.08774
wwwpwwwwwww....o.oooooooooooooot abs/
Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li. St-adapter: Parameter-efficient image-to-video transfer learning. In NeurIPS, 2022.
Jinyoung Park, Jeehye Na, Jinyoung Kim, and Hyunwoo Kim. Deepvideo-rl: Video reinforcement fine-tuning via difficulty-wware regressive grpo. arXiv preprint 0734:22506.0744, 2025.
Jungin Park, Jiyoung Lee, and Kwanghoon Sohn. Dual-path adaptation from image to video trans- formers. In CVPR, 2023.
Yingzhe Peng, Gongrui Zhang, Miaosen Zhang. Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-rl: Empowering 3b Imms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint 25x3v:2503. 250.00336,,, 2025.
Ji Qi, Ming Ding, Weihan Wang, Yushi Bai, Qingsong Lv, Wenyi Hong, Bin Xu, Lei Hou, Juanzi Li, Yuxiao Dong, and Jie Tang. Cogcom: Train large vision-language models diving into details through chain of manipulations, 2024. URL https: / /awxivvvvvv////////////a/aa/iaia org/ abs/ 22//2/42222 04236.
Kangan Qian, Sicong Jiang, Yang Zhong, Ziang Luo, Zilin Huang, Tianze Zhu, Kun Jiang, Meng- meng Yang. Zheng Fu, Jinyu Miao, Yining Shi, He Zhe Lim, Li Liu, Tianbao Zhou, Huang Yu, Yifei Hu, Guang Li, Guang Chen, Hao Ye, Lijun Sun, and Diange Yang. Agentthink: A unified framework for tool-augmented chain-of-thought reasoning in vision-language models for autonomous driving, 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- PMLR, 2021.
Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shah- baz Khan. Fine-tuned clip models are efficient video learners. In CVPR, 2023.
Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hong- sheng Li. Visual cot: Unleashing chain-of-thouge thouhtt reasoning in multi-modal language models, 2024.
Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang. Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangja Zhao, Qianqian Zhang, et al. Vlm-rl: A stable and generalizable rl-style large vision-language model. arXiv preprint arXiv:2504.0766
2025.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. Arxiv e-prints, 2012.
Alex Su, Haozhe Wang, Weiming Ren, Fangzhen Lin, and Wenhu Chen. Pixel reasoner: In- centivizing pixel-space reasoning with curiosity-driven reinforcement learning. arXiv preprint arXiv:2505. 15966,9666666 2025a.
Zhaochen Su, Linjie Li, Mingyang Song, Yunzhuo Hao, Zhengyuan Yang, Jun Zhang, Guanjie Chen, Jiawei Gu, Juntao Li, Xiaoye Qu, et al. Openthinkimg: Learning to think with images via visual tool reinforcement learning. arXiv preprint arXiv:2505.08617, 2025b.
Guangyan Sun, Mingyu Jin, Zhenting Wang, Cheng-Long Wang, Siqi Ma, Qifan Wang, Tong Geng, Ying Nian Wu, Yongfeng Zhang, and Dongfang Liu. Visual agents as fast and slow thinkers. In The Thirteenth International Conference on Learning Representations, 2025.
Jiacong Wang. Zijian Kang, Haochen Wang, Haiyong Jiang, Jiawen Li, Bohong Wu, Ya Wang, Jiao Ran, Xiao Liang Chao Feng, et al. Vgr: Visual grounded reasoning. arXiv preprint arXiv:2506. / 1991, 2025a.
Mengmeng Wang. Jiazheng Xing, and Yong Liu. Actionclip: A new paradigm for video action recognition. Arxiv e-prints, 2021.
14

---

Published as a conference paper at ICLR 2026
Qi Wang, Yanrui Yu, Ye Yuan, Rui Mao, and Tianfei Zhou. Videorft: Incentivizing video reasoning capability in milms via reinforced tine-tuning. arXiv preprint arXiv:2505.12434 2025b.
Ye Wang, Qianglong Chen, Zejun Li, Siyuan Wang, Shijie Guo, Zhirui Zhang, and Zhongyu Wei. Simple o3: Towards interleaved vision-language reasoning. arXiv preprint arXiv:2508. 12109, 2025c.
Ye Wang, Ziheng Wang, Boshen Xu, Yang Du, Kejun Lin, Zihan Xiao, Zihao Yue, Jianzhong Ju, Liang Zhang, Dingyi Yang, al. Time-rl: Post-training large vision language model for temporal video grounding. arXiv preprint arXiv:2503. 13377, 2025d.
Syed Talal Wasim, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. Vita-clip: Video and text adaptive clip via multimodal prompting. In CVPR, 2023.
Zejia Weng, Xitong Yang, Ang Li, Zuxuan Wu, and Yu-Gang Jiang. Open-velip: Transforming clip to an open-vocabulary video model via interpolated weight optimization. In ICML, 2023.
Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li. Aim: Adapting image models for efficient video action recognition. Arxiv e-prints, 2023.
Guangle Yao, Tao Lei, and Jiandan Zhong. A review of convolutional neeerrl-network-aae action recognition. Pattern Recognition Letters, 118, 2019.
Yating Yu, Congqi Cao, Yifan Zhang, and Yan-Ning Zhang. Learning to generalize without bias for open-vocabulary action recognition. CoRR, abs/2502.20158, 2025a.
Yating Yu, Congqi Cao, Yueran Zhang, Qinyi Lx, Lingtong Min, and Yanning Zhang. Building multi-modal spatiotemporal expert for zero-shot action recognition with clip. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 9689-9697, 2025b.
Haoji Zhang, Xin Gu, Jiawen Li, Chixiang Ma. Sule Bai, Chubin Zhang, Bowen Zhang, Zhichao Zhou, Dongliang He, and Yansong Tang. Thinking with videos: Multimodal tool-augmented reinforcement learning for long video reasoning, 2025a. URL https: httttttt/////eette //axivv.//////////////e/er org/ /bs/ 2508. 04416.
Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu. Feng Li, Kun Wang, Qingsong Wen, Zhang Zhang, et al. Mme-realworld: Could your multimodal IIm challenge high-resolution real-world scenarios that are difficult for humans? arXiv preprint arXiv:2408. 13257, 2024.
Yi-Fan Zhang, Xingyu Lu, Shukang Yin, Chaoyou Fu, Wei Chen, Xiao Hu, Bin Wen, Kaiyu Jiang. Changyi Liu, Tianke Zhang, et al. Thyme: Think beyond images. arXiv preprint a50iv:2508. /630, 2025b.
Fei Zhao, Chengeui Zhang, Runlin Zhang, Tianyang Wang, and Li Mitigating image captioning ballucinations in vision-language models, 2025a. URL mattps://arae www.:iiiii.ooooooooooooooooooeee abs/ 2505. 03420.
Shitian Zhao, Haoquan Zhang, Shaoheng Lin, Ming Li, Qilong Wu, Kaipeng Zhang, and Chen Wei. Pyvision: Agentic vision with dynamic tooling. arXiv preprint arXiv:2507.07998, 2025b.
Ziwei Zheng, Michael Yang, Jack Hong, Chenxiao Zhao, Guohai Xu. Le Yang, Chao Shen, and Xing Yu. Deepeyes: Incentivizing" thinking with images" via reinforcement learning. arXiv preprint arXiv:2505.1432 2025.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
In this appendix, we provide more experimental details, related work, tool library, and discussions for comprehensive evaluation and understanding of our method. Detailed contents are follows:
A Experiment
16
A.1 More Experimental Details
16
A.2 More Ablation Studies
17
A.3 More Case Studies
18
Related Work
20
B.1 Open-Vocabulary Action Recognition.
20
B.2 Multimodal LLMs Reasoning....
20
B.3 Tool-Augmented Agentic System...
20
C
Discussion
21
Tool Library Details
21
D.1 Human detection Tools
21
D.2 Pose estimation Tools
D.3 Online Retrieval-Augmmenation Tools
22
E Prompts
22
F LLM clarification
24
A EXPERIMENT
A. MORE EXPERIMENTAL DETAILS
Datasets. Our experiments are conducted on comprehensive set of benchmark datasets designed for action recognition tasks, including UCF-101 (Soomro et al., 2012), HMDB-51 (Kuehne et al., 2011), Kinetics-400 (K-400) (Carreira & Zisserman, 2017), Kinetics-600 (K-600) (Carreira al., V2 (SSv2) (Goyal et al., 2017). The Kinetics series repre- sents large-scale datasets with 400 and 600 distinct action classes in K-400 and K-600, respectively, where the latter expands upon the former by incorporating egories. Specifically, K-400 contains 240k training samples and 20k validation samples, while K- 600 provides 410k training samples and 29k validation samples, all sourced from You'Tube clips depicting human actions. Transitioning to smaller but widely adopted benchmarks, UCF-101 com- prises 13.320 video samples distributed across 101 action classes, with three official train/test splits that allocate 9,537 training and 3,783 testing samples. This dataset captures diverse daily activi- ties through short-duration videos, serving standard evaluation protocol in action recognition research. HMDB-51 follows a similar structure with 6,849 videos spanning 51 action categories, maintaining over 101 samples per class and three canonical splits. Notably, its content integrates clips from cinematic productions, public archives, and online platforms, offering rich contextual variations. For fine-grained interaction analysis, we evaluate on SSv2, which focuses on object- manipulation actions involving 174 semantically distinct categories. This dataset features 168,913 training samples and 24,777 testing samples, emphasizing temporally complex interactions between humans and objects that require precise spatiotemporal modeling.
2018), and Somnething-t-SSometine
additional 220 non-overlapping cat-
Evaluation metrics. Building upon prior work vision-language pre- pre-training (Huang et al., 2024a; Yu et al., 2025a; Weng al., 2023), adopt two complementary evaluation frameworks to bench- mark performance. For the base-to-novel setting, other methods train models exclusively on high- frequency action categories while testing their generalization low-frequency counterparts. Dif-
16

---

Published as a conference paper at ICLR 2026
Table 7: More ablations of Video-STAR under cross-dataset setting. "Split" denote evaluating
across three validation splits. The best and second best performances are highlighted.
UCF-101
HMDB-51
K-600
UCF-101
HMDB-51
K-600
Method
Tool Library
Splict
Splitt
Splitt
Splitt
Splitt
Splict
(a) Qwen2.5-VL-30
58.1+0.2
42.140.33
37.7=1.8
(g) w/o. Human
95.310.3
84,320.2
88.3
(b) STAR-33
96.7110.3
86.2:1022
9..5
(h) Pose
92.8.10.4
81.2:0.2
83.7
(c) SPT: w/o. Val
9.3810.3
83.2.10.2
86.6.0.7
(i) w/o, Action
93.6.10.3
82.310.2
85,5
(d) RI: w/. Some
94.2++0.3
82,4+003
85.7+1.0
(i) w/o. Video
94.7++03
83.7+0.2
87.4+0.7
ferently, our method is fine-tuned only on the high-frequency action categories of HMDB-51 and evaluated in zero-shot manner on K-400, UCF-101, and SSv2, making our evaluation more chal- lenging. For this configuration, we apply the methodology across four benchmark datasets, including UCF-101, 11, K-400, and SSv2, systematically partitioning each into base and novel class subsets. Final results represent averages across these configurations, computed using the standard validation splits: for HMDB-51 and UCF-101, only the first of three available splits is used for novel class evaluation, while K-400 and SSv2 leverage their single validation split full.
For the cross-dateset setting, we establish cross-domain evaluation protocols by strategically pair- ing training and testing datasets. Specifically, models trained on HMDB-51 (Kuchne et al., 20 are evaluated on UCF-101 (Soomro et al., 2012) and K-600 (Carreira et al., 2018), while models trained on UCF-101 are tested on HMDB-51. This bidirectional transfer design enables rigorous analysis of domain adaptation capabilities. Performance characterization for HMDB-51 and UCF- 101 incorporates their native three-validttion-spii structure, reporting both mean top-1 accuracy and standard deviation across splits. For the K-600 evaluation, we focus on 220 exclusive categories absent in Kinetics-400, adopting the standardized three-split protocol from prior work (Ni et al., 2022: Rasheed al., 2023). Each split contains 160 novel categories. with final results representing their average performance. This multi-faceted evaluation strategy provides comprehensive insights into both within-datase generalization and cross crossed transfer capabilities.
More Hyperparameter Configurations. The proposed method was implemented on system equipped with an Intel(R) Xeon(R) Platinum 8480+ CPU and eight NVIDIA H20 GPUs (90 GB memory each). For training, the reinforcement learning (RL) pipeline utilized the TRL framework, with supervised fine-tuning (SFT) and RL stages taking hour and 20 hours, respectively.
A.2 MORE ABLATION STUDIES
Data Assessment. In row (c) of Tab. 7, we remove the data assessment stage from our SFT training. Accuracy drops 1.9 % 3.0%, and 3.9% from the full model in (b) on UCF-101, HMDB-51, and K- 600, respectively. This indicates that filtering out inconsistent reasoning is important for maintaining data reliability and supporting tool-augmented logical coherence. Without data assessment, errors and factual inconsistencies in training data degrade cross-dataset generalization.
Weighting Wik in Sub-Motion Reward R Row (d) evaluates the effect of assigning equal weights to all sub-motions. Compared to the full model in (b), accuracy decreases 2.5%, 3,8%, and 4.8% in (d), showing that sub-motion-specific weighting helps prioritize semantically discrim- inative primitives. This confirms that our strategy of assigning dynamic importance to different mub-motions is crucial for fine-graine action reasoning.
Tool Library. In rows (g)-(j), we analyze the contribution of each tool in our framework. Remov- ing the human detection tool (g) reduces accuracy 1.4%. 1.9%, and 2.2% illustrating its role in isolating action-relevant regions and avoiding visual-semantic confusion. Without pose estimation (h), performance declines 3.9%, 5.0%, and 6.8% confirming that structured kinematic represen- tation is essential for capturing subtle motion cues. Excluding the action-retrieval tool (i) yields a decrease 3.1 %, 3.9% and 5.0%, showing that high-level category-specific definitions help bridge label-motion gaps. Removing the fideo-frame description tool (f) lowers accuracy 2.0%. 2.5 2.5% and 3.1 %, indicating that detailed temporal-segment descriptions contribute to disambiguating multi- phase actions. The full model (b), which integrates the entire tool library, achieves the highest scores, proving that these tools are complementary and essential for robust performance.
17

---

Published as a conference paper at ICLR 2026
Table 8: Ablation on the teacher model for Chain-of-Thought (CoT) data generation.
CoT Generation Model
UCF Acct
Delta
Qwen2.5-VL 1ive2..5-L-722B (Ours)
96.7%
Intern VL-2.-L--2
97.1%
+0.4%
Gemini-1.5
97.5%
+0.8%
Ablation on CoT Data Generation. confirm that our framework's effectiveness not limited to specific model family result of intra-family knowledge distillation, experimented with gen- erating the CoT data using different powerful MLLMs. As shown in Table 8, when using InternVL. 2.5-78B Demini-1.5-PPp
the "teacher" model to generate training data, the performance of Video-STAR further improves to 97.1% % and 97.5%, respectively. This not only demonstrates the ro- bustness and generalizability of our approach but also highlights that our framework's performance can scale with the reasoning capability of the teacher model used for data generation.
Table 9: Comparison with state-of-the-art (SOTA) frontier models.
Method
K-400
HMDB-51
UCF-1011
Qwen2.5-Vi 5---BB
86.3%
45.6%
83.7%
Qwen3-VI--BB
89.5%
68.5%
87.1%
Gemini- 1.5--r.
92.8%
73.6%
93.3%
Video-STAR-7B
96.7%
92.1%
99.7%
Comparison with Frontier SOTA Models. To position work against the latest advancements, we benchmarked Video-STAR against powerful, general-purpose frontier models like Qwen3-VL and Gemini-1.5-Pro The results Table 9 are decisive. Even a highly capable model like Gemini- 1.5-Pro struggles with the fine-grained discrimination required by HMDB-51, scoring only 73.6%. In stark contrast, our B achieves 92.1% %, massive 18.5% absolute improvement. This large performance delta across all benchmarks proves that for specialized tasks like OVAR, a sophisticated, task-specific reasoning framework far more critical and effective than simply relying on a larger, general-purpon model.
A.3 MORE CASE STUDIES
As shown in Fig. 5 and Fig. 6, we present more case studies between Qwen2.5- and our Video-STAR-3B model on the HMDB-51 dataset. Our method can effectively leverage domain- specific tools combined with sub-action decomposition to achieve fine-grained action recognition.
Question: What kind of action shown the video? Given shhot__ball, Punch, Throw, Kick tthinkeeessoning </ /thinlo DntsserrActtoon//swweer
Tool Augmentation
Qwen2. 55-22---VVL
<thinlo Observing the video, the baby is interacting with stuffed enimal. The baby pulling on stuffed cnimal's 's arm, which a form of play. The baby not performing complex actions like running, jumping, throwing The baby's action is more any specific
focused play exploration
PlaysaerPyy.s/ssseeer>
Video-STAR- -R-R-38
<thinlo[1] Observed body parts and movement characteristics:
Sub Motion Decomposition
Arms: The arms are extended forward and then pulled back, indicating farceful movement.
Legs: The legs are in walking running position, suggesting forward motion.
forward, indicating effort push object. [2] Matching candidate actions: Push, Pullup, Punch.
slightly
[3] Pattern comparison for each candidate:
legs' forward slight bend further supports this action, os it helps exerting force on the object.
[Score: 9/10]
movement
clign
pushing,
Pullup: [Score: 4/10] While the arms' movement is similar. the legs' forward motion and the overall body de not match pullup.
movement similar, definition of punching //hinkoo
Punch: [Score: 3/10]
legs' forward
body posture not match
konswer Push- /answer>>
Figure 5: More Case Study between Qwen2.5-VL. VL-3B and our Video-STAR-3B. Qwen2.5-VL VL-
3B misclassifies "push" as "play", while our Video-STAR- accurately identifies the correct action.
18

---

Published as a conference paper at ICLR 2026
Question: What kind of action is shown in the vidco?
Siven Action: _hhot___oaee_tt shal._ball. Punch, Throw, Kick,
cthinloReasoning // think> canswersAction< onswers>
Tool Augmentation)
<thinle Observing the video, the characters entering house. This action involves the characters moving from one location another. which is typicallyassciited associated with walking-// </tiinggt/ttine <answer> </answer>
Te
Video-STAR-- ST:R-3B:
Sub Mob-Motion Decomposition
<thinko[1] Observed bedy parts and
characteristics:
Arns: The arma moving up down in repetitive motion.
Torso: The torse remains relatively stable, with slight forword bockward movements to maintain balance.
Legs: legs are prominently moving, they slightly bent pasitioned support the body's 's weight.
[2] Matching candidate actions: Wave, Push, Throw.
[3] Pattern comparison for each candidata:
repetitive up-and-down motion matches the definition of waving. The and legs remain relatively stable, and the hands make contact with the surface of the water, which is consistent with the action of waving.
Wave: [Score: 9/10]- The
Push: [òcore: 6/10] While the arms' repetitive motion matchas the definition of pushing. the hands are not making contact with
an object, and the mation
relatively stable, which Throw: [Score: 7/10] While the arms' motion matches the definition of throwing, the hands are not making contact with an object. The torso and legs remain relatively stable, which is not consistent with the action of throwing // //think conswer> Wave </answer>
forceful. The
and legr
consistent with action pushing.
Question: What kind of action shown in the vidco? Given Action: shoot_ shoot_ball, Punch, Throw, Kick...
<thinkoReasoning </think
canswersAction//nssn answer>>
Tool Augmentation
Qwen2. VB:
[
Observing the video, the person standing in room with a window behind then. They are wearing hat and jacket, and their hands are their hips. This posture suggests that they are standing still and observing something. There is indication of
movement action taking place s/tink/ tanswer-Stande <ansser>>
Sub Mob-Mottonn Decomposition
Video STAR-3B:
<thinle Observed body parts movement charactenlaties:
Torre: The torso is resting on chair, indicating a seated position. Bedy is not moving significantly. suggesting static posture. the chair, with significant movement interaction with the chair.
position
Logs: The legs positioned
Arms: arma are resting on the cheir, with no significant movement or interaction
chair.
[2)
Matching candidate actions: Sit, Stand, Push.
Pattern comparison for each candidats:
lege are positioned on chair, which clear indication seated pesition. The arms are also resting the chair, which is consistent with static posture.
[Score: 9/10]- The torso resting on the chair,
Stand: [Score: 1/10]- The body not vertical position, and there is significant novement or interaction with object.
Push: [Score: 1/10] - There is no significant force application or interaction with an object. The movement pattern does not match
the action of pushing. o thinne kanswer> Sit /answers>
Question: What kind of oction is shown in the video?
Action shoot ball, Punch. Throw, Kick,..
<thinkoReasoning thhink canswer-Aciinn//ome ansserr
Tool Augmentation
T
<thinlo Observing the video, the person is performing a throwing action. It suggests this is sequence of throws, likely in courtyard. The oction involves extending the arm and releasing the objact, which is the action of throwing. </ think
Throw </answer>
Midee-STAR-333
<thinl[[1] Observed body parts movement cheracterirtics:
Sub Mootiontion Decomposition
Arms: arms are extended forward and upward with the hands gripping a basketball.
Torso: The torse is slightly bent forward, indicating an attempt to control
the shot.
way that supports the body's weight, with the feet planted the ground before jumping. [2] Matching candidate actions: Shoet ball, Punch, Push.
Legs The legs are positioned
[3] Pattern comparison for each candidate:
extended forward and upward position, the torso's 's elight forward bend, and the legs' stable stance all align with the oction of shooting a ball.
Shoot_ ball: [Score: 9/10] The
Punch: [Score: 6/10]- - The lock of a target and the obsence of a forceful impact support the conclusion that this not punch.
Push: [Score: 5/10] Push is forceful action applied to object, wherear the movement more precise and controlled. </thinlo
canswer> Shoot ball s/answer>
Figure 6: More Case Study between Qwen2.5-VL-3B and our Video-STAR-3B. Our Video-
STAR-3B can accurately identify the correct action.
19

---

Published as a conference paper at ICLR 2026
B RELATED WORK
B.1 OPEN- VOCABULARY ACTION RECOGNITION.
Building the robust feature extraction capabiliitiss of CLIP's pre-training, researchers have de- vised numerous video recognition methods leveraging its cross-modal alignment strengths. Initial approaches primarily focused adapting CLIP via full fine-tuning or parameter-effiient strategies. For instance, ActionCLIP (Wang et al., 2021) pioneered the adaptation of CLIP for video recog- nition by fine-tuning and augmenting the entire model to capture motion. Similarly, STAN (Liu et al., 2023a) introduced auxiliary networks to extract temporal features from CLIP's frozen repre- sentations, while ViFi-CLIP (Rasheed al., 2023) simultaneously fine-tuned both visual and text encoders to achieve better performance. These methods, though effective in closed-set scenarios, often overfit to static cues due to extensive parameter updates.
There are two primary paradigms for parameter-efficien paradigms: architecture adaptation and prompt engineering. Adaptation-bal -base methods like Adaptformer (Chen al., 2022), X-CLIP (Ni et al., 2022), and ST-Adapter (Pan et al., 2022) employ compact neural modules in CLIP to dis- till temporal knowledge from static image representations. Prompt engineering approaches like AIM (Yang et al., 2023) and Dual Trans (Park et al., 2023), which inject learnable prefix tokens into CLIP's text encoder to bias the model towards action-specific semantics, while VPT (Ju et al., 2022) and Vita-CLIP (Wasim et al., 2023) explore both prefix and suffix prompting strategies to align video features. Despite these, their performance is limited by adapters' limited capacity, prompt- based modulations, and architectural rigidity, which hinders cross- cross-architecture generalization.
Recent efforts explicitly targeted open open-vocabulary generalization. Open-VCLIn (Weng et al., 2023) introduced weight interpolation regularize fine-tuning, but this approach restricted compatibility to architectures with identical weight dimensions. FROSTER (Huang et al., 2024b) further advanced this by leveraging residual feature distillation. Open-MeDe Meee (Yu et al., 2025a) adopts a meta-learning framework that explicitly mitigates static bias via cross-batch meta- meta--ptimizztion, yet their depen- dency on CLIP's static generalization hindered robustness to motion-centric tasks. Differently, our work diverges from these paradigms by proposing framework that integrates sub-motion decom- position with tool- aol-auumented RL, thereby enabling robust open-vocabulary inference.
B.2 MULTIMODAL LLMS REASONING.
Recent advances in large language models (LLMs) have demonstrated that RL-based post-training can significantly enhance reasoning capabilities, as exemplified by OpenAl-ol (Jaech et al., 2024) and DeepSeek-RI R1 (Guo et al., 2025). Several works have extended these paradigms to multimodal language models (MLLMs) for tasks such as mathematical and scientific image VQA (Peng et al., 2025; Huang et al., 2025), image segmentation and grounding (Liu et al., 2025a; Bai et al., 2025b; Shen et al., 2025; Liu et al., 2025b), video spatial or temporal grounding (Wang et al., 2025d; Ge et al., 2025; Park et al., 2025; Li et al., 2025c) and video understanding VQA (Feng et al., 2025; Wang et al., 2025b; Li et al., 2025b; Cheng et al., 2025). Despite these, existing methods remain limited in cross-modal interaction and hallucination, especially for long video scenarios (Chen al., 2025). To address this, we propose multimodal CoT reasoning through tool lool-augmented RL, which explicitly reduces hallucination through structured perception.
B.3 TOOL-AUGMENTED AGENTIC SYSTEM.
Recent advancements in LLMs have demonstrated that integrating external tools can significantly enhance multimodal reasoning capabilities. Early works like FAST (Sun et al., 2025) and MVoT (Li et al., 2025a) established foundational approaches by incorporating visual evidence into reason- ing processes, forming multimodal CoT for image tasks. Concurrently, Va- (Liu et al., 2023b) pioneered training strategies for tool use in visual reasoning, while Visual Program Dis- tillation (Hu et al., 2024) leveraged program-derived CoT data to transfer tool-use skills. Visual CoT (Shao et al., 2024) further enriched this domain by creating synthetic datasets to boost rea- soning. CogCoM (Qi et al., 2024) introduced six manipulation strategies through synthetic chain- of maniiiiplanioo (CoM) data, and TACO (Ma et al., 2024) provided 273K multimodal reasoning traces from 15 visual tools. Recent methods like Simple (Wang et al., 2025c), THYME (Zhang et al., 2025b), and Py Vision (Zhao et al., 2025b) extended tool use with RI, thus enhancing general
20

---

Published as a conference paper at ICLR 2026
reasoning capacity. Visual search capabilities were advanced by DeepEyes (Zheng et al., 2025), VGR (Wang et al., 2025a), and OpenThinkImg (Su et al., 2025b), which integrated tools such as im- age zoom-in and sketching to improve visual analysis. However, partial methods often rely static tool pipelines, limiting their ability to disentangle complex actions open-vocabulary settings. Our framework addresses this by explicitly modeling contextual sub-motions and tool usage efficiency with hierarchical motion relevance.
a dynamically balancing
C DISCUSSION
Enforcing Visual Grounding via Tool-Augmented Visual CoT. A core objective of our framework is to mitigate the "text-centric reasoning" prevalent in many MILMs. We achieve this by enforcing Visual Chain-offTToughh Thought (Visual CoT), process fundamentally driven by our tool-use mechanism. Instead of allowing the model to generate abstract, text-based reasoning steps, we compel it to build its logic from concrete visual evidence extracted by external tools. Specifically, tools like pose estimation and human detection extract structured, non-textual facts (e.g., skeletal keypoints, bounding boxes) directly from the video frames. These geometric facts become the foundational "thoughts" in the model's reasoning process. For instance, rather than vague thought like "the person is moving their arm," the model is guided to construct a thought grounded in visual evidence, such as: "The wrist keypoint [x1, y]] rises from low position to above the shoulder keypoint [x2, y2], then oscillates horizontally." By designing reward function that prioritizes reasoning chains demonstrably built upon these tool- mool-generated visual facts, we effectively force the model's entire inference process to be anchored to the visual world, preventing it from drifting into pure text-based speculation and reducing cross-modal hallucinations.
Discussion on Computational Cost and Future Work. While Video-STIa demonstrates highly favorable trade-off between performance and cost, we acknowledge the computational overhead in- troduced by tool invocation, which accounts for significant portion of the inference latency (as shown in Table 5). This latency is primarily due to executing external models and making API calls. However, view this promising direction for future optimization rather than fundamental limitation. For practical deployment, inference time can be significantly accelerated through stan- dard engineering techniques. Model distillation (e.g., from a 3B 1B model), quantization (e.g., to 8-bit or 4-bit precision), and parallelization strategies (e.g., tensor and pipeline parallelism) could drastically reduce both memory footprint and latency. Exploring these optimizations create lightweight yet powerful version of Video-STAR presents an exciting avenue for future research.
D TOOL LIBRARY DETAILS
This section provides a detailed specification of tools used by our Video-STAR framework, including human detection tools, pose estimation tools, and online cotrival-angmmeetttion tools.
D.1 HUMAN DETECTION TOOLS
Our framework employs the Ultralyties YOLO (Weng et al., 2023) detector as a human detec- tion model. We chose this model for its optimization for real-time performance and accuracy. Its primary function is to locate people in video frames. The model employs an anchor-free detection architecture combined with spatial attention mechanisms. This design enables it to handle common challenges such as occlusions, varying human scales, and crowded scenarios. The tool's ability to ensure precise localization of human bounding boxes, even in cluttered environments, is critical for our framework. allows our model to isolate action-relevant regions from background distrac- tions, which is key step in reducing visual-semantic ambiguity and preventing model hallucination during the reasoning process.
D.2 POSE ESTIMATION TOOL.
For pose estimation, we utilize the skeletonization capability of Ultralytics YOLO 11, which trans- lates detected person into structured kinematic representation. The model outputs skeleton
21

---

Published as a conference paper at ICLR 2026
with 17 keypoints that follow the COCO format, including joints for the elbows, wrists, and knees. achieves this with sub-pixel precision, which is essential for detailed motion analysis. The model infers these joint positions using its knowledge of hierarchical spatial relationships between body parts. For instance, it understands the natural correlation between hip and knee angles during loco- motion. This underlying knowledge allows it generate coherent skeletons even when body parts are in fast motion or partially occluded. This fine-grained representation enables our system to cap- ture subtle motion patterns. For example, we can analyze the precise wrist rotation in "handshake" or the exact knee flexion during a "jump".
D.3 ONLINE RETRIEVAL--UGGEENTAIN TOOLS
We use the Qwen API as our online ctrrivallaugmeenattion tool. This tool provides our framework with essential contextual information. It has two primary functions, O The first function is to resolve the ambiguity of high-level action definitions. We use this tool to get a detailed, category-specific explanation for given action label. It helps redefine action words through their transitional phases (e.g., "stand" is described as "transit from sit or lay to stand"). This process provides our model with deeper semantic understanding, closing the gap between simple label and its complex physical meaning. @ The second function is to describe the motion within specific key frames, which helps the model understand complex, multi-phase movements. After our framework extracts temporally salient frames (e.g., frames with maximum joint displacement), we send these frames to the tool. The tool then generates concise textual description of the action happening at that moment. This description emphasizes both spatial configurations, like "rapid arm extension," and temporal dy- namics, like "followed by torso rotation". This maps descriptive text precisely informative video segments, allowing the model to disambiguate sequential actions by understanding each component.
E PROMPTS
During the supervised fine-tuning (SFT) stage of Video-STAR, we designed structured input prompt to generate high-quality thain-of-though, (CoT) data. The prompt template is follows:
First Round Prompts:
TURN TERNTTUURNNALLTTTT ("**
will show you a video of human action. Your task is to determine which visual analysis tools would help you better identify the action. You have three independent tools available:
1. Pose Estimation Tool.:: Adds skeleton keypoints and connections show body pose and joint movements
2. **Person Detection Tool**: Adds bounding boxes around detected persons
to highlight human subjects
3. Nouu Explanation Tool**; Prevides detailed explanations of action types to help with classification
4. +iVideo Description Tool++: Provides description input video to help better understand video content
Please analyze the video using step-by-step reasoning and decide for each tool independent
Analysis Requirements:
1.
**For Pose istitttion Assess whether body joint movements and pose
details are crucial for identifying this action
2. *For Person Detection**; Evaluate whether clearly identifying and localizing the person (s) would help with action recognition
3. **For Noun Explanation**: Consider whether you need detailed make accurate classification
explanations of action categories
4. **For Video Deseriptionsw: Judge whether you require detailed
description concerning video to help identifying action
Output Format: sthink-step-by-steg reasoning process:
22

---

Published as a conference paper at ICLR 2026
[1] Video content analysis: [describe what you observe in the video
[2] Fose estimation evaluation: [assess whether joint/skeleton info would helpl
[3] Ferson detection ovaluation: [assess whether persen localization would help]
[4] Noun explanation evaluation: [assess whether action category details would help]
[5] Final tool selection reasoning: [explain your decisions for each tool
</think> <action>
<humandyes or no</human> <pose>yes or no</pose>
<action or no</action> <videc>yes or no</video> </action>
="*)
Second Round Prompts:
RECOND_TURK TEMPLATE
will provide you with a video and ask you to identify the human action shown.
There is also annotated versien of this video available.
The annotated video may contains the same content but with additiona! annotations:
Blue bounding boxes around detected persons
Pose estimation points and skeleten lines showing body keypoints
Your task to follow this three-stage reasoning process:
Input Format:
Original Question: What kind of human action is shown in the video? Description this Video: (VIDEO_DESCRIPTION)
All action types and their explanations are shown as follews: { ALLONED_ANSWERS])
Analysis Requirements:
**Stage 1: Sub-Motion Decompositicat*
1. Identify all body parts showing significam motion in the video (e.g.,
arms, legs, torso)
3. For each identified body part:
Note its movement direction (up/down, rotational, etc.)
Record contact points with objects (if any)
3. List them by importance from top to bottom
Staagg 2: Action Candidate selcctien**
1. Extract the key movement patterns from its description in given action types
2. Compare with the video's observed:
Temporal sequence of movements (which body part moves first)
Interaction patterns petween body parts
Force application points (e.g., hand gripping vs. pulling)
3. Generate 2-3 candidate actions that best match the observed body part movements
**Stage 3: Matching Scoring**
1. Match these observations to the action definitions in given action types
2. Score each candidate based on: (Maximum Score: 10)
Body part involvement precision
Movement pattern similarity Object interaction consistency
Output Format:
sthinkostep-by-ster reasoning process:
23

---

Published as a conference paper at ICLR 2026
[1] Observed body parts and movement characteristics: [Body Part 1]: IDirection/Centact Description]
[Body Part 2]: [Direction/Centact Description]
[2] Matching candidate actions:
[Candidate 1]: Matches [Body Part [Movement Type]
[Candidate 2]: Matches [Body Part B] [Movement Type]
[3] Fattern comparison for each candidate: ICandidate 11: [Score] [Matching Details]
[Candidate 2]: [Score] [Matching Details] </think> aanseeractioo-tyyee...ooo
*"*)
This structured prompt ensures our dataset contains diverse and causally plausible reasoning pro- cesses, which are critical for cultivating the model's foundational perception and planning capabili- ties before RL fine-tuning.
LLM CLARIFICATION
We clarify the role of Large Language Models (LLMs) in the development of this manuscript. Dur- ing the drafting process, LLMs were utilized primarily for two purposes: translating initial technical content from Chinese to English and refining the language the final draft. This included correcting grammatical inconsistencies, optimizing sentence structure, and improving the clarity and coherence of the narrative. is essential to emphasize that all conceptual contributions, including the formu- lation of open-vocabulary action recognition challenges, the design of the Video-STAR framework (e.g., contextual sub-motion decomposition and hierarchical reward mechanisms), the integration of domain-specific tools (e.g., YOLO 11 for pose estimation and Qwen API for explanations), and the experimental design across HMDB-51, UCF-101, and Kinetics-400/600 datasets, are the original work of the human authors. The LLM was employed strictly a linguistic tool enhance the presentation of ideas, with no involvement in hypothesis generation, algorithmic innovation, or data analysis. The core scientific claims, methodological innovations, and empirical validation presented in this paper is entirely attributable to the research team.
24


Reviews:
[{"review_id": "WgXVKdwH9k", "reviewer": "Reviewer_mh4G", "summary": "This paper addresses two fundamental limitations of current multimodal large language models (MLLMs) for open-vocabulary action recognition:\n(1) the over-reliance on textual priors that neglect domain-specific visual cues, and\n(2) the inability to distinguish semantically ambiguous actions in open-vocabulary settings.\nThese issues are indeed of great importance and widely exist across multiple MLLM-based video understanding tasks. The proposed Video-STAR framework attempts to mitigate these problems by constructing multimodal Chain-of-Thought (CoT) data and introducing tool-augmented reasoning with reinforcement learning optimization.", "questions": "1. The paper claims to mitigate “text-centric reasoning,” but the explanation of how this is achieved is somewhat abstract.\nPlease explicitly describe the mechanism by which visual grounding is enforced during training and inference.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper is motivated by a well-defined and practically significant problem—bridging the gap between text-centric reasoning and visually grounded inference.\n\n2.The use of a multimodal CoT to train MLLMs for structured, tool-guided reasoning is a valuable direction.\n\n3.The framework integrates detection, pose estimation, and semantic reasoning tools with a hierarchical reward design, which is novel and effectively demonstrated across multiple benchmarks.", "weaknesses": "Major Comments\n\n1.During data construction, the model is trained using decomposed motion sequences (sub-actions).\nHow does the framework behave when encountering previously unseen actions in the open-vocabulary test set?\nIs there any mechanism ensuring compositional generalization beyond the motion patterns observed during training?\nWithout such a mechanism, the model might overfit to the seen sub-motion combinations.\n\n\n2.How exactly is each action decomposed into sub-actions?\nIs the decomposition manually annotated, automatically generated, or derived from an existing motion ontology?\nMoreover, how do the authors ensure that the set of sub-actions can comprehensively cover unseen action categories during testing?\nSince this decomposition is central to the model’s reasoning ability, more transparency on this process is necessary for reproducibility and understanding its generalization scope.\n\n3.Video-STAR performs task-specific supervised fine-tuning (SFT) and reinforcement learning (RL), while most baselines (e.g., Qwen2.5-VL,) are evaluated without any fine-tuning.\nThis introduces a fairness issue: the superior performance of Video-STAR may partly result from extra supervision rather than the proposed method itself.\nA fairer comparison would include a fine-tuned Qwen2.5 baseline trained on the same dataset but without tool usage and sub-motion decomposition, to isolate the true contribution of the proposed framework.\n\n4.The ablation studies only compare the presence vs. absence of tool usage but do not analyze which tool or combination contributes most.\nHow is the tool selected in practice?\nWhat is the performance when all tools are used simultaneously (pose, detection, action explanation, and video description)?\nA more detailed comparison of tool selection strategies would clarify whether the proposed policy is optimal or if simpler combinations yield similar gains.\n\n5.Since Video-STAR relies on multiple external tools, the inference pipeline likely introduces additional computational overhead.\nThe paper should report both the overall inference latency and the module-wise cost (e.g., tool invocation vs. model reasoning).\nComparing the efficiency of Video-STAR with standard MLLMs would help quantify the trade-off between accuracy improvement and computational expense.\n\n6. Line 213: The symbol T_r appears for the first time without explicit definition. Please clarify its meaning and source.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K1SyTDadzU", "reviewer": "Reviewer_Hc9X", "summary": "The paper discusses this idea: don’t let the model guess in one shot. Make it first break the action into sub-motions (arms, torso, legs, contact), then match those to candidate actions, and finally score them — and let the model call tools (YOLO human detection, pose estimation, Qwen-based action/video explanation) when it thinks vision cues are not enough.", "questions": "The questions are the same as weakness.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper names two real OVAR pain points, cross-modal hallucination and similar-action confusion, and every design choice (tools, sub-motions, hierarchical reward) points back to those, making a coherent motivation.\n2. Treating “shoot ball” as “bend → jump → arm extend → release” matches how actions are actually separable in video; it’s more plausible than pure text-CoT on top of global video tokens.\n3. Tools aren’t a fixed pipeline — the model decides whether to call pose / human / RAG /video description, and the reward penalizes useless tool calls. That’s better than many “agentic VLM” papers that just always run pose.", "weaknesses": "1. They say open-vocab, but the system leans on online RAG / Qwen API to pull category-specific definitions at inference. That narrows the search space. It’s closer to “recognition with external label dictionary + video grounding” than to “truly open” recognition.\n2. YOLO 11 for human + pose, Qwen API for explanation / video description — that’s a very specific tool stack.\n3. First round: “which tool(s)?” Second round: “do sub-motion reasoning.” Plus GRPO sampling 4–6 responses. That might not be cheap for real-time video, and the paper doesn’t talk about latency / streaming.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zgdfYfdvgU", "reviewer": "Reviewer_vdeG", "summary": "This paper proposes, implements and investigates a novel strategy and framework, i.e., Tools Augmented CoT reasoning for zero-shot fine-grained human action recognition. The framework employs VFMs, VLMs and Visual RAG to extract sub-concept vision representations such as human, pose, and video explanation, and generated a combined prompt on proposed format for CoT to MLLM for final stage prediction. The core innovations are the format definition and implementation of sub-action decomposition, candidate selection, and matching scoring for CoT and reinforcement learning. Concrete evaluations are performed on five formal benchmarks and the results shown significant improvements over the SOTA.", "questions": "See Weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "A novel Tool Augmented CoT framework and implementation approach for open-vocabulary action recognition (OVAR), concrete evaluations of 5 benchmarks and new SOTA performance.", "weaknesses": "There are still some uncertain issues. (1) the experiments on only one base MLLM model, Qwen2.5-VL, are reported, where the training prompts of reasoning chain for CoT are generated by Qwen2.5-VL-72B, and then fine-tune the small-size model Qwen2.5-VL-3B and Qwen2.5-VL-7B for experiments, is teacher-student knowledge distillation on the proposed reasoning chain format able to achieve similar effectiveness? May be better to add more results on other leading frontier MLLMs such as InternVL2.5, Gemini-2.5-Pro, Llama, etc. (2) is it applicable to professional actions such as FineGym, Diving? Where expertise sub-action concepts might not well be learned for AGI models. (3) On lines 356-360, are the protocols defined for previous benchmarks? Please cited them. If novel classes Y_N are completed unknown in MLLM, how to generate the novel nouns of the new classes? So that the base VLM and MLLM have been trained on related concepts, and may be not strictly zero-shot performance. (4) As the training sub-concepts and reasoning chains are generated by VFM, VLM, and MLLM, are there hallucinated chains which lead to final correct answers on ground truth? Maybe the discussion on the Visual Grounded CoT is helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ywC3L02ey2", "reviewer": "Reviewer_zTP9", "summary": "The paper proposes Video-STAR, a framework for open-vocabulary action recognition that integrates contextual sub-motion decomposition with tool-augmented reinforcement learning on top of MLLMs. Actions are no longer treated as atomic labels; instead, they are decomposed into discriminative motion primitives, while external tools are invoked to reduce cross-modal hallucinations and enable category-specific reasoning. A hierarchical reward is designed to jointly optimize tool-usage efficiency, structural coherence, and sub-motion relevance. Extensive experiments demonstrate substantial gains over CLIP-based baselines and vanilla MLLMs across multiple settings.", "questions": "Please see the weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The combination of sub-motion decomposition and tool-augmented RL constitutes a meaningful departure from static cross-modal alignment pipelines commonly used in OVAR.\n\nThe method delivers consistent and large improvements across diverse benchmarks and evaluation settings.", "weaknesses": "1. While the integration is well-designed, the core building blocks (tool-augmented CoT, RL-based post-training, and sub-action decomposition) are all known paradigms; the novelty is primarily at the system-level composition rather than at the level of a fundamentally new principle.\n\n2. The approach depends on specific external tools, yet the paper does not analyze robustness to tool inaccuracies or the portability of the method under alternative tool choices.\n\n3. The computational overhead of repeated tool invocation and multi-round RL inference is not reported nor compared against CLIP-based or purely SFT-based OVAR pipelines. Furthermore, the training cost of GRPO fine-tuning is also not quantified or compared with prior methods.\n\n4. Although standard CLIP-based OVAR methods are included, the paper does not benchmark against recent LLM-augmented CLIP paradigms that explicitly incorporate generative priors for action understanding, such as [1–3].\n\n5. The evaluation is conducted primarily against mid-scale or earlier-generation MLLMs (e.g., Qwen2.5-VL), without comparison to state-of-the-art frontier models (e.g., Qwen3-VL, GPT-5, Claude, Gemini), many of which already demonstrate strong video reasoning capabilities.\n\n[1] Building a Multi-modal Spatiotemporal Expert for Zero-shot Action Recognition with CLIP\n\n[2] Generating Action-Conditioned Prompts for Open-Vocabulary Video Action Recognition\n\n[3] VTD-CLIP: Video-to-Text Discretization via Prompting CLIP", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)