STVG-R1: Incentivizing Instance-Level Reasoning and Grounding in Videos via Reinforcement Learning
Authors: Xiaowen Zhang
Zhi Gao
Licheng Jiao
Lingling Li
Qing Li
Venue: ICLR 2026 Poster (2026)
Keywords: spatial-temporal video grounding
vision language model
reinforcement learning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: In vision–language models (VLMs), misalignment between textual descriptions and visual coordinates often induces hallucinations. This issue becomes particularly severe in dense prediction tasks such as spatial–temporal video grounding (STVG). Prior approaches typically focus on enhancing visual–textual alignment or attaching auxiliary decoders. However, these strategies inevitably introduce additional trainable modules, leading to significant annotation costs and computational overhead. In this work, we propose a novel visual prompting paradigm that avoids the difficult problem of aligning coordinates across modalities. Specifically, we reformulate per-frame coordinate prediction as a compact instance-level identification problem by assigning each object a unique, temporally consistent ID. These IDs are embedded into the video as visual prompts, providing explicit and interpretable inputs to the VLMs. Furthermore, we introduce STVG-R1, the first reinforcement learning framework for STVG, which employs a task-driven reward to jointly optimize temporal accuracy, spatial consistency, and structural format regularization. Extensive experiments on six benchmarks demonstrate the effectiveness of our approach. STVG-R1 surpasses the baseline Qwen2.5-VL-7B by a remarkable margin of 20.9% on m_IoU on the HCSTVG-v2 benchmark, establishing a new state of the art (SOTA). Surprisingly, STVG-R1 also exhibits strong zero-shot generalization to multi-object referring video object segmentation task, achieving a SOTA 47.3% J&F on MeViS.

Published as a conference paper at ICLR 2026
STVG-R1: INCENTIVIZING INSTANCE-LEVEI REASONING AND GROUNDING IN VIDEOS VIA REINFORCEMENT LEARNING
Xiaowen Zhang-.2, Zhi 2 Licheng Ji, Lingling Li', Qing Li 1 Xidian University State Key Laboratory of General Artificial Intelligence, BIGAL 3 Beijing Institute of Technology
https://stvg-r1 ri .ii thub. 1o/
ABSTRACT
In vasion-language models (VLMs), misalignment between textual descriptions and visual coordinates often induces ballucinations. This issue becomes partic- ularly severe in dense prediction tasks such as spatial-temporal video grounding (STVG). Prior approaches typically focus enhancing visual-textual alignment or attaching auxiliary decoders. However, these strategies inevitably introduce additional trainable modules, leading significant annotation costs and computa- tional overhead. In this work, we propose a novel visual prompting paradigm that avoids the difficult problem of aligning coordinates across modalities. Specifi- cally, we reformulate per-fraue per-frame coordinate prediction as a compact instance- identification problem by assigning each object unique, temporally consistent ID. These IDs are embedded into the video as visual prompts. providing explicit and interpretable inputs to the VI.Ms. Furthermore, we introduce STVG-R1, the first reinforcement learning framework for STVG, which employs task-driven reward to jointly optimize temporal accuracy, spatial consistency, and structural format regularization. Extensive experiments on six benchmarks demonstrate the effectiveness of our approach. STVG-RI surpasses the baseline Qwen2.5-VL VL- 7B by remarkable margin of 20.9% on IoU the HCSTVG-v2 benchmark, establishing a new state of the art (SOTA). Surprisingly, STVG-RI also exhibits strong zero-shot generalization to multi-object referring video object segmenta- tion task, achieving a SOTA 47.3% J &F on MeViS.
INTRODUCTION
In video grounding task, hallucination in vision-language models (VLMs) is a common phe- nomenon, where timestamps may extend beyond video duration coordinates may exceed the frame resolution (Wang et al., 2024a Liu et al.) 2024a Chen et al. 2024). A widely accepted per- spective is that the hallucinations stem from misalignment between the visual and textual modalities (Lin et al., 20224] Wang et al.) 2024b). Such misalignment leads to greater performance degradation dense prediction tasks, where bounding box segmentation mask for each frame are required.
To reduce the impact of cross-modal misalignment, existing research focuses on improving the alignment capability of VLMs (Wang et al. [2025a) Ye et al.) (024)) or avoiding direct coordinate prediction (Yuan et al. 2025 Sun et al., (025). Despite their success, these strategies typically introduce additional learnable components and exhibit limited generalization. As illustrated in Fig- approaches (Li et al., 2025) directly output explicit frame-level coordinates,
ure alignment-basee but struggle in multi-object scenes and ofieen yield inconsistent or even meaningless predictions, such as [0.00, 0.00, 0.27, 0.00]. In contrast, basoder-based methods alleviate this by introducing segmentation tokens for cross-frame consistent prediction, yet their implicit outputs limit general- ization. Motivated by these challenges and prior attempts, we propose core idea: if the complex fram coordinate prediction can be reformulated into a compact and interpretable formulation, it becomes possible to mitigate visual- textual misalignment and enhance generalization.
Based on the above observation, we draw inspiration from existing research on visual prompts, which demonstrates the effectiveness of simple yet consistent referential cues for object represen-

---

Published as a conference paper at ICLR 2026
Main Task
Zero-shot Task
Spatial- Temporal Video Grounding
Multi-Object Refering Video Object Segmentation
Qwestion: At what point in the vidco can we observe there is a frishee away dog taking place? sujject.ubject locared??
Question: Give the query: rabbits cating leaves. Where is the corresponding subject'objoet located?
cerresponding
The ovent bappens in 0.0- seconds, The frisbce the left of the dog taking place. The coordinates 276, 358, 358).
The rabbits is in the center the image. There are three white rubbits. The courdinates [89, 903, 672].
Qwen
Qwen
During the span of 22.9--13.6x Object hounding box:
Object bounding box: Os: [0.26, 0.00, 0.63, 0.63), 8.4c: [0.00 0.00, 0.27, 0.00), 8.6s: [0.00, 0.00, 0.27, 0.001 8.8s: [0.00, 0.00, 0.00).
12.9c [0.44, 0:45, 0,56, 0.59). 13.4s:
SLaV
[0.39, 0.33, 0.53, 0.51J. 13.6w: [0.47, 0.48, 0.61, 0.65].
LLaV
Our object-centric visual prompting paradiguu
Our object-centric visual prompting paradigm
<thiake- The frishee away from dog (IDI)
cating leaves first frame, and this behavior continaes in the subsequeet frames well. The rabbit eating leaves. </think-
<obduk>> The rabbit
visible on the ground in the first fow seconds of the video. dog interacting
frisbee, </think> <answerc Target ID: 2. Time range: 0.00 to 3.96.
STYG-RI
STVG-RI
</answer>
Sanswer> Target ID: 1.2. </answer>
Figure 1: Comparisons of general VLMs,, specialized VL.Ms, and proposed STVG-RI model. While Qwen2.5-VL--- outputs a single meaningless bounding box without timestamps, LLaVA-ST is re- stricted to one bounding box per frame. In contrast, STVG-RI achieves strong performance on both spatial-temporal video grounding and zero-shot multi-object referring video object segmentation.
tation (Shtedritski et al. 2023 Cai et al. 2024 Yang et al. 2024). Taking GPT4Scene (Qi et al,
2025) as an example, consistent object IDs across multi-view images is embedded enhance 3D understanding. Following this insight, in this paper, we introduce an object-centric visual prompting paradigm for caatial-temporal video grounding (STVG). Specifically, each object is automatically assigned a unique and temporally consistent identifier throughout the video sequence. Concretely, the first frame processed with an object detector (Tian al. 2025 Liu al., 2024b) Xiao al., 2023) to obtain candidate bounding boxes, which are further refined using the segmentation and tracking capabilities of SAM2 (Ravi al. 2024). To handle newly appearing or previously missed objects, re-detection is performed at fixed intervals, and ReID is employed to maintain temporal consistency. Finally, each candidate instance is overlaid with a numeric marker that serves as its object ID on its center, yielding compact yet interpretable formulation for video spatial grounding.
Building on this paradigm, we introduce STVG-RI, the first reinforcement learning framework for STVG. Unlike conventional supervised fine-tuning (SFT), which relies token-level loss, STVG- RI incorporates a task-driven reward that jointly optimizes temporal accuracy, spatial consistency, and structural correctness. A positive spatial consistency reward is obtained when the predicted object ID is aligned with the ground truth and also falls within the localized temporal segment.
The object-centric visual prompting paradigm achieves substantial performance improvements across four general VLMs in zero-shot settings. Specifically, Intern (Zhu et al. 2025). Owen2.5-VL-7B, Qwen2.5-VL--2BB (Bai al. 2025), and Qwen3-VL-8B improve vloU@i by +3.6%, +12.5% +6.0%, and +28.3% on HCSTVG-vi (Tagg et al. 2021). Beyond zero-shot scenar- ios, the enhanced reasoning capability introduced by reinforcement learning establishes new SOTA on five benchmarks. Remarkably, the STVG-R1 also achieves strong performance on the unseen multi-object referring video object segmentation task, highlighting its robust generalization ability. We attribute this generalization to the object-centrie visual prompts, which provide explicit object identifiers during reinforcement learning, enabling instance-level reasoning and grounding.
The main contributions of this paper centric visual prompting paradigm that reformulates dense per-frams coordinate prediction into a compact object ID identification task. (2) We propose STVG-R1, the first reinforcement learning framework for peatial-temporal video grounding, built upon the GRPO algorithm. (3) Extensive ex- periments across six benchmarks demonstrate the effectiveness of approach. Moreover, its strong
follows: (1)
We introduce simple yet effective object-
2

---

Published as a conference paper at ICLR 2026
Large Language Model
Large Language Model
Large Language Model
answer
Frame-level Decoder
Databuse Retrieval
Encuder
Visual
Aligament
Text
Visual Encoder
Encoder
+ answer
databose
Visual Encoder
Engoder
anawer
Video
video
prompted
Test
Figure 2: Comparison of paradigms: (a) VLM produces both timestamps and frame-level coordi- nates with a trainable alignment block; (b) VLM generates segmentation tokens, which are then processed by trainable decoder; (c) method uses training-free object-centric visual prompted video for spatial-temporal video grounding.
performance on the unseen multi-object referring video object segmentation task further highlights its generalization capability.
RELATED WORK
2.1 SPATIAL TEMPORAL VIDEO GROUNDING
In the research peatial-temport video grounding, existing approaches be broadly categorized
into models based on visual-language pretraining (VLP) and models leveraging VLMs. VLP-based
methods typically employ pretrained encoders, such as CLIP (Radford al. 2021 I3D (Carreira &
Zisserman 2017). Intern Video-v2 (Wang et al.) 2024c), and BERT (Devlin et al. 2019). to extract visual and textual features, followed by the design of task-specific modules for multimodal feature fusion and tailored decoding. These approaches (Gu et al. [2024] 2025 Lin et al.
(2022) still demon-
strate dominant performance on several STVG benchmarks (Tang et al. 2021; Zhang et al. 2020). However, despite their effectiveness, these VLP-based task-specific models continue to struggle with generalization, even on simpler spatial-only or temporal-only video grounding tasks.
Recent efforts have increasingly adopted VLMs (Li al., 2024 Bai et al. 20255 Abdin et al. 2024] Zhang et al. 2024) for video spatial grounding, owing to their superior cross-modal reasoning and generalization abilities. Within this line of research, as shown in Figure aa). one direction directly exploits VLMs for dense prediction, producing both temporal segments and frame-level spatial lo- calization results. For example, SLTV (Li al. (025) enhances the aligmment between textual descriptions and visual coordinates by incorporating additional tokens into the input text embed- dings. Subsequently, SpaceVLLM, (Wang al. 2025a) follows similar strategy by introducing spatial-temporal query tokens to address the alignment challenge. However, these additional train- able tokens require large-scale, high-quality dense prediction data and lead substantial computa- tional overhead. As shown in Figure 2])), another direction mitigates the impact of misalignment by prompting VLMs generate segmentation tokens (Yuan et al.) 2025] Sun al. 2025 Munas-
(agure[aa...
inghe et al. (025), which are then passed into trainable decoder (Ravi et al. 2024). However, their reliance on iterative decoding further increases training complexity and time.
2.2 REINFORCEMENT LEARNING IN VLMS
Reinforcement learning (RL) has demonstrated strong potential in improving the reasoning capa- bilities of LLMs, particularly through reinforcement learning with verifiable reward (RLVR) (Guo)
al. 2025
Chen al., 22025a: Jaech et al.) 2024). For VLMs, many works (Liu et al., 2025 Shen
et al.
2025 Zhang et al. 2025a Chen et al., 20255] Zhang et al. 2025b) also apply this reward-
driven training paradigm to tackle complex tasks (Yang al. 2025
Fu et al. 2025). Recent studies further extend this direction to video understanding and multimodal agents (Fan et al. 2024 2025).
More closely related to our setting, Video-R1 (Feng et al. 2025) the first attempt to explore the R1 paradigm in the video domain, introducing the T-GRPO algorithm to explicitly encourage temporal understanding by shuffling the order of input video frames. Building on this foundation, Time-RI (Wang et al., 2025b) proposes novel post-training framework for temporal video ground- ing, also based on the Group Relative Policy Optimization (GRPO) algorithm (Shao et al.) 2024). More encouragingly, Time-RI demonstrates that using continuous metrics such as IoU as rewards provides more intuitive optimization signals and achieves better performance than token-level su- pervised fine-tuning. However, applying RL jointly address patial-tempoot video grounding
3

---

Published as a conference paper at ICLR 2026
Visual Prompts
Accuntely pinpoint the event does the udult wearing jeams pai! the video.
Target ID:
Time range: 38.8 to 39.9
Spatial
D.Deemniie the precise time period.
Ri
A
2.Identify which object ID
correpponding to the deseribed event
R
Policy Model
Temporal Roward
Group Compute
Video Frimes
Prompted
Og
On
Template Reward
R
A
1s
2s
35
1s
28
46
Figure 3: An illustration of our proposed STVG-RI framework. Each object is assigned a unique ID via visual prompts, and the policy model is trained with spatial, temporal, and template rewards.
remains an underexplored yet promising direction. To bridge this gap, STVG-RI integrates GRPO with STVG-specifi rewards, achieving superior performance.
METHOD
3.1 STVG-RI FRAMEWORK
Our approach reformulates spatial-temporal video grounding a paradigm shift from dense per- frame bounding box regression to a compact formulation based on visual prompts. Figure 3 illus- trates the overall architecture of the STVG-RI model. Specifically, given video V (4 ( .73 with 7 frames, each frame 1 is first augmented with a set of visual prompts,
P. {pi..... Pk.
I I...
where E {I...., T indexes frames, K, is the number of instances in frame t, and @ denotes over- laying the visual prompts onto the frame I, yielding the augmented sequence V "ii....IIy!
To control memory consumption, we constrain each video to approximately R 1.6 x 10 pixels in total. Concretely, for a video V with frame resolution H x W and duration D seconds, we resize frames to H' x W zs R/(2D), where frames are uniformly sampled at 2 FPS. For example, a 30-second video yields 60 frames, each with resolution of about 96 x 96 3. Finally, the sequence of visual coompt-augmented frames V and textual query q fed into VLM ne. which jointly predicts the temporal interval [e] and the corresponding object identifie z.
3.2 OBJECT- CENTRIC PROMPTED VIDEO CONSTRUCTION
Data format. Each sample is denoted as (V, q, M, where P (Pt)T= represents the set of visual prompts
frames, M is the segmentation mask database. and A is the ground-truth answer, defined as the target object ID. Concretely, M stores for each frame T, a set of instance IDs paired with their run-length encoded masks (Golomb) 1966). For consistency with the ground-truth annotations, each mask m is further converted into its corresponding bounding box bf.
To formally derive A, establish frame-level correspondence within the ground truth temporal interval. For each frame I, we compute the loU between the ground-truth Or and all candidate bounding boxes {bt [0k]k11 and assign to frame t the ID 4, with the highest overlap:
K.
arg max ToU( b(). kE ke(l....K.)
Over the entire video V. the final answer A is obtained by majority voting:
arg max Σ 1/x - 1],
where 1[:] denotes the indicator function. This defines the target object as the identifier with the highest overall loU consistency across the video.
4

---

Published as a conference paper at ICLR 2026
Data generation pipeline. To construct object-centric visual prompted videos, we integrate several existing vision models into a unified pipeline. The first frame /1 of each video is processed by an off-the-shelf object detector to produce bounding boxes {bk for all candidate instances across COCO categories. These detections serve prompts for SAM2, which generates high-quality seg- mentation masks {m ) k= that are then propagated to subsequent frames via tracking. To capture newly appearing or previously missed objects, periodic re-detection with loU-based matching is performed, where each detection is compared against the masks already tracked the same frame and is treated as a new instance only when its geometric overlap with all existing objects remains consistently low. Once new instance is identified, SAM2 further applied to perform both for- ward and backward tracking from the discovery frame to recover its complete temporal trajectory. For each mask m4. we embed a compact visual prompt p at its centroid (zi. v())
Importantly, although the COCO taxonomy does not fully cover all categories present in the videos, the detector can still provide bounding boxes for nearly all instances. For example, while fish is absent from the COCO label set, such instances are nevertheless detected under alternative cate- gories. These semantic misclassifications do not affect our framework, as supervision depends only on consistent instance identities rather than precise class labels.
Data source. Two widely adopted STVG datasets are used for training. HCSTVG (Tang et al,, 2021) focuses human-centric grounding data. We merge the training splits of v1 and v2, and remove any samples that appear in the validation or test sets. VidSTG (Zhang et al.] 2020) covers both humans and objects with diverse query types, providing both visual and linguistic diversity.
Preprocessing Pipeline Robustness Analysis. The reliability of the object-centric visual prompt- ing pipeline is critical for stable reinforcement learning. We analyze potential failure modes in the object-centric prompting pipeline that could disrupt consistent instance identity construction. Global detection failures, defined as cases where the target object is not detected in any frame of the video, occur in fewer than 1 1% of all samples, indicating that the majority of target objects can be detected and assigned instance IDs. To address local missing detections caused by occlusion or fast motion, the pipeline incorporates periodic re-detection and SAM2's bidirectional propagation to recover full object trajectories. Identity consistency is further reinforced via majority voting during ID as- sigmment, while lightweight ID-repair step at evaluation time resolves occasional re-identification inconsistencies, as detailed in Appendix A..5 Together, these mechanisms mitigate detection and tracking noise, yielding stable instance identities for downstream optimization.
3.3 ENHANCING VLMs WITH REINFORCEMENT LEARNING
Since dense per-frame prediction is reformulated as compact instance-level identification task, re- inforcement learning can be directly applied optimize the policy model with task-specific rewards. These identifiers further enable the model to produce more precise and interpretable reasoning chains during RL training, leading more coherent spatial-temporal predictions.
Reward modeling. Building on DeepSeek-RI (Guo et al. 2025). the reward design in STVG-RI integrates both accuracy and format components. The accuracy reward measures the correctness of predictions, while the format reward enforces structural compliance with a predefined reasoning template. To capture both temporal and spatial accuracy, the accuracy reward is further decomposed into a temporal loU reward and spatial consistency reward. 'Tee temporal IoU reward quantifies the overlap between the predicted interval [ta, te] and the ground-truth segment [t,, f.). defined as:
[La, Le] 14, 4
(0)
[t. t +)
where An B and U B denote the intersection and union of intervals A and B, respectively.
The spatial consistency reward verifies whether the predicted object ID is correct and appears within the localized temporal segment:
11 if 1" and appears in (ts,tel. 0. otherwise,
ro())
where a and r denote the predicted and ground-truth object ID. This design is consistent with the vIoU metric in STVG, defined as -P Eter, ToU(h, b)), where P. and P,, the intersection
5

---

Published as a conference paper at ICLR 2026
and union of the predicted and ground-trut truth temporal segments, and by and bj are the predicted and ground-truth bounding boxes at frame t. Since vloU jointly evaluates temporal and spatial accuracy, constraining the predicted ID to fall within the localized segment prevents trivial solutions and mitigates overfitting to dataset-specifie temporal patterns, improving optimization stability.
Beyond accuracy, the format reward (0) enforces compliance with the predefined reasoning struc- ture, encouraging the model to explicitly generate its reasoning process. A value of is assigned only if the response encloses the reasoning within <t <think>, </think>>> and the final prediction within <answer>.. < annwwr>>. Reasoning traces with timestamps and instance IDs provide clearer references and more precise grounding.
The overall reward is the sum of the three components:
R(o) ro(o) + +r(l))+r((0
(6)
3.4 TRAINING STRATEGIES
The training objective is to optimize the policy model Ty with GRPO (Guo al., 2025). which has shown strong effectiveness in tasks with well-defined evaluation signals. Given a video-texi query pair (V.g) sampled from the training distribution D, the model generates candidate responses o = f on). each assigned a reward R(o,) as defined in Equation In our experiments, n is set to 8. To normalize rewards within a group, the advantage A, of each response is computed as:
A, R(o;) mean (R(o,))}=1)
std((R(o) ))-=1)
The policy update objective encourages the current policy To to assign higher probabilities re- sponses with larger normalized advantages, relative to the previous policy Tew. Formally, the opti- mization objective is defined as:
(O) = ((),))
To(0:14) TTee (0;1g)
A., clip mo(oil4) Te (o:1g)
(8) e, -e) - 3DLL (T
min
where is the clipping parameter, 3 controls the strength of KL regularization, and Tree denotes the frozen reference policy. The clipping term prevents excessively large updates, while the KL penalty constrains policy drift, together stabilizing optimization.
4 EXPERIMENTS
4.1 SETTING
Implementation details. The object detector used during training and evaluation is YOLOv x (Tian et al. 2025) with a confidence threshold of 0.25, and --lrge is the segmentation and tracking model. Re-detection is performed every 15 frames, where detection is treated instance only if its IoU with all tracked objects is below 0,4 and its overlap ratio with all tracked objects below 0.6. We employ the Dwen2.5-VL-13
new
(Bai et al. 2025) as the pre-trained model. We use AdamW (Loshchilov & Hutter 2017) optimizer with a linear learning rate scheduler. The learning rate 1.0e 6 and the batch size per device. The model is trained for epoch on our object-centric visual prompting dataset, and all experiments are conducted on 8 x 100 GPUs.
Benchmarks. For STVG, HCSTVG-v1 and HCSTVG-v2 (Tang al. 202T are widely used for human-centric grounding, while ST-Align (Li et al. 2025) extends evaluation to both humans and objects and supports spatial video grounding (S (S).). To capture fine- fine-grained spatial understanding, MeViS (Ding et al.) 2023) evaluates mask-level grounding under complex multi-object scenarios. Beyond STVG and SVG, we also consider video temporal grounding (VTG) with Charades-STA (Go et al. 2017) and TVGBench (Wang al. 2025b) to evaluate generalization.
Evaluation metrics. For STVG, following Yang et al. 2022 Gu et al. 2024). we report m_tloU for temporal localization accuracy and m.vloU, vloU @ R for joint spatial temporal grounding quality. For VTG, we adopt m tloU and R. For mask-level referring video object segmentation, J is used assess region similarity and F measure contour accuracy.

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison with state-of-the-art models HCSTVG-v1 HCSTVG-v2 val set (%). The results of GroundingGPT-7B are reported from SpaceVLLM, while those of VL3-SV Qwen2.5-VL-7B, Qwen2.5-VL-72B and Qwen3-VL-8B are generated by our experiments. The best and second-best results are shown in bold and underlined.
and
HCSTVG-v1
HCSTVG-v2
Models
m_tloU
vvvooU
vloU@0..
v.oU@0.5
m.tloU
v.v
vloU@0.3
vloU@0.5
TubeDETR
32.4
49.8
23.5
53.9
36.4
58.8
30.6
STVGFormer
62.2
58.1
38.7
33.8
CG-STVG
$2.8
38.4
61.5
36.3
60.0
39.5
64.5
36,3
TA-STVG
39.1
63.1
36.8
60.4
40.2
36.7
GroundingGPT-78
22.2
16.7
15.0
4,9
19.6
14.7
16.6
3.1
SpaceV1..M-77
56.9
39.3
66.6
36.9
58.0
34.0
56.9
24.7
InternVL3333
22.6
11.7
15.3
2.8
24.9
12.8
14,2
3.2
-Viuualrommp
22.9
f+0.3
13.8 142.1
14.9
4.2
1414
140.1
15.4
142.6
143.9
4.5 1.33
Qwen2,5-VL-7B
40.3
19.7
28.2
7,9
45.1
19.3
26.0
8.2
1-166
T+5.1
1+12.5
145.5
1.04
140.
142.7
10.9 142.7
Dwee2.55VVL-77.
40.7
23,9
37,0
15.1
43,9
23.4
36.1
13.4
+VisualPrompp
1-1.9
T22.1
43.0
1+6.0
140.1
42.0
27.3
143.9
147.4
16.3
142.9
Qwen3-VV VL-88
48.6
19.5
28.2
10.6
51.2
17.2
22.8
7.6
+VisualPrompp
48.7 1+0.1
33.0
1+13.5
56.5
1+28.3
1+16.7
1+1.0
35.8
1+18.6
57.9
f+35.1
30.3 1422.7
STVG-RI
56.9
39.1
66,7
38.6
61.3
40.8
67.9
38.8
Table 2: Performance comparison with state-of-the-art models on ST-Align benchmark (%). The
results of Qwen2.5-VL-7B are generated by our experiments.
Spatial-Temporal Video Grounding
Video Sputial Grounding
Models
tloU@0.5
m_tloU
vloU@0.5
m_vloU
vloU@0.3
vloU@0.5
m_vloU
GroundingGPT-7B
7.1
12.2
2.9
9.2
19.7
5.4
17.9
VTimeILMM Grounded- Mide
7.1
15.5
30.0
33.0
LLava-ST-7B
44.6
43.8
21.1
22.8
47.2
30,9
32.5
Qwen2.5-V1-77
35.2
37.4
17.1
14.3
44.6
39.5
+VisualPrompr
36.0
f+0.8
38.3
7+0.9
21.5
1+4.4
19.5
1+5.2
57.7
1+13.1
51.4
1+11.
46.6
1+11.1
STVG-RI
43.6
45.1
25,9
23.4
60.3
53,9
48.6
4.2
EVALUATION RESULTS ON SPATIAL TEMPORAL VIDEO GROUNDING
Table and Table present results on HCSTVG-vI/V2 and ST-Align. TubeDETR (Yang et al. 2022). STVGFormer (Lin et al. 2023). CG-STVG (Gu et al. 2024 and TA-STVG (Gu et al. 2025 are four VLP-based specialized models. Intern InternVL3-88R
Qwen2.5-VL-7BB
and Qwen3 VL-8B
first perform temporal grounding to predict the frame range, and then apply spatial grounding on frames within the intersection of predicted 4, and ground-truth bū for evaluation.
Zero-shot. The object-centric visual prompting paradigm outperforms the two-stage evaluation across VL Qwen2.5-VL--B. Qwen2.5-VL-7228 and Voven--VL-8B, achieving m_vloU scores of 15.4%. 19.5%, 27.3%, and 35.8% HCSTVG-v2, respectively. The improvement can be attributed to the ability our paradigm to leverage information from the entire video sequence when generating spatial predictions. However, temporal performance slightly declines for Qwen2.5- VL models due to occlusion of fine grained details and distributional shifts introduced by visual prompts. Notably, Qwen3-VL--BB exhibits significant performance boost under visual prompt- ing paradigm, as the baseline model often fails to detect any target when only static image inputs are provided for dynamically described scenes.
Fine-tuning. Reinforcement learning yields substantial gains in both temporal and spatial per- formance, establishing new state-of-the-art results on HCSTVG-vI, HCSTVG-v2, and ST-Align. On HCSTVG-v2, compared with the strongest SFT-trained VLM model Space VLLM, STVG-R1 achieves absolute improvements of 4.0%, 6.2%, 10.9% and 4.1% across four evaluation metrics. As shown in Table 2 STYG-RI also surpasses the strongest ST-Align model LLaVA-ST by +0.6% on m vloU. These spatial gains highlight the effectiveness of our object-centric visual prompting paradigm in enforcing consistent object-level predictions, while reinforcement learning further en- hances reasoning ability, leading to more coherent spatial-temporal video grounding.
4.3 EVALUATION RESULTS ON VIDEO SPATIAL GROUNDING

---

Published as a conference paper at ICLR 2026
Since vloU in STVG is inevitably affected by temporal prediction quality, we further evaluate video spatial grounding to isolate spatial capability. As shown Table 2 the proposed object-centric visual prompt- ing paradigm achieves a notable zero-shot gain of 11.1% on m_vIoU on ST-Align video spatial grounding. And after RL, STVG-R1 surpasses the second-best model LLaVA-ST by 13.1 % on m_ vVoU,, More im- portantly, Table 3 reports the results on the multi-object referring video object segmen- tation task. STVG-RI sets a new state-of- the-art of 47.3% on J&FF on MeViS, de- spite being trained only on single-object STVG data. This demonstrates the strong generalization ability of visual prompting paradigm, where the simplified identifier-based formulation facili- tates transfer to more complex multi-object scenarios.
Table 3: Performance comparison with state-of-the- art models on MeViS (%). The results of TrackGPT are generated by VISA. Models
J&F
URVOS (Seo et al. 2020
25.7
29.9
27.8
MTTR (Bttch el a. 2022
28.8 31.2
30.0
ReferFormer LMPM (Ding all 2025)
2022
29.8
32.2
31.0
34.2
40.2
37.2
LISA (Lai et al 2024
35.1
39.4
37.2
TrackGPT (Stron 2024
37.6
42.6
40.1
VISA Yan ar 2024)
40.7
46.3
43.5
VidcoGlaMMM (Munasinghe et al. 2025
42.1
48.2
45.2
STVG-RI
44.7
50.0 47.3
4.4 EVALUATION RESULTS ON VIDEO TEMPORAL GROUNDING
We further evaluate STVG-RI on out-of-distribttion video temporal grounding benchmarks. As shown in Table 4 STVG-RI achieves the best zero-shot performance on Charades-STA, surpassing the second-best model LLaVA-ST by +7.7% t0.5@0.5. Although slightly below the task specific Time-R1, STVG-RI achieves competitive results on TVGBench, highlighting strong generalization.
Table 4: Performance comparison with state-of-the-ar models Charades-STT and TVGBench (%). The results marked with represent models training on corresponding dataset, while others indicate zero-shot settings.
Charades- STA
TVGBench
Models
IloU@0.3
tloU 1.5
1.o
tlU@@0.5
TimeSuite Zeng et al. 2024)
69.9
48.7
31.1
18.0
TRACE Guo er al
40.3
37.0
25.5
LLaVA ST LL al 2025
63.1
44.8
Time-R1 (Wang et al. 2025h)
78.1
60.8"
41.8
29.4
STVG-R1
73.2
52.5
42.5
27.4
4.5 ABLATION
Ablation on visual prompt design. Fol- lowing prior work (Cai et al. 2024 Sht-
Table 5: Ablation study on visual prompt designs on HCSTVG-v1 with zero-shot Qwen2.5- U-
edritski et al. 2023). red-colored visual prompts consistently yield superior perfor- mance in identifier recognition. As shown in Table 5 varying the font size leads to only moderate performance differences.
Letters denotes uppercase letters, L-Letters denotes lowercase letters, and Mix refers to a combination of numbers and uppercase letters.
Size Type
|m.toOU m_vloU vloU@0,
vloU@0.5
Number
38.1
24.6
12.0
Smaller prompts slightly improve temporal grounding. likely due to reduced visual oc-
20
Number
38.0
24.9
40.6
12.2
30
Number
37.5
24.1
38.9
11.8
clusion. Overall, performance remains rel-
40
Number
37.4
23.2
36.4
11.6
atively stable across sizes, and a medium size provides balanced trade-off between visibility and visual interference. Regard- ing prompt types, both letters and numbers
U-Letters
39.0
24.4
38.0
12.6
L-Letters
38.7
24.0
37.4
12.1
Mix
38.7
15.7
20.0
5.7
achieve comparable performance. Letters yield slightly stronger temporal grounding, possibly be- cause they are single-character tokens, whereas multi-digit numbers occupy more space. In contrast, numbers provide marginally better spatial accuracy, likely due to their more consistent visual struc- ture. A mixed design mapping larger numbers letters does not provide additional benefits, sug- gesting that simple and consistent prompts are sufficient. Based on these observations, red-colored numeric prompts with font size 20 are adopted as the default configuration for all experiments.
8

---

Published as a conference paper at ICLR 2026
Ablation on mask filtering thresholds. To reduce visual clutter from dense prompts, we apply mask filtering that removes small instances whose size falls below a fraction of the maximum mask within each category per frame. As shown in Table [6] higher thresholds 0 degrade data quality, with m m_vloU dropping to 65.1% 1/2 on HCSTVG-vI. In contrast, zero-shot evaluation with Qwen2.5- VL-7B shows that moderate filtering improves spa- tial grounding while maintaining temporal accu- racy, threshold of 1/3 achieves the best trade-off between data quality and zero-shot performance, indicating that many small-scale objects are not semantically salient and may even introduce noise.
Table 6: Experimental results of mask filter- ing thresholds on HCSTVG-v1. Values before '/' denote the upper bound, and those after ' are zero-shot results with Qwen2.5-VL-7B
0
m_iloU m_vIoU vloU@0.3 vloU@0.5
100.0/37.55 69.1/23.3 97.2/37.4
89.9/11.9
1/4
100.0/38.0
68.3/24,6 95.8/40.5
88.5/11.6
1/3
100.0/38.0 67.7/24.9
94.6/40.6
87.6/12.1
1/2
100.0/38.1 65.1/24.1 89.9/38.2
83.5/11.4
Ablation on our modules. We further conduct ablation studies explore the individual contri- butions of object-centric visual prompting paradigm and reinforcement learning. Without visual prompts, GRPO follows the zero-shot two-stage evaluation and is optimized only with the temporal reward. As shown in Table and Table g visual prompts primarily enhance spatial localization, while GRPO substantially improves temporal accuracy. VisualPrompt-SSF improves all metrics but slightly reduces temporal grounding on ST-Align, where reasoning critical. The combination of visual prompt and GRPO yields the most consistent gains, achieving state-of-the-art performance.
Table 7: Ablation study with different modules on HCSTVG-v1 and HCSTVG-v2.
HCSTVG-v1
HCSTVG-v2
Models
|__tloU
m_vloU v..
v.o
tlo m_vloU vloU@0.3
vloU @0.5
Qwen2.5-VL-7B
40.3
19.7
28.2
7.9
45.1
19.3
26.0
8.2
+ +VisualPromp
38.7
24.8
40.7
13.4
44.7
19.2
28.7
10.9
+GRPO
57.5
24.7
37.8
17.6
61.4
24.2
37.7
15.4
VisualPrumpt-SFT
50.9
34.3
60.2
28.0
54.4
36.5
60.8
31.3
Visuaa GRPO
56.9
39.1
66.7
38.6
62.0
40.2
67.8
38.8
Table Ablation study with different modules on ST-Align.
Spatial-Temporal Video Grounding
Spatial Video Grounding
Models
tlaU@0.5
m toU
vl.
m. vloU
vl.
vloU@0.5
m. vloU
Owen2.5-VL-TH
35.2
37.4
17.1
14.3
44.6
39.5
35.5
+ +VisualPrompt
36.0
38.3
21.5
19.5
57.7
51.4
46.6
+GRPO
41.8
43.7
17.3
20.0
+ VisuulPromppt-S/
34.6
36.6
21.5
19.3
58.6
52.3
47.2
+ WisualPrompt-GRPP
43.6
45.1
25.9
23.4
60.3
53.9
48.6
Ablation on reward design. In STVG-RI, the temporal reward directly derived from the eval- uation metric, while the spatial reward adopts sparse 0/1 formulation that provides credit only when the predicted instance ID matches the ground truth. To examine the role the spatial compo- nent, tested two alternatives: (1) coupled spatio-temporal reward inspired by vloU, defined as R(o) = (o()) + (o), re(o) + (o), and (2) a continuous spatial reward computed as the average per-frame IoU over the temporal intersection, т, T.. ΣieT, (oU B₁). Neither variant im- proves performance. On HCSTVG-v1, the coupled reward reduces vloU from 39.1% to 38.3% while the continuous spatial reward yields 38.6%. This suggests that the sparse spatial reward better aligns with the objective of selecting a single correct instance, avoiding additional noise.
4.6 IMPACT VISUAL PROMPT OCCLUSION
potential concern is that overlaying numeric identifiers may introduce visual pollution, particularly for OCR-related tasks. To assess this effect, we evaluate Qwen2.5-VL-77B with and without visual prompts on the MME-Viden VideVid Shi et al. (2025) benchmark, which covers ten OCR-centric tasks. As shown in Table 9 performance differences remain consistently small across all tasks. Notably, tasks that require fine-gra -re-grained character recognition, such as TR, show a small performance drop from 69.6% to 69.3%. In contrast, higher-level tasks such as VTQA and TG improve from 76.4% to 77.4% and from 63.2% to 65.3%. suggesting that the added markers can help guide attention toward relevant regions. These results indicate that the visual prompts introduce negligible interference.

---

Published as a conference paper at ICLR 2026
Table 9: Evaluation results MMEE-ideeOOC "TR' denotes Text Recognition, 'VTQA' Visual
Text QA, 'TG' Text Grounding, 'AR' Attribute Recognition, 'CDT' Change Detection & Tracking, 'STP' Special Text Parsing. "CFTU" Cross-Frame Text Understanding, 'TBR' Text-Based Reason- ing. "TBVU" Text-Based Video Understanding, and 'RVT" Robust Video Testing.
Method
TR
VTQA
TG
AR
CDT
STP
CFTU
TBR
TBVU
RVT
Total
Qwen2.5-VL-7B
69.6
76.4
63.2
69.8
45.7
64.4
22.7
54.7
38.2
79.0
59.4
VisualPrompt
69.3
77.4
65.3
68.5
46.2
64.1
22.7
54.8
38.2
78.8
58.9
4.7 VISUALIZATION
In Figure present case of peaataaltemporal video grounding with our object- prompting paradigm. The model first identifies the object IDs relevant to the query and then determines the temporal boundaries. During the reasoning process, complex descriptive appearance expressions are compactly mapped into object IDs, facilitating precise instance-level reasoning and grounding.
The curly man a light gray suit walks towards man in the red suit and stretches his left hand to rest the left shoulder of the man.
Query:
5.00
11.00
Time
Question: Each object the video marked with red number at its center.
To accurately pinpoint the event "[Query]" the video:
1.Determine the precise time period the event occurs. 2.Identify which object ID is corresponding to the described event. Output thought process within the <think </think>, including both a temporal analysis and a spatial analysis. Finally, provide the start and end times, and object ID in the format "Target ID: [ID). Time range: [start time to end time]" within the <answer> </answer> tags.
Reasoning: <think> To determine the event "The curly man in light gray suit walks towards the in the red suit stretches
hand to rest the left shoulder the man," need to identify the objects involved:
The curly man in a light gray suit: Object ID 2. - The man in the red suit: Object ID
The event happens when Object ID 2 approaches Object ID and extends their left hand to touch the left shoulder of Object ID 3. need analyze the sequence of events in the video clips.
1. The curly man in light gray suit (Object ID 2) is initially seen standing still. 2. The man in the red (Object ID 1) is also standing still, 3. The curly man light gray suit begins walk towards the man in the red suit. 4. As he approaches, the curly man in light gray suit extends his left hand and places on the left shoulder of the man in the red suit.
Based this analysis, The event starts around 5.00 seconds when the curly man in light gray suit begins to walk towards the man the red suit. The event ends around 11.00 seconds when curly man in light gray suit places his hand on the left shoulder of the man in the red suit. </think>
<answer> Target ID: 2, Time range: 5.00 11.00 </answer>
Figure 4: Case study of STVG-RI on the spatial- video grounding task.
5 CONCLUSION
This work addresses challenges of coordinates visual-textual misalignment and instance prediction inconsistency across videos. We propose an object-centric visual prompting paradigm that reformu- lates per-frame coordinate prediction into compact and interpretable instance-level identification problem. We further introduce STVG-RI, a reinforcement learning framework optimized with task- driven rewards. Experiments across six benchmarks demonstrate the effectiveness of compact visual prompts and reinforcement learning in enhancing reasoning consistency and generalization. Future work will extend our detector-based framework from natural images broader visual domains.
10

---

Published as a conference paper at ICLR 2026
6 REPRODUCIBILITY STATEMENT
We make every effort to ensure the reproducibility of our work. Detailed descriptions the model architecture, training pipeline, training datasets, and reward design for STVG-R1 provided in Section 3. Implementation details are reported in Section 4.1. The design of visual prompts and filtering thresholds is described in Section 4.5, and the prompts used for training and evaluation across different tasks are presented in Section A.I.
REFERENCES
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, al. Phi-4 techni- cal report. arXiv preprint arXiv:2412.08905, 2024.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-v1 technical report. arXiv preprint wrXiv:2502.13923 2025.
Adam Botach, Evgenii Zheltonozhskii, and Chaim Baskin. End-to-end referring video object seg- mentation with multimodal transformers. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition, pp. 4985-4995, 2022.
Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee. Vip-llava: Making large multimodal models understand arbitrary visual prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
12914-12923, 2024.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp- 6299 6308, 2017.
Liang Chen, Lei Li, Haozhe Zhao, Yifan Song, and Vinci. R1-v: Reinforcing super generalization ability in vision-language models with less than $3. hattss///githubb.cc//eeee//
R1 V 2025a. Accessed: 2025-02-02.
Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408. 10188, 2024.
Yukang Chen, Wei Huang, Baifeng Shi, Qinghao Hu, Hanrong Ye, Ligeng Zhu, Zhijian Liu, Pavlo Molchanov, Jan Kautz, Xiaojuan Qi, et al. Scaling to long videos. arXiv preprint arXiv:2507.07966 2025b.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers). pp. 4171-4186, 2019.
Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy. Mevis: large-scale benchmark for video segmentation with motion expressions. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 2694-2703 2023.
Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory-augmented multimodal agent for video understanding, In European Conference on Com- puter Vision, pp. 75-92. Springer, 2024.
Yue Fan, Xiaojian Ma, Rongpeng Su, Jun Guo, Rujie Wu, Xi Chen, and Qing Li. Embodied videoa- gent: Persistent memory from egocentric videos and embodied sensors enables dynamic scene understanding. In International Conference on Computer Vision, 2025.
Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang. Tianshuo Peng. Junfei Wu, Xiaoying Zhang, Benyou Wang. and Xiangyu Yue. Video-rl: Reinforcing video reasoning in milms, arXiv preprint arXiv:2003.2177,, 2025.

---

Published as a conference paper at ICLR 2026
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalu- ation benchmark of multi-modal Ilms in video analysis. In Proceedings of the Computer Viston and Puttern Recognition Conference, pp. 24108-241 18, 2025.
Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In Proceedings of the IEEE international conference on computer vision, pp- 5267-5275, 2017.
Solomon Golomb. Run-length encodings (corresp.). IEEE transactions on information theory, 12 (3):999400, 1966.
Kristen Grauman, Andrew Westhury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18995-19012, 2022.
Xin Gu, Heng Fan, Yan Huang, Tiejian Luo, and Libo Zhang. Context-guided spatio-temporal video grounding. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18330-18339, 2024.
Xin Gu, Yaojie Shen, Chenxi Luo, Tiejian Luo, Yan Huang, Yuewei Lin, Heng Fan, and Libo Zhang. Knowing your target: Target-aware transformer makes better spatio-temporal video grounding. arXiv preprint 1XXv:25502.1166 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12944 2025.
Yongxin Guo, Jingyu Liu, Mingda Li, Qingbin Liu, Xi Chen, and Xiaoying Tang. Trace: Temporal grounding video IIm via causal event modeling. arXiv preprint arXiv:2410.05643, 2024.
Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.167 2024.
Zeeshan Khan, CV Jawahar, and Makarand Tapaswi. Grounded video situation recognition. Ad- vances in Neural Information Processing Systems, 35:8199-82111 2022.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Rea- soning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. -lavvo-onviision Easy visual task transfer. arXiv preprint SKKV:2408.033222 2024.
Hongyu Li, Jinyu Chen, Ziyu Wei, Shaofei Huang, Tianrui Hui, Jialin Gao, Xiaoming Wei, and Si Liu. Llava-st: A multimodal large language model for fine-grained ppatial-tempont under- standing. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 8592- 8603, 2025.
Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 26689-26699 2024.
Zihang Lin, Chaolei Tan, Jian-Fang Hu, Zhi Jin, Tiancai Ye, and Wei-Shi Zheng. Stvgformer: Spatio-temper temporal video grounding with static-dyname cross-modal understanding. In Proceedings of the 4th on Person in Context Workshop, pp. 1-5, 2022.
Zihang Lin, Chaolei Tan, Hian-Fang Hu, Zhi Jin, Tiancai Ye, and Wei-Shi Zheng. Collaborative static and dynamic vision- language streams for spatio- spatio-temporal video grounding. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern Recognition, pp. 2023.
12

---

Published as a conference paper at ICLR 2026
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 26296-26338 2024a.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang. Hang Su, al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European conference on computer vision, pp. 38-55. Springer, 2024b.
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning, arXiv preprint arXiv:2503.01788 2025.
Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:1711.0500,, 2017.
Shehan Munasinghe, Hanan Gani, Wenqi Zhu, Jiale Cao, Eric Xing, Fahad Shahbaz Khan, and Salman Khan. Videoglamm: A large multimodal model for pixel-level visual grounding in videos. In Proceedings of the Computer Vision and Puttern Recognition Conference, pp. 19036-19046, 2025.
Zhangyang Qi, Zhixiong Zhang, Ye Fang, Jiaqi Wang, and Hengshuang Zhao Gpt4scene: Un- derstand 3d scenes from videos with vision-language models. arXiv preprint arXiv:2501.01428, 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 7748-8765 PmLR, 2021.
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Va- sudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Fe- ichtenhofer. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714. 2024. URL
/tt////bb/2200.07700 00714
Seonguk Seo, Joon- Joon-Young Lee, and Bohyung Han. Urvos: Unified referring video object segmen- tation network with a large-scale benchmark. In European conference on compuler vision, pp- 208-223. Springer, 2020.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepscekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03:: 2000203300000,,, 2024.
Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, al. Vlm-rl: A stable and generalizable rl-style large vision-language model. arXiv preprint arXiv:2504.0777 07615, 2025.
Yang Shi, Huanqian Wang, Wulin Xie, Huanyao Zhang, Lijie Zhao, Yi-Fan Zhang, Xinfeng Li, Chaoyou Fu, Zhuoer Wen, Wenting Liu, et al. Mme-videoocr: Evaluating ocr-based capabilities of multimodal Ilms in video scenarios. arXiv preprint arXiv:2505.21333 2025.
Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about red circle? visual prompt engineering for vlms. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11987-11997, 2023.
Nicholas Stroh. Trackgpt-a generative pre-trained transformer for cross-domain entity trajectory forecasting, arXiv preprint arXiv:2402.00066, 2024.
Ye Sun, Hao Zhang, Henghui Ding, Tiehua Zhang, Xingjun Ma, and Yu- Yu-Gang Jiang. Sama: To- wards multi-turn referential grounded video chat with large language models. arXiv preprint arXiv:2505. 18822 2025.
13

---

Published as a conference paper at ICLR 2026
Zongheng Tang, Yue Liao, Si Liu, Guanbin Li, Xiaojie Jin, Hongxu Jiang, Qian Yu, and Dong Xu. Human-centric spatio-tenne temboral video grounding with visual transformers. IEEE Transactions on Circuits and Systems for Video Technology, 32(12))8238-8244 8248,,8249444 2021.
Yunjie Tian, Qixiang Ye, and David Doermann. Yolov12: ineetioo-centtie real-time object detec-
tors. arXiv preprint arXiv:2502.12524, 2025.
Jiankang Wang, Zhihan Zhang, Zhihang Liu, Yang Li, Jiannan Ge, Hongtao Xie, and Yongdong Zhang. Spacevllm: Endowing multimodal large language model with spatio-temporal video grounding capability, arXiv preprint arXiv:2503.1398, 2025a.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing vision- laniuae model's s perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024a.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, al. Cogvim: Visual expert for pretrained language models. Advances in Neural Information Processing Systems, 37:121475- 1224999 2024b.
Ye Wang, Ziheng Wang, Boshen Xu, Yang Du Kejun Lin, Zihan Xiao, Zihao Yue, Jianzhong Ju, Liang Zhang, Dingyi Yang, et al. Time-rl: Post-training large vision language model for temporal video grounding. arXiv preprint arXiv:2503. /3377, 2025b.
Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, et al. Internvide02: Scaling foundation models for multimodal video understanding. In European Conference on Computer Vision, pp. 396-416. Springer, 2024c.
Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. Language queries for referring video object segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4974 4984, 2022.
Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing unified representation for variety of vision tasks, arXiv preprint arXiv:2311.0022 0.6244,, 2023.
Cilin Yan, Haochen Wang, Shilin Yan, Xiaolong Jiang. Yao Hu, Guoliang Kang, Weidi Xie, and Efstratios Gavves. Visa: Reasoning video object segmentation via large language models. In European Conference on Computer Vision, pp. 98 115. Springer, 2024.
Antoine Yang. Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Tubedetr: Spatio- temporal video grounding with transformers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16442-16453, 2022.
Jianwei Yang. Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of- mark prompting unleashes extraordinary visual grounding in gpt-4v, 2023. URL https://arxiv. arg/abs/3310.11441, 2024.
Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 10632-10643, 2025.
Hanrong Ye, De-An Huang. Yao Lu, Zhiding Yu, Wei Ping, Andrew Tao, Jan Kautz, Song Han, Dan Xu, Pavlo Molchanov, et al. X-vila: Cross-modality alignment for large language model. arXiv preprint 2005 19335, 2024.
Haobo Yuan, Xiangtai Li, Tao Zhang, Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi, Jiashi Feng, and Mling-Hsun Yang. Sa2va: Marrying sam2 with llava for dense grounded under- standing of images and videos. arXiv preprint arXivv2501.044 040014 2025.
Xiangyu Zeng. Kunchang Li. Chenting Wang, Xinhao Li. Tianxiang Jiang. Ziang Yan, Songze Li, Yansong Shi, Zhengrong Yue, Yi Wang, et al. Timesuite: Improving milms for long video understanding via grounded tuning. arXiv preprint arXiv:2410. 19702, 2024.
14

---

Published as a conference paper at ICLR 2026
Jingyi Zhang, Jiaxing Huang, Huanjin Yao. Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503. 2025a.
Xintong Zhang, Zhi Gao, Bofei Zhang, Pengxiang Li, Xiaowen Zhang, Yang Liu, Tao Yuan, Yuwei Wu, Yunde Jia, Song-Chun Zhu, et al. Chain-of-focus: Adaptive visual search and zooming for multimodal reasoning via rl. arXiv preprint arXiv:2505. 15436 2025b.
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024.
Zhu Zhang. Zhou Zhao, Yang Zhao, Qi Wang, Huasheng Liu, and Lianli Gao. Where does it exist: Spatio-temporal video grounding for multi-form sentences. In Proceedings of the 1EEE/CVF Conference Computer Vision and Pattern Recognition, pp- 10668-10680 10688, 2020.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. /0479, 2025.
15

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 PROMPT FOR TRAINING AND EVALUATION
The prompt for spatial-tmmporal video grounding shown in Figure
Each object in the vidco is marked with a red number at its center.
To accurately pinpoint the event "[Query]" in the video: I.Detemmnne the precise time period of the event occurs.
2.Idemtff which object ID is corresponding to the described event.
Output thought process within the <think> </think>, including both temporal analysis and spatial analysis. Finally, provide the start and end times, and object the format "Target ID: [ID], Time range: [start time to end time]" within the <answer> </answer> tags.
Figure 5: Prompt for spatial-temporal video grounding.
The prompt for video spatial grounding and referring video object segmentation is shown in Figure
Each object in the video is marked with a red number at its center.
To accurately pinpoint the event "[Query]" in the video: 1. I.Identffy which object ID is corresponding the described event.
Output thought process within the <think> </think>, including spatial analysis. Finally, provide the object ID in the format "Target ID: [ID]" within the <answer> </answer> tags.
Figure 6: Prompt for video spatial grounding and referring video object segmentation.
The prompt for video temporal grounding is shown in Figure
Each object in the vidco is marked with red number its center.
To accurately pinpoint the event "[Query]" in the video: 1. D.Detrmine the precise time period of the event occurs.
Output thought process within the <think </think>, including temporal analysis. Finally, provide the start and end times in the format "Time range: [start time end time]" within the <answer> </answer> tags.
Figure 7: Prompt for video temporal grounding.
A.2 VISUALIZATIONS OF DIFFERENT TASKS
We provide more cases of ppatial-temporal video grounding and referring video object segmentation tasks, as presented in Figure S and Figure Although some visual prompts are filtered out certain frames, the corresponding instances remain stored in the mask database, thereby enhancing the final spatial localization capability.
16

---

Published as a conference paper at ICLR 2026
10.00
16.99
Time
Query: The blue man walks few steps forward as he speaks, and then sits down.
Reasoning: <think> To determine the event "The blue man walks few steps forward as he speaks, and then sits down," need identify the blue first. The blue is object IDD 4. He wears blue robe and a black hat. In the video, the blue man starts by standing still and then begins to walk forward while speaking. After walking few steps, he stops and sits down the ground. This sequence happens between 10.00 and 16.99 seconds. </think>
<answer> Target ID: Time range: 10.00 16.99 </answer>
(b)
4,00
8.00
Time
Query: The woman the gray dress turns and takes a step forward, looking back at the woman wearing white headscarf, and then takes another step back, looking the woman wearing white headscarf.
<think> To determine the event "The woman in the gray dress turns and takes a step forward, looking back al the woman wearing white headscarf, and then takes another step back, looking the woman wearing white headscarf." need analyze the sequence actions:
Reasoning:
1. The woman in the gray dress (ID 2) initially faces the woman in the white headscarí (ID
2. She turns her body slightly towards the camera.
3. She takes step forward while still facing the woman in the white headscarf.
She looks back at the woman
white headscarf.
5. She takes another step back while still facing the woman in the white headscarf.
Based on this sequence, the event occurs between the moments when she turns her hody and takes the first step forward until she takes the second step back.
Temporal Analysis:
The event starts around 4,00 seconds the woman the gray dress turns body.
The event ends around 8.00 seconds when she takes the second step back.
Spatial Analysis:
The woman in the gray dress (object ID is the target object for the event. </think>
<answer> Target ID: 2, Time range: 4,00 to 8.00 </answer>
Figure 8: Cases for spatial-temporal video grounding.
17

---

Published as a conference paper at ICLR 2026
Query: The birds inside the cage, continuously hopping around.
Reasoning: <think> To determine the event 'The birds inside the cage, continuously hopping around.' need to observe the behavior of the birds in the video. The birds appear to be moving around the cage, which suggests they hopping. The bird labeled 'I' seems be more active, moving around the cage more frequently than the bird labeled as '2. The bird labeled as '2' appears to be less active and mostly stationary. </think>
<answer> Target ID: 1, </answer>
(b)
Query: A cluster of wandering domesticated birds.
Reasoning: <think> The event 'A cluster of wandering domesticated birds.' seems to occur throughout the video as the geese move around enclosure. However, geese appear to be more active and moving together in the middle part of the video, specifically from around 0.00 to 10,00
seconds. </think>
<answer>
Target ID: 3,4, 3,4,5 </answer>
Query:
The white pigeon that hasn't moved from its position.
Reasoning: <think> To determine the event 'The white pigeon that hasn't moved from its position," 'I need to identify the white pigeon in the cage and observe if it remains stationary throughout the video. The white pigeon located in the middle the cage, and it appears be the second pigeon from the left. seems to be the only one that hasn't moved significantly compared to the others. </think>
<answer> Target ID: 2 </answer>
(d)
Query: bigger one of two planes moving left landing.
Reasoning: <think> The event 'bigger one of the two planes moving left and landing' can be observed from the moment the plane starts descending towards the runway until it touches down. The plane labeled as 'I' is the bigger one aronn the two planes. It is positioned on the right side of the frame and moves leftward lands, </think>
<answer> Target ID: </answer>
Figure 9: Cases for referring video object segmentation.
18

---

Published as a conference paper at ICLR 2026
A.3 MORE EXPERIMENTS
Since video temporal grounding task does not inherently require object-centric visual prompts, their addition may even obscure fine-grained video details and slightly hinder temporal accuracy in the zero-shot setting. To further verify this, we compare STVG-RI with and without visual prompts on Charades-STA, and TVGBench. As shown in Table [0] visual prompts bring a marginal decrease in performance, indicating that visual prompts are not essential for temporal-only task.
Table 10: Comparison of STVG-RI with and without visual prompts on temporal grounding bench- marks Charades-STA and TVGBench (%). Adding visual prompts slightly affects temporal perfor- mance, showing that object-centric prompts are less critical for temporal-only tasks.
Charades-STA
TVGBench
Models
tloU@0.3
tl0
tloU@0.3
tloU@0.5
STVG-RI visualprompt
72.2
52.1
41.8
27.2
STVG-R1 w/o. visialprompt
73.2
52.5
42.5
27.4
A.4 VISUALIZATIONS OF UNSEEN CATEGORY IN OBJECT DETECTOR
Figure 10 presents an example where the queried object (fish) is not included in the detector's tax- onomy and is thus misclassified into incorrect category (bird). Nevertheless, our framework assigns a consistent and correctly localizes the target instance. This demonstrates that category misclassification does affect the effectiveness of approach.
Object detection and tracking:
Object-eentric visual prompt video:
Query: Red fish with the most pronounced up-and-down motion
Reasoning: <hink To identify the red fish with the most pronounced up-and-dowe motion, need observe the movement of each fish in the video. The red fish appears to be the one labeled as "5" on the right side of the tank. It seems to have more noticeable up-and-dow motion compared to the other fish. </think>
<answer>> Target ID: </answer>
Final segmentation results:
Figure 10: Visualizations of unseen category in object detector.
19

---

Published as a conference paper at ICLR 2026
A.5 ID-REPAIR MECHANISM DURING EVALUATION
To improve evaluation robustness, we introduce an ID-repair mechanism that corrects missing or inconsistent instance IDs inside the model-predicted temporal segment. Although the detector and SAM2 tracker are generally reliable, occasional ID fragmentation or missed detections can occur, leading to incomplete ID sequences. The ID-repair mechanism algorithm provided below.
Algorithm ID- ID-Repair mechanism during evaluation
Require:
Predicted temporal segment [tx, fe]; detected boxes {Br} and IDs (TD,) for all frames t; target instance ID ID'.
1: Initialize ID-correction set A
2: Let F (-,3....,,
3: for each frame t E F do
4: if ID= e IDe then
5: continue
6: end if
7: Apply ID correction using A
8:
for each old_id E A do
9:
if old_id ID, then
Replace old with ID in ID; 11: end if
10:
12:
end for
if TD" ID, then
14:
continue end if
16:
Find nearest frame fre E F where ID* E DDua
if her exists then
18:
Let beer be the box of ID" in Biat Compute IoU between bief and each b Br Find Inea with the highest IoU to b Area(bret O Ubea))
19:
20:
21:
(b brea) 2 0.4 or min(Arca(Der), (bbea)
2 0.6 then
22:
old_id ID of bnest
23:
Replace old id with ID* in ID; Add old_id to (persist this correction for later frames)
24:
25:
end if
end if
27: if ID
& ID: then
Assign ID* to the box in B, with largest area
29:
end if
30:
end for
A.6 ABLATION OF VISUAL. PROMPTING PIPELINE COMPONENTS
Table 11 reports ablations the major components of the preprocessing pipeline. Periodic detection is essential, as removing it significantly degrades vloU by failing to capture objects that appear after the first frame. In contrast, omitting backward tracking leads smaller performance drop, indicating that forward tracking alone can recover most trajectories. Thus, backward tracking can be omitted when stricter runtime efficiency is required.
Table 11: Ablation study of preprocessing components on HCSTVG-v1 (%).
Method
m n.tloU
vloU
vloU@0.3
v.oU@0.5
w/o re-detection
56.1
27.8
45.4
17.4
w/o backward tracking
56.8
28.4
66.2
37.1
Full pipeline
56.9
39.1
66.7
38.6
20

---

Published as a conference paper at ICLR 2026
A.7 VISUALIZATIONS FOR ADDITIONAL DOWNSTREAM TASKS
To further demonstrate the applicability of object-centric visual prompting beyond STVG, we pro- vide qualitative examples for downstream tasks: video question answering and multi-person video captioning.
Video Question Answering. Figure 11 shows cases where the queried entity lacks distinctive appearance cues, making it difficult for general VLM to localize the correct subject. Without visual prompts, the model fails to answer the question. With object-centric prompts (e.g.. "Person I"), the model correctly grounds the target individual and produces the correct response.
(a) Input the video.
Query: What is person black doing?
Qwen2.5-VL-7BB: The person in black appears be walking of building, followed by another individual dressed similar attire. They both seem to engaged in a conversation or interaction as they exit the building. The setting suggests historical or period drama context, given their clothing and architecture the building.
Input the video with visual prompt.
Query: What is person doing?
Qwen2.5-VL-7B: The person with object ID appears to be standing outside building, dressed in historical attire. They seem observing or waiting for someone, as they have one hand on their hip and the other holding white-gloved hand. The posture suggests moment of pause or contemplation.
(b) Input the raw video.
Query: What is the person in blue dress doing?
Qwen2.5-VL--BB The person in blue appears to be engaged in conversation with another individual, possibly sharing moment of laughter discussion. The setting suggests social gathering, and the person's body language indicates they are actively participating in the interaction. The background shows other individuals dressed in period attire, suggesting this scene might be from historical drama or reenactment.
Input the video with visual prompt.
Query: What is the person 6 doing?
Qwen2.5-VLL 5-VL-B: The person object ID 6 appears to be holding plate and seems serving food or drinks from table that is set up outdoors. The setting suggests social gathering a picnic, where people are enjoying refreshments.
Figure 11: visualization of video question answering with and without visual prompting.
21

---

Published as a conference paper at ICLR 2026
Multi-person Video Captioning. Figure 12 illustrates how visual prompts benefit video caption- ing. Without prompts, the model tends to generate a single global caption with ambiguous refer- ences. With instance IDs, the model produces entity-specific descriptions for each individual.
Input the video.
Query: Describe each person in this video.
@wen2.5-VI. 5-L-7B:: The video depicts group of individuals dressed military uniforms, likely from historical or war-themed setting. The uniforms dark-colored with distinct insignia and caps, suggesting they might be part of specific military unit or organization. One individual seen wearing lighter-colored uniform, possibly indicating a different rank role within the group. The scene transitions show a horse-drawn cart moving through a rural landscape. The cart is being pulled by two horses, and there are people around some standing and others walking alongside. environment appears to a countryside with open fields hills in the background, under clear sky. The overall atmosphere suggests historical or period piece, possibly depicting military civilian scene from a past era.
Input video with visual prompt.
Query: Describe cach person in this video.
Qwen2.5-VL-7B: Here is deseription of cach person the video:
1. **#bbect ID 3, 4**; This person is seated in the back of vehiele, wearing military uniform and a cap. They appear to be looking forward.
2. **Object ID 2**; This person standing next vehicle, wearing military uniform and cap. He appears to looking forward.
3. **Object ID 5**; This person is standing next horse-drawn cart, wearing dark outfit and
appears to be holding onto the cart.
4. **Object ID 6**; This person is standing next horse-drawn cart, wearing dark outfit and
appears to be holding onto the cart.
5. ** **Obeeet ID 7 This person standing top of horse-drawn cart, wearing dark outfit. He
jumped down from the horse-drawn cart.
The video shows these individuals in various positions within and around horse-drawn cart, with some them wearing military uniforms and caps. The setting appears to be outdoors, possibly in rural mountainous area.
Figure 12: A visualization of multi-person video captioning enhanced by visual prompting.
22

---

Published as a conference paper at ICLR 2026
A.8 ADDITIONAL QUALITATIVE RESULTS ON DIVERSE VIDEO DOMAINS
To further assess the generalization ability of our approach, we conduct qualitative evaluations on videos drawn from two distinct domains: ego-centric videos from Ego4D Grauman al. (2022) and movie videos from Grounded-VIDSSine Khan et al. (2022). As shown in Figures 13] and 14 our method maintains stable instance grounding performance despite challenges such as rapid camera motion, complex scene composition, and multiple interacting entities.
(a)
2,00
19.99 Time
Query: A person kicks soccer ball.
<think> The vidco shows person practicing soccer drills an indoor field. The person is seen kicking soccer ball towards a goalpost. The ball is visible in the air during the kick and lands near the goalpost. The ball is marked with the number </think>
Reasoning:
sanswer> Target ID: 3, Time range: 2.00 to 19.99 </answer>
10.00
53.96 Time
Query: A person plays piano.
The event "A piano plays piano." occurs throughout the entire video. The piano is consistently visible in the foreground, and it is the main focus of the video. The piano's keys are being played by the person the video, indicating that piano is actively being used. </think>
Reasoning: <think>
<answer> Target ID: Time range: 10.00 53.96 </answer>
Figure 13: Qualitative results on ego-centric videos from Ego4D Grauman et al. 2022
23

---

Published as a conference paper at ICLR 2026
4.00
7.67
Time
Query: man stand up angrily and throw the book out the window.
Reasoning: <think> The event man stand up angrily throw book out window." occurs when the man in the striped sweater stands and throws the book out of the window, This action happens after he has been speaking gesturing, as indicated by his body language and facial expressions, </think>
<answer> Target ID: 9, Time range: 4.00 to 7.67 </answer>
(
0.00
5.00
Time
Query: bald man is beaten.
Reasoning: <think> The event "A bald man is beaten." " occurs when the bald man is engaged physical altercation with another individual. The bald man is identified by his white tank top and orange shorts. The sequence of frames shows him being pushed and punched by other individuals, indicating that he is being beaten. </think>
<answer> Target 2, Time range: 0.00 to 5.00 </answer>
Figure 14: Qualitative results movie-style videos from Grounded- Srounded-VIDSitte Khan al. (2022).
24

---

Published as a conference paper at ICLR 2026
A.9 REWARD ABLATION
We analyze the effect of the format reward (ry) and coupled reward variant. All experiments are conducted on HCSTVG-vl.
Effect of removing ry- The training curves (Fig. 15) indicate that removing ry produces nearly identical optimization dynamics. Since Qwen2.5-Ve VL naturally supports <think> and <answer> tokens, the marginal benefit of the simple format constraint is limited.
2.54
Figure 15:
Training curves with and without the format reward ry. The brown curve corresponds to the full reward setting, while the orange curve corresponds to the model trained without ryy
Coupled reward variant. The training curves of coupled spatial reward and decoupled spatial reward are as shown in Fig. 16 which indicate that both formulations exhibit nearly identical con- vergence behavior.
1.5
0.5
Step
Figure 16: Training curves comparing the decoupled brown reward and the coupled red reward.
A.10 THE USE OF LARGE LANGUAGE MODELS (LLMs)
We employed large language models for language polishing to improve the clarity and readability of the manuscript. Specifically, LLMs were used to refine grammar, adjust sentence structure, and enhance overall flow, without altering the technical details and experimental results.
25


Reviews:
[{"review_id": "hnEFGsnmrN", "reviewer": "Reviewer_cMwu", "summary": "This work addresses the misalignment between textual descriptions and visual coordinates , which often induces hallucinations in Vision-Language Models (VLMs) for the Spatial-Temporal Video Grounding (STVG) task. The authors propose an \"object-centric visual prompting paradigm\" : via a pre-processing pipeline of detection , tracking , and ReID , each object is assigned a unique, temporally consistent ID. This reformulates per-frame coordinate prediction into a compact instance-level identification problem. Building on this, the paper introduces STVG-R1 , the first reinforcement learning framework for STVG , which employs a task-driven reward to jointly optimize temporal accuracy, spatial consistency, and structural format regularization . Experiments show the approach achieves SOTA and exhibits strong zero-shot generalization on the unseen multi-object referring video object segmentation task.", "questions": "See the questions above", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "•\tNovelty: Reformulating STVG from dense per-frame coordinate prediction into a \"compact instance-level identification task\", a novel idea that effectively avoids the difficult problem of VLMs handling coordinate prediction.\n\n•\tNovel RL Framework: Proposing STVG-R1, the first reinforcement learning framework for STVG, which employs a task-driven reward to optimize the VLM's reasoning.\n\n•\tState-of-the-art Results: Achieves new SOTA performance on multiple STVG benchmarks.\n\n•\tStrong Generalization: Exhibits SOTA zero-shot performance on the unseen multi-object referring video object segmentation task (MeViS), highlighting the method's robust generalization ability.", "weaknesses": "**Regarding the nature and robustness of the \"visual prompting\" pipeline:**\n\n•\tThe pipeline is essentially a complex, training-free data pre-processing pipeline reliant on external SOTA models such as YOLO, SAM2, and ReID, rather than a novel model component.\n\n•\tThe robustness of this pipeline is not discussed. For example, what happens when detection, tracking, or ReID fail? Many critical details of the pipeline, such as the arbitration logic between components, are missing, which hinders reproducibility.\n\n•\tThe visual prompts themselves introduce 'visual pollution' and occlude critical information. Embedding ID characters into video frames is a lossy operation that can obscure key details of an object (e.g., facial expressions, specific markings), thereby hindering the model's understanding. Experiments also show the method is sensitive to hyperparameters such as font size, which further confirms the risk of introducing visual interference.\n\n•\tThe paper does not quantify the additional computational and storage overhead introduced by this complex pipeline. How much (per-video) computational and memory overhead does running YOLO/SAM2/ReID add before using the VLM? This is crucial for assessing the method's practical usability.\n\n**Lack of ablation studies:**\n\n•\tThe paper provides no ablation study for the three components （r_t, r_s r_f） of the RL reward function R(o), making it impossible to determine the key factors driving the performance improvement.\n\n•\tLack of ablation on the necessity of each component in the pre-processing pipeline.\n\n**Contribution Positioning:**\n\n•\tThe paper's RL algorithm (GRPO) is heavily borrowed from DeepSeek-R1, which is more of an \"application-level innovation\" rather than an \"algorithmic innovation\", and this should be clearly stated in the manuscript.\n\n**Clarity and Justification of Key Mathematical Formulations:**\n\n•\tRegarding the spatial reward function r_s(o) in Equation (5): This reward function is designed as a sparse, binary (0 or 1) signal, which prevents the model from receiving \"partially correct\" feedback (e.g., for predicting a spatially adjacent but incorrect ID). The authors should justify why this sparse reward was chosen over a smoother, continuous reward that could reflect the spatial proximity of the predicted ID to the ground truth, and discuss the considerations for RL training stability and efficiency.\n\n•\tRegarding the majority voting rule in Equation (3): This formula determines the global target ID A via majority voting, which is a strong heuristic assumption. This assumption may fail for queries describing transient events (e.g., \"a person who flashes by\"), potentially leading to incorrect training labels. The paper should discuss the limitations of this design and its potential impact on performance.\n\n•\tRegarding the total reward function R(o) in Equation (6): The paper combines temporal and spatial rewards via simple addition, mathematically treating them as independent optimization objectives. However, the evaluation metric for spatio-temporal grounding (vIoU) is inherently coupled. The authors need to explain why a decoupled reward was chosen for training and whether there is a potential misalignment between this design and the final evaluation goal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W4hV7npGJT", "reviewer": "Reviewer_bVvE", "summary": "Brief Summary: The paper tackles the task of video spatio-temporal localization where given a video and a corresponding query, the model should output the bounding boxes + temporal timestamps. The authors propose visual prompting using a combination of yolov12 detector (for object detection) + sam2 (for visual tracking) and overlaying the information on the image itself and providing particular object ids which are also overlayed. Given the visual-prompted video, a Qwen2.5-VL model is trained via GRPO to predict the target id (which has associated bounding box) + time-range. Experiments on spatio-temporal datasets like HCSTVGv1, v2, and temporal grounding like charades-sta show that proposed STVG-R1 outperforms competitive baselines.", "questions": "Q1. In general, it is advisable to do an initial cold-start before GRPO, but I don't see any references on that in the paper. Did the authors try it and didn't give good results?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Pros:\n\n1. The paper poses a nice application of combining spatio-temporal understanding with VLMs. Spatio-temporal understanding is an important sub-topic in video understanding and how to best leverage VLMs for this task is a well motivated problem. \n\n2. The proposed method is conceptually simple in re-using existing detection and tracking pipelines to utilize VLMs inherent understanding of vision without additional tokens or requiring VLM to do additional bounding box predictions. \n\n3. Authors provide visualization (in appendix A.2) and ablation on various prompt designs. In particular, Table 8 is very interesting, that pure SFT without GRPO training leads to worse results.", "weaknesses": "Cons:\n\n1. It seems the absolute improvement over previous baselines is marginal? For instance on hcstvg-v1, performance matches with space-vllm and on v2, it is slightly improved over TA-STVG. On ST-Align, it is same as LLava-ST-7B.\n\n2. The core novelty is slightly limited, the paper suggests doing visual-prompting + grpo training works. This is good to know, but unclear what are the main challenges here. \n\n3. One issue with the visual prompting (assuming the visualization at face value), it is unclear how the approach would tackle things with text (OCR). If you overlay a color bounding box, the text is completely lost. So the model cannot answer questions like \"when did the person look at <some_text>\". \n\n4. The model is essentially restricted to detection quality and classes of yolov12 and the quality of sam2. As such, there is no direct way to leverage VLMs internal association capability. Further dependence on separate models would lead to worse inference times requiring heavy video encoding/decoding. \n\n5. Table 9 in Appendix A.3 seems to suggest direct visual prompting is in fact worse? That seems like a major drawback? \n\n6. (Minor) It would be interesting to see results on more diverse videos, such as some ego-centric datasets (such as ego4d) or movie datasets (such as grounded-vidsitu [Ref1]).\n\n7. The main comparison to baselines is somewhat unfair. The proposed model is able to leverage external models for tracking while baseline models need to do the prediction on their own? I could be missing something obvious here. \n\n8. (Minor) I am slightly confused on the r_s reward, why is it not simply 3d-iou? \n\n9. Authors should show additional downstream tasks which gain from such visual prompting.\n\n---\n\n[Ref1]: Khan, Zeeshan, C. V. Jawahar, and Makarand Tapaswi. \"Grounded video situation recognition.\" Advances in Neural Information Processing Systems 35 (2022): 8199-8210.\n\n---\n\nOverall Rating: 4/10\nThe paper proposes grpo RL training with appropriate visual prompting for spatio-temporal video grounding. The scope however is somewhat narrow and not shown to be applicable for other video understanding tasks, and seems to slightly degrade temporal grounding. The proposed visual prompting itself might interfere with ocr reasoning, and comparison to baselines is not strictly fair.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wur9l1JhTX", "reviewer": "Reviewer_enhd", "summary": "This paper proposes a new visual prompt paradigm without alignment coordinates, which transforms frame by frame coordinate prediction into instance level ID recognition. For the first time, reinforcement learning has been introduced into spatial-temporal video grounding, and task driven reward joint optimization of temporal, spatial, and format constraints has been designed. In multiple benchmark tests, refreshing SOTA resulted in a 20.9% improvement in mIoU for HCSTVG-v2.", "questions": "See [Weaknesses].", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The writing of the paper is clear, and the illustrations in the intro section effectively explain the core contribution points of this paper. In addition, the drawing of Figures 2 and 3 is also quite intuitive.\n\n2. The problem studied in this article (cross modal alignment) is one of the core issues in the field of video grounding, and the alignment effect between the spatial-temporal dimensions directly determines the accuracy of grounding in these two dimensions.\n\n3. The experiment used four commonly used indicators and demonstrated the accuracy of the model on the dataset used.", "weaknesses": "1. The last sentence of the second paragraph of the intro should be a description of the core idea, but this sentence is not clear. What does' a compact and interpretable formulation 'refer to.\n\n2. Insufficient contribution in model design, only introducing pre-segmentation and reinforcement learning for video objects, lacking new method design for this problem.\n\n3. The commonly used datasets in the field of Video Grounding include VidSTG and HC-STVG. This article only uses HC-STVG, and the effect on VidSTG is unknown, especially the results of ablation experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)