Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding
Authors: Zaiquan Yang
Yuhao LIU
Gerhard Petrus Hancke
Rynson W. H. Lau
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Large Language Models
Spatio-Temporal Video Grounding
Zero-shot Learning
Primary area: applications

Abstract: Spatio-temporal video grounding (STVG) aims at localizing the spatio-temporal tube of a video, as specified by the input text query.
In this paper, we utilize multimodal large language models (MLLMs) to explore a zero-shot solution in STVG.
We reveal two key insights about MLLMs: 
(1) MLLMs tend to dynamically assign special tokens, referred to as \textit{grounding tokens}, for grounding the text query; and
(2) MLLMs often suffer from suboptimal grounding due to the inability to fully integrate the cues in the text query (\textit{e.g.}, attributes, actions) for inference. Based on these insights, we propose a MLLM-based zero-shot framework for STVG, which includes novel decomposed spatio-temporal highlighting (DSTH) and temporal-augmented assembling (TAS) strategies to unleash the reasoning ability of MLLMs.
The DSTH strategy first decouples the original query into attribute and action sub-queries for inquiring the existence of the target both spatially and temporally.
It then uses a novel logit-guided re-attention (LRA) module to learn latent variables as spatial and temporal prompts, by regularizing token predictions for each sub-query.
These prompts highlight attribute and action cues, respectively, directing the model's attention to reliable spatial and temporal related visual regions.
In addition, as the spatial grounding by the attribute sub-query should be temporally consistent,
we introduce the TAS strategy to assemble the predictions using the original video frames and the temporal-augmented frames as inputs to help improve temporal consistency.
We evaluate our method on various MLLMs, and show that it outperforms SOTA methods on three common STVG benchmarks.

Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Tempora Video Grounding
Zaiquan Yang
Yuhao Liut
Gerhard Hancke
Rynson W.H. Lauf
Department of Computer Science
City University of Hong Kong
{zaiquyang2-c, yuhliu9-chmy cityu. edu hk {gp hancke, Rynson Lau) edu.
Abstract
video grounding (STVG) aims at localizing the spatio-temporal tube of video, as specified by the input text query. In this paper, we utilize multimodal large language models (MLLMs) to explore a zero-shot solution in STVG. We reveal two key insights about MLLMs: (1) MLLMS tend to dynamically assign special tokens, referred to as grounding tokens, for grounding the text query; and (2) MLLMs often suffer from suboptimal grounding due to the inability to fully integrate the cues in the text query (e.g., attributes, actions) for inference. Based on these insights, we propose a MLLM-based zero-shot framework for STVG, which includes novel decomposed tempool highlighting (DSTH) and temporal- augmented assembling (TAS) strategies to unleash the reasoning ability of MLLMs. The DSTH strategy first decouples the original query into attribute and action sub-queries for inquiring the existence of the target both spatially and temporally. It then uses novel Nogit-guided re-attention (LRA) module to learn latent variables as spatial and temporal prompts, by regularizing token predictions for each sub- query. These prompts highlight attribute and action cues, respectively, directing the model's attention to reliable spatial and temporal related visual regions. In addition, the spatial grounding by the attribute sub-query should be temporally consistent, we introduce the TAS strategy to assemble the predictions using the original video frames and the -aggrenddd frames inputs help improve temporal consistency. We evaluate our method on various MLLMs, and show that it outperforms SOTA methods on three common STVG benchmarks. The code will be available at / com/ wwi.uanyagguaaaaaaaaaaaaaaaaaaa haapg/aaannnnnaaaaaaaaaaa LLaVA_ Baapg//LLLVAA_aaaaaLaaeee BeetrANeet_SSSC
Introduction
Spatio-Tene Temporal Video Grounding (STVG) aims to localize a target object in a video both spatially and temporally, given input text query. This task is fundamental to many different applications (e.g., video surveillance and autonomous driving (67). However, it is also very challenging as requires the model to be able to distinguish the target from distractors over time and identify the precise temporal boundary of the action. While existing methods handle the STVG task mainly in supervied setting [3]
59 54). which often relies costly frame-level annotations,
60 25 attempt to introduce the weakly-supervisee or zero-shot setting alleviate the burden of dense annotations. For example, E3M 3 integrates CLIP (43) and an expectation maximization strategy to optimize spatio-temporal localization zero-shot manner. However, CLIP is known to be weak in localization 75 14 simply aligns the global representation of image-text pairs.
several works 22 3 61
21
Joint corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Bllnnde boy neur plant peciestll.
A small elephant partially hidden tail brush.
(a)
Visual Input
A
Visual Input
The woman in red clothes walks a few steas fonward, then turns around.
Spatial inference error
(b)
The man on the left of the man in the orange shirt sits down then turne his head to the lnft.
Temporal inference
(c)
Figure I: (a) The visual attention maps shows that some special tokens (marked as can precisely attend to the target region of the input query. However, these special tokens, referred to the grounding tokens in our work, underperform on complex STVG, where they often focus on part cues and ignore other cues (marked in red within the input text prompts). Examples (b) and (c) illustrate spatial/temporal grounding errors caused by ignoring the discriminative attribute/action cues. Red and green bounding boxes denote the ground truths and predictions, respectively.
Considering the strong capability of multi-modal large language models (MLLMs)
37 27
g
53 36 in cross-modality alignment, several works explore the application of MLLMS in the visual 44 39 of MLLMs with additional grounding datasets and specific model modifications, which can be difficult to scale and generalize novel visual data [4. Although some recent works 4 [69 have investigated the attention maps language models as done in previous ViT CNN architectures 774 [22 12], they only focus on the generated tokens while neglecting other token components input in the language model (e.g., system tokens and special tokens). In particular, we observe that the special tokens play an important role in structuring the communication between the user input and the language model, and help guide the generation of coherent responses in dialogues.
grounding task. However, they typically require explicit fine-tuning 126
With the above observation in mind, we delve deeper and find that the special tokens following the input instruction have outstanding grounding ability. In particular, few special tokens are characterized by high visual activation and can attend to the region of interest well. Considering the left sample of Fig. [[a as an example, the special token "_A' provides tangible attention to the 'boy' referred to by the given query, while in the right sample of Fig. [a), the token 'IST' is assigned to ground the 'elephant'. These special tokens are referred to as grounding tokens our work. Despite the strong comprehension ability, the grounding tokens cannot optimally adapt STVG they tend to ignore some important cues (e.g., attribute or action) a complex video query. As shown in Fig. Tb), the grounding token fails in the spatial grounding by ignoring the attributes. Similarly, in Fig. c). it leads to the failure of temporal grounding by neglecting the action cues of the target.
In this work, we first conduct systematic analysis to probe the special tokens of various MLLMs (Sec.3.2) which demonstrates our aforementioned empirical findings and enables zero-shot spatio- temporal video grounding. To alleviate the problem of neglecting discriminative cues, propose a novel decomposed spatio-temporal highlighting (DSTH) strategy (Sec. (.3). Specifically, the language query is decomposed into attribute and action sub- sub-queries, which are utilized the text input of the MLLM for inquiring the target's 'ss existence spatially and temporally. respectively. We then propose novel logit-gut logit-guided re-attention (LRA) module to highlight the cues in attribute and action sub-queries. For each sub-query, LRA optimizes the learnable latent variable as the (spatial/temporal) prompts via enhancing the positive response generation while suppressing the negative response. As a result, the DSTH strategy can well adapt the model to mine faithful visual context and concentrate on spatially/eemoorally relevant regions. In addition, to further enhance the temporal consistency during spatial grounding by the attribute sub-query. we develop temporal caaporal-augmentrr assembling (TAS) strategy (Sec. 3.4). The TAS strategy utilizes temporally perturbed frames input assemble different predictions into final spatial grounding result.
In summary, contributions are follows:

---

We reveal that MLLMs dynamically assign the special tokens for precisely grounding text-related regions. We identify the special tokens characterized by the salient visual activation for deriving a novel zero-shot STVG framework.
We propose novel test-time tuning strategy named decomposed spatio-temporal highlighting (DSTH). It introduces an innovative logit-guided re-altention (LRA) module to adapt the grounding token for thorough spatio-temporal localization. We also develop a temporal- -aggmree-egeeeeeed assembling (TAS) strategy to further improve the robustness of spatial localization.
We conduct extensive experiments on various MLLMs to demonstrate empirical findings and validate the effectiveness of the proposed method. Our method outperforms existing methods by a remarkable margin on three STVG benchmarks.
2
Related work
Spatio-Temporal Video Grounding (STVG) aims to localize spatio-temporal tube in video
corresponding to the text query. Unlike image-based visual grounding [10 11
63
34
62
35)
STVG [73 51 59 presents significant challenges, requiring models to distinguish targets from distractors both spatially and temporally. Fully-supervised STVG approaches [51] 54
73 49
have
achieved promising results. However, these methods heavily depend on an extensive collection of labor-intensive annotations. Several recent works [29 21 25 tackle STVG in more efficient leverages pre-trained vision-language models and proposes an expectation maximization framework to optimize spatio- temporal localization. However, contrastive objective based multimodal models are known to be weak in localization [75 [4
manner, which only uses coarse video-level descriptions for training. E3M [3]
they simply align the global representations of image-text pairs.
Multimodal Large Language Models (MLLMs) are capable of handling diverse language and vision tasks. To equip MLLMs with the grounding ability, current works [4] 26
28
666 71
construct grounding-orfe -oriented supervision data for instruction tuning and propose novel architectural
modifications. LLaVA-ST [88 achieves spatio-temporal understanding by introducing progressive training strategy consisting of three sequential stages. However, the large amount of grounding data needed for instruction-tunie toniinnnttttnnggnonn imposes high labeling costs. In addition, changing the focus of MLLMs to grounding tasks can degrade the original dialog capabilities due to catastrophic forgetting (68]. Recent works [4] 19 69) reveal the inherent perception ability of MLLMs obtained by general instruction tounonngtunnn....o Unlike these works that only focus on the generated tokens or in our work, we delve deeper into the more token components and reveal that the MLLMs always dynamically assign the special tokens following the instruction prompt for attending to the regions of interest.
Test-time Tuning (TTT) aims to optimize the inference of test samples online. With the progress of multimodal foundation models [43] 5)), TTT has attracted more attention (76) 40) 18] 48; (4) as it can learn effective prompts for test samples and well adapt the foundation model for zero-shot applications. TPT [48] pioneers the study on TTT by minimizing the prediction entropy between each test sample and its augmented views. HisTPT [70] explores memory learning for test-time prompt tuning by introducing the memory bank. However, the test prompt optimization for MLLMS is barely explored. The most relevant work to our work is ControlMLLM (56), which optimizes the attention map by taking the referring regions as supervision. With the lack of supervision, we propose to learn visual prompts by regularizing the token-level response to instruction input. Our work shows that can rectify where text prompts attend to by altering the outputs of MLLMs.
3 Method
3.1 Preliminaries
Task Formulation. Given an untrimmed video V = (f composed of Ta image frames and a sentence query Q, the goal of the STVG task to localize the spatio-temporal tube of target O = {br Vi+,, described by Q. Here b, represents the bounding box of the target in the f-th frame, t, and l. specify the starting and ending boundaries of the target tubelet, respectively. The STVG task can be solved through two sub-tasks: spatial grounding and temporal grounding. In our work, we first derive a zero-shot solution for the STVG task by unleashing the strong cross-modal comprehension ability of MLLMs.
3

---

The Setup of MLLMs. Current MLLMs typically consist of a visual encoder, a projector, and large language model. Specifically, given an mmage-question pair (I. Q) as imput, the image I first projected into text-aligned visual tokens T, = [V1.................... VM by visual encoder and projector, and the question Q converted into text tokens Tg = [11,.... (N,) by text tokenizer and embedding layer. Here, M and Ng denote the numbers of visual tokens and question tokens, respectively. In practice, MLLMs also introduce system tokens Tays and some special tokens Trole for instruction- following ability. In particular, the special tokens play an important role in structuring the well- organized conversation framework. In our work, the special tokens are positioned subsequent to the instruction prompts by the user. They help guide the generation of coherent responses. The number of special tokens Nrole often differs among different MLLMs. As a result, the language model receives concatenated tokens, T {t....... UN V...... ..... VM: f1..... ,N, (i...... NN as input. Appendix 6. provides visual illustration about the input tokens in MILMs.
Text-fo-visual Attention. The LLM in MILMs typically processes the input tokens through L. transformer blocks (37] with the multi-head attention (MHA) for interactions of different tokens. Particularly, the text-to-visual attention represents the relationships between the visual and the textual tokens. We can derive the text-to-visual attention matrix A E RL where N = Nays + M + Nq + Nrote- L and H denote the numbers of layers and heads in the transformer. Our empirical observation from Fig. iis that the special tokens Trole show outstanding grounding ability with global comprehension. For the simplicity of the following analysis, omit the affect of different layers and heads by the mean operation. We can then obtain the cext-to-visue attention matrix, Arole
[Nay + M + N₂ N, Nays Nays + M].
3.2 Grounding Token Identification
In this section, conduct pilot study to quantitatively analyze the grounding ability of the special tokens. Without loss of generality, we ran-
MLLMS or loage (Ru10003))
KILLM oo Wee (8O STVO)
(a)
domly select a subset of 1, 000 image- text pairs from the RefCOCOg [17] validation set for image MLLMs anal- ysis, and a subset of 1,000 video-
SLaW0-3.3
Qheens
DeepSSonk/1
LLatt
MELMS ee liauge (e00000g)
MILKG on Valeu (HC STVC:
L
text pairs from the HC-STVGv2 (51 dataset for video MILLMs analy- sis. Particularly, we choose three typical MLLMs (i.e., LLaVA-1.5. Qwen-VL, (ee for the study image input and three MLLMS (i.e., LLaVA-Next-Viid Qwen2-VL, VL, LLaVA-OneVisioon for the study on video input. We have two key findings from our studies.
L
(b)
L
LL.00215
QuesXL
DeepSeeb-11
LLVVA-Ne-MM
OwnnA: LLAVN-OaalWae
Figure 2: In (a), the results show the frequency with which different special tokens (such as *_A', 5?" have superior grounding ability than other tokens, i.e., hit ratio. In (b), the results represent the grounding accuracy of tokens ranked by their visual activation degrees. For each MILM,, we select four tokens for visualization.
MILMMs dynamically assign the spe- cial tokens to attend to the text-related regions. To demonstrate the finding, we first define the attention ratio of each special token as the ratio of maximum attention within the ground-truth bounding box byt to that outside it. For example, the attention ratio of the special token '_A" can be computed as:
max(A-A) vole ② fB2M (bgr)) max (A-A @ (1 - fB2M (bgx)))
R-A.
A-A RX
(I)
where fI2M denotes the function that transforms a bounding box into its corresponding binary mask. Here, a higher ratio indicates a better target grounding ability. Given test sample, we can identify the token yielding the highest attention ratio as the superior token for grounding. The hit ratio of a token is then defined as the frequency of being the superior token for grounding across all test samples. Fig. ([aa shows the hit ratio of four special tokens in each MILM. Notably, the fixed token does not consistently exhibit the best grounding ability for different samples. For example, the highest hit ratio achieved by token in LLaVA-1.5 is not more than 50%. In addition, for different MLLMs, the token at fixed position does not always yield the best localization performance. For example, the

---

The rur oF orangg
ofthe diare wiffecsioo
Language Model
Grounding Token Ielentiftration
Speriol Rnforence
Sybo
Qu
Text Query
LLM
Insmartice
Staam
Ingit (p4 (T
V 'yee
Is Ever ron dis down, Wile?
Q
Video Input
Vision Encoder
TILEE
T
Language
Model
Grounding Token
Edletification
Temporal inforence
Figure Overview of the proposed approach for zero-shot STVG. Given video-text pair, we first decompose the text Q into spatially and temporally related sub-queries, Q2 and Q.. The text prompt tokens converted from Q: and Qi are then concatenated with visual tokens Tr for spatial and temporal inferences, respectively. In addition, we introduce learnable variables as visual prompts and optimize them by the logit- re-attention (LRA) module. For spatial grounding, we also develop -angerel-augmented assembling (TAS) strategy by reversing the frames to enhance temporal consistency. After optimization, we obtain the object track score Sobj and frame score Strame based on the grounding token identification. The final prediction is derived by joining Sabj and Strame-
last special token ";,, which is adopted in a previous work [6), obtains a high hit ratio in Qwen-VL. VL, but its hit ratio is quite low in LLaVA-1.5. This observation holds for both image and video inputs.
The special token with higher visual activation tends to show superior grounding performance. Since the superior grounding tokens vary across different samples and MLLMs, identifying them for grounding is problem that needs to be resolved when ground truth is unavailable as prior. With further analysis, we reveal that the superior token for grounding tends to show higher visual activation. For each sample, we rank the special tokens according to the maximum value of visual attention and then evaluate their grounding accuracy by selecting the proposal with the highest attention value. Following the paradigm in previous works (50) 16), we extract box proposals using a detector and evaluate the grounding accuracy by the Acc@0.5 metric. Fig. ([bb shows the results. We can see that the grounding accuracy decreases as the rank of visual activation reduces (from left to right). This supports our hypothesis that the special token with higher visual activation tends to show superior grounding ability. We have also observed that models with better comprehension possess better grounding ability overall. For example, the special tokens in LLaVA-OneViss achieve better grounding performance than those of the other MILMS.
straightforward solution for zero-shot STVG. Inspired by the grounding ability of special tokens subsequent the text prompt, we refer to them as grounding tokens in our work. By identifying the special tokens characterized by high visual activation, we derive a strong training-free framework for STVG. Specifically, given the video-text pair (V, Q), we first extract the object track proposals O pro {O..... Op} from the video V, as done in previous works B 25 where P denotes the T= number of proposals, and Op- {b/ | is the set of bounding boxes of the p-th proposal in T, frames. Based on the foregoing findings, we then select the token with the highest attention value for locating the target object, and denote the text-to-image attention matrix of the selected token as Ag E I M As a result, we obtain each object track score by computing the maximum attention value inside each object track as:
Sobj {s) sg.
s/), with S = max (Ag @ fB2M (Op)).
(2)
where 3 denotes element-wise multiplication. We choose the track with the highest score as spatial prediction Opees In a similar manner, we can compute the frame score Strame = {0f. siy. .... .....t.). By selecting the top-K frames with the highest scores from the temporal prediction Strames we obtain the final spatio-temporal prediction Opeed = (b where f.' and ter denote the starting and ending boundary predictions. Though simple, this solution has achieved comparable or even superior performance than current zero-shot methods. For example, based on LLaVA-OneVisioon model, this solution achieves 23.3% on the m Vl metric on HC-STVGvl dataset, which outperforms previous SOTA result (19.1%) by E3M (3].
5

---

3.3 Decomposed
Highlighting
Despite a strong performance by the above solution, these grounding tokens often neglect some important cues during spatio-temporal localization especially when processing complex video queries. As shown in Fig. [T), the attribute cue 'in red clothes' is overlooked during inference, which causes the incorrect spatial localization. We observe that the language query often contains attribute and action descriptions of the target object, which are beneficial for spatial and temporal localization, respectively. To this end, we propose a novel decomposed spatio-temporal highlighting (DSTH) strategy in our framework, which aims at highlighting the attributes/ actions cues in language query and enhances the spatial/temporal reasoning, respectively.
Generation of target-related cues. For comprehensive spatial and temporal reasoning, it is essential to extract the attribute and action descriptions from the original query Q as textual cues of the target. extract attribute and action descriptions. Following previous works [62 [6], we construct prompt with general instructions and in-context task examples. As shown in Fig. we feed the prompt into L.L,, and then these related descriptions Q, and Qi are generated through the LLM completion. More implementation details can be found in Appendix [7.3
Here, we leverage the strong in-context capability of the LLM
Spatio-temporal prompt learning. With the decomposed attribute and action descriptions as cues for spatial and temporal reasoning, the next question is how efficiently direct the model to focus on the corresponding visual regions. Reliable responses in visual question answering (VQA) necessitate
careful attention to the relevant visual context as pointed out by previous studies [58] 42 Inspired by
this, we innovatively propose regularizing the response to the questions constructed from the attribute and action descriptions for adjusting the visual attention of MLLMs. Specifically, we first transform the descriptions into interrogative queries by fixed template to inquire the existence of the target. As shown Fig. 3 from the original text input Q, we can obtain the attribute description 'a man on the left of the man in the orange shirt', which is further transformed into interrogative sub-query Q5: 'Is there a man on the left of the man the orange shirt in this video?' for spatial inquiry. In a similar way, we can obtain the interrogative sub-query Qu for temporal inquiry. These interrogative sub-queries will be taken as input instructions of MLLMs for response generation.
Logit-guided re-attention, With extracted uub-queries above, we then propose novel eegit-guided re-attention (LRA) module to regularize the token prediction during response generation. Taking the spatial sub-query as an example, we first initialize a learnable variable V, with the same shape as the visual tokens T,,, and then add it to T, as the visual input of the language model. Sub-query Qs is converted as text prompt tokens Tę by the text tokenizer and embedding layer. Given the input token sequence, the next token probability prediction over the vocabulary set is formulated as:
(3) where Ta denotes the parameter of the language model and is frozen in our work. logit,,, is the log probability of the generated token at time step Vęi denotes the text tokens sequence prior to prediction time step i. We define the optimization objective by contrasting the probabilities of positive token 'yes' and negative token 'no':
Py exp (logit,, (w + V., TG. v<<+))
(4) During the inference process, we conduct backpropagation to optimize the learnable variable V, as the spatial prompt. The process is iterated Nep times by test-time tuning paradigm. By enhancing the positive response towards the sub-query Qs. we can prompt the MLLM to effectively mine target-related contextual information during VQA, which in turn highlights the attribute cues in the original text. Similarly, we can obtain the temporal prompt V, by optimizing the temporal inference.
L. = 1 - exp (logit,,, (u, (TT. + V., Ty V<i)) - logit,, (I,"I(T, C.T.+V., V., Tyy
Joint inference. Based on the spatial and temporal visual prompts, we derive the attention maps A and Al E T. xhxw of the special token with high visual activation for spatial and temporal predictions, where h and w denote the token numbers of the height and width. From Eq. 2 we obtain the object track score Sony and the temporal score Strame based on A and A", respectively. Finally,
we integrate the predictions Oped (6)
as the spatio-temporal grounding result.
3.4 Temporal-augmentte Assembling
The attribute sub-query, which provides a static state description, exhibits temporal independence for spatial grounding. In other words, the spatial grounding by the attribute sub-query should be

---

Table 1: Quantitative comparison HCSTVG (v1&v2) and VidSTG (Declarative) benchmarks.
HCSTVG-v1
HCSTVG-v2
VidSTG (Declarative)
Sup
Method
m__ V_ vIoL'@0.3 vIo
m_yloU vIoU@003
vioU@0.5
BLvIoU vloU@00.3 v.o
TubeDETR 39
[CVP99100)
32.4
49.8
23.5
36.4
58.8
30.6
30.4
42.5
28.2
STCAT
20
Neut1P22222)
35.0
57.7
30.0
33.1
46.2
32.6
Full
CSDVL
R3
36.9
62.2
34.8
38.7
65.5
33.8
33.7
47.2
32.8
CG-STVG
15
[CVPX2G0))
38.4
61.5
36.3
39.5
64.5
36.3
34,0
47.7
33.1
WINNER
[CVPRIGG))
14.2
17.2
6.1
14.1
7.4
VEM 21
BOCY2024)
14.6
18.6
5.8
14.5
18.6
8.8
Weak
CoSPal.
25
22.1
31.8
19.6
22.2
31.4
18.9
16.0
20.1
13.1
STPro
R
CVV22202
17.6
27.0
12.9
20.0
31.1
14.6
15.5
19.4
12.7
RedCircle
[UN1503121]
9.2
7.8
1.6
7.6
0.9
ReCLIP
50
(SCU2002)
14.4
18.3
4,9
14.2
17.5
7.9
E3M (63323044
19.1
29.4
10.6
16.2
20.5
11.9
78
Ours LLaV Mikoo7
20.4
33.6
12.4
23.6
36.8
15.5
16.6
26.8
11.1
Ours Staesi
20.0
32.2
10.9
24.4
38.9
15.4
17.1
27.8
11.6
Ours VL-12
23.6
39.0
25.6
40.5
27.4
Ours 2VA Dwn
27.7
temporally consistent for temporal augmentation (e.g., reversing the order of video frames). However, there exists temporal inconsistency when introducing temporal augmentation current MLLMs. Here, propose a metric measure the temporal consistency. By denoting the spatial attention maps before and after reversing the order of input frames A and A the temporal consistency can be measured as:
Scons max ... sp), Sp max
((Ag @ fB2M (0p)) @ (Ag fB2M (0p))), (5)
where higher Scons indicates better temporal consistency. We then divide the testing samples into ten groups descending order of temporal consistency. Fig. 4 shows the average ground- ing accuracy of each group samples. The results demonstrate pronounced association between temporal consistency and spatial grounding per- formance. The temporal inconsistency tends cause worse spatial localization.
Spettal grounding accuracy
8
30
Tenpuuull comsistancy
To this end, we integrate
Figure 4: Spatial grounding accuracy of different groups of samples on the HC-STVGv] dataset. These groups are ranked by descending temporal consistency.
assembling (TAS) strategy in our framework. As shown in Fig. 3 perform frame-level rever- sion operation on the visual tokens and the spatial prompt simultaneously, and then optimize the spa- tial prompts for the input of the original frames and the cemporal-augmenter input, respectively, During inference, derive the spatial prediction by assembling the attention maps of temporal- augmented input frames. The proposed TAS strategy alleviates the effect of temporal inconsistency and improves the robustness of spatial grounding.
Experiments
4.1 Settings
Datasets. We evaluate on three video benchmark datasets: HCSTVG-vI, HCSTVG-v2 [5]] and VidSTG [73]. We provide detailed introduction about them in Appendx 7.1
Implementation Details. We adopt G-DINO 38 and SAM2 45 for detection and tracking
tubelet generation, and use GPT-4o to decompose the original query sentence into spatial and temporal sub-queries. We consider four widely-used video MILMS: LLaVA-Next-Viiee--7 (27). Quen2-VL-7B (53), ShareGPT4Video-B8 (7), LLaVa-OneVisioon
27 for demonstrating the
efficiency of our method. Refer to Appendix 7.2 for more details.

---

Evaluation Metrics. We follow the standard evaluation protocol [59 25 and use m_ vloU, and vioU@R to assess the performance of spatio-temporal grounding. Specifically, let S,, S, denote the intersection and union between the predicted and ground-truth frames. The vIoU is computed by S. Eies, [k, be), where i1 and b, denote the detected and ground-truth bounding box at frame t, respectively. The m_ _loU represents the vIoU averaged over all testing samples, and vloU@R denotes the proportion of data samples in the testing subset with vloU greater than the threshold E {0.3,0.5) 0.5).
4.2 Performance Comparison
Quantitative Comparison. Tab. presents a comparison of our method against 11 meth- ods from three categories, including zero-shot, weakly-supervised, and fully-supervised meth- ods. Specifically, we compare our approach with existing zero-shot SOTA approaches (e.g., E3M (3). ReCLIP [50), RedCircle 477). Our method consistently outperforms these methods by a remarkable margin on all benchmarks. Based on MaVV--eett-Videe
our method outperforms E3M by 4.2% on vIoU@0.3 and 1.8% on vloU@0.5. When integrated into the better
7B model, the corresponding improvements reach 12.1% 1% and 5.7%. This shows that our framework can adapt to various MILMM well, and better performances can be achieved by using better MLLMs. Even on the VidSTG dataset, which contains fewer action cues related to the tar- get and thus makes temporal grounding particularly challeng- ing, our framework still outper- forms the previous SOTA meth- ods overall. This demonstrates the strong generalization capabil- ity of our method. Besides, our method even surpasses current SOTA weakly-supervised meth- ods on most metrics. For exam- ple, on the HCTSVG-v2 bench- mark, our method outperforms CoSPaL 25 by a margin of 5.5% on the m_ vlo metric. Furthermore, compared to the cully-supervised methods, our method can still achieve comparable results, which further validates the superiority of our approach. We provide quantitative comparison on the VidSTG (Interrogative) and image benchmarks in Appendix 8.1
The man in the left of the man the orange shirt sits down, then turns his head to the left.
GT
wo/Tuning
Ours
weronn in yollww cothes looks un woman hold og the newspaper then down at the book
Video
wo/Tuning
Figure 5: Qualitative results the HC-STVGv1 test set. Better spatio-temporal grounding results (green) are obtained when the DSTH strategy is being used for optimization.
Qualitative Comparison. We present qualitative results in Fig. 5 In the example below, before introducing the DSTH optimization strategy, the model neglects the attribute cues in the language query and suffers from the spatial grounding error. By highlighting the attribute cues of the target, our method can direct the MLLMs toward reliable visual context and improve spatial localization.
4.3 Ablation Study
In this section, based on HC-STVGvl dataset, we analyse the effect of different proposed components when integrated into LlaVa-Ness Video and ablation experiments with the syper-parameters based on -llaaaNext--idde
We also conduct extensive
Component analysis. In Tab, 2 we first average the attention maps of all special tokens (Ist row) as the baseline. This solution has achieved outstanding performance. Next, in the 2nd row, we integrate the selection of the superior token introduced in grounding token identification (GTI). It brings different MLLMs. The results show that identifying the superior special
consistent improvements token can effectively unleash the powerful comprehension ability of MILMs. We them validate the efficiency of the decomposed spatio-temporal highlighting strategy (3rd and 4th rows). Sume and T Tume denote adopting the prompt learning in spatial and temporal inferences, respectively. demonstrated that the MLLMs can be directed to focus on the spatial/temporal related regions better

---

Di Pictes
30
Beselune
Vancll-Tusing
Ours
vioU
vioU@0.3
vleUig0.55
In, Invlou
+loUg0..3
vlolig00.5
(a)
(b)
(c)
Figure 6: (a) Comparison on the number of frames N as imput, using v.oU@0.3. (b) Ablation on the number selected frames during temporal prediction. (c) Ablation on different trackers.
when highlighting the attribute/action cues with proper prompts. When utilizing prompt learning the two sub-tasks, the final performance can he further refined (5th row). In addition, we also find that the DSTH strategy improve the grounding performance especially for the less efficient MLLMs (e.g., LLaVA--Net--Video Even for the MILMs (e.g.. LLaVA- Tire-nneVi One Vision with strong temporal comprehension ability, performance improvement still be achieved by the test-time optimization. Finally, we introduce the temporal-augmennted assem- bling (TAS) strategy (6th row). The overall perfor- mance can be further im- proved by refining the spa- tial localization.
Table 2: Component ablation
LLaVA-Next-Video and
Oniiiinneision.
SAVVA-Xext-Videee
LlaVA-OOeVVision
GTI S
T,
TAS
m. vIoU v00 vloU@0.5 m. vloU vloU@0.3 v.0
15.2
36.1
12.6
x
16.3
26.6
23.3
38.8
15.1
Why does spatial and tem- poral prompt learning by
x
18.0
30.1
10.1
24.1
40.3
16.0
x
18.4
29.5
10.1
23.8
39.7
15.5
contrasting logits of 'yes' and 'no' work. Though contrasting logits of 'yes'
19.9
32.1
11.9
24.3
40.7
16.0
v
/
V
20.4
33.6
12.4
24.8
41.5
16.3
and 'no' tokens is heuristic, it is reasonable. We empirically find that given the question prompt, the model often gives ambiguous or even wrong response predictions even though the objects referred to by the text do indeed exist in the video. This indicates that the model cannot understand the video content well in certain cases. By optimizing the logit of the 'yes" token toward positive prediction instead of the 'no' token, we encourage the model to carefully attend to the target-related visual information and achieve better visual localization.
To further validate the hypothesis, we measure the distribution of attention peaks before and after test- time optimization. Based the evaluation the HC-STVG dataset, find that the average attention peak before test-time optimization is about 6.7 (without normalization), and after optimization, increases to 23.9 (without normalization). This change attention values indicates that our proposed optimization strategy can encourage the model to perceive more visual modal information for generating visually-grounde responses, thereby justifying our previous hypothesis.
Comparison of different learning strategies. We propose logit-guided re-attention by contrasting logits of 'yes' and 'no' tokens. Here, also consider entropy minimization (i.e., increasing the pre- dicted probability of the 'yes' token across the entire vocabulary) during test-time tuning. The results obtained by LlaVa-Next-Videe- The Ist row denotes the results of in- troducing grounding token identifica- tion (GTT). The 2nd and 3rd rows denote the test-time tuning results achieved by entropy minimization and contrasting the logits of 'yes' and 'no', respectively. We find that entropy min- imization does bring noticeable improvement. We believe this is be-
on the HC-STVG-v2 dataset shown in the table below.
Table 3: Comparison of different learning strategies.
Methods
m m_yloU vlo vloU @0.5
GTI
19.2
28.5
12.4
GTI entropy minimization
19.5
28.5
12.5
GTI contrasting the logits
22.1
32.9
14.0
cause entropy minimization may be achieved by reducing the token prediction of abundant irrelevant tokens in the vocabulary while ignoring the differentiation between the 'yes' and the 'no" token. As result, less efficient than explicitly contrasting the logits promoting the model to attend to abundant visual information.

---

Generality analysis of our method. We conduct the scalability verification on InternVL3 and Qwen2. 5- series models. With limited computational resources, we evaluate our method on InternVL3-4B. InternVL3-88, Quen2. 5-VL-BB and Qwen2. 5-VL-BB models in the table below.
We average the attention maps of all special tokens as the baseline.
Table 4: Evaluation on different scales of models.
Benefiting from our grounding token identification and test-time optimiza- tion, we can obviously achieve bet- ter grounding performance not only in the smaller but also the larger mod- els, which well demonstrates both the generality and scalability of our method. Moreover, we find that the model size significantly impacts the inherent grounding ability of spe- cial tokens. The performance of small models (e.g., InternVL3-4B) is inferior to the larger models (e.g.,
Methods
vloU@0.3 m_ m_vloU t_ Loouug gt_ viou
InternVI334B Baseline
24.5
17.0
38.4
38.2
InternVL334B3 Ours
41.1
25.4
45.5
49.5
Intern/L3-88 Baseline
36.5
22.9
42.9
46.1
InternVIL3-88 Ours
45.3
27.5
47.2
53.2
Qwen2. 55 + Baseline
28.5
17.3
37.1
39.1
Qwen2. 5--22.5 + Ours
41.7
25.4
45.9
49.3
Twen2.5-VL-- + Baseline
39.7
23.6
44.1
47.2
Qwen2.5-VL--2 + Ours
45.4
27.7
46.9
51.6
InternVL3-8B) is because fewer parameters in language models perform poorly in multimodal alignment.
Effect of input frames N,. Fig. G(a) analyzes the effect of using different numbers of video frames N, as input. We can see that more input frames provide richer visual context for model inference and lead better overall performance. Besides, the performance becomes slightly peaked as the number of input frames increases. To balance between performance and efficiency during inference, we uniformly sample 20 frames the visual input of MLLMs. In Fig. 6 we also evaluate Vanilla-Tuning a solution that directly applies test-time optimization to the entire text query Q instead of decomposing it into attribute/action sub-queries. We observe that the performance of momilla-Tuning is inferior to our method. It demonstrates that decomposing the text query into attributeeaction sub-queries can help facilitate the intensive spatio-temporal comprehension of MLLMs.
Ablation on the number K of predicted frames. Fig. b(b)) analyzes the effect of predicted frame numbers K for temporal grounding based on the HCSTVG-vl dataset. Here, we consider selecting the optimal frame number from the set {3. 5, 7, 9, 11). In general, when K is set as 7, the optimal result can be obtained. Thus, we select top-7 frames as the prediction during temporal grounding.
Trackers ablation. Fig. c(c analyzes the effect of different trackers. Besides SAM2 [45] foundation model for tracking, we also consider two other tracking models (i.e., Byte'Track 772 and BoTSort (2) for analysis. It is reasonable that when a stronger tracking model (e.g., SAM2) is used for generating spatio-temporal tubelets, we can obtain better STVG performances. Besides, when suboptimal tracking models are used, our method still achieve comparable or better performances than the current SOTA methods, which shows the generalization of our approach.
5 Conclusion
In this paper, we have presented novel MLLM-based zero-shot framework for the video grounding (STVG) task. Our approach is initiated by identifying the grounding capability of special tokens in widely used MLLMs during response generation. To leverage and unleash the comprehension ability of MLLMs, we have proposed the decomposed spatio-temporal highlighting (DSTH) strategy. It first decomposes the text query into attributes and actions sub-queries. It then employs a -ggit-guieed re-attention (LRA) module to sharpen the spatial/temporal visual context comprehension. We have also proposed the aumpenel-augmented assembling (TAS) strategy to alleviate the effect of temporal inconsistency. Extensive experiments conducted on three STVG benchmarks demonstrate the effectiveness of our proposed framework.
Our approach does have limitations. For example, it may struggle to process long videos well due to high computational consumption caused by MLLMs. As future work, we would like consider incorporating token pruning and key frame selection techniques into the model design.
10

---

References
[1] Achiam, J., Adler, S., Agarwal, S., Ahmad, I.., Akkaya, I., Aleman, F.I., Almeida, D., Al- tenschmidt, J., Altman, S., Anadkat, S., al:: Gpt-4 technical report. arXiv:2303.08774 (2023)
[2] Aharon, N., Orfaig, R., Bobrovsky, B.Z.: Bot-sort: Robust associations multi-pedestrian tracking. arXiv:2206. 14651 (2022)
[3] Bao, P., Shao, Z., Yang, W., Ng. B.P. Kot, A.C.: E3m: zero-shot spatio-temporal video grounding with expectation maxiiiiiiiommaiiiitatiii multimodal modulation. In: ECCV. pp. 227-243. Springer (2024)
[4] Cao, S., Gui, L.Y., Wang, YX.:: Emerging pixel grounding in large multimodal models without grounding supervision. arXiv:2410.08200 (2024)
[5] Chen, D., Wu, Z., Liu, F,, Yang, Z., Zheng, S., Tan, Y., Zhou, E.: Protoclip: Prototypical contrastive language image pretraining. TNNLS (2023)
[6] Chen, K., Zhang, Z., Zeng, W., Zhang, R., Zhu, F., Zhao, R.: Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv:2306. 15000..51199 15195 (2023)
[7] Chen, I.., Wei, X., Li, J., Dong, X., Zhang. P., Zang, Y., Chen, Z., Duan, H., Lin, B., Tang, Z., al.: Sharegpt4video: Improving video understanding and generation with better captions. In: NeurlPS (2024)
[8] Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing. S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., al.: Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In: CVPR. pp. 24185-24198 (2024)
[9] Darcet, T. Oquab, M., Mairal, J., Bojanowski, P.: Vision transformers need registers. 2XXv:2309.16588 (2023)
[10] Deng, J., Yang, Z., Chen, T., Zhou, W. Li, H.: Transvg: End-to-end visual grounding with transformers. In: ICCV. pp. 1769-1779 (2021)
[11] Ding, H., Liu, C., Wang, S., Jiang, X.: Vision-language transformer and query generation for referring segmentation. In: ICCV. pp. 16321-16330 (2021)
[12] Gao, W., Wan, F., Pan, X., Peng, Z., Tian, Q., Han, Z., Zhou, B., Ye, Q.: Ts-cam: Token semantic coupled attention map for weakly supervised object localization. In: ICCV, pp. 2886-2895 (2021)
[13] Garg, A., Kumar, A., Rawat, Y.S.: Stpro: Spatial and temporal progressive learning for weakly supervised spatio-temporal grounding. arXiv:2502.20673 (2025)
[14] Ghiasi, G., Gu, X., Cui, Y., Lin, T.Y.: Scaling open-vocabulary image segmentation with image-level labels. In: ECCV. pp. 540-557. Springer (2022)
[15) Gu, X., Fan, H., Huang, Y., Luo, T., Zhang, L.: Context-guided spatio-temporal video grounding. In: CVPR. pp. 18330-18339 (2024)
[16] Han, Z., Zhu, F., Lao, Q., Jiang, H.: Zero-shot referring expression comprehension via structural
similarity between images and captions. In: CVPR. pp. 14364-14374 (2024)
[17] Hu, R., Rohrbach, M., Darrell, T.: Segmentation from natural language expressions. In: ECCV. pp. 108-124. Springer (2016)
[18] Jia, M., Tang, L., Chen, B.C., Cardie, C., Belongie, S., Hariharan, B., Lim, S.N.: Visual prompt tuning. In: ECCV. pp. 709-727. Springer (2022)
[19] Jiang, Z., Chen, J., Zhu, B., Luo, T., Shen, Y., Yang, X.: Devils middle layers of large vision- language models: Interpreting, detecting and mitigating object hallucinations via attention lens. arXiv:2411.16724 (2024)
[20] Jin, Y., Li, Y., Yuan, Z., Mu, Y.: Embracing consistency: one-stage approach for spatio- temporal video grounding. In: NeurlPS. pp. 29192-29204 (2022)

---

[21] Jin, Y., Mu, Y: Weakly-supervised spatio-temporal video grounding with variational cross modal alignment. In: ECCV. pp. 412 Springer (2024)
[22] Kamath, A., Singh, M., LeCun, Y., Synnaeve, G., Misra, I., Carion, N.: Mdetr-modulated detection for end-to-end multi-modal understanding. In: CVPR. pp. 1780-1790 (2021)
[23] Kang. S., Kim, J.. Kim, J., Hwang. S.J.: See what you are told: Visual attention sink in large multimodal models. arXiv:2503.08 033221 (2025)
[24] Kang, S., Kim, J., Kim, J., Hwang, S.J.: Your large vision-language model only needs few attention heads for visual grounding DXXiv:2503.0628: (2025)
[25] Kumar, A., Kira, Z., Rawat, Y.S.: Contextual self- elff-pacd learning for weakly supervised spatio- temporal video grounding. arXiv:2501.17053 (2025)
[26] Lai, X., Tian, Z.. Chen, Y., Li, Y.. Yuan, Y., Liu, S., Jia, J.: Lisa: Reasoning segmentation via large language model. In: CVPR. pp. 9579-9589 (2024)
[27] Li, B., Zhang, Y., Guo, D.. Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., al.: Llava-onevision: Easy visual task transfer. arXiv:2408. 23i3:2408.03326 (2024)
[28] Li, H., Chen, Wei. Z., Huang, S., Hui, T., Gao. Wei, X., Liu, S.: Llava-st: multimodal large language model for fine-gras tne-grained spatial-tempores understanding. arXiv:2501 .08282 (2025)
[29] Li, M., Wang, H., Zhang, W., Miao, J., Zhao, Z.,, Zhang, Ji, W. Wu, F. Winner: Weakly- supervised hierarchical decomposition and alignment for spatio-temporal video grounding. In: (2023)
CVPR. pp.
[30] Liang, Y., Cai, Z., Xu, J., Huang, G., Wang. Liang, X., Liu, J., Li, Z., Wang, J., Huang, S.L.: Unleashing region understanding in intermediate layers for filmmbbsse referring expression generation. NeurlPS 37, 120578-120601 (2024)
[31] Lin, B., Ye, Y., Zhu, B., Cui, J., Ning, M., Jin, P., Yuan, L.: Video-llava: Learning united visual representation by alignment before projection. In: EMNLP. pp. 5971-5984 (2024)
[32] Lin, F., Yuan, J., Wu, S., Wang, F., Wang. Z.: Uninext: Exploring unified architecture for vision recognition. In: ACMMM. pp. 3200-3208 (2023)
[33] Lin. Z., Tan, C., Hu, J.F., Jin, Z., Ye. T., Zheng, W.S.: Collaborative static and dynamic
vision-language streams for spatio-temporal video grounding, In: CVPR. pp. 23100-23109 (2023)
[34] Liu, C., Ding, H., Jiang, X.: Gres: Generalized referring expression segmentation. In: CVPR. pp. 23592-23600 (2023)
[35] Liu. F., Liu, Y., Kong, Y.. Xu, K., Zhang, L., Yin. B., Hancke, G., Lau, R.: Referring image segmentation using text supervision. In: ICCV, pp. 22124-22134 (2023)
[36] Liu, F., Liu, Y., Xu, K., Ye, S., Hancke, G.P., Lau, R.W.: Language-guide salient object ranking. In: CVPR. pp. 29803-220 29813 (2025)
[37] Liu, H., C., Wu, Q., Lee, Y.J.: Visual instruction tuning. In: NeurIPS (2023)
[38] Liu, S., Zeng. Z., Ren, T., Li, F., Zhang, H., Yang. J., Jiang, Q., Li, C., Yang. J., Su, H., et al.: Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In: ECCV. pp. 38-55. Springer (2024)
[39] Ma, C., Jiang. Y., Wu, J., Yuan, Z. Qi, X.: Groma: Localized visual tokenization for grounding multimodal large language models. In: ECCV. pp. 417-435. Springer (2024)
[40] Ma, J., Li, H., Xiang, X.: Decoupled entropy minimization. In: NeurIPS (2025)
[41] Munasinghe, S., Thushara, R., Maaz, M., Rasheed, H.A., Khan, S., Shah, M., Khan, E: Pg-video-llava: Pixel grounding large video-language models. arXiv:2311.13433 (2023)
12

---

[42] Prasad, A., Stengel-Eskin, E., Bansal, M.: Rephrase, augment, reason: Visual grounding of questions for vision-language models. arXiv:2310.05861 (2023)
[43] Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell,
A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: ICML. pp. 8748-8763, PmLR (2021)
[44] Rasheed, H., Maaz, M., Shaji, S., Shaker, A., Khan, S., Cholakkal, H., Anwer, R.M., Xing, E., Yang. M.H., Khan, F.S.: Glamm: Pixel grounding large multimodal model. In: CVPR. pp. 13009-13018 (2024)
[45] Ravi, N., Gabeur, V., Hu, Y.T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafsonn L., et al.: Sam 2: Segment anything in images and videos. arXiv:2408.000 000144 (2024)
[46] Shen, H., Zhao. T., Zhu, M., Yin, J.: Groundvip: Harnessing zero-shot visual grounding from vision-language pre-training and open-vocabulary object detection. In: AAAL vol. 38, pp. 4766-4775 (2024)
[47] Shtedritski, A., Rupprecht, C., Vedaldi, A.: What does clip know about red circle? visual prompt engineering for vlms. In: ICCV. pp. 1 11987-11997 (2023)
[48] Shu, M.. Nie, W., Huang, D.A., Yu, Z., Goldstein, T., Anandkumar, A., Xiao, C.: Test-time prompt tuning for zero-shot generalization in vision-language models. In: NeurIPS (2022)
[49] Su, R., Yu, Q. Xu, D.: Stygbert: A lisuul-lig transformer based framework for spatio- temporal video grounding. In: ICCV. pp. 1533-1542 (2021)
[50] Subramanian, S., Merrill, W., Darrell, T., Gardner, M., Singh, S., Rohrbach, A.: Reclip: A strong zero-shot baseline for referring expression comprehension. In: ACL. pp. 5198-5215 (2022)
[51] Tang, Z., Liao, Y., Liu, S., Li, G., Jin, X., Jiang, H., Yu, Q., Xu, D.: Human-centric spatio- temporal video grounding with visual transformers. TCSVT 32(12), 8238-8249 (2021)
[52] Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jégou, H.: Training data-efficient image transformers & distillation through attention. In: ICML. pp. 10347-10357. PMLR (2021)
[53] Wang, P., Bai, S., Tan, S., Wang, S., Fan. Z., Bai, J., Chen, K., Liu, X., Wang. J., Ge. W., et al.: Qwen2-vl: Enhancing vision-language model's s perception of the world at any resolution. 1211:2 (2024)
[54] Wasim, S.T., Naseer, M., Khan, S., Yang, M.H., Khan, F.S.: Videogrounding-dine Towards open-vocabulary spatio-temporal video grounding. In: CVPR. pp. 18909-18918 (2024)
[55] Woo, S., Kim, D., Jang, J., Choi, Y., Kim, C.: Don't 't miss the forest for the trees: Attentional vision calibration for large vision language models. arXiv:2405. 17820 (2024)
[56] Wu, M., Cai, X., Ji, J., Li, J., Huang, O., Luo, G., Fei, H., JIAGG, G., Sun, X., Ji, R.: Controlmllm: Training-free visual prompt learning for multimodal large language models. In: NeurIPS (2024)
[57] Xiao, G., Tian, Y., Chen, B., Han, S., Lewis, M.: Efficient streaming language models with attention sinks. In: ICLR (2024)
[58] Xiao, J., Yao, A.. Li, Y., Chua. T.S.: Can i trust your answer? visually grounded video question answering. In: CVPR. pp. 13204-13214 (2024)
[59] Yang. A., Miech, A., Sivic, J., Laptev, I. Schmid, C.: Tubedetr: Spatio-temporal video grounding with transformers. In: CVPR. pp. 16442-16453 (2022)
[60] Yang, Z., Ke, Z., Hancke, G.P., Lau, R.W.: Cross-domain semantic decoupling for weakly- supervised semantic segmentation. In: BMVC (2023)
[61] Yang, 7., Liu, Y., Xu, W., Huang, C.,, Zhou, I.,, Tong, C.: Learning prototype via placeholder for zero-shot recognition. In: IJCAL pp. 1559-1565 (2022)
13

---

[62] Yang, Z., Liu, Y., Lin, J., Hancke, G., Lau, R.: Boosting weakly supervised referring image segmentation via progressive comprehension. Advances in Neural Information Processing Systems 37, 90299-2219932 93239 (2024)
[63] Yang, Z., Wang. J., Tang. Y., Chen, K., Zhao, H., Torr, P.H.: Lavt: Language-aware vision transformer for referring image segmentation. In: CVPR. pp. 18155-18165 (2022)
[64] Yoon, H.S., Yoon, E., Tee, J.T.., Hasegawa-Johnson, M.A., Li, Y., Yoo, C.D.: C-tpt: Calibrated test-time prompt tuning for vision-language models via text feature dispersion. In: ICLR (2024)
[65] You, H., Zhang, H., Gan, Z., Du, X., Zhang, B., Wang, Z., Cao, L.,, Chang. S.F., Yang, Y.: Ferret: Refer and ground anything anywhere at any granularity. In: ICLR (2024)
[66] Yuan, Y. Li, W., Liu, J., Tang, D., Luo, X., Qin, C., Zhang, L.,, Zhu, J:: Osprey: Pixel understanding with visual instruction tuning. In: CVPR. pp. 18202-28211 (2024)
[67] Zeng, S., Chang, X., Xie, M., Liu, X., Bai, Y., Pan, Z., Xu, M., Wei, X.: Futuresightdrive: Thinking visually with spatio-temporal cot for autonomous driving. arXiv:2505.17688 (2025)
[68] Zhai, Y., Tong, S., Li, X., Cai, M., Qu, Q., Lee, YJ., Ma, Y.: Investigating the catastrophic forgetting in multimodal large language models. urXiv:2309.13311 (2023)
[69] Zhang, J., Khayatkhoei, M., Chhikara, P., llievski, F: Milms know where to look: Training-free perception of small visual details with multimodal 1lms. arXiv:2502.1422 (2025)
[70] Zhang. J., Huang. J., Zhang, X., Shao, L., Lu, S.: Historical test-time prompt tuning for vision foundation models. In: NeurlPS (2024)
[71] Zhang, T., Li. X., Fei, H., Yuan, H., Wu, S., Ji, S., Loy, C.C., Shuicheng, Y.: Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding. In: NeurlPS
[72] Zhang, Y., Sun, P., Jiang, Y., Yu, D., Weng, F., Yuan, Z., Luo, P., Liu, W., Wang. X.: Bytetrack: Multi-object tracking by associating every detection box. In: ECCV. pp. 1-21. Springer (2022)
[73] Zhang. Z., Zhao, Z., Zhao, Y., Wang, Q., Liu, H., Gao, L.: Where does exist: Spatio-temporal video grounding for multi-form sentences. In: CVPR. pp. 10668-10677 (2020)
[74] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: CVPR. pp. 2921-2929 (2016)
[75] Zhou, C., Loy, C.C., Dai, B.: Extract free dense labels from clip. In: ECCV, pp. 696-712. Springer (2022)
[76] Zhou, K., Yang. J., Loy. C.C., Liu, Z.: Learning to prompt for vasion-language models. IJCV 130(9), 2337-23444 (2022)
[77] Zhu, C., Zhou, Y., Shen, Y., Luo, G., Pan, X., Lin, M., Chen, C., Cao, L., Sun, X., Ji, R.: Seqtr: simple yet universal network for visual grounding. In: ECCV. pp. 598-615. Springer (2022)
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In Sec | we make the claims to reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In Appendix 9 we discuss the limitations of the work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide implementation details in Sec. 4. and the Appendix for repro- ducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [No]
Justification:
We will consider releasing the data and code once the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/dde/eeee Cooeeeoodeesssseeeeee //nips.com htc/://bbb///giidee////eeeueassl ssiniiiiccypy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We provide details about the experimental setting (e.g., Sec. and the Appendix for reproducibility.
in The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
Guidelines:
The answer NA means that the paper does not include experiments.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification:
We do not report error bars due to limited computing resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the information on the computer resources in Appendix 7.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Question:
Answer:
[Yes]
Justification:
The research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The paper discusses societal impacts of the work in the Appendix 10
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper properly cites the existing assets.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
21

---

Appendices of Unleashing the Potential of Multimodal LLMs for Zero-Shot Temporal Video Grounding
Introduction
2
Related work
Method
Preliminaries
3.2
Grounding Token Identification
3.3
Decomposed Spatio- Temporal Highlighting
3.4
Assembling
Experiments
4.1
Settings
4.2
Performance Comparison
Ablation Study
5
Conclusion
10
6
Grounding Tokens
24
6.1
Illustration of Input Tokens Components
24
6.2
Quantitative Analysis of Special Tokens
24
6.3
Qualitative Analysis of Special Tokens
26
7
More Implementation Details
29
Datasets
................................
7.2
Method Implementation
................................
29
7.3
Generation of Target-related Cues
29
8
More Results and Ablation Analysis
31
8.1
Comparison VidSTG (Interrogative) and RefCOCO Benchmarks
31
8.2
Temporal Consistency Analysis
8.3
Effect of Optimization Times Nep and Learning Rate Ir
32
8.4
Inference Efficiency
32
8.5
Qualitative Spatio Temporal Video Grounding Visualization
................................
33
9
imitation and Future work
35
22

---

10 Broader Impacts
35
23

---

Grounding Tokens
6.1 Illustration Input Tokens Components
In Fig. 7 we give clear illustration of input token components in MLLMs. In practice, besides the visual and text prompt tokens ( and @), there are also system tokens and some special tokens (① and ()) equipped for language generation. In particular, the grounding ability of special tokens subsequent to the text instruction prompt is significantly undervalued in previous works. Here we focus on exploring the grounding ability of these special tokens. For different multi-modal large language models, there often are different special tokens due to different instruction-tuni process. For example, in Llava-1.. 5, the introduced special tokens include: 1': 'ANT', 'IST', 'SS', "_"). While for the Qwen2-Vi VL, these special tokens include: C, "assistant', << <<im__ ilm_star|>>> <<imm__ end > Note that we also consider the
one of the special tokens considering that located in the end the instruction prompt and characteristic sink tokens [2] [7].
In fact, special tokens are predefined symbols within language model's vocabulary. They focus on guiding the model's processing instead of representing real words and guide the model generate coherent and context-aware responses. In the scenario explored in our work (i.e., dialogue systems), special tokens can differentiate between a user's question and the assistant's answer. By using special tokens, models become better at understanding structure and context.
Generated Tokens
8 there a blonde boy near plant pedestal this image?
Yes. There as blonde
Language Model (L layers)
assistanc
pedestal
System tokens
tokens
Instruction Prompt tokens
LLaVA-Next-Vider
< imm start|
end|>
Qwen2-VL
<|im_
</im, start]
assistant
LLaVA-OnaVision
Special tokens
Figure 7: Illustration of the tokens input in MLLMs-based dialog system. Here, we take the image
question answering an example.
6.2 Quantitative Analysis of Special Tokens
In our work, we randomly selected subset of 1000 image-text pairs from RefCOCOg vali- dation set for image MILMMs analysis, and a subset of 1000 video-text pairs from HC-STVG [5] dataset for vidco MLLMs analysis. Particularly, we choose six typical MLLMs (i.e., LLaVA-1. 1. 5, Qwen-VL. VL, Deepseek- VL, LLaVA-Ness -eeet-eetiiieeeee Qwen2 VL, LIaVA-Oni OneVision) for pilot studies. In the following, we will introduce our findings about special tokens in MLLMs.
Some special tokens show outstanding grounding ability for text prompt input. We compare the grounding accuracy of G-DNIO (38 and the special tokens' ones by probing the attention map. Here we adopt the G-DNIO with Swin-T backbone and it is not finetuned on the refcoco series datasets. Following previous visual grounding works (50 [6], we adopt the loU@0.5 as the metric of grounding accuracy. The results evaluated on refcoco series benchmarks are shown in the Fig. B The yellow horizontal line denotes the result by G-DINO model. Though the G-DINO trained on image-text pairs in cully-supervieed manner, some special tokens still achieve comparable and even

---

MLLMs
1LaV6-1.3
SeepS
11aNN-155
OwenAV
DeepSoobb
Quen-Vi
DesySook-VT
Figure 8: Comparison of grounding accuracy between G-DINO and special tokens on RefCOCO series datasets.
MLLMs (ReCOCOgg
MLLMs (RetCoCOg)
MLLMs
LL
Owizs-LL
LLaVA-NNee-iia
Qwen2VL
LLaVAACneVaako
I. Neat-Vl
Qwes2
Figure 9: The hit ratio of different special tokens in image and video MLLMs.
MILLMs Image (BeCOOCD)
MILLMs Image (RelDOCDa)
MLLMs Vidao (SC.STVG)
L.
LI
LTL
Guoedday ACC
L
L
L
De
Figure 10: Grounding accuracy of special tokens ranked by the visual activation degree.
better performance. The results show that the special token can effectively ground the text prompt input by integrating the textual cues. Besides, we also notice that there exists pronounced difference for grounding ability of different special tokens.
MILMs dynamically assign the special tokens to attend to the text-related regions. For further exploration on the grounding ability of special tokens, we define the attention ratio of each special token as the ratio of maximum attention within the ground truth bounding box byt that outside Here, a higher ratio indicates better target grounding ability. Given test sample, we can identify the token that yields the highest attention ratio as the superior token for grounding. Then a token's 's hit ratio is defined as the frequency of being the superior token for grounding across all test samples. The Fig. a) shows the hit ratio of special tokens in each MILM. We choose four special tokens for visualization. In addition to the findings that MLLMs dynamically assign the special tokens to attend the text-related regions, we also observe that the superior token for grounding in video MLLMs shows more concentrated trend than image MLLMs. For example, the highest hit ratio by Qwen2-VE VL is more than 60%. Besides, for the same special token in the MILM,, the hit ratio the case of different datasets may be significantly different. For example, the token shows a high hit ratio in RefCOCOg dataset, but its hit ratio is quite low the HC-STVG dataset.
The special token with higher visual activation tends to show superior grounding performance. In our work, with further analysis, we reveal that the superior token for grounding tends show higher visual activation. For each sample, we rank the special tokens according to the maximum value of visual attention and evaluate their grounding accuracy by selecting the proposal with the highest attention value as the prediction. In practice, we extract box proposals by the detector (e.g., grounding accuracy by the Acc@0.5 metric. Fig. shows the results.
G-DINO) and evaluate We can see that the grounding accuracy decreases the rank of visual activation reduces (from left to right). We also observe that models with better comprehension overall possess better grounding ability.
25

---

6.3 Qualitative Analysis of Special Tokens
In Fig. we visualize token-to-visuu attention maps using some image-text pairs by LLaVA-14 5-7B model. In particular, for each image-text pair, we visualize the visual attendance of the semantic tokens from the text prompt and special tokens. The green bounding box denotes the ground truth of the target object. Notably, we notice that the semantic tokens in text prompts can provide tangible attention to related visual entities by the limited cues, while the special tokens are often assigned to integrate global instruction cues and attend to the exact target region. For example, given the text prompt 'a man getting ready to cut cake', the token 'man' and 'cake' both show reasonable visual response. The special token AA' can accurately attend to the target person. In addition, we see that the special token with hight visual activation often shows better grounding performance. However, we also see that the special tokens may attend to the wrong instance even though the semantic tokens can properly capture their region of interest. It suggests that there is still room for improvement in inference in current multimodal language models.
In Fig. T2 we visualize attention maps of special tokens using some video-text pairs by ShareCPT4Vides 14e-ieo-iiiB- model. The ground-truth spatio-temporal tube is denoted by red bound- ing boxes. Each row shows the attention maps of specific token. We find that only a few special tokens can well capture the corresponding target regions, which necessitates the efficient selection of grounding tokens. Also, some cases ge, the below sample), there may be more than one special token that will pay attention to the target area. This shows that achieving localization by considering the integration of multiple effective special tokens can direction for improvement. We leave to future research.
26

---

getting ready to cut cake.
000
ANT
leg with jeans on next woman.
1_
L_
woman
A
55
155
ANT
Blue next to the girl feeding the giraffes.
blc
rost
1ood
gramess
A
55
ANT
Brown adult horse with long tail.
briwn
Sone
ANT
Semantic tokens
Special tokens
Figure 11: Visualization of tokens' attention maps on image-text pairs. The tokens with dotted box denote the semantic tokens in text prompts while the tokens with solid box denote the special tokens following the text prompts.
27

---

The man the hat raises his hand, turns around, and walks to the door.
cc:
<[end header, Jej
|sstart_ header_idj
man
stops, speaks him, turns away.
6C'
<lend_heater_ju headerr Jul+
<lsttrt header >
Figure 12: Visualization of tokens' attention maps on video-text pairs.
28

---

7 More Implementation Details
7.1 Datasets
We evaluate on three video benchmark datasets: HCSTVG-v1, HCSTVG-v2 [51, and VidSTG 73
HCSTVG-v1 has 4500 training and 1160 testing videos with sentence descriptions of human at- tributes/actions, HCSTVG-v2 extends vI to 16,544 videos, including 10, 131 training, 2,000 vali- dation, and 4,413 testing videos. Since the test set is unavailable, we evaluate on the validation set following prior works [59] 25). VidSTG includes 99,943 video-sentence pairs (44,808 declarative, 55,135 interrogative), covering 10,303 videos and 80 object categories. Training, validation, and test sets have 80,684, 8,956, and 10,303 pairs, respectively.
7.2 Method Implementation
In this work, we adopt G-DINO 38 with 0.4 for both phrase and box thresholds to detect object tracker for tubelet proposals generation. Considering the information redundancy, we run the detector every 10 frames for efficiency. We utilize GPT-4o to decompose the original query sentence into spatial and temporal sub-queries. To demonstrate the efficiency of our method, we consider four LLaVA-like video MLLMs: LlaVA-Nex: -e--NNe---Viie--77 (27]. Quen2-VL-7B [53], ShareGPTAVideo-88 8B 77. LlaVa-OneVisionn
proposals, and then utilize SAM2 45
[27]. In practice, with the limited tokens context length and computing efficiency, we sample 20 frames by default as the visual input of MIL.S for each video. When adopting the test-time tuning strategy DSTH, we set the learning rate Ir and iteration times Nef 8.0 and 1 according to the ablation analysis in Tab. 7 We conduct all experiments on an A100 GPU with 80G VRAM based on Pytorch framework. To better capture text-to-visual attention, introduce 'Describe this video in details.' as general query prompt Qcap and implement the relative attention strategy [69] to reduce the effect visual registers (9 (55] In Algo. [ we outline the implementation of the proposed decomposed spatio-temporal highlighting strategy during test-time tuning.
7.3 Generation of Target-related Cues
In this work, we extract the attributes and actions descriptions from the original query Q as textual cues enhance the spatial and temporal comprehension, respectively. To obtain multiple target- related cues, we leverage the strong in-context capability of the Large Language Model (LLM). In particular, we construct the in-context instruction to prompt Ilm for completion. The whole prompt used in this work includestgeneral instruction (in brown), output constraints (in blue), in-context task examples (in green) and input question (in yellow), shown Fig.
The undressed man is held by the woman in red, takes out a knife
Given text description of target object, which may contain description part about the 'stabe/ static attribute' of target object, and description part about the actioonbehavioor of target obj ect, I'd like you to decompose sentences. Three are some points note. the decomposed sentences should complete sentence or phrase. Each output sentence include one state or attribute and the should d given the referred object. If there is no attributes or actions description contents, you should output original sentence. For example, given the referring text: woman in white shirt looking down at laptop computer. Y ou should output: given the sentence The yellow-haie haireed man picks up the bottle and pours drinks to the person directly opposite him', you should output ('the yellow- haarredd man', 'the man picks up the bottle and pours drinks to the person directly opposite him') J. Give your output:
into 'ttribute'' and 'action' twoo sub-
LLM
Text clues:
The undressed man
The is held by the woman in red, takes out knife
Then given the text: {
Figure 13: Flow of LLM-based generation of target-related cues.
In Fig. 14 we present some examples of LLM generation. After decomposing the original text description into attributes and actions-related descriptions, we will further transform the descriptions into interrogative queries by fixed template to inquire about the existence of the target. our work, adopt the template 'Is there in this video?'.
29

---

Original Text: The man in purple clothes takes step forward sits the bench.
Altributes Text : the man in purple clothes.
Actions Text The man takes step forward and sits on the bench.
Vo_phVLLTMMtmKK.mpp
Original vellow-haired man opposite woman's hand kisses then
Attributes yellow-haired man.
Actions Text The man takes the opposite woman's 's hand and kisses it, then puts down.
159 vss_ssggs2222mmkk
Original Text: The white-bearded man points the man on right and then points himself again.
Attributes Text : the white- silteebearred man.
Actions Text The man points at the man on his right and then points at himself again.
111, Buu__sddPmmB-ng.mpp
Figure 14: Examples LLM-based cues generation.
Algorithm 1: Decomposed Ppatio-Tempori Highlighting
Input: MILM Teg video query input Q. video X, track proposals Opm {0r ...., test-time tuning epoches Nep
( Op}.
Output: Generated tube pred {b. (b. based query input Q and video X
Decompose the Q into attribute/action sub-queries Q, and Qi for spatial/temporal inquiry. Initialize spatial prompt V. and temporal prompt Vt-
Initialize eep 0.
Embed the video into visual tokens Ty. Embed the Qs and Q into text tokens T and gr
while training do
while dep Nee do
// spatial prompt Learning
Compute positive and negative logit prediction for attribute subquery Q.: px* llgitt,,, (vt (T, + Vs, T M.ii),, pdd = logit,, ( + V.. T Ue:))
Update V, by minimizing Lg = exp (p - "0)).
// temporal prompt Learning
Compute positive and negative logit prediction for action query Q yes (T, V., T Th, (c:))), pdd logit (1/00 (T, + V., T Mci))
1d = logit,, (i4
Update V, by minimizing fa exp (p po)).
Kep
kpp
end end
prepare general query prompt Qcap- while inference do cache the visual attention map Acap by query Qcap as input. // inference with T. V..
cache the visual attention map A by query Q with T, + V, as visual tokens. compute spatial-related visual attention map A A Acap: obtain the object track prediction.
// inference with T, V,-
cache the visual attention map AT by query Q with T, + V, as visual tokens.
compute temporal-related visual attention map A = A /Aepps
obtain the target frames prediction.
Combine the object track prediction and target frames prediction.
end
30

---

8 More Results and Ablation Analysis
8.1 Comparison on VidSTG (Interrogative) and RefCOCO Benchmarks
In Tab.5. compare approach with other SOTA methods the VidSTG (Interrogative) bench- marks. Even given the interrogative sentence as query, our method still show superior performance with 4.0% improvement on the vIoU @05 metric when integrated with LLaVA-Next-Videen model, which show the strong generalization capability of our framework. Undoubtedly, our method can also achieve superior localization performance in the image grounding tasks. In Tab. 6 compare our framework with other SOTA visual grounding methods on RefCOCO series benchmarks. Here we just introduce the selection of superior token for grounding and has achieved comparable and even better performance than current zero-shot SOTA methods. The The 'Oracle' denotes the result of choosing the most exact one of candidate boxes (extracted by G-DINO) with knowledge of the ground truth. We also show the proportion of our method's ' performance relative to oracle's, s, which is indicatd by the number in lower right corner. Note that our insight about grounding tokens We believe that it can be integrated with other works to
is orthogonal to other works (69 30 24 achieve better results.
Table Comparison on VidSTG (Interrogative) benchmark.
VidSTG (Interrogative)
Sup
Method
m_v
vIoU@0.3 vloU@0.5
TubeDETR
59
(CVPR2022)
25.7
35.7
23.2
Full
CSDVL 20 CG-STVG
(VPR22023
28.5
39.9
26.2
[CVPR2024)
29.0
40.5
27.5
WINNER 29
CVPR22023
10.2
12.0
5.5
Weak
VEM 21
BCUV2024
13.3
16.7
7.7
CoSPal.
[CLR2225]
13.5
16.4
10.2
ReCLIP [ACL2002]
8.4
8.0
2.3
E3M 3
[5CC22044]
10.6
12.2
5.4
ZS
Ours 1L.VA-Naaa-Vido--78
14.6
23.1
9.4
Ours Qwen2 VL. 7B
13.0
19.5
7.8
Ours ShureGPT4Viddo
13.1
21.0
9.1
Ours
13.5
21.4
8.2
Table 6: Comparison of different methods RefCOCO series datasets.
RefCOCO testA
ReCCOCO+ testA
RefCOCOg
Sup
Method
val
testR
val
textR
val
test
MDETR
22
86.8
89.6
81.4
79,5
84,1
70.6
81.6
80.9
SeqTR
BCCV30021
87.0
90.2
83.6
78.7
84.5
71.9
82.7
83.4
UNINEXT
85.2
89.6
88.7
Full
Shikra-7R
[xXGr]
Ferret-72 Berret-7BBe
HCT 82008)
87.5
91.4
82.5
80.8
87.4
73.1
83.9
84.8
ReCLIP 50
45.8
46.1
47.1
47.9
50.1
45.1
59.3
59.0
28. REC TO
49.4
47.8
$1.7
48.9
50.0
46.9
61.0
60.0
Zero
GroundVLP
46
[NAADONN]]
65.0
73.5
55.0
68.8
78.1
57.3
74.7
75.0
18.777.45
77.2a0.59
62.0rg.
67.174
76.7ap.2%
57.5gg.78
72.3g
73.7as
95.6
8.2 Temporal Consistency Analysis
In our work, we obtain the attributes and actions related sub-queries. Specially, the decomposed attribute sub-query, which provides static state description, should be temporally consistent for spatial grounding. Interestingly, there exists evident inconsistency when introducing temporal augmentation current ML..Ms. We develop a temporal inconsistency metric by introducing reversing the order of input frames. In Fig. 15 show the relation between spatial grounding accuracy and temporal
31

---

80
Spattal graunding arsuiiic
Spatea groundiog accurnty
60
80
40
20
Temporal consistency
Temporal consistencyy
Figure 15: Spatial grounding accuracy of different groups of samples the HC-STVGv1 dataset. These groups are ranked by descending temporal consistency.
Table 7: Iteration times Nep and learning rate Ir ablation.
m m_v vloU@0.3 v.5
ep
0
15.2
25.1
8.5
4
19.9
32.6
11.6
1
8
20.4
33.6
12.4
16
20.2
32.3
12.0
4
20.5
34.0
12.1
8
20.3
32.6
12.1
2
16
20.3
33.2
12.2
consistency based on Met---eet--iideoo model (left of Fig. 15 and LLaVA- -nAVAA--Oeission-78 model (right of Fig. [1]]. We can find that although the LLaVA-OneVision- localization performance, there is still the pronounced temporal inconsistency caused by temporal augmentation. Our findings provide guidance and insight for subsequent research in video MILMSs
model achieves better
8.3 Effect of Optimization Times Nep and Learning Rate Ir
The test-time tuning strategy DSTH is iterated Nep times for optimization with learning rate Ir. Here we analyse effect of the haper-parameters. As shown Tab. 7 better results can achieved as the optimization progresses and the our optimization strategy relatively robust these hyper- parameters. In particular, when setting Nep and Ir = 8.0, optimal performance achieved in general.
8.4 Inference Efficiency
Inference Time
L. Nost-Vido
EL-.A- EL-NA--iin Vlniission
1.9
wo/Tuning
1.2
w/Tuning
Time/scoods
Figure 16: Comparison of inference time before/after introducing test-time tuning.
32

---

In Fig. 16 we show the inference efficiency of our proposed zero-shot framework. We test the inference speed on all video samples from HC-STVGv1 dataset on a single A100 GPU and then compute the average value. Specifically, before introducing the test-time tuning strategy DSTH for inference, our framework costs about 3.1 seconds for each video based on LLaVA-Next-Video model. After adopting the test-time tuning strategy, the cost is still acceptable though with some additional resource consumption. We believe that it would be more efficient to apply the resource- friendly inference schemes in the future.
8.5 Qualitative Spatio-Temmn Temporal Video Grounding Visualization
In Fig. we present some video grounding examples for qualitative analysis. Here, we compare the results before introducing the test-time tuning (denoted with yellow boxes) with the results (denoted with green boxes) after introducing the optimization. The ground truth boxes are denoted with red boxes. By highlighting the attribute/action cues of the target by test-time tuning, our method can direct the MLLMs toward reliable visual context and improve spatial/temporal localization. We also give some failure cases (e.g., the case (d)). Despite optimization during testing, the current model still pinpoints the wrong object instance. We attribute the less efficient comprehension the poor visual conditions and suboptimal spatial inference in current MLLMs.
33

---

The man in the left the man the orange shirt sits down, themr turns head to the left.
(a) Video
wo/Tuning
Ours
The woman yellow clothes looks up the woman holding the newspaper, then looks down at the book her hand.
(b) Video
wo/Tuning
Ours
left.
(c)
Video
wo/Tuning
the woman the cupboard turns and walks away.
(d) Video
wo/Tuning
Ours
Figure 17: Comparison of attention maps on the HC-STVG dataset.
34

---

Limitation and Future work
Our work is simple yet effective, and also provides insights for related fields (e.g., hallucination detection and ovention-guiden MLLMs pruning). Our work does have limitations. For example, based on MLLMs our framework can only receive limited number of video frames as visual input due to the limit of computing resource. Besides, our framework needs to obtain attention for spatial and temporal inference, which not compatible with the flash-attention mechanism adopted in current MLLMs. In the future, we will consider introducing more efficient solution for the comprehension of long videos by incorporating token pruning and key frame selection techniques.
10 Broader Impacts
While we do not foresee our method causing any direct negative societal impact, it may potentially be leveraged by malicious parties to create applications that could misuse the grounding capabilities for unethical illegal purposes. We urge the readers to limit the usage of this work legal cases.
35


Reviews:
[{"review_id": "c70yUMMwXM", "reviewer": "Reviewer_8wFh", "summary": "This paper introduces novel decomposed spatio-temporal highlighting (DSTH) and TAS for robust zero-shot spatiotemporal video grounding with multimodal LLMs.\nThe DSTH is composed of action and attribute sub-queries, followed by the logit-guided re-attention (LRA) to highlight those cues.\nThe temporal-augmented assembling (TAS) strategy is also proposed to perform frame-level reversion operation. \nIt demonstrates the effectiveness on HCSTVG-v1, HCSTVG-v2, and VidSTG benchmarks.", "questions": "Q1 (for W1):\nThe LRA module contrasts “yes” and “no” logits to guide spatial/temporal attention, but this seems rather heuristic. Have you considered or evaluated alternative learning signals (e.g., entropy minimization, ranking loss)? Could you provide any theoretical or empirical justification for why this particular loss formulation leads to better grounding?\n\nQ2 (for W2):\nTest-time optimization (e.g., LRA) and the use of temporal-augmented inputs (TAS) likely increase inference time and memory usage. Could you report throughput (e.g., GFLOPS) comparison with other zero-shot or weakly-supervised STVG methods? Have you evaluated the practicality of the proposed method for real-time or long-video settings?\n\nQ3 (for W3):\nAll experiments are conducted on 7–8B MLLMs. Have you tried smaller or larger MLLMs to validate the scalability and generality of your method? How sensitive is the performance to the model size?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "S1. This paper presents a novel zero-shot framework for spatio-temporal video grounding (STVG) that effectively exploits the grounding capabilities of special tokens in pre-trained MLLMs. The framework outperforms existing zero-shot methods.\n\nS2. The proposed DSTH strategy decomposes a text query into attribute and action sub-queries and applies a novel LRA module for spatial and temporal prompt tuning. This is a practical solution to adapt frozen MLLMs during inference without finetuning.\n\nS3. The paper offers compelling empirical evidence that MLLMs dynamically assign special tokens that exhibit strong grounding behavior. The analysis of attention maps across models adds a new perspective to the interpretability of MLLMs.\n\nS4. The proposed method is evaluated on three STVG benchmarks and shows significant improvements over zero-shot, and even some weakly-supervised baselines. The method generalizes well across four different MLLMs, demonstrating robustness.\n\nW1. While LRA effectively boosts grounding performance, the theoretical justification behind why contrasting logit responses (“yes” vs. “no”) leads to better grounding remains weak. The mechanism is intuitive but lacks formal analysis.\n\nW2. The method involves prompt optimization at test time and uses additional modules (e.g., TAS). The inference cost (e.g., throughput or latency) comparisons against other zero-shot or weakly-supervised STVG methods are needed.\nAlso, the proposed method involves test-time optimization and uses dense visual tokens per frame. This may lead to high computational costs, making it less practical for long or real-time video processing.\n\nW3. All experiments are conducted on 7–8B scale MLLMs. It would strengthen the claims of generality and scalability if the proposed method were validated across smaller (e.g., 3B) and larger (e.g., 13B, 30B) models as well.\n\nW4. The visual pipeline in Figure 3 is informative, but the boundaries between key modules (e.g., DSTH, LRA, TAS) are not clearly demarcated. A more structured visualization would improve readability and make the method easier to understand.", "was_revised": true, "final_justification": "The authors have addressed my concerns, so I keep my rating for acceptance.", "rebuttal": "**We extend our sincere gratitude for your insightful reviews, which helps emphasize the contributions of our work.\nTo the weaknesses and some questions, our response is as follows.**\n\n> **W1&Q1**: LRA module seems heuristic. Alternative learning signals and any theoretical or empirical justification is expected.\n\n**[Ans]**: \n\n- **Empirical justification**. Though contrasting logits of `yes` and `no` is heuristic, it is reasonable. We find that in some cases, the model often gives ambiguous or even wrong token/response predictions when given the question prompt. Here we show an example below. \n\n  ```Text\n  Query: Is there a woman by the cupboard turns in this video?\n  \n  Answer (by llava-next-video): No, there is no woman visible in the video. The focus is on the group of men dressed in period clothing.\n  ```\n  However, the objects referred to by the text do indeed exist in the video. This indicates that the model cannot understand the video content well in certain cases. By optimizing the logit of the `yes` token toward positive prediction instead of the `no` token, we encourage the model to carefully attend to the target-related visual information and achieve better visual localization.\n\n  To validate our hypothesis, we measure the distribution of attention peaks before and after test-time optimization. Based on the evaluation on the HC-STVG-v2 dataset, we find that the average attention peak before test-time optimization is about **6.7** (without normalization), and after optimization, it increases to **23.9** (without normalization). **This change in attention values indicates that our proposed optimization strategy can encourage the model to perceive more visual modal information for generating visually-grounded responses**, thereby justifying our previous hypothesis.\n\n- **Alternative learning signals**. Here, we compare two approaches: `entropy minimization` (*i.e.,* increasing the predicted probability of the “yes” token across the entire vocabulary) and contrasting the logits of `yes` and `no` during test-time tuning. The results obtained by LLaVa-Next-Video-7B on the HC-STVG-v2 dataset are shown in the table below.\n\n  | Methods                                                      | Vision    | Language  | m_vlou $\\uparrow$ | vlou@0.3 $\\uparrow$ |  vlou@0.5 $\\uparrow$ |\n  | -- | -- | -- | -- | -- | -- |\n  | LlaVA-Next-Video-7B + GTI     | clip-vit-large | vicuna/7B-V1.5 | 19.2   | 28.5     | 12.4     |\n  | LlaVA-Next-Video-7B + entropy minimization    | clip-vit-large | vicuna/7B-V1.5 | 19.5   | 28.5     | 12.5     |\n  | LlaVA-Next-Video-7B + contrasting the logits of `yes` and `no` | clip-vit-large | vicuna/7B-V1.5 | 22.1   | 32.9     | 14.0     |\n\n  The first line denotes the results of introducing grounding token identification (**GTI**). The second and third lines denote the test-time tuning results achieved by `entropy minimization` and contrasting the logits of `yes` and `no`, respectively. We find that `entropy minimization` does not bring noticeable improvement. We believe this is because `entropy minimization` may be achieved by reducing the token prediction of abundant irrelevant tokens in the vocabulary while ignoring the differentiation between the `yes` and the `no` token. As a result, it is less efficient than explicitly contrasting the logits of `yes` and `no` in promoting the model to attend to abundant visual information.\n\n> **W2&Q2**: Report throughput (e.g., Gflops) comparison with other STVG methods. Evaluated the practicality of the proposed method for real-time or long-video settings.\n\n**[Ans]**: \n\n- **Inference efficiency evaluation**.\n\n  Since there are few open-source works on weak supervision or zero-shot learning, we compared ours with recent fully supervised work in terms of inference efficiency in below table. We reproduce the inference time and computational complexity on the HC-STVG-v2 dataset following the setting (sample 20 frames with 336x336 resolution for each video) in manuscript  on our computing device (one A100-GPU).  We notice that CG-STVG provides an analysis of computation comparison on VidSTG in supplementary file. We utilize the information for reference.\n\n  | Methods       | Training | compute (Gflops) | Infer time (seconds) | throughput (Gflops/s) |\n  | :-- | :-: | :-: | :-: | :--: |\n  | CG-STVG        | 32 A100  | ~3000            | 0.6                  | ~5000                 |\n  | TubeDETR            | 16 V100  | ~1500            | 0.4                  | ~3800                 |\n  | STCAT      | 32 A100  | ~2900            | 0.5                  | ~5800                 |\n  | CG-STVG (`reproduction`)            | 32 A100  | ~1800            | 0.8                  | ~2200                 |\n  | Qwen2-VL-7B (`reproduction`)        | None     | ~7400            | 2.8                  | ~2600                 |\n  | LLaVa-OneVision-7B (`reproduction`) | None     | ~4000            | 1.9                  | ~2000                 |\n\n  Specifically, we used the **FlopCounterMode** method to measure the computational requirements of **Qwen2-VL-7B** and **LLaVa-OneVision-7B** when loading the model using 8-bit quantization. We mainly evaluate the required compute during the visual tokens preparation and language model inference. Through comparison, it can be seen that our method does not fall significantly behind current methods in terms of inference latency and computational complexity. Instead, previous works require a significant amount of computational resources during training.\n\n- **Proposals for more efficient inference**  \n\n  In practice, in addition to using lower-quantization loading methods (e.g., 4-bit), we can also optimize the inference process starting from the input. For example, when performing temporal grounding, we can draw inspiration from previous works (e.g., [A] FastV and [B]VisionZip) to optimize less important visual tokens from a spatial perspective. Similarly, for spatial localization, we can selectively process a few key frames (e.g.,  [C] AKS) instead of inputting all video frames. These measures can effectively reduce inference latency in practical applications and enhance usability in long video understanding. Overall, we think that the exploration of long video inference merits its own separate paper, and we leave it to future works.\n\n> **W3&Q3**: Generality and scalability of proposed method can be validated across smaller and larger models as well.\n\n**[Ans]**:  We conduct the scalability verification on Intern-VL series models. With limited computational resources (one A-100), we evaluate our method on **InternVL3-2B**, **InternVL3-8B** and **InternVL3-14B** in the table below. Besides reporting the grounding metric (e.g., `vIoU@0.3`, `m_vIoU`), we also give the config (e.g, `N_hidden_layers`) of different VL models.\n\n- Generality of our method is well validated across different scales of models. As indicated in the main paper, we average the attention maps of all special tokens as the **baseline**. Benefiting from our grounding token identification and test-time optimization, we can obviously achieve better grounding performance not only in the smaller (4B) but also the larger models (14B).\n- Models with larger number of parameters often yield better results. The performance of small models (e.g, `InternVL3-2B`) is far inferior to the larger models. It is because fewer parameters in language models perform poorly in multimodal alignment. By further probing the attention maps, we find that the attention maps are often centered on tokens in background and edge regions. How to unlock the interpretability of small models' attention maps is an interesting research topic, and we leave it for future work. Besides, the performance of  `InternVL3-14B` declines slightly than the `InternVL3-8B`, we think it is because it has too many layers and attention maps with grounding ability are easily overwhelmed by a large number of attention maps with poor grounding ability.\n\n|           |     Vision     |   Language   | N_heads | N_layers | N_kv_heads | vIoU@0.3 $\\uparrow$ |  m_vIoU $\\uparrow$  |   t_iou $\\uparrow$   | gt_viou $\\uparrow$ |\n| :- | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n| InternVL3-2B+baseline   | InternViT-300M | Qwen2.5-0.5B |   12    |    28    |     2      |   25.3   |   16.0   |   35.9   |   38.4   |\n| InternVL3-2B+Ours       | InternViT-300M | Qwen2.5-0.5B |   12    |    28    |     2      | **27.7** | **17.8** | **38.3** | **40.0** |\n| InternVL2.5-4B+baseline | InternViT-300M |  Qwen2.5-3B  |   16    |    36    |     2      |   24.5   |   17.0   |   38.4   |   38.2   |\n| InternVL2.5-4B+Ours     | InternViT-300M |  Qwen2.5-3B  |   16    |    36    |     2      | **41.1** | **25.4** | **45.5** | **49.5** |\n| InternVL3-8B+baseline   | InternViT-300M |  Qwen2.5-7B  |   28    |    28    |     4      |   36.5   |   22.9   |   42.9   |   46.1   |\n| InternVL3-8B+Ours       | InternViT-300M |  Qwen2.5-7B  |   28    |    28    |     4      | **45.3** | **27.5** | **47.2** | **53.2** |\n| InternVL3-14B+baseline  | InternViT-300M | Qwen2.5-14B  |   40    |    48    |     8      |   33.2   |   21.0   |   41.2   |   44.8   |\n| InternVL3-14B+Ours      | InternViT-300M | Qwen2.5-14B  |   40    |    48    |     8      | **42.2** | **26.2** | **45.8** | **51.3** |\n\n> **Q4**: boundaries between key modules in Figure 3 can be more clearly demarcated.\n\n**[Ans]**: Thanks for your constructive comments on paper revision. We will revise the boundaries between key modules in Figure 3, and make it clearly demarcated in the final version. \n\nHope that our response can well addresses your concerns. Please let us know if you have any other concerns.\n\n> [A] An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models, ECCV2024\n>\n> [B] VisionZip: Longer is Better but Not Necessary in Vision Language Models, CVPR2025\n>\n> [C] AKS: Adaptive Keyframe Sampling for Long Video Understanding, CVPR2025"}, {"review_id": "LiJQ8zsvvX", "reviewer": "Reviewer_cWEf", "summary": "This paper proposes a zero-shot spatio-temporal video grounding method. The authors observe that some special tokens within existing multi-modal LLMs (LLaVA-OneVision, Qwen2-VL, etc.) are superior for grounding, which they term _grounding tokens_. By leveraging this, the authors further propose methods to handle complex video queries: Decomposed Spatio-Temporal Highlighting (DSTH) and Logit-guided Re-Attention (LRA) to adapt grounding tokens for refining localization. The proposed method outperforms existing zero-shot methods using diverse off-the-shelf MLLMs on HCSTVG-v1&v2 and VidSTG datasets.", "questions": "- In Figure 6(b), why does using more frames (e.g., 11) lead to a drop in performance compared to using 7 frames? Could the authors elaborate on the possible reasons for this degradation?", "limitations": "The authors briefly mention limitations in the conclusion section, but the discussion is not particularly constructive or detailed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 1, "originality": 1, "strengths_and_weaknesses": "**Strengths:**\n- The paper is clearly written and easy to follow. Figures and graphs are helpful in illustrating the method and results.\n- The study includes a comprehensive evaluation across a diverse set of open-source multimodal LLMs, including LLaVA-Next-Video, ShareGPT4Video, Qwen2-VL, and LLaVA-OneVision.\n\n**Weaknesses:**\n- The strongest existing baseline (E3M) does not use LLMs, and the performance gap when using LLaVA-Next-Video-7B appears marginal. Including analysis on model size and throughput would help contextualize this trade-off.\n- The proposed method lacks clarity in some parts. The observation regarding the grounding behavior of special tokens is empirical, and the reliance on manually identifying grounding tokens per MLLM raises concerns about generalizability and scalability. When using larger models, such as 13B variants, does performance consistently improve? If not, could the authors provide insight into any observed saturation or trade-offs?\n- According to the implementation details, the method depends on several additional modules, including Grounding DINO, SAM2, and GPT-4o. This multi-component design introduces potential efficiency and deployment concerns.", "was_revised": true, "final_justification": "The authors have addressed key concerns through a detailed rebuttal, providing both empirical evidence and analysis. I am raising my rating to borderline acceptance.", "rebuttal": "**Sincerely thanks for insightful comments.   Your valuable reviews on our work  help  revise the manuscript for the final version. To the weaknesses and some questions, our response is as follows:**\n\n> **W1**: The strongest existing baseline (E3M) does not use LLMs, and the performance gap when using LLaVA-Next-Video-7B appears marginal. Including analysis on model size and throughput would help contextualize this trade-off.\n\n**[Ans]**: \n\n- **About the marginal performance gap.**\n  - The E3M adopts the **CLIP [A]** as the foundation model which is optimized toward discriminative pre-training objective and presents some advantages for target localization. Differently, our method is based on the LLMs which is toward generative pre-training objective. Unleashing the perception ability of generative models poses a greater challenge to the multimodal alignment of the foundation model.\n  - Despite the challenge, our work provides insightful observations about the grounding tokens of LLMs and presents a solution based on generative pre-training models. Compared to **E3M** based on discriminative pre-training models (e.g., **CLIP[A]**), our framework is simple and can well adapt to current multimodal LLMs and achieve comparable or even better grounding performance. When adopting the `LLaVA-OneVision-7B` model, our method outperforms 7.9% and 4.0% than the E3M method on the HC-STVG-v2 dataset.\n\n  - When adopting the less strong `LLaVA-Next-Video-7B` model, the grounding performance is inferior to the performance of LLaVA-OneVision-7B. But a substantial improvement has been achieved compared to the baseline using LLMs as shown in Table 2 of our manuscript, which well verifies the effectiveness of our method. In fact, the grounding performance may be further improved when integrating recent observations ([B],[C]) about discriminative layers into our framework. Overall, our work aims to reveal the inherent grounding ability of special tokens and propose a simple and effective grounding framework that can serve as a basis for further exploration in the future.\n\n    \n\n- **Analysis on model size and throughput**\n\n  - Since there are currently few open-source works under weakly supervised and zero-shot settings, we compare our method with recent fully supervised works in terms of inference time and computational complexity in the following table.\n\n    | Methods                             | Training | compute (Gflops) | Infer time (seconds) | throughput (Gflops/s) |\n    | :-- | :-: | :-: | :-: | :--: |\n    | CG-STVG                             | 32 A100  | ~3000            | 0.6                  | ~5000                 |\n    | TubeDETR                            | 16 V100  | ~1500            | 0.4                  | ~3800                 |\n    | STCAT                               | 32 A100  | ~2900            | 0.5                  | ~5800                 |\n    | CG-STVG (`reproduction`)            | 32 A100  | ~1800            | 0.8                  | ~2200                 |\n    | Qwen2-VL-7B (`reproduction`)        | None     | ~7400            | 2.8                  | ~2600                 |\n    | LLaVa-OneVision-7B (`reproduction`) | None     | ~4000            | 1.9                  | ~2000                 |\n\n   - Here we list  the provided analysis results on VitSTG dataset in the supplementary file of CG-STVG for reference. Also, the lines with marker reproduction denote the reproduced results by our computing device (one A100) on HC-STVG-v2 dataset. We used the **FlopCounterMode** method to measure the computational requirements of **Qwen2-VL-7B** and **LLaVa-OneVision-7B** when loading the model using 8-bit quantization. Through comparison, it can be seen that our method does not fall significantly behind current methods in terms of inference latency and computational complexity. Instead, previous works require a significant amount of computational resources during training. \n\n   - **Proposals for more efficient inference**  \n\n     Here we also give some engineering optimization proposals. For example, we can try loading models by lower-quantization method (e.g., 4-bit). We can also optimize the inference process starting from the input. When performing temporal grounding, we can draw inspiration from previous works (e.g., **[D]** FastV and **[E]** VisionZip) to optimize less important visual tokens from a spatial perspective. Similarly, for spatial localization, we can selectively process a few key frames (e.g., **[F]** AKS) instead of inputting all video frames. These measures can effectively reduce inference latency in practical applications and enhance usability in long video understanding. Overall, we think that the exploration of long video inference merits its own separate paper, and we leave it to future works.\n\n     Please let us know if you have further questions and concerns.\n\n\n\n> **W2**: The observation regarding the grounding behavior of special tokens is empirical, and the reliance on manually identifying grounding tokens per MLLM raises concerns about generalizability and scalability.\n\n**[Ans]**: Although the grounding behavior of special tokens is derived from empirical observations, it is well verified across different Multimodal LLMs, as shown in Figure 2 of the paper manuscript and the reply to `W3&Q3` of `Reviewer1-8wFh ` . Besides, in the reply to `W1&Q1`  of `Reviewer1-8wFh` we provide empirical justification.  The theoretical exploration of this phenomenon is also an interesting research topic, and we leave it to future works.\n\nFurthermore, our method does not require manually identifying grounding tokens for each MLLM. The special tokens equipped in each MLLM are fixed. As stated in `Line187–Line190`, we can effectively identify the optimal special token characterized by high visual activation based on our findings introduced in `Line 174`. This implementation is quite simple.\n\n\n\n> **W3**: the method depends on several additional modules, including Grounding DINO, SAM2, and GPT-4o. This multi-component design introduces potential efficiency and deployment concerns.\n\n**[Ans]**:  As in other zero-shot works (such as E3M and RedCircle), we utilize off-the-shelf detectors for tubelet generation. In addition, we also leverage GPT-4o for query decomposition. The overall pipeline design is simple and does not introduce excessive components.\n\nFurthermore, for these components, there are several lightweight alternatives available. For example, **[G]TinySAM** and **[H]EdgeTAM** introduce more lightweight designs based on the original SAM2 and can process video at almost real-time speed. For GPT-4o, there are also many lightweight alternatives, such as GPT‑4o Mini, GPT‑4.1 Mini, GPT‑4.1 Nano, and Gemini 2.5 Flash-Lite, which can reduce latency to the millisecond level. Thus, the introduced components have minimal impact during deployment in practical applications. We will release our code after acceptance.\n\n\n\n> **Q1**: In Figure 6(b), why does using more frames (e.g., 11) lead to a drop in performance compared to using 7 frames? Could the authors elaborate on the possible reasons for this degradation?\n\n**[Ans]**: We apologize for any confusion this may have caused.  Figure 6(b) shows the ablation study on the selected frames **K** during temporal prediction  instead of the input frames number. Since we need to simultaneously focus on both temporal and spatial localization, it means that for a target person we must localize the actions described in the text while ignoring other irrelevant temporal segments. When using more frames (e.g., 11) for temporal grounding prediction, the model tends to include more irrelevant temporal segments, leading to performance degradation. According to ablation results in Figure 6(b), we set the appropriate hyperparameter **K** .\n\nHope that our response can well addresses your concerns. Please let us know if you have any new questions or concerns.\n\n\n\n> [A] Learning transferable visual models from natural language supervision, ICML2021\n>\n> [B] Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach, CVPR2025\n>\n> [C] Layer by Layer: Uncovering Hidden Representations in Language Models, ICML2025\n>\n> [D] An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models, ECCV2024\n>\n> [E] VisionZip: Longer is Better but Not Necessary in Vision Language Models, CVPR2025\n>\n> [F] AKS: Adaptive Keyframe Sampling for Long Video Understanding, CVPR2025\n>\n> [G] TinySAM: Pushing the Envelope for Efficient Segment Anything Model, AAAI2025\n>\n> [H] EdgeTAM: On-Device Track Anything Model, CVPR2025"}, {"review_id": "Dz1O9k2MHq", "reviewer": "Reviewer_LSgz", "summary": "This paper proposes a zero-shot STVG framework based on MLLMs, which localizes target objects in videos according to text queries without requiring frame-level annotations. The framework enhances grounding capabilities through the introduction of DSTH and TAS strategies, and demonstrates strong performance across three benchmark datasets.", "questions": "(1) While the authors report the inference efficiency of their zero-shot framework, a broader comparison with other zero-shot, weakly-supervised, and fully-supervised methods would be valuable. How does the proposed method perform in terms of inference speed and computational complexity relative to these approaches?\n(2) While the method achieves strong zero-shot performance on STVG, there is still a notable gap compared to fully-supervised approaches. Given that MLLMs themselves demonstrate strong performance in image spatial grounding and video temporal grounding, could the authors clarify why such a performance gap still exists?\n(3) While the authors introduce a new method leveraging MLLMs for STVG, a natural baseline would be to directly use an MLLM to first perform temporal grounding, followed by spatial grounding on each frame. Could the authors clarify whether this straightforward approach would achieve comparable or even better performance than the proposed method? A comparison or analysis of this alternative would strengthen the paper.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n(1) The paper demonstrates that MLLMs’ ability to dynamically assign special tokens for visual grounding can be effectively used for precise zero-shot STVG without additional annotations.\n(2) The method enhances performance across various MLLMs, demonstrating generalization capabilities.\n(3) The ablation experiments fully demonstrate the effectiveness of the proposed test-time tuning strategy DSTH.\n(4) The writing is clear and the illustrations are explicit.\n\nWeaknesses\n(1) In comparison with other methods, the reliance on MLLMs introduces higher computational complexity to this approach, which could potentially impact inference speed and efficiency.\n(2) Since STVG concerns both temporal and spatial grounding, it is recommended to include a temporal grounding evaluation metric, such as m_tIoU, in addition to the metric m_vIoU. This approach, which has been adopted by recent methods like CG-STVG[1], CoSPaL[2] and TA-STVG[3], would provide a more comprehensive assessment.\n[1] Context-guided spatio-temporal video grounding CVPR'24\n[2] Contextual self-paced learning for weakly supervised spatio-temporal video grounding ICLR'25\n[3] Knowing Your Target: Target-Aware Transformer Makes Better Spatio-Temporal Video Grounding ICLR'25\n\n(3) The paper introduces a novel approach that leverages special markers in MLLMs for precise target localization. However, to comprehensively assess the model's performance, it is essential to provide a thorough analysis of failure cases. Currently, only a single failure case is presented in the supplementary materials, and there is a lack of in-depth analysis.", "was_revised": true, "final_justification": "The author's response has addressed most of my concerns, and the additional analysis effectively demonstrates the validity of their approach. Therefore, I am inclined to raise the initial score. I appreciate the author's efforts during the rebuttal process.", "rebuttal": "**Sincerely thanks for insightful comments.   Your valuable reviews help  revise the manuscript. To the weaknesses and some questions, our response is as follows:**\n\n> **W1&Q1**: the reliance on MLLMs introduces higher computational complexity to this approach, which could potentially impact inference speed and efficiency.\n\n**[Ans]**: We report the inference time and compute metrics in the table below. Since there are few open-source works under weakly supervised and zero-shot settings, we compare our method with recent fully supervised works.\n\nHere we list the results on VitSTG dataset in the supplementary file of CG-STVG for reference. Also, we reproduce results by our computing device (one A100) on HC-STVG-v2 dataset. We used the **FlopCounterMode** method to measure the computational requirements when loading the model using 8-bit quantization.\n\nOur method does not fall significantly behind current methods in terms of inference efficiency. Instead, previous works require a significant amount of computational resources during training. To reduce inference latency and enhance practical feasibility, we have also provided some engineering optimization proposals in the section `Proposals for More Efficient Inference` as our reply to `W2 & Q2` of `Reviewer1-8wFh`.\nPlease let us know if you have any further questions or concerns.\n\n   | Methods                             | Training | compute (Gflops) | Infer time (seconds) | throughput (Gflops/s) |\n   | :-- | :-: | :--: | :--: | :--: |\n   | CG-STVG                             | 32 A100  |      ~3000       |         0.6          |         ~5000         |\n   | TubeDETR                            | 16 V100  |      ~1500       |         0.4          |         ~3800         |\n   | STCAT                               | 32 A100  |      ~2900       |         0.5          |         ~5800         |\n   | CG-STVG (`reproduction`)            | 32 A100  |      ~1800       |         0.8          |         ~2200         |\n   | Qwen2-VL-7B (`reproduction`)        |   None   |      ~7400       |         2.8          |         ~2600         |\n   | LLaVa-OneVision-7B (`reproduction`) |   None   |      ~4000       |         1.9          |         ~2000         |\n\n\n\n> **W2**: Since STVG concerns both temporal and spatial grounding, it is recommended to include a temporal grounding evaluation metric, such as m_tIoU, in addition to the metric m_vIoU\n\n**[Ans]**: Thanks for careful suggestions!  In the table below, we have integrated the `m_tIoU` metric (**last column**) into the results comparison. Compared to the weakly-supervised SOTA method, our method achieves comparable performance on the temporal grounding evaluation metric `m_tIoU`. However, we notice that although our temporal grounding performance is slightly lower than the previous method **CoSPaL**, while our overall performance is significantly higher than **CoSPaL**. It maybe because that  MLLMs have weak differentiation between attention to different frames.  This indicates that there is still considerable room for improvement in temporal localization. We will add comparisons and discussions with these works to the final version.\n\n| Methods      | Supervision | m_vlou $\\uparrow$ | vlou@0.3 $\\uparrow$ | vlou@0.5 $\\uparrow$ | m_tlou $\\uparrow$ |\n| --- | :---: | :--: | -- | :---: | :--: |\n| CSDVL (CVPR2023)              |    Full     |  38.7  | 65.5     |   33.8   |  **58.1**  |\n| CG-STVG (CVPR2024)            |    Full     |  39.5  | 64.5     |   36.3   |  **60.0**  |\n| TA-STVG (ICLR2025)            |    Full     |  40.2  | 65.8     |   36.7   |  **60.4**  |\n| CoSPaL (ICLR2025)             |    Weak     |  22.2  | 31.4     |   18.9   |  **48.6**  |\n| **Ours (Qwen2-VL-7B)**        |    Zero     |  25.6  | 40.5     |   17.1   |  **45.2**  |\n| **Ours (LlaVa-OneVision-7B)** |    Zero     |  27.7  | 47.7     |   19.5   |  **46.0**  |\n\n\n\n> **W3**: To comprehensively assess the model's performance, it is essential to provide a thorough analysis of failure cases.\n\n**[Ans]**: Thanks for your careful comments! Due to the limitations of the rebuttal format, we regret that more failure cases cannot be provided immediately. Through our observations, in addition to the suboptimal tracking tubelet, the attention on the background token may negatively impacts grounding performance. Beisides, suboptimal spatial comprehension also cause unfaithful attendance to distractors as shown in Figure17(d) of supplement.  We will provide a more thorough analysis of failure cases in the final version.\n\n\n\n> **Q2**:  Given that MLLMs themselves demonstrate strong performance in image spatial grounding and video temporal grounding, could the authors clarify why such a performance gap still exists?\n\n**[Ans]**: \n\n- Though current MLLMs (e.g., Qwen2-VL) can achieve outstanding grounding performance by intorducing grounding datasets during instruction tuning, there is an obvious mechanistic difference between our attention-based, training-free solution and these fully-supervised methods. Our solution leverages the emergent grounding ability of special tokens, while these grounding models explicitly output the coordinates tokens corresponding to bounding boxes.\n\n- Even for MLLMs (e.g., LLaVA, DeepSeek-VL) without fine-tuning for target detection/grounding, they still exhibit emergent grounding abilities, as shown in Figure 2. This indicates that the grounding ability of special tokens may originate from cross-modal comprehension and alignment rather than fine-tuning for grounding tasks. As a result, the grounding ability of special tokens benefits little from the grounding-oriented fine-tuning process.\n\n- Here, we further analyze the bottleneck of grounding performance in our method. In the table below, we show the `m_tIoU`, which measures the performance of temporal grounding, and the `gt_vIoU`, which measures the performance of spatial grounding. We observe that there is a certain gap between our method and fully-supervised methods in temporal localization, which is an important factor affecting the overall performance. Additionally, we provide the upper bound of spatial grounding by selecting the candidate tubelet that best matches the ground truth in spatial grounding. The upper bound of spatial grounding is **65.8**, which suggests that we can improve performance by using better detectors and trackers for candidate tubelet generation.\n\n  | Methods       | Supervision | m_vlou $\\uparrow$ | vlou@0.3 $\\uparrow$ | vlou@0.5 $\\uparrow$ | m_tlou $\\uparrow$ | gt_vlou $\\uparrow$ |\n  | -- | :--: | :--: | :--: | :--: | :--: | :---: |\n  | CSDVL (CVPR2023)                   |    Full     |  38.7  |   65.5   |   33.8   |  **58.1**  |   --    |\n  | CG-STVG (CVPR2024)                 |    Full     |  39.5  |   64.5   |   36.3   |  **60.0**  |   --    |\n  | TA-STVG (ICLR2025)                 |    Full     |  40.2  |   65.8   |   36.7   |  **60.4**  |   --    |\n  | CoSPaL (ICLR2025)                  |    Weak     |  22.2  |   31.4   |   18.9   |  **48.6**  |   --    |\n  | **Ours (Qwen2-VL-7B)**             |    Zero     |  25.6  |   40.5   |   17.1   |  **45.2**  |  50.2   |\n  | **Ours (LlaVa-OneVision-7B)**      |    Zero     |  27.7  |   47.7   |   19.5   |  **46.0**  |  53.1   |\n  | *Upper-Bound of Spatial Grounding* |             |        |          |          |            | *65.8*  |\n\n\n\n> **Q3**: A natural baseline would be to directly use an MLLM to first perform temporal grounding, followed by spatial grounding on each frame. Could the authors clarify whether this straightforward approach would achieve comparable or even better performance than the proposed method? \n\n**[Ans]**: Following your comments, we first perform temporal grounding, followed by spatial grounding on each frame. The results on HC-STVG-v2 by Qwen-2.5-VL (3B and 7B) models are given in this table.\n\n|         | Vision  | Language | N_hidden_layers | N_attention_heads | N_kv_heads | vlou@0.3 $\\uparrow$ |  m_vlou $\\uparrow$  |  m_tlou $\\uparrow$ | gt_vlou $\\uparrow$ |\n| -- | :----: | :---: | :---: | :--: | :--: | :--: | :---: | :---: | :---: |\n| Qwen2.5-VL-3B      |     VIT     |  Qwen2.5-3B   |       36        |        16         |     2      |   13.3   |   14.4   |   39.4   |   36.2   |\n| Qwen2.5-VL-3B+Ours |     VIT     |  Qwen2.5-3B   |       36        |        16         |     2      | **41.7** | **25.4** | **45.9** | **49.3** |\n| Qwen2.5-VL-7B      |     VIT     |  Qwen2.5-7B   |       28        |        28         |     4      |   17.8   |   15.2   |   35.7   |   41.3   |\n| Qwen2.5-VL-7B+Ours |     VIT     |  Qwen2.5-7B   |       28        |        28         |     4      | **45.4** | **27.7** | **46.9** | **51.6** |\n\nThe **first** and **third** lines denote the results from token prediction by `Qwen2.5-VL` models. The **second** and **fourth** lines denote the performance of our method. We find that the temporal grounding (`m_tIoU`) and spatial grounding performance (`gt_vIoU`) are inferior to ours. We attribute this to the following points:\n\n- As we predict a bounding box for each frame individually, the objects localized in consecutive video frames often change, which significantly impacts the performance of spatial grounding.\n- Additionally, the VL models are sensitive to input prompts. Different prompts may lead to significantly different outputs. In the below, we show our prompts for temporal and spatial grounding. As a general-purpose model, Qwen-VL requires further fine-tuning to achieve superior performance on specific scenarios and tasks.\n```\n temporal  prompt : \"Give the query: '{}', when does the described content occur in the video? please give the starting time (seconds) and ending time (seconds) in the json format.\"\n spatial  prompt :  \"In this image there is a person described by query '{}', please give his/her coordinates in the json format.\"\n```\n \n\nHope that our response can well addresses your concerns. Please let us know if you have any new questions or concerns."}, {"review_id": "86ebucJ3as", "reviewer": "Reviewer_bp1T", "summary": "The paper is about zero-shot STVG of multimodal LLMs. The authors first observe that special tokens subsequent to the text prompts used in several MLLMs have grounding abilities. The authors introduces training-free framework for STVG leveraging such special tokens with high visual activations. The framework begins with detecting object track proposals, followed by selecting tokens with highest attention value for locating target objects and getting object track score computed using max attention value within each track. The framework is further improved by decomposed spatio-temporal highlights strategy via interrogative subquery which highlights attributes and actions cues in language query.", "questions": "From the text in the checklist, it's not clear when and whether the authors will make their code available. \nIn the recent time, Qwen2.5-VL is widely used in place of Qwen2-VL. If the authors don't have enough time for new experiments; experiments on subsets can be considered to identify the trend.", "limitations": "The authors acknowledged that long-context scenario isn't addressed in this paper.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The method is simple and training-free. Query decomposition via interrogative sub-query and logit-guided re-attention for Q&A is intuitive and effective as shown in the results. The results on three benchmark datasets are strong, specially for the zero-shot and weakly supervised learning setup.\n\nAs a general note about these benchmarks: the video length is 20 seconds for HC-STVG Dataset and average video length is ~35 seconds for VidOR dataset on which the VidSTG dataset has been made off. Spatio-temporal video grounding as an application makes sense only if the videos are longer. Although this isn't a limitation for the proposed paper, but the benchmarks and the fields in general. \n\nI see the core strength of the paper is in revealing the grounding capabilities of the special tokens. Training large models are expensive and there is a value in extracting meaningful information from pre-trained models.", "was_revised": true, "final_justification": "I read the authors' responses to my and other reviewers' questions. I am satisfied with the responses.", "rebuttal": "**Sincerely appreciate your recognition of our work! To the weaknesses and some questions, our response is as follows:**\n\n> **W1**: Spatio-temporal video grounding as an application makes sense only if the videos are longer. Although this isn't a limitation for the proposed paper, but the benchmarks and the fields in general.\n\n**[Ans]**: As you mentioned, long video comprehension remains a significant challenge with the limited context of LLM and computing resource.  \n\nHere we also give some engineering optimization proposals. For example, we can try loading models by lower-quantization method (e.g., 4-bit). We can also optimize the inference process starting from the input. When performing temporal grounding, we can draw inspiration from previous works (e.g., **[A]** FastV and **[B]** VisionZip) to optimize less important visual tokens from a spatial perspective. Similarly, for spatial localization, we can selectively process a few key frames (e.g., **[C]** AKS) instead of inputting all video frames. These measures can effectively reduce inference latency in practical applications and enhance usability in long video understanding.\n\nWe believe that integrating current research efforts to reduce resource consumption for video inference is necessary, and we will leave it for future works. Please let us know if you have any further questions or concerns.\n\n> **Q1**: From the text in the checklist, it's not clear when and whether the authors will make their code available. \n\n**[Ans]**: The original innovations in our work also have great significance and inspiration for other related fields (visual question answering, hallucination suppression, and explainability in MLLMs). To promote community research, we consider releasing the codebase once the review process is done.\n\n> **Q2**: In the recent time, Qwen2.5-VL is widely used in place of Qwen2-VL. If the authors don't have enough time for new experiments; experiments on subsets can be considered to identify the trend.\n\nWe apologize for any inconvenience this may have caused. Our initial code development was based on version 4.48.0 of the Transformers library, which did not yet incorporate `Qwen2.5-VL`. After upgrading to version 4.50.0, we conducted experiments with `Qwen2.5-VL` on the `HC-STVG-v2` dataset. Considering that the `Qwen2.5` series includes models of various sizes—2B, 7B, 32B, and 72B—due to computational resource constraints (a single A100 GPU), we validated the effectiveness of our method on the smaller model sizes (2B and 7B). The results are shown in the following table:\n\n|                        | Vision | Language | N_hidden_layers | N_attention_heads | N_kv_heads | vlou@0.3 $\\uparrow$ |  m_vlou $\\uparrow$  |  m_tlou $\\uparrow$  | gt_vlou $\\uparrow$  |\n| ---------------------- | :---------: | :-----------: | :-------------: | :---------------: | :--------: | :------: | :------: | :------: | :------: |\n| Qwen2.5-VL-3B+baseline |     VIT     |  Qwen2.5-3B   |       36        |        16         |     2      |   28.5   |   17.3   |   37.1   |   39.1   |\n| Qwen2.5-VL-3B+Ours     |     VIT     |  Qwen2.5-3B   |       36        |        16         |     2      | **41.7** | **25.4** | **45.9** | **49.3** |\n| Qwen2.5-VL-7B+baseline |     VIT     |  Qwen2.5-7B   |       28        |        28         |     4      |   39.7   |   23.6   |   44.1   |   47.2   |\n| Qwen2.5-VL-7B+Ours     |     VIT     |  Qwen2.5-7B   |       28        |        28         |     4      | **45.4** | **27.7** | **46.9** | **51.6** |\n\nIn the table, besides reporting grounding metrics (e.g., `vIoU@0.3`, `m_vIoU`), we also provide the configurations (e.g., `N_hidden_layers`) of different VL models.\n\nIn addition, we introduce two additional metrics: `m_tIoU` and `gt_vIoU`. The `m_tIoU` metric only focuses on temporal grounding performance while ignoring spatial grounding, whereas the `gt_vIoU` metric focuses on spatial grounding performance while ignoring temporal grounding.\n\nFrom the results, we observe that our proposed method generalizes well to recent new MLLMs, demonstrating both the generality and scalability of our method. Moreover, we find that the model size significantly impacts the inherent grounding ability of special tokens. We also provide experiments on the Intern-VL3 series models in our response to `W3&Q3` of `Reviewer1-8wFh` for further verification of generality.\n\n\nHope that our response can well addresses your concerns. Please let us know if you have any new questions or concerns.\n\n\n> [A] An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models, ECCV2024\n>\n> [B] VisionZip: Longer is Better but Not Necessary in Vision Language Models, CVPR2025\n>\n> [C] AKS: Adaptive Keyframe Sampling for Long Video Understanding, CVPR2025"}]

Author rebuttal:

Meta-review: The paper introduces a framework for zero-shot spatiotemporal video grounding using pre-trained multimodal large language models (MLLMs). The central contribution is the empirical observation that pre-trained MLLMs contain special grounding tokens that are particularly effective for localizing concepts expressed by text queries, enabling zero-shot grounding capability. Initial reviews were mixed, but following the rebuttal, all reviewers recommended acceptance. Despite minor concerns regarding some technical and experimental details, the reviewers agreed that the identification of grounding tokens is insightful, the proposed method is novel, and the empirical results are extensive. The AC concurs with the reviewers' post-rebuttal assessment and recommends acceptance. The authors are encouraged to incorporate the rebuttal materials into the camera-ready version.
Decision: Accept (poster)