Knowing Your Target: Target-Aware Transformer Makes Better Spatio-Temporal Video Grounding
Authors: Xin Gu
Yaojie Shen
Chenxi Luo
Tiejian Luo
Yan Huang
Yuewei Lin
Heng Fan
Libo Zhang
Venue: ICLR 2025 Oral (2025)
Keywords: Spatio-Temporal Video Grounding
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Transformer has attracted increasing interest in spatio-temporal video grounding, or STVG, owing to its end-to-end pipeline and promising result. Existing Transformer-based STVG approaches often leverage a set of object queries, which are initialized simply using zeros and then gradually learn target position information via iterative interactions with multimodal features, for spatial and temporal localization. Despite simplicity, these zero object queries, due to lacking target-specific cues, are hard to learn discriminative target information from interactions with multimodal features in complicated scenarios (e.g., with distractors or occlusion), resulting in degradation. Addressing this, we introduce a novel $\textbf{T}$arget-$\textbf{A}$ware Transformer for $\textbf{STVG}$ ($\textbf{TA-STVG}$), which seeks to adaptively generate object queries via exploring target-specific cues from the given video-text pair, for improving STVG. The key lies in two simple yet effective modules, comprising text-guided temporal sampling (TTS) and attribute-aware spatial activation (ASA), working in a cascade. The former focuses on selecting target-relevant temporal cues from a video utilizing holistic text information, while the latter aims at further exploiting the fine-grained visual attribute information of the object from previous target-aware temporal cues, which is applied for object query initialization. Compared to existing methods leveraging zero-initialized queries, object queries in our TA-STVG, directly generated from a given video-text pair, naturally carry target-specific cues, making them adaptive and better interact with multimodal features for learning more discriminative information to improve STVG. In our experiments on three benchmarks, including HCSTVG-v1/-v2 and VidSTG, TA-STVG achieves state-of-the-art performance and significantly outperforms the baseline, validating its efficacy. Moreover, TTS and ASA are designed for general purpose. When applied to existing methods such as TubeDETR and STCAT, we show substantial performance gains, verifying its generality. Code is released at https://github.com/HengLan/TA-STVG.

Published as a conference paper at ICLR 2025
KNOWING YOUR TARGET C TARGET-AWARE TRANSFORMER MAKES BETTER SPATIO-TEMPON VIDEO GROUNDING
Xin Gu Yaojie Shen Chenxi Luo
Tiejian Luo
Yan Huang
Yuewei Lin
Heng Fan
Libo Zhang2.1.1.
1 University of Chinese Academy of Sciences
2 Institute of Software Chinese Academy of Sciences
La Trobe University
*University North Texas 'Brookhaven National Laboratory
1111v25502.11668M [cs.CV] 16 Feb 2025
guxin21 mowinn1@maiii il 1s. www..adu....... en, 1 libol sereesscas... ac.
ABSTRACT
Transformer has attracted increasing interest in spatio-temporal video grounding, or STVG, owing its end-to-end pipeline and promising result. Existing Transformer- based STVG approaches often leverage a set of object queries, which are initialized simply using zeros and then gradually learn target position information via iterative interactions with multimodal features, for spatial and temporal localization. Despite simplicity, these zero object queries, due lacking target-specific cues, are hard learn discriminative target information from interactions with multimodal features in complicated scenarios (e.g., with distractors or occlusion), resulting in degrada- tion. Addressing this, we introduce a novel Target-A ware Transformer for STVG (TA-STVG), which seeks to adaptively generate object queries via exploring target- specific cues from the given video-text pair, for improving STVG. The key lies in two simple yet effective modules, comprising text- rext-guided temporal sampling (TTS) and attribute-awenn spatial activation (ASA), working in cascade. The former focuses on selecting target-relevant temporal cues from a video utilizing holistic text information, while the latter aims at further exploiting the fine-grained visual attribute information of the object from previous target-aware temporal cues, which is applied for object query initialization. Compared existing methods leveraging zero-initialized queries, object queries in our TA-STVG, directly generated from a given video-text pair, naturally carry target-speeific cues, making them adaptive and better interact with multimodal features for learning more discriminative in- formation to improve STVG. In our experiments on three benchmarks, including HCSTVG-v1/-22 and VidSTG, TA-STVG achieves state-of-the-an performance and significantly outperforms the baseline, validating its efficacy. Morcover, TTS and ASA are designed for general purpose. When applied to existing methods such as TubeDETR and STCAT, we show substantial performance gains, verifying its generality. Code is released at https: /////////t thub. com/ HengLan/TA-STT
INTRODUCTION
Spatio-temporal video grounding (or STVG) aims at spatially and temporally localizing the target of interest from an untrimmed video given its textual description (Zhang et al., 2020b). As a multimodal content from videos and accurately connect it to the
task, it requires to understand the apatio-tempont corresponding textual expression. Owing to its importance in multimodal video understanding and key applications such content-based video retrieval, robotics, etc, STVG has attracted increasing interest with many models proposed (Zhang et al., 2020a;b; Tang et al., 2021; Su et al., 2021).
Recently, inspired by the compact end-to-end pipeline of Transformer-based detection (Carion et al., 2020), researchers have introduced the Transformer (Vaswani et al., 2017) for STVG given that they both are localization task, and achieved previously unattainable result (Yang et al., 2022a; Jin et al.,
Equul advising and co-last authors; "Corresponding author

---

Published as a conference paper at ICLR 2025
Decoder
Encoder
Decoder
Video
Object
Object Queries
Output
Queries
The boy dirpped closher rators Air finger firmard, then stepe laak aganeat
Output
Spatial Quary
Zero
'Target-Aaare
Spotial Query
Temgoni Quary
Query Generation
fir door
Query Generation
Tempural Querry
Iuput
(a) Existing Transformed--usedd STVG methods
(b) Our Target-Aware Transformer-bused STVG
Comparison between existing Transformer- based STVG methods applying zero-initialized queries for STVG in (a) and our proposed Target- Aware Transformer-basee STVG generating queries with target-aware cues from video and text for STVG in (b). Best viewed in color for all figures.
Figure 1:
2022; Lin et al., 2023b; Gu et al., 2024a). Similar to Transformer-basee detection (Carion et al., 2020), existing Transformer-basee STVG methods often adopt set of spatial and temporal object queries, and then leverage them to learn target position information by iterative interactions with multimodal features (generated by the encoder) from video and text for spatio-temporal target localization. In these approaches (Yang et al., 2022a; Jin et al., 2022; Lin et al., 2023b; Gu et al., 2024a), the object queries are usually simply initialized using zeros (see Fig. (a)). Despite promising results, such vero-initialized object queries, due to the lack of effective target-specific semantic cues, are difficult to learn discriminative target position information from the interactions with nultimodal features in complicated scenarios, e.g., with distractors occlusion, leading to performance degradation.
In the decoder procedure, object queries expect to learn target position information from multimodal features. object queries know the target from the in other words, they know what to learn, they can employ target-specific cues as a prior guide themselves for better interaction with the multimodal features, which benefits learning more discriminative features for better localization. In order to validate this, we conduct the oracle ex- periment by generating spatial and temporal object queries from groundtruth, and compare the perfor- mance using such groundtruth genuutttee queries and zero-initialized queries. The architecture in this oracle experiment simply consists of feature extraction, an encoder and decoder, and please fer to the supplementary material for details due to limited space. Fig. 2 shows the comparison results on HCSTVG-v1 (Tang et al., 2021), and more comparison on other benchmarks can be found in Sec. A in the supplementary material. From Fig. we can clearly observe that, when introducing the target-specific information from groundtruth to initialize object queries, the STVG performance can be significantly improved, egg. from 49.9% 68.9% in m_ IoU (19.0% absolute gains) and 36.4% to 49.9% in m_ vlo (13.5% absolute gains), fully verifying effectiveness of target-sped sreeet- information in initializing queries for accurate STVG.
Oracle Experiment and Comparison on HCSTVG-v1 prsconseaato.................... www.eoateeeeee..................
100
Zero-India Guery
QT-Qererated Query
81.0
80
very beginning,
9
Accurscy (%)
80
40
32.1
20
IOU VloU
vicUg20.33
viou/go.6
Figure 2: Comparison of the aero-initialized queries and groundtruth- genuttteeeeeeeddd queries for STVG. We see the target-specifit information groundtruth largely enhances results.
Thus motivated by the above, we introduce a novel Target-Award Transformer for STVG (TA-STVG). which seeks to adaptively initialize object queries by exploring target-specific cues from videos for spatio-temporal target localization (see Fig. 1 (b)). The crux of our TA-STVG lies in two simple yet effective modules, including text-guided temporal sampling (TTS) and aftribute-aware spatial activation (ASA). These two modules work in a cascade in the TA-STVG, from coarse to fine, to generate the target-aware queries for subsequent localization. Specifically. TTS, from the temporal perspective, firstly aims to select target-relevant frames from the video using holistic information from the textual expression. To enhance the temporal selection, both appearance and motion cues are jointly considered in TTS. Then, from spatial perspective, ASA focuses exploring fine-grained
2

---

Published as a conference paper at ICLR 2025
visual semantic information of the object from the previous target-aware temporal cues learned by TTS. Particularly, attributes such as color and action will be exploited in the ASA to activate spatial features most relevant to the target object. Considering the specificity of object queries for spatial and temporal decoders, different activation features based on the specific attributes will be generated. Finally, features from ASA are used to initialize queries, which are fed to decoders, interacting with multimodal features from the encoder, to learn target position information for localization.
In this work, we employ the DETR-similar architecture for TA-STVG, similar to existing Transformer- based STVG methods (Yang et al., 2022a; Jin et al., 2022; Gu et al., 2024a), but difference is that, our object queries. for both spatial and temporal localization, are directly generated from the given video-text pair, and thus naturally carry the target-specifer cues, making them adaptive and better interact with multimodal features for learning more discriminative information to improve STVG. Our attribute-aware activation is related to Lin et al. (2023b) that activates the spatially attended target using intra-frame visual cues and Tan et al. (2024) that jointly learns feature aligoment and regression using weak supervision signals. The difference is that, our ASA mines target-relevant spatial cues from features by TTS for learning target-aware queries to enhance target localization. To our best knowledge, TA-STVG is the first to incorporate spegef-pecific cues for object query generation to improve STVG. To validate its efficacy, we conduct experiments on HCSTVG-vI/-v2 (Tang et al., 2021) and VidSTG (Zhang et al., 2020b), and results show that TA-STVG outperforms previous STVG methods by achieving new state-of-the-arts, evidencing effectiveness of our solution. Moreover, our TTS and ASA are designed for general purpose and thus applicable to other architectures. We apply TTS and ASA on top of two approaches, including the seminal Transformer- based TubeDETR (Yang al., 2022a) and STCAT (Jin et al., 2022), significantly improving their performance, as shown in our experiments, further highlighting the generality of our approach.
In summary, the major contributions of this work are as follows: We present the TA-STVG, a novel Tware Transformer for improving STVG by exploring target-specific cues for object queries; We propose text-guided temporal sampling (TTS) for selecting target-relevant temporal cues from the videos; + We present attribute-aware spatial activation (ASA) to exploit fine-grained visual semantic attribute information for object query generation; and TA-STVG in extensive experiments achieves new state-of-ther art performance and shows good generality, demonstrating its efficacy.
2 RELATED WORK
Tentooll Video Grounding. Spatio-temporal video grounding (STVG) (Zhang et al., 2020b) aims to spatially and temporally localize the target of interest, given its free-form of textual description, from an untrimmed sequence. Early STVG approaches (Zhang et al., 2020b;a; Tang et al., 2021) mainly consists of two stages, first generating candidate proposals from the video with a tre-trained detector and then selecting correct proposals based on the textual expression. To eliminate the heavy dependency on the pre-trained detection model, recent methods (Su et al., 2021; Yang et al., 2022a; Jin et al., 2022; Lin et al., 2023b; Talal Wasim al., 2024; Gu et al., 2024a), inspired by Transformer, switch to the one-stage pipeline that directly generates a spatio-temporal tube for target localization, without relying on any external detectors. Owing to the compact end-to-end training pipeline, such one-stage framework demonstrates superior performance compared to previous two-stage algorithms. Our TA-STVG also belongs to the one-stage Transformer-basee type. However, different from the aforementioned Transformer-based approaches that simply follows (Carion et al., 2020) to leverage zero-initialized object queries for target localization, TA-STVG innovatively exploits target-specific cues from the video-text pair for object query generation, making it adaptive to various scenarios and better interact with multimodal features in the decoder for more accurate localization.
Temporal Grounding, Temporal grounding focuses on localizing specific targets or events from the video given the textual expression. Being relevant to STVG, it requires temporally localizing the target of interest, but the difference is that temporal grounding does not require to perform the spatial bounding box localization. In recent years, many methods (Cao et al., 2022; Chen et al., 2021; Mun et al., 2020; Hao et al., 2022; Wang et al., 2023b; Zhang et al., 2023; Barrios et al., 2023; Lin et al., 2023a) have been proposed for temporal grounding. For instance, the work of (Wang et al., 2023b) introduces a pre-training approach for improving video temporal grounding. The method in (Lin et al., 2023a) presents a unified framework for various video temporal grounding tasks. The approach in (Chen al., 2021) )) proposes to learn complementary features from different modalities including
3

---

Published as a conference paper at ICLR 2025
Spatial Decodar Sec.
Bas Kbome
2D
Multimodal Feature Fusion See.
Text-guided Temporal Sampling (TSS) in Sec. 3,2
+Ra
Atribure oo swwure Spatial Activation
A4
AvePoollin
Repeai
20
Spasial
Head
(AS.A) in Sec. 3.3
Video
Backhone
DoC Concat
Inital Spariol
3D
-
Queries
green clothes
walks
man
ArgPooling
laiial Tempond
Rupeon
Qusties
in Svc. 3,4
Temponi Decoder
Tube
the man in
brown
Euuckbome
Text
m
RR mn
A m
20
Temporal Head
Text
Input
Multimedal Encoder
Target-aware Query Generution
Decoder
Output
Figure 3: Overview of TA-STVG, which exploits target-specific information from the video and text (i.e,, features from multimodal encoder) for generating spatial and temporal object queries for STVG.
images, flow, and depth for temporal grounding. Different than these works, focus on the more challenging STVG involved with spatial and temporal localization of the target.
Transformer-basee Detection. Detection is fundamental component in computer vision. Recently, the seminal work DETR (Carion et al., 2020) has applied Transformer (Vaswani et al., 2017) for detection with impressive performance, and later been further improved in numerous extensions (Sun al., 2021; Zhu et al., 2021; Zheng et al., 2023; Ye et al., 2023), Similar to other Transformer-based STVG works, our method employs the DETR-similas architecture for STVG. The difference is that propose to learn adaptive object queries from the video-text pair, instead of utilizing zero object queries following (Carion et al., 2020) as in existing STVG methods, for better target localization.
Vision-Language Modeling, Vision-language modeling (VLM) aims process both visual content and language for multimodal understanding. In recent years, it has attracted great attention from the researchers and been studied in various tasks including visual question answering (Antol et al., 2015; Jiang et al., 2020; Han et al., 2023; Shao et al., 2023; Wang et al., 2024b; Weng et al., 2024), visual captioning (You al., 2016; Gu et al., 2023; 2022; Wang et al., 2024a; Shen al., 2023; Ren et al., 2024; Wang et al., 2023a), navigation (Zhu et al., 2020; Li & Bansal, 2023). text-to-image generation (Gu et al., 2024b; Ramesh al., 2021), referring expression segmentation (Yang et al., 2022b; Liu et al., 2023), vision-language tracking (Guo et al., 2022; Zhou et al., 2023), etc. Different from these tasks, we focus on vision-language modeling for spatio-temporss video grounding.
OUR APPROACH
Overview. In this work, we propose TA-STVG, which aims to generate target-aware object queries for improving STVG. Motivated by DETR (Carion et al., 2020), TA-STVG utilizes the encoder-decoder framework which consists of a multimodal encoder and a spatio-temporal grounding decoder. As shown in Fig. 3, the multimodal encoder ($3.1) extracts and interacts visual and textual features, while the decoder ($3.4) learns target position information using target-awun object queries generated with TTS ($3.2) and ASA ($3.3) and their interaction with the multimodal feature for target localization.
3.1 MULTIMODAL ENCODER
Given video and the textual expression, the multimodal encoder aims at obtaining vision- -siion-lagguage features for target-aware query generation and the subsequent decoder for localization. comprises visual and textual feature extraction and multimodal feature fusion as described in the following.
Visual and Textual Feature Extraction. For the video, we extract its both appearance and motion features to leverage rich static and dynamic information. Specifically, we first sample N,, frames I = (, from the video. Afterwards, ResNet-10f 101 (He et al., 2016) and VidSwin (Liu et al., 2022) are respectively utilized for appearance and motion feature extraction. The appearance feature is denoted F. {fr N. where f E Hx XC. with H, W and C, the height, width and channel [
dimensions, and the motion feature is represented using Fm = {f Ji=1 where f." E R MXX with C., the channel dimension.
4

---

Published as a conference paper at ICLR 2025
MLP
Ma
volla
CAUx2
inner
lays
CADDE
Chroeficatti Look
Subyast Extraction
Moti-labet
AYE
CADx2
CABx2
Finerr
Ra
Truning Onty
(a) Text-guided Temporal Sampling (TTS)
(b) Attribute-aware Spatial Activation (ASA)
Figure 4: Illustration of the architecture for TTS in (a) and ASA in (b).
For the textual expression, we use RoBERTa (Liu et al., 2019) to extract its feature. We first tokenize it to obtain a word sequence W = [w:;)] } containing N, words. Then, we apply RoBERTa on W to generate the textual feature F, = (ff fi where f! E R with C, the textual feature channel.
Multimodal Feature Fusion. For enhancing feature representation for STVG, we fuse multimodal features of appearance feature Fa. motion feature Fm, and text feature Fr, similar to (Gu et al., 2024a). Specifically, we first project them to the same channel number 1) and then concatenate corresponding features to produce the multimodal feature = (fi) as follows,
J Si Jinxw> Lig J12 ..... Jinxw> S. Ji.
appearance Reatures 12
motion features 1!
textual fentures Fi
where fi is the multimodal feature in frame i. Then, we perform feature fusion using the self-attention encoder (Vaswani et al., 2017)<<<>e to obtain the multimodal feature F as follows,
f = Sel1fAtEEncoder(1 + Epos + Etyp)
where Epos and Egyp denote position and type embeddings imposed on F, and SelfAttEncoder. the self-attention encoder with L (L=6) standard self-attention encoder blocks. Please refer to Sec. B in supplementary material for architecture of SelfAttEncoder())
3.2 TEXT-GUIDED TEMPORAL SAMPLING (TTS)
To generate target-specific object queries, we first develop a texl-guided temporal sampling (TTS) module to identify and sample frames relevant to the target guided by holistic textual features. Specifically, will predict a relevance score between each frame and text and then sample video frames with their relevance scores above a predefined threshold. To enhance the frame selection in TTS, both appearance and motion are considered in computing the relevance, as shown in Fig. 4 (a).
More concretely, given multimodal feature f from the multimodal encoder (53.1), we first extract the appearance, motion and textual features, respectively denoted by F... Fr and F,, from F via deconcatenation [F, fm, F.] DeConcat Afterwards, we perform average pooling on F. and Fm and channcl-average operation on F, to obtain frame-level appearance and motion features and the holistic textual feature.
Ge = AvgPooling(Fa)
Gm = AvgPooling(
G. Avg(FF:)
where Ga. 9m (both N. and 9, (E RIxD) are averaged appearance, motion, and text features.
To measure the relevance of each frame with the text, fuse the textual feature into appearance and motion features via two cross-attention blocks, respectively. Specifically, the appearance feature G. or motion feature Gm serves query, while the textual feature G, as key/value. After this, MLP is performed on the appearance and motion features to predict relevance scores for each frame in terms of appearance and motion. As shown in Fig, 4 (a), this process can be expressed as follows.
Sa MLP(CA(CAAG.... G.),
8m MLP(CA G.))
5

---

Published as a conference paper at ICLR 2025
(a) Temporal relevance score predicted by TTS (red rectangle: groundtruth; pink rectangles: sampled frames)
Text: The garl with yellow hatr walks and stops.
(b) Appearance ("yellow") activation Ma by ASA
Motion ("walkss" und "stops") activation Min by ASA
Figure 5: Illustration of temporal relevance score by TTS in (a) and attribution -wwrr spatial activation (in partial selected frames) including appearance and motion activation in (b) and (c). We from (a) that TTS can accurately select target-relevant frames, and from (b) and (c) that ASA precisely localizes attributes, e.g., color "yellow" and action "walks in, stops", related to the target.
where Su and S. are the relevance scores regarding appearance and motion, CA(z, u) denotes the cross-attention block (CAB) (Vaswani et al., 2017) with z generating query and key/value, and M.P(.) is the MLP.
Considering both appearance and motion are critical when determining target-relevant frames, we combine and Sm, controlled by 8, to compute the final relevance score s for each frame, as follows,
= x Su + (1 8) x 8m
(5)
With we leverage it to sample target-relevant appearance and motion features. Specifically, we only keep appearance and motion features in frames whose relevance scores are above the pre-defined threshold 0, and this process can be written as follows,
Ra sample s, (F.())s()) 0}
Rm sample 0, = (F.h (1)((1) > 0}
where i E [1. NN] is the frame index. Ra and R. are sampled target-relevant temporal appearance and motion features, which will be used for extracting target-specific fine-grained attribute information. Fig. 5 (a) shows an example of the temporal relevance score and selected temporal frames by TTS. Please note, for R. and Rm, their quantities are the same, because we the same threshold 0 for both appearance and motion features, and sample R., and RRm from Fa and F.,. (F, and Fm have the same quantities) both based on the relevance score generated by TTS (see Eq. (6)). Thus, their quantities are the same, and consistent with the number of frames sampled by TTS.
3.3 ATTRIBUTE-AWARE SPATIAL ACTIVATION (ASA)
To explore more
information of the target, design attribute spatial activation (ASA) module, which mines fine-grained visual semantic information from previous coarse-grained temporal features R, and Rm for object queries. Particularly, ASA considers appearance attribute like color (e.g., "green" "red", etc) and shape (e.g., "small", "tall", etc) and motion attribute such as action (e.g., "walk", "ride". etc) to generate fine- tine-grained spatial features for the target. The intuition is that, appearance attribute is important for spatial target localization, while motion attribute is crucial in temporal localization, i.e., locating the start and end the tube. In certain cases, even spatial localization of target is good enough in each frame, the accuracy is low due to the incapability of temporally recognizing target action. Considering the specificity of queries for spatial and temporal decoders, appearance and motion activation features based on specific attributes are generated.
Specifically, as shown in Fig. 4 (b), given the textual feature F., we first extract the feature F, of the subject (usually the target to localize) in the textual sentence from it. Due to limited space, please refer to the Sec. D in supplementary material for subject extraction. Then, feature T; to match the number of frames in R, and Rm, and the resulted feature is denoted as T, After that, we learn the fine fine-grain -eee-grrnndd spatial features from R, and Rm by interacting them with Ti respectively using two cross-attention blocks. In specifie, the subject feature Te serves as the query, while R, and Rn as the key/value. By doing so, we can allow the subject feature to learn relevant spatial features from appearance and motion. To enforce the learning of attribute-aware
repeat the subject

---

Published as a conference paper at ICLR 2025
Text:
The man in yellow clothes opens the door
Attention maps of zero-initialization queries
Attention maps of our target-aware queries
Figure 6: Comparison of attention maps for zero pro-iinitialized object queries and the proposed target- aware object queries in video frames in the spatial decoder. The red boxes indicate the foreground target to localize. From this figure, we can clearly observe that the attention maps of our target-aware object queries by TTS and ASA can better focus the target object for localization.
spatial features, we propose to supervise ASA with explici! weak attribute labels generated from the textual expression using multi-label classification losses for appearance and motion. The appearance and motion multi-class labels. Due to space limitation, material for details regrading the attribute label construction. The attribute classification is performed with linear projection layer followed with softmax. As in Fig. (b), this can be written as follows,
refer readers to supplementary
Ca = Sof maa
Ra), R₂)))
Cy Sof tmax (max(LLnear((AA((AA) Rm), Rm)))
where C2 and Cm represent classification scores for appearance and motion attributes, respectively, and Linear(-) and Softmax(.) denote linear projection and softmax classification.
By supervising ca and Cm with appearance and motion attribute labels, the subject feature Tr serving query in cross-attention blocks, is able to adaptively learn fine-grained features relevant to attributes from R. and R. for attribute classification. Therefore, the attention maps of Te with R. and R. can be explained as attribute-specific spatial activation in appearance and motion (Dosovitskiy et al., 2021) 1), and adopted to generate attribute features from R. and Rm as follows,
Aa -Ma & R.
Am Mm & Rm
where Au and Am are the target-specific attribute features and Ma and Mm the attribute-aware spatial activation in appearance and motion. Fig. (b) and (c) show the learned attribute activation.
Discussion. Please note that, besides our method, another way is to directly learn spatial activation of the target (i.e., instance-level activation, not the attribute-specific activation) supervised by binary masks generated from the groundtruth box. In this method, results after softmax in ASA will be directly employed as the appearance and motion spatial activation for feature extraction (see details in Sec. C in supplementary material). Despite simplicity, this will not enable the learning of fine-graa fine-grained unique attribute features, which are discriminative to distinguish the target from background. As shown in experiment later, strategy of ASA shows better performance.
features A, and An used to generate the spatial and temporal object queries. Specifically, we simply use average pooling on An and An (note, reshaping is needed) and then apply the repeat operation to obtain the initial spatial and temporal object queries Qj and O follows,
Generating Object Queries. After the TTS and ASA, the final fine- mme-igained
S repeat (yppeattAvgPooinng(AA..
S = repeat repeat (AngPolingg(AA))))..... N.)
(9)
where repeat (v, k)) denotes repeating by k times. Afterwards, Qi and 2 are sent the decoder for spatial- temporal target localization as described later.
3.4 DECODER FOR
LOCALIZATION
To learn target position information, we employ spatial and temporal decoders, similar to existing methods (Jin et al., 2022; Gu et al., 2024a). In both decoders, object queries Oj and Ø will iteratively interact with the multimodal feature F from the encoder for target information learning, as follows,
2j SpatialDecoder( F..
Ø'x TemporalDeccoerr((( F.)
(10)
where 0j and Qx are learned spatial and temporal queries after decoders, and Spati (eaatderee)))) and Tempora)Deccoerr(( denote spatial and temporal decoders with each containing K (K -)
7

---

Published as a conference paper at ICLR 2025
self-attention blocks and cross- coss-attention blocks. Fig. 6 shows the attention maps of target- aware queries for localization. Due to limited space, please refer to the Sec. B in supplementary material for decoder architectures and additional attention maps of our target-aware queries for localization.
Once generating 2i and O we employ two heads to predict the final object boxes B = ={b;) {b,),-m where b, E R represents central position, width and height of predicted box in frame i, and start and end probabilities of each frame H = {h,, ht)] N. where start and end times are determined by the maximum joint start and end probability.
3.5 OPTIMIZATION
In TA-STVG, we predict: (1) the temporal relevance scores sa and Sm in terms of appearance and motion in TTS. (2) the appearance and motion attribution classification scores C, and Cm in ASA, and (3) the spatial boxes B (b: ) in the spatial decoder, start timestamps Hy {hξ ) and end timestamps Hig = [hf ==11 in the temporal decoder. During training, with the groundtruth of motion attribute c. appearance attribute c start timestamps H. end timestamps H. and the bounding box B", we can calculate the total loss L as
L ÀTTS (Sace(Sa, (H,, ?C.) + Lacu(sm, (())) + AASA (SBCE Cm ) + Laculcas Co))
loss
luss of ASA
(11)
+ Ax (Ht., H.) + Cxt (Hi, Ho)) AB B) + AL
temporal decoder
sputial
where CKL, L1, Lio and LBCE are KL divergence, smooth LI, IoU, and binary cross-entropy losses.
4 EXPERIMENTS
Implementation. TA-STVG is implemented in Python using PyTorch (Paszke et al., 2019). Similar to (Gu al., 2024a), we use pre-trained ResNet-101 (He et al., 2016) and RoBERTa-base (Liu et al., 2019) from MDETR (Kamath et al., 2021) as 2D and text backbones, and VidSwin-ting tiny (Liu et al., 2022) as 3D backbone. The number of attention heads 8, and the hidden dimension of the encoder and decoder is 256. The channel dimensions C., Cm. C. D are 2,048, 768, 768 and 256. 5 and 0 are empirically set to 0.5 and 0.7. We use random resized cropping as augmentation method, producing an output with short side of 420. The video frame length N, is 64, and the text sequence length Ne is 30. During training. we use Adam (Kingma & Ba, 2015) with an initial learning rate of 2e-5 for the pre-trained backbone and 3e-4 for the remaining modules (note that the 3D backbone is frozen). The loss weight parameters ArTS, AASA, Ae. As, and A are set to 1, 1, 10, 5, and 3, respectively.
Datasets and Metrics. We use three datasets, Le., HCSTVG-v1/v2 (Tang al., 2021) and Vid- STG (Zhang et al., 2020b), for experiments. HCSTVG-vI contains 5,660 untrimmed videos, with 4,500 and 1,0 video-text pairs training and testing sets. HCSTVG-v2 expands upon HCSTVG-v1, and comprises 10,131 training. 2,000 validation, and 4,413 33 testing samples. As test set annotations are available, results are reported based the validation et al., 2022; Yang et al.. 2022a; Lin et al., 2023b). VidSTG contains 99,943 video-text pairs. It includes 44,808 declarative and 55,135 interrogative sentences, with training, validation, and test sets containing 80,684, 8,956, and 10,303 sentences, and 5,436, 602, and 732 videos, respectively.
HCSTVG-v2 in other methods (Jin
Following (Yang et al., 2022a), we employ m_ m_tloU, m_ vIoU, and vloU@ R as the evaluation metrics. m_ tloU measures the ability of temporal grounding by computing the average tloU score across all the test set. m_ v compares the spatial grounding performance by calculating the average of vloU scores, and vloU @ measures the performance using ratios of samples with vloU greater than R in test sets. For detailed metrics, please kindly refer to (Yang et al., 2022a).
4.1 STATE-OF-THE ART COMPARISON
HCSTVG Datasets. To validate the efficacy of our method, we compare it against other approaches on HCSTVG-vI/v datasets. Tab. shows the comparison results on the HCSTVG-v1 test set. As shown in Tab. 1, TA-STVG achieves ofate-ofthhe-at performance on the 1.3 metric, where it improves the score by 1.6% absolute gains compared to CG- STVG (Gu al., 2024a). Compared to our baseline that does not employ TTS and ASA modules for
all four metrics, especially
8

---

Published as a conference paper at ICLR 2025
Table 1: Comparison on HCSTVG-vl (%).
Table 2: Comparison on HCSTVG-v2 (%).
Methob
RL
Nelbol:
Violl
SYVelee (001)
30- 2222)
20023
(Wang atal,
55.9
STNGFomer al.
34.5
STVGRoeer (l al, 202313
56.1
56.7
05.5
35.6
CRSSTV (faee al, 2014)
47.8
18.
16.3
CG-TVVI (farrtal 2030a)
00.0
10.5
56.5
16.1
Bracite
576
321
Badlae (AB))
58.3
57.
125
35.0
14-STVG (eam)
3811
36.8
Ts VT-STVG (oas)
664
402
658-412
Table 3: Comparison with existing
methods on VidSTG (%).
Mechods
m. 18-21
Dechrative Semsrnos
vlol]
vlel:/00
m. vioti
Internaaatve Sentenoes
vlal180.3
vlall8O.
STORN (Zhong 2036)
43.5
19.8
25.8
14.6
47.0
18.3
21.1
12.8
OMRN (7hang a al. 2020a)
50.7
23.1
12.6
16.4
49.2
20.6
23.4
14.1
STGVT (Tangettl, al, 2021)
21.6
29.8
18.9
STVGBen al. 2011)
24.00
18.4
22.5
16.0
TubeETTR (Yang ctal, 2022a)
30.4
28.2
25.7
23.2
STCAT (Jin et al.
50.8
05.2
32.6
19.7
28.2
39.2
25.6
SGFDN (Wang atal. 2023g
45.1
28.3
41.7
29.
44.8
25.8
36,9
21.9
STVIIFomer in et al, 2023b)
33.7
47.3
92.8
78.5
30.9
26.2
CG-STV
27.5
Baseline (oun)
4955
52.3
46.9
3166
48.5
27.5
25.6
TA-STVG (nun)
51.7
34.4
48.2
(15.))
X35
50.2
(1.50
29.5
41.3.
28.0
target-specific query generation and instead uses zero-initialized object queries, TA-STVG shows absolute performance gains of 3.1%, 2.7%, 5.5%, and 4.7% on the four metrics, which evidences the effectiveness of our method. Tab. 2 shows the result on HCSTVG-v2 validation set. As shown, TA-STVG achieves SOTA results on all four metrics. Compared to the baseline, TA-STVG improves it with 2.1%, 16, 2.3%, 3.2%, and 3.7% absolute gains on four metrics, validating its efficacy again.
VidSTG Dataset. We evaluate TA-STVG on VidSTG in Tab. 3. Different from HCSTVG, VidSTG includes both declarative and interrogative sentences. As in Tab. 3, TA-STVG achieves state-of-the-ar results on all 8 metrics, using target-aware queries, TA-STVG significantly outperforms the baseline, improving m_tloU and m_vIoU scores by 2.2% and 2. for declarative sentences, and 1.6% and 2.0% for interrogative sentences, respectively, once again validating the effectiveness of TA-STVG.
4.2 ABLATION STUDY
Impact of TTS and ASA. The TTS and ASA are key components of TA-STVG. To vali- date their effectiveness, we conducted ablation experiments on HCSTVG-v1. As shown in Tab. 4, without TTS and ASA (0), our base- line achieves a m _ score of 49.9%. When TTS is applied alone, the m_ tIo score is im- proved 52.2% with 2.3% gains (0 v.s. @). When ASA is used alone, together, the best result of 53.0% is achieved with 3.1 % absolute gains 1 v.s. 2). All these results clearly validate the efficacy of TTS and ASA, working either alone together, for improving STVG.
Table 4: Ablations of TTS and ASA.
TTS ASA m tlo m vloU vloU@0.3 vloU @0.5
49.9
36.4
57.6
32.1
0
52.2
38.4
61.7
36.2
0
51.4
38.0
60.4
34.1
0 /
53.0
39.1
63.1
36.8
tloU is improved to 51.4% with 1.5% gains v.s. 3). Working
Impact of different branches in TTS. In TTS, we consider both appearance and motion, guided by text, for temporal selection. To further analyze TTS, we conduct experiments on its different choices on HCSTVG-v1. The results are reported in Tab. 5. As in Tab. 5, without using TTS (but using ASA), TA-STVG achieves 51.4% m .loU score ). With the help of text-guided appearance branch, the m_ tot score is improved to 51.8% with 0.4% gains
v.s. @). When using text- gext-d motion branch, the m. tloU score is improved to 52.3% with 0.9% gains Y.s. 3). When considering both appearance and motion, the m_ totloU score is boosted to 53.0% with 1.6% gains (O v.S. showing the best performance. It is worth noting that, if not using text as guidance, the m tIoU score will be decreased from 53.0% to 51.8% 1 v.s. θ) validating the importance of textual guidance in TTS.
Impact of different attributes in ASA. In ASA, we consider both appearance and motion attributes for fine-grained feature learning. For in-depth analysis. we conduct ablations for ASA on HCSTVG- v1. As in Tab. 6, without using ASA (but using TTS), TA-STVG achieves 52.2% m tloU score (0). When considering appearance attribute, m tloU is improved to 52.3% (0 @). When using motion attribute, the m tloU score is improved to 22.7% (0 v.s. ①) When applying both appearance and motion attributes, best result of 53.0% m_ n_tloU score is achieved with 0.8% gains (0 Y.s. 1)

---

Published as a conference paper at ICLR 2025
Table 5: Ablations of branches in TTS. "TG", Table 6: Ablations of attributes in ASA. "SG", "AB", and "MB" are the text text-guided, appearance "AA", and "MA" are the subject-guided, appear- and motion branches, respectively.
ance and motion attributes, respectively.
TG AB MB m tlo m_ vLol v.. vl0
SG AA MA m tloU m. vloU
v.0 vloU@0.5
51.4
38,0
60.4
34.1
52.2
38.4
61.7
36.2
/
51.8
38.4
61.1
36.3
/
52.3
38.6
62.4
36.2
/
52.3
38.3
62.0
36.5
v
52.7
38.6
61.3
36.6
51.8
38.5
62.0
36.9
/ 52.6
38.8
61.9
36.8
53.0
39.1
63.1
36.8
/
53.0
39.1
63.1
36.8
Table 7: Ablations activation learning in ASA.
Table 8: Ablation of 8.
Table 9: Ablation of 0.
Activation m_tloU m_ vloU v.. vloU@0.5
5
m
IloU
m vloU
0
tloU
vloU
None 52.2
38.4
61.7
36.2
0.4
52.7
38.8
1 0.6
52.3
38.5
Instance-act. 52.8
38.8
62.0
34.7
0.5
53.0
39.1
0.7
53.0
39.1
L Attribute-act. 53.0
39.1
63.1
36.8
0.6
52.8
39.0
3 0.8
52.6
38.6
Please note, the gain by jointly using appearance and motion attributes is the maximum, which shows the necessity of both appearance and motion attributes for improvement. Similarly, when subject is not used, m__ toollU is decreased from 53.0% to 52.6% ( v.s. 1) showing the need of subject in ASA.
Analysis of activation learning strategy in ASA. In ASA, we propose learning attribute- aware spatial activation for fine-grih fine-grained target features. Another way is to learn the naive instance- activation (not attribute-specifii activation) supervised by binary masks generated from the groundtruth boxes as mentioned earlier (see these two strategies in Sec. C in supplementary material). We conduct experiments in Tab. 7. As shown, the attrifute-ppecific features (0) in our strategy shows better result compared to instance-level spatial activation (0) or none activation (i.e., removing ASA) (0).
Impact of parameter 8 and f in TTS. In TTS, 5 controls the fusion of appearance and motion scores, and controls the sampling of video frames. We conduct ablations in Tab. and Tab. 9. We can see that the model performs best by setting 8 to 0.5 (@ in Tab. 8) and 0 to 0.7 @@ in Tab. 9).
Due to limited space. please refer our supplementary material for more experiments and analysis.
4.3 VALIDATION OF GENERALITY
To validate the generality of our approach, conduct experiments by applying TTS and ASA to other Transformer- cluding TubeDETR and STCAT. Since these two methods only use appearance features, we retain only the appearance branch to accom- modate them. Tab. 10 reports the results on HCSTVG-v1. Please note, we retrain Tube- DETR and STCAT on our platform for fair comparison. Although we try our best to re-implement these methods with provided code and settings, our results differ slightly from the original paper with small discrepancy. As Tab. 10, we observe method efffectively and consistently improves TubeDETR and STCAT, achieving gains of 2..%% 1.9% @ v.s. 3 and 1.7%/1.8%
Table 10: Incorporate the TTS and ASA modules into different methods HCSTVG-vl (%). + results from the original paper. c retrained results.
methods, in-
Method
mLdOU
m_yieU
vieU
41.
12.4
40.3
25.5
TobaDoRR Tub:DETR*
49.1
25.5
455
335
53.0
27.8
STCAT
254
35.1
37.7
301
STCAT*
45.3
34,9
57.2
29,8
STCAT*
367
39.9.075
(1.7
v.s. b) in m tloU/m VIO,, evidencing its generality.
CONCLUSION
This paper proposes a novel Twage Transformer for STVG (TA-STVG) by exploiting target- specific cues for query generation. The key lies in two effective modules of TTS and ASA. TTS aims to sample target-relevant temporal features using text as a guidance, while ASA explores fine- fine-grained spatial features from grained temporal features. Working in a cascade, they allow exploiting crucial target-specific cues for generating target-aware object queries, which learn better target position information for STVG. Experiments on three datasets validate the efficacy of TA-STVG.
Acknowledgements. Libo Zhang is supported by National Natural Science Foundation of China (No. 62476266). Yan Huang, Yuewei Lin, and Heng Fan not supported by any funds for this work.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In ICCV, 2015.
Wayner Barrios, Mattia Soldan, Fabian Caba Heilbron, Alberto Mario Reballos-Arroyy and Bernard Ghanem. Localizing moments in long video via multimodal guidance. In ICCV, 2023.
Meng Cao. Tianyu Yang. Junwu Weng, Can Zhang, Jue Wang, and Yuexian Zou. Loevip: Video-text pre-training for temporal localization. In ECCV, 2022.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
Yi-Wen Chen, Yi-Hsuan Tsai, and Ming- Hasugn Yang. End-to-end multi-modal video temporal grounding. In NeurlPS, 2021.
Alexey Dasovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbomn Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2021.
Xin Gu, Hanhua Ye, Guang Chen, Yufei Wang. Libo Zhang, and Longyin Wen. Dual-stream transformer for generic event boundary captioning. arXiv preprint arXiv:2207.03033 2022.
Xin Gu, Guang Chen, Yufei Wang. Libo Zhang. Tiejian Luo, and Longyin Wen. Text with knowledge graph augmented transformer for video captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18941 18951,, 2023.
Xin Gu, Heng Fan. Yan Huang, Tiejian Luo, and Libo Zhang. Context-guided spatio-temporal video grounding. In CVPR, 2024a.
Xin Gu, Ming Li, Libo Zhang, Fan Chen, Longyin Wen, Tiejian Luo,, and Sijie Zhu. Multi-reward condition for instruction-basee image editing. CoRR, 0bs/2411..04113 2024b.
Mingzhe Guo. Zhipeng Zhang, Heng Fan, and Liping Jing. Divert more attention to vision vision- language tracking. NewrlPS, 2022.
Mingfei Han, Linjie Yang, Xiaojun Chang, and Heng Wang. Shot2story20k: A new benchmark for comprehen- sive understanding of multi-shot vidcos. arXiv preprint arXiv:2312. 10300000,, 2023.
Jiachang Hao, Haifeng Sun, Pengfei Ren, Jingyu Wang. Qi Qi. and Jianxin Liao. Can shuffling video benefit temporal bias problem: novel training framework for temporal grounding. In ECCV, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Huaizu Jiang, Ishan Misra, Marcus Rohrbach, Erik Leamed-Miller, and Xinlei Chen, defense of grid features for visual question answering. In CVPR, 2020.
Yang Jin, Yongzhi Li. Zehuan Yuan, and Yadong Mu. Embracing consistency: A one-stage approach for spatio-temporal video grounding. In NewrlPS, 2022.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr- modulated detection for end-to-end multi-modal understanding. In ICCV, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Jialu Li and Mohit Bansal. Improving vision-and-language navigation by generating future-view image semanties. In CVPR, 2023.
Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shraman Pramanick, Difei Gao, Alex Jinpeng Wang, Rui Yan, and Mike Zheng Shou. Univtg: Towards unified vidco-language temporal grounding. In CVPR, 2023a.
Zihang Lin, Chaolei Tan, Jian-Fang Hu, Zhi Jin, Tiancai Ye, and Wei-Shi Zheng, Collaborative static and dynamic vision-language streams for spatio-temporal video grounding. In CVPR, 2023b.
Chang Liu, Henghui Ding, and Xudong Jiang. Gres: Generalized referring expression segmentation. In CVPR, 2023.
11

---

Published as a conference paper at ICLR 2025
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Dangi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, arXiv preprint 1907. 1692, 2019.
7e Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In CVPR, 2022.
Jonghwan Mun, Minsu Cho, and Bohyung Han. Local-global video-text interactions for temporal grounding. In CVPR, 2020.
Adam Paszke. Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan. Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-perffrmance deep learning lihrary. NeurlPS, 2019.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Iyya Sutskever. Zero-shot texi-to-image generation. In ICML, 2021.
Zhongwei Ren, Zhicheng Huang. Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer 2024.
Vision and Puttern Recognition, pp. 26374-2638 2024.
Zhenwei Shao, Zhou Yu, Meng Wang, and Jun Yu. Prompting large language models with answer heuristics for knowledge-basee visual question answering. In CVPR, 2023.
Yaojic Shen, Xin Gu, Kai Xu, Heng Fan, Longyin Wen, and Libo Zhang. Accurate and fast compressed vidco captioning. In ICCV, 2023.
Rui Su, Qian Yu, and Dong Xu. Styghert: visual-linguistic transformer based framework for spatio-temporal video grounding. ICCV, 2021.
Zhiqing Sun, Shengcao Cao, Yiming Yang. and Kris M Kitani. Rethinking transformer- based set prediction for object detection. In ICCV, 2021.
Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming- Huunn Yang, and Fahad Shahbaz Khan, Video- groundingdino: Towards open opea-vocabulary
video grounding. In CVPR, 2024.
Chaolci Tan, Zihang Lin, Jian-Fang Hu, Xiang Li, and Wci-Shi Zheng. Augmented 2d-tan: A two-stage approach for human-centric spatio-temporal video grounding. arXiv, 2021.
Chaolei Tan, Jianhuang Lai, Wei-Shi Zheng, and Jian-Fang Hu, Siamese learning with joint aligmment and regression for suaeiy-s video paragraph grounding. In CVPR, 2024.
Zongheng Tang, Yue Liao, Si Liu, Guanbin Li, Xiaojie Jin, Hongxu Jiang, Qian Yu, and Dong Xu. Human-centric spatio-temporal video grounding with visual transfomerrs IEEE TCSVT, 32(12)88238888 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, 2017.
Haochen Wang, Kaiyou Song, Junsong Fan, Yuxi Wang. Jin Xie, and Zhaoxiang Zhang. Hard patches mining for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10385- 2023a.
Haochen Wang, Junsong Fan, Yuxi Wang, Kaiyou Song. Tong Wang, and ZHAO XIANG ZHANG. Droppos: Pre-training vision transformers reconstructing dropped positions. Advances in Neural Information Processing Systems, 36, 2024a.
Haochen Wang, Anlin Zheng, Yucheng Zhao, Tiancai Wang, Zheng Ge, Xiangyu Zhang, and Zhaoxiang Zhang. Reconstructive visual instruction tuning. arXiv prepriat arXiv:2410.09575, 2024b.
Lan Wang, Gaurav Mittal, Sandra Sajeev, Ye Yu, Matthew Hall, Vishnu Naresh Boddeti, and Mei Chen. Protege: Untrimmed pretraining for video temporal grounding by video temporal grounding. In CVPR, 2023b.
Weikang Wang, Jing Liu, Yuting Su, and Weizhi Nie. Efficient spatio-temporal video grounding with semantic- guided feature decomposition. In Proceedings of the 31 ACM International Conference on Multimedia, pp. 4867-4876, 2023c.
Zhenzhi Wang, Limin Wang, Tao Wu, Tianhao Li, and Gangshan Wu. Negative sample matters: A renaissance of metrie learning for temporal grounding. In AAAI, 2022.
12

---

Published as a conference paper at ICLR 2025
Yuetian Weng, Mingfei Han, Haoyu He, Xiaojun Chang, and Bohan Zhuang. Longvlm: Efficient long video understanding large language models. European Conference on Computer Vision, pp. 453-470. Springer, 2024.
Antoine Yang, Antoine Miech, Josef Sivic. Ivan Laptey, and Cordelia Schmid. Tubedetr: Spatio-temporal video grounding with transformers. CVPR, 2022a.
Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr. Lavt: Language-awar vision transformer for referring image segmentation. In CVPR, 2022b.
Mingqiao Ye, Ke, Siyuan Li, Wing Chi-Keung Tang. Martin Danelljan, and Fisher Yu. Cascade-detr: Delving into high-quality universal object detection. /CCV, 2023.
Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo, Image captioning with semantic attention. In CVPR, 2016.
Yi Yu, Xinying Wang, Wei Hu, Xun Luo, and Cheng Li. 2rd place solutions in the he-stvg track person in context challenge 2021. arXiv, 2021.
Yimeng Zhang, Xin Chen, Jinghan Jia, Sijia Liu, Ke Ding. Text-visual prompting for efficient 2d temporal video grounding. In CVPR, 2023.
Zhu Zhang, Zhou Zhao, Zhijie Lin, Baoxing Huai, and Jing Yuan. Object-aware multi-branch relation networks for spatio- temporal video grounding. In IJCAI, 2020a.
Zhu Zhang, Zhou Zhao, Yang Zhao. Qi Wang, Huasheng Liu, and Lianli Gao. Where does it exist: Spatio- temporal video grounding for multi-form sentences. In CVPR, 2020b.
Dehua Zheng, Wenhui Dong, Hailin Hu, Xinghao Chen, and Yunhe Wang. Less more: Focus attention for efficient detr. In ICCV, 2023.
Zhou, Zikun Zhou, Kaige Mao, and Zhenyu He. Joint visual grounding and tracking with natural language specification. In CVPR, 2023.
Fengda Zhu, Zhu, Xiaojun Chang, and Xiaodan Liang. Vision-language navigation with self-supervised auxiliary reasoning tasks. In CVPR, 2020.
Xizhou Zhu, Weijie Su, Lewei La, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection, In ICLR, 2021.
13

---

Published as a conference paper at ICLR 2025
SUPPLEMENTAL MATERIAL
For better understanding of this work, we offer additional details, analysis, and results as follow:
A Details and More Results for the Oracle Experiments
In this section, display more detailed architecture for the Oracle experiment and report results on more datasets.
В Architectures for Self-attention Encoder, Spatial and Temporal Decoders In this section, we show the details for the architecture of the self-attention encoder used for multimodal feature fusion. In addition, display the architectures of spatial and temporal decoders.
C Different Architectures of Activation Learning Strategies In this section, present details of architectures for activation learning strategies in ASA.
D Text Pre-processing and Attribute Label Construction This section will illustrate the necessary text pre-processing for subject extraction, In addition, we will detail the construction of vocabulary for attribute labels.
E Clarification for Multi-label Classification in ASA In this section, we offer clarification for the multi-label classification in the ASA module.
F Additional Ablation Experiments In this section, we show additional ablation experiments on different loss weights and the robustness of our method.
G Additional Discussion
this section, we provided additional discussions on our method.
Н Efficiency and Complexity Analysis
In this section, we provide analysis of efficiency and model complexity for our proposed method and its comparisons with other state-of-the- approaches.
Qualitative Analysis and Results
In this section, we display qualitative analysis including relevance score learned by TTS, attribute-specife spatial activation learned by ASA, attention maps of object queries in the decoder, and qualitative results for mpatial-tempore target localization.
Limitation
We discuss the limitation of our method.
A DETAILS AND MORE RESULTS FOR THE ORACLE EXPERIMENTS
To verify whether target-aware queries benefit localization, we conduct an Oracle experiment by generating spatial and temporal object queries from the ground-truth boxes. Similar to our TA-STVG, the architecture this Oracle experiment consists of multimodal encoder, query generation, and decoder, shown in Fig. 7. The difference with our method is that, in the Oracle experiment, object queries are directly generated from the groundtruth boxes of the target using Rol pooling. In contrast, method generates object queries from the trained TTS and ASA modules.
We also conduct Oracle experiments on VidSTG (Zhang et al., 2020b) and HCSTVG-v2 (Tang et al., 2021) 1) datasets, as shown Fig. 8. The results reveal that, compared to zero-in ggro-initialized queries, the gooundruut--emeertted generated queries have significantly improve performance these two datasets, which once again demonstrates the importance of using target-aware queries enhancing the performance of STVG.
14

---

Published as a conference paper at ICLR 2025
Sputial Decnder Sec. 3.4
Backcooee
Multimodal Feuture Fusion in Sec. 3.1
AigPooling Suunn
Repant
2D
Sparial
Pooling
AAa
Head
Video
Backbone
DeC oncat
Iaital Spatial
3D
Quariee
B
The man in green clothes
Inital Temporal
ge
cemparel Decoder
Queries
walker behind
ArgPooling
Repeat
St Sec. 3.4
Tube
the man in
brown
Backhone
Toxt
+
Rol Pooling
>
26
Temporal
Head
Text
Input
Multimodal Encoder
Gromddrath-GGeeerated Query Generation
Decoder
Output
Figure 7: Detailed architecture for the oracle experiment using groundtruth -enerrrtteed object queries for STVG. B denotes the ground truth bounding box used to generate object queries.
Comparton
De Guryy
702
611
711
.8.3
12
542
31.4
2/6
P_3KU
FLAR
wougga3
wuughs
PLBU [I
WAAggo J)
(a) Results Oracle experiment on #CSTVG-2
(b] Results for Orade expectment on VidSG Declarative Sentences:
Interrogative Sentenoes)
Figure 8: Comparison of performance using zero-initialized object queries and the groundtruth- generated object queries for STVG HCSTVG-v2 in (a) and VidSTG in (b).
for
Pm
Cross Cross-Attomin Artention
Self Attention Block
Cross Cross-Artenic Attention
Eps
Multi-Head
Add & Norm
Add Norm
Attention
Forward
Atteation Block
Block
-2
Block
-
Food
F
Ecyp
K*
K
Lx
(a) Spatial Decoder
(b) Temporal Decoder
Figure 9: Architecture of self-attention encoder.
Figure 10: Architecture of decoders.
B ARCHITECTURES FOR SELF-ATTENTION ENCODER, SPATIAL AND TEMPORAL DECODERS
B. ARCHITECTURE OF STTENNITNNNOON ENCODER
The self-attention encoder SelfAttEncoder()) composed of (L 6) standard self-attention encoder blocks, is used to fuse features from multiple modalities. The structure is shown in Fig. 9.
B.2 ARCHITECTURE OF SPATIAL AND TEMPORAL DECODER
Similar to previous methods (Gu et al., 2024a; Jin et al., 2022), we use spatial decoder to learn spatial locations and a temporal decoder to learn temporal locations. The spatial decoder and the temporal decoder each consist of K (K 6) blocks, with each block comprising self-attention module and a atoss-attention module. Fig. 10 illustrates the architectures of the spatial and temporal decoders.
15

---

Published as a conference paper at ICLR 2025
A
Mr
S.Am Nadiisti
key/ value
CABx2
Linear
Softmax
Rag
Query
CABX2
MLP
Cin E
Mmn
"lassification Loss
keys
Binaryy Cross Entropy Less
Molti-llb
value
$t
Quary
Gieed Toak
CABx2
Linear
So Sofimax
value
key!
CABx2
MLP
M,
R
R2
quer3
mad Saece
Whii WXXD
Xxie WXX
M.
>A MX
(a) Sttriute-Speecifi Spatial Activation
(b) Instance-LLeve Spatial Activation
Figure 11: The architectures of attribute-specifer (a) and instance-level (b) spatial activation.
C DIFFERENT ARCHITECTURES OF ACTIVATION LEARNING STRATEGIES
In ASA, we compare two different strategies for learning spatial activation of the target. One is the proposed attribute-awan awrre spatial activation which learns attribute-specific information by attribute classification, and the other one is to learn the naive instance-level spatial activation through binary classification. Fig. (a) and (b) show the detailed architectures of these strategies. Specifically, for the instance-level spatial activation in (b), the classification results can be directly employed as the spatial activation maps Ma and Mn, to generated the spatially activated features. Compared the naive instance-level activation method, our attribute-specific spatial activate enables learning more discriminative fine-grained features, which are crucial to distinguish the target from the background.
D TEXT PRE-PROCESSING AND ATTRIBUTE LABEL CONSTRUCTION
Subject Extraction. In the ASA module, we guide the learning of activation using the subject from the text. To this end, we employ the Stanford CoreNLP tool conduct syntactic analysis sis on the text, transforming it into a structured graph. From the generated graph, we are able to extract the subject along with the related adjectives for appearance such as color and shape and verbs for motion such various actions based on the part-of-speech tagging. The adjectives and verbs extracted are respectively used to produce the appearance and motion attribute labels. It is worth noticing that, since current STVG task focuses localizing single target (and thus there is only one subject in the query), we extract only one subject representation from the textual sentence.
Vocabulary and Attribute Label Construction. The ASA module is trained by predicting attribute classes. For this purpose, we have constructed two vocabularies for each dataset, including an appearance vocabulary and motion vocabulary. Please note that, all information is from the dataset itself, without using extra information. Specifically, we first count the frequency of attribute words (adjectives or verbs related to the subject.) in each textual description in the dataset, and then eliminate words that appear less than 50 times. The remaining words constitute the vocabulary. The lengths of the appearance vocabulary and motion vocabulary for HCSTVG-v1/22 are 25 and 40, respectively, while for VidSTG, they are 36 and 45, respectively. With the vocabulary. we can construct the multi-class attribute label for each dataset based on the words in the textural expression.
E CLARIFICATION FOR MULTI-LABEL CLASSIFICATION IN ASA
In this section, we provide more clarifications for the multi-label classification in our ASA module. Specifically, answer the following questions: (a) Are the extracted subject features for appearance
SraffordippgihhuhioCCCoeN
16

---

Published as a conference paper at ICLR 2025
Table 11: Ablations of different loss weights on HCSTVG-v1.
AFTS
AASA
vloU
VlaU@
53.0
39.1
63.1
36.8
52.9
39.0
63.3
36.5
52.8
38.7
62.8
35.9
52.6
38.8
62.7
36.3
Table 12: Comparison of existing methods in complex scenarios on Hard Hard-HCSTVV-vL.
Method
m m_tloU
m___l _vvlU
@00.3
vloU@00.5
TubeDETR
40.4
28.3
42.3
14.3
STCAT
42.9
28.8
47.3
19.3
CG-STVG
45.5
31.4
50.3
20.0
Baseline (ours)
43.1
29.3
48.8
18.7
TA-STVG (ours)
45.9
32.6
54.7
22.3
and motion identical? (b) What is the relationship between weak attribute labels and subject features, and (c) do the weak labels update dynamically?
For (a): The extracted subject features for appearance and motion identical. Because the subject is usually the target of interest, and we want to extract its related appearance and motion features. Thus, use the identical subject the query for learning both appearance and motion information. For (b): The weak attribute labels and the subject both come from the given textual query (please Sec. D in our supplementary material on how to obtain these two). For example, the textual query 'The tall girl is walking'. 'girl' is the subject, while 'tall' and 'walking' are the weak attribute labels. The subject feature is used as the query to learn relevant attribute information from the visual features for the goal of multi-attribute classification. For (c): Given the textual query, the weak label for the weak attributes is generated from the query and is fixed (not updated).
F ADDITIONAL ABLATION EXPERIMENTS
Impact of different loss weights. Similarly to the previous approaches, our model is trained with multiple losses, including TTS loss, ASA loss, spatial decoder loss, and temporal decoder loss. The weights for the spatial and temporal decoder losses are kept consistent works such as STCAT (Jin et al., 2022) and CG-STVG (Gu et al., 2024a). The weights for TTS and ASA losses are empirically set. We have conducted an ablation on the weights ÀrTS and AASA for TTS and ASA losses, and the results shown in Tab. 11. From Tab. 11, observe that when setting Aris and AASa to I and 1 (see respectively, we achieve the best performance.
Ablation on the robustness of TA-STVG. In order to show the robustness of our TA-STVG again challenges, we manually select videos from a current benchmark HCSTVG-v1, in which the objects suffer from heavy occlusions, similar distractors, noisy text deseriptions. The resulted subset is called Hard-Ht Sorrd-HCSTV-v// (it will be released together with our source code), containing 300 videos selected from the set of HCSTVG-vI. We show our performance on this HCrd--C and compare it with other state-of-the-art models in Tab. 12. From Tab. 12, we observe that, due increased complexity, all the models degrade on the more difficult Hurd--CCTVG--/ (please see Tab. 12 here and Tab. in the paper). Despite this, our proposed TA-STVG still achieves wtwtll//ooooololloooeoeoeo (t.0////oo//0.////llllloo
the best performance (45.9/32.6/) 7/22.22//..// in m_ t/otto//mm www./oloUUOUUo @ ww..///o.......oo/.oeeoo @0.5, see 2) by outperforming CG-STVG (45.5/31 4/50.3/20.0 in m tltlll/o/mmmmmm STCAT (42.9/28 8/473889/4733/99.3 in m_ f__tloU/m_ vIoU/vIoU (1..0/....30......... (40.4/28.3/442./14.3 in m_ m__loU/m__ LttU///VVOOUUUUUUUU CEAVVOOUO03// @0.3/v10. @0.5, see Further more, TA-STVG significantly improves its baseline under these challenges v.s. +). All of these are attributed to our TTS and ASA modules that mine discriminative information of the target, even in challenging scenarios, for localization, showing their efficacy for improving STVG.
see O). see @). and TubeDETR
G ADDITIONAL DISCUSSION
Discussion on construction frame-specific object queries. In this work, the feature output by the ASA module undergoes pooling and repetition to generate the final object queries. The reasons
17

---

Published as a conference paper at ICLR 2025
for not constructing frame-specifee object queries are two-fold: (1) For STVG, we aim to obtain a global-level target-relevant query by pooling, which ensures the consistency in target prediction across different frames, i.e., localizing the same target instance in all frames using global information; and (2) Since the ASA module only extracts target features from frames selected by the previous TTS module, not every frame has its target-specific feature. As result, it is hard to directly construct frame-specific object queries in our work.
Discussion of computational consumption and future directions. Our TA-STVG needs a high de- mand of computational resource. In fact, this is a common issue in the STVG field. The computational complexity of STVG methods is high, because STVG models, particularly current Transformer based ones (including our work), require to receive all sampled video frames at once to make the prediction, leading to the requirement of significant computational resources. That being said, our proposed mod- ules TTS and ASA are lightweight and only bring about 4M parameters and 0.09T FLOPS compared to the baseline method (please see Tab. 13). In the future, we will further explore efficient STVG in the future, such (1) how to develop lightweight STVG architecture by reducing unnecessary network parameters, aiming to improve training and inference efficiency, and (2) how to leverage parameter-efficien fine-tuning (PETR) techniques like adapter or prompt learning for STVG model training, aiming to reduce the training complexity.
Discussion on handling videos with multiple targets. For STVG, the video may comprise multiple targets which are described in a single textual query. Since the target of interest is often the unique subject of the textual query, our model is able to leverage many descriptions such as motion and color attributes via our proposed modules, as well as contextual information like interaction with others by our spatio-temporal modeling in TA-STVG to distinguish the target from other objects.
Discussion on adapting TTS and ASA to other multimodal data. In this work, we introduce two modules TTS and ASA for STVG. Besides STVG, TTS and ASA (with appropriate adaptations) can be applied to work with other multimodal data. The essence of both TTS and ASA is to align two different correlated modalities (in our work, they are visual and text modalities). Specifically, TTS aims to align the frame-level feature with the global sentence-level feature, while ASA works to align the region-level feature with the local word-level feature. For other multimodal data, such as audio description and video, TTS and ASA can be used in similar way. For example, TTS can be used to align the video with the global audio description to remove irrelevant video content, and ASA can be applied to align spatial regions in the video to more fine- aineigranedd local audio descriptions (like attributes in textual description). For 3D videos, TTS and ASA can be used similarly.
Analysis multi-label classification in the ASA. In this work, the extracted subject features for appearance and motion are identical. Because the subject is usually the target of interest, and we want to extract its related appearance and motion features. Thus, we use the identical subject as the query for learning both appearance and motion information. we use the identical subject as the query for learning both appearance and motion information. The attribute labels and the subject both come from the given textual query. Given the textual query, the label for the weak attributes generated from the query, and is fixed. The subject feature is used as the query learn relevant attribute information from the visual features for the goal of multi- attribute classification.
Discussion on sampled target-relevant features on ASA. In our TA-STVG, the information of sampled frames by TTS is crucial for the subsequent ASA (which extracts attribute-related target information from sample frames for generating queries). We discuss the impact of sampled target- (a) Part of the selected frames does not contain the target object, while the other part contains the target. In this case, since the attention mechanism is utilized in the subsequence ASA, it can still extract useful target-aware information from partial sampled frames that contain the target while decreasing the weight of sampled frames without targets. As a result, can still generate effective target-aware query for improving STVG. For the worse case in this situation, the subsequent ASA fails, the sampled frames from TTS alone, even only partial frames contain targets, can still generate useful queries to enhance STVG performance, as shown in Tab. 4 (see and @ in Tab. 4 in the paper). (b) All sampled frames from TTS do not contain the target. If unfortunately this happens, then the final generated queries after ASA will not contain effective target- aware information, which is equivalent to using iawtoomnnns-nniialiiiaeeeee STVG and thus may degrade the final performance. However, we think that this case might be very occasional, as from Tab. 4 in the paper, we can see that TTS alone can significantly improve baseline
relevant frames on ASA in two situations:
query for
18

---

Published as a conference paper at ICLR 2025
Table 13: Comparison on model efficacy and complexity on VidSTG. The inference is conducted on
single A100 GPU, and inference time refers to the duration of a single forward propagation.
Methods
Trainable
Total
Time
Training
GPU Num
Time
Inference
GPU Mem
FLOPS
m_ m_tloU
m_ V__
TubeDETR (Yang et al, 2022n)
185M
185M
48
16 V600
0
24.4
1,45 T
48.1
30.4
STCAI (Jin al., 2022)
207M
207M
12h
32 32.A100
0.51
30.4
2.85
50.8
33.1
STVGFormer (Lin et al., 2023b)
-48
8 8,6000
33.7
CG-STVG al., 2024a)
203M
231M
32 22.A100
s
29.7
Baseline (ours)
202M
230M
14h
A100
0.53
28.2
2.88
49.5
32.3
TA-STVG (ours)
206M
234M
14.5
3200
0.57
28.4
2,97 T
51.7
34.4
performance. This means that TTS usually generate effective sampled frames that contain the target. We may leave the exploration of this to our future work.
innovation and contribution. In this work, we emphasize that, our innovation and contribution is the idea of exploring target-aware queries for improving STVG, which has never been studied in the STVG field. To achieve the target-aware queries for STVG, we proposed two modules, TTS and ASA. The reason why implementing TTS and ASA with attention is because the adopted aooss-attention perfectly meets our demand in extracting object query features and is simply enough (the pursuit of simple but effective architecture is always our motivation). In experiments, we show our proposed TA-STVG is able to achieve the best performance on all three datasets, showing the significant contribution of our idea of exploring target-aware for STVG, Moreover, our idea is general and when applied to other frameworks such as STCA and TubeDETR, we demonstrate consistent improvements, evidencing its efficacy.
Discussion
H EFFICIENCY AND COMPLEXITY ANALYSIS
To analyze the efficacy and complexity of our model, we conduct a comparative analysis involving the number of parameters, training consumption, inference consumption, and FLOPS against other existing methods. As shown in Tab. 13, we can see that, our proposed maintains similar computational complexity (206/234M trainable/total parameters and 2.97T FLOPS) to existing state-of-the-art models such as STCAT (207M/207M trainable/total parameters and 2.85T FLOPS) and CG-STVG (203M/231M 1M trainable/total parameters and 3.03T FLOPS). while achieving better performance with the best m. tloU of 51.7 and m_ vloU of 34.4 on VidSTG. Moreover, compared to the baseline method, TA-STVG brings negligible parameter and computational increments (Baseline v.s. TA-STVG: 202M v.s. 206M in trainable parameters, 230M v.s. 234M in total parameters, and 2.88T v.s. 2.97T in FLOPS), yet significantly enhances the performance in_ improved from 49.5 51.7 in and m_ vlo from 32.3 to 34.4), underscoring its superiority.
I QUALITATIVE ANALYSIS AND RESULTS
1.1 VISUALIZATION OF TEMPORAL RELEVANCE SCORE AND ATTRIBUTE SPECIFIC ACTIVATION
SPPCCIC
In this section, we show more visualizations of the temporal relevance scores learned by TTS and attribute-specific spatial activation learned by ASA in Fig. 12. From Fig. 12, we can see that the frames, selected by the temporal relevance score 8, will be within the groundtruth, which shows the selected features are target-relevant. In addition, the attribute-spen -speiffc spatial attention maps Ma and Mm can well localize the related attributes such as color and different actions the target. instance, in the first example, the text is "The man in green clothes walks behind the man in brown".. The highly relevant areas in the appearance attention map are mainly concentrated around the green clothes, while in the motion attention map focused on the legs. These attribute-spechs -seecffc spatial activation helps with learning more discriminative fine-grained features for improving STVG.
1.2 ATTENTION MAPS OF OBJECT QUERIES IN THE SPATIAL DECODER
To show the role of target-aware queries, we visualize its attention maps in the spatial decoder with comparison the attention maps of zero-initialized queries, shown in Fig. 13. As in Fig. 13, the
19

---

Published as a conference paper at ICLR 2025
Text: The man in green clothes walks behind man brown.
Tempural relevance score predicted by TTS (red: groundiruth; pink: sampled frumes)
Appearance (" ("green") activation Mc by
Motion ("walks") ") activation Ma by ASA
Text: The man in yellow clothes opens the door.
Temporal relevance score predicted by TTS (red: groundtruth; pink: sampled frames)
Appeurance ("yellow") activation M, by ASA
Motion ("opens") activation Mm by ASA
Visualization of temporal relevance score predicted by TTS and attribute- aware activation score predicted by ASA. The red boxes indicate the temporal and spatial foreground object to localize.
Figure 12:
Text: The man in green clothes walks behind the man in brown.
maps vero-inttialization queries
Attention maps furget-aware queries
Text: The girl in pink clothes moves to the man and huge the mane.
Atlention maps of zero-initialzzaton queries
Atlention maps of our fargef-aware queries
Text: The mun the white shirt putr his wine bottle on the table.
Attention mags zoro-initiaiizaioon queries
Attention maps of our target-aware queries
Figure 13: Comparison of attention maps for zero-initialized object queries and the proposed target- aware object queries in video frames in the spatial decoder. The red boxes indicate the foreground target localize. From this figure, we can clearly observe that the attention maps of our target-aware object queries by TTS and ASA can better focus on the target object for localization.
attention maps from zero-initialized queries are scattered, leading to imprecise target localization. In contrast, the attention maps from target-aware queries highly focus the target, demonstrating that larget-aware object queries can better learn target position information for improving STVG.
I.3 QUALITATIVE GROUNDING RESULTS
To further qualitatively validate the effectiveness of our proposed method, we provide its grounding results and comparison with the baseline method (with zero-initialization queries) on HCSTVG- v1/v2, as shown Fig. 14. From Fig. 14, it is evident that our method outperforms the baseline in both temporal and spatial localization. This suggests that queries with target-specifie fine-gna grained visual information can better learn the centio-temporal position of the target, thereby enhancing spatio-temporal grounding capability of the model.
20

---

Published as a conference paper at ICLR 2025
The man green clothes walks behind man brown.
Video
Baseline
Text: The white-collar man goes up steps and walks forward.
Video
TASTVVG
Baseline
Text: The yellow clothes opens the door.
Video
GT
TA-STVG
Baseline
Text: The girl with yellow hair walks in and stops.
Video
GT
TA-STVG
Raseline
Text: The girl in pink clothes moves to the mun and hugs the man.
Video
GT
TASTVG Baseline
Figure 14: Qualitative results of our TA-STVG (red), the baseline (blue), and ground truth (green).
For instance, in the first example, the task locate man wearing green clothes and walking. The baseline model, using zero-initialin inniiiiiiilzzed queries, fails to locate the target due to incorrect identification. contrast, our model, through the proposed TTS and ASA modules, extracts target-related visual information such as "green" and "walk" from the video initial queries, accurately locating the target object. The same comparison is observed in other examples. These results and comparison further show that initializing queries with fine-grained visual information of target improve STVG.
J LIMITATION
Despite achieving state-of-the-art performance multiple datasets, method has two main limitations. First, although TTS and ASA are able to provide better initialization for object queries, these queries still inevitably contain some noise compared to the desired information, resulting in lower performance than the Oracle experiments with groundrruth-generatee object queries. In the future, one direction similar to all other STVG approaches, our method consumes a large amount of computational resources for training, leading lengthy training period. In future, we will study shource-frienddl STVG using techniques from parameter-efficient learning methods.
further improve the quality of target-aware object queries. Second,
21


Reviews:
[{"review_id": "JuAKfBcSY6", "reviewer": "Reviewer_CJ8J", "summary": "In this work, a new method called Target-Aware Transformer for STVG (TA-STVG) is proposed for addressing the Spatio-Temporal Video Grounding (STVG) task. To replace the traditional zero-initialized object queries used in previous methods, TA-STVG is designed to generate target-specific object queries from the given video-text pair, with two main modules named Text-guided Temporal Sampling (TTS) and Attribute-aware Spatial Activation (ASA). Experiments and analysis are conducted to validate the effectiveness of the proposed TA-STVG.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1.The intuitive motivation of TA-STVG is clear and easy to understand, i.e., the adaptively generated object queries containing target-relevant information facilitate the spatio-temporal localization of textual expressions.\n\n2.The proposed method to enhance and keep track of the information about the described subject in the sentence is concise and effective.\n\n3.State-of-the-art performances are achieved by the method across all three datasets.\n\n4.Qualitative results and analysis are adequately given to illustrate the effectiveness of the proposed components.", "weaknesses": "1.It would be better to include more discussions and comparisons on some recent works in the related domain of video grounding. For example, the proposed ASA module adopts similar techniques of spatial activation learning [1] and concept classification [2].\n\n2.In the proposed TA-STVG, object queries are generated from globally pooling the multimodal features and are repeatedly used for each frame in the video. This way of object query construction seems suboptimal since the frame-specific information is ignored for the queries. Is there any consideration or motivation for such design choice? Why not construct the frame-specific object queries?\n\n3.How are the weights for different loss terms determined? Are there any ablation results?\n\n4.Some typos exist in the current manuscript, for example, \"such\" should have been \"such as \" in Line 29.\n\nReferences [1] Collaborative static and dynamic vision-language streams for spatio-temporal video grounding. In CVPR 2023. [2] Siamese Learning with Joint Alignment and Regression for Weakly-Supervised Video Paragraph Grounding. In CVPR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wHGP7qvtvb", "reviewer": "Reviewer_jmHQ", "summary": "The paper introduces an approach to spatio-temporal video grounding (STVG) using a Target-Aware Transformer (TA-STVG). The method aims to improve the localization of targets within videos based on textual descriptions by generating object queries that are adaptively initialized with target-specific cues extracted from video-text pairs. The key contributions include two main modules: Text-Guided Temporal Sampling (TTS) for selecting target-relevant frames, and Attribute-Aware Spatial Activation (ASA) for exploiting fine-grained visual attribute information. The proposed method achieves state-of-the-art performance on three benchmarks: HCSTVG-v1/-v2 and VidSTG, and demonstrates generality by improving the performance of existing Transformer-based methods like TubeDETR and STCAT.", "questions": "1. How does the computational complexity of TA-STVG compare to other state-of-the-art methods in terms of parameters and FLOPS?\n\n\n2. How does the model handle videos with multiple targets that are described in a single text query?\n\n3. Can the TTS and ASA modules be adapted to work with other types of multimodal data, such as audio descriptions or 3D video data?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents a novel approach to STVG by introducing target-aware object queries, which is a significant departure from traditional zero-initialized queries.\n\n2. TA-STVG outperforms existing methods on multiple benchmarks, indicating the effectiveness of the proposed approach.\n\n3. The writing of the paper is good and it is easy to understand.", "weaknesses": "1.The method may require significant computational resources, which may limit its accessibility and practical application, especially for smaller research groups or companies.\n\n2. The paper could provide more discussion on the computational complexity and parameter count compared to other methods, which is crucial for understanding its scalability.\n\n3. The paper could benefit from a discussion on the robustness of the model against various challenges such as occlusion, similar appearances, or noisy text descriptions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QBH7KgIVHv", "reviewer": "Reviewer_g9Sv", "summary": "The paper introduces a Target-Aware Transformer to address the limitations of traditional zero-initialized queries in STVG. The proposed method leverages target-specific information to generate more adaptive object queries that better interact with multimodal features, leading to improved accuracy. Extensive experiments on multiple benchmarks demonstrate that TA-STVG achieves SOTA performance.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Target-specific cues improve object queries over zero-init methods, enhancing feature learning for better localization.\n\n2. Extensive ablation studies support the effectiveness of the TTS and ASA modules, and the generality of modules is further validated by the integration into other models (e.g. TubeDETR, STCAT).", "weaknesses": "1. The method’s contribution is focused primarily on object query initialization for STVG, with the innovation mainly revolving around attention-based filtering for keyframes and target-specific attribute extraction, which makes the innovation limited.\n\n2. The multi-label classification in the ASA module is unclear. Specifically, are the extracted subject features for appearance and motion identical? What is the relationship between weak attribute labels and subject features, and do the weak labels update dynamically? \n\n3. The performance in complex scenarios requires further validation. Since object query initialization relies on the provided video-text pair, how robust is the query generation when visual information is heavily occluded, or the query is vague and lacks clear attributes? The robustness of the model against unclear attribute information should be further explored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B6joh0GJZ4", "reviewer": "Reviewer_oPic", "summary": "This paper addresses the challenge of spatio-temporal video grounding (STVG) in complex scenarios by introducing the Target-Aware Transformer for STVG (TA-STVG). TA-STVG adaptively generates object queries based on target-specific cues from the video-text pair, utilizing two key modules: text-guided temporal sampling (TTS) and attribute-aware spatial activation (ASA). TTS selects relevant temporal cues using text information, while ASA refines these cues to capture detailed visual attributes for initializing object queries. Experiments on benchmarks, including HCSTVG-v1/-v2 and VidSTG, demonstrate that TA-STVG achieves state-of-the-art performance.", "questions": "- Given the textual feature, first extract the subject's feature (usually the target to localize) from the textual sentence. My question is, do you typically obtain just one subject representation or multiple? If multiple are obtained, are they all processed through the same flow for decoding, or is there a further selection step?\n- Sampled target-relevant temporal appearance and motion features—do they need to have the same quantity? If so, how is this controlled?\n- The sampled target-relevant temporal appearance and motion features determine the frame information for the target object. This step can significantly impact subsequent accuracy. What happens if the selected frames do not contain the target object?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Unlike previous methods that use zero-initialized object queries, this paper presents the Target-Aware Transformer, which improves STVG by exploring target-specific cues for generating object queries.\n- This paper proposes text-guided temporal sampling to select target-relevant temporal cues from videos and introduces attribute-aware spatial activation to leverage fine-grained visual semantic attributes for object query generation.\n- Experiments on three datasets demonstrate its effectiveness.", "weaknesses": "- Given the textual feature, first extract the subject's feature (usually the target to localize) from the textual sentence. My question is, do you typically obtain just one subject representation or multiple? If multiple are obtained, are they all processed through the same flow for decoding, or is there a further selection step?\n- Sampled target-relevant temporal appearance and motion features—do they need to have the same quantity? If so, how is this controlled?\n- The sampled target-relevant temporal appearance and motion features determine the frame information for the target object. This step can significantly impact subsequent accuracy. What happens if the selected frames do not contain the target object?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)