Contextual Self-paced Learning for Weakly Supervised Spatio-Temporal Video Grounding
Authors: Akash Kumar
Zsolt Kira
Yogesh S Rawat
Venue: ICLR 2025 Poster (2025)
Keywords: spatio-temporal video grounding
weakly supervised learning
multimodal foundation models
dense video tasks
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: First adaptation of multimodal foundation models for dense video detection task without any labels.

Abstract: In this work, we focus on Weakly Supervised Spatio-Temporal Video Grounding (WSTVG). It is a multimodal task aimed at localizing specific subjects  spatio-temporally based on textual queries without bounding box supervision. Motivated by recent advancements in multi-modal foundation models for grounding tasks, we first explore the potential of state-of-the-art object detection models for WSTVG. Despite their robust zero-shot capabilities, our adaptation reveals significant limitations, including inconsistent temporal predictions, inadequate understanding of complex queries, and challenges in adapting to difficult scenarios. We propose CoSPaL (Contextual Self-Paced Learning), a novel approach which is designed to overcome these limitations. CoSPaL integrates three core components: (1) Tubelet Phrase Grounding (TPG), which introduces spatio-temporal prediction by linking textual queries to tubelets; (2) Contextual Referral Grounding (CRG), which improves comprehension of complex queries by extracting contextual information to refine object identification over time; and (3) Self-Paced Scene Understanding (SPS), a training paradigm that progressively increases task difficulty, enabling the model to adapt to complex scenarios by transitioning from coarse to fine-grained understanding.

Published as a conference paper at ICLR 2025
CONTEXTUAL SELF-PACED LEARNING FOR WEAKLY SUPERVISED SPATIO-TEMPORAL VIDEO GROUNDING
Akash Kumar"
Zsolt Kira
University of Central Florida a akash. kuma r @ucff edu
Georgia Institute of Technology zk i ra @eagggtccch....................e edu
Yogesh Singh Rawat
University of Central Florida
yogesh@ucf edu
Project Page: http:://aass0000..chhh.ooooppppp
Huggingface link: htwpp//pp///igggggggfaaacc////ts
ABSTRACT
In this work, we focus on Weakly Supervised Spatio-Temporal Video Grounding (WSTVG). It is a multimodal task aimed localizing specific subjects spatio- temporally based on textual queries without bounding box supervision. Motivated by recent advancements in multi-modal foundation models for grounding tasks, we first explore the potential of state-of- oftt----tttt-tttt- object detection models for WSTVG. Despite their robust zero-shot capabilities, our adaptation reveals significant limi- tations, including inconsistent temporal predictions, inadequate understanding of complex queries, and challenges in adapting to difficult scenarios. We propose CoSPaL (Contextual Self-Paced Learning). novel approach which is designed to overcome these limitations. CoSPaL integrates three core components: (1) Tubelet Phrase Grounding (TPG), which introduces spatio-temporal prediction by linking textual queries to tubelets; (2) Contextual Referral Grounding (CRG), which im- proves comprehension of complex queries by extracting contextual information to refine object identification over time; and (3) Self-Paced Scene Understanding (SPS), training paradigm that progressively increases task difficulty, enabling the model to adapt to complex scenarios by transitioning from coarse to fine -eeigraieed understanding. We demonstrate the effectiveness of CoSPaL on three benchmark WSTVG datasets, achieving 3.9% absolute improvement on VidSTG and a 7.9% improvement on HCSTVG-v1.
INTRODUCTION
Opatio-temporal video grounding (STVG) is focused on identifying and localizing objects within video frames both spatially and temporally based on textual descriptions. This problem critical for various applications, including video surveillance, autonomous driving, and general scene understanding. However, STVG presents significant challenges. Specifically, requires not only distinguishing objects from irrelevant ones across time but also predicting the start and end timestamps of activities related to those objects. While recent works solve this problem in a fully-supervised setup (Yang al. 2022 Jin al.) 2022 Lin et al. 2023), these approaches require extensive annotations, both temporally and spatially, which are costly and labor-intensive to acquire Therefore, we focus on a weakly supervised setting for spatio-temporal video grounding (WSTVG), where models are trained using only video-level descriptions, eliminating the need for precise spatio-temporal annotations.
Weakly supervised learning has been studied extensively in the image domain, addressing tasks like phrase grounding (Datta et al.) 2019; Wang et al. 2020) Liu et al.) 2021) and referral grounding (Liu et al., 2019 2022b). which locate objects in images based on text. Various methods have been explored, such as those leveraging coarse image-level labels or proposing complex mechanisms to
"Corresponding Author

---

Published as a conference paper at ICLR 2025
Phrase
(o) DALS (d) TVG (() voG
Task PG RG VOG TVG STVG
add bread and fip
Video
x
x
v
Ino guye
slaying guilar
Referring
X
(b) Reterral
the
Spatial
v
V
V
X
V
Temporal
X
The blue shirt walks speaks
Free-form X X
X
V
playing gulter
man gets
Figure 1: Comparison across tasks. (Left) (a) Phrase grounding (PG) refers to grounding all nouns
in the sentence, (b) Referral grounding (RG) makes the task harder by grounding specific subject, (c) Video object grounding (VOG) has fixed number of object categories and query template is fixed (d) Temporal video grounding (TVG) only focuses temporal localization. Contrast to these, (e) STVG requires spatio-temporal grounding of specific subject using free-form query. Green denotes ground truth. Darker shade denotes temporal boundary. (Right) Table summarizes challenges involved in STVG against other tasks.
handle uncertainty in object localization. However, extending these approaches to videos adds new layer of complexity due to dynamic changes in subject poses and scene context over time. As shown in Figure [ STVG involves increased complexity compared to static image tasks, particularly when handling free-form textual queries, where models must understand and localize objects and actions described in natural language. Existing works that address WSTVG rely on computationally expensive solutions, such as hierarchical algorithms (Li et al. 2023) or the inclusion of extra modality data like optical flow (Chen et al.) 2019b). In contrast, we propose more streamlined and efficient approach that simplifies the process by focusing solely on visual and textual modalities.
We build upon recent progress in multimodal learning (Madan et al. 2024) and leverage vision- language foundation models as our baseline, specifically adapting Grounding DINO (G-DINO) (Liu et al. 2023), a model known for its strong zero-shot capabilities in image-level tasks. While this model shows promise for multimodal understanding, extending it to STVG reveals three key limitations (Table D. First, it struggles with temporal consistency, frequently switching object focus across frames, it lacks clear understanding of temporal grounding. Second, despite being trained on large-scale image-ter datasets, it finds it difficult to handle complex or imbalanced queries, particularly when multiple objects activities described simultaneously. Finally, the model's performance declines in dense scenes with numerous objects, where accurate localization becomes critical.
To address these challenges, we propose CoSPaL, a novel approach that enhances both spatial and temporal grounding in STVG. CoSPaL introduces three key components: (a) Tubelet Phrase Grounding (TPG), which links textual queries to spatio-temporal tubelets (bounding boxes that span across frames), thereby improving object tracking over time. (b) Contextual Referral Grounding (CRG), which fine-tunes the network's 's attention to accurately localize the relevant tubelet mentioned in the query, ensuring more precise object identification across both space and time. (c) Self-Paced Scene Understanding (SPS), a training strategy that gradually increases task complexity, allowing the model to start with coarse predictions and refine them progressively. This structured approach significantly improves the model's 's adaptability and robustness in complex scenes.
We summarize our contributions as follows:
We propose CoSPaL, the first to solve weakly supervised spatio-temporal video grounding based on foundation model.
We propose Contextual Referral grounding (CRG) which extracts contextual information from query and enhances spatio-temporal grounding ability of the network.
We introduce Self-paced Scene Understanding (SPS) training scheme that makes network robust for complex challenging scenarios.
We perform our experiments on three different benchmark datasets, ViDSTG and HCSTVG-vl and HCSTVG-V demonstrating effectiveness of our proposed approach. CoSPaL outperform previous state-of-the-ant methods on WSTVG task by an absolute margin of 3.9% on VidSTG and 7.9% on HCSTVG-v1.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Primary research in this area involves unimodal techniques, which single modality, These techniques can be broadly categorized into two groups: CNN-based methods such as FasterRCNN (Ren et al. (017) and Bottom-Up Attention (Anderson et al. 2017), and Transformer-
Object Detection:
based methods like DETR (Carion et al. 2020) and its variants (Zhu et al. 2020, Wang et al.) 2022b
Liu et al., 2022a| Cai et al. 2023 Fang et al. 2022). However, unimodal detectors are trained on limited object categories, making them unsuitable for the STVG task, which involves free-form
queries. Recently, multimodal object detection research (Li et al. 2022 Zhang et al. [022 Yao et al. 2022, Liu et al. 2023) has emerged, taking image and text as inputs to output bounding boxes for objects. Multimodal detection involves: a) Adaptation to open-world scenarios (Minderer ct al.
2022 Feng et al.) 2022; Dou et al. 2022 Zhang et al., 2022; Yao et al. [022 Li et al. 2022).
allowing detection of novel objects at test time, suitable for STVG queries, and b) Strong zero-shot
grounding capabilities. These foundation models (Yan et al. 2023 Wang al. 2202)] Liu et al., 2023
Cheng et al. 2224)) are trained on large-scale datasets like COCO (Lin et al. 2014) and O365 (Shao
al. 2019), showing strong zero-shot performance for various tasks, including referral grounding.
G-DINO (Liu et al. 2023) outperforms previous models (Yan et al.
(023) in image referral tasks.
We base our work on G-DINO. Different from existing setups, we adapt G-DINO video settings
for STVG task.
Spatio-Temporal Video Grounding: This task involves grounding spatio-temporal tubes based on textual queries, addressing spatial and temporal dimensions. Initial solutions developed for STVG use a twoostage process with separate spatial (Rohrbach al.] [2015] Yamaguchi al., 2017; Chen et al.) 2019ch and temporal grounding (Gao et al. 2017] et al. 2019a). However, pre-trained object detectors have fixed number of object categories, limiting their effectiveness for STVG tasks with
free-form queries. Recent multimodal approaches (Su et al. |202] Yang et al. [022 Din et al.) 2022
Lin ct al. 2023 Gu et al. [2224] Wasim et al.
2024) tackle this challenge in a single stage, leveraging
image-based detectors (Kamath et al. 2021 video encoders, and spatio-temporal decoders ( (Yang)
et al. 2022 Wasim et al. 2024), addressing feature alignment inconsistencies (Jin et al. 2022), or utiliziigg static and motion cues (Lin et al.
2023 Gu et al., (2024), These methods typically rely on frame-level bounding box annotations for training. Differently from these, our work adopts a cost-efficient approach by refraining from using spatio-temporal labels.
Weakly Supervised Learning There are some existing works on dense tasks (ummar et al. 2025
Singh et al. 2024 Kumar al. 2022: Dave et al. 2022 Modi et al. 2022). however, they are unimodal and on semi-supervised
2023 Rana & Rawat 2023] Kumar Rawat 2022] Rana & Rawat
or active learning and can't It be extended to solve weakly supervised STVG task. For grounding
techniques, it can be categorized into three main classes. In images. it includes phrase and referral grounding. Phrase BroundingRRohrbaue
al., D Datta et al.) 2019 Chen et al. [018 Akbari et al., 2019 Gupta et al. 2020; Wang et al. |2200]] Liu et al. [202] Wang et al. 2021 a) highlights objects in contrastive optimization
textual queries using margin losses (Datta et al.) 2019: Chen et al.
(Gupta et al.) 2020 Wang et al., 2020). and reconstruction (Rohrbach et al. (016) methods. Referral
grounding (Liu ct al. 2019; 2022b; et al. 2023) adopts reconstruction (Liu et al. 2019 (022b) or
contrastive learning (Jin al. 2023) ground objects. In temporal grounding for videos (Wang et al.
2022; Lin et al. 2020 Zheng et al. 2022a b), both reconstruction and contrastive approaches (Lin et al.) (2020] Zheng ctt al. 2022a b) outperform contrastive ones. We employ a contrastive and reconstructive approach for spatial and temporal grounding respectively. Different from existing works, we incorporate referential capabilities in spatial and temporal grounding for videos which previous work don't 't Our approach induce focusing on specific contextual knowledge to enhance mutual interaction between vision and text.
2021b: Chen et al.
methods are prominent, however recent reconstruction-
3 METHODOLOGY
Problem Formulation: In WSTVG, the input is an untrimmed video V = (v), U2 ...)) of length L frames, accompanied by a query description caption Q describing the subject and activity in the video. The task output is the spatio-temporal tubelet for the main subject, Ar = {ar {arHi where a, represents the main subject in the query, and and t, denote the corresponding starting and ending timestamps of the activity. In weakly-supervisee settings, only video-level annotations are available for training, and there spatio-temporal labels for supervision.
3

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of weakly-supervised G-DINO et al. 2023) with previous approaches.
VidSTG Declarative
VidSTG- Interrogative
HCSTVG-vI
Methods
m_ vloU vloU/00.3 vloU 00055
vloU v.oU@0.3
vloU m_ vloU
v10U@003
v1oU@
AWGU Chen al. 20120
9.0
7.9
8.2
4.5
0.8
Vis-CTX Shi 2019
9.3
7.3
3.3
8.7
2.9
9.8
6.8
1.0
WINNER L al.
2023
11.6
14.1
7.4
10.2
12.0
5.4
14.2
17.2
6.1
W-GDINO Liu 2023)
10.6
13.0
7.8
9.8
12.1
6.7
9.0
11.6
4.6
Varying detection confidence
Skewed query attention scross time
Complexity
Tubelets
Q: The man in brown hat drops the hat of the man in the black hal then pushnee new the opposite man.
clothee...
Kmn2
0. hat of the Tubelet
man the black
0: hat of black
the hat
Q: man yellow veal...
0.60
0.61 0.45 0.58 Tedsilei (0
92 brown
9: brown
U2 the
Kg4
LO.55
0.50
0.50
Turndel 1..63
Cobeiet 2 (0.64)
Q: The
Q The man in while clothes bends down and looks down then sits in the chair next to him.
Tubelet
Os : oppostte man
Se. the man
Qt supposile mon
K=8
0.47
0.43
0.39
0:35
business
vest...
(a) Unrellable Temporal Predictions
(b) Imbalanced Query Attention
(c) Complex Scene Understanding
Figure 2: Illustration of failures of W-GDINO: (a) Unreliable Temporal Predictions: Foundation model predictions are inconsistent across time and switch attention between actors across time. This leads to performance degradation. (b) Imbalanced Query Attention: It showcases that model lacks understanding of complex queries. Across time, query which model attends for each subject tubelet is inconsistent and doesn't t match with ground truth. (c) Complex Scene Understanding: As the number of subjects increase, model's capability focus on the specific subject described in query reduces. This shows it's lack of understanding of challenging scenarios. K denotes total number of subjects. Blue and red denotes predictions and green denotes ground truth (a) and (c), and brown in (b).
3.1 PRELIMINARIES: GROUNDING DINO (G-DINO)
G-DINO (Liu et al. 2023) extends closed-set set object detection to open-world scenarios. It takes an image and query imput, and outputs a bounding box and confidence score. In work, we use text input query Q and video frames 1,- = {V,)T=1= with T denoting the video length. As multi-modal object detectors
image- based and STVG is video task, we first extend G-DINO for videos. To adapt it, we run detections throughout the video, storing each subject's bounding box, confidence score, and features. Applying tracker (Aharon et al., 2022) these detections yields tubelets for each detected subject k as Tog. K represents the total number of subjects throughout the video. This adapted model is termed weakly-supervised Grounding DINO (W-GDINO). To assess W-GDINO's performance, we accumulate and average the confidence scores of each tubelet, selecting the one with the highest score. While Table Taalleddent demonstrates competitive performance, we observe some issues with this approach.
We attribute these issues three major factors: (a) Unreliable Temporal Predictions: Figure (a) shows the model's predictions are inconsistent over time due to factors like varying subject poses and similar spatial features. W-GDINO lacks spatio-temporal localization. (b) Imbalanced Query Attention: GDINO is trained via byte encoding scheme which breaks down the original query and then rebuild it up. Due to this, GDINO is unable to focus on a specific part of query consistently across time, as seen in Fig. (b). This causes confusion about the described subject. (c) Limitations in Complex Scene Understanding: WSTVG datasets present challenging scenes with many objects, as shown in Fig. (c). This complicates spatial and temporal associations. We propose CoSPaL to address these limilations.
3.2 CONTEXTUAL SELE-PACED LEARNING (CoSPAL)
CoSPaL consists of three key components to address the above limitations: Firstly, Tubelet Phrase Grounding (TPG) (Sec. 3.2.1) } induces spatio-temppral localization capability in W-GDINO. It enables to remove unreliable temporal predictions by aligning textual and tubelet features for spatial grounding and textual and video features for temporal grounding. Second, to improve attention relevant parts of query. we propose a novel concept of Contextual Referral grounding (CRG) module extract fine-grained attributes that highlight the subject's contextual information. enhances focus
4

---

Published as a conference paper at ICLR 2025
Temporal Selt Attention
(e) Spatial Grounding
Contextual Referral Grounding (CRG)
Cross-
Noun
Adjective
Verb
Altention
Man
white
bends, looks, sits
Global Query
Query
Detactor Tracker
Encoder
Q: white clathes bends down...
Video Enooder
Cross- Attention
Local Query
Attribute
Verbs
Background
Q: The man in whit
Cross-
4
The man
He bends down. looks down.
clothes bends down looks down sits the chair.
Attention
while clothes
He sits.
The man white clothes
Query Encoder
Predicted Tube
man white clothes bends down The man white cloth looks down The man in white cloth sits
<M> down
CRG Block
Refined Query
(b) Temporal Grounding
Figure 3: Overview of CoSPaL: TPG contains two grounding modules namely, spatial and temporal. Spatial module grounds the correct subject tubelet. Temporal module predicts the temporal action boundary via cross attention between highlighted vision features and masked query features. Contex- tual Referral Grounding (CRG) block shows the breakdown and generation of local (Qnr) and global query (Qog). Green shows predicted bounding box. Darker green shade shows predicted temporal boundary localization.
on the subject in context (Sec. 3.2.2 Finally, since STVG challenging and matching queries with numerous scene subjects is difficult, we introduce self-paced scene understanding (SPS). It progressively increases task difficulty to adapt the network for complex scenarios and enhance the network's ' discriminative ability over time (Sec. 3.2.3
An overview of CoSPaL is shown in Fig. 3
3.2. TUBELET PIRASE GROUNDING (TPG)
TPG adapts W-GDINO to solve spatial and temporal grounding jointly. The spatial grounding module leverages word-level representations to enhance the aligument between textual and tubelet features. Mcanwhile, the temporal grounding module optimizes the correspondence between video and textual features to accurately predict the start and end timestamps of the activity described in the caption. Following previous works in weakly supervised grounding (Datta et al.) 2019 Gupta et al.
2020: Wang et al., 2020 2021a) we incorporate a visual encoder to extract features from pre-trained
object detectors and video encoders and a language encoder (Devlin et al. 2019 2014 to provide rich textual representations of query.
Pennington et al.
Visual encoder:
We extract object level representations for = Fo E R R from G-DINO (based DETR (Carion et al. 2020 where, F, object encoder model, and, os denotes e detected subject. We link these detections via tracking (Aharon et al.) (2022) algorithm to generate 1e where s and denotes starting and ending timestamp of the subject in the video. Tubelet features for a video is represented by Fr = (f(Too)) ) E RT x x256 where K denotes number of objects present in a video. For video features, we utilize video encoder. F, (e.g. I3D (Carreira & Zisserman, 2017)) get clip-level features, fc F.(EV) RC 1024 C' denotes the number of clips in the video.
subject tubelets for subject k, To {or,
Query encoder: We pass the query Q through a language encoder (Fi), BERT Devlin et al. 2019) to get word level embeddings Fw {f 1.V
x where far = Fi FX Im= N denotes
total words in query.
Spatial Grounding Module highlights the correct tubelet. We use multimodal contrastive learning optimization to highlight the relation between words and tubelet. The insight that to find the maximal mutual information shared between two modalities, they first need to be projected into the same space. We start with subject tubelet features video (F₂). The features are extracted from DETR; thus, the features do not have any interaction amongst them temporally. To establish connection between them and enhance the features temporally, we apply a temporal self-attention block (TSA) to generate updated tubelet features, Fr TSA(Fr). This helps the network to highlight frames which provide more contextual information. For example, if the query description is "man brown coat..." TSA give higher weights to the frame when the actor's s coat is visible rather than noisy frames (frames with zoomed in faces, partial body, challenging poses, figure shown in
5

---

Published as a conference paper at ICLR 2025
supplementary). We project (Fr) for each actor into shared space by applying cross- atos-nattetioon block highlight subjects mentioned in the query (Fur). Fr is used as key and value pairs, and, Fw is query. We use simple feed-forward MLP layers to project key and query features. We calculate the similarity (SIM) between individual word f and tubelet feature fT, as SIM(S fri,)
MMLPg(fun " M'MLP (fr, D)/vd, where MLP, and MLPx denote MLP layers for key and query features. Using the features projected into the same space, we calculate aggregated attention for the video with all tubelets T, Ar with each word as Ar(fT, fum) = Ex= softmax is defined in Eq. I
k= sof coftmax (fr.
fwm (f where
exp((IM([ STre)
softmax fwm)
(1)
Σ Lk'= =1 exp(SIM(S wm: (7:))
MLP, is MLP layers to project value features and sof tmax funm indicates normalized attention scores. Word features are used as query since the context in the caption is present the scene, but the reverse may not be true. Lastly, to optimize the learning for spatial module, we apply multimodal InfoNCE loss shown in Eq. 2 to induce discriminative learning in the projection layers to pull regions with higher attention closer and push away negative tubelets farther. To get the compatibility function for loss, update the AT Ar MLP," (fw... Ar. We pick negative tubelets (fr) within the batch.
exp(Ar(JTL fw..)
log Σ Lk'=1 (R) Cxp(Ar((T )))
K
Temporal Grounding Module provides temporal boundary for activity mentioned in query. The limitation of the spatial grounding module its inability to provide start and end timestamps for actions, which reduces its adaptability for the WSTVG task. We incorporate a reconstruction-bart
approach based on its effectiveness for temporal grounding(Lin et al. 2020 Yang et al. 2023] Zheng et al. 2022a b). The main idea is to enforce semantic consistency between video and query. Firstly, original query highlights key segments in video. Then, original query is masked and use the highlighted visual segments features to regenerate masked query features. This enforces the video features semantically correspond to query features at test time. Fig. 3 (b) shows outline for the CRG module.
Since temporal grounding requires understanding of action, and features from the object detector contain only image- information, we therefore acquire clip-level features Se from the video encoder model. We take cross attention (CA) between original query features (f₂) to get highlighted visual features as f² CA(fq, fe). Key and value pairs come from the visual features and query comes from the caption. Then, the original query is passed through a masking module M which looks into specific part-of-speech (POS) tags of the query and mask out noun/adjjcttvees/errb to generate the masked query, q. We use a transformer decoder (DEC) to regenerate the probability distribution of masked query as ((Tur.. See 9jo:m-1)) = DEC (CA(C @)), where, P denotes probability distribution for mh word Fu. The reconstruction loss (C,) to train the model is the difference between regenerated and original query distribution shown in equation where N denotes total number of words.
== Σ
logP(qulfer ( -1)
(3)
3.2.2 CONTEXTUAL REFERRAL GROUNDING (CRG)
Analyzing the original query Q, we observe that it contains descriptions of background objects/scene. It also contains information about attributes and actions related to those objects. equation 2 spatial loss C, applies summation across similarity with all words. This leads to confusion for the network regarding which tubelet is actually the target tubelet (referral subject). CRG addresses this by leveraging referral subject's attributes to improve attention over objects sharing common information with the query. The intuition is that referral subject-related attributes further enhance grounding capability.
We refine this information from free-form text query by decomposing query Q into three sub-parts: a) Referral tubelet and its attributes (Qaa), b) Referral tubelet action verbs (Qor), and, c) background

---

Published as a conference paper at ICLR 2025
information (Qu). We generate queries, Qo that describes referral tubelet using Qoo and Qov- This helps the network associate attributes and actions with correct tubelets (To.). Additionally, for a more fine- tineggrinedd aspect. we look into noun-adjective--veb word features corresponding to referral from generated (Q.) and original query (Q). These features contain relevant information in relation to the whole caption. Thus, we call these referral features Qag. since they contain global knowledge, and earlier query Qo as Qui since they contain local knowledge in relation to the original query. Fig. (c) illustrate the process of generation of Qot and Qag. The updated spatial loss (C.) is shown in equation 4 where [u(Qop:Qaa) denotes words from updated queries.
N
Σ m=1
log
=11K #N) exp(Ar(fI, Qu)
Similarly, for temporal localization module, existing works (Chen al. 2022 Wang et al. 2021b Lin et al. (2020) lacks referential capabilities. Thus, we update original query with these local queries such that attention is more concentrated on beginning and ending timestamps relevant to the referral subject. Eq. shows updated reconstruction loss (Lx).
È, = Σ logP(ennte. fos Qa))))
(5)
3.2.3 SELF-PACED SCENE UNDERSTANDING (SPS)
STVG inherently complex, particularly when dealing with videos lacking explicit spatio-temporal labels and containing multiple subject tubelets. The primary challenge lies in maximizing correlation between query and subject features, especially when their number increases significantly. To address this, we introduce self- celf-paced curriculum learning (SPL) strategy (Wang et al. [002aa] [Soviany et al.) 2022) to enhance optimization. This approach incrementally increases task complexity, beginning with simpler scenarios and progressively introducing more difficult ones as the model improves. By gradually exposing the model to more challenging cases, SPL ensures better convergence and robustness in learning complex spatio-temporal relationships.
SPL utilizes a student-driven difficulty scheme. Firstly, we analyze the scenes where model gets confused and then devise training accordingly. Thus, emulate SPL in stages: (a) Difficulty Measure: We measure difficult based on the scene complexity. Fig. (c) shows drop in attention values correct subject the scene gets complex. and (b) Training scheduler: Based difficulty measure, we design the training schedule of each curriculum step by setting the upper bound on number of tubelets per video. We increase this upper bound by factor and keep including more challenging videos with each stage and finally include all videos in last stage. This facilitates both spatial and temporal grounding module in terms of coarse-to-fin understanding of scenes. In the beginning, the network has lower discriminative power so it can understand easy (coarse) scenes better, and with time we keep increasing the difficulty and the network's ability to understand complex scenes (fine) improves.
4 EXPERIMENT DETAILS
Datasets: For our experiments, we show results on three benchmark datasets, namely VidSTG Zhang al. 2020), HCSTVG-v1 (Tang et al. 2020) and HCSTVG-v2 (Tang et al. 2020). VidSTG pairs, out of which 44,808 are declarative and
distribution comprises of 99,943 videos- 55.135 are interrogative. The total number of videos are 10,303 and it contains 80 different type of object categories. Training, validation and test contains 80,684, 8,956 and 10,303 distinct video- sentence pairs respectively and the amount of unique videos for each distribution is 5,436, 602 and 732 respectively. HCSTVG-v1 contains 4500 videos for training and 1160 vidcos for testing with sentence description referring to human attributes/actions. HCSTVG-v2 dataset extends version to 16,544 videos. The dataset divided into 10,131 training, 2,000 validation and 4,413 testing videos. Since test set is not available, we evaluate and show results on validation set following previous works (Yang et al. 2022 Lin et al. 2023 Gu et al. 2024).
Implementation details: We divide this into three parts: (a) Detection And Tracking: We utilize
G-DINO(LLu et al. 2023 with 0.4 threshold for both phrase and box threshold. We the detector
7

---

Published as a conference paper at ICLR 2025
Table 2: Comparison with existing state-of-the-an methods on HCSTVG-vI and v2 datasets.
HCSTVG
HCSTVG
Methods
doU
m vnv vloU@0.3 vloU@0.5 doU m vloU v.0 vlo 0905
Pully-Superviser
STGVT (TCSVT20)
Tang et al.
2020
18.2
26.8
9.5
STVGBert [OCV211
Su al.
2021
20.4
29.4
11.3
TubeDETR (CVPR22)
Yang et
2022
43.7
32.4
49.8
23.5
53,9
36,4
58.8
30.6
STCAT [NeurIPS22]
et al. 2022
49.4
35.1
57.7
30.1
CSDVL [CNPR223
Lan al 2023
36.9
62.2
34.8
58.1
38.7
65.5
33.8
CG-STVG [CVPR944
Guu ct al 2024
52.8
38.4
61.5
36.3
60.0
39.5
64.5
36.3
VGDINO (CMP124) (Wasim et al 2024
38.3
62.5
36.1
39.9
67.1
34.5
Veckk/y-Supervsee
AWGU (ACMMMED))
(Cenn et al.
2020
8.2
4.5
0.8
Vis-CTX CVPRI9T
Shi ct al
2019
9.8
6.8
1.0
WINNER (CVPR23)
Li ct al.
2023
14.2
17.2
6.1
W-GDINO (Ours- Baschme)
18.0
9.0
11.6
4.6
23.3
9.9
13.3
5.6
CoSPaL (Ours)
41.2
22.1
31.8
19.6
48.6
22.2
31.4
18.9
(+23.2)
(+7.9)
(+14.6)
(+13.5)
(+25.3)
(+12.3)
(+18.1)
(+13.3)
every 5th frame and extract features from the last decoder layer. We use BoTSORT tracker(Aharon al., 2022) algorithm to generate tubes; (b) TPG: We sample 32 frames equally indexed to get tubelet features. We extract video clip level features using I3D (Carreira & Zisserman) 2017) model. (c) CRG and SPS: We use GPT-3.5 to extract quantifier and phrases from original caption for CRG. We show more details and examples in supplementary. For SPS, we incorporate three stages of training with upper bound on four, seven and all tubelets. The model trained for 10 epochs with iterations over the dataset through each sub-phrases. More details about hyperparameters present in supplementary.
Inference:
We infer the subject with highest attention from spatial localization module get the tubelet à. Temporal localization module predicts the start and end temporal bounds at for the predicted tubelet.
Evaluation Metries: We show performance on metrics used by previous approaches (Yang et al. 2022; Jin al. 2022), namely mean average spatio-temporal localization mn_VIo and temporal localization (tloU). VIoU and tloU is calculated as [S.] Eres, [oU(b, be) and [S.] respectively, where [S..] S, and S, implies intersection and union between the predicted timestamp by the model and the ground truth timestamp. loU be) calculates the spatial overlap between the predicted bounding box be and ground truth bounding box b, at frame t. m_ vIoU is calculated by averaging over vloU for all the videos in test set. vloU @R indicates scores for samples whose mean vloU is greater than
R. We show for two values 0.3 and 0.5 following previous works(Yang al. 2022
Li et al. 2023)
5 RESULTS AND ANALYSIS
Comparison with weakly-suupervised baselines: In Tables 2 and 3 we compare our approach with previous weakly-supervised approaches. On HCSTVG-v dataset, we beat AWGU and Vis-CTX on all metrics by margin of 14-15% at mean vloU score. We outperforms the recent approach, WINNER al. 2023) by margin of 8%. Looking closely at different IoUs, we outperform previous SOTA at 0.3 by 2x and at 0.5 by 3x. Against W-GDINO, CoSPaL outperforms it by margin of 5.4% and 12.4% on m_ vloU and tloU respectively. VidSTG is an extremely challenging large-scale dataset. This is also evident by the gain made by fully-supervised approaches in recent years which less than 2%. CoSPal. outperforms previous weakly approach by 4.4% on declarative and 3.3% on interrogative settings. At higher metrics 0.3 and 0.5, our approach achieves gain of 4-6%.
Comparison with fully-supervised baselines: We also compare approach with fully supervised approaches (Tables 2 and 3). On VidSTG dataset, the proposed approach beats a few of the fully-
supervised approaches which combination of spatial and temporal grounding (Gao et al. 2017) modules. On HCSTVG-v1 dataset, we outperforms STGVT and SVGBert on all metrics. Against
recent approaches(Yang al. 2022 [in et al. 2022 Lin al. 2023), our approach within 10% for
8

---

Published as a conference paper at ICLR 2025
Table Comparison with existing state-of-the-ar methods on VidSTG dataset.
Declarative Sentences
Interrogative Sentences
Methods
m vlvloU vIoU@0.3 vloU@0..5 dlU m_vioU vloU@0.3
Fully-Supervise
Groun-R (SCCV16)
(Rohrbach
2015
9.8
11.0
4.1
9.3
11.4
STPR (ONPRIP) Yamaguchi et al.
2017
34.6
10.1
12.4
4.3
33.7
10.0
11.7
4.4
WSSTG (ACLI9) Chen et al.
2019c
11.4
14.6
5,9
10.7
13.9
5.3
STGRN (CNP920)) Zhang et al.
20201)
48.5
19.8
25.8
14.6
46.9
18.3
21.1
12.8
STVGBert (CCV21
(Su al.
2021
24.0
30.9
18.4
22.5
26.0
16.0
TubeDETR (CVPR22) Yang al.
2022
48.1
30.4
42.5
28.2
46.9
25.7
35.7
23.2
STCAA (NeudPS22 Jin
2022
50.8
33.1
46.2
32.6
49.7
28.2
39.2
26.6
CSDVL (CVPR23) Lan
202.3
33.7
47.2
32.8
28.5
39,9
26.2
CG-STVG (CVPR20) GGu
33.1
49,9
29.0
27.5
VGDINO (CNP9244) Wasım ct
2024
52.0
34.7
48.1
34.0
50.8
29.9
41.0
27.6
Wenk/y-Supervised
AWGU JACMMM22
Chen al.
2020
9.0
7.9
3.1
8.6
6.9
2.9
Vis-CTX (CVPR
Shi
2019
9.3
7.3
3.3
8.7
7.2
2.9
WINNER (eVeRzS)
2023
11.6
7.4
10.2
12.0
5.4
W-GDINO MOurs-Basching
28.7
10.6
7.8
29.1
9.8
12.1
6.7
CoSPaL (Ours)
13.1
38.9
13.5
16.4
10.2
(+12.4)
(H14)
(+6.0)
(+5.3)
(+9.8)
(+3.3)
(+1.3)
(-3,5)
Table 4: Ablation on TPG (upper) and SPS
Table Ablation study on proposed sub-modules. We show the effectiveness of each module and their combinations. First row shows W-GDINO performance.
(lower) different factors and stages of training. S & T denotes spatial and temporal grounding module, TSA denotes temporal attention.
S TSA T
IloU
m
vloU
v1.U@0.3 vl.
TPG CRG SPS doU
m vlOU v.0 v
26.2
13.5
17.7
7.3
18.0
9.0
11.6
4.6
27.3
13.9
18.6
6.9
37.6
15.3
35.2
18.0
26.3
35.8
20.2
30.5
17.6
37.6
19.2
28.8
15.3
/
37.8
21.0
31.7
16.8
Stages
m_tloU m. m_ vIOU
v.oU@003 vloU@0.5
38.2
20.1
28.5
17.6
1
34.1
17.7
26.0
14.4
V
38.1
21.1
30.7
18.4
II
36.2
18.5
27.0
14.8
V
41.2
22.1
31.8
19.6
111
38.2
20.1
28.5
17.6
(423.2)
(415.0)
mean tlUU and within 15% at m_ vIoU on both HCSTVG-vl and v2. Fully-supervised approaches utilizes ground truth information optimize the network, whereas our approach does not.
5.1 ABLATION STUDY
Effectiveness of TPG sub-modules: Firstly, look into base model, TPG. From Table 4 observe that temporal grounding module plays a significant role. It boosts the standalone score of spatial grounding module on all metrics. mean tloU and vIoU scores boosted by margin of 9% and 4.5% respectively. At 0.3 score boosts by a margin of 10% and almost 2x at v.oU@0.5. Temporal attention block improves score by 1% additionally on mean vloU.
Impact of SPS stages: Table demonstrates the importance of progressive learning. Increasing the difficulty with each indeed helps the network become more discriminative. We observe gains of 3% and 4% on mean vloU and tloU respectively.
Effectiveness SPS and CRG: We analyze each sub- aub-module in our proposed approach in Table Firstly, our proposed TPG outperforms W-GDINO on all metrics. On the main metric, our method out- performs by 10%. The context refinement grounding aspect standalone boosts the score by % on of VeDNO and 1% TPG module. This shows the impact that contextual referral matters and focus in on attributes related to referral subject helps. When TPG and CRG are combined, that utilize different referral phrases and noun-adjective-vee in performance by 0.8%. Introducing SPS on TPG and CRG standalone, shows a gain of 0.9% on vloU in both. This indicates that the network adapts well when the difficulty of the task is increased
pairs, observe further improvement

---

Published as a conference paper at ICLR 2025
The woman in blue clothes takes something the table and wrape around wrists a times.
The man in blue clothes runs the side the road, stops, turns around and speaks to person white turns again.
Figure 4: Qualitative analysis: Green: ground truth; red: ae::WGDINO, and blue: CoSPaL. (darker shade represents temporal detection boundaries). W-GDINO suffers from temporal localization and imbalanced attention focusing different subjects throughout the video. CoSPaL overcomes these limitations and has better overlap with GT in both scenarios.
progressively. Using both SPS and CRG with TPG performs the best (last row). It boosts the perfor- mance on top of TPS+CRG by margin of 1.1% on mean vloU and 3.4% on m uo_.loU. Against TPG, the addition of proposed sub-modules improves the performance by 2.9% and 3.6% on m_vIoU and m_tloU respectively. Looking specifically at higher loU at 0.5, SPS boost the performance by mar- gin of 2.3, 0.8 and 2.8 on TPG, CRG, and TPG+CRG. This shows substantial evidence that SPS helps both spatial and temporal grounding module increasing its discriminative ability with task complexity.
Impact of detector backbones: Table Bable/Jshows CoSPaL outperforms WINNER with Faster R-CNN (Anderson et 2018) back- bone for fair comparison. Comparing across backbones, DETR outperforms Faster R-CNN by margin of 6% m__ vloU HCSTVG-v1.
Table 6: Comparison against detector backbones.
Methods
Detector
m_vloU vloU@0.3 vlol @0.5
WINNER Faster-RCNN
11.6
14.1
7.4
CoSPaL Faster-RCNN
16.4
23.7
11.1
CoSPal.
DETR
22.1
3188
19.6
Computational Efficiency: Fig. 5 shows
40
80
CoSPaL computationally efficient against all fully-supervised approaches. The main reason is the use of frozen backbone whereas fully- supervised approaches finetune the whole back- bone end-to-end. Against ours, fully-supervised training time is 2-4x with 2.5x-6.5x more GPU memory requirement. We use single gpu against 8 in CSDVL(Lin et al.] 2023). 16 in Tube- DETR DEnggYang&a al. 2022) and 32 STCAT al. 2022) and CG-STVG Gu al. 2024). In terms total memory (number GPUs x GPU memory), our approach uses only 1-3% against fully-supervised approaches.
CG-STVG
STCAT
Number of GPUs
30
Memory (in GB)
60
20
TubeDETR
CSDVL
40
10
Ours
20
5
15
20
Training Time (hours)
Figure 5: Comparison on computational efficiency against fully supervised approaches.
In Fig. 4 show the effectiveness our approach qualitatively. W-GDINO struggles with grounding the right actor as well as provides inaccurate temporal bounds, whereas approach spatio-temporally grounds the actor better. More examples are shared supplementary.
Qualitative Analysis:
6 CONCLUSION
In this work we focus on Weakly Supervised spatio-temporal video grounding (WSTVG), aiming localize specific objects based on textual queries without relying on labeled data. As first step, we provide an extension of G-DINO for WSTVG task, and observe several challenges and limitations. To address these, we introduce Contextual Self-Paced Learning for Weakly Supervised Spatio-temporal Grounding (CoSPaL). It employs Tubelet Phrase Grounding (TPG) module to enhance spatio-temporal prediction coherency in localization and introduces the Contextual Referral Grounding (CRG) module for extracting contextual information from textual queries, improving object localization precision. Additionally, the Self-Paced Scene Understanding (SPS) training scheme progressively increases task complexity, enhancing the network's 's robustness in challenging scenarios. We evaluate the proposed approach on three benchmark datasets, surpassing existing methods and demonstrating its effectiveness.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGMENTS
This research is based upon work supported in part by the Office of the Director of National Intelligence (Intelligence Advanced Research Projects Activity) via 2022-21 102100001 and the National Science Foundation under Grant No. 2239292. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed implied, of ODNI, IARPA, or the US Government. The US Government authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein. The authors would like to thank Aisha Urooj Khan (Lunit) and Rohit Gupta (UCF CRCV) for helpful discussions on weakly supervised grounding and multimodal aspects.
REFERENCES
Nir Aharon, Roy Orfaig, and Ben-Zion Bobrovsky. Bot-sort: Robust associations multi-pedestrian tracking. ArXiv, abs/2206. 14651, 2022. URL httpss//app. www.nticschon...c..c....... org/
CorpusID:25011:0
Hassan Akbari, Svebor Karaman, Surabhi Bhargava, Brian Chen, Carl Vondrick, and Shih-Fu Chang. Multi-level multimodal common semantic space for image-phrase grounding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12486, 2019.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6077-6086, 2017. org/ Cttp//oo:ppuspspssec
URL https: /aap::://pp..p// www.st.csscooooeeoe.........ccee
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 6077-6086, 2018.
Zhi Cai, Songtao Liu, Guodong Wang, Zheng Ge, Xiangyu Zhang, and Di Huang. Align-detr: Improving detr with simple iou-aware bce loss, 2023.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-ent object detection with transformers. ArXiv, abs/2005. 12872, 2020. URL /sttps:/appi.enneee htttps///aal.eennnticchco...... http:///w.aaaaaaaaooooorrorrrrr/
João Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset, 2017 IEEE Conference on Computer Vision and Pattern Recognitiion (CVPR), pp. 4724-
4733, 2017. URL/https: //ap::://p/p.. semanti cwwhohirccholcccccoooooocooo org/ wttpoo/aarr...ro//ooooo.....
Jingyuan Chen, Lin Ma, Xinpeng Chen, Zequn Jie, and Jiebo Luo. Localizing natural language in videos. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):81558811 Jul. 2019a. doi: 10.1609/a 133.01.330011111111111 330111175. URL Rttt/htts:////oo http:pp:://.....a.....ro//.iiiip www.:////.......iiii......... php/ CAAT/article/viee
Junwen Chen, Wentao Bao, and Yu Kong. Activity-driven grounding from untrimmed videos. Proceedings of the 28th ACM International Conference on Multimedia, 2020.
spatio-temporal
Kan Chen, Jiyang Gao, and Ram Nevatia. Knowledge aided consistency for weakly supervised phrase grounding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4042-4050, 2018.
Long Chen, Yulei Niu, Brian Chen, Xudong Lin, Guangxing Han, Christopher Thomas, Hammad Ayyubi, Heng Ji, and Shih-Fu Chang- Weakly-supervised temporal article grounding. In Empirical Methods in Natural Language Processing (EMNLP), 2022, 2022.
Zhenfang Chen, Lin Ma, Wenhan Luo, and Kwan- Yee Kenneth Wong. Weakly-superviser spatio- temporally grounding natural sentence in video. ArXiv, abs/1906.025..5 025449, 2019b.
11

---

Published as a conference paper at ICLR 2025
Zhenfang Chen, Lin Ma, Wenhan Luo, and Kwan- Yee Kenneth Wong. superv-s spatio- temporally grounding natural sentence in video. In Anna Korhonen, David Traum, and Lluís Märquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1884-1894 Florence, Italy, July 2019c. Association for Computational Linguistics. htt.tttss: ac www.nhholll.....oo/g.o.e
doi: 10.18653/v1/119 1183.
Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real-time open voccbuocayu object detection. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2024.
Samyak Datta, Karan Sikka, Anirban Roy, Karuna Ahuja, Devi Parikh, and Ajay Divakaran. Align2ground: Weakly supervised phrase grounding guided by image-caption alignment. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 2601-2610, 2019.
Ishan Dave, Zacchaeus Scheffer, Akash Kumar, Sarah Shiraz, Yogesh Singh Rawat, and Mubarak Shah. Gabriellav2: Towards better generalization in surveillance videos for action detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops. pp. 122-132, January 2022.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Associa- tion for Computational Linguistics, 2019. URL https::/ httpt:ps:/aa...mmme..s. sew.eeisssmmtcccccc...ooo
org/
GoxpusID:529677
Zi-Yi Dou, Aishwarya Kamath, Zhe Gan, Pengchuan Zhang, Jianfeng Wang, Linjie Li, Zicheng Liu, Ce Liu, Yann LeCun, Nanyun Peng, et al. Coarse- fo-nieettt-eeeeeeee vision- langugge pre-training with fusion in the backbone. Advances neural information processing systems, 35:32942-32955 2022.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. arXiv preprint arXiv:2211.07666 2022.
Chengjian Feng Yujie Zhong, Zequn Jie, Xiangxiang Chu, Haibing Ren, Xiaolin Wei, Weidi Xie, and Lin Ma. Promptder: Towards ppen-oocabulary detection using uncurated images. In European Conference on Computer Vision, pp. 701-717. Springer, 2022.
J. Gao, Chen Sun, Zhenheng Yang, and Ramakant Nevatia. Tall: Temporal activity localization via language query. 2017 IEEE International Conference on Computer Vision (ICCV), pp. 5277-5285, 2017.
Xin Gu. Heng Fan, Yan Huang. Tiejian Luo. and Libo Zhang. Context-guided spatio-temporal video grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18330-18333 2024.
Tanmay Gupta, Arash Vahdat, Gal Chechik, Xiaodong Yang, Jan Kautz, and Derek Hoiem. Con- trastive learning for weakly supervised phrase grounding. In European Conference on Computer Vision, pp. 768. Springer, 2020.
Lei Jin, Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Guannan Jiang, Annan Shu, and Rongrong Ji. Refelip: A universal teacher for weakly supervised referring expression comprehension. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2681-2690, June 2023.
Yang Jin, Yongzhi Li, Zehuan Yuan, and Yadong Mu. Embracing consistency: A one-stage approach for spatio-temporal video grounding. ArXiv, abs/2209. 2022.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 1780-1790. 2021.
Akash Kumar and Yogesh Singh Rawat. End-to-end semi-supervised learning for video action detec- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
12

---

Published as a conference paper at ICLR 2025
Akash Kumar, Ashlesha Kumar, Vibhav Vineet, and Yogesh Singh Rawat. Benchmarking self- supervised video representation learning. Neural Information Processing Systems 4th Workshop on Sepf-Supee Learning: Theory and Practice, 2023.
Akash Kumar, Sirshapan Mitra, and Yogesh Singh Rawat. Stable mean teacher for semi-supervised video action detection. Proceedings of the AAAI Conference Artificial Intelligence, Mar. 2025.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded eogguage-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10965-10975, 2022.
Mengze Li, Han Wang, Wenqiao Zhang, Jiaxu Miao, Zhou Zhao, Shengyu Zhang, Wei Ji, and Fei Wu. Winner: Weakly-supervised hierarchical decomposition and alignment for video grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 23090-23099 June 2023.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Part V 13, pp. 740-755. Springer, 2014.
Z. Lin, C. Tan, J. Hu, Z. Jin, T. Ye, and W. Zheng. Collaborative static and dynamic vision- language streams for spatio-temporal video grounding. In 2023 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 23100-23109, Los Alamitos, CA, USA,
jun 2023. IEEE Computer Society, doi: 10. 20.100//CVPP22272.222..22222 20230222.. URL https //doi.
wttp:///....11////........ 20273.2..222...... 02212
Zhijie Lin, Zhou Zhao, Zhu Zhang, Qi Wang, and Huasheng Liu. Weakly-superrised video moment retrieval via semantic completion network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 11539-11546, 2020.
Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang, DAB-DETR: Dynamic anchor boxes are better queries for DETR. In International Confer- ence on Learning Representations, 2022a. TRRphttss:///seeeeeee
net forum?id=
j0b91]
Siyi Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chun yue Li, Jianwei Yang, Hang Su, Jun-Juan Zhu, and Lei Zhang. Grounding dino: Marrying dino with grounded pre- training for open-set object detection. ArXiv, abs/2303.05499, 2023. URL https://api semanticscho..
ortp/oorpusD:25542222
Xuejing Liu, Liang Li, Shuhui Wang, Zheng-Jun Zha, Dechao Meng. and Qingming Huang. Adaptive reconstruction network for weakly supervised referring expression grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2611-2620. 2019.
Xuejing Liu, Liang Li, Shuhui Wang, Zue Zha, Zechao Li, Qi Tian, and Qingming Huang. Entity-enhanced adaptive reconstruction network for weakly supervised referring expression grounding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 4503)3003-3018, 2022b.
Yongfei Liu, Bo Wan, Lin Ma, and Xuming He. Relation-aware instance refinement for weakly supervised visual grounding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 5612-5621, 2021.
Neelu Madan, Andreas Møgelmose, Rajat Modi, Yogesh S Rawat, and Thomas B Moeslund. Founda- tion models for video understanding: A survey. Authorea Preprints, 2024.
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenbom, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Amab, Mostafa Dehghani, Zhuoran Shen, al. Simple vocabuacyy object detection. In European Conference on Computer Vision, pp. 758-755. Springer, 2022.
13

---

Published as a conference paper at ICLR 2025
Rajat Modi, Aayush Jung Rana, Akash Kumar, Praveen Tirupattur, Shruti Vyas, Yogesh Singh Rawat, and Mubarak Shah. Video action detection: Analysing limitations and challenges. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 4907-4916, 2022. URL httpt:////ppp atemant: escholar wtt/Corpus 1D:
Jeffrey Pennington, Richard Socher, and Christopher Manning, GloVe: Global vectors for word representation. In Alessandro Moschitti, Bo Pang, and Walter Daelemans (eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532-1543, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/v1/D14--1 1162. -1w.://1yy.00001-1111111101111
URL
Aayush Rana and Yogesh Rawat. Are all frames equal? active sparse labeling for video action detection. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 14358-14373. Curran Asso- ciates, Inc., 2022. URL http:tttss///pcceeeeiiii wtw.:gg.....uuiieeee rwwr wttpcccpppppp.peeec wttppppprrrilleeppppperpaeeepp wwoCoooofeeecoo Confereeee pdf
Aayush J. Rana and Yogesh S. Rawat. Hybrid active learning via deep clustering for video action de- tection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18867-18877, June 2023.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6):1137-114 2017. doi: 10. 110//PAAM.221122570011.
Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor Darrell, and Bernt Schiele. Grounding of textual phrases in images by reconstruction. In European Conference on Computer Vision, 2015.
URL http:::://///. //ap:://pp... /pp........... happi.eeeme semant www.amiitccccooocooo ar. wwtoor//oooo Ctrr/orrpssss 9926549 9
Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor Darrell, and Bernt Schiele. Grounding of textual phrases images by reconstruction. In Computer TisionEECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14, pp. 817-834. Springer, 2016.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In 20/9 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8429-84488 8438, 2019. doi: 10. 109/ 2019.00852.
Jing Shi, Jia Xu, Boqing Gong, and Chenliang Xu. Not all frames are equal: Weakly-supervised video grounding with contextual similarity and visual clustering losses. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10436-10444, 2019, doi: 10. N0.1109/CVPR.221.01000
Ayush Singh, Aayush J Rana, Akash Kumar, Shruti Vyas. and Yogesh Singh Rawat. Semi-supervised active learning for video action detection. Proceedings of the AAAI Conference on Artificial Intelligence, 488 4899, Mar. 2024. doi: 10. 20..1609aaai..3885.228888 28222.. URL WtRl:/tps:://ff aaai org/ wnwex.iiii.i.cic hwtp://ppp/paaee wwwp://ww.a///aaa http//atiiicleeeeeeeeeee /
Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe. Curriculum learning: A survey, Int. J.
Compur. Vision 13006))1526-155663 jum 2022. ISSN 0920- doi: 10. 12007//11122---22-166111111
Rui Su, Qian Yu, and Dong Xu. Stygbert: A visual-linguistic transformer based framework for spatio-temporal video grounding. In 202/ /EEE/CVF International Conference on Computer Vision (ICCV), pp. 1513-1522, 2021.
doi: 10. N0.1109//CC48892220000 1.0156..
Zongheng Tang, Yue Liao, Si Liu, Guanbin Li, Xiaojie Jin, Hongxu Jiang, Qian Yu, and Dong Xu. Human-centric spatio-temporal video grounding with visual transformers. IEEE Transactions on Circuits and Systems for Video Technology, 32:8238-8244 2020.
Liwei Wang, Jing Huang, Yin Li, Kun Xu, Zhengyuan Yang, and Dong Yu. Improving weakly supervised visual grounding by contrastive knowledge distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14000-14100, 2021a.
14

---

Published as a conference paper at ICLR 2025
Qinxin Wang, Hao Tan, Sheng Shen, Michael Mahoney, and Zhewei Yao. MAF: Multimodal alignment framework for weakly-supervised phrase grounding. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2030-2038, Online, November 2020.
Association for Computational Linguistics. doi: 10. http:///ww/22200..mmmmmmam.
URL
Sutt:s///beaaattoooooyoor////// wwtp://0..//0/////ooooo emplp main. 159
X. Wang, Y. Chen, and W. Zhu. A survey on curriculum learning, JEEE Transactions on Pattern Analysis and Machine Intelligence, 44(09):4555-4556 sep 2022a. ISSN 1939-3539. doi: 10.1109/ TPAMI.2021.300 .3069908.
Yingming Wang, Xiangyu Zhang, Tong Yang, and Jian Sun. Anchor detr: Query design for transformer-based detector. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 2567-2575,
Yuechen Wang. Jiajun Deng, Wen gang Zhou, and Houqiang Li. Weakly supervised temporal adjacent network for language grounding. IEEE Transactions on Multimedia, 24:3276-322 -3286, 2021b. URL
https://app semanticscii ar. wrrg/ Corpus ID: 235683548
Zhenyu Wang, Yali Li, Xi Chen, Ser-Nam Lim, Antonio Torralba, Hengshuang Zhao, and Shengjin Wang. Detecting everything in the open world: Towards universal object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11433-11443, 2023.
Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsual Yang, and Fahad Shahbaz Khan. Videogrounding-ding Towards apen-vocabulary apatio-temporal video grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 18909-18918, 2024.
Masataka Yamaguchi, Kuniaki Saito, Y. Ushiku, and Tatsuya Harada. Spatio-temporal person retrieval via natural language queries. 2017 IEEE International Conference on Computer Vision (ICCV), pp. 1462-1477 1472,171,200 2017. https:/::
10363312
Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu. Universal instance perception as object discovery and retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15325-15336 2023.
Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Tubedetr: Spatio- temporal video grounding with transformers. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16421-16433 2022.
Lijin Yang, Quan Kong, Hsuan-Kung Yang, Wadim Kehl, Yoichi Sato, and Norimasa Kobori. Deco: Decomposition and reconstruction for compositional temporal grounding via coarse-to-fine contrastive ranking. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 23130-231400 2023. doi: 10. 10.1109/CVPR5522992222 202.2225222..2220222155
Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detelip: Dictionary-enriched visual-concept paralleled pre-training for open- world detection. Advances in Neural Information Processing Systems, 35:9125-9138, 2022.
Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision- language understanding. Advances in Neural Information Processing Systems, 35:36067-36080, 2022.
Zhu Zhang, Zhou Zhao, Yang Zhao, Qi Wang. Huasheng Liu, and Lianli Gao. Where does it cxist: Spatio-temporal video grounding for multi-form sentences. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10665-10674, 2020.
Minghang Zheng, Yanjie Huang, Qingchao Chen, and Yang Liu. Weakly supervised video moment localization with contrastive negative sample mining. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022a.
15

---

Published as a conference paper at ICLR 2025
Minghang Zheng, Yanjie Huang, Qingchao Chen, Yuxin Peng, and Yang Liu. Weakly supervised temporal sentence grounding with gaussian- -aased contrastive proposal learning. In 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 15534-15543, 2022b. doi:
N0.1109/CVPR5528882200 20222.011...2.111111.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
16

---

Published as a conference paper at ICLR 2025
A APPENDIX
Here, we provide some more details about our approach along with additional results and visual analysis. We also include tables which we were not able to include main paper due to space limitations.
Section
We address the challenges and limitations of detector and tracker.
Section
Qualitative Analysis on the model's predictions.
Section
We show more discussion and analysis.
Section
Training details about architectures, datasets, and, other hyperparameters.
Section Qualitative Analysis Detection and tracking, success and failure cases and analysis on the video in the wild.
B CHALLENGES AND LIMITATIONS
STVG datasets are extremely challenging, especially the HCSTVG-vl and HCSTVG-v2 where even detection and tracking fails, shown by maximum upper bound achievable in Table 12b The HCSTVG datasets contains sudden zoom shots, scene changes, and defocus, where even good detectors fail. The additional pre-processing to track the detections to generate tubelets introduce more noise and struggles to track the right person with person crossover, scene change (very high displacement in bbox leads it to assign different IDs), view change and only partial body availability. Due to these two main limitations, we propose to solve the task by breaking it into two sub-tasks. A future work involves exploiting temporal modeling associated with each individual object jointly; however, in current approach, we show promising results quantitatively and qualitatively.
C QUALITATIVE ANALYSIS (MAIN ARCHITECTURE)
show the effectiveness of our approach qualitatively. W-GDINO struggles with grounding the right actor as well as no temporal bounds, whereas our approach spatio-temporally grounds the actor better than baseline.
In Fig.
The woman in blue clothes takes something on the table and wraps it around her wrists a few times.
Query
Video
GT
W-GDINO
CoSPaL
The man blue clothes the side of the road, stops, turns around and speaks to the person in white, and then turns again.
Query
Video
GT W-GDINO CoSPaL
Figure 6: Qualitative analysis: We observe that W-GDINO detects without considering the context of the query, which is improved using the proposed method.
17

---

Published as a conference paper at ICLR 2025
WC
Noun
Comparison between GDINO query: Whole caption (WC) vs Noun. The first row shows detection boxes for whole query as input the GDINO against noun extracted from the query in second row. We observe that it focuses on other objects (for eg. suit (shown in orange, pink, yellow)) which may not be the target instance but overlapping with target instance and thus helps in better score. (Tab 7 Query for the above video (WC): The bald man leaves the room pulls the door walks towards the man in the white suit and then turns to face the white suit man. Noun: man'..
Figure 7:
D DISCUSSIONS
We include multiple discussions to support and strengthen the claims in our main paper:
Performance with Whole Caption (GDINO Input): In the main paper, we follow the traditional weakly supervised settings for fair comparison with previous SOTA, where train and test time the detector outputs ALL human/object bounding boxes, and, given the query, the output should be object tubelet with maximum attention. another setting, we analyze sending in the original caption and perform tracking on output detections. We have shown the difference in detection with only sending noun vs whole caption in Fig. 7 WC setting output detections which doesn't 't correspond to all subjects overlapping detections to specifie subject. In Table 7 we compare three settings. Training and testing on noun extracted from query (Noun), Train and test with whole caption (WC), and, finally, Train on WC and test with Noun. Looking second row, input Grounding DINO with extra information helps. To compare it with traditional weakly settings, third row we perform test with detections using Noun output. This study suggests that whole captions as query generates better detections Grounding DINO, although it might not adhere to traditional
settings.
Table 7: Grounding DINO Input: Noun Whole Caption.
Train Test IloU m vIoU
vloU@0.3 vIo
Noun Noun 37.6
19.2
28.8
15.3
WC WC 34.5
22.7
32.5
18.2
WC Noun 35.0
18.6
26.8
15.0
Improvement in performance with SPS: From Table 5 we consistently observe a 2-3% boost for each setting with inclusion of SPS. This shows that increasing scene understanding is complementary both baseline and baseline+CRG settings. Going in-depth analysis, Tables Sa Sc we show the improvement by SPS based training for all three settings TPG only, CRG only, and, TPG + CRG. Self- delf-paced learning boosts score in each of the settings by 2.4, 3.4, and, 5.0 respectively. This shows the efficacy how self- paced scene understanding training paradigm helps network become more discriminative with time both spatially and temporally. This is also corroborated by the fact that training via SPS paradigm outperforms single-stage training on the whole dataset (shared in Table 5 main paper).
Analysis on Text encoder: Grounding DINO finetunes the vision encoder but keeps the text encoder fixed. The vision backbone is fixed to Swin-T. T. For textual features, explore two choices to find the best alignment between vision and text to begin with. From Table BERT outperforms CL.P on the baseline settings, TPG. Thus, choose BERT as encoder for all experiments.
18

---

Published as a conference paper at ICLR 2025
Table 8: Analysis on SPS all three situations.
(a) TPG only.
(b) CRG only.
(c) TPS and CRG,
Stages m_tooU m_vloU v@0.3
vi00.5
Stages HoU m_ vnvil
ve0.3
v@00.
Stages
doU m. vlv v@0.3
v@0.
34.1
17.7
26.0
14.4
33.4
17.7
24.6
14.8
1
32.3
17.1
24.4
14.0
36.2
18.5
27.0
14.8
36.3
19.6
28.8
16.3
11
37.2
19.9
28.9
16.7
20.1
28.5
17.6
19.6
Table Choice of Textual Encoder: CLIP BERT.
Encoder tloU m_ m_vloU v.aU@0.3
v.0
CLIP 35.7
18.8
28.8
14.8
BERT 37.6
19.2
28.8
15.3
Study on Decoder Layer features: We perform analysis on TPG with different decoder layer features. Since G-DINO shares architecture with DETR, we extract features from six layers of decoder and ran our baseline. In Table [0] we show the performance with features from different decoder layers. We observe features from decoder layer performed the best. To further refine background noise, restrict the number of tubelets for our settings to 10. The last row (Table I0) shows that it further boost the performance by 0.8%.
We perform standalone analysis classification accuracy and temporal grounding metrics from previous works (Zheng et al. 2022a by Lin et al. (220)) in Table 11 In classification accuracy, observe approach outperforms W-GDINO by 20% and baseline TPG by 3.2%. For temporal IoU metrics, we observe including contextual phrases boost the performance further all IoUs.
Standalone classification and temporal scores:
Analysis on multiple IoUs: In Table 12a we show performance comparison ranging from 0. till 0.7 HCSTVG dataset. CoSPaL outperforms TPG and W-GDINO at all IoUs. Our proposed approach is more effective at higher IoUs, showing gain of 4.3% and 4.1% at 0.5 and 0.7 IoU respectively. We perform similar analysis on VidSTG dataset comparing performance multiple IoU ranging from 0.1 till 0.7. Tables 13a and 13b shows that proposed approach outperforms both W-GDINO and TPG at all loUs.
Upper bound Analysis: To quantify how challenging HCSTVG-v1, HCSTVG-v2 and VidSTG datasets are, we perform an analysis to find the upper bound, that maximum achievable results. This analysis is necessary since it tells how challenging detection and tracking is on these datasets. set the temporal bound 100% from ground truth. Looking at Table 12b if the network works perfectly, our proposed module can achieve max 62.3, 52.5, 45.3, 39.8 19.8m m vIoU on HCSTVG-v1, HCSTVG-v2,
and, HudSTG-Interrogativy respectively. With respect to that our current approach achieves effective performance of 35.4, 42.3, 28.5, 28.6 percentage of maximum achievable.
E EXPERIMENT DETAILS
E. DETECTION AND TRACKING
Grounding DINO involves two hyperparameters namely text and box threshold. We set to 0.4 for both. Setting a lower or higher values leads to oversampling or missed detections. Since dataset contains multiple resolution of images, set the image width to 480 if original frame width less than 550, else 800.
Detector:
Tracker: The parameters set for BoTSORT tracker are: 1) new track threshold: 0.21, 2) Low track threshold: 0.1. 3) High track threshold: 0.34, 4) Matching threshold: 0.21, 5) Appearance threshold: 0.48, and, 6) Buffer frames: 60 keep track of the object id for 60 number of frames.
19

---

Published as a conference paper at ICLR 2025
Table 10: Comparison with different decoder layer features, Last row f shows further refinement to restrict upper bound on number of tubelets help.
Layer m_tloU
vloU
v.0 vloU @0.5
35.8
18.4
26.7
15.3
35.4
18.0
26.9
15.0
III
35.6
17.7
25.7
14.3
17.8
26.2
33.5
18.1
26.4
14.9
VI
34.6
17.9
26.1
15.2
It
37.6
19.2
28.8
15.3
Table 11: Analysis standalone classification accuracy and temporal IoU.
(a) Classification Accuracy.
(b) Temporal IoU.
Method
TPGrQuery) NAViPhrases) IoU @00.
1oU@D
IDIIO 18.7
74.1
54.1
23.0
TPG 35,5
76.2
55.6
23,8
CoSPaL 38.7
Table 12: Analysis on multiple factors showcasing effective of our proposed approach. In Table 12b VidSTG-D means VidSTG Declarative and VidSTG-I means VidSTG Interrogative.
(b) Upper-bound Analysis.
(a) Analysis on multiple IoUs on HCSTVG dataset.
Method vIoU v@0.1
v@0.2
v@0.3
v@0.5 v@0.7
Dataset
m. tlu m vloU v.oU@0.5
HCSTVG-vI
79.2
62.3
69.5
W-GDINO
9.0
25.9
17.3
11.6
4.6
0.7
HCSTVG-v2
76.3
52.5
54.6
TPG 19.2
43.1
36.2
28.8
15.3
5.4
VidSTG-D
66.9
45.3
46.8
CoSPaL 22.1
45.6
38.7
31.6
19.6
9.5
VidSTG-I
66.2
39.8
39.2
Table 13: Analysis on multiple IoUs showcasing effectiveness of our proposed approach.
(a) VidSTG--Declaattie
(b) VidSTGG-Imerrrgaatee
Method m. vloU ve0.1
v10022
v@0,3
v@0.5
v@0.7
Method m vloU ve0.1 v600. v@0.3 v@U.5 ve0.7
W-GDINO
10.6
25.0
17.6
13.0
7.8
4.1
W-GDINO
9.8
23.2
16.5
12.2
6.7
3.5
TPG 12.9
28.2
20.9
16.2
9.9
5.6
TPG
11.4
26.8
18.8
14.0
8.0
4,5
CoSPal.
16.0
33.6
25.8
20.1
13.1
CoSPaL
13.5
30.3
22.0
16.4
10.2
5.7
E.2 ARCHITECTURE HYPERPARAMS SETTINGS
Weakly-GDING For weakly-GDINO. we input whole text as the query and frame from video as image input. Frames sample with stride of 5. To calculate the GDINO predictions for video, Firstly, we run the tracker to generate all tubelets in the video. To evaluate, we average the confidence of each tubelet across temporal dimension. The predicted tubelet is assigned the the tubelet with highest average confidence score. The starting and ending timestamp of the predicted tubelet is used for temporal IoU calculation.
Tubelet Phrase Grounding: contains two modules spatial and temporal grounding. The batch size is set to 32. In spatial grounding module, we use Adam optimizer with a learning rate of 1e-4. The maximum length for number words in text is set to 25 for HCSTVG. Temporal grounding module had Adam optimizer with learning rate 4e-4.
Contextual Referral Grounding We use GPT-3.5 to extract referral tubelet attributes (Qoa) and referral tubelet action verbs (Q.a). The input query Qn and Qu to the GPT to extract Qea and Q.. respectively as below:
20

---

Published as a conference paper at ICLR 2025
Qo: Extract the quantifier phrase describing the main person.
Qu: Break the complex actions into simpler actions.
We provide few examples of original texts and extraction from GPT-3 for both scenarios. For first case, extraction of main obejet in context and attributes related to its are follows:
Q1: The bearded woman walks to the woman in gray clothes and touches her face.
Al: The bearded women.
Q2: The man the brown hat drops the hat of the man in the black hat then pushes the opposite man then turns and punches the man in the back.
A2: The
in the brown hat.
Q3: The woman with yellow hair walks from the right
to the left of the man in leather then pulls his arm away.
A3: The woman with yellow hair.
In case of main actor and it's attribute extraction, GPT-3 worked perfectly. However, breaking complex actions into sub-actions, GPT-3 faced challenges and sometimes hallucinates which activity belongs to which actor. One success case as follows:
Q1: The bald man leaves the room pulls the door walks towards the man in the white suit and then turns to face the white suit man.
Pl: The bald man leaves the room.
P2: He walks towards the man in white suit.
P3: He turns to face the white suit man.
One failure case as follows:
Q1: The man the black military uniform catches the things thrown by the opposite man with both hands turns and bends over pick up his hat and puts on it.
Pl: The man the black military uniform catches the
things.
P2: He throws the thing.
P3: Hе turns and bends over.
P4: He pick up his hat.
In above scenario, relates to the activity by the actor not in main context. We filter out these phrases by looking into verbs in active tense. Those verbs denote activity performed by the main actor.
Self-paced Scene understanding: In SPS curriculum based learning, we set the upper bound on the number of object tubelets per video. The first stage bound is set to videos with only upto tubelets and it's incremented by 3 in each stage for two more stages. In last stage, the number of tubelets is 10 and it contains all the videos.
E.3 COMPUTE REQUIREMENTS
For our work, we run our models on single 16 GB Tesla V100 GPU with batch size of 32. The training time for HCSTVG-v1 is 4-5 hours, HCSTVG-v2 id 7-8 hours and VidSTG it's 10-12 hours.
21

---

Published as a conference paper at ICLR 2025
Thresh=0.3 Thresh=0.4
Figure 8: Comparison between threshold for GDINO: The first row shows detection boxes with threshold set to 0.4 and the second row shows the detection with threshold set to 0.3. We see few missed detections in earlier case, however, later, overlapping detection issues arises. Even in second scenario, in third frame lowering confidence didn't 't help. The detection was missed. Query text: Noun: mann..
WiO
0.5
0.5
0.5
0.5
0.6
0.7
0.5
0.2
0.2 0.1
Figure 9: Effeet of Temporal Attention: Without temporal attention (w/o TA) in first row, observe that each frame gets equal weight, however, utilizing temporal attention (w/ TA, second row) increases weight key frames and decrease weight for non-important frames relation to query. Query: The woman holding the child walks to the side of stone bench stops hands the child the woman next to her and walks to the front of the stone bench
E.4 SOCIETAL IMPACT
The proposed work could be used for surveillance and query is not descriptive enough can ground the wrong person leading to possible harm. However, the positive aspect, the proposed work is free of biasness issues due to use foundation models (trained on bigger datasets) and can be deployed in wild.
QUALITATIVE ANALYSIS
F.I FAILURES IN DETECTION AND TRACKING
In this qualitative analysis, show the inherent failure of Grounding DINO al. 2023) and tracker (Aharon et al. 2022)
F.1.1 DETECTION FAILURE
In Fig. 8 we show that GDINO fails to detect the person. If we reduce threshold, it is able to detect, but, then it leads overlapping detections which will add one another step of post-processing of non-maxima suppression.
F.1.2 TRACKING FAILURE
There two type of failure that happens in tracking: 1) Assigning same ID to different objects, and, 2) Different IDs to same objects. In both scenarios, tubelet features get impacted. Fig. [3] illlsstraees both the failures.
22

---

Published as a conference paper at ICLR 2025
Query The man black shirt goes towards the man in brown coat and picks up the book.
Video
GT
W-GDINO CoSPaL
Figure 10: Qualitative Analysis: W-GDINO struggles to attend to the query and switch between actors across time. Our proposed approach is able to detect the main actor in context (from textual query) almost correctly spatio-tena temporaly..
The man lying on the wall takes something out of his pocket and hands it to the man behind him.
Query
Video
GT CoSPaL
Figure 11: Qualitative Analysis (Failure scenario): In these scenarios, visual features are quite similar and query description challenging extract the attributes related the main actor in context.
F.2 EFFECT OF TEMPORAL ATTENTION
In this analysis show how temporal attention applied tubelet helps. shows impact of with and without temporal attention. With temporal attention across temporal dimension, key frames that has higher mutual information in relation to query given higher weight.
F.3 RANDOM VIDEO ANALYSIS IN THE WILD
We take random video from the internet and run our proposed approach. In Fig. 10) we show the comparison between ours against W-GDINO. We pick a video from movie scene Steve Jobs and detector and tracker and then trained weights to predict the tubelet given the query. We formulate the query and video length on our own for this experiment.
F.4 SUCCESS AND FAILURE CASES
Fig. shows failure scenario of our model. We observe model fails when query description doesn't I explicitly contains specific attributes describing the main actor in context and spatial features of objects are very similar.
Fig. 12 shows success scenario. In first example (top row), since the model doesn't 't contains any information about background or other actors, W-GDINO in this scenario works. However, since doesn" it have understanding of time, our approach is temporally localize the action. Bottom shows a challenging example where our method performs better. In general. proposed approach works good when the query contains attributes related to main actor (referral). This shows that proposed use of Contextual Referral grounding aspect helps in the scenario.
23

---

Published as a conference paper at ICLR 2025
Query
The woman in the red dress takes a cup and takes a sip.
Video
GT
W-GDINO CoSPaL
The man the white shirt walks up to the man in the red suit and takes the whip of the man in the hand of the in red suit.
Query
Video
GT W-GDINO CoSPaL
Figure 12: Qualitative Analysis (Success scenario): The proposed approach able to properly spatio-temporally localize the actor and activity associated with it. Top Row: shows an easy example where W-GDINO also succeeds since the query contains description about one actor. However, lacks temporal understanding and thus unable localize the activity temporally. Bottom row: It shows a very hard example where there are query contains description about multiple actors in context. W-GDINO focuses the background actor whereas work is able to properly spatio-temporally localize the correct tubelet (referral tubelet).
Different IDs, Same Objects
Same IDs, Different Objects
Track #1
Track #1
Track #2
Track #2
Track 48
Track
Track #4
Track
Figure 13: Tracking failures: Left: Different IDs, Same Objects Tracks in red color are repetition of same earlier ID but assigned same IDs, but assigned different track IDs; Right: Same IDs, Different Objects red boxes denotes switching of ID happened. Same id is assigned even if the object/actor is different.
track. Tracks I and 4 are same IDs. and, tracks and are
24


Reviews:
[{"review_id": "06cfIDU7UN", "reviewer": "Reviewer_qQgs", "summary": "This paper introduces CoSPaL (Contextual Self-Paced Learning), a novel approach to Weakly Supervised Spatio-Temporal Video Grounding (WSTVG), which aims to locate objects in videos based on text descriptions without requiring detailed bounding box annotations during training. CoSPaL proposes three main components to overcome limitations of existing methods: Tubelet Phrase Grounding (TPG) for improved object tracking, Contextual Referral Grounding (CRG) for better query comprehension, and Self-Paced Scene Understanding (SPS) for progressive learning of complex scenarios. Results are reported on common grounding benchmarks such as VidSTG and HC-STVG-v1/v2.", "questions": "Please consider the weaknesses section. Essentially, my concerns are regarding the impact of the video encoder and the impact of the choice of tracker. I wish to understand how sensitive the method is to these two components, and whether the improvements are coming from the overall proposed design or certain specific components. I will further discuss this with fellow reviewers before making a final decision on the rating.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea is innovative and well motivated.\n2. The paper is well-written and easy to follow.\n3. The method achieves strong performance improvements with respect to the baselines.", "weaknesses": "I have a few major concerns regarding the proposed method:\n\n1. Regarding the Video Encoder: Existing works such as STCAT and TubeDETR utilize a Resnet-101 backbone to encode each frame. Even recent works such as VGDINO use the Grounding-DINO frozen image encoder. With this method using I3D as a separate video encoder, it could be an unfair comparison with existing works. How does this method perform if the authors utilize the same features as produced by the Grounding-DINO backbone as the video features? To make it consistent with previous works.\n\n2. The method uses a pretrained tracker in the pipeline. I am concerned regarding the impact of this tracker, and how the performance changes if a different tracker is used? There does not seem to be any ablations regarding this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R4Nolzcb2f", "reviewer": "Reviewer_BmrP", "summary": "Targeting the issues of inconsistent predictions in time judgment, difficulty in comprehending complex queries, and the complexity of application scenarios faced by weakly supervised Spatio-Temporal Video Grounding (STVG), this paper introduces the self-paced learning method, which has achieved certain performance improvements on two conventional datasets.", "questions": "Please refer to the Weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The problem addressed in this paper is of certain significance and is interesting.\n2. The introduction of self-paced learning is reasonably justified.\n3. Experimental results on different datasets indicate that the algorithm has achieved certain improvements.", "weaknesses": "1. The motivation of introducing self-paced learning should be illustrated more clearly. Are there other methods that can address the challenges presented in this paper? It is suggested to provide a more detailed explanation in the related work section.\n2. In Section 3.2.1, the tracking algorithm in the TPG module seems to play a vital role in learning the whole module. It makes a good start with GroundingDINO in the whole model, and it would be beneficial to add an analysis of the ablation of the tracking algorithm.\n3. Also, in section 3.2.1, the part of the Spatial Grounding Module, the formula description seems confused. It does not give key dimension information, such as in the similarity calculations: $\\text{SIM}(f_{w_m},f_{\\tilde{T_k}})=(\\mathrm{MLP_{q}}(f_{w_{m}})^{T}\\mathrm{MLP_{k}}(f_{\\tilde{T_{k}}}))/\\sqrt{d}$, where $f_{w_m}\\in \\mathbb{R}^{1\\times768}$ and $f_{\\tilde{T_{k}}}\\in\\mathbb{R}^{T\\times{256}}$ obtains from the above, so the $MLP_q(f_{w_m})\\in\\mathbb{R}^{1\\times{D}}, MLP_{k}(f_{\\tilde{T_{k}}})\\in\\mathbb{R}^{T\\times{D}}$, where $D$ is the MLP output dimension, but if it is like this, then matrix multiplication is failed. Besides, in $\\mathrm{A_T}(f_{T}, f_{w_{m}})=\\sum_{k=1}^{K}\\operatorname{softmax}\\left(f_{\\tilde{T}_{k}}, f_{w_{m}}\\right) \\mathrm{MLP}_{v}\\left(f_{\\tilde{T}_{k}}\\right)$, what is the mean about $f_T$? Is $\\sum_{k=1}^{K}$ meant to be a matrix addition of all tubelet features? Then it will give $ A_\\in\\mathbb {R}^{1\\times{D}}$; at this point, how to get the distribution of scores between different tubelets? This part of the equation is confusing.\n4. The comparison parameters of GPU memory use and training time in Figure 5 are ambiguous because the fully supervised models compared in the figure are experimented on different resolutions. However, the paper does not list the information about the resolution of the fully supervised models compared at the time of statistics and the resolution of their own models and whether the hardware parameters (e.g., information about CPU, GPU, memory) are unified across the different models, which I think are essential settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0hG1CxkYoW", "reviewer": "Reviewer_4gRV", "summary": "This paper proposes CoSPaL (Contextual Self-Paced Learning) for Weakly Supervised Spatio-Temporal Video Grounding (WSTVG). The method introduces three key components: Tubelet Phrase Grounding (TPG), Contextual Referral Grounding (CRG), and Self-Paced Scene Understanding (SPS). CoSPaL aims to improve spatio-temporal video grounding by enhancing the model’s ability to understand complex queries and progressively adapt to more difficult scenarios. The effectiveness of CoSPaL is validated on three benchmark datasets, with significant performance improvements over baselines.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper presents a method CoSPaL to address the key issues in WSTVG.\n2.\tThe paper includes comprehensive quantitative analysis and visualization, providing empirical evidence to support the proposed method.\n3.\tThe proposed CoSPaL model shows strong performance gains on multiple datasets, with notable improvements of 3.9% on VidSTG and 7.9% on HCSTVG-v1.", "weaknesses": "1.\tThe technical contribution and motivation is unclear. For example, TPG module uses cross-modal attention, contrastive learning, and feature reconstruction to facilitate interactions between words and tubelets, these techniques are common in the field. Why this particular implementation contributes to improved spatial and temporal grounding?\n2.\tThe SPS component is not clearly described. It is unclear how sample complexity is determined within the curriculum learning framework, is the object number? But intuitively, videos with more objects, faster changes, and more interactions are more complex. \n3.\tWhat does “Self-Paced” mean? This concept is not clearly explained in the paper.\n4.\tIn Table 5, the introduction of SPS in the TPG module leads to a performance drop in vIoU@0.3 (TPG+SPS vs TPG). This raises concerns about the efficacy of SPS in certain settings. The authors should provide a deeper analysis of why this occurs.\n5.\tIn page 10, the part of Impact on actor localization, the corresponding quantitative results for this claim are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FcsStuMnE8", "reviewer": "Reviewer_tfEJ", "summary": "This paper targets the Weakly Supervised Spatio-Temporal Video Grounding (WSTVG) task, which tries to retrieve specific objects and segment by sentence queries without relying on labeled data. The spatial and temporal grounding modules are used to mine the spatio-temporal information. Also, a contextual referral grounding module is utilized to extracts contextual information from query. Finally, a self-paced curriculum learning strategy is adopted for optimization. Some experiments are conducted.", "questions": "Please address the weakness.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "Three WSTVG benchmarks are used for performance evaluation.\n\nThe compared methods are state-of-the-art. CG-STVG and VGDINO are published in 2024.\n\nThe figures and tables are clear.\n\nVarious figures are used to present the performance.", "weaknesses": "In Abstract, authors state that “we first explore the potential of state-of-the-art object detection models for WSTVG”. In fact, it is wrong since many STVG methods use object detection models, e.g., [1].\n\nWhat is the difference between “self-paced scene understanding” and “self-paced curriculum learning”. In Section 3.2.3, authors use “self-paced scene understanding” as the section title, but all the contents describe the self-paced curriculum learning strategy. Besides, the strategy has been used in many works (Wang et al., 2022a; Soviany et al., 2022). Why you treat it as your third contribution and your title?\n\nThe technological novelty is not enough for ICLR. Most modules in the presented network in Figure 3 are popularly used. For example, cross attention in spatial grounding and temporal grounding.\n\nAuthors should conduct the main abalation study in the main paper.\n\nThe start and end timestamps need to be added in Fig 4.\n\nSome grammatical errors. For example, \"three benchmark WSTVG datasets\" in Abstract.\n\n[1] Weakly-Supervised Spatio-Temporal Video Grounding with Variational Cross-Modal Alignment", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)