OmniSTVG: Toward Spatio-Temporal Omni-Object Video Grounding
Authors: Jiali Yao
Xin Gu
Xinran Deng
Mengrui Dai
Bing Fan
Zhipeng Zhang
Yan Huang
Heng Fan
Libo Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Spatio-Temporal Video Grounding
Spatio-Temporal Omni-Object Video Grounding
Benchmark
Primary area: datasets and benchmarks

Abstract: We introduce spatio-temporal omni-object video grounding, dubbed $\textbf{OmniSTVG}$, a new STVG task aiming to localize spatially and temporally all targets mentioned in the textual query within videos. Compared to classic STVG locating only a single target, OmniSTVG enables localization of not only an arbitrary number of text-referred targets but also their interacting counterparts in the query from the video, making it more flexible and practical in real scenarios for comprehensive understanding. In order to facilitate exploration of OmniSTVG, we propose $\textbf{BOSTVG}$, a large-scale benchmark dedicated to OmniSTVG. Specifically, BOSTVG contains 10,018 videos with 10.2M frames and covers a wide selection of 287 classes from diverse scenarios. Each sequence, paired with a free-form textual query, encompasses a varying number of targets ranging from 1 to 10. To ensure high quality, each video is manually annotated with meticulous inspection and refinement. To our best knowledge, BOSTVG, to date, is the first and the largest benchmark for OmniSTVG. To encourage future research, we present a simple yet effective approach, named $\textbf{OmniTube}$, which, drawing inspiration from Transformer-based STVG methods, is specially designed for OmniSTVG and demonstrates promising results. By releasing BOSTVG, we hope to go beyond classic STVG by locating every object appearing in the query for more comprehensive understanding, opening up a new direction for STVG. Our project is released at https://jellyyao3000.github.io/OmniSTVG/.

Published as a conference paper at ICLR 2026
OMNISTVG: TOWARD SPATIO-TEMPORAL OMNI- OBJECT VIDEO GROUNDING
Jiali Yao Xin 2u Xinran Deng² Mengrui Dai
Bing Fan*
Zhipeng Zhang Yan Huang' Heng Fan Libo Zhang
Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences Uniiveriity of Chinese Academy of Sciences "Notth China University of Technology UUniversity of North Texas SShnghai Jiao Tong University
"Institute Software Chinese Academy of Sciences
yaojiall 124 @0ma i 1s ucass....... ac..c. en, libeê scas cn
ABSTRACT
We introduce spatio-tempon omni-object video grounding, dubbed OmniSTVG. new STVG task aiming to localize spatially and temporally all targets mentioned in the textual query within videos. Compared to classic STVG locating only a single target, OmniSTVG enables localization of only an arbitrary number of text-referred targets but also their interacting counterparts in the query from the video, making more flexible and practical in real scenarios for compre- hensive understanding. In order to facilitate exploration of OmniSTVG, we pro- pose BOSTVG, large-scale benchmark dedicated to OmniSTVG. Specifically, BOSTVG contains 10,018 videos with 10.2M frames and covers wide selection of 287 classes from diverse scenarios. Each sequence, paired with free-form textual query, encompasses varying number of targets ranging from I to 10. To ensure high quality, each video is manually annotated with meticulous inspec- tion and refinement. To our best knowledge, BOSTVG, to date, is the first and the largest benchmark for OmniSTVG. To encourage future research, we present simple yet effective approach, named OmniTube, which, drawing inspiration from Transformer-basse and demonstrates promising results. By releasing BOSTVG, we hope to go be- yond classic STVG by locating every object appearing in the query for more com- prehensive understanding, opening up new direction for STVG. Our project released at https: http::///elllllyyeellm g thub. 1 Omn 1SVVG/
STVG methods, is specially designed for OmniSTVG
INTRODUCTION
Query: A man in black catches the soccer
(a) Current tentioteempoall vidco grounding (STVG) localizing single target
Query: As elephant steps on seesunn the man holding a basketball jumps up
6.00s
*
10.076
(b) The proposed spatio-temporul omni-object video grounding (OmniSTVG) localizing all mentioned targets
Figure 1: Comparison existing STVG that localizes a single object in the query (a) and our Om- niSTVG localizing all objects in the query (b). The object in the query and its corresponding spatio- temporal tube in the video is highlighted using the same color. Best viewed in color for all figures.
"Equal contribution
Equal advising and corresponding authors

---

Published as a conference paper at ICLR 2026
Spatio-temporae video grounding (STVG) (Zhang et al., 2020b) has been one of the crucial problems in multimodal video understanding. Given a free-form textual query, it aims at locating the target of interest in space and time from the video (see Fig. (a)). Owing to its crucial applications, including human-machine interaction, robotics, etc., STVG has gained increasing interest in recent years (Su et al., 2021; Yang et al., 2022; Jin et al., 2022; Lin et al., 2023; Wasim al., 2024; Gu al., 2025).
While significant progress has been witnessed, localizing only a single object, as it is done in current STVG, insufficient for video understanding in many real-world scenarios. For instance, in daily life, since an event or an activity often involves various objects (see Fig. (b)), it is common a textual query contains maltiple targets of interest (e.g.. "elephant and man" in query in Fig. 1 (b)). For such query, it is essential to localize every queried objects, spatially and temporally, within the video. Yet, existing STVG localizes only a single target in query (see Fig. I (a)). and hence is restricted multi- objective query localization, degrading practicability. To alleviate this, a straightforward solution is to employ the STVG model repeatedly for multiple single-object textual queries. Nevertheless, this markedly increases the computational burden, leading to the lack of the scalability in existing STVG.
In addition to the restriction in locating multiple queried targets, another limitation of current STVG is the ignorance of interacting counterparts for queried objects. In practice, the object of interest is usually not alone but interacts with other targets (see textual queries in Fig. 1 (a) and (b)). Local- ization of objects of interest, together with the interacting counterparts, provides richer contextual information for the objects, and thus allows more comprehensive spatio-temporal understanding of the video, which greatly benefits many applications, including surveillance, robotics, sport analysis, and so forth. In existing STVG, however, the crucial interacting counterparts are often neglected in localization, restricting the classic STVG for more comprehensive analysis.
Contribution. To mitigate aforementioned limitations of current STVG, the key is to have the abil- ity to locate every target mentioned in the textual query, much like how humans do. Thus motivated, we introduce a new type of STVG task, dubbed Spatio-Te Tiiio--Teenn Temporal (or OmniSTVG). Different from existing STVG locating only a single object, OmniSTVG aims at localizing all targets mentioned in the given query within the video. For each object in the query, spatio-temporal tube is predicted as the localization result. By doing so, OmniSTVG enables local- ization of not only arbitrary number (e.g., one or multiple) of targets of interest but also their inter- acting counterparts in the video, which simultaneously resolves two limitations of current STVG and therefore leads to more practical applications. It is worth noting, OmniSTVG is natural extension of classic STVG task, aiming to further push its frontier for more comprehensive multimodal video understanding. Concept-wise, OmniSTVG, to some extent, is inspired by the idea of segmentation anything (SA) (Kirillov et al., 2023). The difference is, SA aims to segment any regions in an image, while OmniSTVG locates any mentioned objects in the textual query from untrimmed video.
Oojni-Object Video-Grounding
To facilitate exploration of OmniSTVG, we propose BOSTVG, a novel large-scale dataset dedicated to spatio-temporal omni-object video grounding. Specifically, our BOSTVG contains 10,018 videos with 10.2 million frames, and covers wide selection of 287 categories from diverse scenarios. Each sequence in BOSTVG, paired with a free-form textual query. contains a varying number of objects to localize, ranging from to 10 with an average of 2.4. Each object is manually annotated with a spatio-temporal tube (i.e., a set of bounding boxes). To ensure high quality, all the tube annotations in each sequence are carefully inspected and refined when needed through multiple rounds. To our best knowledge, BOSTVG by far is the first and the largest benchmark dedicated to OmniSTVG.
Furthermore, to encourage future research in developing OmniSTVG methods on BOSTVG, we pro- pose a simple yet effective model, dubbed OmniTube. Specifically, OmniTube built upon current Transformer- based STVG method (Yang et al., 2022). It comprises multimodal encoder for video and text fcature fusion and a decoder for localization. Different from current STVG methods ( (Yang et al., 2022: Gu et al., 2024; Jin et al., 2022; Lin et al., 2023) that locate only single target, our Omni learns simultaneously multiple sets of object queries in the decoder to ground all objects in the video. To improve localization, we apply visual information in video guided by textual feature to generate queries, which benefits learning better query features for target localization. To form a spatio-temporal box tube for each target, a simple strategy is introduced to match detection results across different frames from the video. Despite simplicity, OmniTube Tube shows promising results and expects to provide reference for future research on our OmniSTVG task.
2

---

Published as a conference paper at ICLR 2026
Table I: Comparison of BOSTVG with other benchmarks. SO: Single-Object; MO: Multi-Ojecc; AO: All-Object. 4: Since DVD-ST is not released, we report the statistics from its original paper.
Object
Mean
Total
Total
Min
Mean
Max
Total
Num. of
Dataset
Benchmark
Videos
classes
frames
frames
dur.
obj-
obj-
obj.
obj.
queries
focus
STPR (Yamaguchi el., 2017)
5.293
260
1.4M
hrs
1.0
5,828
30.365
so
VID-Sentence (Chen et al., 2019)
5.318
30
294
2.3M
21 hrs
1
1.0
1
7.654
7.654
SO
VidSTG (Zhang et al., 2020b)
6,924
79
798
5.5M
53 hrs
1.0
1
6,924
99,943
SO
HCSTVG-vl (Tang et ul,, 2021)
5,660
522
3.0M
31 hrs
1.0
5,660
5,660
SO
HCSTVG-12 (Tang et ul., 2021)
16,544
522
8.6M
92 hrs
1
1.0
1
16,544
16,544
SO
DVD-ST (Ji et al., 2024)
2.750
163
1.8
12
4,950
5,734
SO, SO,MO
BOSTVG (ours)
10,018
287
1,014
10.2M
102 hrs
2.4
10
24,175
10,018
AO
We notice a concurrent work (Ji et al., 2024) which has similar spirit with this work by supporting grounding multiples in videos. Compared (Ji et al., 2024), our work mainly differs in three aspects. First, concept- wise, OmniSTVG grounds all objects mentioned in query, while the work of (Ji et al., 2024) localizes only partial targets, leading to limitations in comprehensive understanding. Second, lask- and method-wise, the work of (Ji et al., 2024) locates only target objects of the same class, while our OmniSTVG and OmniTube enable the localization of objects of different categories, making more flexible well practical. Third, dataset-wise, our BOSTVG contains 10,018 videos, which is much larger than the dataset in (Ji et al., 2024) with 2,750 sequences.
In summary, we make the following contributions: We propose OmniSTVG, new STVG task that locates all objects mentioned in the query toward more flexible and comprehensive understanding; & We present BOSTVG, a large-scale dataset with 10,018 videos and more than 10 million frames from 287 categories for OmniSTVG; We introduce OmniTube, a simple but effective method to facilitate more future research of OmniSTVG; + We demonstrate that OmniTube achieves promising performance, aiming to offer a reference and provide guidance for future research on OmniSTVG.
RELATED WORK
STVG Benchmarks. Benchmarks play an important role for the development of STVG. The work of (Yamaguchi al., 2017) introduces the STPR for spatially and temporally grounding pedestrians from trimmed videos. Similarly, VIDDSentence (Chen et al., 2019) is proposed for grounding within trimmed videos, but comprises more classes. For more practical STVG, HCSTVG-v1 (Tang al., 2021) is presented to locate human objects in untrimmed videos, making it more challenging. Later, HCSTVG-v2 (Tang et al., 2021) is introduced via expanding from HCSTVG-v1 using extra videos. Different from HCSTVG-vIAA 1/vv, VidSTG (Zhang et al., 2020b), collected from VidOR (Shang et al., 2019) for video object relation detection. aims spatio-temporal video grounding from both declar- ative and interrogative sentences. In addition, besides human category. VidSTG offers other classes in the query and video for localization, aiming at generic STVG. Unlike previous datasets only for single-target localization, the recent DVD-ST (Ji et al., 2024) introduces platform for grounding multiple targets while ignoring interacting counterparts in localization. Different from all the afore- mentioned benchmarks, BOSTVG is specially developed for new STVG task, OmniSTVG. which aims to ground all targets mentioned in the textual query. Therefore, in our BOSTVG, each target in the query is annotated with spatio-temporal tube (see Fig. (b) again for an example in BOSTVG).
STVG Algorithms. STVG algorithms have witnessed great progress recently. Early methods (Tang al., 1021; Zhang et al., 2020a;b) typically adopt two-stage pipeline, which first detects candidate region proposals with a pre-trained detector (Ren et al., 2015) and then finds correct region proposals with an extra model. Despite straightforwardness trained detection model, and their performance is thus restricted by the capacity of the used detector. To overcome this limitation, recent STVG methods (Su et al., 2021; Gu et al.., 2024; Yang et al., 2022; Jin al., 2022; Lin et al., 2023; Wasim et al., 2024; Gu et al., 2025), inspired by DETR (Carion et al., 2020), switch to a one-stage design directly predicting a spatial-temporal tube for localization using Transformer (Vaswani et al., 2017), without adopting any external detectors. Compared to two-stage methods, such one-stage framework shows superior performance for its end-to-end training pipeline. Our OmniTube is also one-stage Transformer- based architecture. Nonetheless, different from the aforementioned approaches that localize only a single target from the video, our OmniTube aims to locate all objects mentioned in the query for more comprehensive multimodal video understanding.
these two-stage methods heavily rely on the pre-
3

---

Published as a conference paper at ICLR 2026
3 THE PROPOSED BOSTVG
3.1 DESIGN PRINCIPLE
BOSTVG aims to offer a dedicated platform to facilitate the development of OmniSTVG. For such purpose, we follow several principles in developing BOSTVG: (i) Dedicated benchmark. The motivation of BOSTVG is to provide a novel benchmark dedicated to OmniSTVG. The video and its paired textual query are required to comprise varying number of targets (e.g. one multiple) to localize, aligning with the goal of OmniSTVG. (ii) Large scale. Developing the deep learning- based methods for OmniSTVG requires abundant training samples. Besides training, a real system needs evaluation on various situations. Hence we expect BOSTVG to contain at least 10K videos with each corresponding to a textual query, which benefits both large-scale training and assessment of deep models. (iii) Diverse object classes. An important aim of BOSTVG is to facilitate the development of general OmniSTVG models that can locate target objects from different classes. To this end, the new dataset expects to contain at least 200 categories, collected from various scenarios, for grounding. (iv) High quality. High-quality annotations are essential for a benchmark in both training and evaluation. To ensure the high quality, each video of BOSTVG is manually labeled with precise
box tubes through multiple rounds of inspections and refinements.
3.2 DATA ACQUISITION
BOSTVG aims to foster general and comprehensive spatial-temp temporal video grounding by containing rich classes from diverse scenarios. To this end, 287 categories that are appropriate for OmniSTVG are selected in BOSTVG. These object classes are chosen from different sources, mainly including ImageNet (Deng et al., 2009) and V3Det (Wang al.. 2023), and organized a coarse-to- hierarchical structure. Due to limited space, we show detailed classes in the supplementary material.
After determining all object categories in our BOSTVG, we then search for raw videos of each class under various scenarios from YouTube, currently the largest and most popular video platform with many real-world videos. All videos are collected under the Creative Commons License and used for research purpose only. Initially, we have collected over 15K sequences using keywords aligned with object classes. Then, we conduct careful inspections on each video sequence to verify its suitability for OmniSTVG. More specifically, if there is at least one video clip suitable for our task, we keep this video; otherwise, we discard the video. This process is carried out by our experts (e.g., students working related field). For the qualified videos, we select one clip from each of them. Eventually, we gather 10,018 videos for BOSTVG, with each provided a textual query by our experts.
Finally, create large-scale benchmark, called BOSTVG, for OmniSTVG. BOSTVG covers 287 classes and contains 10,018 videos with 10.2 million frames from diverse scenarios. Its average video length is 1,014 frames. Each video contains a varying number of targets ranging from to 10. is worth noting, we do not consider the case of none target in the video in BOSTVG and focus on localizing targets that appear within the video. Tab. summarizes BOSTVG and its comparison to other benchmarks, Due to limited space, we provide detailed discussions of maintenance, ethical issue, and responsible usage of BOSTVG in the supplementary material.
3.3 DATA ANNOTATION
In BOSTVG, our experts first generate textual query for each video. After that, we offer two types of annotations based on the query, including start and end timestamps for temporal localization and bounding box tubes of objects for spatial localization. Specifically, given pair of video and textual query, we first identify a temporal segment in the video that corresponds to the query description, and mark it with start and end timestamps. Afterwards, we label each object mentioned in query with consistent spatio-temporal tube formed by a set of boxes in each frame of temporal segment.
To ensure high-quality annotations of BOSTYG, we use a multi-round strategy, More specifically, a few experts who work on related problems first manually annotate the start and end timestamps for each sequence. Then, the temporal segment of each video, marked by start and end timestamps, is manually labeled by annotation team that is formed by a few volunteers and an expert. After this initial round, the eeatio-temporal tube annotations will be sent to a validation team formed by three experts for inspection. If the initial annotations are not unanimously agreed by all experts, they will
4

---

Published as a conference paper at ICLR 2026
Average 35.5
0.32
Pobabaay
0.12
Probahiikg 10bahii
0.24
Average:
Probabiityy
0.3
Avesage: 24
0.08
0.16
0.2
0.04
0.08
0.00
0.00
0.0
Sequence Length
Segment Length
Objocts
(a) Distrihution of video length
(h) Distribution of segment length
(c) Distributiin of number of objects
Averae
0.32
woman and
shir
0.15
snal
Prebabliyy
6.12
Probability
0.24
Ascrage: 5162
another
blue
Leve Low
0.05
0.16
person
0.04
0.00
16
32
18
6-
60
96
112
0.00
100
800
1200 1600 2000 2400 2600
Mire nove Whiteb phiiigeee
mater
Textual Query Length
Number of Boxes
(d) Distribution of textual query longth
Distribution num. boxes videos
Wordeloud of all textual queries
Figure 2: Representative statistics on BOSTVG, including distributions of video length (in seconds) in (a), temporal segment length (in seconds) in (b), number of target objects for grounding in (c), query length (in characters) in (d), number of boxes in videos in (e), and wordcloud of queries in (f).
be returned back to the original labeling team for refinement. We repeat this inspection-refinementt process until the annotations of all videos are qualified. Due to limited space, we show the pipeline of our annotation process in the supplementary material. Fig. 1 (b) displays an annotation example in BOSTVG, and more can be seen in the supplementary material due to limited space.
Annotation accuracy analysis. To analyze our annotation accuracy, we randomly select 100 videos from BOSTVG and ask an independent group of external experts to inspect and re-label them. Then, we compute the Intersection over Union (IoU) of new and original spatio-temporal tubes. The IoU for these selected sequences is 0.90, which validates accuracy and reliability of our annotations.
Statistics of annotation. To better understand BOSTVG, we display some staties in Fig. 2, includ- ing distributions of video length, temporal segment length, and number of objects, textual query length, number of boxes in videos, and the query wordcloud. From Fig. 2 (c), we can see that most videos contain to objects, which is close to real scenarios and makes BOSTVG more suitable for practical applications. From Fig. 2 (e). we can observe that the average number of boxes in each video 516.2, showing the challenge of BOSTVG in localizing more objects.
3.4 DATASET SPLIT AND EVALUATION METRIC
Dataset Split. BOSTVG contains 10,018 videos in total. Among them, 8,106 videos selected for the training set, dubbed BOSTVG and the rest 1,912 for test set, dubbed BOSTVGTs. Tab. 2 displays the comparison between training and test sets. Please note, in split, we try our best to keep distributions of training and test sets similar. For both training and test sets, each video is paired with a textual involving a varying number of targets for grounding, meeting the aim of OmniSTVG.
Table 2: Comparison of training and test sets.
Mean
Total
Mean
Total
Videos
frames
frames
obj.
obj.
BOSTVGT. 8,106
1,014
8.22M
2.4
19,567
BOSTVGG 1,912
1,015
1.94M
2.4
4,608
In addition, to enable in-depth analysis, we further divide the test set into three subsets based on the number of targets in the video, including BOSTVGTu-Low with 1-3 targets each video (1 ,,566 Mediium with 4-6 targets in each video (273 samples), and BOSTVGra: with more than targets in each video (73 samples).
samples),
Evaluation Metric. Following current benchmarks (Zhang al., 2020b), we utilize multiple met- rics, including m_tloU, m_vloU, and vlo @ R for evaluation. Please note, the calculation of m-vIoU and vloU @R here needs to consider all pratial-tempora box tubes in the video, instead of a single one as in other benchmarks, because the video in our BOSTVG may contain multiple objects. Due to limited space, please refer to the supplementary material for the detailed formulations of metrics.
5

---

Published as a conference paper at ICLR 2026
Classita liwe Hice
Belloome
Greacaton
Sprita Quer
Tex gudeed
Sparii Hord
ft
Video
Gocorzion
Teaport Quat
grised
00- CO
Tempunl
Hlov
Ain
with buseioll Bet
Test
Quiput
Mukimadal Ensoder
Spatio-Tappnd Deuola
Granding
(See.
Figure 3: Overview of OmniTube, which contains a multimodal encoder, a spatio-temporal decoder, and spatial-temporal tube generation module to locate all mentioned targets in the textual query.
4 OMNITUBE: A NEW METHOD FOR OMNISTVG
Overview. We propose OmniTube, new framework specially designed for OmniSTVG. Similar to current STVG models (Yang et al., 2022; Jin et al., 2022; Lin et al., 2023), inspired by DETR (Carion al., 2020), OmniTube adopts encoder-decoder architecture. As shown in Fig. 3, OmniTTube mainly consists of three components, including multimodal encoder for feature extraction and fusion (Sec. 4.1), a spatio-temporal decoder for target object position learning (Sec. 4.2), and a box tube generation module (Sec. 4.3), to localize all mentioned objects in the text for OmniSTVG.
4.1 MULTIMODAL ENCODER
Given the video and the text, the multimodal encoder first extracts their features and then performs multimodal feature fusion as described the following.
Feature Extraction. Given the video, sample N. frames from it and then adopt ResNet- 101 (He et al., 2016) and VidSwin (Liu et al., 2022a) for 2D appearance and 3D motion feature extraction. The appearance feature is represented as Fa = { D. the height, width, and channel dimensions, and the motion feature as Fm {frm Ji=1 where f E IR HW DD with Dm the channel dimension. For the text, we use RoBERTa (Liu et al., 2019) for feature extraction. We tokenize it to sequence with N, words, and then apply RoBERTa on the sequence produce textual feature J, ( where S RP with D, the feature channel.
( N. where f E IR #x Da with H, W and N.
Multimodal Feature Fusion. We generate the multimodal feature by fusing appearance, motion, and textual features. Similar (Gu et al., 2024), we first project them the same channel di- mension D and then concatenate them obtain the initial multimodal feature F (f:)) where f: = Concat (Ln( L.n"),,,."" Ln(,,."), Ln(f)))
) is the multumodal feature in frame i. Then, we fuse the features using the self-ateention encoder (Waswani et al., 2017) to generate the multimodal feature F SAEncoder (F Epos + Eype)). where Epos and Eyype denote position and type embeddings and SAEncoder() is self-attention encoder with L (I=6) standard self-attention encoder blocks. Due to space limitation, please see its architecture in the supplementary material.
4.2
DECODER FOR GROUNDING
To obtain target position information from multimodal feature F, we design spatio-temporal de- coder composed of spatial decoder and temporal decoder. The former learns spatial information for all objects in the text, while the later aims to obtain the temporal information for grounding.
4.2. SPATIAL OMNI-OBJECT DECODER FOR GROUNDING
Text-guided Spatial Query Generation. Unlike current STVG methods locating single target, Omni localizes all objects text, and thus introduces multiple queries for each frame. To ex- plore target cue for better localization, we leverage in video for generating object queries. To this end, we first extract features from F by deconcatena-
as guidance to select target-relevant features

---

Published as a conference paper at ICLR 2026
tion Fin. where F./FF/FF are appearanee/mottooleettue features. Then, we utilize appearance and textual features to generate the spatial queries. Specifically, we first aver- age textual feature Fr via FF=Avg(Fi). Then, we calculate similarity between Fr and Fa, and adopt the M most similar features generate initial query 2o by average pooling, as follows,
di ArgPool.
where yho denotes the feature of the j query in frame i, and Ng is the number of queries per frame. Sim() and ()pn() are the operations to calculate the similarities and to pick up top M elements, re- spectively. Compared with previous approaches, our text-guided object queries by exploring target- specific cues can effectively improve localization as shown in our experiments.
Spatial Decoding. After generating 2o. we send to spatial decoder with K (K=6) layers for interaction with the multimodal feature. To enhance queries, in each layer we design two spatial and temporal attention blocks to capture their spatial and temporal relation before interacting with the multimodal feature. Specifically, let Qx-1 represent query features sent to the hthh (1 I k S K) layer for learning 2x. We first perform spatial attention on queries of the same frame, as follows,
1 1h-111 س = SABlock((g.) - ) i= 1,2, No
N. is query feature after spatial attention. SABlock(-) is spatial attention block implemented by self-attention in supplementary material. After this, to capture temporal relation, we use temporal attention on query features of the same object across different frames via
where -x-1= 1 Jj=1Ji=1
(1--11 [ho د TABlock( (aanock((at,) ( j 1 N₂
where êx-1= 3 ) is the query features after the temporal attention block TABlock(.) implemented with self-att att-etttenion as in the supplementary material.
Next, we learn the spatial position of objects by interacting queries with multimodal feature. In OmniTube, spatial localization leverages the appearance and text features. Specifically, we interact Q with the multimodal feature via cross-attention for learning Ôk, as follows,
Q CrossAttBlock (Q-1, F.))
where CrossAttBlock(z. u) is the cross-attention block with z generating query and u key/value.
Finally, with Qx after the Tthh layer in decoder, to predict the bounding boxes of the targets via B = SpatialHead Qx). where B E RR N.N,, XN, and D3 = is the central position, width and height of predicted box. In addition, inspired by MDETR (Kamath et al., 2021), we predict the index for each bounding box, which corresponds to positional index of words in the original text, and is used determine the class of each bounding box, via g ClsHead @k). where Cl (lsHeed(.) is MLP module and G E I N. N,xN, with Ne denoting the maximum positional indexes for any given sentence.
a spatial head, consisting of an MLP module,
4.2.2 TEMPORAL DECODER FOR GROUNDING.
Text-guided Temporal Query Generation. Temporal decoder predicts start and end timestamps. Similar to spatial decoder, we use text as guidance select target-relevant motion features to pro- duce the initial query Po as follows,
Po N
p. = =AAgPooling(TooppSiin F:)))
where pf is query feature in frame i. F. is the pooled textual feature and F. the motion feature extracted from F. It is worth noting that, in OmniSTVG, all targets share the same start and end timestamps with the textual expression. Thus, each frame assigned with single initial query p.
Temporal Decoding, In temporal decoding, we send Po to decoder with K layers for interaction with multimodal feature. Specifically, let (p- 1 Jiz be query features fed to the thth (1 < A < K) layer for learning Pk. where p k-1 is the feature of frame i. To capture temporal relation, we first perform temporal attention on PA-1. 1. as follows,
N.
TABlock (v) (1N,)) )

---

Published as a conference paper at ICLR 2026
where Px- Ji=1 is the query feature after temporal attention. After this, we interact Pk-1 with the multimodal feature using cross-attention for learning Pk. as follows,
Pk CrossAttBBocck(PP--1 [Fm;
where Fm and Fr are motion and textual features from F.
After K layers in temporal decoder. we obtain Pк and employ a temporal head implemented by MIP to predict the start and end timestamps through H = Headoraaleadd(PA where H E R.N. x2 contains the start and end probabilities H. and He of N₂ frames for temporal localization of targets.
4.3 SPATIAL-TEMPORAL TUBE GENERATION
In OmniTube, we perform two steps, including tubelet matching and filtering, to generate the spatial- temporal tube for each target. (i) Tubelet matching. Spatial grounding predicts N, bounding boxes for each frame. To match these boxes across different frames, we apply Hungarian matching (Kuhn, 1955) using spatial positions and object class of bounding boxes. ultimately generating N₂ initial tubelets. (ii) Tubelet filtering. To further optimize tubelets, we first refine their temporal boundaries using start and end timestamps. For each tubelet, we average class probabilities of all its bounding boxes for determining the class. After that, we remove tubelets whose classes are not present in text.
4.4 OPTIMIZATION
In OmniTube, we predict both the spatial bounding boxes and temporal start and end timestamps for loss optimization. Due to limited space, please see our loss function in the supplementary material.
EXPERIMENTS
Implementation. OmniTube is implemented
Table 3: Comparison with current approaches.
using PyTorch (Paszke et al., 2019). We adopt ResNet-101 (He et al., 2016), VidSwin (Liu et al., 2022b), and RoBERTa (Liu et al., 2019) to extract appearance, motion, and text features. Following (Jin et al., 2022; Lin et al., 2023;
indicates algorithm adapted for OmniSTVG.
Methods m tlu
m vloU vIoU@0.3
v.0U@0.5
(a) BOSTVGIa-Low
TubeDETR:
31.20
7.99
3.79
0,21
STCAT
33.68
8.52
4.03
0.38
Gu et al., 2024), part of parameters, including 2D/text backbones and multimodal encoder, are initialized using pre-trained MDETR (Kamath et al., 2021). We train OmniTube end-to-end, keeping the 3D backbone frozen while training all other parameters. During training, we use the Adam optimizer (Kingma Ba, 2014) with an initial learning rate of 1e 5 for backbone and 1e-4 for remaining modules. Additionally, data augmentations such as random resizing and cropping are applied to all training videos, with the shorter side resized to 320 pixels. The video length N, depends on the video duration, with frames extracted FPS=2, and the text length N₂ is set to 30. The channel dimensions Da, Dm. Dr, and 1) are set to 2,048, 768, 768, and 256.
CG-STVG:
32.70
0.32
Omniiuuee
36.16
16.f1
7.16
I.09
(b) BOSTVGr..
TubeDETR:
30.40
5.81
0.00
0.00
STCAT:
31.72
6.20
0.00
0.00
CG-STVG
29.70
5.30
0.37
0.00
OmniTube
34.89
7.24
1.85
0.00
(c) BOSTVG
TubeDETR:
30.27
3.91
0.00
0.00
STCAT
31.56
4.36
1.37
0.00
CG-STVG
28.09
3.22
1.37
0.00
OmniTube
32.27
4.42
1.37
0.00
(d)
COOSTVGr.-Full
TubeDETR:
31.05
7.52
3.14
0.17
STCAT
33.31
8.03
3.35
0.31
CG-STVG:
32.09
7.66
3.56
0.26
OmniTube
35.83
9.47
6.17
0.89
The A and d are set and I.
5.1 OTATE-OF-THE-AN COMPARISON
Since there are no available approaches specially designed for OmniSTVG task, adapt three STVG frameworks with source codes, including TubeDETR (Yang et al., 2022), STCAT (Jin et al., 2022), and CG-STVG (Gu et al., 2024), with appropriate modifications to their implementations, and compare our OmniTube to these approaches on the BOSTVG Please notice that, all methods in comparison trained on BOSTVGTa for fairness, and their codes will be released.
8

---

Published as a conference paper at ICLR 2026
Tab. 3 demonstrates the results and comparison of OmniTube with other methods. We can that Omni outperforms other methods on all metrics. Specifically, in the full BOSTVG OmniTube achieves 35.83% m tloU and 9.47% m vloU scores, which largely surpasses CG-STVG with 32.09% m tloU and 7.66% m vloU scores, STCAT with 33.31% m tloU and 8.03% m vloU scores, and TubeDETR with 31.05% m_tloU and 7.52% m_vloU scores, showing superiority. In addition, under other settings with low-, medium-, and high-density, OmniTube largely surpasses other models, showing its consistent advantages.
5.2 ABLATION STUDY AND DISCUSSION
Ablation of spatial decoder. To study different modules in spatial decoder, conduct ablation in Tab. 4. In our spatial decoder, the text-guided spatial query genera- tion (TG-SQG) aims to learn target-specific spatial query for better interaction with multimodal feature. The spa- tial attention block (SAB) and temporal attention block (TAB) are applied to model spatial and temporal context within the video. As in Tab. 4. without TG-SQG, SAB and TAB, the m_tloU and m_vIoU scores are 34.33% and 8.25% (0). When SAB and TAB are added, the model better captures crucial spa- tial and temporal cues across frames, achieving comparable m tloU of 34.13% but higher m vloU V.S. @). This suggests that the spatial and temporal models in videos are crucial for target localization. More importantly, as shown in Tab. 4, when incorporating TG-SQG with either SAB or TAB, both m_iloU and m_vIoU are improved (0 the text- rext-guided spatial query contains more discriminative better interact with and learn from the multimodal features for target localization. Finally, when all modules are applicd, we obtain the best performance with with 35.83% m tloU and 9.47% m vloU scores (G), which validates the effectiveness of our decoder design in modeling spatial and temporal cues and learning target-specific query feature for improving grounding performance.
Table 4: Ablation of spatial decoder.
TG-SQG SAB
TAB
m. m_tloU
m. m_yToU
34.33
8.25
34,13
9.00
34.98
8.89
35.42
9.15
35.83
9.47
of 9.00%
and 1 2). This indicates that spgee--pecific inforamtion and is able to
Ablation of temporal decoder. To analyze temporal de- coder, we conduct an ablation in Tab. 5. In our tem- poral decoder, the text-guided temporal query genera- tion (TG-TQG) is designed to generate target-aware tem- poral query for localization, and the temporal attention block (TAB) is adopted to capture temporal relation across frames. As shown in Tab. 5, without using TG-TQG and TAB, the m tloU and vloU scores are When adding TAB for temporal modeling, we can see that the m_tloU and
Table 5: Ablation of temporal decoder.
TG-TQG
TAB
m HoUU
m vloU
0
26.06
6,66
35.00
8.98
26,00
6.82
35,83
9.47
26.06% and 6.66% m_vIoU scores are significantly improved to 35.00% and 8. 8.98% v.s. @). This indicates that mod- eling temporal dependencies with TAB is essential for accurate temporal localization. When using TG-TQG alone, we achieve the similar m_tloU score of 26.00% and m_vIoU score of 6.82% (O v.s. This suggests that although TG-TQG provides more discriminative target-aware cues, its benefit cannot be fully realized without temporal modeling. When combining TG-TQG and TAB, we obtain the best 35.83% m_tloU and 9.47% m_vIoU scores (0). This demonstrates that TG-TQG and TAB play complementary roles and their synergy leads to more accurate spatio-temporal grounding
Ablation of different class predictions. In OmniTube, rather than directly predicting the bounding-box class for tube generation, we predict the position index of the class in the text as the bounding box class. To compare these
Table 6: Ablation of box class.
Classification
m flo
m vloU
Box Class
35.36
8.82
Position Index
35.83
9.47
two strategies, we conduct an ablation in Tab. 6. As shown in Tab. 6, predicting the position index
yields consistently better performance (0 v.y. @). We argue the reason is that, directly selecting from 287 classes is a challenging high-cardinality classification problem, predicting the position index simplifies category localization by linking each box to its corresponding noun phrase in the text, therefby leading to better performance.
Ablation of parameter M in the decoder. In the de- coder, M controls how many video features are selected as relevant to the textual query. When M is too small, the selected video features are insufficient to capture target- specific information from videos, which may results in
Table 7: Ablation of M.
m HoUl
m vloU
M =
35.30
9.14
M = 5
35.83 10 35.45
9.47
9.34
9

---

Published as a conference paper at ICLR 2026
Table Ablation of training data.
Training Data
Objects
Classes
# Size
m_tloU
m_vloU
BOSTVG1a
1-10
287
8K
35.83
9.47
HCSTVG-v2 (training set)
10K
22.24
4.76
BOSTVGT. IICSTVV-v2 (training set)
1-10
287
18K
35.47
9.31
performance degeneration. Conversely, when M is too large, the selected video features may con- tain noise, leading to suboptimal performance. To study the effect of M, we conduct ablation in Tab. 7. As shown in Tab. 7, we can observe that the best result is obtained when M is 5 (@), indicating that five query-relevant video features sufficient for optimal grounding.
To examine whether in- corporating existing datasets can further improve performance, we evaluate the impact of adding existing HCSTVG-v2 with single-object data for training. The results are reported in Tab, 8. As shown in Tab. 8, training with our multi-object BOSTVG1a alone achieves the best results (0), indicating that multi- supervision by BOSTVG is essential for OmniTTube In contrast, training solely with HCSTVG-v2 leads to the lowest performance (@), because lacks multi-object interactions and provides a weaker supervisory signal for the spatial-temporal grounding of mul- tiple targets. When combining BOSTVGTa and HCSTVG-v2 for training, the model performance slightly decreases )). This degradation is likely due to inconsistent training distributions for dif- ferent tasks. HCSTVG-v2 contains only one human class and lacks multi-object queries. When merging BOSTVGTa and HCSTVG-v2, the model is exposed to a large amount of samples with very narrow semantic scope, which may bias learning toward human-centric patterns and weaken its ability generalize across diverse object categories. Moreover, since HCSTVG-v2 lacks multi- object queries, it cannot provide supervision for joint localization of multiple entities. These two factors together result in the final slight performance drop, which also indicates that training on diverse, multi-class, multi-object data is essential for solving the omni-target problem. We believe that this result underscores the importance of our dataset design: BOSTVG is not just an extension of existing work, but necessary shift toward richer, more complex training signals.
Ablation of training data. OmniTube is trained with our BOSTVGTa-
Impact of motion information. In OmniTube, we leverage motion information as an additional cue for target localization. To evaluate effect, we conduct ablation in Tab. 9. As shown in Tab. 9, incorporating motion feature consis- tently improves localization performance (0 v.s. 2 This is because motion encodes strong tempo- ral signals that help link entities across frames and clarify the interactions among targets. Without motion cues, the model depends solely on appearance, which makes it harder capture dynamic relationships and maintain temporal consistency. The results in Tab. 9 confirm that motion features play complementary role to appearance cues and are essential for spatio-temporal grounding.
Table 9: Ablation of the motion cue.
Motion
tloU
vloU
OmniTTuee w/o motion
35.29
8.88
OmniTTube w/ motion
35,83
9.47
Due to space limitations, we show additional results and analyzes in the supplementary material.
6 CONCLUSION AND ETHICAL STATEMENT
In this work, we present OmniSTVG that aims to locate all targets in the query. To foster research on OmniSTVG, we propose BOSTVG by offering 10,018 sequences with 10.2 million frames. To our best knowledge. BOSTVG is the first benchmark dedicated to OmniSTVG. Moreover, to encourage future research, we introduce OmniTube, a simple yet highly effective method for OmniSTVG. Our results show the advantages of OmniTube over other models.
Ethical Statement. The construction of BOSTVG strictly adheres to ethical standards, ensuring that all annotators provided informed consent. All videos in BOSTVG are collected under Creative Commons licenses and used for research purpose only. Due to limited space, we provide more detailed ethical statement in the supplementary material.
Acknowledgments. Libo Zhang is supported by National Natural Science Foundation of China (No. 62476266). Zhipeng Zhang is supported by Natural Science Foundation of China (No. 62503323). Bing Fan, Yan Huang, and Heng Fan are not supported by any funds for this work.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Siho Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-v1 technical report. arXiv preprint arXiv:2502.1392. 2025.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
Zhenfang Chen, Lin Ma, Wenhan Luo, and Kwan- Yee Kenneth Wong. Weakly-supervised spatio-
temporally grounding natural sentence in video. In ACL, 2019.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06261, 2025.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Xin Gu, Heng Fan, Yan Huang, Tiejian Luo, and Libo Zhang. Context-guided spatio-temporal video grounding. In CVPR, 2024.
Xin Gu, Yaojie Shen, Chenxi Luo, Tiejian Luo, Yan Huang, Yuewei Lin, Heng Fan, and Libo Zhang. Knowing your target: Target-aware transformer makes better spatio-temporal video grounding. In ICLR, 2025.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Wei Ji, Xiangyan Liu, Yingfei Sun, Jiajun Deng, You Qin, Ammar Nuwanna, Mengyao Qiu, Lina Wei, and Roger Zimmermann. Described spatial-temporal video detection. arXiv, 2024.
Yang Jin, Yongzhi Li, Zehuan Yuan, and Yadong Mu. Embracing consistency: A one-stage approach for spatio-temporal video grounding. In NeurIPS, 2022.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Car- ion. Mdetr- modula-dd detection for end-to-end multi-modal understanding. In ICCV, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412::1412.6990, 2014.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In ICCV, 2023.
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
Jie Lei, Licheng Yu. Tamara Berg, and Mohit Bansal. Tvr: A large-scale dataset for video-subtitle moment retrieval. In ECCV, 2020.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Zihang Lin, Chaolei Tan, Jian-Fang Hu, Zhi Jin, Tiancai Ye, and Wei-Shi Zheng. Collaborative static and dynamic vision-language streams for spatio-temporal video grounding. In CVPR, 2023.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. arXiv, 2019.
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In CVPR, 2022a.
11

---

Published as a conference paper at ICLR 2026
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In CVPR, 2022b.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. NeurIPS, 2019.
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. In NeurlPS, 2015.
Xindi Shang, Donglin Di, Junbin Xiao, Yu Cao, Xun Yang, and Tat-Seng Chua. Annotating objects and relations in user-generated videos. In ICMR, 2019.
Rui Su, Qian Yu, and Dong Xu. Stygbert: A visual-linguisti transformer based framework for spatio-temporal video grounding. In ICCV, 2021.
Zongheng Tang, Yue Liao, Si Liu, Guanbin Li, Xiaojie Jin, Hongxu Jiang, Qian Yu, and Dong Xu. Human-centric spatio-temporal video grounding with visual transformers. IEEE T-CSVT, 32(12): 0238-8249, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurIPS, 2017.
Jiaqi Wang, Pan Zhang, Tao Chu, Yuhang Cao, Yujie Zhou, Tong Wu, Bin Wang, Conghui He, and Dahua Lin. V3det: Vast vocabulary visual detection dataset. In ICCV, 2023.
Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang. and Fahad Shahbaz Khan. Videogrounding-dino: Towards open-vocabulary spatio-temporal video grounding. In CVPR, 2024.
Waswani, N Shazeer, N Parmar, Uszkoreit, Jones, A Gomez, Kaiser, and Polosukhin. Attention is all you need. In NeurlPS, 2017.
Masataka Yamaguchi, Kuniaki Saito, Yoshitaka Ushiku, and Tatsuya Harada. Spatio-temporal per- son retrieval via natural language queries. In ICCV, 2017.
Antoine Yang. Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Tubedetr: Spatio- temporal video grounding with transformers. In CVPR, 2022.
Zhu Zhang, Zhou Zhao, Zhijie Lin, Baoxing Huai, and Jing Yuan. Object-aware multi-branch relation networks for spatio-temporal video grounding. In IJCAI, 2020a.
Zhu Zhang, Zhou Zhao, Yang Zhao, Qi Wang, Huasheng Liu, and Lianli Gao. Where does it exist: Spatio-temporal video grounding for multi-form sentences. In CVPR, 2020b.
12

---

Published as a conference paper at ICLR 2026
SUPPLEMENTAL MATERIAL
To better understanding of this work, we provide additional details, analysis, results, and discussions:
A Details of Object Categories
We present detailed object categories in the proposed BOSTVG, along with the number of sequences in each category.
B Construction Pipeline for BOSTVG and Additional Annotation Examples
This section introduces the detailed construction pipeline of BOSTVG and showcases more annotation examples.
C Detailed Architectures of Modules
We show architectures for SAEncoder(-). SABlock(+), and TABlock(-) in the paper.
D Detailed Architecture of Baseline
We show the detailed architecture of our baseline.
E Details of Evaluation Metrics
We describe evaluation metrics used for assessing different approaches.
F Details of Loss Function for Optimization
We provide details of the loss function for optimization in our method.
G Additional Results and Analysis
In this section, provide additional comparison results and analysis of our work.
H Limitation of OmniTube
We discuss the limitations of our proposed OmniTube.
Ethical Statement and Dataset Specification
We discuss the ethical concerns, annotator protection, and specifications of BOSTVG.
A DETAILS OF OBJECT CATEGORIES
BOSTVG contains 287 object categories, aiming to provide diverse platform for the OmniSTVG task. The categories are organized hierarchically to ensure comprehensive coverage. Specifically, first collect 23 coarse object classes, comprising "Appliance", "Bird", "Body Part", "Building", "Clothes", "Container", "Environment Flement", "Fish", "Food", "Furniture", "Geometric", "Hu- man". "Invertebrate Animal", "Item", "Kitchenware". "Machine". "Mammal", "Music Instrument", "Plant", "Reptile Animal", "Sport Equipment", "Tool", and "Vehicle". Please note, since "Human" a special category, we separate it from "Mammal". After this, we further gather 287 fine categories from coarse classes. Fig. 4 shows the category organization of BOSTVG (please zoom in). We will provide and release the category information with our BOSTVG on our website.
B CONSTRUCTION PIPELINE FOR BOSTVG AND ADDITIONAL ANNOTATION EXAMPLES
The construction of BOSTVG comprises five steps: (i) Overall design. In the first step, we determine the overall goal, all object categories to include in BOSTVG, and strategies to search for videos from YouTube. (ii) Video collection from YouTube. The second step is to collect videos using strategies in the first step. In our work, we choose source videos from YouTube because it is currently the largest video platform and contains many real-world videos. Please note that, all videos are collected under the Creative Commons license for research purpose only. (iii) Video selection. After video collection, the third step is to select videos which are qualified for our OmniSTVG task. Specifically, our experts (e.g., students working on related fields) will manually inspect each video for ensure all selected videos meet the requirement. (iv) Initial labeling. The fourth step is to conduct the initial data labeling by our experts. Specifically, the experts will first provide a textual query for each video. Based on the query, two types of annotations will be provided, including start and end timestamps of the temporal segment and the spatial bounding boxes for the targets of interest. (v) Multi-round inspection and refinement. In the final fifth step, multiple rounds of inspection and refinement if needed are performed by our experts and volunteers to ensure high-quality annotations. Fig. 5 shows the construction pipeline of BOSTVG. In Fig. we provide more annotation examples in BOSTVG.
13

---

Published as a conference paper at ICLR 2026
mammal
Hartion
Plant
contaiter
clateer
vehicle
reptdle apimal
food
fumiiture
BOSTVG
item
wil
enviromment element
tool
Aa
sport equipmentt
body part
bird
invertobrate animal
applisnce
Figure 4: Category organization BOSTVG, The inner circle of the pie chart displays 23 coarser object classes, while the outer circle displays 287 fine object categories. Best viewed in pdf and by zooming in.
Step 1: Overall Design
Step Video Selection
Strp 4: Initial L. Labeling
Grals:
Sourch Texus:
tectral query:
A. Varyza punzbor abjocts Eire Eireee cotegertie
Und:
Requirements
Tri luzis fabitoo
Viaeo Source:
You
Teedi
mésbel screes
dow, spanion, partot
feeding fighting training
Video Durrotion: uxape
temporal stan time 24 245999 end line 34.7-759
bounding bouss
annotatioon
Step Video Collection from YouTube
Video Quallty: video resclittoon 770p
Stop S: M Multt-round Inspection RefInement
Ingpct quary clritt Inspec arnotitin aormmey arsetation retuning
Objects desctieble Intorattion
Vidcos
more videos...
labler
arnecalions
Figure 5: Illustration of construction pipeline of the proposed BOSTVG, including five key steps, ie., overall design, (ii) video collection from YouTube, (iii) video selection, (iv) initial labeling. and (v) multi-round inspection and refinement.
14

---

Published as a conference paper at ICLR 2026
Query: fion pounces on black
16.01s
21.
Query: ponguii seaguil
5.20
8.77
Queery: women knoeling on nwo dilphines
Query: boy hiss ball wibh basehall hut
S
7.54s K the robot
10.98u
Query: A woman tobes
23.40s
Query: drone delivery lands
locker
Qwery: excavator dumps soil into trick
0.5G
Query: boat paised red and green sailboats
21.31a
#27.000
Figure 6: Additional annotation samples on our BOSTVG.
qary
Spos
Aftention
Multi-Hent
Add Norm
Feed Forward
Add & Norm
Attention
Multi-Heul
Add & Norm
Forward
Add & Norm
F
Estr y (or Eot 1/11
Feed
(0t (pt 1)m))
Exype
2
(a) attenttnn encoder with & blocks
(b) self- affeattttioo block for the spatial/tempoaal attention blocks
Figure 7: Detailed architectures for the self-attention encoder in (a) and spatial/temporal attention blocks in (b).
15

---

Published as a conference paper at ICLR 2026
ZD Beekbhone
Spriil Deodlng ayer(
tion Head
Clessifiir-
Spotiia Htmmppral Tube Gonzation
Mutimodai Faaure Fusion
f
Crosa
Commt
Backborre
3D
Dect Conseet
Tlaid
Sparial
Video
En Fcl
Temponm Hend
Bacckbone
Text
funehalll
Texporil Deodling Layer
Text
Output
Mukimodal
Licoodrr
Spatio cmpori Decoder for Grounding
Tube Genen fiom
Figure 8: Illustration of our baseline, which share similar architecture with OmniTube but without text-guided query generation modules, spatial and temporal attention blocks.
C DETAILED ARCHITECTURES OF MODULES
In Fig. 7 (a), show the architectures for the self-attention encoder, which is composed of L (L 6) standard self-attention encoder blocks and used to fuse features from multiple modalities. In Fig. 7 (b) display the architectures for the spatial attention block and temporal attention block, which are both implemented with a self-attention block.
D DETAILED ARCHITECTURE OF BASELINE
Our baseline method mentioned the main text shares a similar architecture with OmniTube but does contain the key text-guided query generation modules and spatial and temporal attention blocks. Its detailed architecture is shown in Fig. 8. From the experiments described in the main text, the performance the baseline inferior to our OmniTube, which validates the effectiveness of our text- text-guided query generation and spatial and temporal attention blocks for target localization.
E DETAILS OF EVALUATION METRICS
Following current STVG benchmarks (Chen et al., 2019; Zhang et al., 2020b), we utilize multiple metries, including m tloU, vloU, and vloU@ for evaluation. Specifically, m tloU aims assess the temporal localization performance and is calculated by averaging the temporal loU scores tloU all test videos. The iloU is calculated [Pu] where P; and Pu represent the intersection and union between the groundtruth and predicted segments, respectively. m_vloU is utilized measure the spatial localization performance, which is calculated by averaging spatial IoU scores vloU. The vIOU is calculated as follows,
[Py]
cIoU
Σ IoU bt,)) VEN,
[Pu
where bi, and b,, are the groundtruth bounding box and the predicted bounding box the 7th target in th frame. The vloU@ R is defined ratio of samples with spatial IoU scores above threshold R.
F DETAILS OF LOss FUNCTION FOR OPTIMIZATION
Given a video and its textual expression, OmniTube predicts two types of localization, comprising (1) the spatial position and class G of the bounding in the spatial grounding, and (2) the start timestamps H. and end timestamps H. in the temporal grounding. During training, given the ground truth for the spatial location and class 9 the bounding box, as well as the start timestamps H and end timestamps,
calculate the total loss as
Ao ((B, (B", (8))")) + do (H),, H.) + C,,H,, He))
16

---

Published as a conference paper at ICLR 2026
where CA, denotes the Hungarian loss in (Carion et al., 2020) explained later and La represents the KL divergence loss. The parameters A and d are used to balance the loss.
Explanation of C. In spatial grounding, predict the spatial location and class of the bounding box, which can be denoted as
y = - = (1,,
where N and Ng denote the number of frames and the number of predicted targets, respectively. yi. = (br.) Rt.). where bis and 9i,j are the spatial location and class of the jth bounding box in the 7th frame. Suppose that the ground truth for spatial location and class can be denoted as
y" ( ((,,
where N2 ( Ng) denotes the number of ground truth targets. Following DETR (Carion et al., 2020), to compute the pair-wise matching cost between the prediction results and ground truth, first utilize the Hungarian matching algorithm to establish a one-to-one correspondence between the prediction and ground truth, follows,
y HungarianMatchhrr(
where Y represents the successfully matched bounding boxes, while the unmatched bounding boxes are excluded from the loss calculation. After this, we can calculate the loss Ln. as follows,
LA C. (B","'" B²) + (B,, B²) + X 9")
where Lus, L1 and L. are IoU, smooth L1, and binary cross-entropy losses. The parameters A A Ao are set to 3, 5,
G ADDITIONAL RESULTS AND ANALYSIS
G.1 COMPARISON WITH EXISTING METHODS ON CLASSIC STVG BENCHMARK
Besides OmniSTVG, OmniTube can
Table 10: Comparison with existing STVG methods on the
also be used for conventional single- object STVG. In order to validate this, we conduct an experiment on the clas- sic single-object STVG benchmark HCSTVG-v2 (Tang et al., 2021). Tab. 10 shows the results. Since Om- niTube focuses on multi-object local- ization, including multi-object inter- action and multi-object tube match- ing, it does not achieve state-of-the-art the single-object STVG benchmark. That said, method outperforms most existing approaches, showing its effectiveness.
classic STVG dataset HCSTVG-v2.
Methods m tloU
vlo vIoU@0.3 v.5
MMN (Lei et al., 2020)
30.3
49.0
25.6
TubeDETR (Yang al., 2022)
53.9
36.4
58.8
30.6
STCAT (Jin al., 2022)
56.6
36.9
60.3
33.6
STVGFormer (Lin al., 2023)
58.1
38.7
65.5
33.8
CG-STVG (Gu 2024)
60.0
39.5
64.5
36.3
OmniTuhe
38.5
"38.7
"63.
"35.4
G.2 COMPARISON WITH SINGLE-OBJECT STVG METHOD ON BOSTVG
As discussed earlier in the main text, to adapt existing STVG to OmniSTVG. a straightfor- ward solution is to employ the STVG model repeatedly for multiple single-object textual queries. Here we conduct an experiment by repeatedly running recent STVG method CG-STVG (Gu et al., 2024) trained on Vid- STG (Zhang et al., 2020b) and comparing it with OmniTube. Specifically, we first build new test set by selecting videos from BOSTVGTe that have same classes as in VidSTG so that CG-STVG can work. There are 283 sequences selected. Then we provide textual query for each single object in these videos CG-STVG can success- fully run. Tab. 11 reports the results. As Tab. 11, our OmniTube achieves better results due to multi-object relationship modeling and runs much faster without the need of running multiple times.
Table 11: Comparison with the single-object STVG method CG-STVG on OmniSTVG. Please note that the results here reported from BOSTVGT with 283 videos.
newly built test
Methods m m.tloU v. vloU@0033 v.0 Time Cost
CG-STVG 32.5
8.6
558.1s
OmniTube 37.9
10.5
134.1s
17

---

Published as a conference paper at ICLR 2026
G.3 COMPARISON WITH EXISTING MLLM- BALSED METHODS ON BOSTVG
To investigate the STVG capabilities of multimodal large models, we have
Table 12: Comparison with existing milm-based meth-
ods on BOSTVGTe-Full
added experiments evaluating two popular MLLMs, including Gemini-2.5 Pro (Co- manici et al., 2025) and Qwen2.5-VL. (Bai et al., 2025), on OmniSTVG. Please note that, since Gemini-2.5 Pro is not open-
Method
m.tloU
m.vlol
v.oU@0.3
v.0U@0.5
Gemini-2.3 Pro
35.5
7.2
5.1
Qwen2.5-VL
15.2
0,0
Swean.55VL-SSFF
29.1
5.2
2.2
0.1
OmniTube (ours)
35.8
9.5
source, directly apply Gemini-2.5 Pro to predict spatial-temporal locations of targets in videos without any tuning. For Qwen2.5-VL, we report results of two versions, include zero-shot version (named Qwen2.5-VL.) without tuning and supervised fine-tuned version (named Qwen2.5-VI--SFT) The results and comparison on BOSTVG1a-Full are shown in Tab. 12. From Tab. 12, we observe that, our proposed OmniTube
()) outperforms all MLLMMbaseed methods (0: Gemini-2.5 Pro, @: Qwen2.5 LL without tuning, @ Owwn2.55VL-SPF with supervised fine-tuning). We argue that the reasons why MLLM-based methods underperform are two fold: (1) without task-specific tuning (see and @) ,, MLLMs- basdd method are not equipped to perform fine lin-ggrinnd
local- ization for multiple objects in videos; and (2) even with fine-tuning (see θ), the training objective of MILMM treats structured outputs such as timestamps and bounding boxes as discrete text tokens, making them particularly sensitive to even small lexical differences (e.g., "1-9s" versus "0-8s")- These findings highlight the challenges of our OmniSTVG task and the necessity of dedicated ar- chitectures like the proposed OmniTube.
G.4 SERO-SHOT GROUNDING PERFORMANCE COMPARISON
To evaluate the zero-shot object grounding capability of different methods, we conduct a new zero- shot experiment by spliting BOSTVG into non-overlapped training and test sets. Specificaly, the training set contains 237 classes, and the test set contains the rest 50 held-out classes, with no overlap between training and test sets. As shown in Tab. 13, both our OmniTube and existing methods are degraded to some extend (see Tab. 13 here and Tab. 3 (d) the paper) in this zero-shot setting due to lack of training samples for unseen cases. That being said, the proposed OmniTube still achieves superior performance than existing methods, which validates the effectiveness and generality of our architecture in grounding zero-shot objects.
Table 13: Comparison of zero-shot grounding ability of different methods.
Method
#Train Classes
#Test Classes
m tlo
m vloU
vloU @0.3
vloU 0.5
STCAT
237
50
28.71
6.92
2.75
0.25
CG-STVG
237
50
29.84
7.24
2.98
0.26
OmniTube (ours)
237
50
30.98
7.58
3,56
0.37
G.5
ABLATION OF DIFFERENT FEATURES
To explore the role of different features, we show the experiments by studying the ResNet and VidSwin feature for performance in Tab.. 14. As shown in Tab. 14, when using ResNet feature only (i.e., removing VidSwin feature; see i), the m_tloU and m_vloU are 35.29% and 8.88%, respec- tively, indicating that the ResNet-based appearance information is essential for grounding. This is expected because the ResNet-based appearance backbone is initialized from a pre-trained vision- language model, MDETR (Kamath et al., 2021), and provides strong object-level semanties needed for target localization. In contrast, when using VidSwin feature only (i.e., removing ResNet feature; see @), the model fails converge, and therefore no valid results can be reported. This indicates that, when using only VidSwin feature, the model loses its ability to ground textual references to vi- sual regions and fails converge meaningfully. When combining ResNet and VidSwin features, the model achieves the best performance with 35.83% m.tloU and 9.47% m_vIoU, demonstrating that motion cues from VidSwin complement appearance information from ResNet and further enhance target localization.
G.6 DISCUSSION OF TASK INHERENT DIFFICULTIES.
18

---

Published as a conference paper at ICLR 2026
Table 14: Ablation of different features. "x" means that the model fails to converge.
Appearance Feature (ResNet)
Motion Feature (VidSwin)
Ulo
m. vlol
35.29
8.88
35.83
9.47
For the proposed BOSTVG benchmark, several inherent difficulties that go be- yond conventional single-object STVG are introduced, including (i) multi-object grounding: In benchmark and task, the query often refers to multiple ob- jects. To localize all the mentioned ob-
Query: goollooper saving the soceer ball from two players, wearing green vest and the other with dark shin.
4.78s Query: The remaining four athietes standing by jumped into the water:
5.92a
jects in the query, the grounding model needs to understand the complex rela- tionships among multiple objects for ac- curate localization, which is inherently challenging: (ii) dynamic and crowded scenes: Compared to conventional single- proposed benchmark brings in more frequent and appearance changes, e.g., motion blur, partial or full occlusions among objects, and close object-interactions videos, thereby demanding precise spatio- temporal alignment under visual ambigu- ity, particularly in the context of multi- object grounding; and (iii) more complex textual query: more complicated by containing more entities, due to its aim at localizing multiple objects. For robust performance, the grounding model needs to be equipped with strong scene-level reasoning.
6.40s
9.66s
Query:: adult goose water with five goslings, while staved on the graxs.
object STVG benchmark,
11.5s
Figure 9: Examples illustrating the inherent challenges of the BOSTVG benchmark.
our benchmark, the textual query is
Critically, these challenges co-occur in real-world scenarios, as illustrated in Fig, 9. In the soccer scene (Top), the query "one wearing a green vest, the other with dark skin" refers to two players among many. requiring the model to ground multiple targets amid crowd clutter, motion blur, and partial occlusion while correctly parsing fine-graine one-grained visual attributes from a linguistically complex description. The diving sequence (Middle) involves four athletes jumping simultaneously. The query must be interpreted to identify all four individuals despite their near-identical appearance, synchronized motion, and rapid entry into the water setting that combines dense multi-object grounding, severe visual ambiguity, and query implicitly referencing a group action. Similarly, in the goose scene (Bottom), the model must localize an adult goose and five goslings, which move into the water while another adult goose remains on land. This requires disentangling spatially and temporally varying object states, resolving visually similar instances, and understanding relational language ("another stayed on the grass"), ") again entangling all three core challenges. These examples underscore that BOSTVG reflects the intertwined complexity multi-object reference, visual dynamics, and linguistic sophistication must be addressed jointly, making it a more realistic and demanding benchmark than prior single-target settings.
real-world video grounding, where
G.7 DISCUSSION OF LIMITATIONS OF EXISTING ALGORITHMS ON BOSTVG
Existing STVG methods are designed for single-object grounding. For example, the recent CG- STYG mines visual context the object to improve localization accuracy, but can mine only the visual context of a single object and cannot be directly extended to multi-object scenarios. More- over, existing methods lack explicit modeling of interactions among multiple objects, making them unable to resolve ambiguities when several similar interacting objects appear together, which is common in BOSTVG. In contrast, the proposed OmniTube is built from the ground up for omni- object grounding. supporting joint parsing, localization, and disambiguation of multiple targets.
19

---

Published as a conference paper at ICLR 2026
G.8 DISCUSSION OF DATA DISTRIBUTION BOSTVG
Our BOSTVG test set is partitioned into three complexity-base baaeexttty-beed groups: BOSTVG-Low (1-3 tar- gets, 1,566 samples), BOSTVG-Medium (4-6 targets, 273 samples), and BOSTVG-High (27 tar- gets, 73 samples). Similar to other well-known datasets such as ImageNet (Deng et al., 2009) and COCO (Lin al.. 2014), the data distribution of our BOSTVG follows the long-tailes distribu- tion, reflecting the natural frequeney of multi-object scenarios in the real world. As consequence, queries referring to 1-3 objects are more common, while those with 27 targets are rare still included. To ensure fair evaluation across complexity levels, in addition to the overall average, we report separate metrics for BOSTVG-Low, -Medium, and -High (please see Tab. the paper). However, we agree that adding more samples in BOSTVG-High is crucial for evaluating the model's capability in handling complex, multi- scenarios. Addressing this, we will include more high- density examples (e.g.. 200 more) in BOSTVG- to enable more robust evaluation of scalability. Since this is beyond our current goal, we leave this to our future work.
G.9 DISCUSSION OF THE SHARED TEMPORAL SEGMENTATION ASSUMPTION
In this work, we follow the conventional standard STVG formulation, in which the textual query describes a spatio- temporal event defined by determined start and end time stamps. The only dif- ference in our OmniSTVG task is, the event contains multiple simultaneous interactions among multiple objects, and there all target objects share the same start and end time stamps (i.e., same temporal segmentation) with the event itself. This design reflects how humans typically describe dynamic scenes: multi-object references are anchored single event interval. That being said, we believe more formats spatio-tempon video grounding is worth studying, egg., multiple objects with each having its own temporal segmentation. Since this is currently beyond our goal, we leave its exploration to our future.
G.10 ANALYSIS OF MEMORY REQUIREMENT AND MODEL EFFICIENCY OF OMNITUBE
In this work, we train our OmniTube using 32 Nvidia A100 (48GB) for around 13 hours. In infer- ence, for 50s-video with 100 sampled frames, the GPU memory requirement for running Omni- Tube is 34GB and the inference time is 0.66 seconds.
G.11 QUALITATIVE RESULTS
To further qualitatively validate the effectiveness of our OmniTube, provide the grounding results of our method in Fig. 10. As shown in Fig. 10, we can see that, our method can robustly localize all objects mentioned in the textual query, showing its effectiveness.
H LIMITATION OF OMNITUBE
To encourage more future research OmniSTVG, we propose simple yet effective baseline Om- niTube. Despite achieving promising results, OmniTube has two limitations. First, OmniTube may degrade in highly dense scenarios. highly dense videos, target objects may be heavily occluded, resulting in lower localization performance. To handle this case, special strategies such as leverag- ing rich spatial context information via explicit object-object interaction or target trajectory cues are needed for improving target localization. Second, our OmniTube, similar to existing STVG mod- els, requires considerable computational resources for training because it takes as input the whole video. To alleviate this, we will investigate more light-weight resource-friendy: frameworks for OmniSTVG. Considering that our current goal is to provide a feasible baseline for OmniSTVG, we leave the above explorations for better performance to our future work.
20

---

Published as a conference paper at ICLR 2026
Text: person lying on donkey's back.
Video
GT
OmniTube
Text: Person opens a microwave and checks the contents.
Video
GT
OmniTube
Text: Leopard stands and interacts with a mirror.
Video
GT
OmniTube
Text: Large owl grooms smaller owls.
Video
OmniTube
Figure 10: Qualitative results of our method. Our prediction results are visualized with solid-line bounding boxes, and the groundtruth boxes are shown in the same color with the prediction results but with dashed-line bounding boxes.
21

---

Published as a conference paper at ICLR 2026
1 ETHICAL STATEMENT AND DATASET SPECIFICATION
Ethical Concerns. The construction of BOSTVG strictly follow ethical standards. All videos are collected under Creative Commons licenses. Nevertheless, we understand the license might change in future. Once any notification regarding this is received, we will take appropriate actions to handle it.
Annotator Protection. To ensure annotator well-being, all participants provide informed consent and are pre-warned about potentially distressing content. We allow annotators to skip any video that cause discomfort to withdraw from the study entirely.
Maintenance. BOSTVG will be hosted on the popular Github. This allows to conveniently check feedback from the community and to improve BOSTVG via necessary maintenance and updates by the authors. addition, all experimental results of compared approaches on BOSTVG will be made publicly available. Our ultimate goal is to provide a long-term and stable platform for the community foster research on spatio-temporal omni-object video grounding.
Responsible Use of BOSTVG. Our BOSTVG aims at fostering research and application of video grounding. BOSTVG can be used for research purpose only. Please note that, due to inevitable bias during the data collection process, there may exist geographic and demographic imbalance.
Data Collection Protocol. BOSTVG is constructed from YouTube videos licensed under Creative Commons, used strictly for research purposes. All videos and annotations undergo manual review to exclude offensive or sensitive content. During curation, we prioritize diversity in both scene types and object categories ensure representative and balanced dataset.
Potential Limitations. key limitation of BOSTVG lies in the high cost and complexity of anno- tation. Precisely grounding multiple objects in videos especially under omni-target settings requires fine-grained spatiotemporal labeling which is labor-intensive time-consuming and demands expert annotators. This limits the scalability of the dataset and hinders rapid expansion larger scales or new domains.
22


Reviews:
[{"review_id": "DEGice23iu", "reviewer": "Reviewer_nfHN", "summary": "The paper proposes a new video grounding task, OmniSTVG, which differs from the traditional STVG task in that it can localize all targets mentioned in the textual query as well as the interactive relationships existing among these targets. To this end, the paper constructs a large-scale dataset BOSTVG and presents a simple yet effective model named OmniTube.", "questions": "Why is the performance of any multimodal large language models on this dataset not provided?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The task proposed in the paper expands the scope of the traditional STVG task. Moreover, the proposed dataset has a wide range of sources and is built using a relatively rigorous manual annotation method, combining both scale and quality.\n- The paper puts forward a simple and effective baseline, and has implemented and compared several public models.\n- The paper provides detailed descriptions of details, making it easy to follow.", "weaknesses": "The paper does not mention the performance of multimodal large language models (MLLMs) on this task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gSptecjtXl", "reviewer": "Reviewer_CK93", "summary": "Existing Spatio-Temporal Video Grounding (STVG) tasks are impractical, as they are conventionally trained to localize only a single target, even when multiple targets are referenced in the text.\n\nThis paper proposes a more practical new task, termed OmniSTVG (Omni-target Spatio-Temporal Video Grounding), which aims to localize all objects mentioned in the textual query. The primary contributions include:\n\nNew Task (OmniSTVG): Defining the aforementioned new direction of \"omni-target\" localization.\n\nNew Dataset (BOSTVG): To support this task, a large-scale, high-quality dataset comprising 10,000 videos was constructed, which has undergone multiple rounds of manual refinement.\n\nNew Baseline (OmniTube): An effective Transformer-based model is provided, which utilizes \"text-guided queries\" to concurrently localize all targets.", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Pioneering a New Direction: The work identifies the critical limitation of existing STVG tasks (i.e., single-target grounding) and defines a more complex and practical new research direction (i.e., multi-target grounding).\n\nContribution of a Core Resource (BOSTVG): It provides the field's first large-scale (10,000 videos), high-quality benchmark dataset specifically dedicated to \"omni-target\" localization, establishing a core asset for advancing subsequent research.\n\nProvision of a Strong Baseline (OmniTube): The study does not merely pose the problem but also delivers a well-designed (e.g., \"text-guided queries\") and empirically validated solution, offering a solid baseline for future work.\n\nRigorous and Solid Experimentation: The effectiveness of the model's constituent components is validated through exhaustive ablation studies, while comparative experiments underscore the uniqueness and necessity of the new BOSTVG dataset.\n\nThe manuscript is clearly and normatively written; the figures are highly comprehensible and meticulously detail the proposed methodology.", "weaknesses": "**Insufficient Analysis**:Regarding the benchmark, an analysis of the task's inherent difficulties is absent. The inclusion of illustrative examples to demonstrate these challenges would be beneficial.Furthermore, a deeper analysis is required as to why existing algorithms, including those compared in this study, cannot be directly or effectively applied to this benchmark.In the ablation study, the paper merely enumerates the functional contributions of individual algorithmic components without providing further in-depth analysis.\n\n**Insufficient Representativeness of \"Omni\" (Data Distribution Imbalance)**:In Section 3.4 (Dataset Splits), the test set is partitioned into three groups: BOSTVG-Low (1-3 targets, 1566 samples), BOSTVG-Medium (4-6 targets, 273 samples), and BOSTVG-High (>7 targets, only 73 samples).The core objective of the paper is to address the \"omni-target\" problem; however, the proportion of high-density, genuinely complex \"omni-target\" samples (73) within the test set is extremely low (approximately 3.8% of the test set).This implies that the model's capability in handling complex, multi-target scenarios has not been sufficiently validated. The reported overall average performance (e.g., $m\\_vIoU$) is likely dominated by the simpler \"Low\" group.\n\n**Questionable Model Generalizability and Data Compatibility (Poor Generalizability)**:In Section 5.2 (Table 8: Ablation on Training Data), when attempting to merge BOSTVG with the existing single-target dataset HCSTVG-v2 for training (Row 3), performance paradoxically decreases slightly compared to training on BOSTVG alone (Row 1).Typically, augmenting the training data, even with related datasets, is expected to enhance model robustness.This performance degradation (which the authors attribute to \"data inconsistency\") may suggest that the model (OmniTube) or the annotation style of the dataset (BOSTVG) is prone to overfitting, hindering its ability to generalize or maintain compatibility with other data sources.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qcWazWIgp6", "reviewer": "Reviewer_81gP", "summary": "This paper introduces the OmniSTVG task where multiple objects mentioned in the text query have to be grounded spatially and temporally. The BOSTVG dataset is collected, consisting of 10K videos manually annotated with spatio-temporal tubes for 1-10 objects, including 287 object classes in total. The OmniTube model is presented, including separate spatial and temporal decoders, text-guided query generation and multi-tube prediction. This model outperforms single-object methods adapted to the new task, and is also about 4x fater than naive approaches of running a single-object model for each object in the query.", "questions": "- Out of curiosity, is the architecture able in practice to ground “zero-shot” objects not seen in the training set?\n- How much is each of ResNet and VidSwin feature helpful for the STVG performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- A new dataset that addresses an important limitation of previous STVG approaches\n- OmniTube performs well compared to prior approaches and has been ablated extensively", "weaknesses": "- Assuming all objects share the same temporal segmentation is a strong assumption\n- It is confusing what the “baseline” is meant to show in the paper: it excludes various features present in prior work, e.g. text-guided query generation in https://arxiv.org/abs/2502.11168 or alternative spatial and temporal blocks in https://arxiv.org/abs/2203.16434. Note that these features are rightfully not being presented as contributions in the introduction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Conditions for acceptance have been satisfied.
Decision: Accept (Poster)