Embracing Consistency: A One-Stage Approach for Spatio-Temporal Video Grounding
Authors: Yang Jin
yongzhi li
Zehuan Yuan
Yadong MU
Venue: NeurIPS 2022 Accept (2022)
Keywords: Computer Vision
Video Understanding
Visual Grounding

Abstract: Spatio-Temporal video grounding (STVG) focuses on retrieving the spatio-temporal tube of a specific object depicted by a free-form textual expression. Existing approaches mainly treat this complicated task as a parallel frame-grounding problem and thus suffer from two types of inconsistency drawbacks: feature alignment inconsistency and prediction inconsistency. In this paper, we present an end-to-end one-stage framework, termed Spatio-Temporal Consistency-Aware Transformer (STCAT), to alleviate these issues. Specially, we introduce a novel multi-modal template as the global objective to address this task, which explicitly constricts the grounding region and associates the predictions among all video frames. Moreover, to generate the above template under sufficient video-textual perception, an encoder-decoder architecture is proposed for effective global context modeling. Thanks to these critical designs, STCAT enjoys more consistent cross-modal feature alignment and tube prediction without reliance on any pre-trained object detectors. Extensive experiments show that our method outperforms previous state-of-the-arts with clear margins on two challenging video benchmarks (VidSTG and HC-STVG), illustrating the superiority of the proposed framework to better understanding the association between vision and natural language. Code is publicly available at https://github.com/jy0205/STCAT.

Embracing Consistency: A One-Stage Approach for Spatio-Temporal Video Grounding
Yang Jin', Yongzhi Li², Zehuan Yuan', Yadong Mu Peeking University ByteDance *Peng Cheng Laboratory en, cn,
j jiny@stu.pku.. ekd.ee. edu. yuanzehuan@byreddanes com, mydēpku. edu.on
com,
Abstract
video grounding (STVG) focuses on retrieving the spatio- temporal tube of specific object depicted by a free-ffom textual expression. Exist- ing approaches mainly treat this complicated task as parallel frame-grounding problem and thus suffer from two types of inconsistency drawbacks: feature aligument inconsistency and prediction inconsistency. In this paper, we present an end-to-end one-stage framework, termed Transformer (STCAT), to alleviate these issues. Specially, we introduce a novel multi-modal template as the global objective to address this task, which explicitly constricts the grounding region and associates the predictions among all video frames. Moreover, to generate the above template under sufficient video-textual perception, an encoder-decoder architecture is proposed for effective global con- text modeling. Thanks to these critical designs, STCAT enjoys more consistent cross- moosl feature alignment and tube prediction without reliance any pre- trained object detectors. Extensive experiments show that our method outperforms previous -fate-of-the--e arts with clear margins two challenging video benchmarks (VidSTG and HC-STVG), illustrating the superiority of the proposed framework to better understanding the association between vision and natural language. Code is publicly available at https//githbbcomiyy000////c.
Consistency-AWr
1 Introduction
Visual grounding is prominent and fundamental task in the multi-modal understanding field. It aims to localize region from the visual content specified by given natural language query. As bridge to connect vision and language, this topic has drawn increasing research attention over the past few years [13, 11, 45, 17, 16). Early works mostly focused on static images and have achieved remarkable progress, while visual grounding in videos has not been adequately explored yet. Recently, spatio- temporal video grounding (STVG) was introduced in [48], which is compound task, requiring both spatial and temporal localization. Formally, given an untrimmed video and a textual description of an object, this task aims at producing spatio-temporal tube (i.e., sequence of bounding boxes) for the queried target object (See Figure 1). Compared with previous grounding tasks in images, STVG is substantially more challenging as it delivers high demands for distinguishing subtle spatio-temports status of instance in videos based on the query sentence. Thus, how to effectively align the textual semantics and time-varying visual appearance especially critical for settling this task.
The majority of current approaches [38, 34, 4, 48, 47] streamline the spatio-temporal video grounding two-stage pipeline, where the object proposals tubelets are firstly generated by pre-trained object detector (e.g.. Faster RCNN [25]) and then all the candidates are ranked according to the similarity with the query. However, this simple design leads to an inevitable deficiency: the grounding
"Corresponding Author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Query Sentence:
An adull in blue grabs ball the baskethall court
Paraliel Frame Grounding
Estrrt
Cenett
Output Tube
Inconsistent Prediction
Figure I:
An illustration of spatio-temporal video grounding task (left) and the inconsistency prediction in previous approaches (right). This task is particularly challenging and requires exquisite reasoning about linguistic semantics and the global video context. Existing methods simply treat STVG as a parallel image grounding problem, leading to inconsistent predictions (highlighted in red).
performance is heavily limited by the quality of proposals, especially when a huge domain gap exists between the pre-trained object categories and free-form text. Lately, some approaches [29, 33] try to address these issues by leveraging a one-stage framework without any off-the-shelf object detectors. They directly produce the bounding box at each frame as well as predict the starting and ending boundary based on the fused cross-moda features.
Though promising results have been achieved, most of these one-stage approaches have the following drawbacd wawbacks that need to be concerned. (1) Feature alignment inconsistency. Global context mod- eling is of great significance for the STVG scenario. For example, given the query of "An adult in blue grabs a ball on the basketball court" as in Figure 1, the target "adult" cannot be determined without a perception of the whole video content, since there are two adults in blue but only one of them eventually grabs the ball. Accordingly, the cross-modal feature alignment should be per- formed between the whole video and the querying sentence. However, existing methods [48, 29, 33] only attend the cross-modal fusion with local short-term video context. Even in the very recent method [40], the alignment is still considered at frame-level. (2) Prediction inconsistency. Previous approaches simply treat STVG as parallel frame-level grounding problem. As shown in Figure 1, they separately generate a bounding box with respect the query on every single frame, yet neglect the consistency among frames. It is worth noting that the goal of STVG task is to localize a single target instance specified by the textual description. Although the appearance of the target instance across all video frames may change due to camera motion, scene dynamics, etc., they should have identical semanties (e.g., the guy who grabs the ball). Therefore, conducting grounding independently on each frame regardless of semantic consistency over time will cause the prediction mismatch.
In this paper, we present an effective one-stage transformer-fashioon encoder-decoder based [35] framework, dubbed as Spatio-Tempore Consistency- Aware Transformer (STCAT) to cope with the aforementioned issues in the STVG task. Building upon the prevalent detection network DETR [2], STCAT introduces a novel video-level multi-moda template to modulate the grounding procedure. This template serves as a global target that guides and correlates the predictions among each frame to maintain temporal consistency. Specially, it is formulated as a learnable query as in [2], which comprised of a content term shared by all frames to encode the semantics and a position term characterized by per frame. Such design allows the final grounding results to be determined based on the specific frame content while simultancously considering consistency. Moreover, in order to adaptively obtain the mioveementioned query by considering the interactions between visual and textual information, we propose a cross-modal spatio-temporal encoder to fuse the global video context and textual description. The query in STCAT is then rendered based on the encoded features and fed into a decoder to aggregate the multi-modal representation. During the decoding procedure, the query belonging to each frame is dynamically updated layer by layer. Finally, the grounded temporal tubelet is produced by different prediction heads.
The technical contributions of this work can be summarized as follows. (1) We propose a novel multi- modal template mechanism to mitigate the prediction inconsistency issue in the STVG task. In doing so, the grounding among all video frames can be correlated to yield more precise spatio-temporal

---

prediction. (2) We design a transformer- architecture, named as STCAT, to fully exploit the global video context and thus perform a better cross-modal alignment. Coupling with the proposed template mechanism, our STCAT can directly ground the spatio-temporal tubes without reliance on any pre-trained object detectors. (3) Comprehensive experiments conducted on two challenging video benchmarks (VidSTG [48] and HC-STVG (33]) further demonstrate that our method obtained new state-of-the-art performance compared to other approaches.
2 Related Work
Visual Grounding in Images/ / Videos. Visual grounding is an essential multi-modal task that aims to localize the object of interest in an image/video based on a text description. In the image grounding scenario, most existing methods [18, 44, 20, 36, 41] firstly utilize pre-trained object detector get some object proposals. While some recent works like [16, 43, 23, 42] propose one-stage frameworks that alleviate the reliance on pre-trained detectors. For example, [16] Liao al. proposed to extract cross-modal representations and localize the target objects by an anchor-free object detection method. Yang et al. tried to generate text-conditional visual features by sub-queries in [42], which further promotes the one-stage method.
The video grounding task can be categorized into temporal grounding and spatio-tempore grounding. The former requires to localizes a temporal clip in the video by referring sentence. While the spatio- temporal video grounding lies at the intersection of spatial and temporal localization. Most previous approaches [4, 33, 38, 48] also rely on pre-extracted tube or object proposals. Recently, STVGBert in [29] propose one-stage approach that extend the VilBERT [22] settle this task. However, all these methods suffer from the drawbacks mentioned in Section 1. More recently, a concurrent work in [40] also proposed a one-stage framework that utilizes DETR [2] architecture. However, they only consider the multi-modal interaction at the frame level and still lack an effective design for settling prediction inconsistency in STVG, thus achieving inferior performance to ours.
Vision-Ianguage modeling. Vision-language modeling tasks such as visual question answering, image captioning, and image-text retrieval attract lots of researchers to explore recently. Due to the simplicity and success of Transformers in the natural language processing field, many works [32, 14, 22, 30] borrow this architecture to align the context information between images and sentences. Others like [31, 50, 6, 9, 15, 39] further extend transformers into video-text tasks. But most of them rely either on pre-extracted object features, spatially pooled features, that ignored detailed spatial information within each frame and are not capable to handle the challenging STVG task.
Transformer based detection. Object detection is a traditional task in the computer vision field, early methods [25, 24] usually rely on CNN encoders and region proposal or regression modules to tackle the various objects in images. As Carion el al. [2] first introduce Transformers into this task, lot of transformer-hane hased detection methods [51, 1, 49] have sprung up recently. Most of them follow the encoder-decoder paradigm to transform the visual features into precision object bounding boxes, we borrow this effective architecture to tackle the STVG task in this paper.
3 The Proposed Approach
In this section, we briefly present the formal definition of the STVG task and an overview of proposed framework in Section 3.1. Then we elaborate on the main components STCAT, including the feature extractor (Section 3.2) and the consistency-aware transformer (Section 3.3). Finally, the training and inference details are introduced in Section 3.4.
3.1 Overview
Given an untrimmed video V = -w consisting of T consecutive frames and query textual description S {= depicting a target object existing in V. The goal of the STVG task is to localize a spatio-temporal tube B (b, H-=,, corresponding to the semanties of given textual query, where b, represents bounding box in the 1-th frame, l, and I, specify the starting and ending boundary of the retrieved object tube respectively. STVG an extremely challenging task, which requires not only the spatial interaction with the textual modality for detecting bounding box at the
3

---

E Encoder Block
Prediction Head
Generate Tube
Temporal Interaction Layer
Decoder Block
tpullll Inleradiin Laye
Spaiil
Spulal Laper
Cross-A
Puaart Tea
Framer m1
Feature Concal
Self-A ttention
Inear
Linear
V 'isual Encoder
Text
Time
hlue grabsa ball the bastethell court
Loca
Template G Generator
Clibal
Viuual Tolon
Input Video
Query Sentence
Lingykeii Taken
Figure 2: The Architecture the proposed Temporal Consistency- Transformer (STCAT). Given an input video and query sentence pair, it firstly leverages visual and a linguistic encoder to extract features for cach modality. The extracted fcatures are fed into encoder to perform cross-model interaction. Then generator yiclds the multi-modal templates which is responsible for guiding the dccoder. The retrieved object tube is finally generated based the decoded features via prediction hcad.
frame level, but also the long-range temporal relation modeling the video level for determining the start and end timestamps of query-related video segments.
To mitigate the aforementioned feature alignment and prediction inconsistency drawbacks, effective framework named STCAT is developed in this paper. As illustrated in Figure 2, the proposed model firstly utilizes two feature extractors obtain both visual and textual features from the video frames and querying sentence, respectively. then models the video-text interactions through a well-designed pratio-tempord cross- cross-modal encoder, which introduces global learnable token for the whole vidco to encode target object semantics, and local one for each individual frame to represent the frame- specific appearance. These tokens further leveraged produce template for the target object by a template generator. Finally, the yielded template treated as a query (Like the one DETR [2]) and fed into decoder to aggregate features and predict the retrieved spatio-temporal tube.
3.2 Feature Extractor
Visual Encoder. We start by adopting a vision backhone (e.g., ResNet [10]) to extract the visual features for each frame nt individually in video V= {v: )T" The obtained 2D feature map of per frame is then flattened, bringing a visual feature sequence F. = {ft)] Each f( E N, xC. W and C, the visual feature channel.
serves compact representation for us, where N,
Linguistic Encoder. For the querying sentence S = {s, W with N words, we leverage a pre- trained linguistic embeding encoder, (e.g.BERRT model [5]), encode the linguistic representation. The textual features of input sentence query is denoted F. [ where J E RR. and C, is textual feature channel.
3.3 -onsistency-Aware Transformer
The detailed spatio-temporal video grounding procedure is executed by leveraging four crucial components: cross-modal transformer encoder, template generator, querry-uiieed transformer decoder, and two parallel prediction heads.
3.3.1 Multi-modal Encoding
The proposed encoder is obliged to fully exploit the tross-modal interaction between the video and text, correlating their corresponding semantics in fine- ere-ngr manner. Existing one-stage

---

video grounding approaches only restrict this interaction either in individual frames [40] or short video snippets [29]. This design leads to an inconsistent cross-modal feature alignment as the query sentence depicts a long-term evolutional event in videos.
To this end, the developed encoder aims at performing more consistent feature alignment between two modalities. Specially, given the input features Fu and Fs, a projection layer is first applied to embed them into the same channel dimension C. We denote the projected visual embedding as {P,, J where Pv, E R No xC and linguistic embedding as P, E RN.xC", Besides, we introduce a learnable token denoted as Pg E R² to integrate the global video context during encoding. and T analogous ones dubbed as pt = {p} R attend the local context within individual frame. The proposed encoder reads all above-described embedding tokens as its input and has M stacked encoder blocks. Each block consists of a spatial and temporal interaction layer. Both adopt the transformer-stylle encoder structure [35].
Spatial Interaction Layer. The goal of this layer is to conduct the intra- and inter- mido--tddlltyy relation modeling spatially for each local frame. detail, the input Ty for each interaction layer on the f-th frame can be formulated as:
24 Ipt Po. P
Pot
P
visual tokens Pry.
linguistic tokens
The joint input sequence iy is then fed into one spatial interaction layer to yield the contextualized visual-text representations for each frame respectively. It is worth noting that the state of frame- specific token př is enriched by both visual and linguistic context within frame U/--
Temporal Interaction Layer. The aforementioned spatial interaction layer only attends the local context information at frame level but lacks the modeling of global context across the whole video. To address this, our temporal interaction layer further models the interactions temporally between local frames. Similarly, the input belonging to this layer is fed into another transformer encoder layer, which can be formulated as:
Fg [pg. Pi: Pi. ....PP
frame tokens
To retain the temporal relative information, we also add a positional encoding to Zgg. Through this temporal layer, the local frame tokens pt {pt)T_ attend the whole video content and the global token also aggregates the global video-text context.
After the above encoding procedure, we harvest the contextualized multi-modal features F.t E NN, the global embedding Py for the whole video and the local embedding p: = {pi] with respect to all T video frames.
3.3.2 Template Generation
Unlike previous works that ground each frame separately without considering consistency, we design a template-based mechanism to correlate and restrict the predictions across all video frames. Con- cretely, the developed template is comprised of content term and a position term. The content term qe is shared by all frames to hold identical semantics depieted by the query sentence, while the position term 4p {qp 1 i= is characterized by per frame according to its appearance. Both of them are yielded by template generator based on the encoded local and global tokens.
MLP
0c
P2
MLP
8
Fgg
As illustrated in Figure 3, the content term de RC is generated by jointly considering the global -isual-linguistic context:
The illustration of procedure for yielding content term de and position term up in (3) the template generator.
Figure 3:
4. W.Pg + bc.
where both W. and b are learnable parameters. As for the position term 4p {g serves as reference anchor to restrict the grounding region on each individual frame e; Based on considerations of both global semantics consistency and local characteristic specificity critical
[T 1 where each dp is a 4-dimensional vector dubbed as (Iz, Dt. We, ht), which
5

---

for 4p. we generate it via depending both local frame embedding pt and global embedding Pg (See
Figure 3). In detail, inspired by [46, 42], we firstly leverage P3 to modulate the local embedding
Pt {pi)t=r It fulfilled by two linear layers to generate two modulation vectors y E R and
8 R based on Pgi
of = tanh (W,Ps + b,), 8 = tanh (WaPs + ba),
(4)
where Wy, by. W3 and by are learnable parameters. Finally, the position term of for each specific frame 2t is obtained by modulating p\:
g Sigmoid (fp (7 @ pl +9))
(5)
where fp is learnable mapping layer: RC R
3.3.3 Query-Guided Decoding
The objective of our decoder is to convert the multi-modal representations Fer obtained from the encoder to the target object tube based the generated template. Inspired by recent query- -bseed detectors [37, 19, 8], we formulate the above template (9e= 4p) as object queries to guide the overall decoding procedure, where qe serves as content queries probe target object patterns in encoded embedding and Op acts as positional queries to restrict the possible attending regions [19].
In detail, we denote {Qr) as the object query for each individual frame it. The initial query for feeding the decoder is formulated by Q: [C;; Pt] and we have:
C = qe+
Pt = Lincar(PE(qg.)),
(6)
where PE means the sinusoidal position encoding to 9 (Iz, Vt. wr, wr, h4). For disentangling the spatial and temporal feature aggregation, we introduce a dual-decoder to handle the prediction of bounding- box and temporal boundary separately. Both adopt the same standard architecture in line with DETR [2]. Each decoder block includes a self-attention module for message passing across all video frames, and a cross-attention module for feature probing within the corresponding frame v,. The content query Ct is thus enriched by stacking M decoder layers, resulting the final contextualized representation used for generating object tubes. Following the previous practice 37], the position query Pt in bounding-boo decoder is updated after each layer via predicting a relative offset with respect to of by shared regression head described below. More concrete exposition for the query- guided decoding procedure is presented in supplementary material.
Prediction Head. After decoding. the refined content query C, is utilized for simultaneously predicting spatial and temporal localization. Specially, a regression head implemented as a 3-layer MLP is adopted to perform bounding box coordinates prediction. The output of this prediction head a 4-dimensional coordinate offset (Ar,, Ay,, Aw,, Aht) and the final predicted box 6e E [0, 1]* in frame is obtained by:
b, (z: + AT. Us AWt Wi + Aw,, hr + Ahu).
The temporal boundary [f fe] is generated by predicting the start and end probabilities of each frame through a similar MLP prediction head.
3.4 Training and Inference
Training. At the training stage, we send batch of video-sentence pairs to the proposed model. Each pair has ground-truth bounding box sequence B = (b. (/= and the corresponding start and end timestamps. For spatial localization, we involve the box prediction loss Lsboz as follows:
Litox >r, Lr, (B, B) + Agioa B3, B);
where LL, and Cgion are the smooth L2 loss and generalized IoU loss [26] on the bounding boxes respectively. Note that Lidoz only considers predictions [t., tel. As for temporal localization, we follow [27, 29, 40] to generate two 1-dimensional gaussian heatmaps Hs, T. E RT for starting and ending positions. And the temporal prediction loss is termed as Liemp = Ls ( + La (rey Te), where L, and L. are the KI. divergence between target and predicted distributions. To encourage more accurate temporal prediction, we also add an auxiliary head to predict whether a frame belongs to the ground-truth segment. It is supervised by a binary cross-entropy loss term Leeg. The total training loss is defined as:
C Lsbox + Aromp Aseg
(9)

---

Inference. During inference, our framework produces the bounding boxes and the starting and ending probabilities for all frames. The start and end times of output tube, lę and fe, are determined by selecting segment with maximal joint starting and ending probability. Finally, the bounding boxes within [t,, fe] form the retrieved object tube.
4 Experiments
4.1 Datasets and Metrics
Datasets. To evaluate the proposed method, we follow the previous work [29] and adopt two large video grounding benchmarks: VidSTG [48] and HC-STVG [33]. Both two datasets are annotated with spatio-temporal tubes corresponding to text queries. VidSTG contains in total 6,924 untrimmed videos, which are clipped into 44,808 video-triplet pairs with 99,943 sentences describing 80 types of objects. We follow the pioneer work [48] to split all videos into three parts, 5,563 for training, 618 for validation, and 743 for testing, which paired with 80,684, 8,956, and 10,303 distinct sentences respectively. HC-STVG contains 5,660 untrimmed videos in multi-person scenes, each annotated with one expression related to human attributes or actions. This dataset is divided into the training set and the testing set with 4,500 and 1,160 video-sentence pairs, respectively.
Evaluation Metries. We follow [48] and select m_ mnvlol, m_ tlo and v R as the evaluation criteria. Note that vIoU 18% Eles, loU b1), where S; and S,, are the intersection and union
between the predicted tubes and ground-truth tubes, respectively. And loU by) describes the IoU score between the detected bounding box b, and ground-trut bounding box b, at frame t. While the m_vloU score is defined as the average vloU score over all testing videos. As for vloU @ R, which is the ratio of samples whose vloU > R in the testing subset. Besides, we also use m tloU (the mean of
[S.]
tloU) to evaluate the temporal localization performance only, where tloU
Suu
4.2 Implementation Details
In line with the previous methods, we adopt the ResNet-101 [10] as the visual encoder and RoBERTa [21] as the linguistic encoder. The output from the 4-th residual block of ResNet-101 is adopted as the extracted visual feature. For both encoder and decoder, the number of attention head is set to 8 and the hidden dimension of feed-forward networks in the attention layer is 2, 048. Follow- ing [29, 40J, part of the model parameters are initialized with the pre-trained weights provided in [12] and the whole framework is end-to-end optimized during training. Due to the appearance similarity between adjacent frames, the input video are uniformly down-sampled for computation efficiency. Besides, data augmentations include random resizing and random cropping are also applied to all training videos. The final object tube is obtained by linearly interpolating the predicted bounding- -bx in sampled frames. We empirically set the hyper-parametert in this paper with M 6, C 256 and the loss weights as AL, = 5, Ayaou = 3, Aqemp = 10 and Away = 2. The batch size and base learning rate are set to be 32 and le -4, respectively, More implementation details are shown in supplementary material.
4.3 Performance Comparison
To fully demonstrate the superiority of proposed model, we compare with all previous spatio-temporal video grounding methods on VidSTG and HC-STVG datasets. Specially, there are three types of competitors: (a) Factorized: These approaches factorize the STVG task by performing spatial and temporal grounding separately. They firstly utilize temporal visual grounding techniques like TALL [7] and L-Net [3] to predict the temporal boundary of the target object and then perform spatial visual grounding with methods like GroundeR [28], STPR [38] and WSSTG [4] to generate the bounding boxes. (b) Two-Stage. The two-stage approaches like STGRN [48], STGVT [33] and OMRN [47] accomplish spatio-temporal grounding via firstly generating box proposals in each frame by means of pre-trained object detector and then selecting the best matches from these candidates. (c) One-Stage. The ste-stage methods like STVGBert [29] and very recent concurrent work TubeDETR [40] tackle the STVG task through a unified architecture that directly grounds the final spatio-temporal object tube from the given video-sentence pair.
7

---

Table 1: Performance comparisons of the state-of-the-art on VidSTG test set (%)
Declarative Sentences
Interrogative Sentences
Methods
n_tioU
vlol vloU@0.3
vloU@0.5
doU
n_vloU vloU@0.3
vIoU@0.5
Factorized:
GroundeR [28]+TLLL
9.78
11.04
4.09
9.32
11.39
3.24
STPR [38]+ ESJ-TALLL [7]
34.63
10.40
12.38
33.73
11.74
4.36
WSSIG [4]+TALL [7]
11.36
14,63
5.91
10.65
13.90
5.32
GroundeR [28]+L-Net [3]
11.89
15.32
5.45
11.05
14,28
5.11
STPR [38]+L-Net [3]
40.86
12.93
16.27
39.79
11.94
14.73
WSSTGi [4]+1.-Ner
14.45
18.00
7.89
13.36
17.39
7.06
Two-Stage:
STGRN [48]
48.47
19.75
25.77
14,60
46.98
18.32
21.10
12.83
SIGVT [33]
29.80
OMRN [47]
50.73
23.11
32.61
16.42
49.19
20.63
28.35
14.11
One-Sier Stage:
STVGBert [29]
23.97
30.91
18.39
22.51
25.97
15.95
TubeDETR [40]
48.10
30.40
42.50
28.20
25.70
35.70
23.20
STCAT (Ours)
50.82
33.14
46.20
32.58
49.67
28.22
39.24
26.63
Table 2: Performance comparisons of the state-of-the-art the HC-STVG test set (9)).
Methads
tlot m_vloU v.o
v.oU@0.5
Two-Stage:
STGVT [33]
18.15
26.81
9.48
One-Slage:
STVGBert [29]
20.42
29.37
11.31
TubeDETR [40]
43.70
32.40
49.80
23.50
STCAT (Ours)
49.44
35.09
57.67
30.09
The complete performance comparisons between our model and the aforementioned approaches are presented in Tables and for two video benchmarks respectively. Overall, our STCAT achieves the state-of-the-ee the--ar grounding performance in terms of all evaluation metrics (m (m_vlol vloU, m tloU and v R) for both two complex benchmarks, demonstrating the superiority of the proposed model. From the shown results, we can further obtain the following observations. 1) Separating STVG task into individual spatial and temporal grounding problems achieved unsatisfactory performance in comparison tacking them simultaneously (worse performance than both one-stage and two-stage). 2) Compared with conventional two-stage methods, the removal of object detectors brings a significant performance boost. It is mainly because that one-stage methods end-to-end trainable and break through the restriction of the domain gap imposed by the pre-trained detectors. 3) Our proposed method outperforms all existing one-stage methods two datascts. Although the very recent method [40] also applies the DETR architecture to STVG task, it only performs cross-modal interaction at the frame level in the encoder and implicitly models the temporal correlation through the self-attention layer the origin DETR architecture. In contrast, our framework considers the global video context and maintains template mechanism queries for explicitly modeling consistency. All these novel designs contribute to the empirical superiority to [40] by large margins. Especially for the HC-STVG benchmark (e.g., 30.09% v.s. 23.50% in terms of vloU @.55) since it contains a longer query sentence to describe complicated scene change within movies, delivering higher demand for modeling subtle spatio-temporal clues.
4.4 Ablation Study
In this section, we conduct some ablations on the VidSTG benchmark to further investigate the contributions and design choices of the components in the proposed framework.
Effect of the template mechanism. To demonstrate the validation of the proposed global and local template mechanism, we design three variants of our models 1) Remove the global template in decoder and initialize the content query with zero vector, while remain the local template for generating position query. 2) Remove the local template in decoder and leverage a learnable position query while remain the global template as content query. 3) Remove both two types of template. The quantitative ablation results are shown in Table 3a. One can observe that cancelling either local or global template mechanism will weaken the grounding performance. Besides, the local template contributes more to the whole performance (+1.64% m vloU v.s. +0.95% m_ mlyU) This is basically

---

Query: There is grey bird away from another bird outdoors.
GT
STVGBett
SICAT (Ours)
Query: What does the adult in white clothes hug?
GT
STVGBert
SICAT (Ours)
Figure 4: Some illustration examples of the apatio-tempore video grounding predictions produced by the STVGBert [29] (yellow) and our model (blue), compared with annotated ground truth (green) on VidSTG benchmark.
intuitive since the local template is specified by frame characteristic. In summary, both two types of template are benefitial for the STVG task.
Local
Global
m vloU
vloU@0.3
vIoU@0.5
29.59
41.92
27.21
Setting
m_ vlov
vloU@0.3
vloU@0.5
31.23
44.90
29.21
w/o Temp
31.12
44.36
29.79
30.54
41.52
28.76
w/ Temp
33.14
46.20
32.58
33.14
46.20
32.58
(b) Ablations for the temporal interaction layer.
(a) Ablations for the local and glohal templates.
Table 3: The ablation studies on VidSTG benchmark (declarative sentences) for demonstrating the effectiveness of different components in our proposed STCAT.
Effect of the temporal interaction layer. The temporal interaction layer in our cross cross-modal en- coder obliged to model the global context across the whole video, which crucial for conducting a consistent feature alignment for encoder and maintaining video-level objective for the decoder. We thus explore the function temporal interaction layer eliminating it from each encoder block (denoted w/o temp). For the variant "w/o temp", the global embedding Pg just the average pooling of all local embedding Pt- The detailed ablation results are shown Table 3b. From the presented comparison, we can observe distinct performance drop without this layer, which further validates the effectiveness of our proposed temporal interaction layer.
4.5 Qualitative Analysis
In this section, we illustrate some examples in figure to qualitatively compare our predictions with the one-stage method STVGBert [29] on VidSTG dataset. As shown in Figure 4, our model can predict more meaningful and accurate spatio-temporal tubes than STVGBert. Specially, for the first query, the ground truth tube should contain the top-right bird in the given video specified by "away from". However, due to the lack of long-range context modeling, the STVGBert retrieved unmatched results. In contrast, our STCAT attended the whole video content and thus can precisely localize the desired instance. More visualization examples are provided in the supplementary material.
Conclusion
In this paper, we develop an effective one-stage consistency-aware transformer network (STCAT) for tackling the spatio-temporal video grounding task. To mitigate the feature alignment and prediction

---

inconsistency in previous approaches, our proposed STCAT introduces novel template mechanism to modulate the grounding procedure, which serves as a global target that guides and correlates the predictions among each individual frame to maintain temporal consistency. In addition, The extensive experimental results further demonstrate the superiority of the proposed network on the STVG task.
Acknowledgement: The research is supported by Science and Technology Innovation 2030 New Generation Artificial Intelligence (2020AAA010400), Beijing Natural Science Foundation (2190001), and Peng Cheng Laboratory Key Research Project No.PCL2021A07.
References
[1] Josh Beal, Eric Kim, Eric Tzeng, Dong Huk Park, Andrew Zhai, and Dmitry Kislyuk. Toward transformer-based object detection. arXiv preprint arXiv:2012.09958, 2020.
[2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.
[3] Jingyuan Chen, Lin Ma, Xinpeng Chen, Zequn Jie, and Jiebo Luo. Localizing natural language in videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8175-8182, 2019.
[4] Zhenfang Chen, Lin Ma, Wenhan Luo, and Kwan- Yee K Wong. Weakly-supervisedd spatio- temporally grounding natural sentence in video. arXiv preprint arXiv:1906.02544 025499, 2019.
[5] Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
[6] Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. Multi-modal transformer for video retrieval. In European Conference Computer Vision, pages 214-229. Springer, 2020.
[7] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In Proceedings of the IEEE international conference on computer vision, pages 5267-5275, 2017.
[8] Ziteng Gao, Limin Wang, Bing Han, and Sheng Guo. Adamixer: A fast-converging query- object detector. arXiv preprint arXiv:2203. 16507, 2022.
[9] Simon Ging, Mohammadreza Zolfaghari, Hamed Pirsiavash, and Thomas Brox. Coot: Co- operative hierarchical transformer for video-text text representation learning. Advances in neural information processing systems, 33:22605-223 2200000--228882221 2020.
[10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[11] Ronghang Hu, Mareus Rohrbach, Jacob Andreas, Trevor Darrell, and Kate Saenko. Modeling relationships in referential expressions with compositional modular networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1115-1124, 2017.
[12] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion.
detection for end-to-end multi-modal understanding, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1780-1790, 2021.
[13] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring objects photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787-798, 2014.
[14] Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 1 11336-11444 2020.
10

---

[15] Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. Hero: Hi- erarchical encoder for video+ language omn-rrepresentation pre-training. arXiv preprint 20ti:20005002000, 2020.
[16] Yue Liao, Si Liu, Guanbin Li, Fei Wang, Yanjie Chen, Chen Qian, and Bo Li. A real-time cross- modality correlation filtering method for referring expression comprehension. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10880-10889, 2020.
[17] Daqing Liu, Hanwang Zhang, Feng Wu, and Zheng-Jun Zha. Learning to assemble neural module tree networks for visual grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4673-4682, 2019.
[18] Jingyu Liu, Liang Wang, and Ming-Hsuan Yang. Referring expression generation and com- prehension via attributes. In Proceedings of the IEEE International Conference on Computer Vision, pages 4856-4866 2017.
[19] Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. Dab-detr: Dynamic anchor boxes are better queries for detr. arXiv preprint arXiv:2201. 12329, 2022.
[20] Xihui Liu, Zihao Wang, Jing Shao, Xiaogang Wang, and Hongsheng Li. Improving referring ex- pression grounding with cross-modal attention-guided erasing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1950-1959, 2019.
[21] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint 1907v 1692,, 2019.
[22] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-langenn lanngaag tasks. Advances in neural information processing systems, 32, 2019.
[23] Gen Luo. Yiyi Zhou, Xiaoshuai Sun, Liujuan Cao, Chenglin Wu, Cheng Deng, and Rongrong Ji. Multi-task collaborative network for joint referring expression comprehension and segmentation. In Proceedings of the IEEE/CVF Conference computer vision and pattern recognition, pages 10034-10000 10043, 2020.
[24] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779-788, 2016.
[25] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
(26) Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and loss for bounding box regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 658-666, 2019.
[27] Cristian Rodriguez, Edison Marrese- Marrese-Taylor, Fatemeh Sadat Saleh, Hongdong Li, and Stephen Gould, Proposal-free temporal moment localization of a natural-language query in video using guided attention. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2464-2473, 2020.
[28) Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor Darrell, and Bernt Schiele. Grounding of textual phrases images by reconstruction. In European Conference on Computer Vision, pages 817-834. Springer, 2016.
[29] Rui Su, Qian Yu, and Dong Xu. Stvgbert: A visual-linguistic transformer based framework for spatio-temporal video grounding. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 1533-1542, 2021.

---

[30] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bbrt: Pre-training of generic visual- lingusii representations. arXiv preprint Sed:::99000085300 2019.
[31] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7464-7473, 2019.
[32] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv: (008://908.0749999 2019.
[33] Zongheng Tang. Yue Liao, Si Liu, Guanbin Li, Xiaojie Jin, Hongxu Jiang, Qian Yu, and Dong Xu. Human-centric spatio-temporal video grounding with visual transformers. IEEE Transactions on Circuits and Systems for Video Technology, 2021.
[34] Arun Balajee Vasudevan, Dengxin Dai, and Luc Van Gool. Object referring in videos with language and human gaze. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 4129-4138, 2018
[35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[36] Peng Wang, Qi Wu, Jiewei Cao, Chunhua Shen, Lianli Gao, and Anton van den Hengel. Neigh- bourhood watch: Referring expression comprehension via language-guided graph attention networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1960-1968, 2019.
[37] Yingming Wang, Xiangyu Zhang, Tong Yang, and Jian Sun. Anchor detr: Query design for transformer- based object detection. arXiv preprint arXiv:2109.07107, 2021.
[38] Masataka Yamaguchi, Kuniaki Saito, Yoshitaka Ushiku, and Tatsuya Harada. Spatio-temporal person retrieval via natural language queries. In Proceedings of the IEEE International Confer- ence on Computer Vision, pages 1453-1462, 2017.
[39] Antoine Yang, Antoine Miech, Josef Sivie, Ivan Laptev, and Cordelia Schmid. Just ask: Learning to answer questions from millions of narrated videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1686-1697, 2021.
[40] Antoine Yang, Antoine Miech, Josef Sivic Ivan Laptev, and Cordelia Schmid. Tubeedtt: Spatio-temporal video grounding with transformers. arXiv preprint arXiv:2203. 16434, 2022.
[41] Sibei Yang, Guanbin Li, and Yizhou Yu. Cross-modal relationship inference for grounding referring expressions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4145 4154, 2019.
[42] Zhengyuan Yang, Tianlang Chen, Liwei Wang, and Jiebo Luo. Improving one-stage visual grounding by recursive sub-query construction. In European Conference on Computer Vision, pages 387-404. Springer, 2020.
[43] Zhengyuan Yang, Boqing Gong, Liwei Wang, Wenbing Huang, Dong Yu, and Jiebo Luo. fast and accurate one-stage approach to visual grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4683-4693, 2019.
[44] Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mattnet: Modular attention network for referring expression comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1307-1315, 2018.
[45] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In European Conference on Computer Vision, pages 69-85. Springer, 2016.
[46] Yitian Yuan, Lin Ma, Jingwen Wang, Wei Liu, and Wenwu Zhu. Semantic conditioned dynamic modulation for temporal sentence grounding in videos. Advances in Neural Information Processing Systems, 32, 2019.
12

---

[47] Zhu Zhang, Zhou Zhao, Zhijie Lin, Baoxing Huai, and Jing Yuan. Object-aware multi-branch relation networks for spatio-temporal video grounding, In Proceedings of the Twenty-Nine International Conference on International Joint Conferences on Artificial Intelligence, pages 1069-1075, 2021.
[48] Zhu Zhang, Zhou Zhao, Yang Zhao, Qi Wang, Huasheng Liu, and Lianli Gao. Where does it exist: Spatio-temporee video grounding for multi-form sentences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10668-10677, 2020.
[49] Minghang Zheng, Peng Gao, Renrui Zhang, Kunchang Li, Xiaogang Wang. Hongsheng Li, and Hao Dong. End-to-end object detection with adaptive clustering transformer. arXiv preprini arXiv:2011.09315 2020.
[50] Linchao Zhu and Yi Yang. Actbert: Learning global-local video-text representations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages $756-8755, 2020.
[51] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:0010004159 2020.
13


Reviews:
[{"review_id": "jp-Ey7SAqJ", "reviewer": "Reviewer_wsdA", "summary": "The paper presents an end-to-end one-stage Transformer-based framework for spatio-temporal video grounding (STVG). In particular, the authors introduced a novel multi-modal template to alleviate the alignment inconsistency and prediction inconsistency issues in the task above. The model followed an encoder-decoder design paradigm without any pre-trained object detectors, which reached resonable performance on both VidSTG and HC-STVG datasets.", "questions": "Please see the *Strengths And Weaknesses* section.", "limitations": "Yes,  the authors adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The main contribution is the semantic-consistency flow pattern between the encoder and decoder for the proposed Spatio-Temporal Consistency-Aware Transformer (STCAT) architecture, which mainly consists of an encoder, a template generator, and a corresponding decoder with couples of prediction heads.\n- The encoder utilized two specifically-designed interaction layers (spatial interaction layer and temporal interaction layer) to enhance the visual-linguistic feature fusion and the semantic consistency intra- and inter-frames. The template generator produced two terms for both global query representation and frame-level prediction constraint, which benefitted the decoder for spatial-temporal grounding.\n- STCAT sets new state-of-the-art performance on the two aforementioned datasets under most evaluation metrics.\n- The paper is well-written and easy to follow. The ablation study is also reasonable.\n\n**Weaknesses**\n- I am confused about the selection of ResNet instead of ViT as the visual feature extractor. ViT can naturally provide well-organized and adequately-interacted tokens for each frame, which can promote the feature quality for encoder layers. So why not use a transformer-fashion backbone for visual features?\n- The naming of the “Spatial Interaction Layer” seems a bit imprecise to me. It seems this module is mainly designed for visual and linguistic semantic fusion instead of boosting spatial information interaction in each frame. If the spatial information of visual tokens needs to be enriched before inter-frame interaction, I sincerely suggest you consider ViT-fashion models.\n- Detailed information about the input for the cross-attention layer in each decoder block might need to be supplied.\n- The usage of certain loss functions also seems unreasonable. What’s the aim of combining $L_1$loss with GIoU loss for bbox prediction? If the $L_1$loss is complementary to the GIoU loss for eliminating certain special cases when supervising via IoU,  the CIoU or DIoU might be a better metric than the combination, which also avoids tolerating the slow converge speed of GIoU.\n- The comparison between STCAT with other one-stage attention-based models seems insufficient. For example, what’s the throughput and parameter amount for STCAT to reach better performance than TubeDETR and STVGBert?\n- Could the authors provide an efficiency analysis such as FLOPs, latency and running time on the proposed method and the baselines to provide prove the effectiveness of the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "i2ZjsDNn72x", "reviewer": "Reviewer_JJPQ", "summary": "The authors proposed to approach the problem of spatiotemporal video grounding by a one stage end-to-end framework. The model adopts an encoder-decoder strategy and generates the video tube directly from output. Comparisons are made against other approaches to demonstrate the performance of the proposed approach.", "questions": "1) In terms of the model architecture contributions, if I understood correctly, the analogy is to add the decoder part similar to the one in traditional transformer on top of the encoder in ViT-based models and use it across different modalities, right? If that's the case, could you elaborate what specific modifications need to be done that can impact the performance of the model?\n\n2) For results in Table 1, when compared with OMRN, the other metrics are clearly outperforming, but it's not the case for m_tIoU. It was mentioned that \"OMRN leverages an off-the-shelf detector\", so can you explain why the detector mainly helps with one specific metric, or does this indicate that STCAT and OMRN are actually performing differently on different types of test data? If so, some qualitative comparisons might be helpful for error analysis and intuitive understanding.\n\n3) For the generated tube from the decoder, is there any kind of constraint applied so that the output is more consistent across frames, or is it simply enforced by attention? In other words, for other frame-based models, if some simple heuristic constraints can be added to output frames, would that also solve the inconsistency issue, at least to some extent?", "limitations": "The authors addressed the limitations in Supplementary Section 5.2.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is generally well written, the model architecture is intuitive and the results are reasonably well.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YxHAm7kccOd", "reviewer": "Reviewer_5BXu", "summary": "This paper introduces a new algorithm, STCAT, to solve the Spatio-Temporal video grounding problem. The new approach is based on the DETR implementation and achieves state-of-the-art performance. By considering the video's global information, it is expected to solve the problem of Prediction Inconsistency, i.e., prediction is performed only at the frame-level, ignoring the consistency between frames.", "questions": "1. I find that there is no explicit loss to enforce the feature consistency, I wonder how your designed module can make the feature “consistency”. I think more visualization and analyze deserve to be done for this part. \n2. In the ablation study, the effect of the temporal interaction layer section, what is the global embedding pg if the model is trained without the temporal interaction layer.", "limitations": "None", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The newly proposed method achieves state-of-the-art performance. \n2. The article is very clearly written and easy to understand.\nWeaknesses:\n1. The ablation study is insufficient, and we do not know the effect of the prediction head.\n2. The method is constructed on the top of DETR, so the paper contributes less to the network structure.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept