Knowing Your Target: Target-Aware Transformer Makes Better Spatio-Temporal Video Grounding

Authors: Xin Gu, Yaojie Shen, Chenxi Luo, Tiejian Luo, Yan Huang, Yuewei Lin, Heng Fan, Libo Zhang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: spatio-temporal video grounding

Abstract:
Transformer has attracted increasing interest in spatio-temporal video grounding, or STVG, owing to its end-to-end pipeline and promising result. Existing Transformer-based STVG approaches often leverage a set of object queries, which are initialized simply using zeros and then gradually learn target position information via iterative interactions with multimodal features, for spatial and temporal localization. Despite simplicity, these zero object queries, due to lacking target-specific cues, are hard to learn discriminative target information from interactions with multimodal features in complicated scenarios (e.g., with distractors or occlusion), resulting in degradation. Addressing this, we introduce a novel $\textbf{T}$arget-$\textbf{A}$ware Transformer for $\textbf{STVG}$ ($\textbf{TA-STVG}$), which seeks to adaptively generate object queries via exploring target-specific cues from the given video-text pair, for improving STVG. The key lies in two simple yet effective modules, comprising text-guided temporal sampling (TTS) and attribute-aware spatial activation (ASA), working in a cascade. The former focuses on selecting target-relevant temporal cues from a video utilizing holistic text information, while the latter aims at further exploiting the fine-grained visual attribute information of the object from previous target-aware temporal cues, which is applied for object query initialization. Compared to existing methods leveraging zero-initialized queries, object queries in our TA-STVG, directly generated from a given video-text pair, naturally carry target-specific cues, making them adaptive and better interact with multimodal features for learning more discriminative information to improve STVG. In our experiments on three benchmarks, including HCSTVG-v1/-v2 and VidSTG, TA-STVG achieves state-of-the-art performance and significantly outperforms the baseline, validating its efficacy. Moreover, TTS and ASA are designed for general purpose. When applied to existing methods such as TubeDETR and STCAT, we show substantial performance gains, verifying its generality. Code is released at https://github.com/HengLan/TA-STVG.

Decision: accept (oral)
Reviewer scores: 8, 8, 6, 8

Reviews and discussion:
<thread>
creation date:
1730343763631

thread_type:
official_review

<thread_object>
creator: reviewer_opic

<content>
summary:
This paper addresses the challenge of spatio-temporal video grounding (STVG) in complex scenarios by introducing the Target-Aware Transformer for STVG (TA-STVG). TA-STVG adaptively generates object queries based on target-specific cues from the video-text pair, utilizing two key modules: text-guided temporal sampling (TTS) and attribute-aware spatial activation (ASA). TTS selects relevant temporal cues using text information, while ASA refines these cues to capture detailed visual attributes for initializing object queries. Experiments on benchmarks, including HCSTVG-v1/-v2 and VidSTG, demonstrate that TA-STVG achieves state-of-the-art performance.

strengths:
- Unlike previous methods that use zero-initialized object queries, this paper presents the Target-Aware Transformer, which improves STVG by exploring target-specific cues for generating object queries.
- This paper proposes text-guided temporal sampling to select target-relevant temporal cues from videos and introduces attribute-aware spatial activation to leverage fine-grained visual semantic attributes for object query generation.
- Experiments on three datasets demonstrate its effectiveness.

weaknesses:
- Given the textual feature, first extract the subject's feature (usually the target to localize) from the textual sentence. My question is, do you typically obtain just one subject representation or multiple? If multiple are obtained, are they all processed through the same flow for decoding, or is there a further selection step?
- Sampled target-relevant temporal appearance and motion features—do they need to have the same quantity? If so, how is this controlled?
- The sampled target-relevant temporal appearance and motion features determine the frame information for the target object. This step can significantly impact subsequent accuracy. What happens if the selected frames do not contain the target object?

questions:
- Given the textual feature, first extract the subject's feature (usually the target to localize) from the textual sentence. My question is, do you typically obtain just one subject representation or multiple? If multiple are obtained, are they all processed through the same flow for decoding, or is there a further selection step?
- Sampled target-relevant temporal appearance and motion features—do they need to have the same quantity? If so, how is this controlled?
- The sampled target-relevant temporal appearance and motion features determine the frame information for the target object. This step can significantly impact subsequent accuracy. What happens if the selected frames do not contain the target object?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730512709457

thread_type:
official_review

<thread_object>
creator: reviewer_g9sv

<content>
summary:
The paper introduces a Target-Aware Transformer to address the limitations of traditional zero-initialized queries in STVG. The proposed method leverages target-specific information to generate more adaptive object queries that better interact with multimodal features, leading to improved accuracy. Extensive experiments on multiple benchmarks demonstrate that TA-STVG achieves SOTA performance.

strengths:
1. Target-specific cues improve object queries over zero-init methods, enhancing feature learning for better localization.

2. Extensive ablation studies support the effectiveness of the TTS and ASA modules, and the generality of modules is further validated by the integration into other models (e.g. TubeDETR, STCAT).

weaknesses:
1. The method’s contribution is focused primarily on object query initialization for STVG, with the innovation mainly revolving around attention-based filtering for keyframes and target-specific attribute extraction, which makes the innovation limited.

2. The multi-label classification in the ASA module is unclear. Specifically, are the extracted subject features for appearance and motion identical? What is the relationship between weak attribute labels and subject features, and do the weak labels update dynamically? 

3. The performance in complex scenarios requires further validation. Since object query initialization relies on the provided video-text pair, how robust is the query generation when visual information is heavily occluded, or the query is vague and lacks clear attributes? The robustness of the model against unclear attribute information should be further explored.

questions:
See weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730557443382

thread_type:
official_review

<thread_object>
creator: reviewer_jmhq

<content>
summary:
The paper introduces an approach to spatio-temporal video grounding (STVG) using a Target-Aware Transformer (TA-STVG). The method aims to improve the localization of targets within videos based on textual descriptions by generating object queries that are adaptively initialized with target-specific cues extracted from video-text pairs. The key contributions include two main modules: Text-Guided Temporal Sampling (TTS) for selecting target-relevant frames, and Attribute-Aware Spatial Activation (ASA) for exploiting fine-grained visual attribute information. The proposed method achieves state-of-the-art performance on three benchmarks: HCSTVG-v1/-v2 and VidSTG, and demonstrates generality by improving the performance of existing Transformer-based methods like TubeDETR and STCAT.

strengths:
1. The paper presents a novel approach to STVG by introducing target-aware object queries, which is a significant departure from traditional zero-initialized queries.

2. TA-STVG outperforms existing methods on multiple benchmarks, indicating the effectiveness of the proposed approach.

3. The writing of the paper is good and it is easy to understand.

weaknesses:
1.The method may require significant computational resources, which may limit its accessibility and practical application, especially for smaller research groups or companies.

2. The paper could provide more discussion on the computational complexity and parameter count compared to other methods, which is crucial for understanding its scalability.

3. The paper could benefit from a discussion on the robustness of the model against various challenges such as occlusion, similar appearances, or noisy text descriptions.

questions:
1. How does the computational complexity of TA-STVG compare to other state-of-the-art methods in terms of parameters and FLOPS?


2. How does the model handle videos with multiple targets that are described in a single text query?

3. Can the TTS and ASA modules be adapted to work with other types of multimodal data, such as audio descriptions or 3D video data?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730628272515

thread_type:
official_review

<thread_object>
creator: reviewer_cj8j

<content>
summary:
In this work, a new method called Target-Aware Transformer for STVG (TA-STVG) is proposed for addressing the Spatio-Temporal Video Grounding (STVG) task. To replace the traditional zero-initialized object queries used in previous methods, TA-STVG is designed to generate target-specific object queries from the given video-text pair, with two main modules named Text-guided Temporal Sampling (TTS) and Attribute-aware Spatial Activation (ASA). Experiments and analysis are conducted to validate the effectiveness of the proposed TA-STVG.

strengths:
1.The intuitive motivation of TA-STVG is clear and easy to understand, i.e., the adaptively generated object queries containing target-relevant information facilitate the spatio-temporal localization of textual expressions.

2.The proposed method to enhance and keep track of the information about the described subject in the sentence is concise and effective.

3.State-of-the-art performances are achieved by the method across all three datasets.

4.Qualitative results and analysis are adequately given to illustrate the effectiveness of the proposed components.

weaknesses:
1.It would be better to include more discussions and comparisons on some recent works in the related domain of video grounding. For example, the proposed ASA module adopts similar techniques of spatial activation learning [1] and concept classification [2].

2.In the proposed TA-STVG, object queries are generated from globally pooling the multimodal features and are repeatedly used for each frame in the video. This way of object query construction seems suboptimal since the frame-specific information is ignored for the queries. Is there any consideration or motivation for such design choice? Why not construct the frame-specific object queries?

3.How are the weights for different loss terms determined? Are there any ablation results?

4.Some typos exist in the current manuscript, for example, "such" should have been "such as " in Line 29.

References [1] Collaborative static and dynamic vision-language streams for spatio-temporal video grounding. In CVPR 2023. [2] Siamese Learning with Joint Alignment and Regression for Weakly-Supervised Video Paragraph Grounding. In CVPR 2024.

questions:
Please refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732546052376

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Rebuttal

comment:
We appreciate all the reviewers for their thoughtful and careful feedback. In this paper, we propose a novel Target-Aware Transformer for STVG (TA-STVG) that uses target-specific cues for query generation. Experiments on three datasets validate its efficacy in improving target localization. Additionally, applying it to other frameworks like STCAT and TubeDETR shows consistent improvements, demonstrating its generalizability.

As suggested by the reviewers, we have thoroughly revised our manuscript and address each of the issues raised in the reviews:

- Reviewer `CJ8J`:  We have discussed the missing related works and will cite them in revision (*Q1*), further explained the motivation behind the query construction method (*Q2*), and included ablation experiments on loss weights (*Q3*). We also polished the whole paper to fix typos (*Q4*).
  
- Reviewer `jmHQ`: We have explained the computational requirements of our method (*Q1*), provided a comparison of computational complexity with other methods (*Q2*, *Q4*), and discussed the robustness of the model (*Q3*). We further analyzed handling multiple targets in a single text query (*Q5*) and the adaptability of TTS and ASA modules (*Q6*).
  
- Reviewer `g9Sv`: We have clarified our novelty and core contributions (*Q1*), explained the multi-label classification in the ASA module (*Q2*), and further validated the performance in complex scenarios (*Q3*).

- Reviewer `oPic`: We have clarified the details of extracting subjects from the text (*Q1*), the number of target-relevant features sampled by TTS (*Q2*), and further discussed what happens if the selected frames do not contain the target object (*Q3*).

We once again express our heartfelt gratitude to all the reviewers for their valuable feedback, and we hope that our responses satisfactorily address all concerns. Please feel free to let us know if you have any remaining concerns and we are happy to address them!
</content>
</thread_object>

</thread>


Metareview:
metareview:
All reviewers recognize the novelty and contributions of this paper. The paper presents a novel approach to STVG by introducing target-aware object queries, which is a significant departure from traditional zero-initialized queries.The motivation is clear and easy to understand. After rebuttal, the authors have addressed the reviewer's main concerns. So, this paper is recommended for acceptance.