Let Me Finish My Sentence: Video Temporal Grounding with Holistic Text Understanding

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 12:56:14

Abstract: Video Temporal Grounding (VTG) aims to identify visual frames in a video clip that match text queries. Recent studies in VTG employ cross-attention to correlate visual frames and text queries as individual token sequences. However, these approaches overlook a crucial aspect of the problem: a holistic understanding of the query sentence. A model may capture correlations between individual word tokens and arbitrary visual frames while possibly missing out on the global meaning. To address this, we introduce two primary contributions: (1) a visual frame-level gate mechanism that incorporates holistic textual information, (2) cross-modal alignment loss to learn the fine-grained correlation between query and relevant frames. As a result, we regularize the effect of individual word tokens and suppress irrelevant visual frames. We demonstrate that our method outperforms state-of-the-art approaches in VTG benchmarks, indicating that holistic text understanding guides the model to focus on the semantically important parts within the video.

Main paper: 
 introduction: 
 Recently, the boom of video and streaming platforms such as Dis-ney+, YouTube, TikTok, Netflix, etc., has led to an abundance of online video content. Naturally, the growing pool of videos from various platforms sparked an interest in efficiently searching videos using text inputs. Video Temporal Grounding (VTG) is a prominent research area within video-text search, that focuses on grounding visual frames that correspond to custom queries. Within VTG, various tasks such as Moment Retrieval (MR) [6, 14, 24, 27, 50, 51] and Highlight Detection (HD) [20, 37, 47] has been proposed.The goal of MR is to identify time intervals that are highly relevant to text queries, while HD assesses the significance of each video frame to select the most significant segments. HD can be "A man speaks to the camera while he is wearing shades and black biker's helmet." Unlike previous works that do not take holistic text understanding into account, our method effectively filters out frames that do not correspond to the full context of the query. Here, our model does not predict the final frames due to the absence of the helmet and shades mentioned in the query.categorized into two perspectives: query-independent and querydependent. This paper focuses exclusively on query-dependent HD, which utilizes text queries to analyze video content. Despite the distinct operational focuses of MR and HD, both tasks share the core aim of aligning video content with corresponding natural language queries. Recognizing their shared goal, [13] introduced the QVhighlights dataset. This allows for simultaneous training on both MR and HD, promoting a unified approach to VTG. Prior approaches have focused on developing cross-modal interaction strategies [10, 18] or developed models specifically for the demands of the MR and HD tasks [36, 42] . However, despite these advancements, existing models often treat the text query as a sequence of tokens rather than an entire sentence. These approaches may neglect the overall textual semantics, as individual text tokens either lack the capacity to convey the complete meaning and/or cause the model to attend to unrelated words or frames. This oversight can limit the model's ability to fully capture the intent of the query. For instance, as shown in Fig. 1 , the model without holistic text understanding may highlight the latter part of a video in response to the tokens inside the clause "A man speaks to the camera. " To tackle this issue, we focus on utilizing the global information contained within text queries, emphasizing its importance in accurately identifying the most relevant video frames.To this end, we propose a novel framework that utilizes a holistic, or global text anchor-representing the full input sentence-to selectively suppress less relevant video frames while emphasizing the relevant ones. Leveraging this specialized token, our framework introduces a gated cross-attention mechanism that effectively filters out irrelevant video content. The gated cross-attention employs two gating mechanisms: local and non-local gates. The local gate assigns weights based on channel-wise similarity between the text anchor and individual frames (frame-level). In contrast, the non-local gate assigns weights by evaluating the overall relevance between the text anchor and the entire video (clip-level), prioritizing frames that exhibit greater contextual alignment with the global text query. The overall relevance is computed through an anchorquery cross-attention mechanism that employs the text anchor as a query to interact with video frames. An attention map derived from this interaction effectively assesses clip-level correlation, thereby emphasizing the focus on contextually pertinent frames. To further refine the precision in assessing similarity, we introduce two fine-grained alignment losses that optimize clip-level consistency and frame-level relevance. These losses use the text query as an anchor, enabling a more targeted and accurate alignment between the video content and the corresponding textual information. The clip-level consistency loss aims to minimize the discrepancy between the anchor and the clip-level video feature outputted from the anchor-query cross-attention layer. This reduction aims to enhance the accuracy in determining the relative significance of each video frame in relation to the anchor. On the other hand, the framelevel relevance loss aims to ensure that frames relevant to the text query are closely aligned with the global text query representation. It minimizes the distance between the global query and the corresponding frames, thus enhancing the model's ability to more accurately align relevant video content with the text.To demonstrate the effectiveness of our proposed framework, we conduct extensive experiments on the QVHighlights [13] dataset, as well as on other notable VTG benchmarks, including Charades-STA [6] and TACoS [27] . Our experimental results show that our proposed method, to the best of our knowledge, achieves state-ofthe-art performance compared to previous methods. Additionally, we carry out a detailed ablation study to further assess and validate the advantages of our proposed method. The contributions of our approach are summarized as follows:• We introduce a novel framework that utilizes a global textanchor based approach to enhance video grounding, leveraging holistic textual information to accurately filter and prioritize relevant frames.  
related works: 
 Moment Retrieval (MR) is a task that identifies temporal moments that are highly relevant to a given natural language query. Within MR, previous works have been categorized in two sections: proposal-based and proposal-free. As in the name, proposal-based methods have generally followed a pipeline where a model generates candidate windows from the entirety of the video, then grants a rank based on the matched scores. This approach often relies on predefined temporal structures like sliding windows [1, 6, 18, 31, 52] or temporal anchors [3, 33, 44, 46, 49, 51] to formulate candidate moments. On the other hand, proposal-free methods solve the task as a regression problem where they directly regress the start and end time frames through various multimodal methods. Within this line of works, previous methods have utilized methods such as multimodal coattention [7, 23, 48, 50] , dynamic filters [29] , and additional features [4, 29] to varying degrees of success. Both proposal-based and proposal-free methods are effective but rely on hand-crafted processes such as proposal generation and non-maximum suppression.Highlight Detection (HD) is a task that focuses on the evaluation of individual video clips' significance by assigning them clipwise saliency scores and highlighting the segments with the highest scores. However, HD datasets [30, 34, 38] are usually domainspecific and operate independently of textual queries. The common datasets [20, 47] available for query-based highlight detection offer a limited number of annotated frames for training and evaluation.The scarcity of query-dependent HD datasets underscores the perception of HD as primarily a vision-only problem. Unlike earlier HD datasets that were query-agnostic, we explore and test on an HD task that offers a saliency score for query-relevant clips, facilitating models to perform query-dependent highlight detection.Video Temporal Grounding seeks to combine the two aforementioned tasks. Despite their similar objectives, the absence of a unified dataset supporting both tasks has constrained simultaneous exploration and combination of these two fields. To this end, [13] released QVhighlights and proposed Moment-DETR as a simple baseline. Subsequently, UMT [21] explored the addition of audio cues to enrich query generation, and QD-DETR [22] enhanced query-dependent video representations through a specialized crossattention module. MH-DETR [45] explores further cross-modal integration by merging visual and textual features through a pooling mechanism, while UniVTG [17] proposes a unified grounding model including video summarization. More recent efforts by TR-DETR [36] and UVCOM [42] incorporate distinct task characteristics into their frameworks. Even with these advancements, existing models often interpret text queries as a sequence of individual tokens, neglecting the holistic semantics of the entire query. To address this issue, we introduce a novel framework that emphasizes holistic textual understanding to accurately identify and emphasize relevant video segments. Cross-modal alignment is an important concept in tasks that involve multiple modalities. This type of alignment ensures that representations from various modalities, such as visual and textual data, are positioned in a shared embedding space. This is often achieved through contrastive learning methods [11, 12, 15, 26, 43] .Particularly in the video-text domain, approaches such as [2, 8, 41] aim to align these modalities in a more fine-grained manner. 
method: 
 In this section, we provide an in-depth explanation of our approach that centers on the global text-anchor. As shown in Fig. 2 , our framework integrates four components: feature extraction, cross-modal interaction, fine-grained alignment loss, and prediction. We first outline the task while detailing the feature extraction process in Sec. 3.1. We then introduce the core method of our framework in Sec. 𝐹 ! 𝑊 ! 𝑊 " 𝑊 # ! 𝑄 ! 𝐾 ! SIM Softmax 𝑉 ! MUL 𝑊 " 𝑄 SIM Softmax 𝑊 ! 𝑊 # 𝐾 𝑉 MUL 𝐾 ! (mean pool of 𝐾) ⨀ 𝜎 ⨀ ⊗ 𝑔$ ∈ ℝ $!×& 𝑔 ' ∈ ℝ $! Projector × 𝑁 Fine-Grained Alignment Loss Gated Cross-Attention ϕ Anchor-Query Cross-Attention 𝑡 1 𝐹 # t 1 𝑂 + 𝑂 𝐹 ! $ + 𝐹 2 t 1 3 t 1 4 . 𝑣 3 . 𝑣 4 Figure 2:The pipeline of our framework consists of four components: feature extraction, cross-modal interaction, fine-grained alignment loss, and prediction. First, we extract visual and text features with frozen pre-trained encoders. Since the task requires cross-modal understanding and suppression of irrelevant information, we incorporate the gated cross-attention mechanism for the cross-modal interaction. The encoded features of cross-modal interaction are leveraged through the fine-grained alignment loss, which guides the model to enhance cross-modal alignment. Finally, the visual and textual representations from this aligned embedding space are fed into the prediction section to produce task-specific outputs.two critical cross-attention mechanisms-gated cross-attention and anchor-query cross-attention. Both mechanisms utilize the global text anchor to enhance the interaction between the video content and the text query. We then present newly devised fine-grained alignment losses in Sec. 3.4, aimed at improving the alignment between video content and text queries. Finally, we detail the inference procedure, explaining how our model concurrently predicts for both Moment Retrieval (MR) and Highlight Detection (HD).Given a video 𝑉 ∈ R 𝐿 𝑣 ×𝐻 ×𝑊 ×3 , consisting of 𝐿 𝑣 frames sampled from the original video at specific intervals and a natural language text query 𝑇 comprising 𝐿 𝑡 tokens, the objective is to identify all relevant moments 𝑚 𝑛 = 𝑚 𝑐 𝑛 , 𝑚 𝜎 𝑛 𝑁 𝑛=1 , where 𝑚 𝑐 𝑛 denotes the center coordinate of a moment and 𝑚 𝜎 𝑛 represents its width. Additionally, the model predicts a frame-level saliency score {𝑠 𝑖 } 𝐿 𝑣 𝑖=1 concurrently. For feature extraction, following previous works [13, 17, 22, 36, 42] , we employ pre-trained encoders 𝐸 𝑉 and 𝐸 𝑇 to extract visual features𝐹 𝑣 = [𝑣 1 , 𝑣 2 , . . . , 𝑣 𝐿 𝑣 ] ∈ R 𝐿 𝑣 ×𝑑 𝑣 and text features 𝐹 𝑡 = [𝑡 1 , 𝑡 2 , . . . , 𝑡 𝐿 𝑡 ] ∈ R 𝐿 𝑡 ×𝑑 𝑡 ,respectively. Separate Multi-layer Perceptron (MLP) are used to project the video and text features into a shared embedding space of the same dimension 𝑑. In the following sections, we describe in detail our contributions of cross-modal interaction, fine-grained alignment loss, and how we train our model with our proposed methods.The primary goal of the VTG task is to identify video sequences that correspond to the entire query. In order to do this effectively, an essential aspect of VTG is that it needs to capture the overall meaning of a given text query. Therefore, we propose to use a global text anchor that embodies a holistic understanding of the sentence, as emphasized in our title: Let Me Finish My Sentence. Adopting the global text query as an anchor offers two main advantages: (1) the text modality is generally less noisy compared to the visual modality due to its discreteness, and (2) we can effectively leverage prior knowledge from language models trained on a large corpora, which are well-generalized across various domains. We employ a global mean pooling operation, denoted as 𝜙, to derive a global text anchor from token-level text representations. This anchor is then integrated into our cross-modal interaction module and is further detailed in the following section.As mentioned, we hypothesize the need for holistic understanding of the text query so that the model may take into account the entire text query instead of being biased to certain words in the query. Hence, we propose and introduce a global text anchor to address the aforementioned issues in these ways: 1) suppressing non-essential video frames through a gating mechanism with the global text anchor, 2) enhancing cross-modal alignment between video content and global textual information. The versatile use of the global text-anchor enhance the contextual relevance of videotext interactions and cross-model understanding.Gated Cross-Attention. As shown in Fig. 2 , we extend the conventional cross-attention mechanism [40] by incorporating local and non-local gates. The standard cross-attention is defined as: The local gate weight, denoted by 𝑔 𝐿 , employs element-wise multiplication and sigmoid activation to assess the relevance between the video clip and the text query, utilizing the global key 𝐾 𝐺 ∈ R 𝑑 . This key 𝐾 𝐺 is derived by mean pooling 𝐾 across time dimensions:EQUATIONEQUATIONwhere 𝑊 𝑔 𝑞 and 𝑊 𝑔 𝑘 are trainable weights applied to the query 𝑄 and the global key 𝐾 𝐺 , respectively, to capture channel-wise relevance between single frame and global text. The output, 𝑔 𝐿 , modulates the feature vector 𝑉 ′ through element-wise multiplication, resulting in a relevance-enhanced feature representation 𝑉 ′ 𝐿 :EQUATIONThe non-local gate weight 𝑔 𝑁 , designed for broad relevance assessment across video clips, modifies the interaction to enhance contextually pertinent video frames:EQUATIONThis mechanism, 𝑔 𝑁 ∈ R 𝑑 , computes min-max normalized attention scores between the global text anchor and video frames, effectively weighting the relative significance of each video frame in relation to the global text. It serves as an effective filter that emphasizes frames with substantial contextual relevance while suppressing the less relevant ones. Through the integration of both local and non-local gates, our model emphasizes clips closely aligned with the text query while minimizing the influence of non-relevant clips in subsequent steps.Anchor-Query Cross-Attention. Here, the global text anchor 𝑡 𝑎 , obtained through the mean pooling operator 𝜙 (𝐹 𝑡 ) is encoded into the visually enriched text-query. The video clip representations 𝐹 𝑣 are adapted as both keys and values within the cross-attention mechanism. While the gated cross-attention approach learns the alignment between individual video frames and text tokens, the anchor-query cross-attention addresses the relative relevance between the global text representation and each of the video frames. The operational equation is defined as follows:EQUATIONIn this configuration, 𝑊 𝑄 and 𝑊 𝐾 serve as projection layers for video and text modalities within the gated cross-attention mechanism, respectively. They aid in establishing the understanding between video content and textual information. Since softmax𝑄 𝑡 𝐾 𝑇 𝑡 √ 𝑑 𝑘represents the similarity score between the text query and video frames, we represent this calculation as 𝑔 𝑁 .The two parts in the cross-modal interaction module refine the intermediate features based on cross-modal understanding. The module guides the model to emphasize the frames which are contextually relevant to the text query.Understanding of the fine-grained correlation between text queries and video clips is essential for MR and HD tasks. Moreover, the validity of the local and non-local gates depends on the reliability of cross-modal understanding. To address this, we propose two loss functions to better learn cross-modal alignment: 1) preserving the consistency of the global text representation before and after the anchor-query cross-attention step, and 2) inducing the model to capture the frame-level relevance between the visual frame and global text representation. EQUATION-1EQUATION)Optimizing the model with this objective enhances the crossmodal alignment between the text anchor 𝑡 𝑎 and semantically relevant video clips. By minimizing the distance between 𝑡 𝑎 and the visually enriched text query t𝑝 𝑎 , which incorporates both relevant and irrelevant video frames, the approach guides the text anchor's attention strongly towards semantically relevant video frames.The training objective aims to refine the video clip representation F𝑣 , derived from the gated cross-attention, to align closely with the text anchor 𝑡 𝑎 for relevant text queries, and to diverge when the queries are irrelevant. This method conditions the model to enhance the semantic correlation between video clips and the corresponding text query, ensuring their representations in the embedding space accurately reflect their contextual relevance.Frame-Level Relevance Losses. This loss function refines the representation of video clips F𝑣 , which has been processed through gated cross-attention, by optimizing the alignment between video frames and the text anchor. Specifically, it enhances the similarity between relevant video frames v𝑝 and the text anchor 𝑡 𝑎 , while reducing the similarity with irrelevant frames v𝑛 . This loss guides the model to learn the fine-grained correlation between visual frames and the corresponding text query, ensuring their representations in the embedding space accurately reflect their contextual relevance. The similarity score between the i-th frame v𝑖 and the text anchor 𝑡 𝑎 is given by 𝐷 𝑖 = 𝜎 ( v𝑖 • 𝑎), where 𝜎 denotes a sigmoid that incorporates the similarity score. The loss function then is:EQUATIONHere, 𝐶 𝑖 is a binary indicator reflecting whether the i-th clip is relevant (1) or irrelevant (0) to the text query.In our cross-modal interaction module, which consists of 𝑁 transformer layers, we aim to generate a composite representation. This is achieved by channel-wise concatenating the intermediate outputs from each layer 𝑂 𝑙 with the input video features 𝐹 𝑣 , and then projecting this concatenation into a 𝑑-dimensional space using a linear projection layer 𝑓 :EQUATIONSubsequently, the output t𝑎 is concatenated with 𝑂 in a temporal manner to form the basis for a query-dependent adaptive classifier [22, 35] :EQUATIONThis concatenated output, denoted as 𝑂, is considered the final feature set of the cross-modal interaction module.Highlight Prediction. The final feature set 𝑂 is processed through a transformer encoder 𝐸. Separate Multi-layer Perceptron (MLP) is used to project the video and text features into a shared embedding space of the same dimension 𝑑, producing Ô = { ô1 , . . . , ô𝐿 𝑣 , t′ 𝑎 }. Following QD-DETR [9, 22] , we calculate a vector of saliency scores 𝑆 ∈ R 𝐿 𝑣 for every frame in the video as follows:EQUATIONwhere 𝑤 𝑠 and 𝑤 𝑣 are learnable parameters applied to the query representation and each video frame representation respectively.Moment Retrieval Prediction. Following decoder strategies from prior research [19, 22, 36] , we utilize dynamic anchor boxes to represent moment queries (which is clearly separate from text queries). Together with the output Ô = { ô1 , . . . , ô𝐿 𝑣 }, this input is provided to the decoder 𝐷, resulting in moment features 𝑄. 𝑄 undergoes processing via a Multi-Layer Perceptron (MLP) and a sigmoid activation to generate predictions for moment dimensions ( m), yielding 𝑀 moment predictions. Concurrently, another linear layer equipped with a softmax activation categorizes each predicted moment as foreground or background ( p).Highlight Loss. The highlight loss 𝐿 ℎ𝑙 is comprised of a margin contrastive loss 𝐿 margin and a rank-aware contrastive loss 𝐿 rank . Margin loss contrasts high and low scoring frames within (𝑡 high , 𝑡 low ) and outside (𝑡 in , 𝑡 out ) ground-truth moments, given by:EQUATIONwith Δ denoting the margin. Following QD-DETR [22] , the rankaware loss is:EQUATIONwhere 𝑋 pos 𝑟 and 𝑋 neg 𝑟 are the indexes of positive and negative frames within the 𝑟 -th rank group, respectively, and 𝜏 is a temperature scaling parameter. The total highlight loss is as follows:EQUATION)Moment Retrieval Loss. To address the set prediction challenge in moment retrieval without a direct one-to-one correspondence between ground truth and predictions, we apply the Hungarian matching algorithm. This algorithm pairs ground truth moments with predictions, where σ (𝑖) indexes the predicted moment matched to the 𝑖-th ground truth moment. The span loss for matched pairs is as follows:EQUATION)incorporating the generalized IOU loss [28] . The moment retrieval loss combines classification and span losses:EQUATIONwhere I (•) applies span loss only to non-empty ground truth moments.Overall Loss. The overall loss is defined as:EQUATION)where the coefficients 𝜆 clip and 𝜆 frame are parameters that balance the contribution of clip-level and frame-level losses to the overall loss, respectively. 
experiments: 
 In this section, we outline the experimental setup and list details on the dataset, evaluation metrics, and implementation specifics as discussed in Sec. 4.1. Then, we compare the performance of our framework with established baselines in Sec. 4.2 and show a detailed ablation study in Sec. 4.3. Lastly, we show qualitative results showcasing the effectiveness of our approach in Sec. 4.4.Datasets. As our task is to detect highlights while retrieving moments, we use commonly used MR and HD dataset QVhighlights [13] as our main benchmark. The QVhighlights dataset currently stands as the sole dataset available to test both MR and HD tasks concurrently and comprises of over 10,000 YouTube videos accompanied by human-written, free-form text queries. For moment labels, each video-text pair is annotated with one or more relevant moments, and highlight labels are provided with 5-scale saliency scores (ranging from 1 being very bad to 5 being very good). Additionally, in order to ensure a fair benchmark for evaluation, the performance on the test set is assessed exclusively through submissions to the QVhighlights server. 1 In addition to this, to further test the efficacy of our method, we evaluate it on other VTG datasets, namely Charades-STA [6] and TACoS [27] . Charades-STA features 9,848 videos with 16,128 query-moment pairs focusing on indoor activities. TACoS comprises 127 videos annotated specifically for cooking scenarios.Evaluation Metrics. We follow the conventions established in previous research [10, 13, 17, 21, 22, 36, 42, 45] . In MR, we apply Recall@1 (R@1) at IoU thresholds of 0.5 and 0.7. We also calculate mean Average Precision (mAP) for IoU thresholds [0.5 : 0.05 : 0.95].For HD, we measure mAP and HIT@1, where HIT@1 is determined by the hit ratio of the clip with the highest score. Baseline Architecture. Our framework is built upon the QD-DETR [22] architecture, which is a widely used baseline in Video Temporal Grounding due to its effective use of cross-attention layers for injecting text query information into video frames. We build upon this baseline while retaining its standard cross-attention mechanism, decoder structure, and rank-aware loss. Our method introduces novel approaches aimed to effectively leverage the holistic context of text queries for improved video frame selection and alignment. Implementation Details. We configure the number of layers in the transformer encoder 𝐸 and decoder 𝐷 as 3. We set the crossmodal interaction layer count to 2. The loss balancing parameters are established as 𝜆 L1 = 10, 𝜆 iou = 1, 𝜆 cls = 4, 𝜆 frame = 1, and 𝜆 clip is adjusted to 1 for QVHighlights and Charades-STA, and 0.6 for TACoS. We set the batch size to 32 and the learning rate (LR) to 0.0001 for QVHighlights, maintain a batch size of 32 with an LR of 0.0002 for Charades. We adjust the batch size of 16 with an LR of 0.0002 for TACoS following previous works. Across all datasets, training proceeds for 200 epochs with a learning rate reduction at epoch 100, using the Adam optimizer. Additionally, for all datasets, we set the hidden dimension 𝑑 to 256 and the number of moment queries 𝑀 to 10. Whereas otherwise stated, we employ a pre-trained SlowFast and CLIP [26] model for video feature extraction. Specifically for Charades-STA, additionally features were extracted using VGG [32] , C3D [39] , and GloVe [25] . All models were trained on a single NVIDIA RTX 4090 with an average training time of 3 hours for all 200 epochs on our machines.We compare how our method performs in relation to recent stateof-the-art methods for MR and HD and summarize our findings in the following sections.Results on QVHighlights. In Table 1 , we list the experimental results of our method as well as other established methods on the QVHighlights dataset. Our method diverges from XML [14] , which adopts a proposal-free strategy, and aligns with the transformerbased and end-to-end trainable nature of current baselines [10, 13, 16, 17, 21, 22, 36, 42, 45] . MH-DETR [45] and QD-DETR [22] focus more on cross-modal interaction before transformer encoder and TR-DETR [36] explores the inherent reciprocity between MR and HD, and UVCOM [42] is tailored to address the unique demands of both MR and HD tasks effectively. Although EaTR [10] is a fairly recent work, we do not list them in our main table as they do not evaluate on the QVHighlights test split. Our model capitalizes on global text semantics and novel loss functions within a refined cross-modal interaction framework, surpasses all compared methods. Notably, our method outperforms the baseline model, QD-DETR [22] , in MR by achieving a 3.5% improvement in R@1 at IoU 0.5, 4.76% improvement at IoU 0.7, and 3.7% increase in average mAP, and in HD by 1.63 mAP and 3.20 in HIT@1. Our method outperforms the most recent method, UVCOM [42] , in MR by 2.40% R@0.5, 2.24% R@0.5, and 0.39% average mAP, and in HD by 0.50 mAP and 1.40 HIT@1 respectively. We find that our method outperforms all previous baselines across all metrics in both MR and HD, to the best of our knowledge, setting the new state-of-the-art.Results on Charades-STA and TACoS. To test the generalizability and efficacy of our method, we extend our experimentation to other VTG benchmarks such as Charades-STA [6] and TACoS [27] . On top of our comparisons with transformer-based models, we further test our approach against prior proposal-based approaches [49, 51] . As evidenced in Tables 2 and 3 , our method surpasses previous techniques, establishing new state-of-the-arts on these benchmarks. Particularly on Charades-STA, our approach demonstrates its robustness by consistently outperforming existing methods across a variety of backbones, including learned multimodal features from CLIP [26] , as well as 2D and 3D features from VGG [32] and C3D [39] , respectively. Notably, with features combining SlowFast [5] and CLIP (SF+C, C), our method surpasses the recent state-of-the-art model, UVCOM [42] , by achieving improvements of 1.48% in R@1 at an IoU of 0.5 and 2.85% at an IoU of 0.7. On TACoS, our method once again outperforms all previous baselines, affirming its effectiveness across diverse domains and datasets.To understand the individual components of our framework and its effects, we present a series of ablation studies on QVHighlights validation split.Gated Cross-Attention. We analyse the significance of employing both local and non-local gates within our gated cross-attention framework and present it in Table 4 . Implementing the non-local gate alone enhances performance in MR and HD tasks, underscoring To further assess the validity of our non-local gate weight (𝑔 𝑁 ) within the gated cross-attention framework, we test to see if it can be used directly as a saliency score for Highlight Detection on the QVHighlights validation split. Table 5 demonstrates that using 𝑔 𝑁 directly to predict saliency not only exceeds Moment-DETR's HD performance but also shows the utility of our gate mechanism in emphasizing relevant video sections in cross-modal interaction. Additionally, '𝑔 𝑁 w/o Non-local, ' represents a scenario where 𝑔 𝑁 is computed without applying the non-local gate, which corresponds to the the second row of Table 4 , and this results in a significant drop in performance. This underlines the non-local gate's critical role in emphasizing relevant video frame and enhancing the accuracy of the similarity measure between global text and video frames.Fine-Grained Alignment Losses. We further analyse the impact of the frame-level and clip-level similarity losses in Table 6 . Implementing each loss individually offers noticeable improvements; however, integrating both simultaneously provides a substantial performance boost. Specifically, without these similarity losses, the model achieves lower scores across all metrics. With both losses applied, we observe a 5.29% increase in R@1 at IoU 0.5, a 3.3% rise at IoU 0.7, and a 2.88% improvement in mAP, highlighting their effectiveness in refining the model's capability to align video content with textual queries accurately.Global Text Anchor. We also explore various methods for generating the global text anchor, which is pivotal to our framework as QUERY: A teacher is writing on a white board. QUERY: Woman walks around town with purple mirrored sunglasses. it focuses attention on the relevant parts of the video corresponding to the text query. To determine the most effective approach, as detailed in Table 7 , we compared mean pooling, max pooling, weighted pooling, and the use of a transformer layer. We find that mean pooling outperforms other methods. This underscores the effectiveness of a simple yet powerful mean pooling strategy in capturing the holistic semantics of the text query for VTG.We show qualitative results in Fig. 3 compared with the baseline model, QD-DETR. By adopting a holistic approach to understanding text queries, our method consistently identifies moments that fully align with the intent of the textual queries. This approach allows our model to capture the essence of the entire query, preventing the oversight of integral query components such as 'a white board' and 'walks around', which QD-DETR sometimes overlooks. This underscores the advantage of our method's integration of global text semantics for more accurate video grounding. 
conclusion: 
 In this work, we present a novel approach to Video Temporal Grounding (VTG) that emphasizes holistic understanding of text queries and suppresses irrelevant visual frames. By integrating the entire query text into a global representation and employing visual frame-level gate mechanisms within a cross-modal interaction framework, our approach significantly enhances the alignment of text queries with accurate video segments. We demonstrate stateof-the-art performance on several VTG benchmarks, highlighting the importance of considering the entire text query and selectively focusing on relevant video.  
 
 Appendix: https://codalab.lisn.upsaclay.fr/competitions/6937 


Decision: This paper receives two borderline rejects, and one borderline accept initially. The reviewers raise some questions regarding novelty, hyper-parameter tuning, presentation, results on ActivityNet Caption dataset, the visualized results, computational cost, generalizability, etc. Most of these questions are addressed in the rebuttal and recognized by reviewers. Eventually with the merit of the work, this paper receives one borderline reject, one borderline accept, one borderline accept (initial score without final update). Considering the author response to the reviewers' questions, the AC recommended to accept. Authors are encouraged to revise the paper according to the reviews.

Reviews: review: paper_summary: paper_summary: The article introduces the goal of Video Temporal Grounding (VTG), showing the critical role of holistic text understanding in Video Temporal Grounding. To address this problem, the authors proposes two key contributions: introducing a visual frame-level gating mechanism and cross-modal alignment loss. These solutions help improve the model's performance in VTG tasks and emphasize the importance of holistic text understanding in guiding the model to focus on the semantically important parts within the video.
strength_weakness: strength_weakness: 1. The article introduces visual frame-level gating mechanisms and cross-modal alignment loss, providing new insights for addressing issues in Video Temporal Grounding (VTG) tasks.
2. The article emphasizes the importance of holistic text understanding in guiding the model to focus on the semantically important parts within the video, which helps improve model performance.
3. The introduction of visual frame-level gating mechanisms and cross-modal alignment loss provides new insights for addressing issues in Video Temporal Grounding (VTG) tasks.
4. Extensive experiments are conducted, showing the effectiveness of the proposed method.
limitations: limitations: 1. The paper introduces a cross-attention contribution that closely resembles existing work in the field, such as the cross-gated attended recurrent network described in previous studies [1,2], which also match natural sentences to video sequences for fine-grained interaction. Given these similarities, the claimed novelty of this idea is questionable.
2. While the cross-modal alignment loss is logically presented within the paper, its generality and applicability across different VTG models remain uncertain. The method appears to be highly sensitive to an array of hyper-parameters,  including different loss balancing parameters: $\lambda_{L1} = 10$, $\lambda_{iou} = 1$, $\lambda_{cls} = 4$, $\lambda_{frame} = 1$. Furthermore, the $\lambda_{clip}$ seems need to tuned in the TACoS dataset. More thorough discussion regarding the hyper-parameter tuning process is necessary to strengthen the paper's methodology.
3. The symbol $V_{L}^{'}$ in Formula 4 is not represented in the figure. The author should add appropriate labels, which can make the figure easier to understand.
4. The design of Figure 2 is overly complex, with numerous intersecting lines and unclear meanings, making the figure difficult to interpret.  Additionally, there is excessive white space that could be used more effectively to create a tighter, clearer presentation.  Simplifying or optimizing the diagram can significantly improve the clarity and effectiveness of the visual representation.

[1] Localizing Natural Language in Videos

[2] Fine-grained Iterative Attention Network for Temporal Language Localization in Videos
paper_summary: paper_summary: This paper proposes a new pipeline to deal with the video temporal grounding task. The authors consider that existing works usually neglect the holistic understanding of the query sentence and miss out on the global meaning. As a result, the paper introduces two strategies, frame-level gate mechanism and cross-model alignment loss to solve these problems. The results on three popular banchmarks show the effectiveness of the proposed strategies.
strength_weakness: strength_weakness: 1. The paper is well-written and easy to read. The author(s) introduce a practical pipeline to solve the video grounding task.
2. The results on QVHighlights, Charades-STA, and TACoS surpass the state-of-the-art models and show the effectiveness of the proposed method.
3. The author gives detailed ablation studies to verify the effect of each component of the framework.
limitations: limitations: 1. The proposed local and non-local attention gates somehow lack novelty. Some existing works also consider local and global text features like LGI, TRM[1]. The clip-level consistency loss is also similar to the loss proposed by PLPNet[2] and LGI. Comparisons with these works should be contained in the paper and many references are missing.
2. Could this method also work on ActivityNet Caption dataset? More experiments and comparisons should be conducted.
3. As shown in Fig.3, the visualized results show the attention score of each frame. However, for some of those frames that are not contained in the range of ground truth, the predicted scores are still high or nearly equal to the baseline model. It cannot reflect the effects of the proposed modules.

[1] Phrase-level Temporal Relationship Mining for Temporal Sentence Localization. 2023.
[2] Phrase-level Prediction for Video Temporal Localization. 2022.
paper_summary: paper_summary: This paper introduces a novel framework for Video Temporal Grounding that emphasizes holistic text understanding and selective attention to relevant video frames. The proposed method integrates global text semantics and frame-level gate mechanisms to enhance alignment between text queries and video segments, leading to improved accuracy in video grounding tasks. Qualitative results demonstrate the effectiveness of the approach in capturing text query essence and accurately identifying corresponding video moments.

The framework includes feature extraction, cross-modal interaction, fine-grained alignment loss, and prediction components, incorporating global text semantics and gated cross-attention mechanisms for enhanced video-text interaction. Fine-grained alignment losses are introduced to refine the alignment between video content and text queries, improving semantic correlation between the two modalities. The method outperforms existing models in Video Temporal Grounding benchmarks.
strength_weakness: strength_weakness: 1.The framework introduces a novel method that emphasizes holistic text understanding and selective attention to relevant video frames, showcasing originality in addressing video temporal grounding tasks.

2.By incorporating global text semantics and frame-level gate mechanisms, the model improves alignment between text queries and video segments, leading to more accurate video grounding results.

3.The method outperforms state-of-the-art models on multiple benchmarks, demonstrating its effectiveness in capturing the essence of text queries and accurately identifying corresponding video moments.

4.Leveraging text queries for improved video frame selection and alignment, the framework showcases promising results in moment retrieval and highlight detection tasks, indicating the practical benefits of considering text queries in video temporal grounding.
limitations: limitations: 1.The introduction of global text-anchor and fine-grained alignment loss functions may increase the complexity of the model, potentially leading to longer training times and higher computational costs.

2.While the method outperforms existing models on multiple benchmarks, its generalizability to diverse datasets and real-world applications may require further validation and testing.