Temporal Sentence Grounding with Relevance Feedback in Videos
Authors: Jianfeng Dong
Xiaoman Peng
Daizong Liu
Xiaoye Qu
Xun Yang
Cuizhu Bao
Meng Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Temporal Sentence Grounding
Cross-Modal Retrieval
Vision and Language
Primary area: machine_vision

Abstract: As a widely explored multi-modal task, Temporal Sentence Grounding in videos (TSG) endeavors to retrieve a specific video segment matched with a given query text from a video. The traditional paradigm for TSG generally assumes that relevant segments always exist within a given video. However, this assumption is restrictive and unrealistic in real-world applications where the existence of a query-related segment is uncertain, easily resulting in erroneous grounding. Motivated by the research gap and practical application, this paper introduces a new task, named Temporal Sentence Grounding with Relevance Feedback (TSG-RF) in videos, which accommodates the possibility that a video may or may not include a segment related to the query. This task entails localizing precise video segments that semantically align with the query text when such content is present, while delivering definitive feedback on the non-existence of related segments when absent. Moreover, we propose a novel Relation-aware Temporal Sentence Grounding (RaTSG) network for addressing this challenging task. This network first reformulates the TSG-RF task as a foreground-background detection problem by investigating whether the query-related semantics exist in both frame and video levels. Then, a multi-granularity relevance discriminator is exploited to produce precise video-query relevance feedback and a relation-aware segment grounding module is employed to selectively conduct the grounding process, dynamically adapting to the presence or absence of query-related segments in videos. To validate our RaTSG network, we reconstruct two popular TSG datasets, establishing a rigorous benchmark for TSG-RF. Experimental results demonstrate the effectiveness of our proposed RaTSG for the TSG-RF task. Our source code is available at https://github.com/HuiGuanLab/RaTSG.

Temporal Sentence Grounding with Relevance Feedback in Videos
Jianfeng Dong!
Xiaoman Peng'
Daizong Liu Xiaoye Qu
Xun Yang Cuizhu Bao
Meng Wang
'Zejjang Gongshang University
ZZeejaang Key Laboratory of Big Data and Future E-Commerce Technology
*Peking University "Huazhong University of Science and Technology
$ University of Science and Technology of China Heefii University of Technology
Abstract
As a widely explored multi-modal task, Temporal Sentence Grounding in videos (TSG) endeavors to retrieve a specific video segment matched with a given query text from video. The traditional paradigm for TSG generally assumes that rel- evant segments always exist within given video. However, this assumption is restrictive and unrealistic in real-world applications where the existence of a query-related segment is uncertain, easily resulting in erroneous grounding. Mo- tivated by the research gap and practical application, this paper introduces a new task, named Temporal Sentence Grounding with Relevance Feedback (TSG-RF) in videos, which accommodates the possibility that a video may or may not in- clude a segment related to the query. This task entails localizing precise video segments that semantically align with the query text when such content is present, while delivering definitive feedback the non-existence of related segments when absent. Moreover, we propose a novel Relation- aware Temporal Sentence Ground- ing (RaTSG) network for addressing this challenging task. This network first reformulates the TSG-RF task as a brrgrroudd-bcckrronn investigating whether the query-related semantics exist in both frame and video levels. Then, multi-granularte precise video-query relevance feedback and relation-aware segment grounding module is employed to selectively conduct the grounding process, dynamically adapting the presence absence of query-related segments in videos. To vali- date our RaTSG network, we reconstruct two popular TSG datasets, establishing a rigorous benchmark for TSG-RF. Experimental results demonstrate the effective- ness of our proposed RaTSG for the TSG-RF task. Our source code is available at https://githin com Hui Gww.Laa/aaaaaaaaaa
detection problem by
relevance discriminator is exploited to produce
Introduction
Grounding target content described by users in videos is fundamental capability that facilitates various multimedia applications, such as intelligent robotic service []] video on demand [2]. and metaverse (3] Following this demand, temporal sentence grounding in videos (TSG) has recently become a research hotspot, attracting wide attention from researchers in various fields (5 Despite the significant advancements made this field, the existing TSG task is overly idealistic. It aims to identify segments that are semantically relevant to given query from given long video, assuming that relevant segments always exist in the given video [10 However, this setting limits the applicability of TSG in real real-world scenarios as the given video may not contain the query-related contents, resulting mismatched or wrong grounding results. As illustrated in Figure [ a), given
"Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Query: A person opens door.
11.12
1940s
Query: A man is drinking water.
(a) Temporal Sentence Grounding (TSG) in videos
Query: A person door.
Relevance feedback: Has relevance
11.12
1940ss
Query: A man is drinking water. Relevance feedback: Lacks relevance
(b) Temporal Sentence Grounding with Relevance Feedback (TSG-RF) in videos
Figure 1: The difference between the TSG and TSG-RF tasks. TSG always predicts the start and end boundaries of the grounded segments, even in the absence of video content relevant to the given query text. By contrast, TSG-RF provides relevance feedback on whether exist query-related content in the given video, and selectively predicts the start and end boundaries of the grounded segments, according to the presence or absence of query-related segments in videos.
video and query, existing TSG methods always predict the start and end times segments, even when the video lacks content relevant to the given query text. This suggests a significant gap between the literature and the real world.
To fill the gap, this paper makes the first attempt to introduce an expanded but challenging task for TSG, named Temporal Sentence Grounding with Relevance Feedback (TSG-RF) in videos. This new task aims to deliver more flexible grounding results for a given query and a video. Specifically, as illustrated in Figure [b), if relevant segments are present in the video, the task is pinpoint the video segment that is semantically relevant to the given query description. Conversely, if no relevant segments exist, should be explicitly conveyed the user that "there are target segments related to the query in the given video, and thus no corresponding segments can be localized."
To tackle this new task, it is necessary first review the failure reason of traditional TSG methods [13 16]. We argue the essential issue of previous TSG methods their inability to discern whether grounding results warranted based the presence absence of relevant segments. To alleviate this issue, straightforward approach to address the TSG-RF task involves two-stage process: Firstly, a relevance discriminator is trained separately to and the video, providing relevant feedback. Then, query-video samples identified as relevant by the discriminator are processed using models designed for the TSG task to predict the target segment. Despite the structural simplicity of this integrated approach, it requires training two distinct models, leading to the redundant consumption of computational resources, increased storage demands, and suboptimal overall inference efficiency. Therefore, how to design an end-to-end relevance-aware TSG-RF method is an emerging problem.
the relevance between the query text
Based on the above considerations, we propose to address the TSG-RF task by developing multi- task learning model, i.e, the Relation-aware Temporal Sentence Grounding (RaTSG) network. This model promotes mutual enhancement between relevance discrimination and video segment grounding through shared knowledge, thus facilitating unified and efficient approach both tasks. As the essence of relevance discrimination lies in accurately measuring the semantic partial relevance between the query text and the video, we propose to first explore the measurement of partial semantic relevance between text and video at two different granularities: fine-grained and coarse-grained. Specifically, to capture fine-grained partial relevance, learning [1]. treating the entire video as bag with each frame as an instance. Following (20), video is considered partially related to the text if contains least one foreground frame; all frames are background and irrelevant, the video is deemed unrelated to the text. Considering the semantic incompleteness of individual video frames, coarse-grained perspective is also adopted to learn relevance between the global video-level semantics and the full query text. Then, multi-granularity relevance discriminator is designed to integrate the discriminative capabilities of both granularities to predict the final relevance score. As for the selective grounding process based on the relevance score,
adopt the concept of multiple instance

---

existing segment grounding heads always make grounding for any query-video input, even when no relevant segments are present in the video, which not suitable for the TSG-RF task. Therefore, we design a relation-aware segment grounding module that takes the previously predicted relevance information of each query-video pair as prior knowledge, and selectively conducts grounding.
We find a recent concurrent work [21] that shares a similar motivation with us. However, it simply treats the irrelevant video contents from an open-set perspective and implicitly utilizes cross-dataset evaluation. Instead, we implement original in-domain data to explicitly learn the potential irrelevant relationship. Further, existing TSG datasets can not be directly used to evaluate our proposed network for TS, since the datasets assume that a given query text can always localize a corresponding relevant segment from the given video. Hence, we further reconstruct two commonly used datasets, Charades- STA [2 and ActivityNet Captions [23], by including samples without grounding results where the video does not contain the content relevant to the query.
The main contributions of this paper are summarized as follows:
We formalize a novel TSG task to explore more flexible segment grounding in videos, i.e., Temporal Sentence Grounding with Relevance Feedback (TSG-RF) RF) in videos, which advances user-specified TSG towards more practical applications.
Targeting the TSG-RF task, we propose a novel Relation- aware Temporal Sentence Ground- ing (RaTSG) network, which mainly depends on a multi suulti-raaullrityy relevance discriminator and a relation-aware segment grounding module. The multi quati-granullrity relevance discrimi- nator is devised to predict the relevance feedback based on fine-graines and coarse- -arriin relevance between query text and videos, and the relation-awes aware segment grounding module selectively predicts the start and end boundaries of the grounded segment.
To facilitate the evaluation of the TSG-RF task, we reconstruct two commonly used TSG datasets and establish appropriate performance evaluation metrics to meet the setting of TSG-RF Extensive experiments conducted on these reconstructed datasets demonstrate the effectiveness of the proposed model.
2 Related Work
2.1 Temporal Sentence Grounding and Highlight Detection in Videos
The Temporal Sentence Grounding in Videos (TSG) task aims to retrieve video segments that match user-input natural language descriptions. Recent efforts for TSG can be typically grouped into proposal-based and proposal-free methods. Proposal-basee methods [22] 24|399 follow two-stage paradigm: generating proposals by dividing the video into clips and then aligning textual semantics with visual features. Proposal-free methods [40 10
13 45) offer an end-to-end paradigm, predicting target segments without proposals, enhancing computational efficiency. Although efficient, these methods struggled with segment-level feature capture. To combine strengths, Xiao et [45] used an anchor-free approach to generate candidates, then matched them with query statements using anchor-baa based methods. However, the above methods cannot be directly applied the TSG-RF task proposed in this paper. They will still predict the start and end times of segments and output retrieved segments, even in the absence of video content relevant to the query text.
Highlight detection task is similar to TSG, which mainly aims to identify the most interesting or important segments within a video based on given natural language query, focusing on segments that are salient or engaging [46]49]. Recent works typically tackle this task using multi-modal inputs, advanced transformers, large-scale pretraining techniques. For instance, Moment-DETR [66] introduce a transformer-baut based approach that simplifies highlight detection by treating it as a set prediction task, eliminating traditional proposal steps. QD-DETR [48] enhances video-text understanding with cross attennattetiion and negative pairs to improve relevance predictions. UniVTG (49 unifies diverse temporal annotations and enables large-scale pretraining to improve generalization across video grounding tasks. Notably, similar to TSG, highlight detection also assumes that every video contains highlights. In contrast, our proposed TSG-RF task not only grounds relevant segments but also accounts for cases where no relevant segments are present.
3

---

2.2 Cross-Modal Semantic Similarity Learning
This paper extends TSG by incorporating relevance feedback to discern the semantic connection between text and video, introducing techniques for learning cross-modal semantic similarity [50 53
The prevailing approach aligns text and video semantics by learning a shared space, using metric to assess similarity. For instance, Miech et al. (50) respectively represent text and video into feature vector, and compute their similarity via cosine similarity metric. Croitoru et al. [ use distillation techniques to combine knowledge from multiple pretrained models, enhancing cross-modal similarity learning. Current mainstream methods involve learning multiple shared spaces, with relevance computed as a weighted sum of similarities across these spaces, Li et al. [54] employe space multi-loss learning framework, enhancing cross-modal similarity by leveraging complementary encoders. While these techniques assess the whole text-video similarity, TSG-RF requires learning partial relevance, where the text relates to only some frames within the video. For non ron-reeriev samples, the text is irrelevant to any frame. Therefore, the focus on learning partial relevance.
3 The Proposed Method
3.1 Problem Definition and Overview
TSG-RF task. Given video V [v1, u....., Un] and query text Q [9t, ......, qm] as inputs, the TSG-RF task is asked to predict relevance feedback on whether exist query-related content the given video. Besides, if the query has relevance with the video, it is asked to conduct grounding to identify the precise boundary indices of video segments that are semantically consistent with the query text in the video V. Otherwise, the grounding results ignored. Compared to the traditional TSG task that always predicts grounding results for all videos, this TSG-RF task is more challenging as it requires only distinguishing the videos containing and not containing query-relevant segments but also generating explicit non-existence signals for non-relevant videos besides the grounding results.
Overall framework. Figure 2 illlustrates the framework of our proposed RaTSG network for the TSG-RF task. Given an input video V and a query text Q encoded by the attention-based mechanisms, the multi-granularity relevance discriminator first captures fine-grat fine-grained and coarse-grained semantic- aware relevance between the video and query text at the frame and segment levels. Subsequently. integrates the relevance determination capabilities of these discrimination score Pe for distinguishing whether the video has a semantic-relate content of the query. Then, relation-aware segment grounding module takes the multimodal enhanced video features as the basic representation for generating the grounding probability distribution P, of the start boundary and Pa of the end boundary, while exploiting relevance feedhack signal as additional input to produce the separate final results of irrelevant and relevant segments. In the following sections, we will provide detailed illustration of each component.
granularities generate the relevance
3.2
Preprocessing of Text and Video Features
Given an input video and a query text, we first use a pre-trained video feature extractor, such as 13D 155) to extract the video feature sequence V = [1i, ; Un] E Raxd, where d, is the feature dimension. Subsequently, the query text features are extracted using text feature extractor, like GloVe (56), producing word embeddings Q = [/1, -2,..., Im]] E mmd, where d, represents the feature dimension. Both the video feature sequence V and the query text embeddings Q are first mapped into the same dimension d through a fully connected layer respectively, resulting in V nxd and Q m To enhance the video semantic with the query contexts, we utilize an attention mechanism to integrate word-wise query features Q with the frame-wise video feature V for generating text- eext-guided enhanced video feature Vg [13. To comprehend the whole semantic of the query for latter global-level reasoning, we use a self-attention layer to perform the intra-modal information interaction on the text feature sequences Q to obtain the sentence-level query representation lig The final contextual multi-modal representation V can be obtained by concatenating the frame-wise video feature V, with sentence-level query feature ly. More details can be found in Appendix A.I

---

Relation-aware Segment Grounding
P. and P. different scenarios
Video Encodie
EnhanesTEnt
Text Ted..Guide Garikeed
Relation pommm Predude
T There are no mlevant segment in video
Relevare Sijestion
Pe
Video
signood
scorer
here are relevant: segment in video
PC
0.8
Pe
Frame-level relevance discriminntoo
Weldled
Quey Test: parooni wilke i dounver chiadio SUEE cotiee
Text Eroodr
Salf- - Atrantion
0.6 ScOTe
M Maximum
Video- relevance disuriminator
A verage
Relevance
Discriminator:
Concatenate
Figure 2: The overall framework of our proposed RaTSG for addressing TSG-RF task. It mainly de- pends on a multi-granularity relevance discriminator that is employed to learn query-video relevance, and relation-aware segment grounding module that is used to selectively perform grounding.
3.3 MuatilGrryyyrrity Relevance Discriminator
Before generating the grounding results for each query-video pair, we need to first investigate the semantic relevance between them. To determine whether a given video contains segments relevant to query text, the key lies in assessing the existence of relevance between the text semantic and the video content. Here, we delve into capturing the multi-grained query-video relevance by matching the global sentence-level textual contexts with frame-level and video-level video semantics, respectively.
Frame-level relevance discriminator, Determining frame-level query-video relevance can be treated as a foreground-backe backgroondd frame discrimination problem. Theoretically, for video containing query-related content, the foreground frames located within the start and end index boundaries should exhibit strong relevance to the query text. Conversely, for a video lacking relevant segments, no foreground frames in the video should relate to the text. To detect foreground frames from the query text, we design a foreground frame detector that finely learns the similarity between the text and each video frame. Specifically, we feed the fused multi-moda feature sequence V into a fully connected layer to predict sequence of query-to-frame similarity scores. These scores are then mapped to a range between and using the sigmoid function, resulting in the foreground frame prediction R" where each score indicates the confidence of frame
score sequence S sigmoid(FC(V")) belonging to the foreground. To train the frame-level relevance discriminator, a binary cross-entropy loss frame (Appendix A.2) is employed each frame, which encourages the discriminator to output large scores for foreground frames while small scores for background frames.
To obtain the relevance between text and video at a line-grained frame-level, we apply the concept of multiple instance learning 17.1. To be specific, the video is treated as bag with video frames as instances, categorized into foreground (positive) and background (negative) frames. Thus, a video containing at least one foreground frame is considered a positive bag, indicating partial relevance to the query text. In contrast, video composed entirely of background frames is deemed negative bag, showing no relevance. Following [20]. we utilize a max- moxx-pooling operator to transfer the instance scores to bag scores, obtaining the fine-grained query-video discrimination score as scorefs = max(S Sy). The low fine-grained discrimination score suggests that there are no foreground frames in the video, while high score indicates the presence of at least one foreground frame.
Video-level relevance discriminator. In addition to capturing the frame-level relevance, we also attempt to perceive the semantic relevance from a more informationally complete video level. To generate the comprehensive video-level visual representation, we need to aggregate all query-related frame-wise features into a global one to determine its semantic similarity with the sentence contexts. Specifically, we apply frame-wise weighted sum operation on the text-guided enhanced video feature V, to obtain the query-related video representation hy Rd where we assign weights to
5

---

each frame by normalizing the previously obtained frame-level similarity scores (reflect the relative relevance score between the text and each frame) using the softmax function. Then, we fuse the sentence-level textual feature li with the global video feature h2 to obtain the video-level relation signal vector g. which then input into a fully connected layer with sigmoid function to obtain the final video-level similarity score ecore sigmoid(FC(g)), which globally describes the relevance between the text and the entire video. To train the video-level relevance discriminator, a binary cross- entropy loss Luideo (Appendix (A.3) ) is employed on each query-video sample, which encourages the discriminator output large scores for samples with grounding results while small scores for that without grounding results.
Multi-level relevance prediction. The previously discussed two relevance discriminators assess the relevance between query and video at the frame and video levels respectively, yielding fine- ane-grained relevance score (scorefgl and coarse-grained relevance score (scoreeg). Considering that both granularity levels are deemed equally important for determining the relevance between text and video, we calculate the average of both fine-grained and coarse saarse-grained relevance scores as the final multi-granularity relevance score P. as:
P. (scorefs + scoreeg)/2 /2.
(1)
3.4 Relation-Aware Segment Grounding
After obtaining the query-video relevance score, we aim to design flexible segment grounding module selectively predict the start and end boundaries of the target segments, according to the presence or absence of query-related segments in videos. However, existing segment grounding heads designed for TSG tasks B T3
57 16 are not suitable for the TSG-RF task. They always generate high-confidence start/end boundaries of segments even when there are relevant segments the video. To this end, we introduce a relation-aware segment grounding module that incorporates previously obtained relevance feedback during the segment grounding process, which accommodates the possibility that there may be no relevant segments within the video.
Specifically, to address situations where the video contains no relevant segments, we prepend special token the general video feature sequence, representing index 0. This token serves both the starting and ending boundary indices for samples that do not have grounding results, indicating that the query text does not align with any start and end boundaries in the given video. Therefore, it is essential to introduce contextual relevance- to ensure that the segment predictor can effectively handle the absence of relevant segments the video. Considering the video-level relation signal vector g obtained from the video-level relevance discriminator contains enough relevance-aware knowledge to determine whether the segment exists or not, we utilize the video-level relation signal vector g as the special token in this segment predictor. Consequently, the range of boundary indices for relevant segment grounding is [1, n].. Thus, as shown in Figure 2 if sample is determined to lack relevant segments, the ground-truth boundary index is assigned as A [0, O]: if a sample is deemed to have grounding results, the ground-truth boundary index is defined as A [a", a²), where S a" a <
representation as the special feature token
To predict the probability distributions of start and end boundaries, following 13 unidirectional LSTMs with two feed-forward layers. The boundary prediction loss function L boundary represented as:
we use two
Σ Y, log(P.) + Σ Ya log(P.)),
Lboundary
2
where Y, and Ye are the one-hot vector representations of the ground-truth start (a,) and end (a.)) boundary indices, respectively. Ps and P. indicate the predicted start and end boundary probability distribution, respectively.
3.5 Training and Inference
In summary, the total training loss be defined
Liotal = Lboundory + 3L + Leideos
(3)
where 3 and y are hyperparameters used to balance the importance of the three losses.

---

During the inference phase, the multi granularity relevance score P. is first compared with a threshold m 0.5 determine whether query-related content exists or not the given video:
Has grounding result, if P. 2 m
No grounding result, if Ps m
For samples judged to have grounding results, the model predicts the start (ū.)) and end (á²) boundary indices of the target segment. Specifically, we first compute the joint probability distribution according to the start and the end boundary probability distributions P.. Pe generated by the Alltton-awarr
segment grounding module. The predicted start (ā₂) and end (ā₂) boundary indices are obtained by maximizing the joint probability:
(ās: (e) arg max (P"x(P Po), Bg
0 S 0g S âe n.
Evaluation
4.1 Experimental Setup
Datasets. To verify the viability of our proposed model for TSG-RF, samples without grounding results where the video does not contain the content relevant to the query are required. Therefore, we reconstruct two commonly used datasets, Charades-STA 22 and ActivityNet Captions [23], by adding samples without grounding results (Details of how we obtain them are illustrated in the Appendix A.5) ). For each sample in both validation and test sets, we add a corresponding sample
without grounding result, resulting in 1:1 ratio of samples with and without grounding results.
The Charades-STA 22 dataset comprises 6,672 videos, where the training set contains 12,408 video-text sample pairs, while the test comprises 3,720 pairs. Since the original Charades-STA does not have a validation set, we randomly halve the original test samples to form a validation and test set. The ActivityNet Captions (33 dataset consists of approximately 20,000 videos featuring diverse and open-content videos. Consistent with previous work [43 for the TSG task, the training set includes 37,421 video-text pairs, while the validation and test sets contain 17,505 and 17,031 samples, respectively. After reconstruction, the validation and test sets were augmented with an equal number of samples without grounding results, doubling the total number of sample pairs to 35,010 and 34,062, respectively.
For ease of reference, name the corresponding reconstructed datasets as Charades-RF and ActivityNer-RF, respectively.
Performance metric. As our proposed TSG-RF task requires models to provide relevance feedback indicating samples with or without grounding results, we use accuracy ("Acc") to measure the ability of relevance feedback. For measuring the grounding ability, referring to the previous TSG works [13 [3], we utilize the "R(n) @ (m)" and "mloU". [n] @ (m]" indicates the percentage of query texts in the top-n segments where at least one instance has an Intersection over Union (IoU) greater than m. "mloU" represents the mean IoU across all test samples. is worth noting that due to the inclusion of samples with no grounding results, Union (loU) in four specific scenarios: (1) When the sample is predicted to have no grounding results but actually has grounding results, the IoU is set to O. (2) When both the prediction and the ground truth indicate that the sample has no grounding results, the loU is set to 1. (3) When the sample is predicted to have grounding results but actually has grounding results, the IoU is set to 0. (4) When both the prediction and the ground truth indicate that the sample has grounding results, the IoU calculated as the Intersection over Union between the predicted segment and the ground truth.
redefine the calculation of Intersection over
Implementation details. The implementation details are presented in Appendix A.6
4.2 Comparison with Baseline Methods
As models specifically designed for TSG-RF are non non-existing, we compare our proposed model with models targeted at conventional TSG, and adapt them to TSG-RF by adding an extra relevance discriminator for relevance feedback. Specifically, we select six traditional TSG models, including
VSI. [3), SeqPAN [4), EAMAT [5). ADPN [16], UniVTG [49, QD-DETR 48] considering their source code released thus ensuring fair and replicable comparisons. For the relevance
7

---

Table 1:
Performance comparison on Charades-RF and ActivityNet-RF dataset. Model denotes that the baseline model adapted to TRF-RF by utilizing an additional trained relevance discriminator, using two cascaded models: a relevance discrimination model and a segment grounding model. Our proposed RaTSG, unified model for both relevance discrimination and segment grounding, achieves the best performance with very lightweight model for TRF-RF.
Charades-RF
AetivityNet-RF
Method
Params (M)
R1
RI@0.
R1@0.7
mloU
Asz
R1@0.3
RI@0.5
R
VSLNet
50.00
33.74
27.31
17.72
24.69
50.00
31.06
21.88
12.82
22.27
1.16
VniVIG
50.00
35.81
30.03
16.67
24.96
50.00
30.89
21.67
11.29
21.35
41.35
QD-DETR
50.00
35.16
29.46
19.27
25.31
50.00
26.50
19.15
11.07
18.99
7.07
ADPN
19.87
25.98
50.00
30.72
20.74
22.05
SoqPAN
50,00
35.35
29.57
20.51
26.14
50.00
31.85
22.65
13.34
22.86
1.19
EAMAT
50,00
37.12
30.59
20.86
27.27
50,00
31.10
20.80
12.07
22.07
94.12
VSLNer++
71.94
56.77
49.65
54.67
81.60
66.15
58.37
50.64
5.34
UniVTG+
71.94
62.58
58.55
48.79
54.65
81.60
66.15
58.36
49.46
58.00
45.53
QD-DETR
71.94
62.18
58:20
50.96
55.13
81.60
62.43
56.13
49.27
55.97
11.25
*
71.94
62.26
57.23
51.16
55.41
81.60
65.85
57.41
50.28
58.47
6.45
SoqPAN++
71.94
62.12
58.01
51.61
55.49
81.60
66.77
58.98
51.11
59.11
5.37
++MAT*
71.94
63.55
59.17
51.96
56.23
81.60
66.
57.36
49,93
58.45
98.30
RaTSG (ours)
76.85
68.17
61.91
$4.19
59.93
84.27
69.02
60.68
$2.88
61.15
1.27
discriminator, we utilize a binary classification model which is trained separately to assess the relevance between the query text and the video (For the details of the relevance discriminator, please refer to Appendix B.I). Then, query-video samples identified relevant by the discriminator are processed using models designed for the TSG task to predict the target segment. For ease of reference, we denote the corresponding models associated with an extra relevance discriminator as VSLNet+*, UniVTG and QD-DETR respectively.
SeqPAN++ EAMAT** ADPN+
Table 1 summarizes the performance comparison and model parameters on the Charades-RF and ActivityNer-RF dataset. is worth noting that traditional TSG models lack the ability to discriminate relevance. These models assume all samples have grounding results, making them incapable of correctly handling samples without grounding results. Hence, this often leads to mismatched grounding prediction, resulting in low recall and mIoU performance for TSG-RF. Additionally, since the test set has an equal ratio (1:1) of samples with and without grounding results, the relevance prediction accuracy of these models is only 50%. Additionally, the enhanced versions of these baseline models, namely VSLNer++, SeqPAN+ EAMAT+ ADPN++ UniVTG and QD- DETR ++ include a relevance diseriminator, which results in consistent performance gains compared to their counterparts without the discriminator. However, these enhanced models require separate and independent training of the relevance discriminator and video grounding components, leading to redundant use of computational resources and increased model size. In contrast, our proposed RaTSG model provides more lightweight and comprehensive solution by integrating the discrimination and grounding modules, achieving the best performance.
4.3 Ablation Studies
4.3.1
Effectiveness of the Multi-Granularity Relevance Diseriminator
To validate the effectiveness of the multi-granulane relevance discriminator, we compare it with degraded models that use only coarse-grained or fine-grained discrimination scores. As shown in Table single-granularity models fail to perceive the various degrees of partial relevance between text and video, therefore performing worse than the multi-granularigg one.
4.3.2 Effectivvness of the Relation-aware Segment Grounding
To assess the viability of the relation-awan segment grounding module, we conduct ablations on by replacing the relation signal vector g with randomly initialized one. As shown in Table 3 the relation-aware segment grounding consistently outperforms the random counterpart. The result demonstrates the benefit of using previously predicted relevance information of each query-video pair as prior knowledge for TSG-RF. Additionally, in order to further explore how relation features are
8

---

Table 2:
The effectiveness of the multi- granularity discriminator on Charades-RF.
Granularity
Acc
R1@0.5
@0.7
mloU
V 75.35
67.34
60.91
53.84
59.27
x
75.73
67.63
60.48
53.25
59.18
76.85
68.17
61.91
54.19
59.93
Table 3: The effectiveness the relation- segment grounding on Charades-RF.
Figure 3: t-SNE visualization of relation sig- nal vectors
test samples on Charades- RF, where red dots represent relevant samples and blue points represent irrelevant samples of queries and videos.
Relation-aware
Acc
010
R160.5
R1@007
mloU
76.40
66.18
59.62
51.96
57.82
76.85
68.17
61.91
54.19
59.93
Table 4: The performance comparison of rel- evance discrimination with and without the segment grounding module on Charades-RLl
Table 5: The performance comparison of seg- ment grounding with and without the rele- vance discriminator on Charades-STi STA.
Segment Grounding
Acc
Discriminator @00.3
R1@0.5
R1@0.77
mloU
75.59
67.47
54.62
35.43
49.37
76.85
74.19
56.61
37.47
53,02
Table 6: Our proposed multi- granularitu relevance discriminator and relation-awarg segment ground- ing enhance can jointly used to enhance traditional TSG methods for TSG-RF tasks.
Method
Acc
RI
R1@0.5
R160.7
mloU
EAMAT
50,00
37.12
30.59
20.86
27.27
EAMAT
71.94
63,55
59.17
51.96
56.23
EAMAT+Ours
76.37
67.47
62.02
54.83
59.58
learned, visualize the relation signal feature g by t-SNE 58 in Figure 3 It is demonstrated that dots with the same color are relatively more clustered than those with different colors. Overall, the result shows the good discrimination ability of the learned relation feature in our model for relevant and irrelevant samples.
4.3.3 Mutual Enhancement between Relevance Discrimination and Segment Grounding
Our proposed RaTSG is unified model for both relevance discrimination and segment grounding tasks, trained in multi-task learning manner. To explore whether the two tasks mutually enhance each other, evaluate the counterparts with the corresponding segment grounding module or relevance discriminator removed. As shown in Table removing the segment grounding module influences the performance of relevance discrimination. Similarly, shown in Table removing the relevance discriminator degrades the performance of segment grounding. The results allow us to conclude that the relevance discriminator and the segment grounding in our model mutually enhance each other, which also demonstrates the effectiveness of our unified dual-branch framework design.
4.3.4 Enhancing traditional TSG methods to work for TSG-RF tasks.
To further investigate the scalability of our proposed gaaliiigranularity relevance discriminator and relation-awe -awore segment grounding modules, we also conduct experiments to explore whether they can be adapted for traditional TSG methods enable them work for the TSG-RF task. Specifically, we integrate these components into an existing traditional TSG model, EAMAT, for comparison. As shown in T EAMAT with our devised multi- granularttyl relevance discriminator and relation- aware segment grounding outperforms the original EAMAT and the enhanced EAMAT+ with an extra relevance discriminator. These results demonstrate the adaptability and effectiveness of our proposed modules for enhancing traditional TSG methods to work for the TSG-RF task.

---

4.4 Analysis of Grounding Examples
4 visualizes grounding examples obtained by our proposed RaTSG and VSL.et** The results show that our RaTSG effectively handles samples with and without grounding results. For the first sample with grounding results, RaTSG localizes video segments more accurately than VSLNet** Furthermore, foreground frame prediction scores obtained by RaTSG are more reasonable. We attribute it to the fact that RaTSG is trained using samples without grounding results, allowing the model to learn the similarity characteristics between background frames and the text, thus enhancing its ability distinguish foreground and background frames in videos.
For the second sample with- out grounding results in Figure RaTSG consistently predicts low foreground frame scores, providing accurate relevance feedback. Besides, the relation- aware segment grounding mod- ule assigns high probability of the special index of O, in- dicating no grounding result. In contrast, VSINet++ pro- vides incorrect relevance feed- back for this sample, showing high foreground frame predic- tion scores and resulting in in- correct grounding segments.
Text: The put the pillow down.
Ground Truth
1.25
10.As (nith grounding reulis)
Relevance Bedlback: Hasrolerance
Ground.
kupntabblly castade and bounlay
Truth
Relevance Boobback Has relevance
Ground
Kiriichl dosistee sau beadeyy dostadee tianqrra poda wooo
Truth
Query Text: man is sittigg on chair.
Ground ratth
There is no relevant vidro segment twithout groanding restts
Relevance Bedtback
rackarn relevance
Lacks
Kipripbliy demis and leanbey stanngrae podactss soceed
In Figure we illustrate two bad examples to discuss the limitations of our model. In model in-
Aibriially Gostadee LSunAo
the first example, correctly judges the relevance feedback due the lack of au- dio cues which are crucial for identifying the action of sneez- ing. In the second example, our model misinterprets the temporal sequence of actions, mistaking the closing action for the opening action. These examples demonstrate that our proposed model struggles with handling audio-related actions and temporal-sensitive content. However, such limitation can be alleviated by integrating au- dio features and temporal mod- eling in the video representa- tion moudel.
Figure 4: Visualization of grounding examples obtained by our proposed RaTSG and the VSLNet baseline.
Ground Tuuth
L.0g
137T
PrPeeer yallder
prikilt
Ground Truth
roleuant signest
Queey Tial: A peseo opantng Taptop
segioott
Figure 5: Visualization of two bad examples obtained by our pro- posed RaTSG.
5 Conclusions
This paper breaks through the bottleneck of existing TSG works that can not handle the cases where query-related segments do exist, by introducing more realistic and natural task, i.e., TSG-RF. By incorporating the multi-grann multi-granularity relevance discriminator into the segment grounding model with further mutual enhancement designs, our proposed method can effectively and efficiently localize precise video segments that closely match the query text for videos containing relevant content, while providing clear feedback indicating the absence of related segments for videos not containing relevant content. Besides, two constructed datasets for TSG-RF are contributed.
10

---

Acknowledgements
This work was supported by the Pioneer and Leading Goose R&D Program of Zhejiang (No. 2024C01 10, No. 2023C01212), Young Elite Scientists Sponsorship Program by China Association for Science and Technology (No. 2022QNRC001), Zhejiang Provincial Natural Science Foundation (No. LZ23F020004), Fundamental Research Funds for the Provincial Universities of Zhejiang (No. FR2402ZD) and the Graduate Research Innovation Fund Project of Zhejiang Gongshang University.
References
[1] Dionisis Andronas, George Apostolopoulos, Nikos Fourtakas, and Sotiris Makris. multi-modal interfaces for natural human-robot interaction. Procedia Manufacturing, 54:197-202, 2021.
[2] Xue Song, Baohan Xu, and Yu-Gang Jiang. Predicting content similarity via multimodal modeling for video-in-videe advertising. IEEE Transactions on Circuits and Systems for Video Technology, (1); 569-581, 2020.
[3] Gareth W Young and Oliver Dawkins. Multimodality: Exploring sensibility ennse-making beyond the metaverse. International Conference on Hlmann-Computer Interaction, pages 307-322. Springer, 2023.
[4] Xiaohan Lan, Yitian Yuan, Xin Wang, Zhi Wang. and Wenwu Zhu. A survey temporal sentence grounding videos. ACM Transactions on Mullimedia Computing, Communications and Applications, 19 -33, 2023.
[5] Zijian Zhang, Zhou Zhao, Zhu Zhang, Zhijie Lin, Qi Wang. and Richang Hong. Temporal textual localization in video via adversarial bi-directional interaction networks. IEEE Transactions on Multimedia. 23:3306-3317, 2020.
[6] Xiaoye Qu, Pengwei Tang, Zhikang Zou, Yu Cheng, Jianfeng Dong, Pan Zhou, and Zichuan Xu, Fine- grained iterative attention network for temporal language localization videos. Proceddings of the 28th ACM International Conference on Multimedia, pages 4280-4288, 2020.
[7) Junyu Gao and Changsheng Xu. Fast video moment retrieval. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1523 15322, 2021.
[8] Dan Guo, Kun Li, Bin Hu, Yan Zhang, and Meng Wang. Benchmarking micro-action recognition: Dataset, method, and application. IEEE Transactions on Circuits and Systems for Video Technology, 34(7y: 6238-6252, 2024.
[9] Qi Zheng, Jianfeng Dong, Xiaoye Qu, Xun Yang, Yabing Wang, Pan Zhou, Baolong Liu, and Xun Wang. Progressive localization networks for language-based moment localization. ACM Transactions on Multimedia Computing, Communications and Applications, 19(2):1-21, 2023.
[10] Soham Ghosh, Anuva Agarwal, Zarana Parekh, and Alexander Hauptmann. Exel: Extractive clip localiza- tion using natural language descriptions. arXiv preprint 194.0,,,
[11] Xin Sun, Xuan Wang, Jialin Gao, Qiong Liu, and Xi Zhou. You need to read again: guani-yra perception network for moment retrieval in videos. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1022-1032, 2022.
[12] Zhe Xu, Da Chen, Kun Wei, Cheng Deng, and Hui Xue. Hisa: Hicrarchically semantic associating for video temporal grounding. IEEE Transactions on Image Processing, 31:5178- 31:5118-5188, 2022.
[13] Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou. Span-based localizing network for natural language video localization. arXiv preprint arXiv:2004.139) 2020.
[14] Hao Zhang, Aixin Sun, Wei Jing, Liangli Zhen, Joey Tianyi Zhou, and Rick Siow Mong Goh. Parallel attention network with sequence matching for video grounding. arXiv preprint arXiv:2105.0848)) 2021.
[15] Shuo Yang and Xinxiao Wu. Entity-aware and motion-aware transformers for language-driven action localization in videos. arXiv preprint arXiv:2205.052 055544 2022.
[16] Houlun Chen, Xin Wang, Xiaohan Lan, Hong Chen, Xuguang Duan, Jia Jia, and Wenwu Zhu. Curriculum- listener: Consistency-and complementarity-wwal audio-enhanced temporal sentence grounding. In Proceedings of the 31st ACM International Conference on Multimedia, pages 3117-3128, 2023.
[17] Thomas G Dietterich, Richard H Lathrop, and Tomás Lozano-Péree Solving the multiple instance problem with axis-paralla rectangles. Artificial intelligence, 89(1-2):31-71 1997.
11

---

[18] Oded Maron and Tomás Lozano-Pérez A framework for multiple-instance learning, Advances in neural information processing systems, 10, 1997.
[19] Zhi-Hua Zhou and Min-Ling Zhang. Neural networks for multi-instance learning. In Proceedings of the International Conference on Intelligent Information Technology, Beijing, China, pages 455-459. Citeseer, 2002.
[20] Jianfeng Dong, Xianke Chen, Minsong Zhang, Xun Yang, Shujic Chen, Xirong Li, and Xun Wang. Partially relevant video retrieval. In Proceedings of the 30th ACM International Conference on Maltimedia, pages 246-257, 2022.
[21] Xiang Fang, Wanlong Fang, Daizong Liu, Xiaoye Qu, Jianfeng Dong, Pan Zhou, Renfu Li, Zichuan Xu, Lixing Chen, Panpan Zheng, et al. Not all imputs are valid: Towards open-set video moment retrieval using language. In Proceedings of the 32nd ACM International Conference on Multimedia, 2024.
[22] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In Proceedings of the IELE international conference on computer vision, pages 5267-5275, 2017.
[23] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In Proceedings of the IEEE international conference on computer viston, pages 706-715, 2017.
[24] Lisa Anne Hendricks, Oliver Wang. Eli Shechtman, Josef Sivie, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In Proceedings of the IEEE international conference on computer vision, pages 5803-5812, 2017.
[25] Meng Liu, Xiang Wang, Liqiang Nie, Xiangnan He, Baoquan Chen, and Tat-Seng Chua. Attentive moment retrieval in videos. In The s/ international ACM SIGIR conference research & development in information retrieval, pages 15-24, 2018.
[26] Huijuan Xu, Kun He, Bryan Plummer, Leonid Sigal, Stan Sclaroff, and Kate Saenko. Multilevel language and vision integration for text-to-clip retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9062-9069, 2019.
[27] Shaoxiang Chen and Yu Yu-Gang Jiang, Semantic proposal for activity localization in videos via sentence query. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8199-8206. 2019.
[28] Daizong Liu, Xiaoye Qu, Jianfeng Dong, Pan Zhou, Yu Cheng, Wei Wei, Zichuan Xu, and Yulai Xic. Context-aware biaffine localizing network for temporal sentence grounding. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 1235-11244, 2021.
[29] Daizong Liu, Xiaoye Qu. Xiao Xiao-Yang Liu, Jianfeng Dong, Pan Zhou, and Zichuan Xu. Jointly cross-and self-modal graph attention network for query-based moment localization. In Proceedings of the 28th ACM International Conference on Multimedia, pages 4070-4078, 2020.
[30] Daizong Liu, Xi Ouyang, Shuangjie Xu, Pan Zhou, Kun Hc, and Shiping Wen. Saanet: Siamese action-umits attention network for improving dynamic facial expression recognition. Neurocomputing. 413:145-157, 2020.
[31] Daizong Liu, Xiaoye Qu, Jianfeng Dong, and Pan Zhou Adaptive proposal generation network for temporal sentence localization in videos. arXiv preprint arXiv: 209:::2:09.0000 109.00099..... 2021.
[32] Daizong Liu, Xiaoye Qu. Jianfeng Dong, and Pan Zhou. Reasoning step-by-step: Temporal sentence localization in vidcos via deep rectification modulation network. In Proceedings of the 28th International Conference on Computational Linguistics, pages
rectficationmmodulatiion
2020.
[33] Daizong Liu, Shuangjie Xu, Xiao Xiao-Yang Liu, Zichuan Xu, Wei Wei, and Pan Zhou. Spatiotemporal graph neural network based mask reconstruction for video object segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2100-2108, 2021.
[34] Daizong Liu, Xiaoye Qu, Jianfeng Dong, Pan Zhou, Zichuan Xu, Haozhao Wang. Xing Di, Weining Lu, and Yu Cheng. Transform-cquie equivariant consistency learning for temporal sentence grounding ACM Transactions on Multimedia Compuring, Communications and Applications, 20(4):1-19, 2024.
cransform--quiiariaan
[35] Daizong Liu, Xiaoye Qu. Jianfeng Dong. Guoshun Nan, Pan Zhou, Zichuan Xu, Lixing Chen, He Yan, and Yu Cheng. Filling the information gap between video and query for language lanuuage-driven moment retrieval. In Proceedings of the 31st ACM International Conference on Multimedia, pages 1190-4199) 2023.
12

---

[36] Xiang Fang, Zeyu Xiong, Wanlong Fang, Xiaoye Qu, Chen Chen, Jianfeng Dong, Keke Tang, Pan Zhou, Yu Cheng, and Daizong Liu. Rethinking weakly-supervised video temporal grounding from game perspective. In European Conference on Computer Vision. Springer. 2024.
[37] Daizong Liu, Xiaoye Qu, Xiang Fang, Jianfeng Dong, Pan Zhou, Guoshun Nan, Keke Tang, Wanlong Fang. and Yu Cheng. Towards robust temporal activity localization learning with noisy labels. In Proceedings of the 2024 Joint International Conference Computational Linguistics, Language Resources and Evaluation (IREC-COLING 2024), pages 16630-16642, 2024.
[38] Daizong Liu, Xiang Fang, Xiaoye Qu, Jianfeng Dong, He Yan, Yang Yang. Pan Zhou, and Yu Cheng. Unsupervised domain adaptative temporal sentence localization with mutual information maximization. In Proceedings of the AAAI Conference Artifical Intelligence, volume 38, pages 3567-3575, 2024.
[39] Zeyu Xiong, Daizong Liu, Xiang Fang, Xiaoye Qu, Jianfeng Dong, Jiahao Zhu, Keke Tang, and Pan Zhou. Rethinking video sentence grounding from tracking perspective with memory network and masked attention. IEEE Transactions on Multimedia, 2024.
[40] Jingyuan Chen, Xinpeng Chen, Lin Ma, Zequn Jie, and Tat-Seng Chua. Temporally grounding natural sentence in video. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 162-171, 2018.
[41] Jingwen Wang, Lin Ma, and Wenhao Jiang. Temporally grounding language queries in vidcos by contextual boundary-aware prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12168-12175, 2020.
[42] Yitian Yuan, Lin Ma, Jingwen Wang. Wei Liu, and Wenwu Zhu. Semantic conditioned dynamic modulation for temporal sentence grounding in videos. Advances in Neural Information Processing Systems, 32, 2019.
[43] Songyang Zhang. Houwen Peng, Jianlong Fu, and Jiebo Luo. Learning 2d temporal adjacent networks for moment localization with natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12870-12877, 2020.
[44] Yitian Yuan, Tao Mei, and Wenwu Zhu. To find where you talk: Temporal sentence localization in video with attention based location regression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9159-9166, 2019.
[45] Shaoning Xiao, Long Chen, Songyang Zhang, Wei Ji. Jian Shao, Lu Ye, and Jun Xiao. Boundary proposal network for two-stage natural language video localization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2986-2994, 2021.
[46] Jie Lei, Tamara L Berg, and Mohit Bansal. Detecting moments and highlights in vidcos via natural language queries. Advances in Neural Information Processing Systems, 34:1 11111184-11858 2021.
[47] Ye Liu, Siyuan Li, Yang Wu, Chang- Wen Chen, Ying Shan, and Xiaohu Qie. Umt: Unified multi-modal transformers for joint video moment retrieval and highlight detection. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3042-303 3051, 2022.
[48] WonJun Moon, Sangeek Hyun, SangUL Uk Park, Dongchan Park, and Jae-Pil Heo. Query-dependent video representation for moment retrieval and highlight detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13023-23033, 2023.
[49] Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shraman Pramanick, Difei Gao, Alex Jinpeng Wang, Rui Yan. and Mike Zheng Shou. Univtg: Towards unified lideo-language temporal grounding. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 2794-2804, 2023.
[50] Antoine Miech, Dimitri Zhukov, Jean- lean-Baptiste Alayrac, Makarand Tapaswi, Ivan I Laptev, and Josef Sivie. Nowto100ms Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEÉE/CVF international conference on computer vision, pages 2630-2640, 2019.
[51] Jianfeng Dong, Xirong Li, Chaoxi Xu, Xun Yang, Gang Yang, Xun Wang, and Meng Wang. Dual encoding for video retrieval by text. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(8): 4065-4080, 2021.
[52] loana Croitoru, Simion- Vlad Bogolin, Marius Leordeanu, Hailin Jin, Andrew Zisserman, Samuel Albanie, and Yang Liu. Teachtext: Crossmodal generalized distillation for text-video retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11583 11593, 2021.
13

---

[53] Jianfeng Dong, Minsong Zhang, Zheng Zhang, Xianke Chen, Daizong Liu, Xiaoye Qu, Xun Wang, and Baolong Liu. Dual learning with dynamic knowledge distillation for partially relevant video retrieval. In Proceedings of IEEE/CVF International Conference on Computer Vision, pages 11302-11312, 2023.
[54] Xirong Li, Fangming Zhou, Chaoxi Xu, Jiaqi Ji, and Gang Yang. Sea: Sentence encoder assembly for video retrieval by textual queries. IEEE Transactions on Multimedia, 23:4351-43666 -4662, 2020.
[55] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299 6308, 2017.
[56] Jeffiey Pennington, Richard Socher. and Christopher Manning. Glove: Global vectors word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543, 2014.
[57] Jingyuan Chen, Lin Ma, Xinpeng Chen, Zequn Jie, and Jiebo Luo. Localizing natural language in videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8175-818 175-8182, 2019.
[58] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (11), 2008.
[59] Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. Bidirectional attention flow for machine comprehension. arXiv preprint 1xXiv.611.01660) 2016.
[60] Caiming Xiong, Victor Zhong, and Richard Socher. Dynamic coattention networks for question answering. arXiv preprint arXiv:1611.0160 2016.
[61] Adams Wei Yu, David Dohan, Miah-Thang Thang Luong. Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc Le. Qanet: Combining local convolution with global self-attention for reading comprehension. arXiv preprint Xrv: /004, 09541, 2018.
[62] Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, volume page 2, 2019.
[63] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 21(140): 2020.
[64] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1977: 2019.
14

---

A More Technical Details
A.1 Video and Text Encoders
Considering the need for intra-modal interaction among features, this paper implements single layer of the simplest Transformer encoder achieve self-attention. Since the Transformer primarily models global information interactions and does not capture local region information within the feature sequence, four convolutional layers are added prior to the multi-head attention mechanism in the Transformer to enhance the capability for local area interaction.
A.1.1 Self-Attention Mechanism
To capture the complete semantic information and ohtain sentence-level features of the query text, a self-attention mechanism is employed to aggregate the word embedding sequence. Specifically, the sentence-level feature ha of the query can be calculated with the word-wise query feature Q follows:
h = Q" softmax(QW);
(6)
where ho RR and W E RxX1 is learnable weight matrix that represents the importance each word in the sentence.
A.1.2 Text-Guided Video Feature Enhancement
To enhance the video semantic with query contexts, we employ a query-video bidirectional attention mechanism [59 60). which allows for simultaneous focus on the interaction between text words and video frames. The query-video bidirectional attention mechanism calculates attention scores from two directions: from video text and from text video. Both directions of attention matrices derived from common similarity matrix S _ which can expressed as:
Ki a (Vi, Qj):
(7)
where 8ij R represents the similarity score between the i-th vidco feature vector Vi and the j-th word vector Qj. The function a(-) ) is trilinear function [6]] used to measure the similarity between the two input vectors V. and Qy:
a(v, q) = Wo je; q; u 1
(8)
where Wo represents a trainable weight matrix, and @ denotes element-wise multiplication. Normaliz- ing the rows of the common similarity matrix yields the attention weights from video to text A
[a]2y, 02 22 ..... 2 R=xm Similarly, normalizing the columns of S gives the attention weights from text to video A [o] aa2h ...., o2221 E mR="" Subsequently, using the attention weights A the weighted text feature sequence Q used to construct the video feature sequence C A Q E Raad Similarly, utilizing the attention weights A the text feature sequence reconstructed from video features C420 A V E mm is obtained. The reconstructed text feature sequence C2220 is then mapped back into the video encoding space using the attention weights 2 producing text-aware video feature sequenee C 22 (222v E nxd Finally, through a simple feedforward network, original video feature sequence V. the video feature sequence reconstructed from text features C", and the text-aware video feature sequence C are fused to obtain the text-guided enhanced video feature Vg:
Vg FFN((V: C" C" VV C*]).
where Vg E RR" FFN(.) denotes the feedforward network; A indicates element- multiplication.
A.1.3 Multi-Modal Feature Fusion
Although the bidirectional attention mechanism facilitates cross-modal interaction and information fusion between query text and vidco, this process primarily enables local interaction between individual words and video frames. Therefore, we further integrate sentence-level features hg, which contain complete semantic information, with the text-guided enhanced video feature V,, to generate the final contextual multi-modal representation V. This is achieved through the feature concatenation operation:
V/" FC((hq: U41, has U.,,... .......,uue hgi Uq..))
(10)
where [:] denotes concatenation along the feature dimension, and FC(-) represents fully connected layer that maps the features d-dimensional space.
A.2 Supervision of the Frame-level Relevance Discriminator
Considering that the similarity score between text and each video frame depends on the fully connected layer's understanding of foreground and background frames, this paper introduces a binary cross-entropy loss Lframe
15

---

supervise the learning of the rrame-level relevance discriminator. This supervision ensures that the discriminator correctly learns the differences between foreground frames and background frames. The loss function is defined as follows:
Tiframe 1 Σ (vs. log(ss,) + (1 yr. log(1 s(,)),
(11)
where ys, is the i-th element of the binary sequence Y,. and ss. is the i-th element of the foreground frame prediction score sequence S,, with n being the length of sequence. The binary sequence Yy is constructed as follows: for a video feature sequence length n. a, and d, denote the start and end boundary indices of the target segment, respectively. Frames within these indices are labeled as (indicating foreground frames), while those outside are labeled as 0 (indicating background frames).
A.3 Supervision of the Video-level Relevance Discriminator
The sentence-level query feature ha and the query-related video representation by are concatenated to produce a vivel-level relation signal vector g E R represented as 9 FC(hq; To train the video-level relevance discriminator and ensure that the video-level relation signal vector accurately learns the semantic relationship between the query text and video segments, this paper introduces a video-level relation constraint Leideo- Specifically, the video-level relation signal vector is input into a fully connected layer with sigmoid function to obtain the video-level similarity score scoresg sigmoid(FC(g)), which globally describes the relevance between the and the video. The video-level relation constraint is calculated as:
Luideo - (yn, log(scoreee)) + (1 ys)) log(1 scorcey)).
(12)
where yo. is the ground truth label indicating whether a sample has retrievable results or not.
A.4 Relation-Aware Segment Predictor
Considering the video-level relation signal vector 9 obtained from the video-level relevance discriminator contains enough relevanco-awar aware knowledge to determine whether the segment cxists or not, we utilize the video- level relation signal vector g as the special token in this segment predictor. Specifically, using the foreground frame prediction score sequence Sy produced by the Frame-level relevance discriminator, the foreground frames in the text-guided enhanced video feature V, are enhanced, while the background features unrelated to the query text are diminished. The foreground-enhance video feature sequence V, is calculated as:
V4- Vg Sr,
(13)
where @ denotes clement-wise multiplication. Subsequently, the video-level relation signal vector g is concate- nated to the front of foreground enhanced video feature sequence Vg. forming the input feature sequence Vi [g: Vg] for the relation- aware segment predictor, in dimension (=x==))xd This sequence is then fed into stacked two-layer LSTM network. The first layer of the LSTM generates the time-sequential feature sequence for
predicting the start boundary H [hā, hi..... hā] in ((++1)x+] (1xd)) and the second layer processes this output to generate the end boundary feature sequence Hf = [hū, Mi....,,.
A.]] E IR (x++))xd as follows:
h LSTM (ug, M)-1);
(14)
hg LSTM2 hf-1),
where h//- denotes the 1-th time-sequente feature vector in Ha/e, and og. represents the t-th video feature vector in V.a. LSTM, indicates the corresponding layer of the LSTM network. The start and end time- seneeequential feature sequences H and Hº are each fed into their respective feedforward networks to obtain the boundary
probability distributions Pa/e E R(++1);
P. softmax(FFF.... (H" V2()),
(15)
P. = softmax Nena HH; V21),,
where P. represents the start boundary probability distribution, and P. the end boundary probability distribution. The boundary prediction loss function Lioundery is represented as:
Lioundaarg
Y. log((.) + Σ Y. log(P.)),
(16)
where Y, and Y. are the one-hot vector representations of the ground-trun start (a.) and end (a.) bound- ary indices. respectively. P. and P. indicate the predicted start and end boundary probability distribution. respectively.
16

---

V1
S11
Query: person opened refrigerator.
Query:
person closed the cupboard.
Sim
V
V3
Sample pair with retrieval results
Sample pair without retrieval results
Figure 6: The organizational structure of reconstructed dataset.
Table 7: The detailed statistics of the different datasets.
Dataset
Domain
#Videos
#Anno.
#Anno RF.
Charades-STA
22
Indoor
6,672
12,408/1,,60/11
ActivityNet Captions
Open
19,207
37,421017,,055117.611
Charades-RF
Indoor
6,672
12,400/3,,20/3,720
-/1 21,86001,50 1,600
ActivityNet-RI
Open
19,207
37.421//5,,00/334.622
-/17.505/17,0
#Anno. denotes the number of video-text annotation pairs different (train/val/tes),
#Anno-RF. denotes the number of viden-text annotation pairs without retrieval results different (train/val/tst).
The original Charades-STA dataset lacked validation
randomly split the test set into two
equal parts create validation set and new test set
A.5 Dataset Reconstruction
Due the absence of datascts specifically created for the TSG-RF task, this paper has reconstructed the validation and test sets of widely used datasets the TSG domain: Charades-STA [2] and Aetivity-Net Captions [23], construct a lesting environment for task. the original datasets, a single video corresponds to multiple query texts, as illustrated in Figure 6 where video Vi corresponds to m query texts with grounding results [S11 S12...,, m.m. For each text with grounding results, corresponding sample without grounding results (i.e., the video does not contain segments relevant the query text) is constructed.
Specifically, the reconstruction process is exemplified by the sample pair (S11, Vi). By randomly selecting another video from the video library to pair the query text S11, sample pair without grounding results (S11. V,, is formed. To ensure quality of reconstructed sample pairs and remove low-quality ones, this paper utilizes Large Language Models 622644 for their deep and precise text modeling capabilities to achieve sample selection. Specifically, the BERT model (22) is used to extract features from the query text. Subsequently, the cosine similarity between the query Sn and the query
with grounding results from randomly selected video [S....... S.m] calculated. If maximum similarity less than a threshold value (empirically set to 0.2), the reconstructed sample pair without grounding results (Snr. V., is added to the dataset. Otherwise, a new vidco is randomly selected until the condition is met. This methodology ensures the relevance and integrity of the dataset for accurately simulating the TSG-RF task.
As shown in Table 7 name the corresponding reconstructed datascts Charades-RE RF and Activity RF for the TSG-RF RF task. these reconstructed datasets, we added equal number of samples without grounding results validation and test sets. This augmentation ensures real-world scenarios where not all query texts correspond to relevant video segments.
datasets more accurately reflect
A.6 Implementation Details
Each word the query is initialized using GloVe3 which remains frozen during training. Visual features of videos are extracted using a pre pre-trained 13D network. The maximum video feature sequence length set to 128. Sequences longer than this are uniformly downsampled to 128, while shorter sequences are zero zero-pade pooo-padedd to the same length. During training, since the training set reconstructed and only contains samples with
17

---

grounding results, each batch includes randomly selected videos paired with the original query text to create samples without grounding results. In Equation! X
empirically set and value providing the highest accuracy on the validation set is chosen: 0.5 for Charades-Re RF and 0.3 for Activity Net-RF. All experiments are conducted on workstation with an NVIDIA GeForce RTX 3090Ti GPU and 256G RAM. Training our proposed model on the Charades- RF datasct takes approximately 2 hours, while training the ActivityNet-RR dataset takes approximately hours.
to balance all loss
functions at start training. For threshold me in Equation
B
More Experiments
B.1
Implement of the Enhanced Baseline with Relevance Feedback
Since there are currently no models specifically designed for the TSG-RF task. we select representative open- source models developed for the TSG task in recent years as baseline models to ensure fair and replicable comparisons, including VSe.Ntt 13. SeqPAN [4), FAMAT [15] ADPN [16), UniVTG [49], QD-DETR
[88 While these baseline models demonstrate strong video segment grounding capabilities, they lack the ability to discern the relevance between the video and contain retrievable results. Considering that directly training an additional relevance determination model the most straightforward and efficient strategy to enable the above baseline models to discern relevance, this paper constructs a simple relevance determination model. By integrating this independently trained relevance determination model with the aforementioned baseline models, compensate for their lack of relevance discernment capability. forming enhanced baseline models: VSLNet** SeqPAN+ EAMAT* ADPN* UniVTG+ and QD-DETR+* ++ Figure illustrates the architecture of the simple relevance discrimination model. Specifically, the query text and video are processed through separate feature encoders. The encoded features of text and video then concatenated along the sequence length dimension, with CLS token added the heginning aid learning the relevance hetween text and video. This concatenated sequence is subsequently input into a Transformer encoder to facilitate the interaction of information between feature vectors. Finally, the CLS token from the input sequence is processed through a fully connected layer to obtain the case discrimination score Po.
input samples, and thus assume that all samples
CLStoken
some coffee.
doorway drinking
0.6 Pp
person walks in
Query Text:
Text Encoder
FC
Transformer Encoder
Video Encoder
Figure 7: The architecture of the simple relevance discrimination Model.
B.2 Grouped Performance Comparison with Baseline Methods
To further explore model performance under unbalanced data conditions, the number of samples without grounding results in the test set was randomly decreased in proportion, yielding six groups of test sets with ratios of non-grounding to retrieval samples at 0, 0.2, 0.4, 0.6, 0.8, and 1.0, respectively. Observing Figure S from left to right, when the proportion of non- non-grounding samples is O. the test set contains only grounding samples, testing all models their TSG task performance. Starting from 0.2 ratio, all models on their TSG-RF task performance. As the proportion nou nou-grrunding samples in the test set increases, although there is general improvement in the mloU of all models, the increase exhibited by RaTSG is the most significant. This is attributed to RaTSG's multi-scal scale learning of the semantic relationships between text and video, effectively integrating relevance feedback during the segment grounding process. Particularly test sets
assessed
18

---

with high proportion of non- non-grounn grounding samples. RaTSG demonstrates superior performance compared to other models, showcasing its exceptional ability in discrimination and fine-grained grounding.
65
VSLNer* SeqPAN*
EAMAT*
QD - DETR
RaTSG((urs)
ADPN**
UniVTG
60
55
mioU
50
45
35
30
0.0
0.2
0.4
0.6
0.8
1.0
Sample quantity ratios
Figure 8: Performance Comparison across different models at varying sample quantity ratios.
B.3 Performance Comparison in the Context of TSG Task
As illustrated in Table X we present performance comparison various models on Charades-STI and ActivityNet Captions datasets for the TSG task. While our model, RaTSG, does not outperform all other models in every metric, it remains highly competitive. On the Charades-STA dataset, RaTSG achieves an R1 @0.3 of 74. 14.1, which is very close the performing EAMAT 74.25%. For RI @0.5, RaTSG scores 56.61% just slightly behind SeqPAN's 59. 14%. terms of mloU, RaTSG records 53.02%, comparable ADPN's 51.96% Similarly, on the ActivityNet Captions dataset, RaTSG attains an R1 @0.3 of 46%, nearly matching ADPN's 61 1.46% and EAMAT's ' 63. 63.20%. For RI @0.5, RaTSG scores 42.36%, close to SeqPAN's 45.31 %. In terms of mloU, RaTSG achieves 43.72%, demonstrating its competitiveness alongside other models. Although RaTSG not consistently the highest performer, it demonstrates competitive results across various metrics. It important to note that our model is specifically designed for the TSG-RF task, which includes the challenge of handling samples without grounding results. This specialized focus may slightly affect its performance on the standard TSG task, yet RaTSG remains highly competitive. This demonstrates the robustness and versatility of approach, indicating its strong potential in more complex real-world scenarios.
Table Comparison on Charades- STA and ActivityNet Captions in the context of TSG task.
Charades-STA
ActivityNet Captions
Method
RI @0.3
RI@0,5
R1@0.
mloU
R1@0.3
R1@0.5
RI@0,7
mloU
VSI
67.47
54.62
35.43
49.37
62.12
43.76
25.64
44.54
SoqPAN
70.70
59.14
41.02
52.32
63.71
45.31
26.69
45.73
EAMAT
74.25
61.18
41.72
54.53
62.20
41.60
24.14
44.15
ADPN
71.24
56.88
39.73
51.96
61.46
41.49
24.78
44.12
QD-DETR
70.32
58.92
38.54
50.62
62.20
41.60
24.14
44.15
UniVTG
71.62
60.06
33.34
49.92
61.78
43.34
22.59
42.71
RaTSG(ours)
74.19
56.61
37.47
53.02
61.46
42.36
24.74
43.72
B.4 Comparison to CLIP as relevance discriminator
As shown in Tablc we also try compare designed relevance discriminator with large-scale pre-trained vision- langual models duc their strong cross-modal representation capabilities. To implement comparison, we choose the CLIP model, large-scale pre- pre-trained vision- lisouneannguae model, to assess the relevance between the query text and video. Specifically, we first utilize the CLIP to measure the cosine similarity score between the query text and each frame of video, and then aggregate the similarity scores over all frames to obtain the final relevance. In particular, we two aggregate methods implemented by the CL.PP model: (1) averaging all scores (CLIP-Avg) and (2) averaging on top-5 scores (CLIP-Top5) We incorporate CLIP with the SOTA method EAMAT to achieve the joint relevance feedback and grounding framework. The experimental results on the Charades RF dataset are summarized in Table 9 We can find that CLIP- based EAMAT models achieve relatively lower performance, demonstrating that the CLIP model has poor ability to predict relevance. We attribute
19

---

to the fact that the CLIP model is simply pre-trained common scenarios without additional fine-tuning on specific downstream task, thus severely suffering from domain shift issues the target dataset.
Table 9: Comparison of relevance discrimination ability with CLIP on Charades- RF dataset.
Method
Acc
RI @0.3
RI @0.5
R1@0.7
mloU
EAMAT
71.94
63.55
59.17
51.96
56.23
EAMATCLIP-AA
60.70
54.92
51.96
47.10
50.21
EAMATCLIP-TOR
62.52
55.75
52.47
46.94
50.25
RaTSG(Ours)
76.85
68.17
61.91
54.19
59.93
20

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction of this paper accurately reflect its contributions and scope.
Guidelines:
The answer NA means that abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match thcorctical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, the paper discusses the limitations of the work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors are encouraged create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results these assumptions (e.g.. independence assumptions, noiseless settings, model well well-specifiennt specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
violations of
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should he articulated.
The authors should reflect the factors that inffuence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
21

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
Formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper fully discloses all the necessary information to reproduce the main experimen- tal results, ensuring that the main claims and conclusions can be independently verified.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is datasct and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, contribution novel architecture, describing the architecture fully might suffice, or the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require relcasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clcarly and fully.
(c) the contribution a new model (e.g.. a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (c.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reprodueibility. In the case of closed-source source models, it may be that access to the model is limited in some way (c.g.. to registered users). but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code. with sufficient instructions faithfully reproduce the main experimental results, as described supplemental material?
Answer: [Yes]
Justification: The code and data have been submitted as supplementary material in ZIP format.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurTPS code and data submission guidelines httpe////ippscccpubbp/
for more details.
While we encourage release of code and data, we understand that this might not possible,
so "Na" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source henchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: : ://nis///.////////// cc/public/ qui.dess/Coodeuubmmeessooooooo.. for more details.
22

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reprodueible, they should state which ones are omitted from script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand results?
Answer: [Yes]
Justification: As mentioned in Section this paper provides comprehensive details on the training and test settings, including data splits, hyperparametens.
Guidelines:
answer NA means that the paper does not include experiments.
The experimental setting should be presented in the of the paper level of detail that
necessary appreciate the results and make sense them.
The full details can provided either with the code, appendix. or as supplemental material.
7. Experiment Statistical Significance
Question: Daes the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [No]
Justification: The paper does not provide error bars, confidence intervals, statistical significance tests for the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, hootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error har than state that they have 96% CI, if the hypothesis of Normality of errors is verified.
For asymmetric distributions, the authors should be careful symmetric error hars would yield results that are out of range (e.g. negative error rates).
show in tables figures
error bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Ycs]
Justification: In Appendix A.6, we provide the type of compute workers (GPUs) and the size of memory. Additionally, we provide the training time of our proposed model on each dataset.
Guidelines:
The answer NA means that the paper does not include experiments.
23

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster. or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethies httph: /ttrpa://neuria...
Answer: [Yes]
Justification:
The research fully adheres to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
authors should make sure preserve anonymity (e.g.. if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: mentioned in Section [ the TSG-RF task proposed in this paper improves upon the limitations of existing TSG tasks, making it more suitable for real- world applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed. If the authors answer NA or No, they should explain why their work has socictal impact why the paper does address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g., deploy- ment technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will foundational research not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (c.g., gated relcase of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g. pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions to access the model implementing safety filters.
24

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: paper does not existing
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version the asset is used and, if possible, include a URL.
The name of the license (c.g., CC-BY 4.0) should be included for cach asset. scraped from particular source (e.g.. website), the copyright terms of service of that source should provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswi thcode. www.coom//aaac /attttttstts has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-per re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the datasct/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations,
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an amonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelincs:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
25

---

Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer means that paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, approval (or equivalent) may be required for any human subjects research. If you ohtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
26


Reviews:
[{"review_id": "6u9TDsUyvX", "reviewer": "Reviewer_3Qyt", "summary": "The work develops a model that localizes query-related segments when present and provides feedback on non-existence when absent.\nIt proposes a Relation-aware Temporal Sentence Grounding (RaTSG) network, which reformulates TSG-RF as a foreground-background detection problem.\nAlso, the work uses a multi-granularity relevance discriminator for precise video-query relevance feedback.\nThe work constructs two popular TSG datasets for TSG-RF benchmarking.\nExperimental results demonstrate RaTSG's effectiveness for the TSG-RF task", "questions": "Please address the questions in weakness.", "limitations": "None", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "The work introduces a new and more realistic task, TSG-RF, which addresses a significant limitation of traditional TSG.\nThe work develops a model that can handle cases where no relevant segment exists, making it more suitable for real-world applications.\nExperimental results demonstrate the effectiveness of the proposed RaTSG network for the TSG-RF task.", "weaknesses": "1. The work claimed to incorporate foreground/background info. However, it was simply adding a binary classification, such binary foreground classification for temporal grounding was already explored in [1]. The video part is done by weighted sum, which is also a standard way of aggregating frame features.\n2. In Table 1, the author explained why the performance is 50%. Why are the other rows all have 71.94% of accuracy and 81.6% accuracy?\n3. Although this work proposes a new RF setting, it will be interesting to know how the model performs in a standard setting to see if there is a performance gap.\n\n\n\n[1] Lin, Kevin Qinghong, et al. \"Univtg: Towards unified video-language temporal grounding.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 3Qyt, \n\nThank you for your thorough review and valuable feedback. We appreciate your insights and would like to address your concerns point by point.\n\n**Q1:  The work claimed to incorporate foreground/background info. However, it was simply adding a binary classification, such binary foreground classification for temporal grounding was already explored in [1]. The video part is done by weighted sum, which is also a standard way of aggregating frame features.**\n\n**A1**:  In our work, we make the first attempt to fill the critical gap in traditional temporal sentence grounding tasks that assume the existence of relevant segments in the corresponding video but fails to provid feedback on non-existence relevant content. In addition to proposing a novel TSG-RF task and a new framework, we claim that we have also developed novel technical designs for the pipeline.\n\nIn particular, as for the foreground/background information learning, our foreground-background detection is not simply implemented with a binary classification. Although the naive binary classification head has been investigated and utilized to detect the foreground clips in previous work [1], it is simply/directly applied to the raw clip-level features for classification. Such design is coarse and fail to capture the long-term dependency in the complex videos. In contrast, we propose a more effective multi-granularity relevance discriminator. By employing both frame-level and video-level relevance discriminators, our dual-level multi-granularity relevance discriminator allows us to effectively balance fine-grained and coarse-grained relevance assessments, thereby enhancing the overall robustness and applicability of our model.\n\nAdditionally, while the weighted sum operation is indeed a standard operation in video analysis, it is effective enough to handle the frame-wise contexts aggregation for determining the query-related video-level content. Moreover, aggregating frame features is not the focus of our work, and any feature aggregation operation can be theoretically used in our proposed framework. Note that our main contribution is proposing a novel TSG-RF task and a new framework that incorporates relevance feedback capability.\n\n\n**Q2:  In Table 1, the author explained why the performance is 50%. Why are the other rows all have 71.94% of accuracy and 81.6% accuracy?**\n\n**A2**:  In Table 1, the performance of traditional TSG methods, such as VSLNet, shows 50% accuracy because our newly constructed test set has an equal ratio (1:1) of samples with and without grounding results (i.e., query-relevant segments exist or do not exist in the corresponding video). Since the traditional TSG methods assume that all samples have grounding results, they lead to a relevance prediction accuracy of 50%.\n\nFor the enhanced versions of the traditional models (i.e., VSLNet++, ADPN++, SeqPAN++,  and EAMAT++) , we incorporate an additional relevance discriminator to their models. This relevance discriminator is trained separately and then directly combined with the baseline methods, endowing them with the same ability to discriminate relevance (Please  refer to Appendix B.1 for more implementation details). This additional relevance discriminator achieves the 71.94% and 81.6% accuracy on Charades-RF and ActivityNet-RF, respectively. It is worth noting that as all enhanced models share the same relevance discriminator, they achive the same accuracy on the same dataset.\n\n**Q3: Although this work proposes a new RF setting, it will be interesting to know how the model performs in a standard setting to see if there is a performance gap.**\n\n**A3**:  Actually we have conducted experimetns with the standard setting in Appendix B.3 of our paper. Below is the Table  from the appendix, summarizing the performance comparison in the context of the TSG task. Although our proposed RaTSG is not consistently the highest performer, it demonstrates competitive results across various metrics. It is important to note that our model is specifically designed for the TSG-RF task, which includes the challenge of handling samples without grounding results. This specialized focus may slightly affect its performance on the standard TSG task, yet RaTSG remains highly competitive. This demonstrates the robustness and versatility of our approach, indicating its strong potential in more complex real-world scenarios.\n\n| Method| Charades-STA (R1\\@0.3) | Charades-STA (R1\\@0.5) | Charades-STA (R1\\@0.7) | Charades-STA (mIoU) | ActivityNet (R1\\@0.3) | ActivityNet (R1\\@0.5) | ActivityNet (R1\\@0.7) | ActivityNet (mIoU)|\n|-----|-----|-----|-----|-----|-----|-----|-----|-----|\n| VSLNet | 67.47| 54.62| 35.43| 49.37| 62.12| 43.76| 25.64| 44.54|\n| SeqPAN | 70.70| 59.14| 41.02| 52.32| 63.71| 45.31| 26.69| 45.73|\n| EAMAT| 74.25| 61.18| 41.72| 54.53| 62.20| 41.60| 24.14| 44.15|\n| ADPN| 71.24| 56.88| 39.73| 51.96| 61.46| 41.49| 24.78| 44.12|\n| RaTSG (Ours) | 74.19| 56.61| 37.47| 53.02| 61.46| 42.36| 24.74| 43.72|\n\n\nWe will incorporate all the above in the final version. Please reconsider our paper. Thank you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "prPBTmOa0e", "reviewer": "Reviewer_dpih", "summary": "This paper introduces Temporal Sentence Grounding with Relevance Feedback (TSG-RF) in videos, a new task that addresses the limitations of traditional Temporal Sentence Grounding (TSG), which assumes relevant segments always exist within a video. TSG-RF accounts for the possibility that a video may not include a segment related to the query, aiming to localize segments that align with the query when present and provide feedback when they are absent. The proposed Relation-aware Temporal Sentence Grounding (RaTSG) network reformulates TSG-RF as a foreground-background detection problem, assessing query-related semantics at both frame and video levels. It utilizes a multi-granularity relevance discriminator for precise relevance feedback and a relation-aware segment grounding module to adaptively ground segments. To validate RaTSG, two popular TSG datasets are reconstructed, establishing a benchmark for TSG-RF. Experimental results demonstrate the effectiveness of RaTSG for this task.", "questions": "Please refer to the Weaknesses section.", "limitations": "The Relevance feedback in Video Moment Retrieval is a trivial research question. Either we could generalize this problem to general video grounding (spatial / temporal) or general video understanding (not just VMR, but also hallucinations in VLMs).", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This paper introduces the Relation-aware Temporal Sentence Grounding (RaTSG) network, which effectively uses a multi-granularity relevance discriminator for predicting relevance feedback and a relation-aware segment grounding module for selectively determining segment boundaries.\n- The paper argument the original datasets for evaluation.", "weaknesses": "- The key motivation of this paper is related to another video temporal task -- highlight detection. However, this paper do not have any discussion.\n- The method is cumbersome, featuring several incremental designs.\n- The experiments utilize outdated datasets. Newer datasets, such as QV-Highlights, should be considered.\n- The baselines are no up to date which do not include sota methods. such as UMT, UniVTG, QD-DETR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer dpih, \n\nThanks for your detailed review and constructive feedback. We appreciate your insights and would like to address your concerns point by point.\n\n\n**Q1:  discussion about highlight detection**\n\n**A1**: We thank the reviewer for sharing the insight. Actually, our RF task is quite different from highlight detection. Specifically, highlight detection mainly aims to identify the most interesting or important segments within a video based on a given natural language query, focusing on segments that are salient or engaging. This task is similar to the traditional temporal sentence grounding, and existing highlight detection methods still naively assume that there are always highlights or interesting segments in the video. In contrast, our RF setting is specifically designed to handle cases where no relevant segments are found. This involves providing relevance feedback to indicate when no relevant segments exist, which is essential for practical applications where not all queries have corresponding relevant segments. We believe that our RF setting can also be plug-and-play to be applied to the highlight moment detection tasks. We will discuss the  highlight detection in the related work.\n\n**Q2: The method is cumbersome, featuring several incremental designs.**\n\n**A2**: Our proposed framework is specifically designed to handle the challenges posed by our new introduced  TSG-RF task, incorporating several new key modules to ensure effective performance (**please see the global response**). Moreover, our RaTSG framework is a multi-task model that achieves high efficiency with relatively low complexity. As shown in Table 1 of our paper, our model has fewer parameters compared to other baseline models, indicating that our approach is not cumbersome. \n\n**Q3: The experiments utilize outdated datasets. Newer datasets, such as QV-Highlights, should be considered.**\n\n**A3**: As shown in the table below, we make a statistics on the number of top-tier conferrence papers (NeurIPS、CVPR、ICCV、ECCV、AAAI、ACM MM) using the three datasets over the past three years, finding that Charades-STA and ActivityNet Captions are more popular than QV-Highlights. Though including QV-Highlights would enhance our work, we believe that the extensive experiments on two widely-used TSG datasets are convincing. Moreover, we try to use QV-Highlights, but we could not complete dataset downloading, reconstruction, and running experiments with multiple baselines within the limited rebuttal time. We leave the use of QV-Highlights for future work.\n\n| Dataset Name | 2021 | 2022 | 2023 |\n|--------------|----- |-----|------|\n| Charades-STA | 10 | 12 | 19 |\n| ActivityNet Captions | 12 | 13 | 14 |\n| QV-Highlights | 1 | 1 | 6 |\n\n**Q4: The baselines are no up to date which do not include sota methods. such as UMT, UniVTG, QD-DETR.**\n\n**A4**: We would like to clarify that our paper does include comparisons with recent state-of-the-art methods:\n\n- The following table summarizes the traditional TSG performance comparison between the suggested UMT, UniVTG, QD-DERT and our compared ADPN on Charades-STA. It shows that our chosen ADPN is almost the SOTA method, performing much better than the other methods, especially in terms of R1\\@0.7. Therefore, we argue that our compared method is not outdated.\n\n| Method| R1\\@0.3 | R1\\@0.5 | R1\\@0.7 | mIoU| \n|----|---|----|---|---|\n| UMT (2022) | -| 48.31  | 29.25  | - |\n| UniVTG (2023)  | 70.81  | **58.01**  | 35.65  | 50.10 | \n| QD-DETR (2023) | -| 57.31  | 32.55  | - |\n| **ADPN (2023)**|**71.99**|57.69|**41.10**|**52.86**|\n\n- Additionally, following the reviewer's suggestion, we have included a comparison with UniVTG and QD-DETR. Note that we did not compare with UMT considering its performance is much worse than the others (please see the above table). The table below summarizes the performance on the Charades-RF dataset. Our proposed model still outperforms the UniVTG and QD-DETR counterparts. We will  update Table 1 by including UniVTG and QD-DETR and revise the corresponding discussion accordingly in the revison.\n\n| Model | Acc|R1\\@0.3 |R1\\@0.5 |R1\\@0.7 | mIoU |\n|------|----|-----|----|---|------|\n| UniVTG | 50.00| 35.81| 30.03| 16.67| 24.96 |\n| QD-DETR | 50.00| 35.16| 29.46| 19.27| 25.31 |\n| UniVTG++ | 71.94|62.58| 58.55| 48.79| 54.65 |\n| QD-DETR++| 71.94| 62.18| 58.20| 50.96| 55.13 |\n| **RaTSG (ours)** | **76.85**| **68.17**| **61.91**| **54.19**| **59.93**|\n\n**Q5: The Relevance feedback in Video Moment Retrieval is a trivial research question. Either we could generalize this problem to general video-related tasks.**\n\n**A5**: We respectfully disagree with the assertion that relevance feedback in video moment retrieval is a trivial research question. Actually, the TSG-RF task addresses a significant gap in the field of temporal sentence grounding by introducing a scenario where no relevant segments might exist in the video. This is a critical aspect that current methods do not adequately address. The relevance feedback mechanism in our work is crucial for several reasons:\n\n- **More Practical**: In real-world applications, not every query will have a relevant segment in the video. Providing accurate feedback on the absence of relevant segments is essential for practical usability.\n\n- **Enhanced Accuracy**: The feedback mechanism improves the precision and reliability of the grounding process by dynamically adjusting based on the presence or absence of relevant segments.\n\n- **Potential for Generalization**: Our framework has the potential to be generalized to broader video grounding tasks, including spatial and temporal grounding, as well as general video understanding tasks. The principles of relevance feedback can be adapted to various contexts, enhancing the robustness and versatility of grounding models across different tasks.\n\nWe will incorporate all the above in the final version (as specified in our response). Please reconsider our paper. Thank you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XKvxoZ4lNh", "reviewer": "Reviewer_eoKJ", "summary": "This paper presents a novel task named Temporal Sentence Grounding with Relevance Feedback (TSG-RF) to overcome the limitations of conventional Temporal Sentence Grounding (TSG), which presumes the existence of relevant segments in videos. This paper introduces the Relation-aware Temporal Sentence Grounding (RaTSG) network, incorporating a multi-granularity relevance discriminator and a relation-aware segment grounding module to generate relevance feedback and segment boundaries. The RaTSG network's efficacy is demonstrated by reconstructing two widely-used datasets for TSG-RF and validating its performance through extensive experiments. Additionally, the source code for RaTSG is made available.", "questions": "Please refer to the weakness.", "limitations": "The paper does not discuss the limitations of the work. However, the introduction of the novel TSG-RF task significantly improves upon the limitations of traditional TSG task, offering positive societal impact. For constructive suggestions on areas needing improvement, please refer to the weaknesses.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1.This paper introduces the TSG-RF task, which addresses the limitations of traditional TSG by considering scenarios where query-related segments may be absent in videos.\n2.The proposed RaTSG network utilizes a multi-granularity relevance discriminator and a relation-aware segment grounding module, showing significant performance improvements and achieving state-of-the-art results in the TSG-RF task.\n3.The experimental validation is comprehensive, featuring reconstructed datasets and extensive experiments that clearly demonstrate the effectiveness of the RaTSG network. The results are presented in a clear and well-organized manner.", "weaknesses": "1.The weaknesses of this paper include a lack of sufficient baselines for comparison with the RaTSG network. For instance, the paper does not investigate the use of Large Multimodal Models (LMMs) to assess the relevance between the query text and video, which could offer a more robust benchmark for evaluating the proposed method. Incorporating such comparisons would enhance the overall evaluation of the RaTSG network.\n2.The paper also lacks adequate qualitative analysis. Including more qualitative examples of both successful and unsuccessful cases would provide deeper insights into the strengths and limitations of the RaTSG network.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer eoKJ, \n\nThank you for your detailed and positive review of our work. We appreciate your insights and suggestions for further improvement. Below, we address your concerns point by point.\n\n**Q1: The weaknesses of this paper include a lack of sufficient baselines for comparison with the RaTSG network. For instance, the paper does not investigate the use of Large Multimodal Models (LMMs) to assess the relevance between the query text and video, which could offer a more robust benchmark for evaluating the proposed method. Incorporating such comparisons would enhance the overall evaluation of the RaTSG network.**\n\n**A1**: While we acknowledge the importance of comparing our method with large multimodal models (LMMs), it is important to note that current state-of-the-art LMMs, such as Llava and GPT-4, do not yet have the capability for tackling the video-based task. Instead, we choose  a large-scale pre-trained vision-language model, i.e., CLIP, to assess the relevance between the query text and video. Specifically, CLIP is utilzed to measure the cosine similarity score between the query text and each frames of a video, and the final relevance is obtained by aggregating the similarity scores over all frames. We use two aggregate methods: averaging all scores (CLIP-Avg) and  averaging on top-5 scores (CLIP-Top5). Then we jointly use CLIP and  EAMAT that is the best performer among our four compared models, to conduct relevance feedback and grouding. The reulsts on Charades-RF are summarized in the following table. CLIP-based models achieve low accuracy, demonstating its poor ability for relevance prediction. We attribute it to the fact that CLIP is pretrained without addtional training, thus suffering from domain shift issues on the traget dataset.\n\n| Model | Acc  | R1\\@0.3 | R1\\@0.5 | R1\\@0.7 | mIoU  |\n|----------------|-----|--------|------    |--------|-------|\n| $EAMAT^{++}$ | 71.94 | 63.55   | 59.17  | 51.96  | 56.23 |\n| $EAMAT^{CLIP-Avg}$ |60.70  | 54.92   | 51.96  | 47.10  | 50.21 |\n| $EAMAT^{CLIP-Top5}$| 62.52| 55.75   | 52.47  | 46.94  | 50.25 |\n| **RaTSG (ours)** | 76.85| 68.17  | 61.91 | 54.19 | 59.93 |59.93 |\n\n\n**Q2: The paper also lacks adequate qualitative analysis. Including more qualitative examples of both successful and unsuccessful cases would provide deeper insights into the strengths and limitations of the RaTSG network.**\n\n**A2**:  To discuss the limitations of the work, we have included two unsuccessful examples in the attached PDF file of the global response. In the first example of the query text  \"A person starts sneezing\", our model incorrectly judges the relevance feedback due to the lack of audio cues, which are crucial for identifying the action of sneezing. In the second example, the query text is \"A person opening a laptop,\" but the video segment actually shows the person closing a laptop. Our model misinterprets the temporal sequence of actions, mistaking the closing action for the opening action. These examples demonstrate that our proposed model struggles with handling audio-related actions and temproal-sensitive content. However, such limitation can be alleviated by integrating audio features and temporal modeling in the video representation moudel. For the successful examples and corresponding discussion, we illustrate them in Figure 4 and Section 4.4.\n\nWe will incorporate all the above in the final version. Thank you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pjIgTrkeSU", "reviewer": "Reviewer_P4nK", "summary": "This paper proposes a new task called Temporal Sentence Grounding with Relevance Feedback (TSG-RF) in videos, which extends the traditional Temporal Sentence Grounding (TSG) by introducing cross-modal video-text semantic relevance prediction. Besides, a new Relation-aware Temporal Sentence Grounding (RaTSG) network is proposed, where a multi-granularity relevance discriminator and a relation-aware segment grounding module are specifically devised for the TSG-RF. Quantitative and qualitative experiments conducted on two reconstructed datasets demonstrate the effectiveness of our proposed RaTSG for the new TSG-RF task.", "questions": "1.\tThe proposed relation-aware segment predictor is key for TSG-RF. Could it be adapted for traditional TSG methods to enable them to work for TSG-RF?\n2.\tAs the proposed method requires additional relevance feedback, I wonder how the extra computational workload would be included.", "limitations": "Although the proposed method could perform extra relevance feedback, its complexity may be increased when compared to traditional TSG methods.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1.\tCompared to the traditional TSG, the proposed new TSG-RF task is interesting and more in line with practical application scenarios. Besides, two TSG datasets are reconstructed to make them suitable for evaluating the TSG-RF methods. The TSG-RF task with corresponding benchmarks, holds significant value for the vision-language community.\n2.\tThe proposed multi-granularity relevance discriminator and the relation-aware segment grounding module are simple and effective. It has been demonstrated that they mutually enhance each other.\n3.\tThe experimental results are convincing. Although there are no existing methods specifically designed for TSG-RF, the paper compares the recent TSG models and their extending variants for TSG-RF. The ablation experiments and visualization results demonstrate the effectiveness of the proposed method.\n4.\tThe writing of the paper is easy to follow. The source code and dataset are released.", "weaknesses": "The paper should illustrate bad examples in the visualization section, and discuss the limitations of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer P4nK, \n\nThank you for your comprehensive and positive review of our work. We appreciate your insights and suggestions for further improvement. Below, we address your concerns point by point.\n\n**Q1: The paper should illustrate bad examples in the visualization section, and discuss the limitations of the proposed method.**\n\n**A1**:  Following the reviewer's suggestion, we have included two bad examples in the attached PDF file of the global response. In the first example of the query text  \"A person starts sneezing\", our model incorrectly judges the relevance feedback due to the lack of audio cues, which are crucial for identifying the action of sneezing. In the second example, the query text is \"A person opening a laptop,\" but the video segment actually shows the person closing a laptop. Our model misinterprets the temporal sequence of actions, mistaking the closing action for the opening action. These examples demonstrate that our proposed model struggles with handling audio-related actions and temproal-sensitive content. However, such limitation can be alleviated by integrating audio features and temporal modeling in the video representation moudel.\n\n\n**Q2: The proposed relation-aware segment predictor is key for TSG-RF. Could it be adapted for traditional TSG methods to enable them to work for TSG-RF?**\n\n**A2**:  Of course, we can adapte our main components to traditional TSG methods to enable them to work for TSG-RF. Note that, to effectively address the TSG-RF task, it is essential to use not only the Relation-aware Segment Predictor (RaSP) but also the Multi-granularity Relevance Discriminator (MgRD). These components work together to assess relevance at multiple granularities and provide accurate segment grounding. Following the reviewer's suggestion, we adapt them into an existing traditional TSG model EAMAT, which is the best performer among our four compared models. As shown in the following table, EAMAT with our devised RaSP and MgRD outperfomes the original EAMAT and the enhanced EAMAT++ with  an extra relevance discriminator. The results demonstrate the adaptability and effectiveness of our proposed modules for enhancing traditional TSG methods to work for the TSG-RF task. \n\n| Model | Acc | R1\\@0.3 |R1\\@0.5|R1\\@0.7| mIoU |\n|------------|----|----------------|-----------------|----------------|-------|\n| EAMAT | 50.00 | 37.12 | 30.59 | 20.86 | 27.27 |\n| EAMAT++ | 71.94 | 63.55 | 59.17 | 51.96 | 56.23 |\n| **EAMAT+Ours** | **76.37** | **67.47** | **62.02** | **54.83** | **59.58** |\n\n**Q3: As the proposed method requires additional relevance feedback, I wonder how the extra computational workload would be included.**\n\n**A3**:  To measure model's computational workload, we use GFLOPs and parameter count. By adding relevance feedback capability to our model, the GFLOPs and parameter count increase from 0.077 to 0.081 G  and 1.21 to 1.27 M , respectively, with a relatively small increase of 5.1% and 4.9%. This negligible increase is due to our multi-task framework, where the additional relevance feedback module shares the feature extraction modules with the segment grounding module. This efficient design ensures that the overall computational workload remains low while enhancing the model with relevance feedback capability.\n\nWe will incorporate all the above in the final version. Thank you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their encouragement and guidance to further improve this work. **Here, we would like to discuss the specific design of our RaTSG framework for TSG-RF task.**

Firstly, we would like to emphasize that the primary innovation of our paper is the introduction of the new and challenging TSG-RF task. This is the first time this task has been proposed, addressing the critical gap in traditional temporal sentence grounding tasks that assume the existence of relevant segments in every video.

Secondly, to tackle this novel TSG-RF task, we propose a new Relation-aware Temporal Sentence Grounding (RaTSG) framework. Our framework is specifically designed to handle the challenges posed by this task, incorporating several new key modules to ensure effective performance:

- **Multi-granularity Relevance Discriminator**:
The multi-granularity relevance discriminator addresses the challenge of relevance feedback by evaluating the relevance between the query text and video at multiple granularities. 
   - **Frame-level relevance discriminator**:  It evaluates the relevance of each video frame with the query text on a fine-grained basis, capturing detailed and nuanced relationships. This ensures a precise measurement of relevance for each frame. 
   - **Video-level relevance discriminator**: It aggregates these frame-level relevance scores using a weighted sum approach, providing a broader context by calculating the relative relevance of each frame within the entire video sequence. 

- **Dynamic Relevance Feedback Mechanism**:
The dynamic relevance feedback mechanism is designed in the relation-aware segment grounding to handle the dynamic nature of relevance feedback. This mechanism dynamically adapts the segment grounding process based on the relevance feedback received. By iteratively refining predictions, the mechanism improves accuracy and ensures the model can handle cases where no relevant segment exists. The feedback integration enhances the model's ability to adapt to varying levels of relevance, ensuring practical usability. 

- **Mutual Enhancement through Joint Training**:
Our RaTSG method employs a joint training approach that promotes mutual enhancement between relevance discrimination and segment grounding. By sharing representations and learning jointly, the model benefits from the complementary strengths of both tasks. Relevance discrimination improves the model's ability to identify relevant segments, while segment grounding enhances its understanding of contextual relationships. As demonstrated in Section 4.3 (Ablation Studies) of our paper, the joint training approach leads to significant performance improvements. 

In summary, our RaTSG framework is carefully designed to address the specific challenges of the TSG-RF task. Each module plays a critical role in ensuring the model's robustness and accuracy in identifying relevant segments and providing relevance feedback when no relevant segments are found.

Meta-review: Initially, the reviewers raise some questions regarding the way to incorporate foreground/background info, the model performance in a standard setting, comparison with another video temporal task-highlight detection, incremental designs, dataset and baseline usage, the lack of adequate qualitative analysis, etc. Most of these questions are addressed in the rebuttal and recognized by reviewers. Eventually with the merit of the work, this paper receives one strong accept, one accept, and two borderline accepts, with agreement for acceptance. The AC recommends to accept. Authors are encouraged to revise the paper according to the reviews.
Decision: Accept (poster)