Temporal Sentence Grounding with Relevance Feedback in Videos

Authors: jianfeng  dong
xiaoman  peng
daizong  liu
xiaoye  qu
xun  yang
cuizhu  bao
meng  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 10:23:47

Abstract: As a widely explored multi-modal task, Temporal Sentence Grounding in videos (TSG) endeavors to retrieve a specific video segment matched with a given query text from a video. The traditional paradigm for TSG generally assumes that relevant segments always exist within a given video. However, this assumption is restrictive and unrealistic in real-world applications where the existence of a query-related segment is uncertain, easily resulting in erroneous grounding. Motivated by the research gap and practical application, this paper introduces a new task, named Temporal Sentence Grounding with Relevance Feedback (TSG-RF) in videos, which accommodates the possibility that a video may or may not include a segment related to the query. This task entails localizing precise video segments that semantically align with the query text when such content is present, while delivering definitive feedback on the non-existence of related segments when absent. Moreover, we propose a novel Relation-aware Temporal Sentence Grounding (RaTSG) network for addressing this challenging task. This network first reformulates the TSG-RF task as a foreground-background detection problem by investigating whether the query-related semantics exist in both frame and video levels. Then, a multi-granularity relevance discriminator is exploited to produce precise video-query relevance feedback and a relation-aware segment grounding module is employed to selectively conduct the grounding process, dynamically adapting to the presence or absence of query-related segments in videos. To validate our RaTSG network, we reconstruct two popular TSG datasets, establishing a rigorous benchmark for TSG-RF. Experimental results demonstrate the effectiveness of our proposed RaTSG for the TSG-RF task. Our source code is available at https://github.com/HuiGuanLab/RaTSG.

Main paper: 
 introduction: 
 Grounding target content described by users in videos is a fundamental capability that facilitates various multimedia applications, such as intelligent robotic service [1] , video on demand [2] , and metaverse [3] . Following this demand, temporal sentence grounding in videos (TSG) [4] has recently become a research hotspot, attracting wide attention from researchers in various fields [5] [6] [7] [8] [9] . Despite the significant advancements made in this field, the existing TSG task is overly idealistic. It aims to identify segments that are semantically relevant to a given query from a given long video, assuming that relevant segments always exist in the given video [10] [11] [12] . However, this setting limits the applicability of TSG in real-world scenarios as the given video may not contain the query-related Figure 1 : The difference between the TSG and TSG-RF tasks. TSG always predicts the start and end boundaries of the grounded segments, even in the absence of video content relevant to the given query text. By contrast, TSG-RF provides relevance feedback on whether exist query-related content in the given video, and selectively predicts the start and end boundaries of the grounded segments, according to the presence or absence of query-related segments in videos. contents, resulting mismatched or wrong grounding results. As illustrated in Figure 1 (a), given a video and a query, existing TSG methods always predict the start and end times of segments, even when the video lacks content relevant to the given query text. This suggests a significant gap between the literature and the real world.To fill the gap, this paper makes the first attempt to introduce an expanded but challenging task for TSG, named Temporal Sentence Grounding with Relevance Feedback (TSG-RF) in videos. This new task aims to deliver more flexible grounding results for a given query and a video. Specifically, as illustrated in Figure 1 (b), if relevant segments are present in the video, the task is to pinpoint the video segment that is semantically relevant to the given query description. Conversely, if no relevant segments exist, it should be explicitly conveyed to the user that "there are no target segments related to the query in the given video, and thus no corresponding segments can be localized."To tackle this new task, it is necessary to first review the failure reason of traditional TSG methods [13] [14] [15] [16] . We argue the essential issue of previous TSG methods is their inability to discern whether grounding results are warranted based on the presence or absence of relevant segments. To alleviate this issue, a straightforward approach to address the TSG-RF task involves a two-stage process: Firstly, a relevance discriminator is trained separately to assess the relevance between the query text and the video, providing relevant feedback. Then, query-video samples identified as relevant by the discriminator are processed using models designed for the TSG task to predict the target segment. Despite the structural simplicity of this integrated approach, it requires training two distinct models, leading to the redundant consumption of computational resources, increased storage demands, and suboptimal overall inference efficiency. Therefore, how to design an end-to-end relevance-aware TSG-RF method is an emerging problem.Based on the above considerations, we propose to address the TSG-RF task by developing a multitask learning model, i.e, the Relation-aware Temporal Sentence Grounding (RaTSG) network. This model promotes mutual enhancement between relevance discrimination and video segment grounding through shared knowledge, thus facilitating a unified and efficient approach to both tasks. As the essence of relevance discrimination lies in accurately measuring the semantic partial relevance between the query text and the video, we propose to first explore the measurement of partial semantic relevance between text and video at two different granularities: fine-grained and coarse-grained. Specifically, to capture fine-grained partial relevance, we adopt the concept of multiple instance learning [17] [18] [19] , treating the entire video as a bag with each frame as an instance. Following [20] , a video is considered partially related to the text if it contains at least one foreground frame; if all frames are background and irrelevant, the video is deemed unrelated to the text. Considering the semantic incompleteness of individual video frames, a coarse-grained perspective is also adopted to learn the relevance between the global video-level semantics and the full query text. Then, a multi-granularity relevance discriminator is designed to integrate the discriminative capabilities of both granularities to predict the final relevance score. As for the selective grounding process based on the relevance score, existing segment grounding heads always make grounding for any query-video input, even when no relevant segments are present in the video, which is not suitable for the TSG-RF task. Therefore, we design a relation-aware segment grounding module that takes the previously predicted relevance information of each query-video pair as prior knowledge, and selectively conducts grounding.We find a recent concurrent work [21] that shares a similar motivation with us. However, it simply treats the irrelevant video contents from an open-set perspective and implicitly utilizes cross-dataset evaluation. Instead, we implement original in-domain data to explicitly learn the potential irrelevant relationship. Further, existing TSG datasets can not be directly used to evaluate our proposed network for TSG-RF, since the datasets assume that a given query text can always localize a corresponding relevant segment from the given video. Hence, we further reconstruct two commonly used datasets, Charades-STA [22] and ActivityNet Captions [23] , by including samples without grounding results where the video does not contain the content relevant to the query.The main contributions of this paper are summarized as follows:• We formalize a novel TSG task to explore more flexible segment grounding in videos, i.e., Temporal Sentence Grounding with Relevance Feedback (TSG-RF) in videos, which advances user-specified TSG towards more practical applications.• Targeting the TSG-RF task, we propose a novel Relation-aware Temporal Sentence Grounding (RaTSG) network, which mainly depends on a multi-granularity relevance discriminator and a relation-aware segment grounding module. The multi-granularity relevance discriminator is devised to predict the relevance feedback based on fine-grained and coarse-grained relevance between query text and videos, and the relation-aware segment grounding module selectively predicts the start and end boundaries of the grounded segment.• To facilitate the evaluation of the TSG-RF task, we reconstruct two commonly used TSG datasets and establish appropriate performance evaluation metrics to meet the setting of TSG-RF. Extensive experiments conducted on these reconstructed datasets demonstrate the effectiveness of the proposed model.2 Related Work 
temporal sentence grounding and highlight detection in videos: 
 The Temporal Sentence Grounding in Videos (TSG) task aims to retrieve video segments that match user-input natural language descriptions. Recent efforts for TSG can be typically grouped into proposal-based and proposal-free methods. Proposal-based methods [22, [24] [25] [26] [27] [28] [29] [30] [31] [32] [33] [34] [35] [36] [37] [38] [39] follow a two-stage paradigm: generating proposals by dividing the video into clips and then aligning textual semantics with visual features. Proposal-free methods [40-44, 10, 13, 45] offer an end-to-end paradigm, predicting target segments without proposals, enhancing computational efficiency. Although efficient, these methods struggled with segment-level feature capture. To combine strengths, Xiao et al. [45] used an anchor-free approach to generate candidates, then matched them with query statements using anchor-based methods. However, the above methods cannot be directly applied to the TSG-RF task proposed in this paper. They will still predict the start and end times of segments and output retrieved segments, even in the absence of video content relevant to the query text.Highlight detection task is similar to TSG, which mainly aims to identify the most interesting or important segments within a video based on a given natural language query, focusing on segments that are salient or engaging [46] [47] [48] [49] . Recent works typically tackle this task using multi-modal inputs, advanced transformers, or large-scale pretraining techniques. For instance, Moment-DETR [46] introduce a transformer-based approach that simplifies highlight detection by treating it as a set prediction task, eliminating traditional proposal steps. QD-DETR [48] enhances video-text understanding with cross-attention and negative pairs to improve relevance predictions. UniVTG [49] unifies diverse temporal annotations and enables large-scale pretraining to improve generalization across video grounding tasks. Notably, similar to TSG, highlight detection also assumes that every video contains highlights. In contrast, our proposed TSG-RF task not only grounds relevant segments but also accounts for cases where no relevant segments are present.This paper extends TSG by incorporating relevance feedback to discern the semantic connection between text and video, introducing techniques for learning cross-modal semantic similarity [50] [51] [52] [53] .The prevailing approach aligns text and video semantics by learning a shared space, using a metric to assess similarity. For instance, Miech et al. [50] respectively represent text and video into a feature vector, and compute their similarity via cosine similarity metric. Croitoru et al. [52] use distillation techniques to combine knowledge from multiple pretrained models, enhancing cross-modal similarity learning. Current mainstream methods involve learning multiple shared spaces, with relevance computed as a weighted sum of similarities across these spaces. Li et al. [54] employe a multi-space, multi-loss learning framework, enhancing cross-modal similarity by leveraging complementary encoders. While these techniques assess the whole text-video similarity, TSG-RF requires learning partial relevance, where the text relates to only some frames within the video. For non-retrievable samples, the text is irrelevant to any frame. Therefore, the focus is on learning partial relevance.3 The Proposed Method 
problem definition and overview: 
 TSG-RF task. Given a video V = [v 1 , v 2 , . . . , v n ] and a query text Q = [q 1 , q 2 , . . . , q m ] as inputs, the TSG-RF task is asked to predict relevance feedback on whether exist query-related content in the given video. Besides, if the query has relevance with the video, it is asked to conduct grounding to identify the precise boundary indices of video segments that are semantically consistent with the query text in the video V . Otherwise, the grounding results are ignored. Compared to the traditional TSG task that always predicts grounding results for all videos, this TSG-RF task is more challenging as it requires not only distinguishing the videos containing and not containing query-relevant segments but also generating explicit non-existence signals for non-relevant videos besides the grounding results.Overall framework. Figure 2 illustrates the framework of our proposed RaTSG network for the TSG-RF task. Given an input video V and a query text Q encoded by the attention-based mechanisms, the multi-granularity relevance discriminator first captures fine-grained and coarse-grained semanticaware relevance between the video and query text at the frame and segment levels. Subsequently, it integrates the relevance determination capabilities of these two granularities to generate the relevance discrimination score P v for distinguishing whether the video has a semantic-related content of the query. Then, a relation-aware segment grounding module takes the multimodal enhanced video features as the basic representation for generating the grounding probability distribution P s of the start boundary and P e of the end boundary, while exploiting relevance feedback signal as additional input to produce the separate final results of irrelevant and relevant segments. In the following sections, we will provide detailed illustration of each component.Given an input video and a query text, we first use a pre-trained video feature extractor, such as I3D [55] , to extract the video feature sequenceV = [v 1 , v 2 , . . . , v n ] ∈ R n×dv, where d v is the feature dimension. Subsequently, the query text features are extracted using a text feature extractor, like GloVe [56] , producing word embeddings Q = [q 1 , q 2 , . . . , q m ] ∈ R m×dq , where d q represents the feature dimension. Both the video feature sequence V and the query text embeddings Q are first mapped into the same dimension d through a fully connected layer respectively, resulting in V ∈ R n×d and Q ∈ R m×d . To enhance the video semantic with the query contexts, we utilize an attention mechanism to integrate word-wise query features Q with the frame-wise video feature V for generating text-guided enhanced video feature V q [13] . To comprehend the whole semantic of the query for latter global-level reasoning, we use a self-attention layer to perform the intra-modal information interaction on the text feature sequences Q to obtain the sentence-level query representation h Q . The final contextual multi-modal representation V ′ q can be obtained by concatenating the frame-wise video feature V q with sentence-level query feature h Q . More details can be found in Appendix A. The overall framework of our proposed RaTSG for addressing TSG-RF task. It mainly depends on a multi-granularity relevance discriminator that is employed to learn query-video relevance, and a relation-aware segment grounding module that is used to selectively perform grounding.Before generating the grounding results for each query-video pair, we need to first investigate the semantic relevance between them. To determine whether a given video contains segments relevant to a query text, the key lies in assessing the existence of relevance between the text semantic and the video content. Here, we delve into capturing the multi-grained query-video relevance by matching the global sentence-level textual contexts with frame-level and video-level video semantics, respectively.Frame-level relevance discriminator. Determining frame-level query-video relevance can be treated as a foreground-background frame discrimination problem. Theoretically, for a video containing query-related content, the foreground frames located within the start and end index boundaries should exhibit strong relevance to the query text. Conversely, for a video lacking relevant segments, no foreground frames in the video should relate to the text. To detect foreground frames from the query text, we design a foreground frame detector that finely learns the similarity between the text and each video frame. Specifically, we feed the fused multi-modal feature sequence V ′ q into a fully connected layer to predict a sequence of query-to-frame similarity scores. These scores are then mapped to a range between 0 and 1 using the sigmoid function, resulting in the foreground frame prediction score sequence S f = sigmoid(FC(V ′ q )) ∈ R n where each score indicates the confidence of a frame belonging to the foreground. To train the frame-level relevance discriminator, a binary cross-entropy loss L f rame (Appendix A.2) is employed on each frame, which encourages the discriminator to output large scores for foreground frames while small scores for background frames.To obtain the relevance between text and video at a fine-grained frame-level, we apply the concept of multiple instance learning [17] [18] [19] . To be specific, the video is treated as a bag with video frames as instances, categorized into foreground (positive) and background (negative) frames. Thus, a video containing at least one foreground frame is considered a positive bag, indicating partial relevance to the query text. In contrast, a video composed entirely of background frames is deemed a negative bag, showing no relevance. Following [20] , we utilize a max-pooling operator to transfer the instance scores to bag scores, obtaining the fine-grained query-video discrimination score as score f g = max(S f ). The low fine-grained discrimination score suggests that there are no foreground frames in the video, while a high score indicates the presence of at least one foreground frame.Video-level relevance discriminator. In addition to capturing the frame-level relevance, we also attempt to perceive the semantic relevance from a more informationally complete video level. To generate the comprehensive video-level visual representation, we need to aggregate all query-related frame-wise features into a global one to determine its semantic similarity with the sentence contexts. Specifically, we apply a frame-wise weighted sum operation on the text-guided enhanced video feature V q to obtain the query-related video representation h V ∈ R d , where we assign weights to each frame by normalizing the previously obtained frame-level similarity scores (reflect the relative relevance score between the text and each frame) using the softmax function. Then, we fuse the sentence-level textual feature h Q with the global video feature h v to obtain the video-level relation signal vector g, which is then input into a fully connected layer with a sigmoid function to obtain the final video-level similarity score score cg = sigmoid(FC(g)), which globally describes the relevance between the text and the entire video. To train the video-level relevance discriminator, a binary crossentropy loss L video (Appendix A.3) is employed on each query-video sample, which encourages the discriminator output large scores for samples with grounding results while small scores for that without grounding results.Multi-level relevance prediction. The previously discussed two relevance discriminators assess the relevance between query and video at the frame and video levels respectively, yielding fine-grained relevance score (score f g ) and coarse-grained relevance score (score cg ). Considering that both granularity levels are deemed equally important for determining the relevance between text and video, we calculate the average of both fine-grained and coarse-grained relevance scores as the final multi-granularity relevance score P v as:P v = (score f g + score cg )/2.(1)After obtaining the query-video relevance score, we aim to design a flexible segment grounding module to selectively predict the start and end boundaries of the target segments, according to the presence or absence of query-related segments in videos. However, existing segment grounding heads designed for TSG tasks [13, 15, 57, 16] are not suitable for the TSG-RF task. They always generate high-confidence start/end boundaries of segments even when there are no relevant segments in the video. To this end, we introduce a relation-aware segment grounding module that incorporates previously obtained relevance feedback during the segment grounding process, which accommodates the possibility that there may be no relevant segments within the video.Specifically, to address situations where the video contains no relevant segments, we prepend a special token to the general video feature sequence, representing index 0. This token serves as both the starting and ending boundary indices for samples that do not have grounding results, indicating that the query text does not align with any start and end boundaries in the given video. Therefore, it is essential to introduce a contextual relevance-aware representation as the special feature token to ensure that the segment predictor can effectively handle the absence of relevant segments in the video. Considering the video-level relation signal vector g obtained from the video-level relevance discriminator contains enough relevance-aware knowledge to determine whether the segment exists or not, we utilize the video-level relation signal vector g as the special token in this segment predictor. Consequently, the range of boundary indices for relevant segment grounding is [1, n] . Thus, as shown in Figure 2 , if a sample is determined to lack relevant segments, the ground-truth boundary index is assigned as A = [0, 0]; if a sample is deemed to have grounding results, the ground-truth boundary index is defined as A = [a s , a e ], where 1 ≤ a s < a e ≤ n.To predict the probability distributions of start and end boundaries, following [13] , we use two unidirectional LSTMs with two feed-forward layers. The boundary prediction loss function L boundary is represented as:EQUATIONwhere Y s and Y e are the one-hot vector representations of the ground-truth start (a s ) and end (a e ) boundary indices, respectively. P s and P e indicate the predicted start and end boundary probability distribution, respectively.In summary, the total training loss can be defined as:EQUATIONwhere β and γ are hyperparameters used to balance the importance of the three losses.During the inference phase, the multi-granularity relevance score P v is first compared with a threshold m = 0.5 to determine whether query-related content exists or not in the given video:EQUATIONFor samples judged to have grounding results, the model predicts the start (â s ) and end (â e ) boundary indices of the target segment. Specifically, we first compute the joint probability distribution according to the start and the end boundary probability distributions P s , P e generated by the relation-aware segment grounding module. The predicted start (â s ) and end (â e ) boundary indices are obtained by maximizing the joint probability:(â s , âe ) = arg max âs,âeEQUATION4 Evaluation 
experimental setup: 
 Datasets. To verify the viability of our proposed model for TSG-RF, samples without grounding results where the video does not contain the content relevant to the query are required. Therefore, we reconstruct two commonly used datasets, Charades-STA [22] and ActivityNet Captions [23] , by adding samples without grounding results (Details of how we obtain them are illustrated in the Appendix A.5.). For each sample in both validation and test sets, we add a corresponding sample without grounding result, resulting in 1:1 ratio of samples with and without grounding results.The Charades-STA [22] For ease of reference, we name the corresponding reconstructed datasets as Charades-RF and ActivityNet-RF, respectively.Performance metric. As our proposed TSG-RF task requires models to provide relevance feedback indicating samples with or without grounding results, we use accuracy ("Acc") to measure the ability of relevance feedback. For measuring the grounding ability, referring to the previous TSG works [13, 43] , we utilize the "R{n}@{m}" and "mIoU". "R{n}@{m}" indicates the percentage of query texts in the top-n segments where at least one instance has an Intersection over Union (IoU) greater than m. "mIoU" represents the mean IoU across all test samples. It is worth noting that due to the inclusion of samples with no grounding results, we redefine the calculation of Intersection over Union (IoU) in four specific scenarios: (1) When the sample is predicted to have no grounding results but actually has grounding results, the IoU is set to 0. (2) When both the prediction and the ground truth indicate that the sample has no grounding results, the IoU is set to 1. (3) When the sample is predicted to have grounding results but actually has no grounding results, the IoU is set to 0. (4) When both the prediction and the ground truth indicate that the sample has grounding results, the IoU is calculated as the Intersection over Union between the predicted segment and the ground truth.Implementation details. The implementation details are presented in Appendix A.6.As models specifically designed for TSG-RF are non-existing, we compare our proposed model with models targeted at conventional TSG, and adapt them to TSG-RF by adding an extra relevance discriminator for relevance feedback. Specifically, we select four traditional TSG models, including VSLNet [13] , SeqPAN [14] , EAMAT [15] , ADPN [16] , UniVTG [49] , QD-DETR [48] considering their source code are released thus ensuring fair and replicable comparisons. For the relevance discriminator, we utilize a binary classification model which is trained separately to assess the relevance between the query text and the video (For the details of the relevance discriminator, please refer to Appendix B.1). Then, query-video samples identified as relevant by the discriminator are processed using models designed for the TSG task to predict the target segment. For ease of reference, we denote the corresponding models associated with an extra relevance discriminator as VSLNet ++ , SeqPAN ++ , EAMAT ++ , ADPN ++ , UniVTG ++ , and QD-DETR ++ respectively.Table 1 summarizes the performance comparison and model parameters on the Charades-RF and ActivityNet-RF dataset. It is worth noting that traditional TSG models lack the ability to discriminate relevance. These models assume all samples have grounding results, making them incapable of correctly handling samples without grounding results. Hence, this often leads to mismatched grounding prediction, resulting in low recall and mIoU performance for TSG-RF. Additionally, since the test set has an equal ratio (1:1) of samples with and without grounding results, the relevance prediction accuracy of these models is only 50%. Additionally, the enhanced versions of these baseline models, namely VSLNet ++ , SeqPAN ++ , EAMAT ++ , ADPN ++ , UniVTG ++ , and QD-DETR ++ , include a relevance discriminator, which results in consistent performance gains compared to their counterparts without the discriminator. However, these enhanced models require separate and independent training of the relevance discriminator and video grounding components, leading to redundant use of computational resources and increased model size. In contrast, our proposed RaTSG model provides a more lightweight and comprehensive solution by integrating the discrimination and grounding modules, achieving the best performance.To validate the effectiveness of the multi-granularity relevance discriminator, we compare it with degraded models that use only coarse-grained or fine-grained discrimination scores. As shown in Table 2 , single-granularity models fail to perceive the various degrees of partial relevance between text and video, therefore performing worse than the multi-granularity one.To assess the viability of the relation-aware segment grounding module, we conduct ablations on it by replacing the relation signal vector g with a randomly initialized one. As shown in Table 3 , the relation-aware segment grounding consistently outperforms the random counterpart. The result demonstrates the benefit of using previously predicted relevance information of each query-video pair as prior knowledge for TSG-RF. Additionally, in order to further explore how relation features are learned, we visualize the relation signal feature g by t-SNE [58] in Figure 3 . It is demonstrated that dots with the same color are relatively more clustered than those with different colors. Overall, the result shows the good discrimination ability of the learned relation feature in our model for relevant and irrelevant samples.Our proposed RaTSG is a unified model for both relevance discrimination and segment grounding tasks, trained in a multi-task learning manner. To explore whether the two tasks mutually enhance each other, we evaluate the counterparts with the corresponding segment grounding module or relevance discriminator removed. As shown in Table 4 , removing the segment grounding module influences the performance of relevance discrimination. Similarly, as shown in Table 5 , removing the relevance discriminator degrades the performance of segment grounding. The results allow us to conclude that the relevance discriminator and the segment grounding in our model mutually enhance each other, which also demonstrates the effectiveness of our unified dual-branch framework design.To further investigate the scalability of our proposed multi-granularity relevance discriminator and relation-aware segment grounding modules, we also conduct experiments to explore whether they can be adapted for traditional TSG methods to enable them to work for the TSG-RF task. Specifically, we integrate these components into an existing traditional TSG model, EAMAT, for comparison. As shown in Table 6 , EAMAT with our devised multi-granularity relevance discriminator and relationaware segment grounding outperforms the original EAMAT and the enhanced EAMAT ++ with an extra relevance discriminator. These results demonstrate the adaptability and effectiveness of our proposed modules for enhancing traditional TSG methods to work for the TSG-RF task.Figure 4 visualizes grounding examples obtained by our proposed RaTSG and VSLNet ++ . The results show that our RaTSG effectively handles samples with and without grounding results. For the first sample with grounding results, RaTSG localizes video segments more accurately than VSLNet ++ . Furthermore, foreground frame prediction scores obtained by RaTSG are more reasonable. We attribute it to the fact that RaTSG is trained using samples without grounding results, allowing the model to learn the similarity characteristics between background frames and the text, thus enhancing its ability to distinguish foreground and background frames in videos.Query Text: The man put the pillow down. There is no relevant video segment Query Text: A person opening a laptop. There is no relevant video segment Query Text: A person opening a laptop. There is no relevant video segment For the second sample without grounding results in Figure 4 , RaTSG consistently predicts low foreground frame scores, providing accurate relevance feedback. Besides, the relationaware segment grounding module assigns a high probability of the special index of 0, indicating no grounding result. In contrast, VSLNet ++ provides incorrect relevance feedback for this sample, showing high foreground frame prediction scores and resulting in incorrect grounding segments.In Figure 5 , we illustrate two bad examples to discuss the limitations of our model. In the first example, our model incorrectly judges the relevance feedback due to the lack of audio cues which are crucial for identifying the action of sneezing. In the second example, our model misinterprets the temporal sequence of actions, mistaking the closing action for the opening action. These examples demonstrate that our proposed model struggles with handling audio-related actions and temporal-sensitive content. However, such limitation can be alleviated by integrating audio features and temporal modeling in the video representation moudel.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: The paper fully discloses all the necessary information to reproduce the main experimental results, ensuring that the main claims and conclusions can be independently verified. 
conclusions: 
 This paper breaks through the bottleneck of existing TSG works that can not handle the cases where query-related segments do not exist, by introducing a more realistic and natural task, i.e., TSG-RF. By incorporating the multi-granularity relevance discriminator into the segment grounding model with further mutual enhancement designs, our proposed method can effectively and efficiently localize precise video segments that closely match the query text for videos containing relevant content, while providing clear feedback indicating the absence of related segments for videos not containing relevant content. Besides, two constructed datasets for TSG-RF are contributed.the learning of the rrame-level relevance discriminator. This supervision ensures that the discriminator correctly learns the differences between foreground frames and background frames. The loss function is defined as follows:EQUATIONwhere y f i is the i-th element of the binary sequence Y f , and s f i is the i-th element of the foreground frame prediction score sequence S f , with n being the length of sequence. The binary sequence Y f is constructed as follows: for a video feature sequence of length n, as and ae denote the start and end boundary indices of the target segment, respectively. Frames within these indices are labeled as 1 (indicating foreground frames), while those outside are labeled as 0 (indicating background frames).Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes]Justification: The code and data have been submitted as supplementary material in ZIP format.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: In Appendix A.6, we provide the type of compute workers (GPUs) and the size of memory. Additionally, we provide the training time of our proposed model on each dataset.Guidelines:• The answer NA means that the paper does not include experiments.• Datasets that have been scraped from the Internet could pose safety risks. The should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [NA] Justification: paper does not use existing assets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? 
 
 Appendix: The sentence-level query feature hQ and the query-related video representation hV are concatenated to produce a video-level relation signal vector g ∈ R d , represented as g = FC([hQ; hV ]). To train the video-level relevance discriminator and ensure that the video-level relation signal vector accurately learns the semantic relationship between the query text and video segments, this paper introduces a video-level relation constraint L video . Specifically, the video-level relation signal vector g is input into a fully connected layer with a sigmoid function to obtain the video-level similarity score scorecg = sigmoid(FC(g)), which globally describes the relevance between the text and the video. The video-level relation constraint is calculated as:EQUATIONwhere y h is the ground truth label indicating whether a sample has retrievable results or not.Considering the video-level relation signal vector g obtained from the video-level relevance discriminator contains enough relevance-aware knowledge to determine whether the segment exists or not, we utilize the videolevel relation signal vector g as the special token in this segment predictor. Specifically, using the foreground frame prediction score sequence S f produced by the Frame-level relevance discriminator, the foreground frames in the text-guided enhanced video feature Vq are enhanced, while the background features unrelated to the query text are diminished. The foreground-enhanced video feature sequence Ṽq is calculated as:EQUATIONwhere ⊙ denotes element-wise multiplication. Subsequently, the video-level relation signal vector g is concatenated to the front of the foreground-enhanced video feature sequence Ṽq, forming the input feature sequence V g q = [g; Ṽq] for the relation-aware segment predictor, in dimension R (n+1)×d . This sequence is then fed into a stacked two-layer LSTM network. The first layer of the LSTM generates the time-sequential feature sequence for predicting the start boundary H s = [h s 0 , h s 1 , . . . , h s n ] in R (n+1)×d , and the second layer processes this output to generate the end boundary feature sequence H e = [h e 0 , h e 1 , . . . , h e n ] ∈ R (n+1)×d , as follows:EQUATIONwhere hs/e t denotes the t-th time-sequential feature vector in H s/e , and v g q t represents the t-th video feature vector in V g q . LSTM * indicates the corresponding layer of the LSTM network. The start and end time-sequential feature sequences H s and H e are each fed into their respective feedforward networks to obtain the boundary probability distributions P s/e ∈ R (n+1) :EQUATIONwhere Ps represents the start boundary probability distribution, and Pe the end boundary probability distribution.The boundary prediction loss function L boundary is represented as:EQUATIONwhere Ys and Ye are the one-hot vector representations of the ground-truth start (as) and end (ae) boundary indices, respectively. Ps and Pe indicate the predicted start and end boundary probability distribution, respectively.person opened the refrigerator. Query: person the refrigerator. Query:person closed the cupboard. Query: person closed the cupboard. Query: ... ...Sample pair without retrieval results Due to the absence of datasets specifically created for the TSG-RF task, this paper has reconstructed the validation and test sets of two widely used datasets in the TSG domain: Charades-STA [22] and ActivityNet Captions [23] , to construct a testing environment for this task. In the original datasets, a single video corresponds to multiple query texts, as illustrated in Figure 6 , where video V1 corresponds to m query texts with grounding results [S11, S12, . . . , S1m]. For each text with grounding results, a corresponding sample without grounding results (i.e., the video does not contain segments relevant to the query text) is constructed.Specifically, the reconstruction process is exemplified by the sample pair (S11, V1). By randomly selecting another video Vn from the video library to pair with the query text S11, a sample pair without grounding results (S11, Vn) is formed. To ensure the quality of reconstructed sample pairs and remove low-quality ones, this paper utilizes Large Language Models [62] [63] [64] for their deep and precise text modeling capabilities to achieve sample selection. Specifically, the BERT model [62] is used to extract features from the query text. Subsequently, the cosine similarity between the query text S11 and the query texts with grounding results from the randomly selected video [Sn1, . . . , Snm] is calculated. If the maximum similarity is less than a threshold value (empirically set to 0.2), the reconstructed sample pair without grounding results (S11, Vn) is added to the dataset. Otherwise, a new video is randomly selected until the condition is met. This methodology ensures the relevance and integrity of the dataset for accurately simulating the TSG-RF task.As shown in Table 7 , we name the corresponding reconstructed datasets as Charades-RF and ActivityNet-RF for the TSG-RF task. In these reconstructed datasets, we added an equal number of samples without grounding results to the validation and test sets. This augmentation ensures that the datasets more accurately reflect real-world scenarios where not all query texts correspond to relevant video segments.Each word in the query text is initialized using GloVe300d, which remains frozen during training. Visual features of videos are extracted using a pre-trained I3D network. The maximum video feature sequence length is set to 128. Sequences longer than this are uniformly downsampled to 128, while shorter sequences are zero-padded to the same length. During training, since the training set was not reconstructed and only contains samples with results, each batch includes randomly selected videos paired with the original query text to create samples without grounding results. In Equation 3, we empirically set β = 6 and γ = 6 to balance all loss functions at the start of training. For the threshold m in Equation 4, the value providing the highest accuracy on the validation set is chosen: 0.5 for Charades-RF and 0.3 for ActivityNet-RF. All experiments are conducted on a workstation with an NVIDIA GeForce RTX 3090Ti GPU and 256G RAM. Training our proposed model on the Charades-RF dataset takes approximately 2 hours, while training it on the ActivityNet-RF dataset takes approximately 5 hours.Since there are currently no models specifically designed for the TSG-RF task, we select representative opensource models developed for the TSG task in recent years as baseline models to ensure fair and replicable comparisons, including VSLNet [13] , SeqPAN [14] , EAMAT [15] , ADPN [16] , UniVTG [49] , QD-DETR [48] . While these baseline models demonstrate strong video segment grounding capabilities, they lack the ability to discern the relevance between the video and text in input samples, and thus assume that all samples contain retrievable results. Considering that directly training an additional relevance determination model is the most straightforward and efficient strategy to enable the above baseline models to discern relevance, this paper constructs a simple relevance determination model. By integrating this independently trained relevance determination model with the aforementioned baseline models, we compensate for their lack of relevance discernment capability, forming enhanced baseline models: VSLNet ++ , SeqPAN ++ , EAMAT ++ , ADPN ++ , UniVTG ++ , and QD-DETR ++ . Figure 7 illustrates the architecture of the simple relevance discrimination model. Specifically, the query text and video are processed through separate feature encoders. The encoded features of text and video are then concatenated along the sequence length dimension, with a CLS token added at the beginning to aid in learning the relevance between text and video. This concatenated sequence is subsequently input into a Transformer encoder to facilitate the interaction of information between feature vectors. Finally, the CLS token from the input sequence is processed through a fully connected layer to obtain the case discrimination score Pv.Encoder To further explore model performance under unbalanced data conditions, the number of samples without grounding results in the test set was randomly decreased in proportion, yielding six groups of test sets with ratios of non-grounding to retrieval samples at 0, 0.2, 0.4, 0.6, 0.8, and 1.0, respectively. Observing Figure 8 from left to right, when the proportion of non-grounding samples is 0, the test set contains only grounding-result samples, testing all models on their TSG task performance. Starting from a 0.2 ratio, all models are assessed on their TSG-RF task performance. As the proportion of non-grounding samples in the test set increases, although there is a general improvement in the mIoU of all models, the increase exhibited by RaTSG is the most significant. This is attributed to RaTSG's multi-scale learning of the semantic relationships between text and video, effectively integrating relevance feedback during the segment grounding process. Particularly in test sets with a high proportion of non-grounding RaTSG demonstrates superior performance compared to other models, showcasing its exceptional ability in case discrimination and fine-grained grounding. It is important to note that our model is specifically designed for the TSG-RF task, which includes the challenge of handling samples without grounding results. This specialized focus may slightly affect its performance on the standard TSG task, yet RaTSG remains highly competitive. This demonstrates the robustness and versatility of our approach, indicating its strong potential in more complex real-world scenarios. R1@0.3 R1@0.5 R1@0.7 mIoU R1@0.3 R1@0.5 R1@0. As shown in Table 9 , we also try to compare our designed relevance discriminator with large-scale pre-trained vision-language models due to their strong cross-modal representation capabilities. To implement the comparison, we choose the CLIP model, a large-scale pre-trained vision-language model, to assess the relevance between the query text and video. Specifically, we first utilize the CLIP to measure the cosine similarity score between the query text and each frame of a video, and then aggregate the similarity scores over all frames to obtain the final relevance. In particular, we use two aggregate methods implemented by the CLIP model: (1) averaging all scores (CLIP-Avg) and ( 2) averaging on top-5 scores (CLIP-Top5). We incorporate CLIP with the SOTA method EAMAT to achieve the joint relevance feedback and grounding framework. The experimental results on the Charades-RF dataset are summarized in Table 9 . We can find that CLIP-based EAMAT models achieve relatively lower performance, demonstrating that the CLIP model has a poor ability to predict relevance. We attribute it the fact that the CLIP model is simply pre-trained on common scenarios without additional fine-tuning on specific downstream task, thus severely suffering from domain shift issues on the target dataset. • The proofs can either appear in main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 


Decision: Initially, the reviewers raise some questions regarding the way to incorporate foreground/background info, the model performance in a standard setting, comparison with another video temporal task-highlight detection, incremental designs, dataset and baseline usage, the lack of adequate qualitative analysis, etc. Most of these questions are addressed in the rebuttal and recognized by reviewers. Eventually with the merit of the work, this paper receives one strong accept, one accept, and two borderline accepts, with agreement for acceptance. The AC recommends to accept. Authors are encouraged to revise the paper according to the reviews.

Reviews: review: paper_summary: paper_summary: The work develops a model that localizes query-related segments when present and provides feedback on non-existence when absent.
It proposes a Relation-aware Temporal Sentence Grounding (RaTSG) network, which reformulates TSG-RF as a foreground-background detection problem.
Also, the work uses a multi-granularity relevance discriminator for precise video-query relevance feedback.
The work constructs two popular TSG datasets for TSG-RF benchmarking.
Experimental results demonstrate RaTSG's effectiveness for the TSG-RF task
strength_weakness: strength_weakness: 1. The work claimed to incorporate foreground/background info. However, it was simply adding a binary classification, such binary foreground classification for temporal grounding was already explored in [1]. The video part is done by weighted sum, which is also a standard way of aggregating frame features.
2. In Table 1, the author explained why the performance is 50%. Why are the other rows all have 71.94% of accuracy and 81.6% accuracy?
3. Although this work proposes a new RF setting, it will be interesting to know how the model performs in a standard setting to see if there is a performance gap.



[1] Lin, Kevin Qinghong, et al. "Univtg: Towards unified video-language temporal grounding." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
questions: questions: Please address the questions in weakness.
limitations: limitations: None
paper_summary: paper_summary: This paper introduces Temporal Sentence Grounding with Relevance Feedback (TSG-RF) in videos, a new task that addresses the limitations of traditional Temporal Sentence Grounding (TSG), which assumes relevant segments always exist within a video. TSG-RF accounts for the possibility that a video may not include a segment related to the query, aiming to localize segments that align with the query when present and provide feedback when they are absent. The proposed Relation-aware Temporal Sentence Grounding (RaTSG) network reformulates TSG-RF as a foreground-background detection problem, assessing query-related semantics at both frame and video levels. It utilizes a multi-granularity relevance discriminator for precise relevance feedback and a relation-aware segment grounding module to adaptively ground segments. To validate RaTSG, two popular TSG datasets are reconstructed, establishing a benchmark for TSG-RF. Experimental results demonstrate the effectiveness of RaTSG for this task.
strength_weakness: strength_weakness: - The key motivation of this paper is related to another video temporal task -- highlight detection. However, this paper do not have any discussion.
- The method is cumbersome, featuring several incremental designs.
- The experiments utilize outdated datasets. Newer datasets, such as QV-Highlights, should be considered.
- The baselines are no up to date which do not include sota methods. such as UMT, UniVTG, QD-DETR.
questions: questions: Please refer to the Weaknesses section.
limitations: limitations: The Relevance feedback in Video Moment Retrieval is a trivial research question. Either we could generalize this problem to general video grounding (spatial / temporal) or general video understanding (not just VMR, but also hallucinations in VLMs).
paper_summary: paper_summary: This paper presents a novel task named Temporal Sentence Grounding with Relevance Feedback (TSG-RF) to overcome the limitations of conventional Temporal Sentence Grounding (TSG), which presumes the existence of relevant segments in videos. This paper introduces the Relation-aware Temporal Sentence Grounding (RaTSG) network, incorporating a multi-granularity relevance discriminator and a relation-aware segment grounding module to generate relevance feedback and segment boundaries. The RaTSG network's efficacy is demonstrated by reconstructing two widely-used datasets for TSG-RF and validating its performance through extensive experiments. Additionally, the source code for RaTSG is made available.
strength_weakness: strength_weakness: 1.The weaknesses of this paper include a lack of sufficient baselines for comparison with the RaTSG network. For instance, the paper does not investigate the use of Large Multimodal Models (LMMs) to assess the relevance between the query text and video, which could offer a more robust benchmark for evaluating the proposed method. Incorporating such comparisons would enhance the overall evaluation of the RaTSG network.
2.The paper also lacks adequate qualitative analysis. Including more qualitative examples of both successful and unsuccessful cases would provide deeper insights into the strengths and limitations of the RaTSG network.
questions: questions: Please refer to the weakness.
limitations: limitations: The paper does not discuss the limitations of the work. However, the introduction of the novel TSG-RF task significantly improves upon the limitations of traditional TSG task, offering positive societal impact. For constructive suggestions on areas needing improvement, please refer to the weaknesses.
paper_summary: paper_summary: This paper proposes a new task called Temporal Sentence Grounding with Relevance Feedback (TSG-RF) in videos, which extends the traditional Temporal Sentence Grounding (TSG) by introducing cross-modal video-text semantic relevance prediction. Besides, a new Relation-aware Temporal Sentence Grounding (RaTSG) network is proposed, where a multi-granularity relevance discriminator and a relation-aware segment grounding module are specifically devised for the TSG-RF. Quantitative and qualitative experiments conducted on two reconstructed datasets demonstrate the effectiveness of our proposed RaTSG for the new TSG-RF task.
strength_weakness: strength_weakness: The paper should illustrate bad examples in the visualization section, and discuss the limitations of the proposed method.
questions: questions: 1.	The proposed relation-aware segment predictor is key for TSG-RF. Could it be adapted for traditional TSG methods to enable them to work for TSG-RF?
2.	As the proposed method requires additional relevance feedback, I wonder how the extra computational workload would be included.
limitations: limitations: Although the proposed method could perform extra relevance feedback, its complexity may be increased when compared to traditional TSG methods.

Comments: title: Post response
Thanks very mauch. We will update our paper accordlingly.
Thanks for your recommendation.
Thanks for you suggestion, and we will update our paper accordlingly.
Thank you for raising the score.
Post response
Post response
comment: Thanks for the response, the author have addressed my concerns.
Thanks for your suggestion, we will include Figure 4 by adding a bad example and updating the discussion accordingly in the revision, and will add the results of Q2 in the appendix. Thanks again for your positive review.
Thanks for your time, and we greatly appreciate your recommendation on our paper.
Many thanks for sharing the insight with us. We will utilize QV-Highlights in our follow-up works. Additionally, we promise that we will incorporate the discussion on highlight detection and appropriately cite relevant papers in the related work. Besides, we will update Table 1 by including UniVTG and QD-DETR for a more comprehensive comparison.
Thank you for raising the score. We also appreciate your detailed review with constructive feedback.
I have read the other reviewer's comments and author's reply, and consider the paper meet the standards required for NeurIPS. I recommend accepting the paper.
My concerns have been addressed. Thanks for the effort from the authors. Lastly, I recommend including Figure 4 by adding a bad example in the revision, and adding the results of Q2 in the appendix. Considering that the paper presents a new interesting and practical task, introduces a simple yet effective method, and provides informative open-source code, I would like to increase my final rating as a strong accept.
Thanks for the rebuttal!

The response to Q2 and Q3 addressed my concerns. I'll raise my score accordingly.
Thank you to the authors for their rebuttal and hard work.

Regarding Q1 on highlight detection: The explanation that RF setting is specifically designed to handle cases where no relevant segments are found makes sense.

For Q2: QVHL is distinct because it includes one or more ground-truth intervals with high-quality resolutions, making it more reliable and challenging compared to Charades-STA and ActivityNet, which many past methods have already overfitted on.

I appreciate the addition of several baseline comparisons in the experiments. Please ensure the discussion on highlight detection is included in the related work and add the baseline (UMT, UniVTG, etc) comparison in the experiments, as omitting this would be disrespectful to the work in this community.

Considering these points, I would like to increase my rating to borderline accept.