Explicit Granularity and Implicit Scale Correspondence Learning for Point-Supervised Video Moment Localization

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-03 07:53:26

Abstract: Video moment localization (VML) aims to identify the temporal boundary of the target moment semantically matching the given query. Existing approaches fall into three paradigms: fully-supervised, weakly-supervised, and point-supervised. Compared to other two paradigms, point-supervised VML strikes a balance between localization accuracy and annotation cost. However, it is still in its infancy due to the following two challenges: explicit granularity alignment and implicit scale perception, especially when facing complex cross-modal correspondences. To this end, we propose a Semantic Granularity and Scale Correspondence Integration (SG-SCI) framework aimed at modeling the semantic alignment between video and text, leveraging limited single-frame annotation information for correspondence learning. It explicitly models semantic relations of different feature granularities and adaptively mines the implicit semantic scale, thereby enhancing and utilizing modal feature representations of varying granularities and scales. SG-SCI employs a granularity correspondence alignment module to align semantic information by leveraging latent prior knowledge. Then we develop a scale correspondence learning strategy to identify and address semantic scale differences. Extensive comparison experiments, ablation studies, and necessary hyperparameter analyses on benchmark datasets have demonstrated the promising performance of our model over several state-of-the-art competitors.

Main paper: 
 introduction: 
 Video moment localization (VML), which refers to localizing a visual moment corresponding to a given textual query, is a fundamental task in video understanding. It benefits many important application scenarios, such as multimedia retrieval [45, 48] and smart human-computer interaction [37, 43] .Current efforts [14, 44, 50, 53] focus on addressing both fully and weakly supervised VML paradigms. As shown in Figure 1 (a), the fully-supervised method requires the ground-truth moment annotations for training, which is laborious and time-consuming to obtain [29] . The weakly-supervised method is more flexible because it does not require moment annotations, but results in a significant drop in performance [10] . To balance accuracy with annotation cost, the point-supervised paradigm is proposed [21, 25] . Pointsupervised methods require a single frame annotation from the ground truth, which is more practical and flexible compared to fully supervised annotation, costing only 1/6 as much [24] .Prior research [2, 4, 18, 35] , to enhance video-language comprehension, can be classified into two primary approaches: granularity modeling and scale modeling. Granularity modeling involves delineating relationships among various features (such as frame-word and moment-sentence) through the explicit boundaries present in video-language, facilitating the differentiation of data across varying entity granularities. In contrast, scale modeling derives from the intricate semantic logics and inherent hierarchical structures within textual and visual domains. This approach acknowledges implicit semantic scale in video and language interaction that exists within an alternative modal space.To be specific, as shown in Figure 1 (b), existing point-supervised labeling inevitably introduces two accompanied challenges: 1) Explicit Granularity Alignment. According to [10, 12] , on the one hand, single-frame annotations are randomized within intervals, causing clear mismatches at a fine-grained level (frame-word); on the other hand, the incomplete information on interval annotations hinders the direct use of correspondences at a coarse-grained level (moment-query). Undoubtedly, this challenge poses a significant obstacle to effective language and visual alignment. 2) Implicit Scale Perception. Prior studies [9, 16] note that query sentences for the same video may vary in semantic scale. This issue is especially noticeable in the point-supervised approach. These methods struggle to capture temporal relationships within a video because of using only single-frame annotations, leading to a lack of effective contextual modeling constraints. As a result, the essential semantic scale information remains unmodeled, impeding temporal learning and video comprehension.To navigate these challenges, we introduce a novel Semantic Granularity and Scale Correspondence Integration (SG-SCI) framework that integrates a Granularity Correspondence Alignment (GCA) module and a Scale Correspondence Learning (SCL) strategy. Firstly, the GCA module is engineered to enhance the interaction of videolanguage. By employing a fine-grained alignment approach, it establishes a more detailed and comprehensive mapping between video frames and textual descriptions. This module not only facilitates a deeper understanding of the video content but also ensures that even the less prominent frames find relevance in the corresponding textual narrative. Secondly, the SCL strategy addresses the disparity in temporal scale. It is designed to learn latent semantics adaptively across varying temporal scales, enabling the model to assimilate and correlate extensive video sequences with succinct textual queries. This strategy ensures a more robust and contextually aware matching process, thereby enhancing the accuracy and adaptability. Consequently, this framework offers a more nuanced and adaptable solution for point-supervised VML.To the best of our knowledge, it is the first work on integrating explicit granularity alignment and implicit scale perception into point-supervised VML. Our approach effectively mitigates the granularity and scale-related challenges by the synergy of the innovative component and strategy. It boosts the precision of aligning video moments to text queries and enhances the model's robustness.In summary, the main contributions are as follows:• Model Contribution. We introduce an innovative Granular Correspondence Alignment module. Specifically, it is designed to improve the explicit correspondence relation across varying granularities among different modalities. [12] and TACoS [33] , validate the effectiveness and superiority of our model. The codes and settings are released at https://anonymous.4open.science/r/SG-SCI. 
related work 2.1 video-language modeling: 
 Current research can be divided into two main categories depending on the use of a modeling approach: feature granularity modeling and semantic scale modeling. In terms of entities, the former is explicit while the latter is implicit.Early explicit modeling approaches [1, 12] mainly used global sentence-level alignment to enhance the semantics of visual features, but this approach overlooked fine-grained semantics such as words. As a result, several studies [4, 35, 49] have investigated various word-frame interactions using attention mechanisms to capture the relationships between visual cues and textual queries. Recent research [38, 47, 52] has recognized the significance of local phrase patterns or tokens in sentences for video moment retrieval.Compared to explicit modeling, implicit scale modeling [2, 11, 18, 19, 39] utilizes multi-scale relations to achieve better grounding results. These relations mainly exist at the video-level and languagelevel. However, all grounding methods overlook the possibility of different semantic scales within the same level (e.g., sentence level). Accordingly, DualMIL [9] extends multiple instance learning into a two-level framework.Although the aforementioned modeling strategies are effective, most methods do not fully utilize their potential when dealing with limited supervisory information. Therefore, our focus is on extracting both implicit and explicit information to accurately locate boundaries in point-supervised VML.In the domain of video-language learning, the misalignment between textual descriptions and corresponding visual content introduces significant challenges, necessitating the adoption of correspondence learning as an innovative approach [15, 17, 20] . This paradigm shift is epitomized by the introduction of MIL-NCE [31] , which pioneers the strategy of aligning video clips with adjacent sentences to diminish the effects of misalignment. In contrast, Tang et al. [40] improves video description quality by integrating an external image captioning model, emphasizing data-driven enhancement over immediate error correction.Our method differs itself from preceding endeavors through two principal innovations. Firstly, we go beyond the traditional focus on either model architectures or supervisory strategies by enhancing improvements through the synergistic interaction between model and strategy learning. Secondly, we expand the use of correspondence learning beyond segmented multi-modal data. This pioneering adaptation allows for precise moment localization with point-supervised for the first time.Different from existing fully and weakly-supervised VML studies [12, 14, 27, 30, 44, 50, 53] , point-supervised VML strikes a delicate balance between annotation efforts and model performance by leveraging a single frame from the localization moment [10, 21, 25] . Compared to fully-supervised VML, point-supervised significantly reduces the cost of annotation data and it provides more comprehensive information than weakly supervised learning. By eliminating the need for precise start and end timestamps for the target moment, a quick "glance" at the video and the selection of a single frame are often sufficient, making point-supervised methods applicable to scenarios with incomplete annotation information [46] . The concept of point-supervised was initially introduced by Bearman et al. [3] in the context of semantic segmentation tasks and Cui et al. [10] applied the concept of point-supervised to VML and introduced the ViGA model, which aligns Gaussian distributions generated from supervised frame positions with cross-modal attention. Subsequent works have built upon ViGA's foundation, D3G [25] is proposed to align features of sentence-moment pairs and dynamically mitigate annotation bias using Gaussian distributions, while CFMR [21] develops a concept-driven multi-modal alignment mechanism to circumvent the need for cross-modal interaction modules during the inference process. Despite efforts to enhance retrieval performance, existing work still falls short in effectively modeling the semantic granularity relationship between two modalities. Furthermore, due to the constraints of the supervised information, it struggles to capture the overall action changes within a moment.𝑠 !! 𝑠 !" 𝑠 #! 𝑠 #" 𝑠 $! 𝑠 $" 𝑠 !% 𝑠 #% 𝑠 $% … … … … … … … … … … … 
methodology: 
 This section commences by defining the problem and outlining the core pipeline of point-supervised video moment localization. After this, we will explore the insights behind our innovative Granularity Correspondence Alignment method, highlighting its main elements. Finally, we conclude with an exposition of our Scale Correspondence Learning strategy, showing its significance in our research. Figure 2 presents an overview of our proposed SG-SCI.In this work, we aim to address the challenge of cross-modal video localization under point-supervised. The dataset comprises a series of quadruples and is denoted as D, which is expressed as follows,EQUATIONwhere V 𝑖 represents an untrimmed video, Q 𝑖 denotes the associated query sentence, and 𝜏 𝑠 𝑖 , 𝜏 𝑒 𝑖 are the start and end timestamps respectively. The target video moment aligns with the query Q 𝑖 .Contrasting with a fully-supervised paradigm, our training is based on a collection of triplet annotations, symbolized asA = {(V 𝑖 , Q 𝑖 , 𝜏 𝑚 𝑖 ) | 𝑖 = 1 to 𝑁 },where 𝜏 𝑚 𝑖 signifies a randomly selected point within the interval defined by 𝜏 𝑠 𝑖 and 𝜏 𝑒 𝑖 . During the inference phase, the objective is to precisely localize the relevant video moment for each query within the dataset D.To fulfil the task of Point-supervised VML, we first introduce the pipeline which can be categorized into three parts: Featurn Extraction, Multi-modal Interaction, and Moment Localization.Before multi-modal interaction, we extract the visual and textual features, and then incorporate learnable positional embedding into the extracted features.Visual Representation. Given an untrimmed video V, our approach utilizes a pre-trained CNN, i.e., C3D [41] or I3D [6] , to extract visual features to ensure fair comparison. The extracted features are then processed through a fully-connected (FC) layer, obtaining a visual representation denoted as 𝑉 = {𝑣 1 , 𝑣 2 , . . . , 𝑣 𝑇 }, where 𝑇 represents the number of frames sampled at intervals in the video and 𝑣 𝑖 denotes the 𝑖-th frame.To perceive the time of the video, we incorporate learnable Positional Embedding (PE) into the model and get enhanced visual representation V ∈ R 𝑇 ×𝑑 , where 𝑑 represents the dimension of the visual representation.Text Representation. For a given query Q, we employ the pretrained Glove model [32] to extract textual features. A bi-directional gated recurrent unit (Bi-GRU) is employed to obtain the latent sequential order of original sentence, and get new textual representation 𝑄 = {𝑞 1 , 𝑞 2 , . . . , 𝑞 𝐿 }, where 𝐿 indicating the number of words in the query. Similar to the visual representation, PE is also added to the text representation, obtaining the enhanced features Q ∈ R 𝐿×𝑑 .To capture the intra-model semantics, we utilize V and Q through the self-attention layer [42] to generate the intra-modal representation V and Q. Similarly, we use the cross-modal multi-head attention to obtain the cross-modalEQUATIONSubsequently, to serve the subsequent modules, we take out the last layer of cross-modal attention scores as the cross-modal attention scores 𝐴 𝑞→𝑣 and 𝐴 𝑣→𝑞 , which will be further illustrated in 3.3. We input the features and attention scores into our proposed granularity correspondence alignment module to obtain the multigranularity representation Ṽ𝑓 and Q𝑤 :EQUATIONwhere GCA(•) is the granularity correspondence alignment module. So far, we have obtained the multi-granularity representation Ṽ𝑓 and Q𝑤 via multi-modal interaction. The representation is compound of multi-level granularities information, assisting crossmodal semantic alignment. In order to integrate the semantic information of the whole sentence, we use the max-pooling function F(•) to obtain the sentence representation Q𝑠 as follows,EQUATIONMoment localization aims to locate the start and end timestamps based on the obtained representation.In the training stage, unlike fully-supervised scenario, a random point within the target video moment is accessed while it is absent during inference. Because of the divergence between the two stages, we will elaborate on both of them subsequently.Training Stage. In the training process, we use sliding windows to slice Ṽ𝑓 , and generate candidate moments. Gaussian distribution is utilized to measure the moments' temporal features and cosine similarity of semantic features. The temporal information is defined as follows,EQUATION)where 𝑖 is the 𝑖-th frame, 𝜏 𝑚 is the index of the supervised frame, 𝜎 is a hyperparameter, and 𝐿 𝑣 is the length of the video. The semantic information 𝑆 𝑖 is the cosine similarity between Ṽ𝑓 (𝑖) and Q𝑠 . These temporal and semantic analyses are integrated and applied in our scale correspondence learning strategy, detailed in Sec. 3.4.Inference Stage. The inference involves two primary steps, consisting of identifying the key point that best matches query and expand from this point to get the predicted video moment most similar to the query.In this section, we introduce a novel granularity correspondence alignment module within the framework of point-supervised VML, which is designed to adaptively regulate the granularity relationship between different modalities. By exploring semantic relationships across modalities, the module enriches the original modal representation and supplements existing semantic alignment with information across various granularities, ensuring symmetry in intermodal interactions.Unlike existing methods that unconsciously interact the smallest granularity units (frames and words) in representations across modalities, our main insights derive from exploiting the attentionperceived matrix and utilizing it to capture potential cross-modal semantic granularity information. Specifically, we use the matrix consisting of cross-modal attention scores as a granularity perception matrix as follows,EQUATIONwhere 𝐴 𝑞→𝑣 ∈ R 𝑇 ×𝐿 is the cross-modal attention score matrix, 𝑇 and 𝐿 are the number of frames and words, respectively, and 𝑠 𝑖 𝑗 is the attention score between the 𝑖-th frame and the 𝑗-th word. The differences of scores in each column represent how much attention the word pays to the video frame. Therefore, we consider using max function to aggregate the information in each row to obtain the potential prior distribution Ṽ𝑝 ∈ R 𝑇 of the complete query with respect to the video frames, formulated as,EQUATION)Next, we calculate the cosine similarity between the query and the original video frames to obtain the prior distribution of the complete query with respect to the video frames,EQUATIONwhere 𝑐𝑜𝑠 (•) is the cosine similarity function. Subsequently, we use the finest granularity information from attentional perception and pooled global granularity information, which are averaged and used as adaptive visual granularity perceptual features Ṽ𝑐 = 1 2 ( Ṽ𝑔 + Ṽ𝑝 ), In this way, we can obtain the visual granularity perception vector Ṽ𝑐 , which can be used to enhance the original visual representation. On this basis, we use the obtained features to remodel the crossmodal fusion with text feature, Ṽ𝑎 = Ṽ𝑐 ⊙ Ṽ𝑓 .After reaggregation, we integrate self-attention-based fine-grained features with multi-scale visual perception features. This enhances fine-grained entity alignment and focuses on overall perception, thereby improving representation. We then apply these features for cross-modal semantic interactions, using prior semantic knowledge to refine visual-word correlations. Finally, we use the weighted mean approach in order to control the original modal feature information, thus preventing noise interference due to excessive introduction of cross-modal information as follows,EQUATION)where 𝛼 is the mean weighted factor. We use the same strategy applied to another modality. Therefore, we can obtain the final crossmodal representation Ṽ𝑓 and Q𝑤 via the multi-modal interaction.To improve the model's understanding ability under different scales of moments, we utilize potential frame-moment correspondence semantic information, as the single-frame supervised information alone is not sufficient for effective moment semantics. To be specific, the target of optimization consists of three parts, the first part models the semantic and temporal information of the fused features from a global perspective. The second part uses point annotation to compare the differences between potential positive and negative samples in different intervals to effectively capture the information about the change of actions within the intervals. The third part aims to exploit the single point information in the global moment that is most similar to the query, which in turn enriches the priori knowledge and enhances the guidance of cross-modal semantics.3.4.1 Global Alignment Loss. We take the cross-entropy loss to force the representation information we get to be close to the information provided by the supervised frames in terms of temporal distance and semantic distance,EQUATIONwhere 𝐺 𝑖 is the Gaussian distribution weight of the 𝑖-th frame, and 𝑆 𝑖 is the semantic similarity score of the 𝑖-th frame.Loss. Due to lacking of boundary annotations, a single point's information inadequately represents the interval matching the query. This makes us to consider using video moments that encompass point annotation as potential positives. We construct binary labels to facilitate interval perception: moments containing the point annotation are labeled 1, while others are labeled 0, as follows,EQUATIONwhere M 𝑖 is the 𝑖-th moment by sliding window. Due to the latent differences in temporal information of these moments, we introduce Gaussian distribution weights to further construct the soft labels ỹ𝑖 , which can be formulated as:EQUATIONwhere W (•) = 𝐺 (𝑠) * 𝐺 (𝑒), 𝐺 (𝑠) and 𝐺 (𝑒) are the Gaussian distribution weights at the beginning and end of M 𝑖 , respectively. Since it is inevitable that there will be multiple positive sample moments belonging to a query, it is inappropriate to view similarity score learning as a 1-in-N classification problem with cross-entropy loss.To this end, we utilize the Kullback-Leibler divergence to construct frame-moment correspondence loss 𝐿, as follows,EQUATIONwhere S 𝑖 is the cosine similarity calculated by moment M 𝑖 and query feature Q𝑠 .In the beginning of location process, we focus on identifying a point that align most closely with the query, enabling our model to discern and emphasize subtle points' differences. We separate essential information from positive sample moments, concentrating on feature variances. Specifically, we extract detailed point data reflecting semantic labels across moments, leveraging this to enrich point annotation with inherent disparities. This process can be represented as follows,EQUATIONwhere 𝑠 and 𝑒 are the start and end index of the moment M 𝑖 .For label construction, we use both hard sample labels and soft labels with Gaussian weights, just as the Eqn. ( 12) and (13) . We then employ KL divergence to analyze differences between positive and negative samples, using these insights to refine our alignment process, as follows,EQUATION)By combining the global loss 𝐿 𝑔 and correspondence loss 𝐿 𝑐 , we obtain the final loss for model optimization, as follows,EQUATION)where 𝛽 and 𝛾 are used to balance the focus between moments and frames, avoiding the model over underscore the sample information. 
experiments: 
 This section will begin by presenting the experimental settings. Subsequently, we will conduct comparative experiments, comprehensive ablation studies, and evaluate the effectiveness of SG-SCI to answer the following three research questions (RQs):• RQ1: Is our proposed SG-SCI able to outperform several stateof-the-art competitors on VML? • RQ2: Is each component of our SG-SCI helpful for boosting the localization performance? • RQ3: How do hyperparameters affect model capability?The experiments that follow are conducted on two benchmark datasets: Charades-STA [12] and TACoS [33] .Charades-STA [12] : It is constructed on the Charades dataset [36] and contains 16,128 "moment-query" pairs with an average video duration of 30 seconds. Following the standard split strategy [12] , we divided the dataset into 12,408 and 3,720 "moment-query" pairs for training and testing, respectively.TACoS [33] : It is built upon MPII Cooking Compositive dataset [34] and only covers cooking activities that contain pairs of queries with very similar visual information. It consists of 127 untrimmed videos with the average duration of 320 seconds and 18,818 queries.In particular, we use point-supervised information instead of boundary supervision information in the original dataset setting, just like existing effort [10] , where the point-supervised information comes from a random point in the boundary.Following the existing work [10, 21, 25] , we choose 𝑅𝑛@𝑚 and mean averaged IoU (mIoU) as protocols to evaluate the performance of moment localization. To be specific, 𝑅𝑛@𝑚 refers to the percentage of predictions for which the temporal Intersection over Union (IoU) surpasses the thresholds 𝑚 within the top-𝑛 of the sorted results, and mean averaged IoU (mIoU) represents the average IoU across all test samples. Since our method localizes the target moment with the highest coordinate probability, we set 𝑛 = 1 and 𝑚 ∈ {0.3, 0.5, 0.7}. The higher 𝑅𝑛@𝑚 and mIoU are better.In our work, we employ the pretrained I3D [6] and C3D [41] network to extract visual features from Charades-STA [12] and TACoS [33] respectively. Consistent with prior research [10] , we keep the word embedding module fixed and utilize the 840B GloVe [32] to construct a comprehensive vocabulary. The model dimension 𝑑 𝑚𝑜𝑑𝑒𝑙 is set to 512, and we employ AdamW [28] with a learning rate of 1𝑒 -4 , which decays by half when reaching a plateau during training. By default, we set 𝛼 = 0.3, 𝛽 = 𝛾 = 0.1, and 𝜎 = 0.04 in both datasets. The batch sizes for the Charades-STA and TACoS are empirically set to 512 and 64, respectively. All experiments are conducted on a NVIDIA GeForce RTX 4090 with 24GB memory.In this subsection, we compare the proposed method with different kinds of state-of-the-art methods, including fully-supervised methods (2D-TAN [51] , SS [11] , FVMR [13] , ADPN [7] , MS-DETR [22] ), weakly-supervised methods (SCN [26] , CNM [54] , CWG [8] , CPL [55] , IRON [5] , PPS [23] ) and point-supervised methods (ViGA [10] , PSVTG [46] , CFMR [21] , D3G [25] ). To ensure the validity of the comparison, the majority of the selected methods are drawn from the last three years. The comparisons on Charades-STA and TACoS are encapsulated in Table 1 and Table 2 respectively. Based on the data presented, the following observations can be derived:• Our method mines more correspondence information compared with existing point-supervised methods. While traditional point-supervised methods focus on the interaction of sentence-moments [25] , our method involves multi-granularity interactions among complex video, concise textual descriptions, and point annotation. Therefore, the model can consider the interaction between sentences and point annotation at multiple granularities during the training process, resulting in better crossmodal information interaction. As illustrated in Table 1 , in the Charades-STA dataset, SG-SCI achieves 4.65% and 6.77% higher than CFMR [21] and D3G [25] in 𝑅1@0.7, respectively. • SG-SCI reduces the performance disparity between fullysupervised methods and point-supervised methods. Lacking of annotated timestamps, the positioning accuracy of pointsupervised methods is lower compared to existing supervised learning methods. By learning features across different scales, SG-SCI exploits the information contained in the supervised frames, to some extent compensating for the performance gap caused by different supervised information. In the Charades-STA dataset, our method even outperforms 2D-TAN [51] , fully-supervised method, by 1.45% in terms of 𝑅1@0.5. • Trivial annotation cost contributes to localization. The existence of frame annotation provides the model with a clear direction during iterative convergence, and our proposed correspondence module enables the model to continuously converge during gradient backpropagation based on the multi-granularity information from query and frame annotation. As evidenced in Table 1 and Table 2 , our approach consistently outperforms the cutting-edge weakly-supervised models. Global F-M M-F R1@0.3 R1@0.5 R1@0. • SG-SCI can handle more challenging task. In contrast to Charades-STA, TACoS contains longer videos with shorter retrieved moments, and each frame within the video exhibits a high degree of semantic approximation, demanding a heightened ability from the model to perceive boundaries and differentiate similar moments. Consequently, existing weakly-supervised methods may struggle to address this scenario, resulting in a significant absence of the 𝑅1@0.7 metric in Table 2 . However, compared to existing point-supervised methods, like ViGA [10] and D3G [25] , our approach exhibits great performance in 𝑅1@0.7.To demonstrate each component of SG-SCI, we conducted extensive ablation experiments on Charades-STA.• Impact of GCA and SCL. To discuss whether the proposed GCA and SCL can achieve better performance, we simplify the pipeline of SG-SCI and set it as the baseline. Specifically, after multi-modal interaction, we train the model by calculating the KL divergence of the similarity distribution between the query and each frame with a Gaussian distribution. To demonstrate the effectiveness of SCL, we conduct experiments by setting the soft labels to 1. From the Figure 3 (a), it can be seen that both GCA and SCL effectively enhance the localization ability of the baseline, especially in 𝑅1@0.7. • Effect of multiple granularities in GCA. As described in Sec 3.3, GCA not only helps to understand the video contents more deeply but also ensures that less significant frames have relevance in the corresponding textual narrative. We validate the effectiveness of multi-granularity information interaction on cross-modal representation, with specific results shown in Figure 3 (b). It can be seen that the multi-granularity interaction proposed in GCA has effectively improved the localization effect by enhancing single-modal representation. At the same time, increasing the multi-granularity interaction of queries can enhance the model's ability to perceive fine-grained boundaries, i.e., improve the 𝑅1@0.7 by 6.10%. • Effectiveness of different loss. The final loss contains three parts: global loss, frame-moment correspondence loss and momentframe correspondence loss. To evaluate the effectiveness of them, we remove some of them for comparison and the results are shown in Table 3 , and we find that scale correspondence learning can inspire the potential of the model for boundary perception.In the presence of global loss, the frame-moment correspondence loss and moment-frame correspondence loss alone can enhance performance. The combination of the two can produce comprehensive performance improvement, and the 𝑅1@0.7 even gets more than 10% improvement than only using global loss, showing that scale correspondence learning can enhance the ability to capture key information and boundary perception. Query: She picks up the rinds and throws them away. {0.05, 0.1, 0.2, 0.5, 1}, and display the numerical relationship between 𝑅1@0.7, as shown in Figure 4 . Similar to the 𝛼 in GCA, 𝛽 and 𝛾 also show an overall trend of first increasing and then decreasing, and the highest value is obtained at 𝛽 = 𝛾 = 0.1. The experimental results indicate that frame-moment correspondence loss and moment-frame correspondence loss have a significant impact on fine-grained boundary perception.• Sliding Window Size. Sliding window is designed to capture multi-granularity features via multi-scale sliding windows. In fact, monotonically increasing or decreasing the size of the sliding window would lead to degradation of the representation discrimination, thus affecting overall performance. For instance, in extreme cases, the sliding window may encompass information from either the entire video or just a frame. In such scenarios the model struggles to learn and comprehend these moments holistically, leading to a decline in overall localization performance. We conducted ablation experiments on the Charades-STA dataset, with a default step size of 4. The specific experimental results are shown in Table 4 . The results indicate that the best performance is achieved when the sliding window size is 8, indicating that the model needs a moderate sliding window size for sufficient learning during Scale Correspondence Learning. Size R1@0.3 R1@0.5 R1@0. . This is because our proposed GCA and SCL do not introduce significant computational complexity. As a result, we consider our methods to be competitive in terms of efficiency.  
conclusion and future work: 
 In this paper, we designed a new framework to understand the correspondence connections between video and text. This framework used a limited amount of data from single frame to learn. Initially, we looked at the problems existing methods had in linking the meaning of video and text that varied in detail. To address these issues, our method included two main strategies: (1) creating a model that used underlying knowledge to connect features of different modalities, and (2) developing a learning strategy that focused on the differences in information between similar samples. These strategies helped our method to clearly define and find the differences in meaning across various details and modalities. This made our method better at representing features of explicit granularity and implicit scales. Experiments conducted on two benchmark datasets demonstrated the effectiveness of our proposed framework.In the future, in order to further improve the performance of VML, we plan to introduce a more advanced visual-language transformer backbone. In addition, inspired by the fully-supervised paradigm, we plan to study the impact of different inference methods on localization performance.  
 
 Appendix:  


Decision: This paper receives one weak accept, three borderline accepts, and one borderline reject initially. The reviewers raise some questions regarding the approach novelty and the experiment results (Sub-item indicator is not SOTA). Most of these questions are addressed in the rebuttal and recognized by reviewers. Eventually with the merit of the work, this paper receives one weak accept, three borderline accepts, and one borderline accept (initial score without final update), with agreement for acceptance. Considering that most sub-item indicators are SOTA with only one exception, AC recommended to accept. Authors are encouraged to revise the paper according to the reviews, and add an explanation/analysis for the underperformed sub-item indicator.

Reviews: review: paper_summary: paper_summary: This work proposes an explicit granularity and implicit scale correspondence learning framework for point-supervised video moment localization. It aims to model the semantic alignment between video and text.
strength_weakness: strength_weakness: 1. The basic idea, explicitly modeling semantic relations of different feature granularities and adaptively mining the implicit semantic scale is good.
2. The experimental results look good.
limitations: limitations: I cannot see the significant weakness of this work. I will just provide some minor suggestions:
1. Figure 2 should clearly demonstrate the interaction between model and strategy learning, to support the claim in the introduction.
2. While Figure 5 qualitatively shows the advantage of the proposed method, it would be better to show some failure cases and analyze the limitations.
3. Some relevant works should be discussed, e.g.
Luo et al., Soc: Semantic-assisted object cluster for referring video object segmentation, NIPS,2023
paper_summary: paper_summary: This paper proposes a granular correspondence alignment module to improve the correspondence relation across varying granularities and proposes a scale correspondence learning strategy. Extensive experiments are constructed to verify the effectiveness of the proposed methods.
strength_weakness: strength_weakness: 1. The writing is easy to follow.
2. Extensive experiments demonstrate the effectiveness of the proposed methods.
3. The proposed methods are interesting.
limitations: limitations: 1. The framework shown in Figure 2 seems to be complex.
2. The author should construct more ablation studies to compare the proposed methods (correspondence alignment module and the scale correspondence learning strategy) with other related methods.
3. It is suggested to construct the computational cost experiments.
paper_summary: paper_summary: The paper introduces the SG-SCI framework, which significantly improves point-supervised VML.

Key contributions:

	1.	Granularity Correspondence Alignment (GCA): Enhances semantic alignment between video frames and text descriptions.
	2.	Scale Correspondence Learning (SCL): Adapts to varying temporal scales, improving query matching accuracy.
	3.	Framework Design: Combines feature extraction, multi-modal interaction, and moment localization using sliding windows and Gaussian distribution.

Validated on Charades-STA and TACoS datasets, SG-SCI outperforms existing methods, showcasing its effectiveness in point-supervised video moment localization. This paper makes a notable contribution by addressing granularity alignment and scale perception challenges.
strength_weakness: strength_weakness: 1. The paper does not use very advanced feature extractors such as CLIP, big foundation models (LLaVA, etc) but still achieves  SOTA results. 
2. Achieve SOTA results and surpass the previous work by a large margin on a complex dataset. 
3. The pipeline is somewhat novel.
limitations: limitations: 1.	The overall approach lacks novelty. The proposed granularity correspondence alignment essentially employs attention mechanisms and cosine similarity, which are well-established techniques.
2.	The paper presents results primarily on two datasets. However, on the Charades-STA dataset, under the point-supervised setting, it fails to achieve state-of-the-art (SOTA) results for R1@0.3. Moreover, the second-best result cited is from 2022, indicating that the current SOTA result for this dataset dates back two years.
3.	Despite conducting extensive analytical experiments, the paper does not perform sufficient experiments on a diverse range of datasets to robustly demonstrate the generalizability and effectiveness of its main findings.
paper_summary: paper_summary: The paper introduces the SG-SCI framework for point-supervised Video Moment Localization (VML), addressing challenges in explicit granularity alignment and implicit scale perception. It uses a Granularity Correspondence Alignment module and a Scale Correspondence Learning strategy to improve the alignment between video frames and textual descriptions, enhancing model performance for VML tasks. Experiments show the framework's effectiveness over existing methods.
strength_weakness: strength_weakness: 1. The paper is well-written and well-motivated.
2. The solution to implicit scale perception and explicit granularity alignment is neat and novel.
3. The ablation study is thorough and the performance gain is significant shown in Table 3.
limitations: limitations: 1. I recommend to simplify Figure 2. Current version is too implicit and not easy to follow the core idea of this paper. 
2. Is there any study to test the robustness of the proposed method? Since the L159 claims that the robustness is enhanced.
paper_summary: paper_summary: This paper introduces framework to understand the correspondence connections between video and text. It models semantic relations of different feature granularities and adaptively mines the implicit semantic scale to enhance modal feature representations.
strength_weakness: strength_weakness: The paper is well organized and easy to follow. The motivation is clear.
limitations: limitations: 1. The proposed method contains several modules and involves several hyper-parameters, increasing the difficulty of the learning process.

2. The time consumption is just a relative value and I worry about whether this method can be applied in practice.

3. Why the Scale Correspondence Learning works? 

4. I notice that, in Tab. 1 and Tab. 2, SG-SCI achieves good performance except the R1@0.3 in Tab. 1. It doesn't seem to make sense.