Maskable Retentive Network for Video Moment Retrieval

Authors: jingjing  hu
xun  yang
meng 2024 wang
dan  guo
kun  li
zhan  si
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-06 03:03:40

Abstract: Video Moment Retrieval (MR) tasks involve predicting the moment described by a given natural language or spoken language query in an untrimmed video. In this paper, we propose a novel Maskable Retentive Network (MRNet) to address two key challenges in MR tasks: cross-modal guidance and video sequence modeling. Our approach introduces a new retention mechanism into the multimodal Transformer architecture, incorporating modality-specific attention modes. Specifically, we employ the Unlimited Attention for language-related attention regions to maximize cross-modal mutual guidance. Then, we introduce the Maskable Retention for video-only attention region to enhance video sequence modeling, that is, recognizing two crucial characteristics of video sequences: 1) bidirectional, decaying, and non-linear temporal associations between video clips, and 2) sparse associations of key information semantically related to the query. We propose a bidirectional decay retention mask to explicitly model temporal-distant context dependencies of video sequences, along with a learnable sparse retention mask to adaptively capture strong associations relevant to the target event. Extensive experiments conducted on five popular benchmarks ActivityNet Captions, TACoS, Charades-STA, ActivityNet Speech, and QVHighlights for MR tasks demonstrate the significant improvements achieved by our method over existing approaches. Code is available at https://github.com/xian-sh/MRNet.

Main paper: 
 introduction: 
 As an important application of multimedia and multimodal retrieval, query-guided video Moment Retrieval (MR) [8, 53] has attracted great attention from the research community in recent years [1, 5, 8, 12, 16-18, 22, 50, 57, 58, 63, 66, 67] . The MR task is initially proposed with the text as the query to retrieve the target moment within an untrimmed video, and this task setting still dominates the field of moment retrieval, which is called NLMR (Natural Language Moment Retrieval). As speech technology evolves, Xia et al. [53] explore the possibility of retrieving video with audio query and propose the ActivityNet Speech dataset and a new MR task, namely SLMR (Spoken Language Moment Retrieval). Fig. 1 shows their example, given a sentence query "the man then rides the bike and it moves across the snow" or an audio query, the goal of MR tasks is to find the moment (80.75s-147.34s) that is semantically described by the query. Recently, some MR-related multi-task work such as moment retrieval and highlight detection (MR+HD) task has also been explored [14, 29] , where HD refers to additionally determining the highlightness (or saliency) score of each clip in the predicted moments. Although these MR tasks are set up differently, they share some key challenges, i.e., how to improve self-modal context modeling (especially "visual contextual association learning") and "cross-modal guidance" for better video content understanding and semantic alignment between language and video.For the above two challenges, we rethink the MR tasks and highlight the following three characteristics: (C1) Temporal association between video clips. Existing works mostly emphasize some kind of relational modeling in terms of temporal sequence, e.g., near-neighbor, recursive, local and global [9, 25, 30, 38] . However, as shown in Fig. 1 , gradually decaying correlations are bidirectional along the timeline, these correlations are complex and non-linear [3] , which may not be adequately expressed for example by weighting techniques like attention [47] . (C2) Redundant background interference. The background contains a lot of redundant information (Fig. 1 ) that can interfere with the recognition of the current event, and this redundancy is even worse in long videos, resulting in the key information semantically related to the target event being sparsely associated. Thus, additional de-redundancy strategies are important to highlight the key information in a video and improve the quality of moment retrieval. (C3) Specific crossmodal attention constraints. Since the video moment retrieval is a typical cross-modal task with the language query as guidance, and inspired by the success of Transformers [47] in cross-modal tasks [4, 6, 20, 35, 39, [70] [71] [72] [73] , many recent MR methods [14, 29, [62] [63] [64] adopt the Transformers to achieve the "cross-modal guidance" with no difference in the attention modes. However, we will no longer treat the different modalities indiscriminately, we impose the C1 & C2 constraints on video-only attention region, but not on the other query-related regions, so as to improve visual modeling and maximize linguistic guidance. Concretely, in this work, we attempt to solve the "cross-modal guidance" challenge based on C3 and the "video sequence modeling" challenge based on C1 & C2.In this paper, we propose a new Maskable Retentive Network (MRNet) for MR tasks. The overview of MRNet is shown in Fig. 3 . We first introduce a new retention mechanism [45] into the multimodal Transformer architecture. We divide the language and video attention map into four regions: A (𝑞 → 𝑞), A (𝑞 → 𝑣), A (𝑣 → 𝑞) and A (𝑣 → 𝑣), so that we can perform modality-specific operations based on the properties of the modality itself (C3): (1) As the linguistic semantics are highly condensed and in order to better maximize the guiding role of language, we set the Unlimited Attention mode for the language-related regions A (𝑞 → 𝑞), A (𝑞 → 𝑣) and A (𝑣 → 𝑞), allowing queries can utilize visual details without restriction, and each video clip can response to each query, thus achieving cross-modal mutual guidance; (2) As for the pure video branch A (𝑣 → 𝑣), we propose a new Maskable Retention mechanism to model two characteristics of video sequences (C1 & C2) together. Specifically, we firstly optimize the original retention mechanism [45] as a bidirectional decay mask along the timeline to control the range within which the current video sequence token can be reviewed before and after with gradually fading attention, which can achieve the bidirectional recurrent sequence modeling effect. Second, in the video branch A (𝑣 → 𝑣) of 𝑇 × 𝑇 size, a large number of redundant responses between the video clips serving as background may interfere with the highlighting of the target moment regions, and cause the key useful responses to be sparse, so we further propose a learnable sparse retention mask to adaptively highlight the strong responses and learn the key specific semantic associations related to the target event. In addition, to ensure the sparsity of the video attention, we design a self-supervised sparsity constraint loss to remove redundant responses.To sum up, our proposed MRNet has parallel training capability like Transformers and aggregates bidirectional recursive reasoning capability like RNNs, enhanced visual modeling methodologies (like maskable retention, bidirectional decay, learnable sparse retention, nonlinearity, etc.). Moreover, our proposed architecture is applicable to NLMR, SLMR and MR+HD tasks and can be implemented in either single or multi-query training modes. In the validation of our experiments, the effect is maximized during multi-query training. The performance of our model is not only good, but also achieves a trade-off between model size and accuracy, as shown in Fig. 2 .The main contributions of this work can be summarized as follows: (1) We propose a new Maskable Retentive Network (MRNet) for three MR tasks (NLMR, SLMR and MR+HD) from a new perspective of deep video sequence modeling. It is a new attempt to introduce the retention mechanism into the multimodal Transformer framework for video sequence modeling, which is proven to have a significant performance advantage over other methods, and the architectural advantages such as scalability, recurrent sequential reasoning and sparsity. (2) We innovatively implement modality-specific attention modes, set Unlimited Attention for language-related attention regions to maximize cross-modal mutual guidance, and propose a new Maskable Retention for pure video branch for enhanced video sequence modeling. (3) We propose core video modeling methods: 1) bidirectional retention mask expresses temporal associations of video sequences (bidirectional, decaying, non-linear), 2) learnable sparse retention mask adaptively learns the strong associative responses related to the target event, and a new self-supervised loss is designed to constrain the sparsity of video attention region and remove redundant associations. (4) Extensive experiments are conducted on five benchmarks (ActivityNet [47] in both natural language processing and computer vision [2, 6, 11, 34, 39, 48, 69] , many effective transformer-based vision modeling strategies for MR tasks have been proposed [14, 16, 25, 29, 37, 49, 53, 63, 64] . Zhang et al. [64] adopt the visual language transformer encoder to jointly learn finegrained associations between words and video clips, Xia et al. [53] propose the video-guided audio transformer decoding representation pre-training, Lei et al. [14] construct the Moment-DETR framework using transformer-based detection pipeline. The transformerbased moment retrieval structure enjoys efficient training and high performance. Recently, there have been some approaches such as RWKV [32] and RetNet [45] that have started to explore the advantages of combining Transformers and RNNs to improve the temporal inference capability of the model. In this paper, we attempt to introduce a new retention mechanism into transformer for MR tasks. The new retention mechanism has the strengths for video sequence modeling such as parallel training like Transformers, bidirectional recurrent reasoning via the design of a new bidirectional decay mask, enhanced association learning through a learnable sparse retention mask and the nonlinearity regulation by updating the gate operation of retention.… … q 1 q 2 … q M v 1 v 2 v 3 v 4 … v 
our method 3.1 preliminary: 
 3.1.1 Problem Definition. Given an untrimmed video consisting of a sequence of 𝑇 video clips, let its corresponding pre-extracted features beV = {𝑣 𝑖 } 𝑇 𝑖=1 ∈ R 𝑇 ×𝑑 𝑣, where 𝑑 𝑣 is the clip feature dimension, and Q = {𝑞 𝑖 } 𝑀 𝑖=1 ∈ R 𝑀 ×𝑑 𝑞 be a query set consisting of a number of 𝑀 language queries with the 𝑑 𝑞 -dim feature. The goal of both MR tasks NLMR (natural language moment retrieval) and SLMR (spoken language moment retrieval) is to predict the temporal boundaries (𝜏 𝑠𝑡𝑎𝑟𝑡 , 𝜏 𝑒𝑛𝑑 ) of target moment described by a given query 𝑞 (text or audio modality).3.1.2 Feature Encoding. We linearly convert each query token and video clip token into a unified feature space. Thus, we obtain the query and video features Q ∈ R 𝑀 ×𝑑 and V ∈ R 𝑇 ×𝑑 , and represent their concatenated long input sequence as Q||V, where || denotes matrix concatenation operation. Considering the differences between video and language modalities, we independently add positional embedding for two modalities. Rotary position embedding method xPos [46] has decomposability and can better express the relative positional relationships of query and video sequences [42] . So we use it to encode two modalities:EQUATIONwhere Θ 𝑞/𝑣 is the complex conjugate of Θ 𝑞/𝑣 = 𝑒 𝑖𝜃 [46] respectively, and 𝑖 is the index of sequence elements.In this section, we conduct modality-specific attention modes, that is, we set Unlimited Attention for language-related attention regions to maximize cross-modal mutual guidance, and perform a new Maskable Retention for video branch A (𝑣 → 𝑣) for enhanced video sequence modeling. The core video modeling methods (Bidirectional Retention Mask, Learnable Sparse Retention Mask and self-supervised sparse constraint loss) in Maskable Retention will be described as follows.3.2.1 Video-Language Attention. We follow the calculation of videolanguage attention A ∈ R (𝑀+𝑇 ) × (𝑀+𝑇 ) in transformer. Eq. 2 divides the attention matrix into four blocks: A 𝑀 ×𝑀 , A 𝑀 ×𝑇 , A 𝑇 ×𝑀 and A 𝑇 ×𝑇 , which represents four attention regions: A (𝑞 → 𝑞), A (𝑞 → 𝑣), A (𝑣 → 𝑞) and A (𝑣 → 𝑣). We adopt different attention modes for different attention regions based on the characteristics of the modality itself. That is, as videos often contain repetitive or closely similar visual images and sparsely response to the query, we apply the Maskable Retention in the visual branch A (𝑣 → 𝑣) to remove redundant vision information and model retentive reasoning in video sequences and enhance temporal context association in the video. For query branch (other three attention regions), as the query semantics is relatively dense and doesn't reach a redundancy bound (each word in the queries has its own contextual linguistics), we leave them free to the query branch, so that queries and queries, and queries and video clip can be associated with each other without limitation in terms of retention calculation, thereby queries and video clips can provide comprehensive mutual guidance and semantic alignment as shown in Fig. 3 .EQUATIONThe Unlimited Attention for query branch is tend to maximize the "linguistic guidance", and the Maskable Retention for visual branch A (𝑣 → 𝑣) is designed to enhance the "vision modeling". Specifically, the additional optimization strategies Maskable Retention includes two important masking mechanisms, i.e., the Bidirectional Retention Mechanism (expressed by a matrix M 𝑅𝐷 ) and Learnable Sparse Retention Mask (expressed by a matrix M 𝐿𝑆 ). The M 𝑅𝐷 is designed for stronger temporal sequential reasoning and the M 𝐿𝑆 is for highlighting key semantic associations in the video. Next we give the detailed explanations of them.For modeling the refined temporal context association between video sequences, we reconsider that a video is a sequence with retention relationships in time. Inspired by the powerful ability in sequence modeling of Retentive Network (RetNet) for language models [45] , we introduce the retention mechanism into retentive reasoning in video sequence for better visual contextual association learning. Unlike the original Ret-Net that employing the unidirectional (only forward) retention, we incorporate a bidirectional explicit retention decay masking M 𝑅𝐷 for A (𝑣 → 𝑣), so that the current token can bidirectionally review temporal dependencies forward and backward:BiRetention(𝑉 𝑣 ) = (A 𝑇 ×𝑇 ⊙ M 𝑅𝐷 ) • 𝑉 𝑣 ;(3)EQUATIONwhere 𝛾 is retention decay factor (e.g., 𝛾 = 0.98), 𝑛 and 𝑚 are the row and column indices of matrix M 𝑅𝐷 , which combines bidirectional temporal masking and exponential decay along relative distance as shown in Fig. 3 . Based on Eq. 4, we construct gradual decay effect from near to far on the video segment at the current time step and explicitly control the valid receptive field range of before and after the current token by setting a specific 𝛾 value. This is used to perform recurrent retention reasoning in the video to enhance the model's temporally contextual learning ability like RNNs with transformer architecture.Mask. In addition to optimizing the temporal modeling capability of the model, we further optimize the model's ability to associate the semantics of video sequences.And we note the fact that in MR tasks, video sequences often contain a large amount of redundant background information, which can lead to excessive interference responses in the cross-modal attention map. And we argue that the strong responses in the attention map should be sparse, thus we further propose the Learnable Sparse Retention Mask to focus this issue for better "vision modeling".To depict the key semantic associations between video sequence tokens, we innovatively introduce a learnable sparse mask M 𝐿𝑆 that allows the model to focus more on relevant video tokens, learning an adaptive attention map to optimize for task-specific performance improvement. Then, we add a new self-supervised sparse constraint loss L 𝑠𝑝𝑎𝑟𝑠𝑒 for region A (𝑣 → 𝑣) to reduce redundant noise interference within the video and learn more critical and useful information. We propose an auxiliary variable 𝑊 𝐿𝑆 of size 𝑇 ×𝑇 to build M 𝐿𝑆 , randomly initialize and optimize it during training. The calculation method for M 𝐿𝑆 and sparsity constraint loss are:M 𝐿𝑆 = (1 -I 𝑇 ×𝑇 ) • 𝜎 (𝑊 𝐿𝑆 ) + I 𝑇 ×𝑇 ;(5)EQUATIONhere I 𝑇 ×𝑇 is an identity matrix, we adopt the sigmoid function 𝜎 with the threshold of 0.5 to scale 𝑊 𝐿𝑆 into [0, 1], to build a soft mask.Here the mask M 𝐿𝑆 is decomposed into two parts: self-token mask I 𝑇 ×𝑇 and inter-token mask (1 -𝐼 𝑇 ×𝑇 ) • 𝜎 (𝑊 𝐿𝑆 ). The self-token mask is explicit and acts as a diagonal matrix. The minimization of the objective loss L 𝑠𝑝𝑎𝑟𝑠𝑒 of Eq. 6 is mainly imposed on the inter-token mask term. Finally, L 𝑠𝑝𝑎𝑟𝑠𝑒 serves as a regularizer to force the mask sparse, this drives some of the mask values towards 0, resulting in effective noise suppression. Acting M 𝐿𝑆 on video branch A (𝑣 → 𝑣), would force the model to learn a sparse attention map that retains only the key correspondences.Block. We adopt the proposed maskable retention to enhance the video sequence inference ability of the model. With the learned matrices M 𝑅𝐷 and M 𝐿𝑆 , we update the representation of language video attention A:EQUATIONEQUATIONTable 1 : The statistics of three widely used natural language moment retrieval (NLMR) datasets, the recent spoken language moment retrieval (SLMR) dataset, and the recent multi-task of moment retrieval and highlight detection (MR+HD) dataset.Attributes Datasets NLMR (the query is text in English) SLMR (the query is audio in English) MR+HD (multi-task) ANetCap [13] TACoS [36] Charades-STA [8] ANetSpeech [53] QVHighlights [ We set up a 𝑁 -layer MRT block to handle unified language and video sequence input 𝑋 . A single-layer MRT block is defined as:EQUATIONThe enhanced language and video features are obtained after passing through 𝑁 -layer MRT blocks, which are denoted as 𝑋 𝑁 . Then, the language and video are represented as Q 𝑁 = MRT(𝑋 ) [1 : 𝑀; : ] ∈ R 𝑀 ×𝑑 and V 𝑁 = MRT(𝑋 ) [𝑀 + 1 : 𝑀 + 𝑇 ; :] ∈ R 𝑇 ×𝑑 , respectively.we follow the backend of 2D-TAN [66] to output the match score map of candidate moments and queries, with the query feature from the output of MRT block. Specifically, after passing through multiple MRT blocks, we obtaine enhanced language and video features, then we use the video features V 𝑁 ∈ R 𝑇 ×𝑑 to build the 2D candidate moments map. We compute the Hadamard product of each query 𝑞 ∈ Q 𝑁 , Q 𝑁 ∈ R 𝑀 ×𝑑 , and 2D moments map to obtain the matching score map 𝑆 ∈ R 𝑇 ×𝑇 with a total of 𝐶 moment candidates. To supervise the score map, we apply the binary crossentropy loss to regress the IoU score of each moment:EQUATIONWe also adopt contrastive learning [52, 67] as an auxiliary positive and negative samples constraint:EQUATIONwhere Q 𝐵 and V 𝐵 are the sets of queries and videos in a training batch. For 𝑣 𝑞 ∈ {𝑣 + 𝑞 , 𝑣 - 𝑞 }, 𝑣 + 𝑞 is the moment matched to query 𝑞 (solo positive sample), 𝑣 - 𝑞 denotes the moment unmatched to 𝑞 in the training batch (multiple negative sample). 𝑞 𝑣 ∈ {𝑞 + 𝑣 , 𝑞 - 𝑣 }, 𝑞 + 𝑣 is the query matched to moment 𝑣, 𝑞 - 𝑣 denotes the query unmatched to 𝑣 in the training batch. Thus the total training loss isEQUATIONwhere 𝜆 𝑠 and 𝜆 𝑐 are loss weights, and unsupervised loss term L 𝑠𝑝𝑎𝑟𝑠𝑒 we propose is defined in Eq. 6. Following the same dataset split as [8] for fair comparisons, it has 12,408 and 3,720 samples for training and testing, respectively. 
experiments 4.1 datasets and evaluation metrics: 
 To validate the scalability of our model, we also conduct experiments on the recently proposed MR-related datasets. For SLMR task, there is currently only one public ActivityNet Speech (ANet-Speech) [53] . The queries of which are audios in English, obtained by volunteers reading text captions, and its dataset division is consistent with ANetCap dataset. For the MR+HD multi-task, The QVHighlights [14] dataset is recently proposed for joint moment retrieval and highlight detection tasks, it contains 10,148 openworld videos and 10,310 queries, we follow its data splits [14, 29] .Following the convention [8, 30, 66] , we evaluate the performance of Moment Retrieval on two main metrics: 1) Recall: We adopt "R@𝑘, IoU@𝜇" as the evaluation metric, which represents the percentage of top-𝑘 predicted moments whose tIoU (temporal Intersection Over Union) with the ground-truth moment is larger than 𝜇. We consider the following tIoU threshold values 𝜇 = {0.1, 0.3, 0.5, 0.7}. 2) mIoU: We also adopt mIoU as the evaluation metric to count the average value of tIoU of all test samples. Additionally, for QVHighlights dataset, we follow official protocol [14] : mean average precision (mAP) is also a metric for Moment Retrieval (MR) and HIT@1 indicates the hit ratio of the top-scoring clip to metric Highlight Detection (HD) task.For the preliminary extraction of video features, we use the C3D feature on ANetCap, ANetSpeech and TACoS from 2D-TAN [66] , the I3D feature on Charades-STA from [30] , as well as SlowFast/CLIP features on QVHighlights provided by [14] for a fair comparison. We set the number of sampled clips 𝑇 to 64 for ANetCap and Charades-STA, 128 for TACoS, and 75 for QVHighlights. For text features, following previous work [14, 52] , we adopt the GloVe [33] , BERT [39] feature and CLIP [35] feature (only on QVHighlights [14] ). We extract the audio feature of ANetSpeech with the pre-trained Data2vec [2] model.For the back-end decoder setup, we adopt the same 2D proposal generation as 2D-TAN [66] and MMN [52] for the NLMR and SLMR tasks, and employ the same back-end (2-layer decoder and task loss settings) as Moment-DETR [14] and QD-DETR [29] for the joint Table 2 : Evaluation results for ANetCap and TACoS datasets. The best results and second best performance are marked with bold and underline, respectively. We list the main visual processing backbones. "MRNet-S" and "MRNet-M" refer to the model with single-query training mode and multi-query mode, respectively.ANetCap TACoS R@1, IoU@ R@5, IoU@ mIoU R@1, IoU@ R@5, IoU@ mIoU 0. MR+HD task. For training, we control the sparse ratio of M 𝐿𝑆 to 25%, and set the loss weight 𝜆 𝑠 of L 𝑠𝑝𝑎𝑟𝑠𝑒 to 10.0 for all datasets. The Table 5 : Evaluation results for QVHighlights dataset. As one query per video, we only test the single-query training mode.HD R@1, IoU@ mAP >= Very Good @0.5 @0.7 @0.5 @0.75 Avg. mAP HIT@1 MCN [1] 11 contrastive loss weight 𝜆 𝑐 is consistent with MMN [52] , i.e., 0.1 for ANetCap and TACoS, 0.05 for Charades-STA. We use AdamW [28] optimizer with a learning rate 8 × 10 -4 for ANetCap, 1 × 10 -3 for Charades-STA and 15 × 10 -4 for TACoS, and batch size 12 for all datasets. The setup for ANetSpeech is the same as for ANetCap.In MR task, one-to-one (1 query to 1 video) training is known as single-query training [63, 66, 68] , many-toone (many queries to one video) training is known as multi-query training [24, 50, 52, 67] . We test the effect of single-query training and adopt a multi-query training model to maximize the linguistic guidance from queries. Noting that during inference, regardless of any training mode, the evaluation is a fair single query input to predict a uniquely corresponding moment, consistent with the convention of the MR tasks [8, 16, 66] . [53] , when the queries are audios in English. We reproduce the evaluations of the typical MR methods VSLNet [63] and MMN [52] , and compare our model with other state-of-the-art methods VGCL [53] and SIL [51] . The results are shown in Tab. 4, our MRNet-M currently has optimal performances on ANetSpeech dataset, e.g., 34.00 on R@1, IoU@0.7, and 51.31 on mIoU, which has significant performance improvement compared to other MR methods, demonstrating our model's good scalability on MR tasks.QVHighlights for the MR+HD multi-task. We further validate our model on the multi-task MR-related dataset QVHighlights (MR+HD) [14] . This joint task includes both finding relevant moments and predicting the highlighted score of each clip in them.The comparisons with existing works are detailed in Tab. 5. From the results, our model demonstrates performance on par with SOTA models, achieving an 𝑅@1, 𝐼𝑜𝑈 @0.7 of 46.63 for moment retrieval, and a HIT@1 of 63.55 for highlight detection. It further affirms the robust universality of our model on MR+HD tasks.In this part, we evaluate the impact of various factors of the proposed MRNet to answer the following research questions (all experiments are based on MRNet-M): Q1: How do two core mask components M 𝑅𝐷 and M 𝐿𝑆 contribute to the performance of our MRNet? Q2: Is a bidirectional retention more effective than a unidirectional one? Which non-linear operation is more effective: Soft-Max or Swish Gate? Q3: Will the self-supervised sparsity loss be beneficial? How to set this loss's weight 𝜆 𝑠 and the sparsity ratio 𝛼 of the M 𝐿𝑆 to be optimal? Q4: Which masking mode is more effective: only masking video or full masking on the entire cross-modal map? An qualitative case of cross-modal retention map. Q5: The quantitative case analysis of video moment retrieval.4.4.1 Main Module Analysis (Q1). We verify the contributions of the main two mask components M 𝑅𝐷 and M 𝐿𝑆 of our proposed MRNet framework, i.e., bidirectional retention mask module and learnable sparse retention mask module. As shown in Tab. 6, we combine the retentive network backbone and the 2D proposal backend of 2D-TAN [66] as the baseline (Row 1), namely removing M 𝑅𝐷 and M 𝐿𝑆 from the model. In our work, the proposed baseline is a new architecture, it is equipped with RetNet [45] in a crossmodal Transformer architecture, which integrates the strengths such as Transformer (Parallel learning, Eq. 2), RetNet (Rerecursive inference, Eq. 3), nonlinearity regulation (Eq. 8). Compared with the SOTA methods in Tab. 2, the performance of the baseline is good enough. Furthermore, with the independent addition of the bidirectional retention mask M 𝑅𝐷 or the learnable sparse retention mask M 𝐿𝑆 on the baseline, the model achieves a considerable performance improvement. When we model bidirectional video temporal associations and then add sparsity mask as a constraint to erase unnecessary dependencies in video sequences, the performance of the model is further improved. The results show that both mask modules of MRNet are coordinated and compatible, and contribute to achieving significant improvement on the representation learning and generalization performance of the MR model.There are two operations that we optimize in the Retention module as shown in Fig. 5 : (1) From unidirectional to bidirectional. If we use unidirectional retention mask instead of bidirectional M 𝑅𝐷 (the unidirectional M 𝑅𝐷 has only forward mask as (M 𝑅𝐷 ) 𝑛,𝑚 = 𝛾 𝑛-𝑚 𝑖 𝑓 𝑛 ≥ 𝑚; 𝑒𝑙𝑠𝑒 0.), the performance of the model is compromised, e.g., R@1, IoU@0.5 decreased by 1.60 point, this indicates that bidirectional M 𝑅𝐷 is indeed stronger in its ability to model video sequences than unidirectional in our MRNet. (2) SoftMax over Gating: we define the "MaskRetention" approach in Eq. 8 that adopts the SoftMax operation (Transformer-like), while Swish Gate is specifically designed to handle the model's nonlinearity in original Retention [45] , we use SoftMax instead, the Fig. 5 (right) shows the advantages of the SoftMax operation over the Swish Gate operation, which promises both complexity reduction and performance improvement.We evaluate the effect of self-supervised loss L 𝑠𝑝𝑎𝑟𝑠𝑒 that is used to constrain the sparsity of learnable mask M 𝐿𝑆 . From Tab. 7, L 𝑠𝑝𝑎𝑟𝑠𝑒 is effective in improving the performance of the model. We further give the ablation of two important hyperparameters of M 𝐿𝑆 in Fig. 6 , i.e., the loss weight 𝜆 𝑠 of L 𝑠𝑝𝑎𝑟𝑠𝑒 and the sparsity ratio 𝛼 (%) of M 𝐿𝑆 , where 𝛼 controls the mask's proportion of M 𝐿𝑆 during training by setting the average convergence cutoff threshold for M 𝐿𝑆 at 0.5𝛼. The optimal setting of 𝜆 𝑠 is 10.0, and the sparsity ratio 𝛼 is 25%.Mode Analysis (Q4). We discuss different masking modes for the vision-language attention map. The "No masking (vanilla attention)", "Only masking video" and "Full masking" modes in Fig. 7 correspond to the first, second and last rows of Tab. 8, respectively. In our solution, we leave the query free and focus on the sparsity of video, with the core idea of removing redundancy within the video and making the most of query semantics. From Tab. 8, we test different mask combinations, whenever it comes to query-related masking, there is some performance degradation, which confirms our above core idea, i.e., only "masking "A (𝑣 → 𝑣) is optimal. Moreover, from Fig. 7 , if we mask the whole vision-language map, the performance is higher than that with "No masking", this indicates that vanilla attention brings redundant correlations (𝑇 × 𝑇 ) and hurts performance. It is reasonable and effective to consider removing redundancy between video attention responses. The visualized mask retention map of our solution "Only masking video" shows that the retention mask is well concentrated in the groundtruth (GT) region, the response in the other regions is close to 0, realizing effective noise suppression. across three datasets, which indicates that broad adaptability of our approach to MR-related tasks. Overall, the visualization results demonstrate the effectiveness of our proposed MRNet. 
conclusion: 
 In this paper, we propose a compact single-stream Maskable Retentive Network (MRNet) as a new structural paradigm for addressing video Moment Retrieval tasks involving video and query interactions, which implements video retentive reasoning in a cross-modal environment. The effectiveness of MRNet framework has been demonstrated on five standard benchmarks for three MR tasks. 
abstract: 
 Video Moment Retrieval (MR) tasks involve predicting the moment described by a given natural language or spoken language query in an untrimmed video. In this paper, we propose a novel Maskable Retentive Network (MRNet) to address two key challenges in MR tasks: cross-modal guidance and video sequence modeling. Our approach introduces a new retention mechanism into the multimodal Transformer architecture, incorporating modality-specific attention modes. Specifically, we employ the Unlimited Attention for language-related attention regions to maximize cross-modal mutual guidance. Then, we introduce the Maskable Retention for video-only attention region to enhance video sequence modeling, that is, recognizing two crucial characteristics of video sequences: 1) bidirectional, decaying, and non-linear temporal associations between video clips, and 2) sparse associations of key information semantically related to the query. We propose a bidirectional decay retention mask to explicitly model temporal-distant context dependencies of video sequences, along with a learnable sparse retention mask to adaptively capture strong associations relevant to the target event. Extensive experiments conducted on five popular benchmarks ActivityNet Captions, TACoS, Charades-STA, Activ-ityNet Speech, and QVHighlights for MR tasks demonstrate the significant improvements achieved by our method over existing approaches. Code is available at https://github.com/xian-sh/MRNet.• Computing methodologies → Activity recognition and understanding; • Information systems → Multimedia and multimodal retrieval.  
 
 Appendix:  


Decision: In summary, the reviewers highlight several strengths of the paper, including its innovative bidirectional retention mask and learnable sparse retention mask, intuitive design for modeling temporal associations in video sequences, significant performance improvement, and comprehensive experimental validation. Minor concerns are raised regarding the clarity and implementation details of the sparse retention mask, the complexity and readability of the paper, the lack of open-sourced code for reproducibility, and specific issues with formula definitions, formatting, and comprehensive explanations of certain methodological choices.

Overall, two reviewers lean towards acceptance, while one leans towards rejection due to presentation issues. Considering this, I recommend accepting the manuscript for publication. However, please revise your manuscript further based on the reviewer's comments on the writing.

Reviews: review: paper_summary: paper_summary: This paper introduces the Maskable Retentive Network (MRNet), a novel architecture designed to address three video moment retrieval tasks (NLMR, SLMR and MR+HD).
The key innovation lies in integrating a retention mechanism into the multimodal Transformer framework, resulting in the Maskable Retention Transformer (MRT) block.
The MRT block divides the language and video attention map into four regions $\mathcal{A}(q \rightarrow q)$, $\mathcal{A}(q \rightarrow v)$, $\mathcal{A}(v \rightarrow q)$ and $\mathcal{A}(v \rightarrow v)$.
The primary contributions of this work are the introduction of the bidirectional retention mask, represented by matrix $\mathcal{M}{RD}$,  and the learnable sparse retention mask, denoted by matrix $\mathcal{M}_{LS}$, which are specifically designed for enhanced video modeling.
Experimental results on five benchmark datasets, particularly ActivityNet Captions, demonstrate the effectiveness of this method.
strength_weakness: strength_weakness: - The main innovation of the article lies in the bidirectional retention mask and the learnable sparse retention mask. The design ideas are very intuitive, aiming to model the bidirectional, decaying, and nonlinear temporal associations between video sequences, as well as to adaptively learn strong associative responses related to the target event and eliminate redundant background interference in the video.
- The method is simple, yet the performance improvement is obvious.  Meanwhile, this method achieves a trade-off between model size and accuracy.
- From the description, the structure does not seem complicated and should be easy to reproduce. However, it would be great if the author could open source the implementation in the future.
- The author provides supplementary materials that discuss more experimental results.
limitations: limitations: - Despite the intuitive motivation of the learnable sparse retention mask, the implementation details are not convincing. Why does Eq. 5 achieve the effect of reducing redundant noise interference and learning more critical information under the supervision of Eq. 6? I have not found any explanation in the article or supplementary materials about how the sparse ratio $\alpha$ controls the proportion of responses.
- The main innovation of the article is the introduction of the bidirectional retention mask and learnable sparse retention mask. However, the baseline in Table 6 on the ANetCap dataset has already achieved state-of-the-art performance. What has led to such strong performance?
- The ablation experiment R2.1 only compares the performance difference between bidirectional and unidirectional, but it does not explain how unidirectional is implemented, such as how to adjust Eq.4 to make it unidirectional?
- There is an issue with Eq.4. $(\mathcal{M}_{RD})_{n,m}$ should be a scalar, but $\mathcal{M}_{RD} \in \mathbb{R}^{T \times T}$.
- The sparsity constraint loss in Table 8 should be $\mathcal{L}_{sparse}$.
- Here are some suggestions: ① It would be helpful to annotate that Figure 2 displays the experimental results of the MRNet in multi-query training mode on the ANetCap dataset, to prevent any confusion; the methods and experimental analysis sections could be expanded with more detail, as the introduction is too verbose - focus on the key points; the right figure of Figure 7 could benefit from a more detailed explanation.
paper_summary: paper_summary: This paper introduces a novel deep learning architecture, Maskable Retentive Network (MRNet), designed to address the challenges in Video Moment Retrieval (MR) tasks. MR tasks involve locating specific moments within untrimmed videos based on natural language or spoken language queries. The authors highlight two key challenges in MR: cross-modal guidance and video sequence modeling.
To tackle these challenges, MRNet integrates a new retention mechanism into the multimodal Transformer architecture. This mechanism employs modality-specific attention modes, including Unlimited Attention for language-related regions to enhance cross-modal guidance and a Maskable Retention mechanism for the video-only attention region to improve video sequence modeling. The proposed system also introduces a bidirectional decay retention mask and a learnable sparse retention mask to model temporal-distant context dependencies and capture strong associations relevant to the target event, respectively.
The authors conducted extensive experiments on five benchmarks: ActivityNet Captions, TACoS, Charades-STA, ActivityNet Speech, and QVHighlights. The results demonstrate that MRNet significantly outperforms existing approaches, showcasing its effectiveness in MR tasks.
strength_weakness: strength_weakness: 1. The paper introduces the retention mechanism, including modality-specific attention modes like Unlimited Attention and Maskable Retention, which is interesting.
2. The paper provides extensive experimental results across multiple benchmarks, which validate the effectiveness of the proposed MRNet.
limitations: limitations: The readability of this paper is reduced due to the following issues.
1. The paper is full of long and difficult sentences, which are difficult to understand and read. For example, lines 145-150, lines 365-372.
2. Why are the characteristics of video sequences bidirectional, attenuated, and nonlinear? The paper does not give a very clear explanation.
3. The symbols of the formula are not clearly defined. For example, X in Formula 3 is not defined. The (·) symbol in formula (1) is not explained clearly.
4. There are many typo errors in the article, such as two "learnables" on lines 382-383 and two R3.3 on lines 660.
5. Why is the attention score between text and video not masked? In fact, the text also contains tokens that are more relevant to the video and tokens that are less relevant, which the author does not seem to have considered.
6. How is the matching score map in line 478 obtained, and how is the query feature obtained?
7. The code is not provided, and I doubt the reproducibility of the article’s results.
8. While the paper shows excellent performance on specific benchmarks, it is unclear of robustness of the model. It is recommended to test on ood-datasets like Charades-CD [1] or Charades-CG [2] to test the generalization ability of the model.
9. It’s weird to me. From the beginning, the input for text is the sequence of sentence feature and the input for video is the sequence of video clips. These two types of features are not at the same granularity to attend to each other. 
10. The CPL in Table 3 is weakly supervised, but your method is fully supervised. It is unreasonable to compare them together.
11. What is the difference between single-query training mode and multi-query mode?

[1] A Closer Look at Temporal Sentence Grounding in Videos

[2] Compositional Temporal Grounding with Structured Variational Cross-Graph Correspondence Learning
paper_summary: paper_summary: In this paper, a novel Maskable Retentive Network (MRNet) is proposed to address two key difficulties in Video Moment Retrieval tasks: cross-modal guidance and video sequence modeling. A novel retention mechanism is introduced into classical multimodal Transformer and divide the attention into four areas. The Unlimited Attention is leveraged to language-related attention regions to maximize cross-modal mutual supervision and the Maskable Retention is employed for video-only attention region to boost video sequence modeling. In addition, extensive experiments results verify the important improvement of MRNet compared with existing methods.
strength_weakness: strength_weakness: In this paper, two learnable masks proposed for the only-video attention region can enhance the model's ability to learn the temporal context of the video, as well as the redundant background information of the video is removed, so that the strong responses on the attention map become sparse, and then highlight the relevant video tokens. The comparison experiment is fully conducted, and the ablation study is also relatively comprehensive, which can clearly highlight the impact of some important module in the MRNet.
limitations: limitations: 1.	This paper has not clearly expressed the benefits of dividing attention map into four regions. If only to perform modality-specific modeling with the properties of the modality itself, why we not directly utilize the vanilla cross/self attention mechanisms. 
2.	This paper seems to only concentrate on the only-video attention ($A(v \rightarrow v)$) region and does not make some modification to query-video ($A(q \rightarrow v)$) and video-query ($A(v \rightarrow q)$) attention regions, but the latter two is also very essential for cross-modal feature fusion.
3.	In formula 1, only Q and V are added with position coding. Why is the calculation of V (V_q and V_v) not appended with position coding?
4.	Formula 11 doesn’t seem to be a representation of the contrastive learning loss function, since there is no obvious contrast between positive and negative samples, it is more like cross entropy loss.
5.	There are some small problems such as formatting in the article, as described below:
   A.	In line 95, the “highlightness” dose not seems to be a valid derivative. 
   B.	In line 454, the abbreviation “MRT” have already been explained before (line 287) and need not be explained more than once.
   C.	In line 487, the citations [45, 46] are actually the same paper.