Saliency-Guided Fine-Grained Temporal Mask Learning for Few-Shot Action Recognition

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 07:51:10

Abstract: Temporal relation modeling is one of the core aspects of few-shot action recognition. Most previous works mainly focus on temporal relation modeling based on coarse-level actions, without considering the atomic action details and fine-grained temporal information. This oversight represents a significant limitation in this task. Specifically, coarse-level temporal relation modeling can make the few-shot models overfit in high-discrepancy temporal context, and ignore the low-discrepancy but high-semantic relevance action details in the video. To address these issues, we propose a saliency-guided fine-grained temporal mask learning method that models the temporal atomic action relation for few-shot action recognition in a finer manner. First, to model the comprehensive temporal relations of video instances, we design a temporal mask learning architecture to automatically search for the best matching of each atomic action snippet. Next, to exploit the low-discrepancy atomic action features, we introduce a saliency-guided temporal mask module to adaptively locate and excavate the atomic action information. After that, the few-shot predictions can be obtained by feeding the embedded rich temporal-relation features to a common feature matcher. Extensive experimental results on standard datasets demonstrate our method’s superior performance compared to existing state-of-the-art methods.

Main paper: 
 introduction: 
 Few-shot learning is able to generalize well to new unseen categories with only a few data samples and thus has gained increasing attention. It is promising to reduce the labor of collecting large-scale training data and the computational cost that many successful action recognition models [1, 11, 12, 40] will suffer when deployed in unfamiliar realistic scenarios where large-scale datasets are difficult to collect. However, video few-shot action recognition persists as a challenge, attributed to the intricate nature of action representation and temporal relationships.Metric-based meta-learning methods [26, 28, 32] are one of the most widely used techniques in few-shot action recognition, where a meta-trained embedding network maps videos into a feature space and then matches the most similar video pairs in the feature space for recognition [3, 15, 41, 43, 44] . Most of the these methods obtain the similarity between videos by aggregating frame-to-frame similarity scores [4, 35, 36] . For example, OTAM [4] performs fixed time-order video frame alignment after obtaining video features to match the closest frame pair in two videos. HyRSM [36] devises a bidirectional mean Hausdorff metric to obtain frame-to-frame correspondences with aggregate matching relationships.Despite the remarkable results, metric-based approaches to fewshot action recognition still exhibit limitations. Frame-level metric methods treat all frames in a video equally in the matching process, ignoring the fact that the semantic salience of each frame is different. Video frames can be classified into salient frames, non-salient frames, and ambiguous frames. However, current video models, trained on video-level semantic labels, tend to excessively prioritize salient frames that exhibit high dissimilarity to other support videos. Consequently, ambiguous frames may not receive adequate attention from the model, increasing the likelihood of false matching, especially when dealing with complex atomic actions [17] or those containing multiple semantic instances.Recently, masked visual modeling has achieved stunning results in self-supervised learning of images and videos [13, 18, 20, 31] , which masks out a part of the input and reconstructs the complete input with the remaining part, and it has been shown that masked visual modeling is capable of learning a more generalized feature potential space [18] . In addition, by designing a specific masking strategy, the model can be directed to focus on particular parts of the data. Building upon the aforementioned concepts, we posit that the introduction of mask feature reconstruction as a self-supervised pretext task into supervised few-shot learning for classification, which guides the model to perceive the semantics of the actions in the video more efficiently by reconstructing the video features, and improves the generalization ability of the few-shot action classification on the new task. However, in few-shot action recognition, indiscriminate escalation of the reconstruction difficulty may lead to an imbalance between the learning objectives of self-supervision and few-shot classification. Therefore, it remains a challenge to design a reasonable masking strategy so as to take full advantage of masked visual modeling to enhance the generalization ability of few-shot action recognition.To address the above issues, we propose a novel saliency guided fine-grained temporal mask learning method for few-shot action recognition. Specifically, we built a temporal mask learning scheme in which fine-grained atomic action details are learned through interaction with the saliency-guided temporal mask. In general, our study revolves around two key ideas. First, we learn a comprehensive temporal relation by forcing ambiguous frames to aggregate the complete video action semantics. Second, we exploit the low-discrepancy atomic action features to locate and excavate the atomic action information. Notably, to the best of our knowledge, our method is the first to learn the mid-level atomic action details compared to existing mask visual modeling work that typically uses the paradigm of reconstructing low-level features such as original pixels and HOGs [18, 37] .In summary, our contribution can be summarized as follows:• we propose a novel saliency guided fine-grained temporal mask learning method that models the temporal atomic action relation for few-shot action recognition in a finer manner. • We propose a temporal mask learning architecture to autonomously explore the optimal alignment for each atomic action snippet while incorporating a saliency-guided temporal mask module to adaptively locate and excavate the atomic action information. • We conduct extensive experiments on five benchmark datasets to verify the effectiveness of the proposed method. The experimental results demonstrate the superior performance of our method compared to existing state-of-the-art methods. 
related works 2.1 few-shot action recognition: 
 Existing few-shot action recognition methods typically employ a metric-based meta-learning paradigm [32] and perform frame-toframe temporal matching [4, 35, 36] to search for the most similar videos. Videos have an additional dimension of time than images, and thus the few-shot action recognition task requires a combination of both video spatio-temporal modeling and inter-video similarity metrics. For instance, AMeFu [15] incorporates the depth modality as supplementary scene information and fuses it with RGB modality for prototype metric [26] . OTAM [4] employs a variant of Dynamic Time Warping (DTW) to enforce the alignment of video frames in temporal order. ITANet [42] introduces a decomposed self-attention mechanism to alleviate intra-class variability in video features. TRX [24] utilizes CrossTransformers [9] to construct query-specific prototype representations. Building upon TRX [24] , STRM [30] enhances local and global features to effectively capture spatiotemporal contextual information in videos. HCL [43] extracts multi-scale video representations from coarse to fine granularity, utilizing these representations for hierarchical matching. HyRSM [36] , through task-aware video relation learning, tailors features specific to the task and employs a set-matching metric. MoLo [35] introduces a long-short contrastive loss to enforce local frame feature prediction with global context and perceives motion details through frame-wise difference reconstruction. SloshNet [39] adaptively integrates spatial features from different levels and integrates long-term and short-term temporal features for rich spatiotemporal characteristics. Our method focuses on providing discriminative video features, reducing potential ambiguities in frame-to-frame matching.Mask autoencoder is fundamentally a denoising autoencoder that learns effective feature representations by reconstructing the complete input from corrupted inputs. Recently, some work has applied Masked Image Modeling to self-supervised image pretraining, achieving remarkable results. For instance, iGPT [6] employs a selfsupervised image pretraining approach by predicting the next pixel in a one-dimensional pixel sequence. BEiT [2] learns the visual semantics of images through predicting discrete visual tokens during pretraining. SimMIM [38] introduces a simple framework for the regression task on original image pixels. MAE [18] introduces an asymmetric encoder-decoder architecture and employs a high mask rate random masking strategy, enhancing the efficiency of masked image pretraining.The tremendous success of Masked Image Modeling has sparked numerous efforts to extend this self-supervised pretraining paradigm to videos. BEVT [34] and VIMPAC [29] attempt a similar approach to BEiT [2] , learning video representations by predicting features exported by a tokenizer. MaskFeat [37] sets the goal of mask reconstruction to HOG [7] handcrafted features instead of raw pixels. VideoMAE [31] and MAE-ST [13] use extremely high mask ratios to increase the reconstruction difficulty. VideoMAE [31] also proposes a tube masking strategy to mitigate information leakage during the reconstruction process, while VideoMAE v2 [33] introduces dual masking to explore larger video-based models, masking the input of the decoder to enhance training efficiency. MAR [25] introduces cell running masking, providing detailed context for the encoder to easily perceive the missing parts. In contrast to the aforementioned methods, our proposed saliency-guided finegrained temporal mask learning is not aimed at achieving better pretraining performance but rather to assist in achieving more accurate video matching for few-shot action recognition. 
method 3.1 problem definition: 
 The goal of few-shot action recognition is to obtain good generalization over new action categories with a small number of labeled samples. We follow the standard metric-based few-shot learning protocol [24, 32, 44] and employ the episode training paradigm. We sample videos from the meta-training set 𝐶 𝑡𝑟𝑎𝑖𝑛 during training to construct multiple episodes, each of which consists of a set of support videos 𝑆 and a set of query videos 𝑄. The support set S contains 𝑁 × 𝐾 samples from 𝑁 action categories, where each action category contains 𝐾 samples (called the 𝑁 -way 𝐾-shot task), and the query set 𝑄 contains several samples in N action categories. The training objective is to correctly recognize the video in 𝑄 as one of the 𝑁 categories. During testing, episodes are built by sampling 𝑁 categories from the meta-test set 𝐶 𝑡𝑒𝑠𝑡 , where the action categories of 𝐶 𝑡𝑟𝑎𝑖𝑛 and 𝐶 𝑡𝑒𝑠𝑡 do not overlap with each other, in order to test the generalization performance of the model on unseen data.Overall architecture. Figure 1 shows an overview of our framework. Our framework comprises two branches: the temporal mask learning branch and the few-shot classification branch. In the temporal mask learning branch, we utilize the saliency-guided temporal mask generator module to adaptively generate a mask map for masking the feature embeddings, and then force the remaining portions to predict the features of the complete video. In the few-shot classification branch, we measure the similarity of action videos through frame-level matching for action classification. We aggregate the losses from both branches to form the overall framework loss.In the episode under the N-way K-shot setting, the support set contains 𝑁 × 𝐾 videos, and the query set contains at least one video. For each video V, it can be represented as𝑉 = {𝐼 1 , 𝐼 2 , • • • , 𝐼 𝑇 } ∈ R 𝑇 ×3×𝐻 ×𝑊, where 𝑇 is the number of video frames obtained by uniform sampling. We use a 2D CNN backbone network to extract frame-level features of the video V to obtain the feature embedding𝑋 𝑒𝑚𝑏 = {𝑋 𝑒𝑚𝑏 1 , 𝑋 𝑒𝑚𝑏 2 , • • • , 𝑋 𝑒𝑚𝑏 𝑇 } ∈ R 𝑇 ×𝐷 .Each video frame is encoded as a feature vector in D dimensions.Saliency-guided temporal mask. Given a frame-level embedded feature sequence 𝑋 𝑒𝑚𝑏 of a video, we obtain the category-aware scores for each frame using the following formula:EQUATIONWhere Φ 𝑐𝑙𝑠 is the composition of several temporal convolutional layers, interspersed with ReLU layers and Dropout layers. The resulting A ∈ R 𝑇 ×𝐶 represents the semantic relevance between the frame-level features of the video and the class labels, where 𝐶 is the number of action categories in the meta-training set. Subsequently, a simple channel-wise summation operation (Φ 𝑠𝑢𝑚 ) and a Sigmoid function are applied for category-agnostic aggregation, resulting in the actionness score A 𝑛𝑒𝑠𝑠 ∈ R 𝑇 . This operation can be expressed as follows:EQUATIONWe consider the value of A 𝑛𝑒𝑠𝑠 as the temporal saliency of all sampled frames in the video, and use it to guide the generation of the mask map. To excavate and learn more details about atomic actions. We focus on the ambiguous snippets (more detailed discussion in Section 4.3.2). We mask both salient and non-salient frames, compelling the network to aggregate comprehensive video semantics from ambiguous frames. Specifically, we perform top-k and bottom-k operations on the feature embedding 𝑋 𝑒𝑚𝑏 based on 𝐴 𝑛𝑒𝑠𝑠 to obtain two sets of frame features: 𝑋 𝑠𝑎𝑙 ∈ R 𝑇 1 ×𝐷 containing the top 𝑇 1 frames with the highest temporal saliency, and 𝑋 𝑛𝑜𝑛 ∈ R 𝑇 1 ×𝐷 comprising the bottom 𝑇 1 frames with the lowest temporal saliency. The remaining frame-level features are considered as features corresponding to ambiguous frames, denoted as 𝑋 𝑎𝑚𝑏 ∈ R 𝑇 2 ×𝐷 , where 𝑇 2 = 𝑇 -2 ×𝑇 1 . We mask 𝑋 𝑠𝑎𝑙 and 𝑋 𝑛𝑜𝑛 and input only 𝑋 𝑎𝑚𝑏 as the visible portion into the encoder, relying on 𝑋 𝑎𝑚𝑏 to reconstruct the features of the entire video.Encoder. Vision Transformer (ViT) [10] has demonstrated its powerful capability for global visual perception and is commonly employed as an encoder in Mask Visual Modeling methods [13, 18, 31] . Therefore, we choose shallow Transformer layers, similar to those in the ViT, as our encoder. In order to make the features extracted by the CNN suitable to be used as inputs to the Transformer block, we decompose the feature embedding into 𝑁 non-overlapping tokens, each of which has the shape 1 × 𝑃 , i.e. 𝑁 = 𝑇 2 𝐷/𝑃. We reshape 𝑋 𝑎𝑚𝑏 ∈ R 𝑇 2 ×𝐷 into X𝑎𝑚𝑏 ∈ R 𝑁 ×𝑃 , add positional embedding to identify the relative position of tokens, and then feed them into the encoder to obtain the encoded latent features 𝑋 𝑙𝑎𝑡 :EQUATIONΦ 𝑒𝑛𝑐 refers to the encoder network, and 𝑃𝐸 denotes positional embedding.Decoder. Similar to the encoder, the decoder consists of shallow Transformer layers. We concatenate 𝑋 𝑙𝑎𝑡 with learnable mask tokens, add fixed positional embedding to indicate the position of each token in the video frame sequence, and utilize this as input for the decoder. Consequently, we obtain the reconstructed features 𝑋 𝑟𝑒𝑐𝑜𝑛 ∈ R 𝑁 ′ ×𝑃 , where 𝑁 ′ = 𝑇 𝐷/𝑃. The process can be expressed by the following equation:EQUATIONΦ 𝑑𝑒𝑐 denotes the decoder network, while 𝑚𝑎𝑠𝑘 represents mask tokens.Reconstruction loss. To extract more semantic information from a small number of ambiguous frames, we introduce the reconstruction loss. The training objective of the temporal mask learning branch is to minimize the Mean Square Error (MSE) loss between the reconstructed and original features. We decompose 𝑋 𝑒𝑚𝑏 ∈ R 𝑇 ×𝐷 into a token sequence X𝑒𝑚𝑏 ∈ R 𝑁 ′ ×𝑃 , where 𝑃 is the dimension of each token, and 𝑁 ′ = 𝑇 𝐷/𝑃 is the number of tokens, achieving the same size as 𝑋 𝑟𝑒𝑐𝑜𝑛 , and then apply the MSE loss:EQUATIONwhere || • ||2 denotes the ℓ 2 loss. Metric-based few-shot classification. The frame-level feature sequence 𝑋 𝑒𝑚𝑏 undergo a notable refinement through the encoder of the aforementioned temporal mask learning branch. Nevertheless, a disparity persists between this refined representation and the requisite representation for the few-shot action recognition. As a result, in preparation for the frame-level metric procedure, we employ shallow Transformer layers [10] to further process the features, facilitating an improved alignment of these features within the metric space. The reshaped token sequence X𝑒𝑚𝑏 , along with positional embeddings, is fed into the Transformer. The output are then reshaped into the size of 𝑇 × 𝐷, obtaining the features 𝑋 𝑚𝑎𝑡 for frame-level matching in videos. The above process can be represented by the following formula:EQUATIONΦ 𝑒𝑛𝑐 represents the encoder in the temporal mask learning branch, Transformer stands for the shallow Transformer layers.Given the features corresponding to the query video, denoted as 𝑋 𝑚𝑎𝑡 𝑞 , and the feature representations corresponding to videos in the support set S with the same labels as the query video, represented as X𝑚𝑎𝑡 , the classification loss is expressed as follows:EQUATION)𝜙 is the distance metric function for video features, used to obtain video-level similarity by aggregating the similarity of frame-level features between support videos and the query video. The final loss can be expressed as:EQUATIONOur training strategy consists of two steps. Firstly, we freeze the feature embedding network and perform supervised training on Φ 𝑐𝑙𝑠 using the class labels of video instances from the meta-training set. Subsequently, we freeze the parameters of Φ 𝑐𝑙𝑠 and train our framework using the loss L from Equation 8. In the inference phase, given a query video 𝑞 from an unknown class and a support set S, we use a similarity metric to find the video in S that is most similar to 𝑞. Subsequently, we assign the action class label of that video to 𝑞. 
experiment 4.1 datasets and experimental setups: 
 Datasets. We evaluate our approach on five standard datasets, including Kinetics [5] , UCF101 [27] , HMDB51 [27] , SSv2-Full [16] , and SSv2-Small [16] . The datasets are partitioned into meta-training, meta-validation, and meta-testing sets based on action categories to meet the requirements of the few-shot classification setting. For Kinetics, we follow the splitting strategy proposed by [44] , selecting 100 action categories, each with 100 samples, and dividing these categories into 64, 12, and 24 for training, validation, and testing, respectively. For UCF101, we split it into 70, 10, and 21 categories for training, validation, and testing, following the same setup as [41] . In the case of HMDB51, we split it into 31, 10, and 10 categories for training, validation, and testing, adhering to the same splitting strategy as in [41] . For SSv2-Full and SSv2-Small, we adopt the split strategies utilized in [4] and [44] , selecting 64 categories for training, 12 for validation, and 24 for testing from the datasets. The distinction lies in the fact that SSv2-Full comprises all samples for each category, whereas SSv2-Small only includes 100 samples per category.Implementation details. Following the common paradigm of existing few-shot action recognition methods [4, 24, 35, 36] , we employ ResNet50 [19] as the backbone network and initialize it with weights pre-trained on ImageNet [8] to extract frame-level features. We sparsely and uniformly sample 8 frames from each video, like previous methods [30, 35] . In the network architecture, the Transformer layers of the Encoder and Decoder are both configured with two layers. Additionally, we set k in the top-k and bottom-k, as well as 𝑇 1 and 𝑇 2 to 2, thus masking 4 frames. During training, we resize each frame in the video into 256 × 256, followed by random horizontal flips and random cropping to a 224 × 224 region. In the testing phase, we first perform resizing and then replace random cropping with center cropping. Similar to prior work [35] , we collect 10,000 episodes from the meta-testing set to evaluate the model's performance and report the average accuracy. The weight parameter 𝜆 in the loss function is set to 0.5. In Section 4.2, we employ two methods, OTAM [4] and Bi-MHM [36] , as feature similarity metrics to train and evaluate our model. In Section 4.3, we exclusively utilize OTAM during the feature matching phase. We implement our framework using PyTorch [23] and conduct training on one RTX 4090 GPU. Experiments are performed under the 5-way task with 1-shot, 3-shot, and 5-shot settings. The best results are denoted in bold black, the second-best results are indicated with an underscore, and "-" signifies that the result is not available in the published works.Reference Kinetics UCF101 HMDB51 1-shot 3-shot 5-shot 1-shot 3-shot 5-shot 1-shot 3-shot 5-shot CMN [44] ECCV ' In the 5-way task with 1-shot, 3-shot, and 5-shot settings, we compare our method with state-of-the-art approaches on Kinetics, UCF101 and HMDB51, as presented in Table 1 . We utilize two commonly used frame-level alignment metrics as feature similarity metrics, namely OTAM [4] and Bi-MHM [36] , to validate the effectiveness of our method. From the experimental results, it can be observed that when using Bi-MHM as the metric strategy, our method surpasses existing approaches across all three datasets in both the 1-shot and 3-shot settings. Specifically, in the 1-shot setting, our method achieves significant improvements of 1.5%, 1.8%, and 0.7% on Kinetics, UCF01, and HMDB51, respectively. Under the 3-shot setting, improvements of 0.1%, 1.5%, and 0.5% are observed on the same datasets, establishing a new state-of-the-art benchmark. When employing OTAM as the metric strategy, our method likewise demonstrates excellent performance outperforming numerous methods. Building upon Bi-MHM, Our method's performance in the 5-shot setting on the Kinetics dataset exceeds the previous best method by 0.6%. However, on UCF101 and HMDB51, it lags behind the best-performing methods. This problem is attributed to two possible reasons: (1) Methods such as STRM [30] and SloshNet [39] , following the metric approach introduced in TRX [24] , which is specifically designed for tasks with a higher number of shots, contributing to its superior performance in the 5-shot setting. (2)The 5-shot setting includes a higher number of samples in support set. Consequently, the number of discrepancy-based salient frames increases, while the number of ambiguous frames decreases. However, our method may obscure some information in salient frames, potentially leading to a decrease in accuracy. This problem is further elaborated in the subsequent experiments. We further compare our approach with state-of-the-art methods on the SSv2-Small and SSv2-Full datasets, as presented in Table 2 .Our method achieves optimal performance under the 5-shot setting on both datasets, with a consistent improvement of 0.8%. However, under the 1-shot setting, our method slightly lags behind the bestperforming approach, i.e. MoLo [35] , which additionally leverages inter-frame difference information, thereby effectively enhancing performance on SSv2-Small and SSv2-Full. However, the framework of MoLo results in a larger parameter count (89.6M). In contrast, our approach achieves a parameter count (37.2M) only 0.42 times that of MoLo, striking a better balance between performance and efficiency.Through analyzing the results of the first two experiments, we derive the following observations: For Kinetics, UCF101, and HMDB51, actions can be easily recognized through several key frames containing scene content, characterizing them as scene-centric datasets. On the other hand, the actions in SSv2 encompass complex temporal variations, classifying it as a temporal-centric dataset. For scene-centric datasets, having more key frames in the samples of support set makes it easier for the model to identify action category. For instance, in the 5-shot setting in Table 1 , several previous methods achieve better results, whereas our approach performs better in low-shot settings. Conversely, in temporal-centric datasets, having more samples of support set results in more variable motion information, making it more challenging to learn. Our method effectively addresses this issue, yielding the best performance in the 5-shot setting. Therefore, in more complex temporal action matching scenarios, our approach demonstrates significant efficacy.The results of the ablation study of network components are displayed in Table 3 . Starting with a comparison of baseline and method 1, we add an Encoder to the baseline method, consisting of 2 Transformer layers without Table 2 : Comparison with state-of-the-art few-shot action recognition methods on the SSv2-Small and SSv2-Full datasets. Experiments are performed under the 5-way task with 1-shot and 5-shot settings.Reference SSv2-Small SSv2-Full 1-shot 5-shot 1-shot 5-shot MatchingNet [32] NeurIPS' 16 31.3 45.5 --MAML [14] ICML'17 30.9 41.9 --CMN [44] ECCV ' mask reconstruction. We observe marginal performance improvements of 0.13% and 0.06% for 1-shot and 5-shot tasks, respectively, indicating that simply increasing the depth of the temporal feature extraction network does not lead to a significant performance improvement. Integrating temporal mask learning architecture (TML) into the aforementioned network for masked feature reconstruction, where the temporal mask is randomly generated, results in performance improvements of 0.57% and 0.34% for 1-shot and 5shot tasks, respectively, demonstrating that the random temporal mask strategy can improve feature robustness to a certain extent. Subsequently, we further incorporate saliency-guided temporal mask module (SGTM). Compared to baseline that without temporal mask, we observe improvements of 0.81% and 0.37% for 1-shot and 5-shot tasks, respectively. Compared to random mask, SGTM can bring about more significant improvements. This suggests that selective temporal mask learning enables the network to selectively focus on learn more detailed actions, ultimately enhancing overall performance. Further elaboration on the learned action details will be provided in Section 4.4. Here, we further explore the impact of masking different regions under the guidance of temporal saliency on the model's performance. Experiments are conducted on the Kinetics dataset under the 5-way 1-shot and 5-way 5-shot settings, and the results are summarized in Table 4 . We observe that masking the top 4 frames with the highest temporal saliency achieves suboptimal performance in the 1-shot setting, masking the middle 4 frames results in the worst performance, and masking the bottom 4 frames yields 1-shot performance between the first two scenarios. In contrast, our method achieves the best performance in both 1-shot and 5-shot scenarios. The comparison results of different mask strategies in Table 4 indicate that:• Masking salient frames can improve overall performance, suggesting that the network learns additional information from ambiguous frames and non-salient frames. • Masking ambiguous frames leads to a performance decrease, indicating that non-salient frames can't provide sufficient information, causing the network to overly rely on salient frames. • Our method's superior performance suggests that additional temporal information not present in the salient frames, which can be utilized for few-shot temporal matching and overall performance enhancement.Analysis of the impact of different loss functions. We further investigate the impact of different loss functions on the performance of the method. 5 , we can observe that, on both datasets, smaller values of the reconstruction loss, i.e. 𝜆 is set to 0.1 and 0.3, show better performance compared to the case where the reconstruction loss is 0. Larger values of the reconstruction loss, i.e., 𝜆 set to 0.7 and 0.9, result in a performance decline. This is attributed to the stronger constraints imposed on the mask feature reconstruction network when the reconstruction loss is larger, enhancing the reconstruction capability of the encoder. The optimal performance is achieved when the value of the reconstruction loss is set between values mentioned aboves. Therefore, we set 𝜆 to 0.5 as a balance point between the two losses to achieve the best performance.To investigate the impact of our proposed method on per-class classification accuracy for specific categories, we conduct a per-class accuracy analysis on the meta-testing set of the Kinetics dataset under the 5-way 1-shot setting. A comparison is made with TRX [24] and the Baseline, where the baseline comprises a ResNet50 backbone followed by 2 Transformer layers, and OTAM [4] is employed for metric learning. As depicted in Figure 2 , in comparison to the baseline, our approach shows a reduction in accuracy for specific categories, such as "playing drums" and "pushing car". These categories involve interactions with large objects and lack distinct ambiguous segments. Consequently, by masking certain highly salient temporal segments, we experience a loss in performance. However, in comparison to our competitors, our method consistently demonstrates an overall performance advantage, which further highlights its applicability to the majority of motion patterns.As shown in Table 6 , we evaluate the accuracy of the model under the N-way 1-shot setting, where 𝑁 is incremented from 5 to 10, and compare it with some previous methods, including OTAM [4] , TRX [24] , HyRSM [36] , and MoLo [35] . The experimental results show that the difficulty of few-shot classification continues to increase with increasing 𝑁 , resulting in a gradual degradation of performance. Under a higher difficulty classification setting, i.e., the 10-way 1-shot classification setting, our method shows an accuracy advantage over the other methods, achieving an improvement of 0.8%, which further illustrates the effectiveness of the proposed method in dealing with difficult classification tasks. Meanwhile, our method consistently delivers the best results under all settings, demonstrating the excellent performance of the proposed method under different classification difficulties.Analysis of different masking ratio. We further investigate the impact of varying masking ratios on model performance. Figure 4 presents the performance fluctuations of our model on Kinetics dataset with four different masking ratios, with experiments conducted under both 5-way 1-shot and 5-way 5-shot scenarios. From Figure 4 , we can observe that trend in performance variation with different masking ratios is consistent across both settings: it initially increases with the growth of the masking ratio, peaks at 50%, and then diminishes upon reaching 75%. This suggests that for the majority of video instances in the dataset, employing a 50% masking ratio effectively captures valuable information, while also learning more from previously overlooked information within ambiguous frames.To further investigate the impact of the saliency-guided fine-grained temporal mask learning strategy on the latent feature representation, we visualize frame saliency as well as attention maps for two action instances in the SSv2-Full meta-testing set. We observe and compare the learned feature content of our method with that of OTAM by studying the distribution of the spatial attention weights within the feature embedding network. As shown in Figure 3 (a), OTAM tends to focus on the correct regions in salient frames, but its attention is somewhat scattered on ambiguous frames, not concentrating on the regions where human limbs interact with objects.This result in OTAM incorrectly interpreting the entire video sequence as the atomic action of "picking up shoes", failing to properly comprehend the atomic action of "placing shoes behind the door" contained in the ambiguous snippet. In contrast, our method consistently aligns its attention regions with the semantic regions of actions in the video, allowing it to correctly focus on the regions where human limbs interact with objects even in ambiguous frames.From Figure 3 (b), it can be noted that, due to OTAM concentrating solely on differential information between frames within video instances, it exhibits attention divergence when there is no significant change in the video content (e.g. the third and fourth frames). It can also be observed that our method interprets actions based on interactions between objects rather than focusing on the objects themselves. This indicates that our method can accurately focus on the interaction between objects in the scene from a semantic perspective, which is particularly crucial in the SSv2 dataset where different objects interactions often occur in video instances. Such results validate the conclusion drawn in Section 4.3.1: Our saliency-guided temporal mask learning enables the network to learn more detailed semantic-level action features. 
conclusion: 
 In this paper, we propose the saliency-guided fine-grained temporal mask learning method for few-shot action recognition. Our approach models the temporal atomic action relationships in videos through the temporal mask learning architecture, facilitating finegrained matching for each atomic action segment. Additionally, we employ a saliency-guided temporal mask module to locate and excavate the fine-grained action snippet information. Extensive experiments demonstrate that our method effectively extracts temporal semantic information beyond the discrepancy-based salient frames. This enables a holistic understanding of action patterns from the perspective of overall motion, achieving a more comprehensive and refined modeling of temporal relationships. Consequently, our method exhibits excellent performance and holds a performance advantage over existing few-shot action recognition methods.  
 
 Appendix:  


Decision: This paper proposes a novel saliency-guided fine-grained temporal mask learning method to model the temporal atomic action relation for few-shot action recognition. The effectiveness of the proposed methods is demonstrated through experiments on five datasets. All the reviewers are inclined to accept. The AC agrees with the reviewers. 

Quality: The  paper is well-structured, providing detailed and comprehensive explanations.  

Clarity: The paper is easy to read, but some parts are not very clear. 

Originality: The exploration of applying mask visual modeling to few-shot action recognition is interesting and effective. 

Significance: Temporal modeling of atomic action relation is crucial for action recognition.

Reviews: review: paper_summary: paper_summary: This paper proposes a saliency-guided fine-grained temporal mask learning method that aims to model the temporal atomic action relation for few-shot action recognition in a more detailed manner. The effectiveness of the proposed method is demonstrated through experiments conducted on five datasets.
strength_weakness: strength_weakness: 1. This paper proposes a novel saliency-guided fine-grained temporal mask learning method, which significantly improves performance when combined with OTAM and Bi-MHM.
2. The experimental analysis is thorough, providing a comprehensive evaluation of the strengths and weaknesses of the proposed method.
3. The writing is clear, providing detailed and comprehensive explanations. The figures and tables are visually intuitive.
limitations: limitations: 1. The combination of OTAM and Bi-MHM in this paper leads to a noticeable improvement in performance. However, the proposed method does not achieve the best results on some datasets, which suggests the need to consider integration with more advanced models.
1. The method proposed in this paper is relatively simple and lacks significant innovation.
paper_summary: paper_summary: This paper proposes a saliency-guided fine-grained temporal mask learning method that models the temporal atomic action relation for few-shot action recognition in a finer manner.
strength_weakness: strength_weakness: To model the comprehensive temporal relations of video instances, we design a temporal mask learning architecture to automatically search for the best matching of each atomic action snippet.
limitations: limitations: 1. Please explain in detail how the saliency-guided temporal mask module is able to locate the features of keyframe without being guided by the loss function.
2. In the comparison results of the Table 1, why does the proposed method in this paper show sub-optimal results under the 5-shot setting on UCF101 and HMDB51 datasets.
paper_summary: paper_summary: This work focuses on few-shot action recognition task. Authors propose saliency-guided fine-grained temporal mask learning to help the temporal relation modeling，which introduces mask feature reconstruction as a self-supervised pretext task.
strength_weakness: strength_weakness: This paper explore the help of self-supervised tasks in few-shot action recognition tasks.
The experimental results demonstrate the effectiveness on the proposed method on several datasets (eg. kinetics, ucf)
limitations: limitations: ——Why is the score obtained by formula (1) category-aware? Why can the subsequent actionness score reflect the salient or ambiguous nature of the frame? Lack of explanation and evidence.

——It can also be seen from the results in Table 4 that the differences between different masking strategies are actually very weak, and may just be experimental fluctuations. Neither intuitively nor empirically can explain the effectiveness of the saliency-guided mask proposed in the article.

——The related work in the paper also mentions work such as MaskFeat or VideoMAE. I think using their methods can also gain points. There is no strong evidence to convince me that the methods in this article are better than these. In addition, MoLo also used the self-supervised pretext task. They used decoder to reconstruct the frame difference and achieved better results on the SSv2 data set than this article.

——Line 342-345, I don’t understand why this kind of reshape is required. What is the difference between this and directly inputting the feature into the transformer?

——The main experiment table lacks some methods that need to be compared [1, 2]

[1] Khoi D Nguyen, Quoc-Huy Tran, Khoi Nguyen, Binh-Son Hua, and Rang Nguyen. 2022. Inductive and transductive few-shot video classification via appearance and temporal alignments. In European Conference on Computer Vision. Springer, 471 –487.

[2] Jiazheng Xing, Mengmeng Wang, Yudi Ruan, Bofan Chen, Yaowei Guo, Boyu Mu, Guang Dai, Jingdong Wang, and Yong Liu. 2023. Boosting Few-shot Action Recognition with Graph-guided Hybrid Matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 1740–1750
paper_summary: paper_summary: This paper proposes a saliency guided fine-grained temporal mask learning method for few-shot action recognition. The proposed approach employs a temporal mask learning architecture to model the temporal atomic action relationships in videos, and utilizes a saliency guided temporal mask generation module to generate mask maps, adaptively masking the features of salient and non-salient frames. By reconstructing the original video features from the features of ambiguous frames, it aims to extract temporal semantic information beyond salient frames, achieving more comprehensive and refined temporal relationship modeling. The experimental results verify the favorable performance of the proposed approach.
strength_weakness: strength_weakness: -This paper presents a novel exploration of applying mask visual modeling to few-shot action recognition. The authors leverage temporal saliency guidance of video frames and employ a temporal mask learning architecture to uncover often overlooked action relationships beyond the salient frames. This approach offers a new perspective and methodology for modeling temporal action relationships in few-shot action recognition.
-- The authors differentiate the frame sequences sampled from the video based on semantic saliency, with a particular focus on the low-discrepancy ambiguous frames. By extracting fine-grained temporal information from these frames, the representation of action features is enriched. Existing methods often overlook these ambiguous frames, whereas the authors provide a refreshing and effective insight in this regard.
-- In the section of method, the presentation of the problem definition provides readers with a clearer understanding of the issue. The method framework diagram has a clear flow and is easy to comprehend, the description of the method is well-structured.
-The proposed method exhibits excellent performance compared to existing state-of-the-art methods in multiple settings.
-The paper conducts comprehensive ablation studies, further validating the reliability and effectiveness of the proposed method. Additionally, the visualization results intuitively demonstrate the superiority of the method.
The article is clear in writing, easy to understand and follow.
limitations: limitations: -It is suggested to supplement the implementation details by specifying the specific dimensions of parameters such as $D$ and $P$ in the Section 3.2, and to clarify whether the Temporal Mask Learning Branch is inactive during the inference phase.
-The performance under the 5-way 5-shot setting on UCF101 and HMDB51 is limited.
-The abbreviations of modules such as TML and SGTM should be introduced earlier in the paper rather than being mentioned only in Section 4.3.
-In Figure 6, it is suggested to label "blue", "green" and "red" in the captions with their respective colors for better clarity and visual understanding.
-  The statements and explanations in Section 4.2 may be somewhat redundant. A more concise version is encouraged.

-  Some studies utilize multimodal frameworks for few-shot action recognition, such as CLIP-FSAR[1] and AMFAR[2]. Please include a discussion of these methods in Section2.1.

[1] Wang, Xiang, et al. "CLIP-guided prototype modulating for few-shot action recognition." International Journal of Computer Vision. 2023.
[2] Wanyan, Yuyang, et al. "Active exploration of multimodal complementarity for few-shot action recognition." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.