SOAP: Enhancing Spatio-Temporal Relation and Motion Information Capturing for Few-Shot Action Recognition

Authors: wenbo  huang
jinghui  zhang
xuwei  qian
meng  wang
lei  zhang
zhen  wu
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-05 15:12:34

Abstract: High frame-rate (HFR) videos of action recognition improve fine-grained expression while reducing the spatio-temporal relation and motion information density. Thus, large amounts of video samples are continuously required for traditional data-driven training. However, samples are not always sufficient in real-world scenarios promotes few-shot action recognition (FSAR) research. We observe that most recent FSAR works build spatio-temporal relation of video samples via temporal alignment after spatial feature extraction, cutting apart spatial and temporal features within samples. They also capture motion information via narrow perspectives between adjacent frames without considering density, leading to insufficient motion information capturing. Therefore, we propose a novel plug-and-play architecture for FSAR called $\underline{\textbf{S}}$patio-temp$\underline{\textbf{O}}$ral fr$\underline{\textbf{A}}$me tu$\underline{\textbf{P}}$le enhancer ($\textbf{SOAP}$) in this paper. The model we designed with such architecture refers to SOAP-Net. Temporal connections between different feature channels and spatio-temporal relation of features are considered instead of simple feature extraction. Comprehensive motion information is also captured, using frame tuples with multiple frames containing more motion information than adjacent frames. Combining frame tuples of frame counts further provides a broader perspective. SOAP-Net achieves new state-of-the-art performance across well-known benchmarks such as SthSthV2, Kinetics, UCF101, and HMDB51. Extensive empirical evaluations underscore the competitiveness, pluggability, generalization, and robustness of SOAP. The code will be released.

Main paper: 
 introduction: 
 Ubiquitous videos in daily lives are rapidly accelerating the development of multimedia analytic research. As a fundamental task, action recognition is experiencing an explosive demand in a wide range of applications including intelligent surveillance, video understanding, and health monitoring [4, 15, 20] . Progress in video recorders contributes to high frame-rate (HFR) videos, with more similar frames per second improving the expression of fine-grained actions [17] . As the example shown in Figure 1 , we can explicitly observe that the timeline and displacement of object in HFR video frames are much subtler than those in low frame-rate (LFR) video frames, better reflecting fine-grained actions. However, the spatiotemporal relation and motion information density decrease with the improvement of video fluency [36] . Therefore, a larger amount of video samples are continuously required to train data-driven models. Unfortunately, samples for target actions such as "falling down" are usually insufficient and hard to collect in real-world scenarios. Contemporary few-shot learning mitigates data dependence by transferring knowledge from a few samples, promoting few-shot action recognition (FSAR) research.According to data characteristics of HFR videos, two prevailing challenges of FSAR exist. Challenge 1: Optimizing spatio-temporal relation construction. Spatial and temporal features work as a whole in video samples, only focusing on spatial information makes models misidentify horizontal or vertical actions such as "pushing", "pulling", "putting up", and "putting down". However, the spatiotemporal relation of HFR videos is subtle, making the construction challenging. Challenge 2: Comprehensive motion information capturing. As an exclusive characteristic of videos, motion information plays a crucial role in helping models recognize target actions in a dynamic manner. However, the difficulties in motion information capturing is exaggerated by low motion information density from HFR videos and limited frames processed at one time with mainstream methods. In summary, the challenges mentioned above not only stem from HFR videos but are also aggravated by the insufficient samples in few-shot settings.In FSAR, the metric-based paradigm predominates due to its simplicity and efficacy. It embeds samples into action prototypes to calculate support-query distances for classification in an episodic task [1, 18, 30, 41, 43] . We observe that these works prioritize temporal alignment after spatial feature extraction, cutting apart spatial and temporal features within samples and overlooking the importance of motion information capture. Some works [29, 31, 33] incorporate motion information into action recognition and achieve remarkable performance. However, the motion information is captured between adjacent frames. This narrow perspective inevitably overlooks density of motion information and results in inadequate capturing. So far, no solution exists for both challenges.Motivated by aforementioned challenges, we propose a novel plug-and-play architecture for FSAR called Spatio-tempOral frAme tuPle enhancer (SOAP). The model we designed with such architecture is defined as SOAP-Net. The cores are optimizing construction of spatio-temporal relation and capturing comprehensive motion information. For the first goal, simply extracting features from video frames is insufficient. These features are located in different channels and have temporal connections between each channel. Spatio-temporal relation within features also play a key role in FSAR. For the second goal, we consider motion information density and find that frame tuples with multiple frames contain richer motion information than adjacent frames. Combining frame tuples of various frame counts further provides a broader perspective. To be specific, SOAP has three components: 3-Dimension Enhancement Module (3DEM) uses a 3D convolution for spatio-temporal relation construction and Channel-Wise Enhancement Module (CWEM) calibrates channel-wise feature responses adaptively while Hybrid Motion Enhancement Module (HMEM) applies a broader perspective to help models capture comprehensive motion information. These three modules work in parallel, adding priors into raw input.To our best knowledge, SOAP is the first to address all challenges simultaneously. Our main contribution are three-fold:• Spatio-Temporal Relation Construction. The proposed SOAP optimizes spatio-temporal relation construction, avoiding simply operating temporal alignment after extracting spatial features. • Motion Information Capturing. Taking motion information density and the processing method into account, our SOAP finds a comprehensive solution, overcoming the second challenge with a broader perspective that combines frame tuples with various frame counts.  
related works 2.1 few-shot learning: 
 The core goal of few-shot learning is to recognize unseen classes from only few samples. Unlike traditional deep learning, few-shot learning utilizes episodic training where training units are structured as similar tasks with small labeled sets. Three primary groups of few-shot learning are data-augmentation based, optimizationbased, and metric-based paradigms. In the first data-augmentation paradigm, generating additional samples to supplement available data is a key feature. Specifically, MetaGAN [39] utilizes generative adversarial networks (GANs) and statistics of existing samples to synthesize data. The most representative optimization-based method is MAML [7] , which identifies a model parameters set and adapts it to individual tasks via gradient descent. The metric-based paradigm is well-known and widely used for its simplicity and effectiveness. It leverages similarity between support samples to classify query samples into corresponding classes. Prototypical Networks [22] construct prototypes based on class centroids and then classify samples by measuring distance to each prototype. Most few-shot learning paradigms are applied to image classification, with fewer in the field of videos. Our SOAP belongs to the metric-based paradigm for FSAR, focusing on improving prototype representation ability.Compared to image classification, action recognition is a more complex and extensively researched problem in the community due to the spatio-temporal relation and motion information. Previous works [3, 19, 26, 32] have typically utilized 3D backbones to construct spatio-temporal relations, while optical flow is applied by additional networks to inject video motion information for action recognition, resulting in promising results. However, these works with high acquisition costs are all designed for traditional data-driven training without considering insufficient samples in real-world scenarios. In contrast, SOAP is specifically designed for a more realistic few-shot setting.Existing methods of FSAR mainly focus on metric-based paradigm for effective prototype construction. Among them, CMN [43, 44] introduces a multi-saliency embedding algorithm for key frame encoding, improving the prototype representation; ARN [38] captures short-range dependencies using 3D backbones with a selftrained permutation invariant attention; OTAM [1] aligns support and query with a dynamic time warping (DTW) algorithm. Joint spatio-temporal modeling approaches such as TA 2 N [13] , ITA [40] , STRM [24] , and SloshNet [34] employ spatial-temporal frameworks to address the support-query misalignment from multiple perspectives; Temporal relation is emphasized by TRX [18] , using a Cross Transformer for sub-sequence alignment and achieving notable improvement; HyRSM [30] learns the task-specific embedding by exploiting the relation within and cross videos; SA-CT [41] complement the temporal information by learning the spatial relation.The temporal alignment all operated after spatial feature extraction. Diverse types of information including optical flow [31] and depth [8] are introduced by extra networks, increasing computations. MoLo [29] and MTFAN [33] explicitly extract motion information from raw videos and leverage temporal context for FSAR.Although achieving promising performance, these works are either cut apart spatial and temporal features or ignore motion information density. Therefore, our SOAP focuses on spatio-temporal relation and motion information capturing. 
methodology 3.1 problem formulation: 
 Classifying an unlabeled query into one of the support classes is the inference goal of FSAR. The support set has at least one labeled sample per class. Following the prior works [1, 18, 29, 30] , we randomly select few-shot tasks from training set for episodic training. In each task, the support set 𝑆 contains 𝑁 classes and 𝐾 samples per class, i.e., 𝑁 -way 𝐾-shot setting. The query set 𝑄 in each task has samples to be classified. Uniformly sampling 𝐹 frames of each video, we define theEQUATIONNotions used are 𝐹 (frames), 𝐶 (channels), 𝐻 (height), and𝑊 (width), while the sample with 𝐹 frames of query is defined as:EQUATIONModels of FSAR need to predict labels for query samples with the guidance of the support set.An overview of SOAP-Net is provided in Figure 2 . For clear description, we take the 𝑄 and 𝑐 th of 𝑆 as an example to illustrate the whole process of our method. First, video samples are decoded into frames at fixed intervals to serve as the input. Then, support and query are sent into three main modules of SOAP, i.e., 3DEM, CWEM, and HMEM. To be specific, 3DEM establishes spatio-temporal relation of features, while CWEM adaptively calibrates temporal connections between channels. Instead of solely concentrating on motion information between adjacent frames, HMEM adopts a broader perspective on frame tuples of varying frame counts, delivering comprehensive motion information through a hybrid approach. The three modules are arranged in parallel to generate triple prior guidance before feature extraction. Several linear layers are subsequently adopted for prototype construction. Finally, the distance between query and prototype can be calculated for classification.For the relation construction between spatial and temporal information, 3DEM is designed based on 3D convolution. The structure of this module is shown in Figure 3 . Firstly, 3DEM averages the support and query across the channels, resulting in spatio-temporal tensors. Then the corresponding tensors are reshaped. The above operation can be formulated as:EQUATIONA 3D convolution is used for constructing the spatio-temporal relation and another reshape operation is used for shape recovery:EQUATIONFinally, spatio-temporal relation are fed into a Sigmoid activation and residually connected with the input to generate the 3D prior knowledge before feature extraction:EQUATIONFor each input channel, 3DEM can assist the spatio-temporal relation construction.Features located in different channels have temporal connections between each channel. Inspired by SE [11] , we design this module with the addition of a simple 1D convolution. The structure of CWEM is detailed in Figure 4 . First, a spatial average pooling is performed on the support and query, followed by a 2D convolution that expands the channel number to 𝐶 𝑟 , represented as: The third reshape operation prepares for the 1D convolution, which adaptively calibrates channel-wise feature responses, as:EQUATIONEQUATIONOnce channel-wise feature responses are calibrated, the next steps are reshape and 2D convolution. Together, these operations recover the original dimension and channel numbers of the feature maps. This process can be formulated as follows:EQUATIONThe final step of CWEM is formulated similarly to Eqn (5), using Sigmoid and residual connection, to generate channel prior knowledge before feature extraction:EQUATIONIn previous works [29, 31] , motion information plays a significant role. However, we find that focusing only on the motion information between adjacent frames is insufficient, as subtle displacements are hard to detect. In order to capture more motion information using HMEM, we extend perspective from adjacent frames to frame tuples and apply the combination of multiple scales. The structure of HMEM is demonstrated in EQUATION)We concatenate tuple differences in the first dimension, preparing for motion information calculation. For a simple description, we choose the 𝑒 th (𝑒 ∈ [1, |O|] and 𝑒 ∈ N * ) branch as an example:EQUATIONMotion information with different scales can be captured by setting multiple branches. Then we concatenate them along the first dimension, achieving tensors belong to R 𝑋 ×𝐶 ×𝐻 ×𝑊 . Function 𝑍 (•) that performs flatten (R 𝑋 ×𝐶 ×𝐻 ×𝑊 ↦ → R 𝑋 × (𝐶 ×𝐻 ×𝑊 ) ), linear transformation (R 𝑋 × (𝐶 ×𝐻 ×𝑊 ) ↦ → R 𝐹 × (𝐶 ×𝐻 ×𝑊 ) ), and reshape operation (R 𝐹 × (𝐶 ×𝐻 ×𝑊 ) ↦ → R 𝐹 ×𝐶 ×𝐻 ×𝑊 ) in sequence, acquiring comprehensive motion information. This process is concluded as:EQUATIONA spatial average pooling is operated for reducing computation, as:EQUATIONSimilar with the prior knowledge calculated by two previous modules, the hybrid motion prior can be represented as:EQUATIONTriple prior guidance are added into the raw support and query before feature extraction:EQUATION)S𝑐𝑘 and Q are sent to the backbone network. Features of support and query are defined as: 𝑆 𝑐𝑘 𝑓 = 𝑓 𝜃 s𝑐𝑘 1 + 𝑓 𝑝𝑒 (EQUATION)The LN (•) means the layer normalization and Softmax (•) represents the Softmax function.The linear layer Λ (•) is applied on 𝑄 𝑓 to ensure the same operation as 𝑆 𝑐𝑘 𝑓 . By obtaining the closest distance between 𝑄 𝑓 and 𝑃 𝑐 , model can predict the label ỹ of 𝑄, as:EQUATIONThe cross-entropy loss L ce is selected as the objective loss for training, which can be calculated with the ground truth 𝑦:EQUATION4 Experiments 4.1 Experimental Configuration 4.1.1 Datasets Processing. We select four widely used datasets, i.e., SthSthV2 [9] , Kinetics [2] , UCF101 [23] , and HMDB51 [12] . Apart from SthSthV2, which is temporal-related, other three datasets are all spatial-related. When decoding videos, we set the sampling intervals to every 1 frame. For simulating various fluency, the intervals can be adjusted. Building upon data split from prior works [1, 38, 43] , we split our datasets into training, validation, and testing sets. A task within each set, classifying query samples into one of the support classes serves as a training unit.Following TSN [27] , 8 frames (𝐹 = 8) resized to 3 × 256 × 256 are uniformly and sparsely sampled each time. Data augmentations in training include random crops to 3 × 224 × 224 and horizontal flipping, while only a center crop is used during testing. As pointed out in previous works [1] , many action classes in SthSthV2 are sensitive to horizontal direction (e.g., "Pulling S from left to right"1 ). Hence, we avoid horizontal flipping on this dataset.Table 1 : Performance (↑ Acc. %) comparison. "★": our implementation, " †": multimodal methods, "N/A": not available in the publication, Bold texts: the best results, underline texts: the previous best results.Backbone SthSthV2 Kinetics UCF101 HMDB51 1-shot 5-shot 1-shot 5-shot 1-shot 5-shot 1-shot 5-shot CMN [43, 44] ResNet 4.1.2 Implementation Details and Evaluation. In 3DEM, the size of Conv3D is 3 × 3 × 3. For CWEM, the expand channel number 𝐶 𝑟 is set to 16, while Conv1D size is 3. We set O = {1, 2, 3} in HMEM, the Conv2D 𝑀 with shared parameters are also 3 × 3 in each branch. All 2D convolution for changing channel number is set to 1 × 1. Except for these channel recovery convolutions, other convolutions maintain the same shape of input and output. We employ widely-used ResNet-50 [10] or ViT-B [6] as the backbone initialized with pre-trained weights on ImageNet [5] . The final outputs of the backbone are 2048-dimensional vectors (𝐷 = 2048).For three linear layers, the parameters are randomly initialized while 𝑑 𝑘 and 𝑑 𝑣 are both set to 1152. We utilize the standard 5-way 5-shot and 1-shot few-shot settings. Due to its larger size, SthSthV2 requires 75,000 tasks for training, while other datasets employ 10,000 tasks. An SGD optimizer is applied to train our model, with an initial learning rate of 10 -3 . The training process is conducted on a deep learning server equipped with two NVIDIA 24GB RTX3090 GPUs. Validation set determines hyperparameters. During testing, we report average accuracy across 10,000 random tasks from the testing set.Comparison with ResNet-50 Backbone Methods. Based on the average accuracy reported in Table 1 , we have the following observation. Our SOAP-Net outperforms other methods and achieves SOTA performance. For example, on the Kinetics dataset under the 1-shot setting, SOAP improves the current SOTA performance of MoLo [29] from 75.2% to 81.1%. We believe the motion information within adjacent frames is insufficient, despite MoLo introducing it. Similar improvements are also found in other datasets under diverse few-shot settings, showing that spatio-temporal relation and comprehensive motion information from SOAP lead to significant improvement in FSAR. It is worth mentioning that SOAP-Net even surpasses multimodal methods. 
comparison with: 
 ViT-B Backbone Methods. ResNet-50 serves as the backbone network in most previous works, while ViT-B is rarely used in FSAR. For a comprehensive comparison, we implemented several methods and replaced their backbones with ViT-B, finding it outperformed ResNet-50 in FSAR due to larger model capacity. The previous SOTA performance is also achieved by MoLo, benefiting from optimizing spatio-temporal relation construction and comprehensive motion information. A similar trend is observed on ResNet-50 and ViT-B. These comparisons reveal the competitiveness of our proposed method. 2 , results indicate that each SOAP component can improve FSAR performance. However, the most significant boost comes from HMEM, likely due to fuller motion information usage. Combining two components with HMEM performs better than without HMEM. We conclude that comprehensive motion information plays a more significant role in FSAR. Using all SOAP components yields the best results, proving all are essential. The CAM [21] for "crossing river" from the Kinetics dataset is shown in Figure 6 . The first row shows raw HFR video frames. We observe that the timeline and displacement of the moving peoples are very subtle, making it difficult to detect the spatio-temporal relation and motion information. In the second row, we see that the model attends more to the background than moving peoples. This result supports our observation in HFR videos. With the help of SOAP, we detect that the focus shifts from the background to the moving peoples, despite the subtle timeline and displacement. Based on this analysis of the CAM, we conclude that motion information is crucial in FSAR. Spatio-temporal relation and comprehensive motion information from SOAP can significantly improve feature representation. In the 5-way task, feature distributions can be visualized using t-SNE [25] . We select five representative yet challenging support action classes from Kinetics: "ski jumping", "snowboarding", "skateboarding", "crossing river", and "driving tractor". The t-SNE visualization is shown in Figure 7 , with different colors and markers representing distinct action classes. Without SOAP (Figure 7 (a)), the five classes are generally separable but with some overlapping, such as in the right ("driving tractor" Blue Rabbits: "ski jumping", Orange Dogs: "crossing river", Green Cats: "driving tractor", Red Monkeys: "snowboarding", Purple Pigs: "skateboarding".and "crossing river"), left ("skateboarding" and "snowboarding"), and center ("ski jumping"). By contrast, with SOAP (Figure 7 (b)), the five classes are more distinctly separated and same-class samples are more tightly clustered. This strong evidence demonstrates the enhanced representational ability of SOAP for support features.4.5.1 On RGB-Based Methods. In Table 5 , we experimentally demonstrate that the SOAP generalizes well to other methods by inserting it into widely used methods including TRX, HyRSM, and MoLo. Using Kinetics as an example, TRX benefits from comprehensive motion information and achieves 7.6% gains in the 1-shot setting and 8.5% in the 5-shot setting. Similar improvements are also observed in the other two methods. These results fully prove that optimizing spatio-temporal relation and comprehensive motion information are particularly useful for feature extraction. In this study, we aim to simulate different levels of frame-rate by varying the sampling interval from a minimum of 1 to a maximum of 6 and conduct a series of experiments under a well-established 5-way 5-shot setting. As shown in Figure 8 , the performance tends to decrease gradually with the increase of frame-rate. We find that recent FSAR methods all experience drastic degradation in performance with HFR videos while SOAP-Net indicates performance stability, demonstrating the superiority of SOAP across varying frame-rate. These phenomena also highlight the key role of spatiotemporal relation optimization and comprehensive motion information in advancing FSAR performance. 6 
conclusion: 
 In this paper, we propose the plug-and-play SOAP to optimize spatio-temporal relation construction and capture comprehensive motion information for FSAR. SOAP takes into account temporal connections across feature channels and spatio-temporal relation within features. It combines frame tuples of various frame counts to provide a broader perspective for motion information. Considering the competitiveness, pluggability, generalization, and robustness of SOAP, we hope and believe that our work will offer valuable insights for future research in multimedia analytic. 
 
 Appendix: "S" refers to "something" 


Decision: This work is focused on few-shot action recognition. It received weak accept, borderline reject and borderline accept ratings initially and the authors provided a rebuttal. After the rebuttal most of the concerns were addressed and the reviewers provide 2x weak accept and borderline accept as final ratings. The AC agree with the assessment and recommend this work for acceptance. The authors should address the comments from the reviewers in preparing the final submission.

Reviews: review: paper_summary: paper_summary: This paper proposes a novel plug-and-play architecture called Spatio-temporal frame tuple enhancer (SOAP) for few-shot action recognition. In this paper, temporal connections between different feature channels and spatio-temporal relation of features are considered instead of simple feature extraction.
strength_weakness: strength_weakness: SOAP-Net achieves new state-of-the-art performance across well-known benchmarks such as SthSthV2, Kinetics, UCF101, and HMDB51. Extensive empirical evaluations underscore the competitiveness, pluggability, generalization, and robustness of SOAP.
limitations: limitations: 1. The timeline and displacement of object in HFR video frames are much subtler than those in low frame-rate (LFR) video frames, better reflecting fine-grained actions. However, the spatio-temporal relation and motion information density decrease with the improvement of video fluency. How can this method extract accurately fine-grained features from high motion information density LFR videos with sparse fine-grained features?
2. What multimodal tasks does the multimodal methods address in Section 4.5.2?
3. In Figure 6, the difference between the video frames is very small. In this case, how can the model capture the temporal features or relationships of the videos?
paper_summary: paper_summary: In order to better construct spatiotemporal relations and focus on motion information in FSAR, the paper designs spatial, channel, and temporal attention models. The proposed method achieves new SOTA performance on several well-known FSAR benchmarks.
strength_weakness: strength_weakness: -  Technical correctness.
- Evaluation: the proposed method achieves new SOTA performance on several well-known FSAR benchmarks. Comprehensive experiments demonstrate the competitiveness, pluggability, generalization, and robustness of the proposed method.
- Clear presentation.
limitations: limitations: -  Novelty: 1. This article places attentions before the backbone, which is different from the previous methods. Is this a reason for the superior performance of the proposed method? In other words, just adding other attention before the backbone will produce good results? 2. The authors' use of prototype construction is also different from the previous methods . And in Table 2, only using prototype construction surpasses many methods. Does this mean that the outstanding performance of proposed method comes from the prototype construction?
paper_summary: paper_summary: The paper introduces a novel plug-and-play architecture called SOAP-Net, designed to enhance spatio-temporal relation and motion information capturing for few-shot action recognition (FSAR). The authors identify two main challenges in FSAR: optimizing spatio-temporal relation construction and comprehensive motion information capturing. To address these, SOAP-Net employs a 3D convolution-based module for spatio-temporal relation construction, a channel-wise enhancement module for temporal connections, and a hybrid motion enhancement module for capturing motion information across multiple frames. The model achieves state-of-the-art performance on benchmarks such as SthSthV2, Kinetics, UCF101, and HMDB51.
strength_weakness: strength_weakness: ## Strengths

Novelty: SOAP-Net presents a novel approach to FSAR by integrating spatial and temporal features in a more cohesive manner than previous methods. The use of frame tuples and a hybrid approach for motion information is particularly innovative.

Technical Approach:  The paper's technical approach is sound, with a well-defined problem formulation and a clear explanation of the SOAP-Net architecture. The use of 3D convolutions and channel-wise enhancements is theoretically justified and aligns with the current trends in action recognition research.

Evaluation: The authors provide an extensive experimental evaluation, demonstrating SOAP-Net's effectiveness across multiple datasets. The comparison with state-of-the-art methods and ablation studies further validates the model's performance.

Clarity: The paper is well-organized, with a clear presentation of the methodology, experiments, and results. The figures and tables are informative and enhance the reader's understanding of the proposed approach.

Applications: The potential applications of SOAP-Net are broad, including intelligent surveillance, video understanding, and health monitoring, which demonstrates the practical relevance of the research.
limitations: limitations: CWEM is essentially a form of channel attention mechanism, and there is a lack of experimental comparison with other channel or spatial attention mechanisms such as SENet and CBAM.

SENet: Hu J, Shen L, Sun G. Squeeze-and-excitation networks[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 7132-7141.
CBAM： Woo S, Park J, Lee J Y, et al. Cbam: Convolutional block attention module[C]//Proceedings of the European conference on computer vision (ECCV). 2018: 3-19.