# A Multi-Scale Video Longformer Network for Action Recognition
## 2 Related Works  
Existing action recognition methods can be classified into two categories based on how they extract features: convolution-based methods and attention-based methods. Convolution is effective at capturing local semantic information within a window. [PERSON] et al. [9] proposed the two-stream convolutional network, which uses spatial and temporal stream neural networks to perform 2D convolution operations on RGB and optical flow, respectively, to extract appearance and motion information from videos. [PERSON] et al. [19] proposed the TSN network to process longer video sequences by sparsely sampling the video and extracting a single frame from each segment. C3D [11] demonstrated the effectiveness of 3D convolutional networks in extracting spatiotemporal features from videos and explores the optimal 3D convolutional kernel structure as 3 \(\times\) 3 \(\times\) 3. [PERSON] et al. [12] presented the inflate and bootstrap methods, expanding the 2D network structure and weights trained on ImageNet to a 3D network structure and weights, making it easier to use mature 2D network structures and weights. R(2 + 1)D [20] combined 2D convolution and 3D convolution to reduce computation and memory usage. ACRN [21] modeled spatiotemporal relations with weakly supervised learning to capture interactions between actors, objects, and scenes to differentiate human actions. [PERSON] [22] introduced a long-term feature bank, which involves extracting supportive information from the entire span of a video to augment video models. To better balance the accuracy and speed of the network, [PERSON] et al. [23] proposed the SlowFast network, including a slow and a fast branch, to extract motion and scene information from videos, respectively. Convolutional methods are limited to capturing local information within a fixed window, which poses challenges for comprehensively considering global information. In contrast, transformers can leverage self-attention mechanisms to represent global features. To model long-distance semantic information, [PERSON] et al. [24] proposed the insertable non-local spatiotemporal self-attention module. [PERSON] et al. [16] introduced the TimeSformer, which applies self-attention separately in the temporal and spatial dimensions to reduce computational complexity. [PERSON] et al. [25] proposed the action transformer model to capture semantic context from other people's actions. ViViT [26] investigated the impact of different spatiotemporal attention structures on network performance, while VidTr [27] used separable attention mechanisms to reduce redundancy in videos, thus decreasing memory consumption. TadTR [28] effectively reduced computation costs by selectively attending to sparse key segments in videos. STPT [29] balanced efficiency and accuracy by employing local window attention and global attention at both early and late stages to extract semantic information. MeMViT [30] combined multimodal information by storing past visual and textual features to capture long-term dependencies in videos. MaskFeat [31] employed masks to guide feature learning in videos, enhancing the feature representation of objects in video frames. VideoMAE [32] enhanced the similarity between videos of the same class and distinguishes videos of different classes through a self-supervised paradigm of masking and reconstruction. MViT [17] and VideoSwin [33] reduced transformer resource consumption through multi-scale feature hierarchical structures and shifted window operations. UniFormer [34] combined both CNN and transformer structures to mitigate local redundancy and effectively model global features. Transformers exhibited quadratic computational complexity as the token count increased during calculations, limiting their ability to model global features effectively. To address this issue, Longformer employed sliding window attention to calculate local context, thereby reducing local redundancy and maintaining linear complexity with the sequence length.