# A Multi-Scale Video Longformer Network for Action Recognition
## 1 Introduction  
With the rapid growth of the internet and the rise of short video platforms, there has been an exponential increase in the number of videos available online. Understanding and categorizing these massive volumes of videos has become an urgent challenge [1]. Action recognition is one of the challenging tasks in the field of video understanding [2], with wide-ranging applications in video classification, short video recommendations, and security surveillance [3]. Early action recognition methods primarily depended on handcrafted features, representing videos with multiple local descriptors [4; 5]. Nevertheless, these handcrafted feature extraction methods were effective only in limited scenarios due to their lack of adaptability to different types of data. With the rapid development of deep learning in image recognition and detection, an increasing number of deep learning-based action recognition methods have emerged in recent years. These deep learning-based methods can generally be categorized into two groups: those based on convolutional neural networks (CNNs) [6] and those based on transformers [7]. Unlike static images, videos encompass both spatial and temporal dimensions of feature information. Effectively extracting and utilizing these two types of information from continuous video frames is crucial for action recognition [8]. Two-dimensional (2D) CNNs can effectively extract high-level semantic information in the spatial dimension from video frames, while they lack sufficient capability to integrate spatio-temporal features. [PERSON] et al. [9] proposed a two-stream 2D CNN architecture, which enhances the network's spatio-temporal modeling capabilities by integrating features from both RGBframes and optical flow frames. However, extracting optical flow is time-consuming, and changes in scenes and lighting in the sample may lead to errors in optical flow recognition, thereby reducing the accuracy of action recognition. In contrast to 2D CNNs, 3D CNNs demonstrate greater efficacy in representing video features, directly modeling samples in the spatio-temporal dimension and effectively capturing motion information in videos [10; 11; 12]. However, a prevalent limitation associated with CNN-based methods is their focus on local information. This focus inherently restricts their capacity to model global features, especially those beyond the scope of their receptive fields [13; 14]. Originally developed for natural language processing, transformers compute attention for each token in relation to all other tokens in the input, enabling them to capture long-range dependencies effectively. This characteristic makes them particularly adept at understanding the context within sequences. With the application of vision transformers (ViTs) [15] to image classification, the exceptional capabilities of the self-attention mechanism have been showcased. This has spurred an increasing number of works applying transformers to action recognition tasks. For instance, TimeSformer [16] utilizes temporal and spatial attention to process video data, effectively extracting time-based and spatial features. MViT [17] enhances network performance and reduces computational consumption by integrating the multiscale feature hierarchies with transformer models. Nevertheless, as the sequence length (number of tokens) increases, the computational complexity grows quadratically. This sensitivity to token quantity restricts the transformer's ability to model global features effectively. In this paper, we introduce a 3D Longformer [18] structure and utilize it to construct a Multi-Scale Video Longformer network for action recognition. Our MSVL comprises four stages, progressively reducing video feature resolution while increasing feature dimensions. In stages 1 and 2, it reduces local redundancy through local window attention and maintains global features with global tokens. In stages 3 and 4, the local window attention turns into a dense computation mechanism to enhance performance.