# A Multi-Scale Video Longformer Network for Action Recognition
## 3 Method
### Overall Network Architecture  
The overall structure of the MSVL is illustrated in Figure 1, which is composed of a stack of 3D Longformer structures, consisting of four stages and a final fully connected layer. As video features enter each stage, the spatial resolution decreases while the depth increases. To ensure effective extraction of temporal information from the video, temporal subsampling is applied only in the first stage, reducing the temporal dimension by a factor of 2. The subsequent stages maintain the temporal dimension unchanged. Each stage includes a complete 3D Longformer structure, consisting of three main components: the PatchEmbed, the 3D Longformer AttenBlock, and the MlpBlock, as depicted in Figure 2. In the PatchEmbed module, the input video, represented as \(x\in R^{(H\times W\times 3\times T)}\), is divided into \(N\) patches of size \(P\times P\) in the spatial dimensions, where \(N=H\times W/P^{2}\). Subsequently, it passes through Flatten and LayerNorm layers, and positional encoding is added, along with global tokens. PatchEmbed further reduces spatial resolution and increases depth by combining all tokens from the previous stage's patches. The 3D Longformer AttenBlock extracts features by computing attention scores, while the MlpBlock enhances the extracted spatiotemporal features.  
The detailed configuration parameters of the MSVL are provided in Table 1, where \(n\) represents the number of 3D Longformer AttenBlocks and MlpBlocks in each stage. _patchsize_ refers to the spatial dimensions of the patches, _dims_ is the depth of the output video features, _heads_ and _heads_dims_ denote the number of attention heads and the dimension of each head, where the dimension of each head is equal to the depth of the output video features divided by the number of attention heads. _num_global_ specifies the number of global tokens. \(T\), \(H\), and \(W\) represent the dimensions of the video features, where we sample 16 frames per video, and each frame has a size of 224 \(\times\) 224. _window size_ defines the window size for self-attention in the spatial dimensions.  
\begin{table}
\begin{tabular}{l l l l l} \hline \hline
**Name of Parameter** & **Stage1** & **Stage2** & **Stage3** & **Stage4** \\ \hline \(n\) & \(1\) & \(2\) & \(8\) & \(1\) \\ _patchsize_ & \(4\) & \(2\) & \(2\) & \(2\) \\ _dims_ & \(96\) & \(192\) & \(384\) & \(768\) \\ _heads_ & \(3\) & \(3\) & \(6\) & \(12\) \\ _heads_ & \(32\) & \(64\) & \(64\) & \(64\) \\ \(T\) & \(8\) & \(8\) & \(8\) & \(8\) \\ _num\_global_ & \(8\) & \(8\) & \(8\) & \(8\) \\ \(H\times W\) & \(56\times 56\) & \(28\times 28\) & \(14\times 14\) & \(7\times 7\) \\ _window\_size_ & \(15\times 15\) & \(15\times 15\) & \(15\times 15\) & \(15\times 15\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Network configuration parameters.  
Figure 1: The overall structure of MSVL. MSVL consists of 4 stages, where the spatial resolution of the video decreases, and the depth increases as the video progresses through each stage.  
Figure 2: Structure diagram for each stage, which is composed of three parts: PatchEmbed, 3D Longformer AttenBlock, and MlpBlock. The quantity of 3D Longformer AttenBlocks and MlpBlocks differs between stages, but each stage always begins with a PatchEmbed.