# A Multi-Scale Video Longformer Network for Action Recognition
## 3 Method
### Patchemed  
The PatchEmbed module consists of a Position Embedding Layer, a Flatten Layer, a Layer Normalization Layer, a Position Encoding Layer, a Dropout Layer, and a global token. It combines all tokens from the previous stage's patches into a single token for the current stage, achieving spatial downsampling of video features while increasing their depth. The Position Embedding Layer uses Conv3D to split the input video into different patches in the spatial dimension; the video features sequentially pass through the Flatten layer and the LayerNorm layer, concatenate with positional encoding, and finally undergo Dropout before output. As videos exhibit variations in both spatial and temporal dimensions, encoding the spatial and temporal position information within the video is essential; position encoding is applied to both local tokens and global tokens as shown in Equation (1).  
\[X=[x_{g}^{1},\ldots;x_{g}^{n};LN(x_{1}^{1}E);\ldots;LN(x_{T}^{n}E)]\] \[+Pos(x_{g}^{1},\ldots;x_{g}^{n},x_{1}^{1};\ldots;x_{T}^{n}), \tag{1}\]  
where \(x_{g}^{n}\) represents global tokens, \(LN\) denotes the LayerNorm layer, \(E\) represents the position embedding layer, and \(x_{T}^{n}\) represents the \(n\)th local token in the temporal dimension \(T\).  
We implement both absolute 3D position encoding and relative 3D position encoding. Absolute 3D position encoding initializes the learnable tensors using a truncated normal distribution to encode coordinates (\(t\), \(x\), \(y\)) in the temporal and spatial dimensions. These coordinates are concatenated and added to the input features. During training, these parameters are continuously updated, as illustrated in Equation (2).  
\[Pos_{ape}(x_{g}^{1},\ldots;x_{g}^{n},x_{1}^{1};\ldots;x_{T}^{n})=\] \[\text{Cat}(x_{g}^{1},\ldots;x_{g}^{n},\text{Cat}(X_{\text{pos}},Y_ {\text{pos}})).\text{repeat}(T), \tag{2}\]  
where Cat represents the concatenation operation, \(X_{\text{pos}}\) and \(Y_{\text{pos}}\) denote the \(x\) and \(y\) coordinates in the spatial dimension, and \(T\) represents the temporal dimension.  
The relative 2D position encoding typically involves adding position biases during self-attention calculations, as shown in Equation (3). \(Q=XW^{\text{G}}\), \(K=XW^{\text{K}}\), \(V=XW^{\text{V}}\), and \(d_{k}\) represent the dimensions of matrix \(K\), and \(B\) is the positional bias.  
\[\text{Attention }(Q,K,V)=\text{softmax}\bigg{(}\frac{QK^{T}}{\sqrt{d_{k}}} +B\bigg{)}V. \tag{3}\]  
However, this method lacks absolute spatiotemporal position information and may not be suitable for video features with both temporal and spatial dimensions. CPE [37] has proved that zero padding 3D depthwise separable convolution can overcome permutation invariance, allowing any local token at arbitrary positions to obtain its absolute spatiotemporal position by querying local tokens located at the boundary positions. In this paper, a kernel size of (3, 3, 3), a stride of (1, 1, 1), and padding of (1, 1, 1)--along with the number of groups equal to the dimensions of the current stage--are employed in 3D depthwise separable convolution to calculate the 3D relative positional encoding of local tokens, as shown in Equation (4).  
\[Pos_{ape}(x_{g}^{1},\ldots;x_{g}^{n};x_{1}^{1},\ldots;x_{T}^{n})=\] \[\text{Cat}(\text{DwConv}_{3D}(x_{1}^{1};\ldots;x_{T}^{n}),x_{g}^ {1};\ldots;x_{g}^{n}), \tag{4}\]  
where \(\text{DwConv}_{3D}\) represents 3D depthwise separable convolution, and the global token utilizes absolute positional encoding initialized with truncated normal distribution.