## 3 Methods
### Framework for Spatiotemporal Feature Extraction Based on Dual-Stream Networks
#### 3.1.1 STA-Block  
Multi-Scale Transposed Attention  
As mentioned above, we aim to address two main challenges: significant local redundancy and intricate global dependencies, aiming for efficient and effective spatiotemporal representation learning. However, existing methods, such as popular 3D CNNs and spatiotemporal transformers, often focus solely on one of these challenges. Therefore, we introduce a novel approach called Multi-Scale Transpose Attention (MSTA). Designed in a concise transformer format, MSTA seamlessly unifies 3D convolution and spatiotemporal self-attention, adeptly tackling video redundancy and dependencies at different levels in both shallow and deep layers. Due to the substantial computational overhead of transformers, primarily from the self-attention layer, applying the traditional self-attention mechanism (SA) [30; 31] becomes impractical for most video-understanding tasks. In the conventional self-attention mechanism, the time and memory complexity of key-query dot-product interactions grows quadratically with the spatial resolution of the input. To address this issue, we propose  
Figure 1: Spatial–Temporal Joint Attention Network (STA-NET). Multi-Scale Transpose Attention (MSTA), which exhibits linear complexity, as depicted in Figure 2. The key distinction lies in MSTA applying self-attention across channels, calculating cross-covariance across channels to generate an attention map implicitly encoding global context. As another integral component of MSTA, we introduce depthwise convolution to emphasize 3D local context, performing this operation before computing feature covariance to generate a global attention map. From the Layer Normalization tensor \(\hat{\mathbf{X}}\in\mathbb{R}^{\mathrm{H}\times\mathrm{W}\times\mathrm{C}\times \mathrm{T}}\), our Multi-Scale Transpose Attention (MSTA) initially generates query (\(\mathbf{Q}\), key (\(\mathbf{K}\)), and value (\(\mathbf{V}\)) projections, enriching local contexts. This is achieved by applying a 1 \(\times\) 1 \(\times\) 1 convolution to aggregate spatiotemporal cross-channel context, followed by a 3 \(\times\) 3 \(\times\) 3 depthwise convolution to encode channel-level spatiotemporal context, resulting in \(\mathbf{Q}=\mathbf{W}_{d}^{\mathrm{Q}}\mathbf{W}_{p}^{\mathrm{Q}}\hat{\mathbf{ X}}\), \(\mathbf{K}=\mathbf{W}_{d}^{K}\mathbf{W}_{p}^{K}\hat{\mathbf{X}}\), and \(\mathbf{V}=\mathbf{W}_{d}^{V}\mathbf{W}_{p}^{V}\hat{\mathbf{X}}\). Here, \(\mathbf{W}_{p}^{(\cdot)}\) represents a 1 \(\times\) 1 pointwise convolution, and \(\mathbf{W}_{d}^{(\cdot)}\) represents a 3 \(\times\) 3 \(\times\) 3 depthwise convolution. In summary, the MSTA process is defined as follows:  
\[ \begin{split}&\mathbf{Y}=\mathbf{W}_{p}\mathrm{Attention}( \mathbf{Q},\mathbf{K},\mathbf{V})+\hat{\mathbf{X}},\\ &\mathrm{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})=\mathbf{V} \cdot\mathrm{Soft}\max(\mathbf{K}\cdot\mathbf{Q}/\alpha),\end{split}  \tag{1}\]  
Here, \(\hat{\mathbf{X}}\) and \(\mathbf{Y}\) are the input and output feature maps, respectively. In this context, \(\alpha\) is a learnable scaling parameter used to control the magnitude of the dot product between \(\mathbf{K}\) and \(\mathbf{Q}\) before applying the SoftMax function. Similar to traditional multi-head self-attention [31], we divide the number of channels into \"heads\" and independently learn attention maps in parallel. Multi-Scale Feedforward Network  
A conventional Feedforward Network (FN) [30; 31] performs the same operation at each spatiotemporal position to transform features. This network utilizes two 1 \(\times\) 1 \(\times\) 1 convolutions, where the first convolution layer is employed to expand feature channels (typically expanded by a factor of \(\gamma=4\)), and the second convolution layer reduces the channel count back to the original input dimensions. The relationship between these two operations lies in that the former aims to increase the spatial dimension of features to capture advanced features of spatiotemporal characteristics more effectively.