## 3 Methodology
### Network Architectures
#### 3.2.4 Temporal/Spatial Sparse Structural Attention Network  
In the basic Sparse Attention mechanism, we introduce a Sparse Structural Attention Network (SSANet) that effectively incorporates rich structural patterns of Query-Key correlation into temporal and spatial contextual feature aggregation. As is shown in Figure 4, the SSANet consists of two steps: structural Query-Key attention and contextual value aggregation.  
Figure 4: Temporal/Spatial Sparse Structural Attention. For the input feature map x, in accordance with the classic transformer principle, we utilize linear layers to construct the Query (Q), Key (K), and value (V) matrices. Subsequently, the correlation matrix of Q and K is obtained through matrix multiplication operations. Based on the SQKA module, the multi-head structured features of Q-K are derived. Simultaneously, the value V matrix is processed through a 2D convolutional layer to obtain value features with the same number of heads. Finally, the aggregation feature of each head is obtained through multi-head contextual value aggregation. Thereafter, the multi-head features are summed via an addition operation. Ultimately, a skip connection is added to the original input feature map x to enhance the convergence speed of the model.  
We assume that the inputting of features into SSANet is \(X\in\Re^{S\times N\times N}\), where S represents the number of views in the TSSA module, or S can be replaced with C, which represents the channels of feature maps in the SSSA module. Next, we take the TSSA module as an example, and the calculation flow is the same as that of the TSSA module.  
In the traditional transformer structure, after the correlation calculation of Query-Key, the attention result is directly output, lacking the learning of structural features. To effectively leverage rich structural patterns, an improved Structural Self-Attention Network is proposed. Let \(U\) be the correlation matrix of Query-Key:  
\[U=\frac{QK^{T}}{\sqrt{d_{k}}},\;U\in\Re^{S\times S} \tag{4}\]  
To capture the local temporal (or spatial) structural patterns, the Structural Query-Key Attention (SQKA) module is deployed on top of Query-Key correlation by  
\[U_{d}=\varepsilon\bigg{(}conv\bigg{(}\widehat{U},W_{u}\bigg{)}\bigg{)},\;U_{ d}\in\Re^{D\times S\times S} \tag{5}\]  
where \(\widehat{U}\) is a channel-expanded copy of the correlation matrix \(U\), constructed by adding an extra dimension to enable 2D convolution, \(W_{u}\in\Re^{M\times D}\) represents \(D\) learnable convolutional kernels, and \(\varepsilon\) denotes the softmax function applied to generate attention scores from the convolved features.  
Meanwhile, the value contextual patterns can be calculated by  
\[V_{d}=\varepsilon\bigg{(}conv\bigg{(}\widehat{V},W_{v}\bigg{)}\bigg{)},\;V_{ d}\in\Re^{D\times S\times N_{v}} \tag{6}\]  
where the operation of \(\widehat{V}\) is the same to that of \(\widehat{U}\), and \(W_{v}\in\Re^{M\times D}\) denotes \(D\) convolutional kernels with size \(M\). The \(D\) convolutional kernels are used to expand to multi-head structural attention. Then, the contextual value aggregation matrix \(C\) is deployed by Einstein Summation Convention:  
\[C^{ij}=\sum_{d=1}^{D}\sum_{k=1}^{S}U_{d}^{ik}V_{d}^{kj},C\in\Re^{S\times N_{v}} \tag{7}\]  
where \(N_{v}=N\times N\) in this work, and \(U_{d}\in\Re^{D\times S\times S}\) generates temporal kernels dynamically aggregating the local context of value \(V_{D,S,\cdot}\) in multi-head temporal space.  
Then, the contextual value aggregation matrix \(C\) is reshaped to be \(\widehat{C}\in\Re^{S\times N\times N}\) with a dimensional resetting layer. At last, a residual connection is used to connect the input original features \(X\) and \(\widehat{C}\), which helps address the problems of vanishing gradients and degradation in the self-attention transformer structure.