## 3 Methods
### Encoder
#### 3.2.2 Transformer Based on a Sparse Attention Mechanism  
The Transformer structure utilizes global attention, which has a computational complexity of \(O\left(T^{2}\cdot d_{model}\right)\), where \(T\) is the sequence length. When handing long sequences, the complexity of calculating attention scores and weights becomes prohibitively high, resulting in excessive use of computational resources and memory [32]. Furthermore, the global attention mechanism allows each position to attend to all other positions in the sequence, which can lead to overly dispersed attention. This makes it challenging for the model to focus on the most relevant information, as positions in the sequence are often more closely related to their local neighbors rather than distant positions. Excessive attention may also cause interference from irrelevant information. The sparse attention mechanism reduces the computational complexity to \(O(T\cdot k\cdot d_{model})\), where \(k\) is the sparse range, by restricting each position's attention to a specific range of neighboring position. This significantly reduces computational and memory requirements, allowing the model to process long sequences more efficiently. By limiting the attention range, the sparse attention mechanism enhances the model's ability to capture local dependencies, reduces interference from irrelevant information, and improves overall performance. Therefore, the network model proposed in this paper employs sparse attention instead of global attention, with the computational process as follows:  
The input sequence X has a dimension of \((T,d_{model})\), where \(T\) is the sequence length and \(d_{model}\) is the feature vector dimension. The sparse attention mechanism consists of the following steps: first, the query, key, and value are computed via the weight matrices \(W^{Q}\), \(W^{K}\), and \(W^{V}\). Figure 4: One of the layers of the encoder network. To enable the model to recognize the relative or absolute positions of elements in the sequence, positional information is embedded into the input features. The positional encoding is computed using the following equation:  
\[ \begin{cases}Q=X\cdot W^{Q}\\ K=X\cdot W^{K}\\ V=X\cdot W^{V}\end{cases}  \tag{4}\]  
Attention scores are computed by taking the dot product of the query \(Q\) and the key \(K\) and then scaling the result, where \(S\) represents the attention score:  
\[S=\frac{Q\cdot K^{T}}{\sqrt{d_{model}}} \tag{5}\]  
The sparse mask \(M\) is applied to the attention scores, restricting each location \(i\) to attend only to locations within the sparse range, and \(f\) represents the sparse range factor:  
\[M_{ij}= \begin{cases}0,|i-j|\leq s_{f}\\ -\infty,otherwise\end{cases}  \tag{6}\]  
When applying sparse masks to attention scores, \(MS\) represents the attention score with a sparse mask:  
\[MS=S+M \tag{7}\]  
When performing softmax on sparse attention scores, AW represents the attention weight:  
\[AW=softmax(MS) \tag{8}\]  
The output is then obtained by weighted summation of the value matrix \(V\) via the attention weights:  
\[AO=AW\cdot V \tag{9}\]  
The results are then passed to the feed-forward neural network, which performs two linear transformations followed by \(ReLU\) activations. The output of the feed-forward network is then normalized using layer normalization. A residual connection is applied, and finally, dropout is used to prevent overfitting. The detailed process is as follows:  
\[O=dropout\left(\frac{ReLU(W_{2}\cdot ReLU(W_{1}\cdot AO+b_{1})+b_{2})-\mu_{ AO}}{\sqrt{\sigma_{AO}^{2}+\epsilon}}\cdot\gamma+\beta,p\right) \tag{10}\]  
Figure 5 illustrates the distribution differences between sparse attention and global attention in the feature dimension. Global attention is characterized by each position being able to interact with all other positions in the sequence, allowing the model to learn the dependencies between time steps across the entire sequence. The left image shows a relatively uniform attention distribution, with color variations being fairly balanced.