## 2 Methodology: PartsNet
### Encoder-Net
#### 2.2.2 Improved Sparse Multi-Head Attention Mechanism  
The Transformer [37] is a sequence-to-sequence model that utilizes the multi-head attention mechanism. The attention mechanism assigns different weights to the information from each timestep and compares it to the information from the current timestep in order to better capture implicit structures and long-range dependencies in sequence data. The self-attention mechanism can be represented by the following equation:  
\[Attention(Q,K,V)=softmax\bigg{(}\frac{QK^{T}}{\sqrt{d_{k}}}\bigg{)}V \tag{11}\]  
where \(Q\), \(K\), and \(V\) are obtained by linear transformations applied to the input sequence. \(d_{k}\) represents the dimensionality of each query vector. \(QK^{T}\) represents the similarity matrix between the query and key vectors, and it corresponds to the inner product projection of the tensors. Multi-head attention strengthens the ability of the neural network to capture highly nonlinear relationships. It consists of multiple independent self-attention layers, referred to as \"heads.\" Each head has its own set of learnable weight matrices, allowing it to capture different features and semantic information from the input sequence. The splitting of matrices is performed as follows: \(Q_{i}=XW_{i}^{Q},K_{i}=XW_{i}^{K},V_{i}=XW_{i}^{V}\). The multi-head attention mechanism is calculated as follows:  
\[ \begin{split} MultiHead(Q,K,V)=Concat(head_{1},\cdots,head_{H})W ^{O}\\ \text{where}\ head_{i}=Attention\left(Q_{i}W_{i}^{Q},K_{i}W_{i}^{ K},V_{i}W_{i}^{V}\right)\end{split}  \tag{12}\]  
These transformation matrices belong to the vector space as follows: \(W_{i}^{Q}\in\mathbb{R}^{d_{model}\times d_{k}}\), \(W_{i}^{K}\in\mathbb{R}^{d_{model}\times d_{k}}\), \(W_{i}^{V}\in\mathbb{R}^{d_{model}\times d_{o}}\) and \(W^{O}\in\mathbb{R}^{hd_{o}\times d_{model}}\). \(W^{O}\) is a linear transformation matrix so that the dimensions of the input and output are the same. \(H\) represents the number of heads. The time complexity of the multi-head attention mechanism is \(\mathcal{O}\big{(}n^{2}\cdot d_{model}\big{)}\), and \(n\) represents the length of the sequence. Figure 4: Schematic diagram of GRU module. This paper proposes a sparse multi-head attention mechanism based on a sparse mask, aiming to reduce the time complexity of the Transformer encoder while ensuring learning accuracy. A sparse mask matrix, which has the same shape as \(Qk^{T}\), is designed in this study. The dimensions to be preserved in the original projection matrix are determined based on the hyperparameter sparsity, which yields a sparse threshold. Subsequently, information positions in the sparse mask that are less than or equal to the threshold are set to negative infinity and mapped to \(Qk^{T}\), introducing sparsity into the attention scores. \[\textit{Sparse mask}\Big{(}Qk^{T}\Big{)}=\left\{ \begin{array}{ll}-\infty, \textit{if}&\left\|Qk^{T}\right\|\leq\textit{threshold}\\ 0,&\textit{otherwise}\end{array} \right. \tag{13}\]  
The sparse multi-head attention mechanism involves sorting the attention scores, mask processing, and conventional matrix multiplication. The overall complexity is denoted as \(\mathcal{O}\big{(}n^{2}\cdot\log n\big{)}\), where \(n\) represents the length of the sequence. The performance of the proposed mechanism will be further demonstrated in the ablation experiments conducted in Section 4.2. \[\textit{Sparse Attention}(Q,K,V)=softmax\Big{(}\frac{(Qk^{T})_{ Sparse}}{\sqrt{d_{k}}}\Big{)}V \tag{14}\] \[\textit{Sparse MultiHead}(Q,K,V)=\textit{Concat}(\textit{head}_{1 },\cdots,\textit{head}_{H})W^{O}\] \[\textit{where head}_{i}=\textit{Sparse Attention}\Big{(}Q_{i}W^{O}_{i},K_{i}W^{K}_{i},V_{i}W^{V}_{i}\Big{)}\]  
The processed retention data from the GRU layer is first subjected to layer normalization, making sure that each feature dimension has a comparable mean and variance. After obtaining sparse attention weights, a residual connection [38] is used before the subsequent layer normalization for enhancing model robustness.