## 2 Materials and Methods
### Methods
#### 2.2.2 Bidirectional Long Short-Term Memory Model (Bi-LSTM) and Attention  
Recurrent Neural Networks (RNNs) [12] are designed to process sequential data, capturing temporal dependencies by passing hidden states across time steps. Long short-term memory (LSTM) [14] is an enhanced RNN architecture that mitigates the gradient problem in sequence processing through the use of forget, input, and output gates. The forget gate discards unnecessary old information, the input gate records relevant new information, and the output gate determines how the current memory state is reflected in the output. These gates enable LSTM to effectively retain long-term information, making it ideal for sequence prediction problems. The structure of an LSTM is depicted in Figure 2.  
Figure 2: The structure of LSTM network. The LSTM network structure consists of the following four steps: updating the forget gate, input gate, cell state, and output gate. Here are the specific four steps introduced below. \(f_{t}\), \(i_{t}\), and \(o_{t}\) represent the forget gate, input gate, and output gate at time \(t\), respectively. \(W\) and \(b\) represent the corresponding weight coefficients and biases. \(h_{t-1}\) represents the hidden state at time \(t-1\). \(x_{t}\) represents the input at time \(t\), and \(c_{t}\) represents the current time step's cell state. \(\widetilde{c}_{t}\) is the candidate value vector. \(\sigma\) denotes the Sigmoid activation function [14]. Update the Forget Gate:  
\[f_{t}=\sigma(W_{f}\cdot[h_{t-1},x_{t}]+b_{f}). \tag{1}\]  
Update the Input Gate:  
\[i_{t}=\sigma(W_{i}\cdot[h_{t-1},x_{t}]+b_{i}), \tag{2}\]  
\[\widetilde{c_{t}}=\tanh(W_{c}\cdot[h_{t-1},x_{t}]+b_{c}). \tag{3}\]  
Update cell state:  
\[c_{t}=f_{t}\cdot c_{t-1}+i_{t}\cdot\widetilde{c_{t}}. \tag{4}\]  
Update the Output Gate:  
\[o_{t}=\sigma(W_{o}[h_{t-1},h_{t}]+b_{o}), \tag{5}\]  
\[h_{t}=o_{t}\cdot\tanh(c_{t}). \tag{6}\]  
The limitation of LSTM units lies in transferring information only in a single direction, while Bi-LSTM takes into account both forward and backward information, enabling it to capture richer contextual information. The structure of the bidirectional LSTM model is depicted in Figure 3.  
To enhance the generalization capability of the Bi-LSTM, we introduced the self-attention mechanism, which allows the model to dynamically focus on important parts while processing sequences. This combination enables the model to better understand and utilize sequential data, thereby improving its performance and effectiveness. Self-attention mechanism reduces reliance on external information and is better at capturing internal correlations within data or features. Computing attention involves three  
Figure 3: The structure of Bi-LSTM network. stages. In the first stage, the computation of the similarity or correlation between the query and each key (In our implementation of the Attention mechanism, the Key is the output obtained after a Bi-LSTM layer passes through a fully connected layer--the query vector comes from the intermediate representation generated by a convolutional layer) is conducted as follows:  
The self-attention mechanism reduces reliance on external information and excels at capturing internal correlations within data or features. Computing attention involves three stages. In the first stage, the similarity or correlation between the query and each key is calculated. In our implementation, the key is derived from the output of a Bi-LSTM layer followed by a fully connected layer. The query vector originates from the intermediate representation generated by a convolutional layer. The computation is performed as follows:  
Figure 2:0  
where \(s_{t}\) represents the attention score. \(w_{h},b_{h}\) are the weight and bias of the attention mechanism. Respectively, \(h_{t}\) refers to the input of the attention mechanism and also represents the hidden states in both directions of output by the Bi-LSTM.