## 3 Transformer Encoder-Based BiLSTM Network Model
### Feature Extraction of the Transformer Encoder  
The structure of the Transformer model is illustrated in Figure 2, consisting of an encoder and a decoder [18; 19]. The encoder comprises \(N=6\) identical layers, each containing a multi-head attention mechanism and a position-wise feedforward neural network sublayer. Residual connections and layer normalization are applied to the output of each sublayer, facilitating the capture of dependencies across all positions in the input sequence. The decoder is similarly composed of \(N=6\) identical layers, with each layer consisting of three sublayers: a masked self-attention layer, an encoder-decoder attention layer, and a position-wise feedforward neural network. Residual connections and layer normalization are also applied after each sublayer. This architecture ensures that the decoder incorporates information from previous outputs while preventing the influence of future data during sequence generation. Figure 1: Flowchart of GWO-optimized VMD parameters. The structure of the Transformer encoder employed in this paper is outlined as follows:  
1 Positional Encoding  
Traditional Recurrent Neural Networks (RNNs) can inherently capture positional information within sequences. However, the Transformer model lacks built-in mechanisms for encoding sequence positions. To address this limitation, positional encodings are introduced to inject positional information into each element of the input sequence. This enables the model to differentiate between various positions within the input data and ensures the retention of the complete feature set of the input information. \[\left\{ \begin{array}{l}PE_{(pos,2i)}=\sin(\frac{pos}{\frac{2i}{ 10000^{2}\frac{2i}{\text{pos}}}})\\ PE_{(pos,2i+1)}=\cos(\frac{2i}{10000^{2}\frac{2i}{\text{pos}}})\end{array} \right. \tag{6}\]  
Here, _pos_ represents the position in the sequence. \(i\) represents the dimension. _d\({}_{model}\)_ represents the embedding space dimension. 2 Self-Attention  
The self-attention mechanism assesses the importance of each position in relation to others by calculating the attention scores between the Query, Key, and Value. It then performs a weighted aggregation of the various components of the input sequence based on these scores [20]. \[Attention(\mathbf{Q},\mathbf{K},\mathbf{V})=softmax(\frac{\mathbf{Q}\mathbf{K}^{T}}{ \sqrt{d_{K}}})*\mathbf{V} \tag{7}\]  
Here, \(\mathbf{Q}\), \(\mathbf{K}\), and \(\mathbf{V}\) represent the Query, Key, and Value matrices, respectively. _d\({}_{K}\)_ denotes the dimension of the Key. The sequence information, after positional encoding, is divided into \(\mathbf{Q}\), \(\mathbf{K}\), and \(\mathbf{V}\), which are used as the input to the encoder. To capture richer sequence features, the Transformer model utilizes multiple self-attention mechanisms operating in parallel. Each attention head learns a distinct set of weights, enabling the model to extract information from different subspaces of the input. The outputs of the \(h\) attention heads are concatenated, as shown in the Equation (8). Figure 2: The structure of the Transformer model. However, since this concatenated result does not provide an integrated representation, a linear transformation is applied to fuse the outputs effectively. \[\text{MultiHead}(\mathbf{Q},\mathbf{K},\mathbf{V})=\text{Concat}(\text{head}_{1},\text{head }_{2},...,\text{head}_{h})\mathbf{W}^{O} \tag{8}\]  
\[\text{head}_{i}=Attention(\mathbf{Q}_{i},\mathbf{K}_{i},\mathbf{V}_{i}) \tag{9}\]  
\[\mathbf{Q}_{i},\mathbf{K}_{i},\mathbf{V}_{i}=\mathbf{Q}\mathbf{W}_{i}^{O},\mathbf{K}\mathbf{W}_{i}^{K},\bm {W}\mathbf{W}_{i}^{V} \tag{10}\]  
Here, \(\text{head}_{i}\) represents the output of attention head\({}_{i}\). \(\mathbf{Q}_{i}\), \(\mathbf{K}_{i}\), and \(\mathbf{V}_{i}\) are the inputs to attention head\({}_{i}\).