# Ionospheric TEC Prediction in China Based on the Multiple-Attention LSTM Model
## 2 Data and Proposed Model
### Our Proposed Model
#### 2.5.2 LSTM Based on Multiple-Attention Modules (MA-LSTM) Proposed in This Paper  
When the traditional LSTM model is used to model TEC data, the prediction weights of the data at any position in the historical sequence to the future data are equal, so it is impossible to adaptively and accurately model TEC. To solve this problem, this paper adds three attention modules [29], which can adaptively assign weights to each input sequence, so that the model can selectively focus on the historical sequence and reduce the prediction error. The proposed MA-LSTM TEC prediction model is shown in Figure 7.  
The proposed MA-LSTM model includes four parts: the input layer, the LSTM layer, the multiple-attention layer and the output layer.  
**Input layer:** Each sample includes two parts: feature vector \(x_{n}\) and regression value \(y_{n}\) (i.e., prediction value), which form an input sample in pairs as \([x_{n},y_{n}]\).  
**LSTM Layer:** This layer includes two independent LSTM neuron layers, one of which is provided with an L1 constraint and the other is not. These two layers, respectively, receives a pair of feature and regression value samples of the input layer \([x_{n},y_{n}]\) and process them separately. The processed results are spliced by concatenation function and transmitted to the next layer. The calculation formula of this layer is as follows:  
\[k_{n}=LSTM([x_{n},y_{n}]) \tag{12}\]  
\[m_{n}=LSTM([x_{n},y_{n}]) \tag{13}\]  
\[r_{n}=concat([k_{n},m_{n}]) \tag{14}\]  
where \(k_{n}\) represents the output of the first LSTM layer with an L1 constraint. \(m_{n}\) represents the output of the unconstrained LSTM layer. \(r_{n}\) represents the vector after connecting \(k_{n}\) and \(m_{n}\). All \(r_{n}\) are connected to form the output of this layer, which is \(l_{n}\).  
**Multiple-attention layer:** This layer contains three parallel attention branches, which, respectively, receive the data from the LSTM layer. After receiving the data transmitted from the LSTM layer, each branch first calculates the similarity between each feature in the received data and the regression value through the attention function, and each feature obtains an attention score. The calculation formula of attention function selected in this paper is as follows:  
\[score(h_{n},y_{n})=V^{T}tanh[Wh_{n}+Uy_{n}] \tag{15}\]  
where \(W,V,U\) are the parameters that can be learned in the training process. After obtaining the attention score, then normalize it with the softmax function to obtain the probability distribution of attention. The specific calculation formula is as follows:  
\[a_{1n}=softmax(score(h_{n},y_{n})) \tag{16}\]  
\[a_{2n}=softmax(score(h_{n},y_{n})) \tag{17}\]  
\[a_{3n}=softmax(score(h_{n},y_{n})) \tag{18}\]  
\(a_{1n},a_{2n},a_{3n}\) represents the respective attention distribution values of the three branches. The softmax function is as follows:  
Figure 6: Structure of an LSTM unit.  
(19)  
Then, \(a_{1n},a_{2n},a_{3n}\) are, respectively, multiplied \(h_{n}\) to obtain the output value of each attention branch. The calculation formula is as follows:  
\[t_{1n}=a_{1n}\times h_{n} \tag{20}\]  
\[t_{2n}=a_{2n}\times h_{n} \tag{21}\]  
\[t_{3n}=a_{3n}\times h_{n} \tag{22}\]  
Figure 7: MA-LSTM model structure for TEC prediction.  
Finally, connect the attention values of the three branches as the output of the attention layer.  
**Output layer:** This layer includes a flattened layer and a fully connected layer. It is used to map the previously results into predicted values and then output them.