# Multivariate Time Series Data Prediction Based on ATT-LSTM Network
## 3 ATT-LSTM Model
### ATT-LSTM Model  
In this section, we will describe in detail the ATT-LSTM model structure. First, this section will explain the data processing and experimental procedures of the entire ATT-LSTM model. Secondly, this section will give a detailed introduction to the composition of the ATT-LSTM model and the composition of each module. We aim to analyze and predict multivariate time series data. To solve the large scale and high complexity characteristics of current multivariate time series data, we apply the attention mechanism to the LSTM and proposes the ATT-LSTM network model. The ATT-LSTM model has five processes in the experiment, and the specific process is shown in Figure 5.  
As shown in Figure 5, the ATT-LSTM model is divided into the following five steps in data processing and data analysis and prediction:  
1. Data set selection. We select two real multivariate time series data sets for follow-up experiments. We will elaborate on the details of each data set in the experimental section. Figure 4: Attention mechanism. Figure 5: Model flow chart. Data processing. Since we choose an actual multivariate time series data set, the processed data set can be input to the model. The data set processing mainly includes two parts: 1 missing value processing; 2 mutation information processing. 3 Attention mechanism processing. Multivariate time series data has several or even dozens of related sequences, and changes in each sequence will affect the target sequence. According to historical lows, the attention mechanism calculates the attention value of each sequence, which is the distribution weight of each sequence. Finally, analyze and predict based on the attention value of each sequence. 4 LSTM network prediction. LSTM analyzes and predicts data based on the input data and the attention value processed by the attention mechanism. 5 Model output. According to the data set's attributes, the analysis and prediction results of the ATT-LSTM model are output intuitively. The model is compared with other models according to the two evaluation indicators of RMSE and MAE. As shown in Figure 6, the ATT-LSTM model is mainly composed of the input, ATT-LSTM module, and output. Among them, the input data are multivariate time series data. The red box represents the ATT-LSTM module, which is also the main structure of the ATT-LSTM model. The processed time series data are first input to the ATT module for attention value calculation and the state of the LSTM module at the last moment is also input to the ATT module for attention calculation. The ATT module calculates the attention value and inputs the data to the LSTM module. The LSTM network analyzes and predicts based on the data processed and calculated by the ATT module and outputs the prediction result. The blue box on the right represents the detailed structure of the ATT module. \(K_{1}\), \(K_{2}\), \(K_{3}\)... \(K_{n}\) represent non-target sequences of multivariate time series data. The target on the right represents the target sequence. In the first stage, the function is used to calculate the correlation between the non-target sequence \(K\) and the target list sequence \(T\):  
\[Similarity(Target,K_{n})=Target\cdot K_{n}. \tag{11}\]  
Among them, the commonly used functions are dot product, Cosine of two vectors and neural network, and Formula (11) uses dot product function for calculation. Figure 6: ATT-LSTM model structure diagram. In the second phase, the result of stage one is converted through the Softmax function. The converted weight value is between 0-1, and the sum is 1. Doing so can make the sequences related to the Target sequence more prominent. The weight value of each element is calculated as follows:  
\[w_{i}=Soft\text{max}(Sim_{i})=\frac{e^{Sim_{i}}}{\sum\limits_{j=1}^{L_{x}}e^{Sim _{i}}}. \tag{12}\]  
where \(w_{i}\) is the attention value calculated by the ATT module. After processing by the ATT module, the non-target sequences are assigned their respective Attention values and then input into the LSTM module for analysis and prediction.