## 3 Data and Methods
### Predictive Models for Deep Learning
#### 3.2.2. Informer  
At this stage, Transformer models are constantly applied in various fields of deep learning, and they all show very good performance. In the field of temporal prediction, Transformer shows its profound potential in capturing long-range dependencies [14], which shows a new possibility to solve the above-mentioned problems faced by LSTF. After this, a large number of experimental studies have shown that there is extremely good compatibility between the intrinsic properties of the Transformer and the LSTF, but the Transformer model alone suffers from three major problems [30]: high quadratic computational complexity of self-attention; memory bottleneck of stacked layers under long sequence inputs and slow inference speed when predicting long sequence outputs. Therefore, the original Transformer model is not suitable for direct use in solving problems in LSTF. Therefore, in order to effectively utilize Transformer's grasp of long sequence features and apply it to LSTF, [PERSON]'s team [14] modified the structure of the original Transformer and proposed an improved long sequence time-series prediction model, Informer. They verified the potential value of the model during testing, Transformer's ability to capture individual long-term correlations between the output and input of long sequence time sequences. In addition, in the structure of the model Informer proposed improvements for the three major defects faced by Transformer in solving the LSTF problem, respectively, as follows Figure 4 shows the general architecture of Informer proposed by [PERSON]'s team [14]. The first is the quadratic computational property of self-attention. The dot product operation of Mechanism makes the time complexity and space complexity of each layer \(\text{O}\big{(}L^{2}\big{)}\), which makes the computational processing time of the model rise significantly with the increase of the length of the prediction sequence, and is not suitable for LSTF. In contrast, subsequent work has shown that the distribution of Self-Attention Probability is sparse; in fact, only a few dot product pairs contribute most of the attention, while the rest  
Figure 4: The overall architecture of Informer. Figure 3: (**a**) Comparison of the limit sequence lengths that can be predicted by LSTF and SSTF. (**b**) Prediction length and accuracy of prediction results. is negligible and obeys a long-tailed distribution. Based on such results, Informer proposed ProbSparse self-attention based on the original one, which reduces the spatio-temporal complexity from O(\(L^{2}\)) to O(LlogL)  
\[\mathcal{A}(\text{Q,K,V})=\text{softmax}\left(\frac{\overline{\text{Q}}\text{K }^{T}}{\sqrt{d}}\right)\text{V} \tag{2}\]  
Among them, \(Q\) is the query vector, \(K\) is the key vector, \(V\) is the value vector, and \(d\) is the number of columns of the Q, K matrix which is the dimension of the vector. ProbSparse self-Attention enables the model to have higher arithmetic power when solving the LSTF problem in dealing with a long sequence of inputs, and at the same time its sampling result is the same for each head when sampling the key randomly for each query, that is, the sampled key is the same. But since each layer of self-attention first does a linear transformation of Q, K, and V, this makes the query and key vectors corresponding to different heads at the same position in the different sequence, i.e., each head takes a different optimization strategy. Each head is stitched together and the final output matrix is obtained by fusion as follows:  
\[\text{MHSA}\ =\text{Concat}\left(\text{ head }^{1}\ldots\text{ head }^{h}\right) \tag{3}\]  
Moreover, Informer proposes the Self-Attention Distilling operation permission to control the attention scores in J-Stacking Layers for the memory bottleneck of the Transformer, which reduces the memory usage of J stacking layers.