# Long-Term Structural State Trend Forecasting Based on an FFT-Informer Model
## 2 Long-Term Structural State Trend Forecasting Based on FFT-Informer Model
### LSTF by the Improved Informer Model
#### 2.3.1 Multi-Head ProbHamSparse Attention  
Considering sparsity in the distribution of self-attention probability and the influence of abnormal data on the attention mechanism, the Multi-head ProbHamSparse self-attention is proposed to reduce computation consumption and make prediction more accurate. The ProbHamSparse self-attention is a development based on canonical self-attention and ProbSparse self-attention. Instead of performing a single attention function, the Multi-head attention is a progress from a single side to a multiple side in the Multi-head ProbHamSparse self-attention, as shown in Figure 4 which is applied in the attention mechanism of the Informer model and allows the model to jointly attend to information from different representation subspaces at different positions. **Canonical self-attention:** The canonical self-attention [21] is defined on receiving the tuple input (query, key, value). It performs the scaled dot-product as Equation (5):  
\[ \begin{split}\varphi(Q,K,V)=\text{Softmax}\bigg{(}\frac{QK^{T}}{ \sqrt{d}}\bigg{)}V\\ (Q\in R^{L_{Q}\times d},K\in R^{L_{K}\times d},V\in R^{L_{V}\times d },\text{Softmax}(t)=\frac{1}{1+e^{-t}})\end{split}  \tag{5}\]  
where \(d\) is the input dimension; \(Q\), \(K\), and \(V\) represent query, key, and value and \(\text{Softmax}(-)\) is a normalized exponential function. Equation (5) computes the dot products of the query with all keys, divides each by \(\sqrt{d}\), and applies the Softmax function to obtain the weights on the values. To further discuss the canonical self-attention mechanism, \(q_{i},k_{i},v_{i}\) can represent the \(i\)-th row in \(Q,K,V\); the \(i\)-th query's attention can be defined as a kernel smoother in a probability form:  
\[\varphi(q_{i},K,V)=\sum_{j}\frac{k(q_{i},k_{i})}{\sum_{l}k(q_{i},k_{l})}v_{i} \tag{6}\]  
where \(p(k_{j}|q_{i})=\frac{k(q_{i},k_{i})}{\sum_{l}k(q_{i},k_{l})}\), and \(k(q_{i},k_{j})\) selects the asymmetric exponential kernel \(exp(\frac{qk^{T}}{\sqrt{d}})\). The relationship between \(k_{i}\) and \(q_{i}\) can be obtained based on computing the probability \(p(k_{j}|q_{i})\),which can be applied to improve the prediction ability of the self-attention mechanism. This potential sparsity of distribution of self-attention is recognized by some studies [36; 37]. It reveals that the performance can effectively reduce the computational burden if the irrelevant \(p(k_{j}|q_{i})\) can be distinguished. **ProbHamSparse self-attention:** ProbHamSparse self-attention is proposed to improve the ProbSparse self-attention by combining a Hampel filter. ProbSparse self-attention is an improvement on the basis of canonical self-attention, which considers the sparsity of time series and the dependence of data points. Main contribution points and negligible points can be distinguished by ProbSparse self-attention. So, Informer can focus on the main contribution of a small number of dot products. The \(i\)-th query's attention on all the keys is defined as a probability \(p(k_{j}|q_{i})\), and the output is its composition with values \(V\). The deviation points, as dominant dot-product pairs, encourage the corresponding query's attention probability distribution away from the uniform distribution. If \(p(k_{j}|q_{i})=1/L_{k}\), \(p(k_{j}|q_{i})\) is close to a uniform distribution, the query is redundant to the attention.