## 3 Data and Methods
### Predictive Models for Deep Learning
#### 3.2.2. Informer
In order to enhance the robustness of distillation operations, the encoder architecture of informer also builds multiple encoder stacks, gradually reducing one distillation operation layer and half of the input length at a time, and finally stitching all the stack outputs together, using self-attention distillation operations to reduce the redundancy generated in the attention results and refine the main attention, reducing the total memory usage from O(\(L^{2}\)) to O(\((2-\in)\) LlogL), which also leads to a more scalable model when receiving long sequence inputs. At the same time, the dynamic decoding operation of the original Transformer makes dynamic decoding (step-by-step inference) worse than other prediction models, and the speed drop is more obvious in the case of long-term prediction, so Informer proposes the Generative Style Decoder. Using a standard decoder structure consisting of a stack of two identical multi-head attention layer, generative inference can effectively mitigate the speed drop in the long-term prediction problem by feeding the following vectors into the decode:  
\[X_{feed\_de}^{t}=\text{Concat}(X_{token}^{t},X_{0}^{t})\in\mathbb{R}^{(L_{token }+L_{y})\times d_{model}} \tag{4}\]  
where start token \(X_{token}^{t}\) is the known sequence before the target sequence and \(X_{0}^{t}\) is the timestamp sequence of the target sequence. Compared with the original decoding method, generative inference enables the model to obtain all the predicted data in just one forward step by changing the original decoding method, so that the model only needs one forward step to get all of the predicted data and decode all elements at once, which is more efficient than dynamic decoding. In addition, not only local timing information but also hierarchical and agnostic timestamps are required in the input of LSTF problems. It is difficult to adapt directly to the conventional self-attentive mechanism, and the query-key mismatch between the encoder and decoder degrades the prediction performance. The embedding of the input consists of three independent parts. 1. Local timestamp. The contextual information is preserved by a fixed location embedding, the location encoding of the original Transformer:\[PE(pos,2j)=\sin\left(\frac{pos}{\left(2L_{x}\right)^{\frac{2j}{2}}}\right);PE(pos,2j+1 )=\cos\left(\frac{pos}{\left(2L_{x}\right)^{\frac{2j}{2}}}\right)\] (5) Among them\(j\in\left\{1,\ldots,\left\lfloor\frac{d_{\text{model}}}{2}\right\rfloor\right\}\)
2. Global timestamps. The global timestamp is represented by a learnable stamp embedding \(\text{SE}_{\text{pos}}\) with a restricted vocabulary (up to 60, at the finest granularity per minute). That is, the self-focused similarity computation has access to the global context and the computational consumption is tolerable of long inputs. 3. Alignment dimensionality. The scalar context \(x_{i}^{t}\) is mapped to the dmodel dimensional vector \(u_{i}^{t}\) using a one-dimensional convolutional filter (width 3, step 1). Finally, by using the above three steps, the sequence of feed-in models can be obtained  
\[\mathcal{X}_{\text{feed}\;[i]}^{t}=\alpha u_{i}^{t}+\text{PE}_{\left(L_{x} \times(t-1)+i\right)}+\sum_{p}\Bigl{[}\text{SE}_{\left(L_{x}\times(t-1)+i \right)}\Bigr{]}_{p} \tag{6}\]  
In the above equation, \(i\in\left\{1,\ldots,\text{SE}_{\text{pos}}\right\}\), \(\alpha\) is a factor that balances the size between the scalar mapping and the local/global embedding, and if the sequence input is normalized, then \(\alpha=1\).