# Medium- and Long-Term Load Forecasting for Power Plants Based on Causal Inference and Informer
## 3 Methods
### Improved Informer Model for Non-Stationary Data  
As one of the most popular artificial intelligence models, Transformer [24] has been widely used in numerous fields. However, the model is not directly applicable to the field of time series prediction due to high memory consumption, high time complexity, and the structural constraints of encoder-decoder. In order to overcome the described problems, the Informer model employs an attention mechanism that reduces the time complexity (denoted as 'O') and memory consumption to O(\(L_{k}lnL_{k}\)). The dominant attention is highlighted in self-attention extraction by halving the cascading layer inputs, thereby efficiently handling the ultra-long input sequences. The attention mechanism used in the Informer model is referred to as ProbSparse self-attention. Following the formulation in a previous study [25], the original \(i\)-th query's attention (denoted as '\(\mathcal{A}\)') is defined as a kernel smoother in a probability form:  
\[\mathcal{A}(q_{i},\mathbf{K},\mathbf{V})=\sum_{j}\frac{k(q_{j};\boldsymbol{k} _{j})}{\sum_{j}k(q_{j};\boldsymbol{k}_{j})}\boldsymbol{v}_{j}=\mathbb{E}_{ \mathcal{P}(\boldsymbol{k}_{j}|\boldsymbol{q}_{j})}[\boldsymbol{v}_{j}] \tag{7}\]  
Figure 3: Chart of MI values. where \(p(\mathbf{k}_{j}|\mathbf{q}_{i})=\frac{k\left(\mathbf{q}_{i}\mathbf{k}_{j}\right)}{\sum_{k}k(\mathbf{q} _{i},\mathbf{k}_{j})}\) and \(k(\mathbf{q}_{i},\mathbf{k}_{j})\) are used to select the asymmetric exponential kernel \(\frac{\mathbf{q}_{i}\mathbf{k}_{j}^{T}}{\sqrt{d}}\). Using Kullback-Leibler (denoted as '\(KL\)') divergence to measure the 'likeness' of attention probability \(p(\mathbf{k}_{j}|\mathbf{q}_{i})\) and uniform distribution \(q(\mathbf{k}_{j}|\mathbf{q}_{i})=\frac{1}{L_{k}}\), the following can be obtained:  
\[KL(q||p)=\ln\sum_{l=1}^{L_{k}}\exp\left(\frac{\mathbf{q}_{i}\mathbf{k}_{l}^{T}}{\sqrt{ d}}\right)-\frac{1}{L_{k}}\sum_{j=1}^{L_{k}}\frac{\mathbf{q}_{i}\mathbf{k}_{j}^{T}}{ \sqrt{d}}-\ln L_{k} \tag{8}\]  
Dropping the constant in (8), the \(i\)-th query's sparsity measurement (denoted as '\(M\)') is defined as:  
\[M(\mathbf{q}_{i},\mathbf{k})=\ln\sum_{l=1}^{L_{k}}\exp\left(\frac{\mathbf{q}_{i}\mathbf{k}_{l} ^{T}}{\sqrt{d}}\right)-\frac{1}{L_{k}}\sum_{j=1}^{L_{k}}\frac{\mathbf{q}_{i}\mathbf{k} _{j}^{T}}{\sqrt{d}} \tag{9}\]  
The distribution of the attention probability of the dominant query will be different from the uniform distribution. The number of the dominant query is \(u=c*lnL_{k}\). By allowing each key to pay attention only to the top u queries, the ProbSparse Self-attention is defined as follows:  
\[\mathcal{A}(\mathbf{Q},\mathbf{K},\mathbf{V})=Softmax\left(\frac{\overline{\mathbf{Q}}\mathbf{K}^ {T}}{\sqrt{d}}\right)\mathbf{V} \tag{10}\]  
Finally, the computational complexity of attention is reduced in each layer to O(\(L_{k}lnL_{Q}\)) by sampling the \(L_{k}lnL_{k}\) dot product pairs. Details of the specific derivation of the formula are provided in prior research [26]. The structure of Informer is shown in Figure 4:  
Figure 4: The Structure of Informer. As shown in Figure 4, after input embedding and position encoding, the data were sent to the encoder for valid information extraction and later to the decoder to complete the prediction. Considering the difficulty of dynamic decoding output for fast prediction, Informer uses one forward procedure to calculate the prediction results. For this purpose, a certain length of sequence from the input is needed as 'Start token' [27]. The original Informer model arrives at long time series prediction but does not consider the effect of non-stationary data in real scenarios. In fact, load is affected by various factors and shows significant variations. For instance, thermal power plants need to be shut down first when the electricity demand decreases.