# Penetration Overload Prediction Method Based on a Deep Neural Network with Multiple Inputs
## 3 Method
### Model Introduction
#### 3.2.2 The Improved PF-Informer Model  
Informer Model Architecture  
The Informer model is shown in Figure 4.  
Informer uses the Kullback-Leibler divergence measure [29] (commonly known as KL divergence) to measure the attention similarity. The smaller the metric value is, the closer the two tested distributions are. After simplification, the sparsity measurement formula for the \(i\)th query is defined as  
\[M(q_{i},k)=ln{\sum_{j=1}^{L_{k}}e^{(\frac{q_{i}^{kT}}{\sqrt{d}})}}-\frac{1}{L_{k }}{\sum_{j=1}^{L_{k}}\frac{q_{i}k_{j}^{T}}{\sqrt{d}}} \tag{4}\]  
The first half is the Log-Sum-Exp (LSE) of \(q_{i}\) for all keys. The second half is the arithmetic average of \(q_{i}\) for all keys. According to the above evaluation method, the ProbSparse self-attention formula is obtained, where \(\overline{Q}\) is the sparse version of the original \(Q\) matrix. The Informer model uses ProbSparse self-attention to select appropriate q-k pairs. \[A(q_{i},k_{i},v_{i})=Soft{\max}\left(\frac{\overline{Q}K^{T}}{\sqrt{d}}\right)V \tag{5}\]  
To efficiently solve the time series prediction problem, the encoder inputs multivariate long time series data and replaces the multihead ProbSparse self-attention mechanism with a multihead self-attention mechanism to extract multiple important pieces of attention. The decoder module also uses multihead ProbSparse self-attention to improve the robustness of the algorithm. The multihead ProbSparse self-attention greatly reduces the computational complexity, from O (L'2) to O (LlogL), in terms of time complexity and memory usage (L is the time sequence length), making the model capable of processing long time series. The ProbSparse self-attention used in the model optimizes the transformer's self-attention mechanism, calculates the upper sparse evaluation boundary, and optimizes the computational time and spatial complexity of the algorithm. The Informer solves the computational complexity of the transformer through this method. In the encoder part, a one-dimensional convolution operation is used on the time series extracted by the multihead ProbSparse self-attention, and an exponential linear unit (ELU) is used as the activation function. The model uses the maximum pooling operation at the end. Compared with the rectified linear unit (ReLU) activation function, the ELU has a negative value, which makes the average activation value closer to zero. The closer the average activation value is to zero, the closer the gradient is to the natural gradient, which  
Figure 4: The structure of the Informer model. speeds up the learning process of the model. As an unsaturated function, the gradient of the ELU is nonzero for all its negative values, so no gradient disappearance or neuronal death problems are encountered. Neuronal death means that when an abnormal input occurs, a large number of gradients are generated in the back-propagation process, which causes neurons to die and the gradient to disappear. Compared with the ReLU function and its variants, the ELU can shorten the time required by the deep neural network training process and improve its accuracy [30]. The Informer solves the high memory usage of the transformer through this method. \[f(x)=\left\{ \begin{array}{l}x,\text{ if }x\geq 0\\ \alpha(e^{x}-1),\text{ if }x<0\end{array} \right. \tag{6}\]  
The parallel processing ability of the model improves the speed of long sequence prediction and reasoning and can satisfy the demand of predicting the subsequent short-term overload curve of penetration. When verifying the prediction accuracy, the decoder receives a part of the long input, covers the part of the input to be predicted at the tail, fills the value of the corresponding position of its timing to 0, generates a prediction and compares the result to the actual data to calculate the loss. Modification of the Time Input of the PF-Informer Model  
In the time series prediction task, time, as the main parameter input, is still very important.