## 2 Background Description
### Attention Layer  
Inspired by the successful application of temporary pattern attention in the literature [26], this paper proposes an improved attention network structure. Figure 4 shows the calculation process of the model. The model extracts temporal information \(H_{t-1}^{C}\) in \(H_{t-1}\) by the convolution operations. Where \(H_{t-1}=\{h_{1},h_{2}\cdot\cdot\cdot,h_{t-1}\}\), \(h_{t}\) represents the LSTM layer output at time \(t\). By focusing on the output \(h_{t}\) at the last moment, the model extracts the relevant time sequence information in \(H_{t-1}^{C}\), thereby improving the learning ability of the model. In convolution operation, the Gaussian convolution kernel is used to extract the inertial delay characteristics of data. The convolution kernels corresponding to each feature are independent of each other. Scaled dot-product attention is used to calculate the similarity of time series. The Gaussian convolution kernel formula used in the model is as follows:  
\[ker=K_{ker}\times exp\Big{(}-(a-b)^{2}/c^{2}\Big{)} \tag{13}\]  
\[a_{i}=i \tag{14}\]  
\[K_{ker}=\begin{cases}1,&a-b\geq 0\\ 0,&a-b<0\end{cases} \tag{15}\]  
where \(a,b,c\in\mathbb{R}^{w}\), \(w\) is the length of the convolution kernel; \(a_{i}\) representing the \(i\)-th element in \(a\); \(b\), and \(c\) are trainable parameters. The shape of the Gaussian convolution kernel depends on \(b\) and \(c\). Different Gaussian convolution kernels can extract different inertia anddelay characteristics in the data. \(K_{ker}\) is a step function used to cut the Gaussian convolution kernel. It should be noted that in model training, \(K_{ker}\) does not perform gradient calculation.  
The convolution operation can extract the data delay inertia characteristics through the Gaussian convolution kernel. To better illustrate, we randomly generate a Gaussian convolution kernel for elaboration. We assume that the data has an input of 1 at the moment 0 and 0 at other moments. We pad the input at the 0 moment with \(w-1\) zeros in front of it. The result of the convolution operation is the influence of the input at the moment 0 on the input at subsequent moments. The result of the convolution operation is shown in Figure 5. The data first pass through a purely delayed segment and then gradually makes an impact, reaching a peak and then stopping. This inertial delay response is close to the step response curve of each variable and the target variable in Reference [34]. The trainable parameters in the convolution kernel determine the shape of the convolution kernel so that the model can extract the inertial delay characteristics of different characteristics.  
The calculation process of the model is as follows: The model first performs the convolution operation on the data, and \(H_{t-1}\) forms \(H_{t-1}^{C}\) after the convolution operation.  
\[H_{t-1}^{C}=CNN(H_{t-1}) \tag{16}\]  
The similarity between \(h_{t}\) and the feature at each time step in \(H_{t-1}^{C}\) is calculated. The attention calculation method is scaled dot-product attention, with \(H_{t-1}^{C}\) as the query vector and value vector, and \(h_{t}\) as the key vector. \(v_{t-1}\) is obtained by weighted summation of \(H_{t-1}^{C}\) according to the similarity score. \(v_{t-1}\) and \(h_{t}\) are weighted and summed to obtain the final output \(O\).  
\[\alpha=softmax\Big{(}f\Big{(}H_{t-1}^{C},h_{t}\Big{)}\Big{)} \tag{17}\]  
Figure 4: Attention layer structure diagram. Where \(n=t-w\), \(t\) is the length of time step of input data; \(w\) is the size of the convolution kernel; and \(m\) is the characteristic number of input data.  
Figure 5: Data impact diagram.  
\[f\Big{(}H_{t-1}^{C},h_{t}\Big{)}=H_{t-1}^{C}h_{t}/\sqrt{m} \tag{18}\]  
\[v_{t-1}=\alpha H_{t-1}^{C} \tag{19}\]  
\[O=W_{h}\times h_{t}+W_{v}\times v_{t-1} \tag{20}\]  
where \(H_{t-1}\in\mathbb{R}^{(t-1)\times m}\), \(H_{t-1}^{C}\in\mathbb{R}^{n\times m}\), \(W_{h}\in\mathbb{R}^{m\times m}\), \(W_{v}\in\mathbb{R}^{m\times m}\), \(v_{t-1}\in\mathbb{R}^{m}\), \(\alpha\in\mathbb{R}^{n}\).