# Slipping Trend Prediction Based on Improved Informer
## 3 Methodology
### Back-Propagation Training  
The cross-entropy loss function [25] is widely used in classification tasks. This paper uses this function as the loss estimation of the model, as shown in Formula (18):  
\[loss=\frac{1}{N}\sum_{n=1}^{N}\left[y_{true}\cdot\log y_{pred}+(1-y_{true})\cdot \log(1-y_{pred})\right] \tag{18}\]  
SGD (Stochastic Gradient Descent) [26] is widely used in various neural network learning tasks. The gradient term of this method does not add any momentum coefficient, and it is easy to fall into the local optimum, as shown in Formula (19).  
\[\theta_{t+1}=\theta_{t}-\eta\cdot g_{t} \tag{19}\]  
where \(\theta_{t}\) is the weight parameter matrix at time t, \(\eta\) is the learning rate, and \(g_{t}\) is the gradient of the loss function to the weight. Obviously, the convergence speed and the final convergence effect of Adam (Adaptive Momentum Estimation) [27] are better than other optimizers, as shown in Figure 5. Adam adds both first-order momentum and second-order momentum to SGD. First, the first-order momentum \(m_{t}\) is calculated according to Formula (20):  
\[m_{t}=\beta_{1}m_{t-1}+(1-\beta_{1})\cdot g_{t} \tag{20}\]  
The calculation of second-order momentum is the same as that of RMSProp [28] and Adagrad [29], as shown in Formula (21). This article does not introduce many other optimizers.  
\[v_{t,i}=\beta_{2}v_{t-1,i}+(1-\beta_{2})\cdot g_{t,i}^{2} \tag{21}\]  
where \(\beta_{1}\) and \(\beta_{2}\) are exponential decay rates estimated twice. In order to prevent the gradient weight from being small at the beginning of training, two momentums are added to the correction factors \(\beta_{t1}\) and \(\beta_{t2}\) of the exponential weighted average, respectively, so that the sum of the weights of each time step in the training process is 1. The formula is derived as follows (\(m_{0}\) initialized to 0):  
From Formula (20):  
\[m_{1}=\beta_{1}m_{0}+(1-\beta_{1})g_{0}=(1-\beta_{1})g_{0} \tag{22}\]  
Figure 5: Graph of loss convergence during training.  
From Formula (21):  
\[m_{2}=\beta_{1}m_{1}+(1-\beta_{1})g_{1}=\beta_{1}(1-\beta_{1})g_{0}+(1-\beta_{1})g _{1} \tag{23}\]  
The relationship between the first-order momentum \(m_{t}\) and the gradient \(g_{t}\) is derived by the above formula:  
\[\begin{array}{ll}m_{t}&=\beta_{1}^{t-1}(1-\beta_{1})g_{0}+\beta_{1}^{t-2}(1- \beta_{1})g_{1}+\ldots+\beta_{1}^{1}(1-\beta_{1})g_{t-1}+\beta_{1}^{0}g_{t}\\ &=\sum\limits_{i=1}^{t}\ (1-\beta_{1})\cdot\beta_{1}^{t-i}\cdot g_{i}\end{array} \tag{24}\]  
The weight correction factor is deduced as Formula (25):  
\[\begin{array}{ll}\beta_{t1}&=\sum\limits_{i=1}^{t}\ (1-\beta_{1})\cdot\beta_{1}^{t-i} \\ &=(1-\beta_{1})\cdot(\beta_{1}^{t-1}+\beta_{1}^{t-2}+\ldots+\beta_{1}^{0})\\ &=1-\beta_{1}^{t}\end{array} \tag{25}\]  
The calculation process of \(\beta_{t2}\) is the same as that of \(\beta_{t1}\), so it is not repeated. By dividing the two momentums \(m_{t}\) and \(v_{t}\) by the corresponding correction factor, the corrected momentum is obtained:  
\[\hat{m_{t}}=\frac{m_{t}}{\beta_{t1}} \tag{26}\]  
\[\hat{v_{t}}=\frac{v_{t}}{\beta_{t2}} \tag{27}\]  
Finally, the parameters of the Adam optimization algorithm are updated with Formula (28), where \(\varepsilon\) is a minimum number, preventing the denominator from being 0.  
\[\theta_{t+1}=\theta_{t}-\frac{\eta}{\sqrt{\hat{v_{t}}}+\varepsilon}\cdot\hat{ m_{t}} \tag{28}\]