## 2 Proposed Method
### Improved Dual-Stage Two-Phase Model Based on Attention Mechanism
#### 2.2.2 Models
After \(f^{k}_{t}\) is calculated, the Softmax function is used for normalization:  
\[\alpha^{k}_{t}=\frac{\exp(f^{k}_{t})}{\sum\limits_{j=1}^{n}\exp(f^{j}_{t})}, \tag{16}\]  
where attention weight \(\alpha^{k}_{t}\) is determined by \(h^{f}_{t-1}\), \(s^{f}_{t-1}\), the \(k\)-th feature \(x^{k}\) of the current input, and the \(k\)-th feature \(x^{(best)}_{k}\) of the best matching site, which measures the importance of the \(k\)-th feature at time \(t\). Compared with the input exogenous series \(x_{t}\), for the more important \(k\)-th feature, it will be larger in \(\tilde{x}_{t}\). \(\tilde{x}_{t}\) is the combination of all features at time \(t\), which is defined as follows:  
\[\tilde{x}_{t}=(\alpha^{1}_{t}x^{1}_{t},\alpha^{2}_{t}x^{2}_{t},\ldots,\alpha^ {n}_{t}x^{n}_{t})^{T}. \tag{17}\]  
Then, the hidden state \(h^{f}_{t-1}\) and \(\tilde{x}_{t}\) are input into the LSTM0 layer, and the hidden state \(h^{f}_{t}\) at the current moment is updated. \(\tilde{x}_{t}\) is then input into the second phase attention. (2) Second phase attention  
In the second phase, more stable and concentrated correlations can be obtained through the second learning of spatial correlation. The purpose of this module is to learn the spatial correlation among the exogenous series of the central site collection, its target series and matching site target series. In this study, the specific approach is to combine the target series of the central site collection with the exogenous series of the corresponding time, and the target series of the best matching site is added. The attention weights of the input attention mechanism-II are as follows:  
\[s_{t}^{k}=v_{s}^{T}\tanh(W_{s}[h_{t-1}^{s}:s_{t-1}^{s}]+U_{s}[s^{k}:y^{k}]+M_{s }y_{k}^{(best)}) \tag{18}\]  
Figure 3:0  
where \(v_{s}^{T}\in R^{T},W_{s}\in R^{T\times 2q},U_{s}\in R^{T\times T},M_{s}\in R^{T}\) are parameters to learn; \(h_{t-1}^{s}\in R^{q}\) and \(s_{t-1}^{s}\in R^{q}\) are the hidden state and cell state of the encoder LSTM1 (LSTM layer in the second attention module) unit at the previous time; and \(q\) is the hidden size in the second attention module. The corresponding target variable \(y^{k}\) is concatenated to the \(k\)-th attribute \(\hat{x}^{k}\) to form a new vector \(z^{k}\), that is \(z^{k}=[\hat{x}^{k}:y^{k}]\in R^{(n+1)\times T}\). For each spatial attention module with a target series, the above actions are taken independently. Attention weight \(\beta_{t}^{k}\) measures the importance of \(z^{k}\) at time \(t\), and any attribute value at any time has its corresponding weight:  
Figure 3:1  
Then, \(h_{t-1}^{s}\) and \(\tilde{z}_{t}\) are input into the LSTM1 layer. The purpose is to update the hidden state \(h_{t}^{s}\) at the current moment and \(h_{t}^{s}\) is input into the next stage of temporal attention. (3) Decoder with temporal attention  
The disadvantage of the traditional attention mechanism is that the context vector only relies on the hidden state at the last moment, and cannot pay attention to the important features. However, the decoder with temporal attention can adaptively select the encoder hidden state most relevant to the target series by weighting the encoder hidden state. The encoder with spatial attention outputs the hidden state, and the decoder learns the temporal relationship of the hidden state through the attention mechanism within the window size \(T\). Based on the hidden state \(h_{t-1}^{d}\in R^{p}\) and cell state \(s_{t-1}^{d}\in R^{p}\) of the decoder LSTM2 (LSTM layer in the third attention module) unit at the previous time, the attention weight of each encoder hidden state in the second attention module at time \(t\) can be calculated. The attention weights of the temporal attention mechanism are as follows:  
Figure 3:2  
Figure 3:3  
where \(v_{d},b_{d}\in R^{p},W_{d}\in R^{q\times 2p},U_{d}\in R^{p\times p}\) are parameters to learn, \(p\) is the hidden size in the third attention module and \(h_{t}^{s}\in H^{s}\) is the \(i\)-th encoder hidden state of the second attention module.