# Dynamic Spatial-Temporal Self-Attention Network for Traffic Flow Prediction
## 4 Methodology
### Spatial-Temporal Self-Attention Layer
#### 4.2.3 Temporal Self-Attention Module  
As shown in Figure 5, a temporal self-attention module is employed to capture the dynamic temporal correlations in the traffic data. The model first takes the output of the spatial self-attention module as input and integrates it with the historical temporal position encoding.  
\[\begin{split}\mathbf{\mathcal{X}}_{\textit{input}}=\oplus\Big{(}\mathbf{ \hat{Z}}_{\textit{spa}},\mathbf{\hat{Z}}_{\textit{sem}},\mathbf{\mathcal{X}}_{ \textit{position}}\Big{)}\mathbf{W}^{\textit{in}}\end{split} \tag{12}\]Here, \(\mathbf{W^{in}}\in R^{d\times d}\) is a learnable parameter matrix, and \(\mathbf{\mathcal{X}_{position}}\) represents the positional encoding embedding vector of the current time step in the historical time steps. The positional encoding in this case adopts the same encoding method as the Transformer model [53].  
Assuming the embedding matrix of an arbitrary node \(v\in V\) in the traffic network is denoted as \(\overline{\mathbf{\mathcal{X}}}_{v}\in R^{T\times d}\), the calculation of the temporal self-attention is as follows:  
\[Atten_{tem}(\mathbf{Q}_{t},\mathbf{K}_{t},\mathbf{V}_{t})=\varphi(\mathcal{S}(\mathbf{Q}_{t}, \mathbf{K}_{t}))\cdot\mathbf{V}_{t} \tag{13}\]  
where \(\mathbf{Q}_{t}=\overline{\mathbf{\mathcal{X}}}_{v}\mathbf{W}_{t\_Q^{\prime}}\mathbf{K}_{t}= \overline{\mathbf{\mathcal{X}}}_{v}\mathbf{W}_{t\_K}\), \(\mathbf{V}_{t}=\overline{\mathbf{\mathcal{X}}}_{v}\mathbf{W}_{t\_V}\) denote the Query, Key, and Value matrices, respectively, within the temporal self-attention mechanism. \(\mathcal{S}(\mathbf{Q}_{t},\mathbf{K}_{t})=\frac{\mathbf{Q}_{t}(\mathbf{K}_{t})^{\top}}{ \sqrt{d}}\) denotes the attention score. And \(\mathbf{W}_{t\_Q^{\prime}}\mathbf{W}_{t\_K},\mathbf{W}_{t\_V}\in R^{d\times d^{\prime\prime}}\) are parameter matrices that can be learned, where \(d^{\prime\prime}\) represents the dimension of the Query, Key, and Value matrices within the temporal self-attention module. \(\varphi(\cdot)\) is the \(softmax\) activation function.  
Similarly to the computation process of spatial self-attention, this method can further obtain the final output of the temporal self-attention module:  
\[\mathbf{\mathcal{Z}_{tem}}=\oplus\Big{(}\mathbf{\mathcal{Z}}_{tem}^{h_{1}},\mathbf{ \mathcal{Z}}_{tem}^{h_{2}},\ldots,\mathbf{\mathcal{Z}}_{tem}^{h_{\tau}}\Big{)}\mathbf{ W^{tem}} \tag{14}\]  
where \(h_{\tau}\in[1,h_{tem}]\) represents the number of heads in the temporal self-attention module, and \(h_{tem}\) is the upper bound of the number of heads for the temporal self-attention module. \(\mathbf{W^{tem}}\in R^{h_{tem}d^{\prime\prime}\times d}\) is a learnable parameter matrix.  
In summary, the final output of the spatial-temporal attention layer is calculated as follows:  
\[\mathbf{\mathcal{X}_{output}}=\mathbf{\mathcal{X}_{emb}}+Norm\Big{(}\oplus\Big{(}\bm {\mathcal{Z}}_{tem}^{1},\mathbf{\mathcal{Z}}_{tem}^{2},\ldots,\mathbf{\mathcal{Z}}_{ tem}^{l}\Big{)}\mathbf{W^{out}}\Big{)} \tag{15}\]  
where \(Norm(\cdot)\) is the layer normalisation operation, \(\oplus(\cdot)\) represents the concatenation operation, and \(\mathbf{W^{out}}\in R^{T\times N\times d}\) is the learnable parameter matrix.