# Dynamic Spatial-Temporal Self-Attention Network for Traffic Flow Prediction
## 4 Methodology
### Spatial-Temporal Self-Attention Layer
#### 4.2.2 Spatial Self-Attention Module  
Traffic data exhibit complex spatial correlations, encompassing both local spatial dependencies, such as the traffic conditions at a single node in the traffic network, which often affect the traffic conditions at adjacent road nodes, as well as global spatial dependencies, for example, when there is traffic congestion near one commercial district, residents may travel to another district, thereby impacting the traffic conditions near that district. In order to adequately model the intricate spatial dependencies in the traffic data, a spatial self-attention module is designed as illustrated in Figure 4. Assuming at time step \(t\) that the output of the feature augmentation module is represented as \(\mathbf{\mathcal{X}}_{Aug\_t}\in R^{N\times d}\), the spatial self-attention module can be defined as follows:  
\[Att_{spa}(\mathbf{Q}_{\mathbf{S}},\mathbf{K}_{\mathbf{S}},\mathbf{V}_{\mathbf{S}})=\varphi(\mathcal{S} (\mathbf{Q}_{\mathbf{S}},\mathbf{K}_{\mathbf{S}}))\cdot\mathbf{V}_{\mathbf{S}} \tag{5}\]  
where \(\mathbf{Q}_{\mathbf{S}}=\mathbf{\mathcal{X}}_{Aug\_t}\mathbf{W}_{\mathbf{S}\_Q}\), \(\mathbf{K}_{\mathbf{S}}=\mathbf{\mathcal{X}}_{Aug\_t}\mathbf{W}_{\mathbf{S}\_t}\), \(\mathbf{V}_{\mathbf{S}}=\mathbf{\mathcal{X}}_{Aug\_t}\mathbf{W}_{\mathbf{S}\_t}\) are the Query, Key, and Value matrices in the spatial self-attention mechanism. \(\mathcal{S}(\mathbf{Q}_{\mathbf{S}},\mathbf{K}_{\mathbf{S}})=\frac{\mathbf{Q}_{\mathbf{S}}(\mathbf{K}_\mathbf{{S}})^{\top}}{\sqrt{d}}\) denotes the attention score. \(\mathbf{W}_{\mathbf{S}\_Q},\mathbf{W}_{\mathbf{S}\_K},\mathbf{W}_{\mathbf{S}\_V}\in R^{d\times d^{\prime}}\) are the learnable parameter matrices and \(d^{\prime}\) is the dimension of the Query, Key, and Value matrices. \(\varphi(\cdot)\) is the \(softmax\) activation function. In self-attention mechanisms, each node's receptive field includes all other nodes in the network. However, this situation does not align with reality. In traffic networks, the influence is often effective only between several adjacent nodes. Therefore, this research introduces a masking matrix \(\mathbf{\mathcal{M}}_{\mathbf{hop}}\), based on the concept of k-hop. For the masking matrix \(\mathbf{\mathcal{M}}_{\mathbf{hop}}\), the weight between two nodes is set to 1 when the number of hops between them is less than a threshold \(\epsilon\), and 0 otherwise. By using the masking matrix \(\mathbf{\mathcal{M}}_{\mathbf{hop}}\), distant nodes with weaker influence are masked, ensuring that the model focuses on the local spatial dependencies of traffic flow. Subsequently, after introducing the masking matrix \(\mathbf{\mathcal{M}}_{\mathbf{hop}}\), the computation of local spatial self-attention is as follows:  
\[\sim \tag{6}\]  
where \(\odot\) denotes the Hadamard product and \(\varphi(\cdot)\) is the \(softmax\) activation function. Figure 4: The spatial self-attention module. To capture the global spatial dependencies in the traffic data, this method introduces a semantic masking matrix \(\mathbf{\mathcal{M}}_{\textit{sem}}\), based on the spatial self-attention module. Inspired by previous studies [10; 13; 64], this research utilises the DDTW algorithm [64] to calculate the historical traffic flow similarity for each node. Then, for each node, the top \(\kappa\) semantic neighbours are selected with the highest similarity and assign a weight of 1 to the connections between the node and its semantic neighbours, and 0 otherwise. This process allows us to construct the semantic masking matrix \(\mathbf{\mathcal{M}}_{\textit{sem}}\). The selection of the DDTW algorithm for computing node similarity in this context is motivated by the fact that the first-order derivative of time series data reflects the trends in data development over time. Compared to the irregular distribution of traffic data, the trends in data development are relatively concentrated, providing more supervisory information for traffic flow prediction [10]. Therefore, the computation of global spatial self-attention is as follows:  
\[