# STFEformer: Spatial-Temporal Fusion Embedding Transformer for Traffic Flow Prediction
## 2 Materials and Methods
### Spatial-Temporal Attention Layer
#### 2.4.1 Spatial Self-Attention  
In the spatial dimension, the traffic conditions at various locations interact with each other in a highly dynamic manner. To capture this dynamic spatial dependency in traffic data, we designed a spatial self-attention module.  
We first calculate the query, key, and value (QKV) matrices at time t via the self-attention operation as follows:  
\[Q_{t}^{S}=X_{t:}W_{Q}^{S},K_{t}^{S}=X_{t:}W_{K}^{S},V_{t}^{S}=X_{t:}W_{V}^{S} \tag{9}\]  
where \(W_{Q}^{S}\), \(W_{K}^{S}\), and \(W_{V}^{S}\in\mathbb{R}^{d\times d^{\prime}}\) denote learnable parameter matrices and \(d^{\prime}\) represents the dimension of the query, key and value (QKV) matrices. Then, we apply the spatial self-attention operation to explore the interactions among all nodes and obtain spatial correlations (attention scores) between each pair of nodes at time t as follows:  
\[A_{t}^{S}=\frac{Q_{t}^{S}\times\left(K_{t}^{S}\right)^{T}}{\sqrt{d^{\prime}}} \tag{10}\]  
It is evident that the spatial correlations, \(A_{t}^{S}\in\mathbb{R}^{N\times N}\), between nodes vary across different time slices, and this operation takes into account the connections between all nodes. Therefore, the spatial self-attention module is able to capture long-range and dynamic spatial dependencies. Finally, by normalizing the attention scores and multiplying them with the value matrix, we obtain the output of the spatial self-attention (SSA) module given as follows:  
\[SSA\left(Q_{t}^{S},K_{t}^{S},V_{t}^{S}\right)=softmax(A_{t}^{S})V_{t}^{S} \tag{11}\]  
As mentioned above, each node is involved in interactions with every other node that are equivalent considering the spatial structure in the form of a fully connected graph. However, only the interaction between a few nodes sharing similar functions is crucial. By identifying and focusing attention more on these similar nodes, the model becomes more efficient and accurate. Therefore, we transform the adaptive similarity matrix obtained from the fusion embedding layer into a graph mask matrix, integrating it with the SSA module. Specifically, we choose the top \(K\) nodes that exhibit the greatest similarity for each individual node based on the adaptive similarity matrix. By establishing the weight assigned to the edges connecting the current node with its K similar nodes as 1 and that assigned to the others as 0, we construct a binary mask matrix. Furthermore, we evolve the original SSA into MaskSSA, which can be denoted as follows:  
\[MaskSSA(Q_{t}^{S},K_{t}^{S},V_{t}^{S})=softmax(A_{t}^{S}\odot M_{mask})V_{t}^{S} \tag{12}\]  
where \(\odot\) denotes the Hadamard product. The spatial self-attention module can focus more on the dynamic interactions between similar nodes in this way.  
Finally, we expand MaskSSA into a multi-head spatial self-attention mechanism to gain a more comprehensive understanding of the dynamic and long-range spatial dependencies, thereby enhancing the model's generalization capabilities. The result of the spatial self-attention module can be obtained as follows:  
\[Z_{MaskSSA}=MaskSSA_{1}\parallel MaskSSA_{2}\parallel\dots\parallel MaskSSA_ {h_{ssa}} \tag{13}\]  
where \(h_{ssa}\) is the number of attention heads of the spatial self-attention module.