# STFEformer: Spatial-Temporal Fusion Embedding Transformer for Traffic Flow Prediction
## 1 Introduction
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Study** & **Spatial Component** & **Temporal Component** \\ \hline
[9] & GNN & GRU \\
[10] & GNN & GRU \\
[11] & GNN & GRU \\
[12] & GNN & GRU \\
[13] & GNN & GRU \\
[14] & GNN & GRU \\
[15] & GNN & CNN \\
[16] & GNN & CNN \\
[17] & GNN & CNN \\
[18] & GNN & TCN \\
[19] & GNN & CNN \\
[20] & GNN & TCN \\
[21] & GNN & GRU \\
[22] & GNN & TCN \\
[23] & GNN & CNN \\
[24] & GNN & TCN \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Summary of GNN studies for traffic flow prediction. Moreover, several studies have utilized embedding structures to discover both temporal and spatial features with the goal of efficiently capturing spatial-temporal dependencies. Adaptive Graph Convolutional Recurrent Networks (AGCRNs) [9] are known to constitute a pioneering approach that designs embeddings for each node, thus creating an adaptive graph as opposed to a predefined one, establishing spatial connections and capturing spatial correlations. Both MTGNNs [15] and GMANs [25] utilize spatial embeddings to explore spatial dependencies, and GMANs additionally construct temporal embeddings through one-hot encoding, integrating them with an encoder-decoder architecture. The Adaptive Graph Spatial-Temporal Transformer Network (ASTTN) [26] also obtains temporal embeddings using one-hot encoding and obtains Laplacian positional encoding through the eigenvalue decomposition of the input graph. In addition, other models [27; 28; 29; 30; 31; 32; 33; 34; 35; 36; 37] also demonstrate commendable performance in traffic flow prediction. Details are shown in Table 2. However, there is a lack of comprehensiveness in feature representation when constructing embedding structures. For example, the extraction of native information from traffic data is frequently neglected. The sole utilization of one-hot encoding for representing the day of the week and the time of day of each time step for the purpose of obtaining temporary embeddings falls short of effectively capturing the short-term correlations in traffic data. To effectively explore dynamic spatial-temporal patterns and comprehensively characterize traffic data, this paper introduces a spatial-temporal fusion embedding transformer model, namely, STFEformer. As a pivotal technical contribution, we have developed a novel embedding layer aimed at processing traffic data from multiple perspectives to explore native and spatial-temporal features. Furthermore, we propose a spatial self-attention module based on the graph-masking method, designed to highlight interactions between similar nodes and capture dynamic, long-range spatial dependencies. We also employ a temporal self-attention module to discover the dynamic temporal dependencies in traffic data. In conclusion, the main contributions of this research can be summarized as follows:  
1. We propose STFEformer, a novel transformer-based architecture designed for precise traffic flow prediction, integrating a fusion embedding layer with spatial-temporal self-attention layers. The fusion embedding layer effectively captures native, temporal (both short-term and periodic), and spatial characteristics of traffic data. Subsequently, we utilize a spatial self-attention module based on graph masking to enhance focus on relevant nodes and capture dynamic, long-range spatial correlations. This is complemented by a temporal self-attention module that facilitates the parallel processing of spatial-temporal dependencies, addressing the comprehensive, dynamic, and long-range challenges of traffic data. 2. Comprehensive experiments were carried out on three real-world public traffic datasets: PEMS04, PEMS07, and PEMS08. The results of these experiments clearly indicate that our model significantly surpasses all baseline models in terms of performance. In addition, we conducted ablation experiments to systematically evaluate the impact of the different components of our model on its overall efficacy.