## 3 Methodology
### STA-Blocks
#### 3.4.1 Temporal Attention Layer  
In a traffic network, there is typically a correlation between traffic conditions at different time steps within a sequence at each location. This correlation is known as temporal correlation, and it can be classified into two categories: short-term and long-term. Specifically, \"short-term\" refers to a smooth variation or, in other words, a tendency to  
Figure 2: Overall hierarchical architecture of spatial-temporal attention networks. It consists of stacked STA-Blocks and input/output layers. Each STA-Blocks module consists of a spatial attention layer and a temporal attention layer, which uses a multi-headed self-attention mechanism. remain stable for a relatively extended period of time. On the other hand, \"long-term\" indicates significant variations in impact. The temporal correlations in this layer are modeled using a multi-headed self-attentive mechanism, as shown in the middle part of Figure 2, utilizing an input time series \(\mathbf{\mathcal{X}}\in\mathbb{R}^{T\times N\times D}\). Here, \(T\) represents the length of the time series, and \(D\) denotes the dimensionality after input embedding. This layer includes the multi-headed self-attention module, the residuals and layer normalization module, and the feed-forward network module (FFN). The multi-headed self-attention module computes temporal correlations using three dynamic high-latitude subspaces. These subspaces consist of the query vector, the keyword vector \(K\), and the value vector \(V\), computed as shown in Equations (4) and (5). \[Q=W_{q}\times\mathbf{\mathcal{X}};K=W_{k}\times\mathbf{\mathcal{X}};V=W_{v}\times\mathbf{ \mathcal{X}} \tag{4}\]  
\[Att(\mathbf{\mathcal{X}})=softmax\bigg{(}\frac{QK^{T}}{\sqrt{d_{k}}}\bigg{)}V \tag{5}\]  
We define the parameter matrices of linear mapping as \(W_{q}\in\mathbb{R}^{N\times T\times D}\), \(W_{k}\in\mathbb{R}^{N\times T\times D}\), \(W_{v}\in\mathbb{R}^{N\times T\times D}\). Additionally, the query vector \(Q\) is represented as \(Q=[q_{1},\dots,q_{N}]\), the key vector as \(K=[k_{1},\dots,k_{N}]\), and the value vector as \(V=[v_{1},\dots,v_{N}]\). The self-attention mechanism is computed using a scaled dot product, where each element in the key vector is divided by \(\sqrt{d_{k}}\). This is then followed by applying a _softmax_ function to obtain the weight coefficients corresponding to each time step. The multi-headed self-attention mechanism (MSA) calculates the correlation between all input elements and generates a weight matrix. This mechanism utilizes multiple attention modules, each with different weights assigned to the same query. The concept is to generate multiple queries by linearly transforming the input query using different weight matrices. Consequently, the attention model is capable of incorporating additional information into the computation of contextual vectors. The multi-headed self-attention can be mathematically expressed as Equation (6). \[MSA(\mathbf{\mathcal{X}})=\frac{1}{ht}||_{i=1}^{ht}[Att(\mathbf{\mathcal{X}})] \tag{6}\]  
Here, \(ht\) represents the number of attention heads. The feature vector is upscaled to a higher dimension and subsequently split into \(ht\) multi-dimensional vectors, which are later averaged. The symbol \(\parallel\) corresponds to the operation of the Concat function. In the module of residuals and layer normalization, the output of the MSA module undergoes a projection or dropout operation, resulting in residual and layer normalization. Layer normalization is defined by Equation (7). The first reason for this normalization is to adapt the model to a data distribution with a variance between 0 and 1, improving the neural network's generalization. The second reason is to apply layer normalization before the FFN, preventing data from saturating and reducing the issue of gradient vanishing.