## 3 Methods
### Attention Mechanism  
The attention mechanism, also called soft attention, was primarily designed to improve machine translation in [33]. Importantly, it is designed to mimic the human brain by targeting the most relevant information in a sentence or some specific regions in images where the most important information is given rather than memorizing all the input data. In recent years, attention mechanisms are becoming a necessary component of neural network construction [34]. It has been widely exploited in image processing [35] and neural machine translation [33]. Essentially, the attention mechanism aims to identify the importance of every feature and attribute weight coefficients to highlight the important and unimportant features. To this end, in the training phase, the main purpose is to focus on particular features by a weighted sum procedure described by an attention vector. Specifically, the attention vector at time \(t\), \(\mathbf{s}\), is calculated as,  
\[\mathbf{s}_{t}=\sum_{t}\mathbf{f}_{t}\mathbf{h}_{t}, \tag{1}\]Here, \(\mathbf{h_{t}}\) denotes the hidden states from the model (e.g., a recurrent network) feeding the attention model. \(\alpha_{t}\) refers to the normalized attention model weights calculated by:  
\[\mathbf{ff_{t}}=softmax(\mathbf{e_{t}}), \tag{2}\]  
where \(e_{t}\) denotes the attention model weights, also known as alignment score; it is usually calculated using a feed-forward neural network [33], conditioned on the past hidden state \(\mathbf{h}_{t-1}\) (Figure 1):  
\[\mathbf{e_{t}}=\sigma(\mathbf{w_{a}h_{t-1}+b_{a}}), \tag{3}\]  
\(\mathbf{W_{a}}\) and \(\mathbf{b_{a}}\) represent, respectively, weight matrix and bias vector of the attention model calculated in the training stage. Essentially, the attention vector \(\mathbf{s}\) of the attention model is a dynamic representation of the pertinent portion of the data at time \(t\).  
This mechanism uses a weighted sum for highlighting the important and unimportant data based on the normalized attention model weights that could be explained as a probability (Figure 1). Of course, the idea behind the attention mechanism is inspired by humans' brain functionality that focuses on the distinctive and relevant pieces in case of handling large amounts of information. The attention unit provides the model the ability to concentrate on relevant features. Specifically, it supports the learning to identify a distinctive part of the data sequence by evaluating its memory at prediction. In this study, not all features contribute equitably to traffic flow forecasting. Hence, we should assign more attention to more relevant features.  
Note that the two widely employed attention types are additive [33] (Equation (3)) and multiplicative [36] attentions. The principal distinction between then consists in the way of computing the alignment score:  
\[\mathbf{e_{t}}=\mathbf{W_{a}}\cdot\mathbf{h_{t}}. \tag{4}\]  
Additive attention, also called Bahdanau attention, is essentially based on a single-hidden layer feed-forward network with tanh activation function for computing the attention alignment score. The alignment score in the multiplicative attention is obtained by reducing the hidden states using matrix multiplications [37].