# Multi-View Multi-Attention Graph Neural Network for Traffic Flow Forecasting
## 3 Methodology
### Prediction Network Module
#### 3.3.2 Spatial and Temporal Self-Attention Module  
In a real subway network scenario, passenger flow patterns may exhibit cyclical trends in multi-time granularity, such as time-dependence on weekly, daily, and adjacent time slots. In order to effectively capture trends in passenger flow state evolution over different periodic, three types of time intervals are coded in our model: (1) recent time intervals (e.g., trends over the first few adjacent periods of the forecast period), (2) short-term time intervals (e.g., daily trends), and (3) long-term time intervals (e.g., weekly trends). Semantic embedding of the different time intervals learned is then fed into the spatial and temporal Self-Attention Module. Formally, the formula for the temporal and spatial self-attention module is defined as follows:  
\[P_{(pos,2i)}=\sin\Bigl{(}pos/10000^{2i/d_{model}}\Bigr{)} \tag{11}\]  
\[P_{(pos,2i+1)}=\cos\Bigl{(}pos/10000^{2i/d_{model}}\Bigr{)} \tag{12}\]  
\[e_{ii}=a_{1}\Bigl{(}W^{Q}(X_{i}+P),W^{K}(X_{i}+P)) \tag{13}\]  
\[e_{ij}=a_{2}\Bigl{(}W^{Q}X_{i},W^{K}X_{j}\Bigr{)} \tag{14}\]  
Time interval-specific representations of metro stations are concatenated as \(X_{i}=\left\{x_{i}^{lx},x_{i}^{t_{d}},x_{i}^{t_{r}}\right\}\). Here, \(W^{Q}\in\mathbb{R}^{d\times d}\) and \(W^{K}\in\mathbb{R}^{d\times d}\) represent the parametrized weight matrix of the embedding \(X_{i}\). \(a_{1}\) and \(a_{2}\) are two common forms of attention computing. \(a_{1}\) refers to \"Scaled Dot-Product Attention\", \(P\) refers to where we further embed the position into the node representation in order to segment the chronological information of the sequence. We use \(\sqrt{d}\) in order to scale the input vector, and we can prevent it from getting into saturated areas or making the gradient too small. \(a_{2}\) is a shared attentional mechanism which indicate the importance of node \(j\)'s spatial features to node \(i\). And \(a~{}:~{}\mathbb{R}^{d}\times\mathbb{R}^{d}\rightarrow\mathbb{R}\).  
\[\alpha_{i}=softmax\biggl{(}\frac{e_{ii}}{\sqrt{d}}\biggr{)}(X_{i}+P)W^{V} \tag{15}\]  
In this paper, \(a_{2}\) is a single-layer feedforward neural network, and we further inject graphic structure into the mechanism by performing masking attention, which means only \(e_{ij}\) for nodes \(j\in N_{i}\) could be computed attention coefficients, where \(N_{i}\) is some neighborhood of node \(i\) in the L-space metro network. In addition, softmax function is used for normalization in order to make the coefficient easy to compare between nodes. Finally, the attention coefficient is passed through the LeakyReLU nonlinear function, with the final expansion as follows:  
\[\alpha_{ij}=\frac{\exp\bigl{(}LeakyReLU\bigl{(}e_{ij}\bigr{)}\bigr{)}}{\sum_ {k\in N_{i}}exp\bigl{(}LeakyReLU\bigl{(}e_{ik}\bigr{)}\bigr{)}} \tag{16}\]  
This paper will use a multi-heads mechanism to capture information along temporal dimensions, which will also enhance the capabilities of the prediction module and the robustness of the training process as follows:  
\[H_{t}^{w}=concat\Bigl{(}\alpha_{i}^{1},\;\alpha_{i}^{2},\alpha_{i}^{3}, \cdots\alpha_{i}^{M},\Bigr{)} \tag{17}\]  
\[H_{s}^{w}=ELU\Biggl{(}\frac{1}{M}\sum_{m=1}^{M}\bigl{(}\alpha_{ij}\bigr{)}^{m} W^{m}x_{i}^{tw}\Biggr{)} \tag{18}\]  
\[H_{st}^{w}=(H_{t}^{w}+H_{s}^{w})/2 \tag{19}\]  
\[\mathbf{H}_{2}=att\Bigl{(}H_{st}^{w},\;H_{st}^{d},H_{st}^{r}\Bigr{)} \tag{20}\]where \(ELU\) is activation function, \(M\) represents the number of attention heads in temporal encoder, we concatenate the output of \(M\) heads and average the final result, \(\mathbf{H}_{2}\) is the output feature of the multi-head spatial and temporal self-attention.