## 4 Model Testing and Results Analysis
### Results Analysis and Comparison  
This section first shows the improvement introduced to the traditional VAE by incorporating the self-attention mechanism at the VAE encoder. We compare it with well known recurrent neural networks, LSTM, GRU, BiLSTM, BiGRU, CNN, and ConvLSTM, as well as baseline methods, namely LR, RR, SVR, and Lasso regression, to forecast pedestrian and bicycle traffic flows. The LSTM and GRU are equipped with memory-cell and gating mechanisms, making them powerful models for time-series modeling and suitable for a comparison study. In these experimentations, the set of hyperparameters is fixed for all considered model-based training datasets: optimizer = 'rmsprop', loss function = 'Cross-Entropy', batch size = 250, epochs = 500, and learning rate = 0.001, activation function = 'Rectified Linear Unit (ReLU)'. The configuration of the proposed approach is: [Input: 3, Intermediate: 6, Self-Attention: 6, LSTM: 16, Variance: 16, Mean: 16, Z: 16, Self-Attention: 4, Self-Attention: 4, Predictor: 1]. For the considered models GRU, LSTM, BiLSTM, BiGRU, and ConvLSTM, we set the hidden units to 32. Here, deep recurrent neural networks are built by stacking two recurrent layers as deep temporal feature extractors and a dense layer used for the forecasting task. For example, for LSTM, we have a stacked-LSTM network containing two LSTM layers with 32 hidden units for each layer and a fully connected layer. All hyper-parameters are determined \begin{table}
\begin{tabular}{l l l l l l l} \hline \hline
**Dataset** & **Location** & **Contents** & **Records** \\ \hline Data 1 & Burke Gilman Trail north of NE 70 th St & Bicycle & 57,697 \\ Data 2 & Burke Gilman Trail north of NE 70 th St & Pedestrian & 57,697 \\ Data 3 & MTS Trail west of 1-90 Bridge & Pedestrian & 57,289 \\ Data 4 & MTS Trail west of 1-90 Bridge & Bicycle & 57,289 \\ Data 5 & Seattle Spokane St Bridge & Bicycle & 51,121 \\ Data 6 & 26 th Ave SW Greenway at SW Oregon St & Bicycle & 55,568 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Traffic datasets used to evaluate the considered models. Figure 4: A week of traffic data from Data 1 and Data 2.  
based on a grid search approach. Similarly, the same architecture is used for BiLSTM and BiGRU models: Deep bidirectional temporal feature extractors and a dense layer used for the forecasting task. Generally, the bidirectional models allow the input to be processed in the forward and backward direction, making it possible to extract more complex hidden features. We used a linear kernel for the SVR model, with the regularization parameter \(\mathrm{C}=100\) and gamma ='scale'. For the Lasso regression, we set the constant that multiplies the L1 term, \(alpha=0.1\), the maximum number of iterations is 1000, and the tolerance for the optimization is \(tol=1\times 10^{-3}\). For RR, the value of the regularization strength is chosen as 1, the maximum number of iterations is 1000, and the precision of the solution is chosen to be \(tol=1\times 10^{-3}\). To show the advantage of the proposed GAHD-VAE compared to the traditional VAE, we applied them to the six traffic flow datasets (Table 3). The proposed approach scored the lowest RMSE for the six considered datasets (3.33, 2.39, 1.58, 4.05, 3.641, 2.824) compared to results achieved by the VAE (8.05, 3.63, 1.61, 6.69, 3.707, 3.257). Furthermore, the averaged \(R^{2}\) and EV values for the GAHD-VAE are (0.963, 0.968) and for the VAE are (0.919, 0.94), respectively. Results demonstrate the significant improvement attributed to the high learning quality and capability of GAHD-VAE, brought by the deep self-attention mechanism and the deep hybrid architecture that incorporates recurrent neural networks. Results in Table 3 also revealed that the GAHD-VAE model exhibited superior prediction performance compared to four shallow methods, linear regression, Lasso regression, ridge regression, and support vector regression. This could be attributed to the ability of a deep learning structure to learn complicated patterns from data.