# Performance Evaluation of Graph Neural Network-Based RouteNet Model with Attention Mechanism
## 4 Results and Discussion
### Results Analysis  
The R\({}^{2}\) and MAPE values' comparison of four different RouteNet model configurations are shown in Table 1. The Mean Absolute Percentage Error (MAPE) metric provides insight into the precision of a model's predictions by quantifying the average percentage difference between the predicted values and the actual values. The R\({}^{2}\) value, which can vary from 0 to 1, measures how well the independent variables explain the variance in the dependent variable. A value close to 1 suggests the model predictions are very close to the actual data.  
Figure 10: Modified RouteNet internal architecture with iterative message passing with attention layer.  
Figure 11 illustrates a side-by-side comparative analysis of the R\({}^{2}\) values for four different RouteNet model configurations. Figure 11a,c are the existing RouteNet models, while Figure 11b,d are the models where attention is added to the existing one. The existing baseline RouteNet model, shown in Figure 11a, yields an R\({}^{2}\) value of approximately 0.8807 and a MAPE of 13%, indicating a satisfactory predictive accuracy but also highlighting room for refinement due to the prediction deviation. Introducing an attention mechanism to this baseline model, as shown in Figure 11b, not only increased the R\({}^{2}\) value to 0.9574 but also markedly improved the MAPE to 2.4876%. Another existing model, the Modified RouteNet model presented in Figure 11c, incorporates advanced structural enhancements, leading to an R\({}^{2}\) value of 0.9695 and a MAPE of 4.9476%. This indicates that the model significantly outperforms the original baseline in capturing and predicting the network delays. The integration of the attention mechanism into the Modified RouteNet model, shown in Figure 11d, yields the highest R\({}^{2}\) value of 0.9834 and the lowest MAPE of 2.24887%, this configuration as the most effective and precise among the evaluated models. The Cumulative Distribution Function of the relative error of this model is shown in Figure 12, which represents the relative error between the actual and predicted values. Given the extensive size of the test dataset, which includes 17,100,000 samples, the majority of these samples exhibit errors close to zero, resulting in the CDF curve showing a steep ascent near this value. The curve's steep rise at lower error values indicates that the majority of predictions are accurate, with the plateau near the value of 1.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Model Architecture** & **R\({}^{2}\) Value** & **MAPE Value (\%)** \\ \hline Baseline RouteNet Model (Existing) & 0.8807 & 13 \\ \hline RouteNet Model with Attention & 0.9574 & 2.4876 \\ \hline Modified RouteNet Model (Existing) & 0.9695 & 4.9476 \\ \hline Modified RouteNet Model with Attention & 0.9834 & 2.24887 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparison of R\({}^{2}\) and MAPE values for different models.  
Figure 11: Comparison of R\({}^{2}\) Plots of different models. (**a**) Baseline RouteNet (existing). (**b**) Baseline RouteNet with attention. (**c**) Modified RouteNet (existing). (**d**) Modified RouteNet with attention.  
From the results, it is evident that the incorporation of attention mechanisms in RouteNet models leads to improvements in their ability to predict network delays, as demonstrated by their increased performance.