# Q-RPL: Q-Learning-Based Routing Protocol for Advanced Metering Infrastructure in Smart Grids
## 5 Performance Evaluation
### Montreal Scenario
The CF is the ratio of packets that not only successfully reach their destination but also do so within a predefined delay criteria specific to each application, according to Table 5. The metric is expressed in percentages and is particularly important in networks like Smart Grids, where different applications may have varying and stringent requirements for packet delivery times. The CF has been used in previous works such as [54; 55] for comparing different Smart Grid communication technologies. In summary, the CF can be formulated as follows:  
\[CF=\left(\frac{\text{Total number of packets meeting the delay criteria}}{\text{Total number of successful packets received}}\right)\times 100\% \tag{6}\]  
Figure 6: Performance metrics in the Montreal scenario under traffic load 1. In the case of MR traffic, all the routing protocols under analysis achieve a CF close to 100%. However, Q-RPL displays a clear advantage in AE traffic, improving the CF by 9%, 7%, 7%, and 5% compared to MRHOF, ML-RPL, RI-RPL, and RPL+, respectively. For PQ traffic, Q-RPL maintains the highest CF at 95%. In order to assess the robustness of our novel routing protocol when network load increases, we decreased the sending interval of the MR application by 50% and doubled the percentage of the smart meters sending alarm reports and power quality events. Figure 6(a) shows the PDR achieved by each routing protocol under the new traffic load conditions. Based on the results, Q-RPL demonstrates superior performance across all three applications compared to MRHOF, RPL+, ML-RPL, and RI-RPL. Notably, for the AE traffic, where Q-RPL previously matched ML-RPL's performance, it now surpasses it by 5%, further demonstrating its effectiveness in handling increased network traffic. Figure 6(b) presents the end-to-end delay under the new traffic conditions. Unlike the other routing protocols, which exhibited increased median delay values under higher traffic load, Q-RPL maintained a performance similar to lower traffic conditions, with the median delay hovering around 400 ms across all three traffic applications. In addition, the other protocols showed greater variability in packet delay times, contrasting it with the stability and consistency in the packet delay of Q-RPL. \begin{table}
\begin{tabular}{c c} \hline \hline
**Application** & **Network Transit Time** \\ \hline MR & 2000 ms \\ \hline AEs & 500 ms \\ \hline PQ & 750 ms \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Targeted network transit time for each application. Figure 7: Performance metrics in the Montreal scenario under traffic load 2. Lastly, Figure 7c depicts the CF resulting from the increase in the traffic load. Similar to the outcomes observed with traffic load 1, all protocols perform well in terms of CF for MR traffic. However, a notable decrease in CF is observed for the AE and PQ applications compared to the first traffic load scenario. MRHOF and RI-RPL experienced the most significant reduction. MRHOF dropped its CF by 7% and 10% for AE and PQ traffic, while RI-RPL dropped its CF by 8% for both traffic applications. RPL+ and ML-RPL also saw declines in CF, though to a smaller degree, a 4% and 6% drop for RPL+ and a 2% and 4% CF decrease for ML-RPL in AE and PQ traffic, respectively. Notably, despite a 5% CF reduction in AE traffic, Q-RPL maintained the highest CF among all protocols for AE traffic. In terms of PQ traffic, Q-RPL achieved a CF of 93%, only a slight 2% decrease from traffic load 1, but a still significantly higher CF than MRHOF, RPL+, RI-RPL, and ML-RPL by margins of 13%, 10%, 8%, and 5%, respectively.