# Q-RPL: Q-Learning-Based Routing Protocol for Advanced Metering Infrastructure in Smart Grids
## 5 Performance Evaluation
### Montreal Scenario  
Figure 6a shows the PDR per application measured at the collector for traffic load 1 in the Montreal deployment. Recall that the PDR expresses the ratio of packets successfully delivered to the destination to those generated by the source. Specifically, in the MR application, Q-RPL exceeds the performance of MRHOF, RPL+, RI-RPL, and ML-RPL by 12%, 8%, 8%, and 5%, respectively. In alarm events, Q-RPL matches ML-RPL's performance while \begin{table}
\begin{tabular}{p{42.7 pt} p{56.9 pt} p{56.9 pt} p{56.9 pt} p{56.9 pt}} \hline \hline
**Traffic Load** & **Applications** & **Sending Period** & **Payload (Bytes)** & **Percentage of Meters** \\ \hline \multirow{4}{*}{\(1\)} & Meter reading (MR) & Every 1 h & 400 & 100\% \\ \cline{2-5}  & Alarm events (AE) & Every 1 h & 278 & 25\% \\ \cline{2-5}  & Power Quality (PQ) & Every 1 h & 278 & 25\% \\ \hline \multirow{4}{*}{\(2\)} & Meter reading (MR) & Every 30 min & 400 & 100\% \\ \cline{2-5}  & Alarm events (AEs) & Every 1 h & 278 & 50\% \\ \cline{2-5}  & Power Quality (PQ) & Every 1 h & 278 & 50\% \\ \hline \end{tabular}
\end{table} 
Table 4: SG applications transmitted over each scenario. \begin{table}
\begin{tabular}{p{42.7 pt} p{113.8 pt}} \hline \hline Network simulator & OMNeT++ v6.0.1 \& INET Framework v4.4.1 \\ \hline Simulation runs & 10/per scenario/per traffic load \\ \hline Simulation time & 5 h \\ \hline Smart meters & 200 (Montreal), 355 (Barcelona) \\ \hline Collectors & 1 (Montreal), 1 (Barcelona) \\ \hline \multirow{2}{*}{Channel characteristics} & Path loss, \(\alpha_{p}\) = 3.6 \\ \cline{2-3}  & Shadowing, Lognormal, \(\sigma\) = 7.4 \\ \hline \multirow{8}{*}{PHY Layer} & Standard, 802.15.4g \\ \cline{2-3}  & Frequency band, 2.4 GHz \\ \cline{2-3}  & Transmission rate, 115 Kbps \\ \cline{2-3}  & Transmission power, 14 dBm \\ \cline{2-3}  & Reception sensitivity, –100 dBm \\ \cline{2-3}  & Energy detection, –90 dBm \\ \cline{2-3}  & Min interference power, –120 dBm \\ \hline \multirow{8}{*}{MAC Layer} & Standard, 802.15.4g \\ \cline{2-3}  & Operation mode, Mesh \\ \cline{2-3}  & ACK, Enable \\ \cline{2-3}  & Max re-transmission, 3 \\ \cline{2-3}  & Backoff procedure, Exponential \\ \cline{2-3}  & Min backoff exponent, 3 \\ \cline{2-3}  & Max backoff exponent, 8 \\ \hline \multirow{4}{*}{Learning Layer} & Learning rate, \(\alpha\) = 0.3 \\ \cline{2-3}  & Discount factor, \(\gamma\) = 0.6 \\ \cline{2-3}  & \(\epsilon\) = 1, \(\epsilon_{min}\) = 0.05, \(\epsilon_{decay}\) = 0.95 \\ \hline \multicolumn{2}{p{42.7 pt}}{Packet size \& sending interval} & Application dependent, according to Table 4. \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Simulation Settings. surpassing RPL+ by 6%, RI-RPL by 7%, and MRHOF by 10%. The most significant disparity is observed in the PQ application, where Q-RPL achieves a 99% PDR, outperforming the next closest protocol, ML-RPL, by 5%, and surpassing RI-RPL, RPL+, and MRHOF by 8%, 9%, and 16% respectively. The distribution of the end-to-end delay of the packets that reach the destination is depicted in Figure 5(b) using a box plot representation. In this comparison, MRHOF exhibits the highest median delay across all traffic applications. RI-RPL has the second-worst performance for MR application, but closely matches the median delay values of the other protocols for the AE and PQ applications. RPL+, ML-RPL, and Q-RPL have median delay values within 50 ms of each other across all the applications, however, the box plot reveals that RPL+ and Q-RPL have narrower Interquartile Ranges (IQRs) compared to ML-RPL. This characteristic indicates a more consistent and predictable delay performance of RPL+ and Q-RPL. To complement the end-to-end delay analysis, we present the compliant factor (CF) in Figure 5(c).