# Q-RPL: Q-Learning-Based Routing Protocol for Advanced Metering Infrastructure in Smart Grids
## 5 Performance Evaluation
### General Discussion  
In general, the simulation results from both the Montreal and Barcelona scenarios highlight Q-RPL's superior performance in terms of PDR, network latency, and compliant factor compared to established protocols like MRHOF, RPL+, ML-RPL, and RI-RPL. The core strength of Q-RPL lies in its dynamic adaptability, achieved by integrating Q-learning into its decision-making process. This approach markedly differs from MRHOF's reliance solely on the ETX metric and RPL+'s and ML-RPL's dependence on preset rules and a trained ML model, respectively. When compared to RI-RPL, another Q-learning-based protocol, Q-RPL demonstrates a more advanced and effective learning approach. Specifically, RI-RPL's approach of setting the learning rate in the Q-learning formula to one and the discount factor to zero led to overfitting to recent experiences and favored immediate rewards over long-term strategic routing. This approach made RI-RPL behave more like a multimetric routing protocol rather than leveraging the full potential of Reinforcement Learning, which may explain its performance similarities to RPL+. In contrast, Q-RPL distinguishes itself by continually learning from the network performance and adapting its parent selection in real-time with each packet transmission. This flexibility is crucial for managing the varied traffic patterns typical in AMIs. Figure (a)a displays the evolution of the average PDR in the Montreal and Barcelona scenarios for Q-RPL under traffic load 1. This figure provides a clear illustration of the adaptive learning capability of the Q-RPL algorithm. During the initial stages, which are characterized by an exploratory approach, the PDR starts at a modest level. As the algorithm accumulates experience and refines its decision-making, improvements in PDR are observed. After simulating 2 h of network operation, a significant uptick in PDR performance is noted, with values plateauing at an optimal level. This sustained high performance highlights the efficiency of the Q-RPL's learning mechanisms as they converge towards more effective routing choices over time. Figure 10: Average performance metrics progression in Montreal and Barcelona scenarios. Examining the average end-to-end delay as time progresses reveals an inverse correlation with the previously discussed PDR enhancement. Initially, the delay is notably higher in both scenarios, Figure 9(b). In the Montreal scenario, the average delay in the first 30 min is 1100 ms, and exhibits a steep decline, stabilizing at around the 3 h mark of simulation time. Similarly, Barcelona shows an initial average delay in the first 30 min of 906 ms, which then decreases, leveling off at 285 ms after 3 h. This downward trend in delay underscores Q-RPL's ability to reduce latency as the system progressively adapts to the network's conditions. An important consideration in our approach was how to use some routing metrics to assist the Q-learning algorithm, specifically the RSSI and the ETX. We have illustrated the influence of these metrics on the PDR over time in Figure 10(a),b. These figures clearly demonstrate how the inclusion of the RSSI metric speeds up the learning process. This speed-up was expected due to the metric's role in constraining exploration to nodes with superior link quality. The difference observed between the protocol solely reliant on the Q-learning algorithm and the variant incorporating the RSSI metric is more pronounced in the Barcelona scenario, where the denser network of smart meters presents a wider array of routing choices. Thus, the narrowing of exploration space by the RSSI metric has more impact. Figure 10(a),b also depict the beneficial impact of integrating the ETX metric, used as a tiebreaker as described in Algorithm 2. The inclusion of ETX improves the algorithm's performance in both scenarios. After 3 h of network operation, the data show a consistent performance improvement of 2-3% in the final version of Q-RPL compared to the version employing only Q-learning and RSSI.