# Q-RPL: Q-Learning-Based Routing Protocol for Advanced Metering Infrastructure in Smart Grids
## 7 Conclusions  
The integration of Q-learning into the RPL protocol, named Q-RPL, represents a significant advancement in the adaptability and intelligence of routing decisions for Advanced Metering Infrastructure networks. By retaining the core functionalities of RPL and augmenting them with a learning-based approach for parent selection, we ensure both the reliability of traditional methods and the advantages of adaptive learning. The modifications in DIO messages for Q-value dissemination and the incorporation of ETX and RSSI metrics as auxiliary decision-making tools further refine the routing process. The simulation results from both the Montreal and Barcelona scenarios consistently show the superior performance of Q-RPL compared to MRHOF, RPL+, ML-RPL, and RI-RPL in diverse and dynamic AMI environments. In terms of packet delivery ratio, Q-RPL outperforms several benchmark routing protocols across various traffic applications. In the Montreal scenario under traffic load 1, Q-RPL demonstrates, for instance, a 12% improvement in the MR application compared to MRHOF, while for alarm events, it surpasses RPL+ by 6%, RI-RPL by 7%, and MRHOF by 10%. A similar trend is observed using the Barcelona scenario, where Q-RPL achieves a 10% higher PDR than MRHOF in the MR application and an 8% greater PDR compared to RPL+ and RI-RPL in the PQ application. The end-to-end delay analysis reveals Q-RPL's efficiency in maintaining low and consistent delay values across traffic applications, especially under an increased network load. In the Montreal scenario under traffic load 1, Q-RPL achieves a lower delay compared to the other routing variants of the order of 50-200 ms on average for MR traffic, and the difference in favor of Q-RPL becomes larger for AE and PQ traffic. This trend continues with traffic load 2, where Q-RPL maintains stable and low median delay values while other protocols such as MRHOF, RPL+, ML-RPL, and RI-RPL experience increased variability. The Barcelona scenario exhibits similar results, with Q-RPL consistently showing lower median delay values and higher consistency compared to the MRHOF, RPL+, ML-RPL, and RI-RPL protocols across all traffic applications for both traffic loads, demonstrating its adaptability to varying traffic conditions. The compliant factor metric further highlights Q-RPL's excellent performance in the area of service quality. Q-RPL, in the scenario of Montreal under traffic load 1, consistently achieves high CF values across all applications, outperforming other benchmark protocols such as MRHOF, RPL+, ML-RPL, and RI-RPL, demonstrating its ability to meet specific application transit time requirements. Even under increased traffic load, Q-RPL maintains the highest CF for the AE and PQ applications, showing its robustness and adaptability. The results observed in the Barcelona scenario are consistent with those in Montreal, indicating that the Q-RPL's CF remains consistently above 94% across all traffic applications and under traffic loads 1 and 2. The adaptive learning capability of Q-RPL is shown in the evolution of average PDR and end-to-end delay over time. The algorithm starts with modest performance during the exploratory phase but steadily improves, reaching optimal PDR levels and minimizing delays. This dynamic adaptability is a key strength of Q-RPL, contributing to its high performance in changing network conditions. In summary, the results obtained in this research strongly validate the promise of integrating Reinforcement Learning into communication routing protocols, leading to enhanced performance in AMI networks. Future work will focus on refining Q-RPL with Quality of Service (QoS) considerations and exploring other Reinforcement Learning models for further comparison. Specifically, we plan to investigate the potential of Deep Q-Networks (DQN) to manage high dimensional state spaces effectively and improve decision-making processes in dynamically changing network environments. Conceptualization, C.L.D.S., A.M.M. and J.P.A.L. ; methodology, C.L.D.S., A.M.M. and J.P.A.L.