# Q-RPL: Q-Learning-Based Routing Protocol for Advanced Metering Infrastructure in Smart Grids
## 2 Related Work
These metrics include the ETX, MAC layer losses, channel utilization, and throughput. Based on the assessed importance of each metric, weights are assigned within a forwarding score function, facilitating the identification of the most suitable forwarding node among available candidates. The main limitation of this proposal is the static weight assignment, which can restrict its flexibility and responsiveness to varying network load conditions. That said, the simulation results show that RPL+ achieves a notable improvement in PDR, outperforming standard RPL implementations across various network sizes. The authors of [18] proposed ML-RPL. This research explores the potential of ML to enhance wireless communication networks, specifically in the context of Smart Grids. This proposal integrates CatBoost, a Gradient Boosted Decision Trees (GBDT) algorithm, into RPL to optimize routing decisions. The final ML model was trained and optimized on a dataset of routing metrics obtained from many simulation campaigns, considering a real deployment of smart meters in the city of Montreal. Each smart meter uses the ML model to predict the probability of successfully reaching a destination node, and then the candidate node with the highest probability of effectively being reached is chosen as the preferred next hop. ML-RPL significantly improved the PDR compared to a standard RPL implementation and RPL+. Another ML-based variant of RPL is presented in [19] based on the dataset obtained in [18]. In this case, the authors used the Gaussian Naive Bayes algorithm and integrated it into RPL, resulting in GNB-RPL. The main goal was to take advantage of the benefits of this ML algorithm that the authors considered particularly relevant in SG wireless communication scenarios such as simplicity, scalability, and reduced training data requirements. In [20], is exposed the main limitation of the previous two studies: both Catboost and Naive Bayes are supervised ML methods, so their performance in making the best routing decisions will depend on the dataset on which they are trained. Therefore, a model trained and optimized for a particular scenario could underperform significantly in another type of scenario, as is shown in [20]. A potential solution to avoid the need for frequent retraining or updates to adapt the supervised ML models to changing network scenarios, which can be impractical in dynamic AMI environments, is the application of Reinforcement Learning. RL has been gaining attention in various routing contexts as shown in [36; 37; 38]. For the applicability of this technique on RPL or any other routing protocol utilized in AMI networks, the following works are of special interest. The research presented in [39] introduces a Reinforcement-Learning-based routing protocol for WSNs. The protocol aims to choose the best parent node in a tree topology by using Q-learning. According to the simulation results shown in the article, the proposed method outperforms two linear-weighted-sum-based parent selection algorithms, in terms of packet delivery ratio, end-to-end delay, and energy consumption. However, some aspects of the protocol design could benefit from further elaboration. For instance, a more detailed explanation of the reward function's construction, including how various performance metrics are integrated, would enhance the clarity and robustness of the decision-making process. Additionally, the protocol's approach of utilizing periodic hello messages every 5 s may pose scalability challenges as network size increases. Exploring adaptive strategies for the frequency of these messages, similar to the trickle timer algorithm used in RPL, could potentially optimize network efficiency and scalability. Lastly, adding concerns to the overall network efficiency is the protocol's cycle detection mechanism, which requires each node to send a join request message containing its list of child nodes to prospective parent nodes.