# Q-RPL: Q-Learning-Based Routing Protocol for Advanced Metering Infrastructure in Smart Grids
## 2 Related Work
In a direct attempt to incorporate RL into RPL, the authors of [40] introduce a new RPL variant for Internet-of-Things (IoT) environments, where the parent selection relies on the Q-learning algorithm. The article addresses critical challenges of RPL such as the negative impacts of instantaneous path selection and the need to consider the dynamic conditions of nodes for parent selection. The approach demonstrates improvements in delivery rates, latency, and energy consumption compared to existing methods as shown through simulation tests. Nevertheless, there are some important considerations regarding the design and implementation that raise concerns and merit closer scrutiny. For instance, the authors assign equal weights to all the metrics in the reward function, which does not accurately reflect their relative importance or interdependencies. In addition, we agree that maintaining a stable parent after the algorithm converges, as the authors do, can enhance the consistency and reliability of the routing path. However, this stability might also reduce the protocol's responsiveness to sudden changes in network conditions after convergence. Another study where RL was used along with RPL is [41]. The study introduces QFS-RPL, a novel RPL-based routing protocol enhanced by the Q-learning algorithm and concepts from the Fisheye State Routing protocol. An important element in QFS-RPL's design is the modification of the traditional Q-function by incorporating additional parameters that reflect the state of the network more comprehensively. According to the results presented in the paper, QFS-RPL is particularly suited for scenarios where nodes are mobile. Conversely, in networks with static nodes, QFS-RPL performs similarly to the standard RPL. Lastly, an adaptive control of transmission power for RPL, named ACTOR, is presented in [42]. This article introduces a dynamic approach to optimizing transmission power to improve throughput in dense networks. ACTOR extends the standard RPL by integrating a specific RL strategy, the Upper Confidence Bound (UCB), to efficiently manage the exploration and exploitation trade-off. This method allows for passive exploration of different transmission power levels, aiming to enhance network performance by dynamically adjusting power settings based on real-time conditions. Although ACTOR demonstrates positive results due to adjustments in transmit power, its main drawback is using only the ETX metric for routing decisions, which is similar to the standard RPL implementation. In light of the comprehensive review of existing modifications and enhancements to RPL, it becomes evident that while significant progress has been made, particularly in employing ML-based techniques, there remain inherent limitations to these routing solutions. Table 1 summarizes the main aspects of the studies discussed in this section. \begin{table}
\begin{tabular}{p{42.7 pt} p{113.8 pt} p{113.8 pt} p{113.8 pt} p{113.8 pt}} \hline \hline \begin{tabular}{} \end{tabular}  \\ \cline{1-1} \cline{2-5}  & QWL-RPL targets heterogeneous traffic with dynamic queue and workload-based routing. & Utilizes queue length and MAC transmission rates for parent selection to minimize congestion.