# Q-RPL: Q-Learning-Based Routing Protocol for Advanced Metering Infrastructure in Smart Grids
## 2 Related Work  
RPL was initially defined in RFC 6550 [8] and further shaped by related RFCs such as RFC 6551 [21], RFC 6552 [22], and RFC 6719 [23], which form the core of its functionality. To this day, RPL continues to capture the attention of the scientific community, as evidenced by the following recent studies. These studies reflect ongoing efforts to enhance and adapt RPL's capabilities to meet the evolving demands of current communication networks. Some authors have attempted to improve RPL by exploiting RPL's capability to work with multiple instances. In the context of RPL, multiple instances refer to the capability of the protocol to support more than one distinct routing topology within the same physical network. This feature allows for different routing strategies or criteria to be applied simultaneously for different types of traffic or network conditions. A recent study [24] delved into deploying multiple RPL instances within Wireless Sensor Networks (WSNs). This approach, however, relied on the unmodified standard RPL implementations. A hop-count-based implementation was used for periodic and non-critical traffic, as well as the ETX-based implementation for critical data traffic. Similarly, the research presented in [25] leverages the multi-topology routing capability of RPL to address Quality of Service (QoS) needs for various traffic types. This study introduces a novel parent selection framework using a multi-attribute decision-making method. It aims to overcome the limitations of RPL's single metric approach, demonstrating improved QoS through the multi-topology strategy. Nonetheless, the simulation time used in this work raises questions about the ability to fully assess the long-term effectiveness and adaptability of the proposed solution across diverse network conditions. A further study [26] introduces QWL-RPL, a variant of RPL designed to perform under heterogeneous traffic patterns. This protocol enhancement incorporates a queue and workload-based mechanism for routing decisions. The queue condition is determined by the queue length, i.e., the total number of packets waiting in the queue. In contrast, the workload is assessed by counting the packets transmitted at the MAC layer over set intervals. These metrics serve as indicators of network congestion and traffic load, respectively. Consequently, a node selects its preferred parent based on these criteria, favoring those with lower congestion and lighter traffic loads. Even though performance improvements are observed when compared to the standard RPL implementations, the omission of link quality metrics in the routing decision process could pose a significant drawback. By focusing solely on queue length and workload, there is a potential to overlook the link quality between nodes. This oversight could lead to scenarios where a node with lower packet transmissions and less congestion is selected as a preferred parent, despite having a poor link quality. The authors of [27] tried to improve RPL performance by tackling the load balancing problem. They proposed Weighted Random Forward RPL (WRF-RPL). This variant of RPL combines the energy remaining in the nodes and their corresponding number of parents. These two routing metrics are the base of a weighted random selection algorithm used to choose the best next-hop candidate. WRF-RPL demonstrates improvements in network lifetime and PDR but ignores other important routing metrics related to link quality such as the ETX and RSSI. This could lead to choosing suboptimal routing paths, characterized by higher packet loss rates and reduced overall network reliability. In addition, considering only the number of parents may not accurately reflect the actual load on a node. For instance, nodes with fewer parents could be facing higher traffic loads or processing demands, contrary to those with a larger number of parents that might be underutilized. This approach risks creating an uneven distribution of network load, potentially diminishing network efficiency and performance.