# Q-RPL: Q-Learning-Based Routing Protocol for Advanced Metering Infrastructure in Smart Grids
## 4 Q-RPL: Q-Learning-Based Routing Protocol Design
### Integration into RPL  
The integration of Q-learning into the RPL protocol represents a significant advancement in routing decision-making for AMI. The Q-learning algorithm in our enhanced RPL framework specifically assists in the parent selection process. Each node employs the learned Q-values to make informed decisions on which next hop to choose as its preferred parent for routing packets toward the root according to the policy defined in Section 4.3. This approach leverages the continuous learning capability of Q-learning to dynamically adapt to changing network conditions, thereby optimizing the routing paths over time. We keep the core of RPL functionalities like ranking calculation based on RFC 6550, trickle timer algorithm, loop detection mechanism, and the use of signaling messages such as DIO, Destination Advertisement Object (DAO), and DODAG Information Solicitation (DIS). However, to facilitate the learning-based approach, we propose a modification in the DIO messages. Each node, in addition to the standard RPL information, broadcasts its maximum Q-value. This modification allows neighboring nodes to be aware of the routing efficacy (as measured by the Q-values) of their potential parents and update the Q-learning equation accordingly. The maximum Q-value is included in the DAG Metric Container in the option field of the DIO messages, as shown in Figure 3.  
Figure 4 illustrates the Q-RPL architecture after integrating the Q-learning modules into RPL. It shows the main modules of RPL alongside the Q-learning modules in the same framework. RPL modules are responsible for handling DODAG formation, maintaining network structure, and facilitating basic communication among nodes. In contrast, the Q-learning modules are tasked with optimizing parent selection. Figure 3: DIO message base format. Another important difference in our design with respect to previous studies is the role of the classic routing metrics such as the ETX and RSSI, while the Q-learning algorithm plays the primary role in the decision-making mechanism, the ETX and RSSI serve as a policy enhancement and for guiding exploration, respectively. In cases where two potential parents have identical Q-values, the ETX metric is used as a tiebreaker. Since the ETX represents the number of transmissions expected to deliver a packet over a link successfully, preference is given to the parent with a lower ETX, indicating a more reliable link. The metric is calculated according to the following expression:  
\[ETX=\frac{1}{Df\cdot Dr} \tag{4}\]  
where \(Df\) is the measured probability that a packet is received by the neighbor and \(Dr\) is the measured probability that the acknowledgment packet is successfully received [21]. The metric is assessed after the transmission of actual data and acknowledgment within the re-transmission mechanism at the MAC layer. Thus, it is updated when a packet is successfully acknowledged or when the maximum number of re-transmissions is exceeded. The ETX is also smoothed by using an exponential weighted moving average (EWMA) filter following best practices:  
\[ETX_{new}=\lambda\cdot ETX_{current}+(1-\lambda)\cdot ETX_{prior} \tag{5}\]  
where the value of \(\lambda\) is implementation dependent, and it has been set to 0.8 as in other implementations [46]. Furthermore, we introduce an exploration trigger based on RSSI variations, a critical metric in ensuring optimal routing decisions, as previously discussed in [18]. To this end, each smart meter actively monitors the RSSI value of its neighboring nodes. If the average of the RSSI of one of the neighbors drops below a threshold, the node is not considered in the exploration phase. This way, the algorithm avoids wasting time exploring nodes with poor link quality. In the case that the node with a declining link quality is the current preferred parent, selected based on its high Q-value during the exploitation phase, the algorithm triggers a reassessment of the parent choice, prompting exploration.