# Q-RPL: Q-Learning-Based Routing Protocol for Advanced Metering Infrastructure in Smart Grids
## 1 Introduction  
The transformation of traditional electrical grids into Smart Grids (SGs) represents a major step forward in achieving efficient, reliable, and sustainable energy management. A key component of SGs is the Advanced Metering Infrastructure (AMI), which bridges the gap between electricity consumers and utilities. AMI is instrumental in automating meter reading processes and, more importantly, facilitating two-way communication that enables real-time data collection and analysis. This capability is crucial for enhancing demand response, where consumption patterns are adjusted in response to grid conditions, thus optimizing energy use and reducing costs. Consequently, the efficiency and reliability of the communication network supporting AMI are vital, as any disruption could potentially lead to disturbances or inefficiencies in energy distribution and management. Utilization of wireless technologies has become popular in AMI deployments, offering flexibility and scalability. A distinguishing characteristic of some of those wireless technologies is the capacity to form mesh networks. Two standards have been relevant in this domain, IEEE 802.11s [1] and IEEE 802.15.4g [2]. In particular, the latter is the base standardfor Wireless Smart Metering Utility Networks (Wi-SUN) [3] and was created to establish common and consistent communication specifications for utilities deploying SGs [4]. In wireless mesh networks, nodes transmit their own data and also serve as routers, forwarding data across the network. This brings forth the challenge of efficient and reliable data routing, a critical factor for the operational success of mesh networks. The main routing protocols utilized in AMI deployments reported in the literature are: the Geographic Routing Protocol (GPSR) [5], the Collection Tree Protocol (CTP) [6], the Optimized Link State Routing Protocol (OLSR) [7], the Routing Protocol for Low Power and Lossy Networks (RPL) [8], the Hybrid Routing Protocol (HYDRO) [9], the Lightweight On-Demand Ad hoc Distance-vector Routing Protocol-Next Generation (LOADng) [10], and the Hybrid Wireless Mesh Protocol (HWMP) [11]. Among these protocols, RPL holds a distinct position due to its designed characteristic to optimize data routing in low-power and lossy networks (LLNs), such as those found in AMI deployments [12]. Its suitability for being used in this kind of network has been covered in previous studies [13; 14; 15] which have compared RPL to other routing protocols. Despite its advantages, RPL faces challenges in dynamic and diverse AMI environments. Previous works such as [16; 17] have reflected the main limitations of RPL. One significant challenge is optimizing the parent selection process, which is crucial for ensuring efficient and reliable data transmission. The standard RPL mechanisms can lead to suboptimal routing decisions, affecting overall network performance, especially in dynamically changing environments typical of AMI deployments. Consequently, the problem of how a node selects the best candidate node to send or forward a packet among all possible alternatives remains an open research issue that continues to attract significant attention from the research community. This work contributes to addressing this problem by proposing Q-RPL, a novel approach to enhance RPL's parent selection using Q-learning, a Reinforcement Learning (RL) technique. Our motivation for using RL is built upon acknowledging that Machine Learning (ML) models have demonstrated effectiveness, as evidenced in previous studies [18; 19], where ML-integrated RPL outperforms traditional routing protocols in Smart Grid networks. However, recent findings in [20] indicate that the performance of supervised ML models declines when applied to an AMI scenario different from its training environment. This limitation hinders model transferability between scenarios, and this is where Q-learning becomes a promising solution.