## 5 Challenges and Open Issues of RL-Based Algorithms in IoT Networks  
IoT-based applications have experienced tremendous growth, providing new perspectives on data gathering and transmission. These applications, such as smart and sustainable cities, can potentially affect diverse areas of our lives. Smart city IoT infrastructures span from local area networks to city-wide area networks. These networks include a number of IoT applications, such as smart street lights, smart parking management, smart surveillance systems, and so on. These applications generally depend on the limited resources of IoT devices, particularly energy and efficient communication resources. The RPL protocol can be applied to manage and route data in smart street lights, facilitating real-time control and energy conservation. Similarly, in smart surveillance systems, numerous cameras and sensors are deployed across a city to monitor security. The RPL ensures that the data from these devices are efficiently transmitted to central monitoring stations. RPL-protocol-based routing aided by the RL mechanism can be helpful for such applications to reduce communication overheads and energy consumption. Devices learning to route with minimum overhead support would enable working for longer periods of time. While RL has proven to be a robust methodology to infuse intelligence into IoT devices for communication operations, several issues and challenges still hinder the full exploitation of RL's potential to aid the IoT paradigm. This section discusses the critical challenges and corresponding open issues that need to be addressed to employ RL-based approaches to enhance IoT device capabilities. RL-based algorithms require time and resources to process large volumes of data during the exploration and exploitation phases. Meanwhile, IoT device resources are limited in terms of storage, energy, and computation. Running an RL algorithm for a long time on IoT devices is challenging and, therefore, requires a lightweight and appropriate algorithm design to handle such challenges. This section outlines the following primary challenges and investigates the open issues related to employing RL-based methods for IoT device networking. (1) Synchronization:Devices learning routes through RL-based algorithms such as Q-learning or the policy gradient method can lead to different routing information due to the stochastic nature of the network. The synchronization problem refers to the challenge of ensuring that all devices in the network have consistent and up-to-date information about the optimal routes. (2) HighDimensions of the State-Action Space in Large Networks: The dimensions of the state-action space can considerably affect the performance of RL-based routing mechanisms. Particularly, if the network size increases, it makes it difficult for an IoT device running the RL algorithm to explore all possible state-action pairs. Such a problem is also referred to as the \"curse of dimensionality,\" where the number of possible states and actions grows exponentially. Exploring the large number of samples to explore the state-action space causes poor performance and slow convergence. (3) Accuracyin RL Decision Making: Achieving accuracy in decision-making is one of the critical challenges, especially in scenarios where precise actions are essential. Generally, RL algorithms are concerned with achieving maximum rewards. However, making more optimal decisions, such as in a scenario like autonomous driving, is of utmost importance. Accuracy is also particularly challenging in noisy and uncertain environments. Selecting appropriate metrics for evaluating accuracy in RL is non-trivial. Metrics should align with the specific goals of the RL problem. For example, in the case of IoT networks, the devices may be heterogeneous in nature and often have specific QoS requirements, such as low latency for real-time applications or energy-efficient routing for battery-powered devices.