## 5 Challenges and Open Issues of RL-Based Algorithms in IoT Networks
Finding the right balance between these objectives can be challenging, as it requires considering multiple factors simultaneously. Accuracy can be evaluated on a per-episode basis or aggregated over multiple episodes to obtain a more stable estimate of the agent's overall performance. However, in more complex tasks, accuracy may require a more nuanced definition. (4) ConvergenceDelay: Convergence is one of the critical factors in the operation of the RL algorithm. The convergence rate of an RL-based routing mechanism depends on the network size. As the number of nodes increases, the convergence rate decreases due to the large number of state-action pairs that need to be explored. In addition, traffic patterns and node mobility also affect the convergence rate. Due to convergence delays, the agent or IoT device would take longer to learn and adjust its routing decision. With longer delays, the agent may continue to transmit packets through a sub-optimal path, leading to a high packet loss rate and lower throughput. Suboptimal path choice may also lead to load balancing issues, leading to instability and congestion in the network. Certain RL-based routing models can handle dynamic link properties, such as delay, reliability, and utilization. In addition, they can manage temporary link cancellations by adjusting the attribute weights to zero or infinity. However, they face significant challenges when the structure or configuration of the network topology changes, such as the addition of new links or nodes to the network. The introduction or removal of nodes from the network results in dimensional changes in the state-action pairs of the RL model. Such a scenario increases the computation overload, training time, and complexities. (5) DelayedReward: Generally, immediate feedback is often available in an RL mechanism. However, some IoT applications can lead to a delayed or sparse reward that can complicate the operation of the learning process. For example, adjusting a device's power consumption may not yield instantaneous changes in energy consumption. This delay in receiving the feedback may hinder the agent's ability to make intelligent decisions. The learning agent making decisions in such environments must account for their actions' delayed effects. Similarly, the agent must determine the contribution of past actions to the current outcome. Approaches like value iteration, policy gradients, and model-based RL can help agents make informed decisions in such environments. Temporal difference learning and eligibility traces are also commonly used to handle these situations. (6) HierarchicalLearning: In an IoT network, devices may be organized in hierarchical structures with varying levels of abstraction. This creates a challenge and complexity in coordinating and optimizing actions across different levels of the hierarchy while accounting for local and global objectives. Such structures also introduce temporal dependencies, where actions at one level may have cascading effects on higher levels. RL agents in such scenarios need to be able to learn policies at different hierarchical levels while considering their distinct objectives. (7) DistributedDecision Making: In a large-scale network, multiple IoT devices can collaborate to optimize a common objective, such as reducing the network delay or energy consumption. In such coordination among agents, handling dynamic network conditions and ensuring convergence to optimal solutions are challenging. Generally, distributed decision making for resource-constrained IoT device networks can be achieved with the help of edge computing to process the data. Distributed decision making may or may not involve learning; the IoT devices can make decisions based on fixed, pre-defined rules. (8) Multi-AgentLearning: The typical IoT network consists of a large number of IoT devices with heterogeneous sensing, computation, and communication capabilities. Multi-agent learning problems arise due to the distributed and dynamic nature of IoT networks.