## 5 Challenges and Open Issues of RL-Based Algorithms in IoT Networks
When dealing with multi-agent learning, the RL algorithm must be able to handle limited device resources in changing network conditions. Multi-agent learning can be applied to develop self-organizing routing algorithms that adapt to changing networking conditions and traffic patterns. Multi-agent systems also face problems such as credit assignments, in which it becomes hard to determine which actions of an agent contribute to a particular outcome. Particularly in a large-scale system, agents must be able to determine not only their own learning objectives but also the potential consequences of their actions on other agents. Similarly, the agents may have conflicting objectives that can lead to suboptimal outcomes. In terms of IoT device routing scenarios, the devices can converge to Nash equilibrium, where no device is incentivized to unilaterally change its routing strategy, especially in a non-convex and dynamic environment. In such scenarios, techniques from game theory can be employed to analyze and identify a Nash equilibrium. (9) Real-timeResponsiveness: Some IoT applications, such as in healthcare or factories, may require devices to transmit real-time information. RL algorithms have long processing times when dealing with large-scale state-action-reward tuples, which further increases in scenarios where state and action spaces are multi-dimensional [89]. It becomes even more complex when the IoT network is heterogeneous, where some devices are delay-sensitive while others are delay-tolerant. DRL algorithms can be more efficient and quick in learning such complexities; however, running the DRL in tiny IoT devices is quite challenging. In such scenarios, edge computing can be leveraged to offload some of the computation and memory requirements of DRL algorithms. However, transferring a large amount of training data increases the number of overheads and burdens on the IoT networks. Another way to solve such a problem is through transfer learning, in which the agent shares their learned parameters. However, this would require an effective coordination mechanism among the devices, which would increase the convergence delay. (10) Explorationand Exploitation Performance Trade-off: The exploration-exploitation trade-off involves a decision between selecting already known good actions or choosing actions that are not explored yet. Balancing exploration and exploitation directly impacts the performance of IoT devices. Using an effective exploration strategy such as the upper confidence bound (UCB) algorithm can help the devices to avoid actions that are known to be suboptimal. However, the UCB algorithm requires calculating the confidence bound for each action, which can be computationally expensive. Similarly, stochastic algorithms like UCB usually assume that the probabilities of different outcomes occurring stay the same over time. Such an assumption of _stationarity_ can result in the algorithm selecting suboptimal actions. Similarly, UCB also assumes a fixed reward function, which may not hold in IoT applications. In IoT devices, the reward function can be dynamic due to the varying nature of the application requirements or environmental conditions. Similarly, algorithms like Thompson sampling, which can solve the exploration-exploitation dilemma, are susceptible to local optima, particularly when the number of actions is large or the environment is complex. The large action space and complex environment also face an exploration bias problem. (11) EnergyEfficiency Trade-off in RL-Enabled IoT Environments: RL algorithms require a significant number of samples to effectively learn optimal policies. In IoT networks with limited resources, sample efficiency is crucial to reduce data collection overheads and speed up learning. During learning, the algorithms may involve frequent exchanges of information between nodes, leading to increased communication overheads in IoT networks. Finding lightweight communication mechanisms is essential to minimize energy consumption.