## 5 Challenges and Open Issues of RL-Based Algorithms in IoT Networks
(12) DynamicChanging Environment for RL Algorithms: IoT devices often operate in a non-stationary environment where the underlying distribution of rewards and states can change over time. This leads to poor performance if the environment changes significantly. In such a scenario, _meta-learning_[90] can be useful for an agent to adapt to new environments. Meta-learning also often faces limited generalization problems, i.e., the ability to adapt to a new and unseen scenario. For example, the nodes in IoT networks may join and leave frequently. This dynamic topology can cause instability in RL algorithms, and the routing decisions made by an RL agent may quickly become obsolete as the network topology changes. The quick adaptation to a dynamic changing environment also depends on the sample efficiency. (13) Implementationin Real-World Scenarios: The primary goal of employing RL algorithms is to optimize the network's performance, including improving the quality of service, enhancing the network's energy efficiency, and reducing latency, among other metrics. RL algorithms have proven to be an effective solution for enabling intelligent approaches for IoT device operation and management. However, the majority of current studies solely assess the suggested RL-assisted control strategy through simulations based on random data or models. This approach is far from the practical environment and may not accurately reflect the challenges and complexities of real-world wireless networks. Thus, to enable the widespread deployment of RL-aided schemes in real-world IoT networks, it is necessary to develop robust and efficient algorithms that operate in changing network conditions, such as changes in traffic patterns, user behavior, and device connectivity. (14) Multi-ObjectiveReward Design Optimization: In heterogeneous IoT networks, devices can have different and multiple performance objectives, e.g., energy efficiency optimization or delay minimization. Thus, designing a practical multi-objective reward function is still a challenge. Devices with different performance objectives need to coexist. Designing a practical multi-objective reward function requires careful consideration of the trade-offs between these objectives. One approach is to assign weights to different objectives which determine their relative importance. However, determining these weights can be challenging, as different devices may have different preferences. One of the ways to find optimal multiple-objective solutions is to use Pareto optimization. However, Pareto optimization can also face the curse of yir dimensionality problem as the number of objectives increases. With the increase in decision objective variables, the computational complexity of Pareto optimization algorithms can become prohibitively expensive. (15) Issueswith Enabling FRL: Enabling federated learning is an even bigger challenge for heterogeneous IoT devices with very limited resources. Apart from learning the environment, the devices would need to share their local learning model parameters with other devices, putting a huge burden on a tiny device. Locally computing the learning model and sending the learned model to other devices in a network with a massive number of heterogeneous devices are difficult. Heterogeneity refers to differences in computation resources (i.e., CPU cycles/sec), memory resources, communication, and energy resources. Having heterogeneous parameters can lead to significant differences in learning model accuracy.