# Federated Reinforcement Learning in IoT: Applications, Opportunities and Open Challenges
## 3 Methodology
### FRL Applications in the IoT in Terms of Sustainability and Efficiency
[115] introduce a federated DRL framework focused on multi-objective optimization problems to decrease the delay and energy usage of IoT devices' long-term task completion. The main goal is to handle, in a distributed fashion, the offloading decisions, resources, and transmit power allocation. The experiments demonstrated that the proposed framework is effective in the cases considered. Moreover, resource allocation is another aspect considered in many efforts. The authors of [116] focus on resource allocation in device-to-device (D2D)-enabled 6G using FL. The suggested approach takes into account a D2D-enabled wireless network in the underlay mode and decentralized resource allocation to maximize the capacity while minimizingthe overall energy usage. In fact, resource allocation and further improving spectrum utilization are challenges faced nowadays. Thereupon, the authors considered the quality of service (QoS) requirements of both cellular users and D2D users and demonstrated that this method achieves significant network performance. [PERSON] et al. [117] propose a resource allocation method named concurrent federated reinforcement learning. The challenge considered relies on the limited information faced when resource allocation is planned at edge hosts, whereas lack of privacy is the result of moving this process to central servers. The primary concept is to control the privacy-preserving aspect of FL combined with the RL efficiency. Then, the authors adopt concurrency as joint decision-making to achieve global solutions. Similarly, the authors of [118] adopt a DQN to optimize decisions regarding energy and WiFi channels without any pre-existing network information. This method outperforms baseline approaches and maximizes successful transmissions while reducing energy and channel expenses to a minimum. [PERSON] et al. [119] present an FL protocol to enhance the efficiency of FL systems powered by renewable energy sources. This effort is motivated by the challenges of limited device resources faced by industrial FL deployments. The authors focus on using RL for scheduling devices to adjust to inconsistent renewable energy supply. Moreover, the authors introduce an efficient bandwidth management scheme focused on communication efficiency. The experiments showed that this proposal outperforms state-of-the-art methods. The authors of [120] present a residential energy management system (EMS) using a personalized federated DRL (PFDRL) system to address the issue of reducing standby energy consumption. This privacy-preserving and cloud-free proposal is motivated by (i) the challenges of handling various in standby mode, consuming energy while waiting for wireless communication, and (ii) the potential personal data leakage of existing solutions. Moreover, global collaborative models produce unsatisfactory energy management performance since capturing individual residential characteristics is complex. The authors of [121] present a federated DRL-based cooperative edge caching (FADE) framework to solve the lack of self-adaptivity in dynamic environments of most existing methods. Current proposals focus on centralized solutions, so the authors enable base stations (BSs) to develop a predictive model that can be shared. This strategy provides fast training and separates learning from storage, relying on a distributed-centralized procedure. Similarly, [PERSON] et al. [122] introduce a hierarchical federated DRL (HFDRL) method to predict future users' requests to identify appropriate content replacement strategies. The efficiency of edge caching reduces access time and optimizes content transfer, and novel smart caching solutions have been suggested during recent years. In addition to existing methods, the authors categorize edge devices hierarchically and improve local and global performance. The results obtained in the experiments showed that the proposed strategy presents improvements in, e.g., hit rate and delay have improved over traditional methods.