# Federated Reinforcement Learning in IoT: Applications, Opportunities and Open Challenges
## 3 Methodology
### FRL Applications in the IoT in Terms of Sustainability and Efficiency  
[PERSON] et al. [109] models an energy and carbon allowance trading mechanism as a multiagent reinforcement learning (MARL) problem for a building community. The multi-energy system (MES) is a pivotal pillar towards the future low-carbon energy system, but some challenges harden its current exploration. For example, factors that make this exploration complex include the complex operation of combining multi-energy sectors, the privacy aspect of decentralizing the energy system, and the integration of energy and carbon emission schemes. Thereupon, the authors adopt an abstract critic network using a deep deterministic policy gradient method. This pipeline is, then, integrated into a federated learning (FL) framework to preserve the information private of each building in the community. The experiment showed that the proposed strategy achieves 5.87% lower total energy cost and 8.02% total environment costs compared to baselines. The authors of [110] introduce an edge-based backhaul selection method to enhance traffic delivery based on multi-objective feedback. IoT adoption has substantially increased backhaul traffic congestion. This new scenario demands effective traffic management optimization at the network edge. In fact, edge devices can forward IoT traffic through the backhaul network by choosing appropriate links for collected data flows. This selection challenge requires efficient strategies to learn how to handle partially observable components of the network. The authors employ different advantage-actor-critic deep reinforcement learning (DRL) and federated learning (FL) to train a shared backhaul selection policy. Finally, the proposed solution can solve the backhaul selection problem effectively. The contributions of [111; 112; 113; 114] focus on offloading optimization. The authors of [111] introduce FedAdapt as an adaptive offloading FL framework to tackle the efficiency aspects of FL, e.g., consideration of limited computational capabilities, computational heterogeneity, and variable network bandwidths. To accomplish this, this approach adopts the proximal policy optimization (PPO) to identify which deep neural network (DNN) layers can be offloaded for IoT devices onto a server to handle computational heterogeneity and changing network bandwidth. [PERSON] et al. [112] propose a federated DRL-based online task offloading and resource allocation (FDOR) technique. The provision of cloud-like services to IoT offered by mobile edge computing (MEC) is the target of this contribution. The authors also consider wireless powered communication (WPC) technology, given that a base station (BS) can transmit energy to edge users (EUs) and execute tasks via task off-loading. Thereupon, DRL is executed in EUs with aggregated parameters and an adaptive learning rate. The authors use multiple DRL agents situated on edge nodes to specify the offloading choices of IoT devices in [113]. By transferring computationally demanding tasks to edge nodes, the idea of offloading allows IoT devices to conserve energy and sustain the quality of service. Conversely, federation and intricate resource management are dynamically determined in real-time, considering varying workloads and radio conditions. To tackle this, FL trains DRL agents in a spread approach. Furthermore, [PERSON] et al. 's [114] objective was to reduce the amount of energy used by IoT devices while satisfying the threshold for delaying and resource constraints. They created a collaborative optimization challenge that involved both task offloading and resource allocation in accomplishing this. The authors emphasize that privacy disclosure is a present issue in MEC data exchange and that FL can support these transactions. In this sense, a two-timescale federated DRL technique based on the deep deterministic policy gradient (DDPG) is proposed. Similarly, [PERSON] et al.