# Federated Reinforcement Learning in IoT: Applications, Opportunities and Open Challenges
## 3 Methodology
### FRL Applications in the Industrial Internet of Things  
[PERSON] et al. [132] developed a federated learning-based approach that allows for efficient and adaptable management of mobile edge computing (MEC) in the context of the industrial internet of things (IIoT). More specifically, to handle the problem of network optimization and resource allocation in IIoT networks, the authors deployed a DRL algorithm with FL settings. The proposed system can optimize three essential attributes, i.e., the proportion of task offloading, the allocation of bandwidth, and transmission power. Comprehensive  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Number** & **Work** & **Domain** & **Technology** & **Goal** & **Year** \\ \hline
1 & [PERSON] et al. [124] & Offloading & Deep Q-learning & Optimize energy consumption across Roadside Units (BSUs) & 2022 \\  & & Optimization & Deep Q-learning & and load balancing across fog servers & 2021 \\ \hline
2 & [PERSON] et al. [125] & Offloading & Asynchronous Advantage & Resource slicing and scheduling for & 2021 \\  & & Optimization & actor-critic (AIC) & trustworthy \& cellular services & 2021 \\  
experimental results showed that their optimization approach can decrease the cost of the system while also decreasing the cost of communication, expressed in normalized terms. [PERSON] et al. [133] proposed a DRL-based framework to determine the most efficient control approach and learning effectiveness in practical use cases, such as autonomous driving and robotics. In particular, the multi-agent environment is considered to perform training and share learning parameters (e.g., gradient) to enhance the training quality and performance. Similar to [134], the actor-critic PPO algorithm is deployed within different RL experimental simulations, namely, OpenAI Gym's CartPole, MoutainCar, Acrobot, and Pendulum. The designed weighted federation policy management enables parameter sharing to solve the multi-agent control problem more effectively and efficiently. At last, the authors demonstrated that the proposed policy control system can be used in more complex and realistic application scenarios. [PERSON] et al. [135] focused on the research problem of device assignment and resource allocation in distributed IIoT platforms. They designed a three-layer collaborative architecture in FL settings for optimizing device selection and computational resource allocation. By minimizing the training loss with a stochastic optimization approach, the proposed architecture can reduce the delay in data transmission and reduce long-term energy consumption. Moreover, a reinforcement on federated (RoF) scheme is designed and executed in a decentralized manner at edge servers based on deep multi-agent RL. By utilizing a device refinement subroutine that is integrated into the RoF-based method, the suggested architecture can speed up the rate of convergence while still keeping the on-device energy consumption low. The comprehensive simulated experimental evaluation demonstrated that the proposed scheme outperforms the benchmark techniques with regard to both performance and efficiency. With the explosive availability of smart devices in IIoT, Industry 4.0, and digital twin, many smart devices should be deployed and executed at the same time. As a result, device task assignment and scheduling have become another fundamental problem in industrial applications. [PERSON] et al. [136] introduced a three-layer collaborative FL-based architecture to find a solution for the resource management and the problem of managing the schedule for devices in IIoT. In particular, the DNN model is trained locally at each industrial device side, and then the model parameters (e.g., the gradients) are aggregated, as described in FedAvg algorithm. In order to enhance the efficiency in FL training among resource-limited devices, an optimization algorithm is created to provide a solution for the resource allocation issue while still strictly following the requirements of the FL epoch and device resource consumption.