# Federated Reinforcement Learning in IoT: Applications, Opportunities and Open Challenges
## 3 Methodology
### FRL Applications in the Industrial Internet of Things
Consequently, the resource management problem is transferred to [PERSON]'s decision process based on a DRL model, which can significantly facilitate the FL training process with satisfied performance. [PERSON] et al. [134] studied the issue of organizing the sequence of tasks in automated warehouses by considering the heterogeneous nature of autonomous robotics. Specifically, the authors defined the long-perspective non-convex queuing control system as an optimization problem by reducing the number of tasks waiting to be processed in the system. Unlike the traditional task-scheduling solutions, the proposed techniques utilized the DRL approach, which can employ the proximal policy optimization method (PPO) to handle the stochastic nature of the task flow and the significant quantity of robots in the system. After that, a proximal weighted FL-based algorithm is implemented to enhance the performance of the PPO agents in geographically distributed warehouses. The experimental results showed that the proposed technique outperforms the current approaches based on simulated data. [PERSON] et al. [137] defined a novel digital twin architecture empowered by IIoT, where the real-time status of the industrial devices can be captured and processed for better intelligent decision making. In the Industry 4.0 environment, regarding the prevalence of the availability of smart industrial devices, the efficiency of data transmission and potential privacy leakage poses significant challenges for both researchers and industrial participants. The proposed architecture can cope with the heterogeneous IIoT devices by optimizing the digital twin platform with FL and DRL. Systematic experiments are conducted to present the advantages of the suggested method over the existing solutions. It is shown that an asynchronous FL architecture is better for solving the discrete effects issues due to the heterogeneous nature of the IIoT devices. Additionally, the designed digital twin system can make the convergence faster while still keeping high performance in the training phase. In addition, numerous studies are concentrating on how to enhance the performance of FL-based IIoT applications and the quality of online services as well. [PERSON] et al. [138] deployed a unique structure of digital twin (DT) to authorize IIoT for Industry 4.0 applications. Specifically, in their designed framework, the DT sensors are capable of detecting the features of industrial devices in order to aid and enhance FL performance. In order to alleviate the effects of estimation deviations from the actual value of the DT device state, the authors adaptively adjust and design the aggregation frequency of FL models based on the Lyapunov dynamic deficit queue and DRL under resource constraints. Furthermore, the heterogeneity of IIoT devices is adapted with a clustering-based asynchronous FL framework. Their experimental results confirmed the excellence of their suggested framework over the current benchmark approaches. [PERSON] et al. [139] presented a new DRL scheme for federated and dynamic network management in IIoT applications. More specifically, under the circumstance that IIoT devices have a stronger computational capacity, the designed architecture is developed to solve the quality-of-service (QoS) satisfaction and secure data sharing issues. By taking into account transmission power and spreading factors across IIoT slices, the authors can deploy effective resource allocation solutions for differentiated QoS services. There are major two steps in the proposed architecture. First, a multi-agent deep Q-learning-based technique is designed for maximizing self-QoS requirements. Second, a DRL-based framework is implemented for optimal decision making on transmission power and spreading factors based on the shared information among agents. The simulated results showed that the proposed architecture is more sufficient than the latest techniques.