# Federated Reinforcement Learning in IoT: Applications, Opportunities and Open Challenges
###### Abstract  
The internet of things (IoT) represents a disruptive concept that has been changing society in several ways. There have been several successful applications of IoT in the industry. For example, in transportation systems, the novel internet of vehicles (IoV) concept has enabled new research directions and automation solutions. Moreover, reinforcement learning (RL), federated learning (FL), and federated reinforcement learning (FRL) have demonstrated remarkable success in solving complex problems in different applications. In recent years, new solutions have been developed based on this combined framework (i.e., federated reinforcement learning). Conversely, there is a lack of analysis concerning IoT applications and a standard view of challenges and future directions of the current FRL landscape. Thereupon, the main goal of this research is to present a literature review of federated reinforcement learning (FRL) applications in IoT from multiple perspectives. We focus on analyzing applications in multiple areas (e.g., security, sustainability and efficiency, vehicular solutions, and industrial services) to highlight existing solutions, their characteristics, and research gaps. Additionally, we identify key short- and long-term challenges leading to new opportunities in the field. This research intends to picture the current FRL ecosystem in IoT to foster the development of new solutions based on existing challenges. internet of things (IoT); federated reinforcement learning (FRL); reinforcement learning (RL); federated learning (FL); survey +
Footnote †: journal: _Sciences_  
Furthermore, privacy concerns have become more relevant in the past few years [12]. The interaction with such devices can disclose private information in different forms, e.g., financial transactions [13] and transportation states [14]. The need for privacy-preserving solutions is critical in several applications. In fact, strategies to enable different IoT systems to interact while preserving confidential information can be used in the development of solutions in multiple domains. Moreover, reinforcement learning (RL) has demonstrated remarkable success in solving complex problems in different scenarios [15; 16]. Additionally, federated learning (FL) has enabled global statistical models to be developed based on distributed remote systems in a way to reduce local error [17]. The combination of these two concepts, namely, federated reinforcement learning (FRL), focuses on enabling joint and privacy-preserving learning in sequential decision-making problems [18]. In recent years, new solutions have been developed based on this combined framework [19; 20]. These solutions enable different entities to work collaboratively to achieve faster convergence and more robust results [21; 22]. The same applies to IoT systems, where different systems can improve internal operations based on the experiences collected from global systems. Conversely, although some works focus on analyzing FRL contributions [23], there is a lack of analysis concerning IoT applications. Additionally, an analysis of FRL applications subdomains in the IoT context is necessary to shed light on short- and long-time research directions. There is a lack of understanding regarding challenges and future directions analysis of the current FRL landscape in different IoT applications. Finally, the description of open challenges is important to foster the development of new solutions based on the current issues faced in the intersection between FRL and IoT. Thereupon, the main goal of this research is to present a literature review of federated reinforcement learning (FRL) applications in IoT from multiple perspectives. We focus on analyzing applications in multiple areas (e.g., security, sustainability and efficiency, vehicular solutions, and industrial services) to highlight existing solutions, their characteristics, and research gaps.