# Federated Reinforcement Learning in IoT: Applications, Opportunities and Open Challenges
## 2 Background
### Federated Learning (FL)  
Federated learning (FL) entails the process of learning a shared model from distributed sources on various client systems in order to reduce prediction error [17]. [PERSON] et al., define FL as \"a machine learning setting where multiple entities (clients) collaborate in solving a machine learning problem, under the coordination of a central server or service provider. Each client's raw data is stored locally and not exchanged or transferred; instead, focused updates intended for immediate aggregation are used to achieve the learning objective\" [57]. A common training procedure comprises client identification, broadcast, local computation, aggregation, and local and global updates [57]. All of these phases are pivotal for training and defining a federated model, and they can be performed in different ways. For instance, the aggregation phases include the mixture of various factors (or weights) executed by a centralized system as illustrated in Figure 2. This entity is in charge of considering the various inputs provided by the different clients. An approach used in several efforts is the FedAvg [58; 59], which is based on merging clients' weights considering local updates. FL has been successfully used in multiple scenarios [60; 61; 62; 63] and presents pivotal advantages (e.g., privacy). In reference [60], the authors suggest a federated learning (PEFL) method for IAI that is both efficient and provides enhanced privacy. PEFL is designed to be non-interactive and capable of protecting sensitive data from disclosure, even if multiple entities cooperate to breach it. The authors of [61] introduce a novel architecture that enhances data privacy through security. They propose a privacy-preserving federated learning mechanism, incorporating a two-phase approach involving intelligent data transformation and collaborative data leakage detection to mitigate privacy risks. By combining blockchain technology with on-device learning, this [62] study proposes a novel approach to federated learning. This offers a promising solution for securing machine learning while preserving privacy. Another scenario proposed in [63] is a framework for privacy-preserving federated learning in smart agriculture. The framework employs a deep privacy encoding method to protect the data privacy of each participant in the federated learning process. There are different categories of FR models. Although new approaches are under development by the scientific community, some of the most popular classes are as follows:  
* **Vertical FL:** This strategy refers to scenarios where data samples present in each client's environment share the same target while presenting different features [66]. In other words, clients may have different features referring to the same target [67; 68];
* **Horizontal FL:** Refers to the use of the same feature space by different clients while considering different data samples [66]. This refers to a more structured way of distributed learning as the model weights can be combined due to the reference to the same features [69];  
Figure 2: Federated learning (FL) overview [64; 65]. * **Cross-device FL:** This refers to the utilization of multiple devices (e.g., IoT devices) to train a global model in a likely massive distributed dataset [70]. In other words, the number of training clients can be extremely large [71; 72];
* **Cross-silo FL:** This approach is based on the consideration of entities (e.g., companies or organizations) as training clients in different industrial sectors (e.g., transportation) [73]. Compared to cross-device FL, the number of clients tends to be smaller. Furthermore, each entity participates in the entire training process [74; 75]. There are new federated learning (FL) efforts under active development, e.g., trustworthy federated learning [76; 77], and federated anomaly detection [78; 79; 80].