# Federated Reinforcement Learning in IoT: Applications, Opportunities and Open Challenges
## 2 Background
### Reinforcement Learning (RL)  
Reinforcement learning (RL) is the result of the fusion between the trial-and-error \"law-of-effect\" tradition, optimal control theory, the secondary reinforcement tradition, and the use of different stimulus [24].  
Reinforcement learning techniques have demonstrated their efficacy in various important applications [25]. When applying reinforcement learning (RL) in complex applications, it is common to use generalizing function approximators such as neural networks, decision-trees, or instance-based methods [26]. RL studies how systems can learn and predict the consequences of environmental interactions [27]; FL relies on an agent interacting with these environments, learning an optimal policy in many fields [28], making a series of decisions over time, aiming to achieve goals that may be delayed while also managing uncertainty and randomness. It focuses on making decisions quickly rather than relying on lengthy analysis or higher-level reasoning [29]. To solve problems related to reinforcement learning, there are two primary approaches. The first involves exploring different behaviors to discover one that is effective in a given environment. The second approach is to utilize dynamic programming methods and statistical techniques to gauge the usefulness of actions taken in specific states of the world [30].  
In the past few years, there have been several RL contributions across multiple fields. Furthermore, there are several opportunities to use RL in new applications and develop new RL approaches [15; 16].  
There are many important RL components. A _state_ represents the configuration of the environment for a given task [31; 32], while actions are functions RL agents can execute to change or interact with the environment [33; 34]. These actions can generate new states. In fact, the reward function produces a score for an action executed for a given state [35] (the output--i.e., the reward--can be positive and negative. Although the term \"rewards\" may lead to an understanding that the outcome is always positive, negative signals are also provided to inhibit non-optimized decisions.). Furthermore, a policy represents an association between actions and states from the reward standpoint [36; 37]. Figure 1 illustrates the general RL process model.  
Moreover, several RL techniques have been proposed in the last decade. These models are becoming more complex and solving challenging problems with high performance. Although several efforts have been made in this direction, some of them can be considered applications for different IoT systems.  
Q-learning [40; 41] has been used in IoT monitoring [42] and resource allocation [43]. Deep Q-learning [44] and double deep Q-learning (DDQN) [45] have supported transmission scheduling in IoT [46; 47]. Proximal policy optimization (PPO) [48] and advantage actor-critic (A2C) [49; 50] were adopted for computation offloading [51; 52], while the deep deterministic policy gradient (DDPG) [53] can be adopted for intrusion detection [54] in green IoT. Finally, the asynchronous advantage actor-critic (A3C) [55] has been adopted in service placement [56].  
Figure 1: Reinforcement learning (RL) process [38; 39].