# Self-Organizing Networks for 5G and Beyond: A View from the Top
## 4 Machine Learning Algorithms for SON
### ML Algorithms Taxonomy
#### 4.1.3 Reinforcement Learning (RL)  
This specific subcategory has been already proven as effective for autonomous vehicles, network elements routing, and other applications, including self-optimization. The basic rationale is based on learning through interactions with neighbor nodes. In more detail, the specific algorithms receive as input a reward function indicating that they work as expected [9; 15]. Reinforcement learning is tightly coupled with Markov Decision Processes (MDP) in terms of defining a possible set of network states and a possible set of actions for each state. Possibly the most common examples are Q-learning and Fuzzy Q-learning. The specific algorithms are based on the rationale that for any finite Markov decision process (MDP), an optimal policy for awarding or maximizing the expected value of each step is identified. Deep reinforcement learning is the evolution of RL that seems to be an even more promising technique for 5G and beyond networks [84; 85; 86; 87; 88]. Reinforcement learning (RL) and deep reinforcement learning (DRL) as an advancement of RL is a very promising approach in the field of machine learning for B5G networks dealing with sequential decision making. The difference between the two relates to the fact that the former is based on dynamic learning with a trial-and-error method, while the latter is learning from existing knowledge and applies it to a new data set. However, the operation rationale is very similar and they are expected to have a key role in future telecommunication systems. RL is vital due to the ability that can be provided to the agents to learn in real time with no previous knowledge of the environment and continuously interact with it, contrary to the other legacy ML techniques that require full knowledge of the environment and a training set as input, provided by an external resource to the algorithm. The main rationale is that an agent, which in our case might be a UE or a base station (BTS/NodeB/eNodeB/gNB), learns the optimum behavior by collecting information in real time and tries to obtain the maximum reward at each algorithm step [84; 85; 86; 87; 12]. The key terms related to RL algorithms are the following:  
* Environment (e): A real-life scenario that an agent must interact with. * Agent: An entity or a node that performs specific actions in an environment in order to gain some reward. * State (s): The current situation in the environment. * Reward (R): A return provided to the agent when a specific action is taken. * Policy (\(\pi\)): The rationale that the agent follows in order to decide on the next action according to the current state. * Value (V): The long-term reward to the agent compared to the short-term reward. * Value Function: The value of a state as a total amount of reward gained by the agent. * Q value (Q) or Action Value: An estimation of how good it is to take the action at each state. Figure 10 depicts the interaction between the environment and the agent and correlates the action, state, and reward terms related with RL. Apart from the fact that the agent interacts with an uncertain environment and must face it in a holistic manner, one of the key challenges related to reinforcement learning is the exploitation and exploration tradeoff. This specific tradeoff is related to the selection of the actions already selected in the past from the agent proven to provide a beneficial reward and at the same time explore the environment so that even better actions are selected. We cannot avoid referring to the ancestor of RL, the Bellman equation, which is a functional equation that describes the behavior of a dynamically changing environment over time and led to the development of dynamic programming methods that try to handle optimal control scenarios in such environments [88].