# Machine Learning in Beyond 5G/6G Networks--State-of-the-Art and Future Trends
## 3 Machine Learning
### Reinforcement Learning  
Reinforcement Learning (RL) is based on the principles of behaviourist psychology and the model learns the same way as a child learns to perform a new task. RL is realized on the basis of a feedback performance indicator (reward) conceived from the model's environment. The model pursues the ideal performance of the output by maximizing the indicator of the reward. RL is a hybrid of supervised and unsupervised learning, because (indirect) supervision is required for the model to understand and learn the ideal \begin{table}
\begin{tabular}{l l l l} \hline \hline
**ML Approach** & **Advantages** & **Limitations** \\ \hline \multirow{4}{*}{k-means} & \(\bullet\) & Easy to implement & \(\bullet\) & Manual choice of k \\  & \(\bullet\) & Suitable for large datasets & \(\bullet\) & k greatly impacts performance \\  & \(\bullet\) & Adapts easily to new examples & \(\bullet\) & Can cluster outliers \\  & & & \(\bullet\) & Scales with dimensionality \\ \hline \multirow{4}{*}{SOM} & \(\bullet\) & \(\bullet\) & Needs large number of data \\  & \(\bullet\) & Easily understood & \(\bullet\) & Nearby data points must behave similarly \\  & \(\bullet\) & Capable of clustering large and complex datasets & \(\bullet\) & Finds different similarities among sample \\  & \(\bullet\) & Denoising training & \(\bullet\) & Computationally expensive \\ Auto-encoders & \(\bullet\) & Dimensionality reduction & \(\bullet\) & High complexity \\  & \(\bullet\) & Able to learn non-linear feature representations & \(\bullet\) & Prone to overfitting \\ \hline \multirow{4}{*}{PCA} & \(\bullet\) & Fast & \(\bullet\) & Independent variables are less interpretable \\  & \(\bullet\) & Dimensionality reduction & \(\bullet\) & Needs data standarization beforehand \\  & \(\bullet\) & Reduces overfitting & \(\bullet\) & Incapable of learning non linear feature representations \\ \hline \multirow{4}{*}{HMM} & \(\bullet\) & Can handle inputs of variable lengths & \(\bullet\) & Does not take into account the sequence of \\  & \(\bullet\) & Can combine into libraries & \(\bullet\) & States leading into any given state \\  & \(\bullet\) & Can learn from raw input data & \(\bullet\) & Dependency between appliances cannot be \\ \hline \multirow{2}{*}{RBM} & \(\bullet\) & Can encode any distribution & \(\bullet\) & Difficult training procedure \\  & \(\bullet\) & Computationally efficient & \(\bullet\) & Needs weight adjustment \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Advantages and limitations of unsupervised ML methods. Figure 2: Self Organizing Map Model. system's performance, while there is no available training dataset paired with the desired output [15]. Basically, RL is a trial and error procedure where an agent interacts with the environment and based on whether the action tried was good or bad, gets feedback in terms of reward or penalty. RL tries to learn the best policy that would enable the agent to make an optimal decision at any given state of the environment.Figure 3 displays an example of RF. RL algorithms can be categorized to value-based (e.g., Q-learning, SARSA) and policy-based algorithms (e.g., Policy Gradient (PG), Proximal Policy Optimization (PPO) and Actor-Critic (A2C) [29]. * Q-learning: Q-learning is the most common used RL algorithm. It is an off Policy technique and uses a greedy approach to learn the needed Q-value. The algorithm learns the Q-value given to the agent in a certain state, based on a specific action. The approach creates the Q-table, where the number of rows represent the number of states, and the number of columns represent the number of actions. The Q-value is the reward of the action at a certain state. Once the Q-values are learned the agent can make quick decisions under a current state by taking the action that has the largest Q-value from the table [30]. * SARSA: It is an on-policy algorithm which uses each time the action performed by the current policy of the model, in order to learn the Q-values [19].