## 3 Basics of ML Algorithms
### Reinforcement Learning Algorithms  
For ML and DL algorithms, the training process consists of feeding inputs with labeled or unlabeled data from measurements or observations previously gathered and stored in a dataset. Then, the ML algorithm learns from patterns found in these data. Reinforcement learning (RL) algorithms do not use the same paradigm since they are focused on training an _agent_, which has to be capable of accumulating _rewards_\(r\) through actions \(a\) when observing a desired _environment_. Each action, reward, and information about the environment is stored in a _state_\(s\). Figure 8 describes this process, which consists of a loop whose outputs are data sequences of the state, an action, the reward, and the next state values. Figure 8: Basic scheme of the RL process. Figure 6: Representation of a neuron unit. Figure 7: Basic architecture of a neural network. During training, the agent performs different actions through the environment, considering it a Markov decision process (MDP); each action is penalized or rewarded [48]. The RL agent learns to make smart decisions given a particular state. The function that evaluates the state and decides the best output action is the _policy_\(\pi(s)\). An RL agent aims to collect the maximum sum of rewards \(R(\tau)\), which are commonly weighted by a discount factor as follows:  
\[R(\tau)=\sum_{k=0}^{\infty}\gamma^{k}r_{t+k+1}, \tag{4}\]  
where \(\tau\) represents a sequence of states and actions over time \(t\), \(r_{t}\) is a reward collected at that time, and \(\gamma\) is the discount factor. Even though the objective of RL is to find the optimal policy \(\pi^{*}\), which is the one that leads to the best cumulative reward, there are two main types of RL:  
* Policy-based methods: To learn which action to take given a state, the policy is directly trained. * Value-based methods: A value function is trained to learn the most valuable state, and this value is used to take the action that leads to it. This value function is denoted as \(Q^{*}\) (state value) or \(V^{*}\) (action value). A common approach in RL is Q-learning, an off-policy and value-based method that uses a Q-table to update the action-value function at each step rather than updating it at the end of each episode. Each row of the Q-table consists of a state-action pair value. The \"Q\" means the \"Quality\" of the value of that action at that state. During training, the agent explores the environment to update the Q-table and determine a better approximation of the optimal policy. Figure 9 shows Q-learning and Deep Q-learning approaches (DQL) where the Q table in DQL updates the output values with a DL architecture. In RL, the Bellman equation is a foundation to address the relationship between the value of a state or state-action pair and the expected return or cumulative reward. It is an equation essential to understanding and solving reinforcement learning problems. [PERSON]'s equation is expressed as  
\[V(s)=\max_{a}[R(s,a)+\gamma V(s_{t+1})], \tag{5}\]  
where \(R(s,a)\) refers to the expected cumulative reward given the current state and action, and \(V(s_{t}+1)\) to the associated value of the next state. Given the Bellman equation, it is possible to obtain the optimal policy function to obtain the best action to take given a state. \[\pi^{*}(s)=\arg\max_{a}Q^{*}(s,a). \tag{6}\]  
Among Q-learning value-based RL algorithms, some commonly used approaches in RIS-aided communication systems are the deep-deterministic policy gradient (DDPG) [49],  
Figure 9: Q-learning and deep Q-learning approaches. the twin-delayed deep-deterministic policy gradients (TD3) [50], and the soft actor-critic methods (SAC) [51]. In DDPG, the agent learns deterministic policies from action spaces with high dimensions and continuous outputs. It uses two NNs: the _actor_ network and the _critic_ network. Both networks compute the current state's action predictions and generate an error signal called temporal difference (TD) at each time step.