## 2 ML Methodologies
### Reinforcement Learning
\begin{table}
\begin{tabular}{p{113.8 pt} p{113.8 pt} p{113.8 pt}} \hline \hline
**Algorithm** & **Ref.** & **Description** \\ \hline Q-learning & [185] & Q-learning is a model-free reinforcement learning algorithm that learns an action-value function, known as the Q-function. It iteratively updates the Q-values based on the rewards received and estimates the optimal policy for an agent to maximize its cumulative reward over time. \\ \hline Deep Q-network (DQN) & [186] & DQN is an extension of Q-learning that utilizes deep neural networks to approximate the Q-values. It overcomes the limitations of traditional Q-learning by enabling the agent to handle high-dimensional state spaces. DQN incorporates experience replay and target networks to stabilize and improve learning performance. \\ \hline Policy gradient methods & [187] & Policy gradient methods directly learn a parameterized policy that determines the agent’s actions based on the observed state. These methods use gradient ascent to iteratively update the policy parameters, aiming to maximize the expected cumulative reward. Common variants include REINFORCE, proximal policy optimization (PPO), and trust region policy optimization (TRPO). \\ \hline Actor-critic methods & [188] & Actor–critic methods combine policy gradient and value function estimation. The actor learns the policy, while the critic estimates the value function to evaluate the policy’s performance. This approach provides a balance between exploring new actions and exploiting the current policy, enhancing the stability and efficiency of learning. \\ \hline Proximal policy optimization (PPO) & [189] & PPO is a policy optimization algorithm that employs a surrogate objective function to update the policy parameters. It ensures that policy updates remain within a specified range, preventing drastic policy changes. PPO is known for its sample efficiency and stable learning performance, making it a popular choice for continuous control tasks. \\ \hline Deep deterministic policy gradient (DDPG) & [190] & DDPG is an off-policy actor-critic algorithm that is well suited for continuous action spaces. It uses a deterministic policy to learn the optimal policy, and a deep neural network is employed to approximate both the actor and critic functions. DDPG combines Q-learning and policy gradient methods, enabling stable learning in continuous action domains. \\ \hline Monte Carlo methods & [191] & Monte Carlo methods estimate the value of states or state-action pairs by averaging the observed returns from sampled trajectories. These methods do not rely on a model of the environment and learn directly from episodes of interaction. They are suitable for episodic tasks where the complete trajectory is available. \\ \hline Temporal difference (TD) learning & [192] & TD learning combines ideas from both Monte Carlo methods and dynamic programming. It updates value estimates based on bootstrapping, using estimates from subsequent time steps. TD algorithms, such as SARSA and Q-lambda, enable learning during ongoing interactions without requiring complete episodes of experience. \\ \hline Asynchronous advantage actor–critic (A3C) & [193] & A3C is an actor-critic algorithm that uses multiple agents operating in parallel to learn a policy and value function. Each agent interacts with a separate copy of the environment, and their experiences are asynchronously combined to update the shared network parameters. A3C is known for its scalability and efficient use of computational resources. \\ \hline \hline \end{tabular}
\end{table} 
Table 18: Reinforcement algorithms.