## 4 Selection of Computational Intelligence and Reinforcement Learning Solutions
### Deep Q-Networks  
Traditionally, the Q-Learning algorithm builds a memory table, known as the Q-Table, to store Q-Values for all possible state-action pairs. This value represents the return obtained by executing the action \(A_{t}\) at the time step \(t\), which differs from the one indicated by the current policy, and then, following the policy from the next state onward. After each iteration, the algorithm updates the table using the Bellman Equation:  
\[Q(S_{t},A_{t})=(1-\alpha)Q(S_{t},A_{t})+\alpha\times(R_{t}+\lambda\times\max _{\alpha}Q(S_{t+1},a)) \tag{5}\]  
where \(S\) and \(A\) are, respectively, the state (or observation) and the action that the agent takes at the time step \(t\), \(R\) is the reward received by the agent by taking the action, \(\alpha\) is the learning rate, and \(\lambda\) is the discount factor, which assigns more value to the immediate rewards, making them more important [60]. Basically, the agent updates the current Q-Value with the best estimated future reward, which expects that the agent takes the best current known action. Although this algorithm is simple and quite powerful to create an RL agent, it struggles in dealing with complex problems composed of thousands of actions and states. A simple Q-Table would not suffice to manage reliably thousands of Q-Values, especially regarding memory requirements.  
Deep Q-Learning overcomes the limitations of Q-Learning by approximating the Q-Table with a deep neural network, which optimizes memory usage and gives a solution to the curse-of-dimensionality problem, forming a more-advanced agent called the Deep Q-Network (DQN) [44]. Specifically, the deep neural network receives a representation of the current state as the input, approximates the value of the Q-Value function, and finally, generates the Q-Value for all possible actions as the output. After each iteration, the agent updates the network weights through the Bellman Equation (5). Towards that goal, the DQN first calculates the loss between the optimal and predicted actions:  
\[L(\theta)=L(R_{t}+\gamma\max_{a}Q(S_{t},a),\ Q(S_{t},A_{t}))\]  
where \(\gamma\) is the learning rate parameter. The policy parameters are then updated through backpropagation:  
\[\theta=\theta-\alpha\
abla L(\theta).\]  
Several types of loss functions have been proposed in the literature, including the mean-squared error and cross-entropy loss.  
The trial-and-error mechanism of RL could make off-policy algorithms such as the DQN relatively slow in the training process. To address this problem, the DQN usually relies on experience replay, a replay buffer where it stores experience from the past [61]. This approach collects the most-recent experiences gathered by the agent through its actions in the previous time steps. After each training iteration, the agent randomly samples one or more batches of data from the experience replay buffer, thus making the process more-stable and prone to converge.  
Several optimizations have been proposed for the DQN. To stabilize training, DQN solutions often leverage a separate \"target\" network \(Q^{tar}\) to evaluate the best actions to take. \(Q^{tar}\) is updated less frequently than the policy network \(Q\), and often through soft (Polyak) updates. Equally often, DQN solutions adopt prioritized experience replay, which assigns higher probabilities to actions that lead to higher rewards when sampling from the experience replay buffer. Finally, different DQN variants have been proposed to mitigate the reward overestimation tendencies of the algorithm [62], such as the Dueling-DQN [63] and Double-DQN [64].  
Despite the DQN being one of the first Deep Q-Learning approaches, it is still very widely proposed and highly regarded in the scientific literature, as it has consistently demonstrated a remarkable effectiveness in solving a large number of problems at a reduced implementation complexity.