## 5 Reinforcement Learning for the CEVRP
### Proposed Q-Learning Algorithm  
The Q-learning algorithm is a method of learning by interacting with the environment. It aims to learn strategies that tell agents what actions to take under what conditions. It does not need an environment model and can handle random transformation and reward problems. Q-learning finds the optimal strategy for any finite Markov Decision Process (FMDP). In this sense, it maximizes the total return expectation in all successive steps, starting from the current situation. Given the infinite search time and partially random strategy, Q-learning can determine the optimal action selection strategy for any FMDP. This algorithm is an algorithm that uses a temporal difference algorithm and action-value-based off-policy. The action with the highest value among the possible actions is selected at each step. In other words, a greedy approach is applied. In this algorithm, a table called Q-table is created using situation-action pairs. The table size is the \"number_of_states \(\times\) number_of_actions\". To decide which action to choose from the current situation, the Q values in the table are looked at and the maximum Q value is selected. This algorithm works in discrete action sets and deterministic environments. The update rule that forms the basis of Q-learning is given in Equation (14). \[Q(s_{t},a_{t})=\alpha[r_{t+1}+\gamma maxQ(s_{t+1},a_{t})-Q(s_{t},a_{t})] \tag{14}\]  
\(Q(s_{t},a_{t})\) (state, action) in Equation (14) is the current state, \(\alpha\) is the learning coefficient, \(r_{t+1}\)  is the reward value in the reward table, \(\gamma\) is the discount factor, and \(maxQ(s_{t+1},a_{t})\) represents the highest Q value of the places that can be visited. In Q-learning algorithm, a RL agent engages with an environment over a period. At each time step \(t\), the agent receives a state \(s_{t}\) from a state space \(S\) and chooses an action \(a_{t}\) from an action space \(A\). In our approach, the state representation consists of current node id, current battery capacity, and current cargo capacity information. Action \(a_{t}\) determines the node that the EV goes to at step \(t\). The sequence of actions is generated from the initial step 0 to the final step T. Here, \(a_{0}\) is the action of departing from the depot, \(a_{1}\) is the action selected at the first step, and \(a_{T}\) is the action of returning to the depot. The action space \(A\) represents a set of discrete actions. The agent decides to go to a customer, a charging station, or a depot. An essential aspect of the Q-learning algorithm is the balance between exploration and exploitation. Initially, the agent lacks knowledge about the environment and is more inclined to explore rather than utilize existing knowledge. Over time, as the agent interacts with the environment, it gains understanding and increasingly relies on this acquired knowledge. It is crucial to strike a balance between leveraging this knowledge and continuing to explore new possibilities. Without this balance, the Q-value function may converge to a local minimum rather than the optimal solution. To address this, the exponential decay formula is employed. In the study, several enhancements have been implemented to the state-of-the-art version of Q-learning, as outlined in studies in the literature [12], to better address the CEVRP problem. One of the key modifications is the use of a masking technique, which generates possible actions by masking the nodes that the electric vehicle (EV) cannot reach. This ensures that the EV selects from only the feasible actions at each step. By filtering out infeasible options, the masking mechanism enhances the agent's focus on valid actions, thereby aiding in the construction of feasible solutions. The masking function generates a list of feasible actions based on the current state, considering factors like battery level, vehicle capacity, time windows, and visited customers. Invalid actions, such as moving to a node that cannot be reached due to insufficient battery or capacity, are masked out.