## 5 Reinforcement Learning for the CEVRP
### Proposed Q-Learning Algorithm
* If customer node i has already been visited, it is masked. * If node i represents a customer, its unsatisfied demand exceeds the remaining cargo capacity of the EV, and node i is masked. * If the EV is currently at customer i and its battery level is insufficient to visit the next customer j and return to the depot or any charging stations, customer j is masked. * If the EV is currently at any node i and its battery level is insufficient to visit the next charging station j and return to the depot or any charging stations, charging station j is masked. * If a self-loop is detected, where the agent would revisit charging stations or depots unnecessarily before completing new deliveries, the corresponding action is masked to encourage more efficient routing. Additionally, in contrast to the classical Q-learning approach, where the epsilon-greedy strategy relies on a fixed epsilon value to balance exploration and exploitation, we have adopted an Epsilon Decay Strategy. This dynamic adjustment of epsilon ensures a more adaptive exploration-exploitation trade-off, which can be more effective than a static epsilon throughout the learning process. These updates aim to improve the performance and applicability of Q-learning in solving the CEVRP. The step-by-step representation of the proposed Q-learning algorithm is shown in Figure 10. In the initial step of the algorithm, the depot is designated as \(\left(\mathbf{s}_{0}\right)\) and added to the route list. Accessible actions from this initial state are determined through masking and included in the _feasible_actions_ list. Specifically, customer 5 is masked in this step, as it is unreachable from \(\left(\mathbf{s}_{0}\right)\) with a battery capacity of 3000. The pseudo-code of the proposed Q-learning algorithm to solve the CEVRP is presented in Algorithm 1 and Algorithm 2. First, the algorithm creates a Q table with the \"states_size \(\times\) action_size\". The created Q table is initially filled with zeros because the agent has no experience. The training process begins with the initialization of the neural network weights. Subsequently, entire iteration is executed until the agent reaches a terminal state, which is determined by the completion of the tour with all customers served. At each new state st, the agent selects an action using an epsilon-greedy policy balancing exploration and exploitation. Early in training, it is more likely to choose actions randomly (exploration). Over time, as it becomes smarter, the epsilon value decreases, meaning the agent increasingly exploits the knowledge it has gained so far by picking the best action based on the Q-values predicted by the model. When the agent wants to explore (i.e., the random number is below epsilon), it selects a random action from the set of feasible actions determined by the masking scheme. Otherwise, it computes the Q values for the current state and feasible actions, then selects the action that achieves the highest Q value. The chosen action updates the EV's route, battery level, and capacity based on the selected node (e.g., moving to a customer, charging station, or depot). The reward for the action is calculated using the reward function in Equations (15) and (16), which penalizes the agent for longer distances while incentivizing efficient route planning. After the agent executes an action, the environment transitions to a new state, reflecting the updated EV's status (battery, capacity), location, and current node information. \[R=\sum\mathbf{r_{t}} \tag{15}\]  
\[\mathbf{r_{t}}=-(\mathbf{d_{ij}})\ \ \textbf{if}\ i=\mathbf{V_{t-1}}\ \textbf{and}\ j=\mathbf{V_{t}} \tag{16}\]  
```
1:Initialize Q table with zeros and large negative number for self-loops. 2:Set maximum number of iterations \(max\_iterations\). 3:Set \(\gamma\) and \(\alpha\). 4:Set initial epsilon \(\epsilon\), minimum epsilon \(min\_epsilon\), epsilon_decay
5:Initialize \(i=0\). 6:while\(i\leq max\_iterations\)
7:Initialize visited customers list. 8:Initialize route_list to empty list. 9:Initialize total_distance = 0.