## 6 Experimental Results
### Results of Proposed Q-Learning Algorithm
In this case, in order to proceed even between two customers, the vehicle had to stop at more than one charging station. When the EV battery capacity is 3000, the routes generated by the proposed algorithm are \"R1: D-35-32-115-2-CS9-D R2: D-42B-D\" and the total distance is 3989.19 m. When the battery capacity is 1000, the routes generated are \"R1: D-32-CS8-115-2-CS9-CS7-35-D R2: \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Number of episodes & 20,000 \\ Discount factor \(\gamma\) & 0.6 \\ Learning rate \(\alpha\) & 0.1 \\ Max epsilon & 1.0 \\ Min epsilon & 0.1 \\ Epsilon decay rate & 0.999 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Training parameters. Figure 11: The learning curve of the proposed algorithm. D-42B-D\" and the total distance is 4686.75 m. In this scenario, it is observed that the electric vehicle requires multiple charging stops between two customer locations. In this study, the proposed algorithm is compared with the mathematical model and Double Deep Q-Network (DDQN) algorithm [29]. The DDQN is a reinforcement learning algorithm and an enhancement of the conventional Deep Q-Network (DQN). The DDQN employs two distinct networks, namely the main network and the target network, to mitigate the overestimation bias that can arise during Q-value estimation. The DDQN model is one of the effective reinforcement methods that has been widely applied in many areas, especially in recent years, including routing problems (e.g., VRP, E-VRP, etc.). In the literature, recent studies using the DDQN model for the VRP and EVRP can be found in references [4; 29; 40]. The DDQN model presented in [29] was employed to solve the CEVRP problem and compared with the proposed method in our study. Table 5 shows the comparative results of the mathematical model, DDQN algorithm, and the proposed Q-learning algorithm. The proposed Q-learning model gives results close to the optimal solutions in small test problems, as illustrated in Figure 12. Furthermore, the proposed Q-learning algorithm outperforms the DDQN algorithm, yielding shorter total distances. This superior performance can be attributed to the Q-learning algorithm's ability to effectively balance exploration and exploitation, thereby identifying more optimal routing paths. Compared to the mathematical model, the proposed Q-learning model significantly reduces the execution time. Large-scale problems can be solved in approximately 30 s, in contrast to the 3 h required by the mathematical model. Thus, the proposed method demonstrates markedly shorter execution times, as illustrated in Figure 13. In terms of execution time, both the DDQN and Q-learning algorithms exhibit significantly faster computation times compared to the mathematical model. While the mathematical model, despite providing optimal solutions, requires considerable computational resources and time, the reinforcement learning approaches (Q-learning and DDQN) offer near-instantaneous solutions. This substantial reduction in computation time underscores the practical applicability of reinforcement learning methods in real-world logistics operations, where timely decision making is crucial.