## 1 Introduction
In this study we propose a Q-learning method specifically designed to solve the CEVRP, incorporating well-defined reward structures and fine-tuned hyperparameters for enhanced performance. In this study, route planning is conducted for multiple EVs. However, the number of vehicles is not predetermined at the outset; instead, it is dynamically determined based on customer demand and route optimization. Routes are found where all customer demands are met and the total distance traveled by the fleet is minimized. The number of EVs in the solution is suggested and optimized by the model as in the study [2,4]. This means that a smaller number of EVs will not be enough for customers' demands and a higher number will be excessive and lead to unnecessary energy consumption. In the proposed method, a single-agent approach is adopted to jointly determine both the routes and the number of EVs within an integrated framework, rather than conducting independent planning processes for each EV. To the best of our knowledge, this study is pioneering work using a Q-learning algorithm to solve the CEVRP. The proposed algorithm is tested using a newly presented dataset. The dataset is based on real-world data, thereby addressing a notable gap in the literature, where datasets for the EVRP and its variants are predominantly derived from synthetic data. Synthetic datasets frequently fail to capture the full complexity and intricacies of real-world logistics problems. The existing datasets for the EVRP in the literature typically consist of customer coordinates (x, y), with distances between nodes calculated using the Euclidean distance metric. However, methods that utilize environmental data, such as reinforcement learning, require additional information related to the environment, including streets, intersections, and traffic density. RL is a general method and can work with any data; however, to correctly model the state transitions in the environment, access to environmental data such as street structure, intersection information, and traffic status often plays a more critical role in these approaches. Therefore, the \"need to use environmental data\" is not a requirement specific to RL; however, since the definitions of \"state\" and \"reward\" in RL are directly shaped by environmental factors, the richer these data are, the higher the learning process and decision quality tend to be. Consequently, the richness and quality of the environmental data directly enhance the learning process and the resulting decision-making quality. Optimal solutions for this dataset are obtained through a mathematical model that addresses the CEVRP, where each EV has a limited cargo load and battery capacity to satisfy customers' delivery demands within specified time windows. Experimental results demonstrate that the proposed algorithm achieves near-optimal solutions in a short time for large-scale test problems. The contributions of this study to the literature can be summarized as follows:  
* A dataset fulfilling the 'environment' requirement for researchers working on the EVRP using reinforcement learning is provided. * Optimal solutions are presented for scenarios with and without time windows (TW) in the dataset generated from a last-mile delivery perspective. * A solution to the CEVRP problem using Q-learning is proposed, and comparisons are made with the mathematical model. The remainder of the paper is organized as follows: Section 2 reviews the literature on related problems and presents the place of the subject in the literature. Section 3 outlines the definitions and steps involved in generating the dataset. In Section 4, the CEVPRTW problem is defined, and the corresponding mathematical model is introduced. Section 5 provides a detailed explanation of the Q-learning method developed to address this problem. Section 6 presents computational experiments and analysis. Finally, Section 7 concludes the paper and discusses future research directions.