# Deep Q-Learning-Based Smart Scheduling of EVs for Demand Response in Smart Grids
## 5 Discussion  
Most of existing studies in EV coordination for demand response consider only charging scenarios for renewable energy integration. Moreover, they consider only a limited number of EVs and rather rigid models of the constraints and EV state-of-charge distributions, lacking adaptation features. Our approach introduces significant innovations by employing model-free reinforcement learning for the scheduling and coordination of a fleet of EVs under a V2G operation involving a bidirectional energy flow between the electric grid and EVs. Unlike model-based methods, this approach learns from past experiences in scheduling EV charging and discharging actions based on grid conditions and requirements defined in DR programs. By integrating user preferences and constraints, such as specific time windows for charging or required battery levels, we ensure that the developed schedules are demand-response-program-friendly but also meet user requirements. The key advantage lies in its adaptability to evolving environmental conditions. Model-free solutions are flexible, unlike their rigid model-based counterparts, enabling them to accommodate more easily new constraints and additional resources. This adaptability proves important in complex and uncertain environments or when facing frequent changes, making them particularly well suited for scheduling EVs in renewable energy-powered microgrids. The solution proposed in this paper can consider adaptively different grids and EV states to generate charging schedules, accordingly avoiding overloads and ensuring grid stability. Figures 8 and 9 present the evolution of the average rewards and losses for the model training for charging, whereby different distributions for the SoC of EVs are considered, as initial states. We generated, for a fleet of EVs, three different types of initial SoC distributions: beta distribution, Gaussian distribution, and uniform distribution. In all three cases, the proposed deep Q-learning solution successfully generates changing scenarios for all EVs without requiring any adaptations. The evolution of rewards during the training episodes follows the same pattern. At first, up to 400,000 episodes, there is a very slow increase in rewards, followed by a significant increase in the rewards from one episode to another until a plateau is reached, where the value stabilizes. In the case of the beta distribution, the highest final reward is obtained, and in the case of the uniform distribution, the smallest. This is because the beta distribution provides a distribution of SoC values of electric vehicles (EVs) that is more realistic and closer to a distribution encountered in the real world. This enables the training of improved models with better performance and higher rewards. In the case of the Gaussian distribution, even though the reward evolution is like the beta distribution, the final reward is less, as the Gaussian distribution does not accurately represent scenarios where EVs' SoC values do not follow a bell curve pattern, leading to the learning models being less efficient. In the case of the uniform distribution, the stabilization of the reward is achieved after a larger number of episodes, and the reward obtained is the lowest. Small fluctuations in rewards after stabilization occur in all three cases due to the complexity of the environment in which the agent operates and the exploration vs. exploitation trade-off. In Figure 9, the loss evolution shows the same trend for all three initial SoC distribution models. At first, the loss decreases as the deep Q-learning agent learns, and afterward, it stabilizes. In the case of the Gaussian distribution, stabilization occurs slightly faster than for the other distributions, as the SoC values are more clustered around the mean, providing more consistent training data, which allows the model to learn and reduce loss faster. For the beta and uniform distributions, the process of loss stabilization in the learning process follows a similar curve.