# Deep Q-Learning-Based Smart Scheduling of EVs for Demand Response in Smart Grids
## 6 Conclusions  
In this paper, we proposed a deep Q-learning solution for enabling EVs to coordinate and participate in DR programs by scheduling the EVs' charge and discharge actions to meet a target energy profile. We adapted the Bellman equation to evaluate the EV scheduling state and defined a Q-function to determine the action's effectiveness in terms of rewards. We represented the Q-function using a neural network to learn from for dynamic EV scheduling scenarios by successively training the available actions and using epsilon-greedy algorithm balances' exploitation and exploration of the state space.  
Our proposed solution offers a promising approach to EV participation in DR programs, which can result in significant benefits for both the distribution system operator and EV owners. The results show that the EVs are scheduled to effectively charge and discharge their collective energy profile to successfully meet the demand requirements' adaptation in DR programs. Starting from different initial states with different distributions of SoC of EV batteries, the proposed algorithm can learn energy-operational schedules to accurately follow the energy profile provided as a target in DR, with a Pearson coefficient of 0.99. This relives not only the learning algorithm's effectiveness for a fleet of EVs' energy management but also its adaptability to different EVs and microgrids without prior configurations and fine-tuning.  
In all evaluation cases, the rewards and losses converge to good values, showing the effectiveness of the quality of the learning policy defined for a fleet of EV scheduling as well as of the Q-function in improving the agent's decision making in the defined microgrid environment. Finally, a limitation of our approach is the fact that the learning process needs a substantial number of episodes to achieve losses and rewards because of the complexity of the environment. There are many states to navigate to gather learning experience due to scheduling combinations of EVs, charging stations, and trade-offs between exploration and exploitation.  
Figure 13: DR adaptation by scheduling the energy charge of a fleet of EVs using different SoC distributions as the initial state for the learning process: (**a**) beta distribution, (**b**) Gaussian distribution, (**c**) uniform distribution.  
As future work, we intend to investigate new EV scheduling solutions by iteratively considering each time slot in the DR program window and not the entire interval, as in the current approach. This type of solution may reduce the number of episodes needed for loss and reward convergence while improving the agent's responsiveness to changes in the defined EV scheduling environment and promoting greater dynamism in decision making. In addition, we plan to investigate alternative deep Q-learning network architectures to allow the agent to learn and adapt more efficiently with fewer episodes.  
Conceptualization, V.R.C., C.B.P. and T.C.; methodology, T.C. and V.R.C.; formal analysis, V.R.C. and H.G.R.; investigation, L.A. and H.G.R.; resources, I.A.; data curation, H.G.R. and C.B.P.; writing--original draft preparation, V.R.C., T.C., H.G.R. and C.B.P.; writing--review and editing, C.B.P., V.R.C. and I.A.; visualization, H.G.R., I.A. and C.B.P.; project administration, T.C.; funding acquisition, T.C. All authors have read and agreed to the published version of the manuscript.  
This research received funding from the European Union's Horizon Europe research and innovation program under the Grant Agreement number 101136216. Views and opinions expressed are, however, those of the author(s) only and do not necessarily reflect those of the European Union or the European Climate, Infrastructure, and Environment Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.  
Not applicable.  
The raw data supporting the conclusions of this article will be made available by the authors on request.  
The authors declare no conflicts of interest.