# Deep Q-Learning-Based Smart Scheduling of EVs for Demand Response in Smart Grids
## 4 Evaluation Results
This situation may be due to the complex environment in which the agent operates, which may face situations in which its performance temporarily degrades until it adapts to the new conditions. In the case of the L2 model, a significant increase in rewards is observed between 40,000 and 100,000 episodes, followed by a smaller increase between 100,000 and 120,000 episodes. Starting from episode 120,000, the reward stabilizes, registering only a few small fluctuations. Like the L1 model, these small fluctuations are due to the complexity of the environment in which the agent operates and the exploration vs. exploitation trade-off. Looking at the loss evolution presented in Figure 4, we observe a common trend followed by both the L1 and L2 models. At first, the loss decreases as the models learn, and then, after many episodes, it starts to stabilize. More precisely, in the case of the L1 model, the loss starts to stabilize around 57,000 episodes and, in the case of the L2 model, around 37,000 episodes. The need for many episodes to stabilize loss derives both from the complexity of the environment in which the agent operates and from the trade-off of exploration versus exploitation. The agent needs more episodes to accumulate learning experiences and to strike the right balance between exploration and exploitation. After evaluating the quality of the learned model, the next step is to evaluate the quality of the policy learned. For this, we examined the percentage of allocations of electric vehicles at charging stations with a maximum value of the reward, in relation to the total number of EV allocations per time interval and epoch variations. We also examined the  
Figure 4: (**a**) Average loss for L1 model; (**b**) Average loss for L2 model. Figure 3: (**a**) Average rewards for L1-model training; (**b**) Average rewards for L2-model training. similarity between the curve derived from the scheduling of electric vehicles at charging stations and the target energy curve provided by the grid operator using the Pearson correlation coefficient:  
\[Pearson(E_{MG}(T),E_{Schedule}(T))=\frac{\sum_{i=1}^{n}\left(E_{target}(t_{i})- \overline{E_{target}}\right)\left(E_{EV}(t_{i})-\overline{E_{EV}}\right)}{ \sqrt{\sum_{i=1}^{n}\left(E_{target}(t_{i})-\overline{E_{target}}\right)^{2} *\sqrt{\sum_{i=1}^{n}\left(E_{EV}(t_{i})-\overline{E_{EV}}\right)^{2}}} \tag{17}\]  
In Formula (17), \(n\) is the number of energy values corresponding to the time interval during which the EVs' scheduling is performed; \(E_{target}(t_{i})\) and \(E_{EV}(t_{i})\) are the energy data points in the goal energy curve and the curve resulted by scheduling EVs at charging stations; and \(\overline{E_{target}}\) and \(\overline{E_{SV}}\) are energy sample means, computed as follows:  
\[\overline{E_{target}}=\frac{1}{n}{\sum_{i=1}^{n}\left(E_{target}(t_{i})\right)} \tag{18}\]  
\[\overline{E_{EV}}=\frac{1}{n}{\sum_{i=1}^{n}\left(E_{EV}(t_{i})\right)} \tag{19}\]  
Figure 5 presents the percentage of optimal allocations of EVs versus suboptimal ones for the two learned policies corresponding to the charging and discharging scenarios. In the case of the charging scenario, we observe that in 70.8% of the cases, all vehicles are optimally allocated (that is, the trained model optimally manages EVs); in 17.5% of the cases, one EV is a suboptimal allocation; and in 5.6% of the cases, two EVs are suboptimal allocations. Only a small percentage of cases feature more EV suboptimal allocations (i.e., 6.1%). This means that the learned model makes good decisions, since it can optimally allocate EVs in most cases. Even when suboptimal allocations occur, their frequency is relatively low, indicating that the model has learned a policy that works well for this scenario. In the case of the L2 model, in 82.9% of the cases, EVs are optimally allocated to charging stations; in 14.4% of the cases, there is only one suboptimal allocation; and in the case of 2.5%, there are two suboptimal allocations.