# Deep Q-Learning-Based Smart Scheduling of EVs for Demand Response in Smart Grids
## 4 Evaluation Results
In the case of memory size, we started with a small memory size, which we gradually increased, and looked at how it influenced the agent's learning capacity. In the case of a small size, the number of experiences the agent can store is limited, and as a result, due to insufficient space exploration, the policy learned is suboptimal. In the case of a large memory size, the agent accumulates a wide range of experiences, which leads to an improvement in exploration and a reduction in the occurrence of the phenomenon of overfitting. However, too large values lead to increased computational costs. Based on these findings and considering the dimension of the environment in which the agent learns, we settled on the value of 700,000 for memory size. The epsilon decay value was chosen as 0.99996 to promote exploration at the beginning of training and progressively reduce random decisions as the model learns. Using this value, the agent does more exploration in the initial episodes, thus gathering a wide range of transition experiences. As more episodes pass, epsilon decay gradually decreases, meaning that, over time, the agent takes fewer random actions and begins to make decisions based on information learned from the neural network. The learning rate (\(\eta\)) value of 0.001 was chosen, because smaller updates to model weights lead to more stable learning in complex environments, even though more training episodes are required to converge to the optimal policy. A batch size value equal to 50,000 was chosen, as this gives the best results for loss and reward metrics. A higher value implies a learning process that converges more slowly (i.e., longer training time) and also does not effectively improve the accuracy of the results, while a smaller value would imply a learning process that converges faster but gives results less accurately. After establishing the optimal configuration for network hyperparameters, we proceeded to evaluate the quality of the training model and of the learned policy. Table 3 presents the metrics used in the evaluation process of the trained model and learned policy. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multicolumn{3}{c}{**Hyperparameter Tuning**} & \multicolumn{3}{c}{**Results**} \\ \hline
**Memory Size** & **Batch Size** & \(\epsilon_{decay}\) & \(\eta\) & **Loss** & **Reward** \\ \hline
50,000 & 1000 & 0.99996 & 0.001 & 4,386,0637 & \(-\)107,799 \\
50,000 & 700,000 & 0.9995 & 0.001 & 4,693,810 & \(-\)106,921 \\
500,000 & 15,000 & 0.99996 & 0.001 & 49,102,022 & \(-\)102,242 \\
50,000 & 700,000 & 0.99996 & 0.005 & 88,181,132 & \(-\)104,005 \\
50,000 & 700,000 & 0.99996 & 0.01 & 252,101,836 & \(-\)103,339 \\
50,000 & 700,000 & 0.9997 & 0.001 & 82,704 & \(-\)3852 \\
600,000 & 30,000 & 0.99996 & 0.001 & 78,076 & \(-\)4050 \\ \hline
700,000 & 50,000 & 0.99996 & 0.001 & 14,587 & 654 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Configuration of network hyperparameters and obtained results. \begin{table}
\begin{tabular}{l l} \hline \hline
**What is Evaluated** & **Metrics Used** \\ \hline
**Quality of Trained Model** &
\begin{tabular}{l} Average Reward \\ Loss \\ \end{tabular} \\ \hline
**Quality of Learned Policy** &
\begin{tabular}{l} \% of optimal/suboptimal allocations of EVs \\ \end{tabular} \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Metrics used to evaluate the quality of the trained model and of the learned policy. The first step in the evaluation process is the evaluation of the quality of the learned model. Figure 3 shows the evolution of average rewards during L1- and L2-model training for charging and discharging operations. In the case of the L1 model for charging, we can see that the average rewards of the learning model stabilize around 140,000 episodes; then, at 150,000, there is a temporary worsening of the received rewards, followed by a return and stabilization of the rewards starting from 16,000 episodes.