# Deep Q-Learning-Based Smart Scheduling of EVs for Demand Response in Smart Grids
## 4 Evaluation Results  
In this section we evaluate and test the effectiveness and efficiency of our EV scheduling method described in Section 3 for implementing demand response programs. We have used a data set of EVs, and for each EV, the following information is available: vehicle ID; model; battery power, expressed in kW; the battery capacity, expressed in kWh; the connector type; and the state of charge of the battery, expressed in percentages. The following types of EVs are considered: (a) Renault ZOE, with a battery capacity of 22 kWh and a maximum charge power of 22 kW; (b) Renault ZOE, with a battery capacity of 41 kWh and a maximum charge power of 22 kW; and (c) Nissan LEAF, with a battery capacity of 24 kWh and a maximum charge power of 7 kW. We started the evaluation process by first setting up the configuration environment (see Table 1). The state-of-charge (SoC) values of electric vehicles (EVs) in this environment configuration follows a random distribution. In our study, we explored two demand response-inspired scenarios, with the aim of balancing the electricity supply and demand in a microgrid and reducing the likelihood of failures. In the first scenario, electric vehicles are charged during periods of excess renewable energy in the microgrid, thereby adjusting the total energy demand. This aligns the energy consumption of electric vehicles with the peak of renewable energy supply. In the second scenario, energy stored in electric vehicles is discharged to address grid congestion during periods when the peak energy demand exceeds actual production. We then generated the curves for energy demand and for renewable energy production that need to be balanced through electric vehicle (EV) scheduling. The two curves were generated by simulation. Next, we moved on to Deep Q-Network (DQN) training. In the training process we considered the following input features for model learning: the number of available charging stations; the number of EVs waiting for charging/discharging; the time slots for scheduling; the state of charge (SoC) of each EV; the capacity of each EV; and the energy available in each time slot, representing the energy available for scheduling during that specific hour. This energy value dynamically updates every hour by (i) subtracting the energy needed to charge electric vehicles allocated to charging stations at that hour from the initial target energy value provided by the network operator in the case of charging operations or (ii) adding the energy discharged from electric vehicles allocated to charging stations at that hour to the initial target energy value provided by the network operator in the case of discharging operations. In the training process, the DQN relies on the replay memory, which stores the agent's past experiences. After completing each episode, the agent selects a batch of experiences from this memory and uses it to train the network. The training process of the Deep Q-Network (DQN) involves adjusting several hyperparameters in the learning process, namely, the memory size, learning rate, epsilon decay, and batch size. To identify the optimal values of these hyperparameters, we used a trial-and-error approach that involves testing different values for these parameters, observing their impact on model performance, and selecting the values that produced the best results. Table 2 shows the configurations tested and the results obtained in terms of minimum loss and maximum reward for 160,000 episodes for each considered configuration. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Model ID** & **Environment Configuration** & **Scheduling Actions** & **DR Program** \\ \hline L1 & No. Charging stations = 6, = No. EVs = 30, No. Renault & Charge EVs & Renewable peak mitigation \\  & ZOE 22 = 10, No. Renault ZOE 41 = 10, No. Nissan & & \\ \hline L2 & LEAF = 10, DR program period T = 5/t = 1 & Discharge EVs & Congestion management \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Environment configuration, actions, and DR program.