# Deep Q-Learning-Based Smart Scheduling of EVs for Demand Response in Smart Grids
## 3 Deep Q-Learning for EV Energy Scheduling
The type of action for an EV at a charging station at a time slot can be either a charge or a discharge:  
\[C_{1}:\
abla=\{C,D\},a_{ev_{ID}}=C\lor a_{ev_{ID}}=C \tag{8}\]  
An EV will be considered for scheduling only if its current state of charge falls within a predefined range based on its capacity, and this is correlated with the types of associated actions:  
\[C_{2}:SoC_{ev_{ID}}^{MIN}<SoC_{ev_{ID}}\ll SoC_{ev_{ID}}^{MAX}\to a_{ev_{ID}}=C \tag{9}\]  
\[C_{3}:SoC_{ev_{ID}}^{MIN}\ll SoC_{ev_{ID}}<SoC_{ev_{ID}}^{MAX}\to a_{ev_{ID}}=D \tag{10}\]  
The electric vehicle will only be eligible for charging if its \(SoC\) is significantly less than the maximum capacity and for discharging if the \(SoC\) is significantly higher than the minimum capacity. Two different EVs, \(ev_{IDi}\) and \(ev_{IDj}\), cannot be scheduled at the at the same charging station \(cs\) in the same time slot \(t\):  
\[C_{4}:(\forall)cs,ev_{IDi},ev_{IDj}\to a(cs,ev_{IDi},t,\
abla)\lor a(cs,ev_{IDi },t,\
abla) \tag{11}\]  
Finally, the amount of energy charged or discharged by EVs at a timeslot \(E_{EV}(t)\) must not exceed the target energy provided by the grid operator in the DR program:  
\[C_{5}:\ E_{target}(t)\geq E_{EV}(t) \tag{12}\]  
The goal of the agent is to find new environmental states by successively simulating the execution of actions, such that the distance between the target energy curve and the amount of energy charged and discharged by EVs is minimized over the DR program interval \(T\):  
\[\min_{T}(\ E_{target}-E_{EV}) \tag{13}\]  
Moreover, the agent goal is to learn an optimal policy, \(\pi:\ S\longrightarrow A\), that maximizes the cumulative reward obtained by executing a sequence of actions in the defined environment. To find the optimal policy, the agent explores the environment, observes the rewards obtained when taking specific actions in each state, and uses this experience to update his knowledge of the best actions taken in different states. The optimal policy has associated an optimal \(Q-function\) that represents the maximum return that can be obtained from a state \(S^{k}\), taking an action \(a\) and subsequently following the optimal policy:  
\[\pi:S\longrightarrow A,\pi=\left\{\left(S^{k},a\right)\middle|Q^{*}\left(S^{k},a \right)\text{ is optimal},S^{k}\in S\wedge a\in A\right\} \tag{14}\]  
The optimal action-value function \(Q^{*}\left(S^{k},a\right)\) is described by the Bellman equation, which establishes a link between the optimal action value of one state-action pair and the optimal action value of the next state-action pair. \[(\forall)t\in K_{T},E_{EV}(t)=\sum_{v_{e}\in V_{E}}List<V_{E}> \tag{6}\]0  
where \(Q^{*}\left(S^{k},a\right)\) is the new Q-value calculated for the action \(a\); \(\alpha\) is the learning rate; \(r\left(S^{k},a\right)\) is the immediate reward; \(\gamma\) is the discount rate; and \(max_{a^{\prime}}Q(s^{\prime},a^{\prime})\) is the maximum Q-value selected from all Q -values obtained by executing all possible actions in the next state \(S^{k^{\prime}}\). In our model, we set the initial Q-value of a pair \(\left(S^{k},a\right)\) as usually initialized with an arbitrary fixed value before learning begins. The immediate rewards assigned to a state-action pair are as follows:  
\[r\left(S^{k},a\right)=\left\{ \begin{matrix}MAX_{penalty},\text{ \ }if\text{ }a \rightarrow\
eg C_{1}\vee\
eg C_{2}\
eg C_{3}\vee\
eg C_{4}\\ 100*\left|E_{target}(T)-EV(T)\right|,\text{ \ }where\text{ }EV(T)\text{ is linked to }S^{k}\end{matrix} \right. \tag{16}\]  
Figure 1 shows an overview of the Q-Learning model defined for EV coordination for DR participation. The Q-function is modeled using a deep neural network (Q-network), having, as input, the states (i.e., EV mapping to charging stations and time slot states) and outputs the associated Q-value. To train the neural network, we use the \(\varepsilon\)-greedy algorithm [38] to ensure that the exploration-exploitation strategies are balanced.