# Deep Q-Learning-Based Smart Scheduling of EVs for Demand Response in Smart Grids
## 3 Deep Q-Learning for EV Energy Scheduling  
The objective of EV scheduling is to be able to participate collectively in demand response programs (DRs) used to balance the supply and demand of electricity in the smart grid [36]. We considered V2G-enabled EVs (Vehicle to Grid) [37] that can respond to grid operator signals by modifying their energy consumption or generation patterns. We formalize a DR program as follows:  
\[DR=<P=EVs,E_{target}(T)~{}> \tag{1}\]  
where \(E_{target}\) is a goal energy curve provided to the participant EVs. A Q-learning algorithm determines the collective response of EVs. In Q-learning, one popular assumption is that the environment in which the agent learns is described as a Markov decision process. We will use this simplifying assumption, which means that the future state of the environment depends only on the previous states and actions of the agent. We formally define the environment in which the agent acts as follows:  
\[E=<S,~{}A,~{}r,~{}\pi> \tag{2}\]  
where \(S=\left\{S^{k}\right\}\) is the state space, representing the set of possible states that the agent might be in; \(A\) is the action space, representing the set of all possible actions that the agent can take when it is in each state; \(\pi\) is the policy that the agent learns; and \(r\) is the rewards given to the agents as results of his actions. The state space represents all possible mappings of EVs on charging stations over the demand response program interval \(T\). We model it as a matrix with the dimensions \(NxT\):  
\[S^{k}= \begin{bmatrix}s_{11}&s_{12}&\dots&s_{1T}\\ s_{21}&s_{22}&\dots&s_{2T}\\ \dots&\dots&\dots&\dots\\ s_{N1}&s_{N2}&\dots&s_{NT}\end{bmatrix}  \tag{3}\]  
where \(N\) is the number of available charging stations, and \(T\) is the number of timeslots \(t\) for DR. An element of the matrix stores the identity of the EV, which is assigned to a charging station at specific timeslots or otherwise zero:  
\[s_{it}=\left\{ \begin{array}{cc}0,&if~{}no~{}EV~{}is~{}assigned~{}to~{}cs_{i}~{} at~{}t\\ ev_{1D},&if~{}EV~{}with~{}ev_{1D}~{}is~{}assigned~{}to~{}cs_{i}~{}at~{}t\end{array} \right. \tag{4}\]  
For each environment state, the equivalent energy state is determined and stored in a HashMap \(H\), in which the set of keys \(K\) are the individual timeslots in the DR program interval \(T\), and the list of values \(V_{E}\) represent the energy charged or discharged at each charging point:  
\[H_{Energy}:K_{T}\rightarrow\mathit{List}<V_{E}> \tag{5}\]The amount of energy charged or discharged in total by all charging stations in the microgrid per a time slot \(t\) is determined as follows:  
\[(\forall)t\in K_{T},E_{EV}(t)=\sum_{v_{e}\in V_{E}}List<V_{E}> \tag{6}\]  
The initial state of the agent exploration process is represented by a matrix with zero values, indicating that no EVs are assigned to any charging station within the interval \(T\). The agent will explore and learn by performing actions, receiving rewards based on these actions and updating the Q-values. An action corresponds to assigning an electric vehicle \(ev_{ID}\) to a charging station \(cs_{i}\) in a certain timeslot \(t\):  
\[a=(cs_{i},ev_{ID},\ t,\
abla) \tag{7}\]  
When the agent performs an action, it results in a new state, which is represented as a new matrix containing the new configuration of electric vehicles assigned to charging stations. This iterative process allows the agent to learn the optimal policy for assigning electric vehicles to charging stations while considering the specific constraints and the objective defined for our problem. In the defined environment, we considered several constraints on the set of actions the agent may consider to be driven by the physical limitations of the energy infrastructure.