# Deep Q-Learning-Based Smart Scheduling of EVs for Demand Response in Smart Grids
## 2 Related Work  
Reinforcement learning-based approaches (RLs) for EV scheduling have been studied in the literature to optimize the charge and discharge decisions in a dynamic and uncertain environment. [PERSON] et al. [21] schedule electric vehicles for charging and discharging using a deep Q-learning-based method that considers both the mobility of electric vehicles and the random charging behavior of users. The EV's scheduling is made by considering a distribution model for user charging times, a model for charging demands, a dynamic model for EV state of charge, and a model for travel locations. A dynamic reward function, the electricity price, the charging and discharging cost, and the battery degradation cost are employed to optimally schedule EVs, such that the economic operating cost is reduced, and the efficiency of the charging/discharging process is improved. However, this method suffers from a drawback, namely, the dynamic reward function does not accurately reflect the benefits of charging/discharging operations when user behavior deviates from the assumptions made in the travel location model. [PERSON] et al. [22] also propose a deep RL-based method for EV charging and discharging in real time. The aim is to minimize the EVs' charging cost and to reduce grid loads during peak hours. To model the usage patterns of charging stations, a parametric density function estimation is used that allows the learning agents to make optimal charging and discharging decisions. The effectiveness of this approach has been demonstrated through simulations, and the results demonstrate that a significant reduction in the energy cost and the grid load is obtained. [PERSON] et al. [23] combine a representation network with deep RL to schedule the charging and discharging of electric vehicles in a domestic setting, such that the EVs' charging/discharging cost is minimized. The representation network (i.e., LTSM network) is used to extract features from historical electricity prices that, together with the EVs' battery SoC and user driving patterns (i.e., the arrival and departure times), are given as inputs to the Q-learning network to estimate the optimal action-value function. Like in the previous work, the effectiveness of this approach has also been demonstrated through simulation. In their work, [PERSON] et al. [24] use deep Q-learning to schedule EVs for charging by considering the battery SoC, the battery capacity, the longest trip distance, the distance between the EVs' current location and the charging station, the waiting time until the next charging slot is available, and the charging station power. The aim is to manage electric vehicle congestion by scheduling them in advance at charging stations and optimizing drivers' journeys. Like in the case of previous approaches, this approach's effectiveness has been demonstrated through simulations. [PERSON] et al. [25] address the problem of scheduling electrical vehicles for charging, aiming to reduce carbon emission costs and peak loads in a community. The proposed approach integrates three algorithms: an offline scheduling algorithm that considers EV profiles, the EVs' random arrivals, and power constraints to minimize the carbon emission cost; an online heuristic rolling-based scheduling algorithm that considers the EVs' arrival times and SoC to make carbon emission decisions; and an actor-critic RL scheduling algorithm that updates the EV carbon emission actions. This approach has been tested through simulations. [PERSON] et al. [26] schedule EVs for charging to minimize the cost of charging EVs and stabilizing the voltage of the distribution network. The scheduling and voltage-control problems are modeledas a Markov decision process that considers the uncertainty introduced by the charging behavior of electric vehicle drivers and an uncontrolled load, as well as fluctuations in energy prices and renewable energy production. The deep deterministic policy gradient algorithm is used to return discrete and continuous control actions.