# Deep Q-Learning-Based Smart Scheduling of EVs for Demand Response in Smart Grids
## 1 Introduction  
The ongoing transition towards various economic sectors' decarbonization enables fossil energy-supply substitution with renewable energy. However, the rapid adoption of small-scale renewables at the edge of the grid makes the smart grid management process more complex and exposed to uncertainties related to renewable production [1, 2]. Digitization and decentralization principles bring, to the forefront, energy demand flexibility as a key support to accommodate high shares of variable renewable energy [3, 4]. Leveraging local flexibility is possible to maintain a balance between supply and demand at lower costs using the energy assets of citizens rather than the ones owned by grid operators, which are more expensive to operate [5, 6, 7]. In this context, demand response programs have been defined by grid operators for adjusting the consumer's electricity patterns in response to the fluctuations in renewable energy production. Consumers are requested to shift their flexible energy usage to times when renewable energy is produced in excess or reduce consumption during peak demand. The challenges are even more evident and difficult to tackle in the context of the increased adoption of electrical vehicles (EVs) [8]. Usually, electric vehicles need to be charged daily. They have a substantial amount of flexibility to be used in demand response programs if their charging and discharging are smartly planned. In this respect, grid management should closely cooperate and interact within a low latency context with EV coordination and aggregation services to procure their energy scheduling flexibility to support local network balancing or to achieve self-sufficiency [9; 10]. However, EV usage has several shortcomings such as a limited battery range, a relatively short battery lifespan, averaging 10-20 years or up to 150,000 miles, and a lack of existing infrastructure for charging electric vehicles. Other major issues refer to the impact of EVs on the power grid, encompassing factors such as the rise in short-circuit currents, deviations in voltage levels beyond standard limits, and the potential impact on the lifespan of equipment due to increased power demands [11]. Studies have shown that fully charging an electric vehicle uses the same energy needed to power a home during peak energy use times, and nowadays, the energy grid cannot deal with a major or sudden spike in EV usage [12]. Moreover, this is not only a problem related to the quantity of electricity consumed by EVs but also the timing of their usage [13]. If many electric vehicles were to be charged in the evening, the grid could experience significant strain, potentially leading to blackouts. To address the challenges of EV coordination for smart grid management, demand response programs are used to convince people to change their energy behavior [14]. Dynamic traffic for electricity is well discussed in the literature as a solution to move the charging schedules of EVs to moments of the day with extra solar energy [15]. However, it still faces open challenges related to the complexity of the pricing structure that involves hourly or time-of-use rates, making EV owners reluctant to engage and requiring EV owners to adapt their behavior based on fluctuating electricity prices [16]. Other solutions are leveraging on the setup of demand response programs for EVs aiming to manage and modify their charging or discharging patterns based on grid conditions [17]. However, this presents a major challenge: the need to balance energy production and consumption in the face of intermittent and non-programmable energy sources which are strongly influenced by atmospheric conditions, making this difficult to predict [2]. In terms of algorithms for smart scheduling and the coordination of EV participation in DR programs, many existing state-of-the-art approaches consider variations of the model-based constraint-satisfaction problem [18].