## 4 Proposed Q-Learning Method for Smart Grid Price Forecasting
### Q-Learning-Based Electricity Price Forecasting
#### 4.2.3 Using \(Q\)-Learning for Dynamic Pricing Problem
The algorithm filters through the list through brute force recursion, and it calculates the total weight and value of all the subsets. Moreover, it will only consider the subsets whose total weight is smaller than the maximum capacity W. The values V are shown in Figure 9 and they show the difference in prices amongst four competitive producers. Moreover, the figure also compares the producer prices with the wholesale price. Figure 8: Front end application using React.js. * **Customer Input** The other inputs that the algorithm takes are the dissatisfaction parameters--dmul, alphan, and betan. Moreover, the customers' curtailable demand and critical demand shown in Figures 10-12 are also used for determining the optimal retail price at specific times of the day, given the high demand during peak hours and the low price at off-peak hours. Figure 10: Critical and curtailable load for customer 1. Figure 9: Different prices of producer and provider. ### Numerical Simulation Results

Figures 13-15 show the simulations of the energy demand, consumption, the wholesale price, and the retail price per hour for an entire day. Each of the time slots represents the value of T in the sections above. Here, all the parameter values are specific, and they can be changed as per the characteristics of the market and the providers, producers, and the customers. However, they do not change the overall performance of the simulation in a negative way. Figure 11: Critical and curtailable load for customer 2. Figure 12: Critical and curtailable load for customer 3. From the graphs in Figures 16-18, it can be seen that the agent at the beginning does not know the best actions that result in the higher \(Q\)-values; however, as the algorithm iterates, the agent learns from the environment gradually and then finally converges to the maximum \(Q\)-value. Figure 14: Energy demand and consumption with optimal retail price for Customer 2. Figure 13: Energy demand and consumption with optimal retail price for customer 1. The simulation results in the optimal retail prices for each of the customers in question. Each of the figures shows the optimal retail price along with the prices from the providers for each time slot T. Moreover, the curtailable load and energy consumption is also shown for the customers. The graph also shows that the retail price shows a very similar pattern to the wholesale price but never exceeds it due to the price bounds and dissatisfaction parameters. The price elasticity of the entire day shows a continuing increase in demand as well as the increase in retail price and then a gradual decrease in both. This is because, during peak hours, the electricity demand is maximum and more elastic; therefore, it results in a higher energy consumption. Figure 16: Change of \(Q\) values. Figure 7:0 Energy demand and consumption with optimal retail price for customer 3. ## 6 Conclusions and Future Works

In this paper, the major drawbacks of traditional grids as well as conventional electricity pricing strategy implementations, which were based on abstract models, are discussed. This paper proposes a dynamic DP algorithm, which will not only benefit the SP profit but also minimize CU costs. This model-free approach was performed using RL, where retail prices will be adaptive in nature and change based on the investigation of CUs' and dissatisfaction level and demand profile. This dynamic pricing problem of electricity is approached by converting it to a finite discrete MDP and then formulating the decision-making using \(Q\)-learning. As a result, an SP does not need any specific model of the customers' energy consumption to learn about future retail rates. Rather, it is more convenient as the SPs will learn about CUs' satisfaction and dissatisfaction levels through dynamic online interaction. This approach is feasible as, with the change of environment, which in this case is the variation of CUs and their demand profile, the dynamic pricing model will improvise through learning.