## 4 Proposed Q-Learning Method for Smart Grid Price Forecasting
### Q-Learning-Based Electricity Price Forecasting
#### 4.2.3 Using \(Q\)-Learning for Dynamic Pricing Problem  
\(Q\)-learning is a subsection of RL that is model-free. It can be used to obtain the optimal policy, which is the dynamic policy referred to in this paper. The \(Q\)-learning algorithm is as follows:The main \(Q\)-learning process involves a \(Q\)-value \(Q(ec_{t,c}|ed_{t,c},\lambda_{t,c})\), which is assigned to every state-action pair at a time slot \(t\). Then, it is updated over each episode that promotes good behavior. \(Q^{*}(ec_{t,c}|ed_{t,c},\lambda_{t,c})\) refers to the maximum discounted reward for the future when taking the \(\gamma_{t,c}\) action. This optimal \(Q\) value can be referred to below:  
\[Q^{*}(ec_{t,c}|ed_{t,c},\lambda_{t,c})=r(ec_{t,c}|ed_{t,c},\lambda_{t,c})\quad+ \gamma\cdot\max Q(ec_{t+1,c}|ed_{t+1,c},\lambda_{t+1,c}) \tag{10}\]  
According to the algorithm, the maximum \(Q\)-value is calculated. If the learning rate factor is 0, then the agent learns nothing from the existing data, and in this case, it is the SP who gains no new information regarding the optimal policy. On the other hand, a learning factor of 1 will make the agent reevaluate the recent state of the environment; in this case, it is considering the recent demand response of the customers. In the dynamic electricity market, the service provider interacts with the customer through their dynamic pricing. Then, the demand for CUs change overtime and the SPs receive a new state. The trial and error of these set of actions, which is the dynamic pricing implemented by the SPs, generates \(Q\)-values that are stored and updated from time to time. Eventually, the values are going to converge to a maximum value. If the maximum expected profit of SPs with action \(\lambda_{t,c}\) is at a demand \(ed_{t,c}\), the optimal policy \(v\) can be referred as:  
\[C=\operatorname{argmax}Q(ec_{t_{h}D}|ed_{t_{h}D},\lambda_{t,0}) \tag{11}\]  
Figure 4 shows the flowchart of how the algorithm works:  
Since RL excels at making sequential decisions in an unknown environment, it can adapt to the policy that is required in real-time and learns from past experiences. Therefore, here, \(Q\)-learning is the best RL-method to find the optimal pricing strategy. Here, the flowchart shows that the algorithm begins at 00:00 and ends after 24 h. The inputs that are taken are the prices from producers following the time slot T in an hourly fashion. There are also the coefficients of the price bounds from the third-party providers and all other parameters in the flowchart shown above. After the algorithm runs the inputs, it initializes the \(Q\)-value \(Q(ec|ed)\) to 0, the time to the beginning of the day. The algorithm then finds the optimal prices at each hour of the day using the epsilon-greedy policy, abiding by the price bounds. To be efficient, the epsilon-greedy policy selects an action with uniform distribution from a set of available actions. Using this policy, a random action with a probability \(\epsilon\) that is between 0 and 1 from the 1-values from each state is performed during the iteration. In short, the agent here randomly selects a retail price at each state and scans them amongst the already stored values from the previous states to select the highest value and then choose the retail price. After the SP chooses the retail price, it will receive rewards accordingly from the reward equation. At the same time, the SP also observes the customer's demands for the following time slot and updates the \(Q\)-values using the \(Q\)-learning process. Lastly, if the \(Q\)-value does not converge to the maximum \(Q\)-value, the system then moves on to the next iteration until it finally does converge. This is the termination rule, where the difference between the present and previous \(Q\)-value is less than \(\delta\). After converging, the SP will then obtain the final responsive retail prices for the hourly slots of the day.