## 4 Proposed Q-Learning Method for Smart Grid Price Forecasting  
### Problem Formulation  
#### 4.1.1 Customer Model  
There are two types of energy load profiles that a customer can have. One is the critical load, and the other is the curtailable load. These are classified based on their energy requirement and priorities. Critical load is the load that is of high priority and must be met at any cost for customer satisfaction and priority. An example can be use of power in data centers and power stations. The equation that satisfies the critical load demand is denoted as follows:  
\[ec_{l,c}^{\text{critic}}=ed_{l,c}^{\text{critic}} \tag{1}\]  
where \(t\) is divided into 24 h segments, which represent each hour of the day. The price will be updated every 24 h. \(c\in\{1,2,3\dots C\}\) represents the customers \(c\). \(ed_{l,c}\) refers to the energy demand, and \(ec_{l,c}\) refers to the energy consumption of a customer \(c\) at a specific time \(t\). Curtailable load, on the other hand, is more flexible with price. The demand of customers for curtailable load can change with price as the demand falls with the increase in electricity price. For a certain customer CU \(c\), consuming a certain amount of \(ec_{l,c}\) at time \(t\) will correspond to the customers' amount of load satisfaction. Subtracting that load satisfaction from the total amount of energy demand gives us the dissatisfaction level at time \(t\), which is \(ed_{l,c}-ec_{l,c}\). This dissatisfaction level is denoted as \(\phi_{l,c}\). This signifies the degree of dissatisfaction that customers can experience when prompted to reduce their electricity demand due to high prices. This co-efficient is convex in nature and tends to increase massively if energy reduces significantly. The equation that satisfies the critical load demand is denoted as follows:  
\[ \begin{array}{l}ec_{l,c}^{\text{curt}}=ed_{l,c}^{\text{curt}}\ \cdot\left(1+\xi_{l}\cdot\frac{\lambda_{l,c}-\pi_{l}}{\pi_{l}}\right)\\ \xi_{l}<0\\ \lambda_{l,c}\geqslant\pi_{l}\end{array}  \tag{2}\]  
Here, \(\xi_{l},\lambda_{l,c},\pi_{l}\) denote elasticity coefficient, retail price for customer \(c\) and wholesale price at time \(t\), respectively. In economic terms, elasticity \(\xi_{l}\) measures responsiveness of the change between two variables with respect to one another. Price elasticity of demand can show how the change in product price can impact the energy demand of a particular good. In the context of smart grids, this elasticity refers to the change in the demand for electricity with the 1% increase in price of that particular time. Thus, this elasticity between the demand and price is inversely proportional. Research regarding elasticity on smart grid energy concludes that the demand for electricity is most elastic during peak hours, and long-run elasticity results are better compared to the short one. The elasticity values used in this paper to study the RL implementation were acquired from published papers. The dissatisfaction cost function can be defined as follows:  
\[ \begin{array}{l}\varphi_{l,c}=\frac{\varphi_{c}}{2}\left(ed_{l,c}^{\text{ curt}}\ -ec_{l,c}^{\text{curt}}\ \right)^{2}+\beta_{c}\left(ed_{l,c}^{\text{curt}}\ -ec_{l,c}^{\text{curt}}\ \right)\\ \alpha_{c}>0\\ \beta_{c}>0\\ D_{\min}<ed_{l,c}^{\text{curt}}\ -ec_{l,c}^{\text{curt}}\ <D_{\max}\end{array}  \tag{3}\]  
\(an\) and \(\beta n\) are parameters varying from customer to customer. The former is the response to a customer's consumable energy reduction, where a higher value denotes that a customer is likely to become more dissatisfied if the electricity prices become lower. The latter parameter is a predetermined constant. \(D_{\min}\) and \(D_{\max}\) are the lowest and highest energy reduction, respectively.