## 4 Proposed Q-Learning Method for Smart Grid Price Forecasting
### Q-Learning-Based Electricity Price Forecasting
#### 4.2.3 Using \(Q\)-Learning for Dynamic Pricing Problem
The proposed \(Q\)-learning Algorithm 1 is given below:```
S = State, A = Action Initialize Q(S, A) using arbitrary values for each iteration \(i\),do  At time interval \(t\) Choose an action \(\lambda_{t,c}\) and execute for state \((ed_{t,c})\)  Observe the reward \(r(ec_{t,c}|ed_{t,c},\lambda_{t,c})\) and the new state \((ed_{t+1,c})\) \(Q(ec_{t,c}|ed_{t,c},\lambda_{t,c})\leftarrow\)\(Q(ec_{t,c}|ed_{t,c}\lambda_{t,c})+\theta\cdot[r(ec_{t,c}|ed_{t,c}\lambda_{t,c})+\) \(\gamma\cdot\max Q(ed_{t+1,c}|ed_{t+1,c},\lambda_{t+1,c})-Q(ed_{t,c}|ed_{t,c} \lambda_{t,c})]\)  end for
```

**Algorithm 1**Proposed \(Q\)-learning algorithm

Figure 4: Flowchart for implementing the \(Q\)-learning process for figuring out the optimal price. ## 5 Implementation and Analysis

### Implementation of Blockchain

In this section, an Ethereum-based private blockchain system is introduced that ensures a safe and secure electricity marketplace between the CU and SP. Ethereum provides an open-source blockchain platform that runs based on smart contracts, and the currency it runs with is 'Ether'. It is mainly a DPL that can record, verify, and deploy transactions that occur within the network. A 'DApp' [26] or decentralized application is introduced to complete the transactions, and the application users pay a particular fee that is 'gas' to run the functions, which depends on the amount of computational power needed to compute the whole process. It works on the 'Proof of Work' consensus mechanism. There is a canonical computer in the Ethereum technology (EVM) whose state is followed by the other nodes. The main smart contract is stored in the EVM, and users call out programs into the EVM storage using particular parameters to perform actions. In this implementation, a complete business DApp is demonstrated. In this system, the smart contract was created and deployed in the blockchain system as well as the front end, and the transactions were confirmed and stored in the updated blockchain network. Here, we present the work plan for the implementation system. The basic components of the implemented system and the way they intercommunicate are shown in Figure 5.

### Experimental Setup

#### 5.2.1 Blockchain Network

The core network of the system is Ganache, which is a personal or private blockchain for DApp development [27]. Its primary purpose is to design, launch, and test the DApp securely and easily without costing any real Ether. It provides a local dummy server that holds ten (10) accounts containing a balance of 100 fake Ether to perform the transactions. Each account holds a public key and a private key that is used to conduct the transactions. The server only accepts the RPC connection on host [[http://127.0.0.1:7545](http://127.0.0.1:7545)]([http://127.0.0.1:7545](http://127.0.0.1:7545)) (accessed on 24 January 2022). In Figure 6, 10 fake accounts are displayed, where each has a balance of 100 Ether. Using the private key, the system can connect that particular account with the browser to perform transactions. It shows the visual interface for Ganache CLI, where functions and features are displayed to the users. Figure 5: Workplan of the blockchain implementation. #### 5.2.2 Smart Contract Deploy

The system uses Solidity language to write the smart contract, and the version of the Solidity language is >=0.4.21 <0.6.0. It holds the main functions and events of the whole system. As the smart contract contains the main structure of the performed actions, it is necessary to provide accurate and realistic parameters to meet the particular requirements. After deploying the smart contract, it will hold an address of its own that must be provided to the blockchain while connecting them. In our implementation, a smart contract named 'Marketplace.sol' is initiated that holds the records of the id, name, price, owner's address, and purchase history. It also consists of several events that can run programs and store variables. The functions are createElectricity (name,price) and purchaseElectricity (id).