## 3 Proposed RL-HCPV
\[Agent.Error=r_{t}+((Agent.\gamma*Agent.V)-Agent.Q) \tag{34}\]

Figure 5 depicts the technique of employing fuzzy Q-learning in the RL-HCPV algorithm. Generally, Figure 3 illustrates how fuzzy logic decides on a particular state and reward. The parameters \(\gamma\) and \(\alpha\) are set at 0.9 and 0.1, respectively, and the entries in the Q-value table are set to zeros. All possible actions are listed for each present state, and then an action is randomly selected. When the stated action is finished, the agent will use fuzzy logic in this step to detect the new state and the reward. Equation (33) will calculate the maximum Q-value before updating the state-action Q-value pairs. Finally, the present state will then be the following state. ## 4 Results and Discussion

### Implementing the Proposed Model for the RL-HCPV

The proposed RL-HCPV method is based on the interactions between PV power generation, appliance power consumption, electricity pricing, V2H unit, EV SOC, and availability. A smart meter uses the smart house to gather power data, such as PV power generation, appliance consumption, \(SOC_{SB}\), and energy price signals from the utility, which are then sent to the HEMS. When the EV returns home, HEMS will receive a signal indicating EV availability and the SOC percentage. The data collected help to determine

Figure 5: Flowchart describing the proposed optimal decision-making method. whether the PV-generated power and \(SOC_{SB}\) are sufficient to meet the demand. If the EV can transmit electricity to the home via V2H, HEMS may be the best alternative in a power outage. The simulation runs for 24 h, using an hour-long time step. First, the power utility provides the price tariff signal. Then, the user is expected to depart at 7:00 a.m. and return at 2:00 p.m. There are two peak times: 2:00-5:00 p.m., when the household member returns from work, and 7:00-9:00 p.m., when they begin cooking, watching TV, and other activities. On sunny and cloudy days, the off-peak time begins at midnight and lasts until early morning. The suggested approach sought to understand how using DL, RL, and V2H affected power consumption efficiency while reducing expenses. The two smart MG design scenarios emphasize the necessity of including V2H as a different energy source in the RL-HCPV model. The simulation was tested on a Windows 11 PC with an AMD Ryzen CPU clock speed of 1.80 GHz and 16 GB RAM. The simulation software used was the Real-time Simulink tool from MATLAB R2022a, while the programming software was Python. #### 3.2.1 Case1:

Smart MG without V2H unit

Depending on the relationship between the overall power usage, the price of electricity, and the Q-values, the standard SG, in this case, comprises PV solar panels, SB, and appliances. All of the executed activities are depicted in Figure 6a. For example, during 5:00-7:00 p.m., the total energy consumption is low (3-5.7 kWh), the ToU energy price is 0.24 $/kW, RTP is 0.21 $/kW, and PV generation is low (5 kWh). As a result, the current state is \(s_{t}\) = low; low; expensive; expensive. According to the Q-value, the RTP action has the highest Q-value for this state. Later on after 7:00 a.m., the system switches to another state, since demand exceeds (16 kWh), prices are (ToU 0.28 $/kWh and RTP 0.27 $/kWh), and PV generation is high (10 kWh). The current state is \(s_{t}\) = high; high; expensive; expensive. The PV discharging action has the highest Q-value for this condition according to the Q-value. Furthermore, the PV generation is low, but the SB saves the excess power during sunny days (over 10 kWh). Thus, the current state is \(s_{t}\) = low; high; expensive; expensive. The SB discharge mode provides energy to appliances while lowering power costs. Finally, from 9:00 p.m.-04:00 a.m., consumption is 0.95 kWh, PV generation is 0.3 kWh, and SB power is 3.5 kW. The current states are all low. Thus, ToU (0.21 $/kWh) will be activated, which is cheaper compared to RTP (0.27 $/kWh).