## 3 Proposed RL-HCPV
```
Initialize using \(\hat{pr}_{pq^{\prime}}\), \(\hat{P}_{comp}\), \(\hat{SOC}_{EV}\), \(\hat{loc}_{EV}\), \(\hat{r}\hat{tp}_{app}\), \(\hat{P}_{hour}\), \(time\) for time slot t = 1:T do for HEMS do Execute LSTM for PV Generation End Receive extracted information about future PV Generation Execute ANN for Appliance Consumption End Execute ANN for Appliance Consumption Execute Multi-target regression for EV availability and SOC End Receive extracted information about future EV patterns Execute ANN for pricing End Receive extracted information about future IoU and RTP for each t do Sort descending by T end for Execute Algorithm 2 End End end for end for
```  
**Algorithm 1** Proposed RL-HCPV Algorithm  
```
Initialize Environment ActionSet = [], S = [] Get current state St, ActionSet Initialize Fuzzy Set St, \(u=[]\) Initialize FIS StateSpace Set Member ship function Initialize FQL model St, y,, ee-rate, ActionSet, FIS Set \(x=0\), \(V=0\), \(Error=0\) Set Q-table = matrix([index]) for each episode \(\theta\)do Select Action \(\alpha\) Set Inferred Action \(\alpha\) t if\(\alpha\) t \(\in u[]\)then Reward 1 else Reward 0 end if Receive immediate reward \(r_{t}\) Calculate TruthValue, V, Q, Quality return Q \((S_{t},\alpha_{t})\) Update Q-Value return Q\((S_{(t+1)},\alpha_{(t+1)})\) Keep State History endfor
```  
**Algorithm 2** Fuzzy Q-Learning (Decision Making)  
```
Initialize Environment ActionSet = [], S = [] Get current state St, ActionSet Initialize Fuzzy Set St, \(u=[]\) Initialize FIS StateSpace Set Member ship function Initialize FQL model St, y,, ee-rate, ActionSet, FIS Set \(x=0\), \(V=0\), \(Error=0\) Set Q-table = matrix([index]) for each episode \(\theta\)do Select Action \(\alpha\) Set Inferred Action \(\alpha\) t if\(\alpha\) t \(\in u[]\)then Reward 1 else Reward 0 end if Receive immediate reward \(r_{t}\) Calculate TruthValue, V, Q, Quality return Q \((S_{t},\alpha_{t})\) Update Q-Value return Q\((S_{(t+1)},\alpha_{(t+1)})\) Keep State History endfor
```  
**Algorithm 3** Proposed RL-HCPV Algorithm  
```
Initialize Environment ActionSet = [], S = [] Get current state St, ActionSet Initialize Fuzzy Set St, \(u=[]\) Initialize FIS StateSpace Set Member ship function Initialize FQL model St, y,, ee-rate, ActionSet, FIS Set \(x



### Prediction of Uncertainty Parameters of HEMS

A. DL algorithms for the uncertainty prediction

Applying DLs to predicting PV power generation helps to overcome the limitations of statistical approaches in handling non-linear data caused by weather variations. DL is an effective prediction approach with many applications, such as appliance consumption, weather conditions, and wind speed prediction [31; 46]. The ANN algorithm has become a prominent tool for predicting renewable power generation [60], electricity price [61], and load [62]. ANN is one of the easiest ways to set up building models, even with only minimal physics knowledge. ANNs are constructed by training, validating, and testing datasets. The input, hidden, and output layers are the three primary components of ANNs. The input layer receives the input data, and the hidden layer, that consists of numerous layers, examines the input data. Following the acquisition of analysed information from the hidden layer, the output layer provides the output. Figure 3: FQL for the RL-HCPV system. ### Uncertainty management strategy for RL-HCPV

Every hour, the inputs of the historical data of PV generations, appliance consumption, EV SOC, location, and electricity price data are used for the DL algorithms. The outputs are predicted for 24 h. The projected information will then be placed into the decision-making process, fuzzy Q-learning for efficient power decision-making. Datasets:

This study employs five datasets for the uncertainty prediction. 1. An appliance consumption dataset [63]: A sample smart meter readings dataset was obtained from the Saudi Electricity Company (SEC). The data were collected for each appliance, as shown in Figure 4c during a 30-min time duration;
2.