## 3 Proposed RL-HCPV
The model results displayed the accuracy of MAE at 0.37 and MSE at 0.27. Multi-target regression is advantageous for EV SOC and availability because it predicts several continuous target variables concurrently based on the same input dataset. Two target columns were considered; EV arrival location and SOC at every time interval (one hour). The model displayed the accuracy of MAE at 0.31 and MSE at 0.15. Finally, ANN modelling was applied for ToU and RTP tariffs. The observations, including the trained dataset prices, were regarded as the inputs for the neural networks at every time interval (one-hour). The input layer had 15 neurons for the prices training dataset. Three hidden layers were included, 32, 18, and 9, respectively, to obtain the output layer for the predicted prices. The model displayed the accuracy of MAE at 0.37 and MSE at 0.32. ### RL for Decision-Making in HEMS

Fuzzy Q-Learning Overview

The fuzzy Q-learning algorithm is a fuzzy variation of the Q-learning method. The input states and their related fuzzy sets must be identified before building an FIS to combine with the Q-learning method [67]. As a new ML method, the fuzzy Q-learning algorithm [67] is frequently employed in decision-making to maximize cumulative rewards. This algorithm's primary mechanism is to establish a Q-table in which the Q-value, Q(\(S_{t}\),\(\alpha_{t}\)) in each iteration is modified using Equation (31) until the convergence condition is met. \[Q(s_{t},\alpha_{t})\leftarrow(1-\alpha)Q(s_{t},\alpha_{t})+\alpha[r(s_{t}, \alpha_{t})+\gamma.maxQ(s_{t+1},\alpha_{t+1})] \tag{31}\]

The discount rate \(\gamma\) defines the relationship between future and present rewards. It accepts a value between 0 and 1, where 0 indicates that the agent depends solely on the present reward, whereas 1 implies that the agent will compete for future rewards. While \(\in(0\leq\epsilon\leq 1)\) represents the learning rate and indicates how much the new reward influences the previous value of Q(\(S_{t}\),\(\alpha_{t}\)). For example, \(\epsilon=0\) indicates that the information currently collected is negligible and the received reward will not affect the Q-value. Whereas \(\epsilon=1\) means that only the most recent information is considered. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Uncertainty Parameters** & **DL Algorithm** & **Input** & **Output** \\ \hline PV Generation & LSTM & PV solar generation & Predicted PV solar \\  & LSTM & (training dataset) & generation \\ Appliance consumption & ANN & Appliance consumption & Predicted appliance \\  & ANN & (training dataset) & consumption \\ EV SOC and availability & Multi-target regression & Historical data (SOC-end & Predicted SOC and \\  & Multi-target regression & and End-location) & End-location = 1 \\ ToU tariffs & ANN & ToU Price (training dataset) & Predicted ToU price \\  & ANN & Startup price, intermediate & Predicted startup price, \\ RTP tariffs & ANN & price, and no load price (training dataset for each & intermediate price, and no \\  & & column separately) & load price \\ \hline \hline \end{tabular}
\end{table} 
Table 4: DL algorithms for each uncertainty parameter. ### Fuzzy Q-Learning model for decision making in the RL-HCPV

Under ToU and RTP dynamic power pricing and varied energy consumption patterns, RL produced a real-time smart energy scheduling option for a smart home. The RL process is defined by a state space and a numerical reward given to the agent after performing a specific action to assess the new state. The fuzzy Q-learning in our proposed model consists of state-space and reward function, as follows:

* State-space implementation using fuzzy logic

The state space here was represented by 16 states, as displayed in Table 4. PV power generation and appliance consumption were classified as low or high levels to simplify the model and decrease calculation time. The SB and EV battery SOC was defined as efficient or not efficient, while the EV availability was defined as available or unavailable. In addition, the weather was either cloudy or sunny.