## 3 Results Discussion
### Coupled RL and FC Simulations Outcomes  
The results of two simulations of the coupled control algorithm spanning 5 years are shown in Table 6. The first five rows refer to the H parametrization, while the successive five rows refer to the M parametrization. In either case, after the second year, the SPF\({}_{\text{coll}}\) value gets close to the maximum attained by the reference cases (Tables 2 and 3). Figure 9 shows a clear difference between the first and the second year of the monthly profile of the performance indicator for the five years of learning (reported also in the first five rows of Table 6), while in the following years, the values are close to each other. Most of the learning phase happens during the first seven months of the first year. This was confirmed by the exploration factor (\(\xi\)) reported in Figure 10. It becomes apparent why the performance profile did not change significantly after the first year: the maximum of the exploration factor was reached within this time (in fact the first five months), meaning that what the Q-learning had to learn about the system, it had learnt it within this time. Minor increments in the following seven months were due to states visited rarely during the normal operation of the system. The reader may wonder why the exploration factor did not reach 100% after enough time. This was unavoidable and happens because some states (for example, a temperature of the storage less than 10 \({}^{\circ}\)C with a level of radiation higher that 900 W/m\({}^{2}\)) were not reachable, and therefore, never explored. \begin{table}
\begin{tabular}{c|c|c|c|c|c|c|c|c|c|c} \hline \hline \multirow{2}{*}{**Case**} & \multirow{2}{*}{**Year**} & \multirow{2}{*}{\(\mathbf{\eta_{coll}}\)} & \multirow{2}{*}{**SP\({}_{\text{coll}}\)**} & \multicolumn{2}{c|}{**On/Off**} & \multirow{2}{*}{**GSY**} & \multirow{2}{*}{**R\({}_{\text{G}}\)**} & \multirow{2}{*}{**W\({}_{\text{pump}}\)**} & \multirow{2}{*}{**D\({}_{\text{dhw}}\)**} & \multirow{2}{*}{**SP\({}_{\text{dhw}}\)**} & \multirow{2}{*}{**W\({}_{\text{backup}}\)**} & \multirow{2}{*}{**Q\({}_{\text{loss}}\)**} \\  & & **(-)** & **(-)** & **(-)** & **(kWh/m\({}^{2}\))** & & & & & \\ \hline \multirow{8}{*}{**H-FC**} & 1 st & 0.397 & 370 & 446 & 582 & & 19.1 & & 4.969 & 1523 & 225 \\  & 2 nd & 0.396 & 409 & 425 & 581 & & 17.3 & & 4.936 & 1535 & 217 \\  & 3 rd & 0.396 & 408 & 422 & 581 & 1467 & 17.3 & 7660 & 4.930 & 1537 & 216 \\  & 4 th & 0.396 & 410 & 420 & 581 & & 17.2 & & 4.928 & 1537 & 216 \\  & 5 th & 0.396 & 411 & 422 & 581 & & 17.2 & & 4.924 & 1539 & 215 \\ \hline \multirow{8}{*}{**M-FC**} & 1 st & 0.385 & 611 & 437 & 565 & & 11.2 & & 4.377 & 1739 & 114 \\  & 2 nd & 0.382 & 680 & 414 & 561 & & 10.0 & & 4.285 & 1778 & 104 \\  & 3 rd & 0.382 & 685 & 409 & 560 & 1467 & 9.9 & 7660 & 4.266 & 1786 & 102 \\  & 4 th & 0.382 & 682 & 415 & 560 & & 10.0 & & 4.263 & 1787 & 104 \\  & 5 th & 0.382 & 687 & 416 & 560 & & 9.9 & & 4.278 & 1781 & 102 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Q-learning applied to the H- and M-FCs. Figure 9: Monthly profile of the SPF\({}_{\text{coll}}\) with Q-learning coupled to the H-FC, comparing the monthly performance of the different year during the learning phase. Other tests have been done by varying physical parameters of the solar system, i.e., the efficiency of the collectors and the electrical consumption of the pump, within 75% and 125% of their original value resulting in changes too small to change the optimal control policy. In the rest of this section, the partial reset of the stored information, an essential feature to handle slowly time-variant systems, is discussed. To illustrate the behaviour of the Q-learning algorithm, an artificial use case was constructed. A baseline was created by letting the Q-learning controller explore and exploit the system for 5 years. After this period, a sudden and structural change of the system was simulated by reversing the definition of the agent actions. This modified system was simulated for another period of 5 years.