## 3 Results Discussion
### Reinforced Learning Controller  
The RL was applied to both cases of FC introduced above by means of the modified Q-learning algorithm described in Section 2.1. The RL and FC parts were connected, as described in Section 2.3, because the learning agent (controller) can choose which membership function to employ in the FC. This resulted in the selection of one out of seven characteristic functions (shown in Figure 5 left or right) to be used in the real-time control for the entire period \(\Delta t\) between two iterations of the Q-learning algorithm.  
The definition of the state-space variables for the application of RL techniques was performed, trying to minimize the number of states, and therefore, the number of variables involved. Two parameters were identified as essential to describe the system dynamics: the solar radiation at the collectors and the temperature of the storage. Since no reference was found in literature covering the discretization of the state-space in such an application, the following arbitrary choices were made:  
* The average temperature of the storage was modelled with nine intervals of 10 \({}^{\circ}\)C each from 0 to 90 \({}^{\circ}\)C;
* The radiation on the collectors' plane was modelled with 11 intervals of 100 W/m\({}^{2}\) each from 0 to 1100 W/m\({}^{2}\).  
The Cartesian product of these two grids represents the Q-learning algorithm state-space. The elements of such space were ordered and rearranged as a vector to be addressed linearly. This vector had a total of \(p=|S|=99\) components. The size of the action space was \(l=|A|=7\), as there were seven curves in each FC family to be selected from. The period of the Q-learning iteration \(\Delta t\) was set to 5 min down from the initial 15 min which were initially considered in order to reduce the algorithm convergence time. The parameters affecting the implemented Q-learning algorithm are summarized in the following Table 5.  
The objective of this work was to compare the performance of the coupled RL and FC controller with the reference cases. In particular, the SPFcoll indicator was considered because it is directly influenced by the variables selected to construct the Q-learning state space. The SPFdhw is somewhat more relevant from the economic point of view but it is influenced by a variable (the DHW draws) which is not in the selected state-space, and therefore, the algorithm cannot learn anything about it.  
The reward function was defined assuming that rewards proportional to the current value of the quantity to maximise (SPFcoll) results in the actual maximization of that quantity. A preliminary analysis used a reward equal to SPFcoll measured on the \(\Delta t\) interval, but simulations showed that the learning was not effective, and the best performance of the reference cases could not be achieved. The following additive form of the SPFcoll has proven to give better results than using directly its fractional form (10):  
\[r=\frac{w_{1}\int_{\Delta t}\dot{Q_{coll}}\mathrm{d}t-w_{2}\int_{\Delta t}W_{ pump}\mathrm{d}t}{\int_{\Delta t}I_{G}A_{coll}\mathrm{d}t}[-] \tag{14}\]  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Parameter** & **Unit** & **Value** \\ \hline \(p=|S|\) & - & 99 \\ \(l=|A|\) & - & 7 \\ \(N_{e,A}\) & - & 10 \\ \(\alpha\) & - & 0.5 \\ \(\gamma\) & - & 0.5 \\ Iterative cycle - \(\Delta t\) & min & 5 \\ \(w_{1}\) & - & 1 \\ \(w_{2}\) & - & 370 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Parameters of the Q-learning algorithm used to produce the data (where not otherwise indicated).