## 2 Methods and Performance Figures
### Q-Learning with Guided Exploration  
The application of the standard \(\varepsilon\)-greedy Q-learning algorithm to the test case described in Section 2.4 required modifications to overcome a problem during the explorative iterations. Being that the number of explorative actions was relatively low and the number of trials quite small in the present application, there was no guarantee that the pseudo-random generator would distribute the various actions evenly across the action space within a reasonable time. This can lead to runs where some points of the state-action matrix are not visited for long times. In order to overcome this limitation, a guided exploration algorithm was implemented in place of the random \(\varepsilon\)-greedy approach. The idea is to explore, at each step, the action whose outcome is less known until a sufficient number of explorations has been performed. To implement this idea, a second matrix \(\mathbf{K}\) was introduced. \(\mathbf{K}\). has the same dimensions as \(\mathbf{Q}\) (\(p\) rows and \(l\) columns) and contains the number of times each state-action pair has been visited up to the current time-step. Given a certain state, the new action is selected evaluating the minimum of the matrix \(\mathbf{K}\) along the row corresponding to that state, until a minimum number of trials per action (\(N_{e,a}\)) is performed. In this way, the exploration of each action is ensured in a minimal number of trials. Moreover, if some states are visited more frequently than others, they reach convergence before those visited less frequently. This means that after a state reaches convergence, in that state the following actions are chosen purely based on the Q-value. In this paper, this is called a local convergence criterion because the transition from the exploration to the exploitation phase was evaluated for each state independently. To understand how this was achieved in detail, the vector \(\mathbf{c}\) is introduced. For every state \(s\in S\), \(c_{s}\) is defined to be 1 when the sum of the numbers in the row of \(\mathbf{K}\) referring to state \(s\) is greater than \(N_{e}=N_{e,a}l\) and 0 otherwise. This is concisely expressed by Equation (3). \[\left\{ \begin{array}{l}c_{i}=1\Leftrightarrow\sum\limits_{j=1}^{l}\mathbf{K}_{i, j}\geq N_{e}\\ c_{i}=0\Leftrightarrow\sum\limits_{j=1}^{l}\mathbf{K}_{i,j}<N_{e}\end{array} \right. \tag{3}\]  
The action to be performed at time \(t\) is then selected by means of Equation (4) instead of Equation (2). \[a_{t}=\left\{ \begin{array}{l}\underset{a}{\operatorname{argmax}}\mathbf{Q}_{t} (s_{t},a)\quad\text{if }c_{s_{t}}=1\\ \underset{a}{\operatorname{argmin}}\mathbf{K}_{t}(s_{t},a)\quad\text{if }c_{s_{t}}=0\end{array} \right. \tag{4}\]In either case, matrix \(K\) is updated by incrementing the entry referring to the current state \(s_{t}\) and the so selected action \(a_{t}\), like in Equation (5),  
\[K_{t+1}(s_{t},a_{t})=K_{t}(s_{t},a_{t})+1 \tag{5}\]  
while matrix \(Q\) is update with Equation (1) as usual. From how the \(K\) matrix is updated, it follows that \(c_{s}\) is equal to one if and only if the state \(s\) has been fully explored, that is all the actions in state \(s\) have been tried out at least an \(N_{e,a}\) number of times. The vector \(c\) is used to define a convergence criterion for the RL algorithm. Considering that \(0<\sum_{i=1}^{p}c_{i}<p\) always holds (with \(\sum_{i=1}^{p}c_{i}=0\) at the beginning of the exploration phase and \(\sum_{i=1}^{p}c_{i}=p\) at the end), the _exploration factor_ can be defined (Equation (6)). \[\xi=\frac{\sum_{i=1}^{p}c_{i}}{p} \tag{6}\]  
This parameter indicates the percentage of the matrix explored, and consequently the progress of the learning process toward its end. The RL algorithm is also beneficial for tracking an evolution in the behaviour of the system (or more generally the environment), e.g., due to ageing of components or to the time variance of its boundary conditions. In these circumstances, the agent should be able to recognize the new distribution of the optimal actions to do.