## 4 Simulation Results and Discussion
### Influence of the Reward Window  
Several of the reward strategies calculate the reward applying a time window, that is, considering N previous samples of the error signal, specifically: MSE-RS, MEAN-RS, \(\Delta\)MSE-RS, \(\Delta\)MEAN-RS. To evaluate the influence of the size of this window, several experiments have been carried out varying this parameter. In all of them, the policy update algorithm has been SAR. Figure 10 (left) shows the results when \(\Delta\)MSE-RS is applied and Figure 10 (right) for \(\Delta\)MEAN-RS. Each line is associated with a different window size and is represented in a different color. The reward is a dimensionless parameter as table \(T^{\pi}_{(s_{t},\,a_{t})}\) is dimensionless. The legend shows the size of the window in seconds. The value \(-1\) indicates that all the previous values, from instant 0 of the simulation, have been taken into account to obtain the reward. That is, the size of the window is variable and increases in each control period, and covers from the start of the simulation to the current moment. MSE-RS and MEAN-RS have not been included as, as explained, they are O-P reward strategies and do not converge without forced exploration. Figure 9: Evolution of the MSE (**left**) and error centroid radius (**right**) for different reward strategies and update policy Q-learning (QL). In general, a small window size results in a faster convergence to the minimum, but if the size is too small it can cause oscillations after the absolute minimum. This happens with a window of 0.01s, the MSE oscillates and is even less stable for \(\Delta\)MEAN-RS. A small window size produces noisy rewards. This parameter seems to be related to the control period; a size smaller than the control period produces oscillations. For the \(\Delta\)MSE-RS strategy, the convergence speed decreases with the size of the window up to 1 s. For smaller window sizes it does not converge. This can be explained as if the window is longer than the control period, the window can be divided into two parts: the value of a control period preceding the end of the \(Tw_{2}\) window, and the remaining part from the beginning of the \(Tw_{1}\) window (Figure 11). An action performed at \(t_{i-1}\) produces an effect that is evaluated when the reward is calculated at \(t_{i}\). When the size of the window grows during the control period, the \(Tw_{1}\) part also grows, but \(Tw_{2}\) remains invariant. To produce positive rewards, it is necessary to reduce the MSE, therefore, during \(Tw_{2}\), the increases in \(Tw_{1}\) should be compensated. A larger \(Tw_{1}\) would give a larger accumulated error in this part, which would be more difficult to compensate during \(Tw_{2}\) since only the squared error can be positive. It can then be concluded that the optimal window size for \(\Delta\)MSE-RS is the control period, in this case, 100 ms. The behavior of \(\Delta\)MEAN-RS with respect to the size of the window is similar up to a size of around 20 s; from this value increasing the window size accelerates the convergence and decreases the MSE. This is because a larger window size implies a longer \(Tw_{1}\) part (Figure 11). However, unlike \(\Delta\)MSE-RS, a longer \(Tw_{1}\) produces less accumulated error in \(Tw_{1}\) since, in this case, the positive errors compensate for the negative ones, and the accumulated error tends to 0. Therefore, \(Tw_{2}\) has a greater influence on the window, and learning is faster. Figure 12 shows the variation of the MSE with the size of the reward window, at iteration 5. It is possible to observe how the MSE grows until the size of the window is 1 s, decreases until a size around 10 s, and then it grows again until around 25 s, which continues to grow for \(\Delta\)MSE-RS and decreases for \(\Delta\)MEAN-RS. The numerical values of these local minima and maxima are related to the duration of  
Figure 11: Reward time window regarding the control period. Figure 10: Evolution of the MSE for \(\Delta\)MSE-RS (**left**) and \(\Delta\)MEAN-RS (**right**) and different temporal window.