## 4 Simulation Results and Discussion
### Combination of Individual Reward Strategies  
As discussed above, O-P reward strategies do not converge due to the lack of exploration of the entire space of possible actions. To solve this, \(\epsilon\)-greedy methods can be applied [40] or they can be combined with P-N reward strategies. This last option is explored in this section. Different experiments have been carried out combining reward strategies O-P with P-N and their performance is studied. In all experiments, the policy update algorithm is SAR and the window size is 100 ms.  
Figure 14: Evolution of the MSE (**left**) and variance (**right**) for different reward sizes when \(\Delta\)MSE-RS is applied. Figure 15: Evolution of the MSE (**left**) and variance (**right**) for different reward sizes when \(\Delta\)Mean-RS is applied. Figure 16 shows the results of applying PRS (blue) and VRS (red), PRS-VRS (yellow) and PRS+K-VRS (purple), with K = 2. It is possible to see how the multiplication makes PRS converge. However, this is not true with addition, as this operator cannot convert PRS to a P-N reward strategy in all cases. It depends on the size of each individual reward and on the value of K. Therefore, the addition operator will not be used from now on to combine the rewards. Another interesting result is that the PRS-VRS combination smoothens the VRS curve, the result converges at a slightly slower speed but is more stable for iterations over 15. This combination presents less variance than VRS in general. Figure 17 shows the results of MSE-RS (blue), MEAN-RS (red), MSE-RS-VRS (yellow), Mean-RS -VRS (purple), and VRS (green). All strategies combined converge at roughly the same speed, slightly slower than VRS. During iteration 10, the performance of VRS and MSE-RS-VRS is similar; however, Mean-RS-VRS worsens its performance over VRS. In the following experiment, the P-N \(\Delta\)MSE-RS reward strategy is combined with each O-P reward strategy. Figure 18 shows the results, PRS (dark blue), MSE-RS (red), \(\Delta\)MSE-RS (purple), PRS-\(\Delta\)MSE-RS (green), MSE-RS-\(\Delta\)MSE-RS (light blue) and Mean-RS-\(\Delta\)MSE-RS (magenta). Again it is possible to observe how all the combined strategies and \(\Delta\)MSE-RS converge at the same speed until iteration 5. From this point on \(\Delta\)MSE-RS and PRS-\(\Delta\)MSE-RS give smaller error MSE. On the other hand, the variance decreases until iteration 5, after which it grows, although less for MSE-RS-\(\Delta\)MSE-RS and Mean-RS-\(\Delta\)MSE-RS. Furthermore, PRS-\(\Delta\)MSE-RS tends to be slightly smaller than \(\Delta\)MSE-RS. Figure 16: Evolution of the MSE (**left**) and variance (**right**) for different combinations of position reward strategy (PRS) and velocity reward strategy (VRS). Figure 17: Evolution of the MSE (**left**) and variance (**right**) for combinations of MSE-RS, Mean-RS, and VRS. In this last experiment, the P-N \(\Delta\)Mean-RS strategy is combined with each O-P strategy. Figure 19 shows the results, with PRS (dark blue), MSE-RS (red), \(\Delta\)Mean-RS (purple), PRS-\(\Delta\)Mean-RS (green), Mean-RS-\(\Delta\)MSE-RS (light blue) and Mean-RS-\(\Delta\)Mean-RS (magenta). Again it is possible to observe how all the combined strategies and \(\Delta\)Mean-RS converge at the same speed until approximately iteration 5, where PRS-\(\Delta\)Mean-RS improves the MSE. The combination of these strategies provides a better result than their individual application. Furthermore, the variance also decreases with iterations. The combination of \(\Delta\)Mean-RS with Mean-RS and MSE-RS only offers an appreciable improvement in variance. Table 5 compiles the numerical results of the previous experiments. The KPIs have been measured at iteration 25. The best MSE is obtained by the combination of PRS-VRS and the best mean value and variance by MSE-RS-\(\Delta\)MSE-RS. In general, it is possible to observe how the combination with PRS decreases the MSE and the combination with MEAN-RS and MSE-RS improves the mean value and the variance.