## 4 Evaluation
### Results  
This section presents the set of results that were collected with both emulation and simulation setups with four different tested React-UCB configurations. Testing with different configurations aims to assess the contribution of each React-UCB's feature (e.g., arms correlation, reset functionality) to the overall algorithm's performance. The baseline configuration, referred to as B, represents the implemented discounted KL-UCB (Equation (24)) augmented with \(\epsilon\)-greedy for coverage guarantees in non-stationary reward distributions. This configuration, composed of standard components, serves as a benchmark for the remaining configurations. The second configuration, hereafter referred to BR, extends configuration B by including the reset functionality. Hence, a performance improvement in configuration B over configuration B should be an indication that the reset functionality provides added value to the overall solution. The third configuration, hereafter referred to BRC, builds upon configuration BR by including arms' correlation information in the decision-making process. The goal of this configuration is to assess whether the correlation between arms, that is, knowledge regarding how many links are shared across paths, can make a valuable contribution to the provision of pseudo-rewards. Finally, the fourth configuration, referred to as BRCT, extends configuration BRC by including the mechanism that allows for the algorithm (Lines **7-8** in Algorithm 1) to cancel exploration when rewards rise above a given satisfactory threshold, that is, the rewards are good enough for the task at hand. Table 2 summarizes the average (AVG) and the standard deviation (STD), per time step, of the regret obtained with each of the four tested configurations. These statistics were computed by considering the total regret accumulated over 30 independent runs, divided by the number of time steps in each run (a run lasts \(T\) time steps, with \(T\) varying according to the evaluation setup, as described earlier). A total of 30 independent runs was selected to increase the power of the statistical tests to compare several configurations; that is, to \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Distribution** & **Path (1, 2, 4)** & **Path (1, 3, 4)** & **Path (1, 5, 4)** \\ \hline \(d_{1}\) & **200**\(\pm\) **20.0** & 230 \(\pm\) 23.0 & 270 \(\pm\) 27.0 \\ \(d_{2}\) & 200 \(\pm\) 20.0 & **100**\(\pm\) **10.0** & 240 \(\pm\) 24.0 \\ \(d_{3}\) & 80 \(\pm\) 8.0 & 100 \(\pm\) 10.0 & **20**\(\pm\) **2.0** \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Path delay variation (AVG \(\pm\) STDEV ms) across emulated distributions; the best distribution path in each distribution is in bold. Each path is represented by a tuple composed of the switches along the path. Only a subset of 3 of the 25 possible end-to-end paths are listed. minimise the impact that the randomness in both agent and reward systems may have on the variance in the results. The independence between runs was enforced by using a different random seed in the beginning of each run. As expected, Table 2 shows that including each React-UCB's component results in a gradual decrease in terms of regret. These results are confirmed in both simulation and emulation setups, meaning that these results apply to both network and generic application scenarios. Hence, the devised solutions reset the MAB when a reward distribution change is detected, exploit the arms correlation to build pseudo-rewards, and stop exploration when task-dependent, good-enough rewards are reached. These all positively contribute to the algorithm's performance. In fact, the fourth configuration, BRCT, exhibits only 70% of the regret suffered by the baseline (benchmark) configuration, B, across both setups. Hence, this result indicates an overall important reduction in terms of agent regret.