## 3 Performance Evaluation: Settings  
Tests on the impact of the introduction of the proposed model were carried out through simulations. Performance in terms of regret of six different algorithms was compared.  
The tested algorithms are the following:  
* UCB1 [16];
* muUCB1;
* MLI;
* \(\varepsilon\)-greedy [23];
* \(\varepsilon\)-decreasing [16; 22];
* POKER [17].  
The last three algorithms listed above were not previously presented, and are briefly introduced in the following; in all of them, the action performed is always _use_, but they differ in the rules adopted for selecting the arm.  
The \(\varepsilon\)-greedy is extremely simple: it selects a random arm with probability \(\varepsilon\), and the arm that led to the highest cumulative rewards otherwise.  
The \(\varepsilon\)-decreasing is a variation of the \(\varepsilon\)-greedy in which the probability \(\varepsilon\) to select a random arm decreases with the time index \(n\); one has in particular \(\varepsilon\left(n\right)=\min\left\{1,\frac{\varepsilon_{0}}{n}\right\}\) in the version analyzed in [16] and implemented in this work.  
The Price of Knowledge and Estimated Reward (POKER) algorithm was proposed in [17]; the algorithm combines three ideas in determining the next arm to be selected: (a) assign a value to the exploration of an arm defined in the same units used for rewards; (b) use data collected for other arms to generate a priori knowledge on arms not yet used, assuming correlation between different arms; (c) take into account the time remaining until the time horizon is reached in order to decide whether to exploit or to explore; the algorithm performed quite well when applied to real, captured data. A detailed description of the algorithm and of its performance over captured data can be found in [17].  
All the six algorithms listed above where tested against both synthetic and captured data. The simulations were performed considering different \(T_{U}/T_{M}\) ratios. The case where \(T_{U}=T_{M}\) was included in the analysis as a baseline setting although, as discussed in Section 1, it can be expected that in real scenarios a _measure_ action will require a shorter time than a _use_ action, i.e., usually \(T_{M}<T_{U}\). Increasing \(T_{U}/T_{M}\) ratios can represent scenarios where the network switch process is increasingly costly in terms of overhead or user dissatisfaction, leading to longer use periods before a switching procedure can take place.  
Simulation settings common to both synthetic and real data are the following:  
* the number of steps required to reach the time horizon was set to \(n_{TH}=10^{5}\);
* \(\mathcal{K}=5\) arms were considered;
* the value of \(n_{M}\) was set to 1; therefore, \(T_{M}=T\); the value of \(n_{U}\) was variable, leading to different \(T_{U}/T_{M}\) ratios being considered;
* for \(\varepsilon\)-greedy algorithm, \(\varepsilon\) was set to 0.1 according to the results presented in [17], indicating this value as the one leading to best performance;
* again, according to [17], in the \(\varepsilon\)-decreasing algorithm, \(\varepsilon_{0}\) was set to 5;
* for the MLI algorithm, the number of times that every arm is measured in the Phase 1 was set to \(d_{1}=5\); \(d_{2}\), i.e., the number of use actions after which the first measure is performed, was also set to 5;
* all results were averaged over 500 runs;  
Information specific to synthetic and real data used in the experiments are presented in Sections 3.1 and 3.2, respectively. Results of the experiments are presented and analysed in Section 4.