## 3 System Architecture
### Incremental Formulation and Reset  
To speed up computation, React-UCB avoids the summations in \(\bar{X}_{t}^{std}(i)\) and \(N_{t}^{std}(i)\) with an equivalent incremental formulation:  
\[\bar{X}_{t}(i) =\gamma\bar{X}_{t-1}(i)R_{t-1}+\mathds{1}_{\{I_{t}=i\}}X_{t}(i), \tag{6}\] \[N_{t}(i) =\gamma N_{t-1}(i)R_{t-1}+\mathds{1}_{\{I_{t}=i\}},\] (7) \[\bar{X}_{0}(i) =0,N_{0}(i)=0,R_{0}=0, \tag{8}\]  
where \(R_{t}\in\{0,1\}\) is a modulatory variable that allows for the agent to reset \(\bar{X}_{t}(i)\) and \(N_{t}(i)\), when an abrupt reward distribution change in the pulled arm \(I_{t}\) is detected (such a change would be difficult to accommodate with discounting alone). This change detection occurs on the list of rewards accumulated by each agent's arm \(i\in\mathcal{K}\), whose size \(n_{t}(i)\) increases whenever \(i\) is pulled at time step \(t\), and whose \(j\)-th element is given by \(x_{j}(i)\):  
\[n_{t}(i) =n_{t-1}(i)+\mathds{1}_{\{I_{t}=i\}}, \tag{9}\] \[x_{j<1}(i) =n_{j<1}(i)=0,\] (10) \[x_{n_{t}(i)}(i) =\mathds{1}_{\{I_{t}=i\}}X_{t}(i)+\mathds{1}_{\{I_{t}\
eq i\}}x_ {n_{t-1}(i)}(i). \tag{11}\]  
Similar to M-UCB [23], React-UCB detects changes by comparing some reward statistics across the two most recent and contiguous series of \(w\) rewards gathered by the pulled arm, \(I_{t}\). The two ordered sets of indices of these two series in the rewards list are:  
\[v_{1,t} =\{n_{t}(I_{t})-w,\dots,n_{t}(I_{t}):n_{t}(I_{t})\geq 2w\}, \tag{12}\] \[v_{2,t} =\{n_{t}(I_{t})-2w,\dots,n_{t}(I_{t})-w:n_{t}(I_{t})\geq 2w\}. \tag{13}\]  
M-UCB detects changes based on the absolute difference between average rewards across the two contiguous series as a function of a fixed average reward difference threshold. However, a fixed threshold can rapidly become inaccurate in scenarios in which the rewards'variance changes unpredictably. That is, a low threshold may result in excessive false positives (i.e., resets being issued even when the reward distribution has not changed) if the rewards' variance rises unexpectedly, whereas a high threshold may result in excessive false negatives (i.e., resets not issued when the reward distribution has changed) if the rewards' variance lowers unexpectedly. The practical implication of being unable to properly determine when what has been learned should be forgotten (i.e., a reset should be issued) increases regret. Alternatively, a change detection in React-UCB is only reported if: (i) the average reward difference between \(\{x_{i}(I_{t})\}_{\forall i\in v_{1,t}}\) and \(\{x_{i}(I_{t})\}_{\forall i\in v_{2,t}}\) is above a small threshold \(\zeta\) (small, 0.1 in our tests, to avoid false negatives); (ii) it is statistically significant (not due to chance); and (iii) it is of practical significance for the system optimization goal. Hence, in React-UCB, the chances of triggering a reset are dynamically adjusted according to the rewards' variance, instead of being ruled by a fixed threshold throughout the whole time horizon. Statistical significance holds when \(p_{t}<\alpha_{p}\), where \(p_{t}\) is the probability of the observed difference in average rewards under the null hypothesis \(H_{0}\triangleq(\mu_{1,t}=\mu_{2,t})\) and for a significance level \(\alpha_{p}\) (we opt for the standard \(\alpha_{p}=0.05\)), and is estimated via an independent, two-sample, two-sided [PERSON]'s \(t\)-test:  
\[\mu_{1,t} =w^{-1}\sum_{v\in v_{1,t}}x_{v}(I_{t}), \tag{14}\] \[\mu_{2,t} =w^{-1}\sum_{v\in v_{2,t}}x_{v}(I_{t}). \tag{15}\]  
Practical significance holds when \(\Delta_{t}>\alpha_{\Delta}\), where \(\Delta_{t}\) is the effect size metric known as [PERSON] \(\Delta\) and \(\alpha_{\Delta}\) is the effect size threshold (we used \(\alpha_{\Delta}=0.8\) for large effect size [26]).