SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this paper, we studied a non-stationary MAB problem with an AR structure, which captures the rapid changes commonly observed in real-world dynamics. Our proposed algorithm, AR2, leverages our knowledge or estimate of the temporal dependency to effectively handle the challenges associated with the fast-changing environment, and our techniques can be potentially adapted to more complex temporal series. As the realm of non-stationary bandits with rewards governed by temporal structures remains largely unexplored, there are several exciting avenues for future research. One intriguing direction is to incorporate seasonality into our framework, exploring models such as seasonal ARIMA with both long-term and short-term changes. Additionally, building on the promising numerical results in Section 8, it would be interesting to theoretically characterize the performance of our algorithm when estimation of the AR parameter and online-decision making are performed simultaneously.
We are grateful to the MIT-IBM Watson AI Lab for their generous support.
[1] D. A. Aaker, J. M. Carman, and R. Jacobson. Modeling advertising-sales relationships involving feedback: a time series analysis of six cereal brands. Journal of Marketing Research, 19(1):116-125, 1982.
[2] D. Agarwal, B.-C. Chen, and P. Elango. Spatio-temporal models for estimating click-through rate. In Proceedings of the 18th International Conference on World Wide Web, page 21-30, 2009.
[3] O. Anava, E. Hazan, and A. Zeevi. Online time series prediction with missing data. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pages 2191-2199, 2015.
[4] A. A. Ariyo, A. O. Adewumi, and C. K. Ayo. Stock price prediction using the arima model. In 2014 UKSim-AMSS 16th international conference on computer modelling and simulation, pages 106–112. IEEE, 2014.
[5] P. Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research, 3:397-422, 2003.
[6] P. Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47:235-256, 2002.
[51] W. R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25:285 - 294, 1933.
[52] F. Trovo, S. Paladino, M. Restelli, and N. Gatti. Sliding-window thompson sampling for non-stationary settings. Journal of Artificial Intelligence Research, 68:311-364, 2020.
[53] P. Whittle. Restless bandits: activity allocation in a changing world. Journal of Applied Probability, 25A:287-298, 1988.
Appendices for Non-Stationary Bandits with Auto-Regressive Temporal Dependency
A Numerical Studies on Synthetic Data
Set-up. We consider a non-stationary bandit problem with  $k$  arms and  $T = 10,000$  rounds. To make the setting more general, we let the expected reward  $r_i(t)$  of each arm  $i \in [k]$  follow an independent AR-1 process with heterogeneous AR parameter  $\alpha_i$ , stochastic rate of change  $\sigma_i$ , and truncating boundary  $[-1,1]$ . We consider two different regimes of  $\alpha_i$ 's: (i) Small  $\alpha_i$ 's: the AR-1 parameters  $\{\alpha_i\}_{i=1}^k$  are drawn from a dirichlet distribution with rescaling such that for all  $i \in [k]$ ,  $\mathbb{E}[\alpha_i] = 0.4$  which is less than the threshold value  $\bar{\alpha} = 0.5$  that we defined in Section 5. (ii) Large  $\alpha_i$ 's: the AR-1 parameters  $\{\alpha_i\}_{i=1}^k$  are chosen from a dirichlet distribution with rescaling such that  $\mathbb{E}[\alpha_i] = 0.9$  for all  $i \in [k]$ . The concentration parameters of the dirichlet distributions are all set to be 5. For each regime of  $\alpha_i$ 's (small v.s. large), we consider 100 instances, where each instance corresponds to AR parameters  $(\alpha_1, \ldots, \alpha_k)$  generated as described above; and stochastic rates of change  $(\sigma_1, \ldots, \sigma_k)$  generated from i.i.d. uniform distributions in  $(0,0.5)$ . The initial reward  $r_i(0)$  is drawn independently from the steady state distribution discussed in Appendix D.
Performance comparison. We implement AR2 and compare it against a number of benchmarks designed for stationary and non-stationary bandits respectively. (See Appendix B.1 for the implementation details, where we generalize AR2 to the setting with heterogeneous  $\alpha_{i}$ 's and  $\sigma_{i}$ 's.) All parameters used in the following algorithms are tuned.
The benchmark algorithms designed for stationary settings are: (i) "explore-then-commit" (ETC) policy [24] that first selects each arm for  $m$  rounds, and commits to the arm with the highest average pay-off in the remaining  $T - mk$  rounds; (ii) UCB algorithm [6], which achieves sublinear regret under static settings; (iii)  $\epsilon$ -greedy algorithm [49] with probability of exploration  $\epsilon = 0.1$ , which acts greedily with probability  $(1 - \epsilon)$  and explores a random arm otherwise. (iv) Exp3 algorithm [7], which is designed for the adversarial setting and achieves sublinear regret when evaluated against the weak static benchmark.
Further, we also consider several benchmark algorithms designed for non-stationary settings: (i) Rexp3 algorithm [10], which modifies the Exp3 algorithm for the non-stationary setting by introducing a restarting mechanism, and assumes that the total variation of expected rewards is bounded by a variation budget  $V_{T}$ ; (ii) a modified UCB algorithm (mod-UCB), which modifies the upper confidence bound based on the structure of the AR-1 process; see Appendix C for a description and discussion of the algorithm; (iii) the sliding-window UCB (SW-UCB) algorithm [25] and the sliding-window Thompson Sampling (SW-TS) algorithm [52], both of which adopt sliding-window approaches; (iv) the predictive sampling (PS) algorithm for AR-1 bandits recently introduced by [39], which is shown to outperform the Thompson sampling algorithm in non-stationary environments.
Results. In Table 2, we evaluate and compare the performance of AR2 against the benchmark algorithms in instances with small  $\alpha_{i}$ 's  $(\mathbb{E}[\alpha_i] = 0.4)$  as well as large  $\alpha_{i}$ 's  $(\mathbb{E}[\alpha_i] = 0.9)$ . We also vary the number of arms  $k \in \{2,10,20\}$ . To keep our results consistent across instances, we evaluate each algorithm  $\mathcal{A}$  using the normalized regret<sup>7</sup>:  $\sum_{t=1}^{T} \mathrm{REG}_{\mathcal{A}}(t) / \sum_{t=1}^{T} r^{\star}(t)$ , which normalizes the total regret using the reward of optimal in hindsight dynamic policy.
From Table 2, we see that AR2 clearly outperforms the benchmarks in almost all choices of  $k$  and regimes of  $\alpha_{i}$ 's. The algorithms designed for stationary settings (ETC, UCB, Exp3) perform poorly, as the AR setting includes non-stationarity that differs substantially from the stochastic or adversarial settings they were intended for. Similarly, the algorithms designed for non-stationary setting with limited amount of changes (RExp3, SW-UCB, SW-TS) do not perform well since they are designed to tackle environments either with abrupt changes, or with sublinear amount of changes, and are not suited for the fast changes in the AR environment. Finally, when compared against benchmarks that take into account the AR structure of rewards ( $\epsilon$ -greedy, modified UCB, PS), AR2 also displays superior performance. Here, the PS algorithm indeed dominates TS-based approaches
Note that since the expected rewards can be negative, the normalized regret can exceed one. Further, when  $\alpha_{i}$ 's are small, the normalized regret of each algorithm is in general larger. This is because the optimal reward used in normalization tends to be smaller, and hence does not contradict the fact that our problem is more challenging when the AR parameters are small.
Table 2: The mean and standard deviation (in parenthesis) of normalized regret incurred by each algorithm at  $T = {10},{000}$  . All experiments are run with 100 instances. The algorithm with the smallest normalized regret is highlighted in gray.
(a) Comparison with Stationary Benchmarks
E[αi]	k	AR2	ETC	UCB	ε-greedy	Exp3
0.4	2	0.38 (0.10)	1.00 (0.03)	0.68 (0.09)	0.43 (0.09)	0.99 (0.02)
10	0.67 (0.01)	1.00 (0.01)	0.80 (0.01)	0.76 (0.02)	1.00 (0.01)
20	0.72 (0.01)	1.00 (0.01)	0.84 (0.01)	0.81 (0.01)	1.00 (0.01)
0.9	2	0.18 (0.06)	1.00 (0.11)	0.65 (0.13)	0.36 (0.14)	0.95 (0.12)
10	0.40 (0.04)	1.00 (0.05)	0.58 (0.04)	0.60 (0.06)	0.99 (0.02)
20	0.49 (0.02)	1.00 (0.02)	0.62 (0.02)	0.64 (0.03)	0.99 (0.01)
(b) Comparison with Non-Stationary Benchmarks
E[αi]	k	AR2	Rexp3	mod-UCB	SW-UCB	SW-TS	PS
0.4	2	0.38 (0.10)	0.96 (0.03)	0.45 (0.05)	0.69 (0.05)	1.00 (0.03)	0.51 (0.04)
10	0.67 (0.01)	0.99 (0.01)	0.67 (0.03)	0.90 (0.02)	0.98 (0.01)	0.78 (0.03)
20	0.72 (0.01)	1.00 (0.01)	0.72 (0.02)	0.95 (0.01)	0.99 (0.01)	0.84 (0.02)
0.9	2	0.18 (0.06)	0.87 (0.08)	0.20 (0.07)	0.55 (0.13)	0.99 (0.15)	0.52 (0.33)
10	0.40 (0.04)	0.97 (0.01)	0.43 (0.05)	0.70 (0.03)	0.86 (0.04)	0.47 (0.03)
20	0.49 (0.02)	0.99 (0.01)	0.53 (0.03)	0.84 (0.02)	0.91 (0.02)	0.52 (0.02)
in non-stationary environments, as suggested by [39], but AR2 remains competent in all kinds of AR-based setups; further, [39] also suggests that deriving the conditional probability distribution in the predictive sampling procedure and implementing it can be complicated in general, and hence it is unclear how one can adapt the PS algorithm for more complicated AR- $p$  processes as we did in our case study (see Section 7 and Appendix B.2). Among all stationary benchmarks,  $\epsilon$ -greedy is the most competitive; while mod-UCB stands out as the most competitive within the non-stationary benchmarks. However, there are instances where AR2 can contribute to a decrease in regret as high as  $50\%$  when compared to  $\epsilon$ -greedy, and a  $15.5\%$  decrease in regret when compared to modified UCB. All of the aforementioned results attest to the efficacy of AR2 in rapidly changing environments governed by the AR process.
B Implementation Details and Extensions of Algorithm AR2
B.1 Implementation of AR2
To apply AR2 to the more general setting described in Appendix A, where the expected reward of each arm  $i$  undergoes an AR-1 process with heterogeneous AR parameter  $\alpha_{i}$  and stochastic rate of change  $\sigma_{i}$ , we make a few straightforward updates to AR2: (i) when maintaining estimated reward  $\widehat{r}_i(t)$  of arm  $i$ , we use its AR parameter  $\alpha_{i}$ ; (ii) we assume that an arm  $i \neq i_{\mathrm{sup}}(t)$  gets triggered if at round  $t$ ,
$$
\widehat {r} _ {\mathrm {s u p}} (t) - \widehat {r} _ {i} (t) \leq c _ {1} \sigma_ {i} \sqrt {(\alpha_ {i} ^ {2} - \alpha_ {i} ^ {2 (t - \tau_ {i} + 1)}) / (1 - \alpha_ {i} ^ {2})}.
$$
Empirically, we make two more minor changes to AR2 in its numerical implementation. (i) The definition of the superior arm: instead of considering one of the two most recently pulled arms, we consider all arms and choose the one with the highest estimated reward at round  $t$  to be the superior arm at round  $t$ , i.e.,  $i_{\mathrm{sup}}(t) = \arg \max_{i\in [k]}\widehat{r}_i(t)$ . (ii) The triggered arm to be played: in the exploration round  $t$  (when  $t$  is odd and  $\mathcal{T}\neq \emptyset$ ), we consider the upper confidence bound of the expected reward of each arm  $i$ , defined as:
$$
\mathrm {U C B} _ {i} (t) = \widehat {r} _ {i} (t) + c _ {1} \sigma_ {i} \sqrt {\left(\alpha_ {i} ^ {2} - \alpha_ {i} ^ {2 (t - \tau_ {i} (t))}\right) / \left(1 - \alpha_ {i} ^ {2}\right)}
$$
and pull the triggered arm with the highest upper confidence bound, i.e.  $I_{t} = \arg \max_{j\in \mathcal{T}}\mathrm{UCB}_{j}(t)$ . Note that the arms in the triggered set  $\mathcal{T}$  differ in terms of both triggering time and estimated rewards. Previously, our criteria for selecting the triggered arm to play only takes into account one aspect of such heterogeneity, i.e., the triggering time. The new criteria allows us to also consider the estimated reward of each arm, which is another indicator of the arm's potential.
B.2 Extension of AR2 to AR- $p$  Process with Trend
In this section, we provide an extension of AR2 (Algorithm 1), which would apply to non-stationary bandits where the rewards follow general AR- $p$  processes with trends for any  $p \geq 1$ . This algorithm is applied when we perform our case study on tourism demand prediction in Section 7, and is shown to adapt well to the more complex, rapidly evolving environment.
Let us assume that for each arm  $i \in [k]$ , the reward distribution following an AR- $p$  process, potentially with a trend component. That is, conditioning on past history during rounds  $t - 1, \ldots, 1$ , the expected reward of arm  $i$  at round  $t$  is
$$
r _ {i} (t) = \alpha_ {i, 0} + \sum_ {j = 1} ^ {p} \alpha_ {i, j} R _ {i} (t - j),
$$
where  $\alpha_{i,0}$  is the trend component and  $(\alpha_{i,1},\ldots ,\alpha_{i,p})$  is the set of AR parameters associated with arm  $i$ .
The details of the extension of AR2 is presented in Algorithm 2, which we called AR2-p. At a high level, the main techniques used in AR2-p are very similar to those used in AR2. As AR2-p is specifically designed to address practical scenarios where real-world dynamics are represented by more complex time series, we have incorporated the implementation changes discussed in Appendix B.1 to enhance its practical applicability.
At each round  $t$  of Algorithm 2, we first determine the superior arm  $i_{\mathrm{sup}}(t)$ , which is the arm with the highest expected reward. We then decide if there are any arms that we wish to trigger, depending on whether some untriggered arm has an upper confidence bound that exceeds the expected reward of the superior arm  $\widehat{r}_{\mathrm{sup}}(t)$ . We then alternate between exploration and exploitation: (i) during exploitation rounds, we pull the superior arm; (ii) during exploration rounds, we pull the arm with the highest upper confidence bound out of the set of triggered arms, denoted as  $\mathcal{T}$ . Finally, after we pull arm  $I_t$  at each round, we update the estimate for the state of each arm  $i \in [k]$ .
The main difference between AR2-p and AR2 for AR-1 process lies in how we maintain estimates for each arm. To account for the more complicated structure of an AR- $p$  process with trends, we need to maintain two estimates for each arm—an estimate for the expected reward  $\widehat{r}_i(t)$ , and an estimate for the error bound  $\widehat{E}_i(t)$ , defined as follows:
$$
\widehat {r} _ {i} (t) = \alpha_ {i, 0} + \sum_ {j = 1} ^ {p} \alpha_ {i, j} \widehat {r} _ {i} (t - j) \quad \text {a n d} \quad \widehat {E} _ {i} (t) = \sum_ {j = 1} ^ {p} \alpha_ {i, j} \widehat {E} _ {i} (t - j) + \alpha_ {i, j}.
$$
Here, the estimate for error bound  $\widehat{E}_i(t)$  essentially captures how much error is contained in our estimate for arm  $i$  at round  $t$ , i.e.,  $\widehat{r}_i(t)$ . Since we estimate  $\widehat{r}_i(t)$  based on  $p$  past estimates  $\widehat{r}_i(t - 1), \ldots, \widehat{r}_i(t - p)$ , the error associated with each of these terms, i.e.,  $\widehat{E}_i(t - 1), \ldots, \widehat{E}_i(t - p)$ , would contribute the total error for arm  $i$  at round  $t$ , i.e.,  $\widehat{E}_i(t)$ .
We would then use the error estimate  $\widehat{E}_i(t)$  in determining our triggering condition. We trigger an arm  $i \notin \mathcal{T} \cup \{i_{\mathrm{sup}}(t)\}$  at round  $t$  if
$$
\widehat {r} _ {\sup } (t) - \widehat {r} _ {i} (t) \leq c \sigma_ {i} \sqrt {\widehat {E} _ {i} (t)}.
$$
The error bound estimates would also be useful when we determine which arm to pull out of all the triggering arms. To do that, we form the upper confidence bound for each arm
$$
\mathsf {U C B} _ {i} (t) = \widehat {r} _ {i} (t) + c \sigma_ {i} \sqrt {\widehat {E} _ {i} (t)}
$$
and pick the one with the highest upper confidence bound:  $I_{t} = \arg \max_{i\in \mathcal{T}}\mathrm{UCB}_{i}(t)$ .
C Description of Modified UCB Algorithm
In our numerical studies (Appendix A), we have considered the modified UCB algorithm as one of the non-stationary benchmarks. We now describe this algorithm, which borrows idea from the UCB algorithm [6] for stochastic MAB, and modifies the expression of the upper confidence bound based on the AR temporal structure.
Algorithm 2 Alternating and Restarting algorithm for non-stationary bandits with AR- $p$  reward structure (AR2-p)
Input: AR parameters  $\{\alpha_{i,j}\}_{\substack{i\in [k]\\ j\in [p]}}$  , trend components  $\{\alpha_{i,0}\}_{i\in [k]}$  , stochastic rate of change  $\{\sigma_i\}_{i\in [k]}$  , epoch size  $\Delta_{\mathrm{ep}}$  , parameter  $c$
1. Set the epoch index  $s = 1$ .
2. Repeat while  $s \leq \left[ T / \Delta_{\mathrm{ep}} \right]$ :
(a) Initialization:
- Set  $t_0 = (s - 1)\Delta_{\mathrm{ep}}$ . Set the initial triggered set  $\mathcal{T} = \emptyset$
- For each arm  $i \in [k]$ , set estimates for expected rewards  $\widehat{r}_i(t) = 0$  and error bound estimates  $\widehat{E}_i(t) = \infty$  for  $t \leq 0$ .
(b) For  $t = t_0 + 1, \ldots, \min \{t_0 + \Delta_{\mathrm{ep}}, T\}$
- If  $t - t_0 \leq p \cdot k$ , pull arm  $i = \lfloor (t - t_0 - 1) / p \rfloor + 1$ . // Pull each arm consecutively for  $p$  rounds.
- Else,
- Update the identity of the superior arm and its estimated reward
$$
i _ {\sup } (t) = \underset {i \in [ k ]} {\arg \max } \widehat {r} _ {i} (t) \quad \text {a n d} \quad \widehat {r} _ {\sup } (t) = \widehat {r} _ {i _ {\sup } (t)} (t)
$$
- Trigger arms with potential: For  $i \notin T \cup \{i_{\mathrm{sup}}(t)\}$ , trigger arm  $i$  if
If triggered, add  $i$  to  $\mathcal{T}$ .
i. If  $t$  is odd and  $\mathcal{T} \neq \emptyset$ , pull a triggered arm with the highest upper confidence bound estimate:  $I_{t} = \arg \max_{i\in \mathcal{T}}\widehat{r}_{i}(t) + c\sigma_{i}\sqrt{\widehat{E}_{i}(t)}$ .
ii. Otherwise, pull the superior arm  $I_{t} = i_{\mathrm{sup}}(t)$ .
- Receive a reward  $R_{I_t}(t)$ . Update  $\hat{r}_{I_t}(t) = R_{I_t}(t)$  and  $\widehat{E}_{I_t}(t) = 0$ .
- Maintain Estimates: For each arm  $i \in  \left\lbrack  k\right\rbrack$  ,set estimates for expected rewards and error bounds
Algorithm 3 A modified UCB algorithm for non-stationary AR bandits (mod-UCB)
Input: AR Parameters  $(\alpha_{1},\dots ,\alpha_{k})$  , stochastic rates of change  $(\sigma_{1},\ldots ,\sigma_{k})$  , parameter  $\delta$
1. Initialization: pull each arm  $i \in [k]$  at round  $i$ . Set estimates  $\widehat{r_i} (k + 1) = \alpha^{k - i}\cdot \mathcal{B}(\alpha R_i(i))$  and  $\tau_{i} = i$ .
2. While  $t \leq T$ :
(a) For each arm  $i \in [k]$ , compute its upper confidence bound (UCB):
$$
\mathsf {U C B} _ {i} (t) = \widehat {r} _ {i} (t) + \sqrt {2 \log (2 / \delta)} \cdot \sigma_ {i} \sqrt {(\alpha_ {i} ^ {2} - \alpha_ {i} ^ {2 (t - \tau_ {i} (t))}) / (1 - \alpha_ {i} ^ {2})}.
$$
(b) Pull the arm with the highest UCB:  $I_{t} = \arg \max_{i\in [k]}\mathsf{UCB}_{i}(t)$ . Receive a reward  $R_{I_t}(t)$ , and set  $\tau_{I_t} = t$
(c) Maintain Estimates: Set  $\widehat{r}_{I_t}(t + 1) = \mathcal{B}(\alpha R_{I_t}(t))$  and set  $\widehat{r}_i(t + 1) = \alpha \widehat{r}_i(t)$  for  $i\neq I_t$
The details of modified UCB is presented in Algorithm 3. Similar to the original UCB algorithm, we consider an upper confidence bound of the expected reward of each arm  $i$  at round  $t$ , denoted as  $\mathrm{UCB}_i(t)$ , which consists of two terms. The first term  $\widehat{r}_i(t)$  is the estimated reward of arm  $i$  based on past observations; the second term is the size of the one-sided confidence interval for the expected reward  $r_i(t)$ . The confidence interval is constructed based on Hoeffding's Inequality (see Lemma J.3), and is designed such that the true expected reward  $r_i(t)$  will fall into the confidence interval with probability at least  $1 - \delta$ , where  $\delta$  is an input parameter. To derive the expression of the upper confidence bound in the AR setting, consider a non-stationary MAB problem, in which the expected reward of arm  $i$  follows an AR-1 process with parameter  $\alpha_i \in (0,1)$ , stochastic rate of change  $\sigma_i \in (0,1)$ , and truncating boundary  $[-\mathfrak{R},\mathfrak{R}]$ . (See Section 2 for details about the set-up.) The estimated reward of arm  $i$  at round  $t$ ,  $\widehat{r}_i(t)$ , can be maintained in the same way as done in Algorithm 1.
The modified UCB algorithm always selects the arm with the highest upper confidence bound. To construct the confidence bound, we first recall from Section 6 that we define a new noise term  $\tilde{\epsilon}_i(t)$  (see (5)) that incorporates the influence of the truncating boundary, and the new noise term satisfies the recursive relationship:  $r_i(t + 1) = \alpha r_i(t) + \alpha \tilde{\epsilon}_i(t)$ . Suppose that at round  $t$ , we last observed the reward of arm  $i$  at round  $\tau_i(t)$ . Using the recursive relationship, we have:
$$
\begin{array}{l} r _ {i} (t) = \alpha_ {i} r _ {i} (t - 1) + \alpha_ {i} \tilde {\epsilon} _ {i} (t - 1) = \alpha_ {i} ^ {t - \tau_ {i} (t) - 1} r _ {i} (\tau_ {i} (t) + 1) + \alpha_ {i} ^ {t - \tau_ {i} (t) - 1} \tilde {\epsilon} _ {i} (\tau_ {i} (t) + 1) + \dots + \alpha_ {i} \tilde {\epsilon} _ {i} (t - 1) \\ = \alpha_ {i} ^ {t - \tau_ {i} (t) - 1} r _ {i} \left(\tau_ {i} (t) + 1\right) + \sum_ {t ^ {\prime} = \tau_ {i} (t) + 1} ^ {t - 1} \alpha^ {t - t ^ {\prime}} \tilde {\epsilon} _ {i} \left(t ^ {\prime}\right) = \widehat {r} _ {i} (t) + \sum_ {t ^ {\prime} = \tau_ {i} (t) + 1} ^ {t - 1} \alpha^ {t - t ^ {\prime}} \tilde {\epsilon} _ {i} \left(t ^ {\prime}\right), \tag {8} \\ \end{array}
$$
where the last equality follows from the way we maintain estimates. By Lemma H.1, we have that for any  $s > 0$ ,
$$
\mathbb {P} \left[ \left| \sum_ {t ^ {\prime} = \tau_ {i} (t) + 1} ^ {t - 1} \alpha^ {t - t ^ {\prime}} \tilde {\epsilon} _ {i} \left(t ^ {\prime}\right) \right| \leq s \right] \geq \mathbb {P} \left[ \left| \sum_ {t ^ {\prime} = \tau_ {i} (t) + 1} ^ {t - 1} \alpha^ {t - t ^ {\prime}} \epsilon_ {i} \left(t ^ {\prime}\right) \right| \leq s \right], \tag {9}
$$
where  $\epsilon_{i}(t^{\prime})\sim N(0,\sigma_{i})$  is the independent Gaussian noise applied to arm  $i$  at round  $t^\prime$ . By Hoeffding's Inequality (Lemma J.3), for fixed  $\delta >0$ , we also have the following concentration result:
$$
\mathbb {P} \left[ \left| \sum_ {t ^ {\prime} = \tau_ {i} (t) + 1} ^ {t - 1} \alpha^ {t - t ^ {\prime}} \epsilon_ {i} \left(t ^ {\prime}\right) \right| \leq \sqrt {2 \log (2 / \delta)} \cdot \sigma_ {i} \sqrt {\left(\alpha_ {i} ^ {2} - \alpha_ {i} ^ {2 (t - \tau_ {i} (t))}\right) / \left(1 - \alpha_ {i} ^ {2}\right)} \right] \geq 1 - \delta . \tag {10}
$$
Equations (8), (9) and (10) together establish a confidence bound centered around  $\widehat{r}_i(t)$ , such that the true expected reward  $r_i(t)$  falls into the confidence bound with probability at least  $1 - \delta$ . In Algorithm 3, we thus define the upper confidence bound of arm  $i$  at round  $t$  as:
$$
\mathsf {U C B} _ {i} (t) = \widehat {r} _ {i} (t) + \sqrt {2 \log (2 / \delta)} \cdot \sigma_ {i} \sqrt {(\alpha_ {i} ^ {2} - \alpha_ {i} ^ {2 (t - \tau_ {i} (t))}) / (1 - \alpha_ {i} ^ {2})},
$$
and we have  $\mathbb{P}[r_i(t) > \mathrm{UCB}_i(t)] \leq 1 - \delta$  when implementing the modified UCB algorithm for numerical experiments in Appendix A.