SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Finally, we remark that our algorithm AR2 and the modified UCB algorithm share some similarities. Both algorithms incorporate our knowledge of the AR temporal structure into their design, which makes them more competitive against the rest of the benchmarks. The triggering criteria of AR2 essentially constructs a confidence interval around the estimated reward  $\hat{r}_i(t)$  of arm  $i$  at round  $t$ , and triggers arm  $i$  if the upper confidence bound of arm  $i$  exceeds the estimated reward of the superior arm. On the other hand, AR2 also differs from the modified UCB in two main aspects. (i) By alternating between exploration and exploitation rounds, AR2 avoids potential over-exploration. This is especially common when  $\alpha_i$ 's are close to one, and any arm that has not been explored for a few rounds would have a very high confidence bound. In that case, modified UCB is likely to shift in between arms very often, which leads to too little exploitation. (ii) As discussed in Section 4, unlike the modified UCB algorithm, AR2 also deals with the tradeoff of "remembering" and "forgetting" via the restarting mechanism, which is especially important in a fast changing environment such as the AR setting.
D Steady State Distribution of Rewards
In this section, we characterize the steady state distribution of the expected reward  $r_i(t)$ . Lemma D.1 shows that the steady state distribution depends on two probability masses at the boundaries and a probability density function between the boundaries.
Lemma D.1 (Probability density of steady state distribution). Consider the steady state distribution of  $r_i(t)$ , which follows the AR process with truncating boundaries defined in Section 2. Suppose that at the steady state distribution, we have two probability masses at the boundaries, i.e.,  $\mathbb{P}[r_i(t) = \Re] = \mathbb{P}[r_i(t) = -\Re] = p$ , where  $p \in (0,1/2)$  depends on  $\alpha$  and  $\sigma$ . Then, the steady state distribution between the boundaries can be characterized by the following probability density function (PDF), denoted by  $f_p: (-\Re, \Re) \to \mathbb{R}^+$ .
$$
f _ {p} (x) = (1 - 2 p) C (\alpha , \sigma) \exp \left(\frac {(\alpha - 1) x ^ {2}}{\alpha^ {2} \sigma^ {2}}\right) \quad x \in (- \mathfrak {R}, \mathfrak {R}), \tag {11}
$$
where  $C(\alpha, \sigma) = \frac{\sqrt{1 - \alpha}}{\sqrt{\pi} \alpha \sigma \operatorname{erf}\left(\frac{\mathfrak{R}_{\sqrt{1 - \alpha}}}{\alpha \sigma}\right)}$  is a normalization factor and  $\operatorname{erf}(z) = \frac{2}{\sqrt{\pi}} \int_{0}^{z} e^{-s^2} ds$  is the error function.
Given Lemma D.1, we are now ready to define the steady state distribution of  $r_i(t)$ :
Definition D.2 (Steady state distribution of rewards). Let  $r$  be a random variable with range  $[-\mathfrak{R}, \mathfrak{R}]$  whose probability distribution characterizes the steady state distribution of  $r_i(t)$ . It has probability masses on the boundaries, i.e.,  $\mathbb{P}[r = \mathfrak{R}] = \mathbb{P}[r = -\mathfrak{R}] = p$ , and probability density function  $f_p$  on  $(-\mathfrak{R}, \mathfrak{R})$  as defined in (11), that satisfies the following conditions:
1.  $\mathbb{P}[\alpha (r + \epsilon)\geq \Re ] = \mathbb{P}[\alpha (r + \epsilon)\leq -\Re ] = p.$
2.  $\mathbb{P}[\alpha (r + \epsilon) < y] = p + \int_{-R}^{y}f_{p}(x)dx$  for all  $y\in (-\Re ,\Re)$ .
Note that the probability masses on the boundaries result from the truncating boundary condition, and by symmetry, the masses on upper and lower boundaries are the same. If one adopts a reflecting boundary condition, we would have  $p = 0$ ; if one adopts an absorbing boundary condition, we would have  $p = 1/2$  (see [44] for definitions of the aforementioned boundary conditions).
We remark that for any fixed  $\sigma \in (0,1)$ , when  $\alpha \to 1$ , the probability density function  $f_{p}(x)$  in the steady state distribution of  $r_i(t)$  can be well approximated by a constant function. In Figure 3, we plot the PDF of the steady state distributions for  $\alpha \in \{0.3,0.6,0.9\}$  (with  $\sigma = 0.8$  and  $\mathfrak{R} = 1$ ), and numerically compute the value of  $p$  (see legends in Figure 3)<sup>8</sup>. It can be seen that for larger  $\alpha$ , the distribution within the boundaries becomes almost "uniform". In the other extreme case, when  $\alpha \to 0$ , it can be seen both from Figure 3 and from the definition of  $f_{p}(x)$  in Lemma D.1 that the distribution can be approximated by a normal distribution.
Figure 3: PDF of the steady state distribution of AR process with truncating boundaries for  $\alpha \in \{0.3, 0.6, 0.9\}$ ,  $\sigma = 0.8$ ,  $\Re = 1$ . The probability masses  $p$  associated with the steady state distribution are included in the legends.
Proof of Lemma D.1: We first note that the continuous-time analogue of an AR-1 process is a Ornstein-Uhlenbeck (OU) process [22], defined as the following stochastic differential equation (SDE):
$$
d r _ {t} = \kappa (\theta - r _ {t}) d t + \zeta d W _ {t}, \tag {12}
$$
where  $\kappa, \theta, \zeta$  are constant parameters and  $W_{t}$  denotes the Wiener process. If we perform Euler-Matyama discretization (see [34]) on the above OU process at  $\Delta t, 2\Delta t, \ldots$ , we get
$$
r _ {t + 1} = \kappa \theta \Delta t - (\kappa \Delta t - 1) r _ {t} + \zeta \epsilon_ {t} ^ {\prime} \sqrt {\Delta t}, \tag {13}
$$
where  $\epsilon_t^\prime$  is a standard normal noise. Now, recall from Section 2 that the evolution of expected rewards evolve as an independent AR-1 process:
$$
r _ {i} (t + 1) = \alpha R _ {i} (t) = \alpha \left(r _ {i} (t) + \epsilon_ {i} (t)\right), \tag {14}
$$
where  $\epsilon_{i}(t)\sim N(0,\sigma)$ . Let  $r_t\triangleq r_i(t)$ . By coefficients in (13) and (14), the two equations are equivalent if we let  $\kappa = 1 - \alpha$ ,  $\theta = 0$ ,  $\zeta = \alpha \sigma$ ,  $\Delta t = 1$ . By plugging the values of coefficients into (12), the continuous-time analogue of (14) can be described by the following SDE:
$$
d r _ {t} = (\alpha - 1) r _ {t} d t + \alpha \sigma d W _ {t}.
$$
Let  $f_{p}(x)$  be the PDF for the steady state distribution of  $r_i(t)$  on  $(\mathfrak{R},\mathfrak{R})$ , given that the probability mass on either side of the boundary is  $p$ . We can now solve for  $f_{p}(x)$  by solving the following
For numerically simulating steady state distributions of SDEs, see [34].
<sup>9</sup>We will see that when  $r_t$  follows an AR-1 process with AR parameter  $\alpha$  and stochastic rate of change  $\sigma$ , these constants take the following values:  $\kappa = 1 - \alpha, \theta = 0, \eta = \alpha \sigma$ .
<sup>10</sup>A Wiener process  $W_{t}$  is a continuous-time stochastic process that satisfies the following properties: (i)  $W_{0} = 0$ ; (ii)  $W_{t}$  is continuous in  $t$ ; (iii)  $\{W_{t}\}_{t\geq 0}$  has stationary, independent increments; (iv)  $W_{t + s} - W_{s}\sim N(0,t)$ . (See [20] for more details.)
ordinary differential equation (see [23, 44] for a discussion of forward and backward equations for SDEs):
$$
(1 - \alpha) \left(f _ {p} (x) + x f _ {p} ^ {\prime} (x)\right) + \frac {1}{2} \alpha^ {2} \sigma^ {2} f _ {p} ^ {\prime \prime} (x) = 0, \tag {15}
$$
$$
f _ {p} (x) = \tilde {C} (\alpha , \sigma) \exp (\frac {(\alpha - 1) x ^ {2}}{\alpha^ {2} \sigma^ {2}}),
$$
for  $\tilde{C}$  that depends on  $\alpha, \sigma$ . Since  $f_{p}(x)$  is a probability density, we can solve for  $\tilde{C}(\alpha, \sigma)$  by requiring  $\int_{-\mathbb{R}}^{\mathbb{R}} f_{p}(x) dx = 1 - 2p$ , given that the total probability within the boundaries is  $1 - 2p$ . This then gives
$$
\tilde {C} (\alpha , \sigma) = (1 - 2 p) \sqrt {1 - \alpha} / (\sqrt {\pi} \alpha \sigma \operatorname {e r f} \left(\frac {\Re \sqrt {1 - \alpha}}{\alpha \sigma}\right)) = (1 - 2 p) C (\alpha , \sigma),
$$
where  $C(\alpha, \sigma)$  is defined in the statement of Lemma D.1.
E.1 Illustration of  $g(k, \alpha, \sigma)$  in Regret Lower Bound
Recall that in Section 3, we characterize our regret lower bound to be  $\Omega(g(k, \alpha, \sigma) \alpha \sigma)$ , where  $g(k, \alpha, \sigma)$  is the probability that two best arms are within  $\alpha \sigma$  distance of each other at any given round under the steady state distribution, with its closed form presented in Equation (16) in Appendix F.
To better understand how  $g(k, \alpha, \sigma)$  evolves in terms of our parameters, in Figure 4 we plot the evolution of  $g(k, \alpha, \sigma)$  in terms of  $\sigma$  in two different regimes: (i) a small-  $\alpha$  regime when  $\alpha = 0.4$ , and (ii) a large-  $\alpha$  regime when  $\alpha = 0.9$ . Note that here, we consider the two regimes separately because in Section 5, we will show that the large-  $\alpha$  regime is inherently much more challenging than the small-  $\alpha$  regime.
In Figure 4, we see that in the small- $\alpha$  regime ( $\alpha = 0.4$ ),  $g(k, \alpha, \sigma)$  is roughly constant as  $\sigma$  increases, while in the large- $\alpha$  regime ( $\alpha = 0.9$ ),  $g(k, \alpha, \sigma)$  is roughly linear in terms of  $\sigma$ . To understand why, recall from our discussion in Appendix D that when  $\alpha \to 0$ , the steady state distribution approaches a normal distribution with standard deviation proportional to  $\alpha \sigma$ , while when  $\alpha \to 1$ , the steady state distribution resembles a uniform distribution. Hence, when  $\alpha$  is small, we expect  $g(k, \alpha, \sigma)$  to be roughly constant in terms of  $\sigma$ , while when  $\alpha$  is close to one, we expect  $g(k, \alpha, \sigma)$  to be roughly linearly in terms of  $\sigma$ . Here, our observations in Figure 4 concur with our discussion in Appendix D.
Figure 4: The evolution of function  $g(k, \alpha, \sigma)$ , as defined in Equation (16) in Appendix F, with respect to  $\sigma$ . Here, we set  $\alpha \in \{0.4, 0.9\}$ ,  $k = 5$  and  $\mathfrak{R} = 1$ .
E.2 Illustration of Upper/Lower Bounds in the Small-  $\alpha$  Regime
In Figure 5, we illustrate the evolution of our upper and lower bounds in the small- $\alpha$  regime (i.e.,  $\alpha \in (0, \bar{\alpha})$ ). Here, we take  $\alpha = 0.4$ , and similar plots can be obtained for different values of  $\alpha < \bar{\alpha}$ . Recall from Section 5 that this regime is considered less challenging due to our result in Theorem 5.1, which states that any algorithm can attain near-optimal performance when  $\alpha \in (0, \bar{\alpha})$ . Here, Figure 5 reveals that our upper/upper bounds indeed have the same trend of increase in terms of  $\sigma$  in the small- $\alpha$  regime, which complements our result in Figure 1 (that shows our upper/lower bounds match in terms of  $\alpha$  in the small- $\alpha$  regime).
Figure 5: Comparison of the order of the regret upper bound in Theorem 5.1 and the order of the lower bound in Theorem 3.1, with respect to  $\sigma$ , obtained with  $k = 5, \alpha = 0.4, \Re = 1, C = 0.4$ . This plot complements Figure 1 that shows dependency on  $\alpha$ .
E.3 Illustrations of Upper/Lower Bounds and Per-Round Regrets of AR2
To further highlight the significance of Theorem 5.2 and how well it characterizes the performance of AR2, in Figure 6, we plot the evolution of the upper bound in Theorem 5.2, the lower bound  $O(k\alpha^2\sigma^2)$  for  $\alpha$  close to one, and the per-round regret attained by AR2, at different values of parameters  $\alpha$  and  $\sigma$  respectively. In Figure 6a, the per-round regret of AR2 increases at roughly the same rate as the upper and lower bounds, and stays close to the lower bound, as  $\alpha$  increases. In Figure 6b, the upper and lower bounds follow the same trend of increase with respect to  $\sigma$ . We note that as  $\sigma$  increases, the upper bound loosens up as the per-round regret of AR2 experiences smaller growth and approaches the worst-case lower bound. This especially highlights the strength of AR2 in adapting to a rapidly changing environment.
(a) Evolution of the upper/ lower bounds and per-round regret of AR2 with varying  $\alpha$  .We take  $\alpha \in [0.85,0.95]$  and  $\sigma = 0.4$
(b) Evolution of the upper/ lower bounds and per-round regret of AR2 with varying  $\sigma$  .We take  $\sigma \in [0.1,0.5]$  and  $\alpha = 0.9$
Figure 6: Comparison of the order of regret upper bound of AR2 (Theorem 5.2), the order of the lower bound, and per-round regret attained by AR2 (averaged across 100 experiments). The constants of upper/ lower bounds are tuned for illustration purposes. Here,  $k = 5$  and  $\mathfrak{Q} = 1$
F Missing Proof of Section 3
Before we show the proof for Theorem 3.1, we first present Lemma F.1 [17] which characterizes the joint distribution for order statistics.
Lemma F.1 (The joint distribution of order statistics of a continuous distribution). Let  $X^{(1)} \leq X^{(2)} \leq \dots \leq X^{(k)}$  be the order statistics of  $k$  i.i.d. samples drawn from some continuous distribution with probability density function (PDF)  $f(x)$  and cumulative density function (CDF)  $F(x)$ . The joint PDF of  $X^{(j)}, X^{(j)}$  for any  $1 \leq i < j \leq k$  is
$$
f _ {X ^ {(i)}, X ^ {(j)}} (u, v) = \frac {k ! F (u) ^ {i - 1} (F (v) - F (u)) ^ {j - i - 1} (1 - F (v)) ^ {k - j} f (u) f (v)}{(i - 1) ! (j - i - 1) ! (k - j) !}
$$
In particular, if we take  $i = k - 1$  and  $j = k$ , Lemma F.1 suggests that
$$
f _ {X ^ {(k - 1)}, X ^ {(k)}} (u, v) = k (k - 1) F (u) ^ {k - 2} f (u) f (v),
$$
which will be used in the following proof of Theorem 3.1.
Proof of Theorem 3.1: The proof proceeds in two steps. First, we compute the probability that the expected rewards of two best arms are close to each other at any given round  $t$ . Then, conditioning on this event, we show that at the next round  $t + 1$ , any algorithm incurs  $\Omega(\alpha \sigma)$  regret with constant probability.
Step 1. We first lower bound the probability that the expected rewards of the two best arms are within  $\alpha \sigma$  of each other at round  $t$ . Recall from Appendix D that  $r_i(t)$  follows the steady state distribution with probability mass  $p$  on the boundary and PDF  $f_p(x)$  defined in (11) within the boundaries. Let  $F(x)$  be the CDF of the steady state distribution of  $r_i(t)$ . We can compute the probability that the expected rewards of the two best arms are within  $\alpha \sigma$  by breaking the event into the following cases:
1. Both arms are at the upper boundary. This will happen with probability  $\sum_{j=2}^{k} \binom{k}{j} p^j (1 - p)^{k-j}$ , where  $j$  denote the number of arms at the upper boundary, i.e., have expected reward  $\mathcal{R}$ .
2. The best arm is at the upper boundary, and the second best arm is within the boundaries. This will happen with the following probability:  $k(k - 1) \cdot p \cdot \int_{\mathbb{R} - \alpha \sigma}^{\mathbb{R}} F(x)^{k - 2} f_p(x) dx$
3. Both the best arm and the second best arm are between the boundaries. This will happen with the following probability:  $k(k - 1)\int_0^{\alpha \sigma}\int_{-\mathcal{R}}^{\mathcal{R}}F(x)^{k - 2}f_p(x)f_p(x + z)dxdz$ . To obtain this probability, we used the joint PDF stated in Lemma F.1.
4. The best arm is within the boundaries, and the second best arm is at the lower boundary. This will happen with the following probability  $k \cdot \int_{-\mathcal{R}}^{\alpha \sigma - \mathcal{Q}} f_p(x) dx \cdot p^{k-1}$ .
5. Both arms are at the lower boundary. This will happen with probability  $p^k$ .
We define  $g(k, \alpha, \sigma)$  as the probability that the expected rewards of the two best arms are within  $\alpha \sigma$  of each other at round  $t$ , which is the sum of the terms above:
$$
\begin{array}{l} g(k,\alpha ,\sigma) = \sum_{j = 2}^{k}\binom {k}{j}p^{j}(1 - p)^{k - j} + k(k - 1)\cdot p\cdot \int_{\mathfrak{R} - \alpha \sigma}^{\mathfrak{R}}F(x)^{k - 2}f_{p}(x)dx \\ + k (k - 1) \int_ {0} ^ {\alpha \sigma} \int_ {- \mathfrak {R}} ^ {\mathfrak {R}} F (x) ^ {k - 2} f _ {p} (x) f _ {p} (x + z) d x d z + k \cdot \int_ {- \mathfrak {R}} ^ {\alpha \sigma - \mathfrak {R}} f _ {p} (x) d x \cdot p ^ {k - 1} + p ^ {k} \tag {16} \\ \end{array}
$$
We now make a brief remark about the order of  $g(k, \alpha, \sigma)$  as  $\alpha$  approaches one. First note that among the aforementioned cases, the first three cases would happen with probability at least  $1/2$  given that the steady state distribution of expected reward is symmetric and the expected rewards of the two best arms are skewed towards the upper boundary. Conditioning on the third case, i.e., both the best and the second best arms are within the boundaries, as  $\alpha$  approaches one, the conditional PDF can be approximated as a uniform function (see Remark G.1 in Appendix G for a related discussion), and hence the conditional probability that two best arms are within  $\alpha \sigma$  of each other are of order  $\Omega(k\alpha\sigma)$ . Further, conditioning on the first and the second cases, i.e., the best arm is at the upper boundary, the conditional probability that two best arms are within  $\alpha \sigma$  of each other are lower bounded by  $\Omega(k\alpha \sigma)$ . We thus have  $g(k, \alpha, \sigma) \geq \Omega(k\alpha \sigma)$  as  $\alpha$  approaches one.
Step 2. Let  $\mathcal{A}$  be any algorithm that selects arm  $\mathcal{A}(t)$  at round  $t$ . In this step, we show that conditioning on the event that the two best arms are within  $\alpha \sigma$  of each other at round  $t$ , with constant probability,  $\mathcal{A}$  will make a mistake at round  $t + 1$  and incur regret of order  $O(\alpha \sigma)$ .
Without loss of generality, let us suppose  $r_1(t) \geq \dots \geq r_k(t)$ , and that algorithm  $\mathcal{A}(t)$  selects arm 1 at round  $t$  (if  $\mathcal{A}(t)$  selects a different arm at round  $t$ , the following proof works similarly). We first assume that  $r_1(t) \in [-\Re + 3\sigma, \Re - 3\sigma]$ . Let  $\Phi(x)$  be the CDF of standard normal distribution. Since  $\epsilon_i(t) \stackrel{i.i.d.}{\sim} N(0, \sigma)$ , when  $\mathcal{A}$  pulls arm 1 at round  $t$ , with constant probability  $2\Phi\left(\frac{1}{2}\right) - 1$ , we have  $|\epsilon_1(t)| \leq \frac{\sigma}{2}$ . Let  $\mathcal{A}(t + 1)$  be the arm that  $\mathcal{A}$  chooses to pull at round  $t + 1$ . At round  $t + 1$ , one of the following scenarios will take place:
(i) If  $\mathcal{A}(t + 1) = 1$ , then if  $\epsilon_2(t)\in [2\sigma ,3\sigma ]$ , it will incur regret
$$
\begin{array}{l} r _ {2} (t + 1) - r _ {1} (t + 1) = \alpha \left(r _ {2} (t) + \epsilon_ {2} (t) - r _ {1} (t) - \epsilon_ {1} (t)\right) \\ \geq \alpha (\epsilon_ {2} (t) - | r _ {2} (t) - r _ {1} (t) | - | \epsilon_ {1} (t) |) \\ \geq \alpha (2 \sigma - \alpha \sigma - \frac {\sigma}{2}) \\ \geq \frac {\alpha \sigma}{2}. \\ \end{array}
$$
The event  $\epsilon_{2}(t) \in [2\sigma, 3\sigma]$  happens with constant probability  $\Phi(3) - \Phi(2)$ .
(ii) If  $\mathcal{A}(t + 1) = j$  for any  $j\neq 1$  at  $t + 1$  , then if  $\epsilon_j(t)\in [-2\sigma , - \sigma ]$  , it will incur regret
$$
\begin{array}{l} r _ {1} (t + 1) - r _ {j} (t + 1) = \alpha \left(r _ {1} (t) + \epsilon_ {1} (t) - r _ {j} (t) - \epsilon_ {j} (t)\right) \\ \geq \alpha \left(\epsilon_ {1} (t) - \epsilon_ {j} (t)\right) \\ \geq \alpha (- \frac {\sigma}{2} + \sigma) \\ = \frac {\alpha \sigma}{2}. \\ \end{array}
$$
The event  $\epsilon_{j}(t)\in [-2\sigma , - \sigma ]$  happens with constant probability  $\Phi (2) - \Phi (1)$
In all of the above cases, we show that  $\mathcal{A}$  will incur regret at least  $\frac{\alpha\sigma}{2}$  with constant probability, conditioning on the event that  $r^{(1)} - r^{(2)} \leq \alpha\sigma$ . If  $r_1(t)$  is close to either side of the boundary, a similar analysis can establish the same result. To see that, if  $r_1 \in [\mathfrak{R} - 3\sigma, \mathfrak{R}]$  and  $\mathcal{A}$  pulls arm 1 at round  $t$ , with constant probability we have  $\epsilon_1(t) \in [-2\sigma, -3\sigma]$ . Then, we have one of the following: (i)  $\mathcal{A}(t + 1) = 1$ . If  $|\epsilon_2(t)| \leq \frac{\sigma}{2}$ , arm 1 is sub-optimal and incurs regret by at least  $\frac{\alpha\sigma}{2}$ ; (ii)  $\mathcal{A}(t + 1) = j$  for some  $j \neq 1$ . If  $\epsilon_j(t) \in [-3\sigma, -4\sigma]$ , arm  $j$  is sub-optimal and incurs regret by at least  $\alpha\sigma$ . This, again, shows that  $\mathcal{A}$  incurs regret at least  $\Omega(\alpha\sigma)$  at round  $t + 1$  with constant probability. The proof follows similarly when  $r_1 \in [-\mathfrak{R}, -\mathfrak{R} + 3\sigma]$ .
Combining step 1 and 2, we conclude that the per-round regret of any algorithm  $\mathcal{A}$  is at least  $\Omega(g(k, \alpha, \sigma)\alpha\sigma)$ , which concludes the proof.
G Missing Proof of Section 5
Proof of Theorem 5.1: At any given round  $t$ , recall from Appendix D that  $r_i(t)$  follows a steady state distribution as defined in Definition D.2. Let us define  $r_i'(t) \stackrel{i.i.d.}{\sim} N(0, \frac{\alpha \sigma}{\sqrt{1 - \alpha^2}})$ . The variable  $r_i'(t)$  has the following PDF:
$$
f ^ {\prime} (x) = \frac {\sqrt {1 - \alpha^ {2}}}{\sqrt {2 \pi} \alpha \sigma} \exp \left(\frac {(\alpha^ {2} - 1) x ^ {2}}{2 \alpha^ {2} \sigma^ {2}}\right)
$$
for all  $x \in \mathbb{R}$ . Note that for all  $0 < s \leq \Re$ , we have that  $\mathbb{P}[|r_i(t)| > s] \leq \mathbb{P}[|r_i'(t)| > s]$  which can be obtained by directly computing both probabilities. Intuitively, one can think of the normal distribution above as the steady state distribution of  $r_i(t)$  that follows the same AR-1 process  $r_i(t + 1) = \alpha (r_i(t) + \epsilon_i(t))$ , without the truncating boundaries at  $[-\Re, \Re]$ . As a result of the truncating boundaries, the steady state distribution  $\mathcal{D}$  (characterized in Appendix D) is more "concentrated" than the normal distribution  $N(0, \frac{\alpha\sigma}{\sqrt{1 - \alpha^2}})$ . Now, using the concentration bound for normal random variable (see Lemma J.2), we have
$$
\mathbb {P} \left[ | r _ {i} (t) | > c \frac {\alpha \sigma}{\sqrt {1 - \alpha^ {2}}} \right] \leq \mathbb {P} \left[ | r _ {i} ^ {\prime} (t) | > c \frac {\alpha \sigma}{\sqrt {1 - \alpha^ {2}}} \right] \leq 2 \exp (- \frac {c ^ {2}}{2})
$$
for every arm  $i \in [k]$ . Now let  $\delta = 2\exp(-\frac{c^2}{2})$ . By the union bound, with probability at least  $1 - k\delta$ , we would have  $|r_i(t)| \leq c\frac{\alpha\sigma}{\sqrt{1 - \alpha^2}}$  for all  $i \in [k]$ . And hence  $\Delta r_i(t) \leq 2c\frac{\alpha\sigma}{\sqrt{1 - \alpha^2}}$  for all  $i \in [k]$ . Let us denote the above event as  $\mathcal{E}$ . If we set  $c = \sqrt{2\log(1 / \alpha\sigma) + 2\log(2k)}$ , we would have  $\delta = \frac{\alpha\sigma}{k}$  and hence  $\mathbb{P}[\mathcal{E}] \geq 1 - k\delta = 1 - \alpha\sigma$ .
Let  $\mathcal{A}$  be any non-anticipating algorithm that selects arm  $\mathcal{A}(t)$  at round  $t$ . The expected regret  $\mathcal{A}$  incurs at round  $t$  is
$$
\begin{array}{l} \mathbb {E} [ \Delta r _ {\mathcal {A} (t)} (t) ] = \mathbb {E} [ \Delta r _ {\mathcal {A} (t)} (t) | \mathcal {E} ] \mathbb {P} [ \mathcal {E} ] + \mathbb {E} [ \Delta r _ {\mathcal {A} (t)} (t) | \mathcal {E} ^ {c} ] \mathbb {P} [ \mathcal {E} ^ {c} ] \leq 2 c \frac {\alpha \sigma}{\sqrt {1 - \alpha^ {2}}} + 2 \Re \cdot \alpha \sigma \\ = O \left(c \frac {\alpha \sigma}{\sqrt {1 - \alpha^ {2}}}\right) = O \left(\sqrt {\log (1 / \alpha \sigma) + \log k)} \frac {\alpha \sigma}{\sqrt {1 - \alpha^ {2}}}\right) \\ \end{array}
$$
Given that the per-round regret is also bounded by  $2\mathfrak{R}$ , this naturally gives
$$
\frac {1}{T} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \Delta r _ {\mathcal {A} (t)} (t) \right] = O \left(\min (\sqrt {\log (1 / \alpha \sigma) + \log k} \frac {\alpha \sigma}{\sqrt {1 - \alpha^ {2}}}, 2 \Re)\right).
$$
In Section 5, we choose  $\bar{\alpha} \triangleq 0.5$  as the threshold value such that when  $\alpha \in [\bar{\alpha}, 1)$ , the problem is more challenging and thereby of our interest. Given the threshold  $\bar{\alpha}$ , we further expand on the remark made in Appendix D, which will be useful in the proof of Lemma 6.4 in Appendix H.