SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Remark G.1 (Uniform Approximation). For any fixed  $\sigma \in (0,1)$ , when  $\alpha \to 1$ , the probability density function that characterizes steady state distribution of  $r_i(t)$  within the boundaries can be well approximated by a constant function. This can be seen from
$$
\max _ {x \in (- \mathfrak {R}, \mathfrak {R})} f _ {p} (x) - \min _ {x \in (- \mathfrak {R}, \mathfrak {R})} f _ {p} (x) = (1 - 2 p) \left[ 1 - \exp \left(\frac {(\alpha - 1) \mathfrak {R} ^ {2}}{\alpha^ {2} \sigma^ {2}}\right) \right] C (\alpha , \sigma) \underset {\alpha \to 1} {\longrightarrow} 0.
$$
In particular, by Lemma D.1, for any  $\alpha \in [\bar{\alpha}, 1)$  and  $\sigma \geq c' \sqrt{1 - \alpha}$ , we can bound the PDF  $f_p(x)$  that characterizes the steady state distribution of  $r_i(t)$  for all  $x \in [-\mathfrak{R}, \mathfrak{R}]$  using constants:
$$
\rho_ {-} \leq f _ {p} (x) \leq \rho_ {+}, \tag {17}
$$
where  $\rho_{-} = (1 - 2p)C(\bar{\alpha},c^{\prime}\sqrt{1 - \bar{\alpha}})\exp \left(-\mathfrak{R}^{2} / \bar{\alpha}^{2}c^{\prime 2}\right),\rho_{+} = (1 - 2p)C(\bar{\alpha},c^{\prime}\sqrt{1 - \bar{\alpha}})$  are both constants.
When  $\alpha \in [\bar{\alpha}, 1)$  and  $\sigma = o(\sqrt{1 - \alpha})$ , the steady state distribution of  $r_i(t)$  can be well approximated by a normal distribution, and similar to what we show in Theorem 5.1, in this case even the naive algorithm performs well. We thus focus on the more challenging regime when  $\alpha \in [\bar{\alpha}, 1)$  and  $\sigma = \Omega(\sqrt{1 - \alpha})$  in the following analysis.
H Missing Proofs of Section 6
H.1 Step 2: Good event and its implications.
In Section 6, we define a new noise term in (5) that shows the influence of truncating boundary:
$$
\tilde {\epsilon} _ {i} (t) = \left\{ \begin{array}{l l} \epsilon_ {i} (t) & \text {i f} \alpha (r _ {i} (t) + \epsilon_ {i} (t)) \in [ - \mathfrak {Q}, \mathfrak {Q} ] \\ \frac {1}{\alpha} [ \mathcal {B} \Big (\alpha (r _ {i} (t) + \epsilon_ {i} (t)) \Big) - \alpha r _ {i} (t) ] & \text {o t h e r w i s e} \end{array} \right.
$$
The new noise  $\tilde{\epsilon}_i(t)$  satisfies the recursive relationship:  $r_i(t + 1) = \alpha r_i(t) + \alpha \tilde{\epsilon}_i(t)$ . Recall that here, the boundary function is defined as  $\mathcal{B}(y) = \min \{\max \{y, -\mathfrak{R}\}, \mathfrak{R}\}$ .
Before proceeding with the proof of Lemma 6.3, we first show an auxiliary lemma that establishes the relationship between  $\tilde{\epsilon}_i(t)$  and  $\epsilon_i(t)$ . In particular, Lemma H.1 shows that if the weighted sum of  $\epsilon_i(t)$  is small with high probability, the weighted sum of  $\tilde{\epsilon}_i(t)$  is also small with high probability.
Lemma H.1. For any  $s > 0$ ,  $i \in [k]$ , and rounds  $t_0 < t_1$ , we have
$$
\mathbb {P} \left[ \left| \sum_ {t = t _ {0}} ^ {t _ {1} - 1} \alpha^ {t _ {1} - t} \tilde {\epsilon} _ {i} (t) \right| \leq s \right] \geq \mathbb {P} \left[ \left| \sum_ {t = t _ {0}} ^ {t _ {1} - 1} \alpha^ {t _ {1} - t} \epsilon_ {i} (t) \right| \leq s \right]. \tag {18}
$$
Proof of Lemma H.1: For simplicity of notation, let us drop the notation for dependence on  $i$ . We first show that the following inequality holds for any  $r(t_0)$ :
$$
\mathbb {P} \left[ \left| \sum_ {t = t _ {0}} ^ {t _ {1} - 1} \alpha^ {t _ {1} - t} \tilde {\epsilon} (t) \right| \leq s \mid r (t _ {0}) \right] \geq \mathbb {P} \left[ \left| \sum_ {t = t _ {0}} ^ {t _ {1} - 1} \alpha^ {t _ {1} - t} \epsilon (t) \right| \leq s \mid r (t _ {0}) \right]. \tag {19}
$$
To show (19), we first consider a fictional scenario in which  $r(.)$  only gets truncated at the first time it hits the boundary, and no longer gets truncated afterwards. Let  $t'$  denote the first round at which  $r(.)$  hits the boundary. With slight abuse of notation, we define corresponding fictional noises  $\epsilon_1(t_0), \ldots, \epsilon_1(t_1 - 1)$  such that  $\epsilon_1(t) = \epsilon(t)$  for all  $t \neq t'$  and  $\epsilon_1(t') = \tilde{\epsilon}(t')$ .
Since the expected reward  $r(.)$  hits the boundary for the first time at  $t'$ , we have  $|\alpha r(t') + \alpha \epsilon(t')| > \mathfrak{R}$  for some  $t_0 \leq t' < t_1$ . Recall that in the fictional scenario,  $\epsilon_1(t) = \epsilon(t)$  for all  $t < t'$ . Let  $L = \alpha^{t' - t_0 + 1} r(t_0)$ ,  $E_1 = \sum_{t=t_0}^{t' - 1} \alpha^{t' - t + 1} \epsilon(t)$ . Without loss of generality, let us assume that the expected reward hits the upper boundary  $\mathfrak{R}$ , i.e.
$$
\mathfrak {R} <   \alpha r (t ^ {\prime}) + \alpha \epsilon (t ^ {\prime}) = L + \sum_ {t = t _ {0}} ^ {t ^ {\prime}} \alpha^ {t ^ {\prime} - t + 1} \epsilon (t) = L + E _ {1} + \alpha \epsilon (t ^ {\prime}).
$$
Let  $d = L + E_1 + \alpha \epsilon(t') - \Re > 0$  be the distance by which the expected reward exceeds the boundary at  $t'$ . By (5) and the definition of the truncating boundary in Section 2, we have
$$
\alpha r \left(t ^ {\prime}\right) + \alpha \epsilon_ {1} \left(t ^ {\prime}\right) = \mathcal {B} \left(\alpha r \left(t ^ {\prime}\right) + \alpha \epsilon \left(t ^ {\prime}\right)\right) = \varnothing .
$$
From the discussion above, we have  $E_{1} + \alpha \epsilon(t') = \Re + d - L$  and  $E_{1} + \alpha \epsilon_{1}(t') = \Re - L$ . Since in the fictional scenario, the expected reward only gets reflected at the first time of hitting the boundary, we also have  $\epsilon_{1}(t) = \epsilon(t)$  for all  $t > t'$ . Let  $E_{2} = \sum_{t=t'+1}^{t_{1}-1} \alpha^{t_{1}-t} \epsilon(t)$  denote the weighted sum of the noises after time of reflection  $t'$ . Now conditioning on the initial expected reward  $r(t_{0})$ , we have
$$
\begin{array}{l} \mathbb {P} \left[ \left| \sum_ {t = t _ {0}} ^ {t _ {1} - 1} \alpha^ {t _ {1} - t} \epsilon_ {1} (t) \right| \leq s \mid r (t _ {0}) \right] = \mathbb {P} \left[ \left| \alpha^ {t _ {1} - t ^ {\prime} - 1} E _ {1} + \alpha^ {t _ {1} - t ^ {\prime}} \epsilon_ {1} (t) + E _ {2} \right| \leq s \mid r (t _ {0}) \right] \\ = \mathbb {P} \left[ - s - \alpha^ {t _ {1} - t ^ {\prime} - 1} (\Re - L) \leq E _ {2} \leq s - \alpha^ {t _ {1} - t ^ {\prime} - 1} (\Re - L) \mid r (t _ {0}) \right] \\ \geq \mathbb {P} \left[ - s - \alpha^ {t _ {1} - t ^ {\prime} - 1} (\mathfrak {A} + d - L) \leq E _ {2} \leq s - \alpha^ {t _ {1} - t ^ {\prime} - 1} (\mathfrak {A} + d - L) \mid r (t _ {0}) \right] \\ = \mathbb {P} \left[ \left| \alpha^ {t _ {1} - t ^ {\prime} - 1} E _ {1} + \alpha^ {t _ {1} - t ^ {\prime}} \epsilon (t) + E _ {2} \right| \leq s \mid r \left(t _ {0}\right) \right] = \mathbb {P} \left[ \left| \sum_ {t = t _ {0}} ^ {t _ {1} - 1} \alpha^ {t _ {1} - t} \epsilon (t) \right| \leq s \mid r \left(t _ {0}\right) \right]. \tag {20} \\ \end{array}
$$
Note that the inequality in (20) holds because  $E_{2}$  is normally distributed (it is a weighted sum of normal noises). If we let  $q_{1} = \alpha^{t_{1} - t^{\prime} - 1}(\mathfrak{A} - L)$  and  $q_{2} = \alpha^{t_{1} - t^{\prime} - 1}(\mathfrak{A} + d - L)$ , we must have  $|q_{1}| < |q_{2}|$ . This, along with the normal distribution of  $E_{2}$ , gives
$$
\mathbb {P} \left[ - s - q _ {1} \leq E _ {2} \leq s - q _ {1} \right] \geq \mathbb {P} \left[ - s - q _ {2} \leq E _ {2} \leq s - q _ {2} \right].
$$
Following the same procedure as above, we can define fictional scenarios in which  $r(.)$  only gets truncated when it hits the boundary for the first  $m$  times, for  $1 \leq m \leq t_1 - t_0$ . We can then define corresponding fictional noises  $\epsilon_m(t_0), \ldots, \epsilon_m(t_1 - 1)$  and use the same proof ideas to show
$$
\mathbb {P} \left[ \left| \sum_ {t = t _ {0}} ^ {t _ {1} - 1} \alpha^ {t _ {1} - t} \epsilon_ {m} (t) \right| \leq s \mid r (t _ {0}) \right] \geq \mathbb {P} \left[ \left| \sum_ {t = t _ {0}} ^ {t _ {1} - 1} \alpha^ {t _ {1} - t} \epsilon_ {m - 1} (t) \right| \leq s \mid r (t _ {0}) \right]
$$
for all  $1 \leq m \leq t_1 - t_0$ . Note that if  $m = t_1 - t_0$ , the fictional rewards  $\epsilon_m = \tilde{\epsilon}$  for all  $t \in [t_0, t_1 - 1]$ . Hence,
$$
\mathbb {P} \Big [ \Big | \sum_ {t = t _ {0}} ^ {t _ {1} - 1} \alpha^ {t _ {1} - t} \tilde {\epsilon} (t) \Big | \leq s \mid r (t _ {0}) \Big ] \geq \mathbb {P} \Big [ \Big | \sum_ {t = t _ {0}} ^ {t _ {1} - 1} \alpha^ {t _ {1} - t} \epsilon (t) \Big | \leq s \mid r (t _ {0}) \Big ].
$$
Finally, since (19) holds for any initial expected reward  $r(t_0)$ , taking expectation with respect to  $r(t_0)$  gives (18), which concludes the proof.
Proof of Lemma 6.3: Let us first consider the sequence of noises  $\epsilon_{i}(t)$  (i.e., the noises before applying the truncating boundary). For any fixed  $i\in [k]$ , and for  $t^{\prime}\in [t - \Delta_{\mathrm{ep}},t + \Delta_{\mathrm{ep}}]$ , we have  $\epsilon_i(t')\stackrel {i,i.d.}{\sim}N(0,\sigma)$ . Then, by Hoeffding's inequality (see Lemma J.3, where for  $1\leq j\leq t_1 - t_0$ , we set  $X_{j} = \alpha^{j}\epsilon_{i}(t_{1} - j)$ ,  $\sigma_{j} = \alpha^{j}\sigma$  and  $s = \frac{c_0\sigma}{t_1 - t_0}\sqrt{\frac{\alpha^2 - \alpha^{2}(t_1 - t_0 + 1)}{1 - \alpha^2}}$ ), for  $t - \Delta_{\mathrm{ep}}\leq t_0 < t_1\leq t + \Delta_{\mathrm{ep}}$ , we have
$$
\mathbb {P} \Big [ \Big | \sum_ {t = t _ {0}} ^ {t _ {1} - 1} \alpha^ {t _ {1} - t} \epsilon_ {i} (t) \Big | > c _ {0} \sigma \sqrt {(\alpha^ {2} - \alpha^ {2 (t _ {1} - t _ {0} + 1)}) / (1 - \alpha^ {2})} \Big ] \leq 2 \exp (- \frac {c _ {0} ^ {2}}{2}) = \frac {(\alpha \sigma) ^ {2}}{2 k \Delta_ {\mathrm {e p}} ^ {2}},
$$
where the last equality follows from  $c_{0} = \sqrt{4\log(1 / \alpha\sigma) + 4\log\Delta_{\mathrm{ep}} + 2\log(4k)}$ . By Lemma H.1, we can use the above inequality to also obtain a concentration inequality for the weighted sum of the new noises  $\tilde{c}_i(t)$ , which incorporate the influence of truncating boundary:
$$
\begin{array}{l} \mathbb {P} \left[ \right.\left| \sum_ {t = t _ {0}} ^ {t _ {1} - 1} \alpha^ {t _ {1} - t} \tilde {\epsilon} _ {i} (t) \right| > c _ {0} \sigma \sqrt {\left( \right.\alpha^ {2} - \alpha^ {2 \left(t _ {1} - t _ {0} + 1\right)} / \left(1 - \alpha^ {2}\right)} \left. \right] \\ \leq \mathbb {P} \left[ \left| \sum_ {t = t _ {0}} ^ {t _ {1} - 1} \alpha^ {t _ {1} - t} \epsilon_ {i} (t) \right| > c _ {0} \sigma \sqrt {\left(\alpha^ {2} - \alpha^ {2 \left(t _ {1} - t _ {0} + 1\right)}\right) / \left(1 - \alpha^ {2}\right)} \right] \leq \frac {\left(\alpha \sigma\right) ^ {2}}{2 k \Delta_ {\mathrm {e p}} ^ {2}}. \\ \end{array}
$$
Since we have  $k$  arms and the number of sub-intervals to consider in  $[t - \Delta_{\mathrm{ep}}, t + \Delta_{\mathrm{ep}}]$  is  $\frac{2\Delta_{\mathrm{ep}}(2\Delta_{\mathrm{ep}} - 1)}{2}$ , we can apply union bounds to get
$$
\begin{array}{l} \mathbb {P} \left[ \mathcal {G} ^ {c} (t) \right] \leq \sum_ {i = 1} ^ {k} \sum_ {t - \Delta_ {\mathrm {e p}} \leq t _ {0} <   t _ {1} \leq t + \Delta_ {\mathrm {e p}}} \mathbb {P} \left[ \right.\left| \sum_ {t = t _ {0}} ^ {t _ {1} - 1} \alpha^ {t _ {1} - t} \tilde {\epsilon} _ {i} (t) \right| > c _ {0} \sigma \sqrt {\left( \right.\alpha^ {2} - \alpha^ {2 \left(t _ {1} - t _ {0} + 1\right)} / \left(1 - \alpha^ {2}\right)} \left. \right\rbrack \\ \leq \sum_ {i = 1} ^ {k} \frac {2 \Delta_ {\mathrm {e p}} (2 \Delta_ {\mathrm {e p}} - 1)}{2} \cdot \frac {(\alpha \sigma) ^ {2}}{2 k \Delta_ {\mathrm {e p}} ^ {2}} \leq \sum_ {i = 1} ^ {k} 2 \Delta_ {\mathrm {e p}} ^ {2} \cdot \frac {(\alpha \sigma) ^ {2}}{2 k \Delta_ {\mathrm {e p}} ^ {2}} = (\alpha \sigma) ^ {2}. \\ \end{array}
$$
Having showed that good event  $\mathcal{G}(t)$  happens with high probability, we now establish a number of important implications of the good event:
(i) In Lemma H.2, we show that the expected reward  $r_i(t)$  satisfies a series of concentration inequalities. These inequalities guarantee that our estimated reward  $\widehat{r}_i(t)$  does not deviate too much from the true expected reward  $r_i(t)$ .
(ii) In Lemma H.3, we show that AR2 does not wait too long to play an arm with high expected reward.
(iii) In Lemma H.4, we show that the best expected reward  $r^{\star}(t)$  is not too far away from our estimate reward for the superior arm  $\widehat{r}_{\mathrm{sup}}(t)$ .
These implications will be critical for our proof of Lemma 6.4. Before we proceed, let us define the following:
$$
\mathcal {F} (\Delta t) \triangleq c _ {0} \sigma \sqrt {\alpha^ {2} + \alpha^ {4} + \cdots + \alpha^ {2 \Delta t}} = c _ {0} \sigma \sqrt {\left(\alpha^ {2} - \alpha^ {2 (\Delta t + 1)}\right) / (1 - \alpha^ {2})}, \tag {21}
$$
where the second equality follows from the geometric sum formula. The expression (21) will be used throughout the rest of the analysis. In particular, since  $c_{1} = 24c_{0}$ , the triggering criteria becomes: at round  $t$ , an arm  $i \neq i_{\mathrm{sup}}(t)$  gets triggered if the following is satisfied:
$$
\widehat {r} _ {\sup } (t) - \widehat {r} _ {i} (t) \leq c _ {1} \sigma \sqrt {(\alpha^ {2} - \alpha^ {2 (t - \tau_ {i} (t) + 1)}) / (1 - \alpha^ {2})} = 2 4 \mathcal {F} (t - \tau_ {i} (t)).
$$
Lemma H.2. Given  $\mathcal{G}(t)$ , for any arm  $i\in [k]$  and any rounds  $t_0,t_1$  such that  $t - \Delta_{ep}\leq t_0 < t_1\leq t + \Delta_{ep}$ , the expected rewards satisfy the following inequalities:
$$
\left| r _ {i} \left(t _ {1}\right) - \alpha^ {t _ {1} - t _ {0}} r _ {i} \left(t _ {0}\right) \right| \leq \mathcal {F} \left(t _ {1} - t _ {0}\right) \tag {22}
$$
$$
\left| r ^ {\star} \left(t _ {1}\right) - \alpha^ {t _ {1} - t _ {0}} r ^ {\star} \left(t _ {0}\right) \right| \leq \mathcal {F} \left(t _ {1} - t _ {0}\right) \tag {23}
$$
$$
\left| \Delta r _ {i} \left(t _ {1}\right) - \alpha^ {t _ {1} - t _ {0}} \Delta r _ {i} \left(t _ {0}\right) \right| \leq 2 \mathcal {F} \left(t _ {1} - t _ {0}\right) \tag {24}
$$
Proof of Lemma H.2 Given the AR-1 model and (5), we have the following recursive relationship
$$
\begin{array}{l} r _ {i} (t _ {1}) = \alpha \left[ r _ {i} (t _ {1} - 1) + \tilde {\epsilon} _ {i} (t _ {1} - 1) \right] \\ = \alpha \left[ \alpha \left(r _ {i} \left(t _ {1} - 2\right) + \tilde {\epsilon} _ {i} \left(t _ {1} - 2\right)\right) + \tilde {\epsilon} _ {i} \left(t _ {1} - 1\right) \right] \\ = \alpha^ {t _ {1} - t _ {0}} r _ {i} \left(t _ {0}\right) + \sum_ {t = t _ {0}} ^ {t _ {1} - 1} \alpha^ {t _ {1} - t} \tilde {\epsilon} _ {i} (t) \tag {25} \\ \end{array}
$$
by repeating the above steps. Then by Lemma 6.3, we have that
$$
\left| r _ {i} \left(t _ {1}\right) - \alpha^ {t _ {1} - t _ {0}} r _ {i} \left(t _ {0}\right) \right| = \left| \sum_ {t = t _ {0}} ^ {t _ {1} - 1} \alpha^ {t _ {1} - t} \tilde {\epsilon} _ {i} (t) \right| \leq \mathcal {F} \left(t _ {1} - t _ {0}\right),
$$
which concludes the proof for (22).
As for (23), note that
$$
r ^ {\star} (t _ {1}) = \max  _ {i \in [ k ]} r _ {i} (t _ {1}) \leq \max  _ {i \in [ k ]} \left\{\alpha^ {t _ {1} - t _ {0}} r _ {i} (t _ {0}) + \mathcal {F} (t _ {1} - t _ {0}) \right\} \leq \alpha^ {t _ {1} - t _ {0}} r ^ {\star} (t _ {0}) + \mathcal {F} (t _ {1} - t _ {0})
$$
and the other side of the inequality can be derived similarly.
The last inequality (24) easily follows from (22), (23), and the triangle inequality
$$
\begin{array}{l} \left| \Delta r _ {i} \left(t _ {1}\right) - \alpha^ {t _ {1} - t _ {0}} \Delta r _ {i} \left(t _ {0}\right) \right| \leq \left| \left(r ^ {\star} \left(t _ {1}\right) - r _ {i} \left(t _ {1}\right)\right) - \alpha^ {t _ {1} - t _ {0}} \left(r ^ {\star} \left(t _ {0}\right) - r _ {i} \left(t _ {0}\right)\right) \right| \\ \leq \left| r ^ {\star} \left(t _ {1}\right) - \alpha^ {t _ {1} - t _ {0}} r ^ {\star} \left(t _ {0}\right) \right| + \left| r _ {i} \left(t _ {1}\right) - \alpha^ {t _ {1} - t _ {0}} r _ {i} \left(t _ {0}\right) \right| \leq 2 \mathcal {F} \left(t _ {1} - t _ {0}\right). \\ \end{array}
$$
This concludes the proof of Lemma H.2.
Lemma H.3. Suppose we apply AR2 to the non-stationary MAB problem with  $\alpha \in [\bar{\alpha}, 1)$  and  $k \leq \mathcal{K}(\alpha)$ , given  $\mathcal{G}(t)$ , the following holds for the best arm  $i$  at round  $t$ :  $t - \tau_i(t) \leq 4k - 3$ .
Proof of Lemma H.3 Suppose  $\alpha \in [\bar{\alpha}, 1)$ . Let arm  $i$  be the best arm at round  $t$ , i.e.  $i = \arg \max_{j} r_{j}(t)$ . We let  $\tau = \tau_{i}(t)$ , and let  $\tau^{\mathrm{trig}} \geq \tau$  be the triggering time of arm  $i$  after it gets pulled at round  $\tau$ . First note that by design of AR2, we must have  $t - \tau^{\mathrm{trig}} \leq 2(k - 1)$ , since at any given round there are at most  $k - 1$  triggered arms. We would prove the statement of the lemma by showing the following inequality:
$$
\tau^ {\text {t r i g}} - \tau \leq t - \tau^ {\text {t r i g}} + 1 \tag {26}
$$
If (26) holds, we then have  $t - \tau = (\tau^{\mathrm{trig}} - \tau) + (t - \tau^{\mathrm{trig}})\leq 2(t - \tau^{\mathrm{trig}}) + 1\leq 4k - 3.$
Suppose by contradiction that (26) does not hold. By the triggering criteria, we have
$$
\widehat {r} _ {\sup } \left(\tau^ {\operatorname {t r i g}} - 1\right) - \widehat {r} _ {i} \left(\tau^ {\operatorname {t r i g}} - 1\right) > 2 4 \cdot \mathcal {F} \left(\left(\tau^ {\operatorname {t r i g}} - 1\right) - \tau\right) \tag {27}
$$
Let  $j = i_{\sup}(\tau^{\mathrm{trig}} - 1)$  as defined in (1). Hence we have  $\widehat{r}_{\sup}(\tau^{\mathrm{trig}} - 1) = r_j(\tau^{\mathrm{trig}} - 1)$  or  $\widehat{r}_{\sup}(\tau^{\mathrm{trig}} - 1) = \alpha r_j(\tau^{\mathrm{trig}} - 2)$ . Let us assume  $\widehat{r}_{\sup}(\tau^{\mathrm{trig}} - 1) = r_j(\tau^{\mathrm{trig}} - 1)$  and the proof follows similarly for the other case. Since we also have  $\widehat{r}_i(\tau^{\mathrm{trig}} - 1) = \alpha^{\tau^{\mathrm{trig}} - \tau - 2}\mathcal{B}(\alpha X_i(\tau)) = \alpha^{\tau^{\mathrm{trig}} - \tau - 2}r_i(\tau + 1)$ , (27) can be rewritten as
$$
r _ {j} \left(\tau^ {\operatorname {t r i g}} - 1\right) - \alpha^ {\tau^ {\operatorname {t r i g}} - \tau - 2} r _ {i} (\tau + 1) > 2 4 \cdot \mathcal {F} \left(\left(\tau^ {\operatorname {t r i g}} - 1\right) - \tau\right) \tag {28}
$$
By (22) of Lemma H.2, we also have the following two inequalities:
$$
r _ {i} (t) \leq \alpha^ {t - (\tau + 1)} r _ {i} (\tau + 1) + \mathcal {F} (t - (\tau + 1)) \tag {29}
$$
$$
r _ {j} (t) \geq \alpha^ {t - (\tau^ {\operatorname {t r i g}} - 1)} r _ {j} \left(\tau^ {\operatorname {t r i g}} - 1\right) - \mathcal {F} \left(t - \left(\tau^ {\operatorname {t r i g}} - 1\right)\right) \tag {30}
$$
Then by (29) and (30), we have
$$
\begin{array}{l} r _ {j} (t) - r _ {i} (t) \geq \left(\alpha^ {t - (\tau^ {\operatorname {t r i g}} - 1)} r _ {j} (\tau^ {\operatorname {t r i g}} - 1) - \mathcal {F} (t - (\tau^ {\operatorname {t r i g}} - 1))\right) - \left(\alpha^ {t - (\tau + 1)} r _ {i} (\tau + 1) + \mathcal {F} (t - (\tau + 1))\right) \\ \geq \alpha^ {t - \tau^ {\operatorname {t r i g}} + 1} \left(r _ {j} \left(\tau^ {\operatorname {t r i g}} - 1\right) - \alpha^ {\tau^ {\operatorname {t r i g}} - \tau - 2} r _ {i} (\tau + 1)\right) - \mathcal {F} (t - \left(\tau^ {\operatorname {t r i g}} - 1\right)) - \mathcal {F} (t - (\tau + 1)) \\ \geq 2 4 \cdot \alpha^ {t - \tau^ {\text {t r i g}} + 1} \mathcal {F} \left(\left(\tau^ {\text {t r i g}} - 1\right) - \tau\right) - \mathcal {F} \left(t - \left(\tau^ {\text {t r i g}} - 1\right)\right) - \mathcal {F} (t - (\tau + 1)), \tag {31} \\ \end{array}
$$
where the last inequality follows from (28). Since  $k \leq \mathcal{K}(\alpha)$ , we have  $\alpha^{t - \tau^{\mathrm{trig}} + 1} \geq \alpha^{2k - 1} \geq \frac{1}{8}$  which yields
$$
r _ {j} (t) - r _ {i} (t) \geq 2 4 \cdot \frac {1}{8} \cdot \mathcal {F} \left(\tau^ {\text {t r i g}} - \tau - 1\right) - \mathcal {F} \left(t - \tau^ {\text {t r i g}} + 1\right) - \mathcal {F} (t - \tau - 1).
$$
Since  $\tau^{\mathrm{trig}} - \tau \geq t - \tau^{\mathrm{trig}} + 2$ , we have  $\mathcal{F}(\tau^{\mathrm{trig}} - 1 - \tau) \geq \mathcal{F}(t - \tau^{\mathrm{trig}} + 1)$ . This further yields
$$
r _ {j} (t) - r _ {i} (t) \geq \left(2 4 \cdot \frac {1}{8} - 1\right) \cdot \mathcal {F} \left(\tau^ {\text {t r i g}} - \tau - 1\right) - \mathcal {F} (t - \tau - 1) = 2 \mathcal {F} \left(\tau^ {\text {t r i g}} - \tau - 1\right) - \mathcal {F} (t - \tau - 1). \tag {32}
$$
Note that if  $\tau^{\mathrm{trig}} - \tau > t - \tau^{\mathrm{trig}} + 1$ , then since  $t - \tau = (t - \tau^{\mathrm{trig}}) + (\tau^{\mathrm{trig}} - \tau)$ , we must have  $2(\tau^{\mathrm{trig}} - \tau) > t - \tau + 1$ , and thus
$$
\begin{array}{l} 2 \mathcal {F} (\tau^ {\mathrm {t r i g}} - \tau - 1) = 2 c _ {0} \sigma \sqrt {\alpha^ {2} + \cdots + \alpha^ {2 (\tau^ {\mathrm {t r i g}} - \tau - 1)}} \\ > 2 c _ {0} \sigma \sqrt {\alpha^ {2} + \dots + \alpha^ {t - \tau - 1}} \\ > c _ {0} \sigma \left(\sqrt {\alpha^ {2} + \cdots + \alpha^ {t - \tau - 1}} + \sqrt {\alpha^ {t - \tau + 1} + \cdots + \alpha^ {2 (t - \tau) - 2}}\right) \\ \geq c _ {0} \sigma \sqrt {\alpha^ {2} + \dots + \alpha^ {2 (t - \tau) - 2}} \quad \text {s i n c e} \sqrt {a} + \sqrt {b} \geq \sqrt {a + b} \\ = \mathcal {F} (t - \tau - 1). \\ \end{array}
$$
Plugging this into (32) gives  $r_j(t) - r_i(t) > 0$ , which contradicts that arm  $i$  is the best arm at  $t$ . Hence, we must have  $\tau^{\mathrm{trig}} - \tau \leq t - \tau^{\mathrm{trig}} + 1$ . By our previous argument, this then gives  $t - \tau \leq 4k - 3$ .
Lemma H.4. Suppose we apply AR2 to the non-stationary MAB problem with  $\alpha \in [\bar{\alpha}, 1)$  and  $k \leq \mathcal{K}(\alpha)$ , for any arm  $i \in [k]$  and round  $t$ , we have
$$
r ^ {*} (t) - \widehat {r} _ {\sup } (t) \leq O \left(c _ {0} \sigma \alpha k\right), \tag {33}
$$
where  $\widehat{r}_{sup}(t) = \widehat{r}_{i_{sup}(t)}(t)$  and  $i_{sup}(t)$  is defined in (1).
Proof of Lemma H.4: Without loss of generality, suppose that  $t$  is even, and as a result  $I_{t} = i_{\sup}(t)$ . The proof then proceeds in two steps. In step one, we establish a recursive inequality for the estimated reward of the superior arm. In step two, we show that the estimated reward of the superior arm is in fact close to the expected reward of the best arm.
Step 1: Establish a recursive inequality for  $\widehat{r}_{\mathrm{sup}}(t)$ . We first assume that  $i_{\mathrm{sup}}(t) = I_{t - 2}$  and hence  $\widehat{r}_{\mathrm{sup}}(t) = \alpha \mathcal{B}(\alpha R_{i_{\mathrm{sup}}(t)}(t - 2)) = \alpha r_{i_{\mathrm{sup}}(t)}(t - 1)$ . We also have
$$
r _ {i _ {\mathrm {s u p}} (t)} (t) = \alpha r _ {i _ {\mathrm {s u p}} (t)} (t - 1) + \alpha \epsilon_ {i _ {\mathrm {s u p}} (t)} (t - 1) = \widehat {r} _ {\mathrm {s u p}} (t) + \alpha \epsilon_ {i _ {\mathrm {s u p}} (t)} (t - 1),
$$
which then gives
$$
r _ {i _ {\sup } (t)} (t) \geq \widehat {r} _ {\sup } (t) - \alpha c _ {0} \sigma \tag {34}
$$
by definition of the good event  $\mathcal{G}(t)$  (set  $t_0 = t$  and  $t_1 = t + 1$ ). Since  $I_{t} = i_{\mathrm{sup}}(t)$ , we additionally have
$$
\widehat {r} _ {\sup } (t + 1) \geq \mathcal {B} \left(\alpha R _ {i _ {\sup } (t)} (t)\right) = \alpha \left(r _ {i _ {\sup } (t)} (t) + \epsilon_ {i _ {\sup } (t)} (t)\right) \geq \alpha r _ {i _ {\sup } (t)} (t) - \alpha c _ {0} \sigma \tag {35}
$$
Combining (34) and (35) gives us the following inequality, which is the desired result.
$$
\widehat {r} _ {\sup } (t + 1) \geq \alpha \left(\widehat {r} _ {\sup } (t) - \alpha c _ {0} \sigma\right) - \alpha c _ {0} \sigma \geq \alpha \widehat {r} _ {\sup } (t) - 2 c _ {0} \alpha \sigma . \tag {36}
$$
On the other hand, if  $i_{\mathrm{sup}}(t) = I_{t - 1}$ , we have  $r_{i_{\mathrm{sup}}(t)}(t) = \mathcal{B}(\alpha R_{i_{\mathrm{sup}}(t)}(t - 1)) = \widehat{r}_{\mathrm{sup}}(t)$ , and combining this with (35) also gives
$$
\widehat {r} _ {\sup } (t + 1) \geq \alpha r _ {i _ {\sup } (t)} (t) - \alpha c _ {0} \sigma = \alpha \widehat {r} _ {\sup } (t) - \alpha c _ {0} \sigma \geq \alpha \widehat {r} _ {\sup } (t) - 2 c _ {0} \alpha \sigma .
$$
Overall, we show that  $\widehat{r}_{\mathrm{sup}}(t + 1) \geq \alpha \widehat{r}_{\mathrm{sup}}(t) - 2c_0 \alpha \sigma$ , which completes the proof of the first step.
Step 2: Show that  $\widehat{r}_{\mathrm{sup}}(t)$  and  $r^\star (t)$  are close. For simplicity of notation, let  $j$  be the best arm at round  $t$ , so  $r^{\star}(t) = r_{j}(t)$ . Let  $\tau = \tau_{j}(t)$ . Recall that by Lemma H.3, we have that  $t - \tau \leq 4k - 3 = O(k)$ . First note that