SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \widehat {r} _ {\sup } (t) \geq \alpha \widehat {r} _ {\sup } (t - 1) - 2 c _ {0} \alpha \sigma \quad \text {b y} (36) \\ \geq \alpha^ {t - (\tau + 1)} \widehat {r} _ {\sup } (\tau + 1) - 2 c _ {0} \sigma \left(\alpha + \dots + \alpha^ {t - (\tau + 1)}\right) \quad \text {b y} (36) \\ \geq \alpha^ {t - (\tau + 1)} \widehat {r} _ {\sup } (\tau + 1) - 2 c _ {0} \sigma \left(\alpha + \dots + \alpha^ {4 k - 4}\right) \quad \text {s i n c e} t - \tau \leq 4 k - 3 \\ \geq \alpha^ {t - \tau - 1} \mathcal {B} (\alpha R _ {j} (\tau)) - 2 c _ {0} \sigma \alpha (4 k - 4) \quad \text {s i n c e} \widehat {r} _ {\sup } (\tau + 1) \geq \widehat {r} _ {j} (\tau + 1) = \mathcal {B} (\alpha R _ {j} (\tau)) \\ = \alpha^ {t - \tau - 1} r _ {j} (\tau + 1) - O \left(c _ {0} \sigma \alpha k\right) (37) \\ \end{array}
$$
$$
\begin{array}{l} r ^ {\star} (t) = r _ {j} (t) \leq \alpha^ {t - \tau - 1} r _ {j} (\tau + 1) + \mathcal {F} (t - \tau - 1) \quad \text {b y} (22) \\ = \alpha^ {t - \tau - 1} r _ {j} (\tau + 1) + c _ {0} \sigma \sqrt {\alpha^ {2} + \cdots + \alpha^ {2 (t - \tau - 1)}} (38) \\ \leq \alpha^ {t - \tau - 1} r _ {j} (\tau + 1) + c _ {0} \sigma \alpha \sqrt {t - \tau} \\ = \alpha^ {t - \tau - 1} r _ {j} (\tau + 1) + O \left(c _ {0} \sigma \alpha \sqrt {k}\right) \quad \text {s i n c e} t - \tau \leq 4 k - 3. \\ \end{array}
$$
Combining (37) and (38) then gives  $r^{\star}(t) - \widehat{r}_{\mathrm{sup}}(t)\leq O(c_0\sigma \alpha k)$ . The proof works in a similar fashion for the case when  $t$  is odd.
H.2 Step 3: Distributed regret analysis.
Recall from Definition 6.1 that the distributed regret of arm  $i$  at round  $t$  is defined as
$$
D _ {i} (t) = \left(\frac {\Delta r _ {i} \left(\tau_ {i} (t)\right)}{1 + \alpha^ {2} + \cdots + \alpha^ {2 \left(\Delta \tau_ {i} (t) - 1\right)}}\right) \alpha^ {2 \left(t - \tau_ {i} (t)\right)}. \tag {39}
$$
Before proceeding with the proof of Lemma 6.4, we first establish the following two claims. In Claim H.5 and Claim H.6, we respectively bound the nominator and denominator of  $D_{i}(t)$ , conditioning on the good event  $\mathcal{G}(t)$  and the value of  $\Delta r_{i}(t)$ .
Claim H.5. Suppose we apply AR2 to the non-stationary MAB problem with  $\alpha \in [\bar{\alpha}, 1)$  and  $k \leq \mathcal{K}(\alpha)$ . Given  $\mathcal{G}(t)$ , for any arm  $i \in [k]$ , let  $\eta \triangleq \Delta r_i(t) > 0$  for time  $\tau_i(t) \leq t < \tau_i^{\mathrm{next}}(t)$ , we have that the nominator of (39) satisfies:  $\alpha^{2(t - \tau_i(t))} \Delta r_i(\tau_i(t)) \leq O(\eta + c_0 \sigma \alpha \sqrt{k})$ .
Proof of Claim H.5: For simplicity of notation, let  $\tau = \tau_{i}(t)$  and  $\tau^{\mathrm{next}} = \tau_{i}^{\mathrm{next}}(t)$ . Let  $\tau^{\mathrm{trig}}$  be the first round at which arm  $i$  gets triggered after it gets pulled at round  $\tau$ . We divide our proof into two cases.
Case 1:  $\tau^{\mathrm{trig}} - \tau \leq t - \tau^{\mathrm{trig}}$ . If  $\tau^{\mathrm{trig}} - \tau \leq t - \tau^{\mathrm{trig}} \leq O(k)$ , we must have  $t - \tau = O(k)$ . Then by (22) of Lemma H.2, we have
$$
\begin{array}{l} \left| \Delta r _ {i} (t) - \alpha^ {t - \tau} \Delta r _ {i} (\tau) \right| \leq 2 \mathcal {F} (t - \tau) \leq 2 c _ {0} \sigma \alpha \sqrt {t - \tau} \leq O \left(c _ {0} \sigma \alpha \sqrt {k}\right) \\ \Rightarrow \alpha^ {t - \tau} \Delta r _ {i} (\tau) \leq \eta + O \left(c _ {0} \sigma \alpha \sqrt {k}\right) \\ \Rightarrow \alpha^ {2 (t - \tau)} \Delta r _ {i} (\tau) \leq \alpha^ {t - \tau} O (\eta + c _ {0} \sigma \alpha \sqrt {k}) \leq O (\eta + c _ {0} \sigma \alpha \sqrt {k}). \\ \end{array}
$$
Case 2:  $\tau^{\mathrm{trig}} - \tau > t - \tau^{\mathrm{trig}}$ . Consider time  $t$  such that  $\tau \leq t < \tau^{\mathrm{next}}$ . Then, we have
$$
\begin{array}{l} \widehat {r} _ {\sup } (t) = \max  \left\{\mathcal {B} \left(\alpha R _ {I _ {t - 1}} (t - 1)\right), \alpha \mathcal {B} \left(\alpha R _ {I _ {t - 2}} (t - 2)\right) \right\} = \max  \left\{r _ {I _ {t - 1}} (t), \alpha r _ {I _ {t - 2}} (t - 1) \right\} \\ \leq \max  \left\{r ^ {\star} (t), \alpha r ^ {\star} (t - 1) \right\} \leq \max  \left\{r ^ {\star} (t), r ^ {\star} (t) + 2 c _ {0} \sigma \sqrt {\alpha^ {2}} \right\} \quad \text {b y} (2 3) \\ = r ^ {\star} (t) + 2 c _ {0} \sigma \alpha . \tag {40} \\ \end{array}
$$
Let  $t' = \tau^{\mathrm{trig}} - 1$  denote the round right before the round arm  $i$  gets triggered. Then since at round  $t'$  the arm has not been triggered, we have
$$
\begin{array}{l} 2 4 \mathcal {F} \left(t ^ {\prime} - \tau\right) \leq \widehat {r} _ {\sup } \left(t ^ {\prime}\right) - \widehat {r} _ {i} \left(t ^ {\prime}\right) \\ \leq r ^ {\star} \left(t ^ {\prime}\right) + 2 c _ {0} \sigma \alpha - \alpha^ {t ^ {\prime} - \tau - 1} r _ {i} (\tau + 1) \quad \text {b y} (4 0) \\ \leq \Delta r _ {i} \left(t ^ {\prime}\right) + 2 c _ {0} \sigma \alpha + \mathcal {F} \left(t ^ {\prime} - (\tau + 1)\right) \quad \text {b y (2 2) o f L e m m a H . 2} \\ \leq \Delta r _ {i} \left(t ^ {\prime}\right) + 2 c _ {0} \sigma \alpha + \mathcal {F} \left(t ^ {\prime} - \tau\right) \tag {41} \\ \end{array}
$$
This then gives
$$
\mathcal {F} \left(t ^ {\prime} - \tau\right) \leq \frac {1}{2 3} \left(\Delta r _ {i} \left(t ^ {\prime}\right) + O \left(c _ {0} \sigma \alpha\right)\right) \tag {42}
$$
By (22) of Lemma H.2 and Equation (42), we also have
$$
\alpha^ {t - t ^ {\prime}} \Delta r _ {i} \left(t ^ {\prime}\right) \leq \Delta r _ {i} (t) + 2 \mathcal {F} \left(t - t ^ {\prime}\right) \leq \Delta r _ {i} (t) + \frac {2}{2 3} \Delta r _ {i} \left(t ^ {\prime}\right) + O \left(c _ {0} \alpha \sigma\right) \tag {43}
$$
$$
\alpha^ {2 k - 1} \Delta r _ {i} \left(t ^ {\prime}\right) \leq \alpha^ {t - t ^ {\prime}} \Delta r _ {i} \left(t ^ {\prime}\right), \tag {44}
$$
since  $t - t' = t - \tau^{\mathrm{trig}} + 1 \leq 2(k - 1) + 1 = 2k - 1$ . Now since  $k \leq \mathcal{K}(\alpha) = \frac{1}{2} (\log (\frac{1}{8}) / \log \alpha + 1)$ , we have  $\alpha^{2k - 1} \geq \frac{1}{8}$ , and hence
$$
\left(\frac {1}{8} - \frac {1}{2 3}\right) \Delta r _ {i} \left(t ^ {\prime}\right) \leq \Delta r _ {i} (t) + O \left(c _ {0} \alpha \sigma\right) \tag {45}
$$
Now by (22) of Lemma H.2, we also have
$$
\alpha^ {t ^ {\prime} - \tau} \Delta r _ {i} (\tau) \leq \Delta r _ {i} \left(t ^ {\prime}\right) + 2 \mathcal {F} \left(t ^ {\prime} - \tau\right) \leq \Delta r _ {i} \left(t ^ {\prime}\right) + \frac {2}{2 3} \Delta r _ {i} \left(t ^ {\prime}\right) + O \left(c _ {0} \alpha \sigma\right), \tag {46}
$$
where the last inequality follows from (42) and  $t - t' \leq t' - \tau$ . By multiplying the last equation by  $\alpha^{t - t'}$ , and by applying Equation (45), we then have
$$
\begin{array}{l} \alpha^ {t - \tau} \Delta r _ {i} (\tau) \leq \alpha^ {t - t ^ {\prime}} \left(\Delta r _ {i} (t ^ {\prime}) + \frac {2}{2 3} \Delta r _ {i} (t ^ {\prime}) + O (c _ {0} \alpha \sigma)\right) \leq (1 + \frac {2}{2 3}) \alpha^ {t - t ^ {\prime}} \Delta r _ {i} (t ^ {\prime}) + O (c _ {0} \alpha^ {t - t ^ {\prime} + 1} \sigma) \\ \leq \left(1 + \frac {2}{2 3}\right) \frac {\Delta r _ {i} (t) + O \left(c _ {0} \alpha \sigma\right)}{\frac {1}{8} - \frac {1}{2 3}} + O \left(c _ {0} \alpha^ {t - t ^ {\prime} + 1} \sigma\right) = O \left(\Delta r _ {i} (t) + c _ {0} \alpha \sigma\right) = O (\eta + c _ {0} \alpha) \tag {47} \\ \end{array}
$$
Since  $t - \tau \geq 0$ , we then have  $\alpha^{2(t - \tau)}\Delta r_i(\tau) \leq O(\eta + c_0\alpha \sigma)$ .
In summary of both cases, we have  $\alpha^{2(t - \tau)}\Delta r_i(\tau)\leq O(\eta +c_0\sigma \alpha \sqrt{k})$
Claim H.6. Suppose we apply AR2 to the non-stationary MAB problem with  $\alpha \in [\bar{\alpha},1)$  and  $k\leq \mathcal{K}(\alpha)$ . Given  $\mathcal{G}(t)$ , for any arm  $i\in [k]$ , let  $\eta \triangleq \Delta r_i(t) > 0$  for time  $\tau \leq t < \tau^{next}$ , the denominator of (3) satisfies
$$
1 + \alpha^ {2} + \dots + \alpha^ {2 (\Delta \tau - 1)} = \frac {1 - \alpha^ {2 \Delta \tau}}{1 - \alpha^ {2}} \geq \max  \{\Omega (\frac {\eta^ {2}}{c _ {0} ^ {2} \sigma^ {2} \alpha^ {2} k ^ {2}}), 1 \}.
$$
Proof of Claim H.6 The first equality follows from the geometric sum formula. Since  $1 + \alpha^2 + \dots + \alpha^{2(\Delta \tau - 1)} \geq 1$ , it suffices to show that it is also lower bounded by  $\Omega \left( \frac{\eta^2}{c_0^2 \sigma^2 \alpha^2 k^2} \right)$ . Let  $\tau \leq t < \tau^{\mathrm{next}}$  and recall that  $\eta = \Delta r_i(t)$ . We divide our proof into two cases.
Case 1:  $\alpha^{t - \tau}\Delta r_i(\tau)\leq \eta /2$  . Then by (22) of Lemma H.2, we have
$$
\begin{array}{l} | \eta - \alpha^ {t - \tau} \Delta r _ {i} (\tau) | = | \Delta r _ {i} (t) - \alpha^ {t - \tau} \Delta r _ {i} (\tau) | \leq 2 \mathcal {F} (t - \tau) \\ \Rightarrow 2 \mathcal {F} (t - \tau) \geq \eta / 2 \Rightarrow 2 c _ {0} \sigma \alpha \sqrt {(1 - \alpha^ {2 (t - \tau)}) / (1 - \alpha^ {2})} \geq \eta / 2 \\ \Rightarrow (1 - \alpha^ {2 (t - \tau)}) / (1 - \alpha^ {2}) \geq \Omega (\frac {\eta^ {2}}{c _ {0} ^ {2} \sigma^ {2} \alpha^ {2}}) \\ \end{array}
$$
Since  $\Delta \tau \geq t - \tau$ , we have  $\frac{1 - \alpha^{2\Delta\tau}}{1 - \alpha^2} \geq \frac{1 - \alpha^{2(t - \tau)}}{1 - \alpha^2} \geq \Omega \left( \frac{\eta^2}{c_0^2 \sigma^2 \alpha^2} \right)$ .
Case 2:  $\alpha^{t - \tau}\Delta r_i(\tau) > \eta /2$ . We would show that  $\frac{1 - \alpha^{2\Delta\tau}}{1 - \alpha^2} \geq \Omega \left( \frac{\eta^2}{c_0^2\sigma^2\alpha^2k^2} \right)$ . We have
$$
\begin{array}{l} \widehat {r} _ {\sup } (t) - \widehat {r} _ {i} (t) = \widehat {r} _ {\sup } (t) - \alpha^ {t - \tau} r _ {i} (\tau) \\ \geq \left(r ^ {\star} (t) - O \left(c _ {0} \sigma \alpha k\right)\right) - \alpha^ {t - \tau} \left(r ^ {\star} (\tau) - \Delta r _ {i} (\tau)\right) \quad \text {b y} \\ = \alpha^ {t - \tau} \Delta r _ {i} (\tau) + \left(r ^ {\star} (t) - \alpha^ {t ^ {\prime} - \tau} r ^ {\star} (\tau)\right) - O (c _ {0} \sigma \alpha k) \\ \geq \frac {\eta}{2} - (2 \mathcal {F} (t - \tau) + O \left(c _ {0} \sigma \alpha k\right)) \quad \text {b y} (23) \\ \geq \frac {\eta}{2} - O (k \mathcal {F} (t - \tau)). (48) \\ \end{array}
$$
Now consider  $\tau^{\mathrm{trig}}$ , the round at which arm  $i$  gets triggered. By the triggering criteria, we have
$$
\begin{array}{l} \widehat {r} _ {\sup } \left(\tau^ {\operatorname {t r i g}}\right) - \widehat {r} _ {i} \left(\tau^ {\operatorname {t r i g}}\right) \leq 2 4 \mathcal {F} \left(\tau^ {\operatorname {t r i g}} - \tau\right) \Rightarrow \frac {\eta}{2} - O \left(k \mathcal {F} \left(\tau^ {\operatorname {t r i g}} - \tau\right)\right) \leq 2 4 \mathcal {F} \left(\tau^ {\operatorname {t r i g}} - \tau\right) \quad \text {b y} (48) \\ \Rightarrow O \left(k \mathcal {F} \left(\tau^ {\text {t r i g}} - \tau\right)\right) \geq \frac {\eta}{2} \\ \Rightarrow \frac {\alpha^ {2} - \alpha^ {2 \left(\tau^ {\text {t r i g}} - \tau + 1\right)}}{1 - \alpha^ {2}} \geq O \left(\frac {\eta^ {2}}{c _ {0} ^ {2} \sigma^ {2} k ^ {2}}\right) \quad \text {b y} (21) \\ \Rightarrow \frac {1 - \alpha^ {2 \left(\tau^ {\text {t r i g}} - \tau\right)}}{1 - \alpha^ {2}} \geq O \left(\frac {\eta^ {2}}{c _ {0} ^ {2} \alpha^ {2} \sigma^ {2} k ^ {2}}\right) (49) \\ \end{array}
$$
Since we must have  $\Delta \tau >\tau^{\mathrm{trig}} - \tau$  ,we have
$$
\frac {1 - \alpha^ {2 \Delta \tau}}{1 - \alpha^ {2}} \geq \frac {1 - \alpha^ {2 (\tau^ {\mathrm {t r i g}} - \tau)}}{1 - \alpha^ {2}} \geq \Omega \left(\frac {\eta^ {2}}{c _ {0} ^ {2} \alpha^ {2} \sigma^ {2} k ^ {2}}\right).
$$
Having showed Claim H.5 and Claim H.6, we now formally prove Lemma 6.4.
Proof of Lemma 6.4: Fix arm  $i \in [k]$  and round  $t$ . For simplicity of notation, let  $\tau = \tau_{i}(t)$ ,  $\tau^{\mathrm{next}} = \tau_{i}^{\mathrm{next}}(t)$  and  $\Delta \tau = \Delta \tau_{i}(t)$ . Let us write  $\mathbb{E}[D_i(t)\mathbb{1}_{\mathcal{G}(t)}] = \mathbb{E}[D_i(t)\mathbb{1}_{\mathcal{G}(t)}\mathbb{1}_{\{\Delta r_i(t) = 0\}}] + \mathbb{E}[D_i(t)\mathbb{1}_{\mathcal{G}(t)}\mathbb{1}_{\{\Delta r_i(t) > 0\}}]$ , and in the following analysis, we will bound the two terms on the right-hand side respectively.
First term (Arm  $i$  is optimal at round  $t$ ). We first rewrite the first term as
$$
\mathbb {E} [ D _ {i} (t) \mathbb {1} _ {\mathcal {G} (t)} \mathbb {1} _ {\{\Delta r _ {i} (t) = 0 \}} ] = \mathbb {E} [ D _ {i} (t) | \mathcal {G} (t) \cap \{\Delta r _ {i} (t) = 0 \} ] \cdot \mathbb {P} [ \mathcal {G} (t) \cap \{\Delta r _ {i} (t) = 0 \} ].
$$
Note that if  $\mathcal{G}(t)$  takes place and  $\Delta r_i(t) = 0$ , we have
$$
\alpha^ {t - \tau} \Delta r _ {i} (\tau) = \left| \alpha^ {t - \tau} \Delta r _ {i} (\tau) - \Delta r _ {i} (t) \right| \leq 2 \mathcal {F} (t - \tau). \tag {50}
$$
by (24) in Lemma H.2. By Definition 6.1, we have
$$
\begin{array}{l} D _ {i} (t) = \left(\frac {\Delta r _ {i} (\tau)}{1 + \alpha^ {2} + \cdots + \alpha^ {2 (\Delta \tau - 1)}}\right) \alpha^ {2 (t - \tau)} = \left(\frac {\alpha^ {t - \tau} \Delta r _ {i} (\tau)}{\frac {1}{\alpha^ {2}} (\alpha^ {2} + \cdots + \alpha^ {2 \Delta \tau})}\right) \alpha^ {t - \tau} \\ \leq \left(\frac {2 c _ {0} \sigma \sqrt {\alpha^ {2} + \cdots + \alpha^ {2 (t - \tau)}}}{\alpha^ {2} + \cdots + \alpha^ {2 \Delta \tau}}\right) \alpha^ {t - \tau + 2} \quad \text {b y} (50) \\ \leq \left(\frac {2 c _ {0} \sigma}{\sqrt {\alpha^ {2} + \cdots + \alpha^ {2 (t - \tau)}}}\right) \alpha^ {t - \tau + 2} \quad \text {s i n c e} 1 \leq t - \tau \leq \Delta \tau (51) \\ \end{array}
$$
$$
\mathbb {E} \left[ D _ {i} (t) \mid \mathcal {G} (t) \cap \left\{\Delta r _ {i} (t) = 0 \right\} \right] \leq \left(\frac {2 c _ {0} \sigma}{\sqrt {\alpha^ {2} + \cdots + \alpha^ {2 (t - \tau)}}}\right) \alpha^ {t - \tau + 2}. \tag {52}
$$
Now, let  $j$  be the optimal arm at time  $\tau$ . If  $\mathcal{G}(t)$  takes place and  $\Delta r_i(t) = 0$ , we have
$$
\Delta r _ {j} (t) = \left| \alpha^ {t - \tau} \Delta r _ {j} (\tau) - \Delta r _ {j} (t) \right| \leq 2 \mathcal {F} (t - \tau).
$$
by (24) in Lemma H.2. Hence we have
$$
\begin{array}{l} \mathbb {P} [ \mathcal {G} (t) \cap \left\{\Delta r _ {i} (t) = 0 \right\} ] \leq \mathbb {P} [ \Delta r _ {j} (t) \leq 2 \mathcal {F} (t - \tau), \text {w h e r e} j = \underset {j ^ {\prime}} {\arg \max } r _ {j ^ {\prime}} (\tau) ] \tag {53} \\ \leq \mathbb {P} [ \exists j \neq i \text {s u c h t h a t} \Delta r _ {j} (t) \leq 2 \mathcal {F} (t - \tau) ]. \\ \end{array}
$$
Recall from the discussion in Appendix D and Remark G.1 that the steady state distribution of  $r_i(t)$  can be reasonably approximated by a uniform distribution, and (17) states that the PDF of the steady state distribution  $f(x)$  satisfies  $\rho_{-} \leq f(x) \leq \rho_{+}$  for all  $x \in [-\mathfrak{R}, \mathfrak{R}]$ , where  $\rho_{-}, \rho_{+}$  are constants. Now let  $r_1', \ldots, r_k'$  be  $k$  i.i.d. uniform  $(-1/2\rho_{+}, 1/2\rho_{+})$  random variables. We have
$$
\begin{array}{l} \mathbb {P} [ \exists j \neq i \text {s u c h t h a t} \Delta r _ {j} (t) \leq 2 \mathcal {F} (t - \tau) ] \\ = \mathbb {P} [ \text {t h e t w o b e s t a r m s o f} \left\{r _ {1} (t), \dots , r _ {k} (t) \right\} \text {a r e w i n t h 2} \mathcal {F} (t - \tau) ] \\ \leq \mathbb {P} [ \text {t h e t w o b e s t a r m s o f} \left\{r _ {1} ^ {\prime}, \dots , r _ {k} ^ {\prime} \right\} \text {a r e w i t h i n 2} \mathcal {F} (t - \tau) ] = O (k \cdot \mathcal {F} (t - \tau)) ^ {1 1} \tag {54} \\ \end{array}
$$
Combining (53) and (54) thus gives
$$
\mathbb {P} [ \mathcal {G} (t) \cap \left\{\Delta r _ {i} (t) = 0 \right\} ] \leq O (k \cdot \mathcal {F} (t - \tau)) \tag {55}
$$
Overall, by (52) and (55), for all  $t \geq \tau$  we have
$$
\begin{array}{l} \mathbb {E} \left[ D _ {i} (t) \mathbb {1} _ {\mathcal {G} (t)} \mathbb {1} _ {\{\Delta r _ {i} (t) = 0 \}} \right] = \mathbb {E} \left[ D _ {i} (t) | \mathcal {G} (t) \cap \{\Delta r _ {i} (t) = 0 \} \right] \cdot \mathbb {P} [ \mathcal {G} (t) \cap \{\Delta r _ {i} (t) = 0 \} ] \\ \leq \left(\frac {2 c _ {0} \sigma}{\sqrt {\alpha^ {2} + \cdots + \alpha^ {2 (t - \tau)}}}\right) \alpha^ {t - \tau + 2} \cdot O (k \cdot \mathcal {F} (t - \tau)) = O \left(c _ {0} ^ {2} \sigma^ {2} \alpha^ {2} k\right). \tag {56} \\ \end{array}
$$
Second term (Arm  $i$  is sub-optimal at round  $t$ ). Given  $\mathcal{G}(t)$  and the value of  $\eta \triangleq \Delta r_i(t) > 0$ , we have bounded the nominator and denominator of  $D_{i}(t)$  in Claim H.5 and Claim H.6 respectively. For any time  $\tau \leq t < \tau^{\mathrm{next}}$ , in Claim H.5 we have showed  $\alpha^{2(t - \tau)}\Delta r_i(\tau)\leq O(\eta +c_0\sigma \alpha \sqrt{k})$  and in Claim H.6 we have showed  $1 + \alpha^{2} + \dots +\alpha^{2(\Delta \tau -1)}\geq \max \left\{\Omega (\frac{\eta^{2}}{c_{0}^{2}\sigma^{2}\alpha^{2}k^{2}}),1\right\}$ . These results then give
$$
\mathbb {E} \left[ D _ {i} (t) \mathbb {1} _ {\mathcal {G} (t)} \mathbb {1} _ {\{\Delta r _ {i} (t) > 0 \}} | \eta \right] \lesssim \min  \left\{\frac {\eta + c _ {0} \alpha \sigma \sqrt {k}}{\frac {\eta^ {2}}{c _ {0} ^ {2} \alpha^ {2} \sigma^ {2} k ^ {2}}}, \eta + c _ {0} \alpha \sigma \sqrt {k} \right\}
$$
Let  $\tilde{C} = c_0\alpha \sigma \sqrt{k}$ , this then yields
$$
\mathbb {E} \left[ D _ {i} (t) \mathbb {1} _ {\mathcal {G} (t)} \mathbb {1} _ {\{\Delta r _ {i} (t) > 0 \}} | \eta \right] \lesssim \min  \left\{k \big (\frac {\eta + \tilde {C}}{\frac {\eta^ {2}}{\tilde {C} ^ {2}}} \big), \eta + \tilde {C} \right\} = \min  \left\{k \big (\frac {\tilde {C} ^ {2}}{\eta} + \frac {\tilde {C} ^ {3}}{\eta^ {2}} \big), \eta + \tilde {C} \right\}.
$$
This then further leads to
$$
\mathbb {E} \left[ D _ {i} (t) \mathbb {1} _ {\mathcal {G} (t)} \mathbb {1} _ {\{\Delta r _ {i} (t) > 0 \}} | \eta \right] \lesssim \left\{ \begin{array}{l l} k \frac {\tilde {C} ^ {2}}{\eta} & \mathrm {i f} \eta > \tilde {C} \\ \tilde {C} & \mathrm {i f} 0 <   \eta \leq \tilde {C} \end{array} \right.
$$
We can now bound the second term by taking expectation of the above expression with respect to  $\eta$ . Recall from (17) that the PDF of the steady state distribution of  $r_i(t)$  satisfies  $\rho_{-} \leq f(x) \leq \rho_{+}$  for all  $x \in [-\mathfrak{R}, \mathfrak{R}]$ , where  $\rho_{-}$  and  $\rho_{+}$  are constants. This then gives
$$
\mathbb {E} \left[ D _ {i} (t) \mathbb {1} _ {\mathcal {G} (t)} \mathbb {1} _ {\{\Delta r _ {i} (t) > \tilde {C} \}} \right] \lesssim \int_ {\tilde {C}} ^ {1} k \frac {\tilde {C} ^ {2}}{\eta} d \eta = O \left(k \tilde {C} ^ {2} \log (\tilde {C})\right) = O \left(c _ {0} ^ {2} \alpha^ {2} \sigma^ {2} k ^ {2} \log \left(c _ {0} \alpha \sigma \sqrt {k}\right)\right) \tag {57}
$$
$$
\mathbb {E} \left[ D _ {i} (t) \mathbb {1} _ {\mathcal {G} (t)} \mathbb {1} _ {\{0 <   \Delta r _ {i} (t) \leq \tilde {C} \}} \right] \lesssim \tilde {C} \cdot \mathbb {P} [ 0 <   \Delta r _ {i} (t) \leq \tilde {C} ] = O (\tilde {C} ^ {2}) = O \left(c _ {0} ^ {2} \alpha^ {2} \sigma^ {2} k\right), \tag {58}
$$
which provides an upper bound for the second term.
Finally, summing (56), (57), and (58) gives  $\mathbb{E}[D_i(t)\mathbb{1}_{\mathcal{G}(t)}]\leq O(c_0^2\alpha^2\sigma^2 k^2\log (c_0\alpha \sigma \sqrt{k}))$ , which concludes the proof of Lemma 6.4.
I Missing Proof of Section 8
Proof of Proposition 8.1: Before proceeding, we first make some notational changes that would simplify our proof. Since we pull arm  $i$  consecutively for  $T_{\mathrm{est}}$  rounds for some fixed arm  $i \in [k]$ , in the rest of this proof we omit the dependency of  $R_{i}(t)$  on  $i$  and denote  $R_{t} \triangleq R_{i}(t)$ .
Let us first define the loss function  $\mathcal{L}(\alpha)$  stated in the optimization problem
$$
\widehat {\alpha} = \underset {\alpha \in (0, 1)} {\arg \min } \mathcal {L} (\alpha), \tag {59}
$$
This follows from the fact that if  $X_{(1)} \leq X_{(2)} \leq \dots \leq X_{(k)}$  are the order statistics of  $k$  i.i.d. uniform(0,1) random variables, then  $X_{(k)} - X_{(k-1)}$  follows the Beta(1,k) distribution with CDF  $F(x) = 1 - (1 - x)^{k}$  for  $x \in [0,1]$  (see, e.g., [17]).
which is the negative of the log-likelihood function:
$$
\begin{array}{l} \mathcal {L} (\alpha) = - \frac {1}{T _ {\mathrm {e s t}}} \sum_ {t = 1} ^ {T _ {\mathrm {e s t}}} \mathbb {1} \left\{R _ {t} = \mathfrak {R} \right\} \log \left[ 1 - \Phi \left(\frac {\mathfrak {R} - \alpha R _ {t - 1}}{\sigma}\right) \right] \\ + \mathbb {1} \left\{R _ {t} = - \mathfrak {R} \right\} \log \left[ \Phi \left(\frac {- \mathfrak {R} - \alpha R _ {t - 1}}{\sigma}\right) \right] \tag {60} \\ + \mathbb {1} \left\{- \mathfrak {R} <   R _ {t} <   \mathfrak {R} \right\} \log \left[ \Phi \left(\frac {\mathfrak {R} - \alpha R _ {t - 1}}{\sigma}\right) - \Phi \left(\frac {- \mathfrak {R} - \alpha R _ {t - 1}}{\sigma}\right) \right], \\ \end{array}
$$
where  $\Phi(.)$  is the CDF of standard normal distribution. Note that the three terms in the loss function respectively represent the log likelihood of the observed reward at the upper boundary, at the lower boundary and in between the boundaries.
Recall that  $\widehat{\alpha}$  is the solution to the optimization problem in (59). We can apply the second-order Taylor's Theorem to obtain the following:
$$
\mathcal {L} (\alpha) - \mathcal {L} (\widehat {\alpha}) = - \mathcal {L} ^ {\prime} (\alpha) (\widehat {\alpha} - \alpha) - \frac {1}{2} \mathcal {L} ^ {\prime \prime} (\tilde {\alpha}) (\alpha - \widehat {\alpha}) ^ {2} \tag {61}
$$
for some  $\tilde{\alpha}$  between true AR parameter  $\alpha$  and estimated AR parameter  $\widehat{\alpha}$ . In the rest of this proof, we will provide a bound for  $|\widehat{\alpha} -\alpha |$  via bounding the first and second derivatives of the loss function  $\mathcal{L}(.)$ .
$$
\begin{array}{l} \mathcal {L} ^ {\prime} (\alpha) = - \frac {1}{T _ {\mathrm {e s t}}} \sum_ {t = 1} ^ {T _ {\mathrm {e s t}}} \mathbb {1} \left\{R _ {t} = \mathfrak {R} \right\} \cdot \log^ {\prime} \left[ 1 - \Phi \left(\frac {\mathfrak {R} - \alpha R _ {t - 1}}{\sigma}\right) \right] \cdot \left(\frac {- R _ {t - 1}}{\sigma}\right) \\ + \mathbb {1} \left\{R _ {t} = - \Re \right\} \cdot \log^ {\prime} \left[ \Phi \left(\frac {- \Re - \alpha R _ {t - 1}}{\sigma}\right) \right] \cdot \left(\frac {- R _ {t - 1}}{\sigma}\right) \\ + \mathbb {1} \left\{- \mathfrak {R} <   R _ {t} <   \mathfrak {R} \right\} \cdot \log^ {\prime} \left[ \Phi \left(\frac {\mathfrak {R} - \alpha R _ {t - 1}}{\sigma}\right) - \Phi \left(\frac {- \mathfrak {R} - \alpha R _ {t - 1}}{\sigma}\right) \right] \cdot \left(\frac {- R _ {t - 1}}{\sigma}\right). \\ \end{array}
$$
where  $\log^{\prime}h(y)$  denote the derivative of function  $\log h(y)$  with respect to  $y$ . Let
$$
\begin{array}{l} u _ {t} (\alpha) = \mathbb {1} \left\{R _ {t} = \mathfrak {Q} \right\} \cdot \log^ {\prime} \left[ 1 - \Phi (\frac {\mathfrak {Q} - \alpha R _ {t - 1}}{\sigma}) \right] + \mathbb {1} \left\{R _ {t} = - \mathfrak {Q} \right\} \cdot \log^ {\prime} \left[ \Phi (\frac {- \mathfrak {Q} - \alpha R _ {t - 1}}{\sigma}) \right] \\ + \mathbb {1} \{- \mathfrak {R} <   R _ {t} <   \mathfrak {R} \} \cdot \log^ {\prime} \left[ \Phi \left(\frac {\mathfrak {R} - \alpha R _ {t - 1}}{\sigma}\right) - \Phi \left(\frac {- \mathfrak {R} - \alpha R _ {t - 1}}{\sigma}\right) \right]. \\ \end{array}
$$
Note that  $\mathcal{L}'(\alpha) = \frac{1}{T_{\mathrm{est}}}\sum_{t=1}^{T_{\mathrm{est}}}u_t(\alpha)\cdot\frac{R_{t-1}}{\sigma}$ . To bound the first derivative of the loss function, our next goal is to bound  $\sum_{t=1}^{T_{\mathrm{est}}}u_t(\alpha)$ . Let  $S_j = \sum_{t=1}^{j}u_t(\alpha)$ . Observe that
$$
|S_{j} - S_{j - 1}|\leq \sup_{\substack{y_{1}\in (0,2\mathfrak{R} / \sigma),\\ y_{2}\in -2\mathfrak{R} / \sigma ,0)}}\Bigg\{ \max \Big\{\log^{\prime}[1 - \Phi (y_{1})],\log^{\prime}\Phi (y_{2}),\log^{\prime}(\Phi (y_{1}) - \Phi (y_{2}))\Big\} \Bigg\}.
$$
$$
L _ {1} = \sup  _ {\substack {y _ {1} \in (0, 2 \mathfrak {R} / \sigma), \\ y _ {2} \in - 2 \mathfrak {R} / \sigma , 0)}} \left\{\max  \left\{\log^ {\prime} [ 1 - \Phi (y _ {1}) ], \log^ {\prime} \Phi (y _ {2}), \log^ {\prime} (\Phi (y _ {1}) - \Phi (y _ {2})) \right\} \right\}. \tag{62}
$$