SlowGuess/NeurIPS_2023_OCR huggingface dataset:
and so  $|\hat{\ell}_{t,i}(a) - \ell_{t,i}(a)| \leq \frac{d}{\varepsilon}$ . Therefore since the variance of the estimated losses is a scaled Bernoulli we have that
$$
\operatorname {V a r} \left(\hat {\ell} _ {t, i} (a) - \ell_ {t, i} (a)\right) = \operatorname {V a r} \left(\hat {\ell} _ {t, i} (a)\right) = \mathbf {x} _ {t, i} (a) \left(1 - \mathbf {x} _ {t, i} (a)\right) \left(\frac {\ell_ {t , i} (a)}{\mathbf {x} _ {t , i} (a)}\right) ^ {2} \leq \frac {\ell_ {t , i} ^ {2} (a)}{\mathbf {x} _ {t , i} (a)} \leq \frac {d}{\varepsilon} \tag {53}
$$
We can thus apply a martingale concentration inequality of Fan et al. [23, Corollary 2.1] to the martingale difference sequence (MDS)  $\frac{\varepsilon}{d} (\hat{\ell}_{t,i}(a) - \ell_{t,i}(a))\in [-\frac{\varepsilon}{d},1]$  to obtain
$$
\begin{array}{l} \Pr \left(\sum_ {i = 1} ^ {m} \hat {\ell} _ {t, i} (a) - \ell_ {t, i} (a) \geq \frac {m \Delta_ {a}}{2}\right) = \Pr \left(\frac {\varepsilon}{d} \sum_ {i = 1} ^ {m} \hat {\ell} _ {t, i} (a) - \ell_ {t, i} (a) \geq \frac {\varepsilon m \Delta_ {a}}{2 d}\right) \\ \leq \Pr \left(\max _ {j \in [ m ]} \frac {\varepsilon}{d} \sum_ {i = j} ^ {m} \hat {\ell} _ {t, i} (a) - \ell_ {t, i} (a) \geq \frac {\varepsilon m \Delta_ {a}}{2 d}\right) \\ \leq \exp \left(- \frac {2 \left(\frac {\varepsilon m \Delta_ {a}}{2 d}\right) ^ {2}}{\min  \left\{m (1 + \varepsilon / d) ^ {2} , 4 (\varepsilon m / d + \frac {\varepsilon m \Delta_ {a}}{6}) \right\}}\right) \tag {54} \\ \leq \exp \left(- \frac {2 \left(\frac {\varepsilon m \Delta_ {a}}{2 d}\right) ^ {2}}{4 (\varepsilon m / d + \frac {\varepsilon m \Delta_ {a}}{6})}\right) \\ = \exp \left(- \frac {3 \varepsilon m \Delta_ {a} ^ {2}}{4 d (6 + \Delta_ {a})}\right) \\ \leq \exp \left(- \frac {3 \varepsilon m \Delta_ {a} ^ {2}}{2 8 d}\right) \\ \end{array}
$$
where  $\Delta_{a} = \frac{1}{m}\left|\sum_{i = 1}^{m}\ell_{t,i}(a) - \min_{a^{\prime}\neq a}\sum_{i = 1}^{m}\ell_{t,i}(a^{\prime})\right|$  is the per-arm loss gap in the last step we apply  $\Delta_{a}\leq 1$  . For the symmetric MDS  $-\frac{\varepsilon}{d}\leq \ell_{t,i}(a) - \hat{\ell}_{t,i}(a)\leq 1$  we have
$$
\begin{array}{l} \Pr \left(\sum_ {i = 1} ^ {m} \hat {\ell} _ {t, i} (a) - \ell_ {t, i} (a) \leq - \frac {m \Delta_ {a}}{2}\right) = \Pr \left(\sum_ {i = 1} ^ {m} \ell_ {t, i} (a) - \hat {\ell} _ {t, i} (a) \geq \frac {m \Delta_ {a}}{2}\right) \\ \leq \exp \left(- \frac {2 \left(\frac {m \Delta_ {a}}{2}\right) ^ {2}}{4 \left(\frac {d m}{\varepsilon} + \frac {m \Delta_ {a}}{6}\right)}\right) \tag {55} \\ \leq \exp \left(- \frac {3 \varepsilon m \Delta_ {a} ^ {2} / d}{4 (6 + \varepsilon \Delta_ {a} / d)}\right) \\ \leq \exp \left(- \frac {3 \varepsilon m \Delta_ {a} ^ {2}}{2 8 d}\right) \\ \end{array}
$$
We can then conclude that
$$
\begin{array}{l} \Pr \left(\hat {\mathbf {x}} _ {t} \neq \hat {\mathbf {x}} _ {t}\right) \\ \leq \Pr \left(\exists a \neq \mathring {a} _ {t} t: \sum_ {i = 1} ^ {m} \hat {\ell} _ {t, i} (a) \leq \sum_ {i = 1} ^ {m} \ell_ {t, i} (\mathring {a} _ {t})\right) \\ \leq \Pr \left(\sum_ {i = 1} ^ {m} \hat {\ell} _ {t, i} (\hat {a} _ {t}) \geq \sum_ {i = 1} ^ {m} \ell_ {t, i} (\hat {a} _ {t}) + \frac {m \Delta_ {\hat {a} _ {t}}}{2} \vee \exists a \neq \hat {a} _ {t}: \sum_ {i = 1} ^ {m} \hat {\ell} _ {t, i} (a) \leq \sum_ {i = 1} ^ {m} \ell_ {t, i} (a) - \frac {m \Delta_ {a}}{2}\right) \\ \leq \Pr \left(\sum_ {i = 1} ^ {m} \hat {\ell} _ {t, i} (\hat {a} _ {t}) \geq \sum_ {i = 1} ^ {m} \ell_ {t, i} (\hat {a} _ {t}) + \frac {m \Delta_ {\hat {a} _ {t}}}{2}\right) + \sum_ {a \neq \hat {a} _ {t}} \Pr \left(\sum_ {i = 1} ^ {m} \hat {\ell} _ {t, i} (a) \leq \sum_ {i = 1} ^ {m} \ell_ {t, i} (a) - \frac {m \Delta_ {a}}{2}\right) \\ \leq \exp \left(- \frac {3 \varepsilon m \Delta_ {\tilde {a} _ {t}} ^ {2}}{2 8 d}\right) + \sum_ {a \neq \tilde {a} _ {t}} \exp \left(- \frac {3 \varepsilon m \Delta_ {a} ^ {2}}{2 8 d}\right) \\ \leq d \exp \left(- \frac {3 \varepsilon m \Delta^ {2}}{2 8 d}\right) \tag {56} \\ \end{array}
$$
where the second-to-last line follows by substituting the bounds (54) and (55) into the left and right terms, respectively.
Lemma C.2. Suppose on each task  $t \in [T]$  we run OMD as in Lemma C.1. Then for any  $\beta \in (0,1]$  we have  $\frac{1}{T}\mathbb{E}\sum_{t=1}^{T}\psi_{\beta}(\hat{\mathbf{x}}_t^{(\varepsilon)}) - \psi_{\beta}(\hat{\bar{\mathbf{x}}}^{(\varepsilon)}) \leq -\psi_{\beta}(\frac{\alpha}{\bar{\mathbf{x}}}) + \frac{3d\kappa\beta}{1-\beta}\left(\left(\frac{d}{\varepsilon}\right)^{1-\beta}-1\right)$ .
Proof. We consider the expected divergence of the best initialization under the worst-case distribution of best arm estimation, which satisfies Lemma C.1 and (56). We have by Claim A.1 and the mean-as-minimizer property of Bregman divergences that
$$
\begin{array}{l} \frac {1}{T} \mathbb {E} \sum_ {t = 1} ^ {T} \psi_ {\beta} \left(\hat {\mathbf {x}} _ {t} ^ {(\varepsilon)}\right) - \psi_ {\beta} \left(\hat {\bar {\mathbf {x}}} ^ {(\varepsilon)}\right) = \mathbb {E} \min  _ {\mathbf {x} \in \triangle^ {(\varepsilon)}} \frac {1}{T} \sum_ {t = 1} ^ {T} B _ {\beta} \left(\hat {\mathbf {x}} _ {t} ^ {(\varepsilon)} | | \mathbf {x}\right) \\ \leq \min  _ {\mathbf {x} \in \triangle^ {(\varepsilon)}} \mathbb {E} \frac {1}{T} \sum_ {t = 1} ^ {T} B _ {\beta} \left(\hat {\mathbf {x}} _ {t} ^ {(\varepsilon)} | | \mathbf {x}\right) \\ = \min  _ {\mathbf {x} \in \triangle^ {(\varepsilon)}} \frac {1}{T} \sum_ {t = 1} ^ {T} \sum_ {a = 1} ^ {d} \mathbb {P} (a = \hat {a} _ {t}) B _ {\beta} \left(\mathbf {e} _ {a} ^ {(\varepsilon)} | | \mathbf {x}\right) \\ \leq \max  _ {\substack {\mathbf {p} _ {t} \in \triangle , \forall t \in [ T ] \\ \mathbf {p} _ {t} (a) \leq 2 \kappa , \forall t \in [ T ], a \neq \hat {a} _ {t} \\ 1 - d \kappa \leq \mathbf {p} _ {t} (a), \forall t \in [ T ], a = \hat {a} _ {t}}} \min  _ {\mathbf {x} \in \triangle^ {(\varepsilon)}} \frac {1}{T} \sum_ {t = 1} ^ {T} \sum_ {a = 1} ^ {d} \mathbf {p} _ {t} (a) B _ {\beta} \left(\mathbf {e} _ {a} ^ {(\varepsilon)} \| \mathbf {x}\right) \tag{57} \\ \end{array}
$$
To simplify the last expression, we define  $\bar{\mathbf{p}} = \frac{1}{T}\sum_{t=1}^{T}\mathbf{p}_t$  and again apply the (weighted) mean-as-minimizer property, followed by Claim A.1:
$$
\begin{array}{l} \min  _ {\mathbf {x} \in \triangle^ {(\varepsilon)}} \frac {1}{T} \sum_ {t = 1} ^ {T} \sum_ {a = 1} ^ {d} \mathbf {p} _ {t} (a) B _ {\beta} \left(\mathbf {e} _ {a} ^ {(\varepsilon)} | | \mathbf {x}\right) = \min  _ {\mathbf {x} \in \triangle^ {(\varepsilon)}} \sum_ {a = 1} ^ {d} \bar {\mathbf {p}} (a) B _ {\beta} \left(\mathbf {e} _ {a} ^ {(\varepsilon)} | | \mathbf {x}\right) = \sum_ {a = 1} ^ {d} B _ {\beta} \left(\mathbf {e} _ {a} ^ {(\varepsilon)} | | \bar {\mathbf {p}} ^ {(\varepsilon)}\right) \\ = \psi_ {\beta} \left(\mathbf {e} _ {1} ^ {(\varepsilon)}\right) - \psi_ {\beta} \left(\bar {\mathbf {p}} ^ {(\varepsilon)}\right) \tag {58} \\ \end{array}
$$
By substituting into the previous inequality, we can bound the expected divergence for the worst-case  $\mathbf{p}_t$  as follows:
$$
\begin{array}{l} \frac{1}{T}\mathbb{E}\sum_{t = 1}^{T}\psi_{\beta}(\hat{\mathbf{x}}_{t}^{(\varepsilon)}) - \psi_{\beta}(\hat{\bar{\mathbf{x}}}^{(\varepsilon)})\leq \psi_{\beta}\left(\mathbf{e}_{1}^{(\varepsilon)}\right) + \max_{\substack{\mathbf{p}_{t}\in \triangle ,\forall t\in [T]\\ \mathbf{p}_{t}(a)\leq 2\kappa ,\forall t\in [T],a\neq \tilde{a}_{t}\\ 1 - d\kappa \leq \mathbf{p}_{t}(a),\forall t\in [T],a = \tilde{a}_{t}}} - \psi_{\beta}(\bar{\mathbf{p}}^{(\varepsilon)}) \\ \leq \psi_ {\beta} \left(\mathbf {e} _ {1} ^ {(\varepsilon)}\right) + \quad \max  _ {\sum_ {t = 1} ^ {T} \sum_ {a = 1} ^ {d} \mathbf {p} _ {t} (a) = T} - \psi_ {\beta} \left(\bar {\mathbf {p}} ^ {(\varepsilon)}\right) \tag {59} \\ \sum_ {t = 1} ^ {T} \mathbf {p} _ {t} (a) \geq (1 - d \kappa) \hat {\bar {\mathbf {x}}} (a) T, \forall a \\ \sum_ {t = 1} ^ {T} \mathbf {p} _ {t} (a) \leq (2 \kappa (1 - \stackrel {\circ} {\tilde {\mathbf {x}}} (a)) T + \stackrel {\circ} {\tilde {\mathbf {x}}} (a) T), \forall a \\ = \psi_{\beta}\left(\mathbf{e}_{1}^{(\varepsilon)}\right) - \min_{\substack{\bar{\mathbf{p}}\in \triangle \\ \bar{\mathbf{p}} (a)\geq (1 - d\kappa)^{\frac{\dot{\alpha}}{2}}\bar{\mathbf{x}} (a),\forall a\\ \bar{\mathbf{p}} (a)\leq 2\kappa +(1 - 2\kappa)^{\frac{\dot{\alpha}}{2}}\bar{\mathbf{x}} (a),\forall a}}\psi_{\beta}(\bar{\mathbf{p}}^{(\varepsilon)}) \\ \end{array}
$$
We use the shorthand  $h(\mathbf{x}) = \psi_{\beta}\left((1 - \varepsilon)\mathbf{x} + \frac{\varepsilon}{d}\mathbf{1}_d\right)$ . We have
$$
\begin{array}{l} - \partial_ {\mathbf {x} (a)} (\psi_ {\beta} (\mathbf {x})) = \partial_ {\mathbf {x} (a)} \left(\frac {1}{(1 - \beta)} \left(\sum_ {b = 1} ^ {d} \mathbf {x} (b) ^ {\beta} - 1\right)\right) \\ = \partial_ {\mathbf {x} (a)} \left(\frac {1}{(1 - \beta)} \left(\sum_ {b = 1} ^ {d} \mathbf {x} (b) ^ {\beta} + \beta d ^ {1 - \beta} \left(1 - \sum_ {b = 1} ^ {d} \mathbf {x} (b)\right) - 1\right)\right) \tag {60} \\ = \frac {\beta}{1 - \beta} \cdot \left(\mathbf {x} (a) ^ {\beta - 1} - d ^ {1 - \beta}\right) \\ \end{array}
$$
$$
\begin{array}{l} \| \nabla h (\mathbf {x}) \| _ {\infty} = \max  _ {a = 1, \dots , d} \left| \partial_ {\mathbf {x} (a)} \psi_ {\beta} \left((1 - \varepsilon) \mathbf {x} + \frac {\varepsilon}{d} \mathbf {1} _ {d}\right) \right| \\ \leq \frac {\beta}{1 - \beta} \max  _ {a = 1, \dots , d} | ((1 - \varepsilon) \mathbf {x} (a) + \varepsilon / d) ^ {\beta - 1} - d ^ {1 - \beta} | \tag {61} \\ \leq \frac {\beta}{1 - \beta} \left(\left(\frac {d}{\varepsilon}\right) ^ {1 - \beta} - 1\right) = \beta \log_ {\beta} \left(\frac {d}{\varepsilon}\right) \\ \end{array}
$$
Finally, by convexity of  $h$  we have
$$
\min  _ {\bar {\mathbf {p}} \in \bigtriangleup} \quad h (\bar {\mathbf {p}}) \geq h (\overset {\circ} {\bar {\mathbf {x}}}) - \| \nabla h (\overset {\circ} {\bar {\mathbf {x}}}) \| _ {\infty} \quad \max  _ {\bar {\mathbf {p}} \in \bigtriangleup} \quad \| \bar {\mathbf {p}} - \overset {\circ} {\bar {\mathbf {x}}} \| _ {1}
$$
$$
\bar {\mathbf {p}} (a) \geq (1 - d \kappa) \bar {\hat {\mathbf {x}}} (a), \forall a
$$
$$
\bar {\mathbf {p}} (a) \leq 2 \kappa + (1 - 2 \kappa) \tilde {\bar {\mathbf {x}}} (a), \forall a
$$
$$
\geq h \left(\overset {\circ} {\overline {{\mathbf {x}}}}\right) - 3 d \kappa \| \nabla h \left(\overset {\circ} {\overline {{\mathbf {x}}}}\right) \| _ {\infty} \tag {62}
$$
$$
\geq h (\tilde {\bar {\mathbf {x}}}) - 3 d \kappa \beta \log_ {\beta} \left(\frac {d}{\varepsilon}\right)
$$
so we can substitute into (59) to get
$$
\frac {1}{T} \mathbb {E} \sum_ {t = 1} ^ {T} \psi_ {\beta} \left(\hat {\mathbf {x}} _ {t} ^ {(\varepsilon)}\right) - \psi_ {\beta} \left(\hat {\bar {\mathbf {x}}} ^ {(\varepsilon)}\right) \leq - \psi_ {\beta} \left(\stackrel {\circ} {\bar {\mathbf {x}}} ^ {(\varepsilon)}\right) + \frac {3 d \kappa \beta}{1 - \beta} \left(\left(\frac {d}{\varepsilon}\right) ^ {1 - \beta} - 1\right) \tag {63}
$$
Applying Lemma B.2 completes the proof.
C.2 Guaranteed exploration bounds
Lemma C.3. Suppose we play  $OMD_{\beta, \eta}$  with initialization  $\mathbf{x}_1 \in \triangle^{(\varepsilon)}$ , regularizer  $\psi_{\beta} + I_{\triangle^{(\varepsilon)}}$  for some  $\beta \in (0, 1]$ , and unbiased loss estimators  $(\gamma = 0)$  on the sequence of loss functions  $\ell_1, \ldots, \ell_T \in [0, 1]^d$ . Then for any  $\dot{a} \in [d]$  we have expected regret
$$
\mathbb {E} \sum_ {t = 1} ^ {T} \ell_ {t} \left(a _ {t}\right) - \ell_ {t} (\stackrel {\circ} {a}) \leq \frac {\mathbb {E} B _ {\beta} \left(\hat {\mathbf {x}} ^ {(\varepsilon)} \mid \mid \mathbf {x} _ {1}\right)}{\eta} + \frac {\eta d ^ {\beta} m}{\beta} + \varepsilon m \tag {64}
$$
for  $\hat{\mathbf{x}}$  the estimated optimum of the loss estimators  $\hat{\ell}_1, \dots, \hat{\ell}_T$ .
$$
\begin{array}{l} \mathbb {E} \sum_ {t = 1} ^ {T} \ell_ {t} (a _ {t}) - \ell_ {t} (\mathring {a}) = \mathbb {E} \sum_ {t = 1} ^ {T} \ell_ {t} (a _ {t}) - \langle \ell_ {t}, \mathring {\mathbf {x}} \rangle \\ \leq \mathbb {E} \sum_ {t = 1} ^ {T} \ell_ {t} (a _ {t}) - \langle \ell_ {t}, \mathring {\mathbf {x}} ^ {(\varepsilon)} \rangle + \varepsilon m \\ = \mathbb {E} \sum_ {t = 1} ^ {m} \hat {\ell} _ {t} \left(a _ {t}\right) - \left\langle \hat {\ell} _ {t}, \tilde {\mathbf {x}} ^ {(\varepsilon)} \right\rangle + \varepsilon m \tag {65} \\ \leq \mathbb {E} \sum_ {t = 1} ^ {m} \hat {\ell} _ {t} (a _ {t}) - \langle \hat {\ell} _ {t}, \hat {\mathbf {x}} ^ {(\varepsilon)} \rangle + \varepsilon m \\ \leq \mathbb {E} \left(\frac {B _ {\beta} (\hat {\mathbf {x}} ^ {(\varepsilon)} | | \mathbf {x} _ {1})}{\eta} + \frac {\eta}{\beta} \sum_ {t = 1} ^ {T} \sum_ {a = 1} ^ {d} \hat {\ell} _ {t} ^ {2} (a) \mathbf {x} _ {t} ^ {2 - \beta} (a)\right) + \varepsilon m \\ \leq \frac {\mathbb {E} B _ {\beta} (\hat {\mathbf {x}} ^ {(\varepsilon)} | | \mathbf {x} _ {1})}{\eta} + \frac {\eta d ^ {\beta} m}{\beta} + \varepsilon m \\ \end{array}
$$
where the second inequality follows by optimality of  $\hat{\mathbf{x}}$  for the estimated losses  $\hat{\ell}_t$ , the third by Lemma B.3 constrained to  $\triangle^{(\varepsilon)}$ , and the fourth similarly to Theorem B.1 (note both are also effectively shown in Luo [37]).
Theorem C.1. In Algorithm 1, let  $OMD_{\eta,\beta}$  be online mirror descent with the regularizer  $\psi_{\beta} + I_{\triangle^{(\varepsilon)}}$  over unbiased ( $\gamma = 0$ ) loss estimators,  $\Theta_{k}$  is a subset of  $[\underline{\beta}, \overline{\beta}] \subset [\frac{1}{\log d}, 1]$ , and
$$
U _ {t} (\mathbf {x}, \eta , \beta) = \frac {B _ {\beta} \left(\hat {\mathbf {x}} _ {t} ^ {(\varepsilon)} \right\| \mathbf {x})}{\eta} + \frac {\eta d ^ {\beta} m}{\beta} \tag {66}
$$
where  $\hat{\mathbf{x}}_t^{(\varepsilon)} = (1 - \varepsilon)\hat{\mathbf{x}}_t + \varepsilon \mathbf{1}_d / d$ . Note that  $U_{t}^{(\rho)}(\mathbf{x},\eta ,\beta) = U_{t}(\mathbf{x},\eta ,\beta) + \frac{\rho^{2}(d^{1 - \beta} - 1)}{\eta(1 - \beta)}$ . Then under Assumption 3.1 there exists settings of  $\underline{\eta}$ ,  $\overline{\eta}$ ,  $\alpha$ ,  $\lambda$  s.t. for all  $\varepsilon, \rho \in (0,1)$  we have that
$$
\begin{array}{l} \mathbb {E} \frac {1}{T} \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {m} \ell_ {t, i} \left(a _ {t, i}\right) - \ell_ {t, i} \left(\mathring {a} _ {t}\right) \\ \leq \varepsilon m + \frac {8 d \sqrt {m}}{\rho} \left(1 _ {k > 1} \sqrt {\frac {\log k}{T}} + \frac {1 + \log (T + 1)}{1 6 \rho T}\right) \\ + \min  _ {\beta \in [ \underline {{\beta}}, \bar {\beta} ], \eta > 0} \frac {8 \left(\frac {d}{\varepsilon}\right) ^ {2 - \underline {{\beta}}} (1 + \log T)}{\eta T} + \frac {h _ {\beta} (\Delta)}{\eta} + \frac {\eta d ^ {\beta} m}{\beta} + \frac {L _ {\eta} (\bar {\beta} - \underline {{\beta}})}{2 k} + d \min  \left\{\frac {\rho^ {2}}{2 \eta}, \rho \sqrt {m} \right\} \tag {67} \\ \end{array}
$$
$$
\begin{array}{r l} & {\text {f o r} L _ {\eta} = \left(\frac {\log \frac {d}{\varepsilon}}{\eta} + \eta m \log^ {2} d\right) d a n d h _ {\beta} (\Delta) = (H _ {\beta} + \frac {5 6}{d m}) \iota_ {\Delta} + \frac {d ^ {1 - \beta} - 1}{1 - \beta} (1 - \iota_ {\Delta}) f o r \iota_ {\Delta} = 0,} \\ & {\quad 1 _ {m \geq \frac {7 5 d}{\varepsilon \Delta^ {2}}} \log \frac {d}{\varepsilon \Delta^ {2}}.} \end{array}
$$
Proof. By Lemma C.3 we have
$$
\mathbb {E} \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {m} \ell_ {t, i} \left(a _ {t, i}\right) - \ell_ {t, i} (\hat {a} _ {t}) \leq \varepsilon m T + \mathbb {E} \sum_ {t = 1} ^ {T} \frac {B _ {\beta_ {t}} \left(\hat {\mathbf {x}} _ {t} ^ {(\varepsilon)} \right\rvert   \left| \mathbf {x} _ {t , 1}\right)}{\eta_ {t}} + \frac {\eta_ {t} d ^ {\beta_ {t}} m}{\beta_ {t}} \tag {68}
$$
Since we have the same environment-dependent quantities as in Theorem B.1, we can substitute the above bound into Theorem A.1 and then apply the Lemma C.2 bound
$$
\begin{array}{l} \mathbb {E} \hat {V} _ {\beta} ^ {2} \leq H _ {\beta} + \frac {3 d \kappa \beta}{1 - \beta} \left(\left(\frac {d}{\varepsilon}\right) ^ {1 - \beta} - 1\right) \leq H _ {\beta} + \frac {3 d ^ {2}}{\varepsilon} \exp \left(- \frac {3 \varepsilon \Delta^ {2} m}{2 8 d}\right) \\ = H _ {\beta} + \frac {3 \varepsilon \Delta^ {2}}{d ^ {2}} \exp \left(4 \log \frac {d}{\varepsilon \Delta^ {2}} - \frac {3 \varepsilon \Delta^ {2} m}{2 8 d}\right) \tag {69} \\ \leq H _ {\beta} + \frac {3 \varepsilon \Delta^ {2} / d ^ {2}}{\frac {3 \varepsilon \Delta^ {2} m}{2 8 d} - 4 \log \frac {d}{\varepsilon \Delta^ {2}}} \\ \leq H _ {\beta} + \frac {5 6}{d m} \\ \end{array}
$$
where the last line follows by assuming  $m \geq \frac{75d}{\varepsilon\Delta^2}\log \frac{d}{\varepsilon\Delta^2}$ . If this condition does not hold, then we apply the default bound of  $\mathbb{E}\hat{V}_{\beta}^{2} \leq = \frac{1}{T}\sum_{t=1}^{T}\psi_{\beta}(\hat{\mathbf{x}}_t) - \psi_{\beta}(\hat{\bar{\mathbf{x}}}) \leq \frac{d^{1-\beta}-1}{1-\beta}$ .
Corollary C.1. Let  $\underline{\beta} = \overline{\beta} = 1$ . Then for known  $\Delta$  and assuming  $m \geq \frac{75d}{\Delta^2} \log \frac{d}{\Delta^2}$  we can ensure expected task-averaged regret at most
$$
2 \sqrt {H _ {1} d m + 5 6} + \frac {7 5 d}{\Delta^ {2}} W \left(\frac {m}{7 5}\right) + \tilde {\mathcal {O}} \left(\frac {d ^ {\frac {3}{2}} m ^ {\frac {3}{4}}}{\sqrt {T}} + \frac {d \Delta^ {2} m ^ {2}}{T}\right) \tag {70}
$$
where  $W$  is the Lambert  $W$ -function, while for unknown  $\Delta$  we can ensure expected task-averaged regret at most
$$
2 \sqrt {H _ {1} d m + 5 6} + \frac {3}{\Delta} \sqrt [ 3 ]{5 0 d m \log d \log \frac {d ^ {2} m ^ {2}}{1 5 0 \Delta^ {6} \log d}} + \tilde {\mathcal {O}} \left(\frac {d ^ {\frac {3}{2}} m ^ {\frac {3}{4}}}{\sqrt {T}} + \frac {d ^ {\frac {4}{3}} m ^ {\frac {5}{3}}}{T}\right) \tag {71}
$$
so long as  $m^2 \geq 150d\log d$
Proof. Applying Theorem C.1 and simplifying yields
$$
\varepsilon m + \frac {8 d \sqrt {m} (1 + \log (T + 1))}{1 6 \rho^ {2} T} + \min  _ {\eta > 0} \frac {8 d (1 + \log T)}{\varepsilon \eta T} + \frac {h _ {1} (\Delta)}{\eta} + \eta d m + \frac {d \rho^ {2}}{2 \eta} \tag {72}
$$
Then substitute  $\eta = \sqrt{\frac{h_1(\Delta)}{dm}}$  and set  $\rho = \sqrt[4]{\frac{1}{dT\sqrt{m}}}$  and  $\varepsilon = \frac{75d}{\Delta^2m} W\left(\frac{m}{75}\right)$  (for known  $\Delta$ ) or  $\varepsilon = \sqrt[3]{\frac{150d\log d}{m^2}}$  (otherwise).
Corollary C.2. Let  $\beta = \frac{1}{2}$  and  $\overline{\beta} = 1$ . Then for known  $\Delta$  and assuming  $m \geq \frac{75d}{\Delta^2} \log \frac{d}{\Delta^2}$  we can ensure task-averaged regret at most
$$
\min  _ {\beta \in [ \frac {1}{2}, 1 ]} 2 \sqrt {(H _ {\beta} m + 5 6 / d) d ^ {\beta} / \beta} + \frac {7 5 d}{\Delta^ {2}} W \left(\frac {m}{7 5}\right) + \tilde {\mathcal {O}} \left(\frac {d ^ {\frac {4}{3}} m ^ {\frac {2}{3}}}{\sqrt [ 3 ]{T}} + \frac {d ^ {\frac {5}{3}} m ^ {\frac {5}{6}}}{T ^ {\frac {2}{3}}} + \frac {d \Delta^ {3} m ^ {\frac {5}{2}}}{T}\right) \tag {73}
$$
using  $k = \lceil \sqrt[3]{d^2mT} \rceil$ , while for unknown  $\Delta$  we can ensure expected task-averaged regret at most
$$
\min  _ {\beta \in [ \frac {1}{2}, 1 ]} 2 \sqrt {(H _ {\beta} m + 5 6 / d) d ^ {\beta} / \beta} + \frac {3}{\Delta} \sqrt [ 3 ]{5 0 d ^ {2} m \log \frac {d m ^ {2}}{1 5 0 \Delta^ {6}}} + \tilde {\mathcal {O}} \left(\frac {d ^ {\frac {4}{3}} m ^ {\frac {2}{3}}}{\sqrt [ 3 ]{T}} + \frac {d ^ {\frac {5}{3}} m ^ {\frac {5}{6}}}{T ^ {\frac {2}{3}}} + \frac {d ^ {\frac {3}{2}} m ^ {2}}{T}\right) \tag {74}
$$
so long as  $m \geq 5d\sqrt{6}$ .
$$
\begin{array}{l} \varepsilon m + \frac {8 d \sqrt {m}}{\rho} \left(\sqrt {\frac {\log k}{T}} + \frac {1 + \log (T + 1)}{1 6 \rho T}\right) \\ + \min  _ {\beta \in [ \underline {{\beta}}, \bar {\beta} ], \eta > 0} \frac {8 d ^ {\frac {3}{2}} (1 + \log T)}{\varepsilon^ {\frac {3}{2}} \eta T} + \frac {h _ {\beta} (\Delta)}{\eta} + \frac {\eta d ^ {\beta} m}{\beta} + \frac {d}{4 k} \left(\frac {\log \frac {d}{\varepsilon}}{\eta} + \eta m \log^ {2} d\right) + \frac {d \rho^ {2}}{2 \eta} \tag {75} \\ \end{array}
$$
Then substitute  $\eta = \sqrt{\frac{h_{\beta}(\Delta)}{d^{\beta}m / \beta}}$  and set  $\rho = \sqrt[3]{\frac{1}{d\sqrt{mT}}}$  and  $\varepsilon = \frac{75d}{\Delta^2m} W\left(\frac{m}{75}\right)$  (for known  $\Delta$ ) or  $\varepsilon = \sqrt[3]{\frac{150d^2}{m^2}}$  (otherwise).
Corollary C.3. Let  $\underline{\beta} = \frac{1}{\log d}$  and  $\overline{\beta} = 1$ . Then for known  $\Delta$  and assuming  $m \geq \frac{75d}{\Delta^2} \log \frac{d}{\Delta^2}$  we can ensure task-averaged regret at most
$$
\min  _ {\beta \in (0, 1 ]} 2 \sqrt {\left(H _ {\beta} m + 5 6 / d\right) d ^ {\beta} / \beta} + \frac {7 5 d}{\Delta^ {2}} W \left(\frac {m}{7 5}\right) + \tilde {\mathcal {O}} \left(\frac {d ^ {\frac {4}{3}} m ^ {\frac {2}{3}}}{\sqrt [ 3 ]{T}} + \frac {d ^ {\frac {5}{3}} m ^ {\frac {5}{6}}}{T ^ {\frac {2}{3}}} + \frac {d \Delta^ {4} m ^ {3}}{T}\right) \tag {76}
$$
$$
\min  _ {\beta \in (0, 1 ]} 2 \sqrt {(H _ {\beta} m + 5 6 / d) d ^ {\beta} / \beta} + \frac {3}{\Delta} \sqrt [ 3 ]{5 0 d ^ {2} m \log \frac {d m ^ {2}}{1 5 0 \Delta^ {6}}} + \tilde {\mathcal {O}} \left(\frac {d ^ {\frac {4}{3}} m ^ {\frac {2}{3}}}{\sqrt [ 3 ]{T}} + \frac {d ^ {\frac {5}{3}} m ^ {\frac {5}{6}}}{T ^ {\frac {2}{3}}} + \frac {d ^ {\frac {5}{3}} m ^ {\frac {7}{3}}}{T}\right) \tag {77}
$$
$$
\begin{array}{l} \varepsilon m + \frac {8 d \sqrt {m}}{\rho} \left(\sqrt {\frac {\log k}{T}} + \frac {1 + \log (T + 1)}{1 6 \rho T}\right) \\ + \min  _ {\beta \in [ \underline {{\beta}}, \bar {\beta} ], \eta > 0} \frac {8 d ^ {2} (1 + \log T)}{\varepsilon^ {2} \eta T} + \frac {h _ {\beta} (\Delta)}{\eta} + \frac {\eta d ^ {\beta} m}{\beta} + \frac {d}{2 k} \left(\frac {\log \frac {d}{\varepsilon}}{\eta} + \eta m \log^ {2} d\right) + \frac {d \rho^ {2}}{2 \eta} \tag {78} \\ \end{array}
$$
Corollary C.4. Let  $\underline{\beta} = \frac{1}{\log d}$  and  $\overline{\beta} = 1$ . Then for unknown  $\Delta$  and assuming  $m \geq \max \{d^{\frac{3}{4}}, 56\}$  we can ensure task-averaged regret at most
$$
\min  _ {\beta \in (0, 1 ]} \min  \left\{8 \sqrt {d m}, 2 \sqrt {\left(H _ {\beta} m + \frac {5 6}{d}\right) \frac {d ^ {\beta}}{\beta}} + \frac {2 1 d ^ {\frac {3}{4}} \sqrt [ 3 ]{m}}{\Delta} \sqrt {3 \log \frac {d m}{\Delta^ {2}}} \right\} + \tilde {\mathcal {O}} \left(\frac {d ^ {\frac {4}{3}} m ^ {\frac {2}{3}}}{\sqrt [ 3 ]{T}} + \frac {d ^ {\frac {5}{3}} m ^ {\frac {5}{6}}}{T ^ {\frac {2}{3}}} + \frac {d ^ {2} m ^ {\frac {7}{3}}}{T}\right) \tag {79}
$$
using  $k = \lceil \sqrt[3]{d^2mT}\rceil$
$$
\begin{array}{l} \varepsilon m + \frac {8 d \sqrt {m}}{\rho} \left(\sqrt {\frac {\log k}{T}} + \frac {1 + \log (T + 1)}{1 6 \rho T}\right) \\ + \min  _ {\beta \in [ \underline {{\beta}}, \bar {\beta} ], \eta > 0} \frac {8 d ^ {2} (1 + \log T)}{\varepsilon^ {2} \eta T} + \frac {h _ {\beta} (\Delta)}{\eta} + \frac {\eta d ^ {\beta} m}{\beta} + \frac {d}{2 k} \left(\frac {\log \frac {d}{\varepsilon}}{\eta} + \eta m \log^ {2} d\right) + \frac {d \rho^ {2}}{2 \eta} \tag {80} \\ \end{array}
$$
Then substitute  $\eta = \sqrt{\frac{h_{\beta}(\Delta)}{d^{\beta}m / \beta}}$  and set  $\rho = \sqrt[3]{\frac{1}{d\sqrt{mT}}}$  and  $\varepsilon = \frac{\sqrt{d}}{\sqrt[3]{m^{2}}}$ .
D Robustness to outliers
Proposition D.1. Suppose there exists a constant  $p \in [0, 1]$  and a subset  $S \subset [T]$  of size  $s$  such that  $\mathring{a}_t \in S$  for all but  $\mathcal{O}(T^p)$  MAB tasks  $t \in [T]$ . Then if  $\beta \in [\frac{1}{\log d}, \frac{1}{2}]$  we have  $H_{\beta} = \mathcal{O}(s + \frac{d^{1 - \beta}}{T^{\beta(1 - p)}})$ .
Proof. Define the vector  $\mathbf{e}_S \in [0,1]^d$  s.t.  $\mathbf{e}_{S[a]} = 1_{a \in S}$ . Then by Claim A.1 and the mean-as-minimizer property of Bregman divergences we have