SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 3.2. When using the Shannon entropy regularizer with  $C_{\mathrm{LOG}} = 162 \log K$ ,  $\alpha = 1$ , and  $\theta = \sqrt{1 / \log T}$ , Algorithm 1 ensures  $\operatorname{Reg}^T = \mathcal{O}\left(\sqrt{KT} \log T\right)$  always, and simultaneously the following regret bound under Condition (1):  $\operatorname{Reg}^T = \mathcal{O}\left(\mathbb{G}(\log T)^2 + \sqrt{C\mathbb{G}(\log T)^2} + K^2 \log^{3/2} T + D\right)$ .
The recent work [Ito et al., 2022] is the first to discover that Shannon entropy, used in the very first adversarial MAB algorithm EXP3 [Auer et al., 2002b], in fact also achieves BOBW when configured and analyzed properly (assuming uniqueness). Their results are for the more general setting of MAB with a feedback graph, and when specified to standard MAB, their regret bound under Condition (1) is worse than ours with  $\mathbb{G}$  defined as  $K / \Delta_{\mathrm{MIN}}$ . The key of our improvement comes from our very different and arm-dependent learning rate schedule. Note that there are extra log  $T$  factors in the regret for both the adversarial setting and the stochastic setting, which is also the case in [Ito et al., 2022]. While this makes the bounds worse compared to other regularizers, in the more general setting with a feedback graph, Shannon entropy is known to be critical for achieving the right dependence on the independence number of the feedback graph, and we believe that our results shed light on how to remove the uniqueness requirement in this more general setting using Shannon entropy.
Finally, we present our results for Tsallis entropy.
Theorem 3.3. For any  $\beta \in (0,1)$ , when using the  $\beta$ -Tsallis entropy regularizer with  $C_{\mathrm{LOG}} = \frac{162\beta}{1 - \beta}$ ,  $\alpha = \beta$ , and  $\theta = \sqrt{(1 - \beta) / \beta}$ , Algorithm 1 ensures  $\operatorname{Reg}^T = \mathcal{O}\left(\sqrt{\frac{1}{\beta(1 - \beta)}KT(\log T)^{\mathbb{I}\left\{\beta \neq \frac{1}{2}\right\}}}\right)$  always, and simultaneously the following regret bound under Condition (1):  $\operatorname{Reg}^T = \mathcal{O}\left(\frac{\mathbb{G}\log T}{\beta(1 - \beta)} + \sqrt{\frac{C\mathbb{G}\log T}{\beta(1 - \beta)}} + D + \frac{K^2\sqrt{\beta}}{(1 - \beta)^{3/2}} + \frac{\beta K\log T}{1 - \beta}\right)$ .
When  $\beta = 1/2$ , our learning rate  $\gamma_i^t$  simply becomes  $\sqrt{t}$  (which is arm-independent), and our algorithm exactly recovers Tsallis-INF [Zimmert and Seldin, 2021]. In this case, our result is essentially the same as what the improved analysis of [Ito, 2021] shows, which does not require
Ito [2021] also provides other data-dependent bounds in the adversarial setting, which we do not consider here. Ignoring this part, his algorithm is also slightly different from ours, but we note in passing that our analysis technique also applies if one sets  $\nu_{i}^{t} = p_{i}^{t}$  in his algorithm.
uniqueness. For  $\beta \neq 1/2$ , while such regularizers were also analyzed in [Zimmert and Seldin, 2021] (under uniqueness), their algorithm is infeasible since the learning rates are tuned based on the unknown  $\Delta_i$ 's. On the other hand, our algorithm not only uses a simple and feasible learning rate schedule, but also works without uniqueness. The bound for the adversarial setting has an extra  $\sqrt{\log T}$  factor when  $\beta \neq 1/2$  though, which we conjecture can be removed (as it is the case when using a fixed learning rate [Audibert and Bubeck, 2009, Abernethy et al., 2015]); see Remark C.2.2. We find it surprising that our learning rate exhibits totally different behavior when  $\beta < 1/2$  versus when  $\beta > 1/2$  (recall that  $\alpha$  is set to  $\beta$ ): in the former,  $\gamma_i^t$  increases in the previous  $p_i^\tau$  ( $\tau < t$ ), while in the latter, it decreases in  $p_i^\tau$ .
It might not be clear at this point what the value is to consider  $\beta \neq 1/2$  — after all, our bounds are minimized when  $\beta = 1/2$ . It turns out that, however, other values of  $\beta$  play important roles in other problems, as for example demonstrated by Rouyer and Seldin [2020] in a decoupled exploration and exploitation setting. Below, we generalize our results to this setting, showcasing the broad applicability of our techniques.
Decoupled exploration and exploitation The Decoupled Exploration and Exploitation MAB (DEE-MAB) problem, first considered in [Avner et al., 2012], is a variant of MAB where in each round  $t$ , the learner picks an arm  $i^t$  to exploit and an arm  $j^t$  to explore, and then suffers the loss  $\ell_{it}^{t}$  while observing the feedback  $\ell_{jt}^{t}$ . The performance of the learner is still measured by the same regret definition in terms of the exploitation arms  $i^1, \ldots, i^T$ . The standard MAB can be seen as a special case where  $i^t$  and  $j^t$  must be the same. In DEE-MAB, it turns out that the adversarial setting is as difficult as standard MAB with a lower bound  $\Omega(\sqrt{KT})$ , but one can do much better in the stochastic setting with a  $T$ -independent regret bound. For example, [Rouyer and Seldin, 2020] uses FTRL with  $2/3$ -Tsallis entropy to achieve  $\mathcal{O}(\sqrt{KT})$  in the adversarial setting and simultaneously  $\mathcal{O}\left(\sqrt{\sum_{i \in V} \frac{K}{\Delta_i \cdot \Delta_{\mathrm{MIN}}}}\right)$  in the i.i.d. setting assuming a unique optimal arm.
Using our techniques, we not only remove the uniqueness requirement, but also improve their bounds. Specifically, we consider the exact same algorithm as theirs, which can be described using the framework of Algorithm 1: take  $C_{\mathrm{LOG}} = 0$ ,  $\theta = K^{1/6}$ ,  $\alpha = 1/2$ , and  $\beta = 2/3$  for the Tsallis entropy regularizer; sample the exploitation arm  $i^t$  according to  $p^t$  as before and the exploration arm  $j^t$  according to a different distribution  $g^t$  with  $g_i^t \propto (p_i^t)^{2/3}$ ; finally, construct the importance-weighted estimator using the exploration information:  $\widehat{\ell}_i^t = \frac{\mathbb{I}\{j^t = i\} \ell_i^t}{g_i^t}$ . Our results are as follows.
Theorem 3.4. For the DEE-MAB problem, the algorithm described above ensures  $\mathrm{Reg}^T = \mathcal{O}(\sqrt{KT})$  always, and simultaneously the following regret bound under Condition (1):  $\mathrm{Reg}^T = \mathcal{O}\Big(\sqrt{\sum_{i\in V}\frac{K}{\Delta_i^2}} +\sqrt{C}\cdot \Big(\sum_{i\in V}\frac{K}{\Delta_i^2}\Big)^{1 / 4} + K + D\Big)$ .
Note that in the i.i.d. setting (where  $C = D = 0$ ), we improve their bound from  $\mathcal{O}\left(\sqrt{\sum_{i\in V}\frac{K}{\Delta_i\cdot\Delta_{\mathrm{MIN}}}}\right)$  to  $\mathcal{O}\left(\sqrt{\sum_{i\in V}\frac{K}{\Delta_i^2}}\right)$  (in addition to removing the uniqueness assumption).
In this section, we take  $\beta$ -Tsallis entropy as an example to illustrate the key ideas of our analysis in proving the MAB results under Condition (1). As in all prior work, the analysis relies on a so-called self-bounding technique. Specifically, our goal is to bound the regret as follows (ignoring all minor terms, including the dependence on  $\beta$ ):
$$
\operatorname {R e g} ^ {T} \lesssim \mathbb {E} \left[ \sum_ {i \in V} \sqrt {\left(\log T\right) \sum_ {t = 1} ^ {T} p _ {i} ^ {t}} + \sqrt {\left| U \right| (\log T) \sum_ {i \in V} \sum_ {t = 1} ^ {T} p _ {i} ^ {t}} \right], \tag {3}
$$
where the two terms above enjoy a self-bounding property since they can be related back to the regret under Condition (1). To see this, we apply AM-GM inequality followed by Condition (1) to show the
following for any  $z\geq 0$  ..
$$
\mathbb {E} \left[ \sum_ {i \in V} \sqrt {\log T \sum_ {t = 1} ^ {T} p _ {i} ^ {t}} \right] \leq \mathbb {E} \left[ \sum_ {i \in V} \left(\frac {\log T}{4 z \Delta_ {i}} + z \sum_ {t = 1} ^ {T} p _ {i} ^ {t} \Delta_ {i}\right) \right] \leq z (\operatorname {R e g} ^ {T} + C) + \sum_ {i \in V} \frac {\log T}{4 z \Delta_ {i}},
$$
$$
\mathbb {E} \left[ \sqrt {\left| U \right| \log T \sum_ {i \in V} \sum_ {t = 1} ^ {T} p _ {i} ^ {t}} \right] \leq \mathbb {E} \left[ \frac {\left| U \right| \log T}{4 z \Delta_ {\mathrm {M I N}}} + z \sum_ {i \in V} \sum_ {t = 1} ^ {T} p _ {i} ^ {t} \Delta_ {\mathrm {M I N}} \right] \leq z (\operatorname {R e g} ^ {T} + C) + \frac {\left| U \right| \log T}{4 z \Delta_ {\mathrm {M I N}}}.
$$
Rearranging and picking the optimal  $z$  yields the regret bound under Condition (1) in Theorem 3.3.
The key is thus to prove Eq. (3), which is not that difficult under uniqueness (when  $|V| = K - 1$ ) but turns out to be much more complicated without uniqueness. To proceed, we start with some key concepts and ideas from [Ito, 2021] which we follow. First, define the skewed Bregman divergence for two time steps  $s, t \in \mathbb{N}$  as
$$
D ^ {s, t} (x, y) = \phi^ {s} (x) - \phi^ {t} (y) - \left\langle \nabla \phi^ {t} (y), x - y \right\rangle , \tag {4}
$$
and its variant restricted to any subset  $\mathcal{I} \subseteq [K]$  as  $D_{\mathcal{I}}^{s,t}(x,y) = \phi_{\mathcal{I}}^s (x) - \phi_{\mathcal{I}}^t (y) - \langle \nabla \phi_{\mathcal{I}}^t (y),x - y\rangle$  where  $\phi_{\mathcal{I}}^t (x) = -C_{\mathrm{Log}}\sum_{i\in \mathcal{I}}\log x_i - \frac{1}{1 - \beta}\sum_{i\in \mathcal{I}}\gamma_i^t x_i^\beta$  (that is,  $\phi^t$  restricted to  $\mathcal{I}$ ). The standard Bregman divergence associated with  $\phi^t$ , which we denote by  $D^{t}(x,y)$ , is then a shorthand for  $D^{t,t}(x,y)$ . One key idea of [Ito, 2021] is to carefully choose the right benchmark for the algorithm — when there is a unique optimal arm, the benchmark basically has to be this unique optimal arm, but when multiple optimal arms exist, the benchmark can now be any distribution over these arms, and it can even be varying over time. Indeed, for round  $t$ , the following benchmark was used in [Ito, 2021]:
$$
q ^ {t} = \underset {p \in \Omega_ {U}} {\operatorname {a r g m i n}} \left\{\left\langle p, \sum_ {\tau <   t} \hat {\ell} ^ {\tau} \right\rangle + \phi^ {t} (p) \right\} = \underset {p \in \Omega_ {U}} {\operatorname {a r g m i n}} D ^ {t} (p, p ^ {t}), \tag {5}
$$
which follows the same definition of  $p^t$  but is restricted to  $\Omega_U = \{p \in \Omega_K : \sum_{i \in U} p_i = 1\}$ , the set of distributions over the zero-gap arms. As the second equality shows,  $q^t$  is also the projection of  $p^t$  onto  $\Omega_U$  w.r.t. the Bregman divergence  $D^t$ . With these time-varying benchmarks, Ito [Ito, 2021] proves
$$
\operatorname {R e g} ^ {T} \lesssim \mathbb {E} \left[ \sum_ {t = 1} ^ {T} D ^ {t, t + 1} \left(p ^ {t}, p ^ {t + 1}\right) - D _ {U} ^ {t, t + 1} \left(q ^ {t}, q ^ {t + 1}\right) \right] + D. \tag {6}
$$
The rest of the analysis is where we start to deviate from that of [Ito, 2021] (thought still largely inspired by it), which is critical for our algorithms that use arm-dependent learning rates. First, we introduce an important intermediate point  $\bar{p}^{t + 1} = \bar{p}_U^{t + 1} + \bar{p}_V^{t + 1}$  where  $\bar{p}_U^{t + 1}$  and  $\bar{p}_V^{t + 1}$  are defined as
$$
\begin{array}{l} \bar{p}_{U}^{t + 1} = \operatorname *{argmin}_{x\in \mathbb{R}_{>0}^{K}, \sum_{i\in V}x_{i} = 0,}\left\langle x,\sum_{\tau <  t}\widehat{\ell}^{\tau}\right\rangle +\phi_{U}^{t + 1}(x), \\ \sum_ {i \in U} x _ {i} = \sum_ {i \in U} p _ {i} ^ {t} \tag {7} \\ \end{array}
$$
$$
\vec{p}_{V}^{t + 1} = \operatorname *{argmin}_{x\in \mathbb{R}_{\geq 0}^{K}, \sum_{i\in U}x_{i} = 0,}\left\langle x,\sum_{\tau \leq t}\widehat{\ell}^{\tau}\right\rangle +\phi_{V}^{t + 1}(x).
$$
$$
\sum_ {i \in V} x _ {i} = \sum_ {i \in V} p _ {i} ^ {t}
$$
By definition,  $\bar{p}^{t + 1}$  is obtained from  $p^t$  by redistributing the weights among arms in  $U$  and those in  $V$ , in a way that minimizes an FTRL objective similar to that of  $p^{t + 1}$ . We note that Ito [2021] also uses the same  $\bar{p}_U^{t + 1}$  is his analysis, but we introduce  $\bar{p}_V^{t + 1}$  (and thus  $\bar{p}^{t + 1}$ ) as well since it importantly allows us to decompose each Bregman divergence difference term in Eq. (6) as follows.
Lemma 4.1. For any  $t$ ,  $D^{t,t+1}(p^t, p^{t+1}) - D_U^{t,t+1}(q^t, q^{t+1})$  is bounded by
$$
\underbrace {D _ {V} ^ {t , t + 1} \left(p ^ {t} , \bar {p} ^ {t + 1}\right)} _ {\text {r e g r e t o n s u b - o p t i m a l a r m s}} + \underbrace {D _ {U} ^ {t , t + 1} \left(p ^ {t} , \bar {p} ^ {t + 1}\right) - D _ {U} ^ {t , t + 1} \left(q ^ {t} , q ^ {t + 1}\right)} _ {\text {r e g r e t o n o p t i m a l a r m s}} + \underbrace {D ^ {t + 1} \left(\bar {p} ^ {t + 1} , p ^ {t + 1}\right)} _ {\text {r e s i d u a l r e g r e t}}. \tag {8}
$$
In the rest of this section, we proceed to bound each of the three terms in Eq. (44) (see also Table 3 for a summary of bounds for each of these terms and each of the three regularizers).
Regret on Sub-Optimal Arms The regret related to the sub-optimal arms (or more formally arms in  $V$ ) is the most straightforward to deal with, since our objective is to arrive at the self-bounding terms in Eq. (3) which are exactly only in terms of arms in  $V$ . Indeed, we can write this term as (with  $p_V^t$  being the vector with the same value as  $p^t$  for coordinates in  $V$  and 0 for coordinates in  $U$ )
$$
D _ {V} ^ {t, t + 1} \left(p ^ {t}, \bar {p} ^ {t + 1}\right) = \underbrace {\left\langle p _ {V} ^ {t} - \bar {p} _ {V} ^ {t + 1} , \hat {\ell} ^ {t} \right\rangle - D _ {V} ^ {t} \left(\bar {p} ^ {t + 1} , p ^ {t}\right)} _ {\text {s t a b i l i t y}} + \underbrace {\phi_ {V} ^ {t} \left(\bar {p} ^ {t + 1}\right) - \phi_ {V} ^ {t + 1} \left(\bar {p} ^ {t + 1}\right)} _ {\text {p e n a l t y}}, \tag {9}
$$
and then apply standard arguments to show that the stability term is of order  $\sum_{i\in V}\left(p_i^t\right)^{1 - \beta} / \beta \gamma_i^{t + 1}$  while the penalty term is of order  $\sum_{i\in V}\left(\gamma_i^{t + 1} - \gamma_i^t\right)\left(p_i^t\right)^\beta /1 - \beta$  . In the Tsallis-INF algorithm, we have  $\beta = 1 / 2$  and  $\gamma_i^t = \sqrt{t}$ , and thus the stability term and the penalty term are of the same order. This inspires us to design a learning rate for general  $\beta$  with the same objective. Indeed, it can be verified that our particular learning rate schedule makes sure that the penalty term is of the same order as the stability term, meaning  $D_V^{t,t + 1}(p^t,\bar{p}^{t + 1}) = \mathcal{O}\left(\sum_{i\in V}\left(p_i^t\right)^{1 - \beta} / \beta \gamma_i^{t + 1}\right)$ . Further plugging in the learning rate and summing over  $t$ , we arrive at the following with  $z_i^t = \max \left\{p_i^t,1 / T\right\}$ :
$$
\sum_ {t = 1} ^ {T} \sum_ {i \in V} \frac {\left(p _ {i} ^ {t}\right) ^ {1 - \beta}}{\beta \gamma_ {i} ^ {t + 1}} \leq \sqrt {\frac {1}{\beta (1 - \beta)}} \sum_ {i \in V} \sum_ {t = 1} ^ {T} \frac {\left(z _ {i} ^ {t}\right) ^ {1 - \beta}}{\sqrt {1 + \sum_ {k = 1} ^ {t} \left(z _ {i} ^ {k}\right) ^ {1 - 2 \beta}}}. \tag {10}
$$
Finally, applying the following technical lemma shows that the above is of the same order as the first term in our objective Eq. (3). More details can be found in Section A.2.
Lemma 4.2. Let  $\{x_{t}\}_{t = 1}^{T}$  be a sequence with  $x_{t} > 0$  for all  $t$ . Then, for any  $\alpha \in [0,1]$ , we have
$$
\sum_ {t = 1} ^ {T} \frac {x _ {t} ^ {1 - \alpha}}{\sqrt {1 + \sum_ {s = 1} ^ {t} x _ {s} ^ {1 - 2 \alpha}}} \leq \mathcal {O} \left(\sqrt {\left(\sum_ {t = 1} ^ {T} x _ {t}\right) \log \left(1 + \sum_ {t = 1} ^ {T} x _ {t} ^ {1 - 2 \alpha}\right)}\right). \tag {11}
$$
Regret on Optimal Arms Next, we show that the regret on optimal arms (or more formally arms in  $U$ ),  $D_U^{t,t+1}(p^t, \bar{p}^{t+1}) - D_U^{t,t+1}(q^t, q^{t+1})$ , is nonpositive, which corresponds to the intuition that pulling optimal arms incur no regret. Ito [2021] proves something similar for  $1/2$ -Tsallis entropy via a certain monotonicity property of Bregman divergence, but his proof is highly specific to  $1/2$ -Tsallis entropy. Instead, we develop the following general monotonicity theorem which applies to a broad spectrum of regularizers as long as they satisfy two mild conditions.
Theorem 4.3 (Monotonicity of Bregman divergence). For any  $t \in \mathbb{N}$ , let  $f^t: \Omega \to \mathbb{R}$  be a continuously-differentiable and strictly-convex function defined on  $\Omega \subseteq \mathbb{R}$ . Suppose that the following two conditions hold for all  $z \in \Omega$ : (i)  $(f^t)'(z)$  is differentiable and concave; (ii)  $(f^{t+1})'(z) \leq (f^t)'(z)$ . Then, for any  $x, m \in \mathbb{R}$  with  $x \leq m$ , and  $y, n \in \mathbb{R}$  such that  $(f^{t+1})'(y) - (f^t)'(x) = (f^{t+1})'(n) - (f^t)'(m) = \xi$  for a fixed scalar  $\xi$ , we have  $D^{t,t+1}(x,y) \leq D^{t,t+1}(m,n)$ , where  $D^{t,t+1}(u,v) = f^t(u) - f^{t+1}(v) - (u - v) \cdot (f^{t+1})'(v)$  is the skewed Bregman divergence.
While we state the theorem for the one-dimensional case, it trivially extends to multi-dimensional regularizers as long as they decompose over the coordinates (which is the case for all our regularizers).
Take Tsallis entropy as an example: we only need to apply the theorem with  $f^t(z) = -\frac{\gamma_i^t z^\beta}{1 - \beta}$  for each  $i$  and then sum up the conclusions. The two conditions stated in the theorem also hold for all regularizers we consider. In particular, Condition (ii) holds as long as the learning rate  $\gamma_i^t$  is non-decreasing in  $t$  and the regularizer itself is non-increasing (thus with nonpositive first derivative). This explains the additional linear term in our definition of Shannon entropy: this way it is strictly decreasing. Note that Condition (ii) also trivially holds if  $f^t$  is independent of  $t$ , in which case the theorem states the monotonicity for the standard (non-skewed) Bregman divergence.
After verifying the conditions, we can now apply this theorem to show  $D_U^{t,t+1}(p^t,\bar{p}^{t+1}) \leq D_U^{t,t+1}(q^t,q^{t+1})$ . For each  $i \in U$ , we take  $x = p_i^t$  and  $m = q_i^t$ . Since by definition  $q^t$  is obtained by projecting  $p^t$  onto  $\Omega_U$ , it can be shown via KKT conditions that  $p_i^t \leq q_i^t$  indeed holds for all  $i \in U$ . Then, we define an intermediate point  $z$  such that  $\nabla \phi_U^{t+1}(z) - \nabla \phi_U^t(p^t) = \nabla \phi_U^{t+1}(q^{t+1}) - \nabla \phi_U^t(q^t)$  and show  $D_U^{t,t+1}(p^t,\bar{p}^{t+1}) \leq D_U^{t,t+1}(p^t,z)$ . Finally, taking  $y = z_i$  and  $n = q_i^{t+1}$  and applying Theorem 4.3 finishes the proof; see Section A.3 for details.
Residual Regret Finally, bounding the residual regret  $D^{t + 1}(\bar{p}^{t + 1}, p^{t + 1})$  by the self-bounding terms in Eq. (3) that are only in terms of arms in  $V$  is another key challenge in our analysis, especially given the arm-dependent learning rates. We start by developing a new analysis that leads to tighter bounds compared to [Ito, 2021] on the Lagrangian multipliers associated with Eq. (7), which reveals that the key to analyze  $D^{t + 1}(\bar{p}^{t + 1}, p^{t + 1})$  is to bound the following term (or terms of a similar form)
$$
\frac {\left(\sum_ {i \in V} \frac {\left(p _ {i} ^ {t}\right) ^ {2 - \beta}}{\gamma_ {i} ^ {t}}\right) \left(\sum_ {i \in U} \frac {\left(p _ {i} ^ {t}\right) ^ {3 - 2 \beta}}{\left(\gamma_ {i} ^ {t}\right) ^ {2}}\right)}{\left(\sum_ {i \in [ K ]} \frac {\left(p _ {i} ^ {t}\right) ^ {2 - \beta}}{\gamma_ {i} ^ {t}}\right) \left(\sum_ {i \in U} \frac {\left(p _ {i} ^ {t}\right) ^ {2 - \beta}}{\gamma_ {i} ^ {t}}\right)}. \tag {12}
$$
Again, for the case of  $1/2$ -Tsallis entropy with an arm-independent learning rate  $\gamma_i^t = \sqrt{t}$ , removing all dependence on  $i \in U$  in Eq. (12) is relatively straightforward as shown by Ito [2021]. Indeed, in this case, Eq. (12) simplifies to  $\frac{1}{\sqrt{t}} \frac{\left( \sum_{i \in V} \left( p_i^t \right)^{3/2} \right) \left( \sum_{i \in U} \left( p_i^t \right)^2 \right)}{\left( \sum_{i \in [K]} \left( p_i^t \right)^{3/2} \right) \left( \sum_{i \in U} \left( p_i^t \right)^{3/2} \right)}$ . By splitting  $\sum_{i \in [K]} \left( p_i^t \right)^{3/2}$  into two summations, one over  $i \in V$  and another over  $i \in U$ , and further applying  $x + y \geq x^{2/3}y^{1/3}$  for any  $x, y > 0$ , we have  $\sum_{i \in [K]} \left( p_i^t \right)^{3/2} \geq \left( \sum_{i \in V} \left( p_i^t \right)^{3/2} \right)^{2/3} \left( \sum_{i \in U} \left( p_i^t \right)^{3/2} \right)^{1/3}$  and thus Eq. (12) is bounded by
$$
\frac {1}{\sqrt {t}} \frac {\left(\sum_ {i \in V} \left(p _ {i} ^ {t}\right) ^ {3 / 2}\right) \left(\sum_ {i \in U} \left(p _ {i} ^ {t}\right) ^ {2}\right)}{\left(\sum_ {i \in V} \left(p _ {i} ^ {t}\right) ^ {3 / 2}\right) ^ {2 / 3} \left(\sum_ {i \in U} \left(p _ {i} ^ {t}\right) ^ {3 / 2}\right) ^ {4 / 3}} \leq \frac {1}{\sqrt {t}} \left(\sum_ {i \in V} \left(p _ {i} ^ {t}\right) ^ {3 / 2}\right) ^ {1 / 3} \leq \frac {\sum_ {i \in V} \sqrt {p _ {i} ^ {t}}}{\sqrt {t}},
$$
where importantly, in the second step we use the fact  $\| x\| _2\leq \| x\|_{3 / 2}$  to drop all dependence on  $i\in U$  eventually arriving at the self-bounding term of Eq. (3).
Unfortunately, with an arm-dependent learning rate, it is unclear to us how to analyze Eq. (12) in a similar way. Instead, we propose a different analysis with the following rough idea: we propose a condition under which Eq. (12) can be bounded by  $\sum_{i\in V}\left(p_i^t\right)^{2 - \beta} / \gamma_i^t$  and then further related to a self-bounding term similarly to the analysis of the regret on sub-optimal arms. If, on the other hand, the condition does not hold, then we show that the probability  $p_i^t$  of selecting an optimal arm  $i\in U$  must be no more than the total probability of selecting sub-optimal arms  $\sum_{j\in V}p_j^t$ . Using this fact again allows us to convert dependence on  $i\in U$  to  $i\in V$ . Since we apply this technique to all  $i\in U$ , it leads to the extra  $|U|$  factor in the second self-bounding term of Eq. (3), which eventually translates to  $\frac{|U| \log T}{\Delta_{\min}}$  in our instance complexity  $\mathbb{G}$ . All details can be found in Section A.4.
In this work, we improve and generalize the analysis of [Ito, 2021], showing that many FTRL algorithms can achieve BOBW without the uniqueness assumption. Specifically, we propose a unified arm-dependent learning rate schedule and novel analytical techniques to remove the uniqueness assumption for a broad family of regularizers, including log-barrier,  $\beta$ -Tsallis entropy, and Shannon entropy. With these new techniques, our regret bounds improve upon prior results even when the uniqueness assumption holds. We further apply our results to the decoupled exploration and exploitation setting, showing that our techniques are broadly applicable.
There are many natural future directions, including (1) removing the  $\frac{|U|}{\Delta_{\mathrm{MIN}}}$  term in our regret bounds; (2) improving the dependence on  $D$  (which as mentioned could be as large as the corruption level  $C$ ); (3) understanding whether the extra the log-barrier regularizer is necessary or not; (4) and finally generalizing our results to other problems such as semi-bandits and Markov Decision Processes.
TJ and HL are supported by NSF Award IIS-1943607.