SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We keep details of the dependence on  $S$  and other constants as they are important in applying this result, but in most cases setting  $\rho = \frac{1}{\sqrt[3]{T}}$  yields  $\tilde{\mathcal{O}}(T^{\frac{3}{4}})$  regret. While a slow rate, the losses  $U_{t}$  are non-Lipschitz and non-convex in general, and learning them allows us to tune  $\theta$  over user-specified intervals and  $\eta$  over all positive numbers, which will be crucial later. At the same time, this tuning is what leads to the slow rate, as without tuning  $(k = 1, L_{\eta} = 0)$  the same  $\rho$  yields  $\tilde{\mathcal{O}}(\sqrt{T})$  regret. Lastly, while we focus on learning guarantees, we note that Algorithm 1 is reasonably efficient, requiring a  $2k$  single-dimensional integrals per task; this is discussed in more detail in Section A.3.
3 Multi-armed bandits
We now turn to our first application: the multi-armed bandit problem, where at each round  $i$  of task  $t$  we take action  $a_{t,i} \in [d]$  and observe loss  $\ell_{t,i}(a_{t,i}) \in [0,1]$ . As we are sampling actions from distributions  $\mathbf{x} \in \mathcal{K} = \triangle$  on the  $k$ -simplex, the inner product  $\langle \ell_{t,i}, \mathbf{x}_{t,i} \rangle$  is the expected loss and the optimal arm  $\mathring{a}_t$  on task  $t$  can be encoded as a vector  $\mathbf{\bar{x}}_t$  s.t.  $\mathbf{\bar{x}}_t(a) = 1_{a = \mathring{a}_t}$ .
We use as a base-learner a generalization of Exp3 that uses the negative Tsallis entropy  $\psi_{\beta}(\mathbf{p}) = \frac{1 - \sum_{a=1}^{d} \mathbf{p}^{\beta}(a)}{1 - \beta}$  for some  $\beta \in (0,1]$  as the regularizer; this improves regret from Exp3's  $\mathcal{O}(\sqrt{dm \log d})$  to the optimal  $\mathcal{O}(\sqrt{dm})$  [3]. Note that  $-\psi_{\beta}$  is the Shannon entropy in the limit  $\beta \to 1$  and its Bregman divergence  $B_{\beta}(\mathbf{x}||\cdot)$  is non-convex in the second argument. As the Tsallis entropy is non-Lipschitz at the simplex boundary, which is where the estimated and true optima  $\hat{\mathbf{x}}_t$  and  $\hat{\mathbf{x}}_t$  lie, we will project them using  $\mathbf{c}_{\frac{\varepsilon}{1 - \varepsilon}}(\mathbf{x}) = (1 - \varepsilon)\mathbf{x} + \varepsilon \mathbf{1}_d / d$  to the set  $\mathcal{K}_{\frac{\varepsilon}{1 - \varepsilon}} = \{\mathbf{x} \in \triangle : \min_a \mathbf{x}(a) \geq \varepsilon / d\}$ . We denote the resulting vectors using the superscript  $(\varepsilon)$ , e.g.  $\hat{\mathbf{x}}_t^{(\varepsilon)} = \mathbf{c}_{\frac{\varepsilon}{1 - \varepsilon}}(\hat{\mathbf{x}}_t)$ , and also use  $\triangle^{(\varepsilon)} = \mathcal{K}_{\frac{\varepsilon}{1 - \varepsilon}}$  to denote the constrained simplex. For MAB we also study two base-learners: (1) implicit exploration and (2) guaranteed exploration. The former uses low-variance loss under-estimators  $\hat{\ell}_{t,i}(a) = \frac{\ell_{t,i}(a)1_{a_t,i=a}}{\mathbf{x}_{t,i}(a) + \gamma}$  for  $\gamma > 0$ , where  $\mathbf{x}_{t,i}(a)$  is the probability of sampling  $a$  on task  $t$  round  $i$ , to enable high probability bounds [43]. On the other hand, guaranteed exploration uses unbiased loss estimators (i.e.  $\gamma = 0$ ) but constrains the action space to  $\triangle^{(\varepsilon)}$ , which we will use to adapt to a task-similarity determined by the true optima-in-hindsight.
3.1 Adapting to low estimated entropy with high probability using implicit exploration
In our first setting, the base-learner runs  $\mathrm{OMD}_{\eta_t,\beta_t}(\mathbf{x}_{t,1})$  on  $\gamma$ -regularized estimators with Tsallis regularizer  $\psi_{\beta_t}$ , step-size  $\eta_t$ , and initialization  $\mathbf{x}_{t,1} \in \triangle^{(\varepsilon)}$ . Standard OMD analysis combined with implicit exploration analysis [43] shows (44) that the task-averaged regret is bounded w.h.p. by
$$
(\varepsilon + \gamma d) m + \tilde {\mathcal {O}} \left(\frac {\sqrt {d}}{\gamma T}\right) + \frac {1}{T} \sum_ {t = 1} ^ {T} \frac {B _ {\beta_ {t}} \left(\hat {\mathbf {x}} _ {t} ^ {(\varepsilon)} \mid \mid \mathbf {x} _ {t , 1}\right)}{\eta_ {t}} + \frac {\eta_ {t} d ^ {\beta_ {t}} m}{\beta_ {t}} \tag {6}
$$
The summands have the desired form of  $U_{t}(\mathbf{x}_{t,1},\eta_{t},\beta_{t})$ , so by Theorem 2.1 we can bound their average by
$$
\min  _ {\beta \in [ \underline {{\beta}}, \bar {\beta} ], \eta > 0} \frac {\hat {V} _ {\beta} ^ {2}}{\eta} + \frac {\eta d ^ {\beta} m}{\beta} + \tilde {\mathcal {O}} \left(\frac {L _ {\eta}}{k} + \frac {\left(\frac {d}{\varepsilon}\right) ^ {2 - \underline {{\beta}}}}{\eta T} + \left(\rho + \frac {1}{\rho \sqrt {T}} + \frac {1}{\rho^ {2} T}\right) d \sqrt {m}\right) \tag {7}
$$
where  $\hat{V}_{\beta}^{2} = \frac{1}{T}\sum_{t=1}^{T}\psi_{\beta}(\hat{\mathbf{x}}_{t}^{(\varepsilon)}) - \psi_{\beta}(\hat{\overline{\mathbf{x}}}^{(\varepsilon)})$  is the average difference in Tsallis entropies between the ( $\varepsilon$ -constrained) estimated optima  $\hat{\mathbf{x}}_t$  and their empirical distribution  $\hat{\overline{\mathbf{x}}} = \frac{1}{T}\sum_{t=1}^{T}\hat{\mathbf{x}}_t$ , while  $L_{\eta}$  is the Lipschitz constant of  $\frac{\hat{V}_{\beta}^{2}}{\eta} + \frac{\eta d^{\beta}m}{\beta}$  w.r.t.  $\beta \in [\underline{\beta}, \overline{\beta}]$ . The specific instantiation of Algorithm 1 that (7) holds for is to do the following at each time  $t$ :
1. sample  $\beta_{t}$  via the MW distribution  $\propto \exp (\mathbf{w}_t)$  over the discretization  $\Theta_{k}$  of  $[\beta ,\overline{\beta} ]\subset [0,1]$
2. run  $\mathrm{OMD}_{\eta_t,\beta_t}$  using the initialization  $\mathbf{x}_{t,1} = \frac{1}{t - 1}\sum_{s < t}\hat{\mathbf{x}}_t^{(\varepsilon)} = \frac{\varepsilon}{d}\mathbf{1}_d + \frac{1 - \varepsilon}{t - 1}\sum_{s < t}\hat{\mathbf{x}}_t$  (FTL)
3. update EwOo at each  $\beta \in \Theta_{k}$  with loss  $\frac{B_{\beta}(\hat{\mathbf{x}}_t^{(\varepsilon)}||\mathbf{x}_{t,1}) + \rho^2D_{\beta}^2}{\eta} +\frac{\eta d^{\beta}m}{\beta}$ , where  $D_{\beta}^{2} = \frac{d^{1 - \beta} - 1}{1 - \beta}$  (8)
4. update  $\mathbf{p}_{t + 1}$  using multiplicative weights with expert losses  $\frac{B_{\beta}(\hat{\mathbf{x}}_t^{(\varepsilon)}||\mathbf{x}_{t,1})}{\eta} +\frac{\eta d^{\beta}m}{\beta}$
The final guarantee for this procedure, given in full in Theorem B.1, follows by two properties of the Tsallis entropy  $-\psi_{\beta}$ : (1) its Lipschitzness w.r.t.  $\beta \in [0,1]$  (c.f. Lem B.1) and (2) the fact that  $\hat{V}_{\beta}^{2}$  is bounded by the entropy  $\hat{H}_{\beta} = -\psi_{\beta}(\hat{\mathbf{x}})$  of the empirical distribution of estimated optima (c.f. Lem B.2), which yields our first notion of task-similarity: multi-armed bandit tasks are similar if the empirical distribution of their (estimated) optimal arms has low entropy.
We exemplify the implications of Theorem B.1 in Corollary 3.1, where we consider three regimes of the lower bound  $\underline{\beta}$  on the entropy parameter:  $\underline{\beta} = 1$ , i.e. always using  $\mathrm{Exp3}$ ;  $\underline{\beta} = 1/2$ , which corresponds to the optimal worst-case setting [3]; and  $\underline{\beta} = 1/\log d$ , below which the OMD regret-upper-bound always worsens (and so it does not make sense to try  $\beta < 1/\log d$ ).
Corollary 3.1 (c.f. Cors. B.1, B.2, and B.3). Suppose  $\overline{\beta} = 1$  and we set the initialization, step-size, and entropy parameter of Tsallis OMD with implicit exploration via Algorithm 1 as in Theorem B.1.
1. If  $\underline{\beta} = 1$  and  $T\geq \frac{d^2}{m}$  we can ensure  $\frac{1}{T}\sum_{t = 1}^{T}\sum_{i = 1}^{m}\ell_{t,i}(\mathbf{x}_{t,i}) - \ell_{t,i}(\mathbf{\tilde{x}}_t)\leq 2\sqrt{\hat{H}_1dm} +\tilde{\mathcal{O}}\left(\frac{d^{\frac{2}{3}}m^{\frac{2}{3}}}{\sqrt[3]{T}}\right)w.h.p.$
2. If  $\beta = \frac{1}{2}$  and  $T \geq \frac{d^{5/2}}{m}$  we can set  $k = \lceil \sqrt[4]{d} \sqrt{T} \rceil$  and ensure w.h.p. that task-averaged regret is
$$
\min  _ {\beta \in [ \frac {1}{2}, 1 ]} 2 \sqrt {\hat {H} _ {\beta} d ^ {\beta} m / \beta} + \tilde {\mathcal {O}} \left(\frac {d ^ {5 / 7} m ^ {5 / 7}}{T ^ {2 / 7}} + \frac {d \sqrt {m}}{\sqrt [ 4 ]{T}}\right) \tag {9}
$$
3. If  $\underline{\beta} = \frac{1}{\log d}$  and  $T \geq \frac{d^3}{m}$  we can set  $k = \lceil \sqrt[4]{d}\sqrt{T}\rceil$  and ensure w.h.p. that task-averaged regret is
$$
\min  _ {\beta \in (0, 1 ]} 2 \sqrt {\hat {H} _ {\beta} d ^ {\beta} m / \beta} + \tilde {\mathcal {O}} \left(\frac {d ^ {3 / 4} m ^ {3 / 4} + d \sqrt {m}}{\sqrt [ 4 ]{T}}\right) \tag {10}
$$
In all three settings, as  $T \to \infty$  the regret scales directly with the entropy of the estimated optima-in-hindsight, which is small if most tasks are estimated to be solved by one of a few arms and large if all arms are used roughly equally. Corollary 3.1 demonstrates the importance of tuning  $\beta$ : even if tasks are dissimilar, we asymptotically recover the worst-case optimal guarantee  $\mathcal{O}(\sqrt{dm})$  in cases two and three because the entropy is at most  $\frac{d^{1 - \beta}}{1 - \beta}$ . On the other hand, if a constant  $s \ll d$  actions are always minimizers, i.e. the empirical distribution  $\hat{\tilde{\mathbf{x}}}$  is  $s$ -sparse, then the last bound (10)
implies that Algorithm 1 can achieve task-averaged regret  $o_{T}(md) + \mathcal{O}(\sqrt{sm\log d})$ . At the same time, this tuning is costly, with the last two results having an extra  $\tilde{\mathcal{O}}\left(\frac{d\sqrt{m}}{\sqrt[4]{T}}\right)$  term because of it. Furthermore, the bound of  $\underline{\beta} = \frac{1}{2}$  has a slightly better dependence on  $d$ ,  $m$ , and  $T$  compared to that of  $\underline{\beta} = \frac{1}{\log d}$  due to the  $\left(\frac{d}{\varepsilon}\right)^{\overline{2} - \underline{\beta}}$  term in the bound (7) returned for MAB by our structural result.
We can compare the  $s$ -sparse result to Azizi et al. [10], who achieve task-averaged regret  $\tilde{\mathcal{O}}(m / \sqrt[3]{T} + \sqrt{sm\log T})$  for stochastic MAB. Despite our adversarial setting and no stipulations on how tasks are related, our bounds are asymptotically comparable if the estimated and true optima are roughly equivalent (ignoring their  $\mathcal{O}(\sqrt{\log T})$ -factor), as we also have  $\tilde{\mathcal{O}}(\sqrt{sm})$  average regret as  $T \to \infty$ . Their rate in the number of tasks is better, but at a cost of runtime exponential in  $s$ . Apart from generality, we believe a great strength of our results is their adaptiveness; unlike Azizi et al. [10], we do not need to know how many optimal arms there are to adapt to there being few of them.
3.2 Adapting to the entropy of the true optima-in-hindsight using guaranteed exploration
While the entropy of estimated optima-in-hindsight may be useful in some cases where we wish to actually compute the task-similarity, it is otherwise generally more desirable to adapt to an intrinsic and algorithm-independent measure, e.g. the entropy of the true optima-in-hindsight. However, doing so is difficult without further assumptions, as the optima are both hard to identify and the measure itself may not be fully defined in case of ties. Thus in this section we focus on the setting where we have a nonzero performance gap  $\Delta > 0$  between the best and second-best arms:
Assumption 3.1. For some  $\Delta >0$  and all tasks  $t\in [T]$ $\frac{1}{m}\sum_{i = 1}^{m}\ell_{t,i}(a) - \ell_{t,i}(\mathring{a}_t)\geq \Delta \forall a\neq \mathring{a}_t.$
This assumption is common in the best-arm identification literature [28, 1], which we adapt to show that the estimated optimal arms match the true optima, and thus so do their entropies. To do so, we switch to unbiased loss estimators, i.e.  $\gamma = 0$ , and control their variance by lower-bounding the probability of selecting an arm to be at least  $\frac{\varepsilon}{d}$ ; this can alternatively be expressed as running OMD using the regularizer  $\psi_{\beta} + I_{\triangle^{(\varepsilon)}}$ , where for any  $\mathcal{C} \subset \mathbb{R}^d$  the function  $I_{\mathcal{C}}(\mathbf{x}) = 0$  if  $\mathbf{x} \in \mathcal{C}$  and  $\infty$  otherwise. Guaranteed exploration allows us to extend the analysis of Abbasi-Yadkori et al. [1] to show that the estimated arm is optimal w.h.p.:
Lemma 3.1 (c.f. Lem C.1). Suppose for  $\varepsilon >0$  and any  $\beta \in (0,1]$  we run OMD on task  $t\in [T]$  with regularizer  $\psi_{\beta} + I_{\triangle^{(\varepsilon)}}$ . If  $m = \tilde{\Omega}\left(\frac{d}{\varepsilon\Delta^2}\right)$  then  $\hat{\mathbf{x}}_t = \bar{\mathbf{x}}_t$  w.p.  $\geq 1 - d\exp (-\Omega (\varepsilon \Delta^2 m / d))$ .
However, the constraint that the probabilities are at least  $\frac{\varepsilon}{d}$  does lead to  $\varepsilon m$  additional error on each task, with the upper bound on the task-averaged expected regret becoming
$$
\mathbb {E} \frac {1}{T} \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {m} \ell_ {t, i} \left(a _ {t, i}\right) - \ell_ {t, i} \left(\stackrel {\circ} {a} _ {t}\right) \leq \varepsilon m + \frac {1}{T} \sum_ {t = 1} ^ {T} \frac {\mathbb {E} B _ {\beta_ {t}} \left(\hat {\mathbf {x}} _ {t} ^ {(\varepsilon)} \mid \mid \mathbf {x} _ {t , 1}\right)}{\eta_ {t}} + \frac {\eta_ {t} d ^ {\beta_ {t}} m}{\beta_ {t}} \tag {11}
$$
Moreover, we will no longer set  $\varepsilon = o_{T}(1)$ , as this would require  $m$  to be increasing in  $T$  for the best-arm identification result of Lemma C.1 to hold. Thus, unlike in the previous section, our results will contain "fast" terms—terms in the task-averaged regret that are  $o(\sqrt{m})$  but not decreasing in  $T$  nor affected by the task-similarity. They will still improve upon the  $\Omega(\sqrt{dm})$  MAB lower bound if tasks are similar, but the task-averaged regret will not converge to zero as  $T \to \infty$  if the tasks are identical.
Nevertheless, the tuning-dependent component of the upper bounds in (11) has the appropriate form for our structural result—in fact we can use the same meta-algorithm (8) as for implicit exploration—and so we can again apply Theorem 2.1 to get a bound on the task-averaged regret in terms of the average difference  $\hat{V}_{\beta}^{2} = \frac{1}{T}\sum_{t=1}^{T}\psi_{\beta}(\hat{\mathbf{x}}_{t}^{(\varepsilon)}) - \psi_{\beta}(\hat{\bar{\mathbf{x}}}^{(\varepsilon)})$  of the entropies of the  $\varepsilon$ -constrained estimated task-optima  $\hat{\mathbf{x}}_t^{(\varepsilon)}$  and their mean  $\hat{\bar{\mathbf{x}}}^{(\varepsilon)}$ . The easiest way to apply Lemma C.1 to bound  $\hat{V}_{\beta}^{2}$  in terms of  $H_{\beta} = \frac{1}{T}\sum_{t=1}^{T}\psi_{\beta}(\hat{\mathbf{x}}_t) - \psi_{\beta}(\hat{\bar{\mathbf{x}}})$  is via union bound on all  $T$  tasks to show that  $\hat{\mathbf{x}}_t = \hat{\mathbf{x}}_t \forall t$  w.p.  $\geq 1 - dT\exp(-\Omega(\varepsilon\Delta^2m/d))$ ; however, setting a constant failure probability leads to  $m$  growing, albeit only logarithmically, in  $T$ . Instead, by analyzing the worst-case best-arm identification probabilities, we show in Lemma C.2 that the expectation of  $\hat{V}_{\beta}^{2}$  is bounded by  $H_{\beta} + 3\beta\frac{(d/\varepsilon)^{1-\beta}-1}{1-\beta}\exp\left(-\frac{3\varepsilon\Delta^2m}{28d}\right)$  without resorting to  $m = \omega_T(1)$ . Assuming  $m \geq \frac{75d}{\varepsilon\Delta^2}\log\frac{d}{\varepsilon\Delta^2}$  is enough (69) to bound the second term by  $\frac{56}{dm}$ . Then the final result (c.f. Thm. C.1) bounds the expected task-averaged regret as follows (ignoring terms that become  $o_T(1)$  after setting  $\rho$  and  $k$ ):
$$
\varepsilon m + \min  _ {\beta \in [ \underline {{\beta}}, \overline {{\beta}} ], \eta > 0} \frac {h _ {\beta} (\Delta)}{\eta} + \frac {\eta d ^ {\beta} m}{\beta} \quad \text {f o r} \quad h _ {\beta} (\Delta) = \left\{ \begin{array}{l l} H _ {\beta} + \frac {5 6}{m d} & \text {i f} m \geq \frac {7 5 d}{\varepsilon \Delta^ {2}} \log \frac {d}{\varepsilon \Delta^ {2}} \\ \frac {d ^ {1 - \beta} - 1}{1 - \beta} & \text {o t h e r w i s e} \end{array} \right. \tag {12}
$$
If the gap  $\Delta$  is known and sufficiently large, then we can set  $\varepsilon = \Theta\left(\frac{d}{\Delta^2m}\right)$  to obtain an asymptotic task-averaged regret that scales only with the entropy  $H_{\beta}$  and a fast term that is logarithmic in  $m$ :
Corollary 3.2 (c.f. Cor. C.3). Suppose we set the initialization, step-size, and entropy parameter of Tsallis OMD with guaranteed exploration via Algorithm 1 as specified in Theorem C.1. If  $[\underline{\beta},\overline{\beta}] = [\frac{1}{\log d},1]$  and  $m\geq \frac{75d}{\Delta^2}\log \frac{d}{\Delta^2}$ , then setting  $\varepsilon = \tilde{\Theta}\left(\frac{d}{\Delta^2m}\right)$ ,  $\rho = \frac{1}{\sqrt[3]{d}\sqrt[6]{mT}}$ , and  $k = \lceil \sqrt[3]{d^2mT} \rceil$  ensures that the expected task-averaged regret is at most
$$
\min  _ {\beta \in (0, 1 ]} 2 \sqrt {H _ {\beta} d ^ {\beta} m / \beta} + \tilde {\mathcal {O}} \left(\frac {d}{\Delta^ {2}} + \frac {d ^ {\frac {4}{3}} m ^ {\frac {2}{3}}}{\sqrt [ 3 ]{T}} + \frac {d ^ {\frac {5}{3}} m ^ {\frac {5}{6}}}{T ^ {\frac {2}{3}}} + \frac {d \Delta^ {4} m ^ {3}}{T}\right) \tag {13}
$$
Knowing the gap  $\Delta$  is a strong assumption, as ideally we could set  $\varepsilon$  without it. Note that if  $\varepsilon = \Omega\left(\frac{1}{m^p}\right)$  for some  $p \in (0,1)$  then the condition  $m \geq \frac{75d}{\varepsilon\Delta^2} \log \frac{d}{\varepsilon\Delta^2}$  only fails if  $m \leq \mathrm{poly}\left(\frac{1}{\Delta}\right)$ , i.e. for gap decreasing in  $m$ . We can use this together with the fact that minimizing over  $\eta$  and  $\beta$  in our bound allows us to replace them with any value, even a gap-dependent one, to derive a gap-independent setting of  $\varepsilon$  that ensures a task-similarity-adaptive bound when  $\Delta$  is not too small and falls back to the worst-case optimal guarantee otherwise. Specifically, for indicator  $\iota_{\Delta} = 1_{m \geq \frac{75d}{\varepsilon\Delta^2} \log \frac{d}{\varepsilon\Delta^2}}$ , setting  $\eta = \Theta\left(\sqrt{\frac{h_{\beta}(\Delta)}{d^{\beta} m / \beta}}\right)$  in (12) and using  $\beta = \frac{1}{2}$  if the condition  $\iota_{\Delta}$  fails yields asymptotic regret at most
$$
\left. \varepsilon m + \min  _ {\beta \in (0, 1 ]} \mathcal {O} \left(\iota_ {\Delta} \sqrt {\frac {H _ {\beta} d ^ {\beta} m}{\beta}} + (1 - \iota_ {\Delta}) \sqrt {d m}\right) \leq \varepsilon m + \tilde {\mathcal {O}} \left(\min  \left\{\min  _ {\beta \in (0, 1 ]} \sqrt {\frac {H _ {\beta} d ^ {\beta} m}{\beta}} + \frac {d}{\Delta \sqrt {\varepsilon}}, \sqrt {d m} \right\}\right) \right. \tag {14}
$$
Thus setting  $\varepsilon = \Theta (\sqrt{d} /m^{\frac{2}{3}})$  yields the desired dependence on the entropy  $H_{\beta}$  and a fast term in  $m$
Corollary 3.3 (c.f. Cor. C.4). In the setting of Corollary 3.2 but with  $m = \Omega(d^{\frac{3}{4}})$  and unknown  $\Delta$ , using  $\varepsilon = \Theta(\sqrt{d} / m^{\frac{2}{3}})$  ensures expected task-averaged regret at most
$$
\min  \left\{\min  _ {\beta \in (0, 1 ]} 2 \sqrt {H _ {\beta} d ^ {\beta} m / \beta} + \tilde {\mathcal {O}} \left(\frac {d ^ {\frac {3}{4}} \sqrt [ 3 ]{m}}{\Delta}\right), 8 \sqrt {d m} \right\} + \tilde {\mathcal {O}} \left(\frac {d ^ {\frac {4}{3}} m ^ {\frac {2}{3}}}{\sqrt [ 3 ]{T}} + \frac {d ^ {\frac {5}{3}} m ^ {\frac {5}{6}}}{T ^ {\frac {2}{3}}} + \frac {d ^ {2} m ^ {\frac {7}{3}}}{T}\right) \tag {15}
$$
While not logarithmic, the gap-dependent term is still  $o(\sqrt{m})$ , and moreover the asymptotic regret is no worse than the worst-case optimal  $\mathcal{O}(\sqrt{dm})$ . Note that the latter is only needed if  $\Delta = o(1 / \sqrt[6]{m})$ .
The main improvement in this section is in using the entropy of the true optima, which can be much smaller than that of the estimated optima if there are a few good arms but large noise. Our use of the gap assumption for this seems difficult to avoid for this notion of task-similarity. We can also compare to Corollary 3.1 (10), which did not require  $\Delta > 0$  and had no fast terms but had a worse rate in  $T$ ; in contrast, the  $\mathcal{O}\left(\frac{1}{\sqrt[3]{T}}\right)$  rates above match that of the closest stochastic bandit result [10]. As before, for  $s \ll d$  "good" arms we obtain  $\mathcal{O}(\sqrt{sm\log d})$  asymptotic regret, assuming the gap is not too small. Finally, we can also compare to the classic shifting regret bound for Exp3.S [6], which translated to task-averaged regret is  $\mathcal{O}(\sqrt{dm\log(dmT)})$ . This is worse than even running OMD separately on each task, albeit under weaker assumptions (not knowing task boundaries). It also cannot take advantage of repeated optimal arms, e.g. the case of  $s \ll d$  good arms.
3.3 Adapting to entropic task similarity implies robustness to outliers
While we considered mainly the  $s$ -sparse setting as a way of exemplifying our results and comparing to other work such as Azizi et al. [10], the fact that our approach can adapt to the Tsallis entropy  $\min_{\beta} H_{\beta}$  of the optimal arms implies meaningful guarantees for any low-entropy distribution over the optimal arms, not just sparsely-supported ones. One way to illustrate the importance of this is through an analysis of robustness to outlier tasks. Specifically, suppose that the  $s$ -sparsity assumption—that optima  $\hat{a}_t$  lie in a subset of  $[T]$  of size  $s \ll d$ —only holds for all but  $\mathcal{O}(T^p)$  of the tasks  $t \in [T]$ , where  $p \in [0, 1)$ . Then the best we can do using an asymptotic bound of  $\tilde{\mathcal{O}}(\sqrt{sm})$ —e.g. that of Azizi et al.
[10] in the stochastic case or from naively applying  $\min_{\beta \in (0,1]}H_{\beta}d^{\beta}m / \beta \leq esm\log d$  to any of our previous results—is to substitute  $s + T^{p}$  instead of  $s$ , which will only improve over the single-task bound if  $d = \omega (T^p)$ , i.e. in the regime where the number of arms increases with the number of tasks.
However, our notion of task-similarity allows us to do much better, as we can show (c.f. Prop. D.1) that in the same setting  $H_{\beta} = \mathcal{O}\left(s + \frac{d^{1 - \beta}}{T^{\beta(1 - p)}}\right)$  for any  $\beta \in [\frac{1}{\log d}, \frac{1}{2}]$ . Substituting this result into e.g. Corollary 3.3 yields the same asymptotic result of  $\mathcal{O}(\sqrt{sm\log d})$ , although the rate in  $T$  is a very slow  $\mathcal{O}(\sqrt{dm} / T^{\frac{1 - p}{2\log d}})$ . This demonstrates how our entropic notion of task-similarity simultaneously yields strong results in the  $s$ -sparse setting and is meaningful in more general settings.
4 Bandit linear optimization
Our last application is bandit linear optimization, in which at task  $t$  round  $i$  we play  $\mathbf{x}_{t,i} \in \mathcal{K}$  in some convex  $\mathcal{K} \subset \mathbb{R}^d$  and observe loss  $\langle \ell_{t,i}, \mathbf{x}_{t,i} \rangle \in [-1,1]$ . We will again use a variant of mirror descent, using a self-concordant barrier for  $\psi$  and the specialized loss estimators of Abernethy et al. [2, Alg. 1]. More information on such regularizers can be found in the literature on interior point methods [42]. We pick this class of algorithms because of their optimal dependence on the number of rounds and their applicability to any convex domain  $\mathcal{K}$  via specific barriers  $\psi$ , which will yield interesting notions of task-similarity. Our ability to handle non-smooth regularizers via the structural result (Thm. 2.1) is even more important here, as barriers are infinite at the boundaries. Indeed, we will not learn a  $\beta$  parameterizing the regularizer and instead focus on tuning a boundary offset  $\varepsilon > 0$ . Here we make use of notation from Section 2, where  $\mathbf{c}_{\varepsilon}$  maps points in  $\mathcal{K}$  to a subset  $\mathcal{K}_{\varepsilon}$  defined by the Minkowski function (c.f. Def. E.1) centered at  $\mathbf{x}_1 = \arg \min_{\mathbf{x} \in \mathcal{K}} \psi(\mathbf{x})$ .
From Abernethy et al. [2] we have an upper bound on the expected task-averaged regret of their algorithm run from initializations  $\mathbf{x}_{t,1} \in \mathcal{K}^{\circ}$  with step-sizes  $\eta_t > 0$  and offsets  $\varepsilon_t > 0$ :