SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {E} \frac {1}{T} \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {m} \left\langle \ell_ {t, i}, \mathbf {x} _ {t, i} - \mathring {\mathbf {x}} _ {t} \right\rangle \leq \frac {1}{T} \sum_ {t = 1} ^ {T} \frac {\mathbb {E} B \left(\mathbf {c} _ {\varepsilon_ {t}} (\hat {\mathbf {x}} _ {t}) \mid \mid \mathbf {x} _ {t , 1}\right)}{\eta_ {t}} + (3 2 \eta_ {t} d ^ {2} + \varepsilon_ {t}) m \tag {16}
$$
We can show (88) that  $D_{\varepsilon}^{2} = \max_{\mathbf{x},\mathbf{y}\in \mathcal{K}_{\varepsilon}}B(\mathbf{x}||\mathbf{y})\leq \frac{9\nu^{\frac{3}{2}}K\sqrt{S_{1}}}{\varepsilon}$ , where  $\nu$  is the self-concordance constant of  $\psi$  and  $S_{1} = \| \nabla^{2}\psi (\mathbf{x}_{1})\|_{2}$  is the spectral norm of its Hessian at the center  $\mathbf{x}_1$  of  $\kappa$ . Restricting to tuning  $\varepsilon \in [\frac{1}{m},1]$  which is enough to obtain constant task-averaged regret above if the estimated optima  $\hat{\mathbf{x}}_t$  are identical—we can now apply Algorithm 1 via the following instantiation:
1. sample  $\varepsilon_{t}$  via the MW distribution  $\propto \exp (\mathbf{w}_t)$  over the discretization  $\Theta_{k}$  of  $[\frac{1}{m},1]$
2. run  $\mathrm{OMD}_{\eta_t,\varepsilon_t}$  using the initialization  $\mathbf{x}_{t,1} = \frac{1}{t - 1}\sum_{s < t}\mathbf{c}_{\varepsilon_t}(\hat{\mathbf{x}}_t) = \mathbf{x}_1 + \frac{\sum_{s < t}\hat{\mathbf{x}}_t - \mathbf{x}_1}{(1 + \varepsilon_t)(t - 1)}$  (FTL)
3. update EWOO at each  $\varepsilon \in \Theta_{k}$  with loss  $\frac{B(\mathbf{c}_{\varepsilon}(\hat{\mathbf{x}}_t)||\mathbf{x}_{t,1}) + \rho^2D_{\varepsilon}^2}{\eta} + 32\eta d^2$  for  $D_{\varepsilon}^{2} = \frac{9\nu^{\frac{3}{2}}K\sqrt{S_{1}}}{\varepsilon}$
4. update  $\mathbf{p}_{t + 1}$  using multiplicative weights with expert losses  $\frac{B(\mathbf{c}_{\varepsilon}(\hat{\mathbf{x}}_t)||\mathbf{x}_{t,1})}{\eta} +\varepsilon m$  (17)
Note the similarity to the MAB case (8), with the difference being the upper bound passed to EWOO and MW. Our structural result bounds the expected task-averaged regret as follows (c.f. Thm. E.1):
$$
\mathbb {E} \min  _ {\varepsilon \in [ \frac {1}{m}, 1 ], \eta > 0} \frac {\hat {V} _ {\varepsilon} ^ {2}}{\eta} + (3 2 \eta d ^ {2} + \varepsilon) m + \tilde {\mathcal {O}} \left(\frac {\frac {m ^ {2}}{T} + \frac {1}{k}}{\eta} + \frac {m}{k} + m \min  \left\{\frac {\rho^ {2}}{\eta}, d \rho \right\} + \frac {d m}{\rho} \sqrt {\frac {\log k}{T}} + \frac {d m}{\rho^ {2} T}\right) \tag {18}
$$
For  $\rho = o_{T}(1)$  and  $k = \omega_{T}(1)$  this becomes  $o_{T}(\mathrm{poly}(m)) + \mathbb{E}\min_{\varepsilon \in [\frac{1}{m},1],\eta >0}\frac{\hat{V}_{\varepsilon}^{2}}{\eta} +32\eta d^{2}m + \varepsilon m,$  where  $\hat{V}_{\varepsilon}^{2} = \frac{1}{T}\sum_{t = 1}^{T}\psi (\mathbf{c}_{\varepsilon}(\hat{\mathbf{x}}_t) - \psi (\mathbf{c}_{\varepsilon}(\hat{\bar{\mathbf{x}}} _t))$  . Then by tuning  $\eta$  we get an asymptotic  $(T\to \infty)$  regret of  $4d\hat{V}_{\varepsilon}\sqrt{2} m + \varepsilon m$  for any  $\varepsilon \in [\frac{1}{m},1]$  . Our analysis removes the explicit dependence on  $\sqrt{\nu}$  that appears in the single-task regret [2]; as an example,  $\nu$  equals the number of inequalities defining a polytope  $\mathcal{K}$ , as in the bandit shortest-path application below.
The remaining challenge is to interpret  $\hat{V}_{\varepsilon}^{2}$ , which as we did for MAB we do via specific examples, in this case concrete action domains  $\mathcal{K}$ . Our first example is for BLO over the unit sphere  $\mathcal{K} = \{\mathbf{x} \in \mathbb{R}^d : \| \mathbf{x} \|_2 \leq 1\}$  using the appropriate log-barrier regularizer  $\psi(\mathbf{x}) = -\log (1 - \| \mathbf{x} \|_2^2)$ :
Corollary 4.1 (c.f. Cor. E.1). For BLO on the sphere, Algorithm 1 has expected task-averaged regret
$$
\tilde {\mathcal {O}} \left(\frac {d m ^ {\frac {3}{2}}}{T ^ {\frac {3}{4}}} + \frac {d m}{\sqrt [ 4 ]{T}}\right) + \min  _ {\varepsilon \in \left[ \frac {1}{m}, 1 \right]} 4 d \sqrt {2 m \log \left(1 + \frac {1 - \mathbb {E} \| \hat {\mathbf {x}} \| _ {2} ^ {2}}{2 \varepsilon + \varepsilon^ {2}}\right)} + \varepsilon m \tag {19}
$$
The bound above is decreasing in  $\mathbb{E}\| \hat{\mathbf{x}}\| _2^2$ , the expected squared norm of the average of the estimated optima  $\hat{\mathbf{x}}_t$ . We thus say that bandit linear optimization tasks over the sphere are similar if the norm of the empirical mean of their (estimated) optima is large. This makes intuitive sense: if the tasks' optima are uniformly distributed, we should expect  $\mathbb{E}\| \hat{\mathbf{x}}\| _2^2$  to be small, even decreasing in  $d$ . On the other hand, in the degenerate case where the estimated optima  $\hat{\mathbf{x}}_t$  are the same across all tasks  $t\in [T]$ , we have  $\mathbb{E}\| \hat{\mathbf{x}}\| _2^2 = 1$ , so the asymptotic task-averaged regret is 1 because we can use  $\varepsilon = \frac{1}{m}$ . Perhaps slightly more realistically, if it is  $\frac{1}{m^p}$ -away from 1 for some power  $p\geq \frac{1}{2}$  then setting  $\varepsilon = \frac{1}{\sqrt{m}}$  can remove the logarithmic dependence on  $m$ . These two regimes illustrate the importance of tuning  $\varepsilon$ .
As a last application, we apply our meta-BLO result to the shortest-path problem in online optimization [50, 30]. In its bandit variant [8, 17], at each step  $i = 1, \dots, m$  the player must choose a path  $p_i$  from a fixed source  $u \in V$  to a fixed sink  $v \in V$  in a directed graph  $G(V, E)$ . At the same time the adversary chooses edge-weights  $\ell_i \in \mathbb{R}^{|E|}$  and the player suffers the sum  $\sum_{e \in p_t} \ell_i(e)$  of the weights in their chosen path  $p_t$ . This can be relaxed as BLO over vectors  $\mathbf{x}$  in a set  $\mathcal{K} \subset [0,1]^{|E|}$  defined by a set  $\mathcal{C}$  of  $\mathcal{O}(|E|)$  linear constraints  $(\mathbf{a}, b) \langle \mathbf{a}, \mathbf{x} \rangle \leq b$  enforcing flows from  $u$  to  $v$ ;  $u$  to  $v$  paths can be sampled from any  $\mathbf{x} \in \mathcal{K}$  in an unbiased manner [2, Proposition 1]. On a single-instance, applying the BLO method of Abernethy et al. [2] ensures  $\mathcal{O}(|E|^{\frac{3}{2}}\sqrt{m})$  regret on this problem.
In the multi-instance setting, comprising a sequence  $t = 1,\dots ,T$  of shortest path instances with  $m$  adversarial edge-weight vectors  ${\ell }_{t,i}$  each,we can attempt to achieve better performance by tuning the same method across instances. Notably, we can view this as the problem of learning predictions [33] in the algorithms with predictions paradigm from beyond-worst-case analysis [40], with the OMD initialization on each instance being effectively a prediction of its optimal path. Our meta-learner then has the following average performance across bandit shortest-path instances:
Corollary 4.2 (c.f. Cor. E.2). For multi-task bandit online shortest path, Algorithm 1 with regularizer  $\psi (\mathbf{x}) = -\sum_{\mathbf{a},b\in \mathcal{C}}\log (b - \langle \mathbf{a},\mathbf{x}\rangle)$  attains the following expected average regret across instances
$$
\tilde {\mathcal {O}} \left(\frac {| E | ^ {4} m ^ {\frac {3}{2}}}{T ^ {\frac {3}{4}}} + \frac {| E | ^ {\frac {5}{2}} m ^ {\frac {5}{6}}}{\sqrt [ 4 ]{T}}\right) + \min  _ {\varepsilon \in [ \frac {1}{m}, 1 ]} 4 | E | \mathbb {E} \sqrt {2 m \sum_ {\mathbf {a} , b \in C} \log \left(\frac {\frac {1}{T} \sum_ {t = 1} ^ {T} b - \langle \mathbf {a} , \mathbf {c} _ {\varepsilon} (\hat {\mathbf {x}} _ {t}) \rangle}{\sqrt [ T ]{\prod_ {t = 1} ^ {T} b - \langle \mathbf {a} , \mathbf {c} _ {\varepsilon} (\hat {\mathbf {x}} _ {t}) \rangle}}\right)} + \varepsilon m \tag {20}
$$
Here the asymptotic regret scales with the sum across all constraints  $\mathbf{a}, b \in \mathcal{C}$  of the log of the ratio between the arithmetic and geometric means across tasks of the distances  $b - \langle \mathbf{a}, \mathbf{c}_{\varepsilon}(\hat{\mathbf{x}}_t) \rangle$  from the estimated optimum flow  $\mathbf{c}_{\varepsilon}(\hat{\mathbf{x}}_t)$  to the constraint boundary. As it is difficult to separate the effect of the offset  $\varepsilon$ , we do not state an explicit task-similarity measure like in our previous settings. Nevertheless, since the arithmetic and geometric means are equal exactly when all entries are equal—and otherwise the former is larger—the bound does show that regret is small when the estimated optimal flows  $\hat{\mathbf{x}}_t$  for each task are at similar distances from the constraints, i.e. the boundaries of the polytope. Indeed, just as on the sphere, if the estimated optima are all the same then setting  $\varepsilon = \frac{1}{m}$  again yields constant averaged regret.
We develop and apply a meta-algorithm for learning to initialize and tune bandit algorithms, obtaining task-averaged regret guarantees for both multi-armed and linear bandits that depend on natural, setting-specific notions of task similarity. For MAB, we meta-learn the initialization, step-size, and entropy parameter of Tsallis-entropy OMD and show good performance if the entropy of the optimal arms is small. For BLO, we use OMD with self-concordant regularizers and meta-learn the initialization, step-size, and boundary-offset, yielding interesting domain-specific task-similarity measures. Some natural directions for future work involve overcoming some limitations of our results: can we adapt to a notion of task-similarity that depends on the true optima without assuming a gap for MAB, or at all for BLO? Alternatively, can we design meta-learning algorithms that adapt to both stochastic and adversarial bandits, i.e. a "best-of-both-worlds" guarantee? Beyond this, one could explore other partial information settings, such as contextual bandits or bandit convex optimization.
We thank our anonymous reviewers for helpful suggestions, especially concerning the analysis of robustness to outliers. This material is based on work supported in part by the National Science Foundation under grants CCF-1910321, FAI-1939606, IIS-1901403, SCC-1952085, and SES-1919453; the Defense Advanced Research Projects Agency under cooperative agreement HR00112020003; a Simons Investigator Award; an AWS Machine Learning Research Award; an Amazon Research Award; a Bloomberg Research Grant; a Microsoft Research Faculty Fellowship; a Google Faculty Research Award; a J.P. Morgan Faculty Award; a Facebook Research Award; a Mozilla Research Grant; a Facebook PhD Fellowship; and an NDSEG Fellowship. KYL is supported by the Israel Science Foundation (grant No. 447/20) and the Technion Center for Machine Learning and Intelligent Systems (MLIS). The work of RM was partially supported by the Israel Science Foundation grant number 1693/22.
[1] Yasin Abbasi-Yadkori, Peter Bartlett, Victor Gabillon, Alan Malek, and Michal Valko. Best of both worlds: Stochastic & adversarial best-arm identification. In Proceedings of the 31st Conference On Learning Theory, 2018.
[2] Jacob Abernethy, Elad Hazan, and Alexander Rakhlin. Competing in the dark: An efficient algorithm for bandit linear optimization. In Proceedings of the International Conference on Computational Learning Theory, 2008.
[3] Jacob Abernethy, Chansoo Lee, and Ambuj Tewari. Fighting bandits with a new kind of smoothness. In Advances in Neural Information Processing Systems, 2015.
[4] Oren Anava and Zohar S. Karnin. Multi-armed bandits: Competing with optimal sequences. In Advances in Neural Information Processing Systems, 2016.
[5] Jean-Yves Audibert, Sébastien Bubeck, and Gábor Lugosi. Minimax policies for combinatorial prediction games. In Proceedings of the International Conference on Computational Learning Theory, 2011.
[6] Peter Auer, Nicolò Cesa-Bianchi, Yoav Freund, and Robert E. Schapire. The nonstochastic multiarmed bandit problem. SIAM Journal of Computing, 32:48-77, 2002.
[7] Peter Auer, Pratik Gajane, and Ronald Ortner. Adaptively tracking the best bandit arm with an unknown number of distribution changes. In Proceedings of the 32nd Annual Conference on Learning Theory, 2019.
[8] Baruch Awerbuch and Robert D. Kleinberg. Adaptive routing with end-to-end feedback: Distributed learning and geometric approaches. In Proceedings of the Thirty-Sixth Annual ACM Symposium on Theory of Computing, 2004.
[9] Mohammad Gheshlaghi Azar, Alessandro Lazaric, Emma Brunskill, et al. Sequential transfer in multi-armed bandit with finite set of models. In Advances in Neural Information Processing Systems, 2013.
[10] MohammadJavad Azizi, Thang Duong, Yasin Abbasi-Yadkori, András György, Claire Vernade, and Mohammad Ghavamzadeh. Non-stationary bandits and meta-learning with a small set of optimal arms. arXiv, 2022.
[11] Maria-Florina Balcan, Avrim Blum, and Santosh Vempala. Efficient representations for lifelong learning and autoencoding. In Proceedings of the 28th Annual Conference on Learning Theory, 2015.
[12] Maria-Florina Balcan, Mikhail Khodak, Dravyansh Sharma, and Ameet Talwalkar. Learning-tolearn non-convex piecewise-Lipschitz functions. In Advances in Neural Information Processing Systems, 2021.
[13] Soumya Basu, Branislav Kveton, Manzil Zaheer, and Csaba Szepesvári. No regrets for learning the prior in bandits. In Advances in Neural Information Processing Systems, 2021.
[52] Constantino Tsallis. Possible generalization of Boltzmann-Gibbs statistics. Journal of Statistical Physics, 52:479-487, 1988.
[53] Chen-Yu Wei and Haipeng Luo. Non-stationary reinforcement learning without prior knowledge: An optimal black-box approach. In Proceedings of the 34th Annual Conference on Learning Theory, 2021.
[54] Takuya Yamano. Some properties of q-logarithm and q-exponential functions in Tsallis statistics. Physica A: Statistical Mechanics and its Applications, 305:486-496, 2002.
[55] Peng Zhao, Guanghui Wang, Lijun Zhang, and Zhi-Hua Zhou. Bandit convex optimization in non-stationary environments. Journal of Machine Learning Research, 22(125):1-45, 2021.
A Structural results
A.1 Properties of the Bregman divergence
Lemma A.1. Let  $\psi : \mathcal{C} \mapsto \mathbb{R}$  be a strictly convex function with  $\max_{\mathbf{x} \in \mathcal{C}} \| \nabla^2 \psi(\mathbf{x}) \|_2 \leq S$  over a convex set  $\mathcal{C} \subset \mathbb{R}^d$  over size  $\max_{\mathbf{x} \in \mathcal{C}} \| \mathbf{x} \|_2 \leq K$ , and let  $B(\cdot | | \cdot)$  be the Bregman divergence generated by  $\psi$ . Then for any points  $\mathbf{x}_1, \ldots, \mathbf{x}_T \in \mathcal{C}$  the actions  $\mathbf{y}_1 = \arg \min_{\mathbf{x} \in \mathcal{C}} \psi(\mathbf{x})$  and  $\mathbf{y}_t = \frac{1}{t - 1} \sum_{s < t} \mathbf{x}_s$  have regret
$$
\sum_ {t = 1} ^ {T} B \left(\mathbf {x} _ {t} \mid \left| \mathbf {y} _ {t} \right.\right) - B \left(\mathbf {x} _ {t} \mid \left| \mathbf {y} _ {T + 1} \right.\right) \leq \sum_ {t = 1} ^ {T} \frac {8 S K ^ {2}}{2 t - 1} \leq 8 S K ^ {2} (1 + \log T) \tag {21}
$$
$$
\nabla_ {\mathbf {y}} B (\mathbf {x} | | \mathbf {y}) = - \nabla \psi (\mathbf {y}) - \nabla_ {\mathbf {y}} \langle \nabla \psi (\mathbf {y}), \mathbf {x} \rangle + \nabla_ {\mathbf {y}} \langle \nabla \psi (\mathbf {y}), \mathbf {y} \rangle = \operatorname {d i a g} \left(\nabla^ {2} \psi (\mathbf {y})\right) (\mathbf {y} - \mathbf {x}) \tag {22}
$$
so  $B(\mathbf{x}_t||\mathbf{y})$  is 2SK-Lipschitz w.r.t. the Euclidean norm. Applying Khodak et al. [31, Prop. B.1] yields the result (note that its assumption of strong convexity of the regularizer can be replaced with strict convexity without changing the proof or result).
Claim A.1. Let  $\psi : \mathcal{K} \mapsto \mathbb{R}$  be a strictly-convex function with Bregman divergence  $B(\cdot ||\cdot)$  over a convex set  $\mathcal{K} \subset \mathbb{R}^d$  containing points  $\mathbf{x}_1, \ldots, \mathbf{x}_T$ . Then their mean  $\bar{\mathbf{x}} = \frac{1}{T} \sum_{t=1}^{T} \mathbf{x}_t$  satisfies
$$
\sum_ {t = 1} ^ {T} B \left(\mathbf {x} _ {t} \mid \mid \bar {\mathbf {x}}\right) = \sum_ {t = 1} ^ {T} \psi \left(\mathbf {x} _ {t}\right) - \psi (\bar {\mathbf {x}}) \tag {23}
$$
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} B (\mathbf {x} _ {t} | | \bar {\mathbf {x}}) = \sum_ {t = 1} ^ {T} \psi (\mathbf {x} _ {t}) - \psi (\bar {\mathbf {x}}) - \langle \nabla \psi (\bar {\mathbf {x}}), \mathbf {x} _ {t} - \bar {\mathbf {x}} \rangle \\ = \sum_ {t = 1} ^ {T} \psi (\mathbf {x} _ {t}) - \psi (\bar {\mathbf {x}}) - \left\langle \nabla \psi (\bar {\mathbf {x}}), \sum_ {t = 1} ^ {T} \mathbf {x} _ {t} - \bar {\mathbf {x}} \right\rangle = \sum_ {t = 1} ^ {T} \psi (\mathbf {x} _ {t}) - \psi (\bar {\mathbf {x}}) \\ \end{array}
$$
A.2 Tuning the step-size
Lemma A.2. Let  $\ell_1, \ldots, \ell_T: \mathbb{R}_{>0} \mapsto \mathbb{R}_{>0}$  be a sequence of functions of form  $\ell_t(x) = \frac{B_t^2}{x} + G^2 x$  for adversarially chosen  $B_t \in [0, D]$  and some  $G > 0$ . Then for any  $\rho \geq 0$ , the actions of EWOO [26, Fig. 4] with parameter  $\frac{2\rho^2}{DG}$  run on the modified losses  $\frac{B_t^2 + \rho^2D^2}{x} + G^2 x$  over the domain  $\left[\frac{\rho D}{G}, \frac{D}{G}\sqrt{1 + \rho^2}\right]$  achieves regret w.r.t. any  $x > 0$  of
$$
\sum_ {t = 1} ^ {T} \ell_ {t} (x) - \ell_ {t} (x) \leq \min  \left\{\frac {\rho^ {2} D ^ {2}}{x}, \rho D G \right\} T + \frac {D G (1 + \log (T + 1))}{2 \rho^ {2}} \tag {25}
$$
Proof. By Khodak et al. [31, Prop. C.1] the modified functions are  $\frac{2\rho^2}{DG}$ -exp-concave. Then Khodak et al. [31, Cor. C.2] with  $B_{t}$  set to  $\frac{B_t}{G}$ ,  $D$  to  $\frac{D}{G}$ ,  $\alpha_{t} = G^{2}$ , and  $\varepsilon = \frac{\rho D}{G}$  yields the result.
Lemma A.3. For  $\hat{\mathbf{x}}_1,\dots ,\hat{\mathbf{x}}_T\in \partial \mathcal{K}$  consider a sequence of functions of form
$$
U _ {t} (\mathbf {x}, \eta) = \frac {B \left(\mathbf {c} _ {\varepsilon} \left(\hat {\mathbf {x}} _ {t}\right) \right\| \mathbf {x})}{\eta} + \eta G ^ {2} m \tag {26}
$$
where  $B$  is the Bregman divergence of a strictly convex d.g.f.  $\psi : \mathcal{K}^{\circ} \mapsto \mathbb{R}$  and where  $\mathbf{x}_1 = \arg \min_{\mathbf{x} \in \mathcal{K}} \psi(\mathbf{x})$  defines the projection  $\mathbf{c}_{\varepsilon}(\mathbf{x}) = \mathbf{x}_1 + \frac{\mathbf{x} - \mathbf{x}_1}{1 + \varepsilon}$  for some  $\varepsilon > 0$ . Suppose we play  $\mathbf{x}_{t+1} \gets \mathbf{c}_{\varepsilon}\left(\frac{1}{t} \sum_{s=1}^{t} \hat{\mathbf{x}}_s\right)$  and set  $\eta_t$  using the actions of EWOO [26, Fig. 4] with parameter  $\frac{2\rho^2}{DG}$  for some  $\rho, D_{\varepsilon} > 0$  s.t.  $B(\mathbf{c}_{\varepsilon}(\hat{\mathbf{x}}_t) || \mathbf{x}) \leq D_{\varepsilon}^2 \forall \mathbf{x} \in \mathcal{K}_{\varepsilon}$  on the functions  $\frac{B(\mathbf{c}_{\varepsilon}(\hat{\mathbf{x}}_t) || \mathbf{x}_t) + \rho^2 D_{\varepsilon}^2}{\eta} + \eta G^2 m$  over the domain  $\left[\frac{\rho D_{\varepsilon}}{G \sqrt{m}}, \frac{D_{\varepsilon}}{G} \sqrt{\frac{1 + \rho^2}{m}}\right]$ , with  $\eta_1$  being at the midpoint of the domain. Then  $U_t(\mathbf{x}_t, \eta_t) \leq D_{\varepsilon} G \sqrt{m}\left(\frac{1}{\rho} + \sqrt{1 + \rho^2}\right) \forall t \in [T]$  and
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} U _ {t} (\mathbf {x} _ {t}, \eta_ {t}) \leq \min  _ {\eta > 0, \mathbf {x} \in \mathcal {K}} \sum_ {t = 1} ^ {T} \frac {B (\mathbf {c} _ {\varepsilon} (\hat {\mathbf {x}} _ {t}) | | \mathbf {x})}{\eta} + \eta G ^ {2} m \\ + \min  \left\{\frac {\rho^ {2} D _ {\varepsilon} ^ {2}}{\eta}, \rho D _ {\varepsilon} G \right\} T + \frac {D _ {\varepsilon} G (1 + \log (T + 1))}{2 \rho^ {2}} + \frac {8 S _ {\varepsilon} K ^ {2} (1 + \log T)}{\eta} \tag {27} \\ \end{array}
$$
for  $K = \max_{\mathbf{x}\in \mathcal{K}}\| \mathbf{x}\| _2$  and  $S_{\varepsilon} = \max_{\mathbf{x}\in \mathcal{K}_{\varepsilon}}\| \nabla^{2}\psi (\mathbf{x})\| _2$
Proof. The first claim follows by directly substituting the worst-case values of  $\eta$  into  $U_{t}(\mathbf{x},\eta)$ . For the second, apply Lemma A.2 followed by Lemma A.1:
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} U _ {t} (\mathbf {x} _ {t}, \eta_ {t}) \\ = \sum_ {t = 1} ^ {T} \frac {B (\mathbf {c} _ {\varepsilon} (\hat {\mathbf {x}} _ {t}) | | \mathbf {x} _ {t})}{\eta_ {t}} + \eta_ {t} G ^ {2} m \\ \leq \min  _ {\eta > 0} \min  \left\{\frac {\rho^ {2} D _ {\varepsilon} ^ {2}}{\eta}, \rho D _ {\varepsilon} G \right\} T + \frac {D _ {\varepsilon} G (1 + \log (T + 1))}{2 \rho^ {2}} + \sum_ {t = 1} ^ {T} \frac {B (\mathbf {c} _ {\varepsilon} (\hat {\mathbf {x}} _ {t}) | | \mathbf {x})}{\eta} + \eta G ^ {2} m \\ \leq \min  _ {\eta > 0} \min  \left\{\frac {\rho^ {2} D _ {\varepsilon} ^ {2}}{\eta}, \rho D _ {\varepsilon} G \right\} T + \frac {D _ {\varepsilon} G (1 + \log (T + 1))}{2 \rho^ {2}} + \frac {8 S _ {\varepsilon} K ^ {2} (1 + \log T)}{\eta} \\ + \min  _ {\mathbf {x} \in \mathcal {K} _ {\varepsilon}} \sum_ {t = 1} ^ {T} \frac {B \left(\mathbf {c} _ {\varepsilon} \left(\hat {\mathbf {x}} _ {t}\right) \mid \mid \mathbf {x}\right)}{\eta} + \eta G ^ {2} m \tag {28} \\ \end{array}
$$
Conclude by noting that the sum of Bregman divergence to  $\mathbf{c}_{\varepsilon}(\hat{\mathbf{x}}_t)$  is minimized on their convex hull, a subset of  $\kappa_{\varepsilon}$ .
A.3 Computational and space complexity
Algorithm 1 implicitly maintains a separate copy of FTL for each hyperparameter in the continuous space of EWOO and the grid  $\Theta_{k}$  over the domain of  $\theta$ , but explicitly just needs to average the estimated task-optima  $\hat{\mathbf{x}}_t$ ; this is due to the mean-as-minimizer property of Bregman divergences and the linearity of  $\mathbf{c}_{\varepsilon}$ . Thus the memory it uses is  $\mathcal{O}(d + k)$ , where  $k$  is size of the discretization of  $\Theta$  and should be viewed as sublinear in  $T$ , e.g. for MAB with implicit exploration and BLO  $k = \mathcal{O}(\sqrt[4]{d}\sqrt{T})$ . Computationally, at each timestep  $t$  and for each grid point we must compute two single-dimensional integrals; the integrands are sums of upper bounds that just need to be incremented once per round, leading to a total per-iteration complexity of  $\mathcal{O}(k)$  (ignoring the running of OMD). Although outside the scope of this work, it may be possible to avoid integration by tuning  $\eta$  with MW as well, rather than EWOO, but likely at the cost of worse regret because it would not take advantage of the exp-concavity of  $U_t^{(\rho)}$ .
A.4 Main structural result
Theorem A.1. Consider a family of strictly convex functions  $\psi_{\theta}:\mathcal{K}^{\circ}\mapsto \mathbb{R}$  parameterized by  $\theta$  lying in an interval  $\Theta \subset \mathbb{R}$  of radius  $R_{\Theta}$  that are all minimized at the same  $\mathbf{x}_1\in \mathcal{K}^\circ$ , and for  $\hat{\mathbf{x}}_1,\ldots ,\hat{\mathbf{x}}_T\in$ $\partial \mathcal{K}$  consider a sequence of functions of form  $U_{t}(\mathbf{x},\eta ,\theta)$  (3), as well as the associated regularized upper bounds  $U_{t}^{(\rho)}$  (4). Define the maximum divergence  $D = \max_{\theta \in \Theta}D_{\theta}$ , radius  $K = \max_{\mathbf{x}\in \mathcal{K}}\| \mathbf{x}\| _2$  and  $L_{\eta}$  the Lipschitz constant w.r.t.  $\theta \in \Theta$  of  $\frac{\hat{V}_{\theta}^2}{\eta} +\eta g(\theta)m + f(\theta)m$ . Then Algorithm 1 with  $\Theta_k\subset \Theta$  the uniform discretization of  $\Theta$  s.t.  $\max_{\theta \in \Theta}\min_{\theta '\in \Theta_k}|\theta -\theta '|\leq \frac{R_\Theta}{k}$ ,  $\rho \in (0,1)$ ,  $\underline{\eta} (\theta) = \frac{\rho D_{\theta}}{\sqrt{g(\theta)m}}$ ,  $\overline{\eta} (\theta) = D_{\theta}\sqrt{\frac{1 + \rho^2}{g(\theta)m}}$ ,  $\alpha (\theta) = \frac{2\rho^2}{D_\theta\sqrt{g(\theta)m}}$ , and  $\lambda = \left(M\left(\frac{1}{\rho} +\sqrt{1 + \rho^2}\right) + Fm\right)^{-1}\sqrt{\frac{\log k}{2T}}$  leads to a sequence  $(\mathbf{x}_t,\eta_t(\theta_t),\theta_t)$  s.t.  $\mathbb{E}\sum_{t = 1}^{T}U_{t}(\mathbf{x}_{t},\eta_{t}(\theta_{t}),\theta_{t})$  is bounded by
$$
\begin{array}{l} \mathbb {E} \min  _ {\theta \in \Theta , \eta > 0} \frac {8 S K ^ {2} (1 + \log T)}{\eta} + \left(\frac {\hat {V} _ {\theta} ^ {2}}{\eta} + \eta g (\theta) m + f (\theta) m + \frac {L _ {\eta} R _ {\Theta}}{k} + \min  \left\{\frac {\rho^ {2} D ^ {2}}{\eta}, \rho M \right\}\right) T \\ + \left(\frac {4 M}{\rho} + F m\right) \sqrt {T \log k} + \frac {M (1 + \log (T + 1))}{2 \rho^ {2}} \tag {29} \\ \end{array}
$$