SlowGuess/NeurIPS_2023_OCR huggingface dataset:
and  $\sum_{t=1}^{T} U_t(\mathbf{x}_t, \eta_t(\theta_t), \theta_t)$  is bounded w.p.  $\geq 1 - \delta 1_{k > 1}$  by
$$
\begin{array}{l} \min  _ {\theta \in \Theta , \eta > 0} \frac {8 S K ^ {2} (1 + \log T)}{\eta} + \left(\frac {\hat {V} _ {\theta} ^ {2}}{\eta} + \eta g (\theta) m + f (\theta) m + \frac {L _ {\eta} R _ {\Theta}}{k} + \min  \left\{\frac {\rho^ {2} D ^ {2}}{\eta}, \rho M \right\}\right) T \\ + \left(\frac {4 M}{\rho} + F m\right) \left(\sqrt {T \log k} + 1 _ {k > 1} \sqrt {\frac {T}{2} \log \frac {1}{\delta}}\right) + \frac {M (1 + \log (T + 1))}{2 \rho^ {2}} \tag {30} \\ \end{array}
$$
Proof. In the following proof, we first consider online learning  $U_{t}(\cdot ,\cdot ,\theta)$  for fixed  $\theta \in \Theta_k$ . To tune  $\eta$ , we online learn the one-dimensional losses  $B_{\theta}(\mathbf{c}_{\theta}(\hat{\mathbf{x}}_t)||\mathbf{c}_{\theta}(\mathbf{x}_t)) / \eta +\eta g(\theta)$ , where  $\mathbf{c}_{\theta}(\hat{\mathbf{x}}_t)$  is the  $(\eta_t(\theta)$ -independent) action of FTL at time  $t$ . As discussed, the corresponding regularized losses  $U_{t}^{(\rho)}$  are exp-concave, and so running EWOO yields  $\tilde{\mathcal{O}}\left(M / \rho^2 +\min \left\{\rho^2 D^2 /\eta ,\rho M\right\} T\right)$  regret w.r.t. the original sequence [31, Cor. C.2]. At the same time, we show that FTL has logarithmic regret on the sequence  $B_{\theta}(\mathbf{c}_{\theta}(\hat{\mathbf{x}}_t)||\cdot)$  that scales with the spectral norm  $S$  of  $\nabla^2\psi_\theta$  (c.f. Lem. A.1), and that the average loss of the optimal comparator is  $\hat{V}_{\theta}^{2}$  (c.f. Claim A.1). Thus, since we only care about a fixed comparator  $\eta$ , dividing by  $\eta T$  yields the first and last terms (5). We run a copy of these algorithms for each  $\theta \in \Theta_k$ ; since their losses are bounded by  $\tilde{\mathcal{O}} (M / \rho +Fm)$ , textbook results for MW yield  $\mathcal{O}(\sqrt{T\log k})$  regret w.r.t.  $\theta \in \Theta_k$ , which we then extend to  $\Theta \supset \Theta_k$  using  $L_{\eta}$ -Lipschitzness.
Formally, we have that
$$
\begin{array}{l} \mathbb {E} \sum_ {t = 1} ^ {T} U _ {t} (\mathbf {x} _ {t}, \eta_ {t} (\theta_ {t}), \theta_ {t}) \\ = \mathbb {E} \sum_ {t = 1} ^ {T} \frac {B _ {\theta_ {t}} \left(\mathbf {c} _ {\theta_ {t}} \left(\hat {\mathbf {x}} _ {t}\right) \mid \mid \mathbf {x} _ {t}\right)}{\eta_ {t} \left(\theta_ {t}\right)} + \eta_ {t} \left(\theta_ {t}\right) g (\theta) m + f (\theta) m \\ \leq \left(M \left(\frac {1}{\rho} + \sqrt {2}\right) + F m\right) \sqrt {2 T \log k} + \mathbb {E} \min _ {\theta \in \Theta_ {k}} \sum_ {t = 1} ^ {T} \frac {B _ {\theta} (\mathbf {c} _ {\theta} (\hat {\mathbf {x}} _ {t}) | | \mathbf {x} _ {t})}{\eta_ {t} (\theta)} + \eta_ {t} (\theta) g (\theta) m + f (\theta) m \\ \leq \left(\frac {4 M}{\rho} + F m\right) \sqrt {T \log k} + \mathbb {E} \min  _ {\theta \in \Theta_ {k}, \eta > 0, \mathbf {x} \in \mathcal {K}} \sum_ {t = 1} ^ {T} \frac {B _ {\theta} (\mathbf {c} _ {\theta} (\hat {\mathbf {x}} _ {t}) | | \mathbf {x})}{\eta} + \eta g (\theta) m + f (\theta) m \\ + \min  \left\{\frac {\rho^ {2} D _ {\theta} ^ {2}}{\eta}, \rho D _ {\theta} \sqrt {g (\theta) m} \right\} T + \frac {D _ {\theta} \sqrt {g (\theta) m} (1 + \log (T + 1))}{2 \rho^ {2}} + \frac {8 S K ^ {2} (1 + \log T)}{\eta} \tag {31} \\ \end{array}
$$
where the first inequality is the regret of multiplicative weights with step-size  $\lambda$  [46, Cor. 2.14] and the second is by applying Lemma A.3 for each  $\theta$ . We then simplify and apply the definition of  $\hat{V}_{\theta}^{2}$  via Claim A.1 and conclude by applying Lipschitzness w.r.t.  $\theta$ :
$$
\begin{array}{l} \mathbb {E} \sum_ {t = 1} ^ {T} U _ {t} (\mathbf {x} _ {t}, \eta_ {t} (\theta_ {t}), \theta_ {t}) \\ \leq \left(\frac {4 M}{\rho} + F m\right) \sqrt {T \log k} + \mathbb {E} \min  _ {\theta \in \Theta_ {k}, \eta > 0} \frac {\hat {V} _ {\theta} ^ {2} T}{\eta} + \eta g (\theta) m T + f (\theta) m T \\ + \min  \left\{\frac {\rho^ {2} D ^ {2}}{\eta}, \rho M \right\} T + \frac {M (1 + \log (T + 1))}{2 \rho^ {2}} + \frac {8 S K ^ {2} (1 + \log T)}{\eta} \\ \end{array}
$$
$$
\begin{array}{l} \leq \mathbb {E} \min  _ {\theta \in \Theta , \eta > 0} \frac {8 S K ^ {2} (1 + \log T)}{\eta} + \left(\frac {\hat {V} _ {\theta} ^ {2}}{\eta} + \eta g (\theta) m + f (\theta) m + \frac {L _ {\eta} R _ {\Theta}}{k} + \min  \left\{\frac {\rho^ {2} D ^ {2}}{\eta}, \rho M \right\}\right) T \\ + \left(\frac {4 M}{\rho} + F m\right) \sqrt {T \log k} + \frac {M (1 + \log (T + 1))}{2 \rho^ {2}} \tag {32} \\ \end{array}
$$
The w.h.p. guarantee follows by Cesa-Bianchi and Lugosi [16, Lem. 4.1].
B Implicit exploration
B.1 Properties of the Tsallis entropy
Lemma B.1. For any  $\varepsilon \in (0,1]$  and  $\mathbf{x} \in \triangle s.t.$ $\mathbf{x}(a) \geq \frac{\varepsilon}{d} \forall a \in [d]$  the  $\beta$ -Tsallis entropy  $H_{\beta}(\mathbf{x}) = -\frac{1 - \sum_{a=1}^{d} \mathbf{x}^{\beta}(a)}{1 - \beta}$  is  $d \log \frac{d}{\varepsilon}$ -Lipschitz w.r.t.  $\beta \in [0,1]$ .
Proof. Let  $\log_{\beta}x = \frac{x^{1 - \beta} - 1}{1 - \beta}$  be the  $\beta$ -logarithm function and note that by Yamano [54, Equation 6] we have  $\log_{\beta}x - \log x = (1 - \beta)(\partial_b\log_{\beta}x + \log_{\beta}x\log x)\geq 0\forall \beta \in [0,1]$ . Then we have for  $\beta \in [0,1)$  that
$$
\begin{array}{l} | \partial_ {\beta} H _ {\beta} (\mathbf {x}) | = \left| \frac {- H _ {\beta} (\mathbf {x}) - \sum_ {a = 1} ^ {d} \mathbf {x} ^ {\beta} (a) \log \mathbf {x} (a)}{1 - \beta} \right| \\ = \frac {1}{1 - \beta} \left| \sum_ {a = 1} ^ {d} \mathbf {x} ^ {\beta} (a) \left(\log_ {\beta} \mathbf {x} (a) - \log \mathbf {x} (a)\right) \right| \\ = \frac {1}{1 - \beta} \sum_ {a = 1} ^ {d} \mathbf {x} ^ {\beta} (a) \left(\log_ {\beta} \mathbf {x} (a) - \log \mathbf {x} (a)\right) \tag {33} \\ \leq \frac {1}{1 - \beta} \left(\sum_ {a = 1} ^ {d} \mathbf {x} (a)\right) ^ {\beta} \left(\sum_ {a = 1} ^ {d} \left(\log_ {\beta} \mathbf {x} (a) - \log \mathbf {x} (a)\right) ^ {\frac {1}{1 - \beta}}\right) ^ {1 - \beta} \\ \leq \frac {1}{1 - \beta} \sum_ {a = 1} ^ {d} \log_ {\beta} \mathbf {x} (a) - \log \mathbf {x} (a) \leq \frac {d}{1 - \beta} (\log_ {\beta} \frac {d}{\varepsilon} - \log \frac {d}{\varepsilon}) \leq - d \log \frac {d}{\varepsilon} \\ \end{array}
$$
where the fourth inequality follows by Hölder's inequality, the fifth by subadditivity of  $x^a$  for  $a \in (0,1]$ , the sixth by the fact that  $\partial_x(\log_\beta x - \log x) = x^{-\beta} - 1 / x \leq 0 \forall \beta, x \in [0,1)$ , and the last line by substituting  $\beta = 0$  since  $\partial_\beta \left( \frac{\log_\beta x - \log x}{1 - \beta} \right) = \frac{2(x - x^\beta) - (1 - \beta)(x^\beta + x) \log x}{x^\beta (1 - \beta)^3} \leq 0 \forall \beta \in [0,1), x \in (0,1 / d]$ . For  $\beta = 1$ , applying L'Hôpital's rule yields
$$
\lim  _ {\beta \rightarrow 1} \partial_ {\beta} H _ {\beta} (\mathbf {x}) = - \frac {1}{2} \lim  _ {\beta \rightarrow 1} \sum_ {a = 1} ^ {d} \mathbf {x} ^ {\beta} (a) \log^ {2} \mathbf {x} (a) (1 - (1 - \beta) \log \mathbf {x} (a)) = - \frac {1}{2} \sum_ {a = 1} ^ {d} \mathbf {x} (a) \log^ {2} \mathbf {x} (a) \tag {34}
$$
which is bounded on  $[-2d / e^2, 0]$ .
Lemma B.2. Consider  $\mathbf{x}_1, \ldots, \mathbf{x}_T \in \triangle$  s.t.  $\mathbf{x}_t(a_t) = 1$  for some  $a_t \in [d]$ , and let  $\bar{\mathbf{x}} = \frac{1}{T} \sum_{t=1}^{T} \mathbf{x}_t$  be their average. For any  $\varepsilon \in (0,1]$  and  $\beta \in (0,1]$  we have that for every  $t \in [T]$
$$
H _ {\beta} \left(\bar {\mathbf {x}} ^ {(\varepsilon)}\right) - H _ {\beta} \left(\mathbf {x} _ {t} ^ {(\varepsilon)}\right) \leq H _ {\beta} (\bar {\mathbf {x}}) \tag {35}
$$
where recall that  $\mathbf{x}^{(\varepsilon)} = \mathbf{c}_{\frac{\varepsilon}{1 - \varepsilon}}(\mathbf{x}) = \mathbf{1}_d / d + (1 - \varepsilon)(\mathbf{x} - \mathbf{1}_d / d) = (1 - \varepsilon)\mathbf{x} + \frac{\varepsilon}{d}\mathbf{1}_d.$
Proof. Assume w.l.o.g. that  $\bar{\mathbf{x}}(1) \leq \bar{\mathbf{x}}(2) \leq \ldots \leq \bar{\mathbf{x}}(d)$  and  $a_t = 1$ , so that  $\mathbf{x}_t^{(\varepsilon)} = \mathbf{e}_1^{(\varepsilon)}$ . We take the derivative
$$
\begin{array}{l} \partial_ {\varepsilon} H _ {\beta} \left((1 - \varepsilon) \bar {\mathbf {x}} + \frac {\varepsilon}{d} \mathbf {1} _ {d}\right) - \partial_ {\varepsilon} H _ {\beta} \left(\mathbf {e} _ {1} ^ {(\varepsilon)}\right) \\ = \frac {d}{1 - \beta} \sum_ {a = 1} ^ {d - 1} \left(\frac {1}{((1 - \varepsilon) \bar {\mathbf {x}} (a) + \varepsilon / d) ^ {1 - \beta}} - \frac {1}{(\varepsilon / d) ^ {1 - \beta}}\right) \\ + \frac {d}{1 - \beta} \sum_ {a = 1} ^ {d - 1} \left(\frac {1}{\left((1 - \varepsilon) + \varepsilon / d\right) ^ {1 - \beta}} - \frac {1}{\left((1 - \varepsilon) \bar {\mathbf {x}} (d) + \varepsilon / d\right) ^ {1 - \beta}}\right) \tag {36} \\ + \frac {d ^ {2}}{1 - \beta} \sum_ {a = 1} ^ {d - 1} \bar {\mathbf {x}} (a) \left(\frac {1}{((1 - \varepsilon) \bar {\mathbf {x}} (d) + \varepsilon / d) ^ {1 - \beta}} - \frac {1}{((1 - \varepsilon) \bar {\mathbf {x}} (a) + \varepsilon / d) ^ {1 - \beta}}\right) \\ \end{array}
$$
By the assumption that  $\bar{\mathbf{x}}(a)$  is non-decreasing in  $a$ , each of the summands above become non-positive. So for  $\varepsilon \in (0,1]$  the derivative is non-positive, and for  $\varepsilon \to 0^+$  it goes to  $-\infty$ . Thus the l.h.s. of the bound is monotonically non-increasing in  $\varepsilon$  for all  $\varepsilon \in [0,1]$ . The result then follows from the fact that for  $\varepsilon = 0$  we have  $H_{\beta}\left((1 - \varepsilon)\bar{\mathbf{x}} + \frac{\varepsilon}{d}\mathbf{1}_d\right) - H_{\beta}\left(\mathbf{e}_1^{(\varepsilon)}\right) = H_{\beta}(\bar{\mathbf{x}})$ .
B.2 Implicit exploration bounds
Lemma B.3. Suppose we play  $OMD_{\beta, \eta}$  with regularizer  $\psi_{\beta}$  the negative Tsallis entropy and initialization  $\mathbf{x}_1 \in \triangle$  on the sequence of linear loss functions  $\ell_1, \ldots, \ell_T \in [0, 1]^d$ . Then for any  $\mathbf{x} \in \triangle$  we have
$$
\sum_ {t = 1} ^ {T} \left\langle \ell_ {t}, \mathbf {x} _ {t} - \mathbf {x} \right\rangle \leq \frac {B _ {\beta} (\mathbf {x} \mid \mid \mathbf {x} _ {1})}{\eta} + \frac {\eta}{\beta} \sum_ {a = 1} ^ {d} \mathbf {x} _ {t} ^ {2 - \beta} (a) \ell_ {t} ^ {2} (a) \tag {37}
$$
Proof. Note that the following proof follows parts of the course notes by Luo [37], which we reproduce for completeness. The OMD update at each step  $t$  involves the following two steps: set  $\mathbf{y}_{t + 1}\in \triangle$  s.t.  $\nabla \psi_{\beta}(\mathbf{y}_{t + 1}) = \nabla \psi_{\beta}(\mathbf{x}_t) - \eta \ell_t$  and then set  $\mathbf{x}_{t + 1} = \arg \min_{\mathbf{x}\in \triangle}B_{\beta}(\mathbf{x},\mathbf{y}_{t + 1})$  [25, Algorithm 14]. Note that by Hazan [25, Equation 5.3] and nonnegativity of the Bregman divergence we have
$$
\sum_ {t = 1} ^ {T} \left\langle \ell_ {t}, \mathbf {x} _ {t} - \mathbf {x} \right\rangle \leq \frac {B _ {\beta} (\mathbf {x} | | \mathbf {x} _ {1})}{\eta} + \frac {1}{\eta} \sum_ {t = 1} ^ {T} B _ {\beta} (\mathbf {x} _ {t} | | \mathbf {y} _ {t + 1}) \tag {38}
$$
To bound the second term, note that when  $\psi_{\beta}$  is the negative Tsallis entropy we have
$$
\begin{array}{l} B _ {\beta} (\mathbf {x} _ {t} | | \mathbf {y} _ {t + 1}) \\ = \frac {1}{1 - \beta} \sum_ {a = 1} ^ {d} \left(\mathbf {y} _ {t + 1} ^ {\beta} (a) - \mathbf {x} _ {t} ^ {\beta} (a) + \frac {\beta}{\mathbf {y} _ {t + 1} ^ {1 - \beta} (a)} (\mathbf {x} _ {t} (a) - \mathbf {y} _ {t + 1} (a)\right) \\ = \frac {1}{1 - \beta} \sum_ {a = 1} ^ {d} \left((1 - \beta) \mathbf {y} _ {t + 1} ^ {\beta} (a) - \mathbf {x} _ {t} ^ {\beta} (a) + \beta \left(\frac {1}{\mathbf {x} _ {t} ^ {1 - \beta} (a)} + \frac {1 - \beta}{\beta} \eta \ell_ {t} (a)\right) \mathbf {x} _ {t} (a)\right) \tag {39} \\ = \sum_ {a = 1} ^ {d} \left(\mathbf {y} _ {t + 1} ^ {\beta} (a) - \mathbf {x} _ {t} ^ {\beta} (a) + \eta \mathbf {x} _ {t} (a) \ell_ {t} (a)\right) \\ \end{array}
$$
Plugging the following result, which follows from  $(1 + x)^{\alpha} \leq 1 + \alpha x + \alpha (\alpha - 1)x^{2} \forall x \geq 0, \alpha < 0$ , into the above yields the desired bound.
$$
\begin{array}{l} \mathbf {y} _ {t + 1} ^ {\beta} (a) = \mathbf {x} _ {t} ^ {\beta} (a) \left(\frac {\mathbf {y} _ {t + 1} ^ {\beta - 1} (a)}{\mathbf {x} _ {t} ^ {\beta - 1} (a)}\right) ^ {\frac {\beta}{\beta - 1}} = \mathbf {x} _ {t} ^ {\beta} (a) \left(1 + \frac {1 - \beta}{\beta} \eta \mathbf {x} _ {t} ^ {1 - \beta} (a) \ell_ {t} (a)\right) ^ {\frac {\beta}{\beta - 1}} \\ \leq \mathbf {x} _ {t} ^ {\beta} (a) \left(1 - \eta \mathbf {x} _ {t} ^ {1 - \beta} (a) \ell_ {t} (a) + \frac {\eta^ {2}}{\beta} \mathbf {x} _ {t} ^ {2 - 2 \beta} (a) \ell_ {t} (a) ^ {2}\right) \tag {40} \\ = \mathbf {x} _ {t} ^ {\beta} (a) - \eta \mathbf {x} _ {t} (a) \ell_ {t} (a) + \frac {\eta^ {2}}{\beta} \mathbf {x} _ {t} ^ {2 - \beta} (a) \ell_ {t} (a) ^ {2} \\ \end{array}
$$
Theorem B.1. In Algorithm 1, let  $OMD_{\eta,\beta}$  be online mirror descent with the Tsallis entropy regularizer  $\psi_{\beta}$  over  $\gamma$ -offset loss estimators,  $\Theta_k$  is a subset of  $[\underline{\beta}, \overline{\beta}] \subset [\frac{1}{\log d}, 1]$ , and
$$
U _ {t} (\mathbf {x}, \eta , \beta) = \frac {B _ {\beta} \left(\hat {\mathbf {x}} _ {t} ^ {(\varepsilon)} \right\| \mathbf {x})}{\eta} + \frac {\eta d ^ {\beta} m}{\beta} \tag {41}
$$
where  $\hat{\mathbf{x}}_t^{(\varepsilon)} = (1 - \varepsilon)\hat{\mathbf{x}}_t + \varepsilon \mathbf{1}_d / d$ . Note that  $U_t^{(\rho)}(\mathbf{x},\eta,\beta) = U_t(\mathbf{x},\eta,\beta) + \frac{\rho^2(d^{1-\beta}-1)}{\eta(1-\beta)}$ . Then there exists settings of  $\underline{\eta},\overline{\eta},\alpha,\lambda$  s.t. for all  $\varepsilon,\rho,\gamma\in(0,1)$  we have w.p.  $\geq 1-\delta$  that
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {m} \ell_ {t, i} (a _ {t, i}) - \ell_ {t, i} (\mathring {a} _ {t}) \\ \leq (\varepsilon + \gamma d) m T + \frac {2 + \sqrt {\frac {d \log d}{e m}}}{\gamma} \log \frac {5}{\delta} + \frac {8 d \sqrt {m}}{\rho} \left(1 _ {k > 1} \sqrt {T \log \frac {5 k}{\delta}} + \frac {1 + \log (T + 1)}{1 6 \rho}\right) \\ + \min  _ {\beta \in [ \underline {{\beta}}, \bar {\beta} ], \eta > 0} \frac {8 \left(\frac {d}{\varepsilon}\right) ^ {2 - \underline {{\beta}}} (1 + \log T)}{\eta} + \left(\frac {\hat {H} _ {\beta}}{\eta} + \frac {\eta d ^ {\beta} m}{\beta} + \frac {L _ {\eta} (\bar {\beta} - \underline {{\beta}})}{2 k} + d \min  \left\{\frac {\rho^ {2}}{2 \eta}, \rho \sqrt {m} \right\}\right) T \tag {42} \\ \end{array}
$$
$$
f o r L _ {\eta} = \left(\frac {\log \frac {d}{\varepsilon}}{\eta} + \eta m \log^ {2} d\right) d.
$$
Proof. In this setting we have  $g(\beta) = d^{\beta} / \beta$ ,  $f(\beta) = 0$ ,  $D_{\beta}^{2} = \frac{d^{1 - \beta} - 1}{1 - \beta}$ ,  $D \leq \sqrt{d / 2}$ ,  $M = d\sqrt{m}$ ,  $F = 0$ ,  $S = (d / \varepsilon)^{2 - \beta}$ , and  $K = 1$ . We have that
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {m} \ell_ {t, i} (a _ {t, i}) - \ell_ {t, i} (\mathring {a} _ {t}) \\ = \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {m} \left\langle \hat {\ell} _ {t, i}, \mathbf {x} _ {t, i} \right\rangle - \ell_ {t, i} (\hat {a} _ {t}) + \gamma \sum_ {a = 1} ^ {d} \hat {\ell} _ {t, i} (a) \\ \leq \sum_ {t = 1} ^ {T} \frac {B _ {\beta_ {t}} (\hat {\mathbf {x}} _ {t} ^ {(\varepsilon)} | | \mathbf {x} _ {t , 1})}{\eta_ {t}} + \sum_ {i = 1} ^ {m} \langle \hat {\ell} _ {t, i}, \hat {\mathbf {x}} _ {t} ^ {(\varepsilon)} \rangle - \ell_ {t, i} (\mathring {a} _ {t}) + \frac {\eta_ {t}}{\beta_ {t}} \sum_ {a = 1} ^ {d} \mathbf {x} _ {t, i} ^ {2 - \beta_ {t}} (a) \hat {\ell} _ {t, i} ^ {2} (a) + \gamma \sum_ {a = 1} ^ {d} \hat {\ell} _ {t, i} (a) \\ \leq \varepsilon m T + \sum_ {t = 1} ^ {T} \frac {B _ {\beta_ {t}} (\hat {\mathbf {x}} _ {t} ^ {(\varepsilon)} | | \mathbf {x} _ {t , 1})}{\eta_ {t}} + \sum_ {i = 1} ^ {m} \langle \hat {\ell} _ {t, i}, \hat {\mathbf {x}} _ {t} ^ {(\varepsilon)} \rangle - \langle \ell_ {t, i}, \mathring {\mathbf {x}} _ {t} ^ {(\varepsilon)} \rangle \\ + \sum_ {t = 1} ^ {T} \frac {\eta_ {t}}{\beta_ {t}} \sum_ {i = 1} ^ {m} \sum_ {a = 1} ^ {d} \mathbf {x} _ {t, i} ^ {1 - \beta_ {t}} (a) \hat {\ell} _ {t, i} (a) + \gamma \sum_ {a = 1} ^ {d} \hat {\ell} _ {t, i} (a) \tag {43} \\ \end{array}
$$
where the equality follows similarly to Luo [37] since  $\langle \hat{\ell}_{t,i},\mathbf{x}_{t,i}\rangle = \ell_{t,i}(a_{t,i}) - \gamma \sum_{a = 1}^{d}\hat{\ell}_{t,i}(a)$ , the first inequality follows by Lemma B.3 and the second by Hölder's inequality and the definitions of
$\hat{\ell}_{t,i}$  and  $\hat{\mathbf{x}}_{t,i}^{(\varepsilon)}$ . We next apply the optimality of  $\hat{a}_t$  for  $\sum_{i = 1}^{m}\hat{\ell}_{t,i}$  to get
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {m} \ell_ {t, i} (a _ {t, i}) - \ell_ {t, i} (\mathring {a} _ {t}) \\ \leq \varepsilon m T + \sum_ {t = 1} ^ {T} \frac {B _ {\beta_ {t}} (\hat {\mathbf {x}} _ {t} ^ {(\varepsilon)} | | \mathbf {x} _ {t , 1})}{\eta_ {t}} + (1 - \varepsilon) \sum_ {i = 1} ^ {m} \hat {\ell} _ {t, i} (\mathring {a} _ {t}) - \ell_ {t, i} (\mathring {a} _ {t}) + \frac {\varepsilon}{d} \sum_ {a = 1} ^ {d} \hat {\ell} _ {t, i} (a) - \ell_ {t, i} (a) \\ + \sum_ {t = 1} ^ {T} \frac {\eta_ {t}}{\beta_ {t}} \sum_ {i = 1} ^ {m} \sum_ {a = 1} ^ {d} \mathbf {x} _ {t, i} ^ {1 - \beta_ {t}} (a) \hat {\ell} _ {t, i} (a) + \gamma \sum_ {a = 1} ^ {d} \hat {\ell} _ {t, i} (a) \\ \leq \varepsilon m T + \frac {1 + \frac {\varepsilon}{d} + \frac {\overline {{\eta}}}{\underline {{\beta}}} + \gamma}{2 \gamma} \log \frac {5}{\delta} + \sum_ {t = 1} ^ {T} \frac {B _ {\beta_ {t}} (\hat {\mathbf {x}} _ {t} ^ {(\varepsilon)} | | \mathbf {x} _ {t , 1})}{\eta_ {t}} \\ + \sum_ {t = 1} ^ {T} \frac {\eta_ {t}}{\beta_ {t}} \sum_ {i = 1} ^ {m} \sum_ {a = 1} ^ {d} \mathbf {x} _ {t, i} ^ {1 - \beta_ {t}} (a) \ell_ {t, i} (a) + \gamma \sum_ {a = 1} ^ {d} \ell_ {t, i} (a) \\ \leq \varepsilon m T + \frac {2 + \sqrt {\frac {d \log d}{e m}}}{\gamma} \log \frac {5}{\delta} + \gamma d m T + \sum_ {t = 1} ^ {T} \frac {B _ {\beta_ {t}} \left(\hat {\mathbf {x}} _ {t} ^ {(\varepsilon)} \right| | \mathbf {x} _ {t , 1})}{\eta_ {t}} + \frac {\eta_ {t} d ^ {\beta_ {t}} m}{\beta_ {t}} \tag {44} \\ \end{array}
$$
where the second inequality follows by Neu [43, Lemma 1] applied to each of the last four terms and the fifth by the definition of  $\ell_{t,i}$  and using  $\max_{\beta \in [\frac{1}{\log d},1]}\overline{\eta} (\beta)\leq \sqrt{\frac{d}{em\log d}}$ . Substituting into Theorem A.1 and simplifying yields the result except with  $\hat{V}_{\beta}^{2} = \frac{1}{T}\sum_{t = 1}^{T}\psi_{\beta}(\hat{\mathbf{x}}_{t}^{(\varepsilon)}) - \psi_{\beta}(\hat{\bar{\mathbf{x}}}^{(\varepsilon)})$  in place of  $\hat{H}_{\beta}$ , but the former is bounded by the latter by Lemma B.2.
Corollary B.1. Let  $\underline{\beta} = \overline{\beta} = 1$ . Then w.h.p. we can ensure task-averaged regret at most
$$
2 \sqrt {\hat {H} _ {1} d m} + \tilde {\mathcal {O}} \left(\frac {d \sqrt {m} + d ^ {\frac {2}{3}} m ^ {\frac {2}{3}}}{\sqrt [ 3 ]{T}}\right) \tag {45}
$$
so long as  $mT \geq d^2$  or alternatively ensure
$$
\min  \left\{2 \sqrt {\hat {H} _ {1} d m} + \tilde {\mathcal {O}} \left(\frac {d ^ {\frac {3}{4}} m ^ {\frac {3}{4}} + d \sqrt {m}}{\sqrt [ 4 ]{T}}\right), 2 \sqrt {d m \log d} + \tilde {\mathcal {O}} \left(\frac {d ^ {\frac {3}{2}} \sqrt {m}}{\sqrt {T}}\right) \right\} \tag {46}
$$
so long as  $mT \geq d$ .
Proof. Applying Theorem B.1, simplifying, and dividing by  $T$  yields task-averaged regret at most
$$
\begin{array}{l} (\varepsilon + \gamma d) m + \frac {2 + \sqrt {\frac {d \log d}{e m}}}{\gamma T} \log \frac {5}{\delta} + \left(\frac {1 + \log (T + 1)}{2 \rho^ {2} T} + \min  \left\{\frac {\rho^ {2}}{\eta \sqrt {m}}, \rho \right\}\right) d \sqrt {m} \tag {47} \\ + \min  _ {\eta > 0} \frac {8 d (1 + \log T)}{\varepsilon \eta T} + \left(\frac {\hat {H} _ {1}}{\eta} + \eta d m\right) \\ \end{array}
$$
Set  $\gamma = \frac{1}{\sqrt{dmT}}$ . Then set  $\varepsilon = \sqrt[3]{\frac{d^2}{mT}}$  and  $\rho = \frac{1}{\sqrt[3]{T}}$ , and use  $\eta = \sqrt{\frac{\hat{H}_1}{dm}} + \frac{1}{\sqrt[3]{dmT}}$  to get the first result. Otherwise, set  $\varepsilon = \sqrt{\frac{d}{mT}}$  and  $\rho = \frac{1}{\sqrt[4]{T}}$ , and use the better of  $\eta = \sqrt{\frac{\hat{H}_1}{dm}} + \frac{1}{\sqrt[4]{dmT}}$  and  $\eta = \sqrt{\frac{\log d}{dm}}$  to get the second.
Corollary B.2. Let  $\underline{\beta} = \frac{1}{2}$  and  $\overline{\beta} = 1$  and assume  $mT \geq d^{\frac{5}{2}}$ . Then w.h.p. we can ensure task-averaged regret at most
$$
\min  _ {\beta \in [ \frac {1}{2}, 1 ]} 2 \sqrt {\hat {H} _ {\beta} d ^ {\beta} m / \beta} + \tilde {\mathcal {O}} \left(\frac {d ^ {\frac {5}{7}} m ^ {\frac {5}{7}}}{T ^ {\frac {2}{7}}} + \frac {d \sqrt {m}}{\sqrt [ 4 ]{T}}\right) \tag {48}
$$
using  $k = \left\lceil \sqrt[4]{d}\sqrt{T}\right\rceil$
$$
\begin{array}{l} (\varepsilon + \gamma d) m + \frac {2 + \sqrt {\frac {d \log d}{e m}}}{\gamma T} \log \frac {5}{\delta} + \frac {8 d \sqrt {m}}{\rho} \left(\sqrt {\frac {\log \frac {5 k}{\delta}}{T}} + \frac {1 + \log (T + 1)}{1 6 \rho T}\right) \\ + \min  _ {\beta \in [ \underline {{\beta}}, \bar {\beta} ], \eta > 0} \frac {8 d ^ {\frac {3}{2}} (1 + \log T)}{\varepsilon^ {\frac {3}{2}} \eta T} + \left(\frac {\hat {H} _ {\beta}}{\eta} + \frac {\eta d ^ {\beta} m}{\beta} + \frac {d}{4 k} \left(\frac {\log \frac {d}{\varepsilon}}{\eta} + \eta m \log^ {2} d\right) + \rho d \sqrt {m}\right) \tag {49} \\ \end{array}
$$
Set  $\gamma = \frac{1}{\sqrt{dmT}}, \varepsilon = \frac{d^{\frac{5}{7}}}{(mT)^{\frac{2}{7}}}, \rho = \frac{1}{\sqrt[4]{T}}$ , and use  $\eta = \sqrt{\frac{\beta\tilde{H}_{\beta}}{md^{\beta}}} + \frac{1}{(dmT)^{\frac{2}{7}}}$  to get the result.
Corollary B.3. Let  $\underline{\beta} = \frac{1}{\log d}$  and  $\overline{\beta} = 1$  and assume  $mT \geq d^3$ . Then w.h.p. we can ensure task-averaged regret at most
$$
\min  _ {\beta \in (0, 1 ]} 2 \sqrt {\hat {H} _ {\beta} d ^ {\beta} m / \beta} + \tilde {\mathcal {O}} \left(\frac {d ^ {\frac {3}{4}} m ^ {\frac {3}{4}} + d \sqrt {m}}{\sqrt [ 4 ]{T}}\right) \tag {50}
$$
Proof. Applying Theorem B.1, dividing by  $T$ , and simplifying yields
$$
\begin{array}{l} (\varepsilon + \gamma d) m + \frac {2 + \sqrt {\frac {d \log d}{e m}}}{\gamma T} \log \frac {5}{\delta} + \frac {8 d \sqrt {m}}{\rho} \left(\sqrt {\frac {\log \frac {5 k}{\delta}}{T}} + \frac {1 + \log (T + 1)}{1 6 \rho T}\right) \tag {51} \\ + \min  _ {\beta \in [ \underline {{\beta}}, \overline {{\beta}} ], \eta > 0} \frac {8 d ^ {2} (1 + \log T)}{\varepsilon^ {2} \eta T} + \left(\frac {\hat {H} _ {\beta}}{\eta} + \frac {\eta d ^ {\beta} m}{\beta} + \frac {d}{2 k} \left(\frac {\log \frac {d}{\varepsilon}}{\eta} + \eta \log^ {2} d\right) + \rho d \sqrt {m}\right) \\ \end{array}
$$
Note that  $\hat{H}_{\beta}$  and  $\frac{d^{\beta}}{\beta}$  are both decreasing on  $\beta < \frac{1}{\log d}$ , so  $\beta$  in the chosen interval is optimal over all  $\beta \in (0,1]$ . Set  $\gamma = \frac{1}{\sqrt{dmT}}$ ,  $\varepsilon = \frac{d^{\frac{3}{4}}}{\sqrt[4]{mT}}$ ,  $\rho = \frac{1}{\sqrt[4]{T}}$ , and use  $\eta = \sqrt{\frac{\beta\hat{H}_{\beta}}{md^{\beta}}} + \frac{1}{\sqrt[4]{dmT}}$  to get the result.
C Guaranteed exploration
C.1 Best-arm identification
Lemma C.1. Suppose for  $\varepsilon > 0$  we run OMD on task  $t \in [T]$  with initialization  $\mathbf{x}_{t,1} \in \triangle^{(\varepsilon)}$ , regularizer  $\psi_{\beta_t} + I_{\triangle (\varepsilon)}$  for some  $\beta_t \in (0,1]$ , and unbiased loss estimators  $(\gamma = 0)$ . If Assumption 3.1 holds and  $m > \frac{28d\log d}{3\varepsilon\Delta^2}$  then  $\hat{\mathbf{x}}_t = \mathring{\mathbf{x}}_t$  w.p.  $\geq 1 - d\kappa$ , where  $\kappa = \exp\left(-\frac{3\varepsilon\Delta^2m}{28d}\right)$ .
Proof. We extend the proof by Abbasi-Yadkori et al. [1, Appendices B and F] to arbitrary lower bounds  $\varepsilon / d$  on the probability. First, since  $0 \leq \hat{\ell}_{t,i}(a) \leq \frac{d}{\varepsilon} \ell_{t,i}(a)$  we have that
$$
- \frac {d}{\varepsilon} \leq - 1 \leq - \ell_ {t, i} (a) \leq \hat {\ell} _ {t, i} (a) - \ell_ {t, i} (a) \leq \left(\frac {d}{\varepsilon} - 1\right) \ell_ {t, i} (a) \leq \frac {d}{\varepsilon} \tag {52}
$$