SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \langle q _ {t} - q _ {t + 1}, \widehat {y} _ {t} \rangle - D _ {\Phi_ {t}} (q _ {t + 1}, q _ {t}) = \langle q _ {t} - q _ {t + 1}, \widehat {y} _ {t} \rangle - \frac {1}{\eta_ {t}} D _ {\psi^ {n S}} (q _ {t + 1}, q _ {t}) \\ = \sum_ {i = 1} ^ {k} \left(\widehat {y} _ {t i} \left(q _ {t i} - q _ {t + 1, i}\right) - \frac {1}{\eta_ {t}} D _ {\phi^ {\cap}} \left(q _ {t + 1, i}, q _ {t i}\right)\right) \\ \leq \sum_ {i = 1} ^ {k} \frac {1}{\eta_ {t}} q _ {t i} \xi (\eta_ {t} \widehat {y} _ {t i}) \leq \frac {1}{2} \eta_ {t} \sum_ {i = 1} ^ {k} q _ {t i} \widehat {y} _ {t i} ^ {2} \leq \eta_ {t} \omega_ {t}, \\ \end{array}
$$
where the first inequality follows from (21), the second inequality follows by (19) with  $\widehat{y}_t \geq 0$ , and the last inequality holds since  $\sum_{i=1}^{k} q_{ti} \widehat{y}_{ti}^2 \leq \sum_{i=1}^{k} 2p_{ti} \widehat{y}_{ti}^2 = 2\omega_t$ .
We will confirm that the assumptions for Part (II) of Theorem 1 are indeed satisfied. Using the definition of  $\beta_{t}$  in (6), We have
$$
\begin{array}{l} \beta_ {t} = \beta_ {1} + \frac {1}{\sqrt {h _ {1}}} \sum_ {u = 1} ^ {t - 1} \frac {c _ {1} \omega_ {u}}{\sqrt {\frac {k}{\gamma} + \sum_ {s = 1} ^ {u - 1} \omega_ {s}}} = \beta_ {1} + \frac {2 c _ {1}}{\sqrt {h _ {1}}} \sum_ {u = 1} ^ {t - 1} \frac {\omega_ {u}}{\sqrt {\frac {k}{\gamma} + \sum_ {s = 1} ^ {u - 1} \omega_ {s}} + \sqrt {\frac {k}{\gamma} + \sum_ {s = 1} ^ {u - 1} \omega_ {s}}} \\ \geq \beta_ {1} + \frac {2 c _ {1}}{\sqrt {h _ {1}}} \sum_ {u = 1} ^ {t - 1} \frac {\omega_ {u}}{\sqrt {\frac {k}{\gamma} + \sum_ {s = 1} ^ {u} \omega_ {s}} + \sqrt {\frac {k}{\gamma} + \sum_ {s = 1} ^ {u - 1} \omega_ {s}}} \\ = \beta_ {1} + \frac {2 c _ {1}}{\sqrt {h _ {1}}} \sum_ {u = 1} ^ {t - 1} \left(\sqrt {\frac {k}{\gamma} + \sum_ {s = 1} ^ {u} \omega_ {s}} - \sqrt {\frac {k}{\gamma} + \sum_ {s = 1} ^ {u - 1} \omega_ {s}}\right) \\ = \beta_ {1} + \frac {2 c _ {1}}{\sqrt {h _ {1}}} \left(\sqrt {\frac {k}{\gamma} + \sum_ {s = 1} ^ {t - 1} \omega_ {s}} - \sqrt {\frac {k}{\gamma}}\right) \geq \frac {2 c _ {1}}{\sqrt {h _ {1}}} \sqrt {\sum_ {s = 1} ^ {t} \omega_ {s}}, \\ \end{array}
$$
where the last inequality follows since  $\beta_{1} = \frac{2c_{1}}{\sqrt{h_{1}}}\sqrt{\frac{k}{\gamma}}$  and  $\frac{k}{\gamma} \geq \omega_{t}$ . Hence, stability condition (S2) in Theorem 1 is satisfied with  $a = 2$ , and one can see that the other assumptions are trivially satisfied. Hence, by Part (II) of Theorem 1,
$$
\sum_ {t = 1} ^ {T} \langle \widehat {y} _ {t}, q _ {t} - e _ {i ^ {*}} \rangle \leq \sum_ {t = 1} ^ {T} \left(\frac {1}{\eta_ {t + 1}} - \frac {1}{\eta_ {t}}\right) H (q _ {t + 1}) + \sum_ {t = 1} ^ {T} \eta_ {t} \omega_ {t} + \frac {H (q _ {1})}{\eta_ {1}} \leq 2 \left(c _ {1} + \frac {1}{2 c _ {1}}\right) \sqrt {h _ {1} \sum_ {t = 1} ^ {T} \omega_ {t}} + \frac {\log k}{\eta_ {1}},
$$
where in the last inequality we used  $h_1 \leq \log k$ . Now,
$$
\mathbb {E} \left[ \sqrt {\sum_ {t = 1} ^ {T} \omega_ {t}} \right] \leq \sqrt {\sum_ {t = 1} ^ {T} \mathbb {E} [ \omega_ {t} ]} = \sqrt {\sum_ {t = 1} ^ {T} \mathbb {E} \left[ \frac {\ell_ {t A _ {t}} ^ {2}}{p _ {t A _ {t}}} \right]} = \sqrt {\sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {k} \ell_ {t i} ^ {2}} = \sqrt {\sum_ {t = 1} ^ {T} \| \ell_ {t} \| _ {2} ^ {2}} = \sqrt {L _ {2}}.
$$
Summing up the above arguments and setting  $c_{1} = 1 / \sqrt{2}$ , we have
$$
\operatorname {R e g} _ {T} \leq 2 \sqrt {2} \sqrt {L _ {2} \log k} + \frac {\log k}{\eta_ {1}} + \gamma T = 2 \sqrt {2} \sqrt {L _ {2} \log k} + (\sqrt {2} + 1) (k T \log k) ^ {1 / 3},
$$
which completes the proof of Corollary 2.
G Proof of Corollary 3
We first prove Lemma 3.
Proof of Lemma 3. Recall that  $\Phi_t(p) = \frac{1}{\eta_t} \psi^{\mathsf{nS}}(p) + 2\delta \psi^{\mathsf{LB}}(p)$ . Since  $D_{\Phi_t} = \frac{1}{\eta_t} D_{\psi^{\mathsf{nS}}} + 2\delta D_{\psi^{\mathsf{LB}}}$  and  $D_{\psi^{\mathsf{nS}}}(x,y) = \sum_{i=1}^{k} D_{\phi^{\mathsf{nS}}}(x_i, y_i)$  and  $D_{\psi^{\mathsf{LB}}}(x,y) = \sum_{i=1}^{k} D_{\phi^{\mathsf{LB}}}(x_i, y_i)$ , we can bound the stability term as
$$
\begin{array}{l} \langle q _ {t} - q _ {t + 1}, \widehat {y} _ {t} \rangle - D _ {\Phi_ {t}} (q _ {t + 1}, q _ {t}) \leq \langle q _ {t} - q _ {t + 1}, \widehat {y} _ {t} \rangle - \max  \left\{\frac {1}{\eta_ {t}} D _ {\psi^ {\mathfrak {n S}}} (q _ {t + 1}, q _ {t}), 2 \delta D _ {\psi^ {\mathbb {L B}}} (q _ {t + 1}, q _ {t}) \right\} \\ \leq \sum_ {i = 1} ^ {k} \left(\widehat {y} _ {t i} \left(q _ {t i} - q _ {t + 1, i}\right) - \max  \left\{\frac {1}{\eta_ {t}} D _ {\phi^ {\mathrm {n S}}} \left(q _ {t + 1, i}, q _ {t i}\right), 2 \delta D _ {\phi^ {\mathrm {L B}}} \left(q _ {t + 1, i}, q _ {t i}\right) \right\}\right) \\ \leq \sum_ {i = 1} ^ {k} \min  \left\{\frac {1}{\eta_ {t}} q _ {t i} \xi \left(\eta_ {t} \widehat {y} _ {t i}\right), 2 \delta \zeta \left(\frac {1}{2 \delta} q _ {t i} \widehat {y} _ {t i}\right) \right\}, \tag {23} \\ \end{array}
$$
where in the last inequality we used (21) and (22) with
$$
\frac {\widehat {y} _ {t i}}{2 \delta} \geq - \frac {1}{2 \delta p _ {t i}} \geq - \frac {1}{2 \delta \left(q _ {t i} / \delta\right)} \geq - \frac {1}{q _ {t i}},
$$
where the first inequality follows by the definition of  $\widehat{y}_t$  and the second inequality follows by  $p_{ti} \geq q_{ti} / \delta$ .
Next, we will prove that for any  $i \in [k]$ ,
$$
\left. \min  \left\{\frac {1}{\eta_ {t}} q _ {t i} \xi \left(\eta_ {t} \widehat {y} _ {t i}\right), 2 \delta \zeta \left(\frac {1}{2 \delta} q _ {t i} \widehat {y} _ {t i}\right) \right\} \leq \delta \eta_ {t} \frac {\ell_ {t i} ^ {2}}{p _ {t i}} \min  \left\{1, \frac {p _ {t i}}{2 \eta_ {t}} \right\} \mathbb {1} [ A _ {t} = i ]. \right. \tag {24}
$$
Fix  $i\in [k]$  .By  $q_{ti}\leq \delta p_{ti}$
$$
\frac {1}{2 \delta} q _ {t i} \widehat {y} _ {t i} = \frac {1}{2} p _ {t i} \widehat {y} _ {t i} \leq \frac {1}{2}.
$$
Using this and  $\zeta(x) \leq x^2$  for  $x \in [-\frac{1}{2}, \frac{1}{2}]$  in (20), we have for any  $p_{ti} \in [0, 1]$  that
$$
2 \delta \zeta \left(\frac {1}{2 \delta} q _ {t i} \widehat {y} _ {t i}\right) \leq 2 \delta \left(\frac {1}{2 \delta} q _ {t i} \widehat {y} _ {t i}\right) ^ {2} \leq \frac {\delta}{2} \ell_ {t i} ^ {2} \mathbb {1} [ A _ {t} = i ], \tag {25}
$$
where in the last inequality we used  $q_{ti} \leq \delta p_{ti}$ . In particular, when  $p_{ti} \leq \eta_t$ , i.e., the probability of selecting arm  $i$  is small to some extent, the last inequality can be further bounded as
$$
2 \delta \zeta \left(\frac {1}{2 \delta} q _ {t i} \hat {y} _ {t i}\right) \leq \frac {\eta_ {t}}{p _ {t i}} \frac {\delta}{2} \ell_ {t i} ^ {2} \mathbb {1} [ A _ {t} = i ] \leq \delta \eta_ {t} \frac {\ell_ {t i} ^ {2}}{p _ {t i}} \mathbb {1} [ A _ {t} = i ]. \tag {26}
$$
On the other hand when  $p_{ti} > \eta_t$ , we have  $\eta_t\widehat{y}_{ti}\geq -1$ . Hence, by the inequality  $\xi (x)\leq x^{2}$  for  $x\geq -1$  in (19),
$$
\frac {1}{\eta_ {t}} q _ {t i} \xi \left(\eta_ {t} \widehat {y} _ {t i}\right) \leq \frac {1}{\eta_ {t}} \delta p _ {t i} \left(\eta_ {t} \widehat {y} _ {t i}\right) ^ {2} = \delta \eta_ {t} \frac {\ell_ {t i} ^ {2}}{p _ {t i}} \mathbb {1} \left[ A _ {t} = i \right]. \tag {27}
$$
Hence, combining (25), (26), and (27) completes the proof of (24). Finally, by combining (23) and (24) we completes the proof of Lemma 3.
Remark. When  $\ell_t$  can be negative, the Shannon entropy regularizer alone cannot bound the stability term if the arm selection probability is small, i.e.,  $p_{ti} \leq \eta_t$ . Introducing a time-invariant log-barrier regularizer enables us to bound the stability term even when the arm selection probability is small. This idea was proposed by Bubeck et al. [10], who analyzed the variation of arm selection probability for the change of cumulative losses. Unlike their analysis, our proof directly analyses the stability term, enabling us to obtain the tighter regret bound. More importantly, we will utilize the property  $\nu_t \leq O(1 / \eta_t)$  many times, which directly follows from Lemma 3, in the subsequent sections to prove the BOBW guarantee with the sparsity-dependent bound.
Now, we are ready to prove Corollary 3.
Proof of Corollary 3. Fix  $i^* \in [k]$ . Define  $p^* \in \mathcal{P}_k$  by
$$
p ^ {*} = \left(1 - \frac {k}{T}\right) e _ {i ^ {*}} + \frac {1}{T} \mathbf {1}.
$$
Then, using the definition of the algorithm,
$$
\begin{array}{l} \operatorname {R e g} _ {T} (i ^ {*}) = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \ell_ {t A _ {t}} - \sum_ {t = 1} ^ {T} \ell_ {t i ^ {*}} \right] = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \langle \ell_ {t}, p _ {t} - e _ {i ^ {*}} \rangle \right] \\ = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle \ell_ {t}, p _ {t} - p ^ {*} \right\rangle \right] + \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle \ell_ {t}, p ^ {*} - e _ {i ^ {*}} \right\rangle \right] \\ \leq \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \langle \widehat {y} _ {t}, p _ {t} - p ^ {*} \rangle \right] + k, \\ \end{array}
$$
where the inequality follows from the definition of  $p^*$  and the Cauchy-Schwarz inequality. By the standard analysis of the FTRL, described in Section 3,
$$
\sum_ {t = 1} ^ {T} \langle \widehat {y} _ {t}, p _ {t} - p ^ {*} \rangle \leq \sum_ {t = 1} ^ {T} \Big (\Phi_ {t} (p _ {t + 1}) - \Phi_ {t + 1} (p _ {t + 1}) \Big) + \Phi_ {t + 1} (p ^ {*}) - \Phi_ {1} (p _ {1}) + \sum_ {t = 1} ^ {T} \Big (\langle p _ {t} - p _ {t + 1}, \widehat {y} _ {t} \rangle - D _ {\Phi_ {t}} (p _ {t + 1}, p _ {t}) \Big).
$$
For the penalty term, since  $\Phi_t(p) = \frac{1}{\eta_t}\psi^{\mathsf{nS}}(p) + 2\psi^{\mathsf{LB}}(p)$ ,
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left(\Phi_ {t} \left(p _ {t + 1}\right) - \Phi_ {t + 1} \left(p _ {t + 1}\right)\right) + \Phi_ {t + 1} \left(p ^ {*}\right) - \Phi_ {1} \left(p _ {1}\right) \\ \leq \sum_ {t = 1} ^ {T} \left(\frac {1}{\eta_ {t + 1}} - \frac {1}{\eta_ {t}}\right) H (p _ {t + 1}) + \frac {H (p _ {1})}{\eta_ {1}} + 2 \sum_ {i = 1} ^ {k} \log \left(\frac {1}{p _ {i} ^ {*}}\right) \\ \leq \sum_ {t = 1} ^ {T} \left(\frac {1}{\eta_ {t + 1}} - \frac {1}{\eta_ {t}}\right) H \left(p _ {t + 1}\right) + \frac {\log k}{\eta_ {1}} + 2 k \log T, \\ \end{array}
$$
where in the last inequality we used the fact that  $p_i^* \geq 1 / T$  for all  $i \in [k]$ .
For the stability term, by Lemma 3 with  $\delta = 1$  (since  $p_t = q_t$ ),
$$
\sum_ {t = 1} ^ {T} \left(\langle p _ {t} - p _ {t + 1}, \widehat {y} _ {t} \rangle - D _ {\Phi_ {t}} (p _ {t + 1}, p _ {t})\right) \leq \sum_ {t = 1} ^ {T} \eta_ {t} \nu_ {t}.
$$
We will confirm that the assumptions for Part (II) of Theorem 1 are indeed satisfied. By the definition of the learning rate in (7),
$$
\begin{array}{l} \beta_ {t} = \beta_ {1} + \sum_ {u = 1} ^ {t - 1} \frac {c _ {1} \nu_ {u}}{\sqrt {h _ {1}} \sqrt {\sum_ {s = 1} ^ {u} \nu_ {s}}} \geq \beta_ {1} + \frac {c _ {1}}{\sqrt {h _ {1}}} \sum_ {u = 1} ^ {t - 1} \frac {\nu_ {u}}{\sqrt {\sum_ {s = 1} ^ {u} \nu_ {s}} + \sqrt {\sum_ {s = 1} ^ {u - 1} \nu_ {s}}} \\ \geq \beta_ {1} + \frac {c _ {1}}{\sqrt {h _ {1}}} \sum_ {u = 1} ^ {t - 1} \left(\sqrt {\sum_ {s = 1} ^ {u} \nu_ {s}} - \sqrt {\sum_ {s = 1} ^ {u - 1} \nu_ {s}}\right) = \beta_ {1} + \frac {c _ {1}}{\sqrt {h _ {1}}} \sqrt {\sum_ {s = 1} ^ {t - 1} \nu_ {s}}. \\ \end{array}
$$
Using this inequality,  $\beta_{t}$  is bounded from below as
$$
2 \beta_ {t} = \beta_ {t} + \beta_ {t} \geq 2 \nu_ {t} + \beta_ {1} + \frac {c _ {1}}{\sqrt {h _ {1}}} \sqrt {\sum_ {s = 1} ^ {t - 1} \nu_ {s}} \geq 2 \sqrt {2 \beta_ {1} \nu_ {t}} + \frac {c _ {1}}{\sqrt {h _ {1}}} \sqrt {\sum_ {s = 1} ^ {t - 1} \nu_ {s}} \geq \frac {c _ {1}}{\sqrt {h _ {1}}} \sqrt {\sum_ {s = 1} ^ {t} \nu_ {s}},
$$
where the first inequality follows by  $\nu_t \leq \beta_t / 2$  and the above inequality, the second inequality follows by the AM-GM inequality, and the last inequality follows from  $2\sqrt{2\beta_1} \geq \frac{c_1}{\sqrt{h_1}}$  and  $\sqrt{x} + \sqrt{y} \geq \sqrt{x + y}$  for  $x, y \geq 0$ . Dividing the both sides by 2, we can see stability condition (S2) in Theorem 1 is satisfied with  $a = 1 / 2$ . One can also see that the other assumptions are trivially satisfied. Hence, by Part (II) of Theorem 1,
$$
\sum_ {t = 1} ^ {T} \left(\frac {1}{\eta_ {t + 1}} - \frac {1}{\eta_ {t}}\right) H (p _ {t + 1}) + \sum_ {t = 1} ^ {T} \eta_ {t} \nu_ {t} \leq 2 \left(c _ {1} + \frac {2}{c _ {1}}\right) \sqrt {h _ {1} \sum_ {t = 1} ^ {T} \nu_ {t}}.
$$
Using the last inequality with  $\mathbb{E}\left[\sqrt{\sum_{t=1}^{T} \nu_t}\right] \leq \mathbb{E}\left[\sqrt{\sum_{t=1}^{T} \omega_t}\right] \leq \sqrt{L_2}$ , and setting  $c_1 = \sqrt{2}$ , we have
$$
\begin{array}{l} \operatorname {R e g} _ {T} \left(i ^ {*}\right) \leq \mathbb {E} \left[ 2 \left(c _ {1} + \frac {2}{c _ {1}}\right) \sqrt {h _ {1} \sum_ {t = 1} ^ {T} \nu_ {t}} + \frac {\log k}{\eta_ {1}} + 2 k \log T + k \right] \\ \leq 4 \sqrt {2} \sqrt {L _ {2} \log k} + 2 k \log T + k + \frac {1}{4}, \\ \end{array}
$$
which completes the proof of Corollary 3.
H Proof of results in Section 5.2
Appendix H.1 provides preliminary results, which will be used to quantify the difference between  $q_{t}$  and  $q_{t + 1}$  in Appendix H.2 and will be used to prove the continuity of  $F_{t}$  in Appendix H.3. Appendix H.2 proves Theorem 4 and Appendix H.3 discusses the bisection method to compute  $\beta_{t + 1}$ .
H.1 Some stability results
Before proving Theorem 4, we prove several important lemmas. Consider the following three optimization problems:
$$
p\in \operatorname *{arg  min}_{p^{\prime}\in \mathcal{P}_{k}}\langle L - \xi e_{1},p^{\prime}\rangle +\beta \psi (p^{\prime})  ,
$$
$$
q \in \underset {q ^ {\prime} \in \mathcal {P} _ {k}} {\arg \min } \left\langle L, q ^ {\prime} \right\rangle + \beta \psi \left(q ^ {\prime}\right), \tag {28}
$$
$$
r\in \operatorname *{arg  min}_{r^{\prime}\in \mathcal{P}_{k}}\langle L,r^{\prime}\rangle +\beta^{\prime}\psi^{\prime}(r^{\prime})
$$
with  $L\in \mathbb{R}_+^k$ $0\leq \xi \leq \min_{i\in [k]}L_i$  , and  $\beta ,\beta^{\prime} > 0$  satisfying  $\beta^{\prime}\geq \beta$
$$
\psi (q) = \sum_ {i = 1} ^ {k} \left(q _ {i} \log q _ {i} - q _ {i}\right) - \frac {c}{\beta} \sum_ {i = 1} ^ {k} \log q _ {i} \quad \text {a n d} \quad \psi^ {\prime} (q) = \sum_ {i = 1} ^ {k} \left(q _ {i} \log q _ {i} - q _ {i}\right) - \frac {c}{\beta^ {\prime}} \sum_ {i = 1} ^ {k} \log q _ {i}
$$
for  $c > 0$ . Note that the outputs of FTRL with  $\psi(q)$  and with  $-H(q) - (c/\beta)\sum_{i=1}^{k}\log q_i$  are identical since adding a constant to  $\psi$  does not change the output of the above optimization problems.
In the following lemma, we investigate the relation between  $q$  and  $r$  in (28).
Lemma 4. Consider  $q$  and  $r$  in (28). Then,
$$
r _ {i} \leq q _ {i} ^ {\beta / \beta^ {\prime}}. \tag {29}
$$
Proof. From the KKT conditions there exist  $\mu, \mu' \in \mathbb{R}$  such that
$$
L + \beta \nabla \psi (q) + \mu \mathbf {1} = 0 \quad \text {a n d} \quad L + \beta^ {\prime} \psi^ {\prime} (r) + \mu^ {\prime} \mathbf {1} = 0,
$$
which implies, by  $(\nabla \psi (q))_i = \log q_i - \frac{c}{\beta q_i}$ , that
$$
L _ {i} + \beta \log q _ {i} - \frac {c}{q _ {i}} + \mu = 0 \quad \text {a n d} \quad \eta^ {\prime} L _ {i} + \beta \log r _ {i} - \frac {c}{r _ {i}} + \mu^ {\prime} = 0 \tag {30}
$$
for all  $i\in [k]$  . This is equivalent to
$$
q _ {i} = \exp \left(- \frac {1}{\beta} \left(L _ {i} - \frac {c}{q _ {i}} + \mu\right)\right) \quad \text {a n d} \quad r _ {i} = \exp \left(- \frac {1}{\beta^ {\prime}} \left(L _ {i} - \frac {c}{r _ {i}} + \mu^ {\prime}\right)\right).
$$
Removing  $L_{i}$  from these equalities yields that
$$
r _ {i} = q _ {i} ^ {\beta / \beta^ {\prime}} \exp \left(\frac {c}{\beta^ {\prime}} \left(\frac {1}{r _ {i}} - \frac {1}{q _ {i}}\right)\right) \exp \left(\frac {1}{\beta^ {\prime}} \left(\mu - \mu^ {\prime}\right)\right). \tag {31}
$$
We will prove  $\frac{\mathrm{d}\mu}{\mathrm{d}\beta} > 0$ . Taking derivative with respect to  $\beta$  of (30), we have
$$
\log q _ {i} + \left(\frac {1}{q _ {i}} + \frac {c}{q _ {i} ^ {2}}\right) \frac {\mathrm {d} q _ {i}}{\mathrm {d} \beta} + \frac {\mathrm {d} \mu}{\mathrm {d} \beta} = 0.
$$
Multiplying  $\left(\frac{1}{q_i} +\frac{c}{q_i^2}\right)^{-1}$  and summing over  $i\in [k]$  in the last equality, we have
$$
- \left(\frac {1}{q _ {i}} + \frac {c}{q _ {i} ^ {2}}\right) ^ {- 1} \log (1 / q _ {i}) + \sum_ {i = 1} ^ {k} \frac {\mathrm {d} q _ {i}}{\mathrm {d} \beta} + \left(\frac {1}{q _ {i}} + \frac {c}{q _ {i} ^ {2}}\right) ^ {- 1} \frac {\mathrm {d} \mu}{\mathrm {d} \beta} = 0,
$$
which with the fact  $\sum_{i=1}^{k} \frac{\mathrm{d}q_i}{\mathrm{d}\beta} = 0$  implies  $\frac{\mathrm{d}\mu}{\mathrm{d}\beta} > 0$ . Hence, since  $\beta \leq \beta'$  we have  $\mu \leq \mu'$ .
When  $r_i \leq q_i$ , it is obvious that we get  $r_i \leq q_i^{\beta/\beta'}$ .
When  $r_i > q_i$ , using (31) with the inequalities  $\beta \leq \beta'$  and  $\mu \leq \mu'$ ,
$$
r _ {i} = q _ {i} ^ {\beta / \beta^ {\prime}} \exp \left(\frac {c}{\beta^ {\prime}} \left(\frac {1}{r _ {i}} - \frac {1}{q _ {i}}\right)\right) \exp \left(\frac {1}{\beta^ {\prime}} (\mu - \mu^ {\prime})\right) \leq q _ {i} ^ {\beta / \beta^ {\prime}},
$$
Lemma 5. Consider  $p, q,$  and  $r$  in (28). Then, under  $\eta \coloneqq 1 / \beta \leq \frac{1}{15k}$ , we have
$$
r _ {i} \leq 3 p _ {i} ^ {\beta / \beta^ {\prime}}. \tag {32}
$$
Proof. By Lemma 8 of Bubeck et al. [10] we have  $q_{i} \leq 3p_{i}$  for all  $i \in [k]$ . Using this with Lemma 4, we have
$$
r _ {i} \leq q _ {i} ^ {\beta / \beta^ {\prime}} \leq 3 q _ {i} ^ {\beta / \beta^ {\prime}}.
$$
H.2 Proof of Theorem 4
In this section, we will provide the proof of Theorem 4. We first see that the ratio  $\beta_{t} / \beta_{t + 1}$  is close to one to some extent.
Lemma 6. The learning rate  $\beta_{t}$  in (8) satisfies
$$
1 - \frac {\beta_ {t}}{\beta_ {t + 1}} \in (0, 1 / 1 0 ].
$$
Proof. Recall that  $\beta_{t} = \beta_{1} + \sum_{u = 1}^{t - 1}b_{u}$  with  $b_{u} = \frac{c_{1}\nu_{u}}{U_{u}}$  and  $U_{t} = \sqrt{c_{2} + \bar{z}_{t}h_{1} + \sum_{s = 1}^{t - 1}z_{s}h_{s + 1}}$  for  $t\in \{0\} \cup [T]$ . It suffices to show
$$
\frac {\beta_ {t}}{\beta_ {t + 1}} = \frac {\beta_ {t}}{\beta_ {t} + b _ {t}} \geq \frac {9}{1 0} \Leftrightarrow \beta_ {t} \geq 9 b _ {t}.
$$
This indeed follows since using  $\nu_{t} \leq \beta_{t} / 2$  we have
$$
b _ {t} = \frac {c _ {1} \nu_ {t}}{\sqrt {8 1 c _ {1} ^ {2} + \sum_ {s = 1} ^ {t - 1} z _ {s} h _ {s} + z _ {t} h _ {t + 1}}} \leq \frac {c _ {1} \nu_ {t}}{\sqrt {8 1 c _ {1} ^ {2}}} = \frac {\nu_ {t}}{9} \leq \frac {\beta_ {t}}{9}.
$$
Finally, we are ready to prove one of the key lemmas for proving the BOBW regret bound with the sparsity-dependent bound. Recall that we have  $p_t = \left(1 - \frac{k}{T}\right) q_t + \frac{1}{T} \mathbf{1}$  and  $h_t = \frac{1}{1 - \frac{k}{T}} H(p_t)$ . Using the result in Appendix H.1, we will show that  $h_{t+1}$  is bounded in terms of  $h_t$ .
Lemma 7. Suppose that  $\beta_{t}$  is defined as (8). Then,
$$
h _ {t + 1} \leq 3 h _ {t} + \frac {2 0 k}{9} \left(\frac {\beta_ {t + 1}}{\beta_ {t}} - 1\right) \log \left(\frac {T}{k}\right) h _ {t + 1}.
$$
Proof. Let us recall that  $q_{t}$  and  $q_{t + 1}$  are defined as
$$
q _ {t} \in \operatorname * {a r g   m i n} _ {q \in \mathcal {P} _ {k}} \left\langle \sum_ {s = 1} ^ {t - 1} \widehat {y} _ {s}, q \right\rangle + \Phi_ {t} (q) \quad \text {a n d} \quad q _ {t + 1} \in \operatorname * {a r g   m i n} _ {q \in \mathcal {P} _ {k}} \left\langle \sum_ {s = 1} ^ {t} \widehat {y} _ {s}, q \right\rangle + \Phi_ {t + 1} (q),
$$
which corresponds to optimization problems (28) with  $p = q_t$ ,  $L = \sum_{s=1}^{t} \widehat{y}_s$ ,  $\xi = \widehat{y}_{tA_t}$ ,  $\psi = \Phi_t / \beta_t$ ,  $\eta = 1 / \beta_t$ ,  $r = q_{t+1}$ ,  $\psi' = \Phi_{t+1} / \beta_{t+1}$ , and  $\eta' = 1 / \beta_{t+1}$ .
Since  $H$  is concave, by  $p_{ti} = (1 - \frac{k}{T})q_{ti} + \frac{1}{T}$  and Jensen's inequality we have
$$
\left(1 - \frac {k}{T}\right) h _ {t} = H \left(p _ {t}\right) \geq \left(1 - \frac {k}{T}\right) H \left(q _ {t}\right) + \frac {k}{T} H \left(\frac {1}{k} \mathbf {1}\right) \geq \left(1 - \frac {k}{T}\right) H \left(q _ {t}\right),
$$
which implies  $h_t \geq H(q_t)$ . By Lemma 5 we also have  $q_{t+1,i} \leq 3q_{ti}^{\beta_t / \beta_{t+1}}$ , which implies that
$$
p _ {t + 1, i} = \left(1 - \frac {k}{T}\right) q _ {t + 1, i} + \frac {1}{T} \leq \left(1 - \frac {k}{T}\right) 3 q _ {t i} ^ {\beta_ {t} / \beta_ {t + 1}} + \frac {1}{T} \leq 6 p _ {t i} ^ {\beta_ {t} / \beta_ {t + 1}}.
$$
The last inequality follows since when  $\left(1 - \frac{k}{T}\right) 3{q}_{ti}^{{\beta }_{t}/{\beta }_{t + 1}} \leq  \frac{1}{T}$  ,
$$
\left(1 - \frac {k}{T}\right) 3 q _ {t i} ^ {\beta_ {t} / \beta_ {t + 1}} + \frac {1}{T} \leq \frac {2}{T} \leq 2 \left(\frac {1}{T}\right) ^ {\beta_ {t} / \beta_ {t + 1}} \leq 2 \left(\left(1 - \frac {k}{T}\right) q _ {t i} + \frac {1}{T}\right) ^ {\beta_ {t} / \beta_ {t + 1}} = 2 p _ {t i} ^ {\beta / \beta_ {t + 1}},
$$
and otherwise
$$
\left(1 - \frac {k}{T}\right) 3 q _ {t i} ^ {\beta_ {t} / \beta_ {t + 1}} + \frac {1}{T} \leq 6 \left(1 - \frac {k}{T}\right) ^ {\beta_ {t} / \beta_ {t + 1}} q _ {t i} ^ {\beta_ {t} / \beta_ {t + 1}} \leq 6 p _ {t i} ^ {\beta / \beta_ {t + 1}}.
$$
Using these inequalities, we have
$$
\begin{array}{l} h _ {t + 1} - 3 h _ {t} = \frac {1}{1 - \frac {k}{T}} (H (p _ {t + 1}) - 3 H (p _ {t})) \\ \leq \frac {1}{1 - \frac {k}{T}} \left(H (p _ {t}) + \langle \nabla H (p _ {t}), p _ {t + 1} - p _ {t} \rangle - 3 H (p _ {t})\right) \\ = \frac {1}{1 - \frac {k}{T}} \sum_ {i = 1} ^ {k} \left(p _ {t + 1, i} - 3 p _ {t i}\right) \log \left(\frac {1}{p _ {t i}}\right) \\ \leq \sum_ {i = 1} ^ {k} \left(q _ {t + 1, i} - 3 q _ {t i}\right) \log \left(\frac {1}{p _ {t i}}\right), \tag {33} \\ \end{array}
$$
where the first inequality follows by the concavity of  $H$ , the second inequality follows since  $p_{t+1,i} - 3p_{ti} \leq \left(1 - \frac{k}{T}\right)\left(q_{t+1,i} - q_{ti}\right)$ . Defining  $\mathcal{Q}_t = \{i \in [k] : q_{t+1,i} - 3q_{ti} \geq 0\}$ , (33) is further bounded as
$$
\begin{array}{l} \sum_ {i = 1} ^ {k} \left(q _ {t + 1, i} - 3 q _ {t i}\right) \log \left(\frac {1}{p _ {t i}}\right) = \sum_ {i \in \Omega_ {t}} \left(q _ {t + 1, i} - 3 q _ {t i}\right) \log \left(\frac {1}{p _ {t i}}\right) + \sum_ {i \notin \Omega_ {t}} \left(q _ {t + 1, i} - 3 q _ {t i}\right) \log \left(\frac {1}{p _ {t i}}\right) \\ \leq \frac {\beta_ {t + 1}}{\beta_ {t}} \sum_ {i \in \Omega_ {t}} \left(q _ {t + 1, i} - 3 q _ {t i}\right) \log \left(\frac {1}{p _ {t + 1 , i}}\right) + 0 \\ \leq \frac {1 0}{9} \sum_ {i \in \Omega_ {t}} \left(q _ {t + 1, i} - 3 q _ {t i}\right) \log \left(\frac {1}{p _ {t + 1 , i}}\right) \\ \leq \frac {1 0}{9} \sum_ {i \in \mathbb {Q} _ {t}} \left(q _ {t + 1, i} - q _ {t + 1, i} ^ {\beta_ {t + 1} / \beta_ {t}}\right) \log \left(\frac {1}{p _ {t + 1 , i}}\right) \\ = \frac {1 0}{9} \sum_ {i \in \Omega_ {t}} q _ {t + 1, i} \left(1 - q _ {t + 1, i} ^ {\frac {\beta_ {t + 1}}{\beta_ {t}} - 1}\right) \log \left(\frac {1}{p _ {t + 1 , i}}\right), \tag {34} \\ \end{array}
$$
where the first inequality follows by  $p_{t+1,i} \leq 6p_t^{\beta_t / \beta_{t+1}}$ , the second follows by Lemma 6, and the last inequality follows by  $q_{t+1,i} \leq 3q_{ti}^{\beta_t / \beta_{t+1}}$ . Since for any  $\varepsilon > 0$ ,  $x \in [0,1]$ , and  $\gamma \in [0,1]$ , it holds that
$$
\begin{array}{l} x \left(1 - x ^ {\varepsilon}\right) \leq x \log \left(\frac {1}{x ^ {\varepsilon}}\right) = \varepsilon x \log \left(\frac {1}{x}\right) \\ \leq \varepsilon \left(\left(\log \frac {1}{\gamma} - 1\right) (x - r) + \gamma \log \frac {1}{\gamma}\right) \leq \varepsilon \log \left(\frac {1}{\gamma}\right) (\gamma + (1 - \gamma) x), \tag {35} \\ \end{array}
$$
setting  $\gamma = k / T$  in (35) implies that the RHS of (34) is further bounded as