SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} \left[ \epsilon \sum_ {t = 1} ^ {T} \left\langle \widehat {L} _ {U} ^ {t + 1}, q ^ {t + 1} - q ^ {t} \right\rangle \right] = \epsilon \cdot \mathbb {E} \left[ \left\langle \widehat {L} _ {U} ^ {T + 1}, q ^ {T + 1} \right\rangle - \left\langle \widehat {L} _ {U} ^ {2}, q ^ {1} \right\rangle + \sum_ {t = 2} ^ {T} \left\langle q ^ {t}, \widehat {L} _ {U} ^ {t} - \widehat {L} _ {U} ^ {t + 1} \right\rangle \right] \\ \leq \epsilon \cdot \mathbb {E} \left[ \left\langle \widehat {L} _ {U} ^ {T + 1}, \mathbf {1} _ {U} \right\rangle \right] \leq \mathcal {O} (\epsilon K T), \\ \end{array}
$$
where the second step follows  $\widehat{L}_U^t -\widehat{L}_U^{t + 1} = -\widehat{\ell}_U^t$  and the last step uses  $q_i^t\leq 1$  for  $\forall t,i$
Finally, by telescoping, we have
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} D ^ {t} \left(\widetilde {q} ^ {t}, p ^ {t}\right) - D ^ {t + 1} \left(\widetilde {q} ^ {t + 1}, p ^ {t + 1}\right) \right] = \mathbb {E} \left[ D ^ {1} \left(\widetilde {q} ^ {1}, p ^ {1}\right) - D ^ {t + 1} \left(\widetilde {q} ^ {T + 1}, p ^ {T + 1}\right) \right] \leq \mathbb {E} \left[ D ^ {1} \left(\widetilde {q} ^ {1}, p ^ {1}\right) \right],
$$
where the second step holds since the Bregman divergence is non-negative. Combining all the inequalities above concludes the proof.
Lemma A.1.4 (PRECOST for Shannon Entropy). When using the Shannon entropy regularizer with  $\alpha = 0$  and  $\theta = \sqrt{1 / \log T}$ , Algorithm 1 ensures
- Eq. (17) with  $B_{1} = 4\epsilon T\log K + 1$
Eq. (18) with  $B_{2} = 0$
$D^{1}(\tilde{q}^{1},p^{1})\leq \frac{\log K}{\sqrt{\log T}} +C_{\mathrm{LOG}}|V|\log \left(\frac{|V|}{K\epsilon}\right) + \frac{\epsilon C_{\mathrm{LOG}}|U|}{1 - \epsilon}.$
Therefore, according to Lemma A.1.3, by picking  $\epsilon = \frac{1}{T}$ , our algorithm ensures
$$
\operatorname {P R E C O S T} = \mathcal {O} \left(C _ {\log} K \log T + D\right).
$$
Proof. We first show Eq. (18). For the entries in  $V$ , we have
$$
\begin{array}{l} \phi_ {V} ^ {t + 1} (\widetilde {q} ^ {t + 1}) - \phi_ {V} ^ {t} (\widetilde {q} ^ {t}) = \sum_ {i \in V} \left(\gamma_ {i} ^ {t + 1} \widetilde {q} _ {i} ^ {t + 1} \log \left(\frac {\widetilde {q} _ {i} ^ {t + 1}}{e}\right) - \gamma_ {i} ^ {t} \widetilde {q} _ {i} ^ {t} \log \left(\frac {\widetilde {q} _ {i} ^ {t}}{e}\right)\right) \\ - C _ {\log} \cdot \sum_ {i \in V} \left(\log \hat {q} _ {i} ^ {t + 1} - \log \hat {q} _ {i} ^ {t}\right) \\ = \sum_ {i \in V} \left(\gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t}\right) \cdot \frac {\epsilon}{| V |} \log \left(\frac {\epsilon}{e | V |}\right) \leq 0, \\ \end{array}
$$
where the second step follows from the definition of  $\widetilde{q}$  in Eq. (15) which states  $\widetilde{q}_i^t = \frac{\epsilon}{|V|}$  for any  $i\in V$
For the entries in  $U$ , by direct calculation, we have
$$
\begin{array}{l} \phi_ {U} ^ {t + 1} \left(\tilde {q} ^ {t + 1}\right) - \phi_ {U} ^ {t} \left(\tilde {q} ^ {t}\right) \\ = \sum_ {i \in U} \left(\gamma_ {i} ^ {t + 1} \tilde {q} _ {i} ^ {t + 1} \log \left(\frac {\tilde {q} _ {i} ^ {t + 1}}{e}\right) - \gamma_ {i} ^ {t} \tilde {q} _ {i} ^ {t} \log \left(\frac {\tilde {q} _ {i} ^ {t}}{e}\right)\right) - C _ {\mathrm {L O G}} \sum_ {i \in U} \left(\log \tilde {q} _ {i} ^ {t + 1} - \log \tilde {q} _ {i} ^ {t}\right) \\ = (1 - \epsilon) \sum_ {i \in U} \left(\gamma_ {i} ^ {t + 1} q _ {i} ^ {t + 1} \log \left(\frac {\widetilde {q} _ {i} ^ {t + 1}}{e}\right) - \gamma_ {i} ^ {t} q _ {i} ^ {t} \log \left(\frac {\widetilde {q} _ {i} ^ {t}}{e}\right)\right) - C _ {\mathrm {L O G}} \sum_ {i \in U} \left(\log q _ {i} ^ {t + 1} - \log q _ {i} ^ {t}\right) \\ = (1 - \epsilon) \sum_ {i \in U} \left(\gamma_ {i} ^ {t + 1} q _ {i} ^ {t + 1} \log \left(\frac {q _ {i} ^ {t + 1}}{e}\right) - \gamma_ {i} ^ {t} q _ {i} ^ {t} \log \left(\frac {q _ {i} ^ {t}}{e}\right) + \log (1 - \epsilon) (\gamma_ {i} ^ {t + 1} q _ {i} ^ {t + 1} - \gamma_ {i} ^ {t} q _ {i} ^ {t})\right) \\ - C _ {\log} \sum_ {i \in U} \left(\log q _ {i} ^ {t + 1} - \log q _ {i} ^ {t}\right) \\ \end{array}
$$
$$
\begin{array}{l} = \phi_ {U} ^ {t + 1} \left(q ^ {t + 1}\right) - \phi_ {U} ^ {t} \left(q ^ {t}\right) \\ + (1 - \epsilon) \log (1 - \epsilon) \sum_ {i \in U} \left(\gamma_ {i} ^ {t + 1} q _ {i} ^ {t + 1} - \gamma_ {i} ^ {t} q _ {i} ^ {t}\right) (19) \\ - \epsilon \sum_ {i \in U} \left(\gamma_ {i} ^ {t + 1} q _ {i} ^ {t + 1} \log \left(\frac {q _ {i} ^ {t + 1}}{e}\right) - \gamma_ {i} ^ {t} q _ {i} ^ {t} \log \left(\frac {q _ {i} ^ {t}}{e}\right)\right). (20) \\ \end{array}
$$
Note that, taking the summation of Eq. (19) over all rounds yields
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} (1 - \epsilon) \log (1 - \epsilon) \sum_ {i \in U} \left(\gamma_ {i} ^ {t + 1} q _ {i} ^ {t + 1} - \gamma_ {i} ^ {t} q _ {i} ^ {t}\right) \\ = (1 - \epsilon) \log (1 - \epsilon) \sum_ {i \in U} \left(\gamma_ {i} ^ {T + 1} q _ {i} ^ {T + 1} - \gamma_ {i} ^ {1} q _ {i} ^ {1}\right) \\ \leq - (1 - \epsilon) \log (1 - \epsilon) \sum_ {i \in U} \gamma_ {i} ^ {1} q _ {i} ^ {1} \leq \sum_ {i \in U} \gamma_ {i} ^ {1} q _ {i} ^ {1} \leq 1, \\ \end{array}
$$
where the second and third step follow from the fact  $-1 \leq (1 - \epsilon) \log (1 - \epsilon) \leq 0$ , and the last step uses the definition of the learning rates.
On the other hand, by summing Eq. (20) over all rounds, we have
$$
\begin{array}{l} - \epsilon \sum_ {t = 1} ^ {T} \sum_ {i \in U} \left(\gamma_ {i} ^ {t + 1} q _ {i} ^ {t + 1} \left(\log \left(q _ {i} ^ {t + 1}\right) - 1\right) - \gamma_ {i} ^ {t} q _ {i} ^ {t} \left(\log \left(q _ {i} ^ {t}\right) - 1\right)\right) \\ = - \epsilon \sum_ {i \in U} \left(\gamma_ {i} ^ {T + 1} q _ {i} ^ {T + 1} \left(\log \left(q _ {i} ^ {T + 1}\right) - 1\right) - \gamma_ {i} ^ {1} q _ {i} ^ {1} \left(\log \left(q _ {i} ^ {1}\right) - 1\right)\right) \\ \leq \epsilon \sum_ {i \in U} \gamma_ {i} ^ {T + 1} q _ {i} ^ {T + 1} \left(\log \left(\frac {1}{q _ {i} ^ {T + 1}}\right) + 1\right) \\ \leq 2 \epsilon T \left(1 + \sum_ {i \in U} q _ {i} ^ {T + 1} \log \left(\frac {1}{q _ {i} ^ {T + 1}}\right)\right) \\ \leq 2 \epsilon T (1 + \log | U |), \\ \end{array}
$$
where the second step uses the fact that  $x \log x - x \leq 0$  for any  $x \in [0,1]$ ; the third step follows from the definition of  $\gamma_i^t$  which ensures  $\gamma_i^{T + 1} \leq \sqrt{1 + T \cdot T} \leq 2T$ ; the last step follows from Jensen's inequality.
Finally, we bound  $D^{1}(\widetilde{q}^{1},p^{1})$  as:
$$
\begin{array}{l} D ^ {1} \left(\widetilde {q} ^ {1}, p ^ {1}\right) = \sum_ {i \in [ K ]} \gamma_ {i} ^ {1} \left(\widetilde {q} _ {i} ^ {1} \log \widetilde {q} _ {i} ^ {1} - p _ {i} ^ {1} \log p _ {i} ^ {1}\right) + \sum_ {i \in [ K ]} \gamma_ {i} ^ {1} \left(p _ {i} ^ {1} - \widetilde {q} _ {i} ^ {1}\right) - C _ {\mathrm {L O G}} \sum_ {i \in [ K ]} \left(\log \left(\widetilde {q} _ {i} ^ {1}\right) - \log \left(p _ {i} ^ {1}\right)\right) \\ \leq \sum_ {i \in [ K ]} \gamma_ {i} ^ {1} \left(- p _ {i} ^ {1} \log p _ {i} ^ {1}\right) + C _ {\mathrm {L O G}} | V | \log \left(\frac {| V |}{K \epsilon}\right) + C _ {\mathrm {L O G}} | U | \log \left(\frac {1}{1 - \epsilon}\right) \\ \leq \frac {\log K}{\sqrt {\log T}} + C _ {\mathrm {L O G}} | V | \log \left(\frac {| V |}{K \epsilon}\right) + C _ {\mathrm {L O G}} | U | \log \left(\frac {1}{1 - \epsilon}\right) \\ \leq \frac {\log K}{\sqrt {\log T}} + C _ {\mathrm {L O G}} | V | \log \left(\frac {| V |}{K \epsilon}\right) + \frac {\epsilon C _ {\mathrm {L O G}} | U |}{1 - \epsilon}, \\ \end{array}
$$
where the second step follows from the fact that  $x \log x \leq 0$  for  $x \in [0,1]$ ; the third step follows from the facts that  $p_i^1 = \frac{1}{K}$  and  $\gamma_i^1 = \sqrt{1 / \log T}$  for any  $i \in [K]$ ; the forth step uses the fact that  $\log \left( \frac{1}{1 - \epsilon} \right) = \log \left( 1 + \frac{\epsilon}{1 - \epsilon} \right) \leq \frac{\epsilon}{1 - \epsilon}$  for any  $\epsilon \in (0,1)$ .
Lemma A.1.5 (PRECOST for Log-barrier). When using the log-barrier regularizer with  $\alpha = 1$  and  $\theta = \sqrt{1 / \log T}$ , Algorithm 1 ensures
- Eq. (17) with  $B_{1} = \frac{\epsilon|U|T}{1 - \epsilon}$ ;
- Eq. (18) with  $B_{2} = \mathbb{S}_{2}\left(\frac{\log^{2}\left(\frac{|V|}{\epsilon}\right)}{\log T}\right)$ ;
-  $D^{1}(\tilde{q}^{1},p^{1})\leq \frac{|V|}{\sqrt{\log T}}\log \left(\frac{|V|}{K\epsilon}\right) + \frac{|U|}{\sqrt{\log T}}\log \left(\frac{|U|}{(1 - \epsilon)K}\right) + C_{\mathrm{LOG}}|V|\log \left(\frac{|V|}{K\epsilon}\right) + \frac{\epsilon C_{\mathrm{LOG}}|U|}{1 - \epsilon}.$
$$
\operatorname {P R E C O S T} = \mathcal {O} \left(\mathbb {S} _ {2} (\log T) + C _ {\log} K \log T + D\right).
$$
Proof. We first show that Eq. (18) holds. For the entries in  $V$ , we have
$$
\begin{array}{l} \phi_ {V} ^ {t + 1} (\widetilde {q} ^ {t + 1}) - \phi_ {V} ^ {t} (\widetilde {q} ^ {t}) = - \sum_ {i \in V} \left(\gamma_ {i} ^ {t + 1} \log (\widetilde {q} _ {i} ^ {t + 1}) - \gamma_ {i} ^ {t} \log (\widetilde {q} _ {i} ^ {t})\right) - C _ {\mathrm {L O G}} \cdot \sum_ {i \in V} \left(\log \widetilde {q} _ {i} ^ {t + 1} - \log \widetilde {q} _ {i} ^ {t}\right) \\ = \sum_ {i \in V} \left(\gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t}\right) \log \left(\frac {| V |}{\epsilon}\right), \\ \end{array}
$$
where the second step follows from the definition of  $\hat{q}^t$  in Eq. (15) that  $\widetilde{q}_i^t = \frac{\epsilon}{|V|}$  for all  $i\in V$ . Taking the summation over all rounds, we have
$$
\sum_ {t = 1} ^ {T} \sum_ {i \in V} \left(\gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t}\right) \log \left(\frac {| V |}{\epsilon}\right) \leq \sum_ {i \in V} \gamma_ {i} ^ {T + 1} \log \left(\frac {| V |}{\epsilon}\right),
$$
which could be bounded with  $\mathbb{S}_2(\cdot)$  with our specific learning rate schedule as:
$$
\mathbb {E} \left[ \sum_ {i \in V} \gamma_ {i} ^ {T + 1} \log \left(\frac {| V |}{\epsilon}\right) \right] = \mathcal {O} \left(\sum_ {i \in V} \sqrt {\frac {\log^ {2} \left(\frac {| V |}{\epsilon}\right)}{\log T} \cdot \left(\sum_ {t = 1} ^ {T} p _ {i} ^ {t}\right)}\right) = \mathcal {O} \left(\mathbb {S} _ {2} \left(\frac {\log^ {2} \left(\frac {| V |}{\epsilon}\right)}{\log T}\right)\right).
$$
For the entries in  $U$ , we have
$$
\begin{array}{l} \phi_ {U} ^ {t + 1} \left(\widetilde {q} ^ {t + 1}\right) - \phi_ {U} ^ {t} \left(\widetilde {q} ^ {t}\right) \\ = - \sum_ {i \in U} \left(\gamma_ {i} ^ {t + 1} \log \left(\widetilde {q} _ {i} ^ {t + 1}\right) - \gamma_ {i} ^ {t} \log \left(\widetilde {q} _ {i} ^ {t}\right)\right) - C _ {\mathrm {L O G}} \cdot \sum_ {i \in U} \left(\log \widetilde {q} _ {i} ^ {t + 1} - \log \widehat {q} _ {i} ^ {t}\right) \\ = - \sum_ {i \in U} \left(\gamma_ {i} ^ {t + 1} \log \left(q _ {i} ^ {t + 1}\right) - \gamma_ {i} ^ {t} \log \left(q _ {i} ^ {t}\right)\right) - C _ {\log} \cdot \sum_ {i \in U} \left(\log q _ {i} ^ {t + 1} - \log q _ {i} ^ {t}\right) \\ + \log \left(\frac {1}{1 - \epsilon}\right) \sum_ {i \in U} \left(\gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t}\right) \\ \leq \phi_ {U} ^ {t + 1} (q ^ {t + 1}) - \phi_ {U} ^ {t} (q ^ {t}) + \frac {\epsilon}{1 - \epsilon} \sum_ {i \in U} \left(\gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t}\right). \\ \end{array}
$$
By summing over all rounds and direct calculation, we have
$$
\frac {\epsilon}{1 - \epsilon} \sum_ {t = 1} ^ {T} \sum_ {i \in U} \left(\gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t}\right) \leq \frac {\epsilon}{1 - \epsilon} \sum_ {i \in U} \gamma_ {i} ^ {T + 1} \leq \frac {\epsilon}{1 - \epsilon} | U | T.
$$
Finally, we bound  $D^{1}(\widetilde{q}^{1},p^{1}) = \phi^{1}(\widetilde{q}^{1}) - \phi^{1}(p^{1}) - \left\langle \nabla \phi^{1}(p^{1}),\widetilde{q}^{1} - p^{1}\right\rangle = \phi^{1}(\widetilde{q}^{1}) - \phi^{1}(p^{1})$  , because  $\sum_{i\in [K]}\widetilde{q}_i^1 = \sum_{i\in [K]}p_i^1$  and  $p^1 = \frac{1}{K}\cdot \mathbf{1}_K$  which implies that all entries of  $\nabla \phi^1 (p^1)$  are equal. Note that  $p^1 = \frac{1}{K}\cdot \mathbf{1}_K$ $\widetilde{q}_U^1 = \frac{1 - \epsilon}{|U|}\cdot \mathbf{1}_U$  , and  $\widetilde{q}_V^1 = \frac{\epsilon}{|V|}\cdot \mathbf{1}_V$  . For  $\phi_V^1 (\widetilde{q}^1) - \phi_V^1 (p^1)$  , we have
$$
\begin{array}{l} \phi_ {V} ^ {1} \left(\widetilde {q} ^ {1}\right) - \phi_ {V} ^ {1} \left(p ^ {1}\right) = \sum_ {i \in V} \gamma_ {i} ^ {1} \log \left(\frac {p _ {i} ^ {1}}{\widetilde {\tilde {q}} _ {i} ^ {1}}\right) - C _ {\mathrm {L O G}} \sum_ {i \in V} \left(\log \left(\widetilde {q} _ {i} ^ {1}\right) - \log \left(p _ {i} ^ {1}\right)\right) \\ = \sum_ {i \in V} \gamma_ {i} ^ {1} \log \left(\frac {1 / K}{\epsilon / | V |}\right) + C _ {\mathrm {L O G}} | V | \log \left(\frac {| V |}{K \epsilon}\right) \\ \end{array}
$$
$$
\leq \frac {| V |}{\sqrt {\log T}} \log \left(\frac {| V |}{K \epsilon}\right) + C _ {\mathrm {L O G}} | V | \log \left(\frac {| V |}{K \epsilon}\right),
$$
where  $C_{\mathrm{LOG}} \sum_{i \in V} \left( \log \left( \widetilde{q}_i^1 \right) - \log \left( p_i^1 \right) \right)$  is bounded via a similar way used to bound the Shannon entropy. Similarly, for  $\phi_U^1(\widetilde{q}^1) - \phi_U^1(p^1)$ , we have
$$
\begin{array}{l} \phi_ {U} ^ {1} (\widetilde {q} ^ {1}) - \phi_ {U} ^ {1} (p ^ {1}) = \sum_ {i \in U} \gamma_ {i} ^ {1} \log \left(\frac {p _ {i} ^ {1}}{\widetilde {q} _ {i} ^ {1}}\right) - C _ {\mathrm {L O G}} \sum_ {i \in U} \left(\log \left(\widetilde {q} _ {i} ^ {1}\right) - \log \left(p _ {i} ^ {1}\right)\right) \\ \leq \sum_ {i \in U} \gamma_ {i} ^ {1} \log \left(\frac {1 / K}{(1 - \epsilon) / | U |}\right) + \frac {\epsilon C _ {\mathrm {L O G}} | U |}{1 - \epsilon} \\ \leq \frac {| U |}{\sqrt {\log T}} \log \left(\frac {| U |}{(1 - \epsilon) K}\right) + \frac {\epsilon C _ {\mathrm {L O G}} | U |}{1 - \epsilon}. \\ \end{array}
$$
Combining bounds for  $\phi_V^1 (\widetilde{q}^1) - \phi_V^1 (p^1)$  and  $\phi_U^1 (\widetilde{q}^1) - \phi_U^1 (p^1)$ , we have  $D^{1}(\widetilde{q}^{1},p^{1})$  bounded by
$$
\frac {| V |}{\sqrt {\log T}} \log \left(\frac {| V |}{K \epsilon}\right) + \frac {| U |}{\sqrt {\log T}} \log \left(\frac {| U |}{(1 - \epsilon) K}\right) + C _ {\mathrm {L O G}} | V | \log \left(\frac {| V |}{K \epsilon}\right) + \frac {\epsilon C _ {\mathrm {L O G}} | U |}{1 - \epsilon}.
$$
Lemma A.1.6 (PRECOST for  $\beta$ -Tsallis Entropy). For any  $\beta \in (0,1)$ , when using the  $\beta$ -Tsallis entropy regularizer with  $\alpha = \beta$  and  $\theta = \sqrt{\frac{1 - \beta}{\beta}}$ , Algorithm 1 ensures
Eq. (17) with  $B_{1} = \frac{2\epsilon KT}{\sqrt{\beta(1 - \beta)}}$
Eq. (18) with  $B_{2} = 0$
$D^{1}(\widetilde{q}^{1},p^{1})\leq \frac{K}{\sqrt{\beta(1 - \beta)}} +C_{\mathrm{LOG}}|V|\log \left(\frac{|V|}{K\epsilon}\right) + \frac{\epsilon C_{\mathrm{LOG}}|U|}{1 - \epsilon}.$
Therefore, according to Lemma A.1.3, by picking  $\epsilon = \frac{1}{T}$ , our algorithm further ensures
$$
\operatorname {P R E C O S T} = \mathcal {O} \left(\frac {K}{\sqrt {\beta (1 - \beta)}} + C _ {\log} K \log T + D\right).
$$
Proof. Let us first show that Eq. (18) holds. For the entries in  $V$ , we have
$$
\begin{array}{l} \phi_ {V} ^ {t + 1} (\widetilde {q} ^ {t + 1}) - \phi_ {V} ^ {t} (\widetilde {q} ^ {t}) = - \frac {1}{1 - \beta} \sum_ {i \in V} \left(\gamma_ {i} ^ {t + 1} (\widetilde {q} _ {i} ^ {t + 1}) ^ {\beta} - \gamma_ {i} ^ {t} (\widetilde {q} _ {i} ^ {t}) ^ {\beta}\right) - C _ {\mathrm {L O G}} \cdot \sum_ {i \in V} \left(\log \widetilde {q} _ {i} ^ {t + 1} - \log \widetilde {q} _ {i} ^ {t}\right) \\ = - \frac {1}{1 - \beta} \sum_ {i \in V} \left(\gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t}\right) \cdot \left(\frac {\epsilon}{| V |}\right) ^ {\beta} \leq 0, \\ \end{array}
$$
where the second step follows from the definition of  $\widetilde{q}^t$  in Eq. (15) that  $\widetilde{q}_i^t = \frac{\epsilon}{|V|}$  for any  $i\in V$ . Since  $\phi_V^{t + 1}(\widetilde{q}^{t + 1}) - \phi_V^t (\widetilde{q}^t)\leq 0$ , we have  $\sum_{t = 1}^{T}\phi_V^{t + 1}(\widetilde{q}^{t + 1}) - \phi_V^t (\widetilde{q}^t)\leq 0$ , which gives  $B_{2} = 0$  for Eq. (18).
$$
\begin{array}{l} \phi_ {U} ^ {t + 1} (\tilde {q} ^ {t + 1}) - \phi_ {U} ^ {t} (\tilde {q} ^ {t}) \\ = - \frac {1}{1 - \beta} \sum_ {i \in U} \left(\gamma_ {i} ^ {t + 1} \left(\tilde {q} _ {i} ^ {t + 1}\right) ^ {\beta} - \gamma_ {i} ^ {t} \left(\tilde {q} _ {i} ^ {t}\right) ^ {\beta}\right) - C _ {\log} \cdot \sum_ {i \in U} \left(\log \tilde {q} _ {i} ^ {t + 1} - \log \tilde {q} _ {i} ^ {t}\right) \\ = - \frac {(1 - \epsilon) ^ {\beta}}{1 - \beta} \sum_ {i \in U} \left(\gamma_ {i} ^ {t + 1} \left(q _ {i} ^ {t + 1}\right) ^ {\beta} - \gamma_ {i} ^ {t} \left(q _ {i} ^ {t}\right) ^ {\beta}\right) - C _ {\mathrm {L O G}} \cdot \sum_ {i \in U} \left(\log (1 - \epsilon) + \log q _ {i} ^ {t + 1} - \log (1 - \epsilon) - \log q _ {i} ^ {t}\right) \\ = - \frac {(1 - \epsilon) ^ {\beta}}{1 - \beta} \sum_ {i \in U} \left(\gamma_ {i} ^ {t + 1} \left(q _ {i} ^ {t + 1}\right) ^ {\beta} - \gamma_ {i} ^ {t} \left(q _ {i} ^ {t}\right) ^ {\beta}\right) - C _ {\log} \cdot \sum_ {i \in U} \left(\log q _ {i} ^ {t + 1} - \log q _ {i} ^ {t}\right) \\ \end{array}
$$
$$
= \phi_ {U} ^ {t + 1} (q ^ {t + 1}) - \phi_ {U} ^ {t} (q ^ {t}) + \frac {1 - (1 - \epsilon) ^ {\beta}}{1 - \beta} \sum_ {i \in U} \left(\gamma_ {i} ^ {t + 1} (q _ {i} ^ {t + 1}) ^ {\beta} - \gamma_ {i} ^ {t} (q _ {i} ^ {t}) ^ {\beta}\right).
$$
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \frac {1 - (1 - \epsilon) ^ {\beta}}{1 - \beta} \sum_ {i \in U} \left(\gamma_ {i} ^ {t + 1} \left(q _ {i} ^ {t + 1}\right) ^ {\beta} - \gamma_ {i} ^ {t} \left(q _ {i} ^ {t}\right) ^ {\beta}\right) \right] \\ = \mathbb {E} \left[ \frac {1 - (1 - \epsilon) ^ {\beta}}{1 - \beta} \sum_ {i \in U} \left(\gamma_ {i} ^ {T + 1} \left(q _ {i} ^ {T + 1}\right) ^ {\beta} - \gamma_ {i} ^ {1} \left(q _ {i} ^ {1}\right) ^ {\beta}\right) \right] \\ \leq \mathbb {E} \left[ \frac {1 - (1 - \epsilon) ^ {\beta}}{1 - \beta} \sum_ {i \in U} \gamma_ {i} ^ {T + 1} \right] \\ \leq \frac {1 - (1 - \epsilon) ^ {\beta}}{1 - \beta} \cdot 2 K T \sqrt {\frac {1 - \beta}{\beta}} \\ \leq \frac {2 \epsilon K T}{\sqrt {\beta (1 - \beta)}}, \\ \end{array}
$$
where the third step uses the fact that  $\left(\max \left\{p_i^t, 1 / T\right\}\right)^{1 - 2\beta} \leq \left(\max \left\{p_i^t, 1 / T\right\}\right)^{-1} \leq T$
$$
\gamma_ {i} ^ {T + 1} = \sqrt {\frac {1 - \beta}{\beta}} \cdot \sqrt {1 + \sum_ {t = 1} ^ {T} \left(\max  \left\{p _ {i} ^ {t} , ^ {1} / T \right\}\right) ^ {1 - 2 \beta}} \leq \sqrt {\frac {1 - \beta}{\beta}} \cdot \sqrt {1 + \sum_ {t = 1} ^ {T} T} \leq 2 T \sqrt {\frac {1 - \beta}{\beta}},
$$
and the last step uses the fact that  $(1 - \epsilon)^{\beta} \geq 1 - \epsilon$  for any  $\beta \in (0,1)$ .
Finally, we have by direct calculation:
$$
\begin{array}{l} D ^ {1} \left(\tilde {q} ^ {1}, p ^ {1}\right) = - \frac {1}{1 - \beta} \sum_ {i \in [ K ]} \gamma_ {i} ^ {1} \left(\left(\tilde {q} _ {i} ^ {1}\right) ^ {\beta} - \left(p _ {i} ^ {1}\right) ^ {\beta}\right) - C _ {\log} \sum_ {i \in [ K ]} \left(\log \left(\tilde {q} _ {i} ^ {1}\right) - \log \left(p _ {i} ^ {1}\right)\right) \\ \leq \frac {1}{1 - \beta} \sum_ {i \in [ K ]} \gamma_ {i} ^ {1} + C _ {\mathrm {L O G}} \sum_ {i \in [ K ]} \log \left(\frac {1}{K \widetilde {q} _ {i} ^ {1}}\right) + \frac {\epsilon C _ {\mathrm {L O G}} | U |}{1 - \epsilon} \\ \leq \frac {1}{1 - \beta} \sum_ {i \in [ K ]} \gamma_ {i} ^ {1} + C _ {\log} | V | \log \left(\frac {| V |}{K \epsilon}\right) + \frac {\epsilon C _ {\log} | U |}{1 - \epsilon} \\ = \frac {K}{\sqrt {\beta (1 - \beta)}} + C _ {\mathrm {L O G}} | V | \log \left(\frac {| V |}{K \epsilon}\right) + \frac {\epsilon C _ {\mathrm {L O G}} | U |}{1 - \epsilon}, \\ \end{array}
$$
where the first step uses the fact that  $\left\langle \phi^1 (p^1),\widetilde{q}^1 -p^1\right\rangle = 0$ , since  $\nabla \phi^{1}(p^{1}) = c\cdot \mathbf{1}_{K}$  for some constant  $c\in \mathbb{R}$  and  $\widetilde{q}^1,p^1\in \Omega_K$ ; the third and forth steps follow the definitions that  $p^1 = \frac{1}{K}\cdot \mathbf{1}_K$ $\widetilde{q}_i^1 = \frac{\epsilon}{|V|}$  for any  $i\in V$ , and  $\gamma_i^1 = \sqrt{\frac{1 - \beta}{\beta}}\cdot \mathbf{1}_K$ .
A.2 Regret on Sub-Optimal Arms
One important property of our learning rate schedule is that:
$$
\begin{array}{l} \gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t} = \theta \cdot \frac {\left(\max  \left\{p _ {i} ^ {t} , ^ {1} / T \right\}\right) ^ {1 - 2 \beta}}{\sqrt {1 + \sum_ {k = 1} ^ {t - 1} \left(\max  \left\{p _ {i} ^ {k} , ^ {1} / T \right\}\right) ^ {1 - 2 \beta}} + \sqrt {1 + \sum_ {k = 1} ^ {t} \left(\max  \left\{p _ {i} ^ {k} , ^ {1} / T \right\}\right) ^ {1 - 2 \beta}}} \\ \leq \theta^ {2} \cdot \frac {\left(\max  \left\{p _ {i} ^ {t} , ^ {1} / T \right\}\right) ^ {1 - 2 \beta}}{\theta \cdot \sqrt {1 + \sum_ {k = 1} ^ {t} \left(\max  \left\{p _ {i} ^ {k} , ^ {1} / T \right\}\right) ^ {1 - 2 \beta}}} \\ = \theta^ {2} \cdot \frac {\left(\max  \left\{p _ {i} ^ {t} , ^ {1} / T \right\}\right) ^ {1 - 2 \beta}}{\gamma_ {i} ^ {t + 1}}, \tag {21} \\ \end{array}
$$
holds for any arm  $i \in [K]$  and  $t \in [T]$ , which controls the increase of learning rates from round  $t$  to  $t + 1$ .
On the other hand, our algorithm is somehow stabilized by the extra log-barrier, i.e., it ensures the following multiplicative relation between  $p^t$ ,  $\bar{p}^{t+1}$ , and  $p^{t+1}$ :
$$
\frac {1}{2} p _ {i} ^ {t} \leq \bar {p} _ {i} ^ {t + 1} \leq 2 p _ {i} ^ {t}, \quad \frac {1}{2} p _ {i} ^ {t} \leq p _ {i} ^ {t + 1} \leq 2 p _ {i} ^ {t}, \quad \forall t \in [ T ], \forall i \in [ K ], \tag {22}
$$
according to Lemma C.3.2, Lemma C.3.3, and Lemma C.3.4.
Then, we are ready to analyze the regret related to the sub-optimal arms (REGSUB). For the skewed Bregman divergence  $D_V^{t,t+1}\left(p^t,\bar{p}^{t+1}\right)$ , we can decompose it into the stability and penalty terms as
$$
\begin{array}{l} D _ {V} ^ {t, t + 1} \left(p ^ {t}, \bar {p} ^ {t + 1}\right) = \left\langle p _ {V} ^ {t} - \bar {p} _ {V} ^ {t + 1}, \nabla \phi_ {V} ^ {t + 1} (\bar {p} ^ {t + 1}) - \nabla \phi_ {V} ^ {t} (p ^ {t}) \right\rangle - D _ {V} ^ {t + 1, t} \left(\bar {p} ^ {t + 1}, p ^ {t}\right) \\ = \underbrace {\left\langle p _ {V} ^ {t} - \bar {p} _ {V} ^ {t + 1} , \widehat {\ell} ^ {t} \right\rangle - D _ {V} ^ {t} \left(\bar {p} ^ {t + 1} , p ^ {t}\right)} _ {\text {s t a b i l i t y}} + \underbrace {\phi_ {V} ^ {t} \left(\bar {p} ^ {t + 1}\right) - \phi_ {V} ^ {t + 1} \left(\bar {p} ^ {t + 1}\right)} _ {\text {p e n a l t y}}, \tag {23} \\ \end{array}
$$
where the first step follows from Lemma C.5.1, and the second step follows from the definition of skewed Bregman divergence.
Lemma A.2.1 (Bound on Stability). With  $C_{\mathrm{LOG}} \geq 162$ , Algorithm 1 ensures
$$
\left\langle p _ {V} ^ {t} - \bar {p} _ {V} ^ {t + 1}, \widehat {\ell} ^ {t} \right\rangle - D _ {V} ^ {t} (\bar {p} ^ {t + 1}, p ^ {t}) \leq \mathcal {O} \left(\left\| \widehat {\ell} ^ {t} \right\| _ {\nabla^ {- 2} \phi_ {V} ^ {t} (p ^ {t})} ^ {2}\right),
$$
where  $\| x\| _M\triangleq \sqrt{x^\top Mx}$  is the quadratic norm of  $x\in \mathbb{R}^K$  with respect to some positive semi-definite matrix  $M\in \mathbb{R}^{K\times K}$ , and  $\nabla^{-2}\phi_V^t (p^t)$  denotes the Moore-Penrose inverse of the Hessian  $\nabla^2\phi_V^t (\cdot)$ .
Specifically, the Hessian  $\nabla^2\phi_V^t (x)$  is a diagonal matrix:
$$
\nabla^ {2} \phi_ {V} ^ {t} (x) = \mathrm {d i a g} \left\{\frac {\partial^ {2} \phi_ {V} ^ {t} (x)}{\partial (x _ {i}) ^ {2}}: \forall i \in [ K ] \right\},
$$
where  $\frac{\partial^2\phi_V^t(x)}{\partial(x_i)^2}$  is the second order derivative of  $\phi_V^t (x)$  with respect to the variable  $x_{i}$ . According to the definition of  $\phi_V^t (\cdot),\frac{\partial^2\phi_V^t(x)}{\partial(x_i)^2} = 0$  for any arm  $i\in U$ . Therefore, the Moore-Penrose inverse of  $\nabla^2\phi_V^t (x)$  is also a diagonal matrix, and it holds that
$$
\left(\nabla^ {- 2} \phi_ {V} ^ {t} (x)\right) _ {i, i} = \left\{ \begin{array}{l l} \left(\frac {\partial^ {2} \phi_ {V} ^ {t} (x)}{\partial (x _ {i}) ^ {2}}\right) ^ {- 1}, & i \in V, \\ 0, & i \in U, \end{array} \right.
$$
where  $\left(\nabla^{-2}\phi_V^t (x)\right)_{i,i}$  denotes the  $i$ -th diagonal element of the matrix  $\nabla^{-2}\phi_V^t (x)$ .
Proof. To prove this result, we introduce the minimizer  $z \in \mathbb{R}_{\geq 0}^{K}$  defined as: