SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
z = \underset { \begin{array}{c} x _ {i} = 0, \forall i \in U \\ \sum_ {i \in V} x _ {i} = \sum_ {i \in V} p _ {i} ^ {t} \end{array} } {\operatorname {a r g m i n}} \left\langle \sum_ {\tau \leq t} \widehat {\ell} _ {V} ^ {\tau}, x \right\rangle + \phi_ {V} ^ {t} (x). \tag {24}
$$
By Lemma C.3.5, the extra log-barrier also guarantees the multiplicative relation between  $p^t$  and  $z$ , that is,  $z_i / 2 \leq p_i^t \leq 2z_i$  holds for any arm  $i \in V$ .
According to Lemma C.5.2, we then have
$$
\left\langle p _ {V} ^ {t} - \bar {p} _ {V} ^ {t + 1}, \widehat {\ell} ^ {t} \right\rangle - D _ {V} ^ {t} (\bar {p} ^ {t + 1}, p ^ {t}) \leq \left\langle p _ {V} ^ {t} - z, \widehat {\ell} ^ {t} \right\rangle - D _ {V} ^ {t} (z, p ^ {t}) = \mathcal {O} \left(\left\| \widehat {\ell} _ {V} ^ {t} \right\| _ {\nabla^ {- 2} \phi_ {V} ^ {t} (\xi)} ^ {2}\right),
$$
where  $\xi$  is some intermediate point between  $z$  and  $p^t$ , that is,  $\xi = \rho z + (1 - \rho)p^{t}$  for some  $\rho \in [0,1]$ . With the multiplicative relation between  $z$  and  $p^t$ , we have  $\frac{1}{2} p_i^t\leq \xi_i\leq 2p_i^t$  for any arm  $i\in V$ .
Therefore, one can verify
$$
\frac {1}{4} \cdot \frac {\partial^ {2} \phi_ {V} ^ {t} \left(p _ {i} ^ {t}\right)}{\partial \left(p _ {i} ^ {t}\right) ^ {2}} \leq \frac {\partial^ {2} \phi_ {V} ^ {t} (\xi)}{\partial \left(\xi_ {i}\right) ^ {2}} \leq 4 \cdot \frac {\partial^ {2} \phi_ {V} ^ {t} \left(p _ {i} ^ {t}\right)}{\partial \left(p _ {i} ^ {t}\right) ^ {2}}, \quad \forall i \in V, \tag {25}
$$
for all the regularizers of Algorithm 1. Thus, we can further bound  $\left\| \widehat{\ell}_V^t\right\|_{\nabla^{-2}\phi_V^t (\xi)}^2$  as:
$$
\left\| \widehat {\ell} ^ {t} \right\| _ {\nabla^ {- 2} \phi_ {V} ^ {t} (\xi)} ^ {2} = \sum_ {i \in V} \left(\widehat {\ell} _ {i} ^ {t}\right) ^ {2} \left(\frac {\partial^ {2} \phi_ {V} ^ {t} (\xi)}{\partial (\xi_ {i}) ^ {2}}\right) ^ {- 1} \leq 4 \sum_ {i \in V} \left(\widehat {\ell} _ {i} ^ {t}\right) ^ {2} \left(\frac {\partial^ {2} \phi_ {V} ^ {t} (p _ {i} ^ {t})}{\partial (p _ {i} ^ {t}) ^ {2}}\right) ^ {- 1} = 4 \left\| \widehat {\ell} _ {V} ^ {t} \right\| _ {\nabla^ {- 2} \phi_ {V} ^ {t} (p _ {V} ^ {t})} ^ {2},
$$
where the first step follows from the definition of quadratic norm, and the second step follows from Eq. (25).
Lemma A.2.2 (REGSUB for Log-barrier). When using the log-barrier regularizer with  $C_{\mathrm{LOG}} = 162$ ,  $\alpha = 0$ , and  $\theta = \sqrt{1 / \log T}$ , Algorithm 1 ensures REGSUB = O ( $\mathbb{S}_2(\log T)$ ).
Proof. For the log-barrier regularizer, the multiplicative relation between  $p^t, \bar{p}^{t+1}$ , and  $p^{t+1}$  in Eq. (22) is guaranteed by Lemma C.3.2.
We first consider the stability term. By Lemma A.2.1, we have
$$
\mathbb {E} ^ {t} \left[ \left\langle p _ {V} ^ {t} - \bar {p} _ {V} ^ {t + 1}, \widehat {\ell} ^ {t} \right\rangle - D _ {V} ^ {t} \left(\bar {p} ^ {t + 1}, p ^ {t}\right) \right] \leq \mathcal {O} \left(\mathbb {E} ^ {t} \left[ \sum_ {i \in V} \frac {\left(p _ {i} ^ {t}\right) ^ {2} \left(\widehat {\ell} _ {i} ^ {t}\right) ^ {2}}{\gamma_ {i} ^ {t}} \right]\right) = \mathcal {O} \left(\sum_ {i \in V} \frac {p _ {i} ^ {t}}{\gamma_ {i} ^ {t}}\right),
$$
where the first step follows from the facts that the Hessian  $\nabla^2\phi_V^t (p^t)$  is a diagonal matrix and its diagonal element for arm  $i\in V$  is  $\frac{C_{\mathrm{log}} + \gamma_i^t}{(p_i^t)^2}$ .
On the other hand, we bound the penalty term  $\phi_V^t (\bar{p}^{t + 1}) - \phi_V^{t + 1}(\bar{p}_V^{t + 1})$  as
$$
\begin{array}{l} \phi_ {V} ^ {t} (\bar {p} ^ {t + 1}) - \phi_ {V} ^ {t + 1} (\bar {p} ^ {t + 1}) = \sum_ {i \in V} \gamma_ {i} ^ {t} \log \left(\frac {1}{\bar {p} _ {i} ^ {t + 1}}\right) - \sum_ {i \in V} \gamma_ {i} ^ {t + 1} \log \left(\frac {1}{\bar {p} _ {i} ^ {t + 1}}\right) \\ = \sum_ {i \in V} \left(\gamma_ {i} ^ {t} - \gamma_ {i} ^ {t + 1}\right) \log \left(\frac {1}{\bar {p} _ {i} ^ {t + 1}}\right) \\ \leq 0, \\ \end{array}
$$
where the third step follows from the fact that  $\gamma_i^t\leq \gamma_i^{t + 1}$  for any arm  $i$ . Therefore, we have REGSUB bounded as
$$
\begin{array}{l} \mathbf {R E G S U B} = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle p _ {V} ^ {t} - \bar {p} _ {V} ^ {t + 1}, \widehat {\ell} ^ {t} \right\rangle - D _ {V} ^ {t} (\bar {p} ^ {t + 1}, p ^ {t}) + \phi_ {V} ^ {t} (\bar {p} ^ {t + 1}) - \phi_ {V} ^ {t + 1} (\bar {p} ^ {t + 1}) \right] \\ \leq \mathcal {O} \left(\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {i \in V} \frac {p _ {i} ^ {t}}{\gamma_ {i} ^ {t}} \right]\right) = \mathcal {O} \left(\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {i \in V} \frac {p _ {i} ^ {t}}{\gamma_ {i} ^ {t + 1}} \right]\right), \\ \end{array}
$$
where the last step follows from the multiplicative relation  $p_i^{t + 1} \leq 2p_i^t$  for any arm  $i$ .
By our learning rate schedule, we further have
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {i \in V} \frac {p _ {i} ^ {t}}{\gamma_ {i} ^ {t + 1}} \right] \leq \mathbb {E} \left[ \sqrt {\log T} \sum_ {t = 1} ^ {T} \sum_ {i \in V} \frac {p _ {i} ^ {t}}{\sqrt {1 + \sum_ {s = 1} ^ {t} p _ {i} ^ {s}}} \right] \\ \leq \mathbb {E} \left[ \sqrt {\log T} \sum_ {i \in V} \sum_ {t = 1} ^ {T} \int_ {\sum_ {s = 1} ^ {t - 1} p _ {i} ^ {s}} ^ {\sum_ {s = 1} ^ {t} p _ {i} ^ {s}} \frac {d u}{\sqrt {1 + u}} \right] \\ \end{array}
$$
$$
\begin{array}{l} \leq 2 \mathbb {E} \left[ \sqrt {\log T} \sum_ {i \in V} \sqrt {1 + \sum_ {t = 1} ^ {T} p _ {i} ^ {t}} \right] \\ = \mathcal {O} \left(\mathbb {S} _ {2} (\log T)\right), \\ \end{array}
$$
where the first step follows from the fact  $p_i^t \leq \max \{p_i^t, 1 / T\}$ ; the third step uses the Newton-Leibniz formula  $\int_{a}^{b} \frac{du}{\sqrt{1 + u}} = 2\sqrt{1 + u}\big|_a^b$ .
Lemma A.2.3 (REGSUB for  $\beta$ -Tsallis Entropy). For any  $\beta \in (0,1)$ , when using the  $\beta$ -Tsallis entropy regularizer with  $C_{\mathrm{LOG}} = \frac{162\beta}{1 - \beta}$ ,  $\alpha = \beta$ , and  $\theta = \sqrt{\frac{1 - \beta}{\beta}}$ , Algorithm 1 ensures
$$
\operatorname {R E G S U B} = \mathcal {O} \left(\mathbb {S} _ {2} \left(\frac {\log T}{\beta (1 - \beta)}\right)\right).
$$
Proof. First, the multiplicative relation between  $p^t, \bar{p}^{t+1}$ , and  $p^{t+1}$  in Eq. (22) is ensured by Lemma C.3.2. According to Lemma A.2.1, we have
$$
\mathbb {E} ^ {t} \left[ \left\langle p _ {V} ^ {t} - \bar {p} _ {V} ^ {t + 1}, \hat {\ell} ^ {t} \right\rangle - D _ {V} ^ {t} \left(\bar {p} ^ {t + 1}, p ^ {t}\right) \right] \leq \mathcal {O} \left(\mathbb {E} \left[ \sum_ {i \in V} \frac {\left(p _ {i} ^ {t}\right) ^ {2 - \beta} \left(\hat {\ell} _ {i} ^ {t}\right) ^ {2}}{\beta \gamma_ {i} ^ {t}} \right]\right) = \mathcal {O} \left(\sum_ {i \in V} \frac {\left(p _ {i} ^ {t}\right) ^ {1 - \beta}}{\beta \gamma_ {i} ^ {t}}\right),
$$
since the diagonal of  $\nabla^2\phi_V^t (p^t)$  is  $\frac{\beta\gamma_i^t}{(p_i^t)^{2 - \beta}} +\frac{C_{\mathrm{log}}}{(p_i^t)^2}$  for any arm  $i\in V$
Therefore, we have REGSUB bounded as
$$
\begin{array}{l} \operatorname {R E G S U B} = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \mathbb {E} ^ {t} \left[ \frac {\left(p _ {i} ^ {t}\right) ^ {1 - \beta}}{\beta \gamma_ {i} ^ {t}} + \phi_ {V} ^ {t} (\bar {p} ^ {t + 1}) - \phi_ {V} ^ {t + 1} (\bar {p} ^ {t + 1}) \right] \right] \\ \leq \mathcal {O} \left(\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {i \in V} \left(\frac {\left(p _ {i} ^ {t}\right) ^ {1 - \beta}}{\beta \gamma_ {i} ^ {t}} + \frac {\left(\gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t}\right) \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {\beta}}{1 - \beta}\right) \right]\right) \\ \leq \mathcal {O} \left(\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {i \in V} \left(\frac {\left(p _ {i} ^ {t}\right) ^ {1 - \beta}}{\beta \gamma_ {i} ^ {t}} + \frac {\left(\gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t}\right) \left(p _ {i} ^ {t}\right) ^ {\beta}}{1 - \beta}\right) \right]\right), \\ \end{array}
$$
where the last step follows from the multiplicative relation between  $p^t$  and  $\bar{p}^{t + 1}$  in Eq. (22).
Let  $p_i^0 = \frac{1}{K}$  for any arm  $i$  for notational convenience. For the stability term, we have
$$
\sum_ {i \in V} \frac {\left(p _ {i} ^ {t}\right) ^ {1 - \beta}}{\beta \gamma_ {i} ^ {t}} \leq \sum_ {i \in V} \frac {\left(2 p _ {i} ^ {t - 1}\right) ^ {1 - \beta}}{\beta \gamma_ {i} ^ {t}} \leq 2 \sum_ {i \in V} \frac {\max  \left\{p _ {i} ^ {t - 1} , 1 / T \right\} ^ {1 - \beta}}{\beta \gamma_ {i} ^ {t}}
$$
where the first step follows from the multiplicative relation between  $p_i^{t + 1}$  and  $p_i^t$ . On the other hand, we can bound the penalty term as:
$$
\begin{array}{l} \sum_ {i \in V} \frac {\left(\gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t}\right) (p _ {i} ^ {t}) ^ {\beta}}{1 - \beta} \leq \sum_ {i \in V} \frac {(p _ {i} ^ {t}) ^ {\beta}}{1 - \beta} \cdot \frac {1 - \beta}{\beta} \cdot \frac {\left(\max  \left\{p _ {i} ^ {t} , 1 / T \right\}\right) ^ {1 - 2 \beta}}{\gamma_ {i} ^ {t + 1}} \\ = \sum_ {i \in V} \frac {\left(p _ {i} ^ {t}\right) ^ {\beta} \left(\max  \left\{p _ {i} ^ {t} , 1 / T \right\}\right) ^ {1 - 2 \beta}}{\beta \gamma_ {i} ^ {t + 1}} \leq \sum_ {i \in V} \frac {\left(\max  \left\{p _ {i} ^ {t} , 1 / T \right\}\right) ^ {1 - \beta}}{\beta \gamma_ {i} ^ {t + 1}}, \\ \end{array}
$$
where the first step uses Eq. (21). This shows that the stability term and the penalty term are of the same order, and the rest of the proof boils down to the following final calculation:
$$
\sum_ {t = 1} ^ {T} \sum_ {i \in V} \frac {\left(\max \left\{p _ {i} ^ {t} , 1 / T \right\}\right) ^ {1 - \beta}}{\beta \gamma_ {i} ^ {t + 1}} \leq \frac {1}{\beta} \cdot \sqrt {\frac {\beta}{1 - \beta}} \sum_ {t = 1} ^ {T} \sum_ {i \in V} \frac {\left(\max \left\{p _ {i} ^ {t} , 1 / T \right\}\right) ^ {1 - \beta}}{\sqrt {1 + \sum_ {k = 1} ^ {t} \left(\max \left\{p _ {i} ^ {k} , 1 / T \right\}\right) ^ {1 - 2 \beta}}}
$$
$$
\begin{array}{l} = \mathcal {O} \left(\sqrt {\frac {1}{\beta (1 - \beta)}} \sum_ {i \in V} \sqrt {\log T \sum_ {t = 1} ^ {T} \max  \left\{p _ {i} ^ {t} , ^ {1} / T \right\}}\right) \\ = \mathcal {O} \left(\sqrt {\frac {\log T}{\beta (1 - \beta)}} \sum_ {i \in V} \sqrt {\sum_ {t = 1} ^ {T} p _ {i} ^ {t}}\right), \\ \end{array}
$$
where the first step follows from the definition of  $\gamma_i^t$ ; the second step applies Lemma 4.2; the last step follows from the fact that  $\max \{p_i^t, \frac{1}{T}\} \leq p_i^t + \frac{1}{T}$ .
Lemma A.2.4 (REGSUB for Shannon Entropy). When using the Shannon entropy regularizer with  $C_{\mathrm{LOG}} = 162 \log K$ ,  $\alpha = 1$ , and  $\theta = \sqrt{1 / \log T}$ , Algorithm 1 ensures REGSUB =  $\mathcal{O}\left(\mathbb{S}_2\left(\log^2 T\right)\right)$ .
Proof. Similarly, we apply Lemma A.2.1 and have
$$
\mathbb {E} ^ {t} \left[ \left\langle p _ {V} ^ {t} - \bar {p} _ {V} ^ {t + 1}, \widehat {\ell} ^ {t} \right\rangle - D _ {V} ^ {t} \left(\bar {p} ^ {t + 1}, p ^ {t}\right) \right] \leq \mathcal {O} \left(\mathbb {E} ^ {t} \left[ \sum_ {i \in V} \frac {p _ {i} ^ {t} \left(\widehat {\ell} _ {i} ^ {t}\right) ^ {2}}{\gamma_ {i} ^ {t}} \right]\right) = \mathcal {O} \left(\sum_ {i \in V} \frac {1}{\gamma_ {i} ^ {t}}\right),
$$
since the diagonal of  $\nabla^2\phi_V^t (p^t)$  is  $\frac{\gamma_i^t}{p_i^t} +\frac{C_{\mathrm{log}}}{(p_i^t)^2}$  for arm  $i$
For the penalty term, we have  $\phi_V^t (\bar{p}^{t + 1}) - \phi_V^{t + 1}(\bar{p}_V^{t + 1})$  bounded by
$$
\begin{array}{l} \sum_ {i \in V} \left(\gamma_ {i} ^ {t} - \gamma_ {i} ^ {t + 1}\right) \bar {p} _ {i} ^ {t + 1} \log \left(\frac {\bar {p} _ {i} ^ {t + 1}}{e}\right) \\ = \sum_ {i \in V} \left(\gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t}\right) \bar {p} _ {i} ^ {t + 1} \log \left(\frac {e}{\bar {p} _ {i} ^ {t + 1}}\right) \\ \leq \sum_ {i \in V} \left(\gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t}\right) \max  \left\{\bar {p} _ {i} ^ {t + 1}, ^ {1} / T \right\} \log \left(\frac {e}{\max  \left\{\bar {p} _ {i} ^ {t + 1} , ^ {1} / T \right\}}\right) \\ \leq \sum_ {i \in V} \frac {\left(\max  \left\{p _ {i} ^ {t} , ^ {1} / T \right\}\right) ^ {- 1} \max  \left\{\bar {p} ^ {t + 1} , ^ {1} / T \right\}}{(\log T) \gamma_ {i} ^ {t + 1}} \log \left(\frac {e}{\max  \left\{\bar {p} _ {i} ^ {t + 1} , ^ {1} / T \right\}}\right) \\ \leq \mathcal {O} \left(\sum_ {i \in V} \frac {1}{\gamma_ {i} ^ {t + 1}}\right), \\ \end{array}
$$
where the second step uses the fact that  $x \log \left( \frac{e}{x} \right)$  is monotonically increasing for  $x \in [0,1 / T]$ ; the third step follows from Eq. (21); the last step follows from the multiplicative relation between  $p^t$  and  $\bar{p}^{t + 1}$  so that  $\max \{ p_i^t, 1 / T \} \geq \frac{1}{2} \max \{ \bar{p}_i^{t + 1}, 1 / T \}$ .
$$
\begin{array}{l} \operatorname {R E G S U B} = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \mathbb {E} ^ {t} \left[ \left\langle p _ {V} ^ {t} - \bar {p} _ {V} ^ {t + 1}, \widehat {\ell} ^ {t} \right\rangle - D _ {V} ^ {t} (\bar {p} ^ {t + 1}, p ^ {t}) + \phi_ {V} ^ {t} (\bar {p} ^ {t + 1}) - \phi_ {V} ^ {t + 1} (\bar {p} ^ {t + 1}) \right] \right] \\ \leq \mathcal {O} \left(\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {i \in V} \frac {1}{\gamma_ {i} ^ {t}} + \frac {1}{\gamma_ {i} ^ {t + 1}} \right]\right) = \mathcal {O} \left(\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {i \in V} \frac {1}{\gamma_ {i} ^ {t + 1}} \right]\right), \\ \end{array}
$$
by combining the bounds for the stability and penalty terms.
According to our learning rate schedule, we have
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {i \in V} \frac {1}{\gamma_ {i} ^ {t + 1}} \right] = \mathbb {E} \left[ \sqrt {\log T} \sum_ {i \in V} \sum_ {t = 1} ^ {T} \frac {1}{\sqrt {1 + \sum_ {k = 1} ^ {t} \max \left\{p _ {i} ^ {k} , ^ {1} / T \right\} ^ {- 1}}} \right]
$$
$$
\begin{array}{l} \leq \mathcal {O} \left(\mathbb {E} \left[ \sqrt {\log T} \sum_ {i \in V} \sqrt {\log \left(1 + \sum_ {t = 1} ^ {T} \frac {1}{\max  \left\{p _ {i} ^ {t} , ^ {1} / T \right\}}\right) \sum_ {t = 1} ^ {T} \max  \left\{p _ {i} ^ {t} , ^ {1} / T \right\}} \right]\right) \\ \leq \mathcal {O} \left(\mathbb {E} \left[ \sqrt {\log T} \sum_ {i \in V} \sqrt {\log (1 + T \cdot T) \left(1 + \sum_ {t = 1} ^ {T} p _ {i} ^ {t}\right)} \right]\right) \\ = \mathcal {O} \left(\mathbb {S} _ {2} \left(\log^ {2} T\right)\right), \\ \end{array}
$$
where the second step applies Lemma 4.2; the third step uses the fact that  $\max \{p_i^t, 1 / T\} \geq 1 / T$ .
A.3 Regret on Optimal Arms
We show that  $\mathrm{REGOPT} \leq 0$  holds for all regularizers used in this paper with a specific technique.
Lemma A.3.1 (REGOPT for All Regularizers). For any of the three regularizers, Algorithm 1 ensures  $\mathrm{REGOPT} \leq 0$ .
Proof. According to the KKT conditions of the optimization problem for  $q^t$ , we have
$$
\nabla \phi_ {U} ^ {t + 1} (q ^ {t + 1}) = \nabla \phi_ {U} ^ {t} (q ^ {t}) - \widehat {\ell} _ {U} ^ {t} + \boldsymbol {\xi} ^ {\prime} \cdot \mathbf {1} _ {U},
$$
where  $\xi' \in \mathbb{R}$  is a Lagrange multiplier for  $q^{t+1}$ . Let  $z \in \mathbb{R}_{>0}^{K}$  be a vector that satisfies
$$
\nabla \phi_ {U} ^ {t + 1} (z) = \nabla \phi_ {U} ^ {t} (p ^ {t}) - \widehat {\ell} _ {U} ^ {t} + \boldsymbol {\xi} ^ {\prime} \cdot \mathbf {1} _ {U}.
$$
$$
\nabla \phi_ {U} ^ {t + 1} (z) - \nabla \phi_ {U} ^ {t} \left(p ^ {t}\right) = \nabla \phi_ {U} ^ {t + 1} \left(q ^ {t + 1}\right) - \nabla \phi_ {U} ^ {t} \left(q ^ {t}\right) = - \widehat {\ell} _ {U} ^ {t} + \xi^ {\prime} \cdot \mathbf {1} _ {U}. \tag {26}
$$
Also, by [Ito, 2021, Lemma 18], we have
$$
D _ {U} ^ {t, t + 1} \left(p ^ {t}, \bar {p} ^ {t + 1}\right) = D _ {U} ^ {t, t + 1} \left(p ^ {t}, z\right) - D _ {U} ^ {t + 1} \left(\bar {p} ^ {t + 1}, z\right) \leq D _ {U} ^ {t, t + 1} \left(p ^ {t}, z\right),
$$
where the last step uses the fact that the Bregman divergence is non-negative. Hence, to show  $D_U^{t,t+1}(p^t, \bar{p}^{t+1}) \leq D_U^{t,t+1}(q^t, q^{t+1})$ , it suffices to show  $D_U^{t,t+1}(p^t, z) \leq D_U^{t,t+1}(q^t, q^{t+1})$ , which is done below by applying  $x = p_i^t$ ,  $y = z$ ,  $m = q_i^t$ ,  $n = q_i^{t+1}$ , and  $\xi = -\widehat{\ell}_i^t + \xi'$  in Theorem 4.3.
First, we show that  $p_i^t \leq q_i^t$  for any arm  $i \in U$ . By the KKT conditions for  $q^t$  and  $p^t$ , we know that  $\nabla \phi_U^t(q^t) = \nabla \phi_U^t(p_U^t) + \eta \cdot \mathbf{1}_U$  for a Lagrange multiplier  $\eta \in \mathbb{R}$ . Due to the facts that  $\sum_{i \in U} q_i^t = 1 \geq \sum_{i \in U} p_i^t$  and  $\nabla \phi^t(\cdot)$  is monotone-increasing,  $\eta$  is non-negative and  $q_i^t \geq p_i^t$  holds for any arm  $i \in U$ .
It is clear that all our regularizers can be written in the form of  $\phi^t (p) = \sum_{i\in [K]}f^t (p_i)$ , and we will apply Theorem 4.3 with such  $f^{t}:[0,1]\to \mathbb{R}$ . It suffices to check the two required conditions on  $f^{t}$ . First, all regularizers used in Algorithm 1 satisfy condition (i):  $(f^t)'(z)$  is differentiable and concave. Second, as mentioned in Section 4, we only need to show that the learning rate  $\gamma_i^t$  is non-decreasing in  $t$  (which is true in our case) and the regularizer itself is non-increasing, the latter of which can be verified for all our regularizers as long as we have  $p^t,z,q^t,q^{t + 1}\in (0,1]^K$ . As  $z\in \mathbb{R}_{>0}^{K}$ , we still need to check  $z_{i}\leq 1$ . Indeed, from Eq. (26), we have  $\nabla \phi^{t + 1}(z)\leq \nabla \phi^{t + 1}(q^{t + 1})$  (entry-wise inequality), and since  $q_{i}^{t + 1}\leq 1$  for  $\forall i\in U$  and  $\nabla \phi^{t + 1}(\cdot)$  is monotone-increasing entry-wise, we have  $z_{i}\leq 1$ . To sum up, Theorem 4.3 indeed implies
$$
D _ {U} ^ {t, t + 1} \left(p ^ {t}, z\right) \leq D _ {U} ^ {t, t + 1} \left(q ^ {t}, q ^ {t + 1}\right).
$$
which further implies  $D_U^{t,t+1}(p^t, \bar{p}^{t+1}) \leq D_U^{t,t+1}(q^t, q^{t+1})$ . Taking the summation over all the rounds concludes the proof.
Theorem A.3.2 (Restatement of Theorem 4.3). For any  $t \in \mathbb{N}$ , let  $f^t : \Omega \to \mathbb{R}$  be a continuously-differentiable and strictly-convex function defined on  $\Omega \subseteq \mathbb{R}$ . Suppose that the following two conditions hold for all  $z \in \Omega$ : (i)  $(f^t)'(z)$  is differentiable and concave; (ii)  $(f^{t+1})'(z) \leq (f^t)'(z)$ . Then, for any  $x, m \in \mathbb{R}$  with  $x \leq m$ , and  $y, n \in \mathbb{R}$  such that  $(f^{t+1})'(y) - (f^t)'(x) = (f^{t+1})'(n) - (f^t)'(m) = \xi$  for a fixed scalar  $\xi$ , we have  $D^{t,t+1}(x,y) \leq D^{t,t+1}(m,n)$ , where  $D^{t,t+1}(u,v) = f^t(u) - f^{t+1}(v) - (u - v) \cdot (f^{t+1})'(v)$  is the skewed Bregman divergence.
Proof of Theorem 4.3. For shorthand, we denote  $F^t(z) = (f^t)'(z)$  and use  $(F^t)'(z)$  to denote the first-order derivative of  $F^t$  with respect to  $z$ . Taking the derivative with respect to  $x$  on both sides of  $(f^{t+1})'(y) - (f^t)'(x) = \xi$ , we have
$$
\left(F ^ {t}\right) ^ {\prime} (x) = \left(F ^ {t + 1}\right) ^ {\prime} (y) \frac {d y}{d x}. \tag {27}
$$
We now see  $y$  as a function of  $x$  and  $D^{t,t + 1}(x,y) = f^t (x) - f^{t + 1}(y) - (x - y)F^{t + 1}(y)$  as a function of  $x$  as well. Taking the derivative with respect to  $x$ , we have
$$
\begin{array}{l} (D ^ {t, t + 1}) ^ {\prime} (x, y) \\ = F ^ {t} (x) - F ^ {t + 1} (y) \frac {d y}{d x} - F ^ {t + 1} (y) - x \left(F ^ {t + 1}\right) ^ {\prime} (y) \frac {d y}{d x} + F ^ {t + 1} (y) \frac {d y}{d x} + y \left(F ^ {t + 1}\right) ^ {\prime} (y) \frac {d y}{d x} \\ = F ^ {t} (x) - F ^ {t + 1} (y) - x \left(F ^ {t + 1}\right) ^ {\prime} (y) \frac {d y}{d x} + y \left(F ^ {t + 1}\right) ^ {\prime} (y) \frac {d y}{d x} \\ \geq F ^ {t} (x) - F ^ {t} (y) - x \left(F ^ {t + 1}\right) ^ {\prime} (y) \frac {d y}{d x} + y \left(F ^ {t + 1}\right) ^ {\prime} (y) \frac {d y}{d x} \quad \left(F ^ {t + 1} (y) \leq F ^ {t} (y)\right) \\ = F ^ {t} (x) - F ^ {t} (y) + (y - x) \left(F ^ {t}\right) ^ {\prime} (x) \tag {byEq.(27)} \\ \geq 0. \quad (\text {c o n c a v i t y} F ^ {t} (\cdot)) \\ \end{array}
$$
The implies  $D^{t,t + 1}(x,y)\leq D^{t,t + 1}(m,n)$  given the condition  $x\leq m$
A.4 Residual Regret
Throughout this section, we write all our regularizers in the form  $\phi^t (x) = -C_{\mathrm{Log}}\sum_{i\in [K]}\log x_i + \sum_{i\in [K]}\gamma_i^t\psi (x_i)$  for a proper choice of  $\psi (\cdot)$  (e.g.,  $\psi (x_i) = x_i\log \left(\frac{x_i}{e}\right)$  for the Shannon entropy regularizer). Also, we will present the Hessian  $\nabla^2\phi^t (p^t)$  as a diagonal matrix whose diagonal element for arm  $i$  is denoted as  $w_{i}^{t}$ , that is,  $\nabla^2\phi^t (p^t) = \mathrm{diag}\{w_i^t: i\in [K]\}$ . Clearly, it always holds that  $w_{i}^{t} = \frac{C_{\mathrm{log}}}{(p_{i}^{t})^{2}} +\gamma_{i}^{t}\psi^{\prime \prime}(p_{i}^{t})$ .
With these notations, we are now ready to introduce our decomposition of  $\mathbb{E}^t\left[D^{t + 1}(\bar{p}^{t + 1},p^{t + 1})\right]$  into the term we discussed in Section 4 plus other terms.
Lemma A.4.1. Under Eq. (22), Algorithm 1 ensures that  $\mathbb{E}^t\left[D^{t + 1}(\bar{p}^{t + 1},p^{t + 1})\right]$  is bounded by
$$
\mathcal {O} \left(\frac {\left(\sum_ {i \in V} \frac {1}{w _ {i} ^ {t}}\right) \left(\sum_ {i \in U} \frac {1}{\left(w _ {i} ^ {t}\right) ^ {2} p _ {i} ^ {t}}\right)}{\left(\sum_ {i \in [ K ]} \frac {1}{w _ {i} ^ {t}}\right) \left(\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}\right)} + \sum_ {i \in V} \frac {1}{p _ {i} ^ {t} w _ {i} ^ {t}} + K \sum_ {i \in [ K ]} \frac {\left(\gamma_ {i} ^ {t} - \gamma_ {i} ^ {t + 1}\right) ^ {2} \left(\psi^ {\prime} (p _ {i} ^ {t})\right) ^ {2}}{w _ {i} ^ {t}}\right).
$$
Proof. By Lemma C.5.2, for any  $c \in \mathbb{R}$ , we have  $D^{t+1}(\bar{p}^{t+1}, p^{t+1})$  bounded as
$$
\begin{array}{l} \left\langle \bar {p} ^ {t + 1} - p ^ {t + 1}, \nabla \phi^ {t + 1} (\bar {p} ^ {t + 1}) - \nabla \phi^ {t + 1} (p ^ {t + 1}) \right\rangle - D ^ {t + 1} (p ^ {t + 1}, \bar {p} ^ {t + 1}) \\ = \left\langle \bar {p} ^ {t + 1} - p ^ {t + 1}, \nabla \phi^ {t + 1} \left(\bar {p} ^ {t + 1}\right) - \nabla \phi^ {t + 1} \left(p ^ {t + 1}\right) - c \cdot \mathbf {1} _ {K} \right\rangle - D ^ {t + 1} \left(p ^ {t + 1}, \bar {p} ^ {t + 1}\right) \\ = 2 \left\| \nabla \phi^ {t + 1} \left(\bar {p} ^ {t + 1}\right) - \nabla \phi^ {t + 1} \left(p ^ {t + 1}\right) - c \cdot \mathbf {1} _ {K} \right\| _ {\nabla^ {- 2} \phi^ {t + 1} (\xi)} \\ = \mathcal {O} \left(\left\| \nabla \phi^ {t + 1} (\bar {p} ^ {t + 1}) - \nabla \phi^ {t + 1} (p ^ {t + 1}) - c \cdot \mathbf {1} _ {K} \right\| _ {\nabla^ {- 2} \phi^ {t} (p ^ {t})}\right), \\ \end{array}
$$
where the first step follows from the fact that  $\bar{p}^{t + 1}, p^{t + 1} \in \Omega_K$ , thus  $\langle \bar{p}^{t + 1} - p^{t + 1}, c \cdot \mathbf{1}_K \rangle = 0$  holds for any  $c \in \mathbb{R}$ ; the second step follows from Lemma C.5.2, where  $\xi$  is some intermediate point between  $\bar{p}^{t + 1}$  and  $p^{t + 1}$ ; the third step follows from the multiplicative relation between  $\bar{p}^{t + 1}$  and  $p^{t + 1}$ , and the fact that  $\gamma_i^{t + 1} \geq \gamma_i^t$  for all  $t, i$ .
Then, we analyze the difference  $\nabla \phi^{t + 1}(\bar{p}^{t + 1}) - \nabla \phi^{t + 1}(p^{t + 1})$ . According to the KKT conditions of the optimization problems in the FTRL framework, we have
$$
\nabla \phi_ {U} ^ {t + 1} (\bar {p} ^ {t + 1}) = \nabla \phi_ {U} ^ {t} (p ^ {t}) - \widehat {\ell} _ {U} ^ {t} + \lambda_ {U} \cdot \mathbf {1} _ {U},
$$
$$
\nabla \phi_ {V} ^ {t + 1} (\bar {p} ^ {t + 1}) = \nabla \phi_ {V} ^ {t} (p ^ {t}) - \widehat {\ell} _ {V} ^ {t} + \lambda_ {V} \cdot \mathbf {1} _ {V},
$$
$$
\nabla \phi_ {U} ^ {t + 1} (p ^ {t + 1}) = \nabla \phi_ {U} ^ {t} (p ^ {t}) - \widehat {\ell} _ {U} ^ {t} + \lambda_ {K} \cdot \mathbf {1} _ {U},
$$
$$
\nabla \phi_ {V} ^ {t + 1} (p ^ {t + 1}) = \nabla \phi_ {V} ^ {t} (p ^ {t}) - \widehat {\ell} _ {V} ^ {t} + \lambda_ {K} \cdot \mathbf {1} _ {V},
$$
where  $\lambda_U, \lambda_V, \lambda_K$  are corresponding Lagrange multipliers for the  $\bar{p}^{t+1}$  and  $p^{t+1}$ .
To give a tighter bound of the Lagrange multipliers  $\lambda_U$  and  $\lambda_V$ , we greatly extend the approach of Ito [2021] in Lemma C.4.3. Together with the multiplicative relation between  $p^t, \bar{p}^{t+1}$  and  $p^{t+1}$ , we obtain  $\lambda_U$ 's upper and lower bounds: