SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To prove Lemma 4.2, we first consider a more general version that takes a weight sequence  $\{a_t\}_{t=1}^T$  into consideration. Note that, by simply setting  $a_t = 1$  for all  $t$ , one recovers Lemma 4.2.
Lemma C.2.1. Let  $\{x_{t}\}_{t = 1}^{T}$  and  $\{a_t\}_{t = 1}^T$  be some sequences with that  $x_{t},a_{t} > 0$  for all  $t$ . Then, for any  $\alpha \in [0,1]$ , we have
$$
\sum_ {t = 1} ^ {T} \frac {x _ {t} ^ {1 - \alpha} a _ {t}}{\sqrt {1 + \sum_ {s = 1} ^ {t} x _ {s} ^ {1 - 2 \alpha}}} \leq 2 \sqrt {\left(\sum_ {t = 1} ^ {T} x _ {t} a _ {t} ^ {2}\right) \log \left(1 + \sum_ {t = 1} ^ {T} x _ {t} ^ {1 - 2 \alpha}\right)}.
$$
Proof. For any  $\eta > 0$ , we have
$$
\sum_ {t = 1} ^ {T} \frac {x _ {t} ^ {1 - \alpha} a _ {t}}{\sqrt {1 + \sum_ {s = 1} ^ {t} x _ {s} ^ {1 - 2 \alpha}}} = \sum_ {t = 1} ^ {T} \frac {\left(a _ {t} x _ {t} ^ {1 / 2}\right) \cdot x _ {t} ^ {1 / 2 - \alpha}}{\sqrt {1 + \sum_ {s = 1} ^ {t} x _ {s} ^ {1 - 2 \alpha}}} \leq \sum_ {t = 1} ^ {T} \eta \cdot x _ {t} a _ {t} ^ {2} + \frac {1}{\eta} \cdot \frac {x _ {t} ^ {1 - 2 \alpha}}{1 + \sum_ {s = 1} ^ {t} x _ {s} ^ {1 - 2 \alpha}},
$$
where the second step uses the AM-GM inequality.
Note that, we have
$$
\sum_ {t = 1} ^ {T} \frac {x _ {t} ^ {1 - 2 \alpha}}{1 + \sum_ {s = 1} ^ {t} x _ {s} ^ {1 - 2 \alpha}} \leq \sum_ {t = 1} ^ {T} \int_ {1 + \sum_ {s = 1} ^ {t - 1} x _ {s} ^ {1 - 2 \alpha}} ^ {1 + \sum_ {s = 1} ^ {t} x _ {s} ^ {1 - 2 \alpha}} \frac {d u}{u} = \int_ {1} ^ {1 + \sum_ {s = 1} ^ {T} x _ {s} ^ {1 - 2 \alpha}} \frac {d u}{u} = \log \left(1 + \sum_ {s = 1} ^ {T} x _ {s} ^ {1 - 2 \alpha}\right),
$$
where the first step bounds the fraction with the integral of  $\frac{1}{u}$  from  $1 + \sum_{s=1}^{t-1} x_s^{1-2\alpha}$  to  $1 + \sum_{s=1}^{t} x_s^{1-2\alpha}$ , and the last step follows from the Newton-Leibniz formula.
Therefore, the following bound holds for any  $\eta > 0$
$$
\sum_ {t = 1} ^ {T} \frac {x _ {t} ^ {1 - \alpha} a _ {t}}{\sqrt {1 + \sum_ {s = 1} ^ {t} x _ {s} ^ {1 - 2 \alpha}}} \leq \eta \left(\sum_ {t = 1} ^ {T} x _ {t} a _ {t} ^ {2}\right) + \frac {1}{\eta} \log \left(1 + \sum_ {s = 1} ^ {T} x _ {s} ^ {1 - 2 \alpha}\right).
$$
Finally, picking the optimal  $\eta$  finishes the proof.
Remark C.2.2. For any  $\beta \in (0,1)$ , when using the  $\beta$ -Tsallis entropy regularizer with  $C_{\mathrm{LOG}} \geq \frac{162\beta}{1 - \beta}$ ,  $\alpha = \beta$ , and  $\theta = \sqrt{\frac{1 - \beta}{\beta}}$ , Algorithm 1 ensures (ignoring some lower-order terms)
$$
\mathrm {R e g} ^ {T} = \mathcal {O} \left(\sqrt {\frac {1}{\beta (1 - \beta)}} \sum_ {i \in [ K ]} \sum_ {t = 1} ^ {T} \frac {(p _ {i} ^ {t}) ^ {1 - \beta}}{\sqrt {1 + \sum_ {s \leq t} \max \{p _ {i} ^ {s} , ^ {1} / T \} ^ {1 - 2 \beta}}}\right),
$$
in the adversarial setting. Note that, when  $\beta = \frac{1}{2}$ , our learning rate schedule becomes  $\gamma_i^t = \sqrt{t}$  which is the same as that of the Tsallis-INF algorithm, and the regret bound above can be simplified as
$$
\mathrm {R e g} ^ {T} = \mathcal {O} \left(\sum_ {t = 1} ^ {T} \sum_ {i \in [ K ]} \sqrt {\frac {p _ {i} ^ {t}}{t + 1}}\right) \leq \mathcal {O} \left(\sum_ {t = 1} ^ {T} \sqrt {\frac {K}{t + 1}}\right) = \mathcal {O} \left(\sqrt {K T}\right),
$$
where the second step follows from the Cauchy-Schwarz inequality:  $\sum_{i\in [K]}\sqrt{p_i^t} \leq \sqrt{K\sum_{i\in[K]}p_i^t} = \sqrt{K}$ , and the last step uses the fact that  $\sum_{t=1}^{T}\frac{1}{\sqrt{t+1}} = \mathcal{O}\left(\sqrt{T}\right)$ . On the other hand, for the extreme case where  $\beta = 0$ , we have
$$
\begin{array}{l} \sum_ {i \in [ K ]} \sum_ {t = 1} ^ {T} \frac {p _ {i} ^ {t}}{\sqrt {1 + \sum_ {s \leq t} \max \left\{p _ {i} ^ {s} , ^ {1} / T \right\}}} \leq \sum_ {i \in [ K ]} \sum_ {t = 1} ^ {T} \int_ {\sum_ {s = 1} ^ {t - 1} p _ {i} ^ {s}} ^ {\sum_ {s = 1} ^ {t} p _ {i} ^ {s}} \frac {d u}{\sqrt {1 + u}} \\ \leq 2 \sum_ {i \in K} \sqrt {1 + \sum_ {t = 1} ^ {T} p _ {i} ^ {t}} = \mathcal {O} (\sqrt {K T}), \\ \end{array}
$$
where the first step bounds the fraction by the integral of  $\frac{1}{\sqrt{1 + u}}$  from  $\sum_{s=1}^{t-1} p_i^s$  to  $\sum_{s=1}^{t} p_i^s$ ; the second step follows from the Newton-Leibniz formula; the last step follows from the Cauchy-Schwarz inequality.
These two bounds for  $\beta = 0$  and  $\beta = 1/2$  inspire us to conjecture the following: for any  $\beta \in (0,1)$ , any sequence of distributions  $\{p^t\}_{t=1}^T$ , we have
$$
\sum_ {i \in [ K ]} \sum_ {t = 1} ^ {T} \frac {\left(p _ {i} ^ {t}\right) ^ {1 - \beta}}{\sqrt {1 + \sum_ {s \leq t} \max  \left\{p _ {i} ^ {s} , ^ {1 / T} \right\} ^ {1 - 2 \beta}}} = \mathcal {O} (\sqrt {K T}). \tag {72}
$$
Clearly, if this conjecture holds, then, the extra  $\sqrt{\log T}$  factor in the regret bounds for the adversarial setting can be removed when  $\beta \neq 1/2$ .
C.3 Multiplicative Relation
The added fixed amount log-barrier in the regularizer ensures multiplicative relation among  $\bar{p}^{t + 1}, p^t$ , and  $p^{t + 1}$ . To show this, we start from the following general lemma.
Lemma C.3.1. Consider any constant  $C_{\mathrm{LOG}} > 0$ , any learning rate vectors  $\gamma, \gamma' \in \mathbb{R}_{>0}^{d}$  satisfying  $\gamma_i' \geq \gamma_i$  for all  $i \in [d]$ , any loss vectors  $L, L' \in \mathbb{R}_{>0}^{d}$ , and any convex function  $\psi(x)$  which ensures  $\frac{\psi''(x)}{4} \leq \psi''(z) \leq 4\psi''(x)$  for any  $x \in \mathbb{R}_{>0}$  and any  $z \in \left[\frac{x}{2}, 2x\right]$ . Define  $\phi(p) = \sum_{i \in [d]} \gamma_i \psi(p_i)$  as the regularizer with learning rate  $\gamma$ ,  $\phi'(p) = \sum_{i \in [d]} \gamma_i' \psi(p_i)$  as the regularizer with learning rate  $\gamma'$ ,  $\phi_L(p) = -C_{\mathrm{LOG}} \sum_{i \in [d]} \log p_i$  as the log-barrier regularizer,  $F(p) = \langle p, L \rangle + \phi(p) + \phi_L(p)$ , and  $F'(p) = \langle p, L' \rangle + \phi'(p) + \phi_L(p)$ . Further define  $x, y \in \Omega \subseteq \mathbb{R}_{>0}^{d}$  as:  $x = \operatorname*{argmin}_{p \in \Omega} F(p)$  and  $y = \operatorname*{argmin}_{p \in \Omega} F'(p)$ . If  $C_{\mathrm{LOG}}$  satisfies:
$$
C _ {\mathrm {L O G}} \geq \max  \left\{9, 3 2 \sum_ {i \in [ d ]} \left(L _ {i} ^ {\prime} - L _ {i}\right) ^ {2} x _ {i} ^ {2}, 3 2 \sum_ {i \in [ d ]} \left(\gamma_ {i} ^ {\prime} - \gamma_ {i}\right) ^ {2} \psi^ {\prime} \left(x _ {i}\right) ^ {2} x _ {i} ^ {2} \right\}, \tag {73}
$$
then, for any  $i \in [d]$ , we have
$$
\frac {1}{2} x _ {i} \leq y _ {i} \leq 2 x _ {i}. \tag {74}
$$
Proof. For simplicity, we denote  $H$  as the Hessian  $\nabla^2 F(x)$ , and  $H_{L}$  as the Hessian  $\nabla^2\phi_L(x)$  which is a diagonal matrix with  $\frac{C_{\mathrm{Log}}}{x_i^2}$  on its diagonal for every entry  $i\in [d]$ . Our goal is to show that  $\| y - x\| _H\leq 1$ , which is enough to guarantee Eq. (74) because  $1\geq \| y - x\|_{H}\geq \| y - x\|_{H_{L}}$ , and
$$
\left\| y - x \right\| _ {H _ {L}} \leq 1 \Rightarrow C _ {\mathrm {L O G}} \sum_ {i \in [ d ]} \left(\frac {y _ {i} - x _ {i}}{x _ {i}}\right) ^ {2} \leq 1 \Rightarrow \left| \frac {y _ {i} - x _ {i}}{x _ {i}} \right| \leq \frac {1}{3}, \mathrm {f o r} \forall i \in [ d ],
$$
where the last step follows from the condition  $C_{\mathrm{LOG}} \geq 9$ .
To prove  $\| y - x \|_H \leq 1$ , it suffices to show that for any  $z \in \Omega$  that ensures  $\| z - x \|_H = 1$ , it holds that  $F'(z) \geq F'(x)$ . To see this, note that the level set  $A = \{p \in \Omega : F'(p) \leq F'(x)\}$  is a convex set that contains  $x$  and  $y$ . Clearly, any intermediate point between  $x$  and  $y$  belongs to  $A$ , and does not belong to the boundary set  $\partial B = \{p : \| p - x \|_H = 1\}$  where  $B = \{p : \| p - x \|_H < 1\}$  also contains  $x$ . Therefore,  $y$  belongs to the set  $B$  and guarantees that  $\| y - x \|_H \leq 1$ .
To this end, we first bound  $F'(z)$  for any  $z \in \Omega$  with  $\| z - x \|_H = 1$  as
$$
\begin{array}{l} F ^ {\prime} (z) = F ^ {\prime} (x) + \nabla F ^ {\prime} (x) ^ {\top} (z - x) + \frac {1}{2} \| z - x \| _ {\nabla^ {2} F ^ {\prime} (\xi)} ^ {2} \\ = F ^ {\prime} (x) + \left(\nabla F ^ {\prime} (x) - \nabla F (x)\right) ^ {\top} (z - x) + \nabla F (x) ^ {\top} (z - x) + \frac {1}{2} \| z - x \| _ {\nabla^ {2} F ^ {\prime} (\xi)} ^ {2} \\ \geq F ^ {\prime} (x) + \left(\nabla F ^ {\prime} (x) - \nabla F (x)\right) ^ {\top} (z - x) + \frac {1}{2} \| z - x \| _ {\nabla^ {2} F ^ {\prime} (\xi)} ^ {2} \\ \geq F ^ {\prime} (x) + \left(\nabla F ^ {\prime} (x) - \nabla F (x)\right) ^ {\top} (z - x) + \frac {1}{2} \| z - x \| _ {\nabla^ {2} F (\xi)} ^ {2} \\ \geq F ^ {\prime} (x) + \left(\nabla F ^ {\prime} (x) - \nabla F (x)\right) ^ {\top} (z - x) + \frac {1}{8} \| z - x \| _ {\nabla^ {2} F (x)} ^ {2} \\ = F ^ {\prime} (x) + \left(\nabla F ^ {\prime} (x) - \nabla F (x)\right) ^ {\top} (z - x) + \frac {1}{8} \| z - x \| _ {H} ^ {2} \\ = F ^ {\prime} (x) + \left(\nabla F ^ {\prime} (x) - \nabla F (x)\right) ^ {\top} (z - x) + \frac {1}{8}, \\ \end{array}
$$
where the first step follows the Taylor expansion of  $F'(z)$  at  $x$  with  $\xi$  being an intermediate point between  $x$  and  $z$ ; the third step holds due to the optimality condition of  $x$ , i.e.,  $\nabla F(x)^{\top}(z - x) \geq 0$ ; the forth step uses the fact that  $\nabla^2 F'(\xi) \succeq \nabla^2 F(\xi)$ ; the fifth step applies the multiplicative relation between  $\xi$  and  $x$ :  $\frac{1}{2} x_i \leq \xi_i \leq 2x_i$  for any  $i \in [d]$ , which indicates  $\psi''(\xi_i) \geq \frac{\psi''(x_i)}{4}$  according to the property of  $\psi$ ; the last step uses the condition that  $\| z - x \|_H = 1$ .
To finish the proof, we only need to show that  $(\nabla F'(x) - \nabla F(x))^{\top}(z - x) \geq -\frac{1}{8}$ . We bound  $(\nabla F'(x) - \nabla F(x))^{\top}(z - x)$  as
$$
\begin{array}{l} \left(\nabla F ^ {\prime} (x) - \nabla F (x)\right) ^ {\top} (z - x) \\ = \left(L ^ {\prime} - L + \nabla \phi^ {\prime} (x) - \nabla \phi (x)\right) ^ {\top} (z - x) \\ \geq - \left\| L ^ {\prime} - L + \nabla \phi^ {\prime} (x) - \nabla \phi (x) \right\| _ {H ^ {- 1}} \| z - x \| _ {H} \\ = - \| L ^ {\prime} - L + \nabla \phi^ {\prime} (x) - \nabla \phi (x) \| _ {H ^ {- 1}} ^ {2} \\ \geq - \left\| L ^ {\prime} - L + \nabla \phi^ {\prime} (x) - \nabla \phi (x) \right\| _ {H _ {L} ^ {- 1}} ^ {2} \\ = - \sum_ {i \in [ d ]} \left(L _ {i} ^ {\prime} - L _ {i} + \left(\gamma_ {i} ^ {\prime} - \gamma_ {i}\right) \psi^ {\prime} \left(x _ {i}\right)\right) ^ {2} \frac {x _ {i} ^ {2}}{C _ {\mathrm {L O G}}}, \tag {75} \\ \end{array}
$$
where the second step uses Hölder's inequality, and the forth step follows from the fact that  $H^{-1} \preceq H_{L}^{-1}$ , which ensures  $\| p \|_{H^{-1}} \leq \| p \|_{H_{L}^{-1}}$  for any  $p \in \mathbb{R}^d$ .
$$
\sum_ {i \in [ d ]} \left(L _ {i} ^ {\prime} - L _ {i} + (\gamma_ {i} ^ {\prime} - \gamma_ {i}) \psi^ {\prime} (x _ {i})\right) ^ {2} \frac {x _ {i} ^ {2}}{C _ {\mathrm {L O G}}}
$$
$$
\begin{array}{l} \leq \frac {2}{C _ {\mathrm {L O G}}} \sum_ {i \in [ d ]} \left(L _ {i} ^ {\prime} - L _ {i}\right) ^ {2} x _ {i} ^ {2} + \left(\gamma_ {i} ^ {\prime} - \gamma_ {i}\right) ^ {2} \psi^ {\prime} (x _ {i}) ^ {2} x _ {i} ^ {2} \\ \leq \frac {2}{3 2} + \frac {2}{3 2} = \frac {1}{8}, \\ \end{array}
$$
where the second step follows from the fact that  $(x + y)^2 \leq 2(x^2 + y^2)$  for any  $x, y$ ; the second step uses Eq. (73). Plugging this inequality back to Eq. (75) finishes the proof.
In what follows, we show three applications of Lemma C.3.1 for the  $\beta$ -Tsallis entropy (Lemma C.3.2), the Shannon entropy (Lemma C.3.3), and the log-barrier (Lemma C.3.4), respectively.
Lemma C.3.2 (Multiplicative Relation for  $\beta$ -Tsallis Entropy). For any  $\beta \in (0,1)$ , when using the  $\beta$ -Tsallis entropy regularizer with  $C_{\mathrm{LOG}} \geq \frac{162\beta}{1 - \beta}$ ,  $\alpha = \beta$ , and  $\theta = \sqrt{\frac{1 - \beta}{\beta}}$ , Algorithm 1 guarantees
$$
\frac {1}{2} p _ {i} ^ {t} \leq \bar {p} _ {i} ^ {t + 1} \leq 2 p _ {i} ^ {t}, \quad \frac {1}{2} p _ {i} ^ {t} \leq p _ {i} ^ {t + 1} \leq 2 p _ {i} ^ {t},
$$
for all  $t\in [T]$  and arm  $i\in [K]$
Proof. We first apply Lemma C.3.1 to show  $\frac{1}{2} p_i^t \leq p_i^{t+1} \leq 2p_i^t$ . In particular, we set  $\Omega$  as a probability simplex,  $L = \sum_{\tau < t} \ell_i^\tau$ ,  $L' = \sum_{\tau \leq t} \ell_i^\tau$ ,  $\gamma = \gamma^t$ , and  $\gamma' = \gamma^{t+1}$  where the loss estimators and learning rates are defined in Algorithm 1. These choices naturally give  $x = p^t$  and  $y = p^{t+1}$ . Then, we only need to check that choosing  $C_{\mathrm{LOG}} = \frac{162\beta}{1 - \beta}$  can guarantee that Eq. (73) holds. Clearly, we have  $C_{\mathrm{LOG}} \geq 9$ . From the definition of  $\widehat{\ell}_i^t$ , one can show
$$
3 2 \sum_ {i \in [ K ]} \left(\widehat {\ell} _ {i} ^ {t}\right) ^ {2} \left(p _ {i} ^ {t}\right) ^ {2} \leq 3 2 \sum_ {i \in [ K ]} \frac {\mathbb {I} \{i ^ {t} = i \}}{\left(p _ {i} ^ {t}\right) ^ {2}} \left(p _ {i} ^ {t}\right) ^ {2} = 3 2 \sum_ {i \in [ K ]} \mathbb {I} \{i ^ {t} = i \} = 3 2.
$$
Finally, by Eq. (21), we have
$$
\begin{array}{l} 3 2 \sum_ {i \in [ K ]} \left(\gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t}\right) ^ {2} \left(p _ {i} ^ {t}\right) ^ {2} \left(\psi^ {\prime} \left(p _ {i} ^ {t}\right)\right) ^ {2} \\ = 32 \sum_ {i \in [ K ]} \left(\gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t}\right) ^ {2} \left(p _ {i} ^ {t}\right) ^ {2} \left(\frac {\beta (p _ {i} ^ {t}) ^ {\beta - 1}}{1 - \beta}\right) ^ {2} \\ \leq 32 \sum_ {i \in [ K ]} \frac {\beta^ {2}}{(1 - \beta) ^ {2}} \cdot \frac {(1 - \beta)}{\beta} \cdot \frac {\left(\max  \left\{p _ {i} ^ {t} , 1 / T \right\}\right) ^ {2 - 4 \beta}}{1 + \sum_ {k = 1} ^ {t} \left(\max  \left\{p _ {i} ^ {k} , 1 / T \right\}\right) ^ {1 - 2 \beta}} \left(p _ {i} ^ {t}\right) ^ {2 \beta} \\ \leq 32 \sum_ {i \in [ K ]} \frac {\beta}{(1 - \beta)} \left(\max  \left\{p _ {i} ^ {t}, 1 / T \right\}\right) ^ {1 - 2 \beta} \left(\max  \left\{p _ {i} ^ {t}, 1 / T \right\}\right) ^ {2 \beta} \\ \leq 32 \sum_ {i \in [ K ]} \frac {\beta}{(1 - \beta)} \left(p _ {i} ^ {t} + \frac {1}{T}\right) \\ \leq 32 \cdot \frac {\beta}{1 - \beta} \left(\sum_ {i \in [ K ]} p _ {i} ^ {t} + \frac {K}{T}\right) \leq 6 4 \cdot \frac {\beta}{1 - \beta}, \\ \end{array}
$$
where the third step uses the inequality that  $\max \left\{p_i^t, 1 / T\right\} \leq p_i^t + \frac{1}{T}$ .
To prove  $\frac{1}{2} p_i^t \leq \bar{p}_i^{t+1} \leq 2p_i^t$  for all arms  $i \in [K]$ , we show  $\frac{1}{2} p_i^t \leq \bar{p}_i^{t+1} \leq 2p_i^t$  for all arms  $i \in U$  and all arms  $i \in V$ , respectively. Since the proof ideas of both are the same, we only show the one related to set  $U$ . We set  $\Omega = \{x \in \mathbb{R}_{\geq 0}^K : \sum_{i \in U} x_i = \sum_{i \in U} p_i^t, \sum_{i \in V} x_i = 0\}$  and maintain all settings the same as the above. Note that, Eq. (73) only depends on loss estimators and learning rate schedules, and thus, it holds for new decision space  $\Omega$ . Repeating a similar argument, we can obtain the multiplicative relation related to arms in  $V$ .
Lemma C.3.3 (Multiplicative Relation for Shannon Entropy). When using the Shannon entropy regularizer with  $C_{\mathrm{LOG}} \geq 162 \log K$ ,  $\alpha = 1$ , and  $\theta = \sqrt{\frac{1}{\log T}}$ , Algorithm 1 guarantees
Proof. We consider applying Lemma C.3.1 to prove the multiplicative relation between  $p^t$  and  $p^{t+1}$  with similar setups in the proof Lemma C.3.2. For the Shannon entropy regularizer in Algorithm 1, we have  $\psi(x) = x \log \left( \frac{x}{e} \right)$  and  $\psi'(x) = \log(x)$ .
By the same argument, we know that  $162\log K\geq 32\geq 32\sum_{i\in [K]}\left(\widehat{\ell}_i^t\right)^2 (p_i^t)^2$  and  $162\log K\geq 9$  Then, we only need to verify that  $162\log K\geq 32\sum_{i\in [K]}\left(\gamma_i^{t + 1} - \gamma_i^t\right)^2\psi '(p_i^t)^2 (p_i^t)^2$  as:
$$
\begin{array}{l} 3 2 \sum_ {i \in [ K ]} \left(\gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t}\right) ^ {2} \left(p _ {i} ^ {t} \log p _ {i} ^ {t}\right) ^ {2} \\ \leq 32 \sum_ {i \in [ K ]} \frac {1}{\log T} \frac {\left(\max  \left\{p _ {i} ^ {t} , ^ {1} / T \right\}\right) ^ {- 2}}{1 + \sum_ {k = 1} ^ {t} \left(\max  \left\{p _ {i} ^ {k} , ^ {1} / T \right\}\right) ^ {- 1}} \left(p _ {i} ^ {t} \log p _ {i} ^ {t}\right) ^ {2} \\ \leq 32 \sum_ {i \in [ K ]} \frac {1}{\log T} \left(\max  \left\{p _ {i} ^ {t}, 1 / T \right\}\right) ^ {- 1} \left(\max  \left\{p _ {i} ^ {t}, 1 / T \right\} \log \left(\max  \left\{p _ {i} ^ {t}, 1 / T \right\}\right)\right) ^ {2} \\ \leq 32 \sum_ {i \in [ K ]} \frac {1}{\log T} \max  \left\{p _ {i} ^ {t}, ^ {1} / T \right\} \left(\log \left(\max  \left\{p _ {i} ^ {t}, ^ {1} / T \right\}\right)\right) ^ {2} \\ \leq 32 \sum_ {i \in [ K ]} - \left(\max  \left\{p _ {i} ^ {t}, ^ {1} / T \right\}\right) \log \left(\max  \left\{p _ {i} ^ {t}, ^ {1} / T \right\}\right) \\ \leq 32 K \cdot \left(- \frac {\sum_ {i \in [ K ]} \max  \left\{p _ {i} ^ {t} , ^ {1} / T \right\}}{K} \log \left(\frac {\sum_ {i \in [ K ]} \max  \left\{p _ {i} ^ {t} , ^ {1} / T \right\}}{K}\right)\right) \\ \leq 6 4 \log K, \\ \end{array}
$$
where the first step follows from Eq. (21) that  $\gamma_i^{t + 1} - \gamma_i^t\leq \frac{1}{\log T}\frac{\max\left\{p_i^t,1 / T\right\}^{-1}}{\gamma_i^{t + 1}}$ ; the second step follows from the fact that  $0\geq p_i^t\log (p_i^t)\geq \max \left\{p_i^t,1 / T\right\} \log \left(\max \left\{p_i^t,1 / T\right\}\right)$  as  $x$  log  $x$  is monotonically decreasing in  $[0,1 / T]$ ; the forth step follows from the fact that  $0\leq -\log (\max \left\{p_i^t,1 / T\right\})\leq \log T$ ; the fifth step utilizes the concavity of  $-x\log x$ .
Finally, following the same steps in Lemma C.3.2 finishes the proof of the multiplicative relation between  $\bar{p}^{t + 1}$  and  $p^t$ .
Lemma C.3.4 (Multiplicative Relation for Log-barrier). When using the log-barrier regularizer with  $C_{\mathrm{LOG}} \geq 162$ ,  $\alpha = 0$  and  $\theta = \sqrt{\frac{1}{\log T}}$ , Algorithm 1 guarantees
Proof. Similarly, we only need to verify that  $162\log K \geq 32\sum_{i\in [K]}\left(\gamma_i^{t + 1} - \gamma_i^t\right)^2\psi '(p_i^t)^2 (p_i^t)^2$  as:
$$
\begin{array}{l} 3 2 \sum_ {i \in [ K ]} \left(\gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t}\right) ^ {2} \left(p _ {i} ^ {t}\right) ^ {2} \psi^ {\prime} (p _ {i} ^ {t}) ^ {2} \\ = 3 2 \sum_ {i \in [ K ]} \left(\gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t}\right) ^ {2} \left(p _ {i} ^ {t}\right) ^ {2} \left(\frac {1}{p _ {i} ^ {t}}\right) ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \leq 32 \sum_ {i \in [ K ]} \frac {1}{(\log T)} \frac {\left(\max  \left\{p _ {i} ^ {t} , ^ {1} / T \right\}\right)}{1 + \sum_ {k = 1} ^ {t} \left(\max  \left\{p _ {i} ^ {k} , ^ {1} / T \right\}\right)} \\ \leq \frac {3 2}{\log T} \sum_ {i \in [ K ]} \left(\max  \left\{p _ {i} ^ {t}, 1 / T \right\}\right) \\ \leq \frac {3 2}{\log T} \left(\sum_ {i \in [ K ]} p _ {i} ^ {t} + \frac {K}{T}\right) \leq 6 4, \\ \end{array}
$$
where the first step follows from the definition of the log-barrier regularizer that  $\psi'(p_i^t) = -\frac{1}{p_i^t}$ ; the second step applies Eq. (21).
Lemma C.3.5. With  $C_{\mathrm{LOG}} \geq 162$ , Algorithm 1 guarantees  $\frac{1}{2} p_i^t \leq z_i \leq 2p_i^t$  for any arm  $i \in [K]$  and round  $t \in [T]$  where  $z \in \mathbb{R}_{\geq 0}^K$  is defined as
$$
z = \operatorname *{argmin}_{\substack{x_{i} = 0,\forall i\in U\\ x_{i}\geq 0,\forall i\in V\\ \sum_{i\in V}x_{i} = \sum_{i\in V}p_{i}^{t}}} \left\langle \sum_{\tau \leq t}\widehat{\ell}_{V}^{\tau},x\right\rangle +\phi_{V}^{t}(x).
$$
Proof. Clearly,  $p_V^t$  is the solution of the following optimization problem:
$$
p_{V}^{t} = \operatorname *{argmin}_{\substack{x_{i} = 0,\forall i\in U\\ x_{i}\geq 0,\forall i\in V\\ \sum_{i\in V}x_{i} = \sum_{i\in V}p_{i}^{t}}}\left\langle \sum_{\tau <  t}\widehat{\ell}_{V}^{\tau},x\right\rangle +\phi_{V}^{t}(x),
$$
since  $p_V^t$  satisfies all the KKT conditions.
Therefore, we are able to apply Lemma C.3.1 by setting  $L = \sum_{\tau < t} \widehat{\ell}_V^\tau$ ,  $L' = \sum_{\tau \leq t} \widehat{\ell}_V^\tau$ ,  $\Omega$  being the corresponding simplex,  $\gamma = \gamma' = \gamma^t$  where the loss estimators and learning rates are defined in Algorithm 1. As  $\gamma' = \gamma$  and  $C_{\mathrm{LOG}} = 162 > 9$ , we only need to verify that  $C_{\mathrm{LOG}} \geq 32 \sum_{i \in V} \left( \widehat{\ell}_i^t \right)^2 (p_i^t)^2$ . By the definition of the importance weighted loss estimator  $\widehat{\ell}^t$ , we have
$$
3 2 \sum_ {i \in V} \left(\widehat {\ell} _ {i} ^ {t}\right) ^ {2} \left(p _ {i} ^ {t}\right) ^ {2} \leq 3 2 \sum_ {i \in V} \mathbb {I} \left\{i ^ {t} = i \right\} \leq 3 2 \leq 1 6 2,
$$
C.4 Generalization of [Ito, 2021, Lemma 21]
In this section, we greatly generalize [Ito, 2021, Lemma 21] which is critical to analyze the residual regret. The approach of Ito [2021] requires a closed form solution of the optimization problem from the FTRL framework, which is not always guaranteed. Our approach removes this constraint, and thus, it can be applied to  $\beta$ -Tsallis entropy regularizers when  $\beta \neq 1/2$ , and even more complicated regularizers such as those hybrid ones.
Lemma C.4.1. Given  $\Omega \subseteq \mathbb{R}$  and twice-differentiable functions  $f, g: \Omega \to \mathbb{R}$ , if their corresponding first-order derivatives  $f', g'$  are invertible and the inverse functions  $(f')^{-1}, (g')^{-1}$  are differentiable and convex, then, for any  $p, q \in \Omega$  and  $z \in \mathbb{R}$  satisfying  $g'(q) = f'(p) - z$ , we have
$$
q \geq p + \frac {1}{g ^ {\prime \prime} (p)} \cdot \left(f ^ {\prime} (p) - g ^ {\prime} (p) - z\right), \tag {76}
$$
$$
q \leq p + \frac {1}{g ^ {\prime \prime} (q)} \cdot \left(f ^ {\prime} (p) - g ^ {\prime} (p) - z\right), \tag {77}
$$
$$
q \geq p + \frac {1}{f ^ {\prime \prime} (p)} \cdot \left(f ^ {\prime} (q) - g ^ {\prime} (q) - z\right), \tag {78}
$$
$$
q \leq p + \frac {1}{f ^ {\prime \prime} (q)} \cdot \left(f ^ {\prime} (q) - g ^ {\prime} (q) - z\right). \tag {79}
$$
Proof. We first show Eq. (76):
$$
\begin{array}{l} q = \left(g ^ {\prime}\right) ^ {- 1} \left(f ^ {\prime} (p) - z\right) \\ = \left(g ^ {\prime}\right) ^ {- 1} \left(g ^ {\prime} (p) + \left(f ^ {\prime} (p) - g ^ {\prime} (p)\right) - z\right) \\ \geq p + \left[ \left(\left(g ^ {\prime}\right) ^ {- 1}\right) ^ {\prime} (x) \right] \Bigg | _ {x = g ^ {\prime} (p)} \cdot \left(f ^ {\prime} (p) - g ^ {\prime} (p) - z\right) \\ = p + \frac {1}{g ^ {\prime \prime} (p)} \cdot \left(f ^ {\prime} (p) - g ^ {\prime} (p) - z\right), \\ \end{array}
$$
where the third step follows from the convexity of  $(g')^{-1}$ , which ensures  $(g')^{-1}(u) \geq (g')^{-1}(v) + ((g')^{-1})'(v) \cdot (u - v)$  for any  $u, v$  in the domain of  $(g')^{-1}$ ; the last step uses the inverse function theorem.
$$
\begin{array}{l} p = \left(g ^ {\prime}\right) ^ {- 1} \left(g ^ {\prime} (p)\right) \\ = \left(g ^ {\prime}\right) ^ {- 1} \left(g ^ {\prime} (q) + g ^ {\prime} (p) - g ^ {\prime} (q)\right) \\ = \left(g ^ {\prime}\right) ^ {- 1} \left(g ^ {\prime} (q) + g ^ {\prime} (p) - f ^ {\prime} (p) + f ^ {\prime} (p) - g ^ {\prime} (q)\right) \\ = \left(g ^ {\prime}\right) ^ {- 1} \left(g ^ {\prime} (q) - \left(f ^ {\prime} (p) - g ^ {\prime} (p) - z\right)\right) \\ \geq q - \left[ \left(\left(g ^ {\prime}\right) ^ {- 1}\right) ^ {\prime} (x) \right] \Bigg | _ {x = g ^ {\prime} (q)} \cdot \left(f ^ {\prime} (p) - g ^ {\prime} (p) - z\right) \\ = q - \frac {1}{g ^ {\prime \prime} (q)} \cdot \left(f ^ {\prime} (p) - g ^ {\prime} (p) - z\right), \\ \end{array}
$$
where the forth step uses  $g'(q) = f'(p) - z$ ; the fifth step follows from the convexity of  $(g')^{-1}$ ; the last step uses the inverse function theorem. Rearranging this inequality yields Eq. (77).
By swapping  $p$  with  $q$ ,  $f$  with  $g$ , and flipping  $z$  to  $-z$ , repeating the steps above yields that
$$
p \geq q + \frac {1}{f ^ {\prime \prime} (q)} \cdot \left(g ^ {\prime} (q) - f ^ {\prime} (q) + z\right),
$$
$$
p \leq q + \frac {1}{f ^ {\prime \prime} (p)} \cdot \left(g ^ {\prime} (q) - f ^ {\prime} (q) + z\right).
$$
Rearranging these inequalities finishes the proof of Eq. (79) and Eq. (78), respectively.