SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} K \sum_ {i \in [ K ]} \frac {\left(\gamma_ {i} ^ {t} - \gamma_ {i} ^ {t + 1}\right) ^ {2} \left(\frac {1}{p _ {i} ^ {t}}\right) ^ {2}}{w _ {i} ^ {t}} \leq K \sum_ {i \in [ K ]} \frac {\left(\gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t}\right) ^ {2} \left(p _ {i} ^ {t}\right) ^ {2}}{\gamma_ {i} ^ {t} \left(p _ {i} ^ {t}\right) ^ {2}} \quad (w _ {i} ^ {t} \geq \gamma_ {i} ^ {t} / \left(p _ {i} ^ {t}\right) ^ {2}) \\ \leq \frac {K}{\log T} \sum_ {i \in [ K ]} \frac {\left(\gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t}\right) \max  \left\{p _ {i} ^ {t} , 1 / T \right\}}{\gamma_ {i} ^ {t} \gamma_ {i} ^ {t + 1}} \quad (\text {b y E q . (2 1)}) \\ \leq \frac {K}{\log T} \sum_ {i \in [ K ]} \left(\frac {1}{\gamma_ {i} ^ {t}} - \frac {1}{\gamma_ {i} ^ {t + 1}}\right). \quad (\max  \left\{p _ {i} ^ {t}, ^ {1} / T \right\} \leq 1) \\ \end{array}
$$
Taking the summation over all rounds yields that
$$
\frac {K}{\log T} \sum_ {i \in [ K ]} \sum_ {t = 1} ^ {T} \left(\frac {1}{\gamma_ {i} ^ {t}} - \frac {1}{\gamma_ {i} ^ {t + 1}}\right) = \mathcal {O} \left(\frac {K}{\log T} \sum_ {i \in [ K ]} \frac {1}{\gamma_ {i} ^ {1}}\right) = \mathcal {O} \left(\frac {K ^ {2}}{\sqrt {\log T}}\right),
$$
Lemma A.4.5 (RESREG for Shannon Entropy). When using the Shannon entropy regularizer with  $C_{\mathrm{LOG}} = 162 \log K$ ,  $\alpha = 1$ , and  $\theta = \sqrt{1 / \log T}$ , Algorithm 1 ensures:
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {i \in V} \frac {1}{p _ {i} ^ {t} w _ {i} ^ {t}} \right] = \mathcal {O} \left(\mathbb {S} _ {2} \left(\log^ {2} T\right)\right),
$$
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {i \in U} \frac {\mathbb {I} \left\{p _ {i} ^ {t} \leq \sum_ {j \in V} p _ {j} ^ {t} \right\}}{p _ {i} ^ {t} w _ {i} ^ {t}} \left(\frac {\frac {1}{w _ {i} ^ {t}}}{\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}}\right) \right] = \mathcal {O} \left(\mathbb {S} _ {1} \left(| U | \log^ {2} T\right)\right),
$$
$$
K \sum_ {t = 1} ^ {T} \sum_ {i \in [ K ]} \frac {\left(\gamma_ {i} ^ {t} - \gamma_ {i} ^ {t + 1}\right) ^ {2} \left(\log \left(p _ {i} ^ {t}\right)\right) ^ {2}}{w _ {i} ^ {t}} = \mathcal {O} \left(K ^ {2} \log^ {3 / 2} T\right),
$$
$$
\operatorname {R E S R E G} = \mathcal {O} \left(\mathbb {S} _ {2} \left(\log^ {2} T\right) + \mathbb {S} _ {1} \left(| U | \log^ {2} T\right) + | K | ^ {2} \log^ {3 / 2} T\right).
$$
Proof. Similar to the analysis for  $\beta$ -Tsallis entropy, we also have the first inequality relates to the bound of the stability term as:
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {i \in V} \frac {1}{p _ {i} ^ {t} w _ {i} ^ {t}} \right] \leq \mathbb {E} \left[ \sum_ {i \in V} \sum_ {t = 1} ^ {T} \frac {1}{\gamma_ {i} ^ {t}} \right] = \mathcal {O} \left(\mathbb {E} \left[ \sum_ {i \in V} \sqrt {\log^ {2} T \cdot \sum_ {t = 1} ^ {T} p _ {i} ^ {t}} \right]\right) = \mathcal {O} \left(\mathbb {S} _ {2} \left(\log^ {2} T\right)\right),
$$
where the second step follows from similar analysis of the stability term in Lemma A.2.4.
By the same arguments in the proof of Lemma A.4.3, we have
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {i \in U} \frac {\mathbb {I} \left\{p _ {i} ^ {t} \leq \sum_ {j \in V} p _ {j} ^ {t} \right\}}{p _ {i} ^ {t} w _ {i} ^ {t}} \left(\frac {\frac {1}{w _ {i} ^ {t}}}{\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}}\right) \right] = \mathcal {O} \left(\mathbb {E} \left[ \sqrt {| U | (\log T) ^ {2} \cdot \sum_ {t = 1} ^ {T} \sum_ {i \in V} p _ {i} ^ {t}} \right]\right) \\ = \mathcal {O} \left(\mathbb {S} _ {1} \left(| U | \log^ {2} T\right)\right), \\ \end{array}
$$
according to our learning rate schedule, where the extra  $\sqrt{\log T}$  factor is caused by  $\theta$ .
By direct calculation, for any round  $t$ , we have
$$
\begin{array}{l} K \sum_ {i \in [ K ]} \frac {\left(\gamma_ {i} ^ {t} - \gamma_ {i} ^ {t + 1}\right) ^ {2} (\log (p _ {i} ^ {t})) ^ {2}}{w _ {i} ^ {t}} \\ \leq K \cdot \sum_ {i \in [ K ]} \frac {\left(\gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t}\right) ^ {2} \log^ {2} \left(p _ {i} ^ {t}\right) p _ {i} ^ {t}}{\gamma_ {i} ^ {t}} \quad (w _ {i} ^ {t} \geq \gamma_ {i} ^ {t} / p _ {i} ^ {t}) \\ \leq \frac {K}{\log T} \cdot \sum_ {i \in [ K ]} \frac {\left(\gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t}\right)}{\gamma_ {i} ^ {t} \gamma_ {i} ^ {t + 1}} \cdot \frac {\log^ {2} \left(p _ {i} ^ {t}\right) p _ {i} ^ {t}}{\max  \left\{p _ {i} ^ {t} , 1 / T \right\}} \tag {byEq.21} \\ = \frac {K}{\log T} \sum_ {i \in [ K ]} \left(\frac {1}{\gamma_ {i} ^ {t}} - \frac {1}{\gamma_ {i} ^ {t + 1}}\right) \frac {\log^ {2} \left(p _ {i} ^ {t}\right) p _ {i} ^ {t}}{\max \left\{p _ {i} ^ {t} , ^ {1} / T \right\}}. \\ \end{array}
$$
Note that, for any arm  $i$ , the term  $\frac{\log^2(p_i^t)p_i^t}{\max\{p_i^t,1 / T\}}$  can be bounded as:
$$
\begin{array}{l} \frac {\log^ {2} \left(p _ {i} ^ {t}\right) p _ {i} ^ {t}}{\max  \left\{p _ {i} ^ {t} , 1 / T \right\}} \\ = \frac {\left(\log \left(z _ {i} ^ {t}\right) + \log \left(\frac {p _ {i} ^ {t}}{z _ {i} ^ {t}}\right)\right) ^ {2} p _ {i} ^ {t}}{z _ {i} ^ {t}} \quad (\text {l e t} z _ {i} ^ {t} = \max  \left\{p _ {i} ^ {t}, ^ {1} / T \right\} \text {f o r s i m p l i c i t y}) \\ \leq 2 \left(\left(\log T\right) ^ {2} + \frac {p _ {i} ^ {t}}{z _ {i} ^ {t}} \cdot \log^ {2} \left(\frac {p _ {i} ^ {t}}{z _ {i} ^ {t}}\right)\right) \quad ((x + y) ^ {2} \leq 2 (x ^ {2} + y ^ {2}) \text {a n d} z _ {i} ^ {t} \in [ 1 / T, 1 ]) \\ \leq 2 \left(\left(\log T\right) ^ {2} + 1\right). \quad \text {(s i n c e} x \left(\log x\right) ^ {2} \leq 1 \text {f o r} x \in [ 0, 1 ] \\ \end{array}
$$
Finally, taking the summation over all rounds yields
$$
\frac {K}{\log T} \sum_ {t = 1} ^ {T} \sum_ {i \in [ K ]} \left(\frac {1}{\gamma_ {i} ^ {t}} - \frac {1}{\gamma_ {i} ^ {t + 1}}\right) \frac {\log^ {2} \left(p _ {i} ^ {t}\right) p _ {i} ^ {t}}{\max \left\{p _ {i} ^ {t} , ^ {1} / T \right\}} \leq \frac {2 K \left(\left(\log T\right) ^ {2} + 1\right)}{\log T} \sum_ {t = 1} ^ {T} \sum_ {i \in [ K ]} \left(\frac {1}{\gamma_ {i} ^ {t}} - \frac {1}{\gamma_ {i} ^ {t + 1}}\right)
$$
$$
= \mathcal {O} \left(K \log T \sum_ {i \in [ K ]} \frac {1}{\gamma_ {i} ^ {1}}\right) = \mathcal {O} \left(K ^ {2} \log^ {3 / 2} T\right),
$$
B Proof of Theorem 3.4: Decoupled-Tsallis-INF
For the Decoupled-Tsallis-INF algorithm [Rouyer and Seldin, 2020], the learning rate schedule is  $\gamma_i^t = K^{1/6}\sqrt{t}$  for all  $i \in [K]$  at round  $t$  (henceforth denoted by  $\gamma^t$  for conciseness), and the loss estimator is
$$
\widehat {\ell} _ {i} ^ {t} = \frac {\mathbb {I} \{j ^ {t} = i \} \ell_ {i} ^ {t}}{g _ {i} ^ {t}}, \quad \forall i \in [ K ], \quad \text {w h e r e} \quad g _ {i} ^ {t} = \frac {\left(p _ {i} ^ {t}\right) ^ {2 / 3}}{\sum_ {j \in [ K ]} \left(p _ {j} ^ {t}\right) ^ {2 / 3}} \text {a n d} j ^ {t} \sim g ^ {t}. \tag {35}
$$
Although our analysis of Decoupled-Tsallis-INF follows similar ideas as the case for MAB, we need a finer decomposition of  $D^{t,t + 1}(p^t,p^{t + 1})$  since we do not add an extra log-barrier regularizer to ensure the multiplicative relation between  $\bar{p}^{t + 1}$  and  $p^t$  (which is important to avoid any  $T$ -dependence). To present this new decomposition, we first introduce some definitions. For  $\bar{p}^{t + 1}$ , we maintain the same definition of  $\bar{p}_U^{t + 1}$  in Eq. (7), but slightly adjust the definition of  $\bar{p}_V^{t + 1}$  as:
$$
\bar {p} _ {V} ^ {t + 1} = \underset { \begin{array}{c} x \in \mathbb {R} _ {\geq 0} ^ {K}, \sum_ {i \in U} x _ {i} = 0, \\ \sum_ {i \in V} x _ {i} = \sum_ {i \in V} p _ {i} ^ {t} \end{array} } {\operatorname* {a r g m i n}} \left\langle x, \sum_ {\tau \leq t} \widehat {\ell} ^ {\tau} \right\rangle + \phi_ {V} ^ {t} (x). \tag {36}
$$
Compared to the definition in Eq. (7) used for the analysis of MAB, the new definition here changes  $\phi^{t + 1}$  to  $\phi^t$ . Such a skewness of the regularizer eventually allows us to get  $\sum_{i\in V}(p^t)^{2 / 3}$  rather than  $\sum_{i\in V}(\tilde{p}^{t + 1})^{2 / 3}$  in the regret bound, which is important since we cannot easily convert between these two anymore as we did for the MAB analysis due to the aforementioned lack of multiplicative relation. Furthermore, as  $\phi^{t + 1}$  is used for  $U$  and  $\phi^t$  is used for  $V$ , we introduce another intermediate point  $\widetilde{p}^{t + 1}$  to bridge this skewness, which is defined as:
$$
\widetilde {p} ^ {t + 1} = \underset {x \in \Omega_ {K}} {\operatorname {a r g m i n}} \left\langle x, \sum_ {\tau \leq t} \widehat {\ell} ^ {\tau} \right\rangle + \widetilde {\phi} ^ {t} (x), \tag {37}
$$
where  $\widetilde{\phi}^t (x) = \phi_V^t (x) + \phi_U^{t + 1}(x)$  can be regarded as a specific intermediate regularizer between  $\phi_V^t$  and  $\phi_U^{t + 1}$ . To distinguish the Bregman divergence  $D^{t}(x,y)$  defined on regularizer  $\phi^t$ , we define  $D^t (x,y)$  on regularizer  $\widetilde{\phi}^t$  as:
$$
D ^ {\widetilde {t}} (x, y) = \widetilde {\phi} ^ {t} (x) - \widetilde {\phi} ^ {t} (y) - \left\langle \nabla \widetilde {\phi} ^ {t} (y), x - y \right\rangle . \tag {38}
$$
Similarly, we define the following two notions of skewed Bregman divergence for  $s, t \in \mathbb{N}$ :
$$
D ^ {t, \widetilde {s}} (x, y) = \phi^ {t} (x) - \widetilde {\phi} ^ {s} (y) - \left\langle \nabla \widetilde {\phi} ^ {s} (y), x - y \right\rangle ,
$$
$$
D ^ {\widetilde {t}, s} (x, y) = \widetilde {\phi} ^ {t} (y) - \phi^ {s} (x) - \langle \nabla \phi^ {s} (y), x - y \rangle .
$$
With the help of definitions above, our new decomposition for  $D^{t,t+1}(p^t,p^{t+1}) - D_U^{t,t+1}(q^t,q^{t+1})$  is as follows (see Lemma B.1.1)
$$
\begin{array}{l} \underbrace {D _ {V} ^ {t} \left(p ^ {t} , \bar {p} ^ {t + 1}\right) + \phi_ {V} ^ {t} \left(p ^ {t + 1}\right) - \phi_ {V} ^ {t + 1} \left(p ^ {t + 1}\right)} _ {\text {r e g r e t o n s u b - o p t i m a l a r m s}} \\ + \underbrace {D _ {U} ^ {t , t + 1} \left(p ^ {t} , \bar {p} ^ {t + 1}\right) - D _ {U} ^ {t , t + 1} \left(q ^ {t} , q ^ {t + 1}\right)} _ {\text {r e g r e t o n o p t i m a l a r m s}} + \underbrace {D _ {\text {r e s i d u a l r e g r e t}} ^ {\widetilde {t}} \left(\bar {p} ^ {t + 1} , \tilde {p} ^ {t + 1}\right)} _ {\text {r e s i d u a l r e g r e t}}. \tag {39} \\ \end{array}
$$
Now, we are ready to use this decomposition to analyze the regret bound for Decoupled-Tsallis-INF. As we do not add an extra log-barrier, we just set  $\epsilon = 0$ , i.e.,  $q^{t} = \widetilde{q}^{t}$  where  $\widetilde{q}^t$  is given in Eq. (15). As a result, the only preprocessing cost for Decoupled-Tsallis-INF is  $D$ . To show Theorem 3.4, we present the following theorem.
Theorem B.1. Decoupled-Tsallis-INF ensures
$$
\operatorname {R e g} ^ {T} \leq \mathcal {O} \left(\sqrt {K T _ {0}} + \mathbb {E} \left[ \sum_ {t = T _ {0} + 1} ^ {T} \left(\frac {K ^ {1 / 6}}{\sqrt {t}} \sum_ {i \in V} \left(p _ {i} ^ {t}\right) ^ {2 / 3}\right) \right] + \sqrt {K}\right) + D,
$$
for any  $T_0 \geq 16K$ , any subset  $U \subseteq [K]$ ,  $V = [K] \backslash U$ , and  $D = \mathbb{E}\left[\sum_{t=1}^{T} \max_{i \in U} \mathbb{E}^t\left[\ell_i^t - \ell_{i^\star}^t\right]\right]$ .
Proof. With the decomposition in Lemma B.1.1, we have the following upper bound for  $\mathrm{Reg}^T$ :
$$
\begin{array}{l} \mathbb {E} \left[ D ^ {1} \left(q ^ {1}, p ^ {1}\right) + \sum_ {t = 1} ^ {T} D ^ {t, t + 1} \left(p ^ {t}, p ^ {t + 1}\right) - D _ {U} ^ {t, t + 1} \left(q ^ {t}, q ^ {t + 1}\right) \right] + D \\ \leq \mathbb {E} \left[ \sum_ {t = T _ {0} + 1} ^ {T} D _ {V} ^ {t} \left(p ^ {t}, \bar {p} ^ {t + 1}\right) + \phi_ {V} ^ {t} \left(p ^ {t + 1}\right) - \phi_ {V} ^ {t + 1} \left(p ^ {t + 1}\right) \right] + \mathbb {E} \left[ \sum_ {t = T _ {0} + 1} ^ {T} D _ {U} ^ {t, t + 1} \left(p ^ {t}, \bar {p} ^ {t + 1}\right) - D _ {U} ^ {t, t + 1} \left(q ^ {t}, q ^ {t + 1}\right) \right] \\ + \mathbb {E} \left[ \sum_ {t = T _ {0} + 1} ^ {T} D ^ {\tilde {t}} \left(\bar {p} ^ {t + 1}, \tilde {p} ^ {t + 1}\right) \right] + \mathcal {O} \left(\sqrt {K T _ {0}} + \sqrt {K}\right) + D, \\ \end{array}
$$
where we bound the regret of the first  $T_0$  rounds by  $\mathcal{O}\left(\sqrt{KT_0}\right)$  via Lemma B.5.3, and  $D^{1}(q^{1},p^{1})$  by
$$
D ^ {1} \left(q ^ {1}, p ^ {1}\right) = - 3 \gamma^ {1} \sum_ {i \in [ K ]} \left(\left(q _ {i} ^ {1}\right) ^ {2 / 3} - \left(p _ {i} ^ {1}\right) ^ {2 / 3}\right) \leq 3 K ^ {1 / 6} \sum_ {i \in [ K ]} \left(p _ {i} ^ {1}\right) ^ {2 / 3} = 3 \sqrt {K}.
$$
First, we bound the regret on sub-optimal arms using Lemma B.2.1:
$$
\mathbb {E} \left[ \sum_ {t = T _ {0} + 1} ^ {T} D _ {V} ^ {t} \left(p ^ {t}, \bar {p} ^ {t + 1}\right) + \phi_ {V} ^ {t} \left(p ^ {t + 1}\right) - \phi_ {V} ^ {t + 1} \left(p ^ {t + 1}\right) \right] \leq \mathcal {O} \left(\mathbb {E} \left[ \sum_ {t = T _ {0} + 1} ^ {T} \sum_ {i \in V} \frac {K ^ {1 / 6} \left(p _ {i} ^ {t}\right) ^ {2 / 3}}{\sqrt {t}} \right]\right). \tag {40}
$$
The regret related to the optimal arms for any given  $t$  is again non-positive (see Appendix B.3):
$$
D _ {U} ^ {t, t + 1} \left(p ^ {t}, \bar {p} ^ {t + 1}\right) - D _ {U} ^ {t, t + 1} \left(q ^ {t}, q ^ {t + 1}\right) \leq 0.
$$
Finally, we turn to bound the residual regret (see Appendix B.4). By Lemma B.4.1, the residual regret is bounded as
$$
\mathbb {E} \left[ \sum_ {t = T _ {0} + 1} ^ {T} D ^ {\tilde {t}} \left(\bar {p} ^ {t + 1}, \tilde {p} ^ {t + 1}\right) \right] \leq \mathcal {O} \left(\mathbb {E} \left[ \sum_ {t = T _ {0} + 1} ^ {T} \sum_ {i \in V} \frac {K ^ {1 / 6} \left(p _ {i} ^ {t}\right) ^ {2 / 3}}{\sqrt {t}} \right]\right). \tag {41}
$$
Combining all the above, we complete the proof.
Armed with the result in Theorem B.1, we are now ready to prove Theorem 3.4.
Proof of Theorem 3.4. Theorem B.1 shows that there exists a constant  $z \in \mathbb{R}_{>0}$  such that
$$
\operatorname {R e g} ^ {T} \leq z \left(\sqrt {K T _ {0}} + \mathbb {E} \left[ \sum_ {t = T _ {0} + 1} ^ {T} \frac {K ^ {1 / 6}}{\sqrt {t}} \sum_ {i \in V} \left(p _ {i} ^ {t}\right) ^ {2 / 3} \right] + \sqrt {K}\right) + D
$$
holds for any  $T_0 \geq 16K$ . For the adversarial setting, we simply pick  $T_0 = T$  and  $V = [K] \backslash \{i^\star\}$  (so that  $D = 0$ ), and then the claimed bound  $\mathcal{O}(\sqrt{KT})$  follows.
For the stochastic setting, let  $V = \{i : \Delta_i \neq 0\}$  as in Section 2 and define
$$
\widetilde {\mathbb {G}} = \sum_ {i \in V} \frac {1}{\Delta_ {i} ^ {2}}, \tag {42}
$$
as an instance complexity measure for DEE-MAB. We have the following bound of  $\mathrm{Reg}^T$  for any  $\eta > 0$  and  $T_0 \geq 16K$ :
$$
\begin{array}{l} \operatorname {R e g} ^ {T} = (1 + \eta) \operatorname {R e g} ^ {T} - \eta \operatorname {R e g} ^ {T} \\ \leq (1 + \eta) \left(z \sqrt {K T _ {0}} + \mathbb {E} \left[ \sum_ {t = T _ {0} + 1} ^ {T} \frac {z K ^ {1 / 6}}{\sqrt {t}} \sum_ {i \in V} \left(p _ {i} ^ {t}\right) ^ {2 / 3} \right] + D + z \sqrt {K}\right) \\ - \eta \left(\mathbb {E} \left[ \sum_ {t = T _ {0} + 1} ^ {T} \sum_ {i \in V} \Delta_ {i} p _ {i} ^ {t} - C \right]\right) \\ = \mathbb {E} \left[ \sum_ {t = T _ {0} + 1} ^ {T} \sum_ {i \in V} \left(\frac {(1 + \eta) z K ^ {1 / 6}}{\sqrt {t}} \left(p _ {i} ^ {t}\right) ^ {2 / 3} - \eta \Delta_ {i} p _ {i} ^ {t}\right) \right] \\ + (1 + \eta) \left(z \sqrt {K T _ {0}} + D + z \sqrt {K}\right) + \eta C \\ \leq \sum_ {t = T _ {0} + 1} ^ {T} \sum_ {i \in V} \frac {\left(1 + \eta\right) ^ {3} z ^ {3} \sqrt {K}}{\eta^ {2} \Delta_ {i} ^ {2} t ^ {3 / 2}} + (1 + \eta) \left(z \sqrt {K T _ {0}} + D + z \sqrt {K}\right) + \eta C \\ \leq \frac {2 z ^ {3} (1 + \eta) ^ {3} \widetilde {\mathbb {G}}}{\eta^ {2}} \sqrt {\frac {K}{T _ {0}}} + (1 + \eta) (z \sqrt {K T _ {0}} + D + z \sqrt {K}) + \eta C \\ = \mathcal {O} \left(\frac {\left(1 + \eta^ {3}\right) \widetilde {\mathbb {G}}}{\eta^ {2}} \sqrt {\frac {K}{T _ {0}}} + \eta \left(C + \sqrt {K T _ {0}}\right) + \sqrt {K T _ {0}} + (1 + \eta) \left(D + \sqrt {K}\right)\right) \\ = \mathcal {O} \left(\frac {\widetilde {\mathbb {G}}}{\eta^ {2}} \sqrt {\frac {K}{T _ {0}}} + \eta \left(\widetilde {\mathbb {G}} \sqrt {\frac {K}{T _ {0}}} + C + \sqrt {K T _ {0}} + D\right) + \left(\sqrt {K T _ {0}} + D\right)\right), \\ \end{array}
$$
where the second step uses Condition (1); the forth step uses [Rouyer and Seldin, 2020, Lemma 8]; the fifth step follows from the fact  $\sum_{t = T_0 + 1}^T t^{-3 / 2}\leq \int_{T_0 + 1}^T t^{-3 / 2}dt\leq \frac{2}{\sqrt{T_0 + 1}}\leq \frac{2}{\sqrt{T_0}}$ ; the sixth step uses  $(1 + \eta)^{3}\leq 4(1 + \eta^{3})$  for any  $\eta >0$ ; the last step uses  $\sqrt{K}\leq \sqrt{KT_0}$  as  $T_0\geq 16K$ .
Picking the optimal  $\eta$  yields
$$
\begin{array}{l} \operatorname {R e g} ^ {T} \leq \mathcal {O} \left(\left(\widetilde {\mathbb {G}} \sqrt {\frac {K}{T _ {0}}}\right) ^ {1 / 3} \left(\widetilde {\mathbb {G}} \sqrt {\frac {K}{T _ {0}}} + C + \sqrt {K T _ {0}} + D\right) ^ {2 / 3} + \sqrt {K T _ {0}} + D\right) \\ \leq \mathcal {O} \left(\left(\widetilde {\mathbb {G}} \sqrt {\frac {K}{T _ {0}}}\right) ^ {1 / 3} \left(\widetilde {\mathbb {G}} \sqrt {\frac {K}{T _ {0}}}\right) ^ {2 / 3} + \left(\widetilde {\mathbb {G}} \sqrt {\frac {K}{T _ {0}}}\right) ^ {1 / 3} C ^ {2 / 3} + \left(\widetilde {\mathbb {G}} \sqrt {\frac {K}{T _ {0}}}\right) ^ {1 / 3} \left(\sqrt {K T _ {0}}\right) ^ {2 / 3}\right) \\ + \mathcal {O} \left(\left(\widetilde {\mathbb {G}} \sqrt {\frac {K}{T _ {0}}}\right) ^ {1 / 3} D ^ {2 / 3} + \sqrt {K T _ {0}} + D\right) \\ \leq \mathcal {O} \left(\frac {\widetilde {\mathbb {G}} K ^ {1 / 2}}{T _ {0} ^ {1 / 2}} + \frac {\widetilde {\mathbb {G}} ^ {1 / 3} K ^ {1 / 6} C ^ {2 / 3}}{T _ {0} ^ {1 / 6}} + \widetilde {\mathbb {G}} ^ {1 / 3} K ^ {1 / 2} T _ {0} ^ {1 / 6} + K ^ {1 / 2} T _ {0} ^ {1 / 2} + D\right), \tag {43} \\ \end{array}
$$
where the second step uses the fact  $(x + y)^{2/3} \leq 3(x^{2/3} + y^{2/3})$  for any  $x, y \geq 0$ , and the third step follows from the fact  $x^{1/3}y^{2/3} \leq x + y$  for any  $x, y \geq 0$ .
For  $C \leq \widetilde{\mathbb{G}}^{1/2} K^{1/2}$ , by picking  $T_0 = \max \left\{\widetilde{\mathbb{G}}, 16K\right\}$ , we have
$$
\begin{array}{l} \operatorname {R e g} ^ {T} \leq \mathcal {O} \left(\frac {\widetilde {\mathbb {G}} K ^ {1 / 2}}{T _ {0} ^ {1 / 2}} + \frac {\widetilde {\mathbb {G}} ^ {2 / 3} K ^ {1 / 2}}{T _ {0} ^ {1 / 6}} + \widetilde {\mathbb {G}} ^ {1 / 3} K ^ {1 / 2} T _ {0} ^ {1 / 6} + K ^ {1 / 2} T _ {0} ^ {1 / 2} + D\right) \\ \leq \mathcal {O} \left(\frac {\widetilde {\mathbb {G}} K ^ {1 / 2}}{\widetilde {\mathbb {G}} ^ {1 / 2}} + \frac {\widetilde {\mathbb {G}} ^ {2 / 3} K ^ {1 / 2}}{\widetilde {\mathbb {G}} ^ {1 / 6}} + K ^ {1 / 2} \left(\widetilde {\mathbb {G}} + K\right) ^ {1 / 2} + D\right) \\ \end{array}
$$
$$
\leq \mathcal {O} \left(\widetilde {\mathbb {G}} ^ {1 / 2} K ^ {1 / 2} + K + D\right),
$$
where the first step uses  $C \leq \widetilde{\mathbb{G}}^{1/2} K^{1/2}$ ; the second step follows from  $\widetilde{\mathbb{G}} \leq T_0 \leq \widetilde{\mathbb{G}} + 16K$ ; the last step follows from the fact  $\sqrt{x + y} \leq \sqrt{x} + \sqrt{y}$  for  $x, y \geq 0$ .
For  $C \geq \widetilde{\mathbb{G}}^{1/2}K^{1/2}$ , we set  $T_0 = \max\left\{\frac{\widetilde{\mathbb{G}}^{1/2}C}{K^{1/2}}, 16K\right\}$  and have Eq. (43) further bounded by
$$
\begin{array}{l} \mathcal {O} \left(\frac {\widetilde {\mathbb {G}} K ^ {1 / 2}}{T _ {0} ^ {1 / 2}} + \frac {\widetilde {\mathbb {G}} ^ {1 / 3} K ^ {1 / 6} C ^ {2 / 3}}{T _ {0} ^ {1 / 6}} + \widetilde {\mathbb {G}} ^ {1 / 3} K ^ {1 / 2} T _ {0} ^ {1 / 6} + K ^ {1 / 2} T _ {0} ^ {1 / 2} + D\right) \\ \leq \mathcal {O} \left(\widetilde {\mathbb {G}} K ^ {1 / 2} \left(\frac {K ^ {1 / 2}}{\widetilde {\mathbb {G}} ^ {1 / 2} C}\right) ^ {1 / 2} + \widetilde {\mathbb {G}} ^ {1 / 3} K ^ {1 / 6} C ^ {2 / 3} \left(\frac {K ^ {1 / 2}}{\widetilde {\mathbb {G}} ^ {1 / 2} C}\right) ^ {1 / 6} + \widetilde {\mathbb {G}} ^ {1 / 3} K ^ {1 / 2} \left(\frac {\widetilde {\mathbb {G}} ^ {1 / 2} C}{K ^ {1 / 2}}\right) ^ {1 / 6}\right) \\ + \mathcal {O} \left(\widetilde {\mathbb {G}} ^ {1 / 3} K ^ {1 / 2} \cdot K ^ {1 / 6} + K ^ {1 / 2} \left(\frac {\widetilde {\mathbb {G}} ^ {1 / 2} C}{K ^ {1 / 2}}\right) ^ {1 / 2} + K ^ {1 / 2} \cdot K ^ {1 / 2} + D\right) \\ = \mathcal {O} \left(\widetilde {\mathbb {G}} ^ {3 / 4} K ^ {3 / 4} C ^ {- 1 / 2} + \widetilde {\mathbb {G}} ^ {1 / 4} K ^ {1 / 4} C ^ {1 / 2} + \widetilde {\mathbb {G}} ^ {5 / 1 2} K ^ {5 / 1 2} C ^ {1 / 6} + \widetilde {\mathbb {G}} ^ {1 / 4} K ^ {1 / 4} C ^ {1 / 2}\right) \\ + \mathcal {O} \left(\widetilde {\mathbb {G}} ^ {1 / 2} K ^ {1 / 2} + K + D\right) \\ \leq \mathcal {O} \left(\widetilde {\mathbb {G}} ^ {1 / 4} K ^ {1 / 4} C ^ {1 / 2} + \widetilde {\mathbb {G}} ^ {1 / 2} K ^ {1 / 2} + K + D\right), \\ \end{array}
$$
where the first step uses  $\frac{\widetilde{\mathbb{G}}^{1/2}C}{K^{1/2}} \leq T_0 \leq \frac{\widetilde{\mathbb{G}}^{1/2}C}{K^{1/2}} + 16K$ ; the second step follows from the facts that  $x^{1/3}y^{2/3} \leq x + y$  for any  $x, y \geq 0$ , and that  $\widetilde{\mathbb{G}}^{1/3}K^{2/3} = \left(\widetilde{\mathbb{G}}^{1/2}K^{1/2}\right)^{2/3} \cdot (K)^{1/3}$ ; the third step uses  $C \geq \widetilde{\mathbb{G}}^{1/2}K^{1/2}$ .
Combining these two cases together yields the claimed bound:
$$
\begin{array}{l} \operatorname {R e g} ^ {T} = \mathcal {O} \left(\widetilde {\mathbb {G}} ^ {1 / 4} K ^ {1 / 4} C ^ {1 / 2} + \widetilde {\mathbb {G}} ^ {1 / 2} K ^ {1 / 2} + K + D\right) \\ = \mathcal {O} \left(\sqrt {\sum_ {i \in V} \frac {K}{\Delta_ {i} ^ {2}}} + \sqrt {C} \cdot \left(\sum_ {i \in V} \frac {K}{\Delta_ {i} ^ {2}}\right) ^ {1 / 4} + K + D\right). \\ \end{array}
$$
B.1 Regret Decomposition for Decouple-Tsallis-INF
In this subsection, we present the proposed regret decomposition using  $\bar{p}^{t + 1}$  (with the new definition) and  $\widetilde{p}^t$ .
Lemma B.1.1. For any  $t$ ,  $D^{t,t+1}(p^t, p^{t+1}) - D_U^{t,t+1}(q^t, q^{t+1})$  is bounded by
$$
\underbrace {D _ {V} ^ {t} (p ^ {t} , \bar {p} ^ {t + 1}) + \phi_ {V} ^ {t} (p ^ {t + 1}) - \phi_ {V} ^ {t + 1} (p ^ {t + 1})} _ {\text {r e g r e t o n s u b - o p t i m a l a r m s}} + \underbrace {D _ {U} ^ {t , t + 1} (p ^ {t} , \bar {p} ^ {t + 1}) - D _ {U} ^ {t , t + 1} (q ^ {t} , q ^ {t + 1})} _ {\text {r e g r e t o n o p t i m a l a r m s}} + \underbrace {D _ {\text {r e s i d u a l r e g r e t}} ^ {\tilde {t}} (\bar {p} ^ {t + 1} , \tilde {p} ^ {t + 1})} _ {\text {r e s i d u a l r e g r e t}}.
$$
Proof. We proceed as follows:
$$
\begin{array}{l} D ^ {t, t + 1} \left(p ^ {t}, p ^ {t + 1}\right) \\ = \left\langle p ^ {t} - p ^ {t + 1}, \widehat {\ell} ^ {t} \right\rangle - D ^ {t + 1, t} \left(p ^ {t + 1}, p ^ {t}\right) \\ = \left\langle p ^ {t} - p ^ {t + 1}, \widehat {\ell} ^ {t} \right\rangle - \left(\widetilde {\phi} ^ {t} \left(p ^ {t + 1}\right) - \phi^ {t} \left(p ^ {t}\right) - \left\langle \nabla \phi^ {t} \left(p ^ {t}\right), p ^ {t + 1} - p ^ {t} \right\rangle\right) + \widetilde {\phi} ^ {t} \left(p ^ {t + 1}\right) - \phi^ {t + 1} \left(p ^ {t + 1}\right) \\ = \left\langle p ^ {t} - p ^ {t + 1}, \widehat {\ell} ^ {t} \right\rangle - D ^ {\widetilde {t}, t} (p ^ {t + 1}, p ^ {t}) + \widetilde {\phi} ^ {t} (p ^ {t + 1}) - \phi^ {t + 1} (p ^ {t + 1}) \\ \leq \left\langle p ^ {t} - \widetilde {p} ^ {t + 1}, \widehat {\ell} ^ {t} \right\rangle - D ^ {\widetilde {t}, t} \left(\widetilde {p} ^ {t + 1}, p ^ {t}\right) + \widetilde {\phi} ^ {t} \left(p ^ {t + 1}\right) - \phi^ {t + 1} \left(p ^ {t + 1}\right) \\ \end{array}
$$
$$
\begin{array}{l} = D ^ {t, \bar {t}} \left(p ^ {t}, \widetilde {p} ^ {t + 1}\right) + \widetilde {\phi} ^ {t} \left(p ^ {t + 1}\right) - \phi^ {t + 1} \left(p ^ {t + 1}\right) \\ = D ^ {t, \widetilde {t}} \left(p ^ {t}, \widetilde {p} ^ {t + 1}\right) + \phi_ {V} ^ {t} \left(p ^ {t + 1}\right) - \phi_ {V} ^ {t + 1} \left(p ^ {t + 1}\right), \tag {44} \\ \end{array}
$$
where the first step follows from Lemma C.5.1 according to the definitions of  $p^t$ ; the second step adds and subtracts  $\widetilde{\phi}^t(p^{t+1})$ ; the third step follows from the definition of  $D^{\widetilde{t},t}$ ; the forth step uses Lemma C.5.2; the last step follows from the definition of  $\widetilde{\phi}^t$  which implies  $\widetilde{\phi}_U^t(p^{t+1}) = \phi_U^{t+1}(p^{t+1})$ .
Then, with the help of  $\overline{p}^{t + 1}$ , we can further decompose the term  $D^{t,\widetilde{t}}(p^t,\widetilde{p}^{t + 1})$  as: