SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} D ^ {t, \bar {t}} (p ^ {t}, \tilde {p} ^ {t + 1}) \\ = \phi^ {t} (p ^ {t}) - \widetilde {\phi} ^ {t} (\widetilde {p} ^ {t + 1}) - \left\langle \nabla \widetilde {\phi} ^ {t} (\widetilde {p} ^ {t + 1}), p ^ {t} - \widetilde {p} ^ {t + 1} \right\rangle \\ = \phi^ {t} (p ^ {t}) - \widetilde {\phi} ^ {t} (\bar {p} ^ {t + 1}) - \left\langle \nabla \widetilde {\phi} ^ {t} (\widetilde {p} ^ {t + 1}), p ^ {t} - \bar {p} ^ {t + 1} \right\rangle \\ + \widetilde {\phi} ^ {t} (\bar {p} ^ {t + 1}) - \widetilde {\phi} ^ {t} (\widetilde {p} ^ {t + 1}) - \left\langle \nabla \widetilde {\phi} ^ {t} (\widetilde {p} ^ {t + 1}), \bar {p} ^ {t + 1} - \widetilde {p} ^ {t + 1} \right\rangle \\ = \phi_ {U} ^ {t} (p ^ {t}) - \widetilde {\phi} _ {U} ^ {t} (\bar {p} ^ {t + 1}) - \left\langle \nabla \widetilde {\phi} _ {U} ^ {t} (\widetilde {p} ^ {t + 1}), p _ {U} ^ {t} - \bar {p} _ {U} ^ {t + 1} \right\rangle (45) \\ + \phi_ {V} ^ {t} (p ^ {t}) - \widetilde {\phi} _ {V} ^ {t} (\bar {p} ^ {t + 1}) - \left\langle \nabla \widetilde {\phi} _ {V} ^ {t} (\widetilde {p} ^ {t + 1}), p _ {V} ^ {t} - \bar {p} _ {V} ^ {t + 1} \right\rangle (46) \\ + D ^ {\tilde {t}} \left(\bar {p} ^ {t + 1}, \tilde {p} ^ {t + 1}\right), \\ \end{array}
$$
where the last term is the residual regret.
For the regret on optimal arms, we show that the term  $\phi_U^t (p^t) - \widetilde{\phi}_U^t (\bar{p}^{t + 1}) - \Big\langle \nabla \widetilde{\phi}_U^t (\widetilde{p}^{t + 1}),p_U^t -\bar{p}_U^{t + 1}\Big\rangle$  in Eq. (45) is exactly  $D_U^{t,t + 1}(p^t,\bar{p}^{t + 1})$ :
$$
\begin{array}{l} \phi_ {U} ^ {t} (p ^ {t}) - \widetilde {\phi} _ {U} ^ {t} (\bar {p} ^ {t + 1}) - \left\langle \nabla \widetilde {\phi} _ {U} ^ {t} (\widetilde {p} ^ {t + 1}), p _ {U} ^ {t} - \bar {p} _ {U} ^ {t + 1} \right\rangle \\ = \phi_ {U} ^ {t} \left(p ^ {t}\right) - \widetilde {\phi} _ {U} ^ {t} \left(\bar {p} ^ {t + 1}\right) - \left\langle \nabla \phi_ {U} ^ {t + 1} \left(\bar {p} ^ {t + 1}\right) + c \cdot \mathbf {1} _ {U}, p _ {U} ^ {t} - \bar {p} _ {U} ^ {t + 1} \right\rangle \tag {47} \\ = \phi_ {U} ^ {t} (p ^ {t}) - \tilde {\phi} _ {U} ^ {t} (\bar {p} ^ {t + 1}) - \left\langle \nabla \phi_ {U} ^ {t + 1} (\bar {p} ^ {t + 1}), p _ {U} ^ {t} - \bar {p} _ {U} ^ {t + 1} \right\rangle \\ = D _ {U} ^ {t, t + 1} \left(p ^ {t}, \bar {p} ^ {t + 1}\right), \\ \end{array}
$$
where the first step uses the KKT conditions of  $\bar{p}^{t + 1}$  and  $\widetilde{p}^{t + 1}$ , which indicate  $\nabla \widetilde{\phi}_U^t (\widetilde{p}^{t + 1}) = \nabla \phi_U^{t + 1}(\bar{p}^{t + 1}) + c\cdot \mathbf{1}_U$  for a constant  $c\in \mathbb{R}$ ; the second step follows from the fact  $\sum_{i\in U}p_i^t = \sum_{i\in U}\bar{p}_i^{t + 1}$ , which guarantees  $\langle c\cdot \mathbf{1}_U,p_U^t -\bar{p}_U^{t + 1}\rangle = 0$  for any  $c\in \mathbb{R}$ ; the third step follows from the definition of  $\widetilde{\phi}^t$  which implies  $\widetilde{\phi}_U^t (\bar{p}^{t + 1}) = \phi_U^{t + 1}(\bar{p}^{t + 1})$ .
Following the same idea of handling Eq. (45), we have:
$$
\begin{array}{l} \phi_ {V} ^ {t} (p ^ {t}) - \widetilde {\phi} _ {V} ^ {t} (\bar {p} ^ {t + 1}) - \left\langle \nabla \widetilde {\phi} _ {V} ^ {t} (\widetilde {p} ^ {t + 1}), p _ {V} ^ {t} - \bar {p} _ {V} ^ {t + 1} \right\rangle \\ = \phi_ {V} ^ {t} \left(p ^ {t}\right) - \phi_ {V} ^ {t} \left(\bar {p} ^ {t + 1}\right) - \left\langle \nabla \phi_ {V} ^ {t} \left(\bar {p} ^ {t + 1}\right), p _ {V} ^ {t} - \bar {p} _ {V} ^ {t + 1} \right\rangle \tag {48} \\ = D _ {V} ^ {t} \left(p ^ {t}, \bar {p} ^ {t + 1}\right), \\ \end{array}
$$
where the first step follows from the definition of  $\widetilde{\phi}^t$  that  $\widetilde{\phi}_V^t (\bar{p}^{t + 1}) = \phi_V^t (\bar{p}^{t + 1})$  and the KKT conditions of  $\bar{p}^{t + 1}$  and  $\widetilde{p}^{t + 1}$ , which indicate that  $\nabla \widetilde{\phi}_V^t (\widetilde{p}^{t + 1}) = \nabla \phi_V^t (\bar{p}^{t + 1}) + c'\cdot \mathbf{1}_V$  for a constant  $c'\in \mathbb{R}$ . Finally, combining these bounds together, we have
$$
\begin{array}{l} D ^ {t, t + 1} \left(p ^ {t}, p ^ {t + 1}\right) \\ \leq D ^ {t, \bar {t}} \left(p ^ {t}, \tilde {p} ^ {t + 1}\right) + \phi_ {V} ^ {t} \left(p ^ {t + 1}\right) - \phi_ {V} ^ {t + 1} \left(p ^ {t + 1}\right) \tag {ByEq.(44)} \\ = \phi_ {V} ^ {t} \left(p ^ {t + 1}\right) - \phi_ {V} ^ {t + 1} \left(p ^ {t + 1}\right) + \phi_ {V} ^ {t} \left(p ^ {t}\right) - \widetilde {\phi} _ {V} ^ {t} \left(\bar {p} ^ {t + 1}\right) - \left\langle \nabla \widetilde {\phi} _ {V} ^ {t} \left(\widetilde {p} ^ {t + 1}\right), p _ {V} ^ {t} - \bar {p} _ {V} ^ {t + 1} \right\rangle \\ + \phi_ {U} ^ {t} (p ^ {t}) - \widetilde {\phi} _ {U} ^ {t} (\bar {p} ^ {t + 1}) - \left\langle \nabla \widetilde {\phi} _ {U} ^ {t} (\widetilde {p} ^ {t + 1}), p _ {U} ^ {t} - \bar {p} _ {U} ^ {t + 1} \right\rangle \\ + D ^ {\tilde {t}} \left(\bar {p} ^ {t + 1}, \bar {p} ^ {t + 1}\right) \\ \end{array}
$$
$$
\begin{array}{l} = D _ {V} ^ {t} \left(p ^ {t}, \tilde {p} ^ {t + 1}\right) + \phi_ {V} ^ {t} \left(p ^ {t + 1}\right) - \phi_ {V} ^ {t + 1} \left(p ^ {t + 1}\right) (ByEq.(48)) \\ + D _ {U} ^ {t, t + 1} \left(p ^ {t}, \bar {p} ^ {t + 1}\right) + D ^ {\tilde {t}} \left(\bar {p} ^ {t + 1}, \tilde {p} ^ {t + 1}\right) (ByEq.(47)) \\ \end{array}
$$
B.2 Regret on Sub-Optimal Arms
Lemma B.2.1. For the Decoupled-Tsallis-INF algorithm, the following holds:
$$
\mathbb {E} \left[ \sum_ {t = T _ {0} + 1} ^ {T} D _ {V} ^ {t} \left(p ^ {t}, \bar {p} ^ {t + 1}\right) + \phi_ {V} ^ {t} \left(p ^ {t + 1}\right) - \phi_ {V} ^ {t + 1} \left(p ^ {t + 1}\right) \right] \leq \mathcal {O} \left(\mathbb {E} \left[ \sum_ {t = T _ {0} + 1} ^ {T} \sum_ {i \in V} \frac {K ^ {1 / 6} \left(p _ {i} ^ {t}\right) ^ {2 / 3}}{\sqrt {t}} \right]\right).
$$
for any  $T_0 \geq 16K$  and any subset  $V \subseteq [K]$ .
Proof. The stability term  $D_V^t (p^t,\bar{p}^{t + 1})$ , in conditional expectation, is bounded as:
$$
\begin{array}{l} \mathbb {E} ^ {t} \left[ D _ {V} ^ {t} \left(p ^ {t}, \bar {p} ^ {t + 1}\right) \right] \\ = \mathbb {E} ^ {t} \left[ \left\langle p _ {V} ^ {t} - \bar {p} _ {V} ^ {t + 1}, \widehat {\ell} _ {V} ^ {t} \right\rangle - D _ {V} ^ {t} (\bar {p} ^ {t + 1}, p ^ {t}) \right] \\ \leq \mathcal {O} \left(\frac {1}{\gamma^ {t}} \sum_ {i \in V} \frac {\left(p _ {i} ^ {t}\right) ^ {4 / 3}}{g _ {i} ^ {t}}\right) \\ = \mathcal {O} \left(\frac {1}{\gamma^ {t}} \left(\sum_ {i \in V} \left(p _ {i} ^ {t}\right) ^ {2 / 3}\right) \left(\sum_ {i \in [ K ]} \left(p _ {i} ^ {t}\right) ^ {2 / 3}\right)\right) (49) \\ \leq \mathcal {O} \left(\frac {K ^ {1 / 3}}{\gamma^ {t}} \left(\sum_ {i \in V} \left(p _ {i} ^ {t}\right) ^ {2 / 3}\right)\right) \\ = \mathcal {O} \left(\frac {K ^ {1 / 6}}{\sqrt {t}} \sum_ {i \in V} \left(p _ {i} ^ {t}\right) ^ {2 / 3}\right), (50) \\ \end{array}
$$
where the first step applies Lemma C.5.3; the third step uses the definition of  $g_i^t$ ; the forth step follows from the fact  $\sum_{i\in [K]}(p_i^t)^{2 / 3}\leq K^{1 / 3}$ ; the last step uses the definition of  $\gamma^t = K^{1 / 6}\sqrt{t}$ .
On the other hand, the penalty term can be bounded as
$$
\begin{array}{l} \mathbb {E} ^ {t} \left[ \phi_ {V} ^ {t} \left(p ^ {t + 1}\right) - \phi_ {V} ^ {t + 1} \left(p ^ {t + 1}\right) \right] \\ = \mathcal {O} \left(\left(\gamma^ {t + 1} - \gamma^ {t}\right) \sum_ {i \in V} \left(p _ {i} ^ {t + 1}\right) ^ {2 / 3}\right) \leq \mathcal {O} \left(\frac {K ^ {1 / 6}}{\sqrt {t + 1}} \sum_ {i \in V} \left(p _ {i} ^ {t + 1}\right) ^ {2 / 3}\right), \tag {51} \\ \end{array}
$$
where the second step uses  $\gamma^{t + 1} - \gamma^t\leq \frac{K^{1 / 6}}{\sqrt{t + 1}}$  . By summing up Eq. (50) and Eq. (51) over all  $t$  from  $T_0 + 1$  to  $T$  , we arrive at the desired bound.
B.3 Regret on Optimal Arms
Since the standard  $2/3$ -Tsallis entropy regularizer is twice differentiable and its partial derivatives are concave (that is,  $(-x^{2/3})' = -2/3 \cdot x^{-1/3}$  is concave on  $\mathbb{R}_{>0}$ ), the first condition of Theorem 4.3 holds. As the Decoupled-Tsallis-INF algorithm adopts  $\gamma^t = K^{1/6} \sqrt{t}$ , one can easily verify that the second condition of the theorem also holds. Hence, one can apply Theorem 4.3 to bound the regret on optimal arms by zero, i.e.,  $D_U^{t,t+1}(p^t, \tilde{p}^{t+1}) - D_U^{t,t+1}(q^t, q^{t+1}) \leq 0$  for  $\forall t$ .
B.4 Residual Regret
In this section, our goal is to prove the following lemma, which upper-bounds the residual regret by a self-bounding term.
Lemma B.4.1. For the Decoupled-Tsallis-INF algorithm, the following holds for any  $T_0 \geq 16K$  and any subset  $V \subseteq [K]$ :
$$
\mathbb {E} \left[ \sum_ {t = T _ {0} + 1} ^ {T} D ^ {\widetilde {t}} \left(\bar {p} ^ {t + 1}, \widetilde {p} ^ {t + 1}\right) \right] \leq \mathcal {O} \left(\mathbb {E} \left[ \sum_ {t = T _ {0} + 1} ^ {T} \sum_ {i \in V} \frac {K ^ {1 / 6} \left(p _ {i} ^ {t}\right) ^ {2 / 3}}{\sqrt {t}} \right]\right).
$$
To show Lemma B.4.1, we start from a decomposition of the residual regret. Similar to the analysis of the residual regret for MAB in Section A.4, here, bounding the residual regret requires us to carefully analyze the following KKT conditions:
$$
\nabla \phi_ {V} ^ {t} (\bar {p} ^ {t + 1}) = \nabla \phi_ {V} ^ {t} (p ^ {t}) - \widehat {\ell} _ {V} ^ {t} + \lambda_ {V} \cdot \mathbf {1} _ {V},
$$
$$
\nabla \widetilde {\phi} _ {U} ^ {t} (\widetilde {p} ^ {t + 1}) = \nabla \phi_ {U} ^ {t + 1} (\widetilde {p} ^ {t + 1}) = \nabla \phi_ {U} ^ {t} (p ^ {t}) - \widehat {\ell} _ {U} ^ {t} + \lambda_ {K} \cdot \mathbf {1} _ {U},
$$
$$
\nabla \widetilde {\phi} _ {V} ^ {t} (\widetilde {p} ^ {t + 1}) = \nabla \phi_ {V} ^ {t} (\widetilde {p} ^ {t + 1}) = \nabla \phi_ {V} ^ {t} (p ^ {t}) - \widehat {\ell} _ {V} ^ {t} + \lambda_ {K} \cdot \mathbf {1} _ {V},
$$
where  $\lambda_U, \lambda_V, \lambda_K$  are corresponding Lagrange multipliers for  $\bar{p}^{t+1}$  and  $\widetilde{p}^{t+1}$ . Note that, according to the definition of  $\bar{p}_V^{t+1}$  in Eq. (36) and the fact that  $\widetilde{\phi}^t(x) = \phi_V^t(x) + \phi_U^{t+1}(x)$  for any  $x$ , the second and the forth conditions are slightly different from those in Section A.4.
We start from the following decomposition of  $D^{\tilde{t}}\left(\bar{p}^{t + 1}, \widetilde{p}^{t + 1}\right)$ . For any  $c \in \mathbb{R}$ , we have
$$
D ^ {\tilde {t}} \left(\bar {p} ^ {t + 1}, \tilde {p} ^ {t + 1}\right)
$$
$$
\begin{array}{l} = \widetilde {\phi} ^ {t} (\bar {p} ^ {t + 1}) - \widetilde {\phi} ^ {t} (\widetilde {p} ^ {t + 1}) - \left\langle \nabla \widetilde {\phi} ^ {t} (\widetilde {p} ^ {t + 1}), \bar {p} ^ {t + 1} - \widetilde {p} ^ {t + 1} \right\rangle \\ = \widetilde {\phi} ^ {t} (\bar {p} ^ {t + 1}) - \widetilde {\phi} ^ {t} (\widetilde {p} ^ {t + 1}) - \left\langle \nabla \widetilde {\phi} ^ {t} (\widetilde {p} ^ {t + 1}) + c \cdot \mathbf {1} _ {K}, \bar {p} ^ {t + 1} - \widetilde {p} ^ {t + 1} \right\rangle \\ = \widetilde {\phi} ^ {t} (\bar {p} ^ {t + 1}) - \widetilde {\phi} ^ {t} (\widetilde {p} ^ {t + 1}) - \left\langle \nabla \widetilde {\phi} ^ {t} (\bar {p} ^ {t + 1}) - \lambda_ {V} \cdot \mathbf {1} _ {V} - \lambda_ {U} \cdot \mathbf {1} _ {U} + c \cdot \mathbf {1} _ {K}, \bar {p} ^ {t + 1} - \widetilde {p} ^ {t + 1} \right\rangle \\ = \left\langle \lambda_ {V} \cdot \mathbf {1} _ {V} - c \cdot \mathbf {1} _ {K}, \bar {p} _ {V} ^ {t + 1} - p _ {V} ^ {t + 1} \right\rangle + \widetilde {\phi} _ {V} ^ {t} (\bar {p} ^ {t + 1}) - \widetilde {\phi} _ {V} ^ {t} (\tilde {p} ^ {t + 1}) - \left\langle \nabla \widetilde {\phi} _ {V} ^ {t} (\bar {p} ^ {t + 1}), \bar {p} _ {V} ^ {t + 1} - \widetilde {p} _ {V} ^ {t + 1} \right\rangle \\ \end{array}
$$
$$
\begin{array}{l} + \left\langle \lambda_ {U} \cdot \mathbf {1} _ {U} - c \cdot \mathbf {1} _ {K}, \bar {p} _ {U} ^ {t + 1} - \widetilde {p} _ {U} ^ {t + 1} \right\rangle + \widetilde {\phi} _ {U} ^ {t} (\bar {p} ^ {t + 1}) - \widetilde {\phi} _ {U} ^ {t} (\tilde {p} ^ {t + 1}) - \left\langle \nabla \widetilde {\phi} _ {U} ^ {t} (\bar {p} ^ {t + 1}), \bar {p} _ {U} ^ {t + 1} - \widetilde {p} _ {U} ^ {t + 1} \right\rangle \\ = \left\langle \lambda_ {V} \cdot \mathbf {1} _ {V} - c \cdot \mathbf {1} _ {K}, \bar {p} _ {V} ^ {t + 1} - p _ {V} ^ {t + 1} \right\rangle + \phi_ {V} ^ {t} (\bar {p} ^ {t + 1}) - \phi_ {V} ^ {t} (\widetilde {p} ^ {t + 1}) - \left\langle \nabla \phi_ {V} ^ {t} (\bar {p} ^ {t + 1}), \bar {p} _ {V} ^ {t + 1} - \widetilde {p} _ {V} ^ {t + 1} \right\rangle \\ + \left\langle \lambda_ {U} \cdot \mathbf {1} _ {U} - c \cdot \mathbf {1} _ {K}, \bar {p} _ {U} ^ {t + 1} - \widetilde {p} _ {U} ^ {t + 1} \right\rangle + \phi_ {U} ^ {t + 1} (\bar {p} ^ {t + 1}) - \phi_ {U} ^ {t + 1} (\widetilde {p} ^ {t + 1}) - \left\langle \nabla \phi_ {U} ^ {t + 1} (\bar {p} ^ {t + 1}), \bar {p} _ {U} ^ {t + 1} - \widetilde {p} _ {U} ^ {t + 1} \right\rangle \\ = \left\langle \lambda_ {V} \cdot \mathbf {1} _ {V} - c \cdot \mathbf {1} _ {K}, \bar {p} _ {V} ^ {t + 1} - \tilde {p} _ {V} ^ {t + 1} \right\rangle - D _ {V} ^ {t} (\tilde {p} ^ {t + 1}, \bar {p} ^ {t + 1}) \\ + \left\langle \lambda_ {U} \cdot {\bf 1} _ {U} - c \cdot {\bf 1} _ {K}, \bar {p} _ {U} ^ {t + 1} - \widetilde {p} _ {U} ^ {t + 1} \right\rangle - D _ {U} ^ {t + 1} (\widetilde {p} ^ {t + 1}, \bar {p} ^ {t + 1}), \\ \end{array}
$$
where the second step uses the fact  $\langle c\cdot \mathbf{1}_K,\bar{p}^{t + 1} - \widetilde{p}^{t + 1}\rangle = 0$  for any  $c\in \mathbb{R}$ ; the third step follows from the facts that  $\nabla \widetilde{\phi}_U^{t + 1}(\widetilde{p}^{t + 1}) = \nabla \phi_U^{t + 1}(\widetilde{p}^{t + 1}) = \nabla \phi_U^{t + 1}(\bar{p}^{t + 1}) - \lambda_U\cdot \mathbf{1}_U + \lambda_K\cdot \mathbf{1}_K$  and similarly that  $\nabla \widetilde{\phi}_V^{t + 1}(\widetilde{p}^{t + 1}) = \nabla \phi_V^{t + 1}(\bar{p}^{t + 1}) - \lambda_V\cdot \mathbf{1}_V + \lambda_K\cdot \mathbf{1}_K$ , which are derived from the KKT conditions above, and  $\langle \lambda_K\cdot \mathbf{1}_K,\bar{p}^{t + 1} - \widetilde{p}^{t + 1}\rangle = 0$ ; the forth step follows from the fact that  $\langle \lambda_U\cdot \mathbf{1}_U,\bar{p}_V^{t + 1} - \bar{p}_V^{t + 1}\rangle = \langle \lambda_V\cdot \mathbf{1}_V,\bar{p}_U^{t + 1} - \bar{p}_U^{t + 1}\rangle = 0$ ; the fifth step uses the definition of  $\widetilde{\phi}^t$ .
Now, we choose  $c$  as
$$
c = \frac {\lambda_ {U} \sum_ {i \in U} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3} + \lambda_ {V} \sum_ {i \in V} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}}{\sum_ {i \in [ K ]} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}}, \tag {52}
$$
which minimizes a subsequent bound (Eq. (53)) below. With this choice of  $c$ , we apply Lemma C.5.3 (the condition required by the lemma is verified at the end of Appendix B.5) to get for any  $t \geq T_0 + 1$
$$
\begin{array}{l} D ^ {\tilde {t}} \left(\bar {p} ^ {t + 1}, \tilde {p} ^ {t + 1}\right) \leq \mathcal {O} \left(\sum_ {i \in U} \frac {\left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}}{\gamma^ {t + 1}} \left(\lambda_ {U} - c\right) ^ {2} + \sum_ {i \in V} \frac {\left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}}{\gamma^ {t}} \left(\lambda_ {V} - c\right) ^ {2}\right) \\ \leq \mathcal {O} \left(\sum_ {i \in U} \frac {\left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}}{\gamma^ {t}} (\lambda_ {U} - c) ^ {2} + \sum_ {i \in V} \frac {\left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}}{\gamma^ {t}} (\lambda_ {V} - c) ^ {2}\right) \tag {53} \\ \end{array}
$$
$$
\begin{array}{l} = \mathcal {O} \left(\frac {\left(\sum_ {i \in V} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}\right) \left(\sum_ {i \in U} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}\right)}{\gamma^ {t} \sum_ {i \in [ K ]} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}} \left(\lambda_ {V} - \lambda_ {U}\right) ^ {2}\right) \\ \leq \mathcal {O} \left(\frac {\left(\sum_ {i \in V} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}\right) \left(\sum_ {i \in U} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}\right)}{\gamma^ {t} \sum_ {i \in [ K ]} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}} \left(\lambda_ {U} ^ {2} + \lambda_ {V} ^ {2}\right)\right), \tag {54} \\ \end{array}
$$
where the third step applies the choice of  $c$  and the last step uses  $(x - y)^2 \leq 2(x^2 + y^2)$  for any  $x, y \in \mathbb{R}$ .
To further bound the regret in Eq. (54), we need to know the range of  $\lambda_U$  and  $\lambda_V$ . According to Lemma C.4.2, we use Eq. (80) and Eq. (81) to obtain the upper and lower bounds of  $\lambda_U$ , and apply Eq. (82) and Eq. (83) to get upper and lower bounds of  $\lambda_V$ , which are summarized as:
$$
\lambda_ {U} \leq \frac {\sum_ {i \in U} \left(p _ {i} ^ {t}\right) ^ {4 / 3} \widehat {\ell} _ {i} ^ {t}}{\sum_ {i \in U} \left(p _ {i} ^ {t}\right) ^ {4 / 3}}, \quad \lambda_ {U} \geq - 2 \left(\gamma^ {t + 1} - \gamma^ {t}\right) \frac {\sum_ {i \in U} \vec {p} _ {i} ^ {t + 1}}{\sum_ {i \in U} \left(\vec {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}},
$$
$$
\lambda_ {V} \leq \frac {\sum_ {i \in V} \left(p _ {i} ^ {t}\right) ^ {4 / 3} \widehat {\ell} _ {i} ^ {t}}{\sum_ {i \in V} \left(p _ {i} ^ {t}\right) ^ {4 / 3}}, \quad \lambda_ {V} \geq \frac {\sum_ {i \in V} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3} \widehat {\ell} _ {i} ^ {t}}{\sum_ {i \in V} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}} \geq 0,
$$
where the last step of the lower bound  $\lambda_V\geq 0$  follows from  $\vec{p}_i^{t + 1}\geq 0$  and  $\widehat{\ell}_i^t\geq 0$  for any  $t,i$
In what follows, we continue to bound the  $\lambda_U$ -related part of Eq. (54) and the  $\lambda_V$ -related part respectively. As our goal is to bound  $\mathbb{E}[\sum_{t = T_0 + 1}^T D^{\tilde{t}}(\bar{p}^{t + 1},\tilde{p}^{t + 1})]$ , it suffices to bound these terms in conditional expectation for every round  $t\geq T_0 + 1$ .
Bounding the  $\lambda_U$ -related term First, we consider the term in Eq. (54) related to  $\lambda_U$  and decompose it as:
$$
\begin{array}{l} \mathbb {E} ^ {t} \left[ \frac {\left(\sum_ {i \in V} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}\right) \left(\sum_ {i \in U} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}\right)}{\gamma^ {t} \sum_ {i \in [ K ]} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}} \lambda_ {U} ^ {2} \right] \\ = \mathbb {E} ^ {t} \left[ \frac {\left(\sum_ {i \in V} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}\right) \left(\sum_ {i \in U} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}\right)}{\gamma^ {t} \sum_ {i \in [ K ]} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}} \lambda_ {U} ^ {2} \left(\mathbb {I} \{\lambda_ {U} \geq 0 \} + \mathbb {I} \{\lambda_ {U} <   0 \}\right) \right]. \\ \end{array}
$$
For the case of  $\mathbb{I}\{\lambda_U\geq 0\}$ , we use the upper bound of  $\lambda_U$  to show
$$
\begin{array}{l} \mathbb {E} ^ {t} \left[ \frac {\left(\sum_ {i \in V} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}\right) \left(\sum_ {i \in U} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}\right)}{\gamma^ {t} \sum_ {i \in [ K ]} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}} \lambda_ {U} ^ {2} \mathbb {I} \left\{\lambda_ {U} \geq 0 \right\} \right] \\ \leq \mathcal {O} \left(\mathbb {E} ^ {t} \left[ \frac {\left(\sum_ {i \in V} \left(\vec {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}\right) \left(\sum_ {i \in U} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}\right)}{\gamma^ {t} \sum_ {i \in [ K ]} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}} \left(\frac {\sum_ {i \in U} \left(p _ {i} ^ {t}\right) ^ {4 / 3} \hat {\ell} _ {i} ^ {t}}{\sum_ {i \in U} \left(p _ {i} ^ {t}\right) ^ {4 / 3}}\right) ^ {2} \right]\right). \tag {55} \\ \end{array}
$$
For the case of  $\mathbb{I}\{\lambda_U < 0\}$ , we use the lower bound of  $\lambda_U$  to show
$$
\begin{array}{l} \mathbb {E} ^ {t} \left[ \frac {\left(\sum_ {i \in V} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}\right) \left(\sum_ {i \in U} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}\right)}{\gamma^ {t} \sum_ {i \in [ K ]} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}} \lambda_ {U} ^ {2} \mathbb {I} \left\{\lambda_ {U} <   0 \right\} \right] \\ \leq \mathcal {O} \left(\mathbb {E} ^ {t} \left[ \frac {\left(\gamma^ {t + 1} - \gamma^ {t}\right) ^ {2}}{\gamma^ {t}} \frac {\left(\sum_ {i \in V} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}\right) \left(\sum_ {i \in U} \bar {p} _ {i} ^ {t + 1}\right) ^ {2}}{\left(\sum_ {i \in [ K ]} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}\right) \left(\sum_ {i \in U} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}\right)} \right]\right). \tag {56} \\ \end{array}
$$
Now, our goal is to bound Eq. (55) and Eq. (56), which are shown below respectively.
Bounding Eq. (55) For this part, we bound it as
$$
\begin{array}{l} \mathbb {E} ^ {t} \left[ \frac {\left(\sum_ {i \in V} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}\right) \left(\sum_ {i \in U} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}\right)}{\gamma^ {t} \sum_ {i \in [ K ]} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}} \cdot \left(\frac {\sum_ {i \in U} \left(p _ {i} ^ {t}\right) ^ {4 / 3} \widehat {\ell} _ {i} ^ {t}}{\sum_ {i \in U} \left(p _ {i} ^ {t}\right) ^ {4 / 3}}\right) ^ {2} \right] \\ = \mathbb {E} ^ {t} \left[ \frac {\left(\sum_ {i \in V} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}\right) \left(\sum_ {i \in U} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}\right)}{\gamma^ {t} \sum_ {i \in [ K ]} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}} \cdot \frac {\sum_ {i \in U} \left(\left(p _ {i} ^ {t}\right) ^ {4 / 3} \hat {\ell} _ {i} ^ {t}\right) ^ {2}}{\left(\sum_ {i \in U} \left(p _ {i} ^ {t}\right) ^ {4 / 3}\right) ^ {2}} \right] \\ \leq \mathbb {E} ^ {t} \left[ \left(\sum_ {i \in V} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}\right) ^ {1 / 2} \left(\sum_ {i \in U} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}\right) ^ {1 / 2} \frac {\sum_ {i \in U} \left(\left(p _ {i} ^ {t}\right) ^ {4 / 3} \widehat {\ell_ {i} ^ {t}}\right) ^ {2}}{2 \gamma^ {t} \left(\sum_ {i \in U} \left(p _ {i} ^ {t}\right) ^ {4 / 3}\right) ^ {2}} \right] \\ \leq \mathbb {E} ^ {t} \left[ 8 \left(\sum_ {i \in V} \left(p _ {i} ^ {t}\right) ^ {4 / 3}\right) ^ {1 / 2} \left(\sum_ {i \in U} \left(p _ {i} ^ {t}\right) ^ {4 / 3}\right) ^ {1 / 2} \frac {\sum_ {i \in U} \left(\left(p _ {i} ^ {t}\right) ^ {4 / 3} \widehat {\ell_ {i} ^ {t}}\right) ^ {2}}{2 \gamma^ {t} \left(\sum_ {i \in U} \left(p _ {i} ^ {t}\right) ^ {4 / 3}\right) ^ {2}} \right] \\ = 4 \left(\sum_ {i \in V} \left(p _ {i} ^ {t}\right) ^ {4 / 3}\right) ^ {1 / 2} \frac {\mathbb {E} ^ {t} \left[ \sum_ {i \in U} \left(\left(p _ {i} ^ {t}\right) ^ {4 / 3} \widehat {\ell} _ {i} ^ {t}\right) ^ {2} \right]}{\gamma^ {t} \left(\sum_ {i \in U} \left(p _ {i} ^ {t}\right) ^ {4 / 3}\right) ^ {3 / 2}}, \\ \end{array}
$$
where the first step follows from the fact  $\widehat{\ell}_i^t\cdot \widehat{\ell}_j^t = 0$  for any  $i\neq j$  and the definition of  $\widehat{\ell}_i^t$ ; the second step uses
$$
\left(\sum_ {i \in [ K ]} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}\right) = \left(\sum_ {i \in U} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3} + \sum_ {i \in V} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}\right) \geq 2 \left(\sum_ {i \in U} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}\right) ^ {1 / 2} \left(\sum_ {i \in V} \left(\bar {p} _ {i} ^ {t + 1}\right) ^ {4 / 3}\right) ^ {1 / 2}; \tag {57}
$$
the third step applies Lemma B.5.1 and Corollary B.5.2 to obtain the multiplicative relation on  $U$  and  $V$ , respectively; and the last step holds since  $p^t$  is deterministic given the history.
Furthermore, by direct calculation, we have