SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\lambda_ {U} \leq \mathcal {O} \left(\left(\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}\right) ^ {- 1} \left(\sum_ {i \in U} \frac {\widehat {\ell} _ {i} ^ {t}}{w _ {i} ^ {t}}\right)\right),
$$
$$
\lambda_ {U} \geq \mathcal {O} \left(- \left(\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}\right) ^ {- 1} \left(\sum_ {i \in U} \frac {\left(\gamma_ {i} ^ {t} - \gamma_ {i} ^ {t + 1}\right) \psi^ {\prime} (p _ {i} ^ {t})}{w _ {i} ^ {t}}\right)\right).
$$
Similarly, we have the following upper and lower bounds of  $\lambda_V$ :
$$
\lambda_ {V} \leq \mathcal {O} \left(\left(\sum_ {i \in V} \frac {1}{w _ {i} ^ {t}}\right) ^ {- 1} \left(\sum_ {i \in V} \frac {\widehat {\ell} _ {i} ^ {t}}{w _ {i} ^ {t}}\right)\right)
$$
$$
\lambda_ {V} \geq \mathcal {O} \left(- \left(\sum_ {i \in V} \frac {1}{w _ {i} ^ {t}}\right) ^ {- 1} \left(\sum_ {i \in V} \frac {\left(\gamma_ {i} ^ {t} - \gamma_ {i} ^ {t + 1}\right) \psi^ {\prime} \left(p _ {i} ^ {t}\right)}{w _ {i} ^ {t}}\right)\right).
$$
With the help of these notations, for any  $c\in \mathbb{R}$  , we have
$$
\begin{array}{l} \left\| \nabla \phi^ {t + 1} (\bar {p} ^ {t + 1}) - \nabla \phi^ {t + 1} (p ^ {t + 1}) - c \cdot \mathbf {1} _ {K} \right\| _ {\nabla^ {- 2} \phi^ {t} (p ^ {t})} \\ = \left\| \lambda_ {U} \cdot \mathbf {1} _ {U} + \lambda_ {V} \cdot \mathbf {1} _ {V} - \lambda_ {K} \cdot \mathbf {1} _ {K} - c \cdot \mathbf {1} _ {K} \right\| _ {\nabla^ {- 2} \phi^ {t} (p ^ {t})} ^ {2} \\ = \| \lambda_ {U} \cdot \mathbf {1} _ {U} + \lambda_ {V} \cdot \mathbf {1} _ {V} - c ^ {\prime} \cdot \mathbf {1} _ {K} \| _ {\nabla^ {- 2} \phi^ {t} (p ^ {t})} ^ {2} \\ = \sum_ {i \in U} \frac {1}{w _ {i} ^ {t}} \left(\lambda_ {U} - c ^ {\prime}\right) ^ {2} + \sum_ {i \in V} \frac {1}{w _ {i} ^ {t}} \left(\lambda_ {V} - c ^ {\prime}\right) ^ {2}, \\ \end{array}
$$
where in the second step we define  $c' = c + \lambda_K$ . Picking  $c'$  to minimize this term yields
$$
\begin{array}{l} \mathbb {E} ^ {t} \left[ D ^ {t + 1} \left(\bar {p} ^ {t + 1}, p ^ {t + 1}\right) \right] \\ \leq \mathcal {O} \left(\mathbb {E} ^ {t} \left[ \frac {\left(\sum_ {i \in V} \frac {1}{w _ {i} ^ {t}}\right) \left(\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}\right)}{\sum_ {i \in [ K ]} \frac {1}{w _ {i} ^ {t}}} (\lambda_ {U} - \lambda_ {V}) ^ {2} \right]\right) \\ = \mathcal {O} \left(\mathbb {E} ^ {t} \left[ \frac {\left(\sum_ {i \in V} \frac {1}{w _ {i} ^ {t}}\right) \left(\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}\right)}{\sum_ {i \in [ K ]} \frac {1}{w _ {i} ^ {t}}} \left(\lambda_ {U} ^ {2} + \lambda_ {V} ^ {2}\right) \right]\right) \\ = \mathcal {O} \left(\mathbb {E} ^ {t} \left[ \frac {\left(\sum_ {i \in V} \frac {1}{w _ {i} ^ {t}}\right) \left(\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}\right)}{\sum_ {i \in [ K ]} \frac {1}{w _ {i} ^ {t}}} \lambda_ {U} ^ {2} \right] + \mathbb {E} ^ {t} \left[ \frac {\left(\sum_ {i \in V} \frac {1}{w _ {i} ^ {t}}\right) \left(\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}\right)}{\sum_ {i \in [ K ]} \frac {1}{w _ {i} ^ {t}}} \lambda_ {V} ^ {2} \right]\right), \tag {28} \\ \end{array}
$$
where the second step uses  $(x - y)^2 \leq 2(x^2 + y^2)$  for any  $x, y \in \mathbb{R}$ . In Eq. (28), the first term is only related to  $\lambda_U$ , and the second term depends on  $\lambda_V$ . By upper and lower bounds of  $\lambda_U$  and  $\lambda_V$ ,
we have Eq. (28) bounded as
$$
\begin{array}{l} \mathcal {O} \left(\mathbb {E} ^ {t} \left[ \frac {\left(\sum_ {i \in V} \frac {1}{w _ {i} ^ {t}}\right) \left(\sum_ {i \in U} \frac {\widehat {\ell} _ {i} ^ {t}}{w _ {i} ^ {t}}\right) ^ {2}}{\left(\sum_ {i \in [ K ]} \frac {1}{w _ {i} ^ {t}}\right) \left(\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}\right)} \right] + \frac {\left(\sum_ {i \in U} \frac {\left(\gamma_ {i} ^ {t} - \gamma_ {i} ^ {t + 1}\right) \psi^ {\prime} \left(p _ {i} ^ {t}\right)}{w _ {i} ^ {t}}\right) ^ {2}}{\left(\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}\right)}\right) \tag {29} \\ + \mathcal {O} \left(\mathbb {E} ^ {t} \left[ \frac {\left(\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}\right) \left(\sum_ {i \in V} \frac {\widehat {\ell} _ {i} ^ {t}}{w _ {i} ^ {t}}\right) ^ {2}}{\left(\sum_ {i \in [ K ]} \frac {1}{w _ {i} ^ {t}}\right) \left(\sum_ {i \in V} \frac {1}{w _ {i} ^ {t}}\right)} \right] + \frac {\left(\sum_ {i \in V} \frac {\left(\gamma_ {i} ^ {t} - \gamma_ {i} ^ {t + 1}\right) \psi^ {\prime} (p _ {i} ^ {t})}{w _ {i} ^ {t}}\right) ^ {2}}{\left(\sum_ {i \in V} \frac {1}{w _ {i} ^ {t}}\right)}\right). \\ \end{array}
$$
Then, we further bound the first and the third term in Eq. (29). For the first term, we have
$$
\begin{array}{l} \mathbb {E} ^ {t} \left[ \frac {\left(\sum_ {i \in V} \frac {1}{w _ {i} ^ {t}}\right) \left(\sum_ {i \in U} \frac {\widehat {\ell} _ {i} ^ {t}}{w _ {i} ^ {t}}\right) ^ {2}}{\left(\sum_ {i \in [ K ]} \frac {1}{w _ {i} ^ {t}}\right) \left(\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}\right)} \right] = \mathbb {E} ^ {t} \left[ \frac {\left(\sum_ {i \in V} \frac {1}{w _ {i} ^ {t}}\right) \left(\sum_ {i \in U} \left(\frac {\widehat {\ell} _ {i} ^ {t}}{w _ {i} ^ {t}}\right) ^ {2}\right)}{\left(\sum_ {i \in [ K ]} \frac {1}{w _ {i} ^ {t}}\right) \left(\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}\right)} \right] \\ \leq \frac {\left(\sum_ {i \in V} \frac {1}{w _ {i} ^ {t}}\right) \left(\sum_ {i \in U} \frac {1}{\left(w _ {i} ^ {t}\right) ^ {2} p _ {i} ^ {t}}\right)}{\left(\sum_ {i \in [ K ]} \frac {1}{w _ {i} ^ {t}}\right) \left(\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}\right)}, \tag {30} \\ \end{array}
$$
where the first step uses the fact that  $\widehat{\ell}_i^t\cdot \widehat{\ell}_j^t = 0$  for  $i\neq j$ , and the second step follows from the definition of importance-weighted loss estimator. Similarly, we have the third term bounded as
$$
\mathbb {E} ^ {t} \left[ \frac {\left(\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}\right) \left(\sum_ {i \in V} \frac {\widehat {l} _ {i} ^ {t}}{w _ {i} ^ {t}}\right) ^ {2}}{\left(\sum_ {i \in [ K ]} \frac {1}{w _ {i} ^ {t}}\right) \left(\sum_ {i \in V} \frac {1}{w _ {i} ^ {t}}\right)} \right] \leq \frac {\left(\sum_ {i \in V} \frac {1}{p _ {i} ^ {t} \left(w _ {i} ^ {t}\right) ^ {2}}\right) \left(\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}\right)}{\left(\sum_ {i \in [ K ]} \frac {1}{w _ {i} ^ {t}}\right) \left(\sum_ {i \in V} \frac {1}{w _ {i} ^ {t}}\right)} \leq \sum_ {i \in V} \frac {1}{p _ {i} ^ {t} w _ {i} ^ {t}}. \tag {31}
$$
For the second term in Eq. (29), we have
$$
\frac {\left(\sum_ {i \in U} \frac {\left(\gamma_ {i} ^ {t} - \gamma_ {i} ^ {t + 1}\right) \psi^ {\prime} \left(p _ {i} ^ {t}\right)}{w _ {i} ^ {t}}\right) ^ {2}}{\left(\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}\right)} \leq \frac {| U | \sum_ {i \in U} \left(\frac {\left(\gamma_ {i} ^ {t} - \gamma_ {i} ^ {t + 1}\right) \psi^ {\prime} \left(p _ {i} ^ {t}\right)}{w _ {i} ^ {t}}\right) ^ {2}}{\left(\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}\right)} \leq K \sum_ {i \in U} \frac {\left(\gamma_ {i} ^ {t} - \gamma_ {i} ^ {t + 1}\right) ^ {2} \left(\psi^ {\prime} \left(p _ {i} ^ {t}\right)\right) ^ {2}}{w _ {i} ^ {t}}, \tag {32}
$$
where the first step follows from the Cauchy-Schwarz inequality. Similarly, we have the following bound for the forth term
$$
\frac {\left(\sum_ {i \in V} \frac {\left(\gamma_ {i} ^ {t} - \gamma_ {i} ^ {t + 1}\right) \psi^ {\prime} \left(p _ {i} ^ {t}\right)}{w _ {i} ^ {t}}\right) ^ {2}}{\left(\sum_ {i \in V} \frac {1}{w _ {i} ^ {t}}\right)} \leq K \sum_ {i \in V} \frac {\left(\gamma_ {i} ^ {t} - \gamma_ {i} ^ {t + 1}\right) ^ {2} \left(\psi^ {\prime} \left(p _ {i} ^ {t}\right)\right) ^ {2}}{w _ {i} ^ {t}}. \tag {33}
$$
Plugging Eq. (30), Eq. (31), Eq. (32), and Eq. (33) into Eq. (29) finishes the proof.
As mentioned earlier in Section 4, the first term in Lemma A.4.1 is the leading term in the stochastic setting which will yield a term of the order  $\mathcal{O}\left(\frac{|U| \log T}{\Delta_{\mathrm{MIN}}}\right)$  in the regret bound. In the following lemma, we further decompose this term into two parts where the first part is related to  $\mathbb{S}_2$  and the second part is related to  $\mathbb{S}_1$ .
Lemma A.4.2. The following holds:
$$
\frac {\left(\sum_ {i \in V} \frac {1}{w _ {i} ^ {t}}\right) \left(\sum_ {i \in U} \frac {1}{\left(w _ {i} ^ {t}\right) ^ {2} p _ {i} ^ {t}}\right)}{\left(\sum_ {i \in [ K ]} \frac {1}{w _ {i} ^ {t}}\right) \left(\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}\right)} \leq \sum_ {i \in V} \frac {1}{p _ {i} ^ {t} w _ {i} ^ {t}} + \sum_ {i \in U} \frac {\mathbb {I} \left\{p _ {i} ^ {t} \leq \sum_ {j \in V} p _ {j} ^ {t} \right\}}{p _ {i} ^ {t} w _ {i} ^ {t}} \left(\frac {\frac {1}{w _ {i} ^ {t}}}{\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}}\right).
$$
Proof. We first rewrite the fraction on left hand side as:
$$
\frac {\left(\sum_ {i \in V} \frac {1}{w _ {i} ^ {t}}\right) \left(\sum_ {i \in U} \frac {1}{\left(w _ {i} ^ {t}\right) ^ {2} p _ {i} ^ {t}}\right)}{\left(\sum_ {i \in [ K ]} \frac {1}{w _ {i} ^ {t}}\right) \left(\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}\right)} = \sum_ {i \in U} \frac {1}{p _ {i} ^ {t} w _ {i} ^ {t}} \frac {\sum_ {i \in V} \frac {1}{w _ {i} ^ {t}}}{\sum_ {i \in [ K ]} \frac {1}{w _ {i} ^ {t}}} \left(\frac {\frac {1}{w _ {i} ^ {t}}}{\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}}\right). \tag {34}
$$
To bound the right-hand side of Eq. (34), we introduce an indicator function for any  $t$  and any  $i \in U$ :
$$
A _ {i} ^ {t} = \mathbb {I} \left\{\frac {1}{p _ {i} ^ {t} w _ {i} ^ {t}} \frac {\sum_ {j \in V} \frac {1}{w _ {j} ^ {t}}}{\sum_ {j \in [ K ]} \frac {1}{w _ {j} ^ {t}}} \leq \sum_ {j \in V} \frac {1}{p _ {j} ^ {t} w _ {j} ^ {t}} \right\},
$$
to separate the term into two parts as:
$$
\underbrace {\sum_ {i \in U} \frac {A _ {i} ^ {t}}{p _ {i} ^ {t} w _ {i} ^ {t}} \frac {\sum_ {i \in V} \frac {1}{w _ {i} ^ {t}}}{\sum_ {i \in [ K ]} \frac {1}{w _ {i} ^ {t}}} \left(\frac {\frac {1}{w _ {i} ^ {t}}}{\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}}\right)} _ {\text {C a s e 1}}, \underbrace {\sum_ {i \in U} \frac {1 - A _ {i} ^ {t}}{p _ {i} ^ {t} w _ {i} ^ {t}} \frac {\sum_ {i \in V} \frac {1}{w _ {i} ^ {t}}}{\sum_ {i \in [ K ]} \frac {1}{w _ {i} ^ {t}}} \left(\frac {\frac {1}{w _ {i} ^ {t}}}{\sum  _ {i \in U} \frac {1}{w _ {i} ^ {t}}}\right)} _ {\text {C a s e 2}}.
$$
Case 1. By direct calculation, we have
$$
\sum_ {i \in U} \frac {A _ {i} ^ {t}}{p _ {i} ^ {t} w _ {i} ^ {t}} \frac {\sum_ {i \in V} \frac {1}{w _ {i} ^ {t}}}{\sum_ {i \in [ K ]} \frac {1}{w _ {i} ^ {t}}} \left(\frac {\frac {1}{w _ {i} ^ {t}}}{\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}}\right) \leq \sum_ {i \in U} \left(\sum_ {j \in V} \frac {1}{p _ {j} ^ {t} w _ {j} ^ {t}}\right) \left(\frac {\frac {1}{w _ {i} ^ {t}}}{\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}}\right) = \sum_ {j \in V} \frac {1}{p _ {j} ^ {t} w _ {j} ^ {t}}.
$$
Case 2. For arm  $i$  that  $A_{i}^{t} = 0$ , we have
$$
\begin{array}{l} \frac {1}{p _ {i} ^ {t} w _ {i} ^ {t}} \frac {\sum_ {j \in V} \frac {1}{w _ {j} ^ {t}}}{\sum_ {j \in [ K ]} \frac {1}{w _ {j} ^ {t}}} > \sum_ {j \in V} \frac {1}{p _ {j} ^ {t} w _ {j} ^ {t}} \Rightarrow \frac {1}{p _ {i} ^ {t} w _ {i} ^ {t}} \left(\frac {\sum_ {j \in V} \frac {1}{w _ {j} ^ {t}}}{\sum_ {j \in V} \frac {1}{p _ {j} ^ {t} w _ {j} ^ {t}}}\right) > \sum_ {j \in [ K ]} \frac {1}{w _ {j} ^ {t}} \\ \Rightarrow \frac {1}{p _ {i} ^ {t} w _ {i} ^ {t}} \left(\sum_ {j \in V} p _ {j} ^ {t}\right) > \sum_ {j \in [ K ]} \frac {1}{w _ {j} ^ {t}} \\ \Rightarrow \frac {1}{p _ {i} ^ {t} w _ {i} ^ {t}} \left(\sum_ {j \in V} p _ {j} ^ {t}\right) > \frac {1}{w _ {i} ^ {t}} \\ \Rightarrow \sum_ {j \in V} p _ {j} ^ {t} > p _ {i} ^ {t}, \\ \end{array}
$$
where the second step follows from the fact that  $\left(\sum_{j\in V}p_j^t\right)\left(\sum_{j\in V}\frac{1}{p_j^tw_j^t}\right)\geq \sum_{j\in V}\frac{1}{w_j^t}$ .
$$
\sum_ {i \in U} \frac {1 - A _ {i} ^ {t}}{p _ {i} ^ {t} w _ {i} ^ {t}} \frac {\sum_ {i \in V} \frac {1}{w _ {i} ^ {t}}}{\sum_ {i \in [ K ]} \frac {1}{w _ {i} ^ {t}}} \left(\frac {\frac {1}{w _ {i} ^ {t}}}{\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}}\right) \leq \sum_ {i \in U} \frac {\mathbb {I} \left\{p _ {i} ^ {t} \leq \sum_ {j \in V} p _ {j} ^ {t} \right\}}{p _ {i} ^ {t} w _ {i} ^ {t}} \left(\frac {\frac {1}{w _ {i} ^ {t}}}{\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}}\right),
$$
where  $1 - A_{i}^{t} = \mathbb{I}\{A_{i}^{t} = 0\} \leq \mathbb{I}\left\{p_{i}^{t} \leq \sum_{j \in V} p_{j}^{t}\right\}$  holds due to the fact that  $\mathbb{I}\{A\} \leq \mathbb{I}\{B\}$  for any events  $A, B$  with  $A \subseteq B$ . Finally, combining bounds for these two cases finishes the proof.
Based on Lemma A.4.1 and the careful decomposition in Lemma A.4.2, we now bound RESREG by
$$
\begin{array}{l} \mathcal {O} \left(\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {i \in V} \frac {1}{p _ {i} ^ {t} w _ {i} ^ {t}} + \sum_ {t = 1} ^ {T} \sum_ {i \in U} \frac {\mathbb {I} \left\{p _ {i} ^ {t} \leq \sum_ {j \in V} p _ {j} ^ {t} \right\}}{p _ {i} ^ {t} w _ {i} ^ {t}} \left(\frac {\frac {1}{w _ {i} ^ {t}}}{\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}}\right) \right]\right) \\ + \mathcal {O} \left(\mathbb {E} \left[ K \sum_ {t = 1} ^ {T} \sum_ {i \in [ K ]} \frac {\left(\gamma_ {i} ^ {t} - \gamma_ {i} ^ {t + 1}\right) ^ {2} \left(\psi^ {\prime} (p _ {i} ^ {t})\right) ^ {2}}{w _ {i} ^ {t}} \right]\right). \\ \end{array}
$$
Then, we are ready to bound these three terms with different configurations of Algorithm 1 in Lemma A.4.3, Lemma A.4.4, and Lemma A.4.5. The corresponding  $\psi'(x)$ 's are  $-\frac{\beta x^{\beta - 1}}{1 - \beta}$  for the  $\beta$ -Tsallis entropy regularizer,  $-\frac{1}{x}$  for the log-barrier regularizer, and  $\log(x)$  for the Shannon entropy regularizer.
Lemma A.4.3 (RESREG for  $\beta$ -Tsallis Entropy). For any  $\beta \in (0,1)$ , when using the  $\beta$ -Tsallis entropy regularizer with  $C_{\mathrm{LOG}} = \frac{162\beta}{1 - \beta}$ ,  $\alpha = \beta$ , and  $\theta = \sqrt{\frac{1 - \beta}{\beta}}$ , Algorithm 1 ensures:
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {i \in V} \frac {1}{p _ {i} ^ {t} w _ {i} ^ {t}} \right] = \mathcal {O} \left(\mathbb {S} _ {2} \left(\frac {\log T}{\beta (1 - \beta)}\right)\right),
$$
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {i \in U} \frac {\mathbb {I} \left\{p _ {i} ^ {t} \leq \sum_ {j \in V} p _ {j} ^ {t} \right\}}{p _ {i} ^ {t} w _ {i} ^ {t}} \left(\frac {\frac {1}{w _ {i} ^ {t}}}{\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}}\right) \right] = \mathcal {O} \left(\mathbb {S} _ {1} \left(\frac {| U | \log T}{\beta (1 - \beta)}\right)\right),
$$
$$
K \sum_ {t = 1} ^ {T} \sum_ {i \in [ K ]} \frac {\left(\gamma_ {i} ^ {t} - \gamma_ {i} ^ {t + 1}\right) ^ {2} \left(\frac {\beta (p _ {i} ^ {t}) ^ {\beta - 1}}{1 - \beta}\right) ^ {2}}{w _ {i} ^ {t}} = \mathcal {O} \left(\frac {\sqrt {\beta} K ^ {2}}{(1 - \beta) ^ {3 / 2}}\right),
$$
which leads to the following bound of RESREG:
$$
\operatorname {R e s R e g} = \mathcal {O} \left(\mathbb {S} _ {2} \left(\frac {\log T}{\beta (1 - \beta)}\right) + \mathbb {S} _ {1} \left(\frac {| U | \log T}{\beta (1 - \beta)}\right) + \frac {\sqrt {\beta} K ^ {2}}{(1 - \beta) ^ {3 / 2}}\right).
$$
Proof. Clearly, the first inequality relates to that of the stability term since
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {i \in V} \frac {1}{p _ {i} ^ {t} w _ {i} ^ {t}} \right] \leq \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {i \in V} \frac {\left(p _ {i} ^ {t}\right) ^ {1 - \beta}}{\beta \gamma_ {i} ^ {t}} \right] = \mathcal {O} \left(\mathbb {S} _ {2} \left(\frac {\log T}{\beta (1 - \beta)}\right)\right),
$$
where the first step uses the fact that  $w_{i}^{t} \geq \frac{\gamma_{i}^{t}\beta}{\left(p_{i}^{t}\right)^{2 - \beta}}$ , and the second step follows from the similar argument of the analysis in Lemma A.2.
Next, by direct calculation, we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \sum_ {i \in U} \frac {\mathbb {I} \left\{p _ {i} ^ {t} \leq \sum_ {j \in V} p _ {j} ^ {t} \right\}}{p _ {i} ^ {t} w _ {i} ^ {t}} \left(\frac {\frac {1}{w _ {i} ^ {t}}}{\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}}\right) \\ \leq \sum_ {t = 1} ^ {T} \sum_ {i \in U} \frac {\left(p _ {i} ^ {t}\right) ^ {1 - \beta}}{\beta \gamma_ {i} ^ {t}} \left(\mathbb {I} \left\{p _ {i} ^ {t} \leq \sum_ {j \in V} p _ {j} ^ {t} \right\} \cdot \frac {\frac {1}{w _ {i} ^ {t}}}{\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}}\right) \\ \leq \frac {2}{\sqrt {\beta (1 - \beta)}} \sum_ {i \in U} \sum_ {t = 1} ^ {T} \frac {\left(\max  \left\{p _ {i} ^ {t - 1} , ^ {1} / T \right\}\right) ^ {1 - \beta}}{\sqrt {1 + \sum_ {s <   t} \max  \left\{p _ {i} ^ {s} , ^ {1} / T \right\} ^ {1 - 2 \beta}}} \left(\mathbb {I} \left\{p _ {i} ^ {t} \leq \sum_ {j \in V} p _ {j} ^ {t} \right\} \cdot \frac {\frac {1}{w _ {i} ^ {t}}}{\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}}\right) \\ \leq \mathcal {O} \left(\sum_ {i \in U} \sqrt {\frac {\log \left(1 + \sum_ {t = 1} ^ {T} \max \left\{p _ {i} ^ {t} , 1 / T \right\} ^ {1 - 2 \beta}\right)}{\beta (1 - \beta)} \sum_ {t = 1} ^ {T} \max \left\{p _ {i} ^ {t} , 1 / T \right\} \cdot \mathbb {I} \left\{p _ {i} ^ {t} \leq \sum_ {j \in V} p _ {j} ^ {t} \right\}} \cdot \frac {\frac {1}{w _ {i} ^ {t}}}{\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}}\right) \\ = \mathcal {O} \left(\sum_ {i \in U} \sqrt {\frac {\log T}{\beta (1 - \beta)} \sum_ {t = 1} ^ {T} \left(p _ {i} ^ {t} \mathbb {I} \left\{p _ {i} ^ {t} \leq \sum_ {j \in V} p _ {j} ^ {t} \right\} \frac {\frac {1}{w _ {i} ^ {t}}}{\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}}\right)}\right), \\ \end{array}
$$
where the first step uses the fact that  $w_{i}^{t} \geq \frac{\beta\gamma_{i}^{t}}{\left(p_{i}^{t}\right)^{2 - \beta}}$ ; the second step follows from the multiplicative relation between  $p_{i}^{t}$  and  $p_{i}^{t - 1}$  (we set  $p_{i}^{0} = 1 / K = p_{i}^{1}$  for  $\forall i \in [K]$  for convenience); the third step
applies Lemma C.2.1, a weighted variant of Lemma 4.2, and again the multiplicative relation; the last step follows from  $\max \{p_i^t, 1 / T\}^{1 - 2\beta} \leq T$  for  $\beta \in (0,1)$  and  $\max \{p_i^t, 1 / T\} \leq p_i^t + 1 / T$ .
To further bound this term, we here have to use the Cauchy-Schwarz inequality which leads to the extra  $\frac{|U|}{\Delta_{\mathrm{MIN}}}$  dependency in the final regret bound:
$$
\begin{array}{l} \mathcal {O} \left(\sqrt {\frac {| U | \log T}{\beta (1 - \beta)} \sum_ {t = 1} ^ {T} \sum_ {i \in U} \left(p _ {i} ^ {t} \mathbb {I} \left\{p _ {i} ^ {t} \leq \sum_ {j \in V} p _ {j} ^ {t} \right\} + \frac {1}{T}\right) \frac {\frac {1}{w _ {i} ^ {t}}}{\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}}}\right) \\ \leq \mathcal {O} \left(\sqrt {\frac {| U | \log T}{\beta (1 - \beta)} \sum_ {t = 1} ^ {T} \left(\left(\sum_ {j \in V} p _ {j} ^ {t}\right) + \frac {1}{T}\right) \sum_ {i \in U} \frac {\frac {1}{w _ {i} ^ {t}}}{\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}}}\right) \\ = \mathcal {O} \left(\sqrt {\frac {| U | \log T}{\beta (1 - \beta)} \sum_ {t = 1} ^ {T} \left(\frac {1}{T} + \sum_ {j \in V} p _ {j} ^ {t}\right)}\right), \\ \end{array}
$$
where the first step uses the property of indicator that  $p_i^t \mathbb{I} \left\{ p_i^t \leq \sum_{j \in V} p_j^t \right\} \leq \sum_{j \in V} p_j^t$ . Taking the expectation finishes the proof for the second inequality.
For the last inequality in the statement, for any round  $t$ , we have
$$
\begin{array}{l} K \sum_ {i \in [ K ]} \frac {\left(\gamma_ {i} ^ {t} - \gamma_ {i} ^ {t + 1}\right) ^ {2} \left(\frac {\beta \left(p _ {i} ^ {t}\right) ^ {\beta - 1}}{1 - \beta}\right) ^ {2}}{w _ {i} ^ {t}} \\ \leq \frac {K \beta^ {2}}{(1 - \beta) ^ {2}} \sum_ {i \in [ K ]} \frac {\left(\gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t}\right) ^ {2} \left(p _ {i} ^ {t}\right) ^ {2 - \beta}}{\beta \gamma_ {i} ^ {t} \left(p _ {i} ^ {t}\right) ^ {2 - 2 \beta}} \\ = \frac {K \beta}{(1 - \beta) ^ {2}} \sum_ {i \in [ K ]} \frac {\left(\gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t}\right) ^ {2} (p _ {i} ^ {t}) ^ {\beta}}{\gamma_ {i} ^ {t}} \\ \leq \mathcal {O} \left(\frac {K}{(1 - \beta)} \sum_ {i \in [ K ]} \frac {\left(\gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t}\right) \left(\max  \left\{p _ {i} ^ {t} , 1 / T \right\}\right) ^ {1 - 2 \beta} \left(p _ {i} ^ {t}\right) ^ {\beta}}{\gamma_ {i} ^ {t} \gamma_ {i} ^ {t + 1}}\right) \\ \leq \mathcal {O} \left(\frac {K}{(1 - \beta)} \sum_ {i \in [ K ]} \frac {\left(\gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t}\right) \left(\max  \left\{p _ {i} ^ {t} , ^ {1} / T \right\}\right) ^ {1 - \beta}}{\gamma_ {i} ^ {t} \gamma_ {i} ^ {t + 1}}\right) \quad (p _ {i} ^ {t} \leq \max  \left\{p _ {i} ^ {t}, ^ {1} / T \right\}) \\ \leq \mathcal {O} \left(\frac {K}{(1 - \beta)} \sum_ {i \in [ K ]} \left(\frac {1}{\gamma_ {i} ^ {t}} - \frac {1}{\gamma_ {i} ^ {t + 1}}\right)\right), \quad (\max  \left\{p _ {i} ^ {t}, ^ {1} / T \right\} \leq 1) \\ \end{array}
$$
where the first step applies  $w_{i}^{t}\geq \frac{\beta\gamma_{i}^{t}}{\left(p_{i}^{t}\right)^{2 - \beta}}$  and Eq. (21).
Taking the summation over all the rounds, we have
$$
\sum_ {t = 1} ^ {T} \frac {K}{(1 - \beta)} \sum_ {i \in [ K ]} \left(\frac {1}{\gamma_ {i} ^ {t}} - \frac {1}{\gamma_ {i} ^ {t + 1}}\right) = \frac {K}{(1 - \beta)} \sum_ {i \in [ K ]} \sum_ {t = 1} ^ {T} \left(\frac {1}{\gamma_ {i} ^ {t}} - \frac {1}{\gamma_ {i} ^ {t + 1}}\right) = \mathcal {O} \left(\frac {K ^ {2} \beta^ {1 / 2}}{(1 - \beta) ^ {3 / 2}}\right),
$$
Lemma A.4.4 (RESREG for Log-barrier). When using the log-barrier regularizer with  $C_{\mathrm{LOG}} = 162$ ,  $\alpha = 0$ , and  $\theta = \sqrt{\frac{1}{\log T}}$ , Algorithm 1 ensures:
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {i \in V} \frac {1}{p _ {i} ^ {t} w _ {i} ^ {t}} \right] = \mathcal {O} \left(\mathbb {S} _ {2} (\log T)\right),
$$
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {i \in U} \frac {\mathbb {I} \left\{p _ {i} ^ {t} \leq \sum_ {j \in V} p _ {j} ^ {t} \right\}}{p _ {i} ^ {t} w _ {i} ^ {t}} \left(\frac {\frac {1}{w _ {i} ^ {t}}}{\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}}\right) \right] = \mathcal {O} \left(\mathbb {S} _ {1} \left(| U | \log T\right)\right),
$$
$$
K \sum_ {t = 1} ^ {T} \sum_ {i \in [ K ]} \frac {\left(\gamma_ {i} ^ {t} - \gamma_ {i} ^ {t + 1}\right) ^ {2} \left(\frac {1}{p _ {i} ^ {t}}\right) ^ {2}}{w _ {i} ^ {t}} = \mathcal {O} \left(\frac {K ^ {2}}{\sqrt {\log T}}\right),
$$
$$
\operatorname {R E S R E G} = \mathcal {O} \left(\mathbb {S} _ {2} (\log T) + \mathbb {S} _ {1} (| U | \log T) + \frac {K ^ {2}}{\sqrt {\log T}}\right).
$$
Proof. By direct calculation, we have
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {i \in V} \frac {1}{p _ {i} ^ {t} w _ {i} ^ {t}} \right] \leq \mathbb {E} \left[ \sum_ {i \in V} \sum_ {t = 1} ^ {T} \frac {p _ {i} ^ {t}}{\gamma_ {i} ^ {t}} \right] = \mathcal {O} \left(\mathbb {E} \left[ \sqrt {\log T} \sum_ {i \in V} \sum_ {t = 1} ^ {T} \frac {p _ {i} ^ {t}}{\gamma_ {i} ^ {t + 1}} \right]\right) = \mathcal {O} \left(\mathbb {S} _ {2} (\log T)\right),
$$
where the second step uses the same arguments used in Lemma A.2.2.
Following the same idea used for the  $\beta$ -Tsallis entropy regularizer, we have
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {i \in U} \frac {\mathbb {I} \left\{p _ {i} ^ {t} \leq \sum_ {j \in V} p _ {j} ^ {t} \right\}}{p _ {i} ^ {t} w _ {i} ^ {t}} \left(\frac {\frac {1}{w _ {i} ^ {t}}}{\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}}\right) \right] \\ = \mathcal {O} \left(\mathbb {E} \left[ \sum_ {i \in U} \sqrt {\log T \cdot \sum_ {t = 1} ^ {T} p _ {i} ^ {t} \mathbb {I}} \left\{p _ {i} ^ {t} \leq \sum_ {j \in V} p _ {j} ^ {t} \right\} \left(\frac {\frac {1}{w _ {i} ^ {t}}}{\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}}\right) \right]\right) \\ = \mathcal {O} \left(\mathbb {E} \left[ \sqrt {| U | \log T \cdot \sum_ {t = 1} ^ {T} \sum_ {i \in U} \left(\sum_ {i \in V} p _ {i} ^ {t}\right) \left(\frac {\frac {1}{w _ {i} ^ {t}}}{\sum_ {i \in U} \frac {1}{w _ {i} ^ {t}}}\right)} \right]\right) \\ = \mathcal {O} \left(\mathbb {S} _ {1} \left(| U | \log T\right)\right), \\ \end{array}
$$
where the second step follows from Lemma 4.2 and  $w_{i}^{t} \geq \gamma_{i}^{t} / (p_{i}^{t})^{2}$ ; the second step uses the Cauchy-Schwarz inequality; the last follows from the definition of the self-bounding quantity  $\mathbb{S}_1$  in Eq. (14).
Finally, by direct calculation, for any round  $t$ , we have