SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the following, we show an application of Lemma C.4.1 in the FTRL framework.
Lemma C.4.2. Let  $\mathcal{I} \subseteq [K]$  be an index set. Suppose  $\nabla \phi_{\mathcal{I}}^{t+1}(q) = \nabla \phi_{\mathcal{I}}^t(p) - \widehat{\ell}_{\mathcal{I}}^t + c \cdot \mathbf{1}_{\mathcal{I}}$  where  $\phi^t$  is the regularizer with  $\phi^t(x) = \sum_{i \in S} \gamma_i^t \psi(x_i)$ ,  $\psi : \Omega \to \mathbb{R}$  for  $\Omega \subseteq \mathbb{R}$  is a strictly convex function such that  $\gamma_i^t \psi$  satisfies all conditions in Lemma C.4.1 for all  $t, i$ ,  $\widehat{\ell}_{\mathcal{I}}^t$  is the loss estimator, and  $c \in \mathbb{R}$ . If  $p, q$  satisfy  $\sum_{i \in \mathcal{I}} p_i = \sum_{i \in \mathcal{I}} q_i$ , then we have:
$$
c \leq \left(\sum_ {i \in \mathcal {I}} \frac {1}{\gamma_ {i} ^ {t + 1} \psi^ {\prime \prime} \left(p _ {i}\right)}\right) ^ {- 1} \left(\sum_ {i \in \mathcal {I}} \frac {\left(\gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t}\right) \psi^ {\prime} \left(p _ {i}\right) + \widehat {\ell} _ {i} ^ {t}}{\gamma_ {i} ^ {t + 1} \psi^ {\prime \prime} \left(p _ {i}\right)}\right), \tag {80}
$$
$$
c \geq \left(\sum_ {i \in \mathcal {I}} \frac {1}{\gamma_ {i} ^ {t} \psi^ {\prime \prime} (q _ {i})}\right) ^ {- 1} \left(\sum_ {i \in \mathcal {I}} \frac {\left(\gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t}\right) \psi^ {\prime} (q _ {i}) + \widehat {\ell_ {i} ^ {t}}}{\gamma_ {i} ^ {t} \psi^ {\prime \prime} (q _ {i})}\right). \tag {81}
$$
Consequently, for  $\nabla \phi_{\mathcal{I}}^t (q) = \nabla \phi_{\mathcal{I}}^t (p) - \widehat{\ell}_{\mathcal{I}}^t +c\cdot \mathbf{1}_{\mathcal{I}}$  where all definitions remain the same as above, we have
$$
c \leq \left(\sum_ {i \in \mathcal {I}} \frac {1}{\gamma_ {i} ^ {t} \psi^ {\prime \prime} \left(p _ {i}\right)}\right) ^ {- 1} \left(\sum_ {i \in \mathcal {I}} \frac {\widehat {\ell} _ {i} ^ {t}}{\gamma_ {i} ^ {t} \psi^ {\prime \prime} \left(p _ {i}\right)}\right), \tag {82}
$$
$$
c \geq \left(\sum_ {i \in \mathcal {I}} \frac {1}{\gamma_ {i} ^ {t} \psi^ {\prime \prime} \left(q _ {i}\right)}\right) ^ {- 1} \left(\sum_ {i \in \mathcal {I}} \frac {\widehat {\ell} _ {i} ^ {t}}{\gamma_ {i} ^ {t} \psi^ {\prime \prime} \left(q _ {i}\right)}\right). \tag {83}
$$
Proof. By directly applying Eq. (76) in Lemma C.4.1 with  $f(\cdot) = \gamma_i^t\psi (\cdot), g(\cdot) = \gamma_i^{t + 1}\psi (\cdot), p = p_i, q = q_i$ , and  $z = \widehat{\ell}_i^t -c$ , we have
$$
q _ {i} - p _ {i} \geq \frac {\left(\gamma_ {i} ^ {t} - \gamma_ {i} ^ {t + 1}\right) \psi^ {\prime} \left(p _ {i}\right) - \widehat {\ell} _ {i} ^ {t} + c}{\gamma_ {i} ^ {t + 1} \psi^ {\prime \prime} \left(p _ {i}\right)}. \tag {84}
$$
Since  $\sum_{i\in \mathcal{I}}p_i = \sum_{i\in \mathcal{I}}q_i$ , we sum Eq. (84) over all  $i\in \mathcal{I}$  and then rearrange to get Eq. (80). On the other hand, by applying Eq. (79) in Lemma C.4.1 with the same choices, we have
$$
q _ {i} - p _ {i} \leq \frac {\left(\gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t}\right) \psi^ {\prime} \left(q _ {i}\right) - \widehat {\ell} _ {i} ^ {t} + c}{\gamma_ {i} ^ {t} \psi^ {\prime \prime} \left(q _ {i}\right)}. \tag {85}
$$
We again sum Eq. (85) over all  $i \in \mathcal{I}$  and then rearrange to get Eq. (81).
Lemma C.4.2 is applicable to the analysis of the Decoupled-Tsallis-INF algorithm. For the analysis of the MAB algorithms, since we add some extra log-barrier to the regularizers, we will need the following similar results.
Lemma C.4.3. Under the same setup of Lemma C.4.2 and let  $\theta \in \mathbb{R}_{>0}$  be given, if the regularizer is  $\phi^t (x) = \sum_{i\in S}\gamma_i^t\psi (x_i) + \theta r(x_i)$  where  $\psi (\cdot),r(\cdot)$  are strictly convex and twice differentiable,  $\gamma_i^t\psi (\cdot) + \theta r(\cdot)$  satisfies all conditions in Lemma C.4.1 for all  $t,i$  , we have
$$
c \leq \left(\sum_ {i \in \mathcal {I}} \frac {1}{\gamma_ {i} ^ {t + 1} \psi^ {\prime \prime} (p _ {i}) + \theta r ^ {\prime \prime} (p _ {i})}\right) ^ {- 1} \left(\sum_ {i \in \mathcal {I}} \frac {\left(\gamma^ {t + 1} - \gamma^ {t}\right) \psi^ {\prime} (p _ {i}) + \widehat {\ell} _ {i} ^ {t}}{\gamma_ {i} ^ {t + 1} \psi^ {\prime \prime} (p _ {i}) + \theta r ^ {\prime \prime} (p _ {i})}\right),
$$
$$
c \geq \left(\sum_ {i \in \mathcal {I}} \frac {1}{\gamma_ {i} ^ {t} \psi^ {\prime \prime} (q _ {i}) + \theta r ^ {\prime \prime} (q _ {i})}\right) ^ {- 1} \left(\sum_ {i \in \mathcal {I}} \frac {\left(\gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t}\right) \psi^ {\prime} (q _ {i}) + \widehat {\ell_ {i} ^ {t}}}{\gamma_ {i} ^ {t} \psi^ {\prime \prime} (q _ {i}) + \theta r ^ {\prime \prime} (q _ {i})}\right),
$$
$$
c \leq \left(\sum_ {i \in \mathcal {I}} \frac {1}{\gamma_ {i} ^ {t} \psi^ {\prime \prime} (p _ {i}) + \theta r ^ {\prime \prime} (p _ {i})}\right) ^ {- 1} \left(\sum_ {i \in \mathcal {I}} \frac {\left(\gamma^ {t + 1} - \gamma^ {t}\right) \psi^ {\prime} (q _ {i}) + \widehat {\ell} _ {i} ^ {t}}{\gamma_ {i} ^ {t} \psi^ {\prime \prime} (p _ {i}) + \theta r ^ {\prime \prime} (p _ {i})}\right),
$$
$$
c \geq \left(\sum_ {i \in \mathcal {I}} \frac {1}{\gamma_ {i} ^ {t + 1} \psi^ {\prime \prime} (q _ {i}) + \theta r ^ {\prime \prime} (q _ {i})}\right) ^ {- 1} \left(\sum_ {i \in \mathcal {I}} \frac {\left(\gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t}\right) \psi^ {\prime} (p _ {i}) + \widehat {\ell} _ {i} ^ {t}}{\gamma_ {i} ^ {t + 1} \psi^ {\prime \prime} (q _ {i}) + \theta r ^ {\prime \prime} (q _ {i})}\right).
$$
Proof. For any  $i \in \mathcal{I}$ , applying Eq. (76) in Lemma C.4.1 with  $f = \gamma_i^{t+1} \psi(\cdot) + \theta r(\cdot)$ ,  $g = \gamma_i^t \psi(\cdot) + \theta r(\cdot)$ ,  $p = p_i$ ,  $q = q_i$ , and  $z = \widehat{\ell}_i^t - c$  yields that
$$
q _ {i} - p _ {i} \geq \frac {\gamma_ {i} ^ {t} \psi^ {\prime} (p _ {i}) + \theta r ^ {\prime} (p _ {i}) - \gamma_ {i} ^ {t + 1} \psi^ {\prime} (p _ {i}) - \theta r ^ {\prime} (p _ {i}) - \widehat {\ell} _ {i} ^ {t} + c}{\gamma_ {i} ^ {t + 1} \psi^ {\prime \prime} (p _ {i}) + \theta r ^ {\prime \prime} (p _ {i})} = \frac {\left(\gamma_ {i} ^ {t} - \gamma_ {i} ^ {t + 1}\right) \psi^ {\prime} (p _ {i}) - \widehat {\ell} _ {i} ^ {t} + c}{\gamma_ {i} ^ {t + 1} \psi^ {\prime \prime} (p _ {i}) + \theta r ^ {\prime \prime} (p _ {i})}.
$$
By the condition that  $\sum_{i\in \mathcal{I}}p_i = \sum_{i\in \mathcal{I}}q_i$ , taking the summation on both sides gives us:
$$
0 \geq \sum_ {i \in \mathcal {I}} \frac {\left(\gamma_ {i} ^ {t} - \gamma_ {i} ^ {t + 1}\right) \psi^ {\prime} (p _ {i}) - \widehat {\ell} _ {i} ^ {t} + c}{\gamma_ {i} ^ {t + 1} \psi^ {\prime \prime} (p _ {i}) + \theta r ^ {\prime \prime} (p _ {i})}
$$
$$
\xlongequal {\mathrm {R e a r r a n g i n g}} c \cdot \sum_ {i \in \mathcal {I}} \frac {1}{\gamma_ {i} ^ {t + 1} \psi^ {\prime \prime} (p _ {i}) + \theta r ^ {\prime \prime} (p _ {i})} \leq \sum_ {i \in \mathcal {I}} \frac {\left(\gamma_ {i} ^ {t + 1} - \gamma_ {i} ^ {t}\right) \psi^ {\prime} (p _ {i}) + \widehat {\ell} _ {i} ^ {t}}{\gamma_ {i} ^ {t + 1} \psi^ {\prime \prime} (p _ {i}) + \theta r ^ {\prime \prime} (p _ {i})},
$$
which proves the first inequality. Similarly, the other inequalities in the statement can be obtained by applying Eq. (77), Eq. (78), and Eq. (79) in Lemma C.4.1.
C.5 Analysis of FTRL Framework and Bregman Divergence
Lemma C.5.1 (Lemma 17 in Ito [2021]). For any regularizers  $\phi, \phi'$ , and  $L, L' \in \mathbb{R}^K$ , denote  $z, z'$  as
$$
z = \operatorname * {a r g m i n} _ {x \in \Omega} \langle L, x \rangle + \phi (x), \quad z ^ {\prime} = \operatorname * {a r g m i n} _ {x \in \Omega} \langle L ^ {\prime}, x \rangle + \phi^ {\prime} (x),
$$
where  $\Omega = \{x\in \mathcal{D}\mid Ax = b\}$  for some convex set  $\mathcal{D}$  and  $A\in \mathbb{R}^{K\times K},b\in \mathbb{R}^K$  . Define the divergences  $F^1$  and  $F^2$  as:
$$
F ^ {1} (x, y) = \phi (x) - \phi^ {\prime} (y) - \langle \nabla \phi^ {\prime} (y), x - y \rangle , F ^ {2} (y, x) = \phi^ {\prime} (y) - \phi (x) - \langle \nabla \phi (x), y - x \rangle .
$$
We then have  $F^1(z, z') + F^2(z', z) = -\langle L' - L, z' - z \rangle$ .
Since the following lemma only uses standard Bregman divergence and the analysis is for a given round  $t$ , we drop  $t$  for  $D^{t}(x,y)$  and use  $D(x,y)$ .
Lemma C.5.2. For any  $L, \ell \in \mathbb{R}^K$  and a Legendre regularizer  $\phi$ , let  $p, q$  be the vectors such that
$$
p = \operatorname * {a r g m i n} _ {x \in \Omega_ {K}} \langle x, L \rangle + \phi (x), a n d q = \operatorname * {a r g m i n} _ {x \in \Omega_ {K}} \langle x, L + \ell \rangle + \phi (x).
$$
$$
\langle p - z, \ell \rangle - D (z, p) \leq \langle p - q, \ell \rangle - D (q, p), \forall z \in \Omega_ {K}.
$$
Moreover, there exists an intermediate point  $\xi = \eta \cdot p + (1 - \eta) \cdot q$  for some  $\eta \in [0, 1]$ , such that
$$
\langle p - q, \ell \rangle - D (q, p) \leq 2 \| \ell \| _ {\nabla^ {- 2} \phi (\xi)} ^ {2}.
$$
Proof. Since  $\phi$  is Legendre, we have  $\nabla \phi(p) = -L + c \cdot \mathbf{1}_K$  for some  $c \in \mathbb{R}$ . Thus, it holds for any  $z \in \Omega_K$  that
$$
\begin{array}{l} \langle p - z, \ell \rangle - D (z, p) \\ = \langle p - z, \ell \rangle - \phi (z) + \phi (p) + \langle \nabla \phi (p), z - p \rangle \\ = \langle p - z, \ell \rangle - \phi (z) + \phi (p) + \langle - L, z - p \rangle \\ = \langle p, L + \ell \rangle + \phi (p) - \langle z, L + \ell \rangle - \phi (z) \\ = \langle p, L \rangle + \phi (p) + \langle p, \ell \rangle - \langle z, L + \ell \rangle - \phi (z). \\ \end{array}
$$
Since  $q$  maximizes the last two terms, we have for any  $z \in \Omega_K$ ,
$$
\langle p - z, \ell \rangle - D (z, p) \leq \langle p - q, \ell \rangle - D (q, p).
$$
Let  $F(x) = \langle x, L \rangle + \phi(x)$  and  $G(x) = \langle x, L + \ell \rangle + \phi(x)$  so that  $p$  is the minimizer of  $F(x)$  and  $q$  is the minimizer of  $G(x)$ . By  $\nabla \phi(p) = -L + c \cdot \mathbf{1}_K$  and direct calculation, we have
$$
\begin{array}{l} \langle p - q, \ell \rangle - D (q, p) = F (p) + \langle p, \ell \rangle - G (q) = G (p) - G (q) \\ = \langle p - q, \nabla G (q) \rangle + \frac {1}{2} \| p - q \| _ {\nabla^ {2} G (\xi)} ^ {2} \\ \geq \frac {1}{2} \| p - q \| _ {\nabla^ {2} \phi (\xi)} ^ {2}, \\ \end{array}
$$
where we apply Taylor's expansion with  $\xi = \eta \cdot p + (1 - \eta) \cdot q$  for some  $\eta \in [0,1]$  being an intermediate point between  $p$  and  $q$ , followed by the first-order optimality condition. On the other hand, we have
$$
\begin{array}{l} \langle p - q, \ell \rangle - D (q, p) = \langle p - q, \ell \rangle + F (p) - F (q) \\ \leq \langle p - q, \ell \rangle \\ \leq \left\| \ell \right\| _ {\nabla^ {- 2} \phi (\xi)} \left\| p - q \right\| _ {\nabla^ {- 2} \phi (\xi)}, \\ \end{array}
$$
where the second step follows from the optimality of  $p$ , and the last step applies Hölder's inequality. Combining these two inequalities gives us that
$$
\langle p - q, \ell \rangle - D (q, p) \leq 2 \| \ell \| _ {\nabla^ {- 2} \phi (\xi)} ^ {2},
$$
for some  $\eta \in [0,1]$  such that  $\xi = \eta \cdot p + (1 - \eta)\cdot q$
Lemma C.5.3. Consider the following regularizer for any  $\beta \in (0,1)$  and learning rate  $\gamma_{i} > 0$ :
$$
\phi (x) = - \frac {1}{1 - \beta} \sum_ {i \in [ K ]} \gamma_ {i} x _ {i} ^ {\beta}.
$$
For any  $q, x \in \mathbb{R}_{\geq 0}^{K}$  and  $\ell \in \mathbb{R}^{K}$  satisfying  $\frac{(q_i)^{1 - \beta}\ell_i}{\gamma_i} \geq \frac{\beta}{1 - \beta}\left(e^{\frac{\beta - 1}{\beta}} - 1\right)$  for all arm  $i$ , we have
$$
\langle q - x, \ell \rangle - D (x, q) \leq \sum_ {i \in [ K ]} \frac {\left(q _ {i}\right) ^ {2 - \beta}}{\beta \gamma_ {i}} \left(\ell_ {i}\right) ^ {2}.
$$
Proof. Let  $w = \operatorname{argmin}_{x \in \mathbb{R}_{>0}^{K}} \langle q - x, \ell \rangle - D(x, q)$  be the maximizer. By setting the gradient to zero, we have  $-\ell - \nabla \phi(w) + \nabla \phi(q) = 0$ , which equivalently implies that for any arm  $i \in [K]$ ,
$$
\frac {1}{\left(w _ {i}\right) ^ {1 - \beta}} = \frac {1}{\left(q _ {i}\right) ^ {1 - \beta}} + \frac {1 - \beta}{\gamma_ {i} \beta} \ell_ {i}. \tag {86}
$$
By direct calculation, we have
$$
\begin{array}{l} \langle q - w, \ell \rangle - D (w, q) \\ = \phi (q) - \phi (w) - \langle \nabla \phi (w), q - w \rangle \\ = \frac {1}{1 - \beta} \sum_ {i \in [ K ]} \gamma_ {i} \left((w _ {i}) ^ {\beta} - (q _ {i}) ^ {\beta} + \beta (w _ {i}) ^ {\beta - 1} (q _ {i} - w _ {i})\right) \\ = \frac {1}{1 - \beta} \sum_ {i \in [ K ]} \gamma_ {i} \left((1 - \beta) (w _ {i}) ^ {\beta} - (1 - \beta) (q _ {i}) ^ {\beta} + \beta \left((w _ {i}) ^ {\beta - 1} - (q _ {i}) ^ {\beta - 1}\right) q _ {i}\right) \\ = \frac {1}{1 - \beta} \sum_ {i \in [ K ]} \gamma_ {i} \left((1 - \beta) (w _ {i}) ^ {\beta} - (1 - \beta) (q _ {i}) ^ {\beta} + \frac {1 - \beta}{\gamma_ {i}} \ell_ {i} q _ {i}\right) \\ = \sum_ {i \in [ K ]} \gamma_ {i} \left(\left(w _ {i}\right) ^ {\beta} - \left(q _ {i}\right) ^ {\beta} + \frac {\ell_ {i} q _ {i}}{\gamma_ {i}}\right), \tag {87} \\ \end{array}
$$
where the first step uses  $-\ell - \nabla \phi(w) + \nabla \phi(q) = 0$ , and the fifth step uses Eq. (86).
Moreover, we have for any arm  $i$
$$
\begin{array}{l} (w _ {i}) ^ {\beta} = (q _ {i}) ^ {\beta} \left(\frac {(q _ {i}) ^ {1 - \beta}}{(w _ {i}) ^ {1 - \beta}}\right) ^ {\frac {\beta}{\beta - 1}} \\ = \left(q _ {i}\right) ^ {\beta} \left(1 + \frac {1 - \beta}{\gamma_ {i} \beta} \left(q _ {i}\right) ^ {1 - \beta} \ell_ {i}\right) ^ {\frac {\beta}{\beta - 1}} \\ \leq \left(q _ {i}\right) ^ {\beta} \left(1 - \frac {1}{\gamma_ {i}} \left(q _ {i}\right) ^ {1 - \beta} \ell_ {i} + \frac {1}{\beta \left(\gamma_ {i}\right) ^ {2}} \left(q _ {i}\right) ^ {2 - 2 \beta} \left(\ell_ {i}\right) ^ {2}\right) \\ = \left(q _ {i}\right) ^ {\beta} - \frac {1}{\gamma_ {i}} q _ {i} \ell_ {i} + \frac {1}{\beta (\gamma_ {i}) ^ {2}} \left(q _ {i}\right) ^ {2 - \beta} \left(\ell_ {i}\right) ^ {2}, \tag {88} \\ \end{array}
$$
where the second step uses Eq. (86), and the third step follows from the fact that  $(1 + z)^{\alpha} < 1 + \alpha z + \alpha (\alpha - 1)z^{2}$  for  $\alpha < 0$  and  $z \geq \exp \left(\frac{1}{\alpha}\right) - 1$  with  $\alpha = \frac{\beta}{\beta - 1} = 1 - \frac{1}{1 - \beta} < 0$  for  $\beta \in (0,1)$  and  $z = \frac{1 - \beta}{\gamma_i\beta} (q_i)^{1 - \beta}\ell_i \geq e^{\frac{\beta - 1}{\beta}} - 1 = \exp \left(\frac{1}{\alpha}\right) - 1$ .
Plugging Eq. (88) into Eq. (87) yields
$$
\langle q - w, \ell \rangle - D (w, q) \leq \sum_ {i \in [ K ]} \gamma_ {i} \left(\frac {\ell_ {i} q _ {i}}{\gamma_ {i}} - \frac {1}{\gamma_ {i}} q _ {i} \ell_ {i} + \frac {1}{\beta (\gamma_ {i}) ^ {2}} (q _ {i}) ^ {2 - \beta} (\ell_ {i}) ^ {2}\right) = \sum_ {i \in [ K ]} \frac {(q _ {i}) ^ {2 - \beta}}{\beta \gamma_ {i}} (\ell_ {i}) ^ {2},
$$