SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} h _ {t + 1} - 3 h _ {t} \\ \leq \frac {1 0}{9} \sum_ {i \in \Omega_ {t}} \left(\frac {\beta_ {t + 1}}{\beta_ {t}} - 1\right) \log (T / k) \left(\frac {k}{T} + \left(1 - \frac {k}{T}\right) q _ {t + 1, i}\right) \log \left(\frac {1}{p _ {t + 1 , i}}\right) \\ \leq \frac {1 0 k}{9} \left(\frac {\beta_ {t + 1}}{\beta_ {t}} - 1\right) \log (T / k) \sum_ {i = 1} ^ {k} \left(\frac {1}{T} + \left(1 - \frac {k}{T}\right) q _ {t + 1, i}\right) \log \left(\frac {1}{p _ {t + 1 , i}}\right) \\ \leq \frac {2 0 k}{9} \left(\frac {\beta_ {t + 1}}{\beta_ {t}} - 1\right) \log (T / k) h _ {t + 1}, \\ \end{array}
$$
where the second inequality follows by Lemma 6 and the last inequality follows by the definition of  $h_{t + 1}$ .
Finally we are ready to prove Theorem 4.
Proof of Theorem 4. Fix  $i^{*} \in [k]$  and define  $p^{*} \in \mathcal{P}_{k}$  by
$$
\begin{array}{l} \operatorname {R e g} _ {T} (i ^ {*}) = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \ell_ {t A _ {t}} - \sum_ {t = 1} ^ {T} \ell_ {t i ^ {*}} \right] = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle \ell_ {t}, p _ {t} - e _ {i ^ {*}} \right\rangle \right] \\ = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle \ell_ {t}, q _ {t} - e _ {i ^ {*}} \right\rangle \right] + \mathbb {E} \left[ \gamma \sum_ {t = 1} ^ {T} \left\langle \ell_ {t}, \frac {1}{k} \mathbf {1} - q _ {t} \right\rangle \right] \\ \leq \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \langle \ell_ {t}, q _ {t} - p ^ {*} \rangle \right] + \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \langle \ell_ {t}, p ^ {*} - e _ {i ^ {*}} \rangle \right] + \gamma T \\ \leq \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \langle \widehat {y} _ {t}, q _ {t} - p ^ {*} \rangle \right] + 2 k, \\ \end{array}
$$
where the first inequality follows since  $p_t = (1 - \gamma) q_t + \frac{\gamma}{k} \mathbf{1}$  and the last inequality follows by the definition of  $p^*$  and  $\gamma = \frac{k}{T}$ . By the standard analysis of the FTRL described in Section 3,
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \langle \widehat {y} _ {t}, q _ {t} - p ^ {*} \rangle \leq \sum_ {t = 1} ^ {T} \left(\Phi_ {t} \left(q _ {t + 1}\right) - \Phi_ {t + 1} \left(q _ {t + 1}\right)\right) + \Phi_ {t + 1} \left(p ^ {*}\right) - \Phi_ {1} \left(q _ {1}\right) \\ + \sum_ {t = 1} ^ {T} \left(\left\langle q _ {t} - q _ {t + 1}, \widehat {y} _ {t} \right\rangle - D _ {\Phi_ {t}} \left(q _ {t + 1}, q _ {t}\right)\right). \\ \end{array}
$$
We first consider the penalty term. Since  $\Phi_t = \frac{1}{\eta_t} \psi^{\mathsf{nS}} + 4 \psi^{\mathsf{LB}}$ ,
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left(\Phi_ {t} \left(q _ {t + 1}\right) - \Phi_ {t + 1} \left(q _ {t + 1}\right)\right) + \Phi_ {t + 1} \left(p ^ {*}\right) - \Phi_ {1} \left(q _ {1}\right) \\ \leq \sum_ {t = 1} ^ {T} \left(\frac {1}{\eta_ {t + 1}} - \frac {1}{\eta_ {t}}\right) H (q _ {t + 1}) + \frac {H (q _ {1})}{\eta_ {1}} + 4 \sum_ {i = 1} ^ {k} \log \left(\frac {1}{p _ {i} ^ {*}}\right) \\ \leq \sum_ {t = 1} ^ {T} \left(\frac {1}{\eta_ {t + 1}} - \frac {1}{\eta_ {t}}\right) H \left(q _ {t + 1}\right) + \frac {\log k}{\eta_ {1}} + 4 k \log T, \\ \end{array}
$$
For the stability term, by Lemma 3 with  $\delta = 2$
$$
\sum_ {t = 1} ^ {T} \left(\langle q _ {t} - q _ {t + 1}, \widehat {y} _ {t} \rangle - D _ {\Phi_ {t}} (q _ {t + 1}, q _ {t})\right) \leq 2 \sum_ {t = 1} ^ {T} \eta_ {t} \nu_ {t}.
$$
We will confirm that the assumptions for Part (I) of Theorem 1 are indeed satisfied. By the definition of the learning rate in (8) and  $\nu_{t} \leq \beta_{t} / 2$ ,
$$
\frac {\sqrt {c _ {2}}}{c _ {1}} \left(\beta_ {1} + \beta_ {t}\right) \geq 9 \left(\beta_ {1} + \nu_ {t}\right) \geq \beta_ {1} + \nu_ {t}.
$$
Hence stability condition (S1) of Theorem 1 is satisfied and one can also see that the other assumptions are trivially satisfied. Hence, by Part (I) of Theorem 1,
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left(\frac {1}{\eta_ {t + 1}} - \frac {1}{\eta_ {t}}\right) + 2 \sum_ {t = 1} ^ {T} \eta_ {t} \nu_ {t} \leq 2 \left(c _ {1} + \frac {2}{c _ {1}} \log \left(1 + \sum_ {s = 1} ^ {T} \frac {\nu_ {s}}{\beta_ {1}}\right)\right) \sqrt {c _ {2} + \sum_ {t = 1} ^ {T + 1} \nu_ {t} h _ {t + 1}} \\ \leq 2 \left(c _ {1} + \frac {2}{c _ {1}} \log \left(1 + \frac {T ^ {2}}{\beta_ {1}}\right)\right) \sqrt {c _ {2} + \sum_ {t = 1} ^ {T + 1} \nu_ {t} h _ {t + 1}}, \\ = 2 \sqrt {2 \log \left(1 + \frac {T ^ {2}}{\beta_ {1}}\right)} \sqrt {c _ {2} + \sum_ {t = 1} ^ {T + 1} \nu_ {t} h _ {t + 1}}, \\ \end{array}
$$
where in the last inequality we used  $\nu_{t} \leq T$  and in the equality we set  $c_{1} = \sqrt{2\log\left(1 + \frac{T^{2}}{\beta_{1}}\right)}$ .
By summing up the above arguments and Jensen's inequality, we have
$$
\begin{array}{l} \operatorname {R e g} _ {T} \left(i ^ {*}\right) \leq \mathbb {E} \left[ 2 \sqrt {2 \log \left(1 + \frac {T ^ {2}}{\beta_ {1}}\right)} \sqrt {c _ {2} + \sum_ {t = 1} ^ {T + 1} \nu_ {t} h _ {t + 1}} \right] + 2 k + 4 k \log T + \frac {\log k}{\eta_ {1}} \\ \leq 2 \sqrt {2 \log \left(1 + \frac {T ^ {2}}{\beta_ {1}}\right)} \sqrt {c _ {2} + \mathbb {E} \left[ \sum_ {t = 1} ^ {T + 1} \nu_ {t} h _ {t + 1} \right]} + 2 k + 4 k \log T + 1 5 k \log k \\ \leq 2 \sqrt {2 \log \left(1 + \frac {T ^ {2}}{\beta_ {1}}\right)} \sqrt {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} v _ {t} h _ {t + 1} \right]} + O (k \log T). \tag {36} \\ \end{array}
$$
Adversarial regime We first consider the adversarial regime. Recall that  $\mathbb{E}\left[\sqrt{\sum_{t=1}^{T}\nu_t}\right] \leq \mathbb{E}\left[\sqrt{\sum_{t=1}^{T}\omega_t}\right] \leq \sqrt{L_2}$  as was done in the proof of Corollary 3. Hence (36) with  $h_t \leq 2\log k$  (since  $T \geq 2k$ ) yields that
$$
\operatorname {R e g} _ {T} \leq 4 \sqrt {L _ {2} \log (k) \log \left(1 + \frac {T ^ {2}}{\beta_ {1}}\right)} + O (k \log T).
$$
Adversarial regime with a self-bounding constraint Next we consider the adversarial regime with a self-bounding constraint. We will bound a component of (36). By Lemma 7,  $\sqrt{\mathbb{E}\left[\sum_{t=1}^{T} \nu_t h_{t+1}\right]}$  is bounded as
$$
\begin{array}{l} X _ {t} := \sqrt {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \nu_ {t} h _ {t + 1} \right]} \\ \leq \sqrt {3 \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \nu_ {t} h _ {t} \right] + \frac {2 0 k}{9} \log \left(\frac {T}{k}\right) \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \nu_ {t} \left(\frac {\beta_ {t + 1}}{\beta_ {t}} - 1\right) h _ {t + 1} \right]} \\ \leq \sqrt {3 \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \nu_ {t} h _ {t} \right] + \frac {1 0 k}{9} \log \left(\frac {T}{k}\right) \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left(\beta_ {t + 1} - \beta_ {t}\right) h _ {t + 1} \right]} \\ = \sqrt {3 \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \nu_ {t} h _ {t} \right] + \frac {1 0 k}{9} \log \left(\frac {T}{k}\right) \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \frac {c _ {1} \nu_ {t} h _ {t + 1}}{\sqrt {c _ {2} + \sum_ {t = 1} ^ {T} \nu_ {s} h _ {s + 1}}} \right]} \\ \leq \sqrt {3 \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \nu_ {t} h _ {t} \right] + \frac {2 0 k}{9} \log \left(\frac {T}{k}\right)} \mathbb {E} \left[ \sqrt {\sum_ {t = 1} ^ {T} \nu_ {t} h _ {t + 1}} \right] \\ = \sqrt {3 \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \nu_ {t} h _ {t} \right] + \frac {2 0 k}{9} \log \left(\frac {T}{k}\right) X _ {t}}, \\ \end{array}
$$
where the first inequality follows by Lemma 7, the second inequality follows by  $\nu_t \leq \beta_t / 2$ , the last inequality follows by Lemma 8. Since  $x \leq \sqrt{a + bx}$  for  $x > 0$  implies  $x \leq 2\sqrt{a} + b$ ,
$$
X _ {t} \leq 2 \sqrt {3 \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \nu_ {t} h _ {t} \right]} + \frac {2 0 k}{9} \log \left(\frac {T}{k}\right) = 2 \sqrt {3 \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \mathbb {E} [ \nu_ {t} \mid p _ {t} ] h _ {t} \right]} + \frac {2 0 k}{9} \log \left(\frac {T}{k}\right)
$$
Algorithm 2: BOBW algorithm with a sparsity-dependent bound in Section 5.2
1 for  $t = 1,2,\ldots ,T$  do   
2 Compute  $q_{t}$  using (1).   
3 Sample  $A_{t}\sim p_{t}$  , observe  $\ell_{tA_t}\in [-1,1]$  , and compute  $\widehat{y}_t$    
4 Update  $\beta_{t}$  using (8) based on the bisection method (Algorithm 3).
Algorithm 3: Bisection method for computing  $\beta_{t + 1}$
1 input:  $F_{t}$    
2 left  $\leftarrow \beta_{t}$  , right  $\leftarrow \beta_{t} + T$    
3 while true do   
4 center  $\leftarrow$  (left  $^+$  right)/2   
5 if  $F_{t}$  (center)  $<  0$  then   
6 left  $\leftarrow$  center   
7 else if  $F_{t}$  (center)  $>0$  then   
8 right  $\leftarrow$  center   
9 else   
10 break   
11 return center
$$
\leq 2 \sqrt {6 s \mathbb {E} \left[ \sum_ {t = 1} ^ {T} H \left(p _ {t}\right) \right]} + \frac {2 0 k}{9} \log \left(\frac {T}{k}\right), \tag {37}
$$
where we used  $\mathbb{E}[\nu_t\mid h_t]\leq \mathbb{E}[\sum_{i = 1}^k p_{ti}(\ell_{ti}^2 /p_{ti})] = \mathbb{E}[\sum_{i\in [k]:\ell_{ti}\neq 0}p_{ti}(\ell_{ti}^2 /p_{ti})]\leq s.$
We consider the case of  $P(a^{*}) \geq \mathrm{e}$ , since otherwise Lemma 2 implies  $\sum_{t=1}^{T} H(p_{t}) \leq \mathrm{e}\log (kT)$  and thus the desired bound is trivially obtained. When  $P(a^{*}) \geq \mathrm{e}$ , Lemma 2 implies that  $\sum_{t=1}^{T} H(p_{t}) \leq P(a^{*})\log (kT)$ . Then from the self-bounding technique, for any  $\lambda \in (0,1]$  it holds that
$$
\begin{array}{l} \operatorname {R e g} _ {T} = (1 + \lambda) \operatorname {R e g} _ {T} - \lambda \operatorname {R e g} _ {T} \\ \leq \mathbb {E} \left[ (1 + \lambda) O \left(\sqrt {s \log (T) \log (k T) P \left(a ^ {*}\right)}\right) - \lambda \Delta_ {\min } P \left(a ^ {*}\right) \right] + \lambda C + O (k \log T) \\ \leq O \left(\frac {(1 + \lambda) ^ {2} s \log (T) \log (k T)}{\lambda \Delta_ {\min }} + \lambda C\right) \\ = O \left(\frac {s \log (T) \log (k T)}{\Delta_ {\mathrm {m i n}}} + \lambda \left(\frac {s \log (T) \log (k T)}{\Delta_ {\mathrm {m i n}}} + C\right) + \frac {1}{\bar {\lambda}} \frac {s \log (T) \log (k T)}{\Delta_ {\mathrm {m i n}}}\right), \\ \end{array}
$$
where the first inequality follows by Lemma 1 and the second inequality follows from  $a\sqrt{x} - bx / 2 \leq a^2 / (2b)$  for  $a, b, x \geq 0$ . Setting  $\lambda \in (0, 1]$  to
$$
\lambda = \sqrt {\frac {s \log (T) \log (k T)}{\Delta_ {\mathrm {m i n}}} / \left(\frac {s \log (T) \log (k T)}{\Delta_ {\mathrm {m i n}}} + C\right)}
$$
gives the desired regret bound for the adversarial regime with a self-bounding constraint.
Stochastic regime Using  $\mathbb{E}[\nu_t|h_t]\leq \mathbb{E}[\omega_t|h_t]\leq \mathbb{E}[\sum_{i = 1}^k p_{ti}(\ell_{ti}^2 /p_{ti})] = \mathbb{E}[\sum_{i = 1}^k\ell_{ti}^2 ]$  in the second inequality of (37) and following the same arguments as the analysis for the adversarial regime with a self-bounding constraint, one can obtain the regret bound for the stochastic regime.
H.3 Discussion on bisection method for computing  $\beta_{t + 1}$
This section describes the bisection method to compute  $\beta_{t + 1}$  described in Section 5.2. Recall that  $F_{t}:[\beta_{t},\beta_{t} + T]\to \mathbb{R}$  is defined by the difference of the both sides of the update rule of  $(\beta_{t})$  in (8):
$$
F _ {t} (\alpha) = \alpha - \left(\beta_ {t} + \frac {c _ {1} \nu_ {t}}{\sqrt {c _ {2} + \nu_ {t} h _ {t + 1} (\alpha) + \sum_ {s = 1} ^ {t - 1} \nu_ {s} h _ {s + 1}}}\right), \tag {38}
$$
where  $h_{t + 1}(\alpha) = \frac{1}{1 - \frac{k}{T}} H(p_{t + 1}(\alpha))$ , and  $p_{t + 1}(\alpha)$  is the FTRL output with the regularizer  $\Phi_t = \alpha \psi^{\mathsf{nS}} + 4\psi^{\mathsf{LB}}$ . Note that  $c_1\nu_t / \sqrt{c_2 + \nu_th_{t + 1}(\alpha) + \sum_{s = 1}^{t - 1}\nu_sh_{s + 1}}\leq c_1\nu_t / c_2\leq T / 9$  since  $\nu_{t}\leq T$ .
Assume that  $F_{t}$  is continuous. Then we can see that there exists  $\alpha \in [\beta_{t}, \beta_{t} + T]$  such that  $F_{t}(\alpha) = 0$ . In fact, if  $p_{tA_{t}} = 0$  then  $\beta_{t+1} = \beta_{t}$ , and otherwise, we have  $F_{t}(\beta_{t}) \leq 0$  and  $F_{t}(\beta_{t} + T) > 0$ . Using the intermediate value theorem with the assumption that  $F_{t}$  is continuous, there indeed exists  $\alpha \in [\beta_{t}, \beta_{t} + T]$  satisfying  $F_{t}(\alpha) = 0$ . We can compute such  $\alpha$  by the bisection method. In particular, we first set the range of  $\alpha$  to  $[\beta_{t}, \beta_{t} + T]$ , and then iteratively halve it by evaluating the value of  $F_{t}$  at the middle point. Such a bisection method (binary search) is also used in [50], although the computed target is different. The whole BOBW algorithm with the sparsity-dependent bound in Section 5.2 is given in Algorithm 2, and the concrete procedure of the bisection given in Algorithm 3.
Now, all that remains is to show that  $F_{t}$  is continuous. To prove this, it suffices to prove that  $h_{t + 1}(\alpha) = \frac{1}{1 - \frac{k}{T}} H(p_{t + 1}(\alpha))$  is continuous with respect to  $\alpha$ .
Proposition 2.  $F_{t}$  in (38) is continuous with respect to  $\alpha$ .
Proof of Proposition 2. Take any  $\alpha \in [\beta_t, \beta_t + T]$  and then consider the following optimization problem:
$$
q _ {t + 1} (\alpha) = \underset {q \in \mathcal {P} _ {k}} {\arg \min } \left\langle \sum_ {s = 1} ^ {t} \widehat {y} _ {s}, q \right\rangle + \Phi_ {t + 1} (q),
$$
where  $\Phi_{t + 1} = \alpha \psi^{\mathrm{nS}} + 4\psi^{\mathrm{LB}}$ . Now using Corollary 8.1 of Hogan [20] with the fact that the solution of the above optimization problem is unique,  $q_{t + 1}(\alpha)$  is continuous with respect to  $\alpha$ . This completes the proof since  $p_{t + 1}$  is continuous with respect to  $q_{t + 1}$ ,  $1 / T \leq p_{t + 1,i}(\alpha) \leq 1 - k / T$ , and  $H(p)$  is continuous in a neighborhood of  $p = p_{t + 1}(\alpha)$ .
I Proof of Corollary 6
This section proves Corollary 6, which is the extended result of Corollary 5. Recall that  $B = 1/2$  for FI,  $B = k/2$  for MAB, and  $B = 2mk^2$  for PM-local, and  $r_{\mathcal{M}}$  is 1 if  $\mathcal{M}$  is FI or MAB, and  $2k$  if  $\mathcal{M}$  is PM-local, which are appeared in Appendix B. Let  $\mathrm{Reg}_T(a) = \mathbb{E}\big[\sum_{t=1}^{T} \big(\mathcal{L}_{A_t x_t} - \mathcal{L}_{a x_t}\big)\big] = \mathbb{E}\big[\sum_{t=1}^{T} \langle \ell_{A_t} - \ell_a, e_{x_t} \rangle\big]$  for  $a \in [k]$ .
Proof. Fix  $i^{*} \in [k]$ . From Lemma 7 in Tsuchiya et al. [46], if  $\eta_t > 0$ , we have
$$
\operatorname {R e g} _ {T} \left(i ^ {*}\right) \leq \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left(\frac {1}{\eta_ {t + 1}} - \frac {1}{\eta_ {t}}\right) H \left(q _ {t + 1}\right) + \frac {H \left(q _ {1}\right)}{\eta_ {1}} + \sum_ {t = 1} ^ {T} \eta_ {t} V _ {t} ^ {\prime} \right]. \tag {39}
$$
We will confirm that the assumptions for Part (I) of Theorem 1 are indeed satisfied. Since
$$
\frac {\sqrt {c _ {2} + \bar {z} _ {t} h _ {1}}}{c _ {1}} \left(\beta_ {t} + \beta_ {1}\right) \geq \sqrt {\frac {2 \bar {V} \log k}{\log (1 + T)}} \cdot 2 B \sqrt {\frac {\log (1 + T)}{\log k}} \geq \sqrt {2} \left(\bar {V} + \bar {V} _ {t}\right),
$$
stability condition (S1) is satisfied. One can also see that the other conditions are trivially satisfied. Hence, using Part (I) of Theorem 1, we can bound the RHS of (39) as
$$
\begin{array}{l} \operatorname {R e g} _ {T} (i ^ {*}) \leq \mathbb {E} \left[ \left(2 c _ {1} + \frac {1}{c _ {1}} \log \left(1 + \sum_ {u = 1} ^ {T} \frac {V _ {u} ^ {\prime}}{\bar {V}}\right)\right) \sqrt {\bar {V} H (q _ {1}) + \sum_ {t = 1} ^ {T} V _ {t} ^ {\prime} H (q _ {t + 1})} \right] + \frac {H (q _ {1})}{\eta_ {1}} \\ \leq \left(2 c _ {1} + \frac {1}{c _ {1}} \log (1 + T)\right) \sqrt {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} V _ {t} ^ {\prime} H \left(q _ {t + 1}\right) \right]} + O \left(\sqrt {\bar {V} \log (k) \log (T)} + B \sqrt {\log (k) \log T}\right) \\ = \sqrt {2 \log (1 + T)} \sqrt {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} V _ {t} ^ {\prime} H \left(q _ {t + 1}\right) \right]} + O \left(B \sqrt {\log (k) \log (T)}\right), \tag {40} \\ \end{array}
$$
where the second inequality follows from  $V_{u}^{\prime} / \bar{V} \leq 1$  and in the equality we set  $c_{1} = \sqrt{\frac{\log(1 + T)}{2}}$  and used  $\sqrt{\bar{V}} \leq B$ .
Adversarial regime For the adversarial regime, since  $H(q_{t}) \leq \log k$ , (40) immediately implies
$$
\operatorname {R e g} _ {T} \leq \mathbb {E} \left[ \sqrt {2 \sum_ {t = 1} ^ {T} V _ {t} ^ {\prime} \log (k) \log (1 + T)} + O \left(B \sqrt {\log (k) \log (T)}\right) \right],
$$
Adversarial regime with a self-bounding constraint Next, we consider the adversarial regime with a self-bounding constraint. We consider the case of  $Q(a^{*}) \geq \mathrm{e}$ , since otherwise Lemma 2 implies  $\sum_{t=1}^{T} H(p_t) \leq \mathrm{e}\log (kT)$  and thus the desired bound is trivially obtained. When  $Q(a^{*}) \geq \mathrm{e}$ , Lemma 2 implies that  $\sum_{t=1}^{T} H(q_t) \leq Q(a^{*})\log (kT)$ . Then from the self-bounding technique, for any  $\lambda \in (0,1]$
$$
\begin{array}{l} \operatorname {R e g} _ {T} = (1 + \lambda) \operatorname {R e g} _ {T} - \lambda \operatorname {R e g} _ {T} \\ \leq \mathbb {E} \left[ (1 + \lambda) O \left(\sqrt {\bar {V} \log (T) \log (k T) Q \left(a ^ {*}\right)}\right) - \frac {\lambda \Delta_ {\min } Q \left(a ^ {*}\right)}{r _ {\mathcal {M}}} \right] + \lambda C \\ \leq (1 + \lambda) O \left(\sqrt {\bar {V} \log (T) \log (k T) \bar {Q} (a ^ {*})}\right) - \frac {\lambda \Delta_ {\min} \bar {Q} (a ^ {*})}{r _ {\mathcal {M}}} + \lambda C \\ \leq O \left(\frac {(1 + \lambda) ^ {2} r _ {\mathcal {M}} \log (T) \log (k T)}{\lambda \Delta_ {\min }} + \lambda C\right) \\ = O \left(\frac {r _ {\mathcal {M}} \bar {V} \log (T) \log (k T)}{\Delta_ {\min }} + \lambda \left(\frac {r _ {\mathcal {M}} \bar {V} \log (T) \log (k T)}{\Delta_ {\min }} + C\right) + \frac {1}{\lambda} \frac {r _ {\mathcal {M}} \bar {V} \log (T) \log (k T)}{\Delta_ {\min }}\right), \\ \end{array}
$$
where the first inequality follows by (40) and Lemma 1 with  $c' = r_{\mathcal{M}}$  and the second inequality follows from  $a\sqrt{x} - bx / 2 \leq a^2 / (2b)$  for  $a, b, x \geq 0$ . Setting  $\lambda \in (0,1]$  to
$$
\lambda = \sqrt {\frac {r _ {\mathcal {M}} \bar {V} \log (T) \log (k T)}{\Delta_ {\min}} / \left(\frac {r _ {\mathcal {M}} \bar {V} \log (T) \log (k T)}{\Delta_ {\min}} + C\right)}
$$
gives the desired bound for the adversarial regime with a self-bounding constraint.
J Basic lemma
Lemma 8 ([38, Lemma 4.13]). Let  $a_0 \geq 0$ ,  $(a_t)_{t=1}^T$  be non-negative reals and  $f: \mathbb{R}_+ \to \mathbb{R}_+$  be a non-increasing function. Then,
$$
\sum_ {t = 1} ^ {T} a _ {t} f \left(a _ {0} + \sum_ {s = 1} ^ {t} a _ {s}\right) \leq \int_ {a _ {0}} ^ {\sum_ {t = 0} ^ {T} a _ {t}} f (x) \mathrm {d} x.
$$
We include the proof for the completeness.
Proof. Let  $A_{t} = \sum_{s=0}^{t} a_{s}$ . Then summing the following inequality over  $t$  completes the proof:
$$
a _ {t} f \left(a _ {0} + \sum_ {s = 1} ^ {t} a _ {s}\right) = a _ {t} f (A _ {t}) = \int_ {A _ {t - 1}} ^ {A _ {t}} f (A _ {t}) \mathrm {d} x \leq \int_ {A _ {t - 1}} ^ {A _ {t}} f (x) \mathrm {d} x.
$$
K Comparison of the sparsity-dependent bound and the first-order bound for negative losses
If  $\ell_t \in [0,1]^k$ , the first-order bound by Wei and Luo [49] implies sparsity bounds. This, however, does not hold when  $\ell_t \in [-1,1]^k$ . In fact, let us consider the case where  $\ell_t$  is a zero vector except that only one arm's loss is  $-1$  for some  $t \in [T]$ . Then the sparsity-dependent bound becomes  $O(\sqrt{T})$ . On the other hand, the first-order bound in [49] is not directly applicable, and we need to transform losses to range  $[0,1]$ . This implies that the first-order bound becomes  $O(\sqrt{kT})$ , which is worse than the sparsity-dependent bound.