SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Remark. To the best of our knowledge, this is the first learning rate that depends on both the stability and penalty components. Note that when we set the penalties to their worst-case value, that is,  $h_t = h_1$  for all  $t \in [T]$  (recalling  $h_t \leq h_1$ ), the SPA learning rate in (4) becomes equivalent to the standard type of the learning rate, which depends only on the stability and has the form of  $\beta_t = 1 / \eta_t \simeq \frac{c_1}{\sqrt{h_1}} \sqrt{\bar{z}_1 + \sum_{s=1}^{t-1} z_s}$ . On the other hand, when we set the stabilities to be their worst-case value, that is,  $z \geq \max_{t \in [T]} z_t$ , the SPA learning rate in (4) corresponds to the learning rate dependent only on the penalty in [22, 46].
Using learning rate  $(\eta_t)$  in (4), we can bound  $\widehat{\operatorname{Reg}_T}^{\mathsf{SP}}$  as follows.
Theorem 1 (Stability-penalty-adaptive regret bound). Let  $(\eta_t)_{t=1}^T$  be a SPA learning rate in Definition 2. Then  $\widehat{\mathrm{Reg}}_T^{\mathrm{SP}}$  in (3) is bounded as follows:
(I) If  $((h_t, z_t, \bar{z}_t))_{t=1}^T$  in  $(\eta_t)$  satisfies  $\frac{\sqrt{c_2 + \bar{z}_t h_1}}{c_1} (\beta_1 + \beta_t) \geq \varepsilon + z_t$  for all  $t \in [T]$  for some  $\varepsilon > 0$  (stability condition (S1)), then
$$
\widehat {\operatorname {R e g} _ {T}} ^ {\mathrm {S P}} \leq 2 \left(c _ {1} + \frac {\lambda}{c _ {1}} \log \left(1 + \sum_ {u = 1} ^ {T} \frac {z _ {u}}{\varepsilon}\right)\right) \sqrt {c _ {2} + \bar {z} _ {T} h _ {1} + \sum_ {t = 1} ^ {T} z _ {t} h _ {t + 1}}. \tag {5}
$$
(II) If  $h_t = h_1$  for all  $t \in [T]$ ,  $c_2 = 0$ , and  $((h_t, z_t, \bar{z}_t))_{t=1}^T$  in  $(\eta_t)$  satisfies  $\beta_t \geq \frac{ac_1}{\sqrt{h_1}} \sqrt{\sum_{s=1}^t z_s}$  for some  $a > 0$  (stability condition (S2)), then  $\widehat{\mathsf{Reg}_T^{\mathsf{SP}}} \leq 2 \left( c_1 + \frac{\lambda}{ac_1} \right) \sqrt{h_1 \sum_{t=1}^T z_t}$ .
The proof of Theorem 1 can be found in Appendix D. In Part (I) of Theorem 1, we can see that  $\widehat{\mathrm{Reg}}_T^{\mathrm{SP}}$  is bounded by  $\sqrt{\sum_{t=1}^{T} z_t h_{t+1}}$ , which will enable us to obtain BOBW and data-dependent bounds simultaneously. Note that  $\widehat{\mathrm{Reg}}_T^{\mathrm{SP}}$ , which is the component of the regret, often becomes dominant in particular when we use the Shannon entropy regularizer. Thus, checking the stability condition (S1) and applying Theorem 1 to bound  $\widehat{\mathrm{Reg}}_T^{\mathrm{SP}}$  almost complete the regret analysis. In Section 6, we will see that applying Theorem 1 immediately provides a BOBW bound with a game-dependent bound for PM. In contrast, when deriving BOBW with a sparsity-dependent bound for MAB in Section 5, we will develop additional techniques and conduct further analysis, for example, to satisfy the stability condition (S1), making use of the time-invariant log-barrier regularizer.
5 Sparsity-dependent bounds in multi-armed bandits
This section establishes several sparsity-dependent bounds. We use the FTRL framework in (1) with the inverse weighted estimator  $\widehat{y}_t\in \mathbb{R}^k$  given by  $\widehat{y}_{ti} = \ell_{ti}\mathbb{1}[A_t = i] / p_{ti}$ . This estimator is common in the literature and is useful for its unbiasedness, i.e.,  $\mathbb{E}_{A_t\sim p_t}[\widehat{y}_t|p_t] = \ell_t$ . We first propose algorithms that achieve sparsity-dependent bounds using stability-dependent learning rates in Section 5.1 as preliminaries for the subsequent section. Following that, in Section 5.2, we establish a BOBW algorithm with a sparsity-dependent bound based on the SPA learning rate. More specific steps are summarized as follows.
- Section 5.1.1 discusses the case of  $\ell_t \in [0,1]^k$  and shows that appropriately choosing  $z_t$  in the SPA learning rate (4) with the Shannon entropy regularizer and  $\widetilde{\Theta}((kT)^{-2/3})$  uniform exploration achieves a  $O(\sqrt{L_2\log k})$  regret for  $\ell_t \in [0,1]^k$  without knowing  $L_2$ .
- Section 5.1.2 considers the case of  $\ell_t \in [-1,1]^k$ , which is known to be more challenging than  $\ell_t \in [0,1]^k$ . We show that the time-invariant log-barrier enables us to choose a "tighter"  $z_t$  in (4), which removes the uniform exploration used in Section 5.1.1. This not only results in the bound of  $O(\sqrt{L_2 \log k})$  for  $\ell_t \in [-1,1]^k$  but also becomes one of the key properties to achieve BOBW.
- Section 5.2 presents a BOBW algorithm with a sparsity-dependent bound using the technique developed in Section 5.1 and Theorem 1. While Theorem 1 itself is a strong tool leading directly to the result for PM (Section 6), its application does not lead to the desired bounds. In particular, in this setting the  $\widetilde{O}\left(\sqrt{\sum_{t=1}^{T} z_{t} h_{t+1}}\right)$  term derived through Theorem 1 does not immediately imply a BOBW guarantee with a sparsity-dependent bound. To solve this problem, we develop a novel technique to analyze the variation in FTRL outputs  $q_{t}$  in response to the change in a regularizer (Lemma 7), and prove a BOBW bound with a sparsity-dependent bound of  $O\left(\sqrt{L_{2} \log k \log T}\right)$ .
5.1 Parameter-agnostic sparsity-dependent bounds
This section establishes  $s$ -agnostic algorithms to achieve sparsity-dependent bounds for the adversarial regime, which are preliminaries for Section 5.2.
5.1.1  $L_{2}$ -agnostic algorithm with  $O(\sqrt{L_2\log k})$  bound for  $\ell_t \in [0,1]^k$
Here, we use  $p_t = \mathfrak{T}_t(q_t)$  for  $\mathfrak{T}_t(q) = (1 - \gamma)q + \frac{\gamma}{k}\mathbf{1}$  and  $\gamma = \frac{k^{1/3}(\log k)^{1/3}}{T^{2/3}}$  and assume  $\gamma \in [0,1/2]$  (this holds when  $T \geq \sqrt{8k\log k}$ ), which implies  $2p_t \geq q_t$ . We use the Shannon entropy regularizer  $\Phi_t(p) = -\frac{1}{\eta_t} H(p) = \frac{1}{\eta_t}\psi^{\mathsf{nS}}(p) = \frac{1}{\eta_t}\sum_{i=1}^{k}p_i\log p_i$  with learning rate  $\eta_t = 1/\beta_t$  and
$$
\beta_ {1} = \frac {2 c _ {1}}{\sqrt {h _ {1}}} \sqrt {\frac {k}{\gamma}}, \quad \beta_ {t + 1} = \beta_ {t} + \frac {c _ {1} \omega_ {t}}{\sqrt {\log k} \sqrt {\frac {k}{\gamma} + \sum_ {s = 1} ^ {t - 1} \omega_ {s}}} \quad \text {f o r} \quad \omega_ {t} := \frac {\ell_ {t A _ {t}} ^ {2}}{p _ {t A _ {t}}}, \tag {6}
$$
which corresponds to the learning rate in Definition 2 with  $h_t \gets H(q_1) = \log k$ ,  $z_t \gets \omega_t$ ,  $\bar{z}_t \gets k / \gamma$ , and  $c_2 \gets 0$ . The uniform exploration is used to satisfy stability condition (S2) in Theorem 1, the amount of which is determined by balancing the regret coming from the uniform exploration and stability condition (S2). Theorem 1 immediately gives the following bound.
Corollary 2. When  $T \geq \sqrt{8k\log k}$ , the above algorithm with  $c_{1} = 1 / \sqrt{2}$  achieves  $\mathrm{Reg}_T \leq 2\sqrt{2}\sqrt{L_2\log k} + (2\sqrt{2} + 1)(kT\log k)^{1/3}$  without knowing  $L_{2}$ . In particular, when  $T \geq 7k^2 / s^3$ ,  $\mathrm{Reg}_T \leq (4\sqrt{2} + 1)\sqrt{sT\log k}$ .
The proof is given in Appendix F. The most striking feature of the algorithm is its  $L_{2}$  (or  $s$ )-agnostic property. This is essentially made possible by the learning rate using the data-dependent quantity  $\omega_{t}$  in (6), which satisfies  $\mathbb{E}\left[\sqrt{\sum_{t=1}^{T}\omega_{t}}\right] \leq \sqrt{\sum_{t=1}^{T}\mathbb{E}[\omega_{t}]} = \sqrt{\sum_{t=1}^{T}\sum_{i=1}^{k}\ell_{ti}^{2}} = \sqrt{L_{2}}$ . The leading constant of the bound is better than the existing bounds, as shown in Table 1, despite its agnostic property. Note that while the first-order bound in [49] implies the above sparsity-dependent bound when  $\ell_{t} \in [0,1]^{k}$ , this does not hold when  $\ell_{t} \in [-1,1]$ , which we investigate in the following (see Appendix K for details). We will see in Section 5.1.2 that this assumption can be totally removed by adding a time-invariant log-barrier regularization.
5.1.2  $L_{2}$ -agnostic algorithm with  $O(\sqrt{L_2\log k} +k\log T)$  bound for  $\ell_t\in [-1,1]^k$
Here, we consider the case of  $\ell_t \in [-1, 1]^k$ . It is worth noting that the negative loss cannot be handled by simply shifting the loss since it removes the sparsity from the losses  $(\ell_t)$ ; see [10, 27] and Appendix K for further details. We directly use the output  $q_t$  as  $p_t$ , that is,  $p_t = q_t$ . We use the hybrid regularizer consisting of the negative Shannon entropy and the log-barrier function,  $\Phi_t(p) = \frac{1}{\eta_t} \psi^{\mathrm{nS}}(p) + 2 \psi^{\mathrm{LB}}(p)$ , where  $\psi^{\mathrm{LB}}(p) = \sum_{i=1}^{k} \log(1 / x_i)$ . We use learning rate  $\eta_t = 1 / \beta_t$  and
$$
\beta_ {1} = \frac {c _ {1} ^ {2}}{8 h _ {1}}, \quad \beta_ {t + 1} = \beta_ {t} + \frac {c _ {1} \nu_ {t}}{\sqrt {\log k} \sqrt {\nu_ {t} + \sum_ {s = 1} ^ {t - 1} \nu_ {s}}} \quad \text {f o r} \quad \nu_ {t} := \omega_ {t} \min  \left\{1, \frac {p _ {t} A _ {t}}{2 \eta_ {t}} \right\}, \tag {7}
$$
where  $\omega_{t}$  is defined in (6). Learning rate (7) corresponds to that in Definition 2 with  $h_t\gets H(q_1) = \log k, z_t\gets \nu_t, \bar{z}_t\gets \nu_t,$  and  $c_{2}\gets 0$ . We then have the following bound:
Corollary 3. If we run the above algorithm with  $c_1 = \sqrt{2}$ ,  $\operatorname{Reg}_T \leq 4\sqrt{2}\sqrt{L_2\log k} + 2k\log T + k + 1/4$ , which implies that  $\operatorname{Reg}_T \leq 4\sqrt{2}\sqrt{sT\log k} + 2k\log T + k + 1/4$ .
The proof is given in Appendix G. Corollary 3 removes the assumption of  $T \geq 7k^2 / s^3$  in Corollary 2, and it also improves the leading constant of the regret in [10]. Note that one can prove a bound of the same order, but with a worse leading constant, by setting  $\beta_1 \geq 15k$  and combining the analysis similar to that in Section 5.1.1 and the stability bound in [10]. We successfully remove the assumption of  $T \geq 7k^2 / s^3$  thanks to the following lemma, which serves as one of the key elements in achieving a BOBW guarantee with a sparsity-dependent bound (The proof is given in Appendix G.):
Lemma 3 (Stability bound for negative losses). Let  $\ell_t \in [-1,1]^k$  and  $\widehat{y}_{ti} = \ell_{ti}\mathbb{1}[A_t = i] / p_{ti}$  be the inverse weighted estimator. Assume that  $q_t \leq \delta p_t$  for some  $\delta \geq 1$ . Then the stability term of FTRL with the hybrid regularizer  $\Phi_t = \frac{1}{\eta_t}\psi^{\mathrm{nS}} + 2\delta \psi^{\mathrm{LB}}$  is bounded as
$$
\langle q _ {t} - q _ {t + 1}, \widehat {y} _ {t} \rangle - D _ {\Phi_ {t}} (q _ {t + 1}, q _ {t}) \leq \delta \eta_ {t} \frac {\ell_ {t A _ {t}} ^ {2}}{p _ {t A _ {t}}} \min  \left\{1, \frac {p _ {t A _ {t}}}{2 \eta_ {t}} \right\} = \delta \eta_ {t} \nu_ {t}.
$$
Remark. We can observe from Lemma 3 that the stability term is bounded in terms of  $\nu_{t}$ , and the most important observation is that this  $\nu_{t}$  is bounded by the inverse of the learning rate  $1 / (2\eta_t) = \beta_t / 2$  i.e.,  $\nu_{t} \leq \beta_{t} / 2$ . This enables us to guarantee the stability condition (S2) in Theorem 1 without needing to mix the  $\widetilde{\Theta}((kT)^{-2/3})$  uniform exploration used in Section 5.1.1. Moreover, this will be a key property to prove a BOBW with a sparsity-dependent bound in the next section.
As a minor contribution, by directly bounding the stability component, the RHS of Lemma 3 has a smaller leading constant than the bound obtained by using the bound in [10].
5.2 Best-of-both-worlds guarantee with sparsity-dependent bound
Finally, we are ready to establish a BOBW algorithm with a sparsity-dependent bound and derive its regret bound. We use  $p_t = \mathfrak{T}_t(q_t) = (1 - \gamma)q_t + \frac{\gamma}{k}\mathbf{1}$  with  $\gamma = \frac{k}{T}$  (i.e.,  $\Theta(1 / T)$  uniform exploration).
and assume  $\gamma \in [0,1 / 2]$ , which implies  $2p_{t} \geq q_{t}$  and  $\nu_{t} \leq T$ . We use the hybrid regularizer  $\Phi_t = \frac{1}{\eta_t}\psi^{\mathsf{nS}} + 4\psi^{\mathsf{LB}}$  with learning rate  $\eta_t = 1 / \beta_t$  and
$$
\beta_ {1} = 1 5 k, \beta_ {t + 1} = \beta_ {t} + \frac {c _ {1} \nu_ {t}}{\sqrt {8 1 c _ {1} ^ {2} + \nu_ {t} h _ {t + 1} + \sum_ {s = 1} ^ {t - 1} \nu_ {s} h _ {s + 1}}} \text {w i t h} h _ {t} = \frac {1}{1 - \frac {k}{T}} H \left(p _ {t}\right), \tag {8}
$$
where  $\nu_{t}$  is defined in (7). Note that this learning depends on both the stability and penalty components. This corresponds to the SPA learning rate in Definition 2 with  $z_{t} \gets \nu_{t}$ ,  $\bar{z}_{t} \gets \nu_{t} h_{t+1} / h_{1}$ , and  $c_{2} \gets 81 c_{1}^{2}$ . One can see that stability assumption (S1) in Part (I) of Theorem 1 are satisfied thanks to  $\nu_{t} \leq \beta_{t}$  (see Appendix H for the proof). We then have the following bounds.
Theorem 4 (BOBW with sparsity-dependent bound). Suppose that  $T \geq 2k$ . Then the above algorithm with  $c_{1} = \sqrt{2\log(1 + T / \beta_{1})}$  (Algorithm 2 in Appendix H) achieves
$$
\operatorname {R e g} _ {T} \leq 4 \sqrt {L _ {2} \log (k) \log (1 + T)} + O (k \log T)
$$
in the adversarial regime,
$$
\operatorname {R e g} _ {T} = O \left(\frac {s \log (T) \log (k T)}{\Delta_ {\min}} + \sqrt {\frac {C s \log (T) \log (k T)}{\Delta_ {\min}}}\right)
$$
in the adversarial regime with a  $(\Delta, C, T)$  self-bounding constraint, and  $\mathrm{Reg}_T = O(\mathbb{E}[\sum_{i=1}^k \ell_{ti}^2] \log(T) \log(kT) / \Delta_{\min})$  in the stochastic regime.
The proof is given in Appendix H. This is the first BOBW bound with the sparsity-dependent  $(L_{2}$ -dependent) bound. Note that the bounds in the stochastic regime and the adversarial regime with a self-bounding constraint can also exploit the propoery of the underlying losses. The bound in the stochastic regime is suboptimal in two respects: its dependence on  $\Delta_{\mathrm{min}}$  and  $(\log T)^2$ . Concurrently, two separate studies improve each suboptimality ([25] for  $\Delta_{\mathrm{min}}$  and [13] for  $(\log T)^2$ ), but it is highly uncertain if we can prove a BOBW with a sparsity-dependent bound based on their approach, and it is an important future work to investigate this problem.
Key elements of the proof In the following, we describe some key elements of the proof of Theorem 4. We need to solve one remaining technical issue. Using Part (I) of Theorem 1, we can show that the regret is roughly bounded by  $\mathbb{E}\left[\sqrt{\sum_{t=1}^{T} \nu_t h_{t+1}}\right] \leq \sqrt{\sum_{t=1}^{T} \mathbb{E}[\nu_t h_{t+1}]}$ . However, this quantity cannot be straightforwardly bounded since  $h_{t+1}$  depends on  $\nu_t$ .
To address this issue, we analyze the behavior of arm selection probabilities when the regularizer changes. In particular, we first prove in Lemma 7 that  $h_{t+1} \leq h_t + k(\beta_{t+1} / \beta_t - 1)h_{t+1}$ . This lemma can be proven by a novel analysis evaluating the changes of the FTRL outputs when the learning rate varies (given in Appendices H.1 and H.2) which is not considered and required when we use a time-invariant learning rate (e.g., [10]). Using the last inequality, we have  $\sqrt{\sum_{t=1}^{T}\mathbb{E}[\nu_th_{t+1}]} \lesssim \sqrt{\sum_{t=1}^{T}\mathbb{E}[\nu_th_t] + k\sum_{t=1}^{T}\mathbb{E}[\nu_t(\beta_{t+1} / \beta_t - 1)h_{t+1}]} \lesssim$
$$
\sqrt {\sum_ {t = 1} ^ {T} \mathbb {E} [ \nu_ {t} h _ {t} ] + k \sum_ {t = 1} ^ {T} \mathbb {E} [ (\beta_ {t + 1} - \beta_ {t}) h _ {t + 1} ]} \lesssim \sqrt {\sum_ {t = 1} ^ {T} \mathbb {E} [ \nu_ {t} h _ {t} ] + k \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \sqrt {\sum_ {t = 1} ^ {T} \nu_ {t} h _ {t + 1}} \right]} \lesssim
$$
$\sqrt{\sum_{t=1}^{T} \mathbb{E}[\nu_t h_t]} + k$ , which holds thanks again to  $\nu_t \leq \beta_t$  and based on the fact that  $x \leq \sqrt{a + bx}$  for  $a, b, x > 0$  implies  $x \lesssim \sqrt{b} + a$  (here we ignore some logarithmic factors). This combined with the self-bounding technique leads to a BOBW guarantee in the stochastic regime.
Implementation One may wonder how to compute  $\beta_{t + 1}$  satisfying (8) since  $h_{t + 1} = h_{t + 1}(\beta_{t + 1})$  depends on  $\beta_{t + 1}$ . In fact, this can be computed by defining  $F_{t}:[\beta_{t},\beta_{t} + T]\to \mathbb{R}$  as  $F_{t}(\alpha) = \alpha -\left(\beta_{t} + c_{1}\nu_{t} / \sqrt{81c_{1}^{2} + \nu_{t}h_{t + 1}(\alpha) + \sum_{s = 1}^{t - 1}\nu_{s}h_{s + 1}}\right)$  and setting  $\beta_{t + 1}$  to be a root of  $F_{t}(\alpha) = 0$ . Such  $\alpha$  can be computed using the bisection method because  $F_{t}$  is continuous (proved in Proposition 2 in Appendix H.3). The detailed discussion can be found in Appendix H.3.
6 Best-of-both-worlds with game-dependent bound for partial monitoring
This section discusses the result of a BOBW guarantee with a game-dependent bound for PM. The desired bound is obtained by direct application of the SPA learning rate and Theorem 1, which highlights the usefulness of the SPA learning rate. Due to the space constraints, the background and algorithm are given in Appendix B.
We rely on the Exploration by Optimization (EbO) by Lattimore and Szepesvári [32]. Define
$$
\mathsf {S T} (p, G; q _ {t}, \eta_ {t}) = \max  _ {x \in [ d ]} \left[ \frac {(p - q _ {t}) ^ {\top} \mathcal {L} e _ {x}}{\eta_ {t}} + \frac {\operatorname {b i a s} _ {q _ {t}} (G ; x)}{\eta_ {t}} + \frac {1}{\eta_ {t} ^ {2}} \sum_ {a = 1} ^ {k} p _ {a} \Psi_ {q _ {t}} \left(\frac {\eta_ {t} G (a , \Phi_ {a x})}{p _ {a}}\right) \right], \tag {9}
$$
where  $\eta_t$  is a learning rate,  $\Psi_q(z) = \langle q, \exp(-z) + z - 1 \rangle$  and  $\mathrm{bias}_q(G; x) = \langle q, \mathcal{L}e_x - \sum_{a=1}^k G(a, \Phi_{ax}) \rangle + \max_{c \in \Pi} (\sum_{a=1}^k G(a, \Phi_{ax})_c - \mathcal{L}_{cx})$  is the bias of the estimator. Note that the first and third terms in (9) correspond to the stability and transformation terms. Then in EbO we choose  $(p_t, G_t) \in \mathcal{P}_k \times \mathcal{H}$  by
$$
\left(p _ {t}, G _ {t}\right) = \underset {p \in \mathcal {P} _ {k} ^ {\prime} \left(q _ {t}\right), G \in \mathcal {H}} {\arg \min } \mathsf {S T} \left(p, G; q _ {t}, \eta_ {t}\right) \quad \text {f o r} \quad \mathcal {P} _ {k} ^ {\prime} (q) = \left\{p \in \mathcal {P} _ {k}: p \geq q / (2 k) \right\}, \tag {10}
$$
where  $\mathcal{P}_k'(q)$  is the feasible set proposed in [46]. Let  $\mathrm{opt}_{q_t}'(\eta_t)$  be the optimal value of the optimization problem and  $V_{t}^{\prime} = \max \{0,\mathrm{opt}_{q_{t}}^{\prime}(\eta_{t})\}$  be its truncation.
Remark. It is worth noting that  $V_{t}^{\prime} = \max \{0, \mathrm{opt}_{q_{t}}^{\prime}(\eta_{t})\}$  captures game difficulty the learner is facing as discussed in [32]. As discussed above,  $\mathsf{ST}(p, G)$  in (9), the objective function that determines  $\mathrm{opt}_{q_{t}}^{\prime}(\eta_{t})$ , correspond to the components of the regret upper bound of FTRL. Hence, the smaller  $\mathsf{ST}(p, G)$  can become by optimizing  $p$  and  $G$ , the smaller the regret can become, and thus  $\mathrm{opt}_{q_{t}}^{\prime}(\eta_{t})$  captures the game difficulty.
Now we are ready to state our result.
Corollary 5. For any non-degenerate  $PM$  games, there exists an algorithm based on a SPA learning rate achieving  $\operatorname{Reg}_T = O\left(\mathbb{E}\left[\sqrt{\sum_{t=1}^T V_t' \log(k) \log(1 + T)}\right] + mk^2 \sqrt{\log(k) \log(T)}\right)$  in the adversarial regime, and  $\operatorname{Reg}_T = O(m^2 k^4 \log(T) \log(kT) / \Delta_{\min} + \sqrt{Cm^2 k^4 \log(T) \log(kT) / \Delta_{\min}} + mk^2 \sqrt{\log(k) \log(T)})$  in the adversarial regime with a  $(\Delta, C, T)$  self-bounding constraint.
An extended result and the proof are given in Appendices B and I, respectively. Recall that  $V_{t}^{\prime} = \max \{0,\mathrm{opt}_{q_{t}}^{\prime}(\eta_{t})\}$  in the bound reflects the difficulty of the game the learner is facing, rather than the worst-case difficulty of the class of the game. The bounds in both regimes are optimal up to logarithmic factors, and further detailed comparisons are given in Table 2 and Appendix B.
In this paper, we established the stability-penalty-adaptive (SPA) learning rate (Definition 2), which provides the regret upper bound that jointly depends on the stability and penalty components of FTRL (Theorem 1). This learning rate combined with the technique and analysis for bounding stability terms allows us to achieve BOBW and data-dependent bounds (sparsity- and game-dependent bounds) simultaneously in MAB and PM.
There are some remaining questions. First of all, it would be important future direction to apply the SPA learning rate to other online decision-making problems or regularizers. For example, it is as to investigate online learning with feedback graphs [3], in which the Shannon entropy regularizer (or the Tsallis entropy with the exponent larger than  $1 - 1 / \log k$ ) is necessary to achieve nearly optimal regret bounds. Another interesting example is to employ the Tsallis entropy as a dominant regularizer in the SPA learning rate, for instance, to improve logarithmic dependences on the regret bounds, while in this paper we only focused on the Shannon entropy. Second, it is an open question whether we can achieve the bound of  $O(\sqrt{sT\log(k / s)})$  in the sparsity-dependent bound without knowing the sparsity level  $s$ . Finally, while we only considered PM games with local observability, investigating if the game-dependent bound with the BOBW guarantee is possible for PM with global observability is important future work.
TT was supported by JST, ACT-X Grant Number JPMJAX210E, Japan and JSPS, KAKENHI Grant Number JP21J21272, Japan. JH was supported by JSPS, KAKENHI Grant Number JP21K11747, Japan.