SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Taira Tsuchiya*
tsuchiya@mist.i.u-tokyo.ac.jp
NEC Corporation / RIKEN
Junya Honda
Kyoto University / RIKEN
honda@i.kyoto-u.ac.jp
Adaptivity to the difficulties of a problem is a key property in sequential decision-making problems to broaden the applicability of algorithms. Follow-the-regularized-leader (FTRL) has recently emerged as one of the most promising approaches for obtaining various types of adaptivity in bandit problems. Aiming to further generalize this adaptivity, we develop a generic adaptive learning rate, called stability-penalty-adaptive (SPA) learning rate for FTRL. This learning rate yields a regret bound jointly depending on stability and penalty of the algorithm, into which the regret of FTRL is typically decomposed. With this result, we establish several algorithms with three types of adaptivity: sparsity, game-dependency, and best-of-both-worlds (BOBW). Despite the fact that sparsity appears frequently in real problems, existing sparse multi-armed bandit algorithms with  $k$ -arms assume that the sparsity level  $s \leq k$  is known in advance, which is often not the case in real-world scenarios. To address this issue, we first establish  $s$ -agnostic algorithms with regret bounds of  $\widetilde{O}(\sqrt{sT})$  in the adversarial regime for  $T$  rounds, which matches the existing lower bound up to a logarithmic factor. Meanwhile, BOBW algorithms aim to achieve a near-optimal regret in both the stochastic and adversarial regimes. Leveraging the SPA learning rate and the technique for  $s$ -agnostic algorithms combined with a new analysis to bound the variation in FTRL output in response to changes in a regularizer, we establish the first BOBW algorithm with a sparsity-dependent bound. Additionally, we explore partial monitoring and demonstrate that the proposed SPA learning rate framework allows us to achieve a game-dependent bound and the BOBW simultaneously.
This study considers the multi-armed bandits (MAB) and partial monitoring (PM). In the MAB problem, the learner selects one of  $k$  arms, and the adversary simultaneously determines the loss of each arm,  $\ell_t = (\ell_{t1},\dots,\ell_{tk})^\top$  in  $[0,1]^k$  or  $[-1,1]^k$ . After that, the learner observes only the loss for the chosen arm. The learner's goal is to minimize the regret, which is the difference between the learner's total loss and the total loss of an optimal arm fixed in hindsight. PM is a generalization of MAB, and the learner observes feedback symbols instead of the losses.
*This work was done when the author was with Kyoto University and RIKEN.
One of the most promising frameworks for MABs and PM is follow-the-regularized-leader (FTRL) [5, 12], which determines the arm selection probability at each round by minimizing the sum of the cumulative (estimated) losses so far plus a convex regularizer. Note that the well-known Exp3 algorithm developed in [5] is equivalent to FTRL with the (negative) Shannon entropy regularizer. FTRL is also known to perform well for the classic expert problem [17] and reinforcement learning [52]. Furthermore, when the problem is "benign", it is known that FTRL can exploit the underlying structure to adaptively improve its performance. Typical examples of such adaptive improvements are (i) data-dependent bounds and (ii) best-of-both-worlds (BOBW).
Data-dependent bounds have been investigated to enhance the adaptivity of algorithms to a given structure of losses in the adversarial regime, where feedback (e.g., losses in MABs) is decided in an arbitrary manner. There are various examples of data-dependent bounds, and this study considers sparsity-dependent bounds and game-dependent bounds.
A sparsity-dependent bound is an important example of data-dependent bounds, as sparsity frequently appears in real-world problems. For example, in online advertisement allocation, it is often the case that only a fraction of the ads is clicked. Although there are some studies for sparse MABs [10, 27, 51], all of them assume that (an upper bound of) sparsity level  $s \geq \| \ell_t\| _0 = |\{i\in [k]\colon \ell_{ti}\neq 0\} |$  is known beforehand, which in many practical scenarios does not hold.
The concept of a game-dependent bound was recently introduced by Lattimore and Szepesvári [32] to derive a regret upper bound that depends on the game the learner is facing. As the authors suggest, one of the motivations for the game-dependent bound is that previous PM algorithms are "quite conservative and not practical for normal problems". For example, whereas the Bernoulli MAB is expressed as a PM, algorithms for PM do not always achieve the minimax regret of MAB [5]. The game-dependent bound enables the learner to automatically adapt to the essential difficulty of the game the algorithm is actually facing.
The BOBW algorithm aims to achieve near-optimal regret bounds in stochastic and adversarial regimes, where the feedback is stochastically generated in the stochastic regime. Since we often do not know the underlying regime, it is desirable for an algorithm to simultaneously obtain a near-optimal performance both for the stochastic and adversarial regimes. For multi-armed bandits, Bubeck and Slivkins [9] developed the first BOBW algorithm, and Zimmert and Seldin [53] proposed the well-known Tsallis-INF algorithm, which achieves the optimal regret for both regimes. The Tsallis-INF algorithm also achieves favorable regret guarantees in the adversarial regime with a self-bounding constraint, which interpolates between the stochastic and adversarial regimes.
To realize the aforementioned adaptivity in FTRL, the adaptive learning rate (a.k.a. time-varying learning rate) is one of the most representative approaches. This approach adjusts the learning rate based on previous observations. In the literature, adaptive learning rates have been designed to depend on stability or penalty, which are components of a regret upper bound of FTRL. The stability term increases if the variation of FTRL outputs in the adjacent rounds is large, and stability-dependent learning rates have been used in a considerable number of algorithms available in the literature, e.g., [32, 37, 38] and references therein. In contrast, the penalty term comes from the strength of the regularization, and recently penalty-dependent learning rates were considered to achieve BOBW guarantees [22, 46]. However, existing stability-dependent (resp. penalty-dependent) learning rates are designed with the worst-case penalty (resp. stability), which could potentially limit the adaptivity and performance of FTRL. (There are numerous studies related to this paper and we include additional related work in Appendix C.)
1.1 Contribution of this study
In this paper, in order to further broaden the applicability of FTRL, we establish a generic framework for designing an adaptive learning rate that depends on both the stability and penalty components simultaneously, which we call a stability-penalty-adaptive (SPA) learning rate (Definition 2). This enables us to bound the regret approximately by  $\widetilde{O}\left(\sqrt{\sum_{t=1}^{T} z_{t} h_{t+1}}\right)$  for stability component  $(z_{t})_{t}$  and a penalty component  $(h_{t})_{t}$ , which we call a SPA regret bound (Theorem 1). With appropriate selections of  $z_{t}$  and  $h_{t}$ , this result yields the three important adaptive bounds mentioned earlier, namely sparsity, game-dependency, and BOBW. In particular, our contributions are as follows (see also Tables 1 and 2):
Table 1: Regret upper bounds with sparsity-dependent bounds in multi-armed bandits.  $T$  is the time horizon.  $s \leq k$  is the level of sparsity in losses. Let  $L_{2} = \sum_{t=1}^{T} \| \ell_{t} \|^2$ , and  $\| \ell_{t} \|_{0} \leq s$  implies  $L_{2} = \sum_{t=1}^{T} \| \ell_{t} \|^2 \leq sT$  since  $\| \ell_{t} \|_{\infty} \leq 1$ .  $\Delta_{\min}$  is the minimum suboptimality gap. Adv. and Stoc. are the abbreviations of the adversarial and stochastic regime, respectively.
Reference	s-agnostic?	Range of ℓti	Regime	Regret bound
Kwon and Perchet [27]	-	[0, 1]	Adv.	Ω(√sT)
Kwon and Perchet [27]	No	[0, 1]	Adv.	2√e√sT log(k/s)
Ours (Sec. 5.1.1, Cor. 2)	Yes	[0, 1]	Adv.	2√2√L2 log k + O((kT log k)1/3)
Bubeck et al. [10]	No	[-1, 1]	Adv.	10√L2 log k + 20k log T
Ours (Sec. 5.1.2, Cor. 3)	Yes	[-1, 1]	Adv.	4√2√L2 log k + 2k log T
Ours (Sec. 5.2, Thm. 4)	Yes	[-1, 1]	Adv.	4√L2 log k log T + O(k log T)
Stoc.	O(s log(T) log(kT)/Δmin)
Table 2: Regret bounds for non-degenerate local PM games.  $V_{t}$ ,  $V_{t}^{\prime}$ , and  $\bar{V}^{\prime}$  are game-dependent quantities satisfying  $V_{t} \leq V_{t}^{\prime} \leq \bar{V}$  (see Section 6 and Appendix B for definitions).  $H(q_{t})$  is the Shannon entropy for FTRL output  $q_{t}$ .
Reference	Game-dependent?	BOBW?	Order of regret bound
Many existing studies on PM	No	No	-
Lattimore and Szepesváři [32]	Yes	No	√∑t=1T Vt log k
Tsuchiya et al. [46]	No (only game-class-dependent)	Yes	√V ∑t=1T H(qt+1)
Ours (Sec. 6, Cor. 5)	Yes	Yes	√∑t=1T Vt' H(qt+1) log T
- (Section 5.1) We initially provide new algorithms for sparse MABs as preliminaries for establishing a BOBW algorithm with a sparsity-dependent bound. In Section 5.1.1, we propose a novel estimator of the sparsity level, which is linked to a stability component and induces  $L_{2} = \sum_{t=1}^{T} \|\ell_{t}\|^{2} \leq sT$ . We demonstrate that a learning rate using this estimator with the Shannon entropy regularizer and  $\widetilde{\Theta}((kT)^{-2/3})$  uniform exploration immediately results in an  $O(\sqrt{L_{2} \log k})$  regret bound for  $\ell_{t} \in [0,1]^{k}$ . In Section 5.1.2, we investigate possibly negative losses  $\ell_{t} \in [-1,1]^{k}$ . We employ the time-invariant log-barrier proposed in [10] to control the stability term. This allows us to achieve an  $O(\sqrt{L_{2} \log k})$  regret bound for losses in  $[-1,1]^{k}$  even without the  $\widetilde{\Theta}((kT)^{-2/3})$  uniform exploration. This is a key component for developing the BOBW guarantee that we discuss next. Note that Section 5.1 serves as preliminary findings for the subsequent section.
- (Section 5.2) We establish a BOBW algorithm with a sparsity-dependent bound. In order to achieve this goal, we make another major technical development: we analyze the variation in the FTRL output when the regularizer changes (Lemma 7), which holds thanks to the time-invariant log-barrier and may be of independent interest. This analysis is necessary since we use a time-varying learning rate to obtain a BOBW guarantee, whereas Bubeck et al. [10] uses a constant learning rate. This technical development successfully allows us to achieve the goal (Theorem 4) in combination with the SPA learning rate developed in Section 4 and a technique for exploiting sparsity in Section 5.1.2.
- (Section 6) We show that the SPA learning rate established in Section 4 can also be used to achieve a game-dependent bound and a BOBW guarantee simultaneously, which further highlights the usefulness of the SPA learning rate.
This section introduces the preliminaries of this study. Sections 2.1 and 2.2 formulate the MAB and PM problems, respectively, and Section 2.3 defines regimes considered in this paper.
Notation Let  $\| x\|$ $\| x\| _1$  , and  $\| x\|_{\infty}$  be the Euclidian,  $\ell_1$  , and  $\ell_{\infty}$  -norms for a vector  $x$  , respectively. Let  $\| x\| _0$  be the number of non-zero elements for a vector  $x$  . Let  $\mathcal{P}_k = \{p\in [0,1]^k:\| p\| _1 = 1\}$  be the  $(k - 1)$  -dimensional probability simplex. A vector  $e_i\in \{0,1\} ^k$  is the  $i$  -th standard basis of  $\mathbb{R}^k$  , and 1 is the all-one vector. Let  $D_{\Phi}$  be the Bregman divergence induced by differentiable convex function  $\Phi$  , i.e.,  $D_{\Phi}(p,q) = \Phi (p) - \Phi (q) - \langle \nabla \Phi (q),p - q\rangle$  . Table 3 in Appendix A summarizes the notation used in this paper.
2.1 Multi-armed bandits
In MAB with  $k$ -arms, at each round  $t \in [T] := \{1, 2, \dots, T\}$ , the environment determines the loss vector  $\ell_t = (\ell_{t1}, \ell_{t2}, \dots, \ell_{tk})^\top$  in  $[0, 1]^k$  or  $[-1, 1]^k$ , and the learner simultaneously chooses an arm  $A_t \in [k]$  without knowing  $\ell_t$ . After that, the learner observes only the loss  $\ell_{tA_t}$  for the chosen arm. The performance of the learner is evaluated by the regret  $\mathrm{Reg}_T$ , which is the difference between the cumulative loss of the learner and of the single optimal arm, that is,  $a^* = \arg \min_{a \in [k]} \mathbb{E}\left[\sum_{t=1}^{T} \ell_{ta}\right]$  and  $\mathrm{Reg}_T = \mathbb{E}\left[\sum_{t=1}^{T} (\ell_{tA_t} - \ell_{ta^*})\right]$ , where the expectation is taken with respect to the internal randomness of the algorithm and the randomness of the loss vectors  $(\ell_t)_{t=1}^T$ .
2.2 Partial monitoring
Formulation A PM game  $\mathcal{G} = (\mathcal{L},\Phi)$  with  $k$ -actions and  $d$ -outcomes is defined by a pair of a loss matrix  $\mathcal{L}\in [0,1]^{k\times d}$  and feedback matrix  $\Phi \in \Sigma^{k\times d}$ , where  $\Sigma$  is a set of feedback symbols. The game is played in a sequential manner by a learner and an opponent across  $T$  rounds. The learner begins the game with knowledge of  $\mathcal{L}$  and  $\Phi$ . For every round  $t\in [T]$ , the opponent selects an outcome  $x_{t}\in [d]$ , and the learner simultaneously chooses an action  $A_{t}\in [k]$ . Then the learner suffers an unobserved loss  $\mathcal{L}_{A_tx_t}$  and receives only a feedback symbol  $\sigma_t = \Phi_{A_tx_t}$ , where  $\mathcal{L}_{ax}$  is the  $(a,x)$ -th element of  $\mathcal{L}$ . The learner's performance in the game is evaluated by the regret  $\mathrm{Reg}_T$  as in the MAB case:  $a^* = \arg \min_{a\in [k]}\mathbb{E}\big[\sum_{t = 1}^{T}\mathcal{L}_{ax_t}\big]$  and  $\mathrm{Reg}_T = \mathbb{E}\big[\sum_{t = 1}^T (\mathcal{L}_{A_tx_t} - \mathcal{L}_{a^* x_t})\big] = \mathbb{E}\big[\sum_{t = 1}^T\langle \ell_{A_t} - \ell_{a^*},e_{x_t}\rangle \big]$ , where  $\ell_a\in \mathbb{R}^d$  is the  $a$ -th row of  $\mathcal{L}$ .
Several concepts in PM Let  $m \leq |\Sigma|$  be the maximum number of distinct symbols in a single row of  $\Phi \in \Sigma^{k \times d}$ . Different actions  $a$  and  $b$  are duplicate if  $\ell_a = \ell_b$ . We can decompose possible distributions of  $d$  outcomes in  $\mathcal{P}_d$  based on the loss matrix. For every action  $a \in [k]$ , cell  $\mathcal{C}_a = \{u \in \mathcal{P}_d : \max_{b \in [k]} (\ell_a - \ell_b)^{\top} u \leq 0\}$  is the set of probability vectors in  $\mathcal{P}_d$  for which action  $a$  is optimal. Each cell is a closed convex polytope.
Define  $\dim(\mathcal{C}_a)$  as the dimension of the affine hull of  $\mathcal{C}_a$ . Action  $a$  is said to be dominated if  $\mathcal{C}_a = \emptyset$ . For non-dominated actions, action  $a$  is said to be Pareto optimal if  $\dim(\mathcal{C}_a) = d - 1$ , and degenerate if  $\dim(\mathcal{C}_a) < d - 1$ . Let  $\Pi$  be the set of Pareto optimal actions. Two Pareto optimal actions  $a, b \in \Pi$  are called neighbors if  $\dim(\mathcal{C}_a \cap \mathcal{C}_b) = d - 2$ , which is used to define the difficulty of PM games. A PM game is said to be non-degenerate if it has no degenerate actions. We assume that PM game  $\mathcal{G}$  is non-degenerate and contains no duplicate actions.
The difficulty of PM games is characterized by the following observability conditions. Neighbouring actions  $a$  and  $b$  are locally observable if there exists  $w_{ab}:[k]\times \Sigma \to \mathbb{R}$  such that  $w_{ab}(c,\sigma) = 0$  for  $c\notin \{a,b\}$  and  $\sum_{c = 1}^{k}w_{ab}(c,\Phi_{cx}) = \mathcal{L}_{ax} - \mathcal{L}_{bx}$  for all  $x\in [d]$ . A PM game is locally observable if all neighboring actions are locally observable, and this study considers locally observable games.
Loss difference estimation Let  $\mathcal{H}$  be the set of all functions from  $[k] \times \Sigma$  to  $\mathbb{R}^d$ . For any locally observable games, there exists  $G \in \mathcal{H}$  such that for any  $b, c \in \Pi$ ,  $\sum_{a=1}^{k} (G(a, \Phi_{ax})_b - G(a, \Phi_{ax})_c) = \mathcal{L}_{bx} - \mathcal{L}_{cx}$  for all  $x \in [d]$  [32]. For example, we can take  $G = G_0$  defined by  $G_0(a, \sigma)_b = \sum_{e \in \text{path}_{\mathcal{T}}(b)} w_e(a, \sigma)$  for  $a \in \Pi$ , where  $\mathcal{T}$  is a tree over  $\Pi$  induced by neighborhood relations and path  $\mathcal{T}(b)$  is the set of edges from  $b \in \Pi$  to an arbitrarily chosen root  $c \in \Pi$  on  $\mathcal{T}$  [32]. See Appendix C and [31, Chapter 37] for a more detailed explanation and background of PM.
2.3 Considered regimes
We consider three regimes on the assumptions for losses in MABs and outcomes in PM. In the stochastic regime, a sequence of loss vector  $(\ell_t)$  in MAB and that of outcome vector  $(x_{t})$  in PM follow an unknown distribution  $\nu^{*}$  in an i.i.d. manner. Define the minimum suboptimality gap in  $\Delta_{\mathrm{min}} = \min_{a\neq a^*}\Delta_a$  for  $\Delta_{a} = \mathbb{E}_{\ell_{t}\sim \nu^{*}}[(\ell_{ta} - \ell_{ta^{*}})]$  in MAB and  $\Delta_{a} = \mathbb{E}_{x_{t}\sim \nu^{*}}[(\ell_{a} - \ell_{a^{*}})^{\top}e_{x_{t}}]$  in PM. Note that the definitions of  $\ell$  in MAB and PM are different.
In contrast, the adversarial regime does not assume any stochastic structure for the losses or outcomes, and they can be chosen in an arbitrary manner. In this regime, the environment can choose  $\ell_t$  for MAB and  $x_{t}$  for PM depending on the past history until the  $(t - 1)$ -th round,  $(A_{s})_{s = 1}^{t - 1}$ .
We also consider, the adversarial regime with a self-bounding constraint [53], an intermediate regime between the stochastic and adversarial regimes.
Definition 1. Let  $\Delta \in [0,2]^k$  and  $C\geq 0$ . The environment is in an adversarial regime with a  $(\Delta ,C,T)$  self-bounding constraint if it holds for any algorithm that  $\mathsf{Reg}_T\geq \mathbb{E}\big[\sum_{t = 1}^T\Delta_{A_t} - C\big]$ .
One can see that the stochastic and adversarial regimes are indeed instances of this regime, and that well-known stochastic regimes with adversarial corruptions [36] are also in this regime (see [53] and [46] for definitions in MAB and PM, respectively).
We assume that there exists a unique optimal arm (or action)  $a^*$ , which was employed by many studies aiming at developing BOBW algorithms [18, 34, 49, 53].
This section provides preliminaries for developing and analyzing algorithms. We first introduce FTRL, upon which we develop our algorithms, and then describe the self-bounding technique, which is a common technique for proving a BOBW guarantee.
Follow-the-regularized-leader In the FTRL framework, an arm selection probability  $p_t \in \mathcal{P}_k$  at round  $t$  is given by
$$
q _ {t} = \underset {q \in \mathcal {P} _ {k}} {\arg \min } \left\langle \sum_ {s = 1} ^ {t - 1} \widehat {y} _ {s}, q \right\rangle + \Phi_ {t} (q) \quad \text {a n d} \quad p _ {t} = \mathfrak {T} _ {t} \left(q _ {t}\right), \tag {1}
$$
where  $\widehat{y}_t\in \mathbb{R}^k$  is an estimator of loss  $\ell_t$  at round  $t$ ,  $\Phi_t\colon \mathcal{P}_k\to \mathbb{R}$  is a strongly-convex regularizer, and  $\mathfrak{T}_t\colon \mathcal{P}_k\to \mathcal{P}_k$  is a map from the output of FTRL  $q_{t}$  to an arm selection probability vector  $p_t$ .
In the analysis of FTRL, it is common to evaluate  $\sum_{t=1}^{T}\langle\widehat{y}_t,p_t-p\rangle=\sum_{t=1}^{T}\langle\widehat{y}_t,q_t-p\rangle+\sum_{t=1}^{T}\langle\widehat{y}_t,p_t-q_t\rangle$  for some  $p\in\mathcal{P}_k$ . It is known (see e.g., [31, Exercise 28.12]) that quantity  $\sum_{t=1}^{T}\langle\widehat{y}_t,q_t-p\rangle$  is bounded from above by
$$
\sum_ {t = 1} ^ {T} (\underbrace {\Phi_ {t} \left(q _ {t + 1}\right) - \Phi_ {t + 1} \left(q _ {t + 1}\right)} _ {\text {p e n a l t y t e r m}}) + \Phi_ {T + 1} (p) - \Phi_ {1} \left(q _ {1}\right) + \sum_ {t = 1} ^ {T} (\underbrace {\langle q _ {t} - q _ {t + 1} , \widehat {y} _ {t} \rangle - D _ {\Phi_ {t}} \left(q _ {t + 1} , q _ {t}\right)} _ {\text {s t a b i l i t y t e r m}}). \tag {2}
$$
We refer to the terms in (2) as a penalty and stability terms, and to the quantity  $\langle \widehat{y}_t,p_t - q_t\rangle$  as a transformation term. Note that, though this study focuses on examples in which  $\Phi_{T + 1}(p)$  is not dominant, this term may be dominant dependent on the choice of regularizers.
Self-bounding technique A self-bounding technique is a common method for proving a BOBW guarantee [18, 49, 53]. In the self-bounding technique, we first derive regret upper and lower bounds in terms of a variable dependent on the arm selection probabilities  $(p_t)_t$  or the FTRL outputs  $(q_t)_t$ , and then derive a regret bound by combining the upper and lower bounds. We use a version proposed in [22]. We consider  $Q(i)$ ,  $\bar{Q}(i)$ ,  $P(i)$ , and  $\bar{P}(i)$  for  $i \in [k]$  defined by  $Q(i) = \sum_{t=1}^{T} (1 - q_{ti})$ ,  $\bar{Q}(i) = \mathbb{E}[Q(i)]$ ,  $P(i) = \sum_{t=1}^{T} (1 - p_{ti})$ , and  $\bar{P}(i) = \mathbb{E}[P(i)]$ . Note that  $\bar{Q}(i)$ ,  $\bar{P}(i) \in [0, T]$  for any  $i \in [k]$ . In terms of  $\bar{Q}(i)$  or  $\bar{P}(i)$ , we can obtain the lower bound of the regret for the adversarial regime with a self-bounding constraint as follows:
Lemma 1 ([46, Lemma 4]). In the adversarial regime with a self-bounding constraint (Definition 1), if there exists  $c' \in (0,1]$  such that  $p_{ti} \geq c' q_{ti}$  for all  $t \in [T]$  and  $i \in [k]$ , then  $\mathrm{Reg}_T \geq \Delta_{\min} \bar{P}(a^*) - C \geq c' \Delta_{\min} \bar{Q}(a^*) - C$ .
It is known that the sums of the entropy  $H(\cdot)$  of  $(p_t)_t$  is bounded by  $P(i)$  as follows:
Lemma 2 ([22, Lemma 4]). Let  $(q_{t})_{t = 1}^{T}$  be any sequence of probability vectors and define  $Q(i) = \sum_{t = 1}^{T}(1 - q_{ti})$ . Then for any  $i\in [k]$ ,  $\sum_{t = 1}^{T}H(q_{t})\leq Q(i)\log (\mathrm{ek}T / Q(i))$
Based on Lemmas 1 and 2, it suffices to show  $\mathrm{Reg}_T \lesssim \mathbb{E}\left[\sqrt{\sum_{t=1}^T H(q_t) \mathrm{polylog}(T)}\right]$  to prove a BOBW guarantee in MAB. This is because, for the adversarial regime, using  $H(q_t) \leq \log k$  immediately implies a  $\widetilde{O}(\sqrt{T})$  bound, and for the stochastic regime, using Lemmas 1 and 2 roughly bounds the regret as  $\mathrm{Reg}_T = 2\mathrm{Reg}_T - \mathrm{Reg}_T \lesssim \sqrt{Q(a^*) \mathrm{polylog}(T)} - \Delta_{\min} \bar{Q}(a^*) \lesssim \mathrm{polylog}(T) / \Delta_{\min}$ .
4 Stability-penalty-adaptive (SPA) learning rate and regret bound
This section proposes a new adaptive learning rate, which yields a regret upper bound dependent on both the stability component  $z_{t}$  and penalty component  $h_t$  for various choices of  $z_{t}$  and  $h_t$ . When we use a learning rate  $\eta_t$ , the analysis of FTRL boils down to the evaluation of
$$
\widehat {\operatorname {R e g}} _ {T} ^ {\mathrm {S P}} = \sum_ {t = 1} ^ {T} \left(\frac {1}{\eta_ {t + 1}} - \frac {1}{\eta_ {t}}\right) h _ {t + 1} + \lambda \sum_ {t = 1} ^ {T} \eta_ {t} z _ {t} \quad \text {f o r s o m e} \quad \lambda > 0. \tag {3}
$$
In particular, when we use the Exp3 algorithm,  $h_t$  is the Shannon entropy of the FTRL output at round  $t$ . This can be confirmed by checking the existing studies (e.g., [22, 46]) or the proofs in Appendices F, G, H.2, and I. To favorably bound  $\widehat{\mathrm{Reg}_T}^{\mathrm{SP}}$ , we develop a new learning rate framework, which we call the jointly stability- and penalty-adaptive learning rate, or the stability-penalty-adaptive (SPA) learning rate for short:
Definition 2 (Stability-penalty-adaptive learning rate). Let  $((h_t, z_t, \bar{z}_t))_{t=1}^T$  be non-negative reals such that  $h_1 \geq h_t$  for all  $t \in [T]$ ,  $(\bar{z}_t h_1 + \sum_{s=1}^t z_s h_{s+1})_{t=1}^T$  is non-decreasing, and  $\bar{z}_t h_1 \geq z_t h_{t+1}$  for all  $t \in [T]$ . Let  $c_1, c_2 > 0$ . Then, a sequence of  $(\eta_t)_{t=1}^T$  is a SPA learning rate if it has a form of
$$
\eta_ {t} = \frac {1}{\beta_ {t}}, \quad \beta_ {1} > 0, \quad a n d \quad \beta_ {t + 1} = \beta_ {t} + \frac {c _ {1} z _ {t}}{\sqrt {c _ {2} + \bar {z} _ {t} h _ {1} + \sum_ {s = 1} ^ {t - 1} z _ {s} h _ {s + 1}}}. \tag {4}
$$