SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Bandit algorithms with worst-case regrets that depend on the optimal reward. Recent linear logistic bandit works have shown worst-case regret bounds that depend on the variance of the best arm [32, 1]. When the arms are standard basis vectors, logistic bandits are equivalent to Bernoulli bandits, and the bounds of Abeille et al. [1] become  $\tilde{O}\left(K\sqrt{\hat{\mu}^{*}T} +\frac{K^{2}}{\hat{\mu}_{\min}}\wedge (K^{2} + A)\right)$  where  $\dot{\mu}_{\mathrm{min}} = \min_{i\in [K]}\dot{\mu}_i$  and  $A$  is an instance dependent quantity that can be as large as  $T$ . This bound, compared to ours, has an extra factor of  $\sqrt{K}$  in the leading term and the lower order term has an extra factor of  $K$ . Even worse, it has the term  $\dot{\mu}_{\mathrm{min}}^{-1}$  in the lower order term, which can be arbitrarily large. The bound in Mason et al. [32] becomes  $\tilde{O}\left(\sqrt{\hat{\mu}^{*}KT} +\dot{\mu}_{\mathrm{min}}^{-1}K^{2}\right)$ , which matches our bound in the leading term up to logarithmic factors yet still have extra factors of  $K$  and  $\dot{\mu}_{\mathrm{min}}^{-1}$  in the lower order term.
The KL Maillard Sampling Algorithm. We propose an algorithm called KL Maillard sampling (KL-MS) for bounded reward distributions (Algorithm 1). For the first  $K$  times steps, the algorithm pulls each arm once (steps 3 to 4); this ensures that starting from time step  $K + 1$ , the estimates of the reward distribution of all arms are well-defined. From time step  $t = K + 1$  on, the learner computes the empirical mean  $\hat{\mu}_{t - 1,a}$  of all arms  $a$ . For each arm  $a$ , the learner computes the binary KL divergence between  $\hat{\mu}_{t - 1,a}$  and  $\hat{\mu}_{t - 1,\max}$ ,  $\mathsf{kl}(\hat{\mu}_{t - 1,a},\hat{\mu}_{t - 1,\max})$ , as a measure of empirical suboptimality of that arm. The sampling probability of arm  $a$ , denoted by  $p_{t,a}$ , is proportional to the exponential of negative product between  $N_{t - 1,a}$  and  $\mathsf{kl}(\hat{\mu}_{t - 1,a},\hat{\mu}_{t - 1,\max})$  (Eq. (3) of step 6). This policy naturally trades off between exploration and exploitation: arm  $a$  is sampled with higher probability, if either it has not been pulled many times ( $N_{t - 1,a}$  is small) or it appears to be close to optimal empirically ( $\mathsf{kl}(\hat{\mu}_{t - 1,a},\hat{\mu}_{t - 1,\max})$ ) is small). The algorithm samples an arm  $I_t$  from  $p_t$ , and observe a reward  $y_t$  of the arm chosen.
We remark that if the reward distributions  $\nu_{i}$ 's are Bernoulli, KL-MS is equivalent to the MED algorithm [19] since in this case, all reward distributions have a binary support of  $\{0,1\}$ . However, KL-MS is different from MED in general: MED computes the empirical distributions of arm rewards  $\hat{F}_{t - 1,a}$ , and chooses action according to probabilities  $p_{t,a} \propto \exp(-N_{t - 1,a}D_{t - 1,a})$ ; here,  $D_{t - 1,a} \coloneqq \mathsf{KL}(\hat{F}_{t - 1,a},\hat{\mu}_{t - 1,\max})$  (recall its definition in Section 3) is the "minimum empirical divergence" between arm  $a$  and the highest empirical mean reward, which is different from the binary KL divergence of the mean rewards used in KL-MS.
4.1 Main Regret Theorem
Our main result of this paper is the following theorem on the regret guarantee of KL-MS (Algorithm 1). Without loss of generality, throughout the rest of the paper, we assume  $\mu_1 \geq \mu_2 \geq \dots \geq \mu_K$ .
Theorem 1. For any  $K$ -arm bandit problem with reward distribution supported on  $[0,1]$ ,  $KL-MS$  has regret bounded as follows. For any  $\Delta \geq 0$  and  $c \in (0,\frac{1}{4}]$ :
$$
\begin{array}{l} \mathrm {R e g} (T) \leq T \Delta + \sum_ {a: \Delta_ {a} > \Delta} \frac {\Delta_ {a} \ln (T \mathsf {k l} (\mu_ {a} + c \Delta_ {a} , \mu_ {1} - c \Delta_ {a}) \vee e ^ {2})}{\mathsf {k l} (\mu_ {a} + c \Delta_ {a} , \mu_ {1} - c \Delta_ {a})} \\ + 5 6 0 \sum_ {a: \Delta_ {a} > \Delta} \left(\frac {\dot {\mu} _ {1} + \Delta_ {a}}{c ^ {2} \Delta_ {a}}\right) \ln \left(\left(\frac {\dot {\mu} _ {1} + \Delta_ {a}}{c ^ {2} \Delta_ {a} ^ {2}} \wedge \frac {c ^ {2} T \Delta_ {a} ^ {2}}{\dot {\mu} _ {1} + \Delta_ {a}}\right) \vee e ^ {2}\right) \tag {4} \\ \end{array}
$$
The regret bound of Theorem 1 is composed of three terms. The first term is  $T\Delta$ , which controls the contribution of regret from all  $\Delta$ -near-optimal arms. The second term is asymptotically  $(1 + o(1))\sum_{a:\Delta_a > 0}\frac{\Delta_a}{kl(\mu_a,\mu_1)}\ln (T)$  with an appropriate choice of  $c$ , which is a term that grows in  $T$  in a logarithmic rate. The third term is simultaneously upper bounded by two expressions. One is  $\sum_{a:\Delta_a > 0}\left(\frac{\dot{\mu}_1 + \Delta_a}{c^2\Delta_a}\right)\ln \left(\frac{c^2T\Delta_a^2}{\mu_1 + \Delta_a}\vee e^2\right)$ , which is of order  $\ln T$  and helps establish a tight worst-case regret bound (Theorem 3); the other is  $\sum_{a:\Delta_a > 0}\left(\frac{\dot{\mu}_1 + \Delta_a}{c^2\Delta_a}\right)\ln \left(\left(\frac{\dot{\mu}_1 + \Delta_a}{c^2\Delta_a^2}\right)\vee e^2\right)$ , which does not grow in  $T$  and helps establish a tight asymptotic upper bound on the regret (Theorem 4).
To the best of our knowledge, existing regret analysis on Bernoulli bandits or bandits with bounded support have regret bounds of the form
$$
\operatorname {R e g} (T) \leq T \Delta + \sum_ {a: \Delta_ {a} > \Delta} \frac {\Delta_ {a} \ln (T)}{\operatorname {k l} (\mu_ {a} + c \Delta_ {a} , \mu_ {1} - c \Delta_ {a})} + O \left(\sum_ {a: \Delta_ {a} > \Delta} \frac {1}{c ^ {2} \Delta_ {a}}\right),
$$
for some  $c > 0$ , where the third term is much larger than its counterpart given by Theorem 1 when  $\Delta_{a}$  and  $\dot{\mu}_{1}$  are small. As we will see shortly, as a consequence of its tighter bounds, our regret theorem yields a superior worst-case regret guarantee over previous works.
Theorem 2 (Sub-UCB).  $KL$ -MS's regret is bounded by  $\operatorname{Reg}(T) \lesssim \sum_{a: \Delta_a > 0} \frac{\ln T}{\Delta_a}$ . Therefore,  $KL$ -MS is sub-UCB.
Sub-UCB criterion is important for measuring a bandit algorithm's finite-time instance-dependent performance. Indeed, Lattimore [28, §3] points out that MOSS [6] does not satisfy sub-UCB and that it leads to a strictly suboptimal regret in a specific instance compared to the standard UCB algorithm [9]. A close inspection of the finite-time regret bounds of existing asymptotically optimal and minimax optimal algorithms for the  $[0,1]$ -reward setting, such as KL-UCB++ [33] and KL-UCB-switch [18], reveals that they are not sub-UCB. Thus, we speculate that they would also have a suboptimal performance in the aforementioned instance.
In light of Theorem 1, our first corollary is that KL Maillard sampling achieves the following adaptive worst-case regret guarantee.
Theorem 3 (Adaptive worst-case regret). For any  $K$ -arm bandit problem with reward distribution supported on  $[0,1]$ ,  $KL-MS$  has regret bounded as:  $\operatorname{Reg}(T) \lesssim \sqrt{\mu_1KT\ln K} + K\ln T$ .
An immediate corollary is that KL Maillard sampling has a regret of order  $O(\sqrt{KT\ln K})$ , which is a factor of  $O(\sqrt{\ln K})$  within the minimax optimal regret  $\Theta (\sqrt{KT})$  [33, 5]. This also matches the worst-case regret bound  $O(\sqrt{VKT\ln(K)})$  of Jin et al. [24] where  $V = \frac{1}{4}$  is the worst-case variance for Bernoulli bandits using a Thompson sampling-style algorithm. Another main feature of this regret bound is its adaptivity to  $\dot{\mu}_1$ , the variance of the reward of the optimal arm for the Bernoulli bandit setting, or its upper bound in the general bounded reward setting (see Lemma 16). Specifically, if  $\mu_1$  is close to 0 or 1,  $\dot{\mu}_1$  is very small, which results in the regret being much smaller than  $O(\sqrt{KT\ln K})$ .
Note that UCB-V [7] and KL-UCB/KL-UCB++, while not reported, enjoy a worst-case regret bound of  $O(\sqrt{\dot{\mu}_1KT\ln T})$ , which is worse than our bound in its logarithmic factor; see Appendix F.3 and F.1 for the proofs. Among these, UCB-V does not achieve the asymptotic optimality for the Bernoulli case. While logistic linear bandits [1, 32] can be applied to Bernoulli  $K$ -armed bandits and achieve similar worst-case regret bounds involving  $\dot{\mu}_1$ , their lower order term can be much worse as discussed in Section 3.
Our second corollary is that KL Maillard sampling achieves a tight asymptotic regret guarantee for the special case of Bernoulli rewards:
Theorem 4. (Asymptotic Optimality) For any  $K$ -arm bandit problem with reward distribution supported on  $[0,1]$ , KL-MS satisfies the following asymptotic regret upper bound:
$$
\lim  _ {T \rightarrow \infty} \sup  _ {\Delta_ {a} > 0} \frac {\operatorname {R e g} (T)}{\ln (T)} = \sum_ {a \in [ K ]: \Delta_ {a} > 0} \frac {\Delta_ {a}}{\mathrm {k l} \left(\mu_ {a} , \mu_ {1}\right)} \tag {5}
$$
Specialized to the Bernoulli bandit setting, in light of the asymptotic lower bounds [27, 12], the above asymptotic regret upper bound implies that KL-MS is asymptotically optimal.
While the regret guarantee of KL-MS is not asymptotically optimal for the general  $[0,1]$  bounded reward setting, it nevertheless is a better regret guarantee than naively viewing this problem as a sub-Gaussian bandit problem and applying sub-Gaussian bandit algorithms on it. To see this, note that any reward distribution supported on  $[0,1]$  is  $\frac{1}{4}$ -sub-Gaussian; therefore, standard sub-Gaussian bandit algorithms will yield an asymptotic regret  $(1 + o(1))\sum_{a\in [K]:\Delta_a > 0}\frac{\ln T}{2\Delta_a}$ . This is always no better than the asymptotic regret provided by Eq. (5), in view of Pinsker's inequality that  $\mathrm{kl}(\mu_a,\mu_1)\geq 2\Delta_a^2$
5 Proof Sketch of Theorem 1
We provide an outline of our proof of Theorem 1, with full proof details deferred to Appendix C. Our approach is akin to the recent analysis of the sub-Gaussian Maillard Sampling algorithm in Bian and Jun [11] with several refinements tailored to the bounded reward setting and achieving  $\sqrt{\ln K}$  minimax ratio. First, for any time horizon length  $T$ ,  $\operatorname{Reg}(T)$  can be bounded by:
$$
\operatorname {R e g} (T) = \sum_ {a \in [ K ]: \Delta_ {a} > 0} \Delta_ {a} \mathbb {E} \left[ N _ {T, a} \right] \leq \Delta T + \sum_ {a \in [ K ]: \Delta_ {a} > \Delta} \Delta_ {a} \mathbb {E} \left[ N _ {T, a} \right], \tag {6}
$$
i.e., the total regret can be decomposed to a  $T\Delta$  term and the sum of regret  $\Delta_{a}\mathbb{E}\left[N_{T,a}\right]$  from pulling  $\Delta$ -suboptimal arms  $a$ . Therefore, in subsequent analysis, we focus on bounding  $\mathbb{E}\left[N_{T,a}\right]$ . To this end, we show the following lemma.
Lemma 5. For any suboptimal arm  $a$ , let  $\varepsilon_1, \varepsilon_2 > 0$  be such that  $\varepsilon_1 + \varepsilon_2 < \Delta_a$ . Then its expected number of pulls is bounded as:
$$
\begin{array}{l} \mathbb {E} \left[ N _ {T, a} \right] \leq 1 + \frac {\ln \left(T \mathrm {k l} \left(\mu_ {a} + \varepsilon_ {1} , \mu_ {1} - \varepsilon_ {2}\right) \vee e ^ {2}\right)}{\mathrm {k l} \left(\mu_ {a} + \varepsilon_ {1} , \mu_ {1} - \varepsilon_ {2}\right)} + \frac {1}{\mathrm {k l} \left(\mu_ {a} + \varepsilon_ {1} , \mu_ {1} - \varepsilon_ {2}\right)} + \frac {1}{\mathrm {k l} \left(\mu_ {a} + \varepsilon_ {1} , \mu_ {a}\right)} \\ + 6 H \ln \left(\left(\frac {T}{H} \wedge H\right) \vee e ^ {2}\right) + \frac {4}{\mathrm {k l} \left(\mu_ {1} - \varepsilon_ {2} , \mu_ {1}\right)}, \tag {7} \\ \end{array}
$$
where  $H \coloneqq \frac{1}{(1 - \mu_1 + \varepsilon_2)(\mu_1 - \varepsilon_2)h^2(\mu_1,\varepsilon_2)} \lesssim \frac{2\dot{\mu}_1 + \varepsilon_2}{\varepsilon_2^2}$  and  $h(\mu_1, \varepsilon_2) \coloneqq \ln \left(\frac{(1 - \mu_1 + \varepsilon_2)\mu_1}{(1 - \mu_1)(\mu_1 - \varepsilon_2)}\right)$ .
Theorem 1 follows immediately from Lemma 5. See section C for details; we show a sketch here.
Proof sketch of Theorem 1. Fix any  $c \in (0, \frac{1}{4}]$ . Let  $\varepsilon_1 = \varepsilon_2 = c\Delta_a$ ; by the choice of  $c, \varepsilon_1 + \varepsilon_2 < \Delta_a$ . From Lemma 5,  $\mathbb{E}\left[N_{T,a}\right]$  is bounded by Eq. (7). Plugging in the values of  $\varepsilon_1 = \varepsilon_2$ , and using Lemma 26 that lower bounds the binary KL divergence, along with Lemma 22 that gives  $H \lesssim \frac{2\dot{\mu}_1 + \varepsilon_2}{\varepsilon_2^2}$  and algebra, all terms except the second term on the right hand side of Eq. (7) are bounded by
$$
\left(\frac {3 4}{c ^ {2}} + \frac {4}{(1 - 2 c) ^ {2}}\right) \left(\frac {\dot {\mu} _ {1} + \Delta_ {a}}{c ^ {2} \Delta_ {a} ^ {2}}\right) \ln \left(\left(\frac {\dot {\mu} _ {1} + \Delta_ {a}}{c ^ {2} \Delta_ {a} ^ {2}} \wedge \frac {c ^ {2} T \Delta_ {a} ^ {2}}{\dot {\mu} _ {1} + \Delta_ {a}}\right) \vee e ^ {2}\right).
$$
As a result, KL-MS satisfies that, for any arm  $a$ , for any  $c \in (0, \frac{1}{4}]$ :
$$
\begin{array}{l} \mathbb {E} \left[ N _ {T, a} \right] \leq \frac {\ln (T \mathsf {k l} (\mu_ {a} + c \Delta_ {a} , \mu_ {1} - c \Delta_ {a}) \vee e ^ {2})}{\mathsf {k l} (\mu_ {a} + c \Delta_ {a} , \mu_ {1} - c \Delta_ {a})} \\ + \left(\frac {3 4}{c ^ {2}} + \frac {4}{(1 - 2 c) ^ {2}}\right) \left(\frac {\dot {\mu} _ {1} + \Delta_ {a}}{c ^ {2} \Delta_ {a} ^ {2}}\right) \ln \left(\left(\frac {\dot {\mu} _ {1} + \Delta_ {a}}{c ^ {2} \Delta_ {a} ^ {2}} \wedge \frac {c ^ {2} T \Delta_ {a} ^ {2}}{\dot {\mu} _ {1} + \Delta_ {a}}\right) \vee e ^ {2}\right). \\ \end{array}
$$
Theorem 1 follows by plugging the above bound to Eq. (6) for arms  $a$  s.t.  $\Delta_{a} > \Delta$  with  $c = \frac{1}{4}$ .
5.1 Proof sketch of Lemma 5
We sketch the proof of Lemma 5 in this subsection. For full details of the proof, please refer to Appendix C.2. We first set up some useful notations that will be used throughout the proof. Let  $u := \lceil \frac{\ln(T\mathrm{kl}(\mu_a + \varepsilon_1,\mu_1 - \varepsilon_2)\vee e^2)}{\mathrm{kl}(\mu_a + \varepsilon_1,\mu_1 - \varepsilon_2)} \rceil$ . We define the following events
$$
A _ {t} := \left\{I _ {t} = a \right\}, \quad B _ {t} := \left\{N _ {t, a} <   u \right\}, \quad C _ {t} := \left\{\hat {\mu} _ {t, \max } \geq \mu_ {1} - \varepsilon_ {2} \right\}, \quad D _ {t} := \left\{\hat {\mu} _ {t, a} \leq \mu_ {a} + \varepsilon_ {1} \right\},
$$
By algebra, one has the following elementary upper bound on  $\mathbb{E}\left[N_{T,a}\right]$ :  $\mathbb{E}\left[N_{T,a}\right] \leq u + \mathbb{E}\left[\sum_{t = K + 1}^{T} \mathbf{1}\left\{A_{t}, B_{t - 1}^{c}\right\}\right]$ . Intuitively, the  $u$  term serves to control the length of a "burn-in" phase when the number of pulls to arm  $a$  is at most  $u$ . It now remains to control the second term, the number of pulls to arm  $a$  after it is large enough, i.e.,  $N_{t - 1,a} \geq u$ . We decompose it to  $F1$ ,  $F2$ , and  $F3$ , resulting in the following inequality:
$$
\begin{array}{l} \mathbb {E} \left[ N _ {T, a} \right] \leq u + \underbrace {\mathbb {E} \left[ \sum_ {t = K + 1} ^ {T} \mathbf {1} \left\{A _ {t} , B _ {t - 1} ^ {c} , C _ {t - 1} , D _ {t - 1} \right\} \right]} _ {=: F 1} \\ + \underbrace {\mathbb {E} \left[ \sum_ {t = K + 1} ^ {T} \mathbf {1} \left\{A _ {t} , B _ {t - 1} ^ {c} , C _ {t - 1} , D _ {t - 1} ^ {c} , \right\} \right]} _ {=: F 2} + \underbrace {\mathbb {E} \left[ \sum_ {t = K + 1} ^ {T} \mathbf {1} \left\{A _ {t} , B _ {t - 1} ^ {c} , C _ {t - 1} ^ {c} \right\} \right]} _ {=: F 3} \\ \end{array}
$$
Here:
-  $F1$  corresponds to the "steady state" when the empirical means of arm  $a$  and the optimal arm are both estimated accurately, i.e.,  $\hat{\mu}_{t - 1,\max} \geq \mu_1 - \varepsilon_2$  and  $\hat{\mu}_{t - 1,a} \leq \mu_a + \varepsilon_1$ . It can be straightforwardly bounded by  $\frac{1}{\mathrm{kl}(\mu_a + \varepsilon_1, \mu_1 - \varepsilon_2)}$ , as we show in Lemma 10 (section D.1).
-  $F2$  corresponds to the case when the empirical mean of arm  $a$  is abnormally high, i.e.,  $\hat{\mu}_{t - 1,a} > \mu_{a} + \varepsilon_{1}$ . It can be straightforwardly bounded by  $\frac{1}{\mathrm{kl}(\mu_a + \varepsilon_1,\mu_a)}$ , as we show in Lemma 11 (section D.2).
-  $F3$  corresponds to the case when the empirical mean of the optimal arm is abnormally low, i.e.,  $\hat{\mu}_{t - 1,\max} \leq \mu_1 - \varepsilon_2$ ; it is the most challenging term and we discuss our techniques in bounding it in detail below (section D.3).
We provide an outline of our analysis of  $F3$  in Appendix D.3.1 and sketch its main ideas and technical challenges here.
We follow the derivation from Bian and Jun [11] by first using a probability transferring argument (Lemma 23) to bound the expected counts of pulling suboptimal arm  $a$  by the expectation of indicators of pulling the optimal arm with a multiplicative factor and then change the counting from global time step  $t$  to local count of pulling the optimal arm. Then,  $F3$  is bounded by,
$$
\sum_ {k = 1} ^ {\infty} \underbrace {\mathbb {E} \left[ \mathbf {1} \left\{\hat {\mu} _ {(k) , 1} \leq \mu_ {1} - \varepsilon_ {2} \right\} \exp (k \cdot \mathrm {k l} (\hat {\mu} _ {(k) , 1} , \mu_ {1} - \varepsilon_ {2}) \right]}. _ {M _ {k}}
$$
Intuitively, each  $M_{k}$  should be controlled: when  $\exp (k\cdot \mathrm{kl}(\hat{\mu}_{(k),1},\mu_1 - \varepsilon_2))$  is large,  $\hat{\mu}_{(k),1}$  must significantly negatively deviate from  $\mu_{1} - \varepsilon_{2}$ , which happens with low probability by Chernoff bound (Lemma 25). Using a double integration argument, we can bound each  $M_{k}$  by
$$
M _ {k} \leq \left(\frac {2 H}{k} + 1\right) \exp (- k \mathrm {k l} (\mu_ {1} - \varepsilon_ {2}, \mu_ {1})).
$$
Summing over all  $k$ , we can bound  $F3_{1}$  by  $O\left(H\ln \left(H\vee e^{2}\right) + \frac{1}{\mathrm{kl}(\mu_{1} - \varepsilon_{2},\mu_{1})}\right)$ . Combining the bounds on  $F1$  and  $F2$ , we can show a bound on  $\mathbb{E}\left[N_{T,a}\right]$  similar to Eq. (7) without the “ $\frac{T}{H}\wedge$ ” term in the logarithmic factor. This yields a regret bound of KL-MS, in the form of Eq. (4) without the “ $\bigwedge \frac{c^2T\Delta_a^2}{\dot{\mu}_1 + \Delta_a}$ ” term in the logarithmic factor. Such a regret bound can be readily used to show KL-MS's
Bernoulli asymptotic optimality and sub-UCB property. An adaptive worst-case regret bound of  $\sqrt{\dot{\mu}_1KT\ln(T)}$  also follows immediately.
To show that MS has a tighter adaptive worst-case regret bound of  $\sqrt{\dot{\mu}_1KT\ln(K)}$ , we adopt a technique in [33, 24]. First, we observe that the looseness of the above bound on  $F3$  comes from small  $k$  (denoted as  $F3_1 \coloneqq \sum_{k \leq H} M_k$ ), as the summation of  $M_k$  for large  $k$  (denoted as  $F3_2 \coloneqq \sum_{k > H} M_k$ ) is well-controlled. The key challenge in a better control of  $F3_1$  comes from the difficulty in bounding the tail probability of  $\hat{\mu}_{(k),1}$  for  $k < H$  beyond Chernoff bound. To cope with this, we observe that a modified version of  $F3_1$  that contains an extra favorable indicator of  $\mathrm{kl}(\hat{\mu}_{(k),1}, \mu_1) \leq \frac{2\ln(T / k)}{k}$ , denoted as:
$$
\sum_ {k \leq H} \mathbb {E} \left[ 1 \left\{\hat {\mu} _ {(k), 1} \leq \mu_ {1} - \varepsilon_ {2}, \operatorname {k l} \left(\hat {\mu} _ {(k), 1}, \mu_ {1}\right) \leq \frac {2 \ln (T / k)}{k} \right\} \exp \left(k \cdot \operatorname {k l} \left(\hat {\mu} _ {(k), 1}, \mu_ {1} - \varepsilon_ {2}\right) \right. \right]
$$
can be well-controlled. Utilizing this introduces another term in the regret analysis,  $T \cdot \mathbb{P}(\mathcal{E}^C)$ , where  $\mathcal{E} = \{\forall k \in [1, H], \mathrm{kl}(\hat{\mu}_{(k),1}, \mu_1) \leq \frac{2\ln(T / k)}{k}\}$ , which we bound by  $O(H)$  via a time-uniform version of Chernoff bound. Putting everything together, we prove a bound of  $F3$  of  $O\left(H\ln\left(\left(\frac{T}{H} \wedge H\right) \vee e^2\right) + \frac{1}{\mathrm{kl}(\mu_1 - \varepsilon_2, \mu_1)}\right)$ , which yields our final regret bound of KL-MS in Theorem 1 and the refined minimax ratio.
Remark 6. Although our technique is inspired by [24, 33], we carefully set the case splitting threshold for  $N_{t-1,1}$  (to obtain  $F3_1$  and  $F3_2$ ) to be  $H = O\left(\frac{\mu_1 + \epsilon_2}{\epsilon_2^2}\right)$ , which is significantly different from prior works ( $\tilde{O}\left(\frac{1}{\epsilon_2^2}\right)$ ).
Remark 7. One can port our proof strategy back to sub-Gaussian MS and show that it achieves a minimax ratio of  $\sqrt{\ln K}$  as opposed to  $\sqrt{\ln T}$  reported in Bian and Jun [11]; a sketch of the proof is in Appendix G. Recall that Bian and Jun [11] proposed another algorithm  $MS^{+}$  that achieved the minimax ratio of  $\sqrt{\ln K}$  at the price of extra exploration. Our result makes  $MS^{+}$  obsolete; MS should be preferred over  $MS^{+}$  at all times.
We have proposed KL-MS, a KL version of Maillard sampling for stochastic multi-armed bandits in the  $[0,1]$ -bounded reward setting, with a closed-form probability computation, which is highly amenable to off-policy evaluation. Our algorithm requires constant time complexity with respect to the target numerical precision in computing the action probabilities, and our regret analysis shows that KL-MS achieves the best regret bound among those in the literature that allows computing the action probabilities with  $O(\mathrm{polylog}(1 / \mathrm{precision}))$  time complexity, for example, Tsallis-INF [43], EXP3++ [40], in the stochastic setting.
Our study opens up numerous open problems. One immediate open problem is to generalize KL-MS to handle exponential family reward distributions. Another exciting direction is to design randomized and off-policy-amenable algorithms that achieve the asymptotic optimality for bounded rewards (i.e., as good as IMED [21]).
One possible avenue is to extend MED [19] and remove the restriction that the reward distribution must have bounded support. Furthermore, it would be interesting to extend MS to structured bandits and find connections to the Decision-Estimation Coefficient [16], which have recently been reported to characterize the optimal minimax regret rate for structured bandits. Finally, we believe MS is practical by incorporating the booster hyperparameter introduced in Bian and Jun [11]. Extensive empirical evaluations on real-world problems would be an interesting future research direction.
[1] M. Abeille, L. Faury, and C. Calauzenes. Instance-Wise Minimax-Optimal Algorithms for Logistic Bandits. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), pages 3691–3699, 2021.
[42] C.-Y. Wei and H. Luo. More Adaptive Algorithms for Adversarial Bandits. In Proceedings of the Conference on Learning Theory (COLT), pages 1263-1291, 2018. URL http://arxiv.org/abs/1801.03265.
[43] J. Zimmert and Y. Seldin. Tsallis-inf: An optimal algorithm for stochastic and adversarial bandits. The Journal of Machine Learning Research, 22(1):1310-1358, 2021.