SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hao Qin
University of Arizona hqin@arizona.edu
Kwang-Sung Jun
University of Arizona kjun@cs.arizona.edu
Chicheng Zhang
University of Arizona  
chichengz@cs.arizona.edu
We study  $K$ -armed bandit problems where the reward distributions of the arms are all supported on the [0, 1] interval. Maillard sampling [30], an attractive alternative to Thompson sampling, has recently been shown to achieve competitive regret guarantees in the sub-Gaussian reward setting [11] while maintaining closed-form action probabilities, which is useful for offline policy evaluation. In this work, we analyze the Kullback-Leibler Maillard Sampling (KL-MS) algorithm, a natural extension of Maillard sampling and a special case of Minimum Empirical Divergence (MED) [19] for achieving a KL-style finite-time gap-dependent regret bound. We show that KL-MS enjoys the asymptotic optimality when the rewards are Bernoulli and has an adaptive worst-case regret bound of the form  $O(\sqrt{\mu^{*}(1 - \mu^{*})KT\ln K} +K\ln T)$ , where  $\mu^{*}$  is the expected reward of the optimal arm, and  $T$  is the time horizon length; this is the first time such adaptivity is reported in the literature for an algorithm with asymptotic optimality guarantees.
The multi-armed bandit (abbrev. MAB) problem [41, 27, 29], a stateless version of the reinforcement learning problem, has received much attention by the research community, due to its relevance in many applications such as online advertising, recommendation, and clinical trials. In a multi-armed bandit problem, a learning agent has access to a set of  $K$  arms (also known as actions), where for each  $i \in [K] := \{1, \dots, K\}$ , arm  $i$  is associated with a distribution  $\nu_{i}$  with mean  $\mu_{i}$ ; at each time step  $t$ , the agent adaptively chooses an arm  $I_{t} \in [K]$  by sampling from a probability distribution  $p_{t} \in \Delta^{K-1}$  and receives reward  $y_{t} \sim \nu_{I_{t}}$ , based on the information the agent has so far. The goal of the agent is to minimize its pseudo-regret over  $T$  time steps:  $\mathrm{Reg}(T) = T\mu^{*} - \mathbb{E}\sum_{t=1}^{T} y_{t}$ , where  $\mu^{*} = \max_{i} \mu_{i}$  is the optimal expected reward.
In this paper, we study the multi-armed bandit setting where reward distributions of all arms are supported on  $[0,1]$ . An important special case is Bernoulli bandits, where for each arm  $i$ ,  $\nu_{i} = \operatorname{Bernoulli}(\mu_{i})$  for some  $\mu_{i} \in [0,1]$ . It has practical relevance in settings such as computational advertising, where the reward feedback is oftentimes binary (click vs. not-click, buy vs. not-buy).
Broadly speaking, there are two popular families of provably regret-efficient algorithms for bounded-reward bandit problems: deterministic exploration algorithms (such as KL-UCB [17, 13, 31]) and randomized exploration algorithms (such as Thompson sampling (TS) [41]). Randomized exploration algorithms such as TS have been very popular, perhaps due to its excellent empirical performance and the ability to cope with delayed rewards better than deterministic counterparts [15]. In addition, the logged data collected from randomized exploration, of the form  $(I_{t}, p_{t, I_{t}}, y_{t})_{t=1}^{T}$ , where  $p_{t, I_{t}}$  is the probability with which arm  $I_{t}$  was chosen, are useful for offline evaluation purposes by
<sup>1</sup>All of our results can be extended to distributions supported in  $[L, U]$  for any known  $L \leq U$  by shifting and scaling the rewards to lie in  $[0, 1]$ .
Figure 1: Histogram of the average rewards computed from the offline evaluation where the logged data is collected from Bernoulli TS and KL-MS (Algorithm 1) in a Bernoulli bandit environment with the mean reward (0.8, 0.9) with time horizon  $T = 10,000$ . For Bernoulli TS's log, we approximate the action probability by Monte Carlo Sampling with 1000 samples for each step. Here we estimate the expected reward of the uniform policy which has expected average reward of 0.85 (black dashed line). Across 2000 trials, the logged data of KL-MS induces an MSE of 0.00796; however, for half of the trials, the IPW estimator induced by Bernoulli TS's log returns invalid values due to the action probability estimates being zero. Even excluding those invalid values, the Bernoulli TS's logged data induces an MSE of 0.02015. See Appendix H for additional experiments.
employing the inverse propensity weighting (IPW) estimator [22] or the doubly robust estimator [38]. However, calculating the arm sampling probability distribution  $p_t$  for Thompson sampling is nontrivial. Specifically, there is no known closed-form  $^2$ , and generic numerical integration methods and Monte-Carlo approximations suffer from instability issues: the time complexity for obtaining a numerical precision of  $\epsilon$  is  $\Omega(\mathrm{poly}(1/\epsilon))$  [36]. This is too slow to be useful especially for web-scale deployments; e.g., Google AdWords receives  $\sim 237\mathrm{M}$  clicks per day. Furthermore, the computed probability will be used after taking the inversion, which means that even moderate amount of errors are intolerable. Indeed, Figure 1 shows that the offline evaluation with Thompson sampling as the behavioral policy will be largely biased and inaccurate due to the errors from the Monte Carlo approximation.
Recently, many studies have introduced alternative randomized algorithms that allow an efficient computation of  $p_t$  [19, 30, 14, 43]. Of these, Maillard sampling (MS) [30, 11], a Gaussian adaptation of the Minimum Empirical Divergence (MED) algorithm [19] originally designed for finite-support reward distributions, provides a simple algorithm for the sub-Gaussian bandit setting that computes  $p_t$  in a closed form:
$$
p _ {t, a} \propto \exp \left(- N _ {t - 1, a} \frac {\hat {\Delta} _ {t - 1 , a} ^ {2}}{2 \sigma^ {2}}\right) \tag {1}
$$
where at time step  $t$ ,  $N_{t,a}$  is the number of pulling arm  $a$ . We define the estimator of  $\mu_{a}$  as  $\hat{\mu}_{t,a} \coloneqq \frac{\sum_{s=1}^{t} \mathbf{1}\{I_t = a\} y_t}{N_{t,a}}$  and the best performed mean value as  $\hat{\mu}_{t,\max} \coloneqq \max_{a \in [K]} \mu_{t,a}$ .  $\hat{\Delta}_{t-1,a} = \max_{a'} \hat{\mu}_{t-1,a'} - \hat{\mu}_{t-1,a}$  is the empirical suboptimality gap of arm  $a$ , and  $\sigma$  is the subgaussian parameter of the reward distribution of all arms. For sub-Gaussian reward distributions, MS enjoys the asymptotic optimality under the special case of Gaussian rewards and a near-minimax optimality [11], making it an attractive alternative to Thompson sampling. Also, MS satisfies the sub-UCB criterion (see Section 2 for a precise definition) to help establish sharp finite-time instance-dependent regret guarantees. Can we adapt MS to the bounded reward setting and achieve the asymptotic, minimax optimality and sub-UCB criterion while computing the sampling probability in a closed-form? In this paper, we make significant progress on this question.
Our contributions. We focus on a Bernoulli adaptation of MS that we call Kullback-Leibler Maillard Sampling (abbrev. KL-MS) and perform a finite-time analysis of it in the bounded-reward bandit problem. KL-MS uses a sampling probability similar to MS but tailored to the  $[0,1]$ -bounded reward setting:
$$
p _ {t, a} \propto \exp \left(- N _ {t - 1, a} k l \left(\hat {\mu} _ {t - 1, a}, \hat {\mu} _ {t - 1, \max }\right)\right),
$$
where  $\mathrm{kl}(\mu, \mu') \coloneqq \mu \ln \frac{\mu}{\mu'} + (1 - \mu) \ln \frac{1 - \mu}{1 - \mu'}$  is the binary Kullback-Leibler (KL) divergence. We can also view KL-MS as an instantiation of MED [19] for Bernoulli rewards; See Section 3 for a
Suppose the arms' mean reward posterior distributions' PMFs and PDFs are  $(p_1, \ldots, p_K)$  and  $(F_1, \ldots, F_K)$  respectively; for example, they are Beta distributions with different parameters, i.e.  $p_i(x) = x^{a_i - 1}(1 - x)^{b_i - 1} I(x \in [0, 1])$  for some  $a_i, b_i$ . To the best of our knowledge, the action probabilities have the following integral expression:  $\mathbb{P}(I_t = a) = \int_{\mathbb{R}} p_a(x) \prod_{i \neq a} F_i(x) \mathrm{d}x$  and cannot be further simplified.
Algorithm& Analysis	Finite-Time Regret	Closed-form Probability	Reference
Minimax Ratio	Sub-UCB
TS	√lnK	yes	no	See the caption
ExpTS	√lnK	yes	no	Jin et al. [24]
ExpTS+	1	-**	no	Jin et al. [24]
kl-UCB	√lnT	yes	N/A	Cappé et al. [13]
kl-UCB++	1	-**	N/A	Ménard and Garivier [33]
kl-UCB-switch	1	-**	N/A	Garivier et al. [18]
MED	-	-	no*	Honda and Takemura [19]
DMED	-	-	N/A	Honda and Takemura [20]
IMED	-	-	N/A	Honda and Takemura [21]
KL-MS	√lnK	yes	yes	this paper
Table 1: Comparison of regret bounds for bounded reward distributions; for space constraints we only include those that achieves the asymptotic optimality for the special case of Bernoulli distributions (this excludes, e.g., Maillard Sampling [30, 11], Tsallis-INF [43] and UCB-V [8]).  $-$ indicates that the corresponding analysis is not reported. N/A'indicates that the algorithm does have closed-form, but it is deterministic.  $\star$  indicates that its computational complexity for calculating the action probability is  $\ln(1/\text{precision})$ .  $\star\star$  indicates that we conjecture that the algorithm is not sub-UCB. The results on TS are reported by Agrawal and Goyal [3, 4], Korda et al. [26].
detailed comparison. KL-MS performs an efficient exploration for bounded rewards since one can use  $\mathrm{kl}(a,b)\geq 2(a - b)^2$  to verify that the probability being assigned to each empirical non-best arm by KL-MS is never larger than that of MS with  $\sigma^2 = 1 / 4$ , the best sub-Gaussian parameter for the bounded rewards in [0, 1]. We show that KL-MS achieves a sharp finite-time regret guarantee (Theorem 1) that can be simultaneously converted to:
- an asymptotic regret upper bound (Theorem 4), which is asymptotically optimal when specialized to the Bernoulli bandit setting;
- a  $\sqrt{T}$ -style regret guarantee of  $O(\sqrt{\mu^{*}(1 - \mu^{*})KT\ln K} + K\ln (T))$  (Theorem 3) where  $\mu^{*}$  is the mean reward of the best arm. This bound has two salient features. First, in the worst case, it is at most a  $\sqrt{\ln K}$  factor suboptimal than the minimax optimal regret of  $\Theta (\sqrt{KT})$  [5, 10]. Second, its  $\tilde{O} (\sqrt{\mu^{*}(1 - \mu^{*})}$  coefficient adapts to the variance of the optimal arm reward; this is the first time such adaptivity is reported in the literature for an algorithm with asymptotical optimality guarantees.
- a sub-UCB regret guarantee, which many existing minimax optimal algorithms [33, 18] have not been proven to satisfy.
We also conduct experiments that show that thanks to its closed-form action probabilities, KL-MS generates much more reliable logged data than Bernoulli TS with Monte Carlo estimation of action probabilities; this is reflected in their offline evaluation performance using the IPW estimator; see Figure 1 and Appendix H for more details.
Let  $N_{t,a}$  be the number of times arm  $a$  has been pulled until time step  $t$  (inclusively). Denote the suboptimality gap of arm  $a$  by  $\Delta_a := \mu^* - \mu_a$ , where  $\mu^* = \max_{i \in [K]} \mu_i$  is the optimal expected reward. Denote the empirical suboptimality gap of arm  $a$  by  $\hat{\Delta}_{t,a} := \hat{\mu}_{t,\max} - \hat{\mu}_{t,a}$ ; here,  $\hat{\mu}_{t,a}$  is the empirical estimation to  $\mu_a$  up to time step  $t$ , i.e.,  $\hat{\mu}_{t,a} := \frac{1}{N_{t,a}} \sum_{s=1}^{t} y_s \mathbf{1}\{I_s = a\}$ , and  $\hat{\mu}_{t,\max} = \max_{a \in [K]} \hat{\mu}_{t,a}$  is the best empirical reward at time step  $t$ . For arm  $a$ , define  $\tau_a(s) := \min\{t \geq 1 : N_{t,a} = s\}$  at the time step when arm  $a$  is pulled for the  $s$ -th time, which is a stopping time; we also use  $\hat{\mu}_{(s),a} := \hat{\mu}_{\tau_a(s),a}$  to denote empirical mean of the first  $s$  reward values received from pulling arm  $a$ .
<sup>3</sup>As side results, we show in Appendix F that with some modifications of the analysis, existing algorithms [7, 13, 33] also achieve regret of the form  $\tilde{O}(\sqrt{\mu^{*}(1 - \mu^{*})\mathrm{poly}(K)T})$  for [0, 1]-bounded reward MABs.
We define the Kullback-Leibler divergence between two distributions  $\nu$  and  $\rho$  as  $\mathsf{KL}(\nu, \rho) = \mathbb{E}_{X \sim \nu}\left[\ln \frac{\mathrm{d}\nu}{\mathrm{d}\rho}(X)\right]$  if  $\nu$  is absolutely continuous w.r.t.  $\rho$ , and  $= +\infty$  otherwise. Recall that we define the binary Kullback-Leibler divergence between two numbers  $\mu, \mu'$  in  $[0,1]$  as  $\mathsf{kl}(\mu, \mu') := \mu \ln \frac{\mu}{\mu'} + (1 - \mu) \ln \frac{1 - \mu}{1 - \mu'}$ , which is also the KL divergence between two Bernoulli distributions with mean parameters  $\mu$  and  $\mu'$  respectively. We define  $\dot{\mu} = \mu(1 - \mu)$ , which is the variance of Bernoulli  $(\mu)$  but otherwise an upper bound on any distribution supported on  $[0,1]$  with mean  $\mu$ ; see Lemma 16 for a formal justification.
In the regret analysis, we will oftentimes use the following notation for comparison up to constant factors: define  $f \lesssim g$  (resp.  $f \gtrsim g$ ) to denote that  $f \leq Cg$  (resp.  $f \geq Cg$ ) for some numerical constant  $C > 0$ . We define  $a \lor b$  and  $a \land b$  as  $\max(a, b)$  and  $\min(a, b)$ , respectively. For an event  $E$ , we use  $E^c$  to denote its complement.
Below, we define some useful criteria for measuring the performance of bandit algorithms, specialized to the  $[0,1]$  bounded reward setting.
Asymptotic optimality in the Bernoulli reward setting An algorithm is asymptotically optimal in the Bernoulli reward setting [27, 12] if for any Bernoulli bandit instance  $(\nu_{a} = \mathrm{Bernoulli}(\mu_{a}))_{a\in [K]}$ $\lim \sup_{T\to \infty}\frac{\mathrm{Reg}(T)}{\ln T} = \sum a:\Delta_a > 0\frac{\Delta_a}{\mathrm{kl}(\mu_a,\mu^*)}.$
Minimax ratio The minimax optimal regret of the  $[0,1]$  bounded reward bandit problem is  $\Theta\left(\sqrt{KT}\right)$  [5, 10]. Given a  $K$ -armed bandit problem with time horizon  $T$ , an algorithm has a minimax ratio of  $f(T,K)$  if its has a worst-case regret bound of  $O(\sqrt{KT} f(T,K))$ .
Sub-UCB Sub-UCB is originally defined in the context of sub-Gaussian bandits [29]: given a bandit problem with  $K$  arms whose reward distributions are all sub-Gaussian, an algorithm is said to be sub-UCB if there exists some positive constants  $C_1$  and  $C_2$ , such that for all  $\sigma^2$ -sub-Gaussian bandit instances,  $\operatorname{Reg}(T) \leq C_1 \sum_{a: \Delta_a > 0} \Delta_a + C_2 \sum_{a: \Delta_a > 0} \frac{\sigma^2}{\Delta_a} \ln T$ . Specialized to our setting, as any distribution supported on  $[0,1]$  is also  $\frac{1}{4}$ -sub-Gaussian, and all suboptimal arm gaps  $\Delta_a \in (0,1]$  are such that  $\Delta_a < \frac{1}{\Delta_a}$ , the above sub-UCB criterion simplifies to: there exists some positive constant  $C$ , such that for all  $[0,1]$ -bounded reward bandit instances,  $\operatorname{Reg}(T) \leq C \sum_{a: \Delta_a > 0} \frac{\ln T}{\Delta_a}$ .
Bandits with bounded rewards. Early works of Lai et al. [27], Burnetas and Katehakis [12] show that in the bounded reward setting, for any consistent stochastic bandit algorithm, the regret is lower bounded by  $(1 + o(1))\sum_{a:\Delta_a > 0}\frac{\Delta_a\ln T}{KL_{inf}(\nu_a,\mu^*)}$  and  $KL_{inf}(\nu_{a},\mu^{*})$  is defined as
$$
\mathrm {K L} _ {\inf } (\nu , \mu^ {*}) := \inf  \left\{\mathrm {K L} (\nu , \rho): \mathbb {E} _ {X \sim \rho} [ X ] > \mu^ {*}, \operatorname {s u p p} (\rho) \subset [ 0, 1 ] \right\}, \tag {2}
$$
where the random variable follows a distribution  $\rho$  bounded in  $[0,1]$ . Therefore, any algorithm whose regret upper bound matches the lower bound is said to achieve asymptotic optimality. Cappé et al. [13] propose the KL-UCB algorithm and provide a finite time regret analysis, which is further refined by Lattimore and Szepesvári [29, Chapter 10]. Another line of work establishes asymptotic and finite-time regret guarantees for Thompson sampling algorithms and its variants [2, 4, 25, 24], which, when specialized to the Bernoulli bandit setting, can be combined with Beta priors for the Bernoulli parameters to design efficient algorithms.
A number of studies even go beyond the Bernoulli-KL-type regret bound and adapt to the variance of each arm in the bounded reward setting. UCB-V [7] achieves a regret bound that adapts to the variance. Efficient-UCBV [35] achieves a variance-adaptive regret bound and also an optimal minimax regret bound  $O(\sqrt{KT})$ , but it is not sub-UCB. Honda and Takemura [19] propose the MED algorithm that is asymptotically optimal for bounded rewards, but it only works for rewards that with finite supports. Honda and Takemura [21] propose the Indexed MED (IMED) algorithm that can handle a more challenging case where the reward distributions are supported in  $(- \infty, 1]$ .
As with worst-case regret bounds, first, it is well-known that for Bernoulli bandits as well as bandits with  $[0,1]$  bounded rewards, the minimax optimal regrets are of order  $\Theta (\sqrt{KT})$  [10, 5]. Of the algorithms that enjoy asymptotic optimality under the Bernoulli reward setting described above, KL-UCB [13] has a worst-case regret bound of  $O(\sqrt{KT\ln T})$ , which is refined by the KL-UCB++
algorithm [33] that has a worst-case regret bound of  $O(\sqrt{KT})$ . We also show in Appendix F.1 and F.2 that with some modifications of existing analysis, KL-UCB and KL-UCB++ enjoy a regret bound of  $O(\sqrt{\mu^{*}(1 - \mu^{*})KT\ln T})$  and  $O(\sqrt{\mu^{*}(1 - \mu^{*})K^3T\ln T})$  respectively. Although the regret is worse in the order of  $K$ , it adapts to  $\mu^{*}$  and will have a better regret when  $\mu^{*}$  is small (say,  $\mu^{*} \leq 1 / K^{2}$ ). KL-UCB++[34] and KL-UCB-Switch[18] achieves  $O(\sqrt{KT})$  regret in the finite-time regime and asymptotic optimality, while the sub-UCB criterion has not been satisfied. However, Lattimore [28, §3] shows that MOSS [6] suffers a sub-optimal regret worse than UCB-like algorithms because of not satisfying sub-UCB criteria, and we suspect that KL-UCB-switch experience the same issue as MOSS. For Thompson Sampling style algorithms, Agrawal and Goyal [3] shows that the original Thompson Sampling algorithm has a worst-case regret of  $O(\sqrt{KT\ln K})$ , and the ExpTS+ algorithm [24] has a worst-case regret of  $O(\sqrt{KT})$ .
Randomized exploration for bandits. Many randomized exploration methods have been proposed for multi-armed bandits. Perhaps the most well-known is Thompson sampling [41], which is shown to achieve Bayesian and frequentist-style regret bounds in a broad range of settings [39, 2, 25, 26, 23, 24]. A drawback of Thompson sampling, as mentioned above, is that the action probabilities cannot be obtained easily and robustly. To cope with this, a line of works design randomized exploration algorithms with action probabilities in closed forms. For sub-Gaussian bandits, Cesa-Bianchi et al. [14] propose a variant of the Boltzmann exploration rule (that is, the action probabilities are proportional to exponential to empirical rewards, scaled by some positive numbers), and show that it has  $O\left(\frac{K\ln^2T}{\Delta}\right)$  instance-dependent and  $O\left(\sqrt{KT}\ln K\right)$  worst-case regret bounds respectively, where  $\Delta = \min_{a:\Delta_a > 0}\Delta_a$  is the minimum suboptimality gap. Maillard sampling (MS; Eq. (1)) is an algorithm proposed by the thesis of Maillard [30] where the author reports that MS achieves the asymptotic optimality and has a finite-time regret of order  $\sum_{a:\Delta_a > 0}\left(\frac{\ln T}{\Delta_a} +\frac{1}{\Delta_a^3}\right)$  from which a worst-case regret bound of  $O(\sqrt{KT^{3 / 4}})$  can be derived. MED [19], albeit achieves asymptotic optimality for a broad family of bandits with finitely supported reward distributions, also has a high finite-time regret bound of at least  $\sum_{a:\Delta_a > 0}\left(\frac{\ln T}{\Delta_a} +\frac{1}{\Delta_a^{2|\mathrm{supp}(\nu_1)| - 1}}\right)$ . Recently, Bian and Jun [11] report a refined analysis of Maillard [30]'s sampling rule, showing that it has a finite time regret of order  $\sum_{a:\Delta_a > 0}\frac{\ln(T\Delta_a^2)}{\Delta_a} +O\left(\sum_{a:\Delta_a > 0}\frac{1}{\Delta_a}\ln (\frac{1}{\Delta_a})\right)$ , and additionally enjoys a  $O\left(\sqrt{KT\ln T}\right)$  worst-case regret, and by inflating the exploration slightly (called  $\mathrm{MS}^+$ ), the bound can be improved and enjoy the minimax regret of  $O\left(\sqrt{KT\ln K}\right)$ , which matches the best-known regret bound among those that satisfy sub-UCB criterion, except for AdaUCB. In fact, it is easy to adapt our proof technique in this paper to show that MS, without any further modification, achieves a  $O\left(\sqrt{KT\ln K}\right)$  worst-case regret.
Randomized exploration has also been studied from a nonstochastic bandit perspective [10, 5], where randomization serves both as a tool for exploration and a way to hedge bets against the nonstationarity of the arm rewards. Many recent efforts focus on designing randomized exploration bandit algorithms that achieve "best of both worlds" adaptive guarantees, i.e., achieving logarithmic regret for stochastic environments while achieving  $\sqrt{T}$  regret for adversarial environments [e.g. 43, 42].
Binarization trick. It is a folklore result that bandits with  $[0,1]$  bounded reward distributions can be reduced to Bernoulli bandits via a simple binarization trick: at each time step  $t$ , the learner sees reward  $r_t \in [0,1]$ , draws  $\tilde{r}_t \sim \mathrm{Bernoulli}(r_t)$  and feeds it to a Bernoulli bandit algorithm. However, this reduction does not result in asymptotic optimality for the general bounded reward setting, where the asymptotic optimal regret is of the form  $(1 + o(1)) \sum_{a:\Delta_a > 0} \frac{\Delta_a \ln T}{\mathsf{KL}_{\inf}(\nu_a,\mu^*)}$  with  $\mathsf{KL}_{\inf}(\nu_a,\mu^*)$  defined in the Eq (2). If we combine the binarization trick and the MED algorithm in the bounded reward setting, the size of the support set is viewed as 2, the finite-time regret bound is at best as  $O(K^{1/4}T^{3/4})$  (ignoring logarithmic factors), which is much higher than  $O(\sqrt{KT})$ .
4A close examination of [19]'s Lemma 9 (specifically, equation (20)) shows that for each suboptimal arm  $a$ , the authors bound  $\mathbb{E}\left[N_{T,a}\right]$  by a term at least  $\sum_{t=1}^{T} K(t+1)^{\left|\operatorname{supp}(\nu_1)\right|} \cdot \exp\left(-tC(\mu_1, \mu_1 - \varepsilon)\right)$ , where  $C(\mu, \mu') := \frac{(\mu - \mu')^2}{2\mu'(1 + \mu)}$  and  $\varepsilon \leq \Delta_a$ ; this is  $\Omega\left(\frac{1}{\Delta_a^{2|\operatorname{supp}(\nu_1)|}}\right)$  when  $\mu_1$  is bounded away from 0 and 1.
Algorithm 1 KL Maillard Sampling (KL-MS)
1: Input:  $K \geq 2$   
2: for  $t = 1, 2, \dots, T$  do  
3: if  $t \leq K$  then  
4: Pull the arm  $I_{t} = t$  and observe reward  $y_{t} \sim \nu_{i}$ .  
5: else  
6: For every  $a \in [K]$ , compute  
[ p_{t,a} = \frac{1}{M_t} \exp \left( -N_{t-1,a} \cdot \mathrm{kl}(\hat{\mu}_{t-1,a}, \hat{\mu}_{t-1,\max}) \right) ]  
where  $M_t = \sum_{a=1}^{K} \exp(-N_{t-1,a} \mathrm{kl}(\hat{\mu}_{t-1,a}, \hat{\mu}_{t-1,\max}))$  is the normalizer.  
7: Pull the arm  $I_{t} \sim p_t$ .  
8: Observe reward  $y_t \sim \nu_{I_t}$ .  
9: end if  
10: end for