SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Bounding Regret from Committing to the Wrong Action: As for the regret accumulated from commitment, the key is to prove the  $\mathcal{O}(T^{-1})$  confidence level upper bound presented in Corollary 1. We follow a procedure similar to the proof of Theorem 1 in [2] to utilize the adaptivity of UCB exploration and the pessimistic LCB commitment. We first show that with high probability, the number of pulls  $N_{t}(a)$  for any sub-optimal actions in the exploration phase is upper-bounded. This is because after a certain number of pulls, the uncertainty bonus  $b_{t-1}(a)$  for any sub-optimal action will be so small that the upper confidence bound  $\mathsf{UCB}_{t-1}(a)$  cannot be larger than  $\mu_{a*}$ , thus less than the upper confidence bound of the optimal action. Therefore, sub-optimal actions will not be chosen in future rounds. After our carefully designed  $T_{\mathrm{c}}$ , we make sure that the optimal action has the largest number of pulls  $N_{T_{\mathrm{c}}}(a)$  among all actions, thus its bonus is so small so that its lower confidence bound  $\mathsf{LCB}_{T_{\mathrm{c}}}(a^{*})$  is larger than the expectations  $\mu_{a}$  of any sub-optimal action  $a$ , and thus larger than the lower confidence bound of other actions. So with high probability, we will commit to the optimal action. Then, the  $\mathcal{O}(T^{-1})$  confidence level will provide us with a constant regret in commitment, and the overall dominating regret comes from exploration.
Combining both bounds, we are able to show the regret performance upper bound in Theorem. 1.
4.2 Regret Optimality for Gaussian Bandits with Adaptive Stopping Time
We present the regret performance of EOCP-UG in Theorem 2. Compared to Theorem 1, EOCP-UG has exactly the same regret performance guarantees as EOCP even without the knowledge of a lower bound  $\Delta_{\mathrm{lb}}$  on the minimum reward gap. This implies that EOCP-UG adapts to the reward gap.
Theorem 2 Let  $l = \log (T) + \mathcal{O}(\sqrt{\log T})$  and when  $T$  is large enough, the expected regret of the EOCP-UG algorithm in Algorithm. 2 with adaptive stopping time is upper-bounded by:
$$
\operatorname {R e g} _ {\mu} ^ {\text {E O C P - U G}} (T) \leqslant \sum_ {a: \Delta_ {a} > 0} \frac {2 \log T}{\Delta_ {a}} + \mathcal {O} \left(\frac {\sqrt {\log T}}{\Delta_ {\min }}\right).
$$
We can also directly derive an asymptotic regret bound which shows that EOCP-UG is regret optimal in Gaussian bandits:
$$
\limsup_{T\to \infty}\frac{\operatorname{Reg}_{\boldsymbol{\mu}}^{\text{EOCP - UG}}(T)}{\log T}\leqslant \sum_{a:\Delta_{a} > 0}\frac{2}{\Delta_{a}}.
$$
With adaptive stopping, the sample complexity until the commitment is not a pre-determined value. However, we can still extract similar guarantees along with the confidence level from the proof of Theorem 2. We present these results in the following corollary:
Corollary 2 Let  $l = \log (T) + \mathcal{O}(\sqrt{\log T})$ , the sample complexity until commitment for EOCP-UG algorithm in Algorithm. 2 is upper-bounded by:
$$
\mathsf {S C C} _ {\boldsymbol {\mu}} ^ {\mathsf {E O C P - U G}} (T) \leqslant \sum_ {a: \Delta_ {a} > 0} \frac {8 \log^ {2} T}{\Delta_ {a} ^ {2}} + \mathcal {O} \left(\frac {\log^ {\frac {3}{2}} T}{\Delta_ {\min} ^ {2}}\right),
$$
and the confidence level is upper bounded by  $\mathcal{O}(T^{-1})$ .
Proof Roadmap: The complete proofs of Theorem 2 and Corollary 2 are provided in the supplementary material. Compared to the proof of Theorem 1, the major difference lies in bounding the regret of commitment. Similarly, we are required to bound the probability of committing to a sub-optimal action. We first show that when the agent stops exploration according to Line 2 of Algorithm 2, the action that has the maximum number of pulls is the optimal action with high probability. Then, we show that under this event we will commit to the optimal action if we use LCB commitment.
Magic Choice of Exploration Function: Bounding the regret in both exploration and commitment requires a delicate analysis with any time concentration inequalities. Our choice of exploration function  $l$  plays an important role which manages the trade-off between low-regret exploration and high-probability optimal action commitment. If the exploration function is too large, i.e., if  $l = 2\log T$ , the regret in exploration will not be optimal. If the exploration function is too small, i.e., if  $l = \log T$ , the probability of committing to the wrong action can not be bounded by  $\mathcal{O}(T^{-1})$ . Our magic choice of exploration function  $l$  achieves the best of both worlds.
Loss of SCC from Unknown Gap: Even though we have the same regret performance, the guarantee for commitment time is  $\mathcal{O}(\log^2 T)$  which is worse than  $\mathcal{O}(\log T)$  in the pre-determined setting. So, is this order fundamental with adaptive stopping time? In the next section, we provide the answer by investigating the theoretical limits of commitment time for asymptotic regret optimal algorithms.
4.3 Fundamental Limits of Sample Complexity until Commitment
In order to answer the question regarding the fundamental sample complexity until commitment for regret optimal algorithms with both pre-determined and adaptive stopping times, we consider a simplified Gaussian bandit model where there are only 2 arms. The reward gap between the two actions is  $\Delta$ , so  $\Delta_{\min} = \Delta$ . From Eq. (1), it is clear that the optimal regret of any algorithm is asymptotically  $2\Delta^{-1}\log(T)$ , so the set of all regret optimal algorithms is characterized by:
$$
\Pi_ {\mathrm {R O}} = \left\{\mathsf {A l g} \Bigg | \lim  _ {T \to \infty} \frac {\mathsf {R e g} _ {\pmb {\mu}} ^ {\mathsf {A l g}} (T)}{\log T} \leqslant \frac {2}{\Delta} \mathrm {a n d} \mathbb {P} _ {\pmb {\mu}} (\hat {a} \neq a ^ {*}) = \mathcal {O} \left(T ^ {- 1}\right) \right\}.
$$
Furthermore, for each regret optimal algorithm, we say it has  $c$ -logarithm regret violation if there exists a constant  $c \in (0,1)$  (choose the minimum  $c$  if there exists multiple) such that the following inequality is satisfied when  $T$  is large enough:
$$
\left| \mathrm {R e g} _ {\mu} ^ {\mathrm {A l g}} (T) - \frac {2 \log (T)}{\Delta} \right| = \mathcal {O} (\log^ {c} T).
$$
If an algorithm has  $c$ -logarithm regret violation, the largest additional lower order term in its finite-time regret bound should be  $\mathcal{O}(\log^c T)$ . On the other hand, it also characterizes the convergence rate of the regret to its asymptote. For example, the vanilla UCB algorithm has at most 1/2-logarithm regret violation, and our EOCP and EOCP-UG algorithms both have at most 1/2-logarithm regret violation [22, Theorem. 8]. We then provide the following theorem to characterize the fundamental commitment time limits for algorithms in  $\Pi_{\mathrm{RO}}$ :
Theorem 3 (Information-Theoretic Limits of SCC) Consider a 2-armed Gaussian bandits, for any asymptotically regret optimal algorithm  $\mathsf{Alg}$  which has  $c$ -logarithm regret violation, in order to guarantee  $\mathcal{O}(T^{-1})$  confidence level, the sample complexity until commitment with pre-determined stopping time is lower bounded by
$$
T _ {\mathrm {c}} = \Omega (\Delta^ {- 2} \log (T)),
$$
and with adaptive stopping time, the sample complexity until commitment is lower-bounded by:
$$
\mathbb {E} _ {\boldsymbol {\mu}} \left[ T _ {\mathrm {c}} \right] = \Omega \left(\Delta^ {- 2} \log^ {2 - c} (T)\right).
$$
Proof Roadmap: The proof is provided in the supplementary material. The proof idea relies on the well-known "transportation" lemma [32, Lemma. 1] originally derived to prove the theoretical limits of best arm identification algorithms. This lemma characterizes the expected number of pulls for each action by hypothesis testing between the original bandit problem and another bandit instance with a different optimal action. Then by finding a proper bandit instance and combining the lemma with regret optimal algorithms, we will be able to prove Theorem. 3 for both settings.
Sample Optimality: It is shown by Corollary 1 and Theorem 3 together that our proposed EOCP algorithm achieves the optimal commitment time with  $\mathcal{O}(\log T)$  with pre-determined stopping time if the reward gap  $\Delta$  is known a priori. However, with adaptive stopping time, our EOCP-UG algorithm has  $1/2$ -logarithm regret violation and  $\mathcal{O}(\log^2 T)$  commitment time which is larger than the  $\mathcal{O}(\log^{1.5}T)$  lower bound indicated by Theorem 3. Even though EOCP-UG is not shown to be exactly sample optimal, we conjure that the performance gap comes from our analysis techniques which makes one of the bounds (maybe both) not tight. Whether EOCP-UG is indeed sample optimal, and if not, how to design regret optimal algorithms with  $\mathcal{O}(\log^{1.5}T)$  commitment time remains open.
5 Regret Optimality for General Bandits
Even though EOCP is applicable in sub-Gaussian bandits, it is not regret optimal any more beyond Gaussian bandits. This can be seen by comparing Theorem 1 and the fundamental regret limit (1) with Pinsker's inequality, and it is clear that the lower bound is smaller than our upper bound asymptotically beyond Gaussian bandits. To close this gap, we propose an improved algorithm called KL-EOCP which is provably regret optimal in general bandits.
Natural Exponential Family: We assume the reward distributions of each action belong to a natural exponential family, i.e.,  $\mathcal{P} = \{(\nu_{\theta})_{\theta \in \Theta}:d\nu_{\theta} / d\xi = \exp (\theta x - b(\theta))h(x)\}$ , where  $\Theta \subset \mathbb{R}$  is the set of all parameters  $\theta$  such that the expectation  $\mu$  is positive and bounded, i.e.,  $\mu \in [0,1]$ .  $\xi$  is some reference measure on  $\mathbb{R}$  and  $b:\Theta \to \mathbb{R}$  is a convex twice differentiable function. This distribution  $\nu_{\theta}$  can also be parameterized by its expectation  $\mu = b^{\prime}(\theta)$ , the derivative of  $b(\cdot)$ , and for every  $\mu$  we denote by  $\nu^{\mu}$  the unique distribution in  $\mathcal{P}$  with expectation  $\mu$  and by  $\theta^{\mu}$  its corresponding parameter. Gaussian distribution with unit variance is an example of this family. Moreover, the Kullback-Leibler divergence from  $\nu_{\theta_1}$  to  $\nu_{\theta_2}$  (with a little abuse of notation) can be expressed as [12]:
$$
\mathsf {K L} (\mu_ {1}, \mu_ {2}) = \mathsf {K L} (\nu_ {\theta_ {1}}, \nu_ {\theta_ {2}}) = b (\theta_ {2}) - b (\theta_ {1}) - b ^ {\prime} (\theta_ {1}) (\theta_ {2} - \theta_ {1}).
$$
The set of exponential family bandit models  $\pmb{\nu} = (\nu_{\theta_1}, \dots, \nu_{\theta_A})$  can be characterized by the expectations of the actions  $\pmb{\mu} = (\mu_1, \dots, \mu_A)$ . We assume for all  $\lambda \in \mathbb{R}$ , and  $\theta \in \Theta$  the moment generating function  $M_{\nu_\theta}(\lambda) = \mathbb{E}_{\nu_\theta}[\exp(\lambda W)]$  for the distribution  $\nu_\theta$  is well-defined and is finite.
Algorithm: Analog to  $\Delta_{\mathrm{lb}}$  in Algorithm. 1, the KL-EOCP algorithm requires the knowledge of a strictly positive lower bound  $\mathsf{KL}_{\mathrm{lb}}$  on the "minimum KL divergence", denoted as  $\mathsf{KL}_{\min}$ , which captures the minimum reward distribution gap (distance) between the optimal action and any suboptimal action. Considering the asymmetry of the KL divergence, we define  $\mathsf{KL}_{\min}$  as follows:
$$
\mathsf {K L} _ {\min } = \min  _ {a \neq a ^ {*}} \min  \left\{\mathsf {K L} (\mu_ {a}, \mu_ {a ^ {*}}), 4 \mathsf {K L} (\mu_ {a} ^ {\prime}, \mu_ {a}) \right\},
$$
where  $\mu_{a}^{\prime}\in (\mu_{a},\mu_{a*})$  such that  $4\mathrm{KL}(\mu_a^\prime ,\mu_a*) = \mathrm{KL}(\mu_a,\mu_a*)$ . The term  $4\mathrm{KL}(\mu_a^\prime ,\mu_a)$  reflects the skew of KL divergence when the two distributions are switched. A simple lower bound to  $\mathrm{KL}_{\mathrm{min}}$  can be easily computed given a lower bound  $\Delta_{\mathrm{lb}}$  on the minimum reward gap  $\Delta_{\mathrm{min}}$  with the expression of the exponential family, while our KL-EOCP algorithm can operate with any lower bound  $\mathrm{KL}_{\mathrm{lb}}$ .
The KL-EOCP algorithm is summarized in Algorithm 3. It designs the UCB and LCB bonuses based on the KL divergence of the reward distributions for all actions. In general, these designs would lead to smaller confidence intervals with the same  $\mathcal{O}(T^{-1})$  concentration guarantees as first adopted in the KL-UCB algorithm proposed in [12]. If the lower bound information  $\mathsf{KL}_{\mathrm{lb}}$  is not known a priori, we can combine Algorithm 3 with the adaptive stopping time of EOCP-UG to deal with this setting.
Algorithm 3 KL-EOCP with Pre-Determined Stopping Time
Require: Exploration function  $l$ ; Lower bound  $\mathrm{KL}_{\mathrm{lb}}$  on the minimum KL divergence.
1: Let  $T_{\mathrm{c}} = \frac{4Al}{KL_{\mathrm{lb}}^{2}} + A$  be the length of the exploration phase.
2: Initialize by pulling each arm  $a$  once.
3: for  $t = A + 1 : T_{\mathrm{c}}$  do
4: Set upper confidence bound:
$$
\mathsf{UCB}_{t - 1}(a) = \operatorname *{arg  max}_{\mu \geqslant \bar{r}_{t - 1}(a)}\left\{N_{t - 1}(a)\mathsf{KL}(\bar{r}_{t - 1}(a),\mu)\leqslant l\right\} .
$$
5: Take action  $A_{t} = \arg \max_{a}\mathsf{UCB}_{t - 1}(a)$ . // UCB Exploration
7: Set lower confidence bound:
$$
\mathsf{LCB}_{T_{\mathrm{c}}}(a) = \operatorname *{arg  min}_{\mu \leqslant \bar{r}_{T_{\mathrm{c}}}(a)}\left\{N_{T_{\mathrm{c}}} (a)\mathsf{KL}(\bar{r}_{T_{\mathrm{c}}} (a),\mu)\leqslant l\right\} .
$$
8: For  $t \in [T_{\mathrm{c}} + 1, T]$ , commit to action  $\hat{a} = \arg \max_{a} \mathsf{LCB}_{T_{\mathrm{c}}}(\hat{a})$ . // LCB Commitment
Regret Optimality in General Bandits: The theoretical regret performance of the KL-EOCP is summarized in the following Theorem. To the best of our knowledge, it is the first result achieving asymptotic regret optimality in general bandit problems with commitment.
Theorem 4 Let  $l = \log (T) + \mathcal{O}(\sqrt{\log T})$ , when  $T$  is large enough and the reward distributions  $\nu$  of each action belong to the same natural exponential family, the expected regret of KL-EOCP in Algorithm 3 is upper-bounded by:
$$
\operatorname {R e g} _ {\boldsymbol {\mu}} (T) \leqslant \sum_ {a: \Delta_ {a} > 0} \frac {\Delta_ {a} \log T}{\operatorname {K L} (\mu_ {a} , \mu_ {1})} + \mathcal {O} \left(\frac {\log^ {\frac {3}{4}} T}{\operatorname {K L} _ {\min }}\right).
$$
Therefore, it is clear that we can derive an asymptotic upper bound as follows:
$$
\limsup_{T\to \infty}\frac{\operatorname{Reg}_{\boldsymbol{\mu}}(T)}{\log T}\leqslant \sum_{a:\Delta_{a} > 0}\frac{\Delta_{a}}{\mathsf{KL}(\mu_{a},\mu_{1})},
$$
which exactly matches the regret limit in Eq. (1). The complete proof of Theorem. 4 is provided in the supplementary materials. Even though the proof roadmap is similar to the proof of Theorem. 1, the major difference comes from the use of a tighter concentration lemma modified from [12, Theorem. 11] which captures the low probability event when the KL divergence of the empirical mean is far away from its expectation. The sample complexity until commitment and confidence level guarantees can also be extracted from the proof. Specifically,  $\mathsf{SCC}_{\mu}^{\mathsf{KL - EOCP}}(T) = \mathcal{O}(\mathsf{KL}_{\mathrm{lb}}^{-1}\log T)$  and the confidence level is upper bounded by  $\mathcal{O}(T^{-1})$
In this section, we study the empirical performance of our proposed EOCP algorithm with variants compared to existing algorithms in the literature, including BAI-ETC [22], UCB [10], KL-UCB [12], and DETC [23] in both Gaussian and Bernoulli bandit settings. In this section, we only test the algorithms on a two-armed bandit problem because some baselines are only applicable in two-armed settings. The performance of our algorithms in bandit models with multiple arms is demonstrated in Appendix. E. In the Gaussian setting, we test all the algorithms with distributions  $\mathcal{N}(\mu_i, 1)$  for arm
(a) Gaussian Bandits
(b) Bernoulli Bandits
Figure 1: Comparison of regret performance of EOCP with variants and existing algorithms in the literature. The gap  $\Delta$  between the two arms is 0.5 and results are averaged over  $10^{5}$  iterations.
$i = 1,2$  with a total of  $10^{6}$  rounds, and in the Bernoulli bandit setting, we test the algorithms with distribution  $\mathsf{Ber}(\mu_{\mathrm{i}})$  for arm  $i = 1,2$  with a total of  $10^{5}$  rounds. We set  $\mu_{1} = 0.7$  and  $\mu_{2} = 0.2$ , so the gap between the arms is  $\Delta = 0.5$ . The results are averaged over  $10^{5}$  iterations and in Fig. 1.
In the Gaussian bandit setting, it shows that both BAI-ETC and DETC algorithms exhibit unsatisfactorily high regret. On the contrary, our proposed algorithms EOCP and EOCP-UG have lower final regret even compared to the popular UCB algorithm, surprisingly. Even though our algorithms use a larger exploration function than the vanilla UCB algorithm, and accumulate regret more quickly in exploration, i.e., approximately the first 1000 rounds, it allows us to commit to the optimal action. Our algorithms almost find the optimal action in all simulated traces which gives rise to the very slowly-increasing behavior of regret in the commitment phase. This phenomenon coincides with our theoretical analysis which shows that the regret in commitment is  $\mathcal{O}(1)$ . However, the UCB algorithm continues to explore sub-optimal actions, so its regret continues to grow when the EOCP algorithm has already committed to the optimal action. Numerically, the EOCP algorithm reduces  $20\%$  of the final regret compared to UCB through early commitment, and we conjure that preventing the "over-exploration" phenomenon of the UCB algorithm is behind the reason for such empirical regret reduction. Comparing the commitment time of EOCP-UG and EOCP, we can see that both algorithms stop exploration at approximately 1000 rounds. This means that not knowing the gap information won't harm the empirical sample complexity until commitment too much, which in turn may imply that our theoretical analysis of commitment time upper bound in Corollary. 2 is not tight enough. The same trend can be witnessed in the results of the Bernoulli bandits. However, in Bernoulli bandits, KL-UCB and KL-EOCP algorithms have a much better regret performance than other algorithms, which shows that knowledge of the reward distribution family improves the performance significantly.
We studied ROBAI which intends to both minimize regret and commit to the optimal action. We proposed EOCP with variants, which combine UCB exploration and LCB commitment with novel stopping criteria in both pre-determined and adaptive settings. We showed that both EOCP and EOCP-UG are regret asymptotic optimal in Gaussian bandits with  $\mathcal{O}(\log T)$  and  $\mathcal{O}(\log^2 T)$  commitment time respectively, almost matching the theoretical limits we derived. For general bandits, we proposed KL-EOCP which is provably regret optimal. Numerical experiments confirmed the superiority of our algorithms and revealed the "over-exploration" phenomenon of UCB algorithms.
The work of Qining Zhang and Lei Ying is supported in part by NSF under grants 2112471, 2134081, 2207548, and 222897.
[37] Rayadurgam Srikant and Lei Ying. Communication networks: an optimization, control, and stochastic networks perspective. Cambridge University Press, 2013.
[38] Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
[39] Guanyu Nie, Mridul Agarwal, Abhishek Kumar Umrawal, Vaneet Aggarwal, and Christopher John Quinn. An explore-then-commit algorithm for submodular maximization under full-bandit feedback. In Uncertainty in Artificial Intelligence, pages 1541–1551. PMLR, 2022.
[40] Ali Yekkekhhany, Ebrahim Arian, Mohammad Hajiesmaili, and Rakesh Nagi. Risk-averse explore-then-commit algorithms for finite-time bandits. In 2019 IEEE 58th Conference on Decision and Control (CDC), pages 8441–8446. IEEE, 2019.
[41] Ali Yekkekhany, Ebrahim Arian, Rakesh Nagi, and Ilan Shomorony. A cost-based analysis for risk-averse explore-then-commit finite-time bandits. IISE Transactions, 53(10):1094-1108, 2021.
[42] Eyal Even-Dar, Shie Mannor, Yishay Mansour, and Sridhar Mahadevan. Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. Journal of machine learning research, 7(6), 2006.
In this section, we provide a more detailed review of related works. We first review two classic problem formulations of the multi-armed bandit model: the regret minimization problem and the best arm identification problem. Then we review previous works on explore-then-commit algorithms which takes commitment into account. Finally, we review the offline stochastic bandit literature which motivated our choice of pessimistic principle in action identification.
A.1 Regret Minimization
The theoretical limits of regret minimization have been revealed by [13, 14], which shows that the expected regret of any algorithm is lower bounded when horizon  $T$  approaches infinity:
$$
\liminf_{T\to \infty}\frac{\operatorname{Reg}_{\boldsymbol{\mu}}^{\mathrm{Alg}}(T)}{\log T}\geqslant \sum_{a:\Delta_{a} > 0}\frac{\Delta_{a}}{\operatorname{KL}(\nu_{a},\nu_{a*})},
$$
where  $\mathrm{KL}(\cdot ,\cdot)$  denotes the Kullback-Leibler divergence between two distributions. Based on the asymptotic lower bound, we say an algorithm is asymptotically regret optimal if its regret performance achieves the regret lower bound asymptotically. Two sets of algorithms prevail in the regret minimization literature. One is the family of UCB algorithms [10, 12, 33], which reflects the principal of optimism in action selection to encourage exploration. To be specific, the UCB algorithms will select the action which has the largest upper confidence bound of reward estimation. This upper confidence bound represents the highest possible expected reward given the samples collected from previous rounds under a high probability event. Usually the additional bonus from the empirical mean to the upper confidence bound for a specific action decreases as the number of pulls increases. Therefore, actions which have not been tried frequently in previous rounds will have larger bonus. This trait encourages exploration. By designing the bonus carefully, one can find the optimal trade-off between exploration and exploitation. The other set of prevailing algorithms is the family of Thompson Sampling Algorithms [1, 34, 11, 15, 35]. The TS algorithm assumes each action is associated with a posterior distribution given the reward feedback from previous rounds. Then the agent will collect one virtual sample from each posterior distribution and choose the action which has the largest virtual sample. As the number of pulls grows, the posterior distribution will be more and more concentrated around the expectation so the virtual sample will also be closer to its expectation. On the other hand, the remaining randomness encourages exploration of other under-explored actions. However, both algorithms do not commit to a single action because their action selection policies have to be re-evaluated at each round based on new observations.
A.2 Best Arm Identification