SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \sup  _ {\theta} \mathbb {P} (R _ {\theta} ^ {\pi} (T) \geq x) \leq 6 K \exp \left(- \frac {\left(x - K - 4 \eta_ {1} K ^ {1 - \alpha} T ^ {\alpha} \sqrt {\ln T}\right) _ {+} ^ {2}}{3 2 \sigma^ {2} K T}\right) \\ + 6 K ^ {2} T \exp \left(- \frac {\sqrt {\ln T}}{8 \sigma^ {2}} \left(\frac {\eta_ {1} (x - K) _ {+}}{2 K ^ {\alpha} T ^ {1 - \alpha}} \wedge \eta_ {2} ^ {2} T ^ {\beta} \sqrt {\ln T}\right)\right). \tag {3} \\ \end{array}
$$
2. (instance-dependent regret tail)
$$
\begin{array}{l} \mathbb {P} (R _ {\theta} ^ {\pi} (T) \geq x) \leq 3 K \exp \left(- \frac {\big ((x - K) \Delta_ {0} - 4 \eta_ {2} ^ {2} T ^ {\beta} \ln T \big) _ {+}}{8 \sigma^ {2}}\right) \\ + 3 K T \sum_ {k: \Delta_ {k} > 0} \exp \left(- \frac {\left(\eta_ {1} (T / K) ^ {\alpha} \Delta_ {k} \wedge \eta_ {2} ^ {2} T ^ {\beta} \sqrt {\ln T}\right) \sqrt {\ln T}}{8 \sigma^ {2}}\right), \tag {4} \\ \end{array}
$$
The design of our bonus term is novel and hopefully provides additional insights, as follows. The first component can be interpreted as controlling the worst-case tail risk, while the second one can be regarded as controlling the instance-dependent tail risk. There exhibits a phase transition with respect to the size of confidence interval. At the beginning  $\tilde{\Theta} (T^{2\alpha -\beta})$  time periods, the second term dominates, and so the confidence interval shrinks at a rate of  $1 / \sqrt{n}$ , suggesting that we focus more on exploration within the consistency constraint. While in the remaining time periods, the first term dominates, and so the confidence interval shrinks at a rate of  $1 / n$ , suggesting that we focus more on exploitation within the optimality condition. Our policy design suggests that in real-world practice, to achieve more light-tailed risk, it might be beneficial to have two different phases in the policy design: more exploration at the beginning, and more exploitation afterwards.
The following proposition shows that our policy has  $\tilde{O}(T^{\alpha})$  worst-case expected regret and  $\tilde{O}(T^{\beta})$  instance-dependent expected regret.
Proposition 3.4. For the  $K$ -armed bandit problem,  $\pi = \mathrm{SEwRP}$  with (2) satisfies the following expected regret bounds (ignoring additive and multiplicative constant terms). If  $\beta > 0$  or  $\beta = 0$  and  $\eta_{2} = \Omega(1 \vee \sigma)$ , then
$$
\sup _ {\theta} \mathbb {E} \left[ R _ {\theta} ^ {\pi} (T) \right] = O \left(K ^ {1 - \alpha} T ^ {\alpha} \sqrt {\ln T}\right) a n d \mathbb {E} \left[ R _ {\theta} ^ {\pi} (T) \right] = O \left(T ^ {\beta} \ln T / \Delta_ {0}\right).
$$
Proof road-map. Without loss of generality, we assume arm 1 is optimal. Let the regret threshold  $x \geq K$ . For any  $k \neq 1$ , we define
$$
S _ {k} = \{\text {A r m 1 i s n o t e l i m i n a t e d b e f o r e a r m} k \}.
$$
In the following, we differentiate between two scenarios.
For the worst-case scenario, we define
$$
\mathcal {A} ^ {*} = \left\{k \neq 1: n _ {k} \leq 1 + T / K \right\}.
$$
Then with some basic algebra, one can build
$$
\mathbb {P} \left(R _ {\theta} ^ {\pi} (T) \geq x\right) \leq \sum_ {k \neq 1} \mathbb {P} \left((n _ {k} - 1) \Delta_ {k} \geq \frac {x - K}{2 K}, k \in \mathcal {A} ^ {*}\right) + \sum_ {k \neq 1} \mathbb {P} \left(\Delta_ {k} \geq \frac {x - K}{2 T}, k \notin \mathcal {A} ^ {*}\right) \tag {5}
$$
We need to emphasize that introducing  $\mathcal{A}^*$  allows us to establish  $\Delta_k = \Omega(x/T)$ , otherwise we may only have  $\Delta_k = \Omega(x/KT)$ . The goal of defining  $\mathcal{A}^*$  is to make sure our analysis aligns with our new bonus design (we have a  $1/K^\alpha$  factor in the numerator) such that our tail probabilities have optimal dependence on the number of arms  $K$ .
We decompose the tail risk of an algorithm by two types of events: (i) spending too much time before correctly discarding a sub-optimal arm; (ii) wrongly discarding the optimal arm.
1. Consider the case when  $S_{k}$  happens. This corresponds to the tail risk of spending too much time before correctly discarding a sub-optimal arm [24]. For either  $k \in \mathcal{A}^*$  or  $k \notin \mathcal{A}^*$ , we fix some non-random number  $n_0 < n_k$  (the choice of  $n_0$  is important: to be precise, we let  $n_0 = \lceil (x - K) / (2K\Delta_k) \rceil$  when  $j \in \mathcal{A}^*$  and  $n_0 = T / K$  when  $j \notin \mathcal{A}^*$ ) where  $n_0$  is a phase when a sub-optimal arm  $k$  is not eliminated by the optimal arm 1. Then arm 1 and  $k$  are both not eliminated after each of them being pulled  $n_0$  times, which means
$$
\hat {\mu} _ {t _ {1} (n _ {0}), 1} - \operatorname {r a d} (n _ {0}) \leq \hat {\mu} _ {t _ {k} (n _ {0}), k} + \operatorname {r a d} (n _ {0}).
$$
The probability of this event can be bounded using concentration properties of sub-Gaussian random variables.
2. Now consider the case when  $\bar{S}_k$  happens. This corresponds to the tail risk of wrongly discarding the optimal arm [11, 24]. Then after some phase  $n$ , the optimal arm 1 is eliminated by some arm  $k'$  while arm  $k$  is not eliminated. We emphasize that  $k = k'$  does not necessarily hold when  $K > 2$ . As a result, the following two events hold simultaneously:
$$
\hat {\mu} _ {t _ {k ^ {\prime}} (n), k ^ {\prime}} - \operatorname {r a d} (n) \geq \hat {\mu} _ {t _ {1} (n), 1} + \operatorname {r a d} (n) \quad \text {a n d} \quad \hat {\mu} _ {t _ {k} (n), k} + \operatorname {r a d} (n) \geq \hat {\mu} _ {t _ {1} (n), 1} + \operatorname {r a d} (n).
$$
The two events leads to, respectively,
$$
\text {m e a n o f} n \text {n o i s e t e r m s} \geq \operatorname {r a d} (n) = \tilde {\Theta} \left(\frac {(T / K) ^ {\alpha}}{n} \wedge \sqrt {\frac {T ^ {\beta}}{n}}\right)
$$
$$
\text {m e a n o f} n \text {n o i s e t e r m s} \geq \Delta_ {k} = \Omega \left(\frac {x}{T}\right).
$$
Then the tail probability can be further bounded by
$$
O \left(\exp \left(- \left(\frac {(T / K) ^ {2 \alpha}}{n} \vee T ^ {\beta}\right) \wedge \frac {n x ^ {2}}{T ^ {2}}\right)\right) = O \left(\exp \left(- \frac {x}{K ^ {\alpha} T ^ {1 - \alpha}} \wedge T ^ {\beta}\right)\right).
$$
which yields the second term in (3).
For the instance-dependent scenario, the proof follows similarly to that for the worst-case scenario, but with some amendment. For each  $k \neq 1$ , we treat  $\Delta_k$  as a constant, and instead of (5), we use
$$
\begin{array}{l} \mathbb {P} \left(R _ {\theta} ^ {\pi} (T) \geq x\right) \\ \leq \sum_ {k: \Delta_ {k} > 0} \mathbb {P} \left((n _ {k} - 1) \Delta_ {k} \geq \frac {(x - K) / \Delta_ {k}}{\sum_ {k ^ {\prime}: \Delta_ {k ^ {\prime}} > 0} 1 / \Delta_ {k ^ {\prime}}}, S _ {k}\right) \\ + \sum_ {k: \Delta_ {k} > 0} \mathbb {P} \left((n _ {k} - 1) \Delta_ {k} \geq \frac {(x - K) / \Delta_ {k}}{\sum_ {k ^ {\prime}: \Delta_ {k ^ {\prime}} > 0} 1 / \Delta_ {k ^ {\prime}}}, \bar {S} _ {k}\right). \\ \end{array}
$$
Here, we do not need  $\mathcal{A}^*$  to control the dependence on  $K$ . Also, when bounding the probability when  $S_{k}$  happens, we accordingly define
$$
n _ {0} = \left\lceil (x - K) / \Delta_ {k} ^ {2} \cdot \Delta_ {0} \right\rceil \leq n _ {k} - 1.
$$
The detailed proof is left to the supplementary material. We would like to emphasize the technical novelty compared to that in [24]. In general, since  $\alpha$  and  $\beta$  become flexible constants, the proof requires more delicate controls on probability bounds. When proving the worst-case upper bound, we need a careful manipulation on  $\mathrm{rad}(n)$  since we are dealing with the minimum of two different types of bonus terms. When proving the instance-dependent upper bound, we require a careful division of the tail event to make the bound as tight as possible, depending on specific instances  $(\theta)$ . In [24], the tail bound is only concerned with the worst-case scenario with  $\alpha = 1/2$ , and hence the aforementioned challenges do not exist.
Finally, we would like to note that although applying the naive explore-then-commit (ETC) strategy might be able to circumvent heavy-tailed risk, the approach has two main issues:
1. From the regret expectation perspective, the ETC policy can only achieve worst-case  $O(T^{2/3})$  expected regret bound. When  $\alpha \in [1/2, 2/3)$ , we still have to consider other types of policies. Further, it seems unlikely that without knowing the arm gaps  $\{\Delta_k\}$ , the explore-then-commit policy can achieve  $\tilde{O}(T^\alpha)$  worst-case regret and  $\tilde{O}(T^\beta)$  instance-dependent regret simultaneously with  $\beta < \alpha$ .
2. From the regret tail risk perspective, the tail probability of incurring a large regret may not be optimal in the worst-case. Consider a simple 2-armed bandit case with arm gap  $\Delta$ . An ETC policy with  $m = \Theta(T^{\alpha})$  ( $\alpha \in [2/3,1)$ ) steps of exploration has a probability of  $\Omega(\exp(-m\Delta^{2}))$  committing to a wrong arm. In the worst-case scenario, let  $\Delta = T^{(\alpha - 1)/2}$  and the regret threshold be  $x = T^{(\alpha + 1)/2} / 4 \in (m\Delta, (T - m)\Delta)$ . Then if we incur a regret of  $x$ , it means we commit to a wrong arm after the exploration, which suggests  $\mathbb{P}(R_{\theta}^{\pi}(T) > x) = \Omega(\exp(-m\Delta^{2})) = \Omega(\exp(-T^{2\alpha - 1})) = \omega(\exp(-x / T^{1 - \alpha}))$ .
4 Extension with Structured Non-stationarity
In this section, we extend our results to a more general stochastic MAB model. In certain applications, the assumption that the rewards for one arm are i.i.d. generated may fail to hold. On the other hand, the non-stationary MAB models are at the other extreme being a bit too conservative, where no structural properties exist. The stochastic MAB model with non-stationary baseline rewards tries to strike a balance between stationary stochastic MAB and fully adversarial MAB. Such a structure is receiving attention in the literature (see, e.g., [13, 16, 15, 23]). We restate the setting as follows and highlights those different from the standard stochastic MAB model introduced in Section 2.
At each time  $t$ , upon pulling an arm  $a_{t}$ , a reward is independently sampled as  $r_{t,a_t} = b_t + \theta_{a_t} + \epsilon_{t,a_t}$  where  $b_{t}$  is the baseline reward chosen by the nature at time  $t$ ,  $\theta_{a_t}$  is the mean reward of arm  $a_{t}$ , and  $\epsilon_{t,a_t}$  is a zero-mean random noise, independent across time periods. The random noise  $\epsilon_{t,a_t}$  is assumed to be  $\sigma$ -subGaussian. Let  $\theta = (\theta_1,\dots ,\theta_K)$  be the vector of means and  $\theta_{*} = \max \{\theta_{1},\dots ,\theta_{K}\}$  be the optimal mean reward among the  $K$  arms. Note that the DM does not know  $b_{t}$  or  $\theta$  a priori, except that  $b_{t}\in [0,B]$  (for some  $B\geq 0$ ) and  $\theta \in [0,1]^K$ . The empirical regret of the policy  $\pi = (\pi_1,\dots ,\pi_T)$  under the mean vector  $\theta$  and the noise parameter  $\sigma$  over a time horizon of  $T$  is defined as
$$
\hat {R} _ {\theta} ^ {\pi} (T) = \theta_ {*} \cdot T + \sum_ {t = 1} ^ {T} b _ {t} - \sum_ {t = 1} ^ {T} \left(b _ {t} + \theta_ {a _ {t}} + \epsilon_ {t, a _ {t}}\right) = \theta_ {*} \cdot T - \sum_ {t = 1} ^ {T} \left(\theta_ {a _ {t}} + \epsilon_ {t, a _ {t}}\right).
$$
Note that here although  $b_{t}$  appears in the model, the term gets cancelled in the regret. Let  $\Delta_k = \theta_* - \theta_k$  denote the gap between the optimal arm and the  $k$ th arm. Same as Section 2.1,  $\Delta_0$ , the pseudo regret and the aggregated random noise are defined accordingly.
Theorem 4.1. For the  $K$ -armed bandit problem with baseline rewards,  $\pi = \mathrm{SEwRP}$  with (2) satisfies the following properties: fix any  $\eta_1, \eta_2 \geq 0$ , for any  $x > 0$ , we have
$$
\begin{array}{l} \sup  _ {\theta} \mathbb {P} (R _ {\theta} ^ {\pi} (T) \geq x) \leq 6 K \exp \left(- \frac {\left(x - K - 4 \eta_ {1} K ^ {1 - \alpha} T ^ {\alpha} \sqrt {\ln T}\right) _ {+} ^ {2}}{8 (B + 2 \sigma) ^ {2} K T}\right) \\ + 6 K ^ {2} T \exp \left(- \frac {\sqrt {\ln T}}{2 (B + 2 \sigma) ^ {2}} \left(\frac {\eta_ {1} (x - K) _ {+}}{2 K ^ {\alpha} T ^ {1 - \alpha}} \wedge \eta_ {2} ^ {2} T ^ {\beta} \sqrt {\ln T}\right)\right). \tag {6} \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {P} (R _ {\theta} ^ {\pi} (T) \geq x) \leq 3 K \exp \left(- \frac {\big ((x - K) \Delta_ {0} - 4 \eta_ {2} ^ {2} T ^ {\beta} \ln T \big) _ {+}}{2 (B + 2 \sigma) ^ {2}}\right) \\ + 3 K T \sum_ {k: \Delta_ {k} > 0} \exp \left(- \frac {\left(\eta_ {1} (T / K) ^ {\alpha} \Delta_ {k} \wedge \eta_ {2} ^ {2} T ^ {\beta} \sqrt {\ln T}\right) \sqrt {\ln T}}{2 (B + 2 \sigma) ^ {2}}\right), \tag {7} \\ \end{array}
$$
Proposition 4.2. For the  $K$ -armed bandit problem with baseline rewards,  $\pi = \mathrm{SEwRP}$  with (2) satisfies the following expected regret bounds (ignoring additive and multiplicative constant terms). If  $\beta > 0$  or  $\beta = 0$  and  $\eta_{2} = \Omega(1 \vee \sigma)$ , then
The proof of Theorem 4.1 follows that of Theorem 3.3, except that when bounding the mean of error terms, we now have not only the true noise  $\epsilon$ , but also a baseline  $b_{t}$ . Lemma 4.3 is established to characterize the concentration of these combined error terms. We would like to note that the randomization step in SEwRP is crucial to hedge against the baseline rewards  $b_{t}$ , for which deterministic algorithms may fail. With uniform permutation, although the estimation of one arm is still biased, the difference of estimations between two different arms becomes unbiased.
Lemma 4.3. Let  $\tau_{i}$  be the (random) time period that we pull the last arm in  $\mathcal{A}_i$ . For any  $n \geq 1$  and  $k, k' \in [K]$ , define
$$
E _ {n} (k, k ^ {\prime}) = \frac {1}{n} \sum_ {t = 1} ^ {\tau_ {n} \wedge T} (b _ {t} + \epsilon_ {t, a _ {t}}) (\mathbb {1} \{a _ {t} = k \} - \mathbb {1} \{a _ {t} = k ^ {\prime} \}).
$$
We have, for any  $x > 0$
$$
\begin{array}{l} \max  \left\{\mathbb {P} \left(E _ {n} \left(k, k ^ {\prime}\right) \geq x; \tau_ {n} \leq T; k, k ^ {\prime} \in \mathcal {A} _ {n}\right), \right. \\ \left. \mathbb {P} \left(E _ {n} \left(k, k ^ {\prime}\right) \leq - x; \tau_ {n} \leq T; k, k ^ {\prime} \in \mathcal {A} _ {n}\right) \right\} \\ \leq 3 \exp \left(- \frac {n x ^ {2}}{2 (B + 2 \sigma) ^ {2}}\right). \\ \end{array}
$$
In this work, we study the stochastic MAB problem with the goal of obtaining light-tailed regret distribution under given degree of worst-case optimality and instant-dependent consistency. We fully characterize the tradeoff among optimality, consistency, and tail risk via (i) proving lower bounds for decaying rate of incurring a large regret and (ii) proposing a novel algorithm with explicit regret tail upper bounds that match the lower bounds. We extend our analysis to the stochastic MAB model with non-stationary baseline rewards, and obtain similarly tight regret tail bounds. We study the empirical performance via synthetic simulations, and present insights of our results.
An open question for this work is whether our results can be generalized to the any-time case where the policy has no knowledge of  $T$ . For the lower bound, it has been pointed out in [21] that not knowing  $T$  may have a significant impact on the concentration of regret distribution for standard UCB policies. For the upper bound, as is pointed out in [24], an SE type policy will fail to obtain light-tailed regret distribution under the any-time design even for the basic MAB model. Meanwhile, it seems nontrivial to incorporate randomization into confidence bound based policies to handle the stochastic MAB model with non-stationary baseline rewards. We leave the question for future work.
[1] K. Ashutosh, J. Nair, A. Kagrecha, and K. Jagannathan. Bandit algorithms: Letting go of logarithmic regret for statistical robustness. In International Conference on Artificial Intelligence and Statistics, pages 622–630. PMLR, 2021.
[2] J.-Y. Audibert, R. Munos, and C. Szepesvári. Exploration-exploitation tradeoff using variance estimates in multi-armed bandits. Theoretical Computer Science, 410(19):1876-1902, 2009.
[3] D. Baudry, R. Gautron, E. Kaufmann, and O. Maillard. Optimal thompson sampling strategies for support-aware cvar bandits. In International Conference on Machine Learning, pages 716-726. PMLR, 2021.
[4] S. Bubeck and N. Cesa-Bianchi. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. arXiv preprint arXiv:1204.5721, 2012.
[5] A. Cassel, S. Mannor, and A. Zeevi. A general approach to multi-armed bandits under risk criteria. In Conference On Learning Theory, pages 1295–1306. PMLR, 2018.
[6] J. Q. Chang and V. Y. Tan. A unifying theory of thompson sampling for continuous risk-averse bandits. In Proc. of the 36th AAAI Conference on Artificial Intelligence. AAAI Press, 2022.
[7] N. Chen and S. Yang. Bridging adversarial and nonstationary multi-armed bandit. arXiv preprint arXiv:2201.01628, 2022.
[29] J. Yao, E. Brunskill, W. Pan, S. Murphy, and F. Doshi-Velez. Power constrained bandits. In Machine Learning for Healthcare Conference, pages 209-259. PMLR, 2021.
[30] Q. Zhu and V. Tan. Thompson sampling algorithms for mean-variance bandits. In International Conference on Machine Learning, pages 11599-11608. PMLR, 2020.
[31] A. Zimin, R. Ibsen-Jensen, and K. Chatterjee. Generalized risk-aversion in stochastic multiarmed bandits. arXiv preprint arXiv:1405.0833, 2014.