SlowGuess/NeurIPS_2023_OCR huggingface dataset:
<sup>1</sup>Ziyi Huang, <sup>1</sup>Henry Lam, <sup>2</sup>Amirhossein Meisami, <sup>1</sup>Haofeng Zhang *
<sup>1</sup> Columbia University, New York, NY, USA
2 Adobe Inc., San Jose, CA, USA
zh2354, khl2114, hz2553@columbia.edu, meisami@adobe.com
Bayesian bandit algorithms with approximate Bayesian inference have been widely used in real-world applications. However, there is a large discrepancy between the superior practical performance of these approaches and their theoretical justification. Previous research only indicates a negative theoretical result: Thompson sampling could have a worst-case linear regret  $\Omega (T)$  with a constant threshold on the inference error measured by one  $\alpha$ -divergence. To bridge this gap, we propose an Enhanced Bayesian Upper Confidence Bound (EBUCB) framework that can efficiently accommodate bandit problems in the presence of approximate inference. Our theoretical analysis demonstrates that for Bernoulli multi-armed bandits, EBUCB can achieve the optimal regret order  $O(\log T)$  if the inference error measured by two different  $\alpha$ -divergences is less than a constant, regardless of how large this constant is. To our best knowledge, our study provides the first theoretical regret bound that is better than  $o(T)$  in the setting of constant approximate inference error. Furthermore, in concordance with the negative results in previous studies, we show that only one bounded  $\alpha$ -divergence is insufficient to guarantee a sub-linear regret.
The stochastic bandit problem, dated back to [37], is an important sequential decision-making problem that aims to find optimal adaptive strategies to maximize cumulative reward. At each time step, the learning agent chooses an action among all possible actions and observes its corresponding reward (but not others), and thus requires a balance between exploration and exploitation. Previous theoretical studies mainly focus on exact Bayesian bandit problems, requiring access to exact posterior distributions. However, their work cannot be easily applied to complex models such as deep neural networks, where maintaining exact posterior distributions tends to be intractable [36]. In contrast, approximate Bayesian bandit methods are widely employed in real-world applications with state-of-the-art performances [36, 41, 33, 45, 17, 50]. In comparison with exact algorithms, approximate Bayesian bandit algorithms are more challenging to analyze, as the inaccessibility of exact posterior sampling adds another level of discrepancy, and the resulting theory and solutions hence also differ substantially.
Few theoretical studies have been developed around Bayesian bandit approaches with approximate inference, despite their superior practical performance. [28] gave a theoretical analysis of an approximate sampling method called Ensemble Sampling, which possessed constant Kullback-Leibler divergence (KL divergence) error from the exact posterior and thus indicated a linear regret. [34]
showed that with a constant threshold on the inference error in terms of  $\alpha$ -divergence (a generalization of KL divergence), regardless of how small the threshold is, Thompson sampling with general approximate inference could have a linear regret in the worst case. Their work also showed that Thompson sampling combined with a small amount of forced exploration could achieve a  $o(T)$  regret upper bound, but no better result than  $o(T)$  was shown. Moreover, this improvement was mostly credited to the forced exploration rather than the intrinsic property of Thompson sampling. It appears that [34] illustrated a paradox that approximate Bayesian bandit methods worked well empirically but failed theoretically. Thus, further study regarding the fundamental understanding of approximate Bayesian bandit methods is necessary.
Motivated by the negative results in [34], [30] leveraged an efficient Markov chain Monte Carlo (MCMC) Langevin algorithm in multi-armed Thompson sampling so that the inference error would vanish along with increased sample numbers. [48] extended [30] to the contextual bandit problems and integrated contextual Thompson sampling with the Langevin algorithm that allowed the approximate posterior distribution to be sufficiently close to the exact posterior distribution. Hence, both works [48, 30] had a similar feature in terms of vanishing inference error. However, in other inference approaches, such as variational inference [8], the approximate posteriors might incur a systematic computational bias. To accommodate the latter scenario, we consider a general Bayesian inference approach that allows bounded inference error:
Is it possible to achieve the optimal regret order  $O(\log T)$  with a constant (non-vanishing) threshold on the inference error?
This question is not well investigated in previous literature, even for Bernoulli bandits. [34] showed that the answer is No for Thompson sampling when the inference error measured by only one  $\alpha$ -divergence is bounded. In this study, we will provide a novel theoretical framework and point out that the answer could be Yes when the inference error measured by two different  $\alpha$ -divergences is bounded where one  $\alpha$  is greater than 1, and the other  $\alpha$  is less than 0. This assumption guarantees that the approximate posterior is close to the exact posterior from two different "directions". Our finding implies that the problem of sub-optimal regret in the presence of approximate inference may not arise from the constant but from the design of the inference error. Our study takes the first step in deriving positive answers in the presence of constant approximate inference error, which provides some theoretical support for the superior performance of approximate Bayesian bandit methods.
In this study, we extend the work of the Bayesian Upper Confidence Bound (BUCB) [21, 20, 42, 17] to the setting of approximate inference and propose an enhanced Bayesian bandit algorithm that can efficiently accommodate approximate inference, termed as Enhanced Bayesian Upper Confidence Bound (EBUCB). In particular, we redesign the quantile choice in the algorithm to address the challenge of approximate inference: The original choice of  $t^{-1}$  provides the best regret bound without approximate inference, but in the presence of approximate inference, it leads to an undesirable quantile shift which degrades the performance. By adjusting the quantile choice, we theoretically demonstrate that EBUCB can achieve the optimal regret order  $O(\log T)$  if the inference error measured by two different  $\alpha$ -divergences  $(\alpha_{1} > 1$  and  $\alpha_{2} < 0$ ) is bounded. We also provide insights in the other direction: Instead of two different  $\alpha$ -divergences, controlling one  $\alpha$ -divergence alone is not sufficient to guarantee a sub-linear regret for both Thompson sampling, BUCB, and EBUCB. This further suggests that naive approximate inference methods that only minimize one  $\alpha$ -divergence alone could perform poorly, and thus it is critical to design approaches with two different  $\alpha$ -divergences reduced.
1) We propose a general Bayesian bandit framework, named EBUCB, to address the challenge of approximate inference. Our theoretical study shows that for Bernoulli bandits, EBUCB can achieve a  $O(\log T)$  regret upper bound when the inference error measured by two different  $\alpha$ -divergences is bounded. To the best of our knowledge, with constant approximate inference error, there is no existing result showing a regret upper bound that is better than  $o(T)$ , even for Bernoulli bandits.
2) We develop a novel sensitivity analysis of quantile shift with respect to inference error. This provides a fundamental tool to analyze Bayesian quantiles in the presence of approximate inference, which holds promise for broader applications, e.g., when the inference error is time-dependent.
3) We demonstrate that one bounded  $\alpha$ -divergence alone is insufficient to guarantee a sub-linear regret. Worst-case examples are constructed and illustrated where Thompson sampling/BUCB/EBUCB has  $\Omega(T)$  regret if only one  $\alpha$ -divergence is bounded. Hence, special consideration on reducing two different  $\alpha$ -divergences is necessary for real-world applications.
4) Our experimental evaluations corroborate our theory well, showing that our EBUCB is consistently superior to BUCB and Thompson sampling on multiple approximate inference settings.
Related Work. Bandit problems and their theoretical optimality have been extensively studied over decades [27, 10]. The seminal paper [25] (and subsequently [11]) established the first problem-dependent frequentist regret lower bound, showing that without any prior knowledge on the distributions, a regret of order  $(\log T)$  is unavoidable. Two popular lines of Bayesian bandit algorithms, Thompson sampling [1, 22, 16] and BUCB [21, 20], had been shown to match the lower bound, which indicated the theoretical optimality of those algorithms. Beyond Gaussian processes [42] and linear models [3, 39], exact computation of the posterior distribution is generally intractable, and thus, the approximate Bayesian inference techniques are necessary.
Some recent work focused on designing specialized methods to construct Bayesian indices since previous studies had demonstrated that Thompson sampling with constant inference error could exhibit linear regret in the worst-case scenario [28, 34]. [30] constructed Langevin algorithms to generate approximate samples with decreasing inference error and showed an optimal problem-dependent frequentist regret. [32] proposed variational Bayesian optimistic sampling, suggesting solving a convex optimization problem over the simplex at every time step. Unlike these researches, our study presents general results that only depend on the error threshold of approximate inference, rather than some specific approximate inference approaches.
Beyond Bayesian, another mainstream of bandit algorithms to address the exploration-exploitation tradeoff is upper confidence bound (UCB)-type frequentist algorithms [5-7, 13, 15, 40, 51, 24, 29, 43]. [12] revealed that Thompson sampling empirically outperformed UCB algorithms in practice, partly because UCB was typically conservative, as its configuration was data-independent which led to over-exploration [18]. BUCB [21] could be viewed as a middle ground between Thompson sampling and UCB. On the other hand, empirical studies [20] showed that Thompson sampling and BUCB performed similarly well in general.
The stochastic multi-armed bandit problem consists of a set of  $K$  actions (arms), each with a stochastic scalar reward following a probability distribution  $\nu_{i}$  ( $i = 1, \dots, K$ ). At each time step  $t = 1, \dots, T$ , where  $T$  is the time horizon, the agent chooses an action  $A_{t} \in [K]$  and in return observes an independent reward  $X_{t}$  drawn from the associated probability distribution  $\nu_{A_{t}}$ . The goal is to devise a strategy  $\mathcal{A} = (A_{t})_{t \in [T]}$ , to maximize the accumulated rewards through the observations from historical interactions.
In general, a wise strategy should be sequential, in the sense that the upcoming actions are determined and adjusted by the past observations: letting  $\mathcal{F}_t = \sigma(A_1, X_1, \ldots, A_t, X_t)$  be the  $\sigma$ -field generated by the observations up to time  $t$ ,  $A_t$  is  $\sigma(\mathcal{F}_{t-1}, U_t)$ -measurable, where  $U_t$  is a uniform random variable independent from  $\mathcal{F}_{t-1}$  (as algorithms may be randomized). More precisely, let  $\mu_j$  ( $j \in [K]$ ) denote the mean reward of the action  $j$  (i.e., the mean of the distribution  $\nu_i$ ), and without loss of generality, we assume that  $\mu_1 = \max_{j \in [K]} \mu_j$ . Then maximizing the rewards is equivalent to minimizing the (frequentist) regret, which is defined as the expected difference between the reward accumulated by an "ideal" strategy (a strategy that always playing the best action), and the reward accumulated by a strategy  $\mathcal{A}$ :
$$
R (T, \mathcal {A}) := \mathbb {E} \left[ T \mu_ {1} - \sum_ {t = 1} ^ {T} X _ {t} \right] = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left(\mu_ {1} - \mu_ {A _ {t}}\right) \right]. \tag {1}
$$
The expectation is taken with respect to both the randomness in the sequence of successive rewards from each action  $j$ , denoted by  $(Y_{j,s})_{s\in \mathbb{N}}$ , and the possible randomization of the algorithm,  $(U_{t})_{t\in [T]}$ . Let  $N_{j}(t) = \sum_{s = 1}^{t}\mathbb{1}(A_{s} = j)$  denote the number of draws from action  $j$  up to time  $t$ , so that  $X_{t} = Y_{A_{t},N_{A_{t}}(t)}$ . Moreover, let  $\hat{\mu}_{j,s} = \frac{1}{s}\sum_{k = 1}^{s}Y_{j,k}$  be the empirical mean of the first  $s$  rewards from action  $j$  and let  $\hat{\mu}_j(t)$  be the empirical mean of action  $j$  after  $t$  rounds of the bandit algorithm. Therefore  $\hat{\mu}_j(t) = 0$  if  $N_{j}(t) = 0$ ,  $\hat{\mu}_j(t) = \hat{\mu}_{j,N_j(t)}$  otherwise.
Note that the true mean rewards  $\pmb{\mu} = (\mu_{1},\dots,\mu_{K})$  are fixed and unknown to the agent. In order to perform Thompson sampling, or more generally, Bayesian approaches, we artificially define a prior distribution  $\Pi_0$  on  $\pmb{\mu}$ . Let  $\Pi_t$  be the exact posterior distribution of  $\pmb{\mu}|\mathcal{F}_{t-1}$  with density function  $\pi_t$
Algorithm 1 Enhanced Bayesian Upper Confidence Bound (EBUCB) with Approximate Inference
Input:  $T$  (time horizon),  $\Pi_0 = Q_0$  (initial prior on  $\mu$ ),  $c$  (parameters of the quantile), and a real-value increasing sequence  $\{\gamma_t\}$  such that  $\gamma_t \to 1$  as  $t \to \infty$   
for  $t = 1$  to  $T$  do  
    for each action  $j = 1, \dots, K$  do  
        Compute  $qu_j(t) = Qu(\gamma_t, Q_{t-1,j})$ .  
    end for  
Draw action  $A_t = \arg \max_{j=1,\dots,K} qu_j(t)$   
    Get reward  $X_t = Y_{A_t, N_{A_t}(t)}$   
    Obtain the approximate distribution  $Q_t$   
end for
with marginal distributions  $\pi_{t,1},\dots,\pi_{t,K}$  for actions  $1,\ldots ,K$  . Specifically, if at time step  $t$  , the agent chooses action  $A_{t} = j$  and consequently observes  $X_{t} = Y_{A_{t},N_{A_{t}}(t)}$  , the Bayesian update for action  $j$  is
$$
\pi_ {t, j} (\theta) \propto \nu_ {\theta} \left(X _ {t}\right) \pi_ {t - 1, j} (\theta), \tag {2}
$$
whereas for  $i \neq j$ ,  $\pi_{t,i} = \pi_{t-1,i}$ . At each time step  $t$ , we assume that the exact posterior computation in (2) cannot be obtained explicitly and an approximate inference method is able to give us an approximate distribution  $Q_t$  (instead of  $\Pi_t$ ). We use  $q_t$  to denote the density function of  $Q_t$ .
First, we consider a standard case where the exact posterior is accessible. In Thompson sampling [1, 44], a sample  $\hat{m}$  is drawn from the posterior distribution  $\Pi_{t-1}$  and then an action  $A_t$  is selected using the following strategy:  $A_t = i$  if  $\hat{m}_i = \max_j \hat{m}_j$ . In BUCB [21], we compute the quantile of the posterior distribution  $qu_j(t) = Qu(1 - \frac{1}{t(\log T)^c}, \Pi_{t-1,j})$  for each action  $j$ , where  $Qu(\gamma, \rho)$  is the quantile function associated to the distribution  $\rho$ , such that  $P_\rho(X \leq Qu(\gamma, \rho)) = \gamma$ . Then we select action  $A_t$  as follows:  $A_t = i$  if  $qu_i(t) = \max_j qu_j(t)$ .
Next, we move to a more concrete example, Bernoulli multi-armed bandit problems with a standard setting used in seminal papers [1, 2, 21, 22]. In these problems, each (stochastic) reward follows a Bernoulli distribution  $\nu_{i} \sim \mathrm{Bernoulli}(\mu_{i})$  and these distributions are independent of each other. The prior  $\Pi_{0,j}$  is typically chosen to be the independent and identically distributed (i.i.d.) Beta(1,1), or the uniform distribution for every action  $j$ . Then the posterior distribution for action  $j$  is a Beta distribution  $\Pi_{t,j} = \mathrm{Beta}(1 + S_j(t), 1 + N_j(t) - S_j(t))$ , where  $S_j(t) = \sum_{s=1}^{t} \mathbb{1}\{A_s = j\} X_t$  is the empirical cumulative reward from action  $j$  up to time  $t$ . Then, Thompson sampling/BUCB chooses the samples/quantiles of the posterior  $\Pi_{t,j} = \mathrm{Beta}(1 + S_j(t), 1 + N_j(t) - S_j(t))$  respectively at each time step.
In the presence of approximate inference, Thompson sampling draws the sample  $\hat{m}$  from  $Q_{t-1}$ , as the exact  $\Pi_{t-1}$  is not accessible. Correspondingly, we modify the specific sequence of quantiles chosen by the BUCB algorithm with a general sequence of  $\{\gamma_t\}$ -quantiles and term it as Enhanced Bayesian Upper Confidence Bound (EBUCB) algorithm. The detailed pseudo algorithm of EBUCB is described in Algorithm 1. Note that the choice of  $\gamma_t$  should address the presence of inference error and should be trailed to the specific definition of inference error; See Remark 3.10 in Section 3.
In this section, we present a theoretical analysis of EBUCB. In Section 3.1, we provide the necessary background of  $\alpha$ -divergence on approximate inference error measurement. Then in Section 3.2, we develop a novel sensitivity analysis of quantile shift with respect to inference error. This provides a fundamental tool to analyze Bayesian quantiles in the presence of approximate inference. The general results therein will be used for our derivation for the regret upper bound of EBUCB in Section 3.3, and are also potentially useful for broad applications, e.g., when the inference error is time-dependent. Lastly, in Section 3.4, we provide examples where Thompson sampling/BUCB/EBUCB has a linear regret with arbitrarily small inference error measured by one  $\alpha$ -divergence alone. All proofs are given in the Appendix.
3.1 The Alpha Divergence for Inference Error Measurement
The  $\alpha$ -divergence, generalizing the KL divergence, is a common way to measure errors in inference methods.
Definition 3.1. The  $\alpha$ -divergence between two distributions  $P_{1}$  and  $P_{2}$  with density functions  $p_{1}(x)$  and  $p_{2}(x)$  is defined as:  $D_{\alpha}(P_1,P_2) = \frac{1}{\alpha(\alpha - 1)}\left(\int p_1(x)^\alpha p_2(x)^{1 - \alpha}dx - 1\right)$ , where  $\alpha \in \mathbb{R}$  and the case of  $\alpha = 0$  and 1 is defined as the limit.
Note that different studies use the  $\alpha$  parameter in different ways. Herein, our definition of  $\alpha$ -divergence does not follow Renyi's definition of  $\alpha$ -divergence [35]; Instead, we follow a generalized version of Tsallis's  $\alpha$ -divergence, which is adopted by [52, 31, 34]. Compared with Renyi's  $\alpha$ -divergence, Tsallis's  $\alpha$ -divergence does not involve a log function, and it has the following property:
Proposition 3.2 (Positivity and symmetry). For any  $\alpha \in \mathbb{R}$ ,  $D_{\alpha}(P_1, P_2) \geq 0$  and  $D_{\alpha}(P_1, P_2) = D_{1 - \alpha}(P_2, P_1)$ .
The  $\alpha$ -divergence contains many distances such as  $KL(P_2, P_1)(\alpha \rightarrow 0)$ ,  $KL(P_1, P_2)(\alpha \rightarrow 1)$ , Hellinger distance ( $\alpha = 0.5$ ), and  $\chi^2$  divergence ( $\alpha = 2$ ).  $\alpha$ -divergence is widely used in variational inference [8, 23, 26], which is one of the most popular approaches in Bayesian approximate inference. Moreover, it was also adopted in previous studies on Thompson sampling with approximate inference [34, 28]. In particular, the KL divergence (( $\alpha = 1$ )-divergence) is:  $KL(P_1, P_2) = \int p_1(x) \log \left( \frac{p_1(x)}{p_2(x)} \right) dx$ . In approximate Bayesian inference, the exact posterior distribution  $\Pi_t$  and the approximate distribution  $Q_t$  may differ from each other. To provide a statistical analysis of approximate sampling methods, we use the  $\alpha$ -divergence as the measurement of inference error (statistical distance) between  $\Pi_t$  and  $Q_t$ . Our starting point is the following:
Assumption 3.3. Suppose that there exists a positive value  $\epsilon \in (0, +\infty)$  and two different parameters  $\alpha_{1} > 1$  and  $\alpha_{2} < 0$  such that
$$
\begin{array}{l} D _ {\alpha_ {1}} \left(Q _ {t, j}, \Pi_ {t, j}\right) \leq \epsilon , \forall t \in [ T ], j \in [ K ], \\ P _ {\alpha_ {1}} \left(Q _ {t, j}, \Pi_ {t, j}\right) <   \dots \forall t, s \in [ T ], i \in [ K ]. \end{array} \tag {3}
$$
$$
D _ {\alpha_ {2}} \left(Q _ {t, j}, \Pi_ {t, j}\right) \leq \epsilon , \forall t \in [ T ], j \in [ K ].
$$
This assumption is adapted from [34] but we enhance theirs with two bounded  $\alpha$ -divergences, as [34] showed one bounded  $\alpha$ -divergence was not sufficient to guarantee the sublinear regret. However, in the following, we show that the optimal regret order  $O(\log T)$  is indeed achievable under Assumption 3.3 with two bounded  $\alpha$ -divergences. Intuitively,  $P_{2}$  is flatten to cover  $P_{1}$ 's entire support when minimizing  $D_{\alpha}(P_1,P_2)$  with a large  $\alpha$  (greater than 1), while when  $\alpha$  is small (less than 0),  $P_{2}$  fits the  $P_{1}$ 's dominant mode; See [31] for the implication of  $\alpha$ -divergence. Therefore, Assumption 3.3 guarantees that the approximate posterior is close to the exact posterior from two different "directions". It is worth mentioning that when one  $\alpha$ -divergence is small, it does not necessarily imply that any other  $\alpha$ -divergences are large or infinite. In fact, as long as the two distributions have densities with the same support, then any  $\alpha$ -divergence between them is finite. Note that Assumption 3.1 does not require the threshold  $\epsilon$  to be small; instead,  $\epsilon$  can be any finite positive number. We pinpoint that this assumption, as well as our subsequent results, are very general in the sense that it does not depend on any specific methods of approximate inference. To enhance credibility on Assumption 3.3, we make several additional remarks in Section A.
3.2 Quantile Shift with Inference Error
In this section, we develop a novel sensitivity analysis of quantile shift with respect to inference error, which implies that under Assumption 3.3, the  $\gamma$ -quantiles of  $\Pi_{t,j}$  and  $Q_{t,j}$  only differs from a bound depending on  $\epsilon$ . We provide a general result first, which is rigorously stated as follows:
Theorem 3.4. Consider any two distributions  $P_{1}$  and  $P_{2}$  with densities  $p_1(x)$  and  $p_2(x)$ . Let  $R_{i}$  denote the quantile function of the distribution  $P_{i}$ , i.e.,  $R_{i}(p)\coloneqq Qu(p,P_{i})$  ( $i = 1,2$ ). Let  $0 < \gamma < 1$ . Let  $\delta_{\gamma ,\epsilon}$  satisfy that  $R_{1}(\gamma) = R_{2}(\gamma +\delta_{\gamma ,\epsilon})$  where  $-\gamma \leq \delta_{\gamma ,\epsilon}\leq 1 - \gamma$ .
a) If  $D_{\alpha}(P_1, P_2) \leq \epsilon$  where  $\alpha > 1$ , then
$$
\delta_ {\gamma , \epsilon} \leq 1 - \gamma - (\epsilon \alpha (\alpha - 1) + 1) ^ {\frac {1}{1 - \alpha}} (1 - \gamma) ^ {\frac {\alpha}{\alpha - 1}}.
$$
Note that when  $\alpha > 1$ ,  $(\epsilon \alpha (\alpha - 1) + 1)^{\frac{1}{1 - \alpha}} < 1$  and  $(1 - \gamma)^{\frac{\alpha}{\alpha - 1}} < 1 - \gamma$ .
b) If  $D_{\alpha}(P_1, P_2) \leq \epsilon$  where  $\alpha < 0$ , then
$$
\delta_ {\gamma , \epsilon} \geq 1 - \gamma - (\epsilon \alpha (\alpha - 1) + 1) ^ {\frac {1}{1 - \alpha}} (1 - \gamma) ^ {\frac {\alpha}{\alpha - 1}}.
$$
Note that when  $\alpha < 0$ ,  $(\epsilon \alpha (\alpha -1) + 1)^{\frac{1}{1 - \alpha}} > 1$  and  $(1 - \gamma)^{\frac{\alpha}{\alpha - 1}} > 1 - \gamma$ .
c) Suppose that  $\alpha \in (0,1)$  and  $\epsilon \geq \frac{-1}{\alpha(\alpha - 1)}$ . Then for any  $\delta_{\gamma,\epsilon} \in [-\gamma,1 - \gamma]$ , there exist two distributions  $P_1$  and  $P_2$  such that  $D_{\alpha}(P_1,P_2) \leq \epsilon$ . This implies that the condition  $D_{\alpha}(P_1,P_2) \leq \epsilon$  cannot control the quantile shift between  $P_1$  and  $P_2$  in general when  $\alpha \in (0,1)$ .