SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sanath Kumar Krishnamurthy
Management Science and Engineering
sanathsk@stanford.edu
Ruohan Zhan
Industrial Engineering and Decision Analytics
rhzhan@ust.hk
Susan Athey
athey@stanford.edu
ebrun@cs.stanford.edu
In many applications, e.g. in healthcare and e-commerce, the goal of a contextual bandit may be to learn an optimal treatment assignment policy at the end of the experiment. That is, to minimize simple regret. However, this objective remains understudied. We propose a new family of computationally efficient bandit algorithms for the stochastic contextual bandit setting, where a tuning parameter determines the weight placed on cumulative regret minimization (where we establish near-optimal minimax guarantees) versus simple regret minimization (where we establish state-of-the-art guarantees). Our algorithms work with any function class, are robust to model misspecification, and can be used in continuous arm settings. This flexibility comes from constructing and relying on "conformal arm sets" (CASs). CASs provide a set of arms for every context, encompassing the context-specific optimal arm with a certain probability across the context distribution. Our positive results on simple and cumulative regret guarantees are contrasted with a negative result, which shows that no algorithm can achieve instance-dependent simple regret guarantees while simultaneously achieving minimax optimal cumulative regret guarantees.
Learning and deploying personalized treatment assignment policies is crucial across domains such as healthcare and e-commerce [29, 25]. Traditional randomized control trials (RCTs), while foundational for policy learning [4, 7], can be inefficient and costly [30]. This motivates the study of adaptive sequential experimentation algorithms for the stochastic contextual bandit (CB) settings. The algorithm interacts with a finite sequence of users drawn stochastically from a fixed but unknown distribution. At each round, the algorithm receives a context (a user's feature vector), selects an action, and gets a corresponding reward. At the end of this adaptive experiment, the algorithm outputs a learned policy (mapping between contexts and actions).
Our algorithms are designed with the dual objectives of minimizing simple regret and cumulative regret. Simple regret quantifies the difference between the expected rewards achieved by the optimal policy and the policy learned at the conclusion of the experimental process. In contrast, cumulative regret encapsulates the summation of differences between the expected rewards generated by the
optimal policy and the exploration policies employed at each sequential round of decision-making.1 Although there are many settings where simple regret is an important consideration, the majority of research in the contextual bandit field has focused on the minimization of cumulative regret. To the best of our knowledge, there is no general-purpose computationally efficient algorithm for pure exploration objectives like simple regret minimization in the contextual bandit setting. Further, there has been relatively little work so far into algorithms that explore the trade-off between multiple objectives like cumulative regret and simple regret (though see [3, 9, 37] for studies that address this empirically or juxtapose minimizing cumulative regret with estimating treatment effects or arm parameters). Our work seeks to address these gaps. We show that there is a trade-off between simple and cumulative regret minimization (formalized later in a lower-bound result). To navigate this trade-off, we proposes a new algorithm called Risk Adjusted Proportional Response (RAPR) with a tuning parameter  $\omega \in [1,K]$ , which governs the weight placed on the two objectives.2 The algorithm is general-purpose (in that it can address any user-specified reward and policy classes), ensures near-optimal guarantees, and is also computationally efficient.
Types of guarantees. In our analysis, we consider two different types of bounds on simple and cumulative regret, worst-case and instance-dependent guarantees. Here instance-dependent guarantees refer to bounds that surpass worst-case rates by exploiting instances with large gaps between the conditional expected rewards of the optimal and sub-optimal arms. Recent work by [11] has shown that it is not possible for contextual algorithms to have instance-dependent guarantees on cumulative regret (without suffering an exponential dependence on model class complexity); the authors instead develop algorithms that achieve minimax optimal (worst case optimal) cumulative regret guarantees (with square-root dependence on model class complexity). [26] developed the first general-purpose contextual bandit algorithm for pure exploration, and their algorithm achieved instance-dependent guarantees. They also show that instance-dependent best policy identification guarantees must come at the cost of worse than minimax optimal cumulative regret (discussed in detail later). We show a similar lower bound on cumulative regret for algorithms that achieve better instance-dependent simple regret guarantees, and propose the first family of algorithms that flexibly navigate such trade-offs.
Overview of our guarantees. The simple regret guarantees of RAPR are never worse than the minimax optimal rates (Theorem 2). Depending on the instance, RAPR achieves simple regret guarantees that are up to  $O(1 / \sqrt{\omega})$  times smaller compared to minimax optimal rates (Theorem 2). This improvement factor of  $O(1 / \sqrt{\omega})$  over minimax optimal rates is asymptotically achieved for instances where realizability holds (the reward model class is well specified) and the gap between the best and second best arm in terms of conditional expected reward is at least  $\Delta > 0$  at every context (best-case instance in Theorem 2). RAPR provides these instance-dependent guarantees without the knowledge of any instance information. Unfortunately, the corresponding cumulative regret for the above instances is a factor of  $O(\sqrt{\omega})$  times larger compared to minimax optimal rates (Theorem 1). The cumulative regret guarantees of our algorithm only degrade relative to the minimax optimal rate if the instance allows for better simple regret guarantees. Our lower bound (Theorem 3) considers the instances described above with  $\Delta = 0.24$  (the gap between best and second best arm in terms of conditional expected reward). Theorem 3 shows that, for any algorithm that bounds the simple regret on these instances to  $O(1 / \sqrt{\omega})$  of the minimax optimal rates, its cumulative regret will be at least  $\Omega (\sqrt{\omega})$  times the minimax optimal rates. RAPR thus achieves a near-optimal trade-off between guarantees on simple vs cumulative regrets when  $T$  is large enough. The trade-off contrasts with non-contextual bandits, where successive elimination ensures improved (compared to minimax) instance-dependent guarantees for both simple and cumulative regret [10, 35].
Types of CB algorithms. Contextual bandit algorithms broadly fall into two categories: regression-free and regression-based. Regression-free algorithms create an explicit policy distribution, randomly choosing a policy for decision-making at any time-step [2, 5, 8, 26]. While these algorithms provide worst-case cumulative regret guarantees [2, 5, 8] or instance-dependent PAC guarantees for policy learning [26] without additional assumptions, they can be computationally intensive [11]: they require solving and storing the output of  $\Omega(\mathrm{poly}(T))$  cost-sensitive classification (CSC) problems [21] at every epoch (or update step). In contrast, regression-based algorithms [e.g., 1, 11, 34] construct a
<sup>1</sup>Our formal definition of simple regret compares against the best policy in our policy class, while our cumulative regret definition compares against the global optimal policy (induced by the true conditional expected reward model). The reason for this discrepancy is because we use a regression based approach (due to computational considerations) for constructing our exploration policies.
${}^{2}K$  is the number of arms for the finite arm setting.
conditional arm distribution using regression estimates of the expected reward, allowing for methods that need only solve  $\mathcal{O}(1)$  regression or CSC problems at every epoch (or update step). Traditionally, these algorithms relied on realizability assumptions for optimal regret guarantees, but recent advances allow for misspecified reward model classes [6, 12, 22]. We develop regression-based algorithms and do not assume realizability. RAPR is the first general-purpose regression-based algorithm with attractive pure exploration (simple regret) guarantees.
Overview of our algorithm. We now describe the RAPR algorithm in more detail. We first define a surrogate objective for simple regret, the optimal cover, which is inversely proportional to the probability that the bandit exploration policy chooses the arm recommended by the unknown optimal policy. The optimal cover bounds the variance of evaluating the unknown optimal policy under our exploration policy. This surrogate objective can be minimized by appropriately designing our exploration policy/action selection kernels. To maintain the attractive computational properties of regression-based algorithms, RAPR does not construct an explicit distribution over policies as that distribution would have large support and would be computationally and memory intensive to maintain. Instead, the goal of minimizing the optimal cover is attained by directly constructing a distribution over arms for each arriving context. This in turn builds on a novel general-purpose uncertainty quantification at each context. Much of the existing literature constructs confidence intervals with point-wise guarantees, but existing approaches to constructing them rely on assumptions like linear realizability. For general function classes, these intervals may be too wide and are often computationally expensive to construct. To overcome this issue, we develop Conformal Arm Sets (CASs), which are a set of potentially optimal arms at each context. This uncertainty quantification is regression-based and computationally efficient to construct; it's general-purpose and shrinks at "fast rates" (with square-root dependency on expected squared error bounds for regression). Unfortunately, these sets come with some risk of not containing the arm recommended by the optimal policy at every context. Nevertheless, we can use this uncertainty quantification to construct a distribution over arms at each context that helps us minimize the optimal cover by balancing the benefits and risks of relying on these CASs. The unavoidable trade-off between our simple and cumulative regret guarantees is an artifact of these risky sets. Beyond allowing us to trade off simple and cumulative regret guarantees, the flexibility of the approach also helps us extend to continuous arm settings and allows us to handle model misspecification.
Other Related Work. Our work connects to the literature on pure exploration, extensively studied in MAB settings (see overview in [24]). [10, 15] study elimination-based algorithms for fixed confidence best-arm identification (BAI). [19, 32] study variants of Thompson Sampling with optimal asymptotic designs for BAI. [18] propose sequential halving for fixed budget BAI. Our algorithm provides fixed confidence simple regret guarantees and can be seen as a generalization of successive elimination [10] to the contextual bandit setting. The key technical difference is that it is often impossible to construct sub-gaussian confidence intervals on conditional expected rewards. The uncertainty quantification we use is similar to the notion of conformal prediction (see [36] for a detailed exposition). Until recently, pure exploration had been nearly unstudied in contextual bandits. [38] provide a static exploration algorithm that achieves the minimax lower bound on sample complexity for linear contextual bandits. [26] then provided the first algorithm with instance-dependent  $(\epsilon, \delta)$ -PAC guarantees for contextual bandits. This algorithm is regression-free (adapts techniques from [2]) and requires a sufficiently large dataset of offline contexts as input. Hence, unfortunately, it inherits high memory and runtime requirements [See 11, for a more detailed discussion]. However, these costs come with the benefit that their notion of instance dependence leverages structure not only in the true conditional expected reward (as in Theorem 2) but also in the policy class (similar to policy disagreement coefficient [13]). They also prove a negative result, showing that it is not possible for an algorithm to have instance-dependent  $(0, \delta)$ -PAC guarantees and achieve minimax optimal cumulative regret guarantees. Our hardness result is similar but complementary to their result, for we show a similar result for simple regret (rather than their  $(0, \delta)$ -PAC sample complexity).<sup>3</sup> Our work also recovers some cumulative regret guarantees for the continuous arm case [27, 40], with new guarantees on simple regret and robustness to misspecification. Note that our restriction to "slightly randomized" policies for the continuous arm case results in regret bounds with respect to a "slightly randomized" (smooth) benchmark [see 40, for smooth regret].
<sup>3</sup>In  $(\epsilon, \delta)$  PAC sample complexity results, given an input  $(\epsilon, \delta)$ , the objective is to minimize the number of samples needed in order to output an  $\epsilon$ -optimal policy with probability at least  $1 - \delta$  (a "fix accuracy, compute budget" setting). In contrast, in our simple regret case, we consider how to minimize the error  $\epsilon$  as the number of samples increases.
1.1 Stochastic Contextual Bandits
We consider the stochastic contextual bandit setting, with context space  $\mathcal{X}$ , (compact) arm space  $\mathcal{A}$ , and a fixed but unknown distribution  $D$  over contexts and arm rewards.  $D_{\mathcal{X}}$  refers to the marginal distribution over contexts, and  $T$  signifies the number of rounds or sample size. At each time  $t \in [T]^4$ , the environment draws a context  $x_t$  and a reward vector  $r_t \in [0,1]^\mathcal{A}$  from  $D$ ; the learner chooses an arm  $a_t$  and observes a reward  $r_t(a_t)$ . To streamline notation for discrete and continuous arm spaces, we consider a finite measure space  $(\mathcal{A}, \Sigma, \mu)$  over the set of arms, with  $K$  shorthand for  $\mu(\mathcal{A})$ . For ease of exposition, we focus on the finite/discrete arm setting. Here  $\mathcal{A} = [K]$  and  $\mu$  is the count measure, and  $\mu(S) = |S|$  for any  $S \subseteq \mathcal{A}$ . A (deterministic) policy  $\pi$  maps contexts to singleton arm sets  $\Sigma_1 := \{a | a \in \mathcal{A}\}^6$ . With some abuse of notation, we also let  $\pi$  refer to the kernel given by  $\pi(a|x) = I(a \in \pi(x))$ . An action selection kernel (randomized policy)  $p: \mathcal{A} \times \mathcal{X} \to [0,1]$  is a probability kernel that describes a distribution  $p(\cdot|x)$  over arms at every context  $x$ . We let  $D(p)$  be the induced distribution over  $\mathcal{X} \times \mathcal{A} \times [0,1]$ , where sampling  $(x,a,r(a)) \sim D(p)$  is equivalent to sampling  $(x,r) \sim D$  and then sampling  $a \sim p(\cdot|x)$ .
A reward model  $f$  maps  $\mathcal{X} \times \mathcal{A}$  to  $[0,1]$ , with  $f^{*}(x,a) \coloneqq \mathbb{E}_{D}[r_{t}(a)|x_{t} = x]$  denoting the true conditional expected reward model. Our algorithm works with a reward model class  $\mathcal{F}$  and a policy class  $\Pi$ . For a given model  $f$  and an action selection kernel  $p$ , we denote the expected instantaneous reward of  $p$  with  $f$  as  $R_{f}(p)$ . We write  $R_{f^{*}}(p)$  as  $R(p)$  to simplify notation when no confusion arises. The optimal policy associated with reward function  $f$  is defined as  $\pi_f^7$ .
$$
R_{f}(p):= \mathop{\mathbb{E}}_{x\sim D_{\mathcal{X}}}\mathop{\mathbb{E}}_{a\sim p(\cdot |x)}[f(x,a)],\text{and}\pi_{f}\in \arg \max_{\pi}R_{f}(\pi).
$$
The policy  $\pi_f$  induced by  $f \in \mathcal{F}$  is assumed to be within policy class  $\Pi$  without loss of generality. For any  $S \subseteq \mathcal{A}$ , with some abuse of notation, we let  $f(x, S) = \int_{a \in S} f(x, a) d\mu(a) / \mu(S)$ . Note that  $\pi_f(x) \in \arg \max_{S \in \Sigma_1} f(x, S)$  for all  $x$ . The regret of a policy  $\pi$  with respect to  $f$  is the difference between the optimal value and the actual value of  $\pi$ , denoted as  $\operatorname{Reg}_f(\pi) := R_f(\pi_f) - R_f(\pi)$ . Finally, we let  $\pi^*$  denote the optimal policy in the class  $\Pi$  and let  $\operatorname{Reg}_{\Pi}(\cdot)$  denote the regret with respect to  $\pi^*$ . That is,  $\pi^* \in \arg \max_{\pi \in \Pi} R(\pi)$  and  $\operatorname{Reg}_{\Pi}(\pi) := R(\pi^*) - R(\pi)$ .
Objectives. Contextual bandit algorithms adaptively construct action sampling kernels (exploration policies)  $\{p_t\}_{t\in [T]}$  used to collect data over the  $T$  rounds. At the end of the adaptive experiment, the adaptively collected data is used to learn a policy  $\hat{\pi} \in \Pi$ . We study two main objectives to measure quality of these outputs: [Objective 1] Cumulative regret minimization. Cumulative regret  $(\mathrm{CReg}_T)$  is given by  $\mathrm{CReg}_T := \sum_{t=1}^T \mathrm{Reg}_{f^*}(p_t)$ . It compares the cumulative expected reward obtained during the experiment with the expected reward of the policy  $(\pi_{f^*})$  induced by the true conditional expected reward  $(f^*)$ . We seek to minimize cumulative regret which is equivalent to maximizing cumulative expected reward during the experiment. [Objective 2] Simple regret minimization. Simple regret is given by  $\mathrm{Reg}_{\Pi}(\hat{\pi})$ . It compares the expected reward of the learnt policy  $\hat{\pi} \in \Pi$  against the value of the optimal policy in the class  $\Pi$ . We seek to minimize simple regret which is equivalent to maximizing expected reward of the policy learnt at the end of the experiment. To understand the kind of exploration kernels  $(\{p_t\}_{t\in [T]})$  that help with policy learning, we now identify a surrogate objective for simple regret (called optimal cover) that is in terms of the kernels used for exploration.
Definition 1 (Cover). Given a kernel  $p$  and a policy  $\pi$ , we define the cover of policy  $\pi$  under the kernel  $p$  to be,
$$
V (p, \pi) := \underset {x \sim D _ {\mathcal {X}}, a \sim \pi (\cdot | x)} {\mathbb {E}} \left[ \frac {\pi (a | x)}{p (a | x)} \right]. \tag {1}
$$
Additionally, for any pair of kernels  $(p,q)$ , we let  $V(p,q) \coloneqq \mathbb{E}_{x \sim D_{\mathcal{X}}, a \sim q(\cdot|x)}[q(a|x)/p(a|x)]$ . Finally, we use the term optimal cover for kernel  $p$  to refer to  $V(p,\pi^{*})$ .
The cover measures the quality of data collected under the action selection kernel  $p$  for evaluating a given policy  $\pi$  and bounds the variance of commonly used unbiased estimators for policy value
4For any  $n\in \mathbb{N}^+$  , we use notation  $[n]$  to denote the set  $\{1,\dots,n\}$
Here  $\Sigma$  is a  $\sigma$ -algebra over  $\mathcal{A}$  and  $\mu$  is a bounded set function from  $\Sigma$  to the real line.
The introduction of  $\Sigma_{1}$  is to allow for easy generalization to the continuous arm setting.
${}^{7}$  subject to any tie-breaking rule.
Note that  $\Pi$  may contain policies that are not induced by models in the class  $\mathcal{F}$ .
[e.g., 2, 14, 39]. In particular, the cover under optimal policy  $\frac{1}{T}\sum_{t=1}^{T} V(p_t, \pi^*)$  can be treated as a surrogate objective for simple regret minimization (proven in Appendix E.2), which is particularly instructional in designing our algorithm to minimize simple regret.
Extending notation to continuous arms. In the continuous arm setting, evaluating arbitrary deterministic policies can be infeasible without extra assumptions [28]. Thus, we focus on "slightly randomized" policies by generalizing  $\Sigma_{1}$  to be the arm sets with measure one ( $\Sigma_{1} := \{S \in \Sigma | \mu(S) = 1\}$ ). The granularity of these sets can be adjusted by scaling the finite measure  $\mu$ , which also affects the value of  $K = \mu(\mathcal{A})$ . We then continue defining policies to maps from  $\mathcal{X}$  to  $\Sigma_{1}$  and  $\Pi$  is a class of such policies. We overload notation and define the induced kernel as  $\pi(a|x) = I(a \in \pi(x))$ , which is a valid definition since  $\int_{a} I(a \in \pi(x)) d\mu(a) = \mu(\pi(x)) = 1$ . All the remaining definitions, including  $R_{f}(\pi)$ ,  $\pi_{f}$ ,  $\pi^{*}$  and  $V(p, \pi)$ , relied on these induced kernels and continue to hold. While there are some measure theoretic issues that remain to be discussed, we defer these details to Appendix A.
Uniform sampling. Our algorithm frequently selects an arm uniformly from a constructed set of arms. In the context of a set  $S \subseteq \mathcal{A}$ , uniform sampling refers to selecting an arm from the distribution  $q(a) := I(a \in S) / \mu(S)$ . This constitutes a probability measure since its integral over  $\mathcal{A}$  equals 1. In the discrete arm setting, uniform sampling from a set  $S \subseteq \mathcal{A}$  implies selecting an arm according to the distribution  $I(a \in S) / |S|$ .
1.2 Oracle Assumptions
Our algorithm relies on two sub-routines. For generality, we abstract away these sub-routines by stating them as oracle assumptions, for which we describe two oracles, EstOracle and EvalOracle, in Assumptions 1 and 2 respectively. The EstOracle sub-routine is for estimating conditional expected reward models (Assumption 1), and the EvalOracle sub-routine is for estimating policy values (Assumption 2) according to the true and estimated reward models.
These sub-routine tasks are supervised learning problems. Hence, the average errors for the corresponding tasks can be bounded in terms of the number of samples  $(n)$  and a confidence parameter  $(\delta')$ . The oracle assumptions specify the estimation rates. We let  $\xi : \mathbb{N} \times [0,1] \to [0,1]$  denote the estimation rate for these oracles. For simplicity, we assume that they share the same rate and that  $\xi(n,\delta')$  scales polynomially in  $1/n$  and  $\log(1/\delta')$ . In order to simplify the analysis, we also require  $\xi(n/3,\delta'/n^3)$  be non-increasing in  $n$ .<sup>10</sup> We now formally describe these oracle assumptions, starting with EstOracle.
Assumption 1 (Estimation Oracle). We assume access to a reward model estimation oracle (EstOracle) that takes as input an action selection kernel  $p$ , and  $n$  independently and identically drawn samples from the distribution  $D(p)$ . The oracle then outputs an estimated model  $\hat{f} \in \mathcal{F}$  such that for any  $\delta' \in (0,1)$ , the following holds with probability at least  $1 - \delta'$ :
$$
\underset {x \sim D _ {\mathcal {X}}} {\mathbb {E}} \underset {a \sim p (\cdot | x)} {\mathbb {E}} \left[ \left(\hat {f} (x, a) - f ^ {*} (x, a)\right) ^ {2} \right] \leq B + \xi (n, \delta^ {\prime})
$$
Where  $B \geq 0$  is a fixed but unknown constant that may depend on the model class  $\mathcal{F}$  and distribution  $D$ , but is independent of the action selection kernel  $p$ .
In Assumption 1, the parameter  $B$  measures the bias of model class  $\mathcal{F}$ ; under realizability,  $B$  equals 0. The function  $\xi$  characterizes the estimation variance, which decreases with increasing sample size. As long as the variance term (which shrinks as we gather more data) is larger than the fixed unknown bias  $(B)$ , we have from Assumption 1 that the expected squared error for the estimated reward model is bounded by  $2\xi$ . We use this bound on expected squared error to further bound how accurately the estimated reward model evaluates policies in the class  $\Pi$  (Lemma 6). However, since  $B$  is unknown, we need a test to detect when this policy evaluation bounds starts failing (which can only happen after the variance term gets dominated by the unknown bias term). To construct this test,