Title: Proportional Response: Contextual Bandits for Simple and Cumulative Regret Minimization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Contextual Bandits; Adaptive Experimentation; Simple Regret; Reinforcement Learning
Abstract: In many applications, e.g. in healthcare and e-commerce, the goal of a contextual bandit may be to learn an optimal treatment assignment policy at the end of the experiment. That is, to minimize simple regret. However, this objective remains understudied. We propose a new family of computationally efficient bandit algorithms for the stochastic contextual bandit setting, where a tuning parameter determines the weight placed on cumulative regret minimization (where we establish near-optimal minimax guarantees) versus simple regret minimization (where we establish state-of-the-art guarantees). Our algorithms work with any function class, are robust to model misspecification, and can be used in continuous arm settings. This flexibility comes from constructing and relying on “conformal arm sets" (CASs). CASs provide a set of arms for every context, encompassing the context-specific optimal arm with a certain probability across the context distribution. Our positive results on simple and cumulative regret guarantees are contrasted with a negative result, which shows that no algorithm can achieve instance-dependent simple regret guarantees while simultaneously achieving minimax optimal cumulative regret guarantees.

Proportional Response: Contextual Bandits for Simple and Cumulative Regret Minimization

Abstract

In many applications, e.g. in healthcare and e-commerce, the goal of a contextual bandit may be to learn an optimal treatment assignment policy at the end of the experiment. That is, to minimize simple regret. However, this objective remains understudied. We propose a new family of computationally efficient bandit algorithms for the stochastic contextual bandit setting, where a tuning parameter determines the weight placed on cumulative regret minimization (where we establish near-optimal minimax guarantees) versus simple regret minimization (where we establish state-of-the-art guarantees). Our algorithms work with any function class, are robust to model misspecification, and can be used in continuous arm settings. This flexibility comes from constructing and relying on 'conformal arm sets" (CASs). CASs provide a set of arms for every context, encompassing the contextspecific optimal arm with a certain probability across the context distribution. Our positive results on simple and cumulative regret guarantees are contrasted with a negative result, which shows that no algorithm can achieve instance-dependent simple regret guarantees while simultaneously achieving minimax optimal cumulative regret guarantees.

1 Introduction

Learning and deploying personalized treatment assignment policies is crucial across domains such as healthcare and e-commerce [29, 25]. Traditional randomized control trials (RCTs), while foundational for policy learning [4, 7], can be inefficient and costly [30]. This motivates the study of adaptive sequential experimentation algorithms for the stochastic contextual bandit (CB) settings. The algorithm interacts with a finite sequence of users drawn stochastically from a fixed but unknown distribution. At each round, the algorithm receives a context (a user's feature vector), selects an action, and gets a corresponding reward. At the end of this adaptive experiment, the algorithm outputs a learned policy (mapping between contexts and actions).

Our algorithms are designed with the dual objectives of minimizing simple regret and cumulative regret . Simple regret quantifies the difference between the expected rewards achieved by the optimal policy and the policy learned at the conclusion of the experimental process. In contrast, cumulative regret encapsulates the summation of differences between the expected rewards generated by the

optimal policy and the exploration policies employed at each sequential round of decision-making. 1 Although there are many settings where simple regret is an important consideration, the majority of research in the contextual bandit field has focused on the minimization of cumulative regret. To the best of our knowledge, there is no general-purpose computationally efficient algorithm for pure exploration objectives like simple regret minimization in the contextual bandit setting. Further, there has been relatively little work so far into algorithms that explore the trade-off between multiple objectives like cumulative regret and simple regret (though see [3, 9, 37] for studies that address this empirically or juxtapose minimizing cumulative regret with estimating treatment effects or arm parameters). Our work seeks to address these gaps. We show that there is a trade-off between simple and cumulative regret minimization (formalized later in a lower-bound result). To navigate this trade-off, we proposes a new algorithm called Risk Adjusted Proportional Response (RAPR) with a tuning parameter ω ∈ [1 , K ] , which governs the weight placed on the two objectives. 2 The algorithm is general-purpose (in that it can address any user-specified reward and policy classes), ensures near-optimal guarantees, and is also computationally efficient.

Types of guarantees. In our analysis, we consider two different types of bounds on simple and cumulative regret, worst-case and instance-dependent guarantees. Here instance-dependent guarantees refer to bounds that surpass worst-case rates by exploiting instances with large gaps between the conditional expected rewards of the optimal and sub-optimal arms. Recent work by [11] has shown that it is not possible for contextual algorithms to have instance-dependent guarantees on cumulative regret (without suffering an exponential dependence on model class complexity); the authors instead develop algorithms that achieve minimax optimal (worst case optimal) cumulative regret guarantees (with square-root dependence on model class complexity). [26] developed the first general-purpose contextual bandit algorithm for pure exploration, and their algorithm achieved instance-dependent guarantees. They also show that instance-dependent best policy identification guarantees must come at the cost of worse than minimax optimal cumulative regret (discussed in detail later). We show a similar lower bound on cumulative regret for algorithms that achieve better instance-dependent simple regret guarantees, and propose the first family of algorithms that flexibly navigate such trade-offs.

Overview of our guarantees. The simple regret guarantees of RAPR are never worse than the minimax optimal rates (Theorem 2). Depending on the instance, RAPR achieves simple regret guarantees that are up to O (1 / √ ω ) times smaller compared to minimax optimal rates (Theorem 2). This improvement factor of O (1 / √ ω ) over minimax optimal rates is asymptotically achieved for instances where realizability holds (the reward model class is well specified) and the gap between the best and second best arm in terms of conditional expected reward is at least ∆ > 0 at every context (best-case instance in Theorem 2). RAPR provides these instance-dependent guarantees without the knowledge of any instance information. Unfortunately, the corresponding cumulative regret for the above instances is a factor of O ( √ ω ) times larger compared to minimax optimal rates (Theorem 1). The cumulative regret guarantees of our algorithm only degrade relative to the minimax optimal rate if the instance allows for better simple regret guarantees. Our lower bound (Theorem 3) considers the instances described above with ∆ = 0 . 24 (the gap between best and second best arm in terms of conditional expected reward). Theorem 3 shows that, for any algorithm that bounds the simple regret on these instances to O (1 / √ ω ) of the minimax optimal rates, its cumulative regret will be at least Ω( √ ω ) times the minimax optimal rates. RAPR thus achieves a near-optimal trade-off between guarantees on simple vs cumulative regrets when T is large enough. The trade-off contrasts with non-contextual bandits, where successive elimination ensures improved (compared to minimax) instance-dependent guarantees for both simple and cumulative regret [10, 35].

Types of CB algorithms. Contextual bandit algorithms broadly fall into two categories: regressionfree and regression-based. Regression-free algorithms create an explicit policy distribution, randomly choosing a policy for decision-making at any time-step [2, 5, 8, 26]. While these algorithms provide worst-case cumulative regret guarantees [2, 5, 8] or instance-dependent PAC guarantees for policy learning [26] without additional assumptions, they can be computationally intensive [11]: they require solving and storing the output of Ω( poly ( T )) cost-sensitive classification (CSC) problems [21] at every epoch (or update step). In contrast, regression-based algorithms [e.g., 1, 11, 34] construct a

conditional arm distribution using regression estimates of the expected reward, allowing for methods that need only solve O (1) regression or CSC problems at every epoch (or update step). Traditionally, these algorithms relied on realizability assumptions for optimal regret guarantees, but recent advances allow for misspecified reward model classes [6, 12, 22]. We develop regression-based algorithms and do not assume realizability. RAPR is the first general-purpose regression-based algorithm with attractive pure exploration (simple regret) guarantees.

Overview of our algorithm. We now describe the RAPR algorithm in more detail. We first define a surrogate objective for simple regret, the optimal cover, which is inversely proportional to the probability that the bandit exploration policy chooses the arm recommended by the unknown optimal policy. The optimal cover bounds the variance of evaluating the unknown optimal policy under our exploration policy. This surrogate objective can be minimized by appropriately designing our exploration policy/action selection kernels. To maintain the attractive computational properties of regression-based algorithms, RAPR does not construct an explicit distribution over policies as that distribution would have large support and would be computationally and memory intensive to maintain. Instead, the goal of minimizing the optimal cover is attained by directly constructing a distribution over arms for each arriving context. This in turn builds on a novel general-purpose uncertainty quantification at each context. Much of the existing literature constructs confidence intervals with point-wise guarantees, but existing approaches to constructing them rely on assumptions like linear realizability. For general function classes, these intervals may be too wide and are often computationally expensive to construct. To overcome this issue, we develop Conformal Arm Sets (CASs), which are a set of potentially optimal arms at each context. This uncertainty quantification is regression-based and computationally efficient to construct; it's general-purpose and shrinks at 'fast rates' (with square-root dependency on expected squared error bounds for regression). Unfortunately, these sets come with some risk of not containing the arm recommended by the optimal policy at every context. Nevertheless, we can use this uncertainty quantification to construct a distribution over arms at each context that helps us minimize the optimal cover by balancing the benefits and risks of relying on these CASs. The unavoidable trade-off between our simple and cumulative regret guarantees is an artifact of these risky sets. Beyond allowing us to trade off simple and cumulative regret guarantees, the flexibility of the approach also helps us extend to continuous arm settings and allows us to handle model misspecification.

Other Related Work. Our work connects to the literature on pure exploration, extensively studied in MABsettings (see overview in [24]). [10, 15] study elimination-based algorithms for fixed confidence best-arm identification (BAI). [19, 32] study variants of Thompson Sampling with optimal asymptotic designs for BAI. [18] propose sequential halving for fixed budget BAI. Our algorithm provides fixed confidence simple regret guarantees and can be seen as a generalization of successive elimination [10] to the contextual bandit setting. The key technical difference is that it is often impossible to construct sub-gaussian confidence intervals on conditional expected rewards. The uncertainty quantification we use is similar to the notion of conformal prediction (see [36] for a detailed exposition). Until recently, pure exploration had been nearly unstudied in contextual bandits. [38] provide a static exploration algorithm that achieves the minimax lower bound on sample complexity for linear contextual bandits. [26] then provided the first algorithm with instance-dependent ( ϵ, δ ) -PAC guarantees for contextual bandits. This algorithm is regression-free (adapts techniques from [2]) and requires a sufficiently large dataset of offline contexts as input. Hence, unfortunately, it inherits high memory and runtime requirements [See 11, for a more detailed discussion]. However, these costs come with the benefit that their notion of instance dependence leverages structure not only in the true conditional expected reward (as in Theorem 2) but also in the policy class (similar to policy disagreement coefficient [13]). They also prove a negative result, showing that it is not possible for an algorithm to have instance-dependent (0 , δ ) -PAC guarantees and achieve minimax optimal cumulative regret guarantees. Our hardness result is similar but complementary to their result, for we show a similar result for simple regret (rather than their (0 , δ ) -PAC sample complexity). 3 Our work also recovers some cumulative regret guarantees for the continuous arm case [27, 40], with new guarantees on simple regret and robustness to misspecification. Note that our restriction to 'slightly randomized" policies for the continuous arm case results in regret bounds with respect to a 'slightly randomized" (smooth) benchmark [see 40, for smooth regret].

1.1 Stochastic Contextual Bandits

We consider the stochastic contextual bandit setting, with context space X , (compact) arm space A , and a fixed but unknown distribution D over contexts and arm rewards. D X refers to the marginal distribution over contexts, and T signifies the number of rounds or sample size. At each time t ∈ [ T ] 4 , the environment draws a context x t and a reward vector r t ∈ [0 , 1] A from D ; the learner chooses an arm a t and observes a reward r t ( a t ) . To streamline notation for discrete and continuous arm spaces, we consider a finite measure space ( A , Σ , µ ) over the set of arms, with K shorthand for µ ( A ) . 5 For ease of exposition, we focus on the finite/discrete arm setting. Here A = [ K ] and µ is the count measure, and µ ( S ) = | S | for any S ⊆ A . A (deterministic) policy π maps contexts to singleton arm sets Σ 1 := { a | a ∈ A} 6 . With some abuse of notation, we also let π refer to the kernel given by π ( a | x ) = I ( a ∈ π ( x )) . An action selection kernel (randomized policy) p : A×X → [0 , 1] is a probability kernel that describes a distribution p ( ·| x ) over arms at every context x . We let D ( p ) be the induced distribution over X × A × [0 , 1] , where sampling ( x, a, r ( a )) ∼ D ( p ) is equivalent to sampling ( x, r ) ∼ D and then sampling a ∼ p ( ·| x ) .

A reward model f maps X × A to [0 , 1] , with f ∗ ( x, a ) := E D [ r t ( a ) | x t = x ] denoting the true conditional expected reward model. Our algorithm works with a reward model class F and a policy class Π . For a given model f and an action selection kernel p , we denote the expected instantaneous reward of p with f as R f ( p ) . We write R f ∗ ( p ) as R ( p ) to simplify notation when no confusion arises. The optimal policy associated with reward function f is defined as π f 7 .

The policy π f induced by f ∈ F is assumed to be within policy class Π without loss of generality. 8 For any S ⊆ A , with some abuse of notation, we let f ( x, S ) = ∫ a ∈ S f ( x, a ) dµ ( a ) /µ ( S ) . Note that π f ( x ) ∈ arg max S ∈ Σ 1 f ( x, S ) for all x . The regret of a policy π with respect to f is the difference between the optimal value and the actual value of π , denoted as Reg f ( π ) := R f ( π f ) -R f ( π ) . Finally, we let π ∗ denote the optimal policy in the class Π and let Reg Π ( · ) denote the regret with respect to π ∗ . That is, π ∗ ∈ arg max π ∈ Π R ( π ) and Reg Π ( π ) := R ( π ∗ ) -R ( π ) .

Objectives. Contextual bandit algorithms adaptively construct action sampling kernels (exploration policies) { p t } t ∈ [ T ] used to collect data over the T rounds. At the end of the adaptive experiment, the adaptively collected data is used to learn a policy ˆ π ∈ Π . We study two main objectives to measure quality of these outputs: [Objective 1] Cumulative regret minimization. Cumulative regret (CReg T ) is given by CReg T := ∑ T t =1 Reg f ∗ ( p t ) . It compares the cumulative expected reward obtained during the experiment with the expected reward of the policy ( π f ∗ ) induced by the true conditional expected reward ( f ∗ ). We seek to minimize cumulative regret which is equivalent to maximizing cumulative expected reward during the experiment. [Objective 2] Simple regret minimization. Simple regret is given by Reg Π (ˆ π ) . It compares the expected reward of the learnt policy ˆ π ∈ Π against the value of the optimal policy in the class Π . We seek to minimize simple regret which is equivalent to maximizing expected reward of the policy learnt at the end of the experiment. To understand the kind of exploration kernels ( { p t } t ∈ [ T ] ) that help with policy learning, we now identify a surrogate objective for simple regret (called optimal cover) that is in terms of the kernels used for exploration.

Definition 1 (Cover) . Given a kernel p and a policy π , we define the cover of policy π under the kernel p to be,

Additionally, for any pair of kernels ( p, q ) , we let V ( p, q ) := E x ∼ D X ,a ∼ q ( ·| x ) [ q ( a | x ) /p ( a | x )] . Finally, we use the term optimal cover for kernel p to refer to V ( p, π ∗ ) .

The cover measures the quality of data collected under the action selection kernel p for evaluating a given policy π and bounds the variance of commonly used unbiased estimators for policy value

[e.g., 2, 14, 39]. In particular, the cover under optimal policy 1 T ∑ T t =1 V ( p t , π ∗ ) can be treated as a surrogate objective for simple regret minimization (proven in Appendix E.2), which is particularly instructional in designing our algorithm to minimize simple regret.

Extending notation to continuous arms. In the continuous arm setting, evaluating arbitrary deterministic policies can be infeasible without extra assumptions [28]. Thus, we focus on 'slightly randomized' policies by generalizing Σ 1 to be the arm sets with measure one ( Σ 1 := { S ∈ Σ | µ ( S ) = 1 } ). 9 The granularity of these sets can be adjusted by scaling the finite measure µ , which also affects the value of K = µ ( A ) . We then continue defining policies be maps from X to Σ 1 and Π is a class of such policies. We overload notation and define the induced kernel as π ( a | x ) = I ( a ∈ π ( x )) , which is a valid definition since ∫ a I ( a ∈ π ( x )) dµ ( a ) = µ ( π ( x )) = 1 . All the remaining definitions, including R f ( π ) , π f , π ∗ and V ( p, π ) , relied on these induced kernels and continue to hold. While there are some measure theoretic issues that remain to be discussed, we defer these details to Appendix A.

Uniform sampling. Our algorithm frequently selects an arm uniformly from a constructed set of arms. In the context of a set S ⊆ A , uniform sampling refers to selecting an arm from the distribution q ( a ) := I ( a ∈ S ) /µ ( S ) . This constitutes a probability measure since its integral over A equals 1. In the discrete arm setting, uniform sampling from a set S ⊆ A implies selecting an arm according to the distribution I ( a ∈ S ) / | S | .

1.2 Oracle Assumptions

Our algorithm relies on two sub-routines. For generality, we abstract away these sub-routines by stating them as oracle assumptions, for which we describe two oracles, EstOracle and EvalOracle, in Assumptions 1 and 2 respectively. The EstOracle sub-routine is for estimating conditional expected reward models (Assumption 1), and the EvalOracle sub-routine is for estimating policy values (Assumption 2) according to the true and estimated reward models.

These sub-routine tasks are supervised learning problems. Hence, the average errors for the corresponding tasks can be bounded in terms of the number of samples ( n ) and a confidence parameter ( δ ′ ). The oracle assumptions specify the estimation rates. We let ξ : N × [0 , 1] → [0 , 1] denote the estimation rate for these oracles. For simplicity, we assume that they share the same rate and that ξ ( n, δ ′ ) scales polynomially in 1 /n and log(1 /δ ′ ) . In order to simplify the analysis, we also require ξ ( n/ 3 , δ ′ /n 3 ) be non-increasing in n . 10 We now formally describe these oracle assumptions, starting with EstOracle.

Assumption 1 (Estimation Oracle) . We assume access to a reward model estimation oracle (EstOracle) that takes as input an action selection kernel p , and n independently and identically drawn samples from the distribution D ( p ) . The oracle then outputs an estimated model ˆ f ∈ F such that for any δ ′ ∈ (0 , 1) , the following holds with probability at least 1 -δ ′ :

Where B ≥ 0 is a fixed but unknown constant that may depend on the model class F and distribution D , but is independent of the action selection kernel p .

In Assumption 1, the parameter B measures the bias of model class F ; under realizability, B equals 0. The function ξ characterizes the estimation variance, which decreases with increasing sample size. As long as the variance term (which shrinks as we gather more data) is larger than the fixed unknown bias ( B ) , we have from Assumption 1 that the expected squared error for the estimated reward model is bounded by 2 ξ . We use this bound on expected squared error to further bound how accurately the estimated reward model evaluates policies in the class Π (Lemma 6). However, since B is unknown, we need a test to detect when this policy evaluation bounds starts failing (which can only happen after the variance term gets dominated by the unknown bias term). To construct this test,

our algorithm relies on EvalOracle, which provides consistent independent policy value estimates and helps compare them with policy value estimates with respect to the estimated reward model.

Assumption 2 (Evaluation Oracle) . We assume access to an oracle (EvalOracle) that takes as input an action selection kernel p , n independently and identically drawn samples from the distribution D ( p ) , a set of m models { g i | i ∈ [ m ] } ⊆ F , and another action selection kernel q . The oracle then outputs a policy evaluation estimator ˆ R of true policy value, and a set of m policy evaluation estimators { ˆ R g i | i ∈ [ m ] }} that estimate policy value with respect to the models g 1 , g 2 , . . . , g m respectively. Such that for any δ ′ ∈ (0 , 1) , the following conditions simultaneously hold with probability at least 1 -( m +1) δ ′ :

| ˆ R ( π ) -R ( π ) | ≤ √ 2 V ( p, π ) ξ ( n, δ ′ ) + 2 ξ ( n, δ ′ ) / (min ( x,a ) ∈X×A p ( a | x )) for all π ∈ Π ∪ { q } .

| ˆ R f ( π ) -R f ( π ) | ≤ √ 2 ξ ( n, δ ′ ) for all π ∈ Π ∪ { q } and for all f ∈ { g i | i ∈ [ m ] } .

When F and Π are finite, one can construct oracles such that Assumptions 1 and 2 hold with ξ ( n, δ ′ ) = O (log(max( |F| , | Π | ) /δ ′ ) /n ) . One example of such a construction is given by using empirical squared loss minimization for EstOracle, using inverse propensity scores (IPS) for estimating R ( π ) in EvalOracle, and using the empirical average for estimating R f ( π ) in EvalOracle. The guarantees of these assumptions can be derived using Bernstein's inequality and union bounding. When F has pseudo-dimension [20] bounded by d and Π has the Natarajan-dimension bounded by d [17, 16], one can construct oracles such that Assumptions 1 and 2 hold with ξ ( n, δ ′ ) = O ( d log( nK/δ ′ ) /n ) .

2 Algorithm

Algorithm 1 ω Risk Adjusted Proportional Response ( ω -RAPR)

input: Trade-off parameter ω ∈ [1 , K ] , proportional response threshold β max = 1 / 2 , and confidence parameter δ (used in definition of ξ m ).

At a high level, our algorithm operates in two modes, indicated by a Boolean variable ' safe ". During mode one ( safe = True ), where estimated reward models are sufficiently accurate at evaluating policies in the class Π , 11 we use our estimated models to update our action selection kernel used

during exploration. During mode two ( safe = False ), where the condition for mode one no longer holds, we stop updating the action selection kernel used for exploration. Operationally our algorithm runs in epochs/batches indexed by m . Epoch m begins at round t = τ m -1 +1 and ends at t = τ m , and we use m ( t ) to denote the epoch index containing round t . We let ˆ m denote the critical epoch, at the end of which our algorithm changes mode (with ' safe " being updated from ' True " to ' False "); we refer to ˆ m as the algorithmic safe epoch . For all rounds in epoch m ≤ ˆ m , our algorithm samples action using the action selection kernel p m defined later in (4). For m> ˆ m , our algorithm samples action using p ˆ m -the action selection kernel used in the algorithmic safe epoch ˆ m .

We now describe the critical components of our algorithm. These include (i) data splitting and using oracle sub-routines; (ii) misspecification tests , which we use to identify the safe -mode switching epoch ˆ m ; and (iii) conformal arm sets , which presents a new form of uncertainty quantification that is critical in constructing p m +1 at the end of each epoch m ∈ [ ˆ m ] . Finally, we use these components to describe our final algorithm.

Data splitting and oracle sub-routines. Consider an epoch m ∈ [ ˆ m ] . Let S m denote the set of samples collected in this epoch: S m = { ( x t , a t , r t ( a t )) | t ∈ [ τ m -1 , τ m ] } . Our algorithm splits S m into three equally-sized subsets: S m, 1 , S m, 2 and S m, 3 . Algorithm 1 outlines using these subsets and the oracles (described in Section 1.2) to estimate reward models and evaluate policies. Based on Assumptions 1 and 2, we bound the errors for these estimates in terms of ξ m +1 = 2 ξ (( τ m -τ m -1 ) / 3 , δ/ (16 m 3 )) , where δ is a specified confidence parameter. As we will see later, our algorithm relies on these bounds to test for misspecification and construct action selection kernels.

Misspecification test. We first discuss the need for our misspecification test. Note that Assumption 1 is flexible and allows our reward model class F to be misspecified. In particular, the squared error of our reward model estimate may depend on an unknown bias term B . To account for this unknown B , it is useful to center our analysis around the safe epoch m ∗ := arg max { m ≥ 1 | ξ m +1 ≥ 2 B } , which denotes the last epoch where variance dominates bias. We show that for any epoch m ∈ [ m ∗ ] , the estimated reward model ˆ f m +1 is 'sufficiently accurate' at evaluating the expected reward of any policy in Π ∪{ p m +1 } . This property is critical in ensuring that the constructed action selection kernel p m +1 has low exploration regret Reg f ∗ ( p m +1 ) and a small optimal cover ( V ( p m +1 , π ∗ ) ). Since B and m ∗ are unknown, we need to test whether the estimated reward model is sufficiently accurate at evaluating these policies. When the test fails, the algorithm sets the variable 'safe' to False and stops updating the action selection kernel used for exploration. The core idea for this test comes from [23] although its application to simple regret minimization is new, and the form of our test differs a bit. We now state our misspecification test (2). At the end of each epoch m , the test is passed if (2) holds:

where α ¯ m empirically bounds V ( p ¯ m , π ∗ ) , the optimal cover for the action selection kernel used in epoch ¯ m (see (49)). The first term in (2) measures how well the estimated reward model ˆ f m +1 evaluates the policy π , and the second term accounts for under-explored policies (policies that have high regret under the reward model ˆ f m would be less explored in epoch m ).

Conformal arm sets. We proceed to introduce the notion of conformal arm sets (CASs), based on which we construct the action selection kernels employed by our algorithms. At the beginning of each epoch m , we construct CASs, denoted as { C m ( x, ζ ) | x ∈ X , ζ ∈ [0 , 1] } ; here ζ controls the probability with which the set C m contains the optimal arm. The construction of these sets rely on the models ( ˆ f 1 , . . . , ˆ f m ) estimated from data up to epoch m -1 , as defined below.

Definition 2 (Conformal Arm Sets) . Consider ζ ∈ (0 , 1) . At epoch m , for context x , the arm set C m ( x, ζ ) is given by (3) .

Similar to conformal prediction (CP) [36, 33], CASs have marginal coverage guarantees. We show that with high probability, we have π ∗ ( x ) lies in C m ( x, ζ ) with probability at least 1 -ζ over the context distribution. That is, Pr x ∼ D X ( π ∗ ( x ) ∈ C m ( x, ζ )) ≥ 1 -ζ with high-probability (see Appendix E.1). However, there is also a key technical difference. While CP provides coverage guarantees for the conditional random outcome, CASs provide coverage guarantees for π ∗ ( x ) -which is not a random variable given the context x . Hence, intervals estimated by CP need to be wide enough to account for conditional outcome noise, whereas CASs do not. CASs also have several advantages compared to pointwise confidence intervals used in UCB algorithms. First, CASs are computationally easier to construct. Second, CAS widths have a polynomial dependency on model class complexity, whereas pointwise intervals may have an exponential dependence for some function classes [see lower bound examples in 13]. Third, pointwise intervals require realizability, whereas the guarantees of CASs hold even without realizability (as long as the misspecification test in (2) holds). However, it's important to remember that these benefits of CASs come with the risk of only covering π ∗ ( x ) marginally over the context distribution - that is, these sets may not contain π ∗ ( x ) at all x .

Risk Adjusted Proportional Response Algorithm. We now describe the design of our algorithm, which is summarized in Algorithm 1. The algorithm depends on the following input parameters: ω ∈ [1 , K ] which controls the trade-off between simple and cumulative regret, the proportional response threshold β max = 1 / 2 , and confidence parameter δ . The algorithm also computes η m +1 (risk adjustment parameter for p m +1 ), α m +1 (empirical bound on optimal cover for p m +1 ), and λ m +1 ( · ) (empirical bound on average CAS size). At the end of every epoch m ∈ [ ˆ m ] , we construct the action selection kernel p m +1 given by (4).

At any context x , sampling arm a from p m +1 ( ·| x ) is equivalent to the following. Sample β uniformly from [0 , 1] , then sample arm a uniformly from the set C m +1 ( x, min( β max , β ) /η m +1 ) . A small β results in a larger CAS and a higher probability of containing the optimal arm for the sampled context. However, uniformly sampling an arm from a larger CAS also implies a lower probability on every arm in the set. Sampling β uniformly allows us to respond proportionately to the risk of not sampling the optimal arm while enjoying the benefits of smaller CASs. We refer to this as the Proportional Response Principle .

Similarly note that, a larger risk-adjustment parameter η m +1 encourages reliance on less risky albeit larger CASs. We want to choose η m +1 to tightly bound the the optimal cover (surrogate for simple regret), subject to cumulative regret constrains imposed by the trade-off parameter ω . To do this, we first let λ m +1 ( η ) be a high-probability empirical upper bound on E x ∼ D X [ µ ( C m +1 ( x, β max /η ))] . Hence, using (49), we can upper bound the optimal cover ( V ( p m +1 , π ∗ ) ) by λ m +1 ( η m +1 ) 1 -β max + K η m +1 . Our choice of η m +1 approximately minimizes this upper bound on the optimal cover, by choosing the largest feasible η ∈ [ η m , √ ωK/α m ] such that λ m +1 ( η ) ≤ K η (see (5)). Note that this choice of

η m +1 balances the risk of a small η (large K η ) with the benefits of a small λ m +1 ( η ) (small λ m +1 ( η ) 1 -β max ).

With η m +1 chosen, the action selection kernel p m +1 is decided. Now let α m +1 = 3 K/η m +1 , which is a high-probability empirical upper bound on V ( p m +1 , π ∗ ) . We then use α m +1 at the end of epoch m +1 to construct CASs, compute the risk-adjustment parameter, and test for misspecification.

Computation. We have O (log( T )) epochs. At the end of any epoch m ∈ [ ˆ m ] , we solve three optimization problems. The first is for estimating ˆ f m +1 , which often reduces to empirical squared loss minimization and is computationally tractable for several function classes F . The second is for computing the risk-adjustment parameter in (5) which can be solved via binary search. The third is for the misspecification test in (2), which can be solved via two calls to a cost-sensitive classification (CSC) solver (don't need this when assuming realizability, further if we only care about cumulative

regret, sufficient to use the simpler test in [22]). Finally, to learn a policy ˆ π at the end of T rounds, we need to solve (8) using a CSC solver (under realizability we can set ˆ π = π ˆ f m ( T ) -1 ). Hence, overall, ω -RAPR makes exponentially fewer calls to solvers compared to regression-free algorithms like [26].

3 Main Results

Our algorithm/analysis/results hold for both the discrete and continuous arm cases. As discussed before, minimizing optimal cover helps us ensure improved simple regret guarantees. Hence α m ∈ [1 , 3 K ] (the high-probability empirical upper bound on the optimal cover V ( p m , π ∗ ) ) will play a crititcal role thoughout this results section. We start with stating our cumulative regret bounds.

Theorem 1. Suppose Assumptions 1 and 2 hold. Then with probability 1 -δ , ω -RAPR attains the following cumulative regret guarantee. Here ξ m +1 = ξ (( τ m -τ m -1 ) / 3 , δ/ (16 m 3 ) for all m .

Where we use ˜ O to hide terms logarithmic in T, K, ξ ( T, δ ) .

√

We start with discussing (6b). The first part ωKBT comes from the bias of the regression oracle with model class F and will vanish under the realizability assumption. The second part ∑ T t = τ 1 +1 √ ωKξ m ( t ) , when setting ω = 1 , recovers near-optimal (upto logarithmic factors) minimax cumulative regret guarantees for common model classes, as demonstrated by the following examples.

Corollary 1. We consider ω -RAPR with appropriate oracles in the following cases and let B denote the corresponding bias terms. When F and Π are finite, CReg T ≤ ˜ O ( √ ωKBT + √ ωKT log(max( |F| , | Π | ) /δ )) with probability at least 1 -δ . When F has a finite pseudo dimension d , Π has a finite Natarajan dimension d , and A is finite, CReg T ≤ ˜ O ( √ ωKBT + √ ωKTd log( TK/δ )) with probability at least 1 -δ . Note that under realizability ( B = 0 ), 1 -RAPR achieves near-optimal minimax cumulative guarantees.

In (6a), we observe that the multiplicative √ ω cost to cumulative regret is only incurred if the empirical bound on optimal cover ( α m ∈ [1 , 3 K ] ) can get small. That is, our cumulative regret bounds degrade only if our algorithm better bounds the optimal cover and thus ensures better simple regret guarantees. 12 We now provide instance dependent simple regret guarantees for our algorithm.

Theorem 2. Suppose Assumptions 1 and 2 hold. For some ( λ, ∆ , A ) ∈ [0 , 1] × (0 , 1] × [1 , K ] , consider instances where for 1 -λ fraction of contexts at most A arms are ∆ optimal (i.e. (7) holds).

Let m ′ = min( ˆ m,m ( T )) -1 . Let the learned policy ˆ π be given by (8) (equivalent to variance penalized policy optimization).

Then with probability 1 -δ , ω -RAPR has the following simple regret bound when T samples.

Under (7), we can only argue that the expected (over context distribution) measure of ∆ optimal arms is at most (1 -λ ) A + Kλ = O ( A + Kλ ) . Hence for large T , the best we can hope for is instance-dependant simple regret guarantees that shrink/improve over minimax guarantees by a factor of O ( √ ( A + Kλ ) /K ) . We show that this is guaranteed by Theorem 2. Suppose ω = K , F has a finite pseudo dimension bounded by d , and Π has a finite Natarajan dimension bounded by d . The simple regret guarantee of Theorem 2 reduces to ˜ O (min(( √ Kd/T, √ ( A + Kλ ) d/T + ( K/ √ ∆) √ d/T 4 √ B + d/T )) . When the reward model estimation bias B is small enough, the term ( K/ √ ∆) √ d/T 4 √ B + d/T ) is dominated by the remaining terms for large T . Hence, in this case, we get a simple regret bound of ˜ O ( √ ( A + Kλ ) d/T ) for large T . As promised, this improves upon the minimax guarantees by a factor of O ( √ ( A + Kλ ) /K ) .

Note that Theorem 1 guarantees are better for ω closer to 1 whereas Theorem 2 guarantees are better for ω closer to K . Hence these theorems show a tradeoff between the cumulative and simple regret guarantees for ω -RAPR. Theorem 3 shows that improving upon minimax simple regret guarantees for instances satisfying (7) may come at the unavoidable cost of worse than minimax optimal cumulative regret guarantees. This contrasts with non-contextual bandits, where successive elimination ensures improved (compared to minimax) gap-dependent guarantees for both simple and cumulative regret.

Theorem 3. Given parameters K,F,T ∈ N and ϕ ∈ [1 , ∞ ) . There exists a context space X and a function class F ⊆ ( X × A → [0 , 1]) with K actions such that |F| ≤ F and the following lower bound on cumulative regret holds:

Here ( a 1 , . . . a T ) denotes the actions selected by an algorithm A . D denotes the set of environments such that f ∗ ∈ F and (7) hold with ( A,λ, ∆) = (1 , 0 , 0 . 24) . Π denotes policies induced by F . Ψ ϕ denotes the set of CB algorithms that run for T rounds and output a learned policy with a simple regret guarantee of √ ϕ log F/T for any instance in D with confidence at least 0 . 95 , i.e., Ψ ϕ := { A : P ( Reg (ˆ π A ) ≤ √ ϕ log F/T ) ≥ 0 . 95 for any instance in D} . Finally, ˜ Ω( · ) hides factors logarithmic in K and T .

Near optimal trade-off of RAPR. Note that the environments constructed in Theorem 3 satisfy f ∗ ∈ F with max( |F| , | Π | ) ≤ F and also satisfy (7) with ( A,λ, ∆) = (1 , 0 , 0 . 24) . With appropriate oracles, Assumptions 1 and 2 are satisfied with B = 0 (i.e. m ∗ = ∞ ) and ξ ( n, δ ′ ) = O (log( F/δ ′ ) /n ) . Hence for large enough T , ω -RAPR achieves a simple regret bound of ˜ O ( √ ( K/ω ) log F/T ) with probability at least 0 . 95 and thus is a member of Ψ ϕ for some ϕ = ˜ O ( K/ω ) . Theorem 3 lower bounds the cumulative regret of such algorithms by ˜ Ω( √ K/ϕ √ KT log F ) = ˜ Ω( √ ω √ KT log F ) . Up to logarithmic factors, this matches the cumulative regret upper bound for ω -RAPR. Re-emphasizing that the trade-off observed in Theorems 1 and 2 is near optimal for large T .

Simulations. To demonstrate the computational tractability of our approach, we ran a simulation on setting within a R 2 context space, eight arms, linear models, and an exploration horizon of 5000 . Our algorithms ran in less than 9 seconds on a Macbook M1 Pro. We also compare with other baselines on simple/cumulative regret. See Appendix E.4 for details.

Conclusion. We develop Risk Adjusted Proportional Response (RAPR), a computationally efficient regression-based contextual bandit algorithm. It is the first contextual bandit algorithm capable of trading-off worst-case cumulative regret guarantees with instance-dependent simple regret guarantees. The versatility of our algorithm allows for general reward models, handles misspecification, extends to finite and continuous arm settings, and allows us to choose the trade-off between simple and cumulative regret guarantees. The key ideas underlying RAPR are conformal arm sets (CASs) to quantify uncertainty, proportional response principle for cumulative regret minimization, optimal cover as a surrogate for simple regret, and risk adjustment for better bounds on the optimal cover. 13 Limitations. A limitation of our approach is that we do not utilize the structure of the policy class being explored. Further refining CASs with other forms of uncertainty quantification that leverage such structure can lead to significant improvements, and potentially avoid trade-offs between simple/cumulative regret when policy class structure allows for it.

References

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems , pages 2312-2320, 2011.

Alekh Agarwal, Daniel Hsu, Satyen Kale, John Langford, Lihong Li, and Robert Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. In International Conference on Machine Learning , pages 1638-1646, 2014.

Susan Athey, Undral Byambadalai, Vitor Hadad, Sanath Kumar Krishnamurthy, Weiwen Leung, and Joseph Jay Williams. Contextual bandits in a survey experiment on charitable giving: Within-experiment outcomes versus policy learning. arXiv preprint arXiv:2211.12004 , 2022.

Abhijit Vinayak Banerjee, Esther Duflo, and Michael Kremer. The influence of randomized controlled trials on development economics research and on development policy. The state of Economics, the state of the world , pages 482-488, 2016.

Alina Beygelzimer, John Langford, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandit algorithms with supervised learning guarantees. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 19-26, 2011.

Aldo Gael Carranza, Sanath Kumar Krishnamurthy, and Susan Athey. Flexible and efficient contextual bandits with heterogeneous treatment effect oracles. In International Conference on Artificial Intelligence and Statistics , pages 7190-7212. PMLR, 2023.

Jishnu Das, Abhijit Chowdhury, Reshmaan Hussam, and Abhijit V Banerjee. The impact of training informal health care providers in india: A randomized controlled trial. Science , 354(6308):aaf7384, 2016.

Miroslav Dudik, Daniel Hsu, Satyen Kale, Nikos Karampatziakis, John Langford, Lev Reyzin, and Tong Zhang. Efficient optimal learning for contextual bandits. In Proceedings of the Twenty-Seventh Conference on Uncertainty in Artificial Intelligence , UAI'11, page 169-178, Arlington, Virginia, USA, 2011. AUAI Press.

Akram Erraqabi, Alessandro Lazaric, Michal Valko, Emma Brunskill, and Yun-En Liu. Trading off rewards and errors in multi-armed bandits. In Artificial Intelligence and Statistics , pages 709-717. PMLR, 2017.

Eyal Even-Dar, Shie Mannor, Yishay Mansour, and Sridhar Mahadevan. Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. Journal of machine learning research , 7(6), 2006.

Dylan Foster and Alexander Rakhlin. Beyond ucb: Optimal and efficient contextual bandits with regression oracles. In International Conference on Machine Learning , pages 3199-3210. PMLR, 2020.

Dylan J Foster, Claudio Gentile, Mehryar Mohri, and Julian Zimmert. Adapting to misspecification in contextual bandits. Advances in Neural Information Processing Systems , 33, 2020.

Dylan J Foster, Alexander Rakhlin, David Simchi-Levi, and Yunzong Xu. Instance-dependent complexity of contextual bandits and reinforcement learning: A disagreement-based perspective. arXiv preprint arXiv:2010.03104 , 2020.

Vitor Hadad, David A Hirshberg, Ruohan Zhan, Stefan Wager, and Susan Athey. Confidence intervals for policy evaluation in adaptive experiments. Proceedings of the national academy of sciences , 118(15):e2014602118, 2021.

Avinatan Hassidim, Ron Kupfer, and Yaron Singer. An optimal elimination algorithm for learning a best arm. Advances in Neural Information Processing Systems , 33:10788-10798, 2020.

Ying Jin. Upper bounds on the natarajan dimensions of some function classes. arXiv preprint arXiv:2209.07015 , 2022.

Ying Jin, Zhimei Ren, Zhuoran Yang, and Zhaoran Wang. Policy learning" without'overlap: Pessimism and generalized empirical bernstein's inequality. arXiv preprint arXiv:2212.09900 , 2022.

Zohar Karnin, Tomer Koren, and Oren Somekh. Almost optimal exploration in multi-armed bandits. In International Conference on Machine Learning , pages 1238-1246. PMLR, 2013.

Maximilian Kasy and Anja Sautmann. Adaptive treatment assignment in experiments for policy choice. Econometrica , 89(1):113-132, 2021.

Vladimir Koltchinskii. Oracle Inequalities in Empirical Risk Minimization and Sparse Recovery Problems: Ecole d'Eté de Probabilités de Saint-Flour XXXVIII-2008 , volume 2033. Springer Science & Business Media, 2011.

Akshay Krishnamurthy, Alekh Agarwal, Tzu-Kuo Huang, Hal Daumé III, and John Langford. Active learning for cost-sensitive classification. In International Conference on Machine Learning , pages 19151924. PMLR, 2017.

Sanath Kumar Krishnamurthy, Vitor Hadad, and Susan Athey. Adapting to misspecification in contextual bandits with offline regression oracles. In International Conference on Machine Learning , pages 58055814. PMLR, 2021.

Sanath Kumar Krishnamurthy, Adrienne Propp, and Susan Athey. Towards costless model selection in contextual bandits: A bias-variance perspective. arXiv preprint arXiv:2106.06483 , 2023.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web , pages 661-670. ACM, 2010.

Zhaoqi Li, Lillian Ratliff, Kevin G Jamieson, Lalit Jain, et al. Instance-optimal pac algorithms for contextual bandits. Advances in Neural Information Processing Systems , 35:37590-37603, 2022.

Maryam Majzoubi, Chicheng Zhang, Rajan Chari, Akshay Krishnamurthy, John Langford, and Aleksandrs Slivkins. Efficient contextual bandits with continuous actions. Advances in Neural Information Processing Systems , 33:349-360, 2020.

Wenlong Mou, Peng Ding, Martin J Wainwright, and Peter L Bartlett. Kernel-based off-policy estimation without overlap: Instance optimality beyond semiparametric efficiency. arXiv preprint arXiv:2301.06240 , 2023.

Susan A Murphy. Optimal dynamic treatment regimes. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 65(2):331-355, 2003.

Molly Offer-Westort, Alexander Coppock, and Donald P Green. Adaptive experimental design: Prospects and applications in political science. American Journal of Political Science , 65(4):826-844, 2021.

Maxim Raginsky and Alexander Rakhlin. Lower bounds for passive and active learning. Advances in Neural Information Processing Systems , 24, 2011.

Daniel Russo. Simple bayesian algorithms for best arm identification. In Conference on Learning Theory , pages 1417-1418. PMLR, 2016.

Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research , 9(3), 2008.

David Simchi-Levi and Yunzong Xu. Bypassing the monster: A faster and simpler optimal algorithm for contextual bandits under realizability. Available at SSRN , 2020.

Aleksandrs Slivkins et al. Introduction to multi-armed bandits. Foundations and Trends® in Machine Learning , 12(1-2):1-286, 2019.

Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world . Springer Science & Business Media, 2005.

Jiayu Yao, Emma Brunskill, Weiwei Pan, Susan Murphy, and Finale Doshi-Velez. Power constrained bandits. In Machine Learning for Healthcare Conference , pages 209-259. PMLR, 2021.

Andrea Zanette, Kefan Dong, Jonathan N Lee, and Emma Brunskill. Design of experiments for stochastic contextual linear bandits. Advances in Neural Information Processing Systems , 34:22720-22731, 2021.

Ruohan Zhan, Vitor Hadad, David A Hirshberg, and Susan Athey. Off-policy evaluation via adaptive weighting with data from contextual bandits. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 2125-2135, 2021.

Yinglun Zhu and Paul Mineiro. Contextual bandits with smooth regret: Efficient learning in continuous action spaces. In International Conference on Machine Learning , pages 27574-27590. PMLR, 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel contextual bandit algorithm that balances cumulative and simple regret guarantees using a surrogate objective and optimal cover. By constructing the conformal arms sets (CASs) which improve the coverage guarantees for the optimal policy, the algorithm efficiently finds the policy that minimizes simple regret while balancing with the cumulative regret. Cumulative and simple regret upper bounds are considered in the case when the model can be misspecified and the hardness result shows that the trade-off between simple and cumulative regret is unavoidable.\n\nSTRENGTHS:\nThe paper provides a theoretical analysis of the balance and trade-off between cumulative and simple regrets, which are not considered in the previous literature. The definition of cover and conformal arm sets is new and paves the way to characterize simple regret minimization. Generalizing regret bounds to the case when the model can be misspecified extends the currently existing literature.\n\nWEAKNESSES:\nW1. More discussions on theoretical results are required. Specifically, (a) the role of the action selection kernel q in Assumption 2, (b) what happens when $\\min_{(x,a)\\in\\mathcal{X}\\times\\mathcal{A}} p(a|x)=0$in Assumption 2 (line 180), (c) how the bias $B$ is multiplied to $\\sqrt{1/T}$ term rather than $O(1)$, (d) examples of the setting that satisfies the assumptions in Theorem 3.\n\nW2. The experiment section does not include the balance of cumulative regret and simple regret, which is the key part of the paper. More diverse settings are required to evaluate the algorithm.\n\nQUESTIONS:\nQ1. My main concern is in Theorem 3. The assumption in Theorem 3 implies a positive gap between the optimal and the second-best arm (e.g., Section 5.2 in [1] ) which enables the algorithm to attain zero simple regrets after $O(\\log T)$ exploration rounds and thus logarithmic cumulative regret bound. Does this result contradict Theorem 3?\n\nQ2. I couldn't understand the sentence \"it (the algorithm) does not utilize the structure of the policy class being explored\" in line 349. Could you elaborate more on this?\n\nQ3. In line 109, the integral in $f(x,S)=\\int_{a} f(x,a) d\\mu(a) / \\mu(S)$ ranges over $S$? Should it be $f(x,S)=\\int_{S} f(x,a) d\\mu(a) / \\mu(S)$?\n\n\n[1] Abbasi-Yadkori, Yasin, Dávid Pál, and Csaba Szepesvári. \"Improved algorithms for linear stochastic bandits.\" Advances in neural information processing systems 24 (2011).\n\nLIMITATIONS:\nYes. This work is mostly theoretical and the negative societal impact of the work is unseen.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-02"}

comments: SUMMARY:
This paper studies contextual bandits and develops a flexible algorithm that can be adapted to both cumulative regret minimization and simple regret minimization. New techniques are developed to analyze the proposed algorithm, e.g., constructing the conformal arm sets. The authors also prove a negative result stating that one cannot simultaneously achieve optimal simple regret and cumulative regret.

STRENGTHS:
This paper proposes a novel algorithm called RAPR, which utilizes two computational subroutines, an estimation subroutione and an evaluation subroutine. The RAPR algorithm takes a hyperparameter as input, which can be tuned to adapt to either cumulative regret minimization or simple regret minimization. This result seems quite interesting to me. The authors also complement the theoretical upper bound with a lower bound stating that one cannot simultaneously achieve optimal cumulative regret bound and simple regret bound.

WEAKNESSES:
I have questions regarding some wording used in the paper. More specifically:
1. The paper claims that they develop a new computationally efficient algorithm. However, it seems that the proposed algorithm is not entirely efficient under model misspecification since a cost-sensitive classification (CSC) subroutine is used in this case. It is known that CSC oracle can be NP-hard to implement even in simple cases.
2. The paper claims instance-dependent guarantees for simple regret minimization. However, it seems that the guarantees stated in Theorem 2 have a worst-case flavor, compared to instance-dependent results stated in Li et al, 2022.

QUESTIONS:
See the weaknesses part.

LIMITATIONS:
As stated on Page 9.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
This paper focus on the continues objective between simple regret minmization (SRM) and cumulative regret minimization (CRM) in contextual bandits, and propose an algorithm that can be either optimal in SRM, or optimal in CRM. Meanwhile, a new uncertainty quantification is propsed. Numerical simulations validate the efficacy.

STRENGTHS:
- Novelty: the paper focus on the objective between best arm identification (BAI) and regret minimization (RM), which is rarely studied but of great practical significance, as most works focus on either RM or BAI.
- Technical contribution: the conformal arm set as a new type of confidence bound is new, and is of independent interests beyond the topic.
- The results are strong, as a single algorithm can optimally change from one objective to another is not easy. It's also valuable for multi-objective related topics.

WEAKNESSES:
- The geometic covering in the algorithm (doubling the epochs) is common in RM, however, when it comes to simple regret minimization, there might be the case that only limited round $T$ is given, and thus requires an any-time algorithm.

- $\beta_{max}$ as the threshold of $\beta$ constrains that $\beta$ can not be too big, and restrains the smallest possible CAS. I'm confused with the counter-intuitive parameter, as for the objective of BAI, CAS should be as small as possible eventually. Why do we need the trucation threshold and why it's selected as $1/4$ should be explained.

- Though it is mentioned that the optimal cover can be seen as the surrogate objective for BAI, it is mainly used to estimate arbitrary policy $\pi$, while how it can be optimized to achieve a better $\pi$ is not clear.

QUESTIONS:
See weaknesses above.

LIMITATIONS:
\
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a computational efficient algorithm for contextual bandit that is capable of trading off between near-optimal minimax cumulative regret guarantees and instance-dependent simple regret guarantees. Additionally, It handles model misspecification, works in the continuous arm setting, and provides a lower bound that shows there is always a trade off in achieving simple regret and the cumulative regret.\n\nSTRENGTHS:\n- A significant contribution in achieving a trade-off between simple regret and cumulative regret.\n\n- One advantage of the algorithm is that it is regression based which makes it computationally efficient, while it does not rely on the realizability assumption, as it can adapt to model misspecification.\n\nWEAKNESSES:\nThe weaknesses mainly lie in the paper's poor writing and presentation, as well as in the insufficiently conducted simulations, detailed as bellow.\n\n- The motivation in the introduction lacks conviction and the main story is postponed to the related work section.\n\n- The introduction lacks a clear image of the achieved bounds and the bounds of related works, which could have been presented together in a single table along with all limitations.\n\n- in Section Preliminaries, the notations are listed in the footnote, which make it difficult for readers to follow the setup. In general, the flow in this section is confusing and needs improvement. Moreover, the authors used $\\Pi$ and $\\mathcal{F}$ in the \"objectives\" subsection without defining them before in the problem setup. Also definition of \"CB algorithm\" is missing.\n\n- Lack of clarity regarding the specific location in the appendix for referenced proofs and details.\n\n- Algorithm section is tedious with numerous links to various definitions, making it challenging to follow the flow. Besides that, the provided intuition is not sufficient for a non-expert reader to understand the complex formulas used for variables such as $\\lambda$, $\\eta$, and $\\alpha$. This can hinder comprehension for readers. Moreover, line 15 of algorithm $\\omega$-RAPR seems to be incorrect.\n\n- There are many arXiv papers listed in the references, whereas for most of them there are already the published version in some journal or conference.\n\n- The simulation part is extremely brief and lacks comparison to existing baselines, as well as an illustration of the results.\n\nI believe the paper requires a major revision due to various identified weaknesses.\n\nQUESTIONS:\n- I request the authors to elaborate further on the optimality of the achieved trade-off between simple regret and cumulative regret. Additionally, I am curious to understand how the provided lower bound sheds light on this trade-off and its implications in the context of the paper's contributions.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-19"}