Title: Efficient Batched Algorithm for Contextual Linear Bandits with Large Action Space via Soft Elimination
Venue: NeurIPS 2023
Accepted: yes
Keywords: efficient bandit algorithms

contextual linear bandits
Abstract: In this paper, we provide the first efficient batched algorithm for contextual linear bandits with large action spaces. Unlike existing batched algorithms that rely on action elimination, which are not implementable for large action sets, our algorithm only uses a linear optimization oracle over the action set to design the policy. The proposed algorithm achieves a regret upper bound $\tilde{O}(\sqrt{T})$ with high probability,  and uses $O(\log\log T)$ batches, matching the lower bound on the number of batches (Gao et al., 2019). When specialized to linear bandits, our algorithm can achieve a high probability gap-dependent regret bound of $\tilde{O}(1/\Delta_{\min})$ with the optimal  $\log T$ number of batches, where $\Delta_{\min}$ is the minimum reward gap between a suboptimal arm and the optimal. Our result is achieved via a novel soft elimination approach, that entails $\text{``}$shaping$\text{"}$ the action sets at each batch so that we can efficiently identify (near) optimal actions.

Efficient Batched Algorithm for Contextual Linear Bandits with Large Action Space via Soft Elimination

Abstract

In this paper, we provide the first efficient batched algorithm for contextual linear bandits with large action spaces. Unlike existing batched algorithms that rely on action elimination, which are not implementable for large action sets, our algorithm only uses a linear optimization oracle over the action set to design the policy. The proposed algorithm achieves a regret upper bound ˜ O ( √ T ) with high probability, and uses O (log log T ) batches, matching the lower bound on the number of batches [13]. When specialized to linear bandits, our algorithm can achieve a high probability gap-dependent regret bound of ˜ O (1 / ∆ min ) with the optimal log T number of batches, where ∆ min is the minimum reward gap between a suboptimal arm and the optimal. Our result is achieved via a novel soft elimination approach, that entails 'shaping" the action sets at each batch so that we can efficiently identify (near) optimal actions.

1 Introduction

In contextual linear bandits, a learner interacts with an environment over T rounds: in each round t the learner observes a (possibly different due to context change) set of actions A t ⊆ R d , plays one of them, and receives a reward that follows a noisy linear function parametrized by an unknown vector in R d . The objective of the learner is to minimize regret - how much reward it loses over the T rounds by not always playing the 'highest reward" (optimal) action. To achieve this, the learner at each round updates its policy (its method to select what action to play) based on what it has learned from all past actions played and rewards observed. Linear bandits form the special case where the action set is always the same, i.e., A t = A for all rounds t . Contextual linear and linear bandits have been widely investigated due to their significance in both theory and practice (eg., see [24]).

Batched Setting. In numerous real-world use cases, the learner may be restricted to change the policy a limited (small) number of times.This constraint may stem from factors such as computation or communication considerations, or may be imposed by the nature of the application, as is the case in multi-stage clinical trials or online marketing campaigns with high response rates, where it is not feasible to update the policy after each response. Similarly, the use of crowdsourcing platforms or the need to conduct time-consuming simulations in reinforcement learning may require policies with limited adaptivity. As a result, there has been significant interest in designing algorithms that can achieve the optimal regret with limited policy switches [30, 3, 31, 13, 12, 21, 32, 16]. This setup is known as the batched contextual linear bandit problem: the T rounds are partitioned into batches, and the learner can collect rewards and update the action selection policy only at the end of each batch.

Large Action Space. Contextual linear bandit applications frequently need to explore an extremely large (even continuous) set of actions, e.g., millions of products to be recommended. As other examples, in the classical bandit problem of clinical trials, each decision involves selecting a treatment option from a potentially infinite set of mixed treatments [15]. In manufacturing problems, the goal is often to maximize revenue by selecting from a very large set of decisions, with the revenue associated with each decision being unknown [36]. Additionally, in applications where actions correspond to images in a database or high-dimensional embeddings of complex documents like webpages, the set of actions can be vast [26, 5]. As a result, there is a strong interest in algorithms that can be efficiently implemented when the action space is large or infinite [11, 7, 19, 39, 41].

While computationally-efficient batched algorithms exist for contextual linear bandits with small action sets, and efficient ones that are not batched exist for contextual linear bandits with large action sets, to date, there are no efficient batched algorithms that can handle large action spaces. Existing batched algorithms for contextual linear bandits [32, 16] rely on action elimination that requires a linear scan of the action set; while efficient non-batched algorithms for large action spaces do not extend to the batched setting [32, 16] (see related work in the following for more details).

Our Contributions. In this paper, we provide the first efficient batched algorithm for contextual linear bandits with nearly optimal regret upper bound of ˜ O ( d 3 / 2 √ T ) with high probability, while using O (log log T ) batches, which matches the lower bound on the number of batches required to achieve √ T -type regret bounds [13]. For linear bandits, our algorithm can attain a high probability gap-dependent regret bound of ˜ O ( d 3 / ∆ min ) with the optimal log T number of batches [13], where ∆ min represents the minimum reward gap between a suboptimal arm and the optimal.

Our algorithm for linear bandits, that we term SoftBatch, builds on a form of 'soft elimination'. Our observation is that, a good algorithm should be able to approximate the gap ∆ ( a ) between each action a ∈ A and the optimal one with O ( ∆ ( a )) accuracy; and if we can do that, then we can use this knowledge to limit the number of times we play suboptimal actions, as well as use this knowledge to select which actions we want to play at all. As essentially all batched algorithms do, at each batch we select and play (a small number of) actions that enable to estimate well the unknown parameter vector without incurring large regret. In particular, for each batch, we choose a set of well-behaved basis actions (e.g., a barycentric spanner [7]), established by calling an optimization oracle polynomial times. However, instead of selecting at batch m , vectors from the 'true' action set A , we consider virtual 'weighted' sets ˜ A m , where each action's magnitude is weighted inversely proportional to the estimated gap ∆ ( a ) , and select vectors guided by these weighted action sets. Then we play each basis action a a number of times inversely proportional to the square of the estimated gap ∆ ( a ) to preserve small regret. This in return provides us an accurate estimator for the optimal parameter by the benign properties of the basis actions. Thus our approach implements a form of soft elimination (shaping) of the action set, where the actions closest to the optimal become increasingly dominant. A crucial part in our design is that we never actually calculate the gaps ∆ ( a ) for all actions a ∈ A (only for the basis actions). The exploration policy we propose uses solely a linear optimization oracle applied to the original action set.

Our contextual bandit algorithm utilizes a recent reduction technique [18, 17] to transform the problem into a linear bandit problem. We incorporate the reduction into our batched linear bandit algorithm, by constructing an efficient linear optimization oracle for the exponentially large action set in the reduced problem using a linear optimization oracle for the original action sets (contexts).

Our proof techniques may be of independent interest. We develop a novel approach to bound regret in linear bandits, we design an efficient exploration policy using inverse squared gap weighting, and a simple method to handle the case where the action set does not span R d , where d is the problem dimension. Our approach avoids the necessity of imposing assumptions, such as the one in [41], which entails having a subset of d actions forming a matrix with determinant at least r d for a constant r . These assumptions can be strong, particularly when dealing with changing action sets, and may not hold after modifying the action set, for instance, by eliminating or weighting actions.

Related Work. Contextual linear and linear bandits have had significant impact both in theory and practice [1, 11, 28, 27, 24, 26, 37, 4, 8, 10]). The best performing algorithms achieve a regret bound ˜ O ( d √ T ) 1 [1, 11], matching the regret lower bound Ω ( d √ T ) [24]. The same algorithms achieve a

nearly optimal regret upper bound ˜ O ( d 2 ∆ min ) if the minimum gap of suboptimal arms is lower bounded by ∆ min . However, the resulting policies require updates at every time step and involve solving a non-convex optimization problem, which is not practical for large action spaces [33, 11]. Batched algorithms. Existing batched algorithms for contextual linear bandits [32, 16, 18] have achieved nearly optimal regret upper bounds of ˜ O ( d √ T ) . However, these algorithms rely on action elimination, which involves either performing a linear scan on the action set or solving an optimization problem over the non-convex set of good (not eliminated) actions to design and implement the policy at each time step. Similarly, batched algorithms for linear bandits [25, 12] also rely on action elimination. Although, unlike contextual bandits, the elimination constraint in linear bandits can be linear, which can be exploited to efficiently compute the policy (under certain assumptions) [7], resulting in an ˜ O ( d 3 / 2 √ T ) regret upper bound, it requires solving an optimization problem over the action set with an elimination constraint. This can be much harder than solving the optimization problem over the action set without additional constraints for some sets, such as the non-convex set resulting from the reduction of contextual to linear bandits [18].

Efficient algorithms for large action spaces. There is a long line of work on efficient algorithms for linear bandits that only rely on a linear optimization oracle over the action set [7, 11, 9, 19, 20]. However, these algorithms cannot be extended to the batched setting without extra assumptions on the action set, and more importantly, they do not extend to the batched contextual setting. Existing efficient algorithms for contextual linear bandits [6, 41, 11] can achieve ˜ O ( d 3 / 2 √ T ) regret bound, but it remains unclear if they can be extended to the batched setting, particularly given the challenge posed by changing action sets. Another line of work attempts to design efficient algorithms using hashing-based methods to approximate the maximum inner product [39, 22], but these methods result in complexity that is sublinear but still polynomial in the number of actions. Table 2 in App. A summarizes how our results position w.r.t. related work.

2 Model and Notation

Notation. We use [ n ] for a natural number n to denote the set { 1 , · · · , n } ; 1 ( E ) , for an event E , to denote the indicator function which returns 1 if E holds and 0 otherwise; B r = { a ∈ R d | ‖ a ‖ 2 ≤ r } to denote the ball of center 0 and radius r ; S r = { a ∈ R d | ‖ a ‖ 2 = r } to denote the sphere of center 0 and radius r ; and ‖ a ‖ V = √ a /latticetop V a to denote the matrix norm of a vector a ∈ R d with respect to a positive semi-definite matrix V . Table 1 in App. A summarizes our notation.

Contextual Linear Bandits. We consider a contextual linear bandit problem over an horizon of length T , where at each round t ∈ [ T ] , the learner receives a set of actions A t ⊆ R d sampled from an unknown distribution D independently from other rounds. The learner plays an action a t ∈ A t and receives a reward r t = 〈 a t , θ /star 〉 + η t , where θ /star is an unknown system parameter vector with θ /star ∈ R d , and η t is noise that is zero mean conditioned on the filtration of historic information ( A 1 , a 1 , r 1 , · · · , A t , a t ) . The learner adopts a policy that maps the history ( A 1 , a 1 , r 1 , · · · , A t ) to a distribution over the action set A t , with the objective of minimizing the pseudo regret defined as

For simplicity, we assume that A t is compact for all t ∈ [ T ] almost surely, which ensures the existence of an action a θ ∈ A t that attains the supremum sup a ∈ A t 〈 a, θ 〉 . Non-compact sets can be handled using sufficiently small approximations. We also adopt the following standard assumption [24].

Assumption 1. (Boundedness.) θ /star ∈ B 1 , A t ⊆ B 1 , and | r t | ≤ 1 almost surely ∀ t ∈ [ T ] .

Linear Bandits. Changing the action set over time enables to model contextual information. If the action space is fixed, namely, A t = A for for all t ∈ [ T ] , the problem is known as Linear Bandits. For Linear Bandits, we denote an optimal action by a /star = arg max a ∈ A 〈 a, θ /star 〉 and define the gap ∆ a = 〈 a /star -a, θ /star 〉 for all actions a ∈ A .

Batched Setting. In a batched setting, the learner is only allowed to change the policy at M prechosen rounds, where M is the number of batches. Batch m includes T m rounds, m ∈ [ M ] , with ∑ M m =1 T m = T . In each batch, the learner adopts a policy π that takes as input the action set A t along with all the previous history except for rewards observed in the current batch, and outputs a

distribution over the action set A t . In particular, the rewards of the actions pulled in the current batch are utilized solely to update the policy at the end of the batch.

Regularized least squares. Let { a i , r i } n i =1 be a sequence of n pulled actions and observed rewards over n rounds. The regularized least squares estimate ˆ θ of θ /star based on this action-reward sequence can be calculated as where V = λ I + ∑ n i =1 a i a /latticetop i , and λ is the regularization parameter.

Goal. Our goal is to design efficient batched algorithms for Contextual Linear and Linear Bandits with large (even infinite) action spaces that achieve (nearly) optimal regret.

We will do so by making use of the linear optimization oracles defined next.

Definition 1. A linear optimization oracle for a set A is a function O ( A ; . ) which takes as input θ ∈ B 1 and outputs O ( A ; θ ) ∈ A with 〈 O ( A ; θ ) , θ 〉 = sup a ∈ A 〈 a, θ 〉 . An approximate linear optimization oracle with additive error at most /epsilon1 for the set A is a function O + /epsilon1 ( A ; . ) : B 1 → A that satisfies 〈 O + /epsilon1 ( A ; θ ) , θ 〉 ≥ sup a ∈ A 〈 a, θ 〉 -/epsilon1 , ∀ θ ∈ B 1 . An approximate linear optimization oracle with multiplicative error 0 < α < 1 for the set A is a function O × α ( A ; . ) : B 1 → A that satisfies 〈 O × α ( A ; θ ) , θ 〉 ≥ (1 -α ) sup a ∈ A 〈 a, θ 〉 , ∀ θ ∈ B 1 .

Assumption 2. (Linear optimization oracle.) We assume that we can access a linear optimization oracle O ( A t ; . ) for each set of actions A t with running time at most T opt and space complexity M opt .

We note that assuming a linear optimization oracle over A t is natural [7, 11, 9, 19, 20, 41] since even if the learner perfectly learns the unknown parameter vector θ /star , the learner still needs to solve sup a ∈ A t 〈 a, θ /star 〉 to minimize the regret in (1).

3 Efficient Soft Elimination Algorithm for Linear Bandits

In this section we propose and analyze an algorithm (which we call SoftBach and describe in Algorithm 1) for linear bandits, that is, when A t = A .

3.1 Main Result

The following two theorems, proved in App. D and E, respectively, formally state that Algorithm 1 achieves (nearly) optimal regret using M = /ceilingleft log log T /ceilingright +1 batches with sample and time complexities polynomial in d and linear in T . We provide the algorithm description in Section 3.2 and a proof outline in Section 3.3.

Theorem 1. Consider a linear bandit instance with action set A ⊆ R d and horizon T . There exists a universal constant C and a choice for the batch lengths such that Algorithm 1 finishes in at most M = /ceilingleft log log T /ceilingright +1 batches with regret bounded as

where γ = 8 d √ C L (log(1 / δ ) + log T ) , C L = e 8 d and δ is a parameter. Moreover, if ∀ a ∈ A with ∆ a > 0 we have ∆ a ≥ ∆ min , then there exists a choice of batch lengths so that Algorithm 1 finishes in at most M = log 4 T batches with regret bounded as

Our regret bounds achieve nearly optimal dependency on T , and match the best known regret bounds of ˜ O ( d 3 / 2 √ T ) for (unbatched) efficient contextual linear bandit algorithms [6, 41, 11], while losing a √ d factor when compared to the Ω ( d √ T ) lower bound [24]. This extra √ d factor is due to relying on the best known method to design a notion of spanner of the set of actions (as we explain in section 3.2) with radius √ C L = O ( √ d ) using linear optimization oracles. Any future improvement that reduces the radius from O ( √ d ) to O (1) will immediately result in nearly optimal regret bounds for Algorithm 1. The following result upper bounds the time and space complexity.

Algorithm 1 [SoftBatch] A Batched Algorithm for Linear Bandits

1: Input: action set A ⊆ R d , horizon T , number of batches M , batch lengths { T m } M m =1 , confidence

parameter δ . 2: Let A ′ = A ∪ B 1 /T , C L = e 8 d, γ = 8 d √ C L (log(1 / δ ) + log T ) . 3: Initialize: θ 1 = 0 , a /star 1 is a random action in A , ∆ 1 ( a ) = 1 ∀ a ∈ A ′ , and T 0 = 1 . 4: for m = 1 : M do 5: Calculate { a 1 , . . . , a d } = LWS ( A ′ , η m = √ T m -1 / (8 γ ) , a /star m , θ m ) . 6: For the set { a 1 , . . . , a d } assign π ( i ) = 1 d , ∀ i ∈ [ d ] . 7: for i = 1 : d do 8: If a i / ∈ B 1 /T , calculate ∆ m ( a i ) = 〈 a /star m -a i , θ m 〉 and pull it n m ( i ) = /ceilingleft π ( i ) T m / 8 (1+ √ T m -1 ∆ m ( a i ) / (8 γ )) 2 /ceilingright times. go to step 10 if the number of pulls in the current batch reaches T m . Terminate Algorithm 1 if the total number of pulls reaches T . 9: Pull action a 0 = a /star m for max { 0 , T m -∑ d i =1 n m ( i ) } times. 10: Compute the regularized (with λ = 1 ) least squares estimator V m = I + ∑ T m i =1 ˜ a i ˜ a /latticetop i and θ m +1 = V -1 m ∑ T m i =1 r i ˜ a i , and ˜ a i is the action pulled in i -th round of the batch. 11: Update a /star m +1 = O + 1 T ( A ; θ m +1 ) .

Theorem 2. Algorithm 1 finishes in ˜ O ( Td 2 + d 4 M + T opt d 3 M ) runtime and uses ˜ O ( d 2 + M opt ) memory, where T opt , M opt are the time and space complexity of the linear optimization oracle.

We observe that unlike algorithms that require a linear scan on the action set, our space and time complexities are polynomial in the parameters d , T , and T opt .

3.2 SoftBatch (Algorithm 1) Description

Intuition. The main intuition behind SoftBatch is that, we do not need to necessarily eliminate suboptimal actions; it suffices to be able to select and play a small set of unique actions C m in each batch m , that allows to estimate increasingly well the parameter vector θ ∗ and the best action a /star while playing suboptimal actions for a small number of times. Our algorithm proposes a novel way to select such sets C m efficiently, through a form of 'action set shaping" that we will describe in this section. Additionally, to learn θ ∗ while achieving a (nearly) optimal regret, SoftBatch plays each action a ∈ C m a number of times ∝ 1 / ∆ 2 a , where ∆ a = 〈 a /star -a, θ /star 〉 is the gap for action a (i.e., we play the suboptimal actions in C m for a small number of times so as not to accumulate regret). SoftBatch enables to estimate the gap ∆ a within a constant factor for any action a (yet only does so for a limited number of actions in each batch), and essentially uses the gaps ∆ a as a guide on which actions to play and for how many rounds each.

Steps. SoftBatch (Algorithm 1) takes as input the action set A ⊆ R d , the horizon T , the number of batches M , and the batch lengths { T m } M m =1 , and operates as follows 2 .

In batch m , the algorithm starts with a current estimate of the parameter vector θ /star , which we call θ m , and an estimate of the optimal action a /star which we call a /star m ; note that given these, we are able to estimate for any action a ∈ A the gap ∆ m ( a ) = 〈 a /star m -a, θ m 〉 (but we will only do so for the actions the algorithm actually plays). The algorithm then calls LWS, a Linear Weighted Spanner subroutine (described in Algorithm 2), that it feeds with an augmented action space A ′ = A ∪ B 1 /T for reasons we will explain later. LWS selects d actions C m = { a 1 , · · · , a d } to play in batch m (note that some of these may belong to B 1 /T and will in this case not be played). Each of these d actions a i is pulled n m ( i ) ∝ π ( i ) ∆ m ( a i ) 2 times, where π ( . ) is a uniform exploration distribution with value 1 /d for all the d actions. We show in the proof of Theorem 1 that ∑ d i =1 n m ( i ) ≤ T m , ∀ m ∈ [ M ] , with high probability. To guarantee that the length of the batch is T m , the algorithm pulls a /star m for the remaining rounds, if needed. At the end of the batch, the algorithm updates its estimate θ m +1 of the unknown parameter vector using regularized least squares.

The remaining core part of the algorithm to discuss is the subroutine LWS, and we do so next. We start by providing our reasoning behind the LWS design.

The LWS Algorithm. Recall that we want LWS at each batch m to select d vectors { a i } ⊆ A ′ such that, by playing each n m ( i ) times, we can create a least-squares estimate θ m +1 of θ ∗ that allows an accurate estimate of the product 〈 a, θ /star 〉 for all a ∈ A . It is well-known (see [24]) that the error in estimating 〈 a, θ /star 〉 is proportional to ‖ a ‖ V -1 m , where V m = I + ∑ T m i =1 a i a /latticetop i is the least squares matrix we used to estimate θ m +1 . Thus, essentially we want LWS to select d vectors { a i } that maintain a small ‖ a ‖ V -1 m for all actions a ∈ A 3 . We can do so using what is called a G-optimal design [23].

Definition 2. (G-optimal design) For any set A ⊆ R d , a subset S ⊆ A , together with a distribution π over S is said to be a C -approximate optimal design for A if for any a ∈ A

Notice that if we were to play each action a i for n π ( i ) times, then V π would be (approximately) a normalized least squares matrix since V π + I /n = V /n , and hence, ‖ a ‖ 2 V -1 ≤ ‖ a ‖ 2 V -1 π /n .

where V π = ∑ a i ∈ S π ( i ) a i a /latticetop i 4 . When C = 1 this is referred to as a G-optimal design .

It is well-known that for any compact set, there exists a 1 -approximate optimal design [23] with |S| = d . However, computing an 1 -approximate optimal design is NP-hard in general [14, 35], even for small action sets. Computing a 2 -approximate optimal design can be done in polynomial time [38], but the complexity scales linearly with the size of the action set. Instead, we adopt an approach introduced in [7], which efficiently constructs an O ( √ d ) -approximate optimal design using only a linear optimization oracle. This relies on the concept of a barycentric spanner, which we define next.

Definition 3. (Barycentric spanner) For any set A ⊆ R d , a subset S = { a 1 , · · · , a d } ⊆ A is said to be a C -approximate barycentric spanner for A if any a ∈ A can be expressed as a linear combination of vectors in S with coeficients in [ -C, C ] .

It is easy to see that a C -approximate barycentric spanner together with a uniform distribution π ( i ) = 1 /d results in a C √ d -approximate optimal design [19, 41]. And importantly, a C -approximate barycentric spanner for a set A can be constructed using at most O ( d 2 log C d ) calls of a linear optimization oracle over the set A [19].

However, this is still not sufficient for us. Even though we can efficiently construct a C √ d -approximate optimal design for A , we do not want to pull these arms according to a uniform distribution; we want to pull action a i with estimated gap ∆ m ( a i ) for n m ( i ) = /ceilingleft π ( i ) T m / (1 + √ T m -1 ∆ m ( a ) / (8 γ )) 2 /ceilingright times to control the regret (which can be thought of as using a weighted distribution 5 ). But if we do not use the uniform distribution, the resulting least squares matrix V m may not satisfy that ‖ a ‖ V -1 m is sufficiently small for all actions a .

To account for this, instead of finding a C -approximate barycentric spanner for the set A , at each batch m we consider a virtual action set ˜ A m , which we define as where η m = √ T m -1 / (8 γ ) and find actions { a 1 , · · · , a d } ∈ A such that { φ m ( a i ) } d i =1 forms a C -approximate barycentric spanner for ˜ A m . The least squares matrix at batch m can be bounded as

with high probability 6 , where ˜ n m ( i ) is the number of times action a i is played in batch m and a 0 = a /star m . That is, playing actions { a 1 , · · · , a d } ∈ A for n m ( i ) times each, can equivalently

Algorithm 2 Linear Weighted Spanner (LWS) Algorithm

be thought of as playing actions { φ ( a 1 ) , · · · , φ ( a d ) } ∈ ˜ A m for π ( i ) T m times each; and since { φ ( a 1 ) , · · · , φ ( a d ) } form an approximate optimal design (through a barycentric spanner) for the set ˜ A m , the resulting least squares matrix will lead to small ‖ φ m ( a ) ‖ V -1 m values. In our proofs we show that a small enough ‖ φ m ( a ) ‖ V -1 m implies ‖ a ‖ V -1 m = O ( ∆ a ) as a result of the scaling in φ ( a ) . We prove in Lemma 5 in App. D that this allows to estimate ∆ a within a constant factor, which is all we need.

Intuitively, the virtual set ˜ A m weighs the actions inversely proportional to the estimated gap ∆ m ( a ) and batch length √ T m -1 : the larger the gap and T m -1 , the smaller magnitude the corresponding action has; this implements a form of soft elimination (shaping) of the action set, where the actions closest to the optimal become increasingly dominant as the batch length increases while the remaining fade out to zero. As a result, as m increases, the span of the optimal design focuses on the space where actions have small gaps, allowing to better distinguish among them.

To complete SoftBach (Algorithm 1), one last step is missing. LWS (Algorithm 2) follows standard steps (in Algorithm 2, see [7] for detailed explanation) to calculate the C-approximate barycentric spanner for ˜ A m . But to follow these steps, it requires the ability to solve the non-linear optimization problem sup a ∈ A 〈 φ m ( a ) , θ 〉 , since φ m ( a ) = a/ (1 + √ T m -1 ∆ m ( a ) / (8 γ )) is nonlinear in a . To do so, we will use 7 an approximate oracle with multiplicative error, that we term LW-ArgMax and describe next.

LW-ArgMax Algorithm. LW-ArgMax (Algorithm 3) constructs an approximate oracle with (1 -α ) -multiplicative error for the optimization sup a ∈ A 〈 φ m ( a ) , θ 〉 . This is sufficient: we show in Lemma 2 that Algorithm 2 can use LW-ArgMax to compute a C/ α -approximate barycentric spanner for ˜ m .

Recall that, before providing the action set A to Algorithm 2, SoftBatch extends to A ′ = A ∪ B 1 /T 8 . This guarantees that: A ′ spans R d (required to find a barycentric spanner [7]), and sup a ∈ A ′ 〈 a, θ 〉 ≥ 1 /T for all θ with ‖ θ ‖ 2 = 1 which implies that any approximate optimization oracle with additive error less than 1 / (2 T ) has multiplicative error of at most 1 / 2 . The extension of the set A results in the barycentric spanner possibly containing points not in A . However, we show that removing these points only affects sup a 1 by a constant factor, since /T has a small radius. Extending the

A a ∈ A ‖ ‖ V -B 1

Algorithm 3 LW-ArgMax Algorithm

1: Input: set of actions A , θ ∈ S 1 , parameter η , estimated best action ˆ a , estimate ˆ θ , horizon T. 2: Let ∆ ( a ) = 〈 ˆ a -a, ˆ θ 〉 , φ ( a ) = a/ (1 + η ∆ ( a )) . 3: Let W = 3log T , N = 36 W log 2 ( T ) , s = 1 -1 / 6 log T , /epsilon1 ′ = (1 -exp( -1)) / (12 T 7+12log T ) . 4: Initialize z = 2 W . 5: for i = 1 , · · · , N +1 do 6: ˜ θ = (1 + 1 /W ) z θ + z 1+1 /W η ˆ θ 7: a i = O + /epsilon1 ′ ( A ; ˜ θ / ‖ ˜ θ ‖ 2 ) . 8: z ← zs . 9: Return: arg max a ∈ { a i } N i =1 〈 φ ( a ) , θ 〉 .

set A to A ′ also handles the case where the span of A is smaller than R d , that was typically handled in literature by constructing a basis of A which can be complicated for some sets.

LW-ArgMax then builds on the following observation (proved as part of the proof of Lemma 1):

By choosing W to be large enough, the left hand side of (8) becomes a good approximation for 〈 φ ( a ) , θ 〉 . For a fixed z , the supremum on the right hand side of (8) reduces to a linear optimization over the set A ′ (that we solve using an approximate linear optimization oracle). Although the optimal value of z is not known, it can be bounded (see equation (25) in App. B); thus LW-ArgMax scans between upper and lower bounds on the optimal z with a constant multiplicative step. The pseudo-code is provided in Algorithm 3.

3.3 Proof Outline for Theorem 1

We start by proving that LW-ArgMax is an approximate linear optimization oracle for the set ˜ A with 1 -exp( -3) multiplicative error. The result is stated in Lemma 1 and proved in App. B.

Lemma 1. Let T ≥ 3 , η ∈ R , ˆ a ∈ R d , ˆ θ ∈ B T be given parameters, and A be a given set. Let ∆ ( a ) , φ ( a ) denote ∆ ( a ) = 〈 ˆ a -a, ˆ θ 〉 , φ ( a ) = a/ (1 + η ∆ ( a )) . If B 1 /T ⊆ A ⊆ B 1 , | η | ≤ T and 1 / 2 ≤ 1 + η ∆ ( a ) ≤ T 2 , ∀ a ∈ A , then for any θ ∈ S 1 , LW-ArgMax outputs an element a ∈ A such that

The conditions of Lemma 1 are easy to verify for all batches m ; namely, B 1 /T ⊆ A ⊆ B 1 holds as we extend the set of actions by adding B 1 /T before feeding it into Algorithm 3 and the condition 1 / 2 ≤ 1 + η ∆ ( a ) ≤ T 2 , ∀ a ∈ A is proved in Theorem 1 for all the inputs fed into Algorithm 3.

Given the result of Lemma 1, we next show that Algorithm 2 finds a C/ α -approximate barycentric spanner of the set ˜ A m , ∀ m ∈ [ M ] . This is done by slightly adapting the proof of Proposition 2 . 5 in [7] to work with approximate linear optimization oracles instead of exact oracles. The result is stated in the following theorem and the proof is provided in App. C for completeness.

Lemma 2. Let η ∈ R , ˆ a ∈ R d , ˆ θ ∈ R d be given parameters, and A be a given set. Let ∆ ( a ) , φ ( a ) denote ∆ ( a ) = 〈 ˆ a -a, ˆ θ 〉 , φ ( a ) = a/ (1 + η ∆ ( a )) . Suppose that 〈 φ ( LW-ArgMax ( θ )) , θ 〉 ≥ α sup a ∈ A 〈 φ ( a ) , θ 〉 , then Algorithm 2 computes a C/ α -approximate barycentric spanner for the set ˜ A = { φ ( a ) | a ∈ A} with at most O ( d 2 log C ( d/ α )) calls to LW-ArgMax.

To build our regret bounds, we essentially prove that a number of pulls of ∝ π ( i ) ∆ m ( a i ) 2 for action a i enables to estimate the gap ∆ m ( a i ) within a constant factor of the real gap ∆ a i . To do so, we start by providing an error bound for estimating 〈 φ m ( a ) , θ /star 〉 using standard sub-Gaussian concentration inequalities. Then, through mathematical induction, we extend this bound to the error of the action mean estimates 〈 a, θ m 〉 . Intuitively, if we believe that 〈 a, θ m 〉 is a good estimate of 〈 a, θ /star 〉 for all

actions, which implies ∆ m ( a ) is a good estimate of ∆ a at batch m , then even though the scale in φ m +1 by ∆ m ( a ) , this property will continue to hold at batch m + 1 . The constants multiplying ∆ m ( a ) in φ m are carefully designed to enable this. Finally, we show that the inverse squared gap weighting of the distribution enables to tightly upper bound the regret. /square

4 Algorithm for Contextual Linear Bandits

Our algorithm for contextual linear bandits is based on a technique proposed in [18], which reduces the contextual linear to a linear bandit setting. However, we cannot directly apply the reduction from [18] in Algorithm 1, as the reduction is not necessarily computationally efficient. Instead, we build a new algorithm (see Algorithm 4 in App. G) that incorporates reduction steps from [18] within Algorithm 1. One challenge we encounter is the introduction of a large, non-convex action set through the reduction process. To address this, we construct a linear optimization oracle over the new action set in order to ensure the efficiency of Algorithm 4. Additionally, the reduction requires estimating the expected value of a function (explained next and in App. G), and we carefully design the batch lengths to perform this estimation effectively. The following theorem describes our main result.

Theorem 3. Consider a contextual linear bandit instance with A t generated from an unknown distribution D . There exists a universal constant C and choice for batch lengths such that Algorithm 4 finishes in O (log log T ) batches with regret upper bounded as

R T ≤ C γ T log log T with probability at least 1 -δ , where γ = 10 √ C L d (log(16 M/ δ ) + 57 d log 2 (6 T )) . Moreover, the running time and space complexity are ˜ O ( d 4 + T opt d 3 T ) , ˜ O ( d 2 + M opt ) respectively.

We next briefly review the reduction and refer the reader to [18] for a detailed description. The basic idea in [18] is to establish a linear bandit action for each possible parameter vector θ of the contextual bandit instance. This is achieved through the use of the function g : R d → R d , which computes the expected best action under the context distribution D with respect to the parameter θ : g ( θ ) = E A ∼ D [ O ( A ; θ )] , where O is an optimization oracle (see Definition 1). A key insight, as stated in Theorem 1 of [18], is that if a t = O ( A t ; θ t ) for some θ t ∈ R d , then the reward generated by the contextual bandit instance can be expressed as r t = 〈 g ( θ t ) , θ /star 〉 + η ′ t , where η ′ t is noise with zero mean conditioned on the history. Consequently, the reward can be viewed as generated by pulling action g ( θ t ) in a linear bandit instance with an action set X = { g ( θ ) | θ ∈ Θ } . Moreover, the same theorem demonstrates that if a linear bandit algorithm is employed to choose g ( θ t ) ∈ X at round t and thus play action a t = O ( A t ; θ t ) , then | R T -R L T | = ˜ O ( √ T ) with high probability, where R L T = ∑ T t =1 sup θ ∈ Θ 〈 g ( θ ) -g ( θ t ) , θ /star 〉 is the regret of the algorithm on the linear bandit instance.

To estimate g , which depends on the unknown context distribution D , [18] proposes using contexts observed in previous batches. Specifically, the function g is replaced by g ( m +1) ( θ ) = 1 | H m | ∑ t ∈ H m O ( A t ; θ ) for all θ ∈ Θ ′ , where H m is the set of indices for rounds in batch m and Θ ′ = [ θ ] q | θ ∈ Θ is a discretization of Θ , [ θ ] q = q /floorleft θ √ d/q /floorright / √ d and q is the discretization parameter. The action set at batch m is correspondingly modified as X m = { g ( m ) ( θ ) | θ ∈ Θ ′ } . It is also shown in [18] that g ( m ) is a good estimate of g for all θ ∈ Θ ′ with high probability.

To leverage this reduction, we modify Algorithm 1 by adapting the action set in each batch based on the estimate of g , i.e., the set X m (note that we do not need to explicitly calculate the sets X m ). However, an issue arises where the estimate of θ /star at batch m depends on the approximate optimal design from batch m -1 , which employs the action set X m -1 estimated from the contexts of batch m -2 . In the proof of Theorem 3, we demonstrate that this leads to regret proportional to T m / √ T m -2 . If the batch lengths grow rapidly, significant regret may occur. To mitigate this, we reduce the growth rate of batch lengths by allowing them to increase only when m is odd (a similar technique was employed in [18]). The pseudo-code is in Algorithm 4.

To implement Algorithm 4 efficiently we need: (i) an approximate linear optimization oracle for the set X m with additive error at most /epsilon1 = (1 -exp( -1)) / (12 T 7+12log T ) : we show in Lemma 8 in App. F that g ( m ) ([ θ ] q ) can be used as our approximate oracle for q ≤ /epsilon1 / 2 ; and (ii) an inverse of the function g ( m ) to find θ t associated with g ( m ) ( θ t ) to play the action a t = O ( A t ; θ t ) : we observe

that all actions played by our algorithms (Algorithm 1 and 4) are the output of the approximate optimization oracle for some θ ; namely, for Algorithm 4 any pulled action is of the form g ( m ) ([ θ ] q ) for some input to the approximate oracle θ . Hence, the inversion of g ( m ) for the actions pulled by Algorithm 4 can be performed by storing [ θ ] q whenever the action g ( m ) ([ θ ] q ) is stored. This increases both the space and time complexity only by a constant factor.

Gap-dependent regret bounds for contextual linear bandits. The main difficulty in extending the gap-dependent regret bounds in Theorem 1 to the contextual case is that a large minimum action gap in the original action sets A t does not imply a large gap in the reduced action set X . As a simple example consider d = 1 , θ /star = 1 , and two action sets A 1 = { -1 , 1 } , and A 2 = { -1 } . At each iteration the learner receives the action set A 1 with probability p and A 2 with probability 1 -p independently from other iterations. Recall that the action set in the reduced instance X = { g ( θ ) | θ ∈ [ -1 , 1] } , where g ( θ ) = E A ∼ D [arg max a ∈ A 〈 a, θ 〉 ] . For θ ≥ 0 we have that g ( θ ) = p (1) + (1 -p )( -1) = 2 p -1 , while for θ < 0 we have g ( θ ) = p ( -1)+(1 -p )( -1) = -1 . Then X = { -1 , 2 p -1 } . Therefore, the suboptimality gap is ∆ min = (2 p -1)(1) -( -1)(1) = 2 p which can be arbitrarily small depending on p . Note that in the original contextual bandit instance, the minimum gap is at least 2 for both action sets.

While it may be possible to provide gap dependent regret bounds for our algorithm in the contextual case, this will require more sophisticated regret analysis that does not only rely on the reduced linear bandit instance.

Numerical results. In App. I, we provide a numerical example to compare the computational complexity of computing the exploration policy of our algorithm versus the complexity of computing the policy in [40].

5 Conclusion

In this paper, we proposed the first efficient batched algorithm for contextual linear bandits with large action spaces. Our algorithm achieves a high-probability regret upper bound of ˜ O ( √ T ) , uses O (log log T ) batches, and has a computational complexity that is linear in T and polynomial in d .

Acknowledgments

This work was partially supported by the NSF grants #2007714 and #2221871, the DARPA grant #HR00112190130, and by the Army Research Laboratory under the Cooperative Agreement W911NF-17-2-0196.

References

Y. Abbasi-Yadkori, D. Pál, and C. Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24, 2011.

N. Abe and P. M. Long. Associative reinforcement learning using linear probabilistic concepts. In ICML , pages 3-11. Citeseer, 1999.

A. Agarwal, S. Agarwal, S. Assadi, and S. Khanna. Learning with limited rounds of adaptivity: Coin tossing, multi-armed bandits, and ranking from pairwise comparisons. In Conference on Learning Theory , pages 39-75. PMLR, 2017.

A. Agarwal, S. Bird, M. Cozowicz, L. Hoang, J. Langford, S. Lee, J. Li, D. Melamed, G. Oshri, O. Ribas, et al. Making contextual decisions with low technical debt. arXiv preprint arXiv:1606.03966 , 2016.

D. Agarwal, B.-C. Chen, P. Elango, N. Motgi, S.-T. Park, R. Ramakrishnan, S. Roy, and J. Zachariah. Online models for content optimization. Advances in Neural Information Processing Systems , 21, 2008.

S. Agrawal and N. Goyal. Thompson sampling for contextual bandits with linear payoffs. In International conference on machine learning , pages 127-135. PMLR, 2013.

B. Awerbuch and R. Kleinberg. Online linear optimization and adaptive routing. Journal of Computer and System Sciences , 74(1):97-114, 2008.

D. Bouneffouf, A. Bouzeghoub, and A. L. Gançarski. A contextual-bandit algorithm for mobile context-aware recommender system. In Neural Information Processing: 19th International Conference, ICONIP 2012, Doha, Qatar, November 12-15, 2012, Proceedings, Part III 19 , pages 324-331. Springer, 2012.

S. Bubeck, N. Cesa-Bianchi, and S. M. Kakade. Towards minimax policies for online linear optimization with bandit feedback. In Conference on Learning Theory , pages 41-1. JMLR Workshop and Conference Proceedings, 2012.

W. Cai, J. Grossman, Z. J. Lin, H. Sheng, J. T.-Z. Wei, J. J. Williams, and S. Goel. Bandit algorithms to personalize educational chatbots. Machine Learning , 110(9):2389-2418, 2021.

V. Dani, T. P. Hayes, and S. M. Kakade. Stochastic linear optimization under bandit feedback. 2008.

H. Esfandiari, A. Karbasi, A. Mehrabian, and V. Mirrokni. Regret bounds for batched bandits. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 7340-7348, 2021.

Z. Gao, Y. Han, Z. Ren, and Z. Zhou. Batched multi-armed bandits problem. Advances in Neural Information Processing Systems , 32, 2019.

M. Grötschel, L. Lovász, and A. Schrijver. Geometric algorithms and combinatorial optimization , volume 2. Springer Science & Business Media, 2012.

I. S. T. C. Group et al. The international stroke trial (ist): a randomised trial of aspirin, subcutaneous heparin, both, or neither among 19 435 patients with acute ischaemic stroke. The Lancet , 349(9065):1569-1581, 1997.

Y. Han, Z. Zhou, Z. Zhou, J. Blanchet, P. W. Glynn, and Y. Ye. Sequential batch learning in finite-action linear contextual bandits. arXiv preprint arXiv:2004.06321 , 2020.

O. Hanna, L. Yang, and C. Fragouli. Learning from distributed users in contextual linear bandits without sharing the context. Advances in Neural Information Processing Systems , 35:11049-11062, 2022.

O. A. Hanna, L. F. Yang, and C. Fragouli. Contexts can be cheap: Solving stochastic contextual bandits with linear bandit algorithms. arXiv preprint arXiv:2211.05632 , 2022.

E. Hazan and Z. Karnin. Volumetric spanners: an efficient exploration basis for learning. Journal of Machine Learning Research , 2016.

S. Ito, D. Hatano, H. Sumita, K. Takemura, T. Fukunaga, N. Kakimura, and K.-I. Kawarabayashi. Oracle-efficient algorithms for online linear optimization with bandit feedback. Advances in Neural Information Processing Systems , 32, 2019.

T. Jin, P. Xu, X. Xiao, and Q. Gu. Double explore-then-commit: Asymptotic optimality and beyond. In Conference on Learning Theory , pages 2584-2633. PMLR, 2021.

K.-S. Jun, A. Bhargava, R. Nowak, and R. Willett. Scalable generalized linear bandits: Online computation and hashing. Advances in Neural Information Processing Systems , 30, 2017.

J. Kiefer and J. Wolfowitz. The equivalence of two extremum problems. Canadian Journal of Mathematics , 12:363-366, 1960.

T. Lattimore and C. Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

T. Lattimore, C. Szepesvari, and G. Weisz. Learning with good feature representations in bandits and in rl with a generative model. In International Conference on Machine Learning , pages 5662-5670. PMLR, 2020.

L. Li, W. Chu, J. Langford, and R. E. Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web , pages 661-670, 2010.

Y. Li, Y. Wang, X. Chen, and Y. Zhou. Tight regret bounds for infinite-armed linear contextual bandits. In International Conference on Artificial Intelligence and Statistics , pages 370-378. PMLR, 2021.

Y. Li, Y. Wang, and Y. Zhou. Nearly minimax-optimal regret for linearly parameterized bandits. In Conference on Learning Theory , pages 2173-2174. PMLR, 2019.

C. D. Meyer. Matrix analysis and applied linear algebra , volume 71. Siam, 2000.

V. Perchet, P. Rigollet, S. Chassang, and E. Snowberg. Batched bandit problems. The Annals of Statistics , pages 660-681, 2016.

C. Pike-Burke, S. Agrawal, C. Szepesvari, and S. Grunewalder. Bandits with delayed, aggregated anonymous feedback. In International Conference on Machine Learning , pages 4105-4113. PMLR, 2018.

Y. Ruan, J. Yang, and Y. Zhou. Linear bandits with limited adaptivity and learning distributional optimal design. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing , pages 74-87, 2021.

S. Sahni. Computationally related problems. SIAM Journal on computing , 3(4):262-279, 1974.

J. Sherman and W. J. Morrison. Adjustment of an inverse matrix corresponding to a change in one element of a given matrix. The Annals of Mathematical Statistics , 21(1):124-127, 1950.

M. D. Summa, F. Eisenbrand, Y. Faenza, and C. Moldenhauer. On largest volume simplices and sub-determinants. In Proceedings of the twenty-sixth annual ACM-SIAM symposium on Discrete algorithms , pages 315-323. SIAM, 2014.

K. T. Talluri, G. Van Ryzin, and G. Van Ryzin. The theory and practice of revenue management , volume 1. Springer, 2004.

A. Tewari and S. A. Murphy. From ads to interventions: Contextual bandits in mobile health. Mobile Health: Sensors, Analytic Methods, and Applications , pages 495-517, 2017.

M. J. Todd. Minimum-volume ellipsoids: Theory and algorithms . SIAM, 2016.

S. Yang, T. Ren, S. Shakkottai, E. Price, I. S. Dhillon, and S. Sanghavi. Linear bandit algorithms with sublinear time complexity. In International Conference on Machine Learning , pages 25241-25260. PMLR, 2022.

Z. Zhang, X. Ji, and Y. Zhou. Almost optimal batch-regret tradeoff for batch linear contextual bandits. arXiv preprint arXiv:2110.08057 , 2021.

Y. Zhu, D. J. Foster, J. Langford, and P. Mineiro. Contextual bandits with large action spaces: Made practical. In International Conference on Machine Learning , pages 27428-27453. PMLR, 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the first efficient batch algorithm for (contextual) linear bandits. Instead of relying on the classical elimination-based algorithmic design, this paper adopts a linear optimization oracle over the large action set. The proposed algorithm achieves $\\tilde{O}(\\sqrt{}T)$ regret with the number of batches matching the lower bound. The specific algorithm for contextual linear bandit is built upon a reduction technique which reduces contextual linear bandits to bandits. Directly applying this reduction incurs high computational cost. To circumvent this, an approximated linear optimization oracle is designed.\n\nSTRENGTHS:\n1. Meaningful theoretical contribution: the first efficient batch algorithm for contextual linear bandits is proposed and analyzed. The number of batches matches the lower bound. \n\n2. Novel idea in algorithmic design: instead of classical design based on hard elimination, this paper uses an interesting ‘soft’ elimination which eliminates candidates for the unknown model parameters in each batch, incurring low regret.\n\nWEAKNESSES:\nI did not detect any major weakness in this paper. However, I do believe this paper would benefit from adding some simple numerical simulation. I understand this is a theoretical paper. But due to the complexity of the theoretical analysis and technical algorithmic design, it would be easier to evaluate the performance of the algorithm via numerical simulations on contextual linear bandits. Specifically, it would be better if it can be demonstrated that the number of batches is indeed of the order $\\log \\log (T)$. It would also be interesting to compare with some benchmark algorithms for contextual linear bandits not subject to the batch constraint, and see whether the proposed algorithm has inferior regret due to the batch constraint. Simple numerical simulations suffice. \nI also have some questions. Please see the Questions section.\n\nQUESTIONS:\n1. I am unclear about what are the major technical novelty/contributions in the reduction discussed in section 4. It seems to me that most ideas follow directly from [18], except for the tuning of the batch length. It would be great if the authors can further elaborate. \n\n2. It is mentioned in line 76 that novel approaches are proposed to bound the regret of the algorithm. Could the authors emphasize/summarize the approaches? The current section 3.2 is quite overwhelming to readers.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an algorithm for the batched (contextual) linear bandit problem with a large action space. The proposed algorithm has low time and space complexity. The proposed algorithm for the batched linear bandit problem selects arms in each batch according to a G-optimal design based on a modified action set and chooses them a number of times, dependent on the estimated sub-optimality gap. As a result, it achieves efficient time and computational complexity, even when the number of actions is large, unlike the existing elimination algorithms. The extension to contextual linear bandits is achieved by modifying part of the existing reduction for linear bandits. The proposed algorithm achieves a near-optimal regret with the optimal number of batches. Although there is a gap between the upper and lower regret bounds concerning the dimension $d$ of the feature vectors, it is due to the inheritance of the approximation ratio of the G-optimal design with a barycentric spanner from existing studies.\n\nSTRENGTHS:\n- The paper proposes a novel and impactful algorithm.\n- The design and motivation of the proposed algorithm are clearly explained.\n- A well-summarized proof sketch is provided in the main body.\n\nWEAKNESSES:\n- The regret upper bound of the proposed algorithm is not optimal for the dimension $d$ of the feature vectors.\n\nQUESTIONS:\n#### Questions\n- No specific questions.\n\n#### Comments\n- Overall, I believe this is a high-quality paper.\n- In my opinion, this paper should include in the main body the choice of batch size and how it is derived (or citations) if space allows.\n\n#### Minor Comments\n- Line 201: an uniform -> a uniform\n- Line 287: $B_1$ -> $B_{1/T}$\n\nLIMITATIONS:\n- In this paper, limitations are well discussed (e.g., assumptions for the problem).\n- While societal impacts are not discussed in this paper, it could be considered a paper with little direct impact on society since it is a theoretical paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new batched contextual linear bandits algorithm that achieves $\\widehat{O}(\\sqrt{T})$ regret while using $O(\\log\\log T)$ batches. The algorithm combines various techniques from previous work including approximate barycentric spanner computation and reduction from contextual linear bandits to linear bandits. Meanwhile, the algorithm is also computationally efficiently.\n\nSTRENGTHS:\n- The algorithm intuition and application of prior techniques of approximate barycentric spanner computation and reduction from contextual linear bandits to linear bandits are considered to be novel.\n- The algorithm is shown to be computationally efficient while maintaining moderately good regret and low number of batches.\n\nWEAKNESSES:\nAlthough it is proven that the proposed algorithm has good computational complexity, it will be better if we can empirically see its computational efficiency is superior over other algorithms listed in Table 1. In particular, it seems that implementing the contextual version of the algorithm (Algorithm 3) is not very trivial. Can we see some empirical comparisons of regret and computational efficiency among Algorithm 3 and some other algorithms listed in Table 1?\n\n### Suggestions on Writing\n- It can be better if specific lemmas are referred when the arguments in main context use any results from the appendix. For example, the arguments in line 248 and 277.\n- It may be better to define $\\Delta(a)$ clearly after defining $L_z(a)$ in line 273.\n- For self-containedness, it will be better to have a brief explanation of how Algorithm 2 works.\n\nQUESTIONS:\n- Why the regret bound based on the minimum gap is not applicable for contextual linear bandits?\n- A better G-optimal design approximation can be obtained when $\\mathcal{A}_t$ is discrete and finite. Will this give better theoretical guarantee?\n\nLIMITATIONS:\nThe theoretical limitations are well-addressed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies batched online learning for linear bandits and linear contextual bandits in large action space. It shows that the learner can use $O(\\log \\log T)$ batches (in each of which the sampling distribution over actions remain the same) to achieve near-optimal regret. Compared to previous work, the main contribution is to introduce an oracle efficient algorithm which only requires linear optimization over the action set, and thus can handle large action space. The regret bound is a $d$ factor worse than the tight bound.\n\nSTRENGTHS:\n- The paper studies an important question for linear (contextual) bandits where the action space is large and the number of policy switch is limited. The proposed algorithm is natural and easy to understand. The regret bound (just a $d$ factor sub-optimal) is the best we can hope for under the computational constraint. \n- There are some other algorithmic ideas developed, which could be of independent interest. For example, adding a little ball around the origin to make the action space full rank, and the \"action set shaping\" technique that balance exploitation and exploration when finding the solution of the approximate G-optimal design.\n\nWEAKNESSES:\n- Though I'm not entirely familiar with the literature, I feel that the contribution might be a bit incremental. The reason is that in batched online learning, ways to achieve optimal performance using $O(\\log \\log T)$ batches has already been established. The main contribution of this work is about using barycentric spanner to efficiently explore. The related technique has been extensively studied in [Zhu et al.], though not for the batched setting. I can see a lot of similarity between the LW-ArgMax in this paper and the IGW-ArgMax in [Zhu et al.], but no technical comparison is provided. The LWS procedure is also closely related to the ReweightedSpanner procedure in [Zhu et al.]. To be fair, the two are still slightly different because this paper uses inverse square gap weighting while theirs is inverse gap weighting, but I think there still needs to be some comparison. \n\n- The proposed algorithm operate on the extended action set $\\mathcal{A}'$, which seems to require a linear optimization oracle over $\\mathcal{A}'$. I don't think it is explained what if the learner only has an oracle for the original action set $\\mathcal{A}$. It is unclear whether an oracle for $\\mathcal{A}'$ can be straightforwardly constructed from that for $\\mathcal{A}$. \n\n- Similarly, in the contextual setting, the algorithm operates over a very artificial action set $g^{(m)}([\\theta]_q)$. It is totally unclear how to leverage the linear optimization oracle for the original action set $\\mathcal{A}_t$ to implement the algorithm. \n\n- Since a inverse square gap weighting is used, maybe it's possible to show an refined bound like in [Lattimore and Szepesvari, 2016]. Now only a $\\Delta_{\\min}$ dependent instance-dependent bound is shown, which can be highly sub-optimal. \n\n\n[Zhu et al., 2022] Zhu, Foster, Langford, Mineiro. Contextual bandits with large action spaces: made practical.  \n\n[Lattimore and Szepesvari, 2016] The End of Optimism? An Asymptotic Analysis of Finite-Armed Linear Bandits\n\nQUESTIONS:\nSee the weakness section.\n\nLIMITATIONS:\nThere is no societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-12"}