Title: Generalized Linear Bandits with Limited Adaptivity
Venue: NeurIPS 2024
Accepted: yes
Keywords: Contextual Bandits

Generalized Linear Contextual Bandits

Logistic Bandits

Optimal Design

Limited Adaptivity

Batched Bandits
Abstract: We study the generalized linear contextual bandit problem within the constraints of limited adaptivity.  In this paper, we present two algorithms, B-GLinCB and RS-GLinCB, that address, respectively, two prevalent limited adaptivity settings. Given a budget $M$ on the number of policy updates, in the first setting, the algorithm needs to decide upfront $M$ rounds at which it will update its policy, while in the second setting it can adaptively perform $M$ policy updates during its course. For the first setting, we design an algorithm B-GLinCB, that incurs $\tilde{O}(\sqrt{T})$ regret when $M = \Omega( \log{\log T} )$ and the arm feature vectors are generated stochastically. For the second setting, we design an algorithm RS-GLinCB that updates its policy $\tilde{O}(\log^2 T)$ times and achieves a regret of $\tilde{O}(\sqrt{T})$ even when the arm feature vectors are adversarially generated. Notably, in these bounds, we manage to eliminate the dependence on a key instance dependent parameter $\kappa$, that captures non-linearity of the underlying reward model. Our novel approach for removing this dependence for generalized linear contextual bandits might be of independent interest.

Generalized Linear Bandits with Limited Adaptivity

Abstract

We study the generalized linear contextual bandit problem within the constraints of limited adaptivity. In this paper, we present two algorithms, B-GLinCB and RS-GLinCB , that address, respectively, two prevalent limited adaptivity settings. Given a budget M on the number of policy updates, in the first setting, the algorithm needs to decide upfront M rounds at which it will update its policy, while in the second setting it can adaptively perform M policy updates during its course. For the first setting, we design an algorithm B-GLinCB , that incurs ˜ O ( √ T ) regret when M = Ω(log log T ) and the arm feature vectors are generated stochastically. For the second setting, we design an algorithm RS-GLinCB that updates its policy ˜ O (log 2 T ) times and achieves a regret of ˜ O ( √ T ) even when the arm feature vectors are adversarially generated. Notably, in these bounds, we manage to eliminate the dependence on a key instance dependent parameter κ , that captures non-linearity of the underlying reward model. Our novel approach for removing this dependence for generalized linear contextual bandits might be of independent interest.

1 Introduction

Contextual Bandits ( CB ) is an archetypal framework that models sequential decision making in time-varying environments. In this framework, the algorithm (decision maker) is presented, in each round, with a set of arms (represented as d -dimensional feature vectors), and it needs to decide which arm to play. Once an arm is played, a reward corresponding to the played arm is accrued. The regret of the round is defined as the difference between the maximum reward possible in that round and the reward of the played arm. The goal is to design a policy for selecting arms that minimizes cumulative regret (referred to as the regret of the algorithm) over a specified number of rounds, T . In the last few decades, much progress has been made in designing algorithms for special classes of reward models, e.g. linear model [3, 4, 1, 16], logistic model [6, 2, 7, 28] and generalized linear models [8, 19].

However, despite this progress, there is a key challenge that prevents deployment of CB algorithms in the real world. Practical situations often allow for very limited adaptivity, i.e., do not allow CB algorithms to update their policy at all rounds. For example, in clinical trials [10], each trial involves administering medical treatments to a cohort of patients, with medical outcomes observed and collected for the entire cohort at the conclusion of the trial. This data is then used to design the treatment for the next phase of the trial. Similarly, in online advertising [25] and recommendations [18], updating the policy after every iteration during deployment is often infeasible due to infrastructural constraints. A recent line of work [23, 22, 11, 12, 21, 9, 26] tries to address this limitation

by developing algorithms that try to minimize cumulative regret while ensuring that only a limited number of policy updates occur. Across these works, two settings (called M1 and M2 from here onwards) of limited adaptivity have been popular. Both M1, M2 provide a budget M to the algorithm, determining the number of times it can update its policy. In M1 [23, 13], the algorithm is required to decide upfront a sub-sequence of M rounds where policy updates will occur. While in M2 [1, 23]), the algorithm is allowed to adaptively decide (during its course) when to update its policy.

Limited adaptivity algorithms were recently proposed for the CB problem with linear reward models under the M1 setting [23, 11], and optimal regret guarantees were obtained when the arm feature vectors were stochastically generated. Similarly, in their seminal work on linear bandits, [1] developed algorithms for the M2 setting and proved optimal regret guarantees with no restrictions on the arm vectors. While these results provide tight regret guarantees for linear reward models, extending them to generalized linear models is quite a challenge. Straightforward extensions lead to sub-optimal regret with a significantly worse dependence on an instance dependent parameter κ (See Section 2 for definition) that captures non-linearity of the problem instance. In fact, to the best of our knowledge, developing optimal algorithms for the CB problem with generalized linear reward models under the limited adaptivity settings M1 , M2 , is an open research question. This is the main focus of our work. We make the following contributions.

1.1 Our Contributions

We propose B-GLinCB , an algorithm that solves the CB problem for bounded (almost surely) generalized linear reward models (Definition 2.1) under the M1 setting of limited adaptivity. We prove that, when the arm feature vectors are generated stochastically, the regret of B-GLinCB at the end of T rounds is ˜ O ( √ T ) , when M = Ω(log log T ) . When M = O (log log T ) , we prove an ˜ O ( T 2 M -1 / (2 M -2) ) regret guarantee. While the algorithm bears a slight resemblance to the one in [23], direct utilization of their key techniques (distributional optimal design) results in a regret guarantee that scales linearly with the instance dependent non-linearity κ . On the other hand, the leading terms in our regret guarantee for B-GLinCB have no dependence on κ . To achieve this, we make novel modifications to the key technique of distributional optimal design in [23]. Along with this, the rounds for policy updates are also chosen more carefully (in a κ dependent fashion), leading to a stronger regret guarantee.

We propose RS-GLinCB , an algorithm that solves the CB problem for bounded (almost surely) generalized linear reward models (Definition 2.1) under the M2 setting of limited adaptivity. RS-GLinCB builds on a similar algorithm in [1] by adding a novel context-dependent criterion for determining if a policy update is needed. This new criterion allows us to prove optimal regret guarantee ( ˜ O ( √ T ) ) with only O (log 2 T ) updates to the policy. It is quite crucial for the generalized linear reward settings since, without it, the resultant regret guarantees have a linear dependence on κ . √

Our work also resolves a conjecture in [17] by proving an optimal ( ˜ O ( T ) ) regret guarantee (for the CB problem with logistic reward model) that does not depend polynomially on S (the known upper bound on the size of the model parameters, i.e. ∥ θ ⋆ ∥ ≤ S , See Section 2) 3 . RS-GLinCB is, to our knowledge, the first CB algorithm for generalized linear reward models that is both computationally efficient (amortized O (log T ) computation per round) and incurs optimal regret. We also perform experiments in Section 5 that validate its superiority both in terms of regret and computational efficiency in comparison to other baseline algorithms proposed in [14] and [6].

1.2 Important Remarks on Contributions and Comparison with Prior Work

Remark 1.1 ( κ -independence ) . For both B-GLinCB and RS-GLinCB , our regret guarantees are free of κ (in their leading term), an instance-dependent parameter that can be exponential in the size of the unknown parameter vector, i.e., ∥ θ ⋆ ∥ (See Section 2 for definition). Our contribution in this regard is two-fold. Not only do we prove κ -independent regret guarantees under the limited adaptivity constraint, we also characterize a broad class of generalized linear reward models for which a κ -independent regret guarantee can be achieved. Specifically, our results imply that the CB problem with generalized linear reward models originally proposed in [8] and subsequently studied in literature [19, 14, 24] admits a κ -independent regret.

Remark 1.2 ( Computational efficiency ) . Efforts to reduce the total time complexity to be linear in T have been active in the CB literature with generealized linear rewards models. For e.g., [14] recently devised computationally efficient algorithms but they suffer from regret dependence on κ . Optimal ( κ -independent) guarantees were recently achieved for logistic reward models [6, 2], and the algorithms were subsequently made computationally efficient in [7, 28]. However, the techniques involved rely heavily on the structure of the logistic model and do not easily extend to more general models. To the best of our knowledge, ours is the first work that achieves optimal κ -independent regret guarantees for bounded generalized linear reward models while remaining computationally efficient 4 .

Remark 1.3 ( Self Concordance of bounded GLMs ) . In order to prove κ -independent regret guarantees, we prove a key result about self concordance of bounded (almost surely) generalized linear models (Definition 2.1) in Lemma 2.2. This result was postulated in [8] for GLMs (with the same definition as ours), but no proof was provided. While [6, 7] partially tackled this issue for logistic reward models 5 , in our work, we prove self concordance for much more general generalized linear models.

2 Notations and Preliminaries

Notations: A policy π is a function that maps any given arm set X to a probability distribution over the same set, i.e., π ( X ) ∈ ∆( X ) , where ∆( X ) is the probability simplex supported on X . We will denote matrices in bold upper case (e.g. M ). ∥ x ∥ denotes the ℓ 2 norm of vector x . We write ∥ x ∥ M to denote √ x ⊤ M x for a positive semi-definite matrix M and vector x . For any two real numbers a and b , we denote by a ∧ b the minimum of a and b . Throughout, ˜ O ( · ) denotes big-O notation but suppresses log factors in all relevant parameters. For m,n ∈ N with m<n , we denote the set { 1 , . . . , n } by [ n ] and { m,... , n } by [ m,n ] .

Definition 2.1 (GLM) . A Generalized Linear Model or GLM with parameter vector θ ⋆ ∈ R d is a real valued random variable r that belongs to the exponential family with density function

Function b (called the log-partition function) is assumed to be twice differentiable and ˙ b is assumed to be monotone. Further, we assume that r ∈ [0 , R ] almost surely for some known R ∈ R .

Important properties of GLMs such as E [ r | x ] = ˙ b ( ⟨ x, θ ⋆ ⟩ ) and variance V [ r | x ] = ¨ b ( ⟨ x, θ ⋆ ⟩ ) are detailed in Appendix C. We define the link function µ as µ ( ⟨ x, θ ∗ ⟩ ) := E [ r | x ] . Thus, µ is also monotone. We now present a key Lemma on GLMs (see Appendix C for details) that enables us to achieve optimal regret guarantees for our algorithms designed in Sections 3 and 4.

Lemma 2.2 (Self-Concordance of GLMs) . For any GLM supported on [0 , R ] almost surely, the link function µ ( · ) satisfies | ¨ µ ( z ) | ≤ R ˙ µ ( z ) , for all z ∈ R .

Next we describe the two CB problems with GLM rewards that we address in this paper. Let T ∈ N be the total number of rounds. At round t ∈ [ T ] , we receive an arm set X t ⊂ R d , with number of arms K = |X t | and must select an arm x t ∈ X t . Following this, we receive a reward r t sampled from the GLM distribution P ( r | x t ) with unknown θ ∗ .

Problem 1: In this problem we assume that at each round t , the set of arms X t ⊂ R d is drawn from an unknown distribution D . Further, we assume the constraints of limited adaptivity setting M1 , i.e., the algorithm is given a budget M ∈ N and needs to decide upfront the M rounds at which it will update its policy. Let supp ( D ) denote the support of distribution D . We want to design an algorithm that minimizes the expected cumulative regret given as

Here, the expectation is taken over the randomness of the algorithm, the distribution of rewards r t , and the distribution of the arm set D .

Problem 2: In this problem we do not make any assumptions on the arm feature vectors, i.e., the arm vectors can be adversarially chosen. However, we assume the constraints of limited adaptivity setting M2 , i.e., the algorithm is given a budget M ∈ N and needs to adaptively decide the M rounds at which it will update its policy (during its course). We want to design an algorithm that minimizes the cumulative regret given as

Finally, for both the problems, the Maximum Likelihood Estimator (MLE) of θ ∗ can be calculated by minimizing the sum of the log-losses. The log-loss is defined for any given arm x , its (stochastic) reward r and vector θ ∈ R d (as the estimator of the true unknown θ ∗ ) as follows: ℓ ( θ, x, r ) := -r · ⟨ x, θ ⟩ + ∫ ⟨ x,θ ⟩ 0 µ ( z ) dz . After t rounds, the MLE ̂ θ is computed as ̂ θ = arg min θ ∑ t s =1 ℓ ( θ, x s , r s ) .

2.1 Instance Dependent Non-Linearity Parameters

As in prior works [6, 7], we define instance dependent parameters that capture non-linearity of the underlying instance and critically impact our algorithm design. The performance of Algorithm 1 ( B-GLinCB ) that solves Problem 1 , can be quantified using three such parameters that are defined using the derivative of the link function ˙ µ ( · ) . Specifically, for any arm set X , write optimal arm x ∗ = arg max x ∈X µ ( ⟨ x, θ ∗ ⟩ ) and define,

Remark 2.3 . These quantities feature prominently in our regret analysis of Algorithm 1. In particular, the dominant term in our regret bound scales as O ( √ T/κ ∗ ) . We also note that ̂ κ ≥ κ ∗ ; in fact, for specific distributions D , the gap between them can be significant. Hence, we also provide a regret upper bound of O ( √ T/ ̂ κ ) . In this latter case, however, we incur a worse dependence on d . Section 3 provides a quantified form of this trade-off.

Algorithm 2 ( RS-GLinCB ) that solves Problem 2 , requires another such non-linearity parameter κ 6 , defined as,

We note that, here, κ is defined considering the parameter vector θ ∗ in contrast to prior work on logistic bandits [7], where its definition involved a maximization over all vectors θ with ∥ θ ∥ ≤ S (known upper bound of ∥ θ ⋆ ∥ ). Hence, κ as defined here is potentially much smaller and can lead to lower regret, compared to prior works. Standard to the CB literature with GLM rewards, we will assume that tight upper bounds on these parameters is known to the algorithms.

Assumption 2.4. We make the following additional assumptions which are standard for the CB problem with linear or GLM reward models.

For every round t ∈ [ T ] , and each arm x ∈ X t , ∥ x ∥ ≤ 1 .

Let θ ∗ be the unknown parameter of the GLM reward, then ∥ θ ∗ ∥ ≤ S for a known constant S .

2.2 Optimal Design Policies

G-optimal Design Given an arm set X , the G-OPTIMAL DESIGN policy π G is the solution of the following optimization problem: arg min λ ∈ ∆( X ) max x ∈X || x || 2 U ( λ ) -1 , where U ( λ ) = E x ∼ λ [ xx T ] . Now consider the following optimization problem, also known as the D-optimal design problem: max λ ∈ ∆( X ) log Det ( U ( λ )) . This is a concave maximization problem as opposed to the G-optimal design which is non-convex. We have the following equivalence theorem due to Kiefer and Wolfowitz [15]:

Algorithm 1 B-GLinCB : Batched Generalized Linear Bandits Algorithm Input: Number of batches M and horizon of play T . 1: Initialize batches T 1 , . . . , T M , as defined in equation (3), and set λ := 20 Rd log T . 2: for rounds t ∈ T 1 do 3: Observe arm set X t , sample arm x t ∼ π G ( X t ) , and observe reward r t . 4: Compute ̂ θ w = arg min θ ∑ s ∈T 1 ℓ ( θ, x s , r s ) and matrix V = λ I + ∑ t ∈T 1 x t x T t . 5: Initialize policy π 1 as G-OPTIMAL DESIGN. 6: for batches k = 2 to M do 7: for each round t ∈ T k do 8: Observe arm set X t . 9: for j = 1 to k -1 do 10: Update arm set X t ←X t \ { x ∈ X t : UCB j ( x ) < max y ∈X t LCB j ( y ) } . 11: Scale X t , as in (4), to obtain ˜ X t . , then sample x t ∼ π k -1 ( ˜ X t ) . 12: Equally divide T k into two sets A and B . 13: Define H k = λ I + ∑ t ∈A ˙ µ ( ⟨ x, ̂ θ w ⟩ ) β ( x t ) x t x T t , and ̂ θ k = arg min θ ∑ s ∈A ℓ ( θ, x s , r s ) . 14: Compute DISTRIBUTIONAL OPTIMAL DESIGN policy π k using the arm sets {X t } t ∈B .

Lemma 2.5 (Keifer-Wolfowitz) . Let X ⊂ R d be any set of arms and W G be the expected design matrix, defined as W G := E x ∼ π G ( X ) [ xx T ] , with π G ( X ) as the solution to the D-optimal design problem. Then, π G ( X ) also solves the G-optimal design problem, and for all x ∈ X , ∥ x ∥ 2 W -1 G ≤ d .

Distributional optimal design Notably, the upper bound on ∥ x ∥ W -1 G specified in Lemma 2.5 holds only for the arms x in X . When the arm set X t varies from round to round, securing a guarantee analogous to Lemma 2.5 is generally challenging. Nonetheless, when the arm sets X t are drawn from a distribution, it is possible to extend the guarantee, albeit with a worse dependence on d ; see Section A.5 in Appendix A. Improving this dependence motivates the need of studying DISTRIBUTIONAL OPTIMAL DESIGN and towards this we utilize the results of [23].

The distributional optimal design policy is defined using a collection of tuples M = { ( p i , M i ) : p 1 , . . . , p n ≥ 0 and ∑ i p i = 1 } , wherein each M i is a d × d positive semi-definite matrix and n ≤ 4 d log d . The collection M is detailed next. Let softmax α ( { s 1 , . . . , s k } ) denote the probability distribution where the i th element is sampled with probability s α i ∑ k j =1 s α j . For a specific M =

{ ( p i , M i ) } n i =1 , and each i ∈ [ n ] write π M i ( X ) = softmax α ( {∥ x ∥ 2 M i : x ∈ X} ) . Finally, with π G as the G-OPTIMAL DESIGN policy (Section 2.2), we define the DISTRIBUTIONAL OPTIMAL DESIGN policy π as

Given a collection of arm sets {X 1 , . . . , X s } (called core set ) sampled from the distribution D , we utilize Algorithm 2 of [23] to find the collection M ; see Algorithm 4 of [23]. Overall, the computed M induces a policy π that upholds the following guarantee.

Lemma 2.6 (Theorem 5, [23]) . Let π be the DISTRIBUTIONAL OPTIMAL DESIGN policy that has been learnt from s independent samples X 1 , . . . X s ∼ D . Also, let W denote the expected design matrix, W = E X∼D [ E x ∼ π ( X ) [ xx T | X ]] . Then,

3 B-GLinCB

In this section, we present B-GLinCB (Algorithm 1) that solves Problem 1 described in Section 2, which enforces constraints of limited adaptivity setting M1 . Given limited adaptivity budget M ∈ N , our algorithm first computes the batch length for each of the M batches (i.e., determine rounds where

the policy remains constant). We build upon the batch length construction in [9]; however, the first batch is chosen to be κ dependent which crucially helps in removing κ from the leading term in the regret. 7

Batch Lengths : For each batch k ∈ [ M ] , let T k denote all the rounds within the k th batch. We will refer to the first batch T 1 as the warm-up batch. The batch lengths τ k := |T k | , k ∈ [ M ] are calculated as follows:

where γ := 30 RS √ d log T 8 and α = T 1 2(1 -2 -M +1 ) if M ≤ log log T and α = 2 √ T otherwise.

During the warm-up batch (Lines 2, 3), the algorithm follows the G-OPTIMAL DESIGN policy, π G . At the end of the warm-up batch (Line 4), the algorithm computes the Maximum Likelihood Estimate (MLE), ̂ θ w , of θ ∗ 9 , and design matrix V := ∑ t ∈T 1 x t x T t + λ I , with parameter λ = 20 Rd log T .

Now, for each batch k ≥ 2 and every round t ∈ T k , the algorithm updates X t by eliminating arms from it using the confidence bounds (see Equation (7)) computed in the previous batches (Line 10). The algorithm next computes ˜ X t , a scaled version of X t , as follows, with β ( x ) define in equation (5),

Finally, we use the distributional optimal design policy π k , on the scaled arm set ˜ X t , to sample the next arm (Line 11). At the end of every batch, we equally divide the batch T k into two sets A and B . We use samples from A to compute the estimator ̂ θ k and the scaled design matrix H k . The rounds in B are used to compute π k +1 , the distributional optimal design policy for the next batch. It is important to note while the policy π k is utilized in each round (Line 11) to draw arms, it is updated (to π k +1 ) only at the end of the batch. Hence, conforming to setting M1 , the algorithm updates the selection policy at M rounds that were decided upfront.

Confidence Bounds: The scaled design matrix H k , an estimator of the Hessian, is computed at the end of each batch k ∈ 2 , . . . , M (Line 13):

where A is the first half of T k . Using this, we define the upper and lower confidence bounds ( UCB k and LCB k ) computed at the end of batch T k :

Remark 3.1 . The confidence bounds employed by the algorithm exhibit a significant distinction between the first batch and subsequent batches. While the first batch's bounds are influenced by the parameter κ , subsequent batches utilize κ -independent bounds. This difference arises from the use of the standard design matrix V in the first batch and a scaled design matrix H k (equation 5) in later batches, leveraging the self-concordance property of GLM rewards to achieve κ -independence. Notably, the first batch's confidence bounds influence the scaling factor β ( x ) in later batches, creating a trade-off (addressed in the regret analysis in Appendix A) where an inaccurate estimate of ̂ θ w can exponentially increase the scaling factor and confidence bounds.

In Theorem 3.2 and Corollary 3.3, we present our regret guarantee for B-GLinCB . Detailed proofs for both are provided in Appendix A. The computational efficiency of B-GLinCB is discussed in Appendix D.

Algorithm 2 RS-GLinCB : Rarely-Switching GLM Bandit Algorithm

Theorem 3.2. Algorithm 1 ( B-GLinCB ) incurs regret R T ≤ ( R 1 + R 2 ) log log T 10 , where

Corollary 3.3. When the number of batches M ≥ log log T , Algorithm 1 achieves a regret bound of

Remark 3.4 . Scaling the arm set (as in (4)) for optimal design is a crucial aspect of our algorithm, allowing us to obtain tight estimates of ˙ µ ( ⟨ x, θ ∗ ⟩ ) (see Lemma A.10). This result relies on multiple novel ideas and techniques, including self-concordance for GLMs, matrix concentration, Bernsteintype concentration for the canonical exponential family (Lemma A.1), and application of distributional optimal design on scaled arm set.

Remark 3.5 . The κ -dependent batch construction is a crucial feature of our algorithm, enabling effective estimation of ˙ µ ( ⟨ x, θ ∗ ⟩ ) at the end of the first batch. Since the first batch incurs regret linear in its length, achieving a κ -independent guarantee requires the first batch to be o ( √ T ) . We demonstrate that choosing τ 1 = O ( T 1 3 ) is sufficient for this purpose (see Appendix A).

4 RS-GLinCB

In this section we present RS-GLinCB (Algorithm 2) that solves Problem 2 described in Section 2, which enforces constraints of limited adaptivity setting M2 . This algorithm incorporates a novel switching criterion (Line 4), extending the determinant-doubling approach of [1]. Additionally, we introduce an arm-elimination step (Line 12) to obtain tighter regret guarantees. Throughout this section, we set λ = d log( T/δ ) /R 2 and γ = 25 RS √ d log ( T/δ ) .

At round t , on receiving an arm set X t , RS-GLinCB first checks the Switching Criterion I (Line 4). This criterion checks whether for any arm x ∈ X t the quantity ∥ x ∥ V -1 is greater than a carefully chosen κ -dependent threshold. Here V is the design matrix corresponding to all arms that have been played in the rounds in T o ( := the set of rounds preceding round t , where Switching Criterion I was triggered). Under this criterion the arm that maximizes ∥ x ∥ V -1 is played (call this arm x t ) and the corresponding reward is obtained. Subsequently in Line 6, the set T o is updated to include t ; the

design matrix V is updated as V ← V + x t x ⊺ t ; and the scaled design matrix H t +1 is set to H t . The MLE is computed (Line 7) based on the data in the rounds in T o to obtain ̂ θ o .

When Switching Criterion I is not triggered, the algorithm first checks (Line 9) the Switching Criterion II, that is whether the determinant of the scaled design matrix H t has become more than double of that of H τ (where τ is the last round before t when Switching Criterion II was triggered). If Switching Criterion II is triggered at round t , then in Line 10, the algorithm sets τ ← t and recomputes the MLE over all the past rounds except those in T o to obtain ˜ θ . Then ˜ θ is projected into an ellipsoid around ̂ θ o to obtain the estimate ̂ θ τ via the following optimization problem 11 ,

Here H ( θ ) := ∑ s ∈T o ˙ µ ( ⟨ x s , θ ⟩ ) x s x T s . After checking Switching Criterion II, the algorithm performs an arm elimination step (Line 12) based on the parameter estimate ̂ θ o as follows: for every arm x ∈ X t , we compute UCB o ( x ) = ⟨ x, ̂ θ o ⟩ + γ √ κ ∥ x ∥ V -1 and LCB o ( x ) = ⟨ x, ̂ θ o ⟩ -γ √ κ ∥ x ∥ V -1 12 . Then, X t is updated by eliminating from it the arms with UCB o ( · ) less than the highest LCB o ( · ) . For arms in the reduced arm set X t , RS-GLinCB computes the index UCB ( x, H τ , ̂ θ τ ) := ⟨ x, ̂ θ τ ⟩ + 150 ∥ x ∥ H -1 τ √ d log ( T/δ ) , and plays the arm x t with the highest index (Line 13). After observing the subsequent reward r t , the algorithm updates the scaled design matrix H t (Line 14) as follows: H t +1 ← H t +(˙ µ ( ⟨ x t , ̂ θ o ⟩ ) /e ) x t x ⊺ t . With this, the round t ends and the algorithm moves to the next round. Next, in Lemma 4.1 and Theorem 4.2 we present the guarantees on number of policy updates and regret, respectively, for RS-GLinCB . Detailed proofs for both are provided in Appendix B.

Lemma 4.1. RS-GLinCB (Algorithm 2), during its entire execution, updates its policy at most O ( R 4 S 2 κd 2 log 2 ( T/δ )) times.

Theorem 4.2. Given δ ∈ (0 , 1) , with probability ≥ 1 -δ , the regret of RS-GLinCB (Algorithm 2) satisfies R T = O ( d √∑ t ∈ [ T ] ˙ µ ( ⟨ x ∗ t , θ ∗ ⟩ ) log ( RT/δ ) + κd 2 R 5 S 2 log 2 ( T/δ ) ) .

Remark 4.3 . Switching Criterion I is essential in delivering tight regret guarantees in the non-linear setting. Unlike existing literature [7], which relies on warm-up rounds based on observed rewards (hence heavily dependent on reward models), RS-GLinCB presents a context-dependent criterion that implicitly checks whether the estimate ˙ µ ( ⟨ x, ̂ θ o ⟩ ) is within a constant factor of ˙ µ ( ⟨ x, θ ∗ ⟩ ) (see Lemmas B.3 and B.4). We show that the number of times Switching Criterion I is triggered is only O ( κd 2 log 2 ( T )) (see Lemma B.11), hence incurring a small regret in these rounds.

Remark 4.4 . Unlike [1], our determinant-doubling Switching Criterion II uses the scaled design matrix H t instead of the unscaled version (similar to V ). The matrix H t , estimating the Hessian of the log-loss, is crucial for achieving optimal regret. This modification is crucial in extending algorithms satisfying limited adaptivity setting M2 for the CB problem with a linear reward model to more general GLM reward models.

Remark 4.5 . The feasible set for the optimization stated in 8 is an ellipsoid around ̂ θ o , which contains θ ∗ with high probability. Deviating from existing literature on GLM Bandits which projects the estimate into the ball set of radius S ( { θ : ∥ θ ∥ ≤ S } ), our projection step leads to tighter regret guarantees; notably, the leading √ T term is free of parameters S (and R ). This resolves the conjecture made in [17] regarding the possibility of obtaining S -free regret in the √ T term in logistic bandits. Remark 4.6 . The regret guarantees of the logistic bandit algorithms in [2, 17] have a second-order term that is minimum of an arm-geometry dependent quantity (see Theorem 3 of [17]) and a κ -dependent term similar to our regret guarantee. Although our analysis is not able to accommodate this arm-geometry dependent quantity, we underscore that our algorithm is computationally efficient while the above works are not. In fact, to the best of our knowledge, the other known efficient algorithms for logistic bandits [7, 28] also do not achieve the arm-geometry dependent regret term. It can be interesting to design an efficient algorithm that is able to achieve the same guarantees in the second-order regret term as in [2, 17].

Cumulative Regretvs # of Rounds

CumulativeRegret vs # of Rounds

3000

RS-GLinUCB (This work)

RS-GLinUCB (This work)

ECOLog (Faury etal.2022)

2000

GLM-UCB (Filippi et al.2010)

Regret

2500

Cumulative Regret

1500

2000

CumulativeP

1500

1000

1000

500

500

0

0

1000

2000

3000

4000

5000

#of Rounds

# of Rounds

Logistic with T = 20000, (avg of 20 trials)

Probit with T = 20000, (avg of 20 trials)

45

35

RS-GLinUCB (This work)

RS-GLinUCB (This work)

40

ECOLog (Faury et al.2022)

0E

Execution Time (s)

35

Execution Time (s)

5

30  25 20 15  10

7

15

st

5

0

T0

LowKappa

High Kappa

Low Kappa

High Kappa

5 Experiments

We tested the practicality of our algorithm RS-GLinCB against various baselines for logistic and generalized linear bandits. For these experiments, we adjusted the Switching Criterion I threshold constant in RS-GLinCB to 0 . 01 and used data from both Switching Criteria (I and II) rounds to estimate ˜ θ . These modifications do not affect the overall efficiency as ˜ θ is calculated only O (log( T )) times. The experiment code is available at https://github.com/nirjhar-das/GLBandit_Limited_Adaptivity .

Logistic. We compared RS-GLinCB against ECOLog [7] and GLOC [14], the only algorithms with overall time complexity ˜ O ( T ) for this setting. The dimension was set to d = 5 , number of arms per round to K = 20 , and θ ∗ was sampled from a d -dimensional sphere of radius S = 5 . Arms were sampled uniformly from the d -dimensional unit ball. We ran simulations for T = 20 , 000 rounds, repeating them 10 times. RS-GLinCB showed the smallest regret with a flattened regret curve, as seen in Fig. 1 (top-left).

Probit. For the probit reward model, we compared RS-GLinCB against GLOC and GLM-UCB [8]. The dimension was set to d = 5 and number of arms per round to K = 20 . θ ∗ was sampled from a d -dimensional sphere of radius S = 3 . Arm features were generated similarly as in the logistic bandit simulation. We ran simulations for T = 5 , 000 rounds, repeating them 10 times. RS-GLinCB outperformed both baselines, as shown in Fig. 1 (top-right).

Comparing Execution Times. We compared the execution times of RS-GLinCB and ECOLog . We created two logistic bandit instances with d = 5 and K = 20 , and different κ values. We ran both algorithms for T = 20 , 000 rounds, repeating each run 20 times. For low κ , RS-GLinCB took about one-fifth of the time of ECOLog , and for high κ , slightly more than one-third, as seen in Fig. 1 (left-bottom). This demonstrates that RS-GLinCB has a significantly lower computational overhead compared to ECOLog . We also compared the execution times of RS-GLinCB and GLOC under the probit reward model, creating two bandit instances with d = 5 and K = 20 , but with differing κ . We ran both algorithms for T = 20 , 000 rounds, repeating each run 20 times. The result is shown in Fig. 1 (bottom-right). We observe that for low κ , RS-GLinCB takes less than half time of GLOC while for high κ , it takes about two-third time of GLOC . A more detailed discussion of these experiments is provided in Appendix D.

6 Conclusion and Future Work

The Contextual Bandit problem with GLM rewards is a ubiquitous framework for studying online decision-making with non-linear rewards. We study this problem with a focus on limited adaptivity. In particular, we design algorithms B-GLinCB and RS-GLinCB that obtain optimal regret guarantees for two prevalant limited adaptivity settings M1 and M2 respectively. A key feature of our guarantees are that their leading terms are independent of an instance dependent parameter κ that captures non-linearity. To the best of our knowledge, our paper provides the first algorithms for the CB problem with GLM rewards under limited adaptivity (and otherwise) that achieve κ -independent regret. The regret guarantee of RS-GLinCB , not only aligns with the best-known guarantees for Logistic Bandits but enhances them by removing the dependence on S (upper bound on ∥ θ ∗ ∥ ) in the leading term of the regret and therefore resolves a conjecture in [17]. The batch learning algorithm B-GLinCB , for M = Ω(log (log T )) , achieves a regret of ˜ O ( dRS ( √ d/ ̂ κ ∧ √ 1 /κ ∗ ) √ T ) . We believe that the dependence on d along with the ̂ κ term is not tight and improving the dependence is a relevant direction for future work.

Acknowledgments and Disclosure of Funding

Siddharth Barman gratefully acknowledges the support of the Walmart Center for Tech Excellence (CSR WMGT-23-0001) and a SERB Core research grant (CRG/2021/006165).

References

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24, 2011.

Marc Abeille, Louis Faury, and Clément Calauzènes. Instance-wise minimax-optimal algorithms for logistic bandits. In International Conference on Artificial Intelligence and Statistics , pages 3691-3699. PMLR, 2021.

Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research , 3(Nov):397-422, 2002.

Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandits with linear payoff functions. In Geoffrey Gordon, David Dunson, and Miroslav Dudík, editors, Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , volume 15 of Proceedings of Machine Learning Research , pages 208-214, Fort Lauderdale, FL, USA, 11-13 Apr 2011. PMLR.

Louis Faury. Variance-sensitive confidence intervals for parametric and offline bandits . Theses, Institut Polytechnique de Paris, Oct 2021.

Louis Faury, Marc Abeille, Clément Calauzènes, and Olivier Fercoq. Improved optimistic algorithms for logistic bandits. In International Conference on Machine Learning , pages 3052-3060. PMLR, 2020.

Louis Faury, Marc Abeille, Kwang-Sung Jun, and Clément Calauzènes. Jointly efficient and optimal algorithms for logistic bandits. In International Conference on Artificial Intelligence and Statistics , pages 546-580. PMLR, 2022.

Sarah Filippi, Olivier Cappe, Aurélien Garivier, and Csaba Szepesvári. Parametric bandits: The generalized linear case. Advances in neural information processing systems , 23, 2010.

Zijun Gao, Yanjun Han, Zhimei Ren, and Zhengqing Zhou. Batched multi-armed bandits problem. Advances in Neural Information Processing Systems , 32, 2019.

International Stroke Trial Collaborative Group et al. The international stroke trial (ist): a randomised trial of aspirin, subcutaneous heparin, both, or neither among 19 435 patients with acute ischaemic stroke. The Lancet , 349(9065):1569-1581, 1997.

Yanjun Han, Zhengqing Zhou, Zhengyuan Zhou, Jose Blanchet, Peter W Glynn, and Yinyu Ye. Sequential batch learning in finite-action linear contextual bandits. arXiv preprint arXiv:2004.06321 , 2020.

Osama Hanna, Lin Yang, and Christina Fragouli. Learning from distributed users in contextual linear bandits without sharing the context. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Osama Hanna, Lin Yang, and Christina Fragouli. Efficient batched algorithm for contextual linear bandits with large action space via soft elimination. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Kwang-Sung Jun, Aniruddha Bhargava, Robert Nowak, and Rebecca Willett. Scalable generalized linear bandits: Online computation and hashing. Advances in Neural Information Processing Systems , 30, 2017.

Jack Kiefer and Jacob Wolfowitz. The equivalence of two extremum problems. Canadian Journal of Mathematics , 12:363-366, 1960.

Tor Lattimore and Csaba Szepesvári. Bandit Algorithms . Cambridge University Press, 2020.

Junghyun Lee, Se-Young Yun, and Kwang-Sung Jun. Improved regret bounds of (multinomial) logistic bandits via regret-to-confidence-set conversion. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics , pages 4474-4482. PMLR, 02-04 May 2024.

Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web , pages 661-670, 2010.

Lihong Li, Yu Lu, and Dengyong Zhou. Provably optimal algorithms for generalized linear contextual bandits. In International Conference on Machine Learning , pages 2071-2080. PMLR, 2017.

Blake Mason, Kwang-Sung Jun, and Lalit Jain. An experimental design approach for regret minimization in logistic bandits. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 7736-7743, 2022.

Vianney Perchet, Philippe Rigollet, Sylvain Chassang, and Erik Snowberg. Batched bandit problems. In Peter Grünwald, Elad Hazan, and Satyen Kale, editors, Proceedings of The 28th Conference on Learning Theory , volume 40 of Proceedings of Machine Learning Research , pages 1456-1456, Paris, France, 03-06 Jul 2015. PMLR.

Zhimei Ren and Zhengyuan Zhou. Dynamic batch learning in high-dimensional sparse linear contextual bandits. Management Science , 70(2):1315-1342, 2024.

Yufei Ruan, Jiaqi Yang, and Yuan Zhou. Linear bandits with limited adaptivity and learning distributional optimal design. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing , pages 74-87, 2021.

Yoan Russac, Olivier Capp'e, and Aurélien Garivier. Algorithms for non-stationary generalized linear bandits. ArXiv , abs/2003.10113, 2020.

Eric M Schwartz, Eric T Bradlow, and Peter S Fader. Customer acquisition via display advertising using multi-armed bandit experiments. Marketing Science , 36(4):500-522, 2017.

David Simchi-Levi and Yunzong Xu. Bypassing the monster: A faster and simpler optimal algorithm for contextual bandits under realizability. Mathematics of Operations Research , 47(3):1904-1931, 2022.

Joel A Tropp et al. An introduction to matrix concentration inequalities. Foundations and Trends® in Machine Learning , 8(1-2):1-230, 2015.

Yu-Jie Zhang and Masashi Sugiyama. Online (multinomial) logistic bandit: Improved regret and constant computation cost. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Generalized Linear Bandits with Limited Adaptivity Appendix

Table of Contents

A Regret Analysis of B-GLinCB

A.1 Additional Notation

We write c to denote absolute constant(s) that appears throughout our analysis. Our analysis also utilizes the following function

Note that γ ( λ ) is a 'parameterized' version of γ (which was defined in section 3). In our proof, we present the arguments using this parameterized version. A direct minimization of the above expression in terms of λ would not suffice since we need λ to be sufficiently large for certain matrix concentration lemmas to hold (see Section A.5). However, later we show that setting λ equal to c Rd log T leads to the desired bounds.

We use ˜ x to denote the scaled versions of the arms (see Line 11 of the algorithm); in particular,

Furthermore, to capture the non-linearity of the problem, we introduce the term ϕ ( λ ) :

Recall that the scaled data matrix H k (for each batch k ) was computed using ̂ θ w as follows

Following the definition of H k and using the true vector θ ∗ we define

We will show that H k dominates H ∗ k with high probability. Furthermore, we assume that the MLE estimator θ ∗ obtained by minimizing the log-loss objective always satisfies ∥ ∥ ∥ ̂ θ ∥ ∥ ∥ ≤ S . In case, that's not true, one can use the non-convex projection described in Appendix E. The projected vector satisfies the same guarantees as described in the subsequent lemmas up to a multiplicative factor of 2 . Hence, the assumption ∥ ∥ ∥ ̂ θ ∥ ∥ ∥ ≤ S is non-limiting.

A.2 Concentration Inequalities and Confidence Intervals

Lemma A.1 (Bernstein's Inequality) . Let X 1 , . . . , X n be a sequence of independent random variables with | X t -E [ X t ] | ≤ b . Also, let sum S := ∑ n t =1 ( X t -E [ X t ]) and v := ∑ m t =1 Var [ X t ] . Then, for any δ ∈ [0 , 1] , we have

Lemma A.2. Let X = { x 1 , x 2 , . . . , x s } ∈ R d be a set of vectors with ∥ x t ∥ ≤ 1 , for all t ∈ [ s ] , and let scalar λ ≥ 0 . Also, let r 1 , r 2 , . . . , r s ∈ [0 , R ] be independent random variables distributed by the canonical exponential family; in particular, E [ r s ] = µ ( ⟨ x s , θ ∗ ⟩ ) for θ ∗ ∈ R d . Further, let ̂ θ = arg min θ ∑ t s =1 ℓ ( θ, x s , r s ) be the maximum likelihood estimator of θ ∗ and let matrix

Then, with probability at least than 1 -1 T 2 , the following inequality holds

Proof. We first define the following quantities

Using Lemma C.2 we have

Hence we write

Now by the optimality condition on ̂ θ we have ∑ s j =1 µ ( ⟨ x j , ̂ θ ⟩ ) x j = ∑ s j =1 r j x j (see equation (3) [8]). Hence, we write

Let B denote the unit ball in R d . We can write

We construct an ε -net for the unit ball, denoted as C ε . For any y ∈ B , we define y ε := arg min b ∈C ε ∥ b -y ∥ 2 . We can now write

Rearranging, we obtain

Next, we use Lemma A.3 (stated below) with δ = T 2 |C ε | and union bound over all vectors in C ε . We also observe that |C ε | ≤ ( 2 ε ) d . Substituting ϵ = 1 / 2 and using Lemma A.3, we obtain that the following holds with probability greater than 1 -1 T 2 ,

Substituting in equations (13), we get the desired inequality in the lemma statement.

Lemma A.3. Let y be a fixed vector with ∥ y ∥ ≤ 1 . Then, with the notation stated in Lemma A.2, the following inequality holds with probability at least 1 -δ

Proof. Let us denote the j th term of the sum as Z j . Note that each random variable Z j has variance Var ( Z j ) = ˙ µ ( ⟨ x j , θ ∗ ⟩ ) ( y T H ∗-1 / 2 x j ) 2 . Hence, we have

Moreover, each Z j is at most R √ λ (since ∥ x j ∥ ≤ 1 , H ∗ ⪰ λ I and r ∈ [0 , R ] ). Now applying Lemma A.1, we have

Corollary A.4. Let x 1 , x 2 , . . . , x τ be the sequence of arms pulled during the warm-up batch and let ̂ θ w be the estimator of θ ∗ computed at the end of the batch. Then, for any vector x and λ ≥ 0 the following bound holds with probability greater than 1 -1 T 2

Proof. This result is derived directly from Lemma A.2 and the definition of γ ( λ ) (see 9). By applying the lemma, we obtain

Considering the definition of κ , we have ˙ µ ( ⟨ x , θ ∗ ⟩ ) ≥ 1 κ . This implies that H ∗ ⪰ 1 κ V 13 which in turn leads to the inequality ∥ x ∥ H ∗-1 ≤ √ κ ∥ x ∥ V -1 .

A.3 Preliminary Lemmas

Lemma A.5. For each batch k ≥ 2 and the scaled data matrix H k computed at the end of batch, the following bound holds with probability at least 1 -1 T 2 :

Proof. If the event stated in Lemma A.4 holds,

From Lemma C.2, we apply the multiplicative bound on ˙ µ to obtain

Via Corollary A.4 we have |⟨ x, ̂ θ w ⟩ - ⟨ x, θ ∗ ⟩| ≤ √ κ ∥ x ∥ V -1 γ ( λ ) . Additionally, given that ∥ ∥ ∥ ̂ θ ∥ ∥ ∥ , ∥ θ ∗ ∥ ≤ S and ∥ x ∥ ≤ 1 we also have |⟨ x, ̂ θ w ⟩ - ⟨ x, θ ∗ ⟩| ≤ 2 S . Hence, we write

Substituting these results into the definitions of H k and H ∗ k proves the lemma statement.

Claim A.6. The Algorithm 1 runs for at most log log T batches.

Proof. When M ≤ log log T then the claim trivially holds. When M ≥ log log T +1 , we define the length of the second batch, τ 2 , as 2 √ T . The length of the M th batch is

Corollary A.7. Let ̂ θ k be the estimator of θ ∗ calculated at the end of the k th batch. Then for any vector x the following holds with probability greater than 1 -log log T T 2 for every batch k ≥ 2 .

Proof. This result is a direct consequence of Lemma A.2 and the definition of γ ( λ ) (see 9). According to the lemma, we have

Using Lemma A.5, we can further bound ∥ x ∥ H ∗ k -1 ≤ ∥ x ∥ H k -1 . Finally, a union bound over all batches and considering the fact that there are at most log log T batches (Claim A.6) we establish the corollary's claim.

Claim A.8. For any x ∈ [0 , M ] the following holds

Proof. The claim follows from the convexity of e x .

Lemma A.9. Let x ∈ X be the selected in any round of batch k ≥ 2 in the algorithm, and let x ∗ be the optimal arm in the arm set X , i.e., x ∗ = arg max x ∈X µ ( ⟨ x, θ ∗ ⟩ ) . With probability greater than 1 -log log T T 2 , the following inequality holds-

Proof. We begin by applying Taylor's theorem, which yields the following for some z between ⟨ x, θ ∗ ⟩ and ⟨ x ∗ , θ ∗ ⟩

We now invoke Lemmas C.2 and A.4 to obtain

Similarly, we also have

Further, we can simplify each term in equation (15) as

Finally, we substitute the above bound in (15) to obtain

For Phase k , the distribution of the remaining arms after the elimination step ( X in line 10 of the Algorithm 1) is represented as D k .

Lemma A.10. During any round in batch k of Algorithm 1, and for an absolute constant c , we have

Proof. The proof here invokes Lemma A.9. We begin by noting that

We also have

Substituting the above bounds in Lemma A.9 we obtained the stated inequality. This completes the proof.

A.4 Proof of Theorem 3.2

We trivially upper bound the regret incurred during the warm-up batch as τ 1 R ; recall that R denotes the upper bound on the rewards and τ 1 denotes the length of the first (warm-up) batch; see equation (5)).

For each batch k and an absolute constant c , Lemma A.10 gives us

Since there are at most log log T batches, we can upper bound the regret as

Setting τ 1 = ( ϕ ( λ ) d 2 α R ) 2 / 3 we get

Now with the choice of λ = 20 dR log T , we have

Substituting α = T 1 2 ( 1 -2 -M ) and ϕ ( λ ) = √ κ e 3 S γ 2 we get

A.5 Optimal Design Guarantees

In this section, we study the optimal design policies utilized in different batches of the algorithm. Specifically, π G denotes the G-OPTIMAL DESIGN policy applied during the warm-up batch, while π k refers to the DISTRIBUTIONAL OPTIMAL DESIGN policy calculated at the end of batch k ( and used in the ( k +1) th batch). Recall that the distribution of the remaining arms after the elimination step ( X in line 10 of the Algorithm) is represented as D k . We define expected design matrices for each policy:

Recall, for all batches starting from the second batch ( k ≥ 2 ), we employ the scaled arm set, denoted as ˜ X , for learning and action selection under the DISTRIBUTIONAL OPTIMAL DESIGN policy. However, during the initial warm-up batch, we utilize the original, unscaled arm set.

Claim A.11. The following holds for any positive semidefinite matrix A and any batch k -

This is due to the fact that the set of surviving arms in batch k is always a smaller set than the previous batches.

Lemma A.12 (Lemma 4 [23]) . The expected data matrix W G satisfies. We have

Lemma A.13 (Theorem 5 [23]) . Let the DISTRIBUTIONAL OPTIMAL DESIGN π which has been learnt from s independent samples X 1 , . . . X s ∼ D and let W denote the expected data matrix, W = E X∼D [ E x ∼ π ( X ) [ xx T |X ]] . We have

Lemma A.14. Under the notation of Lemma A.13, we have

Proof. Recall that the DISTRIBUTIONAL OPTIMAL DESIGN policy samples according to the π G policy with half probability. Hence, we have

Therefore,

Lemma A.15 (Matrix Chernoff [27, 23]) . Let x 1 , x 3 , . . . , x n ∼ D be vectors, with ∥ x t ∥ ≤ 1 , then we have

Corollary A.16. In Algorithm 1 the warm-up matrix V , with λ ≥ 16 log ( Td ) , satisfies the following with probability greater than 1 -1 T 2 .

Similarly H k , with λ ≥ 6 log ( Td ) satisfies the following for each batch k ≥ 2 with probability greater than 1 -1 T 2

Proof. The results for both V and H k are obtained directly by applying Lemma A.15 with ε = log( T ) τ 1 and ε = log( T ) τ k , respectively.

We note that the analysis of [23] gives an optimal guarantee (in expectation) on ∥ x ∥ V -1 , but not on ∥ x ∥ 2 V -1 . We obtain such a bound here and use it in the analysis.

Lemma A.17. The following holds with probability greater than 1 -1 T 2

We also have that for sufficiently large T ≳ O ( d 32 (log 2 T ) 2 ) , the following holds with probability greater than 1 -log log T T

Proof. First, we note from Corollary A.16 that the following holds with high probability

We obtain the first two inequalities, (18) and (19), by a direct use of Corollary A.16. For (20) we note that for every phase we have at least O ( √ T ) samples for learning the DISTRIBUTIONAL OPTIMAL DESIGN policy (for any M). Since, T ≥ d 32 log 2 T 2 the event stated in Lemma A.13 holds with probability greater than 1 -1 T 2 .

B Regret Analysis of RS-GLinCB

Recall T o denotes the set of rounds when switching criterion I is satisfied. We write τ o to denote the size of the set τ o = |T o | . We define the following (scaled) data matrix

We will specify the regularizer λ later. We also define

Below, we state the main concentration bound used in the proof.

Lemma B.1 (Theorem 1 of [6]) . Let {F t } ∞ t =1 be a filtration. Let { x t } ∞ t =1 be a stochastic process in B 1 ( d ) such that x t is F t measurable. Let { η t } ∞ t =1 be a martingale difference sequence such that η t is F t measurable. Furthermore, assume we have | η t | ≤ 1 almost surely, and denote σ 2 t = V [ η t |F t ] . Let λ > 0 and for any t ≥ 1 define:

Then, for any δ ∈ (0 , 1] ,

B.1 Confidence Sets for Switching Criterion I rounds

Lemma B.2. At any round t , let ̂ θ o be the maximum likelihood estimate calculated using set of rewards observed in the rounds T o . With probability at least 1 -δ we have

Proof. Let us define the matrix G w = ∑ s ∈T o α ( x, θ ∗ , ̂ θ o ) x s x ⊺ s + λ I . First, we note that by selfconcordance property of µ (Lemma C.2), G w ⪰ 1 1+2 RS H ∗ w . Hence,

Since ̂ θ o is the maximum likelihood estimate, by optimality condition, we have the following relation: ∑ s ∈T o µ ( ⟨ x s , ̂ θ o ⟩ ) x s + λ ̂ θ o = ∑ s ∈T o r s x s . Substituting this above, we get

where η s := ( r s -µ ( ⟨ θ ∗ , x s ⟩ )) .

We will now apply Lemma B.1 η s scaled by R . First note that ∥ ∥ ∑ s ∈T o η s x s ∥ ∥ ( H ∗ w ) -1 = ∥ ∥ ∑ s ∈T o η s R x s ∥ ∥ ( R 2 H ∗ w ) -1 which, in turn ensures that the noise variable is upper bounded by 1 .

Applying B.1 we get

B.2 Confidence Sets for non-Switching Criterion I rounds

We define E w be the event defined in Lemma B.2, that is, E w = {∥ ̂ θ o -θ ∗ ∥ H ∗ w ≤ γ } . Lemma B.3. If in round t the switching criteria I is not satisfied and the event E w

Proof.

Recall that H t is defined in line 14 of Algorithm 2. Further, we define

.

Corollary B.4. Under event E w , H t ⪯ H ∗ t ⪯ e 2 H t

Proof. For a given s ∈ [ t -1] \ T o , let ̂ θ s o denote the value of ̂ θ o in that round. Then, for all x ∈ X s , by Lemma C.2,

Applying lemma B.3, gives e -1 ˙ µ ( ⟨ x, ̂ θ s o ⟩ ) ≤ ˙ µ ( ⟨ x, θ ∗ ⟩ ) ≤ e 1 ˙ µ ( ⟨ x, ̂ θ s o ⟩ ) . Thus,

holds, we have

Recall that τ is the round when the Switching Criterion II (Line 9) is satisfied. Now we define the following quantities:

Moreover, recall the following definition ̂ θ τ :

Lemma B.5. Under event E w , ∥ ∥ ∥ ̂ θ τ -θ ∗ ∥ ∥ ∥ V ≤ 2 γ √ κ

Proof. First, we observe from Lemma B.2 that ∥ ∥ ∥ ̂ θ o -θ ∗ ∥ ∥ ∥ H ∗ w ≤ γ . Using V ⪯ κ H ∗ w , we can write ∥ ∥ ∥ ̂ θ o -θ ∗ ∥ ∥ ∥ V ≤ γ √ κ . This implies that θ ∗ ∈ Θ . Now, ̂ θ τ ∈ Θ by virtue of being a feasible solution to the optimization in (21). Thus,

Lemma B.6. Let δ ∈ (0 , 1) . Then, under event E w , with probability 1 -δ , ∥ ∥ ∥ ̂ θ τ -θ ∗ ∥ ∥ ∥ H ∗ τ ≤ β .

Proof. We have for all rounds s ∈ [ τ -1] \ T o ,

Also note that θ ∗ ∈ Θ . Hence,

Let the event in lemma B.6 be denoted by E τ , or in other words, E τ = {∥ ̂ θ τ -θ ∗ ∥ H ∗ τ ≤ β } .

B.3 Bounding the instantaneous regret

In this subsection, we will only consider rounds t ∈ [ T ] which does not satisfy Switching Criterion I. Let x t ∈ X t be the played arm defined via line 13 Algorithm 2. Further, let x ∗ t ∈ X t be the best available arm in that round.

Corollary B.7. Under the event E τ , for all x ∈ X t , we have, |⟨ x, ̂ θ τ -θ ∗ ⟩| ≤ β ∥ x ∥ H -1 τ .

Proof.

Lemma B.8. Under event E τ , ⟨ x ∗ t -x t , θ ∗ ⟩ ≤ 2 √ 2 β ∥ x t ∥ H ∗-1 t .

Proof.

Lemma B.9. The arm set X ′ t obtained after eliminating arms from X t (line 12 Algorithm 2), under event E w , satisfies: (a) x ∗ t ∈ X t , (b) ⟨ x ∗ t -x t , θ ∗ ⟩ ≤ 4 R

Proof. Suppose x ′ = arg max x ∈X t LCB o ( x ) . Now, we have, for all x ∈ X t ,

Thus, UCB o ( x ∗ t ) = ⟨ x ∗ t , ̂ θ o ⟩ + γ √ κ ∥ x ∗ t ∥ V -1 ≥ ⟨ x ∗ t , θ ∗ ⟩ ≥ ⟨ x ′ , θ ∗ ⟩ ≥ ⟨ x ′ , ̂ θ o ⟩ -γ √ κ ∥ x ′ ∥ V -1 = LCB o ( x ′ ) , where the second inequality is due to optimality of x ∗ t . Hence, x ∗ t is not eliminated, implying x ∗ t ∈ X ′ t . This completes the proof of (a).

Since x t is also in X ′ t (by definition),

Again, using the fact that ⟨ x ∗ t , ̂ θ o ⟩ ≥ ⟨ x ∗ t , θ ∗ ⟩ -γ √ κ ∥ x ∗ t ∥ V -1 and ⟨ x t , ̂ θ o ⟩ ≤ ⟨ x t , θ ∗ ⟩ + γ √ κ ∥ x t ∥ V -1 , we obtain,

Finally, since Switching Criterion I is not satisfied in this round, ∥ x ∥ V -1 < 1 Rγ √ κ for all x ∈ X t . Plugging this above,

.

B.4 Proof of Theorem 4.2

In this subsection, we complete the proof of the regret bound of RS-GLinCB (Algorithm 2). We first restate Theoreom 4.2 and then prove it. For every round t ∈ [ T ] , we use x t ∈ X t to denote the arm played by the algorithm and x ∗ t to denote the best available arm in that round.

Theorem B.10 (Theorem 4.2) . Given δ ∈ (0 , 1) , with probability ≥ 1 -δ , the regret of RS-GLinCB (Algorithm 2) satisfies R T = O ( d √∑ t ∈ [ T ] ˙ µ ( ⟨ x ∗ t , θ ∗ ⟩ ) log ( RT/δ ) + κd 2 R 5 S 2 log 2 ( T/δ ) ) .

Proof. Firstly, we will assume throughout the proof that E w ∩E τ holds, which happens with probability at least 1 -δ . Thus, regret of Algorithm 2 is upper bounded as:

Now, let R 1 ( T ) = ∑ t ∈ [ T ] \T o ˙ µ ( z ) ⟨ x ∗ t -x t , θ ∗ ⟩ . Hereon, we will slightly abuse notation H τ to denote the H τ matrix last updated before time t for each time step t ∈ [ T ] . This will be clear from the context as we will only use H τ term-wise. With this, we upper bound R 1 ( T ) as follows:

Putting things back,

B.5 Bounding number of policy updates: Proof of Lemma 4.1

We first obtain a bound on the number of rounds when Switching Criterion I is satisfied. Then we restate Lemma 4.1 and present its proof. Here, we use T o to denote the collection of all rounds till T for which Switching Criterion I is satisfied.

Lemma B.11. Algorithm 2, during its entire execution, satisfies the Switching Criterion I at most 2 dR 2 κγ 2 log ( T/δ ) times.

Proof. Recall that Switching Criterion I (Line 4) is satisfied, when ∥ x ∥ 2 V -1 > 1 / ( R 2 κγ 2 ) for some x ∈ X t . Let V m be the sequence of V matrices (line 6 of Algorithm 2) for m ∈ T o . That is, V 1 = λ I , V m = ∑ s ∈ [ m -1] ∩T o x s x ⊺ s + λ I . In these rounds, by Line 5 of Algorithm 2, we have that the arm played x t is such that x t = arg max x ∈X t ∥ x ∥ V -1 t . Therefore,

Furthermore, by the Elliptic Potential Lemma (Lemma B.12) we have

Combining (23) and (22) we have

Lemma (4.1) . Algorithm 2, during its entire execution, updates its policy at most O ( R 4 S 2 κd 2 log 2 ( T/δ )) times.

Proof. Note that in Algorithm 2, policy changes happen only in the rounds when either of the Switching Criteria are triggered. The number of times Switching Criterion I is triggered is bounded by Lemma B.11. On the other hand, the number of times Switching Criterion II is triggered is equal to the number of times determinant of H t doubles, which is bounded by Lemma B.15. Thus in total, the number of policy changes in Algorithm 2 is upper bounded by 2 dR 2 κγ 2 log( T/δ ) + c d log( T ) .

B.6 Some Useful Lemmas

Lemma B.12 (Elliptic Potential Lemma (Lemma 10 [1])) . Let x 1 , x 2 , . . . x t be a sequence of vectors in R d and let ∥ x s ∥ 2 ≤ L for all s ∈ [ t ] . Further, let V s = ∑ s -1 m =1 x m x ⊺ m + λ I . Suppose λ ≥ L 2 . Then,

Lemma B.13 (Lemma 12 of [1]) . Let A ⪰ B ≻ 0 . Then

̸

Lemma B.14 (Lemma 10 of [1]) . Let { x s } t s =1 be a set of vectors. Define the sequence { V s } t s =1 as V 1 = λ I , V s +1 = V s + x s x ⊺ s for s ∈ [ t -1] . Further, let ∥ x s ∥ 2 ≤ L ∀ s ∈ [ t ] . Then,

Lemma B.15. Let { x s } t s =1 be a set of vectors. Define the sequence { V s } t s =1 as V 1 = λ I , V s +1 = V s + x s x ⊺ s for s ∈ [ t -1] . Further, let ∥ x s ∥ 2 ≤ L ∀ s ∈ [ t ] . Define the set { 1 = τ 1 , τ 2 . . . τ m = t } such that: det( V τ i +1 ) ≥ 2 det( V τ i ) but det( V τ i +1 -1 ) < 2 det( V τ i ) for i ∈ { 2 , . . . m -1 } . Then, the number of time doubling happens, i.e. , m , is at most O ( d log( t )) .

Proof. By Lemma B.14, det( V t ) ≤ ( λ + tL 2 /d ) d . But we have that from definition of τ i 's

Thus, 2 m -2 λ d ≤ ( λ + tL 2 /d ) d which implies that

Hence, m ≤ O ( d log( t )) .

C Useful Properties of GLMs

Recall that a Generalized Linear Model is characterized by a canonical exponential family, i.e. , the random variable r has density function p z ( r ) = exp( rz -b ( z ) + c ( r )) , with parameter z , log-partition function b ( · ) , and a function c . Further, ˙ b ( z ) = µ ( z ) is also called the link function.

Hereon, we will assume that the random variable has a bounded non-negative support, i.e. , r ∈ [0 , R ] almost surely. Now, we state the following key Lemmas on GLMs

Lemma C.1 (Self-Concordance for GLMs) . For distributions in the exponential family the function µ ( · ) satisfies that for all z ∈ R , | ¨ µ ( z ) | ≤ R ˙ µ ( z ) .

Proof. Indeed,

As a consequence, we have the following simple modification of the self-concordance results of [6]. Lemma C.2. For an exponential distribution with log-partition function b ( · ) , for all z 1 , z 2 ∈ R , letting µ ( z ) := ˙ b ( z ) , following holds:

Proof. Without loss of generality, assume that z 2 ≥ z 1 . Note that by property of integration ∫ b a f ( x ) dx = ∫ a b f ( b + a -x ) dx , α ( z 1 , z 2 ) = α ( z 2 , z 1 ) . Now, by proposition C.1, and the fact that ¨ µ ( z ) = ... b ( z ) , we have for any v ∈ R and z ≥ z 1 ,

Putting z = z 2 establishes 26. To show 25, we further set z = z 1 + u ( z 2 -z 1 ) for u ∈ [0 , 1] , (note that z ≥ z 1 ) and integrate on u ,

Next, we use the fact that for x > 0 , e -x ≤ (1 + x ) -1 which on rearranging gives (1 -e -x ) /x ≥ 1 / (1 + x ) . Applying this inequality to the LHS above finishes the proof. Note that similar exercise can be repeated with z 2 ≤ z 1 to get the same result for z 2 .

For 27, we have, by application of 26, ˙ µ ( z 1 + v ( z 2 -z 1 )) ≥ ˙ µ ( z 1 ) exp( R | v ( z 2 -z 1 ) | ) . Therefore,

Next we state some nice properties of the GLM family that is the key in deriving Lemma C.1.

Lemma C.3 (Properties of GLMs) . For any random variable r that is distributed by a canonical exponential family, we have

Proof. 1. Indeed, since p z ( r ) is a probability distribution, ∫ r p z ( r ) dr = 1 which in turn implies that b ( z ) = log (∫ r exp( rz + c ( r )) dr ) . Thus, taking derivative,

Let f ( z ) := ∫ r r exp( rz + c ( r )) dr . Thus, ˙ b ( z ) = exp( -b ( z )) f ( z ) . Taking derivative on both sides,

Again let f ( z ) := ∫ r r 2 exp( rz + c ( r )) dr . Thus, ¨ b ( z ) = -˙ b ( z ) 2 +exp( -b ( z )) f ( z ) . Taking derivative on both sides,

Now, let us expand E [( r -E [ r ]) 3 ] .

Corollary C.4. For all exponential family, b ( · ) is a convex function.

Proof. Indeed, note that ¨ b ( z ) = V [ r ] which is always non-negative. Thus, ¨ b ( z ) ≥ 0 implying that b ( · ) is convex.

Remark C.5 . In [5] Section 1.4.1, the author claims that if the GLM parameter z lies in a bounded set, then the GLM is self-concordant, i.e., | ¨ µ ( z ) | ≤ a ˙ µ ( z ) , for some appropriate constant a over this bounded set. Thereafter the author notes that the techniques developed in [5] guarantees κ -free regret rates (in √ T term) for such GLMs (i.e., all GLMs with bounded parameter). However, the claim regarding self-concordance of GLMs is not true in general. There are classes of GLMs whose parameters may be restricted in a bounded set, but for them no constant a exists. One such example is the exponential distribution. The link function µ for exponential distribution is given as µ ( z ) = -1 z . If we allow z to lie in the set ( -c, 0) for some positive c , then we have µ ( z ) strictly increasing (satisfying our assumption on monotonicity of µ , thus a valid example). However, for this GLM,

Note that ¨ µ ( z ) is positive for the assumed support of z . Suppose this GLM is self-concordant, then we must have some positive constant a such that

Simplifying, we obtain the following relation:

However, since z ∈ ( -c, 0) , we have lim z → 0 -2 z →∞ . Hence, no constant a is possible. By this counterexample it can be seen that bounded parameter set is not enough to guarantee self-concordance of GLMs. In this work, we give a characterization of self-concordance of GLMs with bounded support of the random variable. It will be interesting to understand a complete characterization of self-concordance of GLMs.

D Computational Cost

Consider a log-loss minimization oracle that returns the unconstrained MLE for a given GLM class with a computational complexity of C opt · n , when the log-loss is computed over n data points. Let the maximum number of arms available every round be K . Furher, let the computational cost of an oracle that solves the non-convex optimization 8 be NC opt .

Computational Cost of B-GLinCB : In the B-GLinCB algorithm, we employ the log-loss oracle at the end of each batch. The estimator ̂ θ calculated at the end of a batch of length τ incurs a computational cost of C opt τ . Furthermore, this oracle is invoked for a maximum of M ≤ log log T batches. Additionally, the computation of the distributional optimal design at the end of each batch is efficient in d (poly ( d ) ). Moreover, in every round, the algorithm solves the D/G -Optimal Design problem (requiring O ( d log d ) computation) and runs elimination based on prior (at most log log T ) phases. Hence, the amortized cost per round of B-GLinCB is O ( K log log T + d log d + C opt ) .

Computational Cost of RS-GLinCB : In the RS-GLinCB algorithm, the estimator ̂ θ o is computed each time Switching Criterion I is triggered. Additionally, during rounds when Switching Criterion I is not triggered, the estimator ̂ θ τ is computed a maximum of O (log( T )) times. These computations involve utilizing both the log-loss oracle and the non-convex projection oracle. Furthermore, in each non-Switching Criterion I round, the algorithm executes an elimination step. This yields an amortized time complexity of O ( C opt log ( T ) + NC opt log 2 ( T ) + K ) per round.

Performance in Practice : As evident from Fig. 1, RS-GLinCB has much better computational performance in practice. We ran all the experiments on an Azure Data Science VM equipped with AMD EPYC 7V13 64-Core Processor (clock speed of 2.45 GHz) and Linux Ubuntu 20.04 LTS operating system. It was ensured that no other application processes were running while we tested the performance. We implemented and tested our code in Python, and measured the execution times using time.time() command. We allowed no operations for 10 seconds after every run to let the CPU temperature come back to normal, in case the execution heats up the CPU, thereby causing subsequent runs to slow down.

Comparison with ECOLog [7] shows that execution time for RS-GLinCB is significantly smaller. We posit that this is because RS-GLinCB solves a large convex optimization problem but less frequently, resulting into smaller overhead at the implementation level, while ECOLog solves a smaller convex optimization problem, but does so every round. On an implementation level, this translates into more function calls and computation. Further, we observe that with increasing κ , the execution time of RS-GLinCB increases, which is in accordance with Lemma 4.1 that quantifies the number of policy switches as an increasing function of κ .

While comparing with GLOC [14], we observe that RS-GLinCB performs better than GLOC in both high and low κ regimes. Since GLOC runs an online convex optimization (online Newton step) algorithm to generate its confidence sets, the time taken by GLOC is nearly constant with changing κ . On the other hand, in accordance with Lemma 4.1, the computational cost of RS-GLinCB increases with κ . However, after a few initial rounds, when neither of the switching criteria are triggered, RS-GLinCB does not need to solve any computationally intensive optimization problem, hence these rounds execute very fast. In practice, with typical data distribution, RS-GLinCB reaches this stage much before what the worst-case guarantees show, hence we see it perform better than GLOC .

E Projection

We describe the projection step used in Algorithms 1 and 2. We present arguments similar to the ones made in Appendix B.3 of [6]. We write

Recall, H ∗ = H ( θ ∗ ) . Let ̂ θ be the MLE estimator of θ ∗ calculated after the sequence arm pulls x 1 , x 2 , . . . , x t . Let r 1 , r 2 , . . . , r t be the corresponding observed rewards. We project ̂ θ to a set Θ by solving the following optimization problem

Lemma E.1. Using the notations described above, if θ ∗ ∈ Θ and max i ∈ [ t ] |⟨ x i , ˜ θ -θ ∗ ⟩| ≤ c/R , then we have

Proof. First, we note that by self-concordance property of µ (lemma C.2), for any s ∈ [ t ] ,

Similarly, we have α ( x s , ˜ θ, θ ∗ ) ≥ ˙ µ ( ⟨ x s , ˜ θ ⟩ ) 1+ c .

Let us define the matrix G = ∑ s ∈ [ t ] α ( x, ˜ θ, θ ∗ ) x s x ⊺ s . Using the above fact, we obtain the relation: G ⪰ 1 1+ c H ∗ and G ⪰ 1 1+ c H ( ˜ θ ) . Also define the vector g ( θ ) = ∑ s ∈ [ t ] µ ( ⟨ θ, x s ⟩ ) x s . Now,

Let g ( θ ) = ∑ t s =1 ˙ µ ( ⟨ x s , θ ⟩ ) x s for any θ . Therefore, we have,

E.1 Convex Relaxation

The optimization problem in (28) is a non-convex optimization problem and therefore it is not clear what is the computational complexity of the problem. However, it is possible to substitute this optimization problem with a convex one, whose computational complexity can be better tractable. The process is similar to the one detailed in [2, section 6]. Here we briefly outline the steps.

Let L t ( θ ) = ∑ t s =1 ℓ ( θ, x s , r s ) and ˘ θ be defined as follows:

Note that when the set Θ is a convex set, then the above optimization problem is convex by property of the log-likelihood function of GLMs. Hence it can be solved efficiently. With this projected ˘ θ , we have the following guarantee:

Lemma E.2. Suppose ∥ ∥ ∥ g ( ̂ θ ) -g ( θ ∗ ) ∥ ∥ ∥ H ∗-1 ≤ γ and λ = γ/R . If θ ∗ ∈ Θ and max i ∈ [ t ] |⟨ x i , ˘ θ -θ ∗ ⟩| ≤ c/R , then we have

Proof. First we note that by self-concordance property of µ , for any s ∈ [ t ] ,

Let us define ˜ G ( θ ∗ , θ ) := ∑ t s =1 ˜ α ( x s , θ ∗ , θ ) x s x ⊺ s . Using the above fact, we obtain ˜ G ( θ ∗ , θ ) ⪰ 1 2+ c H ∗ .

Wenowfollow closely the proof outlined in Appendix B.3 of [2] with minor changes. By second-order Taylor's expansion, for any θ ∈ R d , we can write

Taking absolute value on both sides, and substituting θ = ˘ θ ,

Recall that ̂ θ is the unconstrained MLE, therefore ∇L t ( ̂ θ ) = 0 . By a similar Taylor expansion as above and some algebraic manipulations (see Appendix B.3 of [2]), we have, for θ ∗ .

We also have, by definition of ˘ θ , whenever θ ∗ ∈ Θ , L t ( ˘ θ ) ≤ L t ( θ ∗ ) , therefore we have L t ( ˘ θ ) -L t ( ̂ θ ) ≤ L t ( θ ∗ ) -L t ( ̂ θ ) ≤ 2 R 3 Sγ Thus, we have,

Using the inequality that for some x 2 ≤ bx + c = ⇒ x ≤ b + √ c , we have,

NeurIPS Paper Checklist

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See Section 1

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Conclusion (Section 6)

Answer: [Yes]

Justification: See Conclusion (Section 6)

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All details are stated in Section 2 and the proofs are given in details in the Appendices A, B, C.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Detailed discussion is done in Section 5 and Appendix D. Code can be accessed at https://github.com/nirjhar-das/GLBandit_Limited_Adaptivity .

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Full code with documentation is provided at https://github.com/ nirjhar-das/GLBandit_Limited_Adaptivity .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Execution Time plots have errorbars, but Regret plots do not as regret plots were observed to get cluttered although the variation was not significant.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: There are no potential harms caused by the research process as it is mainly theoretical and the experiments are all in simulation. There might be Societal Impact of the algorithms developed here when applied in practical decision-making settings. The study of this beyond the scope of current work.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: The problem studied is of a theoretical interest and we do not foresee any negative societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal

impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The data and models are only toy data/fully simulated and therefore of no real impact.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All credits regarding code are given in the README.md file at https: //github.com/nirjhar-das/GLBandit_Limited_Adaptivity .

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: All details are provided in the README.md of our codebase https:// github.com/nirjhar-das/GLBandit_Limited_Adaptivity .

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work is mainly theoretical with only simple simulated experiments.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our work does not involve human participants.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the generalized linear contextual bandit problem under limited adaptivity constraints. In a setting (M1) where the times for updating the agent's policy are predetermined, the first proposed algorithm B-GLinCB divides the entire timeline into batches, updating the policy at the end of each batch. B-GLinCB explores using a G-optimal design policy in the first batch and adjusts the arm set in subsequent batches based on estimated MLE parameters using samples obtained in the first batch. The algorithm guarantees a $\\tilde{O}(\\sqrt{T})$ regret when the number of policy updates is $\\Omega(\\log \\log T)$, independent of the problem-dependent instance $\\kappa$. In a setting (M2) where the agent can adaptively decide when to update the policy, the RS-GLinCB algorithm is proposed. RS-GLinCB uses two criteria to alter action selection: the first criterion allows a tighter estimation of the true parameter's derivative, and the second criterion achieves a $\\kappa$-independent regret bound. The theoretical results of the proposed algorithms were supported through comparisons with baseline algorithms in logistic bandit settings.\n\nSTRENGTHS:\n- The limited adaptivity constraint discussed in the proposed paper is crucial for many real-world decision-making problems. The authors extend results from linear reward models to non-linear reward models, with the proposed algorithms guaranteeing $\\tilde{O}(\\sqrt{T})$ regret under specific conditions.\n- Notably, the leading term of the regret bound for the proposed algorithms is independent of problem-dependent instances. This is, to my knowledge, the first result showing kappa-independent regret bounds for GLM reward models, excluding logistic bandits.\n- The proposed algorithms are computationally efficient since the number of samples used to estimate the reward parameter does not increase over time.\n\nWEAKNESSES:\n- Although the proposed algorithm achieves a $\\kappa$-independent regret bound, it requires prior knowledge of $\\kappa$. The MNL contextual bandit (Perivier & Goyal, 2022) achieved a $\\kappa$-independent regret bound without needing information about $\\kappa$, which might be useful here.\n- The proposed algorithm is computationally efficient concerning time $t$, but there is no explanation of its dependency on the dimension $d$. It lacks details on the computational complexity needed for calculating the optimal design and distributional optimal design at each time step.\n\n* * *\nPerivier & Goyal. \"Dynamic pricing and assortment under a contextual MNL demand.\" Advances in Neural Information Processing Systems 35 (2022): 3461-3474.\n\nQUESTIONS:\n1. In logistic bandits[2, 6, 7], algorithms achieve $\\kappa$-independent regret bounds by leveraging the self-concordance property of log-loss. What specific challenges were encountered when extending this to GLMs?\n\n2. In the experiments, it appears that ECOLog in [7] is a sub-algorithm for reward parameter estimation. If the comparison is with OFU-ECOLog, which also has a computational complexity of $O(\\log t)$, what might explain the significant difference in execution times between OFU-ECOLog and RS-GLinCB? \n3. Additionally, how does the execution time difference vary with increasing context dimensions?\n4. Why were different values for the upper bound of the reward parameter S used in experiments (S=3 for logistic and S=5 for probit)?\n\n- [Minor typos]\n    - The $\\kappa$ in line 137 and the $\\kappa$ in line 144 seem to refer to different concepts; should different symbols be used?\n    - Line 156: G-optimal design policy $\\pi_G = \\arg \\min_\\lambda \\min_x \\| x \\|^2_{U(\\lambda)^{-1}}$\n    - Algorithm 2, line 14: $\\hat{\\theta}_w\\) → \\(\\hat{\\theta}_o$\n    - Constraint in Eq. (6): $|| \\theta - \\hat{\\theta}_w ||_V \\le \\gamma \\sqrt{\\kappa}$ → $|| \\theta - \\hat{\\theta}_o ||_V \\le \\gamma \\sqrt{\\kappa}$\n\nLIMITATIONS:\nThe authors have well-addressed the limitations and further research directions in Section 6.\n\nThe content discussed in this paper appears to have little to no negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider the problem of regret minimization in bounded generalized linear contextual bandits with limited adaptivity. Specifically, they consider two models of limited adaptivity: **M1** in which the update rounds must be chosen before the algorithm is run, and **M2** in which the algorithm can be adaptive as the algorithm proceeds. For **M1** the authors propose B-GLinCB that obtains $\\tilde{O}(dRS\\sqrt{T/\\kappa^*})$, and for **M2**, RS-GLinCB that obtains $\\tilde{O}(d \\sqrt{T/\\kappa^*})$. The efficacy of RS-GLinCB is shown numerically.\n\nSTRENGTHS:\n- Clearly and well-written\n- First $d \\sqrt{T/\\kappa}$-type regret that holds for generalized linear bandits *beyond* logistic bandits + $\\mathrm{poly}(S)$-free leading term for generalized linear bandits\n- Numerous interesting (and important) technical contributions were made to the algorithm design and the regret analysis.\n\nWEAKNESSES:\n- The numerical experiments could benefit from more comparators, namely, randomized algorithms: Thompson sampling [1] (and its follow-up works, e.g., [2]) and recently proposed EVILL [3]. I know (and appreciate) that the authors' algorithms are primarily for the limited adaptivity scenarios. Still, given how one of the main contributions of this paper is state-of-the-art regret analysis, it would be important to have these (practically performing well) randomized algorithms as comparators.\n- (Continuing from the first point) In the prior regret analyses of logistic bandits by [4], they obtained $\\tilde{O}(d\\sqrt{T/\\kappa^*} + \\kappa \\wedge R_-(T))$, where $R_-(T)$ is some arm-geometry-adaptive term that can be much smaller than $\\kappa$. As the algorithm here makes use of warmup, it must incur the worst-case geometry-dependent term. The authors should also compare with this algorithm for the sake of regret comparison.\n- The algorithms involve an explicit warmup stage, which in practice may be not so good. This is shown in the logistic bandits experiments, where although RS-GLinUCB is good eventually, its warmup (that scales with $\\kappa$) forces the algorithm to incur high regret in the beginning (til round ~10000).\n- Some references on logistic bandits missing, namely [5] where in fixed arm-set setting, they proposed a Hessian-based optimal design (H-optimality), then a warmup-based algorithm was proposed that obtains $\\tilde{O}(d \\sqrt{T/\\kappa^*})$ regret, which is $\\mathrm{poly}(S)$-free.\n- Although the regret of RS-GLinCB is indeed $\\mathrm{poly}(S)$-free, it mainly relies on a nonconvex optimization (Eqn. (6)), and it seems that the tractable convex relaxation again introduces factors of $S$ (and $R$) to the leading term. (please correct me here if I'm wrong) -- This point should be made precise in the introduction.\n\n\n\n[1] https://proceedings.mlr.press/v108/kveton20a.html\n\n[2] https://arxiv.org/abs/2209.06983\n\n[3] https://proceedings.mlr.press/v238/janz24a.html\n\n[4] https://proceedings.mlr.press/v130/abeille21a.html\n\n[5] https://arxiv.org/abs/2202.02407\n\nQUESTIONS:\n- The authors mention that the optimization problem in Eqn. (6) is nonconvex, and a convex relaxation results in additional factors of $\\mathrm{poly(R, S)}$. Do the factors appear in the leading term as well?\n- Are the analyses and algorithms amenable to unbounded, generalized linear models that are self-concordant? For instance, Gaussian is self-concordant with a multiplicative factor of $1$ but is unbounded.\n- For the algorithms, the authors use l2-penalized MLE, then project it to the S-ball if necessary. Why not just do l2-constrained MLE, as done in [6]?\n- The authors used an optimal design based on $\\lVert \\cdot \\rVert_{V_t}$, thus incurring explicit dependency on $\\kappa$. Is there any way to make the warmup more efficient by considering geometry-dependent norm, e.g., $\\lVert \\cdot \\rVert_{H_t}$ as in [5]?\n- The authors stated that the ellipsoidal projection is the main technical novelty for obtaining $\\mathrm{poly}(S)$-free regret for RS-GLinCB. Can this then be combined with prior UCB-based algorithms for logistic bandits (or GLM bandits) to obtain similar improvements? Or is it the case that such ellipsoidal projection *combined* with some other techniques for the limited adaptivity allows for $\\mathrm{poly}(S)$-free regret?\n- (minor) Can the intuitions and ideas from kappa-dependent warmup be used for best arm identification, e.g., [7]?\n\n\n\n[6] https://proceedings.mlr.press/v238/lee24c.html\n\n[7] https://proceedings.mlr.press/v139/jun21a.html\n\n(If all my concerns are sufficiently addressed, I'm leaning towards further raising the score)\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considered regret minimization for a generalized linear reward model with limited adaptivity, in which the set of arms $\\mathcal{X}t$ is stochastically generated by unknown distribution $\\mathcal{D}$, and after pulling $x_t \\in \\mathcal{X}_t$ the learner receives a reward $r_t$ sampled from the GLM distribution $P(r|x_t)$ with unknown $\\theta^*$.\nIn the first setting M1, the algorithm is given a budget $M$ and is asked to decide upfront $M$ rounds to update its policy. For M1, B-GlinCB is proposed, whose regret depends on $\\sqrt{\\hat{\\kappa}^{-1}}$ or $\\sqrt{{\\kappa^*}^{-1}}$.  In the second setting M2, the algorithm is given a budget $M$ and needs to decide $M$ rounds to update its policy adaptively. \nFor RS-GlinCB, the authors provided regret bound where $\\kappa$-dependence only appears in $\\log^2T$ term. Experimental results are demonstrated to validate their algorithms.\n\nSTRENGTHS:\nThe first attempt to study GKM with limited adaptivity. The algorithm removes the instance^dependent non-linearity parameter $\\kappa$ which can be exponentially large concerning $|| \\theta^*||$.\n\nWEAKNESSES:\n1. To compute the length of the warm-up batch in M1, the knowledge of $\\kappa$ which depends on the optimal parameter $\\theta^*$ is required. Also, UCB/LCB scores and $\\beta(x)$ also need this knowledge. In M2 setting, Switching Criterion I depends on $\\kappa$. In practice, how can we estimate $\\kappa$ parameterized by unknown $\\theta^*$?\n\n\n2. The reviewer could not understand how significant the benefit of removing $\\kappa$dependence compared with other $\\sqrt{\\hat{\\kappa}^{-1}}$ or $\\sqrt{{\\kappa^*}^{-1}}$ is while allowing the policy to update at most $O(\\kappa \\log^2 T)$ times that is $\\kappa$-dependent. Only when $\\kappa =o(\\log T)$, the amount of adaptivity is reasonable, but now the regret bound does not need to care about $\\kappa$-dependence in such cases. Conversely, when $\\kappa$ is large, removing such dependence from the regret bound is important but now the algorithm requires a large amount of updates.  Could authors discuss the benefits/trade-offs of this point more?\n\nQUESTIONS:\nPlease see questions in Weaknesses.\n\n\nTypo/minor comments:\n\n$x^*$ is dependent on arm set $\\mathcal{X}$, so other notation may be clearer such as $x^*_t$ for $\\mathcal{X}_t$.\n\nSince the def. of $\\kappa$ in Alg 1 and Alg 2 is different, why not introduce different notations?\n\nHow the scaling parameter $\\beta(x)$ is defined in (2)? Is this same value defined later in (3)?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-18"}