Information Directed Sampling for Sparse Linear Bandits

1 Introduction . Standard linear bandits associate each action with a feature vector and assume the mean reward is the inner product between the feature vector and an unknown parameter vector [ Auer , 2002 , Dani et al. , 2008 , Rusmevichientong and Tsitsiklis , 2010 , Chu et al. , 2011 , Abbasi-Yadkori et al. , 2011 ] . Sparse linear bandits generalize linear bandits by assuming the unknown parameter vector is sparse [ Abbasi-Yadkori et al. , 2012 , Carpentier and Munos , 2012 , Hao et al. , 2020b ] and is of great practical significance for modeling high-dimensional online decision-making problems [ Bastani and Bayati , 2020 ] . Lattimore and Szepesvári [ 2020 , §24.3 ] established a Ω ( √ sdn ) regret lower bound for the data-rich regime , where n is the horizon , d is the feature dimension , s is the sparsity and data-rich regime refers to the horizon n ≥ dα for some α > 0 . This means polynomial dependence on d is generally not avoidable without additional assumptions . However , this bound hides much of the rich structure of sparse linear bandits by a crude maximisation over all environments . When the action set admits a well-conditioned exploration distribution , Hao et al . [ 2020b ] discovered the information-regret trade-off phenomenon by establishing an Θ ( poly ( s ) n2/3 ) minimax rate for the data-poor regime . An interpretation for this optimal rate is that the agent needs to acquire enough information for fast sparse learning by pulling informative actions that even have high regret . Explore-then-commit algorithm can achieve this rate for the data-poor regime but is sub-optimal 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . in the data-rich regime . Therefore , our goal is to develop an efficient algorithm that can adapt to different information-regret structures for sparse linear bandits . Contributions Our contribution is three-fold : • We prove that optimism-based algorithms fail to optimally address the information-regret trade-off in sparse linear bandits , which results in a sub-optimal regret bound . • We provide the first analysis using information theory for sparse linear bandits and derive a class of nearly optimal Bayesian regret bounds for IDS that can adapt to information-regret structures . • To approximate the information ratio , we develop an empirical Bayesian approach for sparse posterior sampling using spike-and-slab Gaussian-Laplace prior . Through several experiments , we justify the great empirical performance of sparse IDS with an efficient implementation . 2 Preliminary . We first introduce the basic setup of stochastic sparse linear bandits . The agent receives a compact action set A ⊆ Rd in the beginning where |A| = K. At each round t , the agent chooses an action At ∈ A and receives a reward Yt = 〈At , θ∗〉 + ηt , where ( ηt ) nt=1 is a sequence of independent standard Gaussian random variables and θ∗ ∈ Rd is the true parameter unknown to the agent . We make the mild boundedness assumption that for all a ∈ A , ‖a‖∞ ≤ 1 . The notion of sparsity can be defined through the parameter space Θ : Θ = θ ∈ Rd ∣∣∣∣∣ d∑ j=1 1 { θj 6= 0 } ≤ s , ‖θ‖2 ≤ 1 . We assume s is known and it can be relaxed by putting a prior on it . We consider the Bayesian setting where θ∗ is a random variable taking values in Θ and denote ρ as the prior distribution . The optimal action is x∗ = argmaxa∈A E [ 〈a , θ∗〉|θ∗ ] . The agent chooses At based on the history Ft = ( A1 , Y1 , . . . , At−1 , Yt−1 ) . Let D ( A ) be the space of probability measures over A . A policy π = ( πt ) t∈N is a sequence of deterministic functions where πt ( Ft ) specifies a probability distribution overA . The information-theoretic Bayesian regret of a policy π [ Russo and Van Roy , 2014 ] is defined as BR ( n ; π ) = E [ n∑ t=1 〈x∗ , θ∗〉 − n∑ t=1 Yt ] , where the expectation is over the interaction sequence induced by the agent and environment and the prior distribution over θ∗ . Notation Denote Id as the d × d identity matrix . Let [ n ] = { 1 , 2 , . . . , n } . For a vector x and positive semidefinite matrix A , we let ‖x‖A = √ x > Ax be the weighted ` 2-norm and σmin ( A ) be the minimum eigenvalue of A . The relation x & y means that x is greater or equal to y up to some universial constant and Õ ( · ) hides modest logarithmic factors and universal constant . The cardinality of a setA is denoted by |A| . Given a measure P and jointly distributed random variables X and Y we let PX denote the law ofX and we let PX|Y be the conditional law ofX given Y : PX|Y ( · ) = P ( X ∈ ·|Y ) . The mutual information between X and Y is I ( X ; Y ) = E [ DKL ( PX|Y ||PX ) ] where DKL is the relative entropy . We write Pt ( · ) = P ( ·|Ft ) as the posterior measure where P is the probability measure over θ and the history and Et ( · ) = E ( ·|Ft ) . Denote It ( X ; Y ) = Et [ DKL ( Pt , X|Y ||Pt , X ) ] . 3 Related work . Sparse linear bandits Abbasi-Yadkori et al . [ 2012 ] proposed an inefficient online-to-confidenceset conversion approach that achieves an Õ ( √ sdn ) upper bound for an arbitrary action set . Lattimore et al . [ 2015 ] developed a selective explore-then-commit algorithm that only works when the action set is exactly the binary hypercube and derived an optimal O ( s √ n ) upper bound . Hao et al . [ 2020b ] introduced the notion of an exploratory action set and proved a Θ ( poly ( s ) n2/3 ) minimax rate for the data-poor regime using an explore-then-commit algorithm . Hao et al . [ 2021 ] extended this concept to a MDP setting . Carpentier and Munos [ 2012 ] considered a special case where the action set is the unit sphere and the noise is vector-valued so that the noise becomes smaller as the dimension grows . We summarize the comparison of existing results with our work in Table 3 . Sparse linear contextual bandits It recently became popular to study the contextual setting , where the action set changes from round to round . These results can not be reduced to our setting since they rely on either careful assumptions on the context distribution [ Bastani and Bayati , 2020 , Wang et al. , 2018 , Kim and Paik , 2019 , Wang et al. , 2020 , Ren and Zhou , 2020 , Oh et al. , 2020 ] such that classical high-dimensional statistics can be used , or have polynomial dependency on the number of actions [ Agarwal et al. , 2014 , Foster and Rakhlin , 2020 , Simchi-Levi and Xu , 2020 ] . Information-directed sampling Russo and Van Roy [ 2018 ] introduced IDS and derived Bayesian regret bounds for multi-armed bandits , linear bandits and combinatorial bandits . Liu et al . [ 2018 ] studied IDS for bandits with graph-feedback . Kirschner and Krause [ 2018 ] , Kirschner et al . [ 2020a ] investigated the use of IDS for bandits with heteroscedastic noise and partial monitoring . Kirschner et al . [ 2020b ] proved the asymptotic optimality of frequentist IDS for linear bandits . Information-theoretic analysis Russo and Van Roy [ 2014 ] introduced an information-theoretic analysis of Thompson sampling ( TS ) and Bubeck and Sellke [ 2020 ] strengthened the result with a first-order Bayesian regret analysis . Dong and Roy [ 2018 ] , Dong et al . [ 2019 ] extended the analysis to infinite-many actions and logistic bandits . Lattimore and Szepesvári [ 2019 ] explored the use of information-theoretic analysis for partial monitoring . Lu and Van Roy [ 2019 ] generalized the analysis to reinforcement learning . Bayesian sparse linear regression In the Bayesian framework , spike-and-slab methods are commonly used as probabilistic tools for sparse linear regression but most of prior works focus on variable selection and parameter estimation rather than uncertainty quantification [ Mitchell and Beauchamp , 1988 , George and McCulloch , 1993 , Ročková and George , 2018 ] . Bai et al . [ 2020 ] provided a comprehensive overview . 4 Does the optimism optimally balance information and regret ? . We demonstrate the necessity of balancing the trade-off between information and regret through a simple sparse linear bandit instance . We give an example where the optimal regret is only possible by playing actions that are known to be sub-optimal . This phenomenon has disturbing implications for policies based on the principle of optimism , which is that they can never be minimax optimal in certain regime . Illustrative example Consider a problem instance whereA = I ∪U is the union of an informative action set I and an uninformative action set U : • I is a subset of the hypercube that has three properties . First , |I| = O ( s log ( ed/s ) ) . Second , the last coordinate of actions in I is always -1 . Third , the empirical covariance of the uniform distribution over I has a restricted minimum eigenvalue ( Definition A.1 ) at least 1/4 . We prove such I does exist in Appendix A.1 through a probabilistic argument . • U = { x ∈ Rd|xj ∈ { −1 , 0 , 1 } for j ∈ [ d− 1 ] , ‖x‖1 = s− 1 , xd = 0 } . The true parameter θ∗ = ( ε , . . . , ε , 0 , . . . , 0 , −1 ) , where ε > 0 is a small constant . Information-regret structure Sampling an action uniformly at random from I ensures the covariance matrix is well-conditioned so that sparse learning such as Lasso [ Tibshirani , 1996 ] can be used for learning θ∗ faster than ordinary least squares . This means pulling actions from I provides more information to infer θ∗ than from U . On the other hand , actions from I lead to high regret due to the last coordinate -1 . As a consequence , Hao et al . [ 2020b ] has proven that the minimax regret for this problem is Θ ( poly ( s ) n2/3 ) when the horizon is smaller than the ambient dimension . Sub-optimality of optimism-based algorithms We argue the optimism principle does not take this subtle trade-off into consideration and yields sub-optimal regret in the data-poor regime . In general , optimism-based algorithms chooseAt = argmaxa∈Amaxθ̃∈Ct〈a , θ̃〉 , where Ct ⊆ R d is a confidence set that contains the true θ∗ with high probability . We assume that there exists a constant c > 0 such that Ct ⊆ { θ : ( θ̂t − θ ) > Vt ( θ̂t − θ ) ≤ c √ s log ( n ) } , where Vt = ∑t s=1AsA > s and θ̂t is some sparsity-aware estimator . Such confidence set can be constructed through an online-to-confidence set conversion approach [ Abbasi-Yadkori et al. , 2012 ] . Define Rθ ( n ; π ) = E [ ∑n t=1〈x∗ , θ〉 − ∑n t=1 Yt ] for a fixed θ . Claim 4.1 . Let πopt be such an optimism-based algorithm . There exists a sparse linear bandit instance characterized by θ such that for the data-poor regime , we have Rθ ( n ; π opt ) & n/ ( log ( n ) s log ( ed/s ) ) . The proof is deferred to Appendix A.1 . The reason is that optimism-based algorithms do not choose actions for which they have collected enough statistics to prove these actions are suboptimal , but in the sparse linear setting it can be worth playing these actions when they are informative about other actions for which the statistics are not yet so clear . This phenomenon has been observed before in linear and structured bandits [ Lattimore and Szepesvari , 2017 , Combes et al. , 2017 , Hao et al. , 2020a ] .

Summary:  This paper analyses information-directed sampling (IDS) for the problem of sparse linear bandits. They identify a key issue limiting optimism-based algorithms (or UCB type methods) where, for certain sparse instances, the regret can scale nearly linearly in the time horizon. They demonstrate that IDS does circumvents this information obstacle, by explicitly choosing actions that optimize for the information ratio: i.e. a ratio between the (sub)optimality of the action chosen and the future information gleaned from it. For IDS, they demonstrate an adaptive result, that shows its (up to log factors) optimality in regret both  for short (data-poor) and long (data rich) time horizons. Computationally, IDS is non-trivial to implement in the sparse setting, for which the authors propose approach using mcmc sampling, borrowing from Bayesian sparse linear regression literature.