Information Directed Sampling for Sparse Linear Bandits

1 Introduction . Standard linear bandits associate each action with a feature vector and assume the mean reward is the inner product between the feature vector and an unknown parameter vector [ Auer , 2002 , Dani et al. , 2008 , Rusmevichientong and Tsitsiklis , 2010 , Chu et al. , 2011 , Abbasi-Yadkori et al. , 2011 ] . Sparse linear bandits generalize linear bandits by assuming the unknown parameter vector is sparse [ Abbasi-Yadkori et al. , 2012 , Carpentier and Munos , 2012 , Hao et al. , 2020b ] and is of great practical significance for modeling high-dimensional online decision-making problems [ Bastani and Bayati , 2020 ] . Lattimore and Szepesvári [ 2020 , §24.3 ] established a Ω ( √ sdn ) regret lower bound for the data-rich regime , where n is the horizon , d is the feature dimension , s is the sparsity and data-rich regime refers to the horizon n ≥ dα for some α > 0 . This means polynomial dependence on d is generally not avoidable without additional assumptions . However , this bound hides much of the rich structure of sparse linear bandits by a crude maximisation over all environments . When the action set admits a well-conditioned exploration distribution , Hao et al . [ 2020b ] discovered the information-regret trade-off phenomenon by establishing an Θ ( poly ( s ) n2/3 ) minimax rate for the data-poor regime . An interpretation for this optimal rate is that the agent needs to acquire enough information for fast sparse learning by pulling informative actions that even have high regret . Explore-then-commit algorithm can achieve this rate for the data-poor regime but is sub-optimal 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . in the data-rich regime . Therefore , our goal is to develop an efficient algorithm that can adapt to different information-regret structures for sparse linear bandits . Contributions Our contribution is three-fold : • We prove that optimism-based algorithms fail to optimally address the information-regret trade-off in sparse linear bandits , which results in a sub-optimal regret bound . • We provide the first analysis using information theory for sparse linear bandits and derive a class of nearly optimal Bayesian regret bounds for IDS that can adapt to information-regret structures . • To approximate the information ratio , we develop an empirical Bayesian approach for sparse posterior sampling using spike-and-slab Gaussian-Laplace prior . Through several experiments , we justify the great empirical performance of sparse IDS with an efficient implementation . 2 Preliminary . We first introduce the basic setup of stochastic sparse linear bandits . The agent receives a compact action set A ⊆ Rd in the beginning where |A| = K. At each round t , the agent chooses an action At ∈ A and receives a reward Yt = 〈At , θ∗〉 + ηt , where ( ηt ) nt=1 is a sequence of independent standard Gaussian random variables and θ∗ ∈ Rd is the true parameter unknown to the agent . We make the mild boundedness assumption that for all a ∈ A , ‖a‖∞ ≤ 1 . The notion of sparsity can be defined through the parameter space Θ : Θ = θ ∈ Rd ∣∣∣∣∣ d∑ j=1 1 { θj 6= 0 } ≤ s , ‖θ‖2 ≤ 1 . We assume s is known and it can be relaxed by putting a prior on it . We consider the Bayesian setting where θ∗ is a random variable taking values in Θ and denote ρ as the prior distribution . The optimal action is x∗ = argmaxa∈A E [ 〈a , θ∗〉|θ∗ ] . The agent chooses At based on the history Ft = ( A1 , Y1 , . . . , At−1 , Yt−1 ) . Let D ( A ) be the space of probability measures over A . A policy π = ( πt ) t∈N is a sequence of deterministic functions where πt ( Ft ) specifies a probability distribution overA . The information-theoretic Bayesian regret of a policy π [ Russo and Van Roy , 2014 ] is defined as BR ( n ; π ) = E [ n∑ t=1 〈x∗ , θ∗〉 − n∑ t=1 Yt ] , where the expectation is over the interaction sequence induced by the agent and environment and the prior distribution over θ∗ . Notation Denote Id as the d × d identity matrix . Let [ n ] = { 1 , 2 , . . . , n } . For a vector x and positive semidefinite matrix A , we let ‖x‖A = √ x > Ax be the weighted ` 2-norm and σmin ( A ) be the minimum eigenvalue of A . The relation x & y means that x is greater or equal to y up to some universial constant and Õ ( · ) hides modest logarithmic factors and universal constant . The cardinality of a setA is denoted by |A| . Given a measure P and jointly distributed random variables X and Y we let PX denote the law ofX and we let PX|Y be the conditional law ofX given Y : PX|Y ( · ) = P ( X ∈ ·|Y ) . The mutual information between X and Y is I ( X ; Y ) = E [ DKL ( PX|Y ||PX ) ] where DKL is the relative entropy . We write Pt ( · ) = P ( ·|Ft ) as the posterior measure where P is the probability measure over θ and the history and Et ( · ) = E ( ·|Ft ) . Denote It ( X ; Y ) = Et [ DKL ( Pt , X|Y ||Pt , X ) ] . 3 Related work . Sparse linear bandits Abbasi-Yadkori et al . [ 2012 ] proposed an inefficient online-to-confidenceset conversion approach that achieves an Õ ( √ sdn ) upper bound for an arbitrary action set . Lattimore et al . [ 2015 ] developed a selective explore-then-commit algorithm that only works when the action set is exactly the binary hypercube and derived an optimal O ( s √ n ) upper bound . Hao et al . [ 2020b ] introduced the notion of an exploratory action set and proved a Θ ( poly ( s ) n2/3 ) minimax rate for the data-poor regime using an explore-then-commit algorithm . Hao et al . [ 2021 ] extended this concept to a MDP setting . Carpentier and Munos [ 2012 ] considered a special case where the action set is the unit sphere and the noise is vector-valued so that the noise becomes smaller as the dimension grows . We summarize the comparison of existing results with our work in Table 3 . Sparse linear contextual bandits It recently became popular to study the contextual setting , where the action set changes from round to round . These results can not be reduced to our setting since they rely on either careful assumptions on the context distribution [ Bastani and Bayati , 2020 , Wang et al. , 2018 , Kim and Paik , 2019 , Wang et al. , 2020 , Ren and Zhou , 2020 , Oh et al. , 2020 ] such that classical high-dimensional statistics can be used , or have polynomial dependency on the number of actions [ Agarwal et al. , 2014 , Foster and Rakhlin , 2020 , Simchi-Levi and Xu , 2020 ] . Information-directed sampling Russo and Van Roy [ 2018 ] introduced IDS and derived Bayesian regret bounds for multi-armed bandits , linear bandits and combinatorial bandits . Liu et al . [ 2018 ] studied IDS for bandits with graph-feedback . Kirschner and Krause [ 2018 ] , Kirschner et al . [ 2020a ] investigated the use of IDS for bandits with heteroscedastic noise and partial monitoring . Kirschner et al . [ 2020b ] proved the asymptotic optimality of frequentist IDS for linear bandits . Information-theoretic analysis Russo and Van Roy [ 2014 ] introduced an information-theoretic analysis of Thompson sampling ( TS ) and Bubeck and Sellke [ 2020 ] strengthened the result with a first-order Bayesian regret analysis . Dong and Roy [ 2018 ] , Dong et al . [ 2019 ] extended the analysis to infinite-many actions and logistic bandits . Lattimore and Szepesvári [ 2019 ] explored the use of information-theoretic analysis for partial monitoring . Lu and Van Roy [ 2019 ] generalized the analysis to reinforcement learning . Bayesian sparse linear regression In the Bayesian framework , spike-and-slab methods are commonly used as probabilistic tools for sparse linear regression but most of prior works focus on variable selection and parameter estimation rather than uncertainty quantification [ Mitchell and Beauchamp , 1988 , George and McCulloch , 1993 , Ročková and George , 2018 ] . Bai et al . [ 2020 ] provided a comprehensive overview . 4 Does the optimism optimally balance information and regret ? . We demonstrate the necessity of balancing the trade-off between information and regret through a simple sparse linear bandit instance . We give an example where the optimal regret is only possible by playing actions that are known to be sub-optimal . This phenomenon has disturbing implications for policies based on the principle of optimism , which is that they can never be minimax optimal in certain regime . Illustrative example Consider a problem instance whereA = I ∪U is the union of an informative action set I and an uninformative action set U : • I is a subset of the hypercube that has three properties . First , |I| = O ( s log ( ed/s ) ) . Second , the last coordinate of actions in I is always -1 . Third , the empirical covariance of the uniform distribution over I has a restricted minimum eigenvalue ( Definition A.1 ) at least 1/4 . We prove such I does exist in Appendix A.1 through a probabilistic argument . • U = { x ∈ Rd|xj ∈ { −1 , 0 , 1 } for j ∈ [ d− 1 ] , ‖x‖1 = s− 1 , xd = 0 } . The true parameter θ∗ = ( ε , . . . , ε , 0 , . . . , 0 , −1 ) , where ε > 0 is a small constant . Information-regret structure Sampling an action uniformly at random from I ensures the covariance matrix is well-conditioned so that sparse learning such as Lasso [ Tibshirani , 1996 ] can be used for learning θ∗ faster than ordinary least squares . This means pulling actions from I provides more information to infer θ∗ than from U . On the other hand , actions from I lead to high regret due to the last coordinate -1 . As a consequence , Hao et al . [ 2020b ] has proven that the minimax regret for this problem is Θ ( poly ( s ) n2/3 ) when the horizon is smaller than the ambient dimension . Sub-optimality of optimism-based algorithms We argue the optimism principle does not take this subtle trade-off into consideration and yields sub-optimal regret in the data-poor regime . In general , optimism-based algorithms chooseAt = argmaxa∈Amaxθ̃∈Ct〈a , θ̃〉 , where Ct ⊆ R d is a confidence set that contains the true θ∗ with high probability . We assume that there exists a constant c > 0 such that Ct ⊆ { θ : ( θ̂t − θ ) > Vt ( θ̂t − θ ) ≤ c √ s log ( n ) } , where Vt = ∑t s=1AsA > s and θ̂t is some sparsity-aware estimator . Such confidence set can be constructed through an online-to-confidence set conversion approach [ Abbasi-Yadkori et al. , 2012 ] . Define Rθ ( n ; π ) = E [ ∑n t=1〈x∗ , θ〉 − ∑n t=1 Yt ] for a fixed θ . Claim 4.1 . Let πopt be such an optimism-based algorithm . There exists a sparse linear bandit instance characterized by θ such that for the data-poor regime , we have Rθ ( n ; π opt ) & n/ ( log ( n ) s log ( ed/s ) ) . The proof is deferred to Appendix A.1 . The reason is that optimism-based algorithms do not choose actions for which they have collected enough statistics to prove these actions are suboptimal , but in the sparse linear setting it can be worth playing these actions when they are informative about other actions for which the statistics are not yet so clear . This phenomenon has been observed before in linear and structured bandits [ Lattimore and Szepesvari , 2017 , Combes et al. , 2017 , Hao et al. , 2020a ] .

Summary: The paper considers the sparse linear bandit problem from Bayesian perspective. The authors show that IDS achieves optimal Bayesian regret bounds for both data-poor and data-rich regime. Since their algorithm cannot be implemented efficiently, they propose an algorithm desired to approximate the IDS, which is shown to work well empirically.