Neural Contextual Bandits with Deep Representation and Shallow Exploration

√ T ) finite- time regret , where T is the learning time horizon . Compared with existing neural contextual bandit algorithms , our approach is computationally much more efficient since it only needs to explore in the last layer of the deep neural network . 1 INTRODUCTION . Multi-armed bandits ( MAB ) ( Auer et al. , 2002 ; Audibert et al. , 2009 ; Lattimore & Szepesvári , 2020 ) are a class of online decision-making problems where an agent needs to learn to maximize its expected cumulative reward by repeatedly interacting with a partially known environment . Following a bandit algorithm ( also called a strategy or policy ) , in each round , the agent adaptively chooses an arm , and then receives a reward associated with that arm . Since only the reward of the chosen arm will be observed ( bandit information feedback ) , a good bandit algorithm has to deal with the exploration-exploitation dilemma : trade-off between pulling the best arm based on existing knowledge/history data ( exploitation ) and trying the arms that have not been fully explored ( exploration ) . In many real-world applications , the agent will also be able to access detailed contexts associated with the arms . For example , when a company wants to choose an advertisement to present to a user , the recommendation will be much more accurate if the company takes into consideration the contents , specifications , and other features of the advertisements in the arm set as well as the profile of the user . To encode the contextual information , contextual bandit models and algorithms have been developed , and widely studied both in theory and in practice ( Dani et al. , 2008 ; Rusmevichientong & Tsitsiklis , 2010 ; Li et al. , 2010 ; Chu et al. , 2011 ; Abbasi-Yadkori et al. , 2011 ) . Most existing contextual bandit algorithms assume that the expected reward of an arm at a context is a linear function in a known context-action feature vector , which leads to many useful algorithms such as LinUCB ( Chu et al. , 2011 ) , OFUL ( Abbasi-Yadkori et al. , 2011 ) , etc . The representation power of the linear model can be limited in applications such as marketing , social networking , clinical studies , etc. , where the rewards are usually counts or binary variables . The linear contextual bandit problem has also been extended to richer classes of parametric bandits such as the generalized linear bandits ( Filippi et al. , 2010 ; Li et al. , 2017 ) and kernelised bandits ( Valko et al. , 2013 ; Chowdhury & Gopalan , 2017 ) . With the prevalence of deep neural networks ( DNNs ) and their phenomenal performances in many machine learning tasks ( LeCun et al. , 2015 ; Goodfellow et al. , 2016 ) , there has emerged a line of work that employs DNNs to increase the representation power of contextual bandit algorithms ( Allesiardo et al. , 2014 ; Riquelme et al. , 2018 ; Collier & Llorens , 2018 ; Zahavy & Mannor , 2019 ; Zhou et al. , 2020 ; Deshmukh et al. , 2020 ; Zhang et al. , 2020 ) . The problems they solve are usually referred to as neural contextual bandits . For example , Zhou et al . ( 2020 ) developed the NeuralUCB algorithm , which can be viewed as a natural extension of LinUCB ( Chu et al. , 2011 ; Abbasi-Yadkori et al. , 2011 ) , where they use the output of a deep neural network with the feature vector as input to approximate the reward . Zhang et al . ( 2020 ) adapted neural networks in Thompson Sampling ( Thompson , 1933 ; Chapelle & Li , 2011 ; Russo et al. , 2018 ) for both exploration and exploitation and proposed NeuralTS . For a fixed time horizon T , it has been proved that both NeuralUCB and NeuralTS achieve a O ( d̃ √ T ) regret bound , where d̃ is the effective dimension of a neural tangent kernel matrix which can potentially scale with O ( TK ) for K-armed bandits . This high complexity is mainly due to that the exploration is performed over the entire huge neural network parameter space , which is inefficient and even infeasible when the number of neurons is large . A more realistic and efficient way of learning neural contextual bandits may be to just explore different arms using the last layer as the exploration parameter . More specifically , Riquelme et al . ( 2018 ) provided an extensive empirical study of benchmark algorithms for contextual-bandits through the lens of Thompson Sampling , which suggests decoupling representation learning and uncertainty estimation improves performance . In this paper , we show that the decoupling of representation learning and the exploration can be theoretically validated . We study a new neural contextual bandit algorithm , which learns a mapping to transform the raw features associated with each context-action pair using a deep neural network ( deep representation ) , and then performs an upper confidence bound ( UCB ) -type exploration over the linear output layer of the network ( shallow exploration ) . We prove a sublinear regret of the proposed algorithm by exploiting the UCB exploration techniques in linear contextual bandits ( Abbasi-Yadkori et al. , 2011 ) and the analysis of deep overparameterized neural networks using neural tangent kernels ( Jacot et al. , 2018 ) . Our theory confirms the empirically observed effectiveness of decoupling the deep representation learning and the UCB exploration in contextual bandits ( Riquelme et al. , 2018 ; Zahavy & Mannor , 2019 ) . Contributions we summarize the main contributions of this paper as follows . • We propose a contextual bandit algorithm , Neural-LinUCB , for solving a general class of contextual bandit problems without knowing the specific reward generating function . The proposed algorithm learns a deep representation to transform the raw feature vectors and performs UCBtype exploration in the last layer of the neural network , which we refer to as deep representation and shallow exploration . Compared with LinUCB ( Li et al. , 2010 ; Chu et al. , 2011 ) and neural bandits such as NeuralUCB ( Zhou et al. , 2020 ) and NeuralTS ( Zhang et al. , 2020 ) , our algorithm enjoys the best of two worlds : strong expressiveness due to the deep representation and computational efficiency due to the shallow exploration . • Despite the usage of a DNN as the feature mapping , we prove a Õ ( √ T ) regret for the proposed Neural-LinUCB algorithm , which matches the regret bound of linear contextual bandits ( Chu et al. , 2011 ; Abbasi-Yadkori et al. , 2011 ) . To the best of our knowledge , this is the first work that theoretically shows the convergence of bandits algorithms under the scheme of deep representation and shallow exploration . It is notable that a similar scheme called Neural-Linear was proposed by Riquelme et al . ( 2018 ) for Thompson sampling algorithms , and they empirically showed that decoupling representation learning and uncertainty estimation improves the performance . Our work confirms this observation from a theoretical perspective . • We conduct experiments on contextual bandit problems based on real-world datasets , demonstrating a better performance and computational efficiency of Neural-LinUCB over LinUCB and existing neural bandits algorithms such as NeuralUCB , which well aligns with our theory . 1.1 ADDITIONAL RELATED WORK . There is a line of related work to ours on the recent advance in the optimization and generalization analysis of deep neural networks . In particular , Jacot et al . ( 2018 ) first introduced the neural tangent kernel ( NTK ) to characterize the training dynamics of network outputs in the infinite width limit . From the notion of NTK , a fruitful line of research emerged and showed that loss functions of deep neural networks trained by ( stochastic ) gradient descent can converge to the global minimum ( Du et al. , 2019b ; Allen-Zhu et al. , 2019b ; Du et al. , 2019a ; Zou et al. , 2018 ; Zou & Gu , 2019 ) . The generalization bounds for overparameterized deep neural networks are also established in Arora et al . ( 2019a ; b ) ; Allen-Zhu et al . ( 2019a ) ; Cao & Gu ( 2019a ; b ) . Recently , the NTK based analysis is also extended to the study of sequential decision problems including bandits ( Zhou et al. , 2020 ; Zhang et al. , 2020 ) , and reinforcement learning algorithms ( Cai et al. , 2019 ; Liu et al. , 2019 ; Wang et al. , 2020 ; Xu & Gu , 2020 ) . Our algorithm is also different from Langford & Zhang ( 2008 ) ; Agarwal et al . ( 2014 ) which reduce the bandit problem to supervised learning . Moreover , their algorithms need to access an oracle that returns the optimal policy in a policy class given a sequence of context and reward vectors , whose regret depends on the VC-dimension of the policy class . Notation We use [ k ] to denote a set { 1 , . . . , k } , k ∈ N+ . ‖x‖2 = √ x > x is the Euclidean norm of a vector x ∈ Rd . For a matrix W ∈ Rm×n , we denote by ‖W‖2 and ‖W‖F its operator norm and Frobenius norm respectively . For a semi-definite matrix A ∈ Rd×d and a vector x ∈ Rd , we denote the Mahalanobis norm as ‖x‖A = √ x > Ax . Throughout this paper , we reserve the notations { Ci } i=0,1 , ... to represent absolute positive constants that are independent of problem parameters such as dimension , sample size , iteration number , step size , network length and so on . The specific values of { Ci } i=0,1 , ... can be different in different context . For a parameter of interest T and a function f ( T ) , we use notations such asO ( f ( T ) ) and Ω ( f ( T ) ) to hide constant factors and Õ ( f ( T ) ) to hide constant and logarithmic dependence of T . 2 PRELIMINARIES . In this section , we provide the background of contextual bandits and deep neural networks . 2.1 LINEAR CONTEXTUAL BANDITS . A contextual bandit is characterized by a tuple ( S , A , r ) , where S is the context ( state ) space , A is the arm ( action ) space , and r encodes the unknown reward generating function at all context-arm pairs . A learning agent , who knows S and A but does not know the true reward r ( values bounded in ( 0 , 1 ) for simplicity ) , needs to interact with the contextual bandit for T rounds . At each round t = 1 , . . . , T , the agent first observes a context st ∈ S chosen by the environment ; then it needs to adaptively select an arm at ∈ A based on its past observations ; finally it receives a reward r̂t ( xs , at ) = r ( xs , at ) + ξt , ( 2.1 ) where xs , a ∈ Rd is a known feature vector for context-arm pair ( s , a ) ∈ S ×A , and ξt is a random noise with zero mean . The agent ’ s objective is to maximize its expected total reward over these T rounds , which is equivalent to minimizing the pseudo regret ( Audibert et al. , 2009 ) : RT = E [ T∑ t=1 ( r̂ ( xst , a∗t ) − r̂ ( xst , at ) ) ] , ( 2.2 ) where a∗t ∈ argmaxa∈A { r ( xst , a ) = E [ r̂ ( xst , a ) ] } . To simplify the exposition , we use xt , a to denote xst , a since it only depends on the round index t in most bandit problems , and we assume A = [ K ] . In linear contextual bandits , the reward function in ( 2.1 ) is assumed to have a linear structure r ( xs , a ) = x > s , aθ ∗ for some unknown weight vector θ∗ ∈ Rd . One provably sample efficient algorithm for linear contextual bandits is Linear Upper Confidence Bound ( LinUCB ) ( Chu et al. , 2011 ) or Optimism in the Face of Uncertainty Linear bandit algorithm ( OFUL ) ( Abbasi-Yadkori et al. , 2011 ) . Specifically , at each round t , LinUCB chooses the action at = argmaxa∈ [ K ] { x > t , aθt + αt‖xt , a‖A−1t } , where θt is a point estimate of θ ∗ , At = λI + ∑t i=1 xi , aix > i , ai with some λ > 0 is a matrix defined based on the historical context-arm pairs , and αt > 0 is a tuning parameter that controls the exploration rate in LinUCB .

Summary: The paper studies the contextual bandit without knowing the specific reward generating function. The authors propose to use a deep neural network to model the reward function where the hidden layers of the deep neural network are used to represent the features and the exploration is only performed in the last layer of the neural network. In this way, the reward generating function can be expressed as the inner product between a deeply represented feature vector and an exploration weight parameter. As the author explained, the main advantage of this reward model is to efficiently reduce computation because we only need to explore the last layer of the deep neural network instead of exploring the whole network like existing neural bandit algorithms.  The authors show that the proposed algorithm for this problem obtains a near-optimal regret which matches the regret bound of linear contextual bandits. They also provided experiments based on real-world datasets to show the performance and computational efficiency.