EE-Net: Exploitation-Exploration Neural Networks in Contextual Bandits

√ T log T ) regret , which is tighter than existing state-of- the-art neural bandit algorithms ( O ( √ T log T ) for both UCB-based and TS-based ) . Through extensive experiments on four real-world datasets , we show that EE-Net outperforms existing linear and neural contextual bandit approaches . 1 INTRODUCTION . The stochastic contextual multi-armed bandit ( MAB ) ( Lattimore and Szepesvári , 2020 ) has been studied for decades in machine learning community to solve sequential decision making , with applications in online advertising ( Li et al. , 2010 ) , personal recommendation ( Wu et al. , 2016 ; Ban and He , 2021b ) , etc . In the standard contextual bandit setting , a set of n arms are presented to a learner in each round , where each arm is represented by a context vector . Then by certain strategy , the learner selects and plays one arm , receiving a reward . The goal of this problem is to maximize the cumulative rewards of T rounds . MAB algorithms have principled approaches to address the trade-off between Exploitation and Exploration ( EE ) , as the collected data from past rounds should be exploited to get good rewards but also under-explored arms need to be explored with the hope of getting even better rewards . The most widely-used approaches for EE trade-off can be classified into three main techniques : Epsilon-greedy ( Langford and Zhang , 2008 ) , Thompson Sampling ( TS ) ( Thompson , 1933 ) , and Upper Confidence Bound ( UCB ) ( Auer , 2002 ; Ban and He , 2020 ) . Linear bandits ( Li et al. , 2010 ; Dani et al. , 2008 ; Abbasi-Yadkori et al. , 2011 ) , where the reward is assumed to be a linear function with respect to arm vectors , have been well studied and succeeded both empirically and theoretically . Given an arm , ridge regression is usually adapted to estimate its reward based on collected data from past rounds . UCB-based algorithms ( Li et al. , 2010 ; Chu et al. , 2011 ; Wu et al. , 2016 ; Ban and He , 2021b ) calculate an upper bound for the confidence ellipsoid of estimated reward and determine the arm according to the sum of estimated reward and UCB . TS-based algorithms ( Agrawal and Goyal , 2013 ; Abeille and Lazaric , 2017 ) formulate each arm as a posterior distribution where mean is the estimated reward and choose the one with the maximal sampled reward . However , the linear assumption regarding the reward may not be true in real-world applications ( Valko et al. , 2013 ) . To learn non-linear reward functions , recent works have utilized deep neural networks to learn the underlying reward functions , thanks to its powerful representation ability . Considering the past selected arms and received rewards as training samples , a neural network f1 is built for exploitation . Zhou et al . ( 2020 ) computes a gradient-based upper confidence bound with respect to f1 and uses UCB strategy to select arms . Zhang et al . ( 2021 ) formulates each arm as a normal distribution where the mean is f1 and deviation is calculated based on gradient of f1 , and then uses the TS strategy to choose arms . Both Zhou et al . ( 2020 ) and Zhang et al . ( 2021 ) achieve the near-optimal regret bound of O ( √ T log T ) . In this paper , we propose a novel neural-based exploration strategy , named `` EE-Net '' . Similar to other neural bandits , EE-Net has an exploitation network f1 to estimate rewards for each arm . The crucial difference from existing works is that EE-Net has an exploration network f2 to predict the potential gain for each arm compared to current reward estimate . The input to the exploration network is the gradient of f1 and the ground-truth is residual difference between the true received reward and the estimated reward from f1 . The strategy is inspired by recent advances in the neural UCB strategies ( Ban et al. , 2021 ; Zhou et al. , 2020 ) . Finally , a decision-maker f3 is constructed to select arms . f3 has two modes : linear or nonlinear . In linear mode , f3 is a linear combination of f1 and f2 , inspired by the UCB strategy . In the nonlinear mode , f3 is formulated as a neural network with input ( f1 , f2 ) and the goal is to learn the probability of being an optimal arm for each arm . Table 1 summarizes the selection criterion difference of EE-Net from other neural bandit algorithms . To sum up , the contributions of this paper can be summarized as follows : 1 . We propose a novel neural exploration strategy , EE-Net , where a neural network is assigned to learn the potential gain compared to the current reward estimate . 2 . Under standard assumptions of over-parameterized neural networks , we prove that EENet can achieve the regret upper bound of O ( √ T log T ) , which is tighter than existing state-of-the-art conteextual bandit algorithms . 3 . We conduct extensive experiments on four real-world datasets , showing that EE-Net out- performs baselines including linear and neural versions of -greedy , TS , and UCB , and establishing EE-net as the new state-of-the-art exploration strategy . Next , we discuss the problem definition in Sec.3 , elaborate on the proposed EE-Net in Sec.4 , and present our theoretical analysis in Sec.5 . In the end , we provide the empirical evaluation ( Sec.6 ) and conclusion . 2 RELATED WORK . Constrained Contextual bandits . The common constrain placed on the reward function is the linear assumption , usually calculated by ridge regression ( Li et al. , 2010 ; Abbasi-Yadkori et al. , 2011 ; Valko et al. , 2013 ; Dani et al. , 2008 ) . The linear UCB-based bandit algorithms ( Abbasi-Yadkori et al. , 2011 ; Li et al. , 2016 ) and the linear Thompson Sampling ( Agrawal and Goyal , 2013 ; Abeille and Lazaric , 2017 ) can achieve successful performance and the near-optimal regret bound of Õ ( √ T ) . To break the linear assumption , Filippi et al . ( 2010 ) generalizes the reward function to a composition of linear and non-linear functions and then adopt a UCB-based algorithm to deal with it ; Bubeck et al . ( 2011 ) imposes the Lipschitz property on reward metric space and constructs a hierarchical optimistic optimization to make selections ; Valko et al . ( 2013 ) embeds the reward function into Reproducing Kernel Hilbert Space and proposes the kernelized TS/UCB bandit algorithms . Neural Bandits . To learn non-linear reward functions , deep neural networks have been adapted to bandits with various variants . Riquelme et al . ( 2018 ) ; Lu and Van Roy ( 2017 ) build L-layer DNN to learn the arm embeddings and apply Thompson Sampling on the last layer for exploration . Zhou et al . ( 2020 ) first introduces a provable neural-based contextual bandit algorithm with a UCB exploration strategy and then Zhang et al . ( 2021 ) extends the neural network to Thompson sampling framework . Their regret analysis is built on recent advances on the convergence theory in over-parameterized neural networks ( Du et al. , 2019 ; Allen-Zhu et al. , 2019 ) and utilizes Neural Tangent Kernel ( Jacot et al. , 2018 ; Arora et al. , 2019 ) to construct connections with linear contextual bandits ( AbbasiYadkori et al. , 2011 ) . Ban and He ( 2021a ) further adopts convolutional neural networks with UCB exploration aiming for visual-aware applications . Xu et al . ( 2020 ) performs UCB-based exploration on the last layer of neural networks to reduce the computational cost brought by gradient-based UCB . Different from the above existing works , EE-Net keeps the powerful representation ability of neural networks to learn reward function and first assigns another neural network to determine exploration . 3 PROBLEM DEFINITION . We consider the standard contextual multi-armed bandit with the known number of rounds T ( Zhou et al. , 2020 ; Zhang et al. , 2021 ) . In each round t ∈ [ T ] , where the sequence [ T ] = [ 1 , 2 , . . . , T ] , the learner is presented with n arms , Xt = { xt,1 , . . . , xt , n } , in which each arm is represented by a feature vector xt , i ∈ Rd for each i ∈ [ n ] . After playing one arm xt , i , its reward rt , i is assumed to be generated by the function : rt , i = h ( xt , i ) + ηt , i , ( 3.1 ) where the unknown reward function h ( xt , i ) can be either linear or non-linear and the noise ηt , i is drawn from certain distribution with expectation E [ ηt , i ] = 0 . Following many existing works ( Zhou et al. , 2020 ; Ban et al. , 2021 ; Zhang et al. , 2021 ) , we consider bounded rewards , rt , i ∈ [ a , b ] . For the brevity , we denote the selected arm in round t by xt and the reward received in t by rt . The expected regret of T rounds is defined as : RT = E [ T∑ t=1 ( r∗t − rt ) ] , ( 3.2 ) where E [ r∗t | Xt ] = maxi∈ [ n ] h ( xt , i ) is the maximal expected reward in the round t. The goal of this problem is to minimize RT by certain selection strategy . Notation . We denote by { xi } ti=1 the sequence ( x1 , . . . , xt ) . We use ‖v‖2 to denote the Euclidean norm for a vector v , and ‖W‖2 and ‖W‖F to denote the spectral and Frobenius norm for a matrix W. We use 〈· , ·〉 to denote the standard inner product between two vectors or two matrices . We may use Oθ1t f1 ( xt , i ) or Oθ1t f1 to represent the gradient Oθ1t f1 ( xt , i ; θ 1 t ) for brevity . We use { xτ , rτ } tτ=1 to represent the collected data up to round t . 4 PROPOSED METHOD : EE-NET . EE-Net is composed of three components . The first component is the exploitation network , f1 ( · ; θ1 ) , which focuses on learning the unknown reward function h based on the data collected in past rounds . The second component is the exploration network , f2 ( · ; θ2 ) , which focuses on characterizing the level of exploration needed for each arm in the present round . The third component is the decision-maker , f3 , which focuses on suitably combining the outputs of the exploitation and exploration networks leading to the arm selection . Published as a conference paper at ICLR 2022 1 ) Exploitation Net . The exploitation net f1 is a neural network which learns the mapping from arms to rewards . In round t , denote the network by f1 ( · ; θ1t−1 ) , where the superscript of θ 1 t−1 is the index of network and the subscript represents the round where the parameters of f1 finished the last update . Given an arm xt , i , i ∈ [ n ] , f1 ( xt , i ; θ1t−1 ) is considered the `` exploitation score '' for xt , i . By some criterion , after playing arm xt , we receive a reward rt . Therefore , we can conduct gradient descent to update θ1 based on the collected training samples { xτ , rτ } tτ=1 and denote the updated parameters by θ1t . 2 ) Exploration Net . Our exploration strategy is inspired by existing UCB-based neural bandits ( Zhou et al. , 2020 ; Ban et al. , 2021 ) . Based on the Lemma 5.2 in ( Ban et al. , 2021 ) , given an arm xt , i , with probability at least 1− δ , we have the following UCB form : |h ( xt , i ) − f1 ( xt , i ; θ1t−1 ) | ≤ Ψ ( Oθ1t−1f1 ( xt , i ; θ 1 t−1 ) ) , ( 4.1 ) where h is defined in Eq . ( 3.1 ) and Ψ is an upper confidence bound represented by a function with respect to the gradient Oθ1t−1f1 ( see more details and discussions in Appendix D ) . Then we have the following definition . Definition 4.1 . In round t , given an arm xt , i , we define h ( xt , i ) − f1 ( xt , i ; θ1t−1 ) as the `` expected potential gain '' for xt , i and rt , i − f1 ( xt , i ; θ1t−1 ) as the `` potential gain '' for xt , i . Let yt , i = rt , i − f1 ( xt , i ; θ1t−1 ) . When yt , i > 0 , the arm xt , i has positive potential gain compared to the estimated reward f1 ( xt , i ; θ1t−1 ) . A large positive yt , i makes the arm more suitable for exploration , whereas a small ( or negative ) yt , i makes the arm unsuitable for exploration . Recall that traditional approaches such as UCB intend to estimate such potential gain yt , i using standard tools , e.g. , Markov inequality , Hoeffding bounds , etc. , from large deviation bounds . Instead of calculating a large-deviation based statistical bound for yt , i , we use a neural network f2 ( · ; θ2 ) to represent Ψ , where the input is Oθ1t−1f1 ( xt , i ) and the ground truth is rt , i−f1 ( xt , i ; θ 1 t−1 ) . Adopting gradient Oθ1t−1f1 ( xt , i ) as the input also is due to the fact that it incorporates two aspects of information : the feature of the arm and the discriminative information of f1 . Moreover , in the upper bound of NeuralUCB or the variance of NeuralTS , there is a recursive term At−1 = I + ∑t−1 τ=1∇θ1τ−1f1 ( xτ ) ∇θ1τ−1f1 ( xτ ) > which is a function of past gradients up to ( t− 1 ) and incorporates relevant historical information . On the contrary , in EE-Net , the recursive term which depends on past gradients is θ2t−1 in the exploration network f2 because we have conducted gradient descent for θ2t−1 based on { ∇θ1τ−1f1 ( xτ ) } t−1 τ=1 . Therefore , this form θ 2 t−1 is similar to At−1 in neuralUCB/TS , but EE-net does not ( need to ) make a specific assumption about the functional form of past gradients , and is also more memory-efficient . To sum up , in round t , we consider f2 ( Oθ1t−1f1 ( xt , i ) ; θ 2 t−1 ) as the `` exploration score '' of xt , i , because it indicates the potential gain of xt , i compared to our current exploitation score f1 ( xt , i ; θ1t−1 ) . Therefore , after receiving the reward rt , we can use gradient descent to update θ2 based on collected training samples { Oθ1τ−1f1 ( xτ ) , rτ − f1 ( xτ ; θ 1 τ−1 } tτ=1 . We also provide other two heuristic forms for f2 ’ s ground-truth label : |rt , i − f1 ( xt , i ; θ1t−1 ) | and ReLU ( rt , i − f1 ( xt , i ; θ 1 t−1 ) ) . We compare them in an ablation study in Appendix B . 3 ) Decision-maker . In round t , given an arm xt , i , i ∈ [ n ] , with the computed exploitation score f1 ( xt , i ; θ 1 t−1 ) and exploration score f2 ( Oθ1t−1f1 ; θ 2 t−1 ) , we use a function f3 ( f1 , f2 ; θ 3 ) to trade off between exploitation and exploration and compute the final score for xt , i . The selection criterion Algorithm 1 EE-Net Input : f1 , f2 , f3 , T ( number of rounds ) , η1 ( learning rate for f1 ) , η2 ( learning rate for f2 ) , η3 ( learning rate for f3 ) , K1 ( number of iterations for f1 ) , K2 ( number of iterations for f2 ) , K3 ( number of iterations for f3 ) , c ( normalization parameter ) 1 : Initialize θ10 , θ 2 0 , θ 3 0 ; θ̂ 1 0 = θ 1 0 , θ̂ 2 0 = θ 2 0 , θ̂ 3 0 = θ 3 0 2 : for t = 1 , 2 , . . . , T do 3 : Observe n arms { xt,1 , . . . , xt , n } 4 : for each i ∈ [ n ] do 5 : Compute f1 ( xt , i ; θ1t−1 ) , f2 ( Oθ1t−1f1 ( xt , i ) /c √ mL ; θ2t−1 ) , f3 ( ( f1 , f2 ) ; θ 3 t−1 ) 6 : end for 7 : xt = arg maxi∈ [ n ] f3 ( f1 ( xt , i ; θ 1 t−1 ) , f2 ( Oθ1t−1f1 ; θ 2 t−1 ) ; θ 3 t−1 ) 8 : Play xt and observe reward rt 9 : θ1t , θ 2 t , θ 3 t = GRADIENTDESCENT ( θ0 , { xτ } tτ=1 , { rτ } tτ=1 ) 10 : end for 11 : 12 : procedure GRADIENTDESCENT ( θ0 , { xτ } tτ=1 , { rτ } tτ=1 ) 13 : L1 = 12 ∑t τ=1 ( f1 ( xτ ; θ 1 ) − rτ ) 2 14 : θ1 , ( 0 ) = θ10 15 : for k ∈ { 1 , . . . , K1 } do 16 : θ1 , ( k ) = θ1 , ( k−1 ) − η1Oθ1 , ( k−1 ) L1 17 : end for 18 : θ̂ 1 t = θ 1 , ( K1 ) 19 : L2 = 12 ∑t τ=1 ( f2 ( Oθ1τ−1f1 ( xτ ) /c √ mL ; θ2 ) − ( rτ − f1 ( xτ ; θ1τ−1 ) ) ) 2 20 : θ2 , ( 0 ) = θ20 21 : for k ∈ { 1 , . . . , K2 } do 22 : θ2 , ( k ) = θ2 , ( k−1 ) − η2Oθ2 , ( k−1 ) L2 23 : end for 24 : θ̂ 2 t = θ 2 , ( K2 ) 25 : Determine label pt 26 : L3 = − 1t ∑t i=1 [ pt log f3 ( ( f1 , f2 ) ; θ 3 ) + ( 1− pt ) log ( 1− f3 ( ( f1 , f2 ) ; θ3 ) ) ] 27 : θ3 , ( 0 ) = θ30 28 : for k ∈ { 1 , . . . , K3 } do 29 : θ3 , ( k ) = θ3 , ( k−1 ) − η3Oθ3 , ( k−1 ) L3 30 : end for 31 : θ̂ 3 t = θ 3 , ( K3 ) 32 : Randomly choose ( θ1t , θ 2 t ) uniformly from { ( θ̂ 1 0 , θ̂ 2 0 ) , ( θ̂ 1 1 , θ̂ 2 1 ) , . . . , ( θ̂ 1 t , θ̂ 2 t ) } 33 : Randomly choose θ3t uniformly from { θ̂ 3 0 , θ̂ 3 1 , . . . , θ̂ 3 t } 34 : Return θ1t , θ 2 t , θ 3 t 35 : end procedure is defined as xt = arg max i∈ [ n ] f3 ( f1 ( xt , i ; θ 1 t−1 ) , f2 ( Oθ1t−1f1 ( xt , i ) ; θ 2 t−1 ) ; θ3t−1 ) . Note that f3 can be either linear or non-linear functions . We provide the following two forms . ( 1 ) Linear function . f3 can be formulated as a linear function with respect to f1 and f2 : f3 ( f1 , f2 ; θ 3 ) = w1f1 ( xt , i ; θ 1 ) + w2f2 ( Oθ1f1 ; θ 2 ) where w1 , w2 are two weights preset by the learner . When w1 = w2 = 1 , f3 can be thought of as UCB-type policy , where the estimated reward f1 and potential gain f2 are simply added together . In experiments , we report its empirical performance in ablation study ( Appendix B ) . ( 2 ) Non-linear function . f3 also can be formulated as a neural network to learn the mapping from ( f1 , f2 ) to the optimal arm . We transform the bandit problem into a binary classification problem . Given an arm xt , i , we define pt , i as the probability of being the optimal arm for xt , i in round t. For brevity , we denote by pt the probability of being the optimal arm for the selected arm xt in round t. According to different reward distributions , we have different approaches to determine pt . 1 . Binary reward . ∀t ∈ [ T ] , suppose rt is a binary variable over a , b ( a < b ) , it is straightforward to set : pt = 1.0 if rt = b ; pt = 0.0 , otherwise . 2 . Continuous reward . ∀t ∈ [ T ] , suppose rt is a continuous variable over the range [ a , b ] , we provide two ways to determine pt . ( 1 ) pt can be directly set as rt−ab−a . ( 2 ) The learner can set a threshold θ , ( a < θ < b ) . Then pt = 1.0 if rt > θ ; pt = 0.0 , otherwise . Therefore , with the collected training samples { ( f1 ( xτ ; θ 1 τ−1 ) , f2 ( Oθ1τ−1f1 ; θ 2 τ−1 ) ) , pτ } t τ=1 in round t , we can conduct gradient descent to update parameters of f3 ( · ; θ3 ) . Table 2 details the working structure of EE-Net . Algorithm 1 depicts the workflow of EE-Net , where the input of f2 is normalized , i.e. , Oθ1t−1f1 ( xt , i ) /c √ mL . Algorithm 1 provides a version of gradient descent ( GD ) to update EE-Net , where drawing ( θ1t , θ 2 t ) uniformly from their stored historical parameters is for the sake of analysis . One can easily extend EE-Net to stochastic GD to update the parameters incrementally . Remark 4.1 ( Network structure ) . The networks f1 , f2 , f3 can be different structures according to different applications . For example , in the vision tasks , f1 can be set up as convolutional layers ( LeCun et al. , 1995 ) . For the exploration network f2 , the input Oθ1f1 may have exploding dimensions when the exploitation network f1 becomes wide and deep , which may cause huge computation cost for f2 . To address this challenge , we can apply dimensionality reduction techniques to obtain lowdimensional vectors of Oθ1f1 . In the experiments , we use Roweis and Saul ( 2000 ) to acquire a 10-dimensional vector for Oθ1f1 and achieve the best performance among all baselines . Remark 4.2 ( Exploration direction ) . EE-Net has the ability to determine exploration direction . Given an arm xt , i , when the estimation f1 ( xt , i ) is lower than the expected reward h ( xt , i ) , the learner should make the `` upward '' exploration , i.e. , increase the chance of xt , i being explored ; When f1 ( xt , i ) is higher than h ( xt , i ) , the learner should do the `` downward '' exploration , i.e. , decrease the chance of xt , i being explored . EE-Net uses the neural network f2 to learn h ( xt , i ) − f1 ( xt , i ) ( which has positive and negative scores ) and has the ability to determine the exploration direction . In contrast , NeuralUCB will always make `` upward '' exploration and NeuralTS will randomly choose between `` upward '' exploration and `` downward '' exploration ( see selection criteria in Table 1 and more details in Appendix D ) . Remark 4.3 ( Space complexity ) . NeuralUCB and NeuralTS have to maintain the gradient outer product matrix ( e.g. , At = ∑t τ=1 Oθ1f1 ( xτ ; θ 1 τ ) Oθ1f1 ( xτ ; θ 1 τ ) > ∈ Rp×p ) and , for θ1 ∈ Rp , have a space complexity of O ( p2 ) to store the outer product . On the contrary , EE-Net does not have this matrix and only regards Oθ1f1 as the input of f2 . Thus , EE-Net reduces the space complexity from O ( p2 ) to O ( p ) .

Summary: This paper proposes EE-NeT for contextual bandits which contains two neutral networks: one for learning the underlying reward function and another for learning the potential gains of arms if explored. They prove that  EE-NeT achieves better regret bounds than state-of-the-art neural bandits algorithms for both UCB-based and TS-based. This paper also shows that EE-NeT outperforms existing linear and neural bandit approaches.