EE-Net: Exploitation-Exploration Neural Networks in Contextual Bandits

√ T log T ) regret , which is tighter than existing state-of- the-art neural bandit algorithms ( O ( √ T log T ) for both UCB-based and TS-based ) . Through extensive experiments on four real-world datasets , we show that EE-Net outperforms existing linear and neural contextual bandit approaches . 1 INTRODUCTION . The stochastic contextual multi-armed bandit ( MAB ) ( Lattimore and Szepesvári , 2020 ) has been studied for decades in machine learning community to solve sequential decision making , with applications in online advertising ( Li et al. , 2010 ) , personal recommendation ( Wu et al. , 2016 ; Ban and He , 2021b ) , etc . In the standard contextual bandit setting , a set of n arms are presented to a learner in each round , where each arm is represented by a context vector . Then by certain strategy , the learner selects and plays one arm , receiving a reward . The goal of this problem is to maximize the cumulative rewards of T rounds . MAB algorithms have principled approaches to address the trade-off between Exploitation and Exploration ( EE ) , as the collected data from past rounds should be exploited to get good rewards but also under-explored arms need to be explored with the hope of getting even better rewards . The most widely-used approaches for EE trade-off can be classified into three main techniques : Epsilon-greedy ( Langford and Zhang , 2008 ) , Thompson Sampling ( TS ) ( Thompson , 1933 ) , and Upper Confidence Bound ( UCB ) ( Auer , 2002 ; Ban and He , 2020 ) . Linear bandits ( Li et al. , 2010 ; Dani et al. , 2008 ; Abbasi-Yadkori et al. , 2011 ) , where the reward is assumed to be a linear function with respect to arm vectors , have been well studied and succeeded both empirically and theoretically . Given an arm , ridge regression is usually adapted to estimate its reward based on collected data from past rounds . UCB-based algorithms ( Li et al. , 2010 ; Chu et al. , 2011 ; Wu et al. , 2016 ; Ban and He , 2021b ) calculate an upper bound for the confidence ellipsoid of estimated reward and determine the arm according to the sum of estimated reward and UCB . TS-based algorithms ( Agrawal and Goyal , 2013 ; Abeille and Lazaric , 2017 ) formulate each arm as a posterior distribution where mean is the estimated reward and choose the one with the maximal sampled reward . However , the linear assumption regarding the reward may not be true in real-world applications ( Valko et al. , 2013 ) . To learn non-linear reward functions , recent works have utilized deep neural networks to learn the underlying reward functions , thanks to its powerful representation ability . Considering the past selected arms and received rewards as training samples , a neural network f1 is built for exploitation . Zhou et al . ( 2020 ) computes a gradient-based upper confidence bound with respect to f1 and uses UCB strategy to select arms . Zhang et al . ( 2021 ) formulates each arm as a normal distribution where the mean is f1 and deviation is calculated based on gradient of f1 , and then uses the TS strategy to choose arms . Both Zhou et al . ( 2020 ) and Zhang et al . ( 2021 ) achieve the near-optimal regret bound of O ( √ T log T ) . In this paper , we propose a novel neural-based exploration strategy , named `` EE-Net '' . Similar to other neural bandits , EE-Net has an exploitation network f1 to estimate rewards for each arm . The crucial difference from existing works is that EE-Net has an exploration network f2 to predict the potential gain for each arm compared to current reward estimate . The input to the exploration network is the gradient of f1 and the ground-truth is residual difference between the true received reward and the estimated reward from f1 . The strategy is inspired by recent advances in the neural UCB strategies ( Ban et al. , 2021 ; Zhou et al. , 2020 ) . Finally , a decision-maker f3 is constructed to select arms . f3 has two modes : linear or nonlinear . In linear mode , f3 is a linear combination of f1 and f2 , inspired by the UCB strategy . In the nonlinear mode , f3 is formulated as a neural network with input ( f1 , f2 ) and the goal is to learn the probability of being an optimal arm for each arm . Table 1 summarizes the selection criterion difference of EE-Net from other neural bandit algorithms . To sum up , the contributions of this paper can be summarized as follows : 1 . We propose a novel neural exploration strategy , EE-Net , where a neural network is assigned to learn the potential gain compared to the current reward estimate . 2 . Under standard assumptions of over-parameterized neural networks , we prove that EENet can achieve the regret upper bound of O ( √ T log T ) , which is tighter than existing state-of-the-art conteextual bandit algorithms . 3 . We conduct extensive experiments on four real-world datasets , showing that EE-Net out- performs baselines including linear and neural versions of -greedy , TS , and UCB , and establishing EE-net as the new state-of-the-art exploration strategy . Next , we discuss the problem definition in Sec.3 , elaborate on the proposed EE-Net in Sec.4 , and present our theoretical analysis in Sec.5 . In the end , we provide the empirical evaluation ( Sec.6 ) and conclusion . 2 RELATED WORK . Constrained Contextual bandits . The common constrain placed on the reward function is the linear assumption , usually calculated by ridge regression ( Li et al. , 2010 ; Abbasi-Yadkori et al. , 2011 ; Valko et al. , 2013 ; Dani et al. , 2008 ) . The linear UCB-based bandit algorithms ( Abbasi-Yadkori et al. , 2011 ; Li et al. , 2016 ) and the linear Thompson Sampling ( Agrawal and Goyal , 2013 ; Abeille and Lazaric , 2017 ) can achieve successful performance and the near-optimal regret bound of Õ ( √ T ) . To break the linear assumption , Filippi et al . ( 2010 ) generalizes the reward function to a composition of linear and non-linear functions and then adopt a UCB-based algorithm to deal with it ; Bubeck et al . ( 2011 ) imposes the Lipschitz property on reward metric space and constructs a hierarchical optimistic optimization to make selections ; Valko et al . ( 2013 ) embeds the reward function into Reproducing Kernel Hilbert Space and proposes the kernelized TS/UCB bandit algorithms . Neural Bandits . To learn non-linear reward functions , deep neural networks have been adapted to bandits with various variants . Riquelme et al . ( 2018 ) ; Lu and Van Roy ( 2017 ) build L-layer DNN to learn the arm embeddings and apply Thompson Sampling on the last layer for exploration . Zhou et al . ( 2020 ) first introduces a provable neural-based contextual bandit algorithm with a UCB exploration strategy and then Zhang et al . ( 2021 ) extends the neural network to Thompson sampling framework . Their regret analysis is built on recent advances on the convergence theory in over-parameterized neural networks ( Du et al. , 2019 ; Allen-Zhu et al. , 2019 ) and utilizes Neural Tangent Kernel ( Jacot et al. , 2018 ; Arora et al. , 2019 ) to construct connections with linear contextual bandits ( AbbasiYadkori et al. , 2011 ) . Ban and He ( 2021a ) further adopts convolutional neural networks with UCB exploration aiming for visual-aware applications . Xu et al . ( 2020 ) performs UCB-based exploration on the last layer of neural networks to reduce the computational cost brought by gradient-based UCB . Different from the above existing works , EE-Net keeps the powerful representation ability of neural networks to learn reward function and first assigns another neural network to determine exploration . 3 PROBLEM DEFINITION . We consider the standard contextual multi-armed bandit with the known number of rounds T ( Zhou et al. , 2020 ; Zhang et al. , 2021 ) . In each round t ∈ [ T ] , where the sequence [ T ] = [ 1 , 2 , . . . , T ] , the learner is presented with n arms , Xt = { xt,1 , . . . , xt , n } , in which each arm is represented by a feature vector xt , i ∈ Rd for each i ∈ [ n ] . After playing one arm xt , i , its reward rt , i is assumed to be generated by the function : rt , i = h ( xt , i ) + ηt , i , ( 3.1 ) where the unknown reward function h ( xt , i ) can be either linear or non-linear and the noise ηt , i is drawn from certain distribution with expectation E [ ηt , i ] = 0 . Following many existing works ( Zhou et al. , 2020 ; Ban et al. , 2021 ; Zhang et al. , 2021 ) , we consider bounded rewards , rt , i ∈ [ a , b ] . For the brevity , we denote the selected arm in round t by xt and the reward received in t by rt . The expected regret of T rounds is defined as : RT = E [ T∑ t=1 ( r∗t − rt ) ] , ( 3.2 ) where E [ r∗t | Xt ] = maxi∈ [ n ] h ( xt , i ) is the maximal expected reward in the round t. The goal of this problem is to minimize RT by certain selection strategy . Notation . We denote by { xi } ti=1 the sequence ( x1 , . . . , xt ) . We use ‖v‖2 to denote the Euclidean norm for a vector v , and ‖W‖2 and ‖W‖F to denote the spectral and Frobenius norm for a matrix W. We use 〈· , ·〉 to denote the standard inner product between two vectors or two matrices . We may use Oθ1t f1 ( xt , i ) or Oθ1t f1 to represent the gradient Oθ1t f1 ( xt , i ; θ 1 t ) for brevity . We use { xτ , rτ } tτ=1 to represent the collected data up to round t . 4 PROPOSED METHOD : EE-NET . EE-Net is composed of three components . The first component is the exploitation network , f1 ( · ; θ1 ) , which focuses on learning the unknown reward function h based on the data collected in past rounds . The second component is the exploration network , f2 ( · ; θ2 ) , which focuses on characterizing the level of exploration needed for each arm in the present round . The third component is the decision-maker , f3 , which focuses on suitably combining the outputs of the exploitation and exploration networks leading to the arm selection . Published as a conference paper at ICLR 2022 1 ) Exploitation Net . The exploitation net f1 is a neural network which learns the mapping from arms to rewards . In round t , denote the network by f1 ( · ; θ1t−1 ) , where the superscript of θ 1 t−1 is the index of network and the subscript represents the round where the parameters of f1 finished the last update . Given an arm xt , i , i ∈ [ n ] , f1 ( xt , i ; θ1t−1 ) is considered the `` exploitation score '' for xt , i . By some criterion , after playing arm xt , we receive a reward rt . Therefore , we can conduct gradient descent to update θ1 based on the collected training samples { xτ , rτ } tτ=1 and denote the updated parameters by θ1t . 2 ) Exploration Net . Our exploration strategy is inspired by existing UCB-based neural bandits ( Zhou et al. , 2020 ; Ban et al. , 2021 ) . Based on the Lemma 5.2 in ( Ban et al. , 2021 ) , given an arm xt , i , with probability at least 1− δ , we have the following UCB form : |h ( xt , i ) − f1 ( xt , i ; θ1t−1 ) | ≤ Ψ ( Oθ1t−1f1 ( xt , i ; θ 1 t−1 ) ) , ( 4.1 ) where h is defined in Eq . ( 3.1 ) and Ψ is an upper confidence bound represented by a function with respect to the gradient Oθ1t−1f1 ( see more details and discussions in Appendix D ) . Then we have the following definition . Definition 4.1 . In round t , given an arm xt , i , we define h ( xt , i ) − f1 ( xt , i ; θ1t−1 ) as the `` expected potential gain '' for xt , i and rt , i − f1 ( xt , i ; θ1t−1 ) as the `` potential gain '' for xt , i . Let yt , i = rt , i − f1 ( xt , i ; θ1t−1 ) . When yt , i > 0 , the arm xt , i has positive potential gain compared to the estimated reward f1 ( xt , i ; θ1t−1 ) . A large positive yt , i makes the arm more suitable for exploration , whereas a small ( or negative ) yt , i makes the arm unsuitable for exploration . Recall that traditional approaches such as UCB intend to estimate such potential gain yt , i using standard tools , e.g. , Markov inequality , Hoeffding bounds , etc. , from large deviation bounds . Instead of calculating a large-deviation based statistical bound for yt , i , we use a neural network f2 ( · ; θ2 ) to represent Ψ , where the input is Oθ1t−1f1 ( xt , i ) and the ground truth is rt , i−f1 ( xt , i ; θ 1 t−1 ) . Adopting gradient Oθ1t−1f1 ( xt , i ) as the input also is due to the fact that it incorporates two aspects of information : the feature of the arm and the discriminative information of f1 . Moreover , in the upper bound of NeuralUCB or the variance of NeuralTS , there is a recursive term At−1 = I + ∑t−1 τ=1∇θ1τ−1f1 ( xτ ) ∇θ1τ−1f1 ( xτ ) > which is a function of past gradients up to ( t− 1 ) and incorporates relevant historical information . On the contrary , in EE-Net , the recursive term which depends on past gradients is θ2t−1 in the exploration network f2 because we have conducted gradient descent for θ2t−1 based on { ∇θ1τ−1f1 ( xτ ) } t−1 τ=1 . Therefore , this form θ 2 t−1 is similar to At−1 in neuralUCB/TS , but EE-net does not ( need to ) make a specific assumption about the functional form of past gradients , and is also more memory-efficient . To sum up , in round t , we consider f2 ( Oθ1t−1f1 ( xt , i ) ; θ 2 t−1 ) as the `` exploration score '' of xt , i , because it indicates the potential gain of xt , i compared to our current exploitation score f1 ( xt , i ; θ1t−1 ) . Therefore , after receiving the reward rt , we can use gradient descent to update θ2 based on collected training samples { Oθ1τ−1f1 ( xτ ) , rτ − f1 ( xτ ; θ 1 τ−1 } tτ=1 . We also provide other two heuristic forms for f2 ’ s ground-truth label : |rt , i − f1 ( xt , i ; θ1t−1 ) | and ReLU ( rt , i − f1 ( xt , i ; θ 1 t−1 ) ) . We compare them in an ablation study in Appendix B . 3 ) Decision-maker . In round t , given an arm xt , i , i ∈ [ n ] , with the computed exploitation score f1 ( xt , i ; θ 1 t−1 ) and exploration score f2 ( Oθ1t−1f1 ; θ 2 t−1 ) , we use a function f3 ( f1 , f2 ; θ 3 ) to trade off between exploitation and exploration and compute the final score for xt , i . The selection criterion Algorithm 1 EE-Net Input : f1 , f2 , f3 , T ( number of rounds ) , η1 ( learning rate for f1 ) , η2 ( learning rate for f2 ) , η3 ( learning rate for f3 ) , K1 ( number of iterations for f1 ) , K2 ( number of iterations for f2 ) , K3 ( number of iterations for f3 ) , c ( normalization parameter ) 1 : Initialize θ10 , θ 2 0 , θ 3 0 ; θ̂ 1 0 = θ 1 0 , θ̂ 2 0 = θ 2 0 , θ̂ 3 0 = θ 3 0 2 : for t = 1 , 2 , . . . , T do 3 : Observe n arms { xt,1 , . . . , xt , n } 4 : for each i ∈ [ n ] do 5 : Compute f1 ( xt , i ; θ1t−1 ) , f2 ( Oθ1t−1f1 ( xt , i ) /c √ mL ; θ2t−1 ) , f3 ( ( f1 , f2 ) ; θ 3 t−1 ) 6 : end for 7 : xt = arg maxi∈ [ n ] f3 ( f1 ( xt , i ; θ 1 t−1 ) , f2 ( Oθ1t−1f1 ; θ 2 t−1 ) ; θ 3 t−1 ) 8 : Play xt and observe reward rt 9 : θ1t , θ 2 t , θ 3 t = GRADIENTDESCENT ( θ0 , { xτ } tτ=1 , { rτ } tτ=1 ) 10 : end for 11 : 12 : procedure GRADIENTDESCENT ( θ0 , { xτ } tτ=1 , { rτ } tτ=1 ) 13 : L1 = 12 ∑t τ=1 ( f1 ( xτ ; θ 1 ) − rτ ) 2 14 : θ1 , ( 0 ) = θ10 15 : for k ∈ { 1 , . . . , K1 } do 16 : θ1 , ( k ) = θ1 , ( k−1 ) − η1Oθ1 , ( k−1 ) L1 17 : end for 18 : θ̂ 1 t = θ 1 , ( K1 ) 19 : L2 = 12 ∑t τ=1 ( f2 ( Oθ1τ−1f1 ( xτ ) /c √ mL ; θ2 ) − ( rτ − f1 ( xτ ; θ1τ−1 ) ) ) 2 20 : θ2 , ( 0 ) = θ20 21 : for k ∈ { 1 , . . . , K2 } do 22 : θ2 , ( k ) = θ2 , ( k−1 ) − η2Oθ2 , ( k−1 ) L2 23 : end for 24 : θ̂ 2 t = θ 2 , ( K2 ) 25 : Determine label pt 26 : L3 = − 1t ∑t i=1 [ pt log f3 ( ( f1 , f2 ) ; θ 3 ) + ( 1− pt ) log ( 1− f3 ( ( f1 , f2 ) ; θ3 ) ) ] 27 : θ3 , ( 0 ) = θ30 28 : for k ∈ { 1 , . . . , K3 } do 29 : θ3 , ( k ) = θ3 , ( k−1 ) − η3Oθ3 , ( k−1 ) L3 30 : end for 31 : θ̂ 3 t = θ 3 , ( K3 ) 32 : Randomly choose ( θ1t , θ 2 t ) uniformly from { ( θ̂ 1 0 , θ̂ 2 0 ) , ( θ̂ 1 1 , θ̂ 2 1 ) , . . . , ( θ̂ 1 t , θ̂ 2 t ) } 33 : Randomly choose θ3t uniformly from { θ̂ 3 0 , θ̂ 3 1 , . . . , θ̂ 3 t } 34 : Return θ1t , θ 2 t , θ 3 t 35 : end procedure is defined as xt = arg max i∈ [ n ] f3 ( f1 ( xt , i ; θ 1 t−1 ) , f2 ( Oθ1t−1f1 ( xt , i ) ; θ 2 t−1 ) ; θ3t−1 ) . Note that f3 can be either linear or non-linear functions . We provide the following two forms . ( 1 ) Linear function . f3 can be formulated as a linear function with respect to f1 and f2 : f3 ( f1 , f2 ; θ 3 ) = w1f1 ( xt , i ; θ 1 ) + w2f2 ( Oθ1f1 ; θ 2 ) where w1 , w2 are two weights preset by the learner . When w1 = w2 = 1 , f3 can be thought of as UCB-type policy , where the estimated reward f1 and potential gain f2 are simply added together . In experiments , we report its empirical performance in ablation study ( Appendix B ) . ( 2 ) Non-linear function . f3 also can be formulated as a neural network to learn the mapping from ( f1 , f2 ) to the optimal arm . We transform the bandit problem into a binary classification problem . Given an arm xt , i , we define pt , i as the probability of being the optimal arm for xt , i in round t. For brevity , we denote by pt the probability of being the optimal arm for the selected arm xt in round t. According to different reward distributions , we have different approaches to determine pt . 1 . Binary reward . ∀t ∈ [ T ] , suppose rt is a binary variable over a , b ( a < b ) , it is straightforward to set : pt = 1.0 if rt = b ; pt = 0.0 , otherwise . 2 . Continuous reward . ∀t ∈ [ T ] , suppose rt is a continuous variable over the range [ a , b ] , we provide two ways to determine pt . ( 1 ) pt can be directly set as rt−ab−a . ( 2 ) The learner can set a threshold θ , ( a < θ < b ) . Then pt = 1.0 if rt > θ ; pt = 0.0 , otherwise . Therefore , with the collected training samples { ( f1 ( xτ ; θ 1 τ−1 ) , f2 ( Oθ1τ−1f1 ; θ 2 τ−1 ) ) , pτ } t τ=1 in round t , we can conduct gradient descent to update parameters of f3 ( · ; θ3 ) . Table 2 details the working structure of EE-Net . Algorithm 1 depicts the workflow of EE-Net , where the input of f2 is normalized , i.e. , Oθ1t−1f1 ( xt , i ) /c √ mL . Algorithm 1 provides a version of gradient descent ( GD ) to update EE-Net , where drawing ( θ1t , θ 2 t ) uniformly from their stored historical parameters is for the sake of analysis . One can easily extend EE-Net to stochastic GD to update the parameters incrementally . Remark 4.1 ( Network structure ) . The networks f1 , f2 , f3 can be different structures according to different applications . For example , in the vision tasks , f1 can be set up as convolutional layers ( LeCun et al. , 1995 ) . For the exploration network f2 , the input Oθ1f1 may have exploding dimensions when the exploitation network f1 becomes wide and deep , which may cause huge computation cost for f2 . To address this challenge , we can apply dimensionality reduction techniques to obtain lowdimensional vectors of Oθ1f1 . In the experiments , we use Roweis and Saul ( 2000 ) to acquire a 10-dimensional vector for Oθ1f1 and achieve the best performance among all baselines . Remark 4.2 ( Exploration direction ) . EE-Net has the ability to determine exploration direction . Given an arm xt , i , when the estimation f1 ( xt , i ) is lower than the expected reward h ( xt , i ) , the learner should make the `` upward '' exploration , i.e. , increase the chance of xt , i being explored ; When f1 ( xt , i ) is higher than h ( xt , i ) , the learner should do the `` downward '' exploration , i.e. , decrease the chance of xt , i being explored . EE-Net uses the neural network f2 to learn h ( xt , i ) − f1 ( xt , i ) ( which has positive and negative scores ) and has the ability to determine the exploration direction . In contrast , NeuralUCB will always make `` upward '' exploration and NeuralTS will randomly choose between `` upward '' exploration and `` downward '' exploration ( see selection criteria in Table 1 and more details in Appendix D ) . Remark 4.3 ( Space complexity ) . NeuralUCB and NeuralTS have to maintain the gradient outer product matrix ( e.g. , At = ∑t τ=1 Oθ1f1 ( xτ ; θ 1 τ ) Oθ1f1 ( xτ ; θ 1 τ ) > ∈ Rp×p ) and , for θ1 ∈ Rp , have a space complexity of O ( p2 ) to store the outer product . On the contrary , EE-Net does not have this matrix and only regards Oθ1f1 as the input of f2 . Thus , EE-Net reduces the space complexity from O ( p2 ) to O ( p ) .

Summary: This paper presents a neural net based bandit approach with a novel exploration strategy. Specifically, the solution uses an exploitation network to estimate rewards for each arm and an exploration network to predict the potential gain compared to current reward estimate. It then uses a decision maker to select arms using one of the linear or nonlinear modes. The key differentiator/strength of the paper is using neural net based exploration strategy and demonstrating that the proposed solution has a tighter bound than the existing state-of-the-art bandit algorithms.