Expected Improvement-based Contextual Bandits

1 INTRODUCTION . The stochastic multi-armed bandit ( Bubeck and Cesa-Bianchi , 2012 ; Lattimore and Szepesvári , 2020 ) has been extensively studied as an important model to optimize the trade-off between exploration and exploitation in sequential decision making . Among its many variants , the contextual bandit is widely used in real-world applications such as recommendation Li et al . ( 2010 ) , advertising Graepel et al . ( 2010 ) , robotic control Mahler et al . ( 2016 ) , and healthcare Greenewald et al . ( 2017 ) . In each round of a contextual bandit , the agent observes a feature vector ( the “ context ” ) for each of the K arms , pulls one of them , and in return receives a scalar reward . The goal is to maximize the cumulative reward , or minimize regret ( see our definition in Section 2 ) , in a total of T rounds . To do so , the agent must find a trade-off between exploration and exploitation . Currently , there are two popular techniques to solve this trade-off . The first technique is called the optimism in face of uncertainty which chooses the optimal action by maximizing upper-confidence bounds ( UCB ) , and the second one using Thompson Sampling ( TS ) whose basic idea is to estimate a posterior distribution on the reward , and sample an arm that maximises a random reward drawn from this distribution . A series of work has applied both UCB and TS or their variants to explore in contextual bandits with many forms of reward functions - linear and nonlinear . In the line of UCB , there are works of ( Li et al. , 2010 ; Chu et al. , 2011 ; Abbasi-yadkori et al. , 2011 ) for the linear bandits , works of ( Filippi et al. , 2010 ; Valko et al. , 2013 ) for nonlinear contextual bandits , and very recently Zhou et al . ( 2020 ) , which uses neural networks to learn the reward function . In the line of TS , Agrawal and Goyal ( 2013 ) ; Russo and Roy ( 2014 ) are for linear bandits , Russo and Roy ( 2014 ) ; Kveton et al . ( 2020 ) for generalized linear functions , and Riquelme et al . ( 2018 ) ; Zhang et al . ( 2021 ) for nonlinear bandits using deep neural networks . Besides UCB and TS , the expected improvement ( EI ) ( Močkus , 1975 ) is one of the oldest and popular techniques to handle the tradeoff between exploration and exploitation under uncertainty . Different from UCB and TS , the EI is a greedy improvement-based heuristic that samples an action offering the greatest expected improvement over the incumbent . EI enjoys wide use due to its simplicity and ability to handle uncertainty in Bayesian optimization ( Osborne , 2010 ; Zhan and Xing , 2020 ) - a problem that is closely related to infinite-arm multi-arm bandits . There are also several papers Ryzhov ( 2016 ) ; Qin et al . ( 2017 ) using EI to study the best-arm identification problem ( also known as “ pure exploration ” ) which is a finite variant of Bayesian optimization . However , for a contextual bandit which is a generalization of the standard bandit due to its reward depending on both the actions and contexts , it is yet to be seen whether EI can handle the trade-off between exploration and exploitation well . In fact , to our best knowledge , there are no EI-based algorithms even for the standard bandits in the setting of exploration and exploitation . A key challenge of analyzing EI-based algorithms comes from its improvement function involving nonlinear , nonconvex term unlike UCB and TS . This causes the difficulty of the analysis of EI . Another challenge comes from the fact that this improvement function is mainly designed to reduce uncertainty , and thus , solely focusing on exploration . These reasons together explains why theoretical analyses of EI based methods is limited , compared to that of UCB and TS . Furthermore , these few notable exceptions typically focus on bounding the simple regret in EI based Bayesian optimisation ( Bull , 2011 ) and best-arm identification ( Ryzhov , 2016 ; Qin et al. , 2017 ) . To our best knowledge , none of the existing work has investigated the cumulative regret of EI in the proper bandit setting yet . Against this background , our work proposes the first theoretical analysis of EI based methods in the classical bandit setting . In particular , we consider contextual bandits and are interested in deriving bounds on cumulative regret for our algorithms which is more suitable for contextual bandits . We note that cumulative regret bounds are stronger than the simple regret bounds and are usually more challenging to establish . In summary , our main contributions in this paper are : • We introduce and formalize Expected Improvement as a new strategy for contextual bandits creating a parallel to UCB and TS . • We propose two EI-based algorithms . The first algorithm ( LinEI ) assumes the reward function to be linear whilst the second algorithm ( NeuralEI ) is designed for the case when no assumption can be made about the reward function other than boundedness and we model it by a deep neural network . • For the linear reward function , our LinEI algorithm is able to achieve O ( d √ T ln2 ( T ) lnTδ ) regret which matches the information theoretic lower bound Ω ( d √ T ) for this problem ( up to ln ( T ) ) . For the general reward function , we prove that , under standard assumptions ( see section 4.1 ) , our NeuralEI algorithm is able to achieve Õ ( d̃ √ T ) regret , where d̃ is the “ effective ” dimension of a neural tangent kernel matrix and T is the number of rounds . • Finally , we provide an empirical evaluation of the algorithms on both synthetic functions and various benchmark datasets . Our experiments show that LinEI outperforms other baselines for linear bandits , and when the reward function is non-linear , NeuralEI outperforms all baselines . 2 PROBLEM SETTING . We consider the problem of K-arm contextual bandits . At time t = 1 , 2 , ... , the agent observes K contextual vectors xi , t ∈ Rd , then selects an arm a ( t ) and receives a reward ra ( t ) , t which has a general form as follows : ra ( t ) , t = h ( xa ( t ) , t ) + ξa ( t ) , t , where h is an unknown reward function satisfying 0 ≤ h ( x ) ≤ 1 for any x ∈ Rd , and ξa ( t ) , t is conditionally R-subGaussian for a constant R ≥ 0 , i.e. , ∀λ ∈ R , E [ eλξa ( t ) , t | { xi , t } Ki=1 ] ≤ exp ( λ 2R2 2 ) . In our setting , we assume that these context vectors may be chosen by an adversary in an adaptive manner after observing the arms played and their rewards up to time t− 1 . For the unknown function h , we consider two cases as follows : • the reward function h is linear , i.e. , h ( xt , i ) = xTt , iθ∗ , where θ∗ ∈ Rd are fixed but unknown parameters . Without loss of generality , we here assume that ||xi , t|| ≤ 1 , ||θ∗|| ≤ 1 . • the reward function h is modelled by a fully connected neural network with depth L ≥ 2 defined recursively by f ( x ; θ ) = √ mWLσ ( WL−1σ ( ... σ ( W1x ) ) ) , where σ ( x ) : = max { x , 0 } is the ReLU activation , θ = ( vec ( W1 ) ; ... ; vec ( WL ) ) ∈ Rp is the collection of parameters of the neural network , p = dm + m2 ( L − 2 ) + m. Without loss of generality , we assume that the width of each hidden layer is the same ( i.e. , m ) for convenience in analysis . We denote the gradient of the neural network function by g ( x ; θ ) = 5θf ( x ; θ ) ∈ Rp . Performance Measure . Let a∗ ( t ) denote the optimal arm at time t. The objective is to minimize the cumulative regret R ( T ) = ∑T t=1 ( x T a∗ ( t ) , tθ ∗ − xTa ( t ) , tθ ∗ ) . 3 THE LINEI ALGORITHM FOR LINEAR BANDITS . Prior and Posterior Distributions . We follow the design for priors of the reward function like TS algorithm ( Agrawal and Goyal , 2013 ) . We assume that the likelihood of reward ri , t of each arm i follows a Gaussian distribution N ( x > i , tθ∗ , v2 ) , where the variance v2 will be specified later . Let X ( t ) = λI + ∑t−1 j=1 xa ( j ) , jx > a ( j ) , j , θ̂t = X ( t ) −1 ( ∑t−1 j=1 xa ( j ) , jra ( j ) , j ) . Then if we assume that the prior for θ∗ at time t is given byN ( θ̂t , v2X2 ( t ) ) , then the posterior distribution of θ∗ at time t+ 1 is N ( θ̂t+1 , v2X2 ( t+ 1 ) ) ( see the proof in Appendix A.1 in ( Agrawal and Goyal , 2013 ) ) . Expected Improvement for Linear Contextual Bandits . We now use this posterior distribution update to define the form of the expected improvement of each arm in contextual bandits . We denote r+t = maxi∈K { xTi , tθ̂t } which is the largest mean estimate of reward among all arms at time t. We define the expected improvement of an arm i at time t as αEIi , t = Eµ∼N ( θ̂t , v2tX ( t ) −1 ) [ max { 0 , x > i , tµ− r+t } ] , ( 1 ) The αEIi , t measures the potential of arm i to improve upon the largest posterior reward mean at time t. We note that in Eq ( 1 ) , r+t plays a role as an incumbent . In Bayesian optimization , the incumbent is usually selected as the best reward value so far , or the largest reward mean so far . In our setting with contextual bandits , we choose the latter for convenient in analysis . Further , we define si , t = √ xTi , tX ( t ) −1xi , t . We see that if µ ∼ N ( θ̂t , v2X ( t ) −1 ) then the marginal distribution of xTi , tµ is Gaussian with mean x T i , tθ̂t and standard deviation vsi , t . Thus , we can express the expected improvement in closed form as follows αEIi , t = ( x T i , tθ̂t − r+t ) Φ ( x > i , tθ̂t − r + t vsi , t ) + vsi , tφ ( x > i , tθ̂t − r + t vsi , t ) , ( 2 ) where Φ ( . ) and φ ( . ) are the standard normal cdf and pdf function of the normal distribution respectively . LinEI Algorithm . We use this expected improvement to design our algorithm . We consider two particular arms among all arms : the arm selected by the EI mechanism , denoted by a ( t ) . Formally , a ( t ) = argmaxi∈ [ K ] α EI i , t ; and the best-so-far arm , denoted by ã ( t ) . Formally , ã ( t ) = argmaxi∈ [ K ] { x > i , tθ̂t } . Algorithm 1 The Linear Expected Improvement Algorithm ( LinEI ) Input : parameters C0 , β 1 : for t = 1 to T do 2 : Observe contexts { xi , t } Ki=1 3 : Set a ( t ) : = argmaxi∈Kα EI i , t , ã ( t ) = argmaxi∈K { x > i , tθ̂t } 4 : if αEIa ( t ) , t ≥ C0 tβ then 5 : a ( t ) = a ( t ) 6 : else 7 : a ( t ) = ã ( t ) 8 : end if 9 : Play arm a ( t ) , and observe reward ra ( t ) , t 10 : Update X ( t+ 1 ) = λId + ∑t j=1 xa ( j ) , jx > a ( j ) , j , θ̂t+1 = X ( t+ 1 ) −1 ( ∑t j=1 xa ( j ) , jra ( j ) , j ) 11 : end for Selecting arm a ( t ) by the EI strategy yields potentially the highest expected improvement over maxi∈ [ K ] { x > i , tθ̂t } . However , when the expected improvement ( measured by αEIi , t of all arms ) is small , an empirical observation is that the exploitation using ã ( t ) is often better than the solution a ( t ) . To make use of this intuition , we propose a modification in the EI algorithm by introducing a threshold function g ( t ) . If maxi∈ [ K ] αEIi , t ≥ g ( t ) then the algorithm selects a ( t ) , otherwise selects ã ( t ) for the pure exploitation . We consider g ( t ) = C0 tβ which is controlled by two parameters C0 , β . Our algorithm is summarized in Algorithm 1 . The threshold function is a crucial factor in achieving our optimal convergence rate for our algorithm and a relevant choice of parameters of the threshold function is discussed in the next section . Comparison with several related works . Our LinEI is related to the -Greedy strategy . The -Greedy strategy in the standard bandit plays the best arm based on current estimates with probability 1 − and otherwise explores uniformly at random . The -Greedy EI algorithm in Bayesian optimization ( Bull , 2011 ) ) plays the arm with the highest EI with probability 1− and otherwise explores uniformly at random . Unlike these algorithms , our algorithm removes the exploration at random , and uses a time-varying threshold function for arm-selection instead of a coin flip . Our algorithm is also different from the TTEI algorithm ( Qin et al. , 2017 ) for the best arm identification in a non-contextual setting . TTEI algorithm is designed to allocate a sufficient number of samples to the suboptimal arms to eliminate these arms with high confidence . While this is appropriate to solve BAI but it can not be used for contextual bandit problems . This is because it is not possible to eliminate an arm completely as in contextual bandits , this arm could be the best arm in some context . See Section C in our Supplementary Material for more discussion .

Summary: This paper applies expected improvement to contextual bandits to balance exploration and exploitation. The authors proposed LinEL for linear rewards and NeuralEI for general rewards using neural network for approximation. Theoretical analysis claimed that compared with UCB based approaches, LinEL's regret bound reduces a factor of $\sqrt{\log(T)}$. Empirical results validated the effectiveness of LinEL and NeuralEI.