Expected Improvement-based Contextual Bandits

1 INTRODUCTION . The stochastic multi-armed bandit ( Bubeck and Cesa-Bianchi , 2012 ; Lattimore and Szepesvári , 2020 ) has been extensively studied as an important model to optimize the trade-off between exploration and exploitation in sequential decision making . Among its many variants , the contextual bandit is widely used in real-world applications such as recommendation Li et al . ( 2010 ) , advertising Graepel et al . ( 2010 ) , robotic control Mahler et al . ( 2016 ) , and healthcare Greenewald et al . ( 2017 ) . In each round of a contextual bandit , the agent observes a feature vector ( the “ context ” ) for each of the K arms , pulls one of them , and in return receives a scalar reward . The goal is to maximize the cumulative reward , or minimize regret ( see our definition in Section 2 ) , in a total of T rounds . To do so , the agent must find a trade-off between exploration and exploitation . Currently , there are two popular techniques to solve this trade-off . The first technique is called the optimism in face of uncertainty which chooses the optimal action by maximizing upper-confidence bounds ( UCB ) , and the second one using Thompson Sampling ( TS ) whose basic idea is to estimate a posterior distribution on the reward , and sample an arm that maximises a random reward drawn from this distribution . A series of work has applied both UCB and TS or their variants to explore in contextual bandits with many forms of reward functions - linear and nonlinear . In the line of UCB , there are works of ( Li et al. , 2010 ; Chu et al. , 2011 ; Abbasi-yadkori et al. , 2011 ) for the linear bandits , works of ( Filippi et al. , 2010 ; Valko et al. , 2013 ) for nonlinear contextual bandits , and very recently Zhou et al . ( 2020 ) , which uses neural networks to learn the reward function . In the line of TS , Agrawal and Goyal ( 2013 ) ; Russo and Roy ( 2014 ) are for linear bandits , Russo and Roy ( 2014 ) ; Kveton et al . ( 2020 ) for generalized linear functions , and Riquelme et al . ( 2018 ) ; Zhang et al . ( 2021 ) for nonlinear bandits using deep neural networks . Besides UCB and TS , the expected improvement ( EI ) ( Močkus , 1975 ) is one of the oldest and popular techniques to handle the tradeoff between exploration and exploitation under uncertainty . Different from UCB and TS , the EI is a greedy improvement-based heuristic that samples an action offering the greatest expected improvement over the incumbent . EI enjoys wide use due to its simplicity and ability to handle uncertainty in Bayesian optimization ( Osborne , 2010 ; Zhan and Xing , 2020 ) - a problem that is closely related to infinite-arm multi-arm bandits . There are also several papers Ryzhov ( 2016 ) ; Qin et al . ( 2017 ) using EI to study the best-arm identification problem ( also known as “ pure exploration ” ) which is a finite variant of Bayesian optimization . However , for a contextual bandit which is a generalization of the standard bandit due to its reward depending on both the actions and contexts , it is yet to be seen whether EI can handle the trade-off between exploration and exploitation well . In fact , to our best knowledge , there are no EI-based algorithms even for the standard bandits in the setting of exploration and exploitation . A key challenge of analyzing EI-based algorithms comes from its improvement function involving nonlinear , nonconvex term unlike UCB and TS . This causes the difficulty of the analysis of EI . Another challenge comes from the fact that this improvement function is mainly designed to reduce uncertainty , and thus , solely focusing on exploration . These reasons together explains why theoretical analyses of EI based methods is limited , compared to that of UCB and TS . Furthermore , these few notable exceptions typically focus on bounding the simple regret in EI based Bayesian optimisation ( Bull , 2011 ) and best-arm identification ( Ryzhov , 2016 ; Qin et al. , 2017 ) . To our best knowledge , none of the existing work has investigated the cumulative regret of EI in the proper bandit setting yet . Against this background , our work proposes the first theoretical analysis of EI based methods in the classical bandit setting . In particular , we consider contextual bandits and are interested in deriving bounds on cumulative regret for our algorithms which is more suitable for contextual bandits . We note that cumulative regret bounds are stronger than the simple regret bounds and are usually more challenging to establish . In summary , our main contributions in this paper are : • We introduce and formalize Expected Improvement as a new strategy for contextual bandits creating a parallel to UCB and TS . • We propose two EI-based algorithms . The first algorithm ( LinEI ) assumes the reward function to be linear whilst the second algorithm ( NeuralEI ) is designed for the case when no assumption can be made about the reward function other than boundedness and we model it by a deep neural network . • For the linear reward function , our LinEI algorithm is able to achieve O ( d √ T ln2 ( T ) lnTδ ) regret which matches the information theoretic lower bound Ω ( d √ T ) for this problem ( up to ln ( T ) ) . For the general reward function , we prove that , under standard assumptions ( see section 4.1 ) , our NeuralEI algorithm is able to achieve Õ ( d̃ √ T ) regret , where d̃ is the “ effective ” dimension of a neural tangent kernel matrix and T is the number of rounds . • Finally , we provide an empirical evaluation of the algorithms on both synthetic functions and various benchmark datasets . Our experiments show that LinEI outperforms other baselines for linear bandits , and when the reward function is non-linear , NeuralEI outperforms all baselines . 2 PROBLEM SETTING . We consider the problem of K-arm contextual bandits . At time t = 1 , 2 , ... , the agent observes K contextual vectors xi , t ∈ Rd , then selects an arm a ( t ) and receives a reward ra ( t ) , t which has a general form as follows : ra ( t ) , t = h ( xa ( t ) , t ) + ξa ( t ) , t , where h is an unknown reward function satisfying 0 ≤ h ( x ) ≤ 1 for any x ∈ Rd , and ξa ( t ) , t is conditionally R-subGaussian for a constant R ≥ 0 , i.e. , ∀λ ∈ R , E [ eλξa ( t ) , t | { xi , t } Ki=1 ] ≤ exp ( λ 2R2 2 ) . In our setting , we assume that these context vectors may be chosen by an adversary in an adaptive manner after observing the arms played and their rewards up to time t− 1 . For the unknown function h , we consider two cases as follows : • the reward function h is linear , i.e. , h ( xt , i ) = xTt , iθ∗ , where θ∗ ∈ Rd are fixed but unknown parameters . Without loss of generality , we here assume that ||xi , t|| ≤ 1 , ||θ∗|| ≤ 1 . • the reward function h is modelled by a fully connected neural network with depth L ≥ 2 defined recursively by f ( x ; θ ) = √ mWLσ ( WL−1σ ( ... σ ( W1x ) ) ) , where σ ( x ) : = max { x , 0 } is the ReLU activation , θ = ( vec ( W1 ) ; ... ; vec ( WL ) ) ∈ Rp is the collection of parameters of the neural network , p = dm + m2 ( L − 2 ) + m. Without loss of generality , we assume that the width of each hidden layer is the same ( i.e. , m ) for convenience in analysis . We denote the gradient of the neural network function by g ( x ; θ ) = 5θf ( x ; θ ) ∈ Rp . Performance Measure . Let a∗ ( t ) denote the optimal arm at time t. The objective is to minimize the cumulative regret R ( T ) = ∑T t=1 ( x T a∗ ( t ) , tθ ∗ − xTa ( t ) , tθ ∗ ) . 3 THE LINEI ALGORITHM FOR LINEAR BANDITS . Prior and Posterior Distributions . We follow the design for priors of the reward function like TS algorithm ( Agrawal and Goyal , 2013 ) . We assume that the likelihood of reward ri , t of each arm i follows a Gaussian distribution N ( x > i , tθ∗ , v2 ) , where the variance v2 will be specified later . Let X ( t ) = λI + ∑t−1 j=1 xa ( j ) , jx > a ( j ) , j , θ̂t = X ( t ) −1 ( ∑t−1 j=1 xa ( j ) , jra ( j ) , j ) . Then if we assume that the prior for θ∗ at time t is given byN ( θ̂t , v2X2 ( t ) ) , then the posterior distribution of θ∗ at time t+ 1 is N ( θ̂t+1 , v2X2 ( t+ 1 ) ) ( see the proof in Appendix A.1 in ( Agrawal and Goyal , 2013 ) ) . Expected Improvement for Linear Contextual Bandits . We now use this posterior distribution update to define the form of the expected improvement of each arm in contextual bandits . We denote r+t = maxi∈K { xTi , tθ̂t } which is the largest mean estimate of reward among all arms at time t. We define the expected improvement of an arm i at time t as αEIi , t = Eµ∼N ( θ̂t , v2tX ( t ) −1 ) [ max { 0 , x > i , tµ− r+t } ] , ( 1 ) The αEIi , t measures the potential of arm i to improve upon the largest posterior reward mean at time t. We note that in Eq ( 1 ) , r+t plays a role as an incumbent . In Bayesian optimization , the incumbent is usually selected as the best reward value so far , or the largest reward mean so far . In our setting with contextual bandits , we choose the latter for convenient in analysis . Further , we define si , t = √ xTi , tX ( t ) −1xi , t . We see that if µ ∼ N ( θ̂t , v2X ( t ) −1 ) then the marginal distribution of xTi , tµ is Gaussian with mean x T i , tθ̂t and standard deviation vsi , t . Thus , we can express the expected improvement in closed form as follows αEIi , t = ( x T i , tθ̂t − r+t ) Φ ( x > i , tθ̂t − r + t vsi , t ) + vsi , tφ ( x > i , tθ̂t − r + t vsi , t ) , ( 2 ) where Φ ( . ) and φ ( . ) are the standard normal cdf and pdf function of the normal distribution respectively . LinEI Algorithm . We use this expected improvement to design our algorithm . We consider two particular arms among all arms : the arm selected by the EI mechanism , denoted by a ( t ) . Formally , a ( t ) = argmaxi∈ [ K ] α EI i , t ; and the best-so-far arm , denoted by ã ( t ) . Formally , ã ( t ) = argmaxi∈ [ K ] { x > i , tθ̂t } . Algorithm 1 The Linear Expected Improvement Algorithm ( LinEI ) Input : parameters C0 , β 1 : for t = 1 to T do 2 : Observe contexts { xi , t } Ki=1 3 : Set a ( t ) : = argmaxi∈Kα EI i , t , ã ( t ) = argmaxi∈K { x > i , tθ̂t } 4 : if αEIa ( t ) , t ≥ C0 tβ then 5 : a ( t ) = a ( t ) 6 : else 7 : a ( t ) = ã ( t ) 8 : end if 9 : Play arm a ( t ) , and observe reward ra ( t ) , t 10 : Update X ( t+ 1 ) = λId + ∑t j=1 xa ( j ) , jx > a ( j ) , j , θ̂t+1 = X ( t+ 1 ) −1 ( ∑t j=1 xa ( j ) , jra ( j ) , j ) 11 : end for Selecting arm a ( t ) by the EI strategy yields potentially the highest expected improvement over maxi∈ [ K ] { x > i , tθ̂t } . However , when the expected improvement ( measured by αEIi , t of all arms ) is small , an empirical observation is that the exploitation using ã ( t ) is often better than the solution a ( t ) . To make use of this intuition , we propose a modification in the EI algorithm by introducing a threshold function g ( t ) . If maxi∈ [ K ] αEIi , t ≥ g ( t ) then the algorithm selects a ( t ) , otherwise selects ã ( t ) for the pure exploitation . We consider g ( t ) = C0 tβ which is controlled by two parameters C0 , β . Our algorithm is summarized in Algorithm 1 . The threshold function is a crucial factor in achieving our optimal convergence rate for our algorithm and a relevant choice of parameters of the threshold function is discussed in the next section . Comparison with several related works . Our LinEI is related to the -Greedy strategy . The -Greedy strategy in the standard bandit plays the best arm based on current estimates with probability 1 − and otherwise explores uniformly at random . The -Greedy EI algorithm in Bayesian optimization ( Bull , 2011 ) ) plays the arm with the highest EI with probability 1− and otherwise explores uniformly at random . Unlike these algorithms , our algorithm removes the exploration at random , and uses a time-varying threshold function for arm-selection instead of a coin flip . Our algorithm is also different from the TTEI algorithm ( Qin et al. , 2017 ) for the best arm identification in a non-contextual setting . TTEI algorithm is designed to allocate a sufficient number of samples to the suboptimal arms to eliminate these arms with high confidence . While this is appropriate to solve BAI but it can not be used for contextual bandit problems . This is because it is not possible to eliminate an arm completely as in contextual bandits , this arm could be the best arm in some context . See Section C in our Supplementary Material for more discussion .

Summary: The authors study regret analysis of the expected improvement (EI), a popular but theoretically understudied technique to handle the tradeoff between exploration and exploitation in bandits.  They propose two novel EI-based algorithms for this problem, one for linear payoff and for deep neural networks. With a linear reward function, we demonstrate that our algorithm achieves a near-optimal regret. In particular, our regret bound reduces a factor of \sqrt{log T} from UCB. They also present numerical studies using the proposed algorithm.