Stochastic Multi-Armed Bandits with Control Variates

1 Introduction . Stochastic Multi-Armed Bandits ( MAB ) setting has been extensively used to study decision making under uncertainty ( Thompson , 1933 ; Bubeck et al. , 2012 ; Lattimore and Szepesvári , 2020 ) . In the classical setting , it is assumed that the arm rewards are independent of each other . After playing an arm , the learner observers independent and identically distributed reward samples . The exploration versus exploitation trade-off is a fundamental problem in the bandits setting , and a learner can accumulate more rewards if it is balanced well . A better balance can be achieved if the learner can estimate arm ’ s mean rewards with tighter confidence bounds ( Auer et al. , 2002 ; Auer and Ortner , 2010 ; Garivier and Cappé , 2011 ) and smaller variance ( Audibert et al. , 2009b ; Mukherjee et al. , 2018 ) . Any available side or auxiliary information can aid in building tighter confidence bounds . In this paper , we study the improvement in the performance that can be achieved when side information is available in the form of control variates . Any observable input that is correlated with the random variable of interest can be a Control Variate ( CV ) . When the mean of the CV is known , it becomes useful to reduce the variance of the mean estimator of the random variable . CV method is a popular variance reduction technique to improve the estimates ’ precision without altering the simulation runs or an increase in the number of samples ( Lavenberg et al. , 1982 ) . Many stochastic simulations involve inputs generated using known distributions . These inputs can be potential CVs as output could be correlated with them . It has motivated the development of rich theory to analyze the quality of the estimators resulting from CVs ( Lavenberg and Welch , 1981 ; Lavenberg et al. , 1982 ; Nelson , 1990 ) . We adopt these techniques to study the stochastic multi-armed bandits problem when CVs are available for the arms . In many real-life applications , exogenous variables influence arms ’ rewards and act as CVs . For example , consider a job aggregating platform that assigns jobs to one of the servers/workers in each slot . The jobs are of variable size , and the time to serve them depends on their size and other 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . unknown extraneous factors . The platform observes the size of each job and knows their mean value from historical data . The random job sizes correlate with service times and can be used as a CV to estimate each server ’ s mean service time . Another example is a wireless system where a transmitter can transmit packets on a channel in each slot . Successful transmission of the packets ( quality ) depends on random quantities like fading , interference , and channel noise on the selected channel . The transmitter can observe the fading value using pilot signals in each round and know their mean value from past observations . These fading values on a channel can be used as a CV to estimate the channel ’ s quality . The performance of a MAB algorithm depends on how good is the confidence interval of its mean reward estimators ( Auer et al. , 2002 ; Auer and Ortner , 2010 ; Garivier and Cappé , 2011 ) . The tightness of these intervals depends on the variance of the estimators ( Audibert et al. , 2009b ; Mukherjee et al. , 2018 ) . Naturally , estimators with smaller variance for the same number of samples result in better performance . Thus CVs can be leveraged to improve the performance of bandits algorithms with smaller confidence intervals . The reduction depends on how strongly the reward samples and associated CVs are correlated . Linear methods are widely used for variance reduction using CVs where the new samples are generated taking the weighted sum of reward samples and centered CV samples . Then these new samples are used for the estimation of the mean rewards . The choice of the weight that results in maximum variance reduction depends on the variance of CV and its correlation with reward samples . In practice , both of these quantities are unknown and need to be estimated from the observed samples . However , using the estimated weight makes the new samples highly correlated and makes the analysis of confidence intervals challenging . For multivariate normal distributions , tight confidence intervals can be constructed that hold with high probability for the mean reward estimators resulting from the linear control variates . Using these results , we develop a MAB algorithm that exploits CVs to improve the regret performance . For general distributions , we discuss resampling methods that result in unbiased estimators with better confidence intervals . Specifically , our contributions are as follows : • For the case where rewards and CVs are normally distributed , we develop an algorithm named Upper Confidence Bounds with Control Variates ( UCB-CV ) that uses the estimators based on the linear control variates . • In Section 4 , we show that the regret of UCB-CV is smaller by a factor ( 1− ρ2 ) compared to the existing algorithms when the rewards and CV have a multivariate normal distribution , where ρ is the correlation coefficient of the reward and control variates . • In Section 5 , we discuss how to adapt UCB-CV for the general distributions using estimators and associated confidence intervals based on jackknifing , splitting , and batching methods . • We validate the performance of UCB-CV on synthetically generated data in Section 6 . 1.1 Related Work . Our work uses side or auxiliary information available in the form of CVs to improve the variance of estimates . In the following , we discuss works that incorporate variance estimates and/or side information to improve the performance of multi-armed bandits algorithms . Incorporating variance estimates : Many stochastic multi-armed bandits algorithms like UCB1 ( Auer et al. , 2002 ) , DMED ( Honda and Takemura , 2010 ) , KL-UCB ( Garivier and Cappé , 2011 ) , Thompson Sampling ( Chapelle and Li , 2011 ; Agrawal and Goyal , 2012 ; Kaufmann et al. , 2012 ; Agrawal and Goyal , 2013 ) , assume that the rewards are bounded on some interval [ 0 , b ] and define index of each arm based on its estimates of the mean rewards and ignore the variance estimates . The regret of these algorithms have optimal logarithmic bound in the number of rounds ( T ) , and grows quadratic in b , i.e. , O ( ( b2/∆ ) log T ) , where ∆ is the smallest sub-optimality gap . UCB1-NORMAL by ( Auer et al. , 2002 ) uses the variance estimates in the index of the arms and when rewards are Gaussian with variance σ2 , to achieve regret is of order O ( ( σ2/∆ ) log T ) . This bound is even better than that of UCB1 when the variance is smaller than b2 even though it has considered unbounded support . Further , the authors experimentally demonstrate that the UCB-Tuned algorithm ( Auer et al. , 2002 ) , which uses variance estimates in arm indices , significantly improves regret performance over UCB1 when rewards are bounded . Extending the idea of using ‘ variance aware ’ indices , UCBV ( Audibert et al. , 2009b ) algorithm achieves regret of order O ( ( σ2/∆ + 2b ) log T ) thus confirming that making algorithms variance aware is beneficial . EUCBV ( Mukherjee et al. , 2018 ) improves the performance of UCBV using an arm elimination strategy like used in UCB-Improved ( Auer and Ortner , 2010 ) . Incorporating side information : In the applications like advertising and web search , whether or not users like a displayed item could depend on their profile . Users ’ profile is often available to the platform , which could then be used as side information . Contextual bandits ( Dani et al. , 2008 ; Rusmevichientong and Tsitsiklis , 2010 ; Li et al. , 2010 ; Filippi et al. , 2010 ; Abbasi-Yadkori et al. , 2011 ; Chu et al. , 2011 ; Li et al. , 2017 ; Jun et al. , 2017 ; Zhang et al. , 2016 ) and MAB with covariates ( Perchet and Rigollet , 2013 ) exploit such side information to learn the optimal arm . They assume a correlation structure ( linearity , GLM , etc . ) between mean reward and context where mean reward varies with side-information ( context ) . In contrast to contextual bandits , mean rewards do not vary in our unstructured setup with side information available in the form of CVs . We do not use CVs directly to decide which arm to select in each round but use them only to get better estimates of the mean rewards by exploiting their correlation with rewards . Our motivating examples from jobs scheduling and communication networks capture these unstructured bandits where CVs provide information about the rewards but do not alter their mean values . Our confidence bounds for mean reward estimators are based on the variance of estimators rather than on the variance of the reward sample , as was the case in UCBV ( Audibert et al. , 2009a ) and EUCBV ( Mukherjee et al. , 2018 ) . The samples that we use to estimate mean rewards are not independent and identically distributed . Hence , we can not use the standard Bernstein inequality to get confidence bounds on mean and variance estimates . CVs are used extensively for variance reduction ( James , 1985 ; Botev and Ridder , 2014 ; Chen and Ghahramani , 2016 ; Kreutzer et al. , 2017 ; Vlassis et al. , 2019 ) in the Monte-Carlo simulation of complex systems ( Lavenberg and Welch , 1981 ; Lavenberg et al. , 1982 ; Nelson , 1989 , 1990 ) . To the best of our knowledge , our work is the first to analyze the performance of stochastic bandits algorithms with control variates . 2 Control Variate for Variance Reduction . Let µ be the unknown quantity that needs to be estimated , and X be an unbiased estimator of µ , i.e. , E [ X ] = µ . A random variable W with known expectation ( ω ) is a CV if it is correlated with X . Linear control methods use errors in estimates of known random variables to reduce error in the estimation of an unknown random variable . For any choice of a coefficient β , define a new estimator as X̄ = X + β ( ω −W ) . It is straightforward to verify that its variance is given by Var ( X̄ ) = Var ( X ) + β2Var ( W ) − 2βCov ( X , W ) . and it is minimized by setting β to β ? = Cov ( X , W ) Var ( W ) . The minimum value of the variance is given by Var ( X̄ ) = ( 1−ρ2 ) Var ( X ) , where ρ is the correlation coefficient of X and W . Larger the correlation , the greater the variance reduction achieved by the CV . In practice , the values of Cov ( X , W ) and Var ( W ) are unknown and need to be estimated to compute the best approximation for β ? . 3 Problem Setting . We consider a stochastic multi-armed bandits problem with K arms . The set of arms is represented by [ K ] .= { 1 , 2 , . . . , K } . In round t , the environment generates a vector ( { Xt , i } i∈ [ K ] , { Wt , i } i∈ [ K ] ) . The random variable Xt , i denotes the reward of arm i in round t , and form an Independent and Identically Distributed ( IID ) sequence drawn from an unknown but fixed distribution with mean µi and variance σ2i . The random variable Wt , i is the Control Variate ( CV ) associated with the reward of arm i in round t. These random variables are drawn from an unknown but fixed distribution with mean ωi and variance σ2w , i and form an IID sequence . The learner knows the values { ωi } i∈ [ K ] but not the { σ2w , i } i∈ [ K ] . The correlation coefficient between rewards and associated control variates of an arm i is denoted by ρi . In our setup , the learner observes the reward and associated CVs from the selected arm . We refer to this new variant of the multi-armed bandits problem as Multi-Armed Bandits with Control Variates ( MAB-CV ) . The parameter vectors µσ = { µi , σ2i } i∈ [ K ] , ωσ = { ωi , σ2w , i } i∈ [ K ] , and ρ = { ρi } i∈ [ K ] identify an instance of MAB-CV problem , which we denote henceforth using P = ( µσ , ωσ , ρ ) . The collections of all MAB-CV problems are denoted by P . For a problem instance P ∈ P with mean reward vector µ , we denote the optimal arm as i ? = argmax i∈ [ K ] µi . MAB-CV problem with instance ( µσ , ωσ , ρ ) For round t : 1 . Environment generates a vector Xt = ( Xt,1 , . . . , Xt , K ) ∈ RK and Wt = ( Wt,1 , . . . , Wt , K ) ∈ RK , where E [ Xt , i ] = µi , Var ( Xt , i ) = σ2i , E [ Wt , i ] = ωi , Var ( Wt , i ) = σ2w , i , and the sequence ( Xt , i ) t≥1 and ( Wt , i ) t≥1 are IID for all i ∈ [ K ] . 2 . Learner selects an arm It ∈ [ K ] based on past observation of rewards and CVs samples from the arms till round t− 1 . 3 . Feedback and Regret : The learner observes a random reward Xt , It and associated CV Wt , It , and then incurs penalty ( µi ? − µIt ) . Let It denote the arm selected by the learner in round t based on the observation of past reward and control variate samples . The interaction between the environment and a learner is given in MAB-CV problem . We aim to learn policies that accumulate maximum reward and measure its performance by comparing its cumulative reward with that of an Oracle that plays the optimal arm in each round . Specifically , we measure the performance in terms of regret defined as follows : RT = Tµi ? − E [ T∑ t=1 Xt , It ] . ( 1 ) Note that maximizing the mean cumulative reward of a policy is the same as minimizing the policy ’ s regret . A good policy should have sub-linear expected regret , i.e. , RT /T → 0 as T →∞ . Our goal is to learn a policy that is sub-linear with small regret . To this effect , we use CVs to estimate mean rewards with sharper confidence bounds so that the learner can have a better exploration-exploitation trade-off and start playing the optimal arm more frequently early .

Summary: The paper tackles a bandit setting similar to contextual bandits, where the feedback consists of the reward obtained from playing an arm, but also an additional random variable (the control variate) which is correlated with the reward but whose mean is known to the decision maker. Unlike in contextual bandits, the value of this control variate is not available at decision time but it is only revealed after the arm's play and can thus only be used to produce better estimates of the rewards of arms. The paper proposes the UCB-CV algorithm that builds confidence intervals that exploit this additional information to produce lower regret. The algorithm is accompanied by a regret analysis and experiments demonstrating the algorithm's performance and the decrease in regret that can be achieved based on the correlation coefficient between the reward and the control variates.