Online Multi-Armed Bandits with Adaptive Inference

1 Introduction . Stochastic multi-armed bandits [ Robbins , 1952 , Berry and Fristedt , 1985 , Bubeck and Cesa-Bianchi , 2012 , Sutton and Barto , 2018 , Lattimore and Szepesvári , 2020 ] is the simplest and most wellestablished model for sequential decision making , where a decision maker needs to adaptively select an arm at each time from a set of arms with unknown ( but fixed ) mean rewards , in the hope of maximizing the total accumulated expected returns over a certain time horizon . The existing literature has studied this problem extensively , mainly focusing on developing algorithms that deal with the exploration-exploitation trade-off , yielding at least two broad classes of algorithms that provide optimal ( sometimes up to log factors ) regret guarantees in the worst case . The first is upper confidence bound ( UCB ) based algorithms [ Lai and Robbins , 1985 , Agrawal , 1995 , Auer et al. , 2002 , Auer , 2002 , Garivier and Moulines , 2011 , Garivier and Cappé , 2011 , Carpentier et al. , 2011 ] . Reflecting “ optimism in face of uncertainty ” , UCB algorithms compute confidence bounds of the estimated mean , construct the index for each arm by adding the confidence bound to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the mean ( as the best statistically plausible mean reward ) and select the arm with the highest index . The finite-time regret bounds for these algorithms–Θ ( log T ) gap-dependent bounds and Θ ( √ T ) gap-independent bounds–are both optimal , with the latter in the worst-case sense . On the other hand , UCB algorithms are known to be sensitive to hyper-parameter tuning , and are often hard to tune for stellar performance . This stands in contrast to Thompson sampling ( TS ) based algorithms [ Thompson , 1933 , Kaufmann et al. , 2012 , Ghavamzadeh et al. , 2015 , Russo et al. , 2017 ] , an algorithm that does not require much tuning and that achieves exploration through “ probabilistic optimism in face of uncertainty ” . Further , as a result of this more nuanced exploration scheme , TS has been widely recognized to outperform UCB in empirical applications [ Scott , 2010 , Graepel et al. , 2010 , May and Leslie , 2011 , Chapelle and Li , 2011 ] . However , although an old algorithm [ Thompson , 1933 ] , its finite-time worst-case regret bound was not known at the time . Consequently , driven by its empirical performance , its theoretical guarantee was raised as an open problem in COLT 2012 [ Li and Chapelle , 2012 ] and was subsequently settled by [ Agrawal and Goyal , 2012 , 2013c ] , which yield the same minimax optimal regret guarantees ( up to log factors ) as in UCB . These developments—both the UCB/TS algorithms and the theoretical guarantees—have subsequently been successfully applied to contextual bandits1 [ Li et al. , 2010 , Filippi et al. , 2010 , Chu et al. , 2011 , Jun et al. , 2017 , Li et al. , 2017 , Agrawal and Goyal , 2013a , b , Russo and Van Roy , 2014 , 2016 , Agrawal et al. , 2017 ] . Despite this remarkably fruitful line of work , searching for better multi-armed bandit algorithms is far from over . For one thing , minimax ( i.e. , worst-case ) optimality is often too conservative a metric , and does not serve as a useful indicator of practical performance for average problems 2 . In particular , an important weakness in UCB/TS algorithms is that the true mean reward of each arm is estimated using a simple sample average , which would work if the data were i.i.d . generated . However , the data is adaptively collected and consequently , as pointed out in [ Xu et al. , 2013 , Bowden and Trippa , 2017 , Nie et al. , 2018 , Neel and Roth , 2018 , Hadad et al. , 2019 , Shin et al. , 2019 ] directly using the sample average to estimate the true mean rewards is not guaranteed to be unbiased and asymptotically normal . Intuitively , this is because arms which appear to have worse reward performance than they actually do due to randomness are sampled less often and the downward bias is not corrected . As such , the inferential quality of an estimator–when treating the underlying data as if they are i.i.d.–may be poor , thereby constraining the overall online decision making performance.3 Our goal in this paper is to leverage recent advances in the causal inference literature that tackle the problem of unbiased and asymptotically normal inference from previously collected , offline bandit data [ Hadad et al. , 2019 , Zhang et al. , 2020 , Zhan et al. , 2021 , Bibaut et al. , 2021 ] and obtain practically better online performance by harnessing the strengths of these adaptive inference estimators4 ( originally designed for offline , post-experiment analyses of adaptive experiments ) and the effective exploration-exploitation balance provided by TS , thereby designing a more effective online sequential decision making algorithm . 1.1 Our Contributions . Our contributions are threefold . First , we pinpoint the issues of the existing bandit algorithms ( Section 3.1 ) and design a new algorithm–which we call doubly-adaptive Thompson sampling ( DATS ) –by incorporating the adaptively-weighted doubly robust estimator in [ Hadad et al. , 2019 ] into the online decision making process and making the necessary modifications to the estimated variance of that estimator to render it suitable for the exploration/exploitation trade-off and ensure sufficient exploration . This algorithm mitigates the issues of poor inferential quality inherent in the sample average used in TS/UCB , while retaining the benefits of intelligent exploration ( see 1And to reinforcement learning as well , but contextual bandits and RL are not the focus of this paper . 2The fact that TS performs better in practice than UCB , even though the existing UCB bounds are tighter ( e.g . by log factors ) than those of TS , already attests to that . For instance , Audibert et al . [ 2009 ] showed that a variant of UCB has regret O ( √ KT ) in multi-armed bandits , while the best known regret bound for TS is O ( √ KT log T ) for TS with Beta prior and O ( √ KT logK ) for TS with Gaussian prior [ Agrawal and Goyal , 2017 ] . And for contextual bandits , TS is often at least a factor of d ( context dimension ) worse . 3Work concurrent to our paper sheds new light on the arm pulling properties of UCB that results in more balanced data collection under small-gap regime compared to TS , which drives the estimates from UCB-collected data towards asymptotic normality in the small-gap case [ Kalvit and Zeevi , 2021 ] . 4Such as the adaptively weighted estimator proposed by [ Luedtke and Van Der Laan , 2016 ] in the setting of i.i.d . observational data with multiple optimal policies and later generalized by [ Hadad et al. , 2019 ] in the multi-armed bandit setting . Section 3.2 for a more detailed discussion ) . Previously , inverse propensity weighting ( IPW ) has been used in bandits [ Agarwal et al. , 2014 ] , motivated , analyzed and evaluated in [ Dimakopoulou et al. , 2017 , 2019 ] and systematically benchmarked in [ Bietti et al. , 2018 ] . However , IPW has very high variance and these works did not use variance stabilizing weights , which can yield poor inferential performance–as we will see in section 4.2 . Second , we validate that the desired benefits in the design of DATS do translate into higher quality decision making . In particular , through 20 synthetic domain experiments and one semi-synthetic experiment , we demonstrate that DATS is more effective than TS and UCB . This effectiveness is shown in two metrics : one is regret , which measures the cumulative performance compared to that of the optimal oracle ; the other is sample complexity , which measures how many rounds are needed in order to identify the best arm . Under both metrics , DATS beats all existing bandit algorithms with a clear margin ( see Section 4 for a detailed presentation of results ) . Finally , to complete the picture , we show that DATS achieves this superior practical performance without giving away the comparable worst-case regret guarantees . In particular , we establish that DATS has a finite-time regret of O ( K3/2 √ T log T ) , which is minimax optimal ( up to log factors ) in the horizon T , thus matching that of TS ( K is the number of arms ) . Notably , all existing results on adaptive estimators in the inference literature are asymptotic in nature ( typically in the style of central limit theorem bounds ) , whereas our bound is finite in nature and sheds light in the arena of online decision making , rather than offline inference , on which all prior work on adaptive estimators has focused–to our knowledge . We point out that we do not attempt to be tight in K in our bound , as in the applications we have in mind ( e.g . clinical trials , web-service testing ) , including the one motivating our semi-synthetic experiment , the number of arms is typically a ( small ) constant . That said , our synthetic empirical results ( Fig . 1 ) on 20 domains ( including for large K ) demonstrate that DATS indeed has an optimal O ( √ K ) dependence . We leave tighter analysis on K for future work . 2 Problem Formulation . In stochastic multi-armed bandits , there is a finite set A of arms a ∈ A with |A| = K. At every time t , the environment generates–in an i.i.d . manner–a reward rt ( a ) for every arm a ∈ A with expectation E [ rt ( a ) ] = µa , where µa is the unknown true reward of arm a which is modified by mean-zero noise to generate rt ( a ) . The optimal arm is the arm with the maximum true reward , which is denoted by a∗ : = arg maxa∈A µa and is also unknown . When at time t the decision maker chooses arm at , only the reward of the chosen arm rt : = rt ( at ) is observed . At every time t , the decision maker employs a policy πt that maps the history of arms and rewards observed up to that time , Ht−1 = ( a0 , r0 , . . . , at−1 , rt−1 ) , to a probability distribution over the set of arms A and chooses at ∼ πt . The probability with which arm a is chosen at time t ( often referred to as propensity score of arm a at time t ) is πt , a = P ( at = a|Ht−1 ) . The goal of the decision maker is to make decisions adaptively and learn a sequence of policies ( π1 , . . . , πT ) over the duration of T time periods , so that the expected cumulative regret with respect to the optimal arm selection strategy over T time periods is minimized , where Regret ( T , π ) : = ∑T t=1 ( µa∗ − µat∼πt ) . 2.1 Baseline Algorithms : UCB and TS . Upper confidence bound ( UCB ) is an algorithm that balances the exploration/exploitation tradeoff by forming for each arm a ∈ A at every time period t an upper bound Ut , a that represents the maximum statistically plausible value of the unknown true reward µa given the history Ht−1 . Then , at time t , UCB chooses the arm with the highest bound , at = arg maxa∈A Ut , a . Hence , the policy πt of time t assigns probability one to the arm with the highest bound ( breaking ties deterministically ) and probability zero to all other arms . An example is UCB-Normal for normally distributed rewards [ Auer et al. , 2002 ] , where the upper confidence bound of arm a at time t is given by Ut , a = r̄t , a + β √ σ̂2t , a log ( t− 1 ) , where r̄t , a is the sample average and σ̂2t , a = qt , a−nt , ar̄t , a nt , a ( nt , a−1 ) is an estimate of the mean reward ’ s variance ( qt , a is the sum of squared rewards and nt , a is the number of pulls of arm a up to time t ) while β is an algorithm parameter . Thompson sampling ( TS ) is another algorithm that balances the exploration/exploitation trade-off by forming for each arm a ∈ A at every time period t a posterior distribution P ( µa ∈ ·|Ht−1 ) , drawing a sample r̃t , a from it and choosing the arm with the highest sample , at = arg maxa∈A r̃t , a . Hence , the policy πt of time t is to choose each arm with the probability that it is optimal given the history of observations Ht−1 , πt ( a ) = P ( a = argmaxa∈A r̃t , a|Ht−1 ) . An example is the TS-Normal , in which it is assumed that the true reward µa of arm a is drawn from N ( µ̂0 , a , σ̂20 , a ) , which plays the role of a prior distribution , and the realized reward of arm a at time t , rt ( a ) , is drawn from N ( µa , σ2 ) , where µa is unknown and σ is known . The posterior distribution of arm a at time t is also Normal [ Russo et al. , 2017 ] with mean µ̂t , a = E [ µa|Ht−1 ] = r̄t , aσ̂ 2 t−1 , a+µ̂t−1 , aσ 2/nt , a σ̂2t−1 , a+σ 2/nt , a and variance σ̂2t , a = E [ ( µa−µ̂t , a ) 2|Ht−1 ] = σ̂2t−1 , aσ 2/nt , a σ̂2t−1 , a+σ 2/nt , a . The sample average r̄t , a of arm ’ s a observed rewards up to time t plays a prominent role in UCB and TS , since it is used in Ut , a and P ( µa ∈ ·|Ht−1 ) respectively . However , the observations of a multi-armed bandit algorithm are adaptively collected and , as a result , they are not i.i.d. , which makes these sample averages biased .

Summary: This paper considers the multi-arm bandit problem. It starts from the observation (already present in the literature) that classical bandit algorithms such as UCB or TS estimate the mean rewards of the arms by simple empirical averages without taking into account the fact that they are adaptively sampled and are therefore biased. The paper thus proposes to make these estimators unbiased by using importance weighting (inverse propensity score weighting) as well as variance reduction tools (adaptive doubly robust estimator and clipping). Applying it to TS, they empirically demonstrate that their algorithm significantly outperforms classical TS and UCB while maintaining a theoretical worst-case regret bound of order $O(\sqrt{T})$.