Breaking the Moments Condition Barrier: No-Regret Algorithm for Bandits with Super Heavy-Tailed Payoffs

1 Introduction . Multi-armed bandit ( MAB ) problems have been introduced by Robbins ( Robbins , 1952 ) , and have since become a standard model for modeling sequential decision-making problems . In an MAB instance , there are finite number of arms , pulling each of which an agent receives a random reward ( payoff ) with unknown distribution . An agent then aims to maximize the received rewards by pulling the arms strategically for a number of times . The MAB problems frequently arise in practice : e.g. , clinical trials and online advertising . When the number of arms becomes infinite in practice , the stochastic linear bandit ( Abe and Long , 1999 ; Auer , 2002 ; Dani et al. , 2008 ) generalizes the classical MAB by assuming the underlying reward distribution possesses a linear structure . Linear bandits achieve tremendous success such as online advertisement and recommendation systems ( Li et al. , 2010 ; Chu et al. , 2011 ; Li et al. , 2016 ) . ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Due to the online learning nature of a bandit problem , we measure the performance of an agent via regret , which measures the differences of the rewards collected from the best arm to those collected from the agent . When the reward distribution is benign , e.g. , with sub-Gaussian tails† , there are a number of efficient algorithms ( Bubeck and Cesa-Bianchi , 2012 ; Lattimore and Szepesvári , 2020 ) , which obtain a worst-case regret bound of the form Õ ( √ AT ) ‡ , where A is the number of arms and T is the total number of arm pulls . Any algorithm with regret bound sublinear in T is effectively learning as its average regret tends to 0 when T →∞ . In the linear setting , a number of results ( Dani et al. , 2008 ; Abbasi-Yadkori et al. , 2011 ) achieve Õ ( poly ( d ) √ T ) regret bounds , eliminating the dependence on A . Here d is the ambient dimension of the problem . A related performance measure is about the number of pulls to identify the best arm ( Soare et al. , 2014 ; Tao et al. , 2018 ; Jedra and Proutiere , 2020 ) . We note that usually a regret minimization algorithm can be converted to identify the best arm with high probability . Nevertheless , in many practical scenarios , we encounter non-sub-Gaussian noises in the observed payoffs , e.g. , the price fluctuations in financial markets ( Cont and Bouchaud , 2000 ; Rachev , 2003 ; Hull , 2012 ) , and/or fluctuations of neural oscillations ( Roberts et al. , 2015 ) . In such scenarios , the previously mentioned algorithms may fail . To tackle this problem , Bubeck et al . ( 2013 ) makes the first attempt to study the stochastic MAB problem with heavy-tailed noise . Specifically , for the rewards with a finite second order moment , by utilizing more robust statistical estimators , Bubeck et al . ( 2013 ) achieves regret bound of the same order as in the bounded/sub-Gaussian loss setting . Then , Medina and Yang ( 2016 ) ; Shao et al . ( 2018 ) ; Xue et al . ( 2020 ) study the heavy-tailed linear bandits . They consider a general characterization of heavy-tailed payoffs in bandits , where the reward takes the form r = µ+ η , where µ is an unknown but fixed number and η is a random noise , whose distribution has a finite moment of order 1 + . Here ∈ ( 0 , 1 ] . For this setting , they establish a sublinear regret bound Õ ( T 1 1+ ) . Unfortunately , when = 0 , these regret bounds will be linear in T , failing to learn in such situations . To further account for many such real-world scenarios , where the payoff noise has super-heavy tails ( e.g. , only ( 1 + ) -th moment for ∈ ( 0 , 1 ) exists , or Cauchy distribution whose mean does not exist ) , new algorithms need to be developed : Can we design an efficient algorithm that provably learns for bandits with super heavy-tailed payoffs ? In this paper , we give the affirmative answer to this question . Without loss of generality , we consider the linear setting , which includes MAB as a special case . In this setting , each arm is viewed as a vector in Rd . The random reward of the arm x is specified as θ > x+η , where θ ∈ Rd is an unknown but fixed vector and η is a super heavy-tailed symmetric random noise such that Pr ( |η| > y ) ≤ 1/yα for any y > 0 and some α > 0 . One of the key challenges in this setting is that the mean of y may not exist . The previous robust mean estimators ( Bubeck et al. , 2013 ) , such as truncated empirical mean and median of means ( Bubeck et al. , 2013 ; Medina and Yang , 2016 ; Shao et al. , 2018 ; Xue et al. , 2020 ) which require the estimation of the mean , can not effectively handle this super heavy-tailed noise . On the other hand , since the mean does not exist , we are also required to measure the performance of the agent with high-probability pseudo-regret ( defined in Section 2.2 ) . To tackle these challenges , we propose a novel estimator : mean of medians . We then present a generic algorithmic framework to apply it in any existing bandit algorithm . Below , we summarize our contributions : • We propose a novel robust statistical estimator : mean of medians . Specifically , we simply split ñ samples into k blocks and takes the mean of the median in each block . Theoretically , we can prove that , by utilizing ñ ( α ) samples , the super heavy-tailed noise is reduced to the bounded noise with high probability . Here ñ ( α ) is a constant which depends on α . • For the super heavy-tailed linear bandits , we propose a new algorithmic framework . In detail , by simply combing the above mean of medians estimator and an arbitrary provably efficient bandit algorithm , we obtain a new algorithm that can be proved efficient for regret minimization problems and best arm identification problems . Our obtained sample bounds and regret bounds can be nearly optimal . †For any ζ > 0 , a random variable X is said to be ζ-sub-Gaussian if it holds that E [ et ( X−E [ X ] ) ] ≤ eζ 2t2/2 for any t > 0 . ‡Õ ( · ) ignores logarithm factors . • We instantiate our framework with Student ’ s t-noises and compare with previous methods . Our experiments demonstrate our method can significantly outperform existing algorithms in these environments , and is strictly consistent with our theoretical guarantees . 1.1 Related Works . A line of recent work ( Bubeck et al. , 2013 ; Medina and Yang , 2016 ; Shao et al. , 2018 ; Xue et al. , 2020 ) on the heavy-tailed MAB or linear bandits uses the truncated empirical mean and median of means as the robust estimators . However , without assuming the finite moments of order 1 + for some ∈ ( 0 , 1 ] , it remains unclear whether one can attain equivalent regret/sample complexity for the more heavy-tailed setting , e.g , the noise of the payoff follows a Student ’ s t-distribution , or whether sublinear regret algorithms of any form are even possible at all . In comparison , by incorporating the mean of medians estimator , we can design a general provably efficient algorithmic framework for the super heavy-tailed linear bandits . Our work also adds to the vast body of existing literature on the regret minimization problem on linear bandits ( Auer , 2002 ; Dani et al. , 2008 ; Rusmevichientong and Tsitsiklis , 2010 ; Abbasi-Yadkori et al. , 2011 ; Lattimore and Szepesvari , 2017 ; Combes et al. , 2017 ) . A remarkable analysis is given by Auer ( 2002 ) who builds confidence regions for the true model parameter and then optimistically selects the action minimizing the loss over these sets . In the theoretical view , for the setting where the arm set is finite , Auer ( 2002 ) establishes a Õ ( √ dT ) regret bound . Then , Dani et al . ( 2008 ) ; Rusmevichientong and Tsitsiklis ( 2010 ) ; Abbasi-Yadkori et al . ( 2011 ) construct the confidence ellipsoids for the setting where the arm set is infinite and establish a Õ ( d √ T ) regret bound . Our work is also closely related to another line of work ( Soare et al. , 2014 ; Soare , 2015 ; Garivier and Kaufmann , 2016 ; Tao et al. , 2018 ; Xu et al. , 2018 ; Fiez et al. , 2019 ; Jedra and Proutiere , 2020 ) on the best arm identification problem for linear bandits . This problem is also referred as pure exploration problem since there is no price to be paid for exploring and thus we don ’ t need to carefully balance exploration against exploitation . Hence , an algorithm that is optimal for the best arm identification problem may be suboptimal for the regret minimization since its exploration is too aggressive . The reverse is also true because the exploration for the regret minimization algorithm might be too slow . 2 Preliminaries . 2.1 Notation . For a positive integer K , we use [ K ] to denote { 1 , 2 , · · · , K } . For r ∈ R , its absolute value is |r| , its ceiling integer is dre , and its floor integer is brc . Also , let 1 { · } be the indicator function . 2.2 Linear Bandits . We consider a linear bandit problem specified by a tuple ( X , θ ) , where X is the set of arms , and θ ∈ Rd is an unknown but fixed parameter with ‖θ‖2 ≤ 1 . Without loss of generality , we assume that X ⊂ Rd and ‖x‖ ≤ 1 for any x ∈ X . At each round t , a learner chooses an action xt ∈ X and observes the reward rt = θ > xt + ηt , where ηt is a symmetric and independent noise . We denote x∗ ∈ argmaxx∈X θ > x as an optimal arm . Note that the linear setting includes the MAB problem as a special case . In the MAB setting , each arm a ∈ X is corresponding to a |X |-dimensional standard unit vector ( all entries are 0 except for the a-th entry ) . In the online setting , i.e. , an agent interacts with the bandit instance for at most T ≥ 1 rounds , the performance of the agent is measured by the cumulative ( pseudo ) regret , which is defined by Regret ( T ) = T∑ t=1 ( θ > x∗ − θ > xt ) . In order to obtain a small regret , the agent needs to figure out a vector close to θ , even with the presence of the noise ηt . A related performance measure is the sample complexity of identifying a near-optimal arm . In this case , an agent is asked to output an arm x̂ such that |θ > x∗ − θ > x̂| ≤ with probability at least 1− δ . The sample complexity of the algorithm is measured by the smallest number of steps the agent interacts with the bandit instance . If Regret ( T ) of an algorithm is sublinear , then we can use standard technique to covert it into an algorithm that outputs an -accurate arm in time T such that Regret ( T ) /T = O ( ) . When = 0 , the problem reduces to the best arm identification problem . Note that best arm identification is usually not possible for a continuous action space X .

Summary: The paper addresses the problem where the noise has super-heavy tail, in the framework of online learning. In the previous works, it is assumed that the 1+\epsilon moment of the noise exists, where $\epsilon >0$. However, in this paper, a polynomially decaying tail bound is only assumed, which coves the setting, when the first moment does not exist. Hence, the previous methods like median of means cannot be used. Instead, here the authors develop mean of medians algorithm. With this, the paper first converts the super-heavy tailed noise to a bounded one, and then use this noise to play any online (ex. linear bandit) algorithm. Even when 1+\epsilon moment exists, a comparison study is also provided. Furthermore, the theory is corroborated with experiments.