Breaking the Moments Condition Barrier: No-Regret Algorithm for Bandits with Super Heavy-Tailed Payoffs

1 Introduction . Multi-armed bandit ( MAB ) problems have been introduced by Robbins ( Robbins , 1952 ) , and have since become a standard model for modeling sequential decision-making problems . In an MAB instance , there are finite number of arms , pulling each of which an agent receives a random reward ( payoff ) with unknown distribution . An agent then aims to maximize the received rewards by pulling the arms strategically for a number of times . The MAB problems frequently arise in practice : e.g. , clinical trials and online advertising . When the number of arms becomes infinite in practice , the stochastic linear bandit ( Abe and Long , 1999 ; Auer , 2002 ; Dani et al. , 2008 ) generalizes the classical MAB by assuming the underlying reward distribution possesses a linear structure . Linear bandits achieve tremendous success such as online advertisement and recommendation systems ( Li et al. , 2010 ; Chu et al. , 2011 ; Li et al. , 2016 ) . ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Due to the online learning nature of a bandit problem , we measure the performance of an agent via regret , which measures the differences of the rewards collected from the best arm to those collected from the agent . When the reward distribution is benign , e.g. , with sub-Gaussian tails† , there are a number of efficient algorithms ( Bubeck and Cesa-Bianchi , 2012 ; Lattimore and Szepesvári , 2020 ) , which obtain a worst-case regret bound of the form Õ ( √ AT ) ‡ , where A is the number of arms and T is the total number of arm pulls . Any algorithm with regret bound sublinear in T is effectively learning as its average regret tends to 0 when T →∞ . In the linear setting , a number of results ( Dani et al. , 2008 ; Abbasi-Yadkori et al. , 2011 ) achieve Õ ( poly ( d ) √ T ) regret bounds , eliminating the dependence on A . Here d is the ambient dimension of the problem . A related performance measure is about the number of pulls to identify the best arm ( Soare et al. , 2014 ; Tao et al. , 2018 ; Jedra and Proutiere , 2020 ) . We note that usually a regret minimization algorithm can be converted to identify the best arm with high probability . Nevertheless , in many practical scenarios , we encounter non-sub-Gaussian noises in the observed payoffs , e.g. , the price fluctuations in financial markets ( Cont and Bouchaud , 2000 ; Rachev , 2003 ; Hull , 2012 ) , and/or fluctuations of neural oscillations ( Roberts et al. , 2015 ) . In such scenarios , the previously mentioned algorithms may fail . To tackle this problem , Bubeck et al . ( 2013 ) makes the first attempt to study the stochastic MAB problem with heavy-tailed noise . Specifically , for the rewards with a finite second order moment , by utilizing more robust statistical estimators , Bubeck et al . ( 2013 ) achieves regret bound of the same order as in the bounded/sub-Gaussian loss setting . Then , Medina and Yang ( 2016 ) ; Shao et al . ( 2018 ) ; Xue et al . ( 2020 ) study the heavy-tailed linear bandits . They consider a general characterization of heavy-tailed payoffs in bandits , where the reward takes the form r = µ+ η , where µ is an unknown but fixed number and η is a random noise , whose distribution has a finite moment of order 1 + . Here ∈ ( 0 , 1 ] . For this setting , they establish a sublinear regret bound Õ ( T 1 1+ ) . Unfortunately , when = 0 , these regret bounds will be linear in T , failing to learn in such situations . To further account for many such real-world scenarios , where the payoff noise has super-heavy tails ( e.g. , only ( 1 + ) -th moment for ∈ ( 0 , 1 ) exists , or Cauchy distribution whose mean does not exist ) , new algorithms need to be developed : Can we design an efficient algorithm that provably learns for bandits with super heavy-tailed payoffs ? In this paper , we give the affirmative answer to this question . Without loss of generality , we consider the linear setting , which includes MAB as a special case . In this setting , each arm is viewed as a vector in Rd . The random reward of the arm x is specified as θ > x+η , where θ ∈ Rd is an unknown but fixed vector and η is a super heavy-tailed symmetric random noise such that Pr ( |η| > y ) ≤ 1/yα for any y > 0 and some α > 0 . One of the key challenges in this setting is that the mean of y may not exist . The previous robust mean estimators ( Bubeck et al. , 2013 ) , such as truncated empirical mean and median of means ( Bubeck et al. , 2013 ; Medina and Yang , 2016 ; Shao et al. , 2018 ; Xue et al. , 2020 ) which require the estimation of the mean , can not effectively handle this super heavy-tailed noise . On the other hand , since the mean does not exist , we are also required to measure the performance of the agent with high-probability pseudo-regret ( defined in Section 2.2 ) . To tackle these challenges , we propose a novel estimator : mean of medians . We then present a generic algorithmic framework to apply it in any existing bandit algorithm . Below , we summarize our contributions : • We propose a novel robust statistical estimator : mean of medians . Specifically , we simply split ñ samples into k blocks and takes the mean of the median in each block . Theoretically , we can prove that , by utilizing ñ ( α ) samples , the super heavy-tailed noise is reduced to the bounded noise with high probability . Here ñ ( α ) is a constant which depends on α . • For the super heavy-tailed linear bandits , we propose a new algorithmic framework . In detail , by simply combing the above mean of medians estimator and an arbitrary provably efficient bandit algorithm , we obtain a new algorithm that can be proved efficient for regret minimization problems and best arm identification problems . Our obtained sample bounds and regret bounds can be nearly optimal . †For any ζ > 0 , a random variable X is said to be ζ-sub-Gaussian if it holds that E [ et ( X−E [ X ] ) ] ≤ eζ 2t2/2 for any t > 0 . ‡Õ ( · ) ignores logarithm factors . • We instantiate our framework with Student ’ s t-noises and compare with previous methods . Our experiments demonstrate our method can significantly outperform existing algorithms in these environments , and is strictly consistent with our theoretical guarantees . 1.1 Related Works . A line of recent work ( Bubeck et al. , 2013 ; Medina and Yang , 2016 ; Shao et al. , 2018 ; Xue et al. , 2020 ) on the heavy-tailed MAB or linear bandits uses the truncated empirical mean and median of means as the robust estimators . However , without assuming the finite moments of order 1 + for some ∈ ( 0 , 1 ] , it remains unclear whether one can attain equivalent regret/sample complexity for the more heavy-tailed setting , e.g , the noise of the payoff follows a Student ’ s t-distribution , or whether sublinear regret algorithms of any form are even possible at all . In comparison , by incorporating the mean of medians estimator , we can design a general provably efficient algorithmic framework for the super heavy-tailed linear bandits . Our work also adds to the vast body of existing literature on the regret minimization problem on linear bandits ( Auer , 2002 ; Dani et al. , 2008 ; Rusmevichientong and Tsitsiklis , 2010 ; Abbasi-Yadkori et al. , 2011 ; Lattimore and Szepesvari , 2017 ; Combes et al. , 2017 ) . A remarkable analysis is given by Auer ( 2002 ) who builds confidence regions for the true model parameter and then optimistically selects the action minimizing the loss over these sets . In the theoretical view , for the setting where the arm set is finite , Auer ( 2002 ) establishes a Õ ( √ dT ) regret bound . Then , Dani et al . ( 2008 ) ; Rusmevichientong and Tsitsiklis ( 2010 ) ; Abbasi-Yadkori et al . ( 2011 ) construct the confidence ellipsoids for the setting where the arm set is infinite and establish a Õ ( d √ T ) regret bound . Our work is also closely related to another line of work ( Soare et al. , 2014 ; Soare , 2015 ; Garivier and Kaufmann , 2016 ; Tao et al. , 2018 ; Xu et al. , 2018 ; Fiez et al. , 2019 ; Jedra and Proutiere , 2020 ) on the best arm identification problem for linear bandits . This problem is also referred as pure exploration problem since there is no price to be paid for exploring and thus we don ’ t need to carefully balance exploration against exploitation . Hence , an algorithm that is optimal for the best arm identification problem may be suboptimal for the regret minimization since its exploration is too aggressive . The reverse is also true because the exploration for the regret minimization algorithm might be too slow . 2 Preliminaries . 2.1 Notation . For a positive integer K , we use [ K ] to denote { 1 , 2 , · · · , K } . For r ∈ R , its absolute value is |r| , its ceiling integer is dre , and its floor integer is brc . Also , let 1 { · } be the indicator function . 2.2 Linear Bandits . We consider a linear bandit problem specified by a tuple ( X , θ ) , where X is the set of arms , and θ ∈ Rd is an unknown but fixed parameter with ‖θ‖2 ≤ 1 . Without loss of generality , we assume that X ⊂ Rd and ‖x‖ ≤ 1 for any x ∈ X . At each round t , a learner chooses an action xt ∈ X and observes the reward rt = θ > xt + ηt , where ηt is a symmetric and independent noise . We denote x∗ ∈ argmaxx∈X θ > x as an optimal arm . Note that the linear setting includes the MAB problem as a special case . In the MAB setting , each arm a ∈ X is corresponding to a |X |-dimensional standard unit vector ( all entries are 0 except for the a-th entry ) . In the online setting , i.e. , an agent interacts with the bandit instance for at most T ≥ 1 rounds , the performance of the agent is measured by the cumulative ( pseudo ) regret , which is defined by Regret ( T ) = T∑ t=1 ( θ > x∗ − θ > xt ) . In order to obtain a small regret , the agent needs to figure out a vector close to θ , even with the presence of the noise ηt . A related performance measure is the sample complexity of identifying a near-optimal arm . In this case , an agent is asked to output an arm x̂ such that |θ > x∗ − θ > x̂| ≤ with probability at least 1− δ . The sample complexity of the algorithm is measured by the smallest number of steps the agent interacts with the bandit instance . If Regret ( T ) of an algorithm is sublinear , then we can use standard technique to covert it into an algorithm that outputs an -accurate arm in time T such that Regret ( T ) /T = O ( ) . When = 0 , the problem reduces to the best arm identification problem . Note that best arm identification is usually not possible for a continuous action space X .

Summary: The paper gives a novel robust estimator -- mean of medians -- for heavy-tailed noise satisfying $Pr( |Y| > |y|) \leq |y|^{-\alpha}$ for any fixed $\alpha > 0$. In particular, given access to i.i.d. samples $X_1,\ldots,X_n$, the unbiased estimator returns $X_{\sf mom}$ satisfying (i) $E [X_{\sf mom}] = 0$ and (ii) $|X_{\sf mom}| \leq \sqrt{4^{2/\alpha}/n^{1 - \epsilon} \log(4/\delta)}$ with probability at least $1 - \delta$. The paper then uses this as a black box to give a $\tilde{O}(d\sqrt{T})$-regret algorithm for linear bandits in $d$-dimension.