Title: Efficient Algorithms for Generalized Linear Bandits with Heavy-tailed Rewards
Venue: NeurIPS 2023
Accepted: yes
Keywords: linear bandits

heavy-tailed

truncated

mean of medians
Abstract: This paper investigates the problem of generalized linear bandits with heavy-tailed rewards, whose $(1+\epsilon)$-th moment is bounded for some $\epsilon\in (0,1]$. Although there exist methods for generalized linear bandits, most of them focus on bounded or sub-Gaussian rewards and are not well-suited for many real-world scenarios, such as financial markets and web-advertising. To address this issue, we propose two novel algorithms based on truncation and mean of medians. These algorithms achieve an almost optimal regret bound of $\widetilde{O}(dT^{\frac{1}{1+\epsilon}})$, where $d$ is the dimension of contextual information and $T$ is the time horizon. Our truncation-based algorithm supports online learning, distinguishing it from existing truncation-based approaches. Additionally, our mean-of-medians-based algorithm requires only $O(\log T)$ rewards and one estimator per epoch, making it more practical. Moreover, our algorithms improve the regret bounds by a logarithmic factor compared to existing algorithms when $\epsilon=1$. Numerical experimental results confirm the merits of our algorithms.

Efficient Algorithms for Generalized Linear Bandits with Heavy-tailed Rewards

Abstract

This paper investigates the problem of generalized linear bandits with heavy-tailed rewards, whose (1 + ϵ ) -th moment is bounded for some ϵ ∈ (0 , 1] . Although there exist methods for generalized linear bandits, most of them focus on bounded or subGaussian rewards and are not well-suited for many real-world scenarios, such as financial markets and web-advertising. To address this issue, we propose two novel algorithms based on truncation and mean of medians. These algorithms achieve an almost optimal regret bound of ˜ O ( dT 1 1+ ϵ ) , where d is the dimension of contextual information and T is the time horizon. Our truncation-based algorithm supports online learning, distinguishing it from existing truncation-based approaches. Additionally, our mean-of-medians-based algorithm requires only O (log T ) rewards and one estimator per epoch, making it more practical. Moreover, our algorithms improve the regret bounds by a logarithmic factor compared to existing algorithms when ϵ = 1 . Numerical experimental results confirm the merits of our algorithms.

1 Introduction

The multi-armed bandits (MAB) is a powerful framework to model the sequential decision-making process with limited information [Robbins, 1952], which has been found applications in various areas such as medical trails [Villar et al. , 2015] and advertisement placement [Bubeck and Cesa-Bianchi, 2012]. In the classical K -armed bandit problem, an agent selects one of the K arms and receives a reward drawn independently and identically distributed from an unknown distribution associated with the chosen arm. The goal of the agent is to maximize the cumulative rewards through the trade-off between exploration and exploitation, i.e., pulling the arms that may potentially give better outcomes while also exploiting the knowledge gained from previous trials to select the optimal arm.

One fundamental limitation of MAB is that it ignores contextual information in some scenarios such as advertisement placement [Lattimore and Szepesvári, 2020], where features of users and products can provide valuable guidance for decision making. In these cases, decisions should not only rely on rewards from previous epochs but also the contextual information from both past and current epochs. Stochastic Linear Bandits (SLB) has emerged as the most popular model in the last decade to address this limitation, assuming a linear relationship between the contextual vector and the expected reward

[Auer, 2002; Dani et al. , 2008; Abbasi-yadkori et al. , 2011; Hu et al. , 2021; Alieva et al. , 2021; Yang et al. , 2022; He et al. , 2022; Bengs et al. , 2022]. However, in many real-world applications, such as social network [Filippi et al. , 2010], the assumption of Poisson or logistic relation between the expected reward and contextual vector has demonstrated better performance, which motivates the study of generalized linear bandits (GLB). In each round, the agent first observes a decision set D t ⊂ R d composed of contextual vectors. Then, the agent selects an arm x t ∈ D t and receives a reward y t satisfying the expectation,

where θ ∗ is the inherent vector and µ ( · ) is the link function, such as the identity function or the logistic function. The performance of the agent is measured by the regret such that

where ˜ x t = argmax x ∈D t µ ( x ⊤ θ ∗ ) represents the optimal arm in the set D t .

Extensive research has been conducted on the GLB, with most assuming sub-Gaussian rewards [Filippi et al. , 2010; Li et al. , 2012, 2017; Jun et al. , 2017; Lu et al. , 2019; Zhou et al. , 2019; Han et al. , 2021; Li and Wang, 2022]. However, it has been observed that in certain sequential decision-making scenarios, such as financial markets [Cont and Bouchaud, 2000], the occurrence of extreme returns is much more frequent than the standard normal distribution. This phenomenon is known as heavy-tailed behavior [Foss et al. , 2013], where the existing algorithms are not suitable. To address this limitation, in this study, we focus on the GLB with heavy-tailed rewards [Bubeck et al. , 2013], i.e., the reward obtained at t -th round satifies the condition

for some ϵ ∈ (0 , 1] and u > 0 . Different from the traditional sub-Gaussian setting, heavy-tailed rewards do not decay exponentially and the estimation of expected rewards is significantly impacted.

According to the distinguishing characteristic of heavy-tailed distributions where extreme values occur with high probability, previous studies have developed three main strategies to address the issue in parameter estimation [Audibert and Catoni, 2011; Hsu and Sabato, 2014; Zhang and Zhou, 2018; Ray Chowdhury and Gopalan, 2019; Lugosi and Mendelson, 2021; Zhong et al. , 2021; Huang et al. , 2022; Diakonikolas et al. , 2022; Gorbunov et al. , 2022; Kamath et al. , 2022; Li and Liu, 2022; Gou et al. , 2023]. One such strategy is truncation Audibert and Catoni [2011], which mitigates the impact of extreme values by truncating them. A recently proposed strategy is the mean of medians approach [Zhong et al. , 2021], which involves partitioning the samples drawn from the heavy-tailed distribution into multiple groups, taking the median within each group, and computing the mean of these medians. It intuitively reduces the impact of extreme samples, as extreme samples are distributed to both sides, thus the median value is more robust. The third strategy is median of means [Hsu and Sabato, 2014], which adjusts the order of calculating mean and taking the median in the mean of medians strategy.

Most existing algorithms for heavy-tailed bandit problems are derived from aforementioned three strategies, with a primary focus on the SLB model [Medina and Yang, 2016; Shao et al. , 2018; Xue et al. , 2020]. To provide a comprehensive overview and facilitate comparison, we present a summary of our results and previous findings on linear bandits with heavy-tailed rewards in Table 1. For the sake of clarity, the presented regret bounds in Table 1 are under the assumption that the rewards have finite variance. The computational complexity only takes into account multiplication and division operations. Although Shao et al. [2018] and Xue et al. [2020] achieve nearly optimal regret for

infinite-armed and finite-armed SLB, respectively, their algorithms are computationally expensive. The latest work utilizing the mean of medians approach demonstrates efficiency but is limited to symmetric rewards [Zhong et al. , 2021]. Therefore, designing efficient heavy-tailed algorithms for GLB with symmetric and asymmetric rewards is an interesting and non-trivial challenge.

Through the delicate employment of heavy-tailed strategies, our contributions to the generalized linear bandit problem with heavy-tailed rewards can be summeraized as follows:

We develop two novel algorithms, CRTM and CRMM, which utilize the truncation strategy and mean of medians strategy, respectively. Both algorithms exhibit a sublinear regret bound of ˜ O ( dT 1 1+ ϵ ) , which is amolst optimal as the lower bound is Ω( dT 1 1+ ϵ ) [Shao et al. , 2018].

CRTM reduces the computational complexity from O ( T 2 ) to O ( T ) when compared to existing truncation-based algorithms [Shao et al. , 2018; Xue et al. , 2020], while CRMM reduces the number of estimator required per round from O (log T ) to only one, as compared to existing median-of-means-based algorithms [Shao et al. , 2018; Xue et al. , 2020].

When ϵ = 1 , the regret bounds of CRTM and CRMM improves a logarithmic factor of order 1 2 α and 1 2 α -1 2 for some α ∈ (0 , 1) , respectively, over the recently proposed method of Zhong et al. [2021] 2 . Notably, CRTM extends the method of Zhong et al. [2021] from symmetric rewards to general case, making CRTM more practical.

We conduct numerical experiments to demonstrate that our proposed algorithms not only achieve a lower regret bound but also require fewer computational resources when applied to heavy-tailed bandit problems.

2 Related Work

In this section, we briefly review the related work on linear bandits. Through out the paper, the p -norm of a vector x ∈ R d is ∥ x ∥ p = ( | x 1 | p + . . . + | x d | p ) 1 /p . Given a positive definite matrix A ∈ R d × d , the weighted Euclidean norm of the vector x is ∥ x ∥ A = √ x ⊤ A x .

2.1 Generalized Linear Bandits

Filippi et al. [2010] was the first to address the generalized linear bandit problem and proposed an algorithm with a regret bound of ˜ O ( d √ T ) . However, their algorithm is not efficient as it requires storing all the action-feedback pairs encountered so far and performing maximum likelihood estimation at each step. A notable improvement was presented by Zhang et al. [2016] with the introduction of an efficient algorithm called OL 2 M, whose space and time complexity at each epoch does not grow over time and achieves a ˜ O ( d √ T ) regret. However, their algorithm is limited to the logistic link function. Later, Jun et al. [2017] extended OL 2 Mto generic link functions while still maintaining the ˜ O ( d √ T ) regret bound. Ding et al. [2021] proposed another efficient generalized linear bandit algorithm following the line of Thompson sampling scheme.

The main challenge in the bandit problem is the trade-off between exploration and exploitation. To address this issue, the most commonly used approach is the confidence-region-based method, specifically for the linear bandit model with infinite arms [Dani et al. , 2008; Abbasi-yadkori et al. , 2011; Zhang et al. , 2016]. Here we take the algorithm OL 2 Mto give a brife introduction to this approach [Zhang et al. , 2016]. With the arrival of a new trial ( x t , y t ) in the t -th epoch, OL 2 Mfirst constructs a surrogate loss ℓ t ( θ ) satisfying ∇ ℓ t ( θ ) = ( -y t + µ ( x ⊤ t θ )) x t . Then, OL 2 Memploys a variant of the online Newton step (ONS) to update the estimated parameters, i.e.,

Here, V t +1 = V t + κ 2 x t x ⊤ t for κ > 0 , and the initialized matrix V 1 = λ I d for λ > 0 . Subsequently, OL 2 Mconstructs a confidence region C t +1 centered at the estimated parameter ˆ θ N t +1 , such that

where γ t +1 = O ( d log t ) indicating the uncertainty of the estimation and the unknown parameter θ ∗ lies in this region with high probability. Finally, OL 2 M selects the most promising arm x t +1 according to the principle of 'optimization in the face of uncertainty', i.e.,

2.2 Bandit Learning with Heavy-tailed Rewards

Most of the existing work developed heavy-tailed bandit algorithms using truncation and median of means strategies [Bubeck et al. , 2013; Medina and Yang, 2016; Shao et al. , 2018; Xue et al. , 2020; Huang et al. , 2022]. Bubeck et al. [2013] first conducted extensive research on multi-armed bandits with heavy-tailed rewards and achieved a logarithmic regret bound. Medina and Yang [2016] extended it to the SLB model and introduced two algorithms that achieve regret bounds of ˜ O ( dT 2+ ϵ 2(1+ ϵ ) ) and ˜ O ( d 1 2 T 1+2 ϵ 1+3 ϵ + dT 1+ ϵ 1+3 ϵ ) , respectively. Shao et al. [2018] improved upon the results of Medina and Yang [2016] by a more delicate application of heavy-tailed strategies, achieving a regret bound of ˜ O ( dT 1 1+ ϵ ) . Xue et al. [2020] investigated the case with finite arms and provided two algorithms that attained regret bounds of ˜ O ( d 1 2 T 1 1+ ϵ ) . Recently, Zhong et al. [2021] proposed the mean of medians estimator for the super heavy-tailed bandit problem, but the rewards are limited to symmetric distributions. Applying this estimator to the GLB algorithm of Jun et al. [2017] yields a heavy-tailed GLB algorithm that achieves the regret bound of O ( d (log T ) 1 2 α + 3 2 T 1 2 ) for some α ∈ (0 , 1) . To illustrate the basic idea of adopting different heavy-tailed strategies in the bandit model, we briefly describe three representative algorithms.

For the algorithm exploiting truncation strategy, we take the algorithm TOFU as an instance [Shao et al. , 2018]. With the trials up to round t , TOFU truncates the rewards d times as follows,

where h t = O ( t 1 -ϵ 2(1+ ϵ ) ) is the truncated criterion, and u i τ ( t ) denotes the element in the i -th row and τ -th column of matrix ˜ V -1 / 2 t +1 A t , A t = [ x 1 , x 2 , . . . , x t ] ∈ R d × t is the matrix composed of selected arms and ˜ V t +1 = A t A ⊤ t + I d . Using these truncated rewards, TOFU conducts an estimator as ˜ θ t +1 = ˜ V -1 / 2 t +1 [ u 1 t · Y 1 t , . . . , u d t · Y d t ] with u i t · Y i t = ∑ t τ =1 u i τ ( t ) y τ I | u i τ y τ |≤ h t . TOFUthen constructs a confidence region centered on this estimator and selects the promising arm, similar to (3) and (4). Notice that the scalarized parameters { u i τ ( t ) } t τ =1 are updated at each epoch, requiring TOFU to store the learning history and truncate all rewards at each epoch. Thus, TOFU is not an online method.

For the algorithm exploiting median of means strategy, it's common to play the chosen arm r times and get r sequences of rewards { Y j t } r j =1 , where Y j t = [ y j 1 , . . . , y j t ] is the j -th reward sequence up to epoch t . MENU executes least square estimation for each reward sequence and get r estimators, i.e.,

where r = O (log T ) [Shao et al. , 2018]. Then, the median of means strategy adopted by MENU is operated as follows,

Then, MENU takes the estimator ˆ θ k ∗ t +1 with k ∗ = argmin j ∈{ 1 , 2 ,...,r } { m j } as the center of confidence region. Finally, MENU selects the most promising arm similar to (4).

For the mean of medians method proposed by Zhong et al. [2021], at each epoch t , the agent first plays the selected arm ¯ r times, with a value of ¯ r = O ((log T ) 1 /α ) for some α ∈ (0 , 1) , and then receives rewards { y j t } ¯ r j =1 for these plays. Subsequently, the agent randomly divides the rewards into multiple groups, with each group contains ⌈ ¯ r α ⌉ rewards. The agent then takes the median of each group and uses the mean of these medians to update the estimator. Notice that the expectation of the median has a bias to the expected reward other than the symmetric distribution. Thus, mean of medians strategy is limited to symmetric distribution. Another point worth mentioning is that ¯ r is too

large to try sufficient different arms. For example, the agent can only play ⌈ T/ ¯ r ⌉ = 100 different arms with T = 10 6 and α = 0 . 62 , which is obviously unreasonable 3 .

3 Algorithms

In this section, we first introduce the generalized linear bandit model and then demonstrate two novel algorithms based on truncation and mean of medians, respectively.

3.1 Learning Model

The formal description of the generalized linear bandit model is as follows. In each round t , an agent plays an arm x t ∈ D t and obtains a stochastic reward y t , which is generated from a generalized linear model represented by the following equation,

where θ ∗ is the inherent parameters, τ > 0 is a known scale parameter, and g ( · ) and h ( · , · ) are normalizers [P. McCullagh, 1989]. The expectation of y t is given by

Thus, m ′ ( · ) is the link function in (1), such that µ ( · ) = m ′ ( · ) . The reward model can be rewritten as

where η t is a random noise satisfying the condition

Here, G t -1 ≜ { x 1 , y 1 , . . . , x t -1 , y t -1 , x t } is a σ -filtration and G 0 = ∅ . Following the existing work [Filippi et al. , 2010; Jun et al. , 2017; Li et al. , 2017], we make standard assumptions as follows.

Assumption 1 The coefficients θ ∗ and contextual vectors x are bounded, such that ∥ θ ∗ ∥ 2 ≤ S and ∥ x ∥ 2 ≤ 1 for all x ∈ D t , where S is a known constant.

Assumption 2 The link function µ ( · ) is L -Lipschitz on [ -S, S ] , and continuously differentiable on ( -S, S ) . Moreover, there exists some κ > 0 such that µ ′ ( z ) ≥ κ and | µ ( z ) | ≤ U for any z ∈ ( -S, S ) .

3.2 Truncation

Our first algorithm is called Confidence Region with Truncated Mean (CRTM). The complete procedure is provided in Algorithm 1. Here, we consider the heavy-tailed setting, i.e., there exists a constant u > 0 , the rewards admit

As we have mentioned earlier in Section 2.1, to design effective algorithms for GLB model, constructing a narrow confidence region for the underlying coefficients θ ∗ is necessary. However, heavy-tailed rewards that satisfy (10) produce extreme values with high probability, resulting in a confidence region with a large radius. Therefore, a straightforward approach to settle this problem is to truncate the extreme reward to reduce its impact.

In each round t , CRTM first plays an arm x t ∈ D t and observes the corresponding reward y t . Then, CRTM truncates the reward y using a uniform criterion Γ = O ( T 1 -ϵ 2(1+ ϵ ) ) , such that t ˜

Algorithm 1 Confidence Region with Truncated Mean (CRTM)

Input : δ, ϵ, u, κ, S, λ = max { 1 , κ/ 2 } and T ∈ N +

1: Initialize ˆ θ 1 = 0 and V 1 = λ I d

3: for t = 1 , 2 , . . . , T do

2: Define the truncation criterion Γ = 2 ( u/ ln(4 T/δ )) 1 1+ ϵ ( d ln ( 1 + κT 2 λd ) /κ ) 1 2 T 1 -ϵ 2(1+ ϵ )

4: ( x t , ˜ θ t ) = argmax x ∈D t , θ ∈C t ⟨ x , θ ⟩

5: Play the arm x t and observe the payoff y t

6: Truncate the observed payoff ˜ y t = y t I ∥ x t ∥ V -1 t | y t |≤ Γ

7: Compute the gradient ∇ ˜ ℓ t ( ˆ θ t ) = ( -˜ y t + µ ( x ⊤ t ˆ θ t )) x t

9: Update the estimator

8: Update V t +1 = V t + κ 2 x t x ⊤ t

10: Construct the confidence region

11: end for

where V t = V t -1 + κ 2 x t -1 x ⊤ t -1 with V 1 = λ I d . Here, κ is defined in Assumption 2 and λ = max { 1 , κ/ 2 } . With the processed action-reward pair ( x t , ˜ y t ) , CRTM computes the gradient of the loss function as where ˜ ℓ t ( · ) is the negative log-likelihood of the generalized linear model (8). After that, CRTM employs a variant of online Newton step (ONS) to update its estimator, given by

Equipped with above estimation, CRTM constructs the confidence region C t +1 where the inherent parameters θ ∗ lies in with high probability, such that

where γ = ˜ O ( T 1 -ϵ 1+ ϵ ) denotes the width of the confidence region, and details are shown in Theorem 1. Given the confidence region C t +1 , the most promising arm x t +1 can be obtained through the following maximize operation,

since µ ( · ) is monotonically increasing according to Assumption 2.

Although there exists several heavy-tailed linear bandit algorithms based on the truncation strategy, such as TOFU [Shao et al. , 2018] and BTC [Xue et al. , 2020], CRTM differs from them in two aspects. Firstly, both TOFU and BTC have to store the historical rewards and truncate them at each epoch, resulting in a computational complexity of O ( T 2 ) . In contrast, CRTM achieves online learning by processing only the reward of current round, whose computational complexity is O ( T ) . Secondly, while TOFU and BTC are designed for SLB model and calculate the estimator via least-squares estimation, CRTM is designed for the GLB model and updates the estimator using the ONS method, which makes the analytical techniques fundamentally different. Theorem 1 provides a tight confidence region, and its proof relies on the induced method because ONS is an iteratively updated method. Due to the page limit, we provide the detailed proof in the Appendix A.2.

Theorem 1 If the rewards satisfy (9) and (10) , then with probability as least 1 -δ , the confidence region in CRTM is

Algorithm 2 Confidence Region with Mean of Medians (CRMM)

Input : δ, ϵ, v, κ, S, λ = max { 1 , κ/ 2 } and T ∈ N +

1: Initialize ˆ θ 1 = 0 , V 1 = λ I d and γ 1 = λS 2

3: for t = 1 , 2 , . . . , T 0 do

2: r = ⌈ 16 ln 4 T δ ⌉ and T 0 = ⌊ T/r ⌋

4: ( x t , ˜ θ t ) = argmax x ∈D , θ ∈C t ⟨ x , θ ⟩

5: Play the arm x t r times and observe the rewards { y 1 t , y 2 t , . . . , y r t }

7: Compute the gradient ∇ ¯ ℓ t ( ˆ θ t ) = ( -¯ y t + µ ( x ⊤ t ˆ θ t )) x t

6: Take the median of { y 1 t , y 2 t , . . . , y r t } as ¯ y t

8: Update V t +1 = V t + κ 2 x t x ⊤ t

9: Compute the center of confidence region

10: Construct the confidence region

11: end for where

With above confidence region, the regret bound of CRTM is explicitly given as follows.

Theorem 2 If the rewards satisfy (9) and (10) , then with probability at least 1 -δ , the regret of CRTM satisfies

Remark: The above theorem establishes a ˜ O ( dT 1 1+ ϵ ) regret bound with the assumption that the (1 + ϵ ) -th moment of the rewards is bounded for some ϵ ∈ (0 , 1] . Existing algorithms based on truncation is time-consuming because they need to store the learning history and truncate all historical rewards at each epoch [Shao et al. , 2018; Xue et al. , 2020]. Unlike the recently proposed mean of medians method which is limited in symmetric rewards [Zhong et al. , 2021], CRTM expands it to asymmetric and achieves an improved regret bound by a factor of O ((log T ) 1 2 α ) for some α ∈ (0 , 1) if ϵ = 1 . Furthermore, CRTM is almost optimal as the lower bound is Ω( dT 1 1+ ϵ ) [Shao et al. , 2018].

3.3 Mean of Medians

In this section, we present our second algorithm, referred to as Confidence Region with Mean of Medians (CRMM), which shares a similar framework with CRTM but uses a different mean of medians estimator. The complete procedure is outlined in Algorithm 2. CRMM requires that for some ϵ ∈ (0 , 1] , the 1+ ϵ central moment of the rewards is bounded, and the distribution of rewards is symmetric. Precisely, for some ϵ ∈ (0 , 1] , there exists a constant v > 0 such that the rewards satisfy

At each epoch t , CRMM plays the selected arm x t r times, generating rewards { y 1 t , . . . , y r t } with r = O (log T ) . To obtain a robust estimation using mean of medians strategy, CRMM first takes the median of { y 1 t , . . . , y r t } , denoted by ¯ y t . Subsequently, CRMM computes the gradient with the arm-reward pair ( x t , ¯ y t ) through the operation similar to (11). Then, CRMM employs a variant of ONS to update the estimator and construct the confidence region C t +1 centered on the new estimator. The details about the constructed confidence region is given in Theorem 3.

Compared to existing bandit algorithms that utilize the median of means strategy, the primary difference lies in the item chosen as the 'means'. As we have introduced in (7), MENU of Shao et al. [2018] uses the distance between different estimators as the 'means'. BMM of Xue et al. [2020] calculates multiple estimated rewards for each arm and treats them as the 'means'. Both MENU and BMMrequire O (log T ) estimators during each round, whereas CRMM only requires one estimator. Moreover, compared to the mean of medians approach [Zhong et al. , 2021], CRMM plays each selected arm fewer times, leading to more model updates, which is critical based on experimental results. Since the chosen arm has to be played multiple times, we assume the arm set for CRMM is static, such that D t = D for t > 0 , which is a common assumption [Medina and Yang, 2016; Zhang et al. , 2016; Lu et al. , 2019]. The following theorem guarantees a tight confidence region.

Theorem 3 If the rewards satisfy (9) and (12) , then with probability as least 1 -2 δ , the confidence region in CRMM is

where

With above confidence region, we prove the following regret bound for CRMM.

Theorem 4 If the rewards satisfy (9) and (12) , then with probability at least 1 -2 δ , the regret of CRMM satisfies

Remark: Theorem 3 clarifies that if the rewards have a finite 1 + ϵ central moment for some ϵ ∈ (0 , 1] , CRMM achieves a regret bound of ˜ O ( dT 1 1+ ϵ ) . This bound reduces to ˜ O ( d √ T ) when ϵ = 1 , indicating that CRMM achieves the same order as the bounded rewards assumption regarding both d and T [Zhang et al. , 2016; Jun et al. , 2017]. Compared to the approach of Zhong et al. [2021], CRMMenhances the bound by an order of O ((log T ) 1 2 α -1 2 ) for a fixed α ∈ (0 , 1) if ϵ = 1 .

4 Experiments

This section demonstrates the improvement of our algorithms by numerical experiments. Firstly, we show the effectiveness of our algorithms in dealing with heavy-tailed problems by comparing their regret to that of existing generalized linear bandit algorithms. Secondly, we evaluate the efficiency of our algorithms by comparing their time consumption to other existing algorithms designed for heavy-tailed bandit problems. All algorithms are implemented using PyCharm 2022 and tested on a laptop with a 2.5GHz CPU and 32GB of memory.

4.1 Regret Comparison

To assess the enhancement of our algorithms in handling heavy-tailed problems, we utilize the vanilla GLB algorithms, specifically OL 2 M[Zhang et al. , 2016] and GLOC [Jun et al. , 2017], as baselines. Additionally, we incorporate the mean of medians method proposed by Zhong et al. [2021] into OL 2 Mand GLOC, resulting in another two baselines OL 2 M_mom and GLOC_mom, respectively. All algorithms are configured with ϵ = 1 , δ = 0 . 01 , and T = 10 6 .

Let θ ∗ = 1 / √ d ∈ R d , where 1 is an all1 vector and ∥ θ ∗ ∥ 2 = 1 . The number of arms is set to K = 20 , and the feature dimension is d = 10 . Each component of the contextual vector x t is uniformly sampled from the interval [0 , 1] , and then normalized to be unit norm, i.e., ∥ x t ∥ 2 = 1 . We tune the width of the confidence region following the common practice in bandit learning [Zhang et al. , 2016; Jun et al. , 2017]. Precisely, with c being a tuning parameter searched within [1 e -4 , 1] , the width of the confidence region for OL 2 Mand GLOC are set as γ t = cd ln( t/λ +1) and γ t = c ∑ t τ =1 ( µ ( x ⊤ τ ˆ θ τ ) -y τ ) 2 ∥ x τ ∥ 2 V -1 τ , respectively. In addition, the radius of the confidence region is set as cd ln(4 T/δ ) 2 ϵ 1+ ϵ ln( T/ ( dλ ) + 1) T 1 -ϵ 1+ ϵ for CRTM, and cd ln( t/ ( dλ ) + 1) t 1 -ϵ 1+ ϵ for

0.0

0.2

0.4

0.6

0.8

1.0

Iteration

1e6

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Cumulative Regret

1e4

OL

2

M

GLOC

OL

2

M_mom

GLOC_mom

CRMM(ours)

CRTM(ours)

(a) Student's

t

-noises (symmetric)

0.0

0.2

0.4

0.6

0.8

1.0

Iteration

1e6

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Cumulative Regret

1e4

OL

2

M

GLOC

OL

2

M_mom

GLOC_mom

CRMM(ours)

CRTM(ours)

(b) Pareto noises (asymmetric)

CRMM. For OL 2 M_mom and GLOC_mom, the chosen arm is played ¯ r = (16 ln(2 T/δ )) 1 /α times per round, and α = 0 . 62 is close to optimal according to the experiments of Zhong et al. [2021].

Werun 10 repetitions for each algorithm and display the average regret with time evolution. According to the generalized linear bandit model, the observed reward y t is given by

where µ ( x ) = 1 1+ e -x is the logit model and η t is the heavy-tailed noises. To evaluate the algorithms performance under both symmetric and asymmetric rewards, η t fits the following two distributions,

(i) Student's t -Noise: η t ∼ G (2) √ 3 πG (1 . 5) ( 1 + x 2 3 ) -2 where G ( · ) is the Gamma function;

(ii) Pareto Noise: η t ∼ sx s m x s +1 I x ≥ x m where s = 3 and x m = 0 . 01 .

Fig. 1 compares our algorithms against two vanilla GLB algorithms (OL 2 Mand GLOC), as well as these two algorithms exploiting mean of medians estimators (OL 2 M_mom and GLOC_mom). Fig. 1(a) shows that CRTM and CRMM outperform the other four algorithms. CRTM provides the best performance, which is consistent with the theoretical guarantees. OL 2 M_mom and GLOC_mom appear ineffective at handling heavy-tailed problems, because they update estimator only 100 times with the chosen arm played ¯ r times [Zhong et al. , 2021]. Fig. 1(b) presents the cumulative regrets under asymmetric noises, with CRTM still having the lowest regret curve, demonstrating its generality and effectiveness in handling heavy-tailed bandit problems. On the other hand, CRMM, GLOC_mom, and OL 2 M_mom performs poorly in Fig. 1(b), as they can not deal with the asymmetric rewards.

4.2 Runtime Comparison

To demonstrate the efficiency improvement of our algorithms, we compare them with existing heavytailed bandit algorithms such as CRT and MoM [Medina and Yang, 2016], TOFU and MENU [Shao et al. , 2018], and SupBTC and SupBMM [Xue et al. , 2020]. Among them, CRT, TOFU and SupBTC employ truncation strategy, while MoM, MENU and SupBMM utilize the median of means strategy.

The experimental settings are the same as described in Regret Comparison section, except for the time horizon and feature dimension. We use a smaller time horizon T = 10 4 since TOFU is time-consuming. The feature dimension is increased to d = 100 to highlight the difference between SupBTC and TOFU. The computational runtimes are shown in Table 2.

For the truncation-based algorithms, CRTM

consumes the least time, while TOFU and SupBTC takes over a hundred times longer to execute than CRTM, representing a significant improvement. CRT takes only slightly longer than CRTM as both algorithms update the model online, but the regret bound of CRT is ˜ O ( dT 3 4 ) , which is ˜ O ( T 1 4 ) worse

than the bound of CRTM. For median of means algorithms, CRMM has the shortest runtime. MENU takes significantly longer than the other algorithms because MENU needs to calculate the distance between O (log T ) estimators.

5 Conclusion and Future Work

We present two algorithms, CRTM and CRMM, for the generalized linear bandit model with heavytailed rewards, which utilize the truncation and mean of medians strategies, respectively. Both algorithms achieve the regret bound of ˜ O ( dT 1 1+ ϵ ) conditioned on a bounded (1 + ϵ ) -th moment of rewards, where ϵ ∈ (0 , 1] . This bound is almost optimal since the lower bound of the stochastic liear bandit problem is Ω( dT 1 1+ ϵ ) [Shao et al. , 2018]. CRTM is the first truncation-based online algorithm for the heavy-tailed bandit problem that handles both symmetric and asymmetric rewards and approaches the optimal regret bound. CRMM enhances the regret bound of the the most related work by a logarithmic factor [Zhong et al. , 2021]. However, CRMM is limited to symmetric rewards and we will investigate to overcome this restriction in the future.

Acknowledgments and Disclosure of Funding

This work was partially supported by the National Key R&D Program of China (2022ZD0114801), the Key Basic Research Foundation of Shenzhen (JCYJ20220818100005011), NSFC (62122037, 61921006) and the major key project of PCL (PCL2021A12).

References

Yasin Abbasi-yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems 24 , pages 2312-2320, 2011.

Ayya Alieva, Ashok Cutkosky, and Abhimanyu Das. Robust pure exploration in linear bandits with limited budget. In Proceedings of the 38th International Conference on Machine Learning , pages 187-195, 2021.

Jean-Yves Audibert and Olivier Catoni. Robust linear least squares regression. The Annals of Statistics , 39(5):2766-2794, 2011.

Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research , 3(11):397-422, 2002.

Kazuoki Azuma. Weighted sums of certain dependent random variables. Tohoku Math. J. (2) , 19(3):357-367, 1967.

Viktor Bengs, Aadirupa Saha, and Eyke Hüllermeier. Stochastic contextual dueling bandits under linear stochastic transitivity models. In Proceedings of the 39th International Conference on Machine Learning , pages 1764-1786, 2022.

Sébastien Bubeck and Nicolò Cesa-Bianchi. Regret analysis of stochastic and nonstochastic multiarmed bandit problems. Foundations and Trends in Machine Learning , 5(1):1-122, 2012.

Sébastien Bubeck, Nicolò Cesa-Bianchi, and Gábor Lugosi. Bandits with heavy tail. IEEE Transactions on Information Theory , 59(11):7711-7717, 2013.

Rama Cont and Jean-Philipe Bouchaud. Herd behavior and aggregate fluctuations in financial markets. Macroeconomic Dynamics , 4(2):170-196, 2000.

Varsha Dani, Thomas P. Hayes, and Sham M. Kakade. Stochastic linear optimization under bandit feedback. In Proceedings of the 21st Annual Conference on Learning , pages 355-366, 2008.

Ilias Diakonikolas, Daniel Kane, Jasper Lee, and Ankit Pensia. Outlier-robust sparse mean estimation for heavy-tailed distributions. In Advances in Neural Information Processing Systems 35 , pages 5164-5177, 2022.

Qin Ding, Cho-Jui Hsieh, and James Sharpnack. An efficient algorithm for generalized linear bandit: Online stochastic gradient descent and thompson sampling. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics , pages 1585-1593, 2021.

Sarah Filippi, Olivier Cappe, Aurélien Garivier, and Csaba Szepesvári. Parametric bandits: The generalized linear case. In Advances in Neural Information Processing Systems 23 , pages 586-594. 2010.

Sergey Foss, Dmitry Korshunov, and Stan Zachary. An Introduction to Heavy-Tailed and Subexponential Distributions . Springer, 2013.

Eduard Gorbunov, Marina Danilova, David Dobre, Pavel Dvurechenskii, Alexander Gasnikov, and Gauthier Gidel. Clipped stochastic methods for variational inequalities with heavy-tailed noise. In Advances in Neural Information Processing Systems 35 , pages 31319-31332, 2022.

Yutian Gou, Jinfeng Yi, and Lijun Zhang. Stochastic graphical bandits with heavy-tailed rewards. In Proceedings of the 39th Conference on Uncertainty in Artificial Intelligence , pages 734-744, 2023.

Yuxuan Han, Zhipeng Liang, Yang Wang, and Jiheng Zhang. Generalized linear bandits with local differential privacy. In Advances in Neural Information Processing Systems 34 , pages 26511-26522, 2021.

Jiafan He, Dongruo Zhou, Tong Zhang, and Quanquan Gu. Nearly optimal algorithms for linear contextual bandits with adversarial corruptions. In Advances in Neural Information Processing Systems 35 , pages 34614-34625, 2022.

Daniel Hsu and Sivan Sabato. Heavy-tailed regression with a generalized median-of-means. In Proceedings of the 31st International Conference on Machine Learning , pages 37-45, 2014.

Jiachen Hu, Xiaoyu Chen, Chi Jin, Lihong Li, and Liwei Wang. Near-optimal representation learning for linear bandits and linear rl. In Proceedings of the 38th International Conference on Machine Learning , pages 4349-4358, 2021.

Jiatai Huang, Yan Dai, and Longbo Huang. Adaptive best-of-both-worlds algorithm for heavy-tailed multi-armed bandits. In Proceedings of the 39th International Conference on Machine Learning , pages 9173-9200, 2022.

Kwang-Sung Jun, Aniruddha Bhargava, Robert Nowak, and Rebecca Willett. Scalable generalized linear bandits: Online computation and hashing. In Advances in Neural Information Processing Systems 30 , pages 99-109, 2017.

Gautam Kamath, Xingtu Liu, and Huanyu Zhang. Improved rates for differentially private stochastic convex optimization with heavy-tailed data. In Proceedings of the 39th International Conference on Machine Learning , pages 10633-10660, 2022.

Tor Lattimore and Csaba Szepesvári. Bandit Algorithms . Cambridge University Press, 2020.

Shaojie Li and Yong Liu. High probability guarantees for nonconvex stochastic gradient descent with heavy tails. In Proceedings of the 39th International Conference on Machine Learning , pages 12931-12963, 2022.

Chuanhao Li and Hongning Wang. Communication efficient federated learning for generalized linear bandits. In Advances in Neural Information Processing Systems 35 , pages 38411-38423, 2022.

Lihong Li, Wei Chu, John Langford, Taesup Moon, and Xuanhui Wang. An unbiased offline evaluation of contextual bandit algorithms with generalized linear models. In Proceedings of the Workshop on On-line Trading of Exploration and Exploitation 2 , 2012.

Lihong Li, Yu Lu, and Dengyong Zhou. Provably optimal algorithms for generalized linear contextual bandits. In Proceedings of the 34th International Conference on Machine Learning , pages 20712080, 2017.

Shiyin Lu, Guanghui Wang, Yao Hu, and Lijun Zhang. Multi-objective generalized linear bandits. In Proceedings of the 28th International Joint Conference on Artificial Intelligence , pages 3080-3086, 2019.

Gábor Lugosi and Shahar Mendelson. Robust multivariate mean estimation: The optimality of trimmed mean. The Annals of Statistics , 49(1):393-410, 2021.

Andres Munoz Medina and Scott Yang. No-regret algorithms for heavy-tailed linear bandits. In Proceedings of the 33rd International Conference on International Conference on Machine Learning , pages 1642-1650, 2016.

William Mendenhall, Robert.J. Beaver, and Barara.M. Beaver. Introduction to Probability and Statistics . Cengage Learning, 2012.

John A. Nelder P. McCullagh. Generalized linear models. 1989.

Sayak Ray Chowdhury and Aditya Gopalan. Bayesian optimization under heavy-tailed payoffs. In Advances in Neural Information Processing Systems 32 , pages 13790-13801, 2019.

Herbert Robbins. Some aspects of the sequential design of experiments. Bull. Amer. Math. Soc. , 58(5):527-535, 1952.

Yevgeny Seldin, François Laviolette, Nicolò Cesa-Bianchi, John Shawe-Taylor, and Peter Auer. Pac-bayesian inequalities for martingales. IEEE Transactions on Information Theory , 58(12):70867093, 2012.

Han Shao, Xiaotian Yu, Irwin King, and Michael R. Lyu. Almost optimal algorithms for linear stochastic bandits with heavy-tailed payoffs. In Advances in Neural Information Processing Systems 31 , pages 8430-8439, 2018.

Sofía S. Villar, Jack Bowden, and James Wason. Multi-armed Bandit Models for the Optimal Design of Clinical Trials: Benefits and Challenges. Statistical Science , 30(2):199-215, 2015.

Bo Xue, Guanghui Wang, Yimu Wang, and Lijun Zhang. Nearly optimal regret for stochastic linear bandits with heavy-tailed payoffs. In Proceedings of the 29th International Joint Conference on Artificial Intelligence , pages 2936-2942, 2020.

Shuo Yang, Tongzheng Ren, Sanjay Shakkottai, Eric Price, Inderjit S. Dhillon, and Sujay Sanghavi. Linear bandit algorithms with sublinear time complexity. In Proceedings of the 39th International Conference on Machine Learning , pages 25241-25260, 2022.

Lijun Zhang and Zhi-Hua Zhou. ℓ 1 -regression with heavy-tailed distributions. In Advances in Neural Information Processing Systems 31 , pages 1084-1094, 2018.

Lijun Zhang, Tianbao Yang, Rong Jin, Yichi Xiao, and Zhi-Hua Zhou. Online stochastic linear optimization under one-bit feedback. In Proceedings of the 33rd International Conference on Machine Learning , pages 392-401, 2016.

Han Zhong, Jiayi Huang, Lin Yang, and Liwei Wang. Breaking the moments condition barrier: No-regret algorithm for bandits with super heavy-tailed payoffs. In Advances in Neural Information Processing Systems 34 , pages 15710-15720, 2021.

Zhengyuan Zhou, Renyuan Xu, and Jose Blanchet. Learning in generalized linear contextual bandits with stochastic delays. In Advances in Neural Information Processing Systems 32 , pages 51975208, 2019.

Peer reviews:
comments: SUMMARY:
In this paper, the authors study the problem of generalized linear bandits with heavy tailed rewards. They propose two algorithms based on truncation and mean of medians. The algorithms both achieve near optimal regret bound of $\tilde{O}(dT^{\frac{1}{1+\epsilon}})$. These regret bounds improve upon previous results in the sense of regret bound or computational complexity. Finally, the authors run simulations to support their claims.

STRENGTHS:
1. The problem of generalized linear bandits with heavy-tailed rewards is well-motivated.
2. Both algorithms achieve near optimal regret bounds, the proof looks correct to me.
3. The writing is clear, the related work part is really helpful.

WEAKNESSES:
1. According to the introduction of previous methods for generalized linear bandits and bandits with heavy-tailed rewards, the algorithms are direct combinations of previous techniques. Therefore the algorithmic novelty is limited.

QUESTIONS:
1. For CRMM, since the assumption of symmetric distribution is used to guarantee that the median is 0, I am curious about whether it is possible to replace the condition with the median of the distribution equals 0.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-15

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers Generalized Linear Bandit (GLB) with heavy-tailed rewards, i.e., the rewards $y_t=\\mu(\\langle x_t,\\theta^\\ast\\rangle)+\\eta_t$ only allows a bounded $(1+\\epsilon)$-order moment. By utilizing the truncation and the mean of medians technique (both for handling heavy-tailed r.v.s), the authors establish efficient algorithms for GLBs with infinite arms and heavy-tailed rewards.\n\nSTRENGTHS:\n1. Compared to previous works on LB with heavy-tailed rewards, this paper considers Generalized LB instead of standard Stochastic LB, where the function $\\mu$ can be any Lipschitz and uniformly bounded function.\n2. This paper has a lower computational complexity: previous truncation-based works achieving $\\sqrt T$-style regret require performing an iteration over all previous observations every time they update the confidence set, which costs $\\mathcal O(T)$ time. In contrast, by utilizing the ONS step used in the $\\text{OL}^2\\text{M}$ algorithm, this algorithm only needs $\\mathcal O(d^2)$ time for each round.\n3. The presentation is pretty clear, and the algorithms are easy to understand.\n4. The results are supported by various numerical illustrations.\n\nWEAKNESSES:\nI am not sure what is the main technical contribution of this paper. For example, the CRTM algorithm looks pretty like equipping the $\\text{OL}^2\\text{M}$ algorithm by Zhang et al. (2016) with the previous truncation techniques by Shao et al. (2018) or Xue et al. (2020). Without any part in the main text devoted to explaining the technical hardness of showing Theorems 1 & 2, it is hard to evaluate the contribution of this paper. The same also holds for the mean-of-medians method CRMM.\n\nQUESTIONS:\nSee Weaknesses. I am willing to increase my score if the authors can address it convincingly.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of generalized linear bandits with heavy-tail rewards. Due to the heavy-tailedness, methods and algorithms for linear bandits with sub-gaussian rewards cannot be directly applied. To handle such issues, existing works have developed certain strategies, two of which are the truncation strategy and the mean-of-medians strategy. To this end, this paper proposes two novel algorithms, CRTM and CRMM, which utilize the aforementioned strategies and achieve sublinear regret bounds. CRTM reduces the computational complexity of previously best known truncation-based methods, while CRMM reduces the number of estimators required. Furthermore, CRTM does not require the reward to have symmetric distribution, contrasting existing works. Experimental results demonstrate the low regret and computational complexity of the proposed methods.\n\nSTRENGTHS:\n**1. Clarity**\nThe paper has a clear presentation. The GLB bandit problem and heavy-tailedness are rigorously defined. \n\n**2. Meaningful results for an important problem**\nHeavy-tailedness is an important aspect due to its board application in many real-world scenarios. This paper proposed computationally efficient near-optimal algorithms for such a setting, which is a meaningful result. \n\n**3. Experiments**\nThe experimental study covers the regret bound as well as the computational complexity.\n\nWEAKNESSES:\nThere is no major technical weakness/flaws in this paper, as far as I can tell.\n\nHowever, I do challenge the technical contribution of this paper: in terms of the theoretical analysis, does this paper improves over existing results or does this paper develop novel analytical techniques to facilitate the analysis? For example, in line 209, this paper mentions that the theoretical analysis is fundamentally different from that of TOFU/BTC. However, it seems to me that this is not elaborated in the later part of the paper, so I have to assume that the technical contribution is incremental compared to existing works.  Therefore, I am concerned about the technical contribution of this paper.\n\nQUESTIONS:\nIn the theoretical proof, what are the major differences from previous works? For example, is there any non-trivial ideas applied? \nMore details in  the Weakness section.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

comments: SUMMARY:
This paper proposes two novel algorithms which improve computational complexity and regret bounds over previous algorithms for generalized linear bandits with heavy-tailed rewards by combining the truncation strategy (or means of medians strategy) with the online Newton step.

STRENGTHS:
This work improves the computational complexity and regrets bound of the generalized linear contextual bandits with heavy-tailed rewards which are closely related to the real-world application.

WEAKNESSES:
1. Among the input values for the algorithm, $S$, $\epsilon$ and $v$ is not known in practice.

QUESTIONS:
1. Can dependency on $\kappa$ in the regret bound be relaxed?

LIMITATIONS:
Yes. This work is mostly theoretical and potential negative societal impact is unseen.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-04