SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Bo Xue $^{1,2}$ , Yimu Wang $^{3}$ , Yuanyu Wan $^{4}$ , Jinfeng Yi $^{5}$ , Lijun Zhang $^{6,7,*}$
$^{1}$ Department of Computer Science, City University of Hong Kong, Hong Kong, China
<sup>2</sup>The City University of Hong Kong Shenzhen Research Institute, Shenzhen, China
<sup>3</sup>Cheriton School of Computer Science, University of Waterloo, Waterloo, Canada
$^{4}$ School of Software Technology, Zhejiang University, Ningbo, China
$^{5}$ JD AI Research, Beijing, China
$^{6}$ National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China
$^{7}$ Peng Cheng Laboratory, Shenzhen, China
boxue4-c@my.cityu.edu.hk, yimu.wang@uwaterloo.ca, wanyy@zju.edu.cn
yijinfeng@jd.com, zhanglj@lamda.nju.edu.cn
This paper investigates the problem of generalized linear bandits with heavy-tailed rewards, whose  $(1 + \epsilon)$ -th moment is bounded for some  $\epsilon \in (0, 1]$ . Although there exist methods for generalized linear bandits, most of them focus on bounded or sub-Gaussian rewards and are not well-suited for many real-world scenarios, such as financial markets and web-advertising. To address this issue, we propose two novel algorithms based on truncation and mean of medians. These algorithms achieve an almost optimal regret bound of  $\widetilde{O}(dT^{\frac{1}{1 + \epsilon}})$ , where  $d$  is the dimension of contextual information and  $T$  is the time horizon. Our truncation-based algorithm supports online learning, distinguishing it from existing truncation-based approaches. Additionally, our mean-of-medians-based algorithm requires only  $O(\log T)$  rewards and one estimator per epoch, making it more practical. Moreover, our algorithms improve the regret bounds by a logarithmic factor compared to existing algorithms when  $\epsilon = 1$ . Numerical experimental results confirm the merits of our algorithms.
The multi-armed bandits (MAB) is a powerful framework to model the sequential decision-making process with limited information [Robbins, 1952], which has been found applications in various areas such as medical trails [Villar et al., 2015] and advertisement placement [Bubeck and Cesa-Bianchi, 2012]. In the classical  $K$ -armed bandit problem, an agent selects one of the  $K$  arms and receives a reward drawn independently and identically distributed from an unknown distribution associated with the chosen arm. The goal of the agent is to maximize the cumulative rewards through the trade-off between exploration and exploitation, i.e., pulling the arms that may potentially give better outcomes while also exploiting the knowledge gained from previous trials to select the optimal arm.
One fundamental limitation of MAB is that it ignores contextual information in some scenarios such as advertisement placement [Lattimore and Szepesvári, 2020], where features of users and products can provide valuable guidance for decision making. In these cases, decisions should not only rely on rewards from previous epochs but also the contextual information from both past and current epochs. Stochastic Linear Bandits (SLB) has emerged as the most popular model in the last decade to address this limitation, assuming a linear relationship between the contextual vector and the expected reward
*Lijun Zhang is the corresponding author.
Table 1: Summary of the existing work for the linear bandits with heavy-tailed rewards. CC is the abbreviation of computational complexity.
Regret	CC-Truncation	CC_MoM	Arms	Model
Medina and Yang [2016]	O(dT3/4)	O(d2T)	O(d2T/ log T)	infinite	SLB
Shao et al. [2018]	O(d√T)	O(d3T + d2T2)	O(d2T · log T)	infinite	SLB
Xue et al. [2020]	O(√dT)	O(d2T2)	O(d2T)	finite	SLB
This work	O(d√T)	O(d2T)	O(d2T/ log T)	infinite	GLB
[Auer, 2002; Dani et al., 2008; Abbasi-yadkori et al., 2011; Hu et al., 2021; Alieva et al., 2021; Yang et al., 2022; He et al., 2022; Bengs et al., 2022]. However, in many real-world applications, such as social network [Filippi et al., 2010], the assumption of Poisson or logistic relation between the expected reward and contextual vector has demonstrated better performance, which motivates the study of generalized linear bandits (GLB). In each round, the agent first observes a decision set  $\mathcal{D}_t \subset \mathbb{R}^d$  composed of contextual vectors. Then, the agent selects an arm  $x_t \in \mathcal{D}_t$  and receives a reward  $y_t$  satisfying the expectation,
$$
\operatorname {E} \left[ y _ {t} \mid \boldsymbol {x} _ {t} \right] = \mu \left(\boldsymbol {x} _ {t} ^ {\top} \boldsymbol {\theta} _ {*}\right) \tag {1}
$$
where  $\theta_{*}$  is the inherent vector and  $\mu (\cdot)$  is the link function, such as the identity function or the logistic function. The performance of the agent is measured by the regret such that
$$
R (T) = \sum_ {t = 1} ^ {T} \left(\mu \left(\tilde {\boldsymbol {x}} _ {t} ^ {\top} \boldsymbol {\theta} _ {*}\right) - \mu \left(\boldsymbol {x} _ {t} ^ {\top} \boldsymbol {\theta} _ {*}\right)\right)
$$
where  $\tilde{\pmb{x}}_t = \operatorname{argmax}_{\pmb{x} \in \mathcal{D}_t} \mu(\pmb{x}^\top \pmb{\theta}_*)$  represents the optimal arm in the set  $\mathcal{D}_t$ .
Extensive research has been conducted on the GLB, with most assuming sub-Gaussian rewards [Filippi et al., 2010; Li et al., 2012, 2017; Jun et al., 2017; Lu et al., 2019; Zhou et al., 2019; Han et al., 2021; Li and Wang, 2022]. However, it has been observed that in certain sequential decision-making scenarios, such as financial markets [Cont and Bouchaud, 2000], the occurrence of extreme returns is much more frequent than the standard normal distribution. This phenomenon is known as heavy-tailed behavior [Foss et al., 2013], where the existing algorithms are not suitable. To address this limitation, in this study, we focus on the GLB with heavy-tailed rewards [Bubeck et al., 2013], i.e., the reward obtained at  $t$ -th round satisfies the condition
$$
\operatorname {E} \left[ \left| y _ {t} \right| ^ {1 + \epsilon} \right] \leq u
$$
for some  $\epsilon \in (0,1]$  and  $u > 0$ . Different from the traditional sub-Gaussian setting, heavy-tailed rewards do not decay exponentially and the estimation of expected rewards is significantly impacted.
According to the distinguishing characteristic of heavy-tailed distributions where extreme values occur with high probability, previous studies have developed three main strategies to address the issue in parameter estimation [Audibert and Catoni, 2011; Hsu and Sabato, 2014; Zhang and Zhou, 2018; Ray Chowdhury and Gopalan, 2019; Lugosi and Mendelson, 2021; Zhong et al., 2021; Huang et al., 2022; Diakonikolas et al., 2022; Gorbunov et al., 2022; Kamath et al., 2022; Li and Liu, 2022; Gou et al., 2023]. One such strategy is truncation Audibert and Catoni [2011], which mitigates the impact of extreme values by truncating them. A recently proposed strategy is the mean of medians approach [Zhong et al., 2021], which involves partitioning the samples drawn from the heavy-tailed distribution into multiple groups, taking the median within each group, and computing the mean of these medians. It intuitively reduces the impact of extreme samples, as extreme samples are distributed to both sides, thus the median value is more robust. The third strategy is median of means [Hsu and Sabato, 2014], which adjusts the order of calculating mean and taking the median in the mean of medians strategy.
Most existing algorithms for heavy-tailed bandit problems are derived from aforementioned three strategies, with a primary focus on the SLB model [Medina and Yang, 2016; Shao et al., 2018; Xue et al., 2020]. To provide a comprehensive overview and facilitate comparison, we present a summary of our results and previous findings on linear bandits with heavy-tailed rewards in Table 1. For the sake of clarity, the presented regret bounds in Table 1 are under the assumption that the rewards have finite variance. The computational complexity only takes into account multiplication and division operations. Although Shao et al. [2018] and Xue et al. [2020] achieve nearly optimal regret for
infinite-armed and finite-armed SLB, respectively, their algorithms are computationally expensive. The latest work utilizing the mean of medians approach demonstrates efficiency but is limited to symmetric rewards [Zhong et al., 2021]. Therefore, designing efficient heavy-tailed algorithms for GLB with symmetric and asymmetric rewards is an interesting and non-trivial challenge.
Through the delicate employment of heavy-tailed strategies, our contributions to the generalized linear bandit problem with heavy-tailed rewards can be summarized as follows:
- We develop two novel algorithms, CRTM and CRMM, which utilize the truncation strategy and mean of medians strategy, respectively. Both algorithms exhibit a sublinear regret bound of  $\widetilde{O}(dT^{\frac{1}{1+\epsilon}})$ , which is amolst optimal as the lower bound is  $\Omega(dT^{\frac{1}{1+\epsilon}})$  [Shao et al., 2018].
- CRTM reduces the computational complexity from  $O(T^2)$  to  $O(T)$  when compared to existing truncation-based algorithms [Shao et al., 2018; Xue et al., 2020], while CRMM reduces the number of estimator required per round from  $O(\log T)$  to only one, as compared to existing median-of-means-based algorithms [Shao et al., 2018; Xue et al., 2020].
- When  $\epsilon = 1$ , the regret bounds of CRTM and CRMM improves a logarithmic factor of order  $\frac{1}{2\alpha}$  and  $\frac{1}{2\alpha} - \frac{1}{2}$  for some  $\alpha \in (0,1)$ , respectively, over the recently proposed method of Zhong et al. [2021] $^2$ . Notably, CRTM extends the method of Zhong et al. [2021] from symmetric rewards to general case, making CRTM more practical.
- We conduct numerical experiments to demonstrate that our proposed algorithms not only achieve a lower regret bound but also require fewer computational resources when applied to heavy-tailed bandit problems.
In this section, we briefly review the related work on linear bandits. Through out the paper, the  $p$ -norm of a vector  $\boldsymbol{x} \in \mathbb{R}^d$  is  $\| \boldsymbol{x} \|_p = (|x_1|^p + \ldots + |x_d|^p)^{1/p}$ . Given a positive definite matrix  $\mathbf{A} \in \mathbb{R}^{d \times d}$ , the weighted Euclidean norm of the vector  $\boldsymbol{x}$  is  $\| \boldsymbol{x} \|_{\mathbf{A}} = \sqrt{\boldsymbol{x}^\top \mathbf{A}\boldsymbol{x}}$ .
2.1 Generalized Linear Bandits
Filippi et al. [2010] was the first to address the generalized linear bandit problem and proposed an algorithm with a regret bound of  $\widetilde{O}(d\sqrt{T})$ . However, their algorithm is not efficient as it requires storing all the action-feedback pairs encountered so far and performing maximum likelihood estimation at each step. A notable improvement was presented by Zhang et al. [2016] with the introduction of an efficient algorithm called  $\mathrm{OL}^2\mathbf{M}$ , whose space and time complexity at each epoch does not grow over time and achieves a  $\widetilde{O}(d\sqrt{T})$  regret. However, their algorithm is limited to the logistic link function. Later, Jun et al. [2017] extended  $\mathrm{OL}^2\mathbf{M}$  to generic link functions while still maintaining the  $\widetilde{O}(d\sqrt{T})$  regret bound. Ding et al. [2021] proposed another efficient generalized linear bandit algorithm following the line of Thompson sampling scheme.
The main challenge in the bandit problem is the trade-off between exploration and exploitation. To address this issue, the most commonly used approach is the confidence-region-based method, specifically for the linear bandit model with infinite arms [Dani et al., 2008; Abbasi-yadkori et al., 2011; Zhang et al., 2016]. Here we take the algorithm  $\mathrm{OL}^2\mathbf{M}$  to give a brife introduction to this approach [Zhang et al., 2016]. With the arrival of a new trial  $(x_{t},y_{t})$  in the  $t$ -th epoch,  $\mathrm{OL}^2\mathbf{M}$  first constructs a surrogate loss  $\ell_t(\pmb {\theta})$  satisfying  $\nabla \ell_t(\pmb {\theta}) = (-y_t + \mu (\pmb {x}_t^\top \pmb {\theta}))\pmb {x}_t$ . Then,  $\mathrm{OL}^2\mathbf{M}$  employs a variant of the online Newton step (ONS) to update the estimated parameters, i.e.,
$$
\hat {\boldsymbol {\theta}} _ {t + 1} ^ {N} = \underset {\boldsymbol {\theta} \in \mathbb {R} ^ {d}} {\operatorname {a r g m i n}} \frac {\left\| \boldsymbol {\theta} - \hat {\boldsymbol {\theta}} _ {t} ^ {N} \right\| _ {\mathbf {V} _ {t + 1}} ^ {2}}{2} + \langle \boldsymbol {\theta} - \hat {\boldsymbol {\theta}} _ {t} ^ {N}, \nabla \ell_ {t} \left(\hat {\boldsymbol {\theta}} _ {t} ^ {N}\right) \rangle . \tag {2}
$$
Here,  $\mathbf{V}_{t + 1} = \mathbf{V}_t + \frac{\kappa}{2}\pmb {x}_t\pmb{x}_t^\top$  for  $\kappa >0$ , and the initialized matrix  $\mathbf{V}_1 = \lambda \mathbf{I}_d$  for  $\lambda >0$ . Subsequently,  $\mathrm{OL}^2\mathrm{M}$  constructs a confidence region  $\mathcal{C}_{t + 1}$  centered at the estimated parameter  $\hat{\theta}_{t + 1}^{N}$ , such that
$$
\mathcal {C} _ {t + 1} = \left\{\boldsymbol {\theta} \in \mathbb {R} ^ {d} \mid \| \boldsymbol {\theta} - \hat {\boldsymbol {\theta}} _ {t + 1} ^ {N} \| _ {\mathbf {V} _ {t + 1}} ^ {2} \leq \gamma_ {t + 1} \right\} \tag {3}
$$
For  $\epsilon_{1} > \epsilon_{2} > 0$ , if the  $(1 + \epsilon_{1})$ -th moment of rewards exists, then the  $(1 + \epsilon_{2})$ -th moment of rewards is bounded [Xue et al., 2020]. Thus, CRTM and CRMM achieve this regret improvement when  $\epsilon \geq 1$ .
where  $\gamma_{t + 1} = O(d\log t)$  indicating the uncertainty of the estimation and the unknown parameter  $\theta_{*}$  lies in this region with high probability. Finally,  $\mathrm{OL}^2\mathbf{M}$  selects the most promising arm  $x_{t + 1}$  according to the principle of "optimization in the face of uncertainty", i.e.,
$$
\left(\boldsymbol {x} _ {t + 1}, \tilde {\boldsymbol {\theta}} _ {t + 1}\right) = \underset {\boldsymbol {x} \in \mathcal {D} _ {t + 1}, \boldsymbol {\theta} \in \mathcal {C} _ {t + 1}} {\operatorname {a r g m a x}} \langle \boldsymbol {x}, \boldsymbol {\theta} \rangle . \tag {4}
$$
2.2 Bandit Learning with Heavy-tailed Rewards
Most of the existing work developed heavy-tailed bandit algorithms using truncation and median of means strategies [Bubeck et al., 2013; Medina and Yang, 2016; Shao et al., 2018; Xue et al., 2020; Huang et al., 2022]. Bubeck et al. [2013] first conducted extensive research on multi-armed bandits with heavy-tailed rewards and achieved a logarithmic regret bound. Medina and Yang [2016] extended it to the SLB model and introduced two algorithms that achieve regret bounds of  $\widetilde{O}(dT^{\frac{2+\epsilon}{2(1+\epsilon)}})$  and  $\widetilde{O}(d^{\frac{1}{2}}T^{\frac{1+2\epsilon}{1+3\epsilon}} + dT^{\frac{1+\epsilon}{1+3\epsilon}})$ , respectively. Shao et al. [2018] improved upon the results of Medina and Yang [2016] by a more delicate application of heavy-tailed strategies, achieving a regret bound of  $\widetilde{O}(dT^{\frac{1}{1+\epsilon}})$ . Xue et al. [2020] investigated the case with finite arms and provided two algorithms that attained regret bounds of  $\widetilde{O}(d^{\frac{1}{2}}T^{\frac{1}{1+\epsilon}})$ . Recently, Zhong et al. [2021] proposed the mean of medians estimator for the super heavy-tailed bandit problem, but the rewards are limited to symmetric distributions. Applying this estimator to the GLB algorithm of Jun et al. [2017] yields a heavy-tailed GLB algorithm that achieves the regret bound of  $O(d(\log T)^{\frac{1}{2\alpha} + \frac{3}{2}}T^{\frac{1}{2}})$  for some  $\alpha \in (0,1)$ . To illustrate the basic idea of adopting different heavy-tailed strategies in the bandit model, we briefly describe three representative algorithms.
For the algorithm exploiting truncation strategy, we take the algorithm TOFU as an instance [Shao et al., 2018]. With the trials up to round  $t$ , TOFU truncates the rewards  $d$  times as follows,
$$
\bar {Y} _ {t} ^ {i} = \left[ y _ {1} \mathbb {I} _ {\left| u _ {1} ^ {i} (t) y _ {1} \right| \leq h _ {t}}, \dots , y _ {t} \mathbb {I} _ {\left| u _ {t} ^ {i} (t) y _ {t} \right| \leq h _ {t}} \right], i = 1, 2, \dots , d \tag {5}
$$
where  $h_t = O(t^{\frac{1 - t}{2(1 + \epsilon)}})$  is the truncated criterion, and  $u_{\tau}^{i}(t)$  denotes the element in the  $i$ -th row and  $\tau$ -th column of matrix  $\widetilde{\mathbf{V}}_{t+1}^{-1/2}\mathbf{A}_t$ ,  $\mathbf{A}_t = [\pmb{x}_1, \pmb{x}_2, \dots, \pmb{x}_t] \in \mathbb{R}^{d \times t}$  is the matrix composed of selected arms and  $\widetilde{\mathbf{V}}_{t+1} = \mathbf{A}_t\mathbf{A}_t^\top + \mathbf{I}_d$ . Using these truncated rewards, TOFU conducts an estimator as  $\tilde{\pmb{\theta}}_{t+1} = \widetilde{\mathbf{V}}_{t+1}^{-1/2}[u_t^1 \cdot \overline{Y}_t^1, \dots, u_t^d \cdot \overline{Y}_t^d]$  with  $u_t^i \cdot \overline{Y}_t^i = \sum_{\tau=1}^t u_\tau^i(t) y_\tau \mathbb{I}_{|u_\tau^i y_\tau| \leq h_t}$ . TOFU then constructs a confidence region centered on this estimator and selects the promising arm, similar to (3) and (4). Notice that the scalarized parameters  $\{u_\tau^i(t)\}_{\tau=1}^t$  are updated at each epoch, requiring TOFU to store the learning history and truncate all rewards at each epoch. Thus, TOFU is not an online method.
For the algorithm exploiting median of means strategy, it's common to play the chosen arm  $r$  times and get  $r$  sequences of rewards  $\{Y_t^j\}_{j=1}^r$ , where  $Y_t^j = [y_1^j, \ldots, y_t^j]$  is the  $j$ -th reward sequence up to epoch  $t$ . MENU executes least square estimation for each reward sequence and gets estimates, i.e.,
$$
\hat {\boldsymbol {\theta}} _ {t + 1} ^ {j} = \underset {\boldsymbol {\theta} \in \mathbb {R} ^ {d}} {\operatorname {a r g m i n}} \| \mathbf {A} _ {t} ^ {\top} \boldsymbol {\theta} - Y _ {t} ^ {j} \| _ {2} ^ {2} + \| \boldsymbol {\theta} \| _ {2} ^ {2}, j = 1, 2, \dots , r \tag {6}
$$
where  $r = O(\log T)$  [Shao et al., 2018]. Then, the median of means strategy adopted by MENU is operated as follows,
$$
m _ {j} = \operatorname {m e d i a n} \left\{\| \hat {\boldsymbol {\theta}} _ {t + 1} ^ {j} - \hat {\boldsymbol {\theta}} _ {t + 1} ^ {s} \| _ {\widetilde {\mathbf {V}} _ {t + 1}}: s = 1, \dots , r \right\}. \tag {7}
$$
Then, MENU takes the estimator  $\hat{\pmb{\theta}}_{t + 1}^{k_*}$  with  $k_{*} = \mathrm{argmin}_{j\in \{1,2,\dots,r\}}\{m_j\}$  as the center of confidence region. Finally, MENU selects the most promising arm similar to (4).
For the mean of medians method proposed by Zhong et al. [2021], at each epoch  $t$ , the agent first plays the selected arm  $\bar{r}$  times, with a value of  $\bar{r} = O((\log T)^{1 / \alpha})$  for some  $\alpha \in (0,1)$ , and then receives rewards  $\{y_t^j\}_{j=1}^{\bar{r}}$  for these plays. Subsequently, the agent randomly divides the rewards into multiple groups, with each group contains  $\lceil \bar{r}^\alpha \rceil$  rewards. The agent then takes the median of each group and uses the mean of these medians to update the estimator. Notice that the expectation of the median has a bias to the expected reward other than the symmetric distribution. Thus, mean of medians strategy is limited to symmetric distribution. Another point worth mentioning is that  $\bar{r}$  is too
large to try sufficient different arms. For example, the agent can only play  $\lceil T / \bar{r} \rceil = 100$  different arms with  $T = 10^6$  and  $\alpha = 0.62$ , which is obviously unreasonable<sup>3</sup>.
3 Algorithms
In this section, we first introduce the generalized linear bandit model and then demonstrate two novel algorithms based on truncation and mean of medians, respectively.
3.1 Learning Model
The formal description of the generalized linear bandit model is as follows. In each round  $t$ , an agent plays an arm  $x_{t} \in \mathcal{D}_{t}$  and obtains a stochastic reward  $y_{t}$ , which is generated from a generalized linear model represented by the following equation,
$$
\Pr \left(y _ {t} \mid \boldsymbol {x} _ {t}\right) = \exp \left(\frac {y _ {t} \boldsymbol {x} _ {t} ^ {\top} \boldsymbol {\theta} _ {*} - m \left(\boldsymbol {x} _ {t} ^ {\top} \boldsymbol {\theta} _ {*}\right)}{g (\tau)} + h \left(y _ {t}, \tau\right)\right) \tag {8}
$$
where  $\theta_{*}$  is the inherent parameters,  $\tau > 0$  is a known scale parameter, and  $g(\cdot)$  and  $h(\cdot,\cdot)$  are normalizers [P. McCullagh, 1989]. The expectation of  $y_{t}$  is given by
$$
\operatorname {E} \left[ y _ {t} \mid \boldsymbol {x} _ {t} \right] = m ^ {\prime} \left(\boldsymbol {x} _ {t} ^ {\top} \boldsymbol {\theta} _ {*}\right).
$$
Thus,  $m^{\prime}(\cdot)$  is the link function in (1), such that  $\mu (\cdot) = m^{\prime}(\cdot)$ . The reward model can be rewritten as
$$
y _ {t} = \mu \left(\boldsymbol {x} _ {t} ^ {\top} \boldsymbol {\theta} _ {*}\right) + \eta_ {t}
$$
where  $\eta_{t}$  is a random noise satisfying the condition
$$
\mathrm {E} \left[ \eta_ {t} \mid \mathcal {G} _ {t - 1} \right] = 0. \tag {9}
$$
Here,  $\mathcal{G}_{t - 1} \triangleq \{\pmb{x}_1, y_1, \dots, \pmb{x}_{t - 1}, y_{t - 1}, \pmb{x}_t\}$  is a  $\sigma$ -filtration and  $\mathcal{G}_0 = \emptyset$ . Following the existing work [Filippi et al., 2010; Jun et al., 2017; Li et al., 2017], we make standard assumptions as follows.
Assumption 1 The coefficients  $\pmb{\theta}_{*}$  and contextual vectors  $\pmb{x}$  are bounded, such that  $\| \pmb{\theta}_{*}\|_{2} \leq S$  and  $\| \pmb{x}\|_{2} \leq 1$  for all  $\pmb{x} \in \mathcal{D}_t$ , where  $S$  is a known constant.
Assumption 2 The link function  $\mu(\cdot)$  is  $L$ -Lipschitz on  $[-S, S]$ , and continuously differentiable on  $(-S, S)$ . Moreover, there exists some  $\kappa > 0$  such that  $\mu'(z) \geq \kappa$  and  $|\mu(z)| \leq U$  for any  $z \in (-S, S)$ .
3.2 Truncation
Our first algorithm is called Confidence Region with Truncated Mean (CRTM). The complete procedure is provided in Algorithm 1. Here, we consider the heavy-tailed setting, i.e., there exists a constant  $u > 0$ , the rewards admit
$$
\mathrm {E} \left[ | y _ {t} | ^ {1 + \epsilon} \mid \mathcal {G} _ {t - 1} \right] \leq u. \tag {10}
$$
As we have mentioned earlier in Section 2.1, to design effective algorithms for GLB model, constructing a narrow confidence region for the underlying coefficients  $\theta_{*}$  is necessary. However, heavy-tailed rewards that satisfy (10) produce extreme values with high probability, resulting in a confidence region with a large radius. Therefore, a straightforward approach to settle this problem is to truncate the extreme reward to reduce its impact.
In each round  $t$ , CRTM first plays an arm  $x_{t} \in \mathcal{D}_{t}$  and observes the corresponding reward  $y_{t}$ . Then, CRTM truncates the reward  $y_{t}$  using a uniform criterion  $\Gamma = \widetilde{O}\left(T^{\frac{1 - \epsilon}{2(1 + \epsilon)}}\right)$ , such that
$$
\tilde {y} _ {t} = y _ {t} \mathbb {I} _ {\| \boldsymbol {x} _ {t} \| _ {\mathbf {V} _ {t} ^ {- 1}} | y _ {t} | \leq \Gamma}
$$
$^3\alpha = 0.62$  is nearly optimal for  $\epsilon = 1$  according to the experiments of Zhong et al. [2021].
Algorithm 1 Confidence Region with Truncated Mean (CRTM)
Input:  $\delta, \epsilon, u, \kappa, S, \lambda = \max\{1, \kappa/2\}$  and  $T \in \mathbb{N}_+$
1: Initialize  $\hat{\theta}_{1} = \mathbf{0}$  and  $\mathbf{V}_1 = \lambda \mathbf{I}_d$
2: Define the truncation criterion  $\Gamma = 2\left(u / \ln (4T / \delta)\right)^{\frac{1}{1 + \epsilon}}\left(d\ln \left(1 + \frac{\kappa T}{2\lambda d}\right) / \kappa\right)^{\frac{1}{2}}T^{\frac{1 - \epsilon}{2(1 + \epsilon)}}$
3: for  $t = 1,2,\ldots ,T$  do
4:  $(\pmb{x}_t,\tilde{\pmb{\theta}}_t) = \mathrm{argmax}_{\pmb{x}\in \mathcal{D}_t,\pmb {\theta}\in \mathcal{C}_t}\langle \pmb {x},\pmb {\theta}\rangle$
5: Play the arm  $\pmb{x}_t$  and observe the payoff  $y_t$
6: Truncate the observed payoff  $\tilde{y}_t = y_t\mathbb{I}_{\| \boldsymbol{x}_t\|_{\mathbf{V}_t^{-1}}|y_t|\leq \Gamma}$
7: Compute the gradient  $\nabla \tilde{\ell}_t(\hat{\pmb{\theta}}_t) = (-\tilde{y}_t + \mu (\pmb{x}_t^\top \hat{\pmb{\theta}}_t))\pmb{x}_t$
8: Update  $\mathbf{V}_{t + 1} = \mathbf{V}_t + \frac{\kappa}{2}\pmb {x}_t\pmb{x}_t^\top$
9: Update the estimator