SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiayi Huang $^{1}$  Han Zhong $^{1}$  Liwei Wang $^{1,2}$  Lin F. Yang $^{3}$
$^{2}$ National Key Laboratory of General Artificial Intelligence,
<sup>3</sup>University of California, Los Angeles
{jyhuang，hanzhong}@stu.pku.edu.cn,
wanglw@cis.pku.edu.cn, linyang@ee.ucla.edu
While numerous works have focused on devising efficient algorithms for reinforcement learning (RL) with uniformly bounded rewards, it remains an open question whether sample or time-efficient algorithms for RL with large state-action space exist when the rewards are heavy-tailed, i.e., with only finite  $(1 + \epsilon)$ -th moments for some  $\epsilon \in (0,1]$ . In this work, we address the challenge of such rewards in RL with linear function approximation. We first design an algorithm, HEAVY-OFUL, for heavy-tailed linear bandits, achieving an instance-dependent  $T$ -round regret of  $\widetilde{O}\left(dT^{\frac{1 - \epsilon}{2(1 + \epsilon)}}\sqrt{\sum_{t = 1}^{T}\nu_{t}^{2}} + dT^{\frac{1 - \epsilon}{2(1 + \epsilon)}}\right)$ , the first of this kind. Here,  $d$  is the feature dimension, and  $\nu_{t}^{1 + \epsilon}$  is the  $(1 + \epsilon)$ -th central moment of the reward at the  $t$ -th round. We further show the above bound is minimax optimal when applied to the worst-case instances in stochastic and deterministic linear bandits. We then extend this algorithm to the RL settings with linear function approximation. Our algorithm, termed as HEAVY-LSVI-UCB, achieves the first computationally efficient instance-dependent  $K$ -episode regret of  $\widetilde{O}\left(d\sqrt{H\mathcal{U}^{*}} K^{\frac{1}{1 + \epsilon}} + d\sqrt{H\mathcal{V}^{*}K}\right)$ . Here,  $H$  is length of the episode, and  $\mathcal{U}^{*},\mathcal{V}^{*}$  are instance-dependent quantities scaling with the central moment of reward and value functions, respectively. We also provide a matching minimax lower bound  $\Omega(dHK^{\frac{1}{1 + \epsilon}} + d\sqrt{H^{3}K})$  to demonstrate the optimality of our algorithm in the worst case. Our result is achieved via a novel robust self-normalized concentration inequality that may be of independent interest in handling heavy-tailed noise in general online regression problems.
Designing efficient reinforcement learning (RL) algorithms for large state-action space is a significant challenge within the RL community. A crucial aspect of RL is understanding the reward functions, which directly impacts the quality of the agent's policy. In certain real-world situations, reward distributions may exhibit heavy-tailed behavior, characterized by the occurrence of extremely large values at a higher frequency than expected in a normal distribution. Examples include image noise in signal processing [14], stock price fluctuations in financial markets [12, 18], and value functions in online advertising [11, 19]. However, much of the existing RL literature assumes rewards to be either uniformly bounded or light-tailed (e.g., sub-Gaussian). In such light-tailed settings, the primary challenge lies in learning the transition probabilities, leading most studies to assume deterministic rewards for ease of analysis [5, 22, 15]. As we will demonstrate, the complexity of learning reward
functions may dominate in heavy-tailed settings. Consequently, the performance of traditional algorithms may decline, emphasizing the need for the development of new, efficient algorithms specifically designed to handle heavy-tailed rewards.
Heavy-tailed distributions have been extensively studied in the field of statistics [9, 27] and in more specific online learning scenarios, such as bandits [7, 28, 31, 34, 40]. However, there is a dearth of theoretical research in RL concerning heavy-tailed rewards, whose distributions only admit finite  $(1 + \epsilon)$ -th moment for some  $\epsilon \in (0,1]$ . One notable exception is Zhuang and Sui [44], which made a pioneering effort in establishing worst-case regret guarantees in tabular Markov Decision Processes (MDPs) with heavy-tailed rewards. However, their algorithm cannot handle RL settings with large state-action space. Moreover, their reliance on truncation-based methods is sub-optimal as these methods heavily depend on raw moments, which do not vanish in deterministic cases. Therefore, a natural question arises:
Can we derive sample and time-efficient algorithms for RL with large state-action space that achieve instance-dependent regret in the presence of heavy-tailed rewards?
In this work, we focus on linear MDPs [35, 22] with heavy-tailed rewards and answer the above question affirmatively. We say a distribution is heavy-tailed if it only admits finite  $(1 + \epsilon)$ -th moment for some  $\epsilon \in (0,1]$ . Our contributions are summarized as follows.
- We first propose a computationally efficient algorithm HEAVY-OFUL for heavy-tailed linear bandits. Such a setting can be regarded as a special case of linear MDPs. HEAVY-OFUL achieves an instance-dependent  $T$ -round regret of  $\widetilde{O}\left(dT^{\frac{1 - \epsilon}{2(1 + \epsilon)}}\sqrt{\sum_{t = 1}^{T}\nu_{t}^{2}} +dT^{\frac{1 - \epsilon}{2(1 + \epsilon)}}\right)$ , the first of this kind. Here  $d$  is the feature dimension and  $\nu_{t}^{1 + \epsilon}$  is the  $(1 + \epsilon)$ -th central moment of the reward at the  $t$ -th round. The instance-dependent regret bound has a main term that only depends on the summation of central moments, and therefore does not have a  $\sqrt{T}$  term. Our regret bound is shown to be minimax optimal in both stochastic and deterministic linear bandits (See Remark 4.2 for details).
- We then extend this algorithm to time-inhomogeneous linear MDPs with heavy-tailed rewards, resulting in a new computationally efficient algorithm HEAVY-LSVI-UCB, which achieves a  $K$ -episode regret scaling as  $\widetilde{O}\left(d\sqrt{H\mathcal{U}^{*}} K^{\frac{1}{1 + \epsilon}} + d\sqrt{H\mathcal{V}^{*}K}\right)$  for the first time. Here,  $H$  is the length of the episode and  $\mathcal{U}^*$ ,  $\mathcal{V}^*$  are quantities measuring the central moment of the reward functions and transition probabilities, respectively (See Theorem 5.2 for details). Our regret bound is instance-dependent since the main term only relies on the instance-dependent quantities, which vanishes when the dynamics and rewards are deterministic. When specialized to special cases, our instance-dependent regret recovers the variance-aware regret in Li and Sun [26] (See Remark 5.3 for details) and improves existing first-order regret bounds [33, 26] (See Corollary 5.6 for details).
- We provide a minimax regret lower bound  $\Omega(dHK^{\frac{1}{1+\epsilon}} + d\sqrt{H^3K})$  for linear MDPs with heavy-tailed rewards, which matches the worst-case regret bound implied by our instance-dependent regret, thereby demonstrating the minimax optimality of HEAVY-LSVI-UCB in the worst case.
For better comparisons between our algorithms and state-of-the-art results, we summarize the regrets in Table 1 and 2 for linear bandits and linear MDPs, respectively. More related works are deferred to Appendix A. Remarkably, our results demonstrate that  $\epsilon = 1$  (i.e. finite variance) is sufficient to obtain variance-aware regret bounds of the same order as the case where rewards are uniformly bounded for both linear bandits and linear MDPs. The main technique contribution behind our results is a novel robust self-normalized concentration inequality inspired by Sun et al. [32]. To be more specific, it is a non-trivial generalization of adaptive Huber regression from independent and identically distributed (i.i.d.) case to heavy-tailed online regression settings and gives a self-normalized bound instead of the  $\ell_2$ -norm bound in Sun et al. [32]. Our result is computationally efficient and only scales with the feature dimension,  $d$ ,  $(1 + \epsilon)$ -th central moment of the noise,  $\nu$ , and does not depend on the absolute magnitude as in other self-normalized concentration inequalities [42, 41].
Road Map The rest of the paper is organized as follows. Section 2 introduces heavy-tailed linear bandits and linear MDPs. Section 3 presents the robust self-normalized concentration inequality for general online regression problems with heavy-tailed noise. Section 4 and 5 give the main results for heavy-tailed linear bandits and linear MDPs, respectively. We then conclude in Section 6. Related work, experiments and all proofs can be found in Appendix.
Table 1: Comparisons with previous works on linear bandits.  $d$ ,  $T$ ,  $\{\sigma_t\}_{t\in [T]}$ ,  $\{\nu_t\}_{t\in [T]}$  are feature dimension, the number of rounds, the variance or central moment of the reward at the  $t$ -th round.
Algorithm	Regret	Instance-dependent?	Minimax Optimal?	Deterministic-Optimal?	Heavy-Tailed Rewards?
OFUL [1]	\(\widetilde{O}\left(d\sqrt{T}\right)\)	No	Yes	No	No
IDS-UCB [24]	\(\widetilde{O}\left(d\sqrt{\sum_{t=1}^{T}\sigma_{t}^{2}}+d\right)\)	Yes	Yes	Yes	No
Weighted OFUL+ [41]	No
AdaOFUL [26]	No
MENU and TOFU [31]	\(\widetilde{O}\left(dT^{\frac{1}{1+\epsilon}}\right)\)	No	Yes	No	Yes
HEAVY-OFUL (Ours)	\(\widetilde{O}\left(dT^{\frac{1-\epsilon}{2(1+\epsilon)}}\sqrt{\sum_{t=1}^{T}\nu_{t}^{2}}+dT^{\frac{1-\epsilon}{2(1+\epsilon)}}\right)\)	Yes	Yes	Yes	Yes
Table 2: Comparisons with previous works on time-inhomogeneous linear MDPs.  $d$ ,  $H$ ,  $K$ ,  $V_1^*$ ,  $\mathcal{G}^*$  are feature dimension, the length of the episode, the number of episodes, optimal value function, variance-dependent quantity defined in Li and Sun [26].  $\mathcal{U}^*$ ,  $\mathcal{V}^*$  are defined in Theorem 5.2.
Algorithm	Regret	Central Moment-Dependent?	First-Order?	Minimax Optimal?	Computationally Efficient?	Heavy-Tailed Rewards?
LSVI-UCB[22]	\(\widetilde{O}\left(\sqrt{d^{3}H^{4}K}\right)\)	No	No	No	Yes	No
FORCE [33]	\(\widetilde{O}\left(\sqrt{d^{3}H^{3}V_{1}^{*}K}\right)\)	No	Yes	No	No	No
VOQL [2]LSVI-UCB++ [15]	\(\widetilde{O}\left(d\sqrt{H^{3}K}\right)\)	No	No	Yes	Yes	No
VARA [26]	\(\widetilde{O}\left(d\sqrt{HG^{*}K}\right)\)	Yes	Yes	Yes	Yes	\(\epsilon=1\)
HEAVY-LSVI-UCB (Ours)	\(\widetilde{O}\left(d\sqrt{HU^{*}K^{\frac{1}{1+\epsilon}}+d\sqrt{HV^{*}K}}\right)\)	Yes	Yes	Yes	Yes	Yes
Notations Let  $\| \pmb{a}\| \coloneqq \| \pmb{a}\| _2$ . Let  $[t]\coloneqq \{1,2,\dots ,t\}$ . Let  $\mathcal{B}_d(r)\coloneqq \{\pmb {x}\in \mathbb{R}^d ||\pmb {x}\| \leq r\}$ . Let  $x_{[a,b]}\coloneqq \max \{a,\min \{x,b\}\}$  denote the projection of  $x$  onto the close interval  $[a,b]$ . Let  $\sigma (\{X_s\}_{s\in [t]})$  be the  $\sigma$ -field generated by random vectors  $\{X_s\}_{s\in [t]}$ .
2.1 Heavy-Tailed Linear Bandits
Definition 2.1 (Heterogeneous linear bandits with heavy-tailed rewards). Let  $\{\mathcal{D}_t\}_{t\geq 1}$  denote a series of fixed decision sets, where all  $\phi_t\in \mathcal{D}_t$  satisfy  $\| \phi_t\| \leq L$  for some known upper bound  $L$ . At each round  $t$ , the agent chooses  $\phi_t\in \mathcal{D}_t$ , then receives a reward  $R_{t}$  from the environment. We define the filtration  $\{\mathcal{F}_t\}_{t\geq 1}$  as  $\mathcal{F}_t = \sigma (\{\phi_s,R_s\}_{s\in [t]}\cup \{\phi_{t + 1}\})$  for any  $t\geq 1$ . We assume  $R_{t} = \langle \phi_{t},\pmb{\theta}^{*}\rangle +\varepsilon_{t}$  with the unknown coefficient  $\pmb {\theta}^*\in \mathcal{B}_d(B)$  for some known upper bound  $B$ . The random variable  $\varepsilon_t\in \mathbb{R}$  is  $\mathcal{F}_t$ -measurable and satisfies  $\mathbb{E}[\varepsilon_t|\mathcal{F}_{t - 1}] = 0,\mathbb{E}[|\varepsilon_t|^{1 + \epsilon}|\mathcal{F}_{t - 1}] = \nu_t^{1 + \epsilon}$  for some  $\epsilon \in (0,1]$  with  $\nu_{t}$  being  $\mathcal{F}_{t - 1}$ -measurable.
The agent aims to minimize the  $T$ -round pseudo-regret defined as  $\mathrm{Regret}(T) = \sum_{t=1}^{T} [\langle \phi_t^*, \pmb{\theta}^* \rangle - \langle \phi_t, \pmb{\theta}^* \rangle]$ , where  $\phi_t^* = \operatorname{argmax}_{\pmb{\phi} \in \mathcal{D}_t} \langle \phi, \pmb{\theta}^* \rangle$ .
2.2 Linear MDPs with Heavy-Tailed Rewards
We use a tuple  $M = M(\mathcal{S}, \mathcal{A}, H, \{R_h\}_{h \in [H]}, \{\mathbb{P}_h\}_{h \in [H]})$  to describe the time-inhomogeneous finite-horizon MDP, where  $\mathcal{S}$  and  $\mathcal{A}$  are state space and action space, respectively,  $H$  is the length of the episode,  $R_h: \mathcal{S} \times \mathcal{A} \to \mathbb{R}$  is the random reward function with expectation  $r_h: \mathcal{S} \times \mathcal{A} \to \mathbb{R}$ , and  $\mathbb{P}_h: \mathcal{S} \times \mathcal{A} \to \Delta(\mathcal{S})$  is the transition probability function. More details can be found in Puterman [30]. A time-dependent policy  $\pi = \{\pi_h\}_{h \in H}$  satisfies  $\pi_h: \mathcal{S} \to \Delta(\mathcal{A})$  for any  $h \in [H]$ . When the policy is deterministic, we use  $\pi_h(s_h)$  to denote the action chosen at the  $h$ -th step given  $s_h$  by policy  $\pi$ . For any state-action pair  $(s, a) \in \mathcal{S} \times \mathcal{A}$ , we define the state-action value function  $Q_h^\pi(s, a)$  and state value function  $V_h^\pi(s)$  as follows:  $Q_h^\pi(s, a) = \mathbb{E}\left[\sum_{h' = h}^{H} r(s_{h'}, a_{h'})\right| s_h = s, a_h = a]$ ,  $V_h^\pi(s) = Q_h^\pi(s, \pi_h(s))$ , where the expectation is taken with respect to the transition probability of
$M$  and the agent's policy  $\pi$ . If  $\pi$  is randomized, then the definition of  $V$  should have an expectation. Denote the optimal value functions as  $V_h^*(s) = \sup_{\pi} V_h^\pi(s)$  and  $Q_h^*(s,a) = \sup_{\pi} Q_h^\pi(s,a)$ .
We introduce the following shorthands for simplicity. At the  $h$ -th step, for any value function  $V: \mathcal{S} \to \mathbb{R}$ , let  $[\mathbb{P}_hV](s,a) = \mathbb{E}_{s' \sim \mathbb{P}_h(\cdot|s,a)} V(s')$ ,  $[\mathbb{V}_hV](s,a) = [\mathbb{P}_hV^2](s,a) - [\mathbb{P}_hV]^2(s,a)$  denote the expectation and the variance of the next-state value function at the  $h$ -th step given  $(s,a)$ .
We aim to minimize the  $K$ -episode regret defined as  $\mathrm{Regret}(K) = \sum_{k=1}^{K} [V_1^*(s_1^k) - V_1^{\pi^k}(s_1^k)]$ .
In the rest of this section, we introduce linear MDPs with heavy-tailed rewards. We first give the definition of linear MDPs studied in Yang and Wang [35], Jin et al. [22], with emphasis that the rewards in their settings are deterministic or uniformly bounded. Then we focus on the heavy-tailed random rewards.
Definition 2.2. An MDP  $M = M(\mathcal{S},\mathcal{A},H,\{R_h\}_{h\in [H]},\{\mathbb{P}_h\}_{h\in [H]})$  is a time-inhomogeneous finite-horizon linear MDP, if there exist known feature maps  $\phi (s,a):\mathcal{S}\times \mathcal{A}\to \mathcal{B}_d(1)$ , unknown  $d$ -dimensional signed measures  $\{\pmb {\mu}_h^*\}_{h\in [H]}$  over  $\mathcal{S}$  with  $\| \pmb {\mu}_h^* (\mathcal{S})\| := \int_{s\in \mathcal{S}}|\pmb {\mu}(s)|\mathrm{d}s\leq \sqrt{d}$  and unknown coefficients  $\{\pmb {\theta}_h^*\}_{h\in [H]}\subseteq \mathcal{B}_d(B)$  for some known upper bound  $B$  such that
$$
r _ {h} (s, a) = \left\langle \phi (s, a), \boldsymbol {\theta} _ {h} ^ {*} \right\rangle , \quad \mathbb {P} _ {h} (\cdot | s, a) = \left\langle \phi (s, a), \boldsymbol {\mu} _ {h} ^ {*} (\cdot) \right\rangle
$$
for any state-action pair  $(s,a)\in S\times \mathcal{A}$  and timestep  $h\in [H]$ .
Assumption 2.3 (Realizable rewards). For all  $(s,a,h)\in S\times \mathcal{A}\times [H]$ , the random reward  $R_{h}(s,a)$  is independent of next state  $s^{\prime}\sim \mathbb{P}_{h}(\cdot |s,a)$  and admits the linear structure
$$
R _ {h} (s, a) = \left\langle \phi (s, a), \boldsymbol {\theta} _ {h} ^ {*} \right\rangle + \varepsilon_ {h} (s, a),
$$
where  $\varepsilon_h(s,a)$  is a mean-zero heavy-tailed random variable specified below.
We introduce the notation  $\nu_{n}[X] = \mathbb{E}[|X - \mathbb{E}X|^{n}]$  for the  $n$ -th central moment of any random variable  $X$ . And for any random reward function at the  $h$ -th step  $R_{h}: S \times \mathcal{A} \to \mathbb{R}$ , let
$$
\left[ \mathbb {E} _ {h} R _ {h} \right] (s, a) = \mathbb {E} \left[ R _ {h} \left(s _ {h}, a _ {h}\right) \mid \left(s _ {h}, a _ {h}\right) = (s, a) \right],
$$
$$
\left[ \nu_ {1 + \epsilon} R _ {h} \right] (s, a) = \mathbb {E} \left[ \left| \left[ R _ {h} - \mathbb {E} _ {h} R _ {h} \right] \left(s _ {h}, a _ {h}\right) \right| ^ {1 + \epsilon} \right| \left(s _ {h}, a _ {h}\right) = (s, a) ]
$$
denote its expectation and the  $(1 + \epsilon)$ -th central moment given  $(s_h, a_h) = (s, a)$  for short.
Assumption 2.4 (Heavy-tailedness of rewards). Random variable  $\varepsilon_h(s,a)$  satisfies  $[\mathbb{E}_h\varepsilon_h](s,a) = 0$ . And for some known  $\epsilon, \epsilon' \in (0,1]$  and constants  $\nu_R, \nu_{R^\epsilon} \geq 0$ , the following unknown moments of  $\varepsilon_h(s,a)$  satisfy
$$
\left[ \mathbb {E} _ {h} \left| \varepsilon_ {h} \right| ^ {1 + \epsilon} \right] (s, a) \leq \nu_ {R} ^ {1 + \epsilon}, \quad \left[ \nu_ {1 + \epsilon^ {\prime}} \left| \varepsilon_ {h} \right| ^ {1 + \epsilon} \right] (s, a) \leq \nu_ {R ^ {\epsilon}} ^ {1 + \epsilon^ {\prime}}
$$
for all  $(s,a,h)\in S\times \mathcal{A}\times [H]$
Assumption 2.4 generalizes Assumption 2.2 of Li and Sun [26], which is the weakest moment condition on random rewards in the current literature of RL with function approximation. Setting  $\epsilon = 1$  and  $\epsilon' = 1$  immediately recovers their settings.
Assumption 2.5 (Realizable central moments). There are some unknown coefficients  $\{\psi_h^*\}_{h\in [H]}\subseteq$ $\mathcal{B}_d(W)$  for some known upper bound  $W$  such that
$$
\left[ \mathbb {E} _ {h} \left| \varepsilon_ {h} \right| ^ {1 + \epsilon} \right] (s, a) = \langle \phi (s, a), \psi_ {h} ^ {*} \rangle
$$
Remark 2.6. When  $\epsilon = 1$ , that is the rewards have finite variance, Li and Sun [26] use the fact that  $[\nu_{2}R_{h}](s,a) = [\mathbb{V}_{h}R_{h}](s,a) = [\mathbb{E}_{h}R_{h}^{2}](s,a) - [\mathbb{E}_{h}R_{h}]^{2}(s,a)$ , assume the linear realizability of the second moment  $[\mathbb{E}_{h}R_{h}^{2}](s,a)$ , and estimate it instead. However, when  $\epsilon < 1$ , there is no such relationship between the  $(1 + \epsilon)$ -th central moment  $[\nu_{1 + \epsilon}R_{h}](s,a)$  and the  $(1 + \epsilon)$ -th raw moment  $[\mathbb{E}_{h}R_{h}^{1 + \epsilon}](s,a)$ . Thus, we adopt a new approach to estimate  $[\nu_{1 + \epsilon}R_{h}](s,a)$  directly, and bound the error by a novel perturbation analysis of adaptive Huber regression in Appendix C.3.
Assumption 2.7 (Bounded cumulative rewards). For any policy  $\pi$ , let  $\{s_h, a_h, R_h\}_{h \in [H]}$  be a random trajectory following policy  $\pi$ . And define  $r_{\pi} = \sum_{h=1}^{H} [\mathbb{E}_h R_h](s_h, a_h) = \sum_{h=1}^{H} r_h(s_h, a_h)$ . We assume (1)  $0 \leq r_{\pi} \leq \mathcal{H}$ . (2)  $\sum_{h=1}^{H} [\pmb{\nu}_{1+\epsilon} R_h]^{\frac{2}{1+\epsilon}}(s_h, a_h) \leq \mathcal{U}$ . (3)  $\operatorname{Var}(r_{\pi}) \leq \mathcal{V}$ .
Here, (1) gives an upper bound of cumulative expected rewards  $r_{\pi}$ . (2) assumes the summation of  $(1 + \epsilon)$ -th central moment of rewards  $[\pmb{\nu}_{1 + \epsilon}R_h](s_h,a_h)$  is bounded since  $\left[\sum_{h = 1}^{H}[\pmb{\nu}_{1 + \epsilon}R_h](s_h,a_h)\right]^{\frac{2}{1 + \epsilon}} \leq \sum_{h = 1}^{H}[\pmb{\nu}_{1 + \epsilon}R_h]^{\frac{2}{1 + \epsilon}}(s_h,a_h) \leq \mathcal{U}$  due to Jensen's inequality. And (3) is to bound the variance of  $r_{\pi}$  along the trajectory following policy  $\pi$ .
Algorithm 1 Adaptive Huber Regression
Require: Number of total rounds  $T$ , confidence level  $\delta$ , regularization parameter  $\lambda$ ,  $\sigma_{\mathrm{min}}$ , parameters for adaptive Huber regression  $c_0, c_1, \tau_0$ , estimated central moment  $\widehat{\nu}_t$  and moment parameter  $b$  that satisfy  $\nu_t / \widehat{\nu}_t \leq b$  for all  $t \leq T$ .
Ensure: The estimated coefficient  $\theta_{t}$
1:  $\kappa = d\log (1 + TL^2 /(d\lambda \sigma_{\mathrm{min}}^2))$
2: Set  $\pmb{H}_{t - 1} = \lambda \pmb{I} + \sum_{s = 1}^{t - 1}\sigma_s^{-2}\phi_s\phi_s^\top$
3: Set  $\sigma_{t} = \max \left\{\widehat{\nu}_{t},\sigma_{\min},\frac{\|\phi_{t}\|_{\pmb{H}_{t - 1}^{-1}}}{c_{0}},\frac{\sqrt{LB}}{c_{1}^{\frac{1}{4}}(2\kappa b^{2})^{\frac{1}{4}}}\|\phi_{t}\|_{\pmb{H}_{t - 1}^{-1}}^{\frac{1}{2}}\right\}$ .
4: Set  $\tau_{t} = \tau_{0}\frac{\sqrt{1 + w_{t}^{2}}}{w_{t}} t^{\frac{1 - \epsilon}{2(1 + \epsilon)}}$  with  $w_{t} = \left\| \phi_{t} / \sigma_{t}\right\|_{H_{t - 1}^{-1}}$ .
5: Define the loss function  $L_{t}(\pmb {\theta}):= \frac{\lambda}{2}\| \pmb {\theta}\|^{2} + \sum_{s = 1}^{t}\ell_{\tau_{s}}(\frac{y_{s} - \langle\phi_{s},\pmb{\theta}\rangle}{\sigma_{s}}).$
6: Compute  $\pmb{\theta}_t = \mathrm{argmin}_{\pmb{\theta} \in \mathcal{B}_d(B)} L_t(\pmb{\theta})$ .
3 Adaptive Huber Regression
At the core of our algorithms for both heavy-tailed linear bandits and linear MDPs is a new approach - adaptive Huber regression - to handle heavy-tailed noise. Sun et al. [32] imposed adaptive Huber regression to handle i.i.d. heavy-tailed noise by utilizing Huber loss [17] as a surrogate of squared loss. Li and Sun [26] modified adaptive Huber regression for heterogeneous online settings, where the variances in each round are different. However, it is not readily applicable to deal with heavy-tailed noise. Our contribution in this section is to construct a new self-normalized concentration inequality for general online regression problems with heavy-tailed noise.
We first give a brief introduction to Huber loss function and its properties.
Definition 3.1 (Huber loss). Huber loss is defined as
$$
\ell_ {\tau} (x) = \left\{ \begin{array}{l l} \frac {x ^ {2}}{2} & \text {i f} | x | \leq \tau , \\ \tau | x | - \frac {\tau^ {2}}{2} & \text {i f} | x | > \tau , \end{array} \right. \tag {3.1}
$$
where  $\tau > 0$  is referred as a robustness parameter.
Huber loss is first proposed by Huber [17] as a robust version of squared loss while preserving the convex property. Specifically, Huber loss is a quadratic function of  $x$  when  $|x|$  is less than the threshold  $\tau$ , while becomes linearly dependent on  $|x|$  when  $|x|$  grows larger than  $\tau$ . It has the property of strongly convex near zero point and is not sensitive to outliers. See Appendix C.1 for more properties of Huber loss.
Next, we define general online regression problems with heavy-tailed noise, which include heavily-tailed linear bandits as a special case. Then we utilize Huber loss to estimate  $\theta^{*}$ . Below we give the main theorem to bound the deviation of the estimated  $\theta_{t}$  in Algorithm 1 from the ground truth  $\theta^{*}$ .
Definition 3.2. Let  $\{\mathcal{F}_t\}_{t\geq 1}$  be a filtration. For all  $t > 0$ , let random variables  $y_{t},\varepsilon_{t}$  be  $\mathcal{F}_t$ -measurable and random vector  $\phi_t\in \mathcal{B}_d(L)$  be  $\mathcal{F}_{t - 1}$ -measurable. Suppose  $y_{t} = \langle \phi_{t},\pmb{\theta}^{*}\rangle +\varepsilon_{t}$ , where  $\pmb{\theta}^{*}\in \mathcal{B}_{d}(B)$  is an unknown coefficient and
$$
\mathbb {E} [ \varepsilon_ {t} | \mathcal {F} _ {t - 1} ] = 0, \quad \mathbb {E} [ | \varepsilon_ {t} | ^ {1 + \epsilon} | \mathcal {F} _ {t - 1} ] = \nu_ {t} ^ {1 + \epsilon}
$$
for some  $\epsilon \in (0,1]$ . The goal is to estimate  $\theta^{*}$  at any round  $t$  given the realizations of  $\{\phi_s,y_s\}_{s\in [t]}$ .
Theorem 3.3. For the online regression problems in Definition 3.2, we solve for  $\pmb{\theta}_t$  by adaptive Huber regression in Algorithm 1 with  $c_0, c_1, \tau_0$  in Appendix C.2. Then for any  $\delta \in (0,1)$ , with probability at least  $1 - 3\delta$ , for all  $t \leq T$ , we have  $\| \pmb{\theta}_t - \pmb{\theta}^* \|_{\pmb{H}_t} \leq \beta_t$ , where  $\pmb{H}_t$  is defined in Algorithm 1 and
$$
\beta_ {t} = 3 \sqrt {\lambda} B + 2 4 t ^ {\frac {1 - \epsilon}{2 (1 + \epsilon)}} \sqrt {2 \kappa} b (\log 3 T) ^ {\frac {1 - \epsilon}{2 (1 + \epsilon)}} (\log (2 T ^ {2} / \delta)) ^ {\frac {\epsilon}{1 + \epsilon}}. \tag {3.2}
$$
Proof. To derive a tight high-probability bound, we take the most advantage of the properties of Huber loss. A Chernoff bounding technique is used to bound the main error term, which requires a careful analysis of the moment generating function. See Appendix C.2 for a detailed proof.  $\square$
Algorithm 2 HEAVY-OFUL