SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Require: Number of total rounds  $T$ , confidence level  $\delta$ , regularization parameter  $\lambda$ ,  $\sigma_{\mathrm{min}}$ , parameters for adaptive Huber regression  $c_0, c_1, \tau_0$ , confidence radius  $\beta_t$ .
1:  $\kappa = d\log (1 + \frac{TL^2}{d\lambda\sigma_{\mathrm{min}}^2}),\mathcal{C}_0 = \mathcal{B}_d(B),H_0 = \lambda I.$
2: for  $t = 1, \dots, T$  do
3: Observe  $\mathcal{D}_t$
4: Set  $(\phi_t,\cdot) = \mathrm{argmax}_{\boldsymbol {\phi}\in \mathcal{D}_t,\boldsymbol {\theta}\in \mathcal{C}_{t - 1}}\langle \boldsymbol {\phi},\boldsymbol {\theta}\rangle$
5: Play  $\phi_t$  and observe  $R_t, \nu_t$ .
6: Set  $\sigma_{t} = \max \left\{\nu_{t},\sigma_{\min},\frac{\|\phi_{t}\|_{H_{t - 1}^{-1}}}{c_{0}},\frac{\sqrt{LB}}{c_{1}^{\frac{1}{4}}(2\kappa)^{\frac{1}{4}}}\|\phi_{t}\|_{H_{t - 1}^{-1}}^{\frac{1}{2}}\right\}$ .
7: Set  $\tau_{t} = \tau_{0}\frac{\sqrt{1 + w_{t}^{2}}}{w_{t}} t^{\frac{1 - \epsilon}{2(1 + \epsilon)}}$  with  $w_{t} = \left\| \phi_{t} / \sigma_{t}\right\|_{H_{t - 1}^{-1}}$ .
8: Update  $\pmb{H}_t = \pmb{H}_{t-1} + \sigma_t^{-2}\phi_t\phi_t^\top$ .
9: Solve for  $\pmb{\theta}_t$  by Algorithm 1 and set  $\mathcal{C}_t = \{\pmb {\theta}\in \mathbb{R}^d ||\pmb {\theta} - \pmb {\theta}_t||_{\pmb {H}_t}\leq \beta_t\}$
0: end for
We refer to the regression process in Line 6 of Algorithm 1 as adaptive Huber regression in line with Sun et al. [32] to emphasize that the value of robustness parameter  $\tau_{t}$  is chosen to adapt to data for a better trade-off between bias and robustness. Specifically, since we are in the online setting,  $\phi_t$  are dependent on  $\{\phi_s\}_{s < t}$ , which is the key difference from the i.i.d. case in Sun et al. [32] where they set  $\tau_{t} = \tau_{0}$ , for all  $t \leq T$ . Thus, as shown in Line 4 of Algorithm 1, inspired by Li and Sun [26], we adjust  $\tau_{t}$  according to the importance of observations  $w_{t} = \| \phi_{t} / \sigma_{t}\|_{H_{t - 1}^{-1}}$ , where  $\sigma_t$  is specified below. In the case where  $\epsilon < 1$ , different from Li and Sun [26], we first choose  $\tau_{t}$  to be small for robust purposes, then gradually increase it with  $t$  to reduce the bias.
Next, we illustrate the reason for setting  $\sigma_{t}$  via Line 3 of Algorithm 1. We use  $\widehat{\nu}_{t} \in \mathcal{F}_{t-1}$  to estimate the central moment  $\nu_{t}$  and use moment parameter  $b$  to measure the closeness between  $\widehat{\nu}_{t}$  and  $\nu_{t}$ . When we choose  $\widehat{\nu}_{t}$  as an upper bound of  $\nu_{t}$ ,  $b$  becomes a constant that equals to 1. And  $\sigma_{\min}$  is a small positive constant to avoid singularity. The last two terms with respect to  $c_{0}$  and  $c_{1}$  are set according to the uncertainty  $\| \phi_{t} \|_{H_{t-1}^{-1}}$ . In addition, setting the parameter  $c_{0} \leq 1$  yields  $w_{t} \leq 1$ , which is essential to meet the condition of elliptical potential lemma [1].
Remark 3.4. The error bound  $\beta_{t}$  in (3.2) is only related to the feature dimension  $d$  and moment parameter  $b$ . While the Bernstein-style self-normalized concentration bounds [42, 41] depend on the magnitude of  $\varepsilon_{t}$ , thus cannot handle heavy-tailed errors.
4 Linear Bandits
In this section, we show the algorithm HEAVY-OFUL in Algorithm 2 for heavy-tailed linear bandits in Definition 2.1. We first give a brief algorithm description, and then provide a theoretical regret analysis.
HEAVY-OFUL follows the principle of Optimism in the Face of Uncertainty (OFU) [1], and uses adaptive Huber regression in Section 3 to maintain a set  $\mathcal{C}_t$  that contains the unknown coefficient  $\theta^*$  with high probability. Specifically, at the  $t$ -th round, HEAVY-OFUL estimates the expected reward of any arm  $\phi$  as  $\max_{\theta \in \mathcal{C}_{t - 1}}\langle \phi ,\theta \rangle$ , and selects the arm that maximizes the estimated reward. The agent then receives the reward  $R_{t}$  and updates the confidence set  $\mathcal{C}_t$  based on the information up to round  $t$  with its center  $\theta_t$  computed by adaptive Huber regression as in Line 9 of Algorithm 2.
4.2 Regret Analysis
We next give the instance-dependent regret upper bound of HEAVY-OFUL in Theorem 4.1.
Theorem 4.1. For the heavy-tailed linear bandits in Definition 2.1, we set  $c_{0}, c_{1}, \tau_{0}, \beta_{t}$  in Algorithm 2 according to Theorem 3.3 with  $b = 1$ . Besides, let  $\lambda = d / B^{2}$ , and  $\sigma_{\min} = 1 / \sqrt{T}$ . Then with
probability at least  $1 - 3\delta$ , the regret of HEAVY-OFUL is bounded by
$$
\operatorname {R e g r e t} (T) = \widetilde {O} \left(d T ^ {\frac {1 - \epsilon}{2 (1 + \epsilon)}} \sqrt {\sum_ {t = 1} ^ {T} \nu_ {t} ^ {2}} + d T ^ {\frac {1 - \epsilon}{2 (1 + \epsilon)}}\right).
$$
Proof. The proof uses the self-normalized concentration inequality of adaptive Huber regression and a careful analysis to bound the summation of bonuses. See Appendix D.1 for a detailed proof.  $\square$
Remark 4.2. Theorem 4.1 shows HEAVY-OFUL achieves an instance-dependent regret bound. When we assume  $\nu_{t},\forall t\geq 1$  have uniform upper bound  $\nu$  (which can be treated as a constant), then the bound is reduced to  $\widetilde{O} (dT^{\frac{1}{1 + \epsilon}})$ . It matches the lower bound  $\Omega (dT^{\frac{1}{1 + \epsilon}})$  by Shao et al. [31] up to logarithmic factors. In the deterministic scenario, where  $\epsilon = 1$  and  $\nu_{t} = 0$ , for all  $t\geq 1$ , the bound is reduced to  $\widetilde{O} (d)$ . It matches the lower bound  $\Omega (d)^{1}$  up to logarithmic factors.
5 Linear MDPs
In this section, we show the algorithm HEAVY-LSVI-UCB in Algorithm 3 for linear MDP with heavy-tailed rewards defined in Section 2.2. Let  $\phi_{k,h} \coloneqq \phi(s_{k,h}, a_{k,h})$  for short. We first give the algorithm description intuitively, then provide the computational complexity and regret bound.
HEAVY-LSVI-UCB features a novel combination of adaptive Huber regression in Section 3 and existing algorithmic frameworks for linear MDPs with bounded rewards [22, 15]. At a high level, HEAVY-LSVI-UCB employs separate estimation techniques to handle heavy-tailed rewards and transition kernels. Specifically, we utilize adaptive Huber regression proposed in Section 3 to estimate heavy-tailed rewards and weighted ridge regression [42, 15] to estimate the expected next-state value functions. Then, it follows the value iteration scheme to update the optimistic and pessimistic estimation of the optimal value function  $Q_h^k$ ,  $V_h^k$  and  $\check{Q}_h^k$ ,  $\check{V}_h^k$ , respectively, via a rare-switching policy as in Line 7 to 15 of Algorithm 3. We highlight the key steps of HEAVY-LSVI-UCB as follows.
Estimation for expected heavy-tailed rewards Since the expected rewards have linear structure in linear MDPs, i.e.,  $r_h(s,a) = \langle \phi (s,a),\pmb{\theta}_h^*\rangle$  , we use adaptive Huber regression to estimate  $\pmb{\theta}_{h}^{*}$  ..
$$
\boldsymbol {\theta} _ {k, h} = \underset {\boldsymbol {\theta} \in \mathcal {B} _ {d} (B)} {\operatorname {a r g m i n}} \frac {\lambda_ {R}}{2} \| \boldsymbol {\theta} \| ^ {2} + \sum_ {i = 1} ^ {k} \ell_ {\tau_ {i, h}} \left(\frac {R _ {i , h} - \langle \phi_ {i , h} , \boldsymbol {\theta} \rangle}{\nu_ {i , h}}\right), \tag {5.1}
$$
where  $\nu_{i,h}$  will be specified later.
Estimation for central moment of rewards By Assumption 2.5, the  $(1 + \epsilon)$ -th central moment of rewards is linear in  $\phi$ , i.e.,  $[\nu_{1 + \epsilon}R_h](s,a) = \langle \phi (s,a),\psi_h^*\rangle$ . Motivated by this, we estimate  $\psi_h^*$  by adaptive Huber regression as
$$
\psi_ {k, h} = \underset {\psi \in \mathcal {B} _ {d} (W)} {\operatorname {a r g m i n}} \frac {\lambda_ {R}}{2} \| \psi \| ^ {2} + \sum_ {i = 1} ^ {k} \ell_ {\widetilde {\tau} _ {i, h}} \left(\frac {\left| \varepsilon_ {i , h} \right| ^ {1 + \epsilon} - \langle \phi_ {i , h} , \psi \rangle}{\nu_ {i , h}}\right), \tag {5.2}
$$
where  $W$  is the upper bound of  $\| \psi_h^*\|$  defined in Assumption 2.5. Since  $\varepsilon_{i,h}$  is intractable, we estimate it by  $\widehat{\varepsilon}_{i,h} = R_{i,h} - \langle \phi_{i,h},\pmb{\theta}_{i,h}\rangle$ , which gives  $\widehat{\psi}_{k,h}$  as
$$
\widehat {\psi} _ {k, h} = \underset {\boldsymbol {\psi} \in \mathcal {B} _ {d} (W)} {\operatorname {a r g m i n}} \frac {\lambda_ {R}}{2} \| \boldsymbol {\psi} \| ^ {2} + \sum_ {i = 1} ^ {k} \ell_ {\widetilde {\tau} _ {i, h}} \left(\frac {\left| \widehat {\varepsilon} _ {i , h} \right| ^ {1 + \epsilon} - \langle \boldsymbol {\phi} _ {i , h} , \boldsymbol {\psi} \rangle}{\nu_ {i , h}}\right). \tag {5.3}
$$
The inevitable error between  $\psi_{k,h}$  and  $\widehat{\psi}_{k,h}$  can be quantified by a novel perturbation analysis of adaptive Huber regression in Appendix C.3.
<sup>1</sup>Consider the decision set consisting of unit bases of  $\mathbb{R}^d$ . Given that each arm pull can only yield information about a single coordinate, it is inevitable that  $d$  pulls are required for exploration.
Algorithm 3 HEAVY-LSVI-UCB
Require: Number of episodes  $K$ , confidence level  $\delta$ , regularization parameter  $\lambda_R, \lambda_V, \nu_{\mathrm{min}}, \sigma_{\mathrm{min}}$ , confidence radius  $\beta_{R^e}, \beta_0, \beta_R, \beta_V$ .
$$
\begin{array}{l} 1: \kappa = d \log \left(1 + \frac {K}{d \lambda_ {R} \nu_ {\min} ^ {2}}\right). \\ 2: \boldsymbol {\theta} _ {0, h} = \widehat {\boldsymbol {w}} _ {0, h} = \tilde {\boldsymbol {w}} _ {0, h} = \mathbf {0}, \boldsymbol {H} _ {0, h} = \lambda_ {R} \boldsymbol {I}, \boldsymbol {\Sigma} _ {0, h} = \lambda_ {V} \boldsymbol {I}, \text {U P D A T E} = \text {T R U E}. \\ 3: \text {f o r} k = 1, \dots , K \text {d o} \\ 4: \quad V _ {H + 1} ^ {k} (\cdot) = \tilde {V} _ {H + 1} ^ {k} (\cdot) = 0. \\ 5: \quad \text {f o r} h = H, \dots , 1 \mathbf {d o} \\ \end{array}
$$
6: Compute  $\theta_{k - 1,h},\widehat{w}_{k - 1,h}$  and  $\check{w}_{k - 1,h}$  via (5.1) and (5.8).
7: if UPDATE then
$$
\begin{array}{l} Q _ {h} ^ {k} (\cdot , \cdot) = \left\langle \phi (\cdot , \cdot), \boldsymbol {\theta} _ {k - 1, h} + \widehat {\boldsymbol {w}} _ {k - 1, h} \right\rangle + \beta_ {R, k - 1} \| \phi (\cdot , \cdot) \| _ {\boldsymbol {H} _ {k - 1, h} ^ {- 1}} + \beta_ {V} \| \phi (\cdot , \cdot) \| _ {\boldsymbol {\Sigma} _ {k - 1, h} ^ {- 1}}. \\ 9: \quad \check {Q} _ {h} ^ {k} (\cdot , \cdot) = \left\langle \phi (\cdot , \cdot), \boldsymbol {\theta} _ {k - 1, h} + \check {\boldsymbol {w}} _ {k - 1, h} \right\rangle - \beta_ {R, k - 1} \| \phi (\cdot , \cdot) \| _ {\boldsymbol {H} _ {k - 1, h} ^ {- 1}} - \beta_ {V} \| \phi (\cdot , \cdot) \| _ {\boldsymbol {\Sigma} _ {k - 1, h} ^ {- 1}}. \\ 0: \quad Q _ {h} ^ {k} (\cdot , \cdot) = \min  \left\{Q _ {h} ^ {k} (\cdot , \cdot), Q _ {h} ^ {k - 1} (\cdot , \cdot), \mathcal {H} \right\}, \check {Q} _ {h} ^ {k} (\cdot , \cdot) = \max  \left\{\check {Q} _ {h} ^ {k} (\cdot , \cdot), \check {Q} _ {h} ^ {k - 1} (\cdot , \cdot), 0 \right\}. \\ 1: \quad \text {S e t} k _ {\text {l a s t}} = k. \\ \end{array}
$$
2: else
$$
3: \quad Q _ {h} ^ {k} (\cdot , \cdot) = Q _ {h} ^ {k - 1} (\cdot , \cdot), \tilde {Q} _ {h} ^ {k} (\cdot , \cdot) = \tilde {Q} _ {h} ^ {k - 1} (\cdot , \cdot).
$$
4: end if
$$
5: \quad V _ {h} ^ {k} (\cdot) = \max  _ {a} Q _ {h} ^ {k} (\cdot , a), \tilde {V} _ {h} ^ {k} (\cdot) = \max  _ {a} \tilde {Q} _ {h} ^ {k} (\cdot , a), \pi_ {h} ^ {k} (\cdot) = \operatorname {a r g m a x} _ {a} Q _ {h} ^ {k} (\cdot , a).
$$
$$
7: \quad \text {O b s e r v e i n i t i a l s t a t e} s _ {k, 1}.
$$
$$
8: \quad \mathbf {f o r} h = 1, \dots , H \mathbf {d o}
$$
$$
9: \quad \text {T a k e} a _ {k, h} = \pi_ {h} ^ {k} \left(s _ {k, h}\right) \text {a n d o b s e r v e} R _ {k, h}, s _ {k, h + 1}.
$$
$$
2 0: \quad \text {S e t} \nu_ {k, h} \text {a n d} \sigma_ {k, h} \text {a c c o r d i n g t o (5 . 4) a n d (5 . 9) r e s p e c i t y l y}.
$$
$$
\begin{array}{l} \text {2 1 :} \quad \text {S e t} \tau_ {k, h} = \tau_ {0} \frac {\sqrt {1 + w _ {k , h} ^ {2}}}{w _ {k , h}} k ^ {\frac {1 - \epsilon}{2 (1 + \epsilon)}}, \widetilde {\tau} _ {k, h} = \widetilde {\tau} _ {0} \frac {\sqrt {1 + w _ {k , h} ^ {2}}}{w _ {k , h}} k ^ {\frac {1 - \epsilon^ {\prime}}{2 (1 + \epsilon^ {\prime})}} \text {w i t h} w _ {k, h} = \| \phi_ {k, h} / \nu_ {k, h} \| _ {\boldsymbol {H} _ {k, h} ^ {- 1}}. \end{array}
$$
$$
2: \quad \text {U p d a t e} \boldsymbol {H} _ {k, h} = \boldsymbol {H} _ {k - 1, h} + \frac {1}{\nu_ {k , h} ^ {2}} \phi_ {k, h} \phi_ {k, h} ^ {\top} \text {a n d} \boldsymbol {\Sigma} _ {k, h} = \boldsymbol {\Sigma} _ {k - 1, h} + \frac {1}{\sigma_ {k , h} ^ {2}} \phi_ {k, h} \phi_ {k, h} ^ {\top}.
$$
$$
4: \quad \text {i f} \exists h ^ {\prime} \in [ H ] \text {s u c h t h a t} \det  (\boldsymbol {H} _ {k, h ^ {\prime}}) \geq 2 \det  (\boldsymbol {H} _ {k _ {\text {l a s t}}, h ^ {\prime}}) \text {o r} \det  (\boldsymbol {\Sigma} _ {k, h ^ {\prime}}) \geq 2 \det  (\boldsymbol {\Sigma} _ {k _ {\text {l a s t}}, h ^ {\prime}}) \text {t h e n}
$$
$$
\text {5 :} \quad \text {S e t U P D A T E} = \text {T R U E}.
$$
6: else
$$
\begin{array}{l} 2 7: \quad \text {S e t} = \text {F A L S E}. \\ 8: \quad e n d i f \\ \text {e 9 : e n d f o r} \\ \end{array}
$$
We then set the weight  $\nu_{k,h}$  for adaptive Huber regression as
$$
\nu_ {k, h} = \max  \left\{\widehat {\nu} _ {k, h}, \nu_ {\min }, \frac {\left\| \phi_ {k , h} \right\| _ {\boldsymbol {H} _ {k - 1 , h} ^ {- 1}}}{c _ {0}}, \frac {\sqrt {\operatorname* {m a x} \{B , W \}}}{c _ {1} ^ {\frac {1}{4}} (2 \kappa) ^ {\frac {1}{4}}} \left\| \phi_ {k, h} \right\| _ {\boldsymbol {H} _ {k - 1, h} ^ {- 1}} ^ {\frac {1}{2}} \right\}, \tag {5.4}
$$
where  $\nu_{\mathrm{min}}$  is a small positive constant to avoid the singularity,  $\widehat{\nu}_{k,h}^{1 + \epsilon} = [\widehat{\nu}_{1 + \epsilon}R_h](s_{k,h},a_{k,h}) + W_{k,h}$  is a high-probability upper bound of rewards' central moment  $[\pmb {\nu}_{1 + \epsilon}R_h](s_{k,h},a_{k,h})$  with
$$
\left[ \widehat {\boldsymbol {\nu}} _ {1 + \epsilon} R _ {h} \right] \left(s _ {k, h}, a _ {k, h}\right) = \left\langle \phi_ {k, h}, \widehat {\psi} _ {k - 1, h} \right\rangle , \tag {5.5}
$$
$$
W _ {k, h} = \left(\beta_ {R ^ {\epsilon}, k - 1} + 6 \mathcal {H} ^ {\epsilon} \beta_ {R, k - 1} \kappa\right) \| \phi_ {k, h} \| _ {\boldsymbol {H} _ {k - 1, h} ^ {- 1}}, \tag {5.6}
$$
where  $\kappa$  is defined in Algorithm 3,  $\beta_{R^{\epsilon},k} = \widetilde{O}\big(\sqrt{d}\nu_{R^{\epsilon}}k^{\frac{1 - \epsilon'}{2(1 + \epsilon')}}/\nu_{\min}\big)$  and  $\beta_{R,k} = \widetilde{O}\big(\sqrt{d} k^{\frac{1 - \epsilon}{2(1 + \epsilon)}}\big)$ .
Estimation for expected next-state value functions For any value function  $f:\mathcal{S}\to \mathbb{R}$ , we define the following notations for simplicity:
$$
\boldsymbol {w} _ {h} [ f ] = \int_ {s \in \mathcal {S}} \boldsymbol {\mu} _ {h} ^ {*} (s) f (s) \mathrm {d} s, \quad \widehat {\boldsymbol {w}} _ {k, h} [ f ] = \Sigma_ {k, h} ^ {- 1} \sum_ {i = 1} ^ {k} \sigma_ {i, h} ^ {- 2} \phi_ {i, h} f \left(s _ {i, h + 1}\right), \tag {5.7}
$$
where  $\sigma_{i,h}$  will be specified later. Note for any state-action pair  $(s,a)\in S\times \mathcal{A}$ , by linear structure of transition probabilities, we have
$$
[ \mathbb {P} _ {h} f ] (s, a) = \int_ {s ^ {\prime} \in \mathcal {S}} \langle \phi (s, a), \boldsymbol {\mu} _ {h} ^ {*} (s ^ {\prime}) \rangle f (s ^ {\prime}) \mathrm {d} s ^ {\prime} = \langle \boldsymbol {\phi} (s, a), \boldsymbol {w} _ {h} [ f ] \rangle .
$$
In addition, for any  $f, g: \mathcal{S} \to \mathbb{R}$ , it holds that  $\pmb{w}_h[f + g] = \pmb{w}_h[f] + \pmb{w}_h[g]$  and  $\widehat{\pmb{w}}_{k,h}[f + g] = \widehat{\pmb{w}}_{k,h}[f] + \widehat{\pmb{w}}_{k,h}[g]$  due to the linear property of integration and ridge regression.
We remark  $\widehat{\pmb{w}}_{k,h}[f]$  is the estimation of  $\pmb{w}_h[f]$  by weighted ridge regression on  $\{\phi_{i,h},f(s_{i,h + 1})\}_{i\in [k]}$ . And we estimate the coefficients  $\widehat{\pmb{w}}_{k,h},\widetilde{\pmb{w}}_{k,h},\widetilde{\pmb{w}}_{k,h}$
$$
\widehat {\boldsymbol {w}} _ {k, h} = \widehat {\boldsymbol {w}} _ {k, h} \left[ V _ {h + 1} ^ {k} \right], \quad \breve {\boldsymbol {w}} _ {k, h} = \widehat {\boldsymbol {w}} _ {k, h} \left[ \check {V} _ {h + 1} ^ {k} \right], \quad \widetilde {\boldsymbol {w}} _ {k, h} = \widehat {\boldsymbol {w}} _ {k, h} \left[ \left(V _ {h + 1} ^ {k}\right) ^ {2} \right], \tag {5.8}
$$
where  $V_h^k$  and  $\tilde{V}_h^k$  are optimistic and pessimistic estimation of the optimal value functions.
Estimation for variance of next-state value functions Inspired by He et al. [15], we set the weight  $\sigma_{k,h}$  for weighted ridge regression in (5.7) as
$$
\sigma_ {k, h} = \max  \left\{\widehat {\sigma} _ {k, h}, \sqrt {d ^ {3} H D _ {k , h}}, \sigma_ {\min }, \| \phi_ {k, h} \| _ {\boldsymbol {\Sigma} _ {k - 1, h} ^ {- 1}}, \sqrt {d ^ {\frac {5}{2}} H \mathcal {H}} \| \phi_ {k, h} \| _ {\boldsymbol {\Sigma} _ {k - 1, h} ^ {- 1}} ^ {\frac {1}{2}} \right\}, \tag {5.9}
$$
where  $\sigma_{\mathrm{min}}$  is a small constant to avoid singularity,  $\widehat{\sigma}_{k,h}^2 = [\widehat{\mathbb{V}}_h V_{h+1}^k](s_{k,h}, a_{k,h}) + E_{k,h}$  with
$$
\left[ \widehat {\mathbb {V}} _ {h} V _ {h + 1} ^ {k} \right] \left(s _ {k, h}, a _ {k, h}\right) = \left\langle \phi_ {k, h}, \widetilde {\boldsymbol {w}} _ {k - 1, h} \right\rangle_ {[ 0, \mathcal {H} ^ {2} ]} - \left\langle \phi_ {k, h}, \widehat {\boldsymbol {w}} _ {k - 1, h} \right\rangle_ {[ 0, \mathcal {H} ]} ^ {2}, \tag {5.10}
$$
$$
E _ {k, h} = \min  \left\{4 \mathcal {H} \left\langle \phi_ {k, h}, \widehat {\boldsymbol {w}} _ {k - 1, h} - \breve {\boldsymbol {w}} _ {k - 1, h} \right\rangle + 1 1 \mathcal {H} \beta_ {0} \left\| \phi_ {k, h} \right\| _ {\boldsymbol {\Sigma} _ {k - 1, h} ^ {- 1}}, \mathcal {H} ^ {2} \right\}. \tag {5.11}
$$
$$
D _ {k, h} = \min  \left\{2 \mathcal {H} \left\langle \phi_ {k, h}, \widehat {\boldsymbol {w}} _ {k - 1, h} - \check {\boldsymbol {w}} _ {k - 1, h} \right\rangle + 4 \mathcal {H} \beta_ {0} \| \phi_ {k, h} \| _ {\boldsymbol {\Sigma} _ {k - 1, h} ^ {- 1}}, \mathcal {H} ^ {2} \right\}, \tag {5.12}
$$
where  $\beta_0 = \widetilde{O} (\sqrt{d^3H\mathcal{H}^2} /\sigma_{\mathrm{min}})$ . Here  $\widehat{\sigma}_{k,h}^{2}$  and  $D_{k,h}$  are upper bounds of  $[\mathbb{V}_hV_{h + 1}^* ](s_{k,h},a_{k,h})$  and  $\max \{[\mathbb{V}_h(V_{h + 1}^k -V_{h + 1}^*)](s_{k,h},a_{k,h}),[\mathbb{V}_h(V_{h + 1}^* -\check{V}_{h + 1}^k)](s_{k,h},a_{k,h})\}$ , respectively.
5.2 Computational Complexity
Theorem 5.1. For the linear MDPs with heavy-tailed rewards defined in Section 2.2, the computational complexity of HEAVY-LSVI-UCB is  $\widetilde{O}(d^4 |\mathcal{A}|H^3 K + HK\mathcal{R})$ . Here  $\mathcal{R}$  is the cost of the optimization algorithm for solving adaptive Huber regression in (5.1). Furthermore, we can specialize  $\mathcal{R}$  by adopting the Nesterov accelerated method, which gives  $\mathcal{R} = \widetilde{O}(d + d^{-\frac{1 - \epsilon}{2(1 + \epsilon)}}H^{\frac{1 - \epsilon}{2(1 + \epsilon)}}K^{\frac{1 + 2\epsilon}{2(1 + \epsilon)})}$ .
Proof. See Appendix E for a detailed proof.
Such a complexity allows us to focus on the complexity introduced by the RL algorithm rather than the optimization subroutine for solving adaptive Huber regression. Compared to that of LSVI-UCB++ [15],  $\widetilde{O}(d^4|\mathcal{A}|H^3K)$ , the extra term  $\widetilde{O}(HK\mathcal{R})$  causes a slightly worse computational time in terms of  $K$ . This is due to the absence of a closed-form solution of adaptive Huber regression in (5.1). Thus extra optimization steps are unavoidable. Nevertheless, Nesterov accelerated method gives  $\mathcal{R} = \widetilde{O}\left(K^{\frac{1 + 2\epsilon}{2(1 + \epsilon)}}\right)$  with respect to  $K$ , which implies the computational complexity of HEAVY-LSVI-UCB is better than that of LSVI-UCB [22],  $\widetilde{O}(d^2|\mathcal{A}|HK^2)$  in terms of  $K$ , thanks to the rare-switching updating policy. We conduct numerical experiments in Appendix B to further corroborate the computational efficiency of adaptive Huber regression.
5.3 Regret Bound
Theorem 5.2 (Informal). For the linear MDPs with heavy-tailed rewards defined in Section 2.2, we set parameters in Algorithm 3 as follows:  $\lambda_{R} = d / \max \left\{B^{2},W^{2}\right\}$ ,  $\lambda_{V} = 1 / \mathcal{H}^{2}$ ,  $\nu_{\mathrm{min}}$ ,  $\sigma_{\mathrm{min}}$ ,  $c_{0}$ ,  $c_{1}$ ,  $\tau_0$ ,  $\widetilde{\tau}_0$ ,  $\beta_{R^{\epsilon}}$ ,  $\beta_0$ ,  $\beta_{R}$ ,  $\beta_{V}$  in Appendix F.1. Then for any  $\delta \in (0,1)$ , with probability at least  $1 - 16\delta$ , the regret of HEAVY-LSVI-UCB is bounded by
$$
\operatorname {R e g r e t} (K) = \widetilde {O} \left(d \sqrt {H \mathcal {U} ^ {*}} K ^ {\frac {1}{1 + \epsilon}} + d \sqrt {H \mathcal {V} ^ {*} K}\right),
$$
where  $\epsilon \in (0,1],\mathcal{U}^{*} = \min \{\mathcal{U}_{0}^{*},\mathcal{U}\}$ ,  $\mathcal{V}^{*} = \min \{\mathcal{V}_{0}^{*},\mathcal{V}\}$  with  $\mathcal{U}_0^*$ ,  $\mathcal{V}_0^*$  defined in Appendix F.2 and  $\mathcal{H},\mathcal{U},\mathcal{V}$  defined in Assumption 2.7.
Proof. See Appendix F.2 for a formal version of Theorem 5.2 and its detailed proof.
Quantities  $\mathcal{U}^*$ ,  $\mathcal{V}^*$  We make a few explanations for the quantities  $\mathcal{U}^*$ ,  $\mathcal{V}^*$ . On one hand,  $\mathcal{U}^*$  is upper bounded by  $\mathcal{U}$ , which is the upper bound of the sum of the  $(1 + \epsilon)$ -th central moments of reward functions along a single trajectory. On the other hand,  $\mathcal{U}^*$  is no more than  $\mathcal{U}_0^*$ , which is the sum of the  $(1 + \epsilon)$ -th central moments with respect to the averaged occupancy measure of the first  $K$  episodes.  $\mathcal{V}^*$  is defined similar to  $\mathcal{U}^*$ , but measures the randomness of transition probabilities.
Remark 5.3. When  $\epsilon = 1$ , we can show that this regret is bounded by  $\widetilde{O}(d\sqrt{H\mathcal{G}^*K})$ , where  $\mathcal{G}^*$  is an variance-dependent quantity defined by Li and Sun [26]. Thus, our result recovers their variance-aware regret bound. See Remark F.15 in Appendix F.2 for a detailed proof.
To demonstrate the optimality of our results and establish connections with existing literature, we can specialize Theorem 5.2 to obtain the worst-case regret [22, 2, 15] and first-order regret [33].
Corollary 5.4 (Worst-case regret). For the linear MDPs with heavy-tailed rewards defined in Section 2.2 and for any  $\delta \in (0,1)$ , with probability at least  $1 - 16\delta$ , the regret of HEAVY-LSVI-UCB is bounded by
$$
\tilde {O} \left(d H K ^ {\frac {1}{1 + \epsilon}} + d \sqrt {H ^ {3} K}\right).
$$
Proof. Notice  $\mathcal{U}^*$  and  $\mathcal{V}^*$  are upper bounded by  $H\nu_R^2$  and  $\mathcal{H}^2$  (total variance lemma in Jin et al. [21]) respectively. When  $\mathcal{H} = H$ , and we treat  $\nu_{R}$  as a constant, the result follows.
Next, we give the regret lower bound of linear MDPs with heavy-tailed rewards in Theorem 5.5, which shows our proposed HEAVY-LSVI-UCB is minimax optimal in the worst case.
Theorem 5.5. For any algorithm, there exists an  $H$ -episodic,  $d$ -dimensional linear MDP with heavy-tailed rewards such that for any  $K$ , the algorithm's regret is
$$
\Omega \left(d H K ^ {\frac {1}{1 + \epsilon}} + d \sqrt {H ^ {3} K}\right).
$$
Proof. Intuitively, the proof of Theorem 5.5 follows from a combination of the lower bound constructions for heavy-tailed linear bandits in Shao et al. [31] and linear MDPs in Zhou et al. [42]. See Appendix G for a detailed proof.  $\square$
Theorem 5.5 shows that for sufficiently large  $K$ , the reward term dominates in the regret bound. Thus, in heavy-tailed settings, the main difficulty is learning the reward functions.
Corollary 5.6 (First-order regret). For the linear MDPs with heavy-tailed rewards defined in Section 2.2 and for any  $\delta \in (0,1)$ , with probability at least  $1 - 16\delta$ , the regret of HEAVY-LSVI-UCB is bounded by
$$
\widetilde {O} \left(d \sqrt {H \mathcal {U} ^ {*}} K ^ {\frac {1}{1 + \epsilon}} + d \sqrt {H \mathcal {H} V _ {1} ^ {*} K}\right).
$$
And when the rewards are uniformly bounded in  $[0,1]$ , the result is reduced to the first-order regret bound of  $\widetilde{O}(d\sqrt{H^2V_1^*K})$ .