Title: Tackling Heavy-Tailed Rewards in Reinforcement Learning with Function Approximation: Minimax Optimal and Instance-Dependent Regret Bounds
Venue: NeurIPS 2023
Accepted: yes
Keywords: machine learning

reinforcement learning

linear bandits

heavy-tailed rewards

instance-dependent regret
Abstract: While numerous works have focused on devising efficient algorithms for reinforcement learning (RL) with uniformly bounded rewards, it remains an open question whether sample or time-efficient algorithms for RL with large state-action space exist when the rewards are \emph{heavy-tailed}, i.e., with only finite $(1+\epsilon)$-th moments for some $\epsilon\in(0,1]$. In this work, we address the challenge of such rewards in RL with linear function approximation. We first design an algorithm, \textsc{Heavy-OFUL}, for heavy-tailed linear bandits, achieving an \emph{instance-dependent} $T$-round regret of $\tilde{O}\big(d T^{\frac{1-\epsilon}{2(1+\epsilon)}} \sqrt{\sum_{t=1}^T \nu_t^2} + d T^{\frac{1-\epsilon}{2(1+\epsilon)}}\big)$, the \emph{first} of this kind. Here, $d$ is the feature dimension, and $\nu_t^{1+\epsilon}$ is the $(1+\epsilon)$-th central moment of the reward at the $t$-th round. We further show the above bound is minimax optimal when applied to the worst-case instances in stochastic and deterministic linear bandits. We then extend this algorithm to the RL settings with linear function approximation. Our algorithm, termed as \textsc{Heavy-LSVI-UCB}, achieves the \emph{first} computationally efficient \emph{instance-dependent} $K$-episode regret of $\tilde{O}(d \sqrt{H \mathcal{U}^*} K^\frac{1}{1+\epsilon} + d \sqrt{H \mathcal{V}^* K})$. Here, $H$ is length of the episode, and $\mathcal{U}^*, \mathcal{V}^*$ are instance-dependent quantities scaling with the central moment of reward and value functions, respectively. We also provide a matching minimax lower bound $\Omega(d H K^{\frac{1}{1+\epsilon}} + d \sqrt{H^3 K})$ to demonstrate the optimality of our algorithm in the worst case. Our result is achieved via a novel robust self-normalized concentration inequality that may be of independent interest in handling heavy-tailed noise in general online regression problems.

Tackling Heavy-Tailed Rewards in Reinforcement Learning with Function Approximation: Minimax Optimal and Instance-Dependent Regret Bounds

Abstract

While numerous works have focused on devising efficient algorithms for reinforcement learning (RL) with uniformly bounded rewards, it remains an open question whether sample or time-efficient algorithms for RL with large state-action space exist when the rewards are heavy-tailed , i.e., with only finite (1 + ϵ ) -th moments for some ϵ ∈ (0 , 1] . In this work, we address the challenge of such rewards in RL with linear function approximation. We first design an algorithm, HEAVY-OFUL, for heavy-tailed linear bandits, achieving an instance-dependent T -round regret of ˜ O ( dT 1 -ϵ 2(1+ ϵ ) √ ∑ T t =1 ν 2 t + dT 1 -ϵ 2(1+ ϵ ) ) , the first of this kind. Here, d is the feature dimension, and ν 1+ ϵ t is the (1 + ϵ ) -th central moment of the reward at the t -th round. We further show the above bound is minimax optimal when applied to the worst-case instances in stochastic and deterministic linear bandits. We then extend this algorithm to the RL settings with linear function approximation. Our algorithm, termed as HEAVY-LSVI-UCB, achieves the first computationally efficient instance-dependent K -episode regret of ˜ O ( d √ H U ∗ K 1 1+ ϵ + d √ H V ∗ K ) . Here, H is length of the episode, and U ∗ , V ∗ are instance-dependent quantities scaling with the central moment of reward and value functions, respectively. We also provide a matching minimax lower bound Ω( dHK 1 1+ ϵ + d √ H 3 K ) to demonstrate the optimality of our algorithm in the worst case. Our result is achieved via a novel robust self-normalized concentration inequality that may be of independent interest in handling heavy-tailed noise in general online regression problems.

1 Introduction

Designing efficient reinforcement learning (RL) algorithms for large state-action space is a significant challenge within the RL community. A crucial aspect of RL is understanding the reward functions, which directly impacts the quality of the agent's policy. In certain real-world situations, reward distributions may exhibit heavy-tailed behavior, characterized by the occurrence of extremely large values at a higher frequency than expected in a normal distribution. Examples include image noise in signal processing [14], stock price fluctuations in financial markets [12, 18], and value functions in online advertising [11, 19]. However, much of the existing RL literature assumes rewards to be either uniformly bounded or light-tailed (e.g., sub-Gaussian). In such light-tailed settings, the primary challenge lies in learning the transition probabilities, leading most studies to assume deterministic rewards for ease of analysis [5, 22, 15]. As we will demonstrate, the complexity of learning reward

functions may dominate in heavy-tailed settings. Consequently, the performance of traditional algorithms may decline, emphasizing the need for the development of new, efficient algorithms specifically designed to handle heavy-tailed rewards.

Heavy-tailed distributions have been extensively studied in the field of statistics [9, 27] and in more specific online learning scenarios, such as bandits [7, 28, 31, 34, 40]. However, there is a dearth of theoretical research in RL concerning heavy-tailed rewards, whose distributions only admit finite (1 + ϵ ) -th moment for some ϵ ∈ (0 , 1] . One notable exception is Zhuang and Sui [44], which made a pioneering effort in establishing worst-case regret guarantees in tabular Markov Decision Processes (MDPs) with heavy-tailed rewards. However, their algorithm cannot handle RL settings with large state-action space. Moreover, their reliance on truncation-based methods is sub-optimal as these methods heavily depend on raw moments, which do not vanish in deterministic cases. Therefore, a natural question arises:

Can we derive sample and time-efficient algorithms for RL with large state-action space that achieve instance-dependent regret in the presence of heavy-tailed rewards?

In this work, we focus on linear MDPs [35, 22] with heavy-tailed rewards and answer the above question affirmatively. We say a distribution is heavy-tailed if it only admits finite (1 + ϵ ) -th moment for some ϵ ∈ (0 , 1] . Our contributions are summarized as follows.

We first propose a computationally efficient algorithm HEAVY-OFUL for heavy-tailed linear bandits. Such a setting can be regarded as a special case of linear MDPs. HEAVY-OFUL achieves an instance-dependent T -round regret of ˜ O ( dT 1 -ϵ 2(1+ ϵ ) √ ∑ T t =1 ν 2 t + dT 1 -ϵ 2(1+ ϵ ) ) , the first of this kind. Here d is the feature dimension and ν 1+ ϵ t is the (1 + ϵ ) -th central moment of the reward at the t -th round. The instance-dependent regret bound has a main term that only depends on the summation of central moments, and therefore does not have a √ T term. Our regret bound is shown to be minimax optimal in both stochastic and deterministic linear bandits (See Remark 4.2 for details).

We then extend this algorithm to time-inhomogeneous linear MDPs with heavy-tailed rewards, resulting in a new computationally efficient algorithm HEAVY-LSVI-UCB, which achieves a K -episode regret scaling as ˜ O ( d √ H U ∗ K 1 1+ ϵ + d √ H V ∗ K ) for the first time. Here, H is the length of the episode and U ∗ , V ∗ are quantities measuring the central moment of the reward functions and transition probabilities, respectively (See Theorem 5.2 for details). Our regret bound is instance-dependent since the main term only relies on the instance-dependent quantities, which vanishes when the dynamics and rewards are deterministic. When specialized to special cases, our instance-dependent regret recovers the variance-aware regret in Li and Sun [26] (See Remark 5.3 for details) and improves existing first-order regret bounds [33, 26] (See Corollary 5.6 for details).

We provide a minimax regret lower bound Ω( dHK 1 1+ ϵ + d H 3 K ) for linear MDPs with heavytailed rewards, which matches the worst-case regret bound implied by our instance-dependent regret, thereby demonstrating the minimax optimality of HEAVY-LSVI-UCB in the worst case.

√

For better comparisons between our algorithms and state-of-the-art results, we summarize the regrets in Table 1 and 2 for linear bandits and linear MDPs, respectively. More related works are deferred to Appendix A. Remarkably, our results demonstrate that ϵ = 1 (i.e. finite variance) is sufficient to obtain variance-aware regret bounds of the same order as the case where rewards are uniformly bounded for both linear bandits and linear MDPs. The main technique contribution behind our results is a novel robust self-normalized concentration inequality inspired by Sun et al. [32]. To be more specific, it is a non-trivial generalization of adaptive Huber regression from independent and identically distributed (i.i.d.) case to heavy-tailed online regression settings and gives a self-normalized bound instead of the ℓ 2 -norm bound in Sun et al. [32]. Our result is computationally efficient and only scales with the feature dimension, d , (1 + ϵ ) -th central moment of the noise, ν , and does not depend on the absolute magnitude as in other self-normalized concentration inequalities [42, 41].

Road Map The rest of the paper is organized as follows. Section 2 introduces heavy-tailed linear bandits and linear MDPs. Section 3 presents the robust self-normalized concentration inequality for general online regression problems with heavy-tailed noise. Section 4 and 5 give the main results for heavy-tailed linear bandits and linear MDPs, respectively. We then conclude in Section 6. Related work, experiments and all proofs can be found in Appendix.

Notations Let ∥ a ∥ := ∥ a ∥ 2 . Let [ t ] := { 1 , 2 , . . . , t } . Let B d ( r ) := { x ∈ R d |∥ x ∥ ≤ r } . Let x [ a,b ] := max { a, min { x, b }} denote the projection of x onto the close interval [ a, b ] . Let σ ( { X s } s ∈ [ t ] ) be the σ -field generated by random vectors { X s } s ∈ [ t ] .

2 Preliminaries

2.1 Heavy-Tailed Linear Bandits

Definition 2.1 (Heterogeneous linear bandits with heavy-tailed rewards) . Let {D t } t ≥ 1 denote a series of fixed decision sets, where all ϕ t ∈ D t satisfy ∥ ϕ t ∥ ≤ L for some known upper bound L . At each round t , the agent chooses ϕ t ∈ D t , then receives a reward R t from the environment. We define the filtration {F t } t ≥ 1 as F t = σ ( { ϕ s , R s } s ∈ [ t ] ∪ { ϕ t +1 } ) for any t ≥ 1 . We assume R t = ⟨ ϕ t , θ ∗ ⟩ + ε t with the unknown coefficient θ ∗ ∈ B d ( B ) for some known upper bound B . The random variable ε t ∈ R is F t -measurable and satisfies E [ ε t |F t -1 ] = 0 , E [ | ε t | 1+ ϵ |F t -1 ] = ν 1+ ϵ t for some ϵ ∈ (0 , 1] with ν t being F t -1 -measurable.

The agent aims to minimize the T -round pseudo-regret defined as Regret( T ) = ∑ T t =1 [ ⟨ ϕ ∗ t , θ ∗ ⟩ -⟨ ϕ t , θ ∗ ⟩ ] , where ϕ ∗ t = argmax ϕ ∈D t ⟨ ϕ , θ ∗ ⟩ .

2.2 Linear MDPs with Heavy-Tailed Rewards

We use a tuple M = M ( S , A , H, { R h } h ∈ [ H ] , { P h } h ∈ [ H ] ) to describe the time-inhomogeneous finitehorizon MDP , where S and A are state space and action space, respectively, H is the length of the episode, R h : S × A → R is the random reward function with expectation r h : S × A → R , and P h : S × A → ∆( S ) is the transition probability function. More details can be found in Puterman [30]. A time-dependent policy π = { π h } h ∈ H satisfies π h : S → ∆( A ) for any h ∈ [ H ] . When the policy is deterministic, we use π h ( s h ) to denote the action chosen at the h -th step given s h by policy π . For any state-action pair ( s, a ) ∈ S × A , we define the state-action value function Q π h ( s, a ) and state value function V π h ( s ) as follows: Q π h ( s, a ) = E [ ∑ H h ′ = h r ( s h ′ , a h ′ ) | s h = s, a h = a ] , V π h ( s ) = Q π h ( s, π h ( s )) , where the expectation is taken with respect to the transition probability of

M and the agent's policy π . If π is randomized, then the definition of V should have an expectation. Denote the optimal value functions as V ∗ h ( s ) = sup π V π h ( s ) and Q ∗ h ( s, a ) = sup π Q π h ( s, a ) .

We introduce the following shorthands for simplicity. At the h -th step, for any value function V : S → R , let [ P h V ]( s, a ) = E s ′ ∼ P h ( ·| s,a ) V ( s ′ ) , [ V h V ]( s, a ) = [ P h V 2 ]( s, a ) -[ P h V ] 2 ( s, a ) denote the expectation and the variance of the next-state value function at the h -th step given ( s, a ) .

We aim to minimize the K -episode regret defined as Regret( K ) = ∑ K k =1 [ V ∗ 1 ( s k 1 ) -V π k 1 ( s k 1 )] .

In the rest of this section, we introduce linear MDPs with heavy-tailed rewards. We first give the definition of linear MDPs studied in Yang and Wang [35], Jin et al. [22], with emphasis that the rewards in their settings are deterministic or uniformly bounded. Then we focus on the heavy-tailed random rewards.

Definition 2.2. An MDP M = M ( S , A , H, { R h } h ∈ [ H ] , { P h } h ∈ [ H ] ) is a time-inhomogeneous finitehorizon linear MDP , if there exist known feature maps ϕ ( s, a ) : S × A → B d (1) , unknown d -dimensional signed measures { µ ∗ h } h ∈ [ H ] over S with ∥ µ ∗ h ( S ) ∥ := ∫ s ∈S | µ ( s ) | d s ≤ √ d and unknown coefficients { θ ∗ h } h ∈ [ H ] ⊆ B d ( B ) for some known upper bound B such that

for any state-action pair ( s, a ) ∈ S × A and timestep h ∈ [ H ] .

Assumption 2.3 (Realizable rewards) . For all ( s, a, h ) ∈ S × A × [ H ] , the random reward R h ( s, a ) is independent of next state s ′ ∼ P h ( ·| s, a ) and admits the linear structure

where ε h ( s, a ) is a mean-zero heavy-tailed random variable specified below.

We introduce the notation ν n [ X ] = E [ | X -E X | n ] for the n -th central moment of any random variable X . And for any random reward function at the h -th step R h : S × A → R , let

denote its expectation and the (1 + ϵ ) -th central moment given ( s h , a h ) = ( s, a ) for short.

Assumption 2.4 (Heavy-tailedness of rewards) . Random variable ε h ( s, a ) satisfies [ E h ε h ]( s, a ) = 0 . And for some known ϵ, ϵ ′ ∈ (0 , 1] and constants ν R , ν R ϵ ≥ 0 , the following unknown moments of ε h ( s, a ) satisfy

for all ( s, a, h ) ∈ S × A × [ H ] .

Assumption 2.4 generalizes Assumption 2.2 of Li and Sun [26], which is the weakest moment condition on random rewards in the current literature of RL with function approximation. Setting ϵ = 1 and ϵ ′ = 1 immediately recovers their settings.

Assumption 2.5 (Realizable central moments) . There are some unknown coefficients { ψ ∗ h } h ∈ [ H ] ⊆ B d ( W ) for some known upper bound W such that

for all ( s, a, h ) ∈ S × A × [ H ] .

Remark 2.6. When ϵ = 1 , that is the rewards have finite variance, Li and Sun [26] use the fact that [ ν 2 R h ]( s, a ) = [ V h R h ]( s, a ) = [ E h R 2 h ]( s, a ) -[ E h R h ] 2 ( s, a ) , assume the linear realizability of the second moment [ E h R 2 h ]( s, a ) , and estimate it instead. However, when ϵ < 1 , there is no such relationship between the (1 + ϵ ) -th central moment [ ν 1+ ϵ R h ]( s, a ) and the (1 + ϵ ) -th raw moment [ E h R 1+ ϵ h ]( s, a ) . Thus, we adopt a new approach to estimate [ ν 1+ ϵ R h ]( s, a ) directly, and bound the error by a novel perturbation analysis of adaptive Huber regression in Appendix C.3.

Assumption 2.7 (Bounded cumulative rewards) . For any policy π , let { s h , a h , R h } h ∈ [ H ] be a random trajectory following policy π . And define r π = ∑ H h =1 [ E h R h ]( s h , a h ) = ∑ H h =1 r h ( s h , a h ) . We assume (1) 0 ≤ r π ≤ H . (2) ∑ H h =1 [ ν 1+ ϵ R h ] 2 1+ ϵ ( s h , a h ) ≤ U . (3) Var( r π ) ≤ V .

Here, (1) gives an upper bound of cumulative expected rewards r π . (2) assumes the summation of (1+ ϵ ) -th central moment of rewards [ ν 1+ ϵ R h ]( s h , a h ) is bounded since [ ∑ H h =1 [ ν 1+ ϵ R h ]( s h , a h )] 2 1+ ϵ ≤ ∑ H h =1 [ ν 1+ ϵ R h ] 2 1+ ϵ ( s h , a h ) ≤ U due to Jensen's inequality. And (3) is to bound the variance of r π along the trajectory following policy π .

Algorithm 1 Adaptive Huber Regression

Require: Number of total rounds T , confidence level δ , regularization parameter λ , σ min , parameters for adaptive Huber regression c 0 , c 1 , τ 0 , estimated central moment ̂ ν t and moment parameter b that satisfy ν t / ν t ≤ b for all t ≤ T .

1: κ = d log(1 + TL 2 / ( dλσ 2 min )) .

̂ Ensure: The estimated coefficient θ t .

2: Set H t -1 = λ I + ∑ t -1 s =1 σ -2 s ϕ s ϕ ⊤ s .

3: Set σ t = max { ̂ ν t , σ min , ∥ ϕ t ∥ H -1 t -1 c 0 , √ LB c 1 4 1 (2 κb 2 ) 1 4 ∥ ϕ t ∥ 1 2 H -1 t -1 } . √

4: Set τ t = τ 0 1+ w 2 t w t t 1 -ϵ 2(1+ ϵ ) with w t = ∥ ϕ t /σ t ∥ H -1 t -1 .

5: Define the loss function L t ( θ ) := λ 2 ∥ θ ∥ 2 + ∑ t s =1 ℓ τ s ( y s -⟨ ϕ s , θ ⟩ σ s ) .

6: Compute θ t = argmin θ ∈B d ( B ) L t ( θ ) .

3 Adaptive Huber Regression

At the core of our algorithms for both heavy-tailed linear bandits and linear MDPs is a new approach - adaptive Huber regression - to handle heavy-tailed noise. Sun et al. [32] imposed adaptive Huber regression to handle i.i.d. heavy-tailed noise by utilizing Huber loss [17] as a surrogate of squared loss. Li and Sun [26] modified adaptive Huber regression for heterogeneous online settings, where the variances in each round are different. However, it is not readily applicable to deal with heavy-tailed noise. Our contribution in this section is to construct a new self-normalized concentration inequality for general online regression problems with heavy-tailed noise.

We first give a brief introduction to Huber loss function and its properties.

Definition 3.1 (Huber loss) . Huber loss is defined as

where τ > 0 is referred as a robustness parameter.

Huber loss is first proposed by Huber [17] as a robust version of squared loss while preserving the convex property. Specifically, Huber loss is a quadratic function of x when | x | is less than the threshold τ , while becomes linearly dependent on | x | when | x | grows larger than τ . It has the property of strongly convex near zero point and is not sensitive to outliers. See Appendix C.1 for more properties of Huber loss.

Next, we define general online regression problems with heavy-tailed noise, which include heavytailed linear bandits as a special case. Then we utilize Huber loss to estimate θ ∗ . Below we give the main theorem to bound the deviation of the estimated θ t in Algorithm 1 from the ground truth θ ∗ .

Definition 3.2. Let {F t } t ≥ 1 be a filtration. For all t > 0 , let random variables y t , ε t be F t -measurable and random vector ϕ t ∈ B d ( L ) be F t -1 -measurable. Suppose y t = ⟨ ϕ t , θ ∗ ⟩ + ε t , where θ ∗ ∈ B d ( B ) is an unknown coefficient and

for some ϵ ∈ (0 , 1] . The goal is to estimate θ ∗ at any round t given the realizations of { ϕ s , y s } s ∈ [ t ] .

Theorem 3.3. For the online regression problems in Definition 3.2, we solve for θ t by adaptive Huber regression in Algorithm 1 with c 0 , c 1 , τ 0 in Appendix C.2. Then for any δ ∈ (0 , 1) , with probability at least 1 -3 δ , for all t ≤ T , we have ∥ θ t -θ ∗ ∥ H t ≤ β t , where H t is defined in Algorithm 1 and

Proof. To derive a tight high-probability bound, we take the most advantage of the properties of Huber loss. A Chernoff bounding technique is used to bound the main error term, which requires a careful analysis of the moment generating function. See Appendix C.2 for a detailed proof.

Algorithm 2 HEAVY-OFUL

Require: Number of total rounds T , confidence level δ , regularization parameter λ , σ min , parameters for adptive Huber regression c 0 , c 1 , τ 0 , confidence radius β t .

We refer to the regression process in Line 6 of Algorithm 1 as adaptive Huber regression in line with Sun et al. [32] to emphasize that the value of robustness parameter τ t is chosen to adapt to data for a better trade-off between bias and robustness. Specifically, since we are in the online setting, ϕ t are dependent on { ϕ s } s<t , which is the key difference from the i.i.d. case in Sun et al. [32] where they set τ t = τ 0 , for all t ≤ T . Thus, as shown in Line 4 of Algorithm 1, inspired by Li and Sun [26], we adjust τ t according to the importance of observations w t = ∥ ϕ t /σ t ∥ H -1 t -1 , where σ t is specified below. In the case where ϵ < 1 , different from Li and Sun [26], we first choose τ t to be small for robust purposes, then gradually increase it with t to reduce the bias.

Next, we illustrate the reason for setting σ t via Line 3 of Algorithm 1. We use ̂ ν t ∈ F t -1 to estimate the central moment ν t and use moment parameter b to measure the closeness between ̂ ν t and ν t . When we choose ̂ ν t as an upper bound of ν t , b becomes a constant that equals to 1 . And σ min is a small positive constant to avoid singularity. The last two terms with respect to c 0 and c 1 are set according to the uncertainty ∥ ϕ t ∥ H -1 t -1 . In addition, setting the parameter c 0 ≤ 1 yields w t ≤ 1 , which is essential to meet the condition of elliptical potential lemma [1].

Remark 3.4. The error bound β t in (3.2) is only related to the feature dimension d and moment parameter b . While the Bernstein-style self-normalized concentration bounds [42, 41] depend on the magnitude of ε t , thus cannot handle heavy-tailed errors.

4 Linear Bandits

In this section, we show the algorithm HEAVY-OFUL in Algorithm 2 for heavy-tailed linear bandits in Definition 2.1. We first give a brief algorithm description, and then provide a theoretical regret analysis.

4.1 Algorithm Description

HEAVY-OFUL follows the principle of Optimism in the Face of Uncertainty (OFU) [1], and uses adaptive Huber regression in Section 3 to maintain a set C t that contains the unknown coefficient θ ∗ with high probability. Specifically, at the t -th round, HEAVY-OFUL estimates the expected reward of any arm ϕ as max θ ∈C t -1 ⟨ ϕ , θ ⟩ , and selects the arm that maximizes the estimated reward. The agent then receives the reward R t and updates the confidence set C t based on the information up to round t with its center θ t computed by adaptive Huber regression as in Line 9 of Algorithm 2.

4.2 Regret Analysis

We next give the instance-dependent regret upper bound of HEAVY-OFUL in Theorem 4.1.

Theorem 4.1. For the heavy-tailed linear bandits in Definition 2.1, we set c 0 , c 1 , τ 0 , β t in Algorithm 2 according to Theorem 3.3 with b = 1 . Besides, let λ = d/B 2 , and σ min = 1 / √ T . Then with

probability at least 1 -3 δ , the regret of HEAVY-OFUL is bounded by

Proof. The proof uses the self-normalized concentration inequality of adaptive Huber regression and a careful analysis to bound the summation of bonuses. See Appendix D.1 for a detailed proof.

Remark 4.2. Theorem 4.1 shows HEAVY-OFUL achieves an instance-dependent regret bound. When we assume ν t , ∀ t ≥ 1 have uniform upper bound ν (which can be treated as a constant), then the bound is reduced to ˜ O ( dT 1 1+ ϵ ) . It matches the lower bound Ω( dT 1 1+ ϵ ) by Shao et al. [31] up to logarithmic factors. In the deterministic scenario, where ϵ = 1 and ν t = 0 , for all t ≥ 1 , the bound is reduced to ˜ O ( d ) . It matches the lower bound Ω( d ) 1 up to logarithmic factors.

5 Linear MDPs

In this section, we show the algorithm HEAVY-LSVI-UCB in Algorithm 3 for linear MDP with heavy-tailed rewards defined in Section 2.2. Let ϕ k,h := ϕ ( s k,h , a k,h ) for short. We first give the algorithm description intuitively, then provide the computational complexity and regret bound.

5.1 Algorithm Description

HEAVY-LSVI-UCB features a novel combination of adaptive Huber regression in Section 3 and existing algorithmic frameworks for linear MDPs with bounded rewards [22, 15]. At a high level, HEAVY-LSVI-UCB employs separate estimation techniques to handle heavy-tailed rewards and transition kernels. Specifically, we utilize adaptive Huber regression proposed in Section 3 to estimate heavy-tailed rewards and weighted ridge regression [42, 15] to estimate the expected next-state value functions. Then, it follows the value iteration scheme to update the optimistic and pessimistic estimation of the optimal value function Q k h , V k h and ˇ Q k h , ˇ V k h , respectively, via a rare-switching policy as in Line 7 to 15 of Algorithm 3. We highlight the key steps of HEAVY-LSVI-UCB as follows.

Estimation for expected heavy-tailed rewards Since the expected rewards have linear structure in linear MDPs, i.e., r h ( s, a ) = ⟨ ϕ ( s, a ) , θ ∗ h ⟩ , we use adaptive Huber regression to estimate θ ∗ h :

where ν i,h will be specified later.

Estimation for central moment of rewards By Assumption 2.5, the (1 + ϵ ) -th central moment of rewards is linear in ϕ , i.e., [ ν 1+ ϵ R h ]( s, a ) = ⟨ ϕ ( s, a ) , ψ ∗ h ⟩ . Motivated by this, we estimate ψ ∗ h by adaptive Huber regression as

where W is the upper bound of ∥ ψ ∗ h ∥ defined in Assumption 2.5. Since ε i,h is intractable, we estimate it by ̂ ε i,h = R i,h -⟨ ϕ i,h , θ i,h ⟩ , which gives ̂ ψ k,h as

The inevitable error between ψ k,h and ̂ ψ k,h can be quantified by a novel perturbation analysis of adaptive Huber regression in Appendix C.3.

Algorithm 3 HEAVY-LSVI-UCB

Require: Number of episodes K , confidence level δ , regularization parameter λ R , λ V , ν min , σ min , confidence radius β R ϵ , β 0 , β R , β V .

4: V k H +1 ( · ) = ˇ V k H +1 ( · ) = 0 .

̂ 3: for k = 1 , . . . , K do

5: for h = H,.. . , 1 do

6: Compute θ k -1 ,h , ̂ w k -1 ,h and ˇ w k -1 ,h via (5.1) and (5.8).

7: if UPDATE then

12: else

14: end if

16: end for

17: Observe initial state s k, 1 .

18: for h = 1 , . . . , H do

19: Take action a k,h = π k h ( s k,h ) and observe R k,h , s k,h +1 .

20:

Set

ν

and

σ

according to (5.4) and (5.9) respectively.

23: end for

24:

if

∃

h

′

∈

[

H

]

such that det(

25: Set UPDATE = TRUE.

26: else

27: Set UPDATE = FALSE.

28: end if

29: end for

We then set the weight ν k,h for adaptive Huber regression as

where ν min is a small positive constant to avoid the singularity, ̂ ν 1+ ϵ k,h = [ ̂ ν 1+ ϵ R h ]( s k,h , a k,h ) + W k,h is a high-probability upper bound of rewards' central moment [ ν 1+ ϵ R h ]( s k,h , a k,h ) with

where κ is defined in Algorithm 3, β R ϵ ,k = ˜ O ( √ dν R ϵ k 1 -ϵ ′ 2(1+ ϵ ′ ) /ν min ) and β R,k = ˜ O ( √ dk 1 -ϵ 2(1+ ϵ ) ) .

Estimation for expected next-state value functions For any value function f : S → R , we define the following notations for simplicity:

where σ i,h will be specified later. Note for any state-action pair ( s, a ) ∈ S × A , by linear structure of transition probabilities, we have

H

k,h

′

)

2 det(

H

k

last

,h

′

)

Σ

k,h

′

)

2 det(

Σ

k

last

,h

′

)

then

In addition, for any f, g : S → R , it holds that w h [ f + g ] = w h [ f ] + w h [ g ] and ̂ w k,h [ f + g ] = ̂ w k,h [ f ] + ̂ w k,h [ g ] due to the linear property of integration and ridge regression.

We remark ̂ w k,h [ f ] is the estimation of w h [ f ] by weighted ridge regression on { ϕ i,h , f ( s i,h +1 ) } i ∈ [ k ] . And we estimate the coefficients ̂ w k,h , ˇ w k,h , ˜ w k,h

where V k h and ˇ V k h are optimistic and pessimistic estimation of the optimal value functions.

Estimation for variance of next-state value functions Inspired by He et al. [15], we set the weight σ k,h for weighted ridge regression in (5.7) as

where σ min is a small constant to avoid singularity, ̂ σ 2 k,h = [ ̂ V h V k h +1 ]( s k,h , a k,h ) + E k,h with

where β 0 = ˜ O ( √ d 3 H H 2 /σ min ) . Here ̂ σ 2 k,h and D k,h are upper bounds of [ V h V ∗ h +1 ]( s k,h , a k,h ) and max { [ V h ( V k h +1 -V ∗ h +1 ) ]( s k,h , a k,h ) , [ V h ( V ∗ h +1 -ˇ V k h +1 ) ]( s k,h , a k,h ) } , respectively.

5.2 Computational Complexity

Theorem 5.1. For the linear MDPs with heavy-tailed rewards defined in Section 2.2, the computational complexity of HEAVY-LSVI-UCB is ˜ O ( d 4 |A| H 3 K + HK R ) . Here R is the cost of the optimization algorithm for solving adaptive Huber regression in (5.1). Furthermore, we can specialize R by adopting the Nesterov accelerated method, which gives R = ˜ O ( d + d -1 -ϵ 2(1+ ϵ ) H 1 -ϵ 2(1+ ϵ ) K 1+2 ϵ 2(1+ ϵ ) ) .

Proof. See Appendix E for a detailed proof.

Such a complexity allows us to focus on the complexity introduced by the RL algorithm rather than the optimization subroutine for solving adaptive Huber regression. Compared to that of LSVI-UCB++ [15], ˜ O ( d 4 |A| H 3 K ) , the extra term ˜ O ( HK R ) causes a slightly worse computational time in terms of K . This is due to the absence of a closed-form solution of adaptive Huber regression in (5.1). Thus extra optimization steps are unavoidable. Nevertheless, Nesterov accelerated method gives R = ˜ O ( K 1+2 ϵ 2(1+ ϵ ) ) with respect to K , which implies the computational complexity of HEAVY-LSVI-UCB

is better than that of LSVI-UCB [22], ˜ O ( d 2 |A| HK 2 ) in terms of K , thanks to the rare-switching updating policy. We conduct numerical experiments in Appendix B to further corroborate the computational efficiency of adaptive Huber regression.

5.3 Regret Bound

Theorem 5.2 (Informal) . For the linear MDPs with heavy-tailed rewards defined in Section 2.2, we set parameters in Algorithm 3 as follows: λ R = d/ max { B 2 , W 2 } , λ V = 1 / H 2 , ν min , σ min , c 0 , c 1 , τ 0 , ˜ τ 0 , β R ϵ , β 0 , β R , β V in Appendix F.1. Then for any δ ∈ (0 , 1) , with probability at least 1 -16 δ , the regret of HEAVY-LSVI-UCB is bounded by

where ϵ ∈ (0 , 1] , U ∗ = min {U ∗ 0 , U} , V ∗ = min {V ∗ 0 , V} with U ∗ 0 , V ∗ 0 defined in Appendix F.2 and H , U , V defined in Assumption 2.7.

Proof. See Appendix F.2 for a formal version of Theorem 5.2 and its detailed proof.

Quantities U ∗ , V ∗ We make a few explanations for the quantities U ∗ , V ∗ . On one hand, U ∗ is upper bounded by U , which is the upper bound of the sum of the (1 + ϵ ) -th central moments of reward functions along a single trajectory. On the other hand, U ∗ is no more than U ∗ 0 , which is the sum of the (1 + ϵ ) -th central moments with respect to the averaged occupancy measure of the first K episodes. V ∗ is defined similar to U ∗ , but measures the randomness of transition probabilities.

Remark 5.3. When ϵ = 1 , we can show that this regret is bounded by ˜ O ( d √ H G ∗ K ) , where G ∗ is an variance-dependent quantity defined by Li and Sun [26]. Thus, our result recovers their variance-aware regret bound. See Remark F.15 in Appendix F.2 for a detailed proof.

To demonstrate the optimality of our results and establish connections with existing literature, we can specialize Theorem 5.2 to obtain the worst-case regret [22, 2, 15] and first-order regret [33].

Corollary 5.4 (Worst-case regret) . For the linear MDPs with heavy-tailed rewards defined in Section 2.2 and for any δ ∈ (0 , 1) , with probability at least 1 -16 δ , the regret of HEAVY-LSVI-UCB is bounded by √

Proof. Notice U ∗ and V ∗ are upper bounded by Hν 2 R and H 2 (total variance lemma in Jin et al. [21]) respectively. When H = H , and we treat ν R as a constant, the result follows.

Next, we give the regret lower bound of linear MDPs with heavy-tailed rewards in Theorem 5.5, which shows our proposed HEAVY-LSVI-UCB is minimax optimal in the worst case.

Theorem 5.5. For any algorithm, there exists an H -episodic, d -dimensional linear MDP with heavy-tailed rewards such that for any K , the algorithm's regret is

Proof. Intuitively, the proof of Theorem 5.5 follows from a combination of the lower bound constructions for heavy-tailed linear bandits in Shao et al. [31] and linear MDPs in Zhou et al. [42]. See Appendix G for a detailed proof.

Theorem 5.5 shows that for sufficiently large K , the reward term dominates in the regret bound. Thus, in heavy-tailed settings, the main difficulty is learning the reward functions.

Corollary 5.6 (First-order regret) . For the linear MDPs with heavy-tailed rewards defined in Section 2.2 and for any δ ∈ (0 , 1) , with probability at least 1 -16 δ , the regret of HEAVY-LSVI-UCB is bounded by √

And when the rewards are uniformly bounded in [0 , 1] , the result is reduced to the first-order regret bound of ˜ O ( d √ H 2 V ∗ 1 K ) .

Proof.

Our first-order regret ˜ O ( d √ H 2 V ∗ 1 K ) is minimax optimal in the worst case since V ∗ 1 ≤ H . And it improves the state-of-the-art result ˜ O ( d √ H 3 V ∗ 1 K ) [26] by a factor of √ H .

6 Conclusion

In this work, we propose two computationally efficient algorithms for heavy-tailed linear bandits and linear MDPs, respectively. Our proposed algorithms, termed as HEAVY-OFUL and HEAVY-LSVI-UCB, are based on a novel self-normalized concentration inequality for adaptive Huber regression, which may be of independent interest. HEAVY-OFUL and HEAVY-LSVI-UCB achieve minimax optimal and instance-dependent regret bounds scaling with the central moments. We also provide a lower bound for linear MDPs with heavy-tailed rewards to demonstrate the optimality of HEAVY-LSVI-UCB. To the best of our knowledge, we are the first to study heavy-tailed rewards in RL with function approximation and provide a new algorithm for this setting which is both statistically and computationally efficient.

Acknowledgments

Liwei Wang is supported in part by NSF IIS 2110170, NSF DMS 2134106, NSF CCF 2212261, NSF IIS 2143493, NSF CCF 2019844, NSF IIS 2229881. Lin F. Yang is supported in part by NSF grant 2221871, and an Amazon Research Grant.

References

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems , volume 24, 2011.

Alekh Agarwal, Yujia Jin, and Tong Zhang. VO Q L: Towards optimal regret in model-free rl with nonlinear function approximation. In The Thirty Sixth Annual Conference on Learning Theory , pages 987-1063. PMLR, 2023.

Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforcement learning. Advances in neural information processing systems , 21, 2008.

Alex Ayoub, Zeyu Jia, Csaba Szepesvari, Mengdi Wang, and Lin Yang. Model-based reinforcement learning with value-targeted regression. In International Conference on Machine Learning , pages 463-474. PMLR, 2020.

Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning , pages 263-272. PMLR, 2017.

Sujay Bhatt, Guanhua Fang, Ping Li, and Gennady Samorodnitsky. Nearly optimal catoni's m-estimator for infinite variance. In International Conference on Machine Learning , pages 1925-1944. PMLR, 2022.

Sébastien Bubeck, Nicolo Cesa-Bianchi, and Gábor Lugosi. Bandits with heavy tail. IEEE Transactions on Information Theory , 59(11):7711-7717, 2013.

Sébastien Bubeck et al. Convex optimization: Algorithms and complexity. Foundations and Trends® in Machine Learning , 8(3-4):231-357, 2015.

Olivier Catoni. Challenging the empirical mean and empirical variance: a deviation study. In Annales de l'IHP Probabilités et statistiques , volume 48, pages 1148-1185, 2012.

Peng Chen, Xinghu Jin, Xiang Li, and Lihu Xu. A generalized catoni's m-estimator under finite α -th moment assumption with α ∈ (1, 2). Electronic Journal of Statistics , 15(2):5523-5544, 2021.

Hana Choi, Carl F Mela, Santiago R Balseiro, and Adam Leary. Online display advertising markets: A literature review and future directions. Information Systems Research , 31(2): 556-575, 2020.

Rama Cont. Empirical properties of asset returns: stylized facts and statistical issues. Quantitative finance , 1(2):223, 2001.

David A Freedman. On tail probabilities for martingales. the Annals of Probability , pages 100-118, 1975.

A Ben Hamza and Hamid Krim. Image denoising: A nonlinear robust statistical approach. IEEE transactions on signal processing , 49(12):3045-3054, 2001.

Jiafan He, Heyang Zhao, Dongruo Zhou, and Quanquan Gu. Nearly minimax optimal reinforcement learning for linear markov decision processes. In International Conference on Machine Learning , pages 12790-12822. PMLR, 2023.

Pihe Hu, Yu Chen, and Longbo Huang. Nearly minimax optimal reinforcement learning with linear function approximation. In International Conference on Machine Learning , pages 8971-9019, 2022.

Peter J Huber. Robust estimation of a location parameter. The Annals of Mathematical Statistics , pages 73-101, 1964.

John Hull. Risk management and financial institutions,+ Web Site , volume 733. John Wiley & Sons, 2012.

Charles Jebarajakirthy, Haroon Iqbal Maseeh, Zakir Morshed, Amit Shankar, Denni Arli, and Robin Pentecost. Mobile advertising: A systematic literature review and future research agenda. International Journal of Consumer Studies , 45(6):1258-1291, 2021.

Zeyu Jia, Lin Yang, Csaba Szepesvari, and Mengdi Wang. Model-based reinforcement learning with value-targeted regression. In Learning for Dynamics and Control , pages 666-686. PMLR, 2020.

Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael I Jordan. Is q-learning provably efficient? Advances in neural information processing systems , 31, 2018.

Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory , pages 21372143. PMLR, 2020.

Yeoneung Kim, Insoon Yang, and Kwang-Sung Jun. Improved regret analysis for varianceadaptive linear bandits and horizon-free linear mixture mdps. Advances in Neural Information Processing Systems , 35:1060-1072, 2022.

Johannes Kirschner and Andreas Krause. Information directed sampling and bandits with heteroscedastic noise. In Conference On Learning Theory , pages 358-384. PMLR, 2018.

Gen Li, Changxiao Cai, Yuxin Chen, Yuting Wei, and Yuejie Chi. Is q-learning minimax optimal? a tight sample complexity analysis. Operations Research , 2023.

Xiang Li and Qiang Sun. Variance-aware robust reinforcement learning with linear function approximation with heavy-tailed rewards. arXiv preprint arXiv:2303.05606 , 2023.

Gábor Lugosi and Shahar Mendelson. Mean estimation and regression under heavy-tailed distributions: A survey. Foundations of Computational Mathematics , 19(5):1145-1190, 2019.

Andres Munoz Medina and Scott Yang. No-regret algorithms for heavy-tailed linear bandits. In International Conference on Machine Learning , pages 1642-1650. PMLR, 2016.

Aditya Modi, Nan Jiang, Ambuj Tewari, and Satinder Singh. Sample complexity of reinforcement learning using linearly combined model ensembles. In International Conference on Artificial Intelligence and Statistics , pages 2010-2020. PMLR, 2020.

Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons, 2014.

Han Shao, Xiaotian Yu, Irwin King, and Michael R Lyu. Almost optimal algorithms for linear stochastic bandits with heavy-tailed payoffs. Advances in Neural Information Processing Systems , 31, 2018.

Qiang Sun, Wen-Xin Zhou, and Jianqing Fan. Adaptive huber regression. Journal of the American Statistical Association , 115(529):254-265, 2020.

Andrew J Wagenmaker, Yifang Chen, Max Simchowitz, Simon Du, and Kevin Jamieson. Firstorder regret in reinforcement learning with linear function approximation: A robust estimation approach. In International Conference on Machine Learning , pages 22384-22429. PMLR, 2022.

Bo Xue, Guanghui Wang, Yimu Wang, and Lijun Zhang. Nearly optimal regret for stochastic linear bandits with heavy-tailed payoffs. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20 , pages 2936-2942. International Joint Conferences on Artificial Intelligence Organization, 2020.

Lin Yang and Mengdi Wang. Sample-optimal parametric q-learning using linearly additive features. In International Conference on Machine Learning , pages 6995-7004. PMLR, 2019.

Andrea Zanette and Emma Brunskill. Tighter problem-dependent regret bounds in reinforcement learning without domain knowledge using value function bounds. In International Conference on Machine Learning , pages 7304-7312. PMLR, 2019.

Zihan Zhang, Jiaqi Yang, Xiangyang Ji, and Simon S Du. Improved variance-aware confidence sets for linear bandits and linear mixture mdp. Advances in Neural Information Processing Systems , 34:4342-4355, 2021.

Heyang Zhao, Jiafan He, Dongruo Zhou, Tong Zhang, and Quanquan Gu. Variance-dependent regret bounds for linear bandits and reinforcement learning: Adaptivity and computational efficiency. In Proceedings of Thirty Sixth Conference on Learning Theory , volume 195 of Proceedings of Machine Learning Research , pages 4977-5020. PMLR, 2023.

Han Zhong and Tong Zhang. A theoretical analysis of optimistic proximal policy optimization in linear markov decision processes. arXiv preprint arXiv:2305.08841 , 2023.

Han Zhong, Jiayi Huang, Lin Yang, and Liwei Wang. Breaking the moments condition barrier: No-regret algorithm for bandits with super heavy-tailed payoffs. Advances in Neural Information Processing Systems , 34:15710-15720, 2021.

Dongruo Zhou and Quanquan Gu. Computationally efficient horizon-free reinforcement learning for linear mixture mdps. Advances in neural information processing systems , 35:36337-36349, 2022.

Dongruo Zhou, Quanquan Gu, and Csaba Szepesvari. Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory , pages 4532-4576. PMLR, 2021.

Runlong Zhou, Zhang Zihan, and Simon Shaolei Du. Sharp variance-dependent bounds in reinforcement learning: Best of both worlds in stochastic and deterministic environments. In International Conference on Machine Learning , pages 42878-42914. PMLR, 2023.

Vincent Zhuang and Yanan Sui. No-regret reinforcement learning with heavy-tailed rewards. In International Conference on Artificial Intelligence and Statistics , pages 3385-3393. PMLR, 2021.

Peer reviews:
comments: SUMMARY:
main contributions include two-folds. For heavy-tailed payoffs, design heavy-tailed linear bandits, derive the variance-dependent T-round regret; In terms of Linear MDPs, instance-dependent K-episode regret is acquired. All paper results substantially depend on Huber loss regression techniques.

STRENGTHS:
For heavy-tailed linear bandits, the author proposed how to tune the robustification parameter that balances bias and robustness on the fly.
for linear MDPs with bounded rewards problem, the author employs separate estimation techniques to handle heavy-tailed rewards and transition kernels, i.e, utilizing adaptive Huber regression to estimate heavy-tailed rewards and weighted ridge regression to estimate the expected next-state value functions.

WEAKNESSES:
The derived regrets depend on the feature dimension, the number of rounds, the variance or central moment of the reward at the t-th round. simultaneously. It seems to have worsened the obtained results comparing to the previous research works. the overall regret scales with feature dimension, again restricting this proposed approach to small dimension problem.

QUESTIONS:
Since the author believes that their proposed algorithm is computationally effective, why not provide specific examples to demonstrate the practical application effect of the proposed algorithm?

LIMITATIONS:
the authors adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-04

comments: SUMMARY:
In this paper, reinforcement learning problem is considered in the episodic setting for linear bandits and linear MDPs under heavy-tailed rewards with potentially infinite variance. Based on adaptive Huber regression and optimism in the face of uncertainty principle, the authors propose algorithms that utilize conditional reward variance.

STRENGTHS:
The paper looks technically sound, and the presentation of the material seems to be good with clearly stated assumptions and theorem statements.

On the technical side, the use of adaptive Huber regression in linear bandit and linear MDP setting to tackle heavy-tailed reward distributions is an interesting idea.

WEAKNESSES:
In Assumption 2.6, the centralized moments of order $1+\epsilon$ of the random rewards at each step are assumed to be realizable for each state-action pair, which looks highly unrealistic. Also, in addition to the regret bounds, it would be interesting to see empirical evaluations of the proposed algorithms.

QUESTIONS:
a)	What happens in the absence of the realizability assumption in Assumption 2.6?
b)	The paper [25] seems to be considering heavy-tailed rewards also, but with finite variance. As such, it would be fair to update the last column of Table 1 and 2 based on finiteness of the variance.

LIMITATIONS:
No substantial discussion on the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper first addresses Reinforcement learning (RL) with function\napproximation in the presence of heavy-tailed noises whose central moment is known. In\ngeneral, these online learning problems rely on the self-normalized inequality to construct a\nconfidence set of optimal parameter . However, the existing self-normalized inequalities\nhave a magnitude of noise term that is intractable with heavy-tailed noises. The authors\nsolve this problem by utilizing adaptive Huber regression and deriving a robust selfnormalized\ninequality without the noise magnitude term. By using the proposed selfnormalized\ninequality, they introduce two algorithms. The first algorithm, HEAVY-OFUL, is\ndesigned for linear bandits and shown to be minimax optimal. Building upon HEAVYOFUL,\nthey presents HEAVY-LSI-UCB for linear MDPs, which has a better first-order\nregret bound than previous works. Furthermore, they provide the minimax lower regret\nbound in linear MDPs with heavy-tailed noises, which implies the minimax optimality of\nHEAVY-LSI-UCB in the worst case.\n\nSTRENGTHS:\nThis paper is the first attempt to deal with heavy-tailed RL with function\napproximation.\n-  (Optimality) The regret bound of HEAVY-OFUL, is minimax optimal in both stochastic and deterministic linear bandits with heavy-tailed rewards. In addition, the regret bound of HEAVY-LSVI-UCB, recovers the previous variance-aware regret in [1] and improves the existing instance-dependent regrets in linear MDPs [1, 2].\n- (Originality) To address the heavy-tailed rewards, the novel robust self-normalized\ninequality is established.\n\nWEAKNESSES:\n- (Computational costs) The first concern is the practical usage of the proposed\nalgorithms. As the authors noted, the regret analyses of the algorithms inherently\ndepend on the robust self-normalized inequality (Theorem 3.3), which bounds the\ndeviation of estimated parameter and the optimal parameter . However, I believe\nthe fact that is obtained from adaptive Huber regression is problematic since it\nrequires iterative optimization steps due to the absence of a closed form for (Line 5\nof Alg. 1). Indeed, the proposed algorithms (Alg. 2, 3) contain the additional iterative\nalgorithm (Alg. 1) to ensure their theoretical results, and thus share the intrinsic\ndrawback of Huber regression, which is the linear computational complexity per\niteration. In particular, HEAVY-LSVI-UCB utilizes the adaptive Huber regression in\norder to optimize both rewards and central moments.\n- (Absent experiments) There are no experiments supporting the theoretical results of\nthe proposed algorithms and addressing concerns about computational costs, even in\nsimple synthetic problems.\n- (Assumption) The authors assumed that the central moment of rewards is known.\n\nQUESTIONS:\n- Since the paper emphasizes the computational contributions of the proposed algorithms comparing with the existing ones, I think empirical results are needed. Can you provide experiments related to this?\n- If possible, please discuss computational aspects of the proposed algorithms.\n- In my understanding, the essential key in handling heavy-tailed noises is the robust self-normalized inequality (Theorem 3.3). However, it seems that the inequality involves the term, $b$, that requires prior information about central moment. While the closet work [1] suppose similar assumption, I think this is a strong condition. Can this be relaxed?\n- In line 219, the authors claim when $\\epsilon=1$ and $\\nu=0$, $\\forall t$, the regret upper bound of HEAVY-OFUL matches the lower bound of K-armed contextual bandit $\\Omega(d)$ [3]. Can you reconsider this argument? To my knowledge, the regret bound proposed in\n[3] is $\\Omega(\\sqrt{dT})$, not $\\Omega(d)$. Moreover, the settings are different each other in that [3] addresses finite-armed bandits, while this work deals with (possibly) infinite-armed bandits.\n\n[1]. Xiang Li and Qiang Sun. Variance-aware robust reinforcement learning with linear function approximation with heavy-tailed rewards. arXiv preprint arXiv:2303.05606, 2023\n[2]. Andrew J Wagenmaker, Yifang Chen, Max Simchowitz, Simon Du, and Kevin Jamieson. First order regret in reinforcement learning with linear function approximation: A robust\nestimation approach. In International Conference on Machine Learning, pages 22384–22429. PMLR, 2022.\n[3]. Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 208–214. JMLR Workshop and Conference Proceedings, 2011.\n\nLIMITATIONS:\nThere are no limitations discussed in the manuscript.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of linear bandits and linear MDPs, when the noise may be heavy tailed. The main technical tool that they use is the Huber regressor, that enables them to detect extremal noise points that are less informative, and be more robust to these. They show how this regressor can be incorporated into optimistic algorithms to provide sublinear regret bounds.\n\nSTRENGTHS:\nQuality: Claims are sound, and arguments appear to check out. The paper is well-contextualized in the literature.\n\nSignificance: Problem is relevant to practitioners, and theoretical tools may be reapplied in similar online problems elsewhere as well.\n\nOriginality: The technical work required to incorporate the huber regressor into the optimistic algorithms is nontrivial, and the efforts are appreciated.\n\nWEAKNESSES:\nClarity: While the writing is well-organized, it is quite dense at times and a slog to parse through: for instances, Sections 2.2 and Section 5. It would be useful to move some things into the appendix and instead of encapsulating commentary into remark environments, including them in the main text to introduce more flow that would make it easier for the reader.\n\nQUESTIONS:\n1. The setup being considered here with heavy-tailed additive noise is different than the situation where the rewards themselves are heavy-tailed. In the current case, the problem is more one of outlier detection to be robust to these extremal noise events. But in the other case, if an arm has a heavy tailed reward, it may be desirable to pull because there is the potential for receiving very high reward from this arm relative to other arms (i.e if an arm like a lottery ticket with low rewards with high probability but immense rewards with small probability).\n\nI am concerned that conflating these two problems may lead to confusion. Could the authors comment on this distinction? And if the authors agree, I would appreciate if the title were changed to accurately describe the setup and the distinction were made clear in the text.\n\n2. Could you comment on whether a different definition of heavy-tailed in terms of tail probabilities\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-09"}