Title: Provably Robust Temporal Difference Learning for Heavy-Tailed Rewards
Venue: NeurIPS 2023
Accepted: yes
Keywords: temporal difference learning

natural actor-critic

reinforcement learning

policy evaluation

policy gradient

markov decision processes
Abstract: In a broad class of reinforcement learning applications, stochastic rewards have heavy-tailed distributions, which lead to infinite second-order moments for stochastic (semi)gradients in policy evaluation and direct policy optimization. In such instances, the existing RL methods may fail miserably due to frequent statistical outliers. In this work, we establish that temporal difference (TD) learning with a dynamic gradient clipping mechanism, and correspondingly operated natural actor-critic (NAC), can be provably robustified against heavy-tailed reward distributions. It is shown in the framework of linear function approximation that a favorable tradeoff between bias and variability of the stochastic gradients can be achieved with this dynamic gradient clipping mechanism. In particular, we prove that robust versions of TD learning achieve sample complexities of order $\mathcal{O}(\varepsilon^{-\frac{1}{p}})$ and $\mathcal{O}(\varepsilon^{-1-\frac{1}{p}})$ with and without the full-rank assumption on the feature matrix, respectively, under heavy-tailed rewards with finite moments of order $(1+p)$ for some $p\in(0,1]$, both in expectation and with high probability. We show that a robust variant of NAC based on Robust TD learning achieves $\tilde{\mathcal{O}}(\varepsilon^{-4-\frac{2}{p}})$ sample complexity. We corroborate our theoretical results with numerical experiments.

Provably Robust Temporal Difference Learning for Heavy-Tailed Rewards

Abstract

In a broad class of reinforcement learning applications, stochastic rewards have heavy-tailed distributions, which lead to infinite second-order moments for stochastic (semi)gradients in policy evaluation and direct policy optimization. In such instances, the existing RL methods may fail miserably due to frequent statistical outliers. In this work, we establish that temporal difference (TD) learning with a dynamic gradient clipping mechanism, and correspondingly operated natural actor-critic (NAC), can be provably robustified against heavy-tailed reward distributions. It is shown in the framework of linear function approximation that a favorable tradeoff between bias and variability of the stochastic gradients can be achieved with this dynamic gradient clipping mechanism. In particular, we prove that robust versions of TD learning achieve sample complexities of order O ( ε -1 p ) and O ( ε -1 -1 p ) with and without the full-rank assumption on the feature matrix, respectively, under heavy-tailed rewards with finite moments of order (1 + p ) for some p ∈ (0 , 1] , both in expectation and with high probability. We show that a robust variant of NAC based on Robust TD learning achieves ˜ O ( ε -4 -2 p ) sample complexity. We corroborate our theoretical results with numerical experiments.

1 Introduction

In this paper, we develop a framework for robust reinforcement learning in the presence of rewards with heavy-tailed distributions. Heavy-tailed phenomena, stemming from frequently observed statistical outliers, have been ubiquitous in decision-making applications under uncertainty. To name a few examples, waiting times in wireless communication networks [44, 24, 58], completion times of SAT solvers [14], numerous payoff quantities (e.g., stock prices, consumer signals) in economics and finance [22, 35, 38, 37] exhibit heavy-tailed behavior. An important characteristic of heavy-tailed random variables is the infinite order of higher moments, which stems from the frequently occurring outliers.

In reinforcement learning (RL), the goal is to maximize expected total reward in a Markov decision process (MDP) by continual interactions with the unknown and dynamic environment. Among policy optimization methods, Natural actor-critic (NAC) method and its variants [51, 30, 46, 17, 27, 5, 29] have become particularly prevalent due to their desirable stability and versatility characteristics, emanating from the use of temporal difference (TD) learning as the critic for the policy evaluation component of the NAC operation. The existing theoretical analyses for temporal difference learning [4, 54] and natural policy gradient/actor-critic methods [62, 1, 57] assume that the stochastic gradients have finite second-order moments, or even they are bounded. In particular, it is unknown whether natural actor-critic with function approximation is robust for stochastic rewards of heavy-tailed distributions with potentially infinite second-order moments. Furthermore, in practice, these methods

are prone to non-vanishing and even increasing error under heavy-tailed reward distributions (see Example 1). This motivates us for the following fundamental question in this paper:

Can temporal difference learning with function approximation be modified to provably achieve global optimality under stochastic rewards with heavy tails?

We provide an affirmative answer to the above question by proposing a simple modification to the TD learning algorithm, which yields robustness against heavy tails. In particular, we show that incorporating a dynamic gradient clipping mechanism with a carefully-chosen sequence of clipping radii can provably robustify TD learning and NAC with linear function approximation, leading to global near-optimality even under stochastic rewards of infinite variance.

Example 1 (Failure of TD learning under heavy-tailed reward) . In this example, we consider a randomly-generated discounted-reward Markov reward process 1 ( X t , R t ) t on a state space X with | X | = 64 states, with the discount factor γ = 0 . 9 and the reward R t ( X t ) = r ( X t ) + N t -E [ N t ] with N t iid ∼ Pareto (1 , 1 . 4) for any t . In order to predict the value function, we use (projected) TD learning (see [4]) with linear function approximation based on Gaussian features of dimension d = 4 and projection radius ρ = 30 . The performance results are shown in Figure 1. Since R t is heavy-tailed

0.80

2.50

TD learning

Mean Squared Error (MSE)

10-

Mean Squared Error (MSE)

0.75

Mean Squared Error (MSE)

2.25

Robust TD learning

2.00

0.70

101

1.75

0.65

1.50

0.60

1.25

100

0.55

1.00

0.50

0.75

10-1

0.45

0.50

0

100000

200000

300000

400000

500000

0

100000

200000

300000

400000

500000

0

100000

200000

300000

400000

500000

Iteration t

Iteration t

Iteration t

(a) Errors for TD Learning

(b) Errors for Robust TD Learning (c) Comparison of the expected errors

with infinite variance, the existing convergence results for traditional TD learning, which assume that R t has finite variance, do not hold. Furthermore, Figure 1 reveals that TD learning is prone to non-varnishing and even increasing error in practice despite the projection step, iterate averaging and small learning rate, due to the statistical outliers that cause extremely large error often as indicated by a non-negligible fraction of green lines in Figure 1a. On the other hand, with the same learning rate, projection radius and state-reward realizations, our robust variant of TD learning provides resilience against outliers (see Figure 1b), and leads to convergence in the expected behavior as in Figure 1c.

Stochastic rewards with heavy-tailed distributions appear in many important applications. Below, we briefly provide two motivating applications that necessitate robust RL methods to handle heavy tails.

Application (1): Algorithm portfolios. In solving complicated problems such as Boolean satisfiability (SAT) and complete search problems, which appear in numerous applications [19, 43], multiple algorithmic solutions with different characteristics are available. The algorithm selection problem is concerned about the minimization of total execution times to solve these problems [32, 49, 31], where different data distributions and machine characteristics, caused by recursive algorithms, are modeled as states, algorithm choices are modeled as actions, and the execution time of a selected algorithm is modeled as the cost (i.e., negative reward). It is well-known that the execution times, i.e., rewards, in the algorithm selection problem have heavy-tailed distributions with infinite-variance (e.g., Pareto (1 , 1 + p ) with 0 < p < 1 as in [13]) similar to the case in Example 1 [13, 15, 49]. Thus, algorithm selection problem requires robust techniques that we consider here.

Application (2): Scheduling for wireless networks. The scheduling problem considers matching the users with random service demands to fading wireless channels (e.g., Gilbert-Elliot model) with stochastic transmission times so as to minimize the expected delay. A widely-adopted approach to

study the scheduling problem is to use MDPs (see, e.g., [40, 21, 10, 2]). It has been observed that the transmission times follow heavy-tailed distributions of infinite variance, due to various factors including the MAC protocol used, packet size, and channel fading [18, 58, 24, 23]. As such, solving this by using RL approach necessitates robust methods to handle heavy-tailed execution times.

Main contributions. Our main contributions in this paper contain the following.

· Robust TD learning with dynamic clipping for heavy-tailed rewards. We propose Robust TD learning with a dynamic gradient clipping mechanism, and prove that this TD learning variant with linear function approximation can achieve arbitrarily small estimation error that vanishes at rates O ( T -p 1+ p ) and ˜ O ( T -p ) without and with full-rank assumption on the feature matrix, respectively, even for heavy-tailed rewards with moments of order 1 + p for p ∈ (0 , 1] . Our proof techniques make use of Lyapunov analysis coupled with martingale techniques for robust statistical estimation in dynamical systems, and can be of independent interest in the analysis of first-order methods.

· Robust NAC under heavy-tailed rewards. Based on Robust TD learning and the compatible function approximation result in [27], we propose a robust NAC variant, and show that O ( ε -4 -2 p ) samples suffice to achieve ε > 0 error under standard concentrability coefficient assumptions.

· High-probability error bounds. We provide high-probability (sub-Gaussian) error bounds for the robust NAC and TD learning methods in addition to the traditional expectation bounds.

From a statistical viewpoint, our analysis in this work indicates a favorable bias-variance tradeoff: by introducing a vanishing bias to the semi-gradient via particular choices of dynamic gradient clipping, one can achieve robustness by eliminating the destructive impacts of statistical outliers even if the semi-gradient has infinite variance, leading to near optimality.

1.1 Related Work

Temporal difference learning. Temporal difference (TD) learning was proposed in [50], and has been the prominent policy evaluation method. The existing theoretical analyses of TD learning consider MDPs with bounded rewards [4, 7], or rewards with finite variance [54], while we consider heavy-tailed rewards. Our analysis utilizes the Lyapunov techniques in [4].

Policy gradient methods. Policy gradient (PG), and its variant natural policy gradient (NPG) have attracted significant attention in RL [59, 52, 27]. Recent theoretical works investigate the local and global convergence of these methods in the exact case, or with stochastic and bounded rewards [1, 57, 34, 39, 61]. As such, heavy-tailed rewards have not been considered in these works.

Bandits with heavy-tailed rewards. Stochastic bandit variants with heavy-tailed payoffs were studied in multiple works [6, 48, 33, 8]. The stochastic bandit setting can be interpreted as a very simple single-state (i.e., stateless or memoryless) model-based and tabular RL problem. The model we consider in this paper is a model-free RL setting on an MDP with a large state space, which is considerably more complicated than the bandit setting.

Stochastic gradient descent with heavy-tailed noise. There has been an increasing interest in the analysis of SGD with heavy-tailed gradient noise recently [56, 11, 16], following the seminal work of [45]. In our work, we consider the RL problem, which has significantly different dynamics than stochastic convex optimization.

Robust mean and covariance estimation. In basic statistical problems of mean and covariance estimation [42, 41, 36] and regression [20], the traditional methods do not yield the optimal convergence rates for heavy-tailed random variables, which led to the development of robust mean and covariance estimation techniques (for reference, see [36, 28]). Our paper utilizes tools from robust mean estimation literature (particularly, truncated mean estimator analysis in [6]), but considers the more complicated problem of TD learning and policy optimization in a dynamic environment rather than a static mean or covariance estimation problem with iid observations.

1.2 Notation

For a symmetric matrix A ∈ R d × d , λ min ( A ) denotes its minimum eigenvalue. B 2 ( x, ρ ) = { y ∈ R d : ∥ x -y ∥ 2 ≤ ρ } and Π C { x } = arg min y ∈C ∥ x -y ∥ 2 2 for any convex C ⊂ R d .

2 Robust TD Learning for Value Prediction under Heavy Tails

First, we consider the problem of predicting the value function for a given discounted-reward Markov reward process with heavy-tailed rewards.

2.1 Value Prediction Problem

For a finite but arbitrarily large state space X , let ( X t ) t ∈ N be an X -valued Markov chain with the transition kernel P : X × X → [0 , 1] . We consider a Markov reward process ( X t , R t ) t ∈ N such that at state X t , a stochastic reward R t = R t ( X t ) is obtained for all t ≥ 0 . For a discount factor γ ∈ [0 , 1) , the value function for the MRP ( X t , R t ) t ∈ N is the following:

Objective. The goal is to learn V without knowing the transition kernel P by using samples from the system. In particular, for a parameterized class of functions { f Θ : X → R : Θ ∈ R d } , the goal is to solve the following stochastic optimization problem with mean squared error:

In order to solve (2) under Assumption 1, next we propose a robust variant of temporal difference (TD) learning with linear function approximation [50, 54].

2.2 Robust TD Learning Algorithm

For a given set of feature vectors { Φ( x ) ∈ R d : x ∈ X } with sup x ∈ X ∥ Φ( x ) ∥ 2 ≤ 1 , we use f Θ ( · ) = ⟨ Θ , Φ( · ) ⟩ as the approximation architecture. For a given dataset D = { ( X t , R t , X ′ t ) ∈ X × R × X : t ∈ N } with X ′ t ∼ P ( X t , · ) , let the stochastic semi-gradient at Θ ∈ R d be defined as

Robust TD learning is summarized in Algorithm 1.

Algorithm 1: Robust TD learning

Inputs: number of steps T ≥ 1 , clipping radii ( b t ) t ∈ [ T ] , projection radius ρ > 0 , step-size η > 0 Set Θ(1) ∈ B 2 (0 , ρ ) \\ initialization

In the following, we will establish finite-time bounds for Robust TD learning by specifying the sequence of dynamic gradient clipping radii ( b t ) t ≥ 1 , projection radius ρ and step-size η .

2.3 Finite-Time Bounds for Robust TD Learning

We make the following assumptions on the Markov reward process.

Assumption 1. The stochastic process ( X t , R t ) t ∈ N satisfies the following:

Ergodicity: ( X t ) t ∈ N is an irreducible and aperiodic Markov chain with stationary distribution µ = µ P . Also, we assume that there are constants m> 0 , ζ ∈ (0 , 1) such that

Heavy-tailed reward: For some p ∈ (0 , 1] and constant u 0 ∈ (0 , ∞ ) ,

Mean reward: For any t ∈ N , E [ R t ( X t ) | X t ] = r ( X t ) ∈ [ -1 , 1] a.s.

We note that the uniform ergodicity and bounded mean reward assumptions are standard in TD learning literature [3, 54, 4].

Assumption 2 (Sampling) . We consider two types of sampling strategies in this work:

2a. IID sampling: X t iid ∼ µ and X ′ t ∼ P ( X t , · ) for all t ≥ 1 .

X

1

∼

µ

and

X

′

t

=

X

t

+1

∼ P

(

X

t

,

·

)

for all

t

1

.

Assumption 3 (Realizability) . There exists Θ ⋆ ∈ B 2 (0 , ρ ) such that V ( · ) = ⟨ Θ ⋆ , Φ( · ) ⟩ .

Remark 1. We note that Assumption 3 holds directly in interesting realizable problem classes, e.g., linear MDPs [25], and allows us to obtain results on the statistical error performance of our design. In cases when it does not hold, our results will continue to hold with an additional function approximation error proportional to inf Θ ∈ B 2 (0 ,ρ ) √ E [ |V ( x ) -⟨ Θ , Φ( x ) ⟩| 2 ] , which is unavoidable due to the limitation of the linear function approximation architecture.

The following lemma is important in bounding the moments of the gradient norm under Robust TD learning in terms of the projection radius ρ > 0 and the upper bound u 0 on E [ | R t ( X t ) | 1+ p ∣ ∣ X t ] .

Lemma 1 (Tail bounds for ∥ g t ( Θ( t ) ) ∥ 2 ) . Let F + t = σ ( Θ(1) , Θ(2) , . . . , Θ( t ) , X t ) for t ∈ Z + . Then, under Assumption 1, we have:

for any t ∈ Z + , where u = min { ( u 1+ p 0 +2 ρ ) 1+ p , u 0 +2 2 p +3 ρ 1+ p } .

Proof. Note that we have E [ ∥ g t (Θ( t )) ∥ 1+ p 2 |F + t ] ≤ E [ | R t ( X t ) + γf Θ( t ) ( X ′ t ) -f Θ( t ) ( X t ) | 1+ p |F + t ] since sup x ∈ X ∥ Φ( x ) ∥ 2 ≤ 1 . The upper bounds then follow by applying Minkowski's inequality and the triangle inequality for L p spaces, respectively, to this inequality.

This lemma will be useful in the analysis of both the expected (Theorems 1-2), and the high-probability bounds (Theorem 3) on the performance of Robust TD learning.

Next, we provide the main theoretical results in this paper: finite-time bounds for Robust TD learning. The proofs are mainly deferred to the appendix, while we provide a proof sketch for Theorem 3. In the following, we provide convergence bounds for the expected mean squared error under Robust TD learning with various choices of b t .

Theorem 1 (Expected error under Robust TD learning - iid sampling) . Under Assumptions 1, 2a, 3, we have the following bounds for Robust TD learning:

b) Let Λ = ∑ x ∈ X µ ( x )Φ( x )Φ ⊤ ( x ) , and C p ( u, λ min , γ, ρ ) = u 1 -γ ( 4 ρ + 1 (1 -γ ) λ min ) . If λ min (Λ) = λ min > 0 , then with the diminishing step-size η t = 1 (1 -γ ) tλ min and b t = t for t ∈ Z + , for the average iterate ¯ Θ( T ) , we have 2 :

and for the last iterate Θ( T +1) , we have:

̸

for any T > 1 , where 1 x,y = 1 if x = y and 0 otherwise.

Remark 2. The convergence rates in Theorem 1 are O ( T -p 1+ p ) and ˜ O ( T -p ) without and with the full-rank assumption λ min > 0 , respectively. For p = 1 , the convergence rates stated in Theorem 1 both match the existing results for TD learning with bounded rewards [4], up to a larger scaling factor of raw second-order moment rather than variance, due to clipping centered around 0.

Remark 3 (Finite-time bounds in the unrealizable case) . We note that our results hold without Assumption 3 as well. In this general case, there will be an additional function approximation error proportional to inf Θ ∈ B 2 (0 ,ρ ) √ E [ |V ( x ) -⟨ Θ , Φ( x ) ⟩| 2 ] (see Remark 1). One would use a richer function approximation scheme (e.g., larger projection radius ρ or dimension d ) to reduce this function approximation error, which will lead to an increase in the statistical error as we characterize in our bounds. For the extension of our analysis to the general case without Assumption 3, please see Appendix A.1.

In the following, we provide convergence bounds for Robust TD learning under Markovian sampling.

Theorem 2 (Expected error under Robust TD learning - Markovian sampling) . Under Assumptions 1,2b and 3, let T > 1 , ρ > 0 be given, and define the mixing time τ = min { t ∈ Z + : mζ t ≤ √ 2 ρ ( uT ) -1 1+ p } . Then, with η t = η = √ 2 ρ ( uT ) -1 1+ p , Robust TD learning yields the following:

The proof of Theorem 2 is based on a similar Lyapunov technique as Theorem 1 in conjunction with the mixing time analysis in [4] for Markovian sampling, and can be found in Appendix A.

The bounds in Theorem 1 involve expectation over the parameters Θ( t ) , t ∈ [ T ] . In the following, we provide a high-probability error bound on the mean squared error under Robust TD learning.

Theorem 3 (High-probability bound for Robust TD learning) . For any δ ∈ (0 , 1) , let L δ = log(4 /δ ) .

Under Assumptions 1, 2a, 3, with step-size η = √ 2(1 -γ ) ρL 1 -p 2(1+ p ) δ ( uT ) 1 1+ p and clipping radius b t = ( ut L δ ) 1 1+ p ,

holds with probability at least 1 -δ .

In the following, we give a proof sketch for Theorem 3. The full proof can be found in Appendix A.

Proof sketch. Let L (Θ) = ∥ Θ -Θ ⋆ ∥ 2 2 be the Lyapunov function, and χ t = 1 -¯ χ t = ✶ {∥ g t ∥ 2 ≤ b t } . Then, the Lyapunov drift can be decomposed as follows:

where

is the bias in the stochastic semi-gradient, and

We can decompose B ( t ) further into a martingale difference sequence

and a bias term

By Freedman's inequality for martingales [12, 53], we have 1 T ∑ T t =1 B 0 ( t ) ≤ 7 ρu 1 1+ p L p 1+ p δ T p 1+ p , and by

Azuma inequality, we have 1 T ∑ t t =1 Z ( t ) ≤ u 2 1+ p T 1 -p 1+ p L 1 -p 1+ p δ , each holding with probability at least 1 -δ/ 2 .

By Hölder's inequality and Lemma 1, we can bound B ⊥ ( t ) ≤ ub -p t and E t [ ∥ g t ∥ 2 2 ¯ χ t ] ≤ ub 1 -p t , both with probability 1. Finally, by Lemma 2 in [54], we have the negative drift term

By telescoping sum of (11) and rearranging the terms, we have:

The proof is concluded by substituting the above high probability bounds on the sample means of B ( t ) , Z ( t ) and E t [ ∥ g t ∥ 2 2 ¯ χ t ] (via union bound and integral upper bounds), and using Jensen's inequality on the left side of the above inequality.

Most notably, this important theorem establishes that, by appropriately controlling the bias term of dynamic gradient clipping to yield a vanishing sample mean with high probability as the number of iterations increases, one can limit the variance of the semi-gradient, thereby resulting in the provided global near-optimality guarantee.

3 Robust Natural Actor-Critic for Policy Optimization under Heavy Tails

In this section, we will study a two-timescale robust natural actor-critic algorithm (Robust NAC, in short) based on Robust TD learning, and provide finite-time bounds.

3.1 Policy Optimization Problem

We consider a discounted-reward Markov decision process (MDP) with a finite but arbitrarily large state space S , finite action space A , transition kernel P and discount factor γ ∈ (0 , 1) . The controlled Markov chain { ( S t , A t ) ∈ S × A : t ∈ N } has the probability transition dynamics P ( S t +1 ∈ s | S t 1 , A t 1 ) = P A t ( S t , s ) , for any s ∈ S . Taking the action A t ∈ A at state S t ∈ S yields a random reward of R t ( S t , A t ) at any t ∈ Z + . For a given stationary randomized policy π = ( π ( a | s )) ( s,a ) ∈ S × A , the value function V π and the state-action value function (also known as Q-function) Q π are defined as:

Remark 4 (From MDP to MRP) . Under any stationary randomized policy π , the process ( S t , A t ) t> 0 =: ( X t ) t> 0 is a Markov chain over the state-space X = S × A , thus ( X t , R t ) with R t ( X t ) = R t ( S t , A t ) is a Markov reward process of the kind that we analyzed in Section 2. As such, we can use Robust TD learning to evaluate V ( x ) = Q π ( x ) for any x = ( s, a ) ∈ S × A .

Heavy-tailed reward. We assume that the process ( X t , R t ) t> 0 with the Markov chain X t = ( S t , A t ) and the reward R t = R t ( X t ) satisfies Assumption 1. We denote the stationary distribution of X t = ( S t , A t ) under π as µ π .

Objective. For an initial state distribution λ , the objective in this work is to find the following:

over the class of stationary randomized policies.

Policy parameterization. In this work, we consider a finite but arbitrarily large state space S , and for such problems, the tabular methods do not scale [51, 3]. In order to address this scalability issue, we consider widely-used softmax parameterization with linear function approximation: for a given set of feature vectors { Φ( s, a ) ∈ R d : s ∈ S , a ∈ A } and policy parameter W ∈ R d ,

In the following subsection, we will describe the robust natural actor-critic algorithm.

3.2 Robust Natural Actor-Critic Algorithm

For any iteration k ≥ 1 , we denote π k := π W ( k ) throughout the policy optimization iterations.

For samples D ( k ) = { ( S t,k , A t,k , R t,k , S ′ t,k , A ′ t,k ) : t ≥ 1 } , given ( b t,k ) t,k ∈ Z + and ρ > 0 , Robust NAC Algorithm is summarized in Algorithm 2.

Algorithm 2: Robust Natural Actor-Critic Algorithm

Remark 5. The optimal solution Θ ⋆ k ∈ arg min Θ ∈ R d E x =( s,a ) ∣ ∣ ∣ ⟨ Θ , Φ( x ) ⟩ - Q π k ( x ) ∣ ∣ ∣ 2 is a good approximation of the natural policy gradient:

which follows from Jensen's inequality and leads to the Q-NPG [1]. For a detailed discussion, refer to Appendix B.

3.3 Finite-Time Bounds for Robust Natural Actor-Critic

In this subsection, we will provide finite-time bounds for Robust NAC.

We assume that the resulting Markov reward process under π k for each k satisfies Assumptions 1-3 with stationary distribution µ π k and u k ≥ E [ | R t,k ( S t,k , A t,k ) | 1+ p | S t,k , A t,k ] . We assume that the dataset D ( k ) is obtained independently at each iteration k ≥ 1 for simplicity, with ( S t,k , A t,k ) iid ∼ µ π k and S ′ t,k ∼ P A t,k ( S t,k , · ) and A t,k ∼ π k ( ·| S t,k ) according to Assumption 2a under the stationary distribution µ π k = [ µ π k ( s, a )] s ∈ S ,a ∈ A under π k . We make the following standard assumption for policy optimization, which is common in the policy gradient literature [1, 57, 34].

Assumption 4 (Concentrability) . For any k ≥ 1 , we assume that there exists C conc < ∞ such that:

where µ π k is the stationary distribution of ( S t,k , A t,k ) t ≥ 1 under π k .

Theorem 4 (Finite-time bounds for Robust NAC) . Under Assumptions 1-4 for any k ≥ 1 , for any δ ∈ (0 , 1) and T, K > 1 , Robust NAC with ρ ≥ max k ∥ Θ ⋆ k ∥ 2 , b t,k = ( u k T log(4 T/δ ) ) 1 1+ p , learning rates

where L δ = log(4 T/δ ) .

The proof of Theorem 4 can be found in Appendix B.

Remark 6 (Sample complexity of Robust NAC) . An immediate consequence of Theorem 4 is as follows: the best iterate error decays at a rate ˜ O ( 1 √ K ) + ˜ O ( 1 T p 2(1+ p ) ) after K iterations of natural policy gradient, which contains T steps of Robust TD learning per iteration. As such, in order to achieve ε > 0 error, one needs T × K = ˜ O ( ε -2 -2(1+ p ) /p ) samples.

Remark 7. Theorem 4 can be easily extended to expected error bounds and the full-rank case, where we would have ˜ O ( K -1 / 2 + T -p ) by using Theorem 1. By extending the analysis in Theorem 2, one can prove results for Markovian sampling as well.

Remark 8 (Exploration in Robust NAC) . Assumption 4 implies that the stationary state-action distribution under π k should be sufficiently exploratory so that it should have a nonzero probability at each state-action pair that is visited by the optimal policy π ⋆ . This assumption on the exploratory behavior of policy gradient methods is standard in reinforcement learning literature [1, 57, 34, 63]. Alternatively, entropy regularization can be used for natural policy gradient methods to encourage exploration, which would imply weaker conditions at the expense of an additional bias term in the function approximation setting [47, 9].

4 Numerical Results

In this section, we present numerical results for Robust TD learning and its non-robust counterpart.

(1) Randomly-Generated MRP. In the first example, we consider a randomly-generated MRP with | X | = 256 . The transition kernel is randomly generated such that P ( x, x ′ ) iid ∼ Unif (0 , 1) , and rowwise normalized to obtain a stochastic matrix. The feature dimension is d = 128 , random features are generated according to the χ -squared distribution Φ( x ) = Φ 0 ( x ) / ∥ Φ 0 ( x ) ∥ 2 with Φ 0 ( x ) ∼ N (0 , I d ) for all x ∈ X , Θ ⋆ ∼ 3 U/ √ d for U ∼ Unif d (0 , 1) and Ψ = [ Φ ⊤ ( x ) ] x ∈ X . The discount factor is γ = 0 . 9 , and the reward is R t ( X t ) = r ( X t ) + N t -E [ N t ] with N t iid ∼ Pareto (1 , 1 . 2) . Mean squared error (2) under Robust TD learning and TD learning with the clipping radius b t = t and diminishing step-size η t = 1 λ min (1 -γ ) t in Theorem 1 and projection radius ρ = 30 are shown in Figure 2. Despite

(a) Errors for TD Learning

1012

105

109

TD leaning

Mean Squared Error (MSE)

1010

Mean Squared Error (MSE)

104

Mean Squared Error (MSE)

Robust TD learning

107

108

103

106

102

105

104

103

101

102

100

101

100

20000

40000

60000

80000

100000

20000

40000

Iteration t

60000

80000

100000

20000

40000

Iteration t

60000

80000

100000

diminishing step-size and projection, TD learning fails miserably often and in expectation due to the outliers in the reward that lead to extremely large errors (Figure 2a). On the other hand, for the same feature vectors, state and reward realizations, Robust TD learning effectively eliminates them in every sample path, and achieves good and consistent performance despite extremely heavy-tailed reward and gradient noise with tail index 1.2 (Figure 2a).

(2) Circular Random Walk. In this example, we consider a circular random walk for X = { 1 , 2 , . . . , 256 } , where each state x is modulo-| X | [60]. The transition matrix is generated as P ( x, x ′ ) = 1 / 3 if x = x ′ and P ( x, x ′ ) = 1 / 24 if 0 < | x -x ′ | ≤ 8 . The reward and random feature generation is the same as the first example. The performances of TD learning and Robust TD learning in this structured case after 1000 trials are given in Figure 3.

A similar behavior as the randomly-generated MRP is observed in this example: due to outliers, TD learning fails miserably, while Robust TD learning achieves good performance consistently.

109

102

106

TD learning

Robust TD learning

Mean Squared Error (MSE)

107

Mean Squared Error (MSE)

Mean Squared Error (MSE)

105

105

101

104

10°

103

100

102

101

101

10-1

101

100

0

10000

20000

30000

40000

50000

0

10000

20000

30000

40000

50000

0

10000

20000

30000

40000

50000

Iteration t

Iteration t

Iteration t

(a) Errors for TD Learning

(b) Errors for Robust TD Learning (c) Comparison of the expected errors

5 Conclusion

In this paper, we considered RL problem with heavy-tailed rewards, and proposed robust TD learning and NAC variants with a dynamic gradient clipping mechanism with provable performance guarantees, both in expectation and with high probability. Motivated by the results in this work, it would be interesting to explore single-timescale robust NAC and off-policy NAC for future work.

Acknowledgments and Disclosure of Funding

Atilla Eryilmaz's research was supported in part by NSF AI Institute (AI-EDGE) 2112471, CNSNeTS-2106679, CNS-NeTS-2007231; and the ONR Grant N00014-19-1-2621.

References

Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. Optimality and approximation with policy gradient methods in markov decision processes. In Conference on Learning Theory , pages 64-66. PMLR, 2020.

Mukul Agarwal, Vivek S Borkar, and Abhay Karandikar. Structural properties of optimal transmission policies over a randomly varying channel. IEEE Transactions on Automatic Control , 53(6):1476-1491, 2008.

Dimitri P Bertsekas and John N Tsitsiklis. Neuro-dynamic programming . Athena Scientific, 1996.

Jalaj Bhandari, Daniel Russo, and Raghav Singal. A finite time analysis of temporal difference learning with linear function approximation. arXiv preprint arXiv:1806.02450 , 2018.

Shalabh Bhatnagar, Richard S Sutton, Mohammad Ghavamzadeh, and Mark Lee. Natural actor-critic algorithms. Automatica , 45(11):2471-2482, 2009.

Sébastien Bubeck, Nicolo Cesa-Bianchi, and Gábor Lugosi. Bandits with heavy tail. IEEE Transactions on Information Theory , 59(11):7711-7717, 2013.

Qi Cai, Zhuoran Yang, Jason D Lee, and Zhaoran Wang. Neural temporal-difference learning converges to global optima. Advances in Neural Information Processing Systems , 32, 2019.

Semih Cayci, Atilla Eryilmaz, and Rayadurgam Srikant. Budget-constrained bandits over general cost and reward distributions. In International Conference on Artificial Intelligence and Statistics , pages 4388-4398. PMLR, 2020.

Semih Cayci, Niao He, and Rayadurgam Srikant. Linear convergence of entropy-regularized natural policy gradient with linear function approximation. arXiv preprint arXiv:2106.04096 , 2021.

Rong-Rong Chen and Sean Meyn. Value iteration and optimization of multiclass queueing networks. Queueing Systems , 32:65-97, 1999.

Ashok Cutkosky and Harsh Mehta. High-probability bounds for non-convex stochastic optimization with heavy tails. Advances in Neural Information Processing Systems , 34:4883-4895, 2021.

David A Freedman. On tail probabilities for martingales. the Annals of Probability , pages 100-118, 1975.

Carla P Gomes and Bart Selman. Algorithm portfolios. Artificial Intelligence , 126(1-2):43-62, 2001.

Carla P Gomes, Bart Selman, Nuno Crato, and Henry Kautz. Heavy-tailed phenomena in satisfiability and constraint satisfaction problems. Journal of automated reasoning , 24(1-2):67, 2000.

Carla P Gomes, Bart Selman, Henry Kautz, et al. Boosting combinatorial search through randomization. AAAI/IAAI , 98:431-437, 1998.

Mert Gurbuzbalaban, Umut Simsekli, and Lingjiong Zhu. The heavy-tail phenomenon in sgd. In International Conference on Machine Learning , pages 3964-3975. PMLR, 2021.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning , pages 1861-1870. PMLR, 2018.

Mor Harchol-Balter. The effect of heavy-tailed job size distributions on computer system design. In Proc. of ASA-IMS Conf. on Applications of Heavy Tailed Distributions in Economics, Engineering and Statistics , 1999.

Holger H Hoos and Thomas Stützle. Stochastic local search: Foundations and applications . Elsevier, 2004.

Daniel Hsu and Sivan Sabato. Heavy-tailed regression with a generalized median-of-means. In International Conference on Machine Learning , pages 37-45. PMLR, 2014.

Yu-Pin Hsu, Eytan Modiano, and Lingjie Duan. Scheduling algorithms for minimizing age of information in wireless broadcast networks with random arrivals. IEEE Transactions on Mobile Computing , 19(12):2903-2915, 2019.

Marat Ibragimov, Rustam Ibragimov, and Johan Walden. Heavy-tailed distributions and robustness in economics and finance , volume 214. Springer, 2015.

Predrag R Jelenkovi´ c and Jian Tan. Is aloha causing power law delays? In Managing Traffic Performance in Converged Networks: 20th International Teletraffic Congress, ITC20 2007, Ottawa, Canada, June 17-21, 2007. Proceedings , pages 1149-1160. Springer, 2007.

Predrag R Jelenkovi´ c and Jian Tan. Characterizing heavy-tailed distributions induced by retransmissions. Advances in Applied Probability , 45(1):106-138, 2013.

Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory , pages 21372143. PMLR, 2020.

Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In In Proc. 19th International Conference on Machine Learning . Citeseer, 2002.

Sham M Kakade. A natural policy gradient. Advances in neural information processing systems , 14, 2001.

Yuan Ke, Stanislav Minsker, Zhao Ren, Qiang Sun, and Wen-Xin Zhou. User-friendly covariance estimation for heavy-tailed distributions. Statistical Science , 34(3):454-471, 2019.

Byungchan Kim, Jooyoung Park, Shinsuk Park, and Sungchul Kang. Impedance learning for robotic contact tasks using natural actor-critic algorithm. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) , 40(2):433-443, 2009.

Vijay R Konda and John N Tsitsiklis. Actor-critic algorithms. In Advances in neural information processing systems , pages 1008-1014. Citeseer, 2000.

Lars Kotthoff. Algorithm selection for combinatorial search problems: A survey. Data mining and constraint programming: Foundations of a cross-disciplinary approach , pages 149-190, 2016.

Michail G Lagoudakis, Michael L Littman, et al. Algorithm selection using reinforcement learning. In ICML , pages 511-518, 2000.

Kyungjae Lee, Hongjun Yang, Sungbin Lim, and Songhwai Oh. Optimal algorithms for stochastic multi-armed bandits with heavy tailed rewards. Advances in Neural Information Processing Systems , 33:8452-8462, 2020.

Boyi Liu, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural proximal/trust region policy optimization attains globally optimal policy. arXiv preprint arXiv:1906.10306 , 2019.

Mico Loretan and Peter CB Phillips. Testing the covariance stationarity of heavy-tailed time series: An overview of the theory with applications to several financial datasets. Journal of empirical finance , 1(2):211-248, 1994.

Gábor Lugosi and Shahar Mendelson. Mean estimation and regression under heavy-tailed distributions: A survey. Foundations of Computational Mathematics , 19(5):1145-1190, 2019.

Benoit Mandelbrot. New methods in statistical economics. Journal of political economy , 71(5):421-440, 1963.

Mark M Meerschaert and Hans-Peter Scheffler. Moving averages of random vectors with regularly varying tails. Journal of Time Series Analysis , 21(3):297-328, 2000.

Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. In International Conference on Machine Learning , pages 6820-6829. PMLR, 2020.

Sean Meyn. Control techniques for complex networks . Cambridge University Press, 2008.

Stanislav Minsker. Geometric median and robust estimation in banach spaces. Bernoulli , pages 2308-2335, 2015.

Stanislav Minsker. Sub-gaussian estimators of the mean of a random matrix with heavy-tailed entries. The Annals of Statistics , 46(6A):2871-2903, 2018.

Rajeev Motwani and Prabhakar Raghavan. Randomized algorithms . Cambridge university press, 1995.

Jayakrishnan Nair, Adam Wierman, and Bert Zwart. The fundamentals of heavy-tails: Properties, emergence, and identification. In Proceedings of the ACM SIGMETRICS/international conference on Measurement and modeling of computer systems , pages 387-388, 2013.

Arkadij Semenoviˇ c Nemirovskij and David Borisovich Yudin. Problem complexity and method efficiency in optimization . Wiley-Interscience, 1983.

Jan Peters and Stefan Schaal. Natural actor-critic. Neurocomputing , 71(7-9):1180-1190, 2008.

Lior Shani, Yonathan Efroni, and Shie Mannor. Adaptive trust region policy optimization: Global convergence and faster rates for regularized mdps. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 5668-5675, 2020.

Han Shao, Xiaotian Yu, Irwin King, and Michael R Lyu. Almost optimal algorithms for linear stochastic bandits with heavy-tailed payoffs. Advances in Neural Information Processing Systems , 31, 2018.

Matthew Streeter and Daniel Golovin. An online algorithm for maximizing submodular functions. Advances in Neural Information Processing Systems , 21, 2008.

Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning , 3(1):9-44, 1988.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Richard S Sutton, David A McAllester, Satinder P Singh, Yishay Mansour, et al. Policy gradient methods for reinforcement learning with function approximation. In NIPs , volume 99, pages 1057-1063. Citeseer, 1999.

Joel A Tropp. Freedman's inequality for matrix martingales. Electronic Communications in Probability , 16(25):262-270, 2011.

John N Tsitsiklis and Benjamin Van Roy. An analysis of temporal-difference learning with function approximation. IEEE transactions on automatic control , 42(5):674-690, 1997.

Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge university press, 2019.

Hongjian Wang, Mert Gurbuzbalaban, Lingjiong Zhu, Umut Simsekli, and Murat A Erdogdu. Convergence rates of stochastic gradient descent under infinite noise variance. Advances in Neural Information Processing Systems , 34:18866-18877, 2021.

Lingxiao Wang, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural policy gradient methods: Global optimality and rates of convergence. arXiv preprint arXiv:1909.01150 , 2019.

Ward Whitt. The impact of a heavy-tailed service-time distribution upon the m/gi/s waiting-time distribution. Queueing Systems , 36(1-3):71-87, 2000.

Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8(3-4):229-256, 1992.

Eric Xia and Martin Wainwright. Krylov-bellman boosting: Super-linear policy evaluation in general state spaces. In International Conference on Artificial Intelligence and Statistics , pages 9137-9166. PMLR, 2023.

Tengyu Xu, Zhe Wang, and Yingbin Liang. Improving sample complexity bounds for (natural) actor-critic algorithms. arXiv preprint arXiv:2004.12956 , 2020.

Tengyu Xu, Zhe Wang, and Yingbin Liang. Non-asymptotic convergence analysis of two time-scale (natural) actor-critic algorithms. arXiv preprint arXiv:2005.03557 , 2020.

Rui Yuan, Simon S Du, Robert M Gower, Alessandro Lazaric, and Lin Xiao. Linear convergence of natural policy gradient methods with log-linear policies. arXiv preprint arXiv:2210.01400 , 2022.

A Proofs for Robust TD Learning

The following lemma will be critical in our proofs.

Lemma 2 (Lemma 4 in [54]) . For any two vectors ̂ V , V ∈ R | X | ,

where is the Bellman operator.

Proof of Theorem. 1. The proof follows the Lyapunov approach in [4]. Let L (Θ) = ∥ Θ -Θ ∗ ∥ 2 2 be the Lyapunov function for any Θ ∈ R d . Then, by the non-expansivity of Π B 2 (0 ,ρ ) , we have:

✶ Taking conditional expectation given F t and using the fact that ✶ {∥ g t (Θ( t )) ∥ 2 > b t } = 1 -✶ {∥ g t (Θ( t )) ∥ 2 ≤ b t } , we get:

in the last term. Now, for E t [ g ⊤ t (Θ( t ))(Θ( t ) -Θ ⋆ )] , we have the following inequality:

where T is the Bellman operator (17). By using the fact that the value function V is the fixed point of the Bellman operator T , we have the following:

By using Lemma 2, we conclude that:

Then, we can rewrite (19) as follows:

✶ The bias introduced by using the gradient clipping can be bounded as follows:

✶ which follows from Cauchy-Schwarz inequality, triangle inequality and the fact that max {∥ Θ( t ) ∥ 2 , ∥ Θ ∗ ∥ 2 } ≤ ρ due to projection. Using Hölder's inequality on the RHS of (24), we obtain:

Using Markov's inequality, we bound the bias due to using the clipped stochastic gradient as:

✶ Substituting (25) into (19), and taking expectation over the trajectory F t , we obtain:

Telescoping sum over t = 1 , 2 , . . . , T yields:

Rearranging the terms, using Jensen's inequality and L (Θ(1)) ≤ 4 ρ 2 , and substituting the step-size η yields the result.

(b) For the full-rank case, note that

which implies (together with (23)) that:

With the step-size choice η t = 1 (1 -γ ) λt , we obtain by induction:

By rearranging the terms and using the integral bound for the summations above, and using the Jensen's inequality for the µ -norm, we obtain the result.

Proof of Theorem 2. Let F ++ t = σ (Θ(1) , . . . , Θ( t ) , X t , X t +1 ) and E ++ t [ · ] = E [ ·|F ++ t ] . Also, let

The bias due to Markovian sampling is:

With the above definitions, the Lyapunov drift at time t ≥ 1 can be bounded as follows:

where χ t = 1 -¯ χ t = ✶ {∥ g t (Θ( t )) ∥ 2 ≤ b t } . Compared to the case of iid sampling in Theorem 1, the difference is Z t (Θ( t )) . In the following, we bound E Z t (Θ( t )) by using the mixing time analysis in [4]. First, we provide two essential properties of Z t (Θ) to verify the conditions in Lemma 10 in [4].

Lemma 3. Under Assumption 1, we have:

Thus, we have:

We have the following inequality:

Taking the expectation above, and using Hölder's inequality:

By using the information theoretic bound in Lemma 9 in [4], we obtain

under the uniform ergodicity assumption in Assumption 1. Using the last two inequalities in (29), we obtain:

By using the above result, we obtain the ultimate inequality for the Lyapunov drift as follows:

The proof follows from identical steps as Theorem 1.

Proof of Theorem 3. The main idea in the proof is to establish a centering argument for both the bias (due to using clipped stochastic gradients) and the variability (controlled by b t ), and to use martingale concentration arguments based on Freedman's inequality and Azuma-Hoeffding inequality to bound the sample mean for the bias and variability, respectively. This strategy extends the approach in [6] for robust mean estimation to reinforcement learning, which has a dynamic behavior unlike the mean estimation problem. Namely, for any t ∈ { 1 , 2 , . . . , T } , we have:

where the first line follows from Lemma 2, and

Z ( t ) = ∥ g t (Θ( t )) ∥ 2 2 ✶ {∥ g t (Θ( t )) ∥ 2 ≤ b t } -E [ ∥ g t (Θ( t )) ∥ 2 2 ✶ {∥ g t (Θ( t )) ∥ 2 ≤ b t }|F t ] is the variability. By telescoping sum over t = 1 , 2 , . . . , T and some algebraic manipulations, we have:

where we used (20) in the second line. In the following, we will bound the empirical processes 1 T ∑ t =1 Z ( t ) and 1 T ∑ t =1 B ( t ) .

Note that { Z ( t ) : t ∈ N } is a martingale difference sequence (MDS) adapted to the filtration {F t : t ∈ N } . Furthermore, note that

almost surely for any t ≤ T . Thus, ∑ n t =1 V ( t ) ✶ { n ≤ T } forms a martingale with bounded differences, and by using Azuma-Hoeffding inequality [55], we have:

with probability at least 1 -δ/ 2 where the last inequality holds since T > L δ u -2 1 -p We decompose B ( t ) into predictable and non-predictable components as follows:

.

✶ where the martingale difference sequence B 0 ( t ) is defined as follows:

✶ By (25) (with b t replaced by b t ), the first term on the RHS of (34) is bounded by 2 ρuc -p t,δ . Thus, we have:

In order to upper bound 1 T ∑ T t =1 B 0 ( t ) , we use Freedman's inequality for martingales [12]. To use Freedman's inequality, we verify the following conditions.

For any t ≤ T , we have:

almost surely.

The normalized quadratic variation process satisfies:

where the first inequality is due to V ar ( Z ) ≤ E [ Z 2 ] for any random variable Z with a finite variance, the second inequality follows from Cauchy-Schwarz inequality and triangle inequality with max {∥ Θ( t ) ∥ 2 , ∥ Θ ∗ ∥ 2 } ≤ ρ due to projection, the third inequality follows from (20) with b t replaced by b t .

Thus, by Freedman's inequality, we have:

with probability at least 1 -δ/ 2 . Therefore, from (36) and the above inequality, with probability at least 1 -δ/ 2 , we have:

Hence, by substituting (33) and (37) into (32) with union bound, and using the specified step-size together with the facts that L (Θ(1)) ≤ 4 ρ 2 and L (Θ( T +1)) ≥ 0 , we conclude the proof.

A.1 Finite-Time Analysis of Robust TD Learning without Realizability

In the following, we release the realizability assumption, and show that an additional function approximation error appears in the bounds for the general case.

Theorem 5 (Performance of Robust TD learning - without realizability) . Let

Then, under Assumptions 1 and 2a we have the following bounds for Robust TD learning:

For b t = ( ut ) 1 1+ p for any t ∈ Z + and η t = η = 2 ρ (1 -γ ) ( uT ) 1 1+ p , we have:

for any T > 1 .

Proof of Theorem 5. Note that we define

By (19), we have:

✶ To eliminate the realizability assumption (Assumption 3), we make the following decomposition:

Note that the last term corresponds to the approximation error, and we can bound its expectation as:

Since |V ( x ) | ≤ 1 1 -γ and | f Θ( t ) ( x ) | ≤ ρ for all x , and γ ∈ (0 , 1) , we have:

for any t . Using this result, and following identical steps as Theorem 1 with the same step-size yields the result.

B Proofs for Robust Natural Actor-Critic

Proof of Theorem 4. We use the following Lyapunov function for the analysis [1, 57, 34]:

For the Lyapunov drift, at any iteration k , we have:

Since sup s,a ∥ Φ( s, a ) ∥ 2 ≤ 1 , ∇ W log π W ( a | s ) is 1-Lipschitz continuous [1]. Thus, we have:

Since W ( k +1) = W ( k ) + α ¯ Θ k ( T ) , we have:

By performance difference lemma [26], we have:

Using the last two inequalities, we have the drift inequality:

For the log-linear policy parameterization, we have

where the first line follows from the fact that V ar ( X ) ≤ E [ X 2 ] for any random variable X with finite second-order moments, and the last line follows from a change of measure argument. Then, by Theorem 3, we have:

with probability at least 1 -δ/K . Furthermore, we have:

for any k, T by the projection. As such, we can bound the drift inequality as follows:

with probability at least 1 -δ/K . By telescoping sum of the above inequality, using union bound, and noting that π 0 ( a | s ) = 1 | A | for any s, a , which leads to L ( π 1 ) = log | A | , we conclude the proof.

Peer reviews:
comments: SUMMARY:
This paper studies the problem of reinforcement learning under heavy-tailed reward. The paper present two algorithms: robust TD learning and robust NPG that achieves polynomial sample complexity in the generative setting.

STRENGTHS:
The problem of heavy tail reward is important and understudied in RL.

WEAKNESSES:
While the paper is well-written, I'm having trouble appreciating the technique contribution in this paper. Could the authors elaborate what techniques/results in the paper goes beyond a straight-forward combination of (1) robust mean estimation under heavy tail distribution and (2) existing analysis of TD learning and NPG?

QUESTIONS:
See above.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

comments: SUMMARY:
This paper studies the TD and NAC with heavy tailed reward noise. The authors provide both theoretical results and the numerical verification to support their claims.

STRENGTHS:
- This paper is well written and easy to follow. The numerical results firmly support their theoretical claims and show the advantage of the robust algorithm. The theoretical results seem correct to me.
- The authors suggest several practical cases where heavy-tailed reward noises are crucial and the robustness estimation algorithm can help to improve the performance.

WEAKNESSES:
- Comparing with the previous TD learning results [4], it's not clear to me what's the major contribution of this paper. Is it simply combining the result and analysis in [4] and the concentration analysis for heavy-tailed noise (usually the robust-mean results)? 
- The algorithm parameter needs the knowledge of $p$, where I wonder how to estimate it in practice.

QUESTIONS:
- My major concern on this paper is in the weakness section regarding the contribution. It could be better highlighted if the authors can demonstrate some non-trivial processes obtaining this $p / (1 + p) $ style result for the heavy-tailed noise.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work discusses the failure of classical TD learning under heavy-tailed reward. This phenomenon motivates the author to propose a new technique, robust TD learning, with a dynamic gradient clipping mechanism to overcome this issue. The theoretical convergence rates for both in-expectation and high-probability cases are provided to ensure the convergence of this new algorithm under heavy-tailed reward. Some empirical experiments are taken to verify the theoretical result.\n\nSTRENGTHS:\nThe question raised at the beginning of this paper, \"Can temporal difference learning with function approximation be modified to provably achieve global optimality under stochastic rewards with heavy tails?\" is well addressed by proposing a new algorithm and providing the convergence upper bounds. According to the related works, this problem has not been solved before.\n\nRegarding the quality and clarity, the whole paper is complete and well-presented except for some minor issues. \n\nThe motivation example is clear, and it well justifies the issue that this paper aims to solve, which makes this paper well-motivated on the theoretical side. Also, two applications are provided and formulated in the described senario. I believe this paper has made a clear contribution.\n\nWEAKNESSES:\nLack of discussions on the assumptions: I agree that these assumptions are very standard in the convergence analysis of TD-based algorithms, but it is necessary to add more backgrounds. More explicitly, about the definition of heavy-tailed, some well-known heavy-tailed distributions may have infinite expectation (e.g. Cauchy distribution and Pareto distribution Pareto(1, 1)). This make me confused about Assumption 1.2.\n\nPresentation: The parameter $p$ has never been clearly explained until Assumption 1 (on page 4), though it has been used in Main contributions and Abstract.\n\nQUESTIONS:\n1. It seems that some heavy-tailed distributions cannot satisfy both Assumption 1.2 and Assumption 1.3. For example, Cauthy distribution and Pareto(1, 1) distribution, because they have an infinite expectation. Do I misunderstand something here?  \n\n2. Does Theorem 3 directly imply Theorem 2? Why is it necessary to derive two bounds here? If I simply take an expectation on both sides of Theorem 3: (1) with the probability $1-\\delta$, it has the same rate as Theorem 2; (2) with the probability $delta$, bounding everything by a sufficiently large constant. Then tuning the parameter $delta$ could lead to the in-expectation bound.\n\nLIMITATIONS:\nThis is a purely theoretical paper so no limitations and potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the value function estimation problem when the reward of the MDP is heavy tail. A robust version of TD learning is proposed which introduces a clipping step to the algorithm. The authors give both the theoretical analysis and numerical experiments for the proposed algorithm. And they further applied the robust TD learning algorithm to the policy optimization problem, and give a theoretical analysis for it.\n\nSTRENGTHS:\nThe paper studies an important and interesting problem. The presentation is clear and the arguments are well supported by both theoretical and empirical evidence.\n\nWEAKNESSES:\nThe presentation can be improved. In algorithm 1, the variables have a subscript k which is not needed in this algorithm. The proof part skips some details, such as the end of the proof of Theorem 1. Some notations are not defined, such as the total variation and the \\mu-norm. The analysis shares a similar framework with that in [4], which limits the theoretical novelty of the analysis. The result mainly applies to the i.i.d. sampling case. And due to the mixing time assumption, the Markovian sampling case is essentially the same as the i.i.d. case, as one can easily get a near-independent sequence by subsampling. The experiments are conducted on artificial problems, and real-world application related experiments are not presented in the paper. The experiments are only about the value estimation, but do not solve the downstream policy optimization task.\n\nQUESTIONS:\nIn the experiment results, the curve for TD learning has a rapid increase in error at the initial phase. This is an interesting phenomenon given that the curve represents the averaged performance. Could the authors provide any comments on it?\n\nLIMITATIONS:\nThere is not a section about limitation. Since the work is mainly focused on the theoretical analysis, it shouldn't have a direct negative social impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose and study variants of TD(0) and Natural Actor-Critic that provably converge under (on-policy in the case of TD(0)) linear function approximation when the rewards may not have finite second moment or variance, but instead a finite (1+p) moment for $p \\in (0,1]$; the authors use this as the definition of \"heavy-tailed\" rewards.\n\nThe techniques used include:\n- Clipping the update direction using a decaying threshold\n- Using Holder's inequality + the decaying threshold to control the bias introduced by clipping the update direction, where in the normal analysis, one would upper-bound the expected noise by zero.\n\nThe rest of the techniques are otherwise similar to well-known previous work, in particular by Bhandari et al. 2018 (at least in essence, e.g., there is no difference in doing a Lyapunov analysis or using the Online-Gradient-Descent update bound for $g_t$ and $\\Theta_t$). Additional assumptions used include:\n1. realizability, and \n2. knowledge of a projection radius containing the optimum.\n\nThe authors complement the analysis with numerical results.\n\nSTRENGTHS:\nThe paper is very well-written. The logic follows clearly and easily. Examples are given when appropriate. The assumptions are laid out clearly. It's a nice, easy-to-follow read for a theoretician.\n\nThe paper contains the typical interesting (even if standard) results: the robust step-size case, the $1/t$ step-size case with knowledge of $\\lambda_\\textnormal{min}$, the simulator/i.i.d. sample case, and the Markov sample case under uniform exponential mixing.\n\nWhen $p=1$, the results naturally fall back to the $O(1/\\sqrt{T})$ and $O(1/T)$ rates of [4].\n\nConsidering NAC strengthens the paper, covering both policy-evaluation and control.\n\nWEAKNESSES:\nFrom a technical point of view, the assumptions are rather strong. Specifically,\n\n- Realizability is assumed, which is not really required for this MRP setting.\n- In addition, it is assumed that we have knowledge of an upper-bound $\\rho$ on the norm of the optimal $\\Theta^*$. This is not required by e.g., Bhandari et. al. 2018.\n- \"Heavy-tail\" is defined as satisfying a relaxed version of the usual (non-heavy-tailed) assumption: degree 2 is replaced by degree $1+p$ for $p \\in (0,1]$. In addition, it is assumed that this value $p$ is known. Does this definition suite the examples of heavy-tailed rewards given in the introduction?\n\nIn terms of related work, I am not entirely convinced that (as written on line 112), the cases of SGD and RL really differ here, at least as far as the main technique for introducing and handling bias (diminishing threshold + Holders on top of the noise term) is concerned. \n\nFinally, a recent (perhaps concurrent?) work (https://arxiv.org/pdf/2205.08464.pdf), considers defining and using robust losses instead of gradient-clipping. The merits and shortcomings of the two different approaches are worth discussing. The references therein also point to previous work (without theoretical guarantees) that use clipping of the rewards, the errors, or the update directions.\n\nQUESTIONS:\nPlease comment on the weaknesses mentioned above. In particular:\n\n- The strength of the heavy-tailed assumption.\n- How hard it is to relax the projection onto ball of radius $\\rho$ for each of the iid and Markov cases?\n- Do you  need the realizability assumption if you instead bound the loss defined by $\\phi^T \\theta*$?\n\nLIMITATIONS:\nThe technical limitations were already discussed in the \"Weaknesses\" and \"Questions\" section. This is a theoretical work so wider societal impacts are not apparent and addressable at this level.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses robust Temporal Difference (TD) methods for RL with heavy-tailed rewards. Specifically, it proposes robust variants of TD(0) and Natural Actor-Critic (NAC) for policy evaluation and RL control, respectively, and then derives their average iterate's convergence rates.\n\nSTRENGTHS:\n1. Convergence rates for TD(0) and NAC with function approximation are obtained in the non-trivial heavy-tailed reward setup.\n\n2. The above results have been obtained both in the generative as well as the Markovian noise model.\n\nWEAKNESSES:\n1. Assumption 3 is extremely strong. While the authors mention that there would be an additional non-vanishing approximation error when this assumption is not the case, it is unclear what the point of convergence would be in that case. \n\n2. No reasons/insights have been provided on why clipping helps deal with the heavy-tailed nature. \n\n3. In the related works section, it is mentioned that existing theoretical analyses for TD learning only provide guarantees in the expected error rather than in high probability. This is incorrect; e.g., see\n\na.) Patil, G., Prashanth, L.A., Nagaraj, D. and Precup, D., 2023, April. Finite time analysis of temporal difference learning with linear function approximation: Tail averaging and regularisation. In International Conference on Artificial Intelligence and Statistics (pp. 5438-5448). PMLR.\n\nb.) Dalal, G., Szörényi, B., Thoppe, G. and Mannor, S., 2018, April. Finite sample analyses for TD (0) with function approximation. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 32, No. 1).\n\nQUESTIONS:\n1. Why should robust NAC converge to the optimal policy (in the value function sense)? I am not well-versed in the actor-critic literature, but is global convergence straightforward even in the function approximation setting? In the case of SARSA with linear-function approximation (Melo-Meyn-Ribeiro, 2008), one required that the policy improvement operator be Lipshitz with a `small' Lipschitz constant to ensure a unique fixed point of the Bellman operator. For example,  this holds for soft-max type policies with a sufficiently small inverse temperature parameter. In this work though, no such assumption is made apart from full realizability of the function approximation space. \n\nMelo, F.S., Meyn, S.P. and Ribeiro, M.I., 2008, July. An analysis of reinforcement learning with function approximation. In Proceedings of the 25th international conference on Machine learning (pp. 664-671).\n\n2. Is the obtained convergence rate of $\\tilde{O}(T^{-p})$ tight for $p < 1?$\n\nMinor:\n3. What is k in the subscript of $\\theta$ in Algorithm 1?\n\n4. In line 199, should $\\bar{\\chi}(t)$ in the definition of $Z(t)$ be $\\chi(t)?$\n\n4. In the displayed equation above line 482, what is $V(t)?$\n\nLIMITATIONS:\nI could not find any limitations section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}