SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Semih Cayci
cayci@mathc.rwth-aachen.de
Atilla Eryilmaz
Columbus, OH 43210
eryilmaz.2@osu.edu
In a broad class of reinforcement learning applications, stochastic rewards have heavy-tailed distributions, which lead to infinite second-order moments for stochastic (semi)gradients in policy evaluation and direct policy optimization. In such instances, the existing RL methods may fail miserably due to frequent statistical outliers. In this work, we establish that temporal difference (TD) learning with a dynamic gradient clipping mechanism, and correspondingly operated natural actor-critic (NAC), can be provably robustified against heavy-tailed reward distributions. It is shown in the framework of linear function approximation that a favorable tradeoff between bias and variability of the stochastic gradients can be achieved with this dynamic gradient clipping mechanism. In particular, we prove that robust versions of TD learning achieve sample complexities of order  $\mathcal{O}(\varepsilon^{-\frac{1}{p}})$  and  $\mathcal{O}(\varepsilon^{-1 - \frac{1}{p}})$  with and without the full-rank assumption on the feature matrix, respectively, under heavy-tailed rewards with finite moments of order  $(1 + p)$  for some  $p\in (0,1]$ , both in expectation and with high probability. We show that a robust variant of NAC based on Robust TD learning achieves  $\tilde{\mathcal{O}} (\varepsilon^{-4 - \frac{2}{p}})$  sample complexity. We corroborate our theoretical results with numerical experiments.
In this paper, we develop a framework for robust reinforcement learning in the presence of rewards with heavy-tailed distributions. Heavy-tailed phenomena, stemming from frequently observed statistical outliers, have been ubiquitous in decision-making applications under uncertainty. To name a few examples, waiting times in wireless communication networks [44, 24, 58], completion times of SAT solvers [14], numerous payoff quantities (e.g., stock prices, consumer signals) in economics and finance [22, 35, 38, 37] exhibit heavy-tailed behavior. An important characteristic of heavy-tailed random variables is the infinite order of higher moments, which stems from the frequently occurring outliers.
In reinforcement learning (RL), the goal is to maximize expected total reward in a Markov decision process (MDP) by continual interactions with the unknown and dynamic environment. Among policy optimization methods, Natural actor-critic (NAC) method and its variants [51, 30, 46, 17, 27, 5, 29] have become particularly prevalent due to their desirable stability and versatility characteristics, emanating from the use of temporal difference (TD) learning as the critic for the policy evaluation component of the NAC operation. The existing theoretical analyses for temporal difference learning [4, 54] and natural policy gradient/actor-critic methods [62, 1, 57] assume that the stochastic gradients have finite second-order moments, or even they are bounded. In particular, it is unknown whether natural actor-critic with function approximation is robust for stochastic rewards of heavy-tailed distributions with potentially infinite second-order moments. Furthermore, in practice, these methods
are prone to non-vanishing and even increasing error under heavy-tailed reward distributions (see Example 1). This motivates us for the following fundamental question in this paper:
Can temporal difference learning with function approximation be modified to provably achieve global optimality under stochastic rewards with heavy tails?
We provide an affirmative answer to the above question by proposing a simple modification to the TD learning algorithm, which yields robustness against heavy tails. In particular, we show that incorporating a dynamic gradient clipping mechanism with a carefully-chosen sequence of clipping radii can provably robustify TD learning and NAC with linear function approximation, leading to global near-optimality even under stochastic rewards of infinite variance.
Example 1 (Failure of TD learning under heavy-tailed reward). In this example, we consider a randomly-generated discounted-reward Markov reward process $^{1}$ $(X_{t},R_{t})_{t}$  on a state space  $\mathbb{X}$  with  $|\mathbb{X}| = 64$  states, with the discount factor  $\gamma = 0.9$  and the reward  $R_{t}(X_{t}) = r(X_{t}) + N_{t} - \mathbb{E}[N_{t}]$  with  $N_{t}\stackrel {iid}{\sim}\mathrm{Pareto}(1,1.4)$  for any  $t$ . In order to predict the value function, we use (projected) TD learning (see [4]) with linear function approximation based on Gaussian features of dimension  $d = 4$  and projection radius  $\rho = 30$ . The performance results are shown in Figure 1. Since  $R_{t}$  is heavy-tailed
(a) Errors for TD Learning
Figure 1: Non-convergent behavior of TD learning under heavy-tailed noise with tail index 1.4. Each faded green line is the MSE for an individual trial, and the solid lines with markers indicate the average mean squared error for TD learning and Robust TD learning.
(b) Errors for Robust TD Learning
(c) Comparison of the expected errors
with infinite variance, the existing convergence results for traditional TD learning, which assume that  $R_{t}$  has finite variance, do not hold. Furthermore, Figure 1 reveals that TD learning is prone to non-varying and even increasing error in practice despite the projection step, iterate averaging and small learning rate, due to the statistical outliers that cause extremely large error often as indicated by a non-negligible fraction of green lines in Figure 1a. On the other hand, with the same learning rate, projection radius and state-reward realizations, our robust variant of TD learning provides resilience against outliers (see Figure 1b), and leads to convergence in the expected behavior as in Figure 1c.
Stochastic rewards with heavy-tailed distributions appear in many important applications. Below, we briefly provide two motivating applications that necessitate robust RL methods to handle heavy tails.
Application (1): Algorithm portfolios. In solving complicated problems such as Boolean satisfiability (SAT) and complete search problems, which appear in numerous applications [19, 43], multiple algorithmic solutions with different characteristics are available. The algorithm selection problem is concerned about the minimization of total execution times to solve these problems [32, 49, 31], where different data distributions and machine characteristics, caused by recursive algorithms, are modeled as states, algorithm choices are modeled as actions, and the execution time of a selected algorithm is modeled as the cost (i.e., negative reward). It is well-known that the execution times, i.e., rewards, in the algorithm selection problem have heavy-tailed distributions with infinite-variance (e.g., Pareto(1, 1 + p) with  $0 < p < 1$  as in [13]) similar to the case in Example 1 [13, 15, 49]. Thus, algorithm selection problem requires robust techniques that we consider here.
Application (2): Scheduling for wireless networks. The scheduling problem considers matching the users with random service demands to fading wireless channels (e.g., Gilbert-Elliot model) with stochastic transmission times so as to minimize the expected delay. A widely-adopted approach to
<sup>1</sup>The details of the setup, along with other numerical examples can be found in Section 4.
study the scheduling problem is to use MDPs (see, e.g., [40, 21, 10, 2]). It has been observed that the transmission times follow heavy-tailed distributions of infinite variance, due to various factors including the MAC protocol used, packet size, and channel fading [18, 58, 24, 23]. As such, solving this by using RL approach necessitates robust methods to handle heavy-tailed execution times.
Main contributions. Our main contributions in this paper contain the following.
- Robust TD learning with dynamic clipping for heavy-tailed rewards. We propose Robust TD learning with a dynamic gradient clipping mechanism, and prove that this TD learning variant with linear function approximation can achieve arbitrarily small estimation error that vanishes at rates  $\mathcal{O}(T^{-\frac{p}{1 + p}})$  and  $\tilde{\mathcal{O}} (T^{-p})$  without and with full-rank assumption on the feature matrix, respectively, even for heavy-tailed rewards with moments of order  $1 + p$  for  $p\in (0,1]$ . Our proof techniques make use of Lyapunov analysis coupled with martingale techniques for robust statistical estimation in dynamical systems, and can be of independent interest in the analysis of first-order methods.
- Robust NAC under heavy-tailed rewards. Based on Robust TD learning and the compatible function approximation result in [27], we propose a robust NAC variant, and show that  $\mathcal{O}\left(\varepsilon^{-4 - \frac{2}{p}}\right)$  samples suffice to achieve  $\varepsilon > 0$  error under standard concentrability coefficient assumptions.
- High-probability error bounds. We provide high-probability (sub-Gaussian) error bounds for the robust NAC and TD learning methods in addition to the traditional expectation bounds.
From a statistical viewpoint, our analysis in this work indicates a favorable bias-variance tradeoff: by introducing a vanishing bias to the semi-gradient via particular choices of dynamic gradient clipping, one can achieve robustness by eliminating the destructive impacts of statistical outliers even if the semi-gradient has infinite variance, leading to near optimality.
Temporal difference learning. Temporal difference (TD) learning was proposed in [50], and has been the prominent policy evaluation method. The existing theoretical analyses of TD learning consider MDPs with bounded rewards [4, 7], or rewards with finite variance [54], while we consider heavy-tailed rewards. Our analysis utilizes the Lyapunov techniques in [4].
Policy gradient methods. Policy gradient (PG), and its variant natural policy gradient (NPG) have attracted significant attention in RL [59, 52, 27]. Recent theoretical works investigate the local and global convergence of these methods in the exact case, or with stochastic and bounded rewards [1, 57, 34, 39, 61]. As such, heavy-tailed rewards have not been considered in these works.
Bandits with heavy-tailed rewards. Stochastic bandit variants with heavy-tailed payoffs were studied in multiple works [6, 48, 33, 8]. The stochastic bandit setting can be interpreted as a very simple single-state (i.e., stateless or memoryless) model-based and tabular RL problem. The model we consider in this paper is a model-free RL setting on an MDP with a large state space, which is considerably more complicated than the bandit setting.
Stochastic gradient descent with heavy-tailed noise. There has been an increasing interest in the analysis of SGD with heavy-tailed gradient noise recently [56, 11, 16], following the seminal work of [45]. In our work, we consider the RL problem, which has significantly different dynamics than stochastic convex optimization.
Robust mean and covariance estimation. In basic statistical problems of mean and covariance estimation [42, 41, 36] and regression [20], the traditional methods do not yield the optimal convergence rates for heavy-tailed random variables, which led to the development of robust mean and covariance estimation techniques (for reference, see [36, 28]). Our paper utilizes tools from robust mean estimation literature (particularly, truncated mean estimator analysis in [6]), but considers the more complicated problem of TD learning and policy optimization in a dynamic environment rather than a static mean or covariance estimation problem with iid observations.
For a symmetric matrix  $A \in \mathbb{R}^{d \times d}$ ,  $\lambda_{\min}(A)$  denotes its minimum eigenvalue.  $B_2(x, \rho) = \{y \in \mathbb{R}^d : \|x - y\|_2 \leq \rho\}$  and  $\Pi_{\mathcal{C}}\{x\} = \arg \min_{y \in \mathcal{C}} \|x - y\|_2^2$  for any convex  $\mathcal{C} \subset \mathbb{R}^d$ .
2 Robust TD Learning for Value Prediction under Heavy Tails
First, we consider the problem of predicting the value function for a given discounted-reward Markov reward process with heavy-tailed rewards.
2.1 Value Prediction Problem
For a finite but arbitrarily large state space  $\mathbb{X}$ , let  $(X_{t})_{t\in \mathbb{N}}$  be an  $\mathbb{X}$ -valued Markov chain with the transition kernel  $\mathcal{P}:\mathbb{X}\times \mathbb{X}\to [0,1]$ . We consider a Markov reward process  $(X_{t},R_{t})_{t\in \mathbb{N}}$  such that at state  $X_{t}$ , a stochastic reward  $R_{t} = R_{t}(X_{t})$  is obtained for all  $t\geq 0$ . For a discount factor  $\gamma \in [0,1)$ , the value function for the MRP  $(X_{t},R_{t})_{t\in \mathbb{N}}$  is the following:
$$
\mathcal {V} (x) = \mathbb {E} \left[ \sum_ {t = 1} ^ {\infty} \gamma^ {t - 1} R _ {t} \left(X _ {t}\right) \mid X _ {1} = x \right], x \in \mathbb {X}. \tag {1}
$$
Objective. The goal is to learn  $\mathcal{V}$  without knowing the transition kernel  $\mathcal{P}$  by using samples from the system. In particular, for a parameterized class of functions  $\{f_{\Theta}:\mathbb{X}\to \mathbb{R}:\Theta \in \mathbb{R}^d\}$ , the goal is to solve the following stochastic optimization problem with mean squared error:
$$
\min  _ {\Theta \in \mathbb {R} ^ {d}} \mathbb {E} _ {x \sim \mu} | f _ {\Theta} (x) - \mathcal {V} (x) | ^ {2}. \tag {2}
$$
In order to solve (2) under Assumption 1, next we propose a robust variant of temporal difference (TD) learning with linear function approximation [50, 54].
2.2 Robust TD Learning Algorithm
For a given set of feature vectors  $\{\Phi(x) \in \mathbb{R}^d : x \in \mathbb{X}\}$  with  $\sup_{x \in \mathbb{X}} \| \Phi(x) \|_2 \leq 1$ , we use  $f_\Theta(\cdot) = \langle \Theta, \Phi(\cdot) \rangle$  as the approximation architecture. For a given dataset  $\mathcal{D} = \{(X_t, R_t, X_t') \in \mathbb{X} \times \mathbb{R} \times \mathbb{X} : t \in \mathbb{N}\}$  with  $X_t' \sim \mathcal{P}(X_t, \cdot)$ , let the stochastic semi-gradient at  $\Theta \in \mathbb{R}^d$  be defined as
$$
g _ {t} (\Theta) = \left(R _ {t} \left(X _ {t}\right) + \gamma f _ {\Theta} \left(X _ {t} ^ {\prime}\right) - f _ {\Theta} \left(X _ {t}\right)\right) \nabla_ {\Theta} f _ {\Theta} \left(X _ {t}\right).
$$
Robust TD learning is summarized in Algorithm 1.
Algorithm 1: Robust TD learning
Inputs: number of steps  $T \geq 1$ , clipping radii  $(b_{t})_{t \in [T]}$ , projection radius  $\rho > 0$ , step-size  $\eta > 0$
Set  $\Theta (1)\in B_2(0,\rho)$
$$
\Theta (t + 1) = \Pi_ {B _ {2} (0, \rho)} \left\{\Theta (t) + \eta_ {t} \cdot g _ {t} (\Theta (t)) \cdot \mathbb {1} \left\{\| g _ {t} (\Theta (t)) \| _ {2} \leq b _ {t} \right\} \right\}
$$
Output:  $f_{\bar{\Theta}(T)}(\cdot) = \langle \bar{\Theta}(T), \Phi(\cdot) \rangle$  where  $\bar{\Theta}(T) = \frac{1}{T} \sum_{t=1}^{T} \Theta(t)$ .
In the following, we will establish finite-time bounds for Robust TD learning by specifying the sequence of dynamic gradient clipping radii  $(b_{t})_{t\geq 1}$ , projection radius  $\rho$  and step-size  $\eta$ .
2.3 Finite-Time Bounds for Robust TD Learning
We make the following assumptions on the Markov reward process.
Assumption 1. The stochastic process  $(X_{t},R_{t})_{t\in \mathbb{N}}$  satisfies the following:
1. Ergodicity:  $(X_{t})_{t\in \mathbb{N}}$  is an irreducible and aperiodic Markov chain with stationary distribution  $\mu = \mu \mathcal{P}$ . Also, we assume that there are constants  $m > 0, \zeta \in (0,1)$  such that
$$
\max  _ {x \in \mathbb {X}} \| \mathcal {P} ^ {t} (x, \cdot) - \mu \| _ {\mathsf {T V}} \leq m \zeta^ {t}, \forall t \in \mathbb {Z} _ {+}. \tag {3}
$$
2. Heavy-tailed reward: For some  $p \in (0,1]$  and constant  $u_0 \in (0,\infty)$ ,
$$
\mathbb {E} \left[ \left| R _ {t} \left(X _ {t}\right) \right| ^ {1 + p} \mid X _ {t} \right] \leq u _ {0} <   \infty , a. s., \forall t \in \mathbb {N}. \tag {4}
$$
3. Mean reward: For any  $t\in \mathbb{N}$ $\mathbb{E}[R_t(X_t)|X_t] = r(X_t)\in [-1,1]$  a.s.
We note that the uniform ergodicity and bounded mean reward assumptions are standard in TD learning literature [3, 54, 4].
Assumption 2 (Sampling). We consider two types of sampling strategies in this work:
2a. IID sampling:  $X_{t}\stackrel {iid}{\sim}\mu$  and  $X_{t}^{\prime}\sim \mathcal{P}(X_{t},\cdot)$  for all  $t\geq 1$
2b. Markovian sampling:  $X_{1}\sim \mu$  and  $X_{t}^{\prime} = X_{t + 1}\sim \mathcal{P}(X_{t},\cdot)$  for all  $t\geq 1$
Assumption 3 (Realizability). There exists  $\Theta^{\star} \in B_{2}(0, \rho)$  such that  $\mathcal{V}(\cdot) = \langle \Theta^{\star}, \Phi(\cdot) \rangle$ .
Remark 1. We note that Assumption 3 holds directly in interesting realizable problem classes, e.g., linear MDPs [25], and allows us to obtain results on the statistical error performance of our design. In cases when it does not hold, our results will continue to hold with an additional function approximation error proportional to  $\inf_{\Theta \in B_2(0,\rho)}\sqrt{\mathbb{E}[|\mathcal{V}(x) - \langle\Theta,\Phi(x)\rangle|^2]}$ , which is unavoidable due to the limitation of the linear function approximation architecture.
The following lemma is important in bounding the moments of the gradient norm under Robust TD learning in terms of the projection radius  $\rho > 0$  and the upper bound  $u_0$  on  $\mathbb{E}\left[|R_t(X_t)|^{1 + p}|X_t\right]$ .
Lemma 1 (Tail bounds for  $\| g_t(\Theta(t))\|_2$ ). Let  $\mathcal{F}_t^+ = \sigma(\Theta(1), \Theta(2), \ldots, \Theta(t), X_t)$  for  $t \in \mathbb{Z}_+$ . Then, under Assumption 1, we have:
$$
\mathbb {E} [ \| g _ {t} (\Theta (t)) \| _ {2} ^ {1 + p} \mid \mathcal {F} _ {t} ^ {+} ] \leq u <   \infty , a. s., \tag {5}
$$
for any  $t\in \mathbb{Z}_+$ , where  $u = \min \{(u_0^{\frac{1}{1 + p}} + 2\rho)^{1 + p},u_0 + 2^{2p + 3}\rho^{1 + p}\}$ .
Proof. Note that we have  $\mathbb{E}[\| g_t(\Theta(t))\|_2^{1+p}|\mathcal{F}_t^+] \leq \mathbb{E}[|R_t(X_t) + \gamma f_{\Theta(t)}(X_t') - f_{\Theta(t)}(X_t)|^{1+p}|\mathcal{F}_t^+]$  since  $\sup_{x \in \mathbb{X}} \| \Phi(x)\|_2 \leq 1$ . The upper bounds then follow by applying Minkowski's inequality and the triangle inequality for  $L^p$  spaces, respectively, to this inequality.
This lemma will be useful in the analysis of both the expected (Theorems 1-2), and the high-probability bounds (Theorem 3) on the performance of Robust TD learning.
Next, we provide the main theoretical results in this paper: finite-time bounds for Robust TD learning. The proofs are mainly deferred to the appendix, while we provide a proof sketch for Theorem 3. In the following, we provide convergence bounds for the expected mean squared error under Robust TD learning with various choices of  $b_{t}$ .
Theorem 1 (Expected error under Robust TD learning - iid sampling). Under Assumptions 1, 2a, 3, we have the following bounds for Robust TD learning:
a) For  $b_{t} = (ut)^{\frac{1}{1 + p}}$  for any  $t \in \mathbb{Z}_{+}$  and  $\eta_t = \eta = \frac{2\rho(1 - \gamma)}{(uT)^{\frac{1}{1 + p}}}$ , we have:
$$
\underset { \begin{array}{c} \Theta (1), \Theta (2), \dots , \Theta (T) \\ x \sim \mu \end{array} } {\mathbb {E}} \left[ \left(\mathcal {V} (x) - \langle \bar {\Theta} (T), \Phi (x) \rangle\right) ^ {2} \right] \leq \frac {6 \rho u ^ {\frac {1}{1 + p}}}{(1 - \gamma) T ^ {\frac {p}{1 + p}}}, \forall T > 1. \tag {6}
$$
b) Let  $\Lambda = \sum_{x\in \mathbb{X}}\mu (x)\Phi (x)\Phi^{\top}(x)$ , and  $\mathfrak{C}_p(u,\lambda_{\min},\gamma ,\rho) = \frac{u}{1 - \gamma}\Big(4\rho +\frac{1}{(1 - \gamma)\lambda_{\min}}\Big)$ . If  $\lambda_{\mathrm{min}}(\Lambda) = \lambda_{\mathrm{min}} > 0$ , then with the diminishing step-size  $\eta_t = \frac{1}{(1 - \gamma)t\lambda_{\mathrm{min}}}$  and  $b_{t} = t$  for  $t\in \mathbb{Z}_{+}$ , for the average iterate  $\bar{\Theta} (T)$ , we have:
$$
\underset {\Theta (1), \underset {x \sim \mu} {\Theta (2)}, \dots , \Theta (T)} {\mathbb {E}} \left(\mathcal {V} (x) - \langle \bar {\Theta} (T), \Phi (x) \rangle\right) ^ {2} \leq \mathfrak {C} _ {p} (u, \lambda_ {\min }, \gamma , \rho) \left[ \frac {\mathbf {1} _ {p , 1} T ^ {- p}}{1 - p} + \frac {(1 - \mathbf {1} _ {p , 1}) \log (e T)}{T} \right], \tag {7}
$$
and for the last iterate  $\Theta (T + 1)$  , we have:
$$
\underset {\Theta (1), \dots , \Theta (T)} {\mathbb {E}} \max  _ {x \in \mathbb {X}} | \mathcal {V} (x) - \left\langle \Phi (x), \Theta (T + 1) \right\rangle | ^ {2} \leq \frac {\mathfrak {C} _ {p} (u , \lambda_ {\min } , \gamma , \rho)}{\lambda_ {\min }} \left[ \frac {\log (e T) (1 - \mathbf {1} _ {p , 1})}{T} + \frac {\mathbf {1} _ {p , 1} T ^ {- p}}{1 - p} \right], \tag {8}
$$
for any  $T > 1$ , where  $\mathbf{1}_{x,y} = 1$  if  $x \neq y$  and 0 otherwise.
2The upper bound is  $\mathfrak{C}_p(u,\lambda_{\min},\gamma ,\rho)\frac{1}{T}\sum_{t = 1}^{T}t^{-p} = \widetilde{\mathcal{O}} (T^{-p})$  , which is further upper bounded as (6) and (7) by using integral bounds.
Remark 2. The convergence rates in Theorem 1 are  $\mathcal{O}(T^{-\frac{p}{1 + p}})$  and  $\tilde{\mathcal{O}} (T^{-p})$  without and with the full-rank assumption  $\lambda_{\mathrm{min}} > 0$ , respectively. For  $p = 1$ , the convergence rates stated in Theorem 1 both match the existing results for TD learning with bounded rewards [4], up to a larger scaling factor of raw second-order moment rather than variance, due to clipping centered around 0.
Remark 3 (Finite-time bounds in the unrealizable case). We note that our results hold without Assumption 3 as well. In this general case, there will be an additional function approximation error proportional to  $\inf_{\Theta \in B_2(0,\rho)}\sqrt{\mathbb{E}[|\mathcal{V}(x) - \langle\Theta,\Phi(x)\rangle|^2]}$  (see Remark 1). One would use a richer function approximation scheme (e.g., larger projection radius  $\rho$  or dimension  $d$ ) to reduce this function approximation error, which will lead to an increase in the statistical error as we characterize in our bounds. For the extension of our analysis to the general case without Assumption 3, please see Appendix A.1.
In the following, we provide convergence bounds for Robust TD learning under Markovian sampling.
Theorem 2 (Expected error under Robust TD learning - Markovian sampling). Under Assumptions 1,2b and 3, let  $T > 1$ ,  $\rho > 0$  be given, and define the mixing time  $\tau = \min \{t \in \mathbb{Z}_+ : m\zeta^t \leq \sqrt{2}\rho(uT)^{-\frac{1}{1 + p}}\}$ . Then, with  $\eta_t = \eta = \sqrt{2}\rho(uT)^{-\frac{1}{1 + p}}$ , Robust TD learning yields the following:
$$
\underset {\Theta (1), \underset {x} {\Theta (2)}, \dots , \Theta (T)} {\mathbb {E}} \left(\mathcal {V} (x) - \langle \bar {\Theta} (T), \Phi (x) \rangle\right) ^ {2} \leq \frac {7 \rho u ^ {\frac {1}{1 + p}}}{(1 - \gamma) T ^ {\frac {p}{1 + p}}} + \frac {2 \sqrt {2} \rho (1 + 2 \rho) (4 \rho + \tau (1 + 6 \rho))}{(1 - \gamma) T ^ {\frac {1}{1 + p}}}. \tag {9}
$$
The proof of Theorem 2 is based on a similar Lyapunov technique as Theorem 1 in conjunction with the mixing time analysis in [4] for Markovian sampling, and can be found in Appendix A.
The bounds in Theorem 1 involve expectation over the parameters  $\Theta(t), t \in [T]$ . In the following, we provide a high-probability error bound on the mean squared error under Robust TD learning.
Theorem 3 (High-probability bound for Robust TD learning). For any  $\delta \in (0,1)$ , let  $L_{\delta} = \log(4 / \delta)$ .
Under Assumptions 1, 2a, 3, with step-size  $\eta = \frac{\sqrt{2}(1 - \gamma)\rho L_{\delta}^{\frac{1 - p}{2(1 + p)}}}{(uT)^{\frac{1}{1 + p}}}$  and clipping radius  $b_{t} = \left(\frac{ut}{L_{\delta}}\right)^{\frac{1}{1 + p}}$
$$
\sum_ {x \in \mathbb {X}} \mu (x) \left(\mathcal {V} (x) - \langle \bar {\Theta} (T), \Phi (x) \rangle\right) ^ {2} \leq \frac {\rho u ^ {\frac {1}{1 + p}}}{(1 - \gamma) T ^ {\frac {p}{1 + p}}} \left(3 L _ {\delta} ^ {- \frac {1 - p}{2 (1 + p)}} + 7 L _ {\delta} ^ {\frac {p}{1 + p}}\right), \tag {10}
$$
In the following, we give a proof sketch for Theorem 3. The full proof can be found in Appendix A.
Proof sketch. Let  $\mathcal{L}(\Theta) = \| \Theta -\Theta^{\star}\| _2^2$  be the Lyapunov function, and  $\chi_t = 1 - \bar{\chi}_t = \mathbb{1}\{\| g_t\| _2\leq b_t\}$ . Then, the Lyapunov drift can be decomposed as follows: