SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {L} (\Theta (t + 1)) - \mathcal {L} (\Theta (t)) \leq 2 \eta \mathbb {E} _ {t} \left[ g _ {t} ^ {\top} (\Theta (t) - \Theta^ {\star}) \right] + \eta^ {2} \mathbb {E} _ {t} \left[ \| g _ {t} \| _ {2} ^ {2} \chi_ {t} \right] + 2 \eta B (t) + \eta^ {2} Z (t), \tag {11}
$$
$$
B (t) = g _ {t} ^ {\top} (\Theta (t) - \Theta^ {\star}) \chi_ {t} - \mathbb {E} _ {t} [ g _ {t} ^ {\top} (\Theta (t) - \Theta^ {\star}) \chi_ {t} ] - \mathbb {E} _ {t} [ g _ {t} ^ {\top} (\Theta (t) - \Theta^ {\star}) \bar {\chi} _ {t} ],
$$
is the bias in the stochastic semi-gradient, and
$$
Z (t) = \left\| g _ {t} \right\| _ {2} ^ {2} \chi_ {t} - \mathbb {E} _ {t} \left[ \left\| g _ {t} \right\| _ {2} ^ {2} \chi_ {t} \right].
$$
We can decompose  $B(t)$  further into a martingale difference sequence
$$
B _ {0} (t) = g _ {t} ^ {\top} (\Theta (t) - \Theta^ {\star}) \chi_ {t} - \mathbb {E} _ {t} [ g _ {t} ^ {\top} (\Theta (t) - \Theta^ {\star}) \chi_ {t} ],
$$
and a bias term
$$
B _ {\perp} (t) = - \mathbb {E} _ {t} \left[ g _ {t} ^ {\top} (\Theta (t) - \Theta^ {\star}) \bar {\chi} _ {t} \right].
$$
By Freedman's inequality for martingales [12, 53], we have  $\frac{1}{T} \sum_{t=1}^{T} B_0(t) \leq \frac{7\rho u^{\frac{1}{1+p}} L_{\delta}^{\frac{p}{1+p}}}{T^{\frac{p}{1+p}}}$ , and by Azuma inequality, we have  $\frac{1}{T} \sum_{t=1}^{t} Z(t) \leq \frac{u^{\frac{2}{1+p}} T^{\frac{1-p}{1+p}}}{L_{\delta}^{\frac{1-p}{1+p}}}$ , each holding with probability at least  $1 - \delta / 2$ .
By Hölder's inequality and Lemma 1, we can bound  $B_{\perp}(t) \leq ub_t^{-p}$  and  $\mathbb{E}_t[\| g_t\|_2^2\bar{\chi}_t] \leq ub_t^{1 - p}$ , both with probability 1. Finally, by Lemma 2 in [54], we have the negative drift term
$$
\mathbb {E} _ {t} \left[ g _ {t} ^ {\top} (\Theta (t) - \Theta^ {\star}) \right] \leq - (1 - \gamma) \sum_ {x} \mu (x) \left(f _ {\Theta (t)} (x) - \mathcal {V} (x)\right) ^ {2}.
$$
By telescoping sum of (11) and rearranging the terms, we have:
$$
\frac {1}{T} \sum_ {t = 1} ^ {T} \| f _ {\Theta (t)} - \mathcal {V} \| _ {\mu} ^ {2} \leq \frac {\mathcal {L} (\Theta (1))}{2 \eta (1 - \gamma) T} + \frac {1}{(1 - \gamma) T} \sum_ {t = 1} ^ {T} B (t) + \frac {\eta}{2 (1 - \gamma) T} \sum_ {t = 1} ^ {T} \left(Z (t) + \mathbb {E} _ {t} [ \| g _ {t} \| _ {2} ^ {2} \bar {\chi} _ {t} ]\right).
$$
The proof is concluded by substituting the above high probability bounds on the sample means of  $B(t)$ ,  $Z(t)$  and  $\mathbb{E}_t[\| g_t\| _2^2\bar{\chi}_t]$  (via union bound and integral upper bounds), and using Jensen's inequality on the left side of the above inequality.
Most notably, this important theorem establishes that, by appropriately controlling the bias term of dynamic gradient clipping to yield a vanishing sample mean with high probability as the number of iterations increases, one can limit the variance of the semi-gradient, thereby resulting in the provided global near-optimality guarantee.
3 Robust Natural Actor-Critic for Policy Optimization under Heavy Tails
In this section, we will study a two-timescale robust natural actor-critic algorithm (Robust NAC, in short) based on Robust TD learning, and provide finite-time bounds.
3.1 Policy Optimization Problem
We consider a discounted-reward Markov decision process (MDP) with a finite but arbitrarily large state space  $\mathbb{S}$ , finite action space  $\mathbb{A}$ , transition kernel  $\mathcal{P}$  and discount factor  $\gamma \in (0,1)$ . The controlled Markov chain  $\{(S_t,A_t) \in \mathbb{S} \times \mathbb{A} : t \in \mathbb{N}\}$  has the probability transition dynamics  $\mathbb{P}(S_{t+1} \in s|S_1^t, A_1^t) = \mathcal{P}_{A_t}(S_t,s)$ , for any  $s \in \mathbb{S}$ . Taking the action  $A_t \in \mathbb{A}$  at state  $S_t \in \mathbb{S}$  yields a random reward of  $R_t(S_t,A_t)$  at any  $t \in \mathbb{Z}_+$ . For a given stationary randomized policy  $\pi = (\pi(a|s))_{(s,a) \in \mathbb{S} \times \mathbb{A}}$ , the value function  $\mathcal{V}^\pi$  and the state-action value function (also known as Q-function)  $\mathcal{Q}^\pi$  are defined as:
$$
\mathcal {V} ^ {\pi} (s) = \mathbb {E} ^ {\pi} \left[ \sum_ {t = 1} ^ {\infty} \gamma^ {t - 1} R _ {t} \left(S _ {t}, A _ {t}\right) \mid S _ {1} = s \right], s \in \mathbb {S} \tag {12}
$$
$$
\mathcal {Q} ^ {\pi} (s, a) = \mathbb {E} ^ {\pi} \left[ \sum_ {t = 1} ^ {\infty} \gamma^ {t - 1} R _ {t} \left(S _ {t}, A _ {t}\right) \mid S _ {1} = s, A _ {1} = a \right], (s, a) \in \mathbb {S} \times \mathbb {A}. \tag {13}
$$
Remark 4 (From MDP to MRP). Under any stationary randomized policy  $\pi$ , the process  $(S_{t}, A_{t})_{t > 0} =: (X_{t})_{t > 0}$  is a Markov chain over the state-space  $\mathbb{X} = \mathbb{S} \times \mathbb{A}$ , thus  $(X_{t}, R_{t})$  with  $R_{t}(X_{t}) = R_{t}(S_{t}, A_{t})$  is a Markov reward process of the kind that we analyzed in Section 2. As such, we can use Robust TD learning to evaluate  $\mathcal{V}(x) = \mathcal{Q}^{\pi}(x)$  for any  $x = (s, a) \in \mathbb{S} \times \mathbb{A}$ .
Heavy-tailed reward. We assume that the process  $(X_{t},R_{t})_{t > 0}$  with the Markov chain  $X_{t} = (S_{t},A_{t})$  and the reward  $R_{t} = R_{t}(X_{t})$  satisfies Assumption 1. We denote the stationary distribution of  $X_{t} = (S_{t},A_{t})$  under  $\pi$  as  $\mu^{\pi}$ .
Objective. For an initial state distribution  $\lambda$ , the objective in this work is to find the following:
$$
\pi^ {\star} \in \arg \max  _ {\pi} \int_ {\mathbb {S}} \mathcal {V} ^ {\pi} (s) \lambda (d s) =: \mathcal {V} ^ {\pi} (\lambda), \tag {14}
$$
over the class of stationary randomized policies.
Policy parameterization. In this work, we consider a finite but arbitrarily large state space  $\mathbb{S}$ , and for such problems, the tabular methods do not scale [51, 3]. In order to address this scalability issue, we consider widely-used softmax parameterization with linear function approximation: for a given set of feature vectors  $\{\Phi(s,a) \in \mathbb{R}^d : s \in \mathbb{S}, a \in \mathbb{A}\}$  and policy parameter  $W \in \mathbb{R}^d$ ,
$$
\pi_ {W} (a | s) = \frac {\exp \left(W ^ {\top} \Phi (s , a)\right)}{\sum_ {a ^ {\prime} \in \mathbb {A}} \exp \left(W ^ {\top} \Phi \left(s , a ^ {\prime}\right)\right)}, (s, a) \in \mathbb {S} \times \mathbb {A}. \tag {15}
$$
In the following subsection, we will describe the robust natural actor-critic algorithm.
3.2 Robust Natural Actor-Critic Algorithm
For any iteration  $k \geq 1$ , we denote  $\pi_k \coloneqq \pi_{W(k)}$  throughout the policy optimization iterations.
For samples  $\mathcal{D}^{(k)} = \{(S_{t,k},A_{t,k},R_{t,k},S_{t,k}',A_{t,k}') : t\geq 1\}$ , given  $(b_{t,k})_{t,k\in \mathbb{Z}_+}$  and  $\rho >0$ , Robust NAC Algorithm is summarized in Algorithm 2.
Algorithm 2: Robust Natural Actor-Critic Algorithm
Inputs: clipping radii  $(b_{t})_{t\geq 1}$  projection radius  $\rho >0$  , learning rate  $\alpha >0$ $L_{\delta} > 0$    
for  $k = 1,2,\ldots ,K$  do   
Set  $\Theta_k(1) = 0 / /$  initialization: max-entropy policy   
for  $t = 1,2,\dots ,T$  do   
Set  $g_{t}^{(k)}(\Theta_{k}(t)) = \Big(R_{t,k} + \gamma f_{\Theta_{k}(t)}(S_{t,k}^{\prime},A_{t,k}^{\prime}) - f_{\Theta_{k}(t)}(S_{t,k},A_{t,k})\Big)\Phi (S_{t,k},A_{t,k}).$ $\Theta_k(t + 1) = \Pi_{B_2(0,\rho)}\Bigl \{\Theta_k(t) + \eta_t\cdot g_t^{(k)}(\Theta_k(t))\cdot \mathbb{1}\{\| g_t^{(k)}(\Theta_k(t))\| _2\leq b_t\} \Bigr \}$    
end for   
 $W(k + 1) = W(k) + \alpha \cdot \frac{1}{T}\sum_{t = 1}^{T}\Theta_{k}(t)$    
end for
Remark 5. The optimal solution  $\Theta_k^\star \in \operatorname*{arg\min}_{\Theta \in \mathbb{R}^d} \underset{x = (s,a)}{\mathbb{E}} \left| \langle \Theta, \Phi(x) \rangle - \mathcal{Q}^{\pi_k}(x) \right|^2$  is a good approximation of the natural policy gradient:
$$
u_{k} = [G(\pi_{k})]^{-1}\nabla_{W}\mathcal{V}^{\pi_{k}}(\lambda)\in \operatorname *{arg  min}_{w\in \mathbb{R}^{d}}\underset {(s,a)\sim \mathsf{d}_{\lambda}^{\pi_{k}}\otimes \pi_{k}(\cdot |s)}{\mathbb{E}}\Big|\big\langle w,\nabla_{W}\log \pi_{k}(a|s)\big\rangle  - \mathcal{A}^{\pi_{k}}(s,a)\Big|^{2},
$$
which follows from Jensen's inequality and leads to the Q-NPG [1]. For a detailed discussion, refer to Appendix B.
3.3 Finite-Time Bounds for Robust Natural Actor-Critic
In this subsection, we will provide finite-time bounds for Robust NAC.
We assume that the resulting Markov reward process under  $\pi_{k}$  for each  $k$  satisfies Assumptions 1-3 with stationary distribution  $\mu^{\pi_k}$  and  $u_{k} \geq \mathbb{E}[|R_{t,k}(S_{t,k},A_{t,k})|^{1 + p}|S_{t,k},A_{t,k}]$ . We assume that the dataset  $\mathcal{D}^{(k)}$  is obtained independently at each iteration  $k \geq 1$  for simplicity, with  $(S_{t,k},A_{t,k}) \stackrel{i:d}{\sim} \mu^{\pi_k}$  and  $S_{t,k}' \sim \mathcal{P}_{A_{t,k}}(S_{t,k},\cdot)$  and  $A_{t,k} \sim \pi_k(\cdot |S_{t,k})$  according to Assumption 2a under the stationary distribution  $\mu^{\pi_k} = [\mu^{\pi_k}(s,a)]_{s \in \mathbb{S}, a \in \mathbb{A}}$  under  $\pi_k$ . We make the following standard assumption for policy optimization, which is common in the policy gradient literature [1, 57, 34].
Assumption 4 (Concentrability). For any  $k \geq 1$ , we assume that there exists  $C_{\mathrm{conc}} < \infty$  such that:
$$
\max  _ {(s, a) \in \mathbb {S} \times \mathbb {A}} \frac {\mathrm {d} _ {\lambda} ^ {\pi^ {\star}} (s) \pi^ {\star} (a | s)}{\mu^ {\pi_ {k}} (s , a)} \leq C _ {\text {c o n c}}, \tag {16}
$$
where  $\mu^{\pi_k}$  is the stationary distribution of  $(S_{t,k},A_{t,k})_{t\geq 1}$  under  $\pi_k$
Theorem 4 (Finite-time bounds for Robust NAC). Under Assumptions 1-4 for any  $k \geq 1$ , for any  $\delta \in (0,1)$  and  $T, K > 1$ , Robust NAC with  $\rho \geq \max_k \| \Theta_k^\star \|_2$ ,  $b_{t,k} = \left( \frac{u_k T}{\log(4T / \delta)} \right)^{\frac{1}{1 + p}}$ , learning rates  $\eta = \frac{\sqrt{2}(1 - \gamma) \rho L_{\delta}^{\frac{1 - p}{2(1 + p)}}}{(\max_{1 \leq k \leq K} u_k T)^{\frac{1}{1 + p}}}$  and  $\alpha = \frac{\sqrt{\log|\mathbb{A}|}}{\rho \sqrt{K}}$  achieves the following with probability at least  $1 - \delta$ :
$$
\min  _ {1 \leq k \leq K} \left\{\mathcal {V} ^ {\pi^ {\star}} (\lambda) - \mathcal {V} ^ {\pi_ {k}} (\lambda) \right\} \leq \frac {2 \rho \sqrt {\log | \mathbb {A} |}}{(1 - \gamma) \sqrt {K}} + \sqrt {\frac {\left(\max  _ {1 \leq k \leq K} u _ {k}\right) ^ {\frac {1}{1 + p}} C _ {\text {c o n c}} \rho}{(1 - \gamma) ^ {3} T ^ {\frac {p}{1 + p}}} \left(3 L _ {\delta} ^ {- \frac {1 - p}{2 (1 + p)}} + 7 L _ {\delta} ^ {\frac {p}{1 + p}}\right)},
$$
where  $L_{\delta} = \log (4T / \delta)$
The proof of Theorem 4 can be found in Appendix B.
Remark 6 (Sample complexity of Robust NAC). An immediate consequence of Theorem 4 is as follows: the best iterate error decays at a rate  $\tilde{\mathcal{O}}\left(\frac{1}{\sqrt{K}}\right) + \tilde{\mathcal{O}}\left(\frac{1}{T^{\frac{p}{2(1 + p)}}}\right)$  after  $K$  iterations of natural policy gradient, which contains  $T$  steps of Robust TD learning per iteration. As such, in order to achieve  $\varepsilon >0$  error, one needs  $T\times K = \tilde{\mathcal{O}} (\varepsilon^{-2 - 2(1 + p) / p})$  samples.
Remark 7. Theorem 4 can be easily extended to expected error bounds and the full-rank case, where we would have  $\tilde{\mathcal{O}}(K^{-1/2} + T^{-p})$  by using Theorem 1. By extending the analysis in Theorem 2, one can prove results for Markovian sampling as well.
Remark 8 (Exploration in Robust NAC). Assumption 4 implies that the stationary state-action distribution under  $\pi_{k}$  should be sufficiently exploratory so that it should have a nonzero probability at each state-action pair that is visited by the optimal policy  $\pi^{\star}$ . This assumption on the exploratory behavior of policy gradient methods is standard in reinforcement learning literature [1, 57, 34, 63]. Alternatively, entropy regularization can be used for natural policy gradient methods to encourage exploration, which would imply weaker conditions at the expense of an additional bias term in the function approximation setting [47, 9].
In this section, we present numerical results for Robust TD learning and its non-robust counterpart.
(1) Randomly-Generated MRP. In the first example, we consider a randomly-generated MRP with  $|\mathbb{X}| = 256$ . The transition kernel is randomly generated such that  $\mathcal{P}(x,x') \stackrel{iid}{\sim} \mathrm{Unif}(0,1)$ , and row-wise normalized to obtain a stochastic matrix. The feature dimension is  $d = 128$ , random features are generated according to the  $\chi$ -squared distribution  $\Phi(x) = \Phi_0(x) / \| \Phi_0(x) \|_2$  with  $\Phi_0(x) \sim \mathcal{N}(0, I_d)$  for all  $x \in \mathbb{X}$ ,  $\Theta^{\star} \sim 3U / \sqrt{d}$  for  $U \sim \mathrm{Unif}^d(0,1)$  and  $\Psi = [-\Phi^\top(x) - ]_{x \in \mathbb{X}}$ . The discount factor is  $\gamma = 0.9$ , and the reward is  $R_t(X_t) = r(X_t) + N_t - \mathbb{E}[N_t]$  with  $N_t \stackrel{iid}{\sim} \mathrm{Pareto}(1,1.2)$ . Mean squared error (2) under Robust TD learning and TD learning with the clipping radius  $b_t = t$  and diminishing step-size  $\eta_t = \frac{1}{\lambda_{\min}(1 - \gamma)t}$  in Theorem 1 and projection radius  $\rho = 30$  are shown in Figure 2. Despite
Figure 2: Performance of TD learning and Robust TD learning under heavy-tailed rewards of tail index 1.2. Each faded green line is the MSE for an individual trial, and the solid lines with markers indicate the average error performance for TD learning and Robust TD learning.
diminishing step-size and projection, TD learning fails miserably often and in expectation due to the outliers in the reward that lead to extremely large errors (Figure 2a). On the other hand, for the same feature vectors, state and reward realizations, Robust TD learning effectively eliminates them in every sample path, and achieves good and consistent performance despite extremely heavy-tailed reward and gradient noise with tail index 1.2 (Figure 2a).
(2) Circular Random Walk. In this example, we consider a circular random walk for  $\mathbb{X} = \{1, 2, \ldots, 256\}$ , where each state  $x$  is modulo- $|\mathbb{X}|$  [60]. The transition matrix is generated as  $\mathcal{P}(x, x') = 1/3$  if  $x = x'$  and  $\mathcal{P}(x, x') = 1/24$  if  $0 < |x - x'| \leq 8$ . The reward and random feature generation is the same as the first example. The performances of TD learning and Robust TD learning in this structured case after 1000 trials are given in Figure 3.
A similar behavior as the randomly-generated MRP is observed in this example: due to outliers, TD learning fails miserably, while Robust TD learning achieves good performance consistently.
Figure 3: Performances of Robust TD learning and TD learning for the circular random walk under heavy-tailed reward with tail index 1.2. Each faded green line is the error trajectory for an individual trial, and the solid lines indicate the expected errors for TD learning and Robust TD learning.
In this paper, we considered RL problem with heavy-tailed rewards, and proposed robust TD learning and NAC variants with a dynamic gradient clipping mechanism with provable performance guarantees, both in expectation and with high probability. Motivated by the results in this work, it would be interesting to explore single-timescale robust NAC and off-policy NAC for future work.
Atilla Eryilmaz's research was supported in part by NSF AI Institute (AI-EDGE) 2112471, CNS-NeTS-2106679, CNS-NeTS-2007231; and the ONR Grant N00014-19-1-2621.
[1] Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. Optimality and approximation with policy gradient methods in markov decision processes. In Conference on Learning Theory, pages 64-66. PMLR, 2020.
[2] Mukul Agarwal, Vivek S Borkar, and Abhay Karandikar. Structural properties of optimal transmission policies over a randomly varying channel. IEEE Transactions on Automatic Control, 53(6):1476-1491, 2008.
[3] Dimitri P Bertsekas and John N Tsitsiklis. Neuro-dynamic programming. Athena Scientific, 1996.
[4] Jalaj Bhandari, Daniel Russo, and Raghav Singal. A finite time analysis of temporal difference learning with linear function approximation. arXiv preprint arXiv:1806.02450, 2018.
[5] Shalabh Bhatnagar, Richard S Sutton, Mohammad Ghavamzadeh, and Mark Lee. Natural actor-critic algorithms. Automatica, 45(11):2471-2482, 2009.
[6] Sébastien Bubeck, Nicolo Cesa-Bianchi, and Gábor Lugosi. Bandits with heavy tail. IEEE Transactions on Information Theory, 59(11):7711-7717, 2013.
[7] Qi Cai, Zhuoran Yang, Jason D Lee, and Zhaoran Wang. Neural temporal-difference learning converges to global optima. Advances in Neural Information Processing Systems, 32, 2019.
[8] Semih Cayci, Atilla Eryilmaz, and Rayadurgam Srikant. Budget-constrained bandits over general cost and reward distributions. In International Conference on Artificial Intelligence and Statistics, pages 4388-4398. PMLR, 2020.
[9] Semih Cayci, Niao He, and Rayadurgam Srikant. Linear convergence of entropy-regularized natural policy gradient with linear function approximation. arXiv preprint arXiv:2106.04096, 2021.
[49] Matthew Streeter and Daniel Golovin. An online algorithm for maximizing submodular functions. Advances in Neural Information Processing Systems, 21, 2008.
[50] Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3(1):9-44, 1988.
[51] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[52] Richard S Sutton, David A McAllester, Satinder P Singh, Yishay Mansour, et al. Policy gradient methods for reinforcement learning with function approximation. In NIPs, volume 99, pages 1057-1063. Citeseer, 1999.
[53] Joel A Tropp. Freedman's inequality for matrix martingales. Electronic Communications in Probability, 16(25):262-270, 2011.
[54] John N Tsitsiklis and Benjamin Van Roy. An analysis of temporal-difference learning with function approximation. IEEE transactions on automatic control, 42(5):674-690, 1997.
[55] Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cambridge university press, 2019.
[56] Hongjian Wang, Mert Gurbuzbalaban, Lingjiong Zhu, Umut Simsekli, and Murat A Erdogdu. Convergence rates of stochastic gradient descent under infinite noise variance. Advances in Neural Information Processing Systems, 34:18866-18877, 2021.
[57] Lingxiao Wang, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural policy gradient methods: Global optimality and rates of convergence. arXiv preprint arXiv:1909.01150, 2019.
[58] Ward Whitt. The impact of a heavy-tailed service-time distribution upon the m/gi/s waiting-time distribution. Queueing Systems, 36(1-3):71-87, 2000.
[59] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256, 1992.
[60] Eric Xia and Martin Wainwright. Krylov-bellman boosting: Super-linear policy evaluation in general state spaces. In International Conference on Artificial Intelligence and Statistics, pages 9137-9166. PMLR, 2023.
[61] Tengyu Xu, Zhe Wang, and Yingbin Liang. Improving sample complexity bounds for (natural) actor-critic algorithms. arXiv preprint arXiv:2004.12956, 2020.
[62] Tengyu Xu, Zhe Wang, and Yingbin Liang. Non-asymptotic convergence analysis of two time-scale (natural) actor-critic algorithms. arXiv preprint arXiv:2005.03557, 2020.
[63] Rui Yuan, Simon S Du, Robert M Gower, Alessandro Lazaric, and Lin Xiao. Linear convergence of natural policy gradient methods with log-linear policies. arXiv preprint arXiv:2210.01400, 2022.
A Proofs for Robust TD Learning
The following lemma will be critical in our proofs.
Lemma 2 (Lemma 4 in [54]). For any two vectors  $\widehat{V}, V \in \mathbb{R}^{|\mathbb{X}|}$ ,
$$
\| \mathcal {T} \widehat {V} - \mathcal {T} V \| _ {\mu} \leq \gamma \cdot \| \widehat {V} - V \| _ {\mu},
$$
$$
(\mathcal {T} V) (x) = r (x) + \gamma \sum_ {x ^ {\prime} \in \mathbb {X}} \mathcal {P} \left(x, x ^ {\prime}\right) V \left(x ^ {\prime}\right), \tag {17}
$$
is the Bellman operator.
Proof of Theorem 1. The proof follows the Lyapunov approach in [4]. Let  $\mathcal{L}(\Theta) = \| \Theta -\Theta^{*}\|_{2}^{2}$  be the Lyapunov function for any  $\Theta \in \mathbb{R}^d$ . Then, by the non-expansivity of  $\Pi_{\mathcal{B}_2(0,\rho)}$ , we have:
$$
\begin{array}{l} \mathcal {L} (\Theta (t + 1)) \leq \mathcal {L} (\Theta (t)) + \eta^ {2} \| g _ {t} (\Theta (t)) \| _ {2} ^ {2} \mathbb {1} \left\{\| g _ {t} (\Theta (t)) \| _ {2} \leq b _ {t} \right\} \\ - 2 \eta g _ {t} (\Theta (t)) ^ {\top} (\Theta (t) - \Theta^ {*}) \mathbb {1} \left\{\| g _ {t} (\Theta (t)) \| _ {2} \leq b _ {t} \right\}. \tag {18} \\ \end{array}
$$
Taking conditional expectation given  $\mathcal{F}_t$  and using the fact that  $\mathbb{1}\{\| g_t(\Theta (t))\| _2 > b_t\} = 1 - \mathbb{1}\{\| g_t(\Theta (t))\| _2\leq b_t\}$ , we get:
$$
\begin{array}{l} \mathbb {E} \left[ \mathcal {L} (\Theta (t + 1)) \mid \mathcal {F} _ {t} \right] \leq \mathcal {L} (\Theta (t)) + 2 \eta \mathbb {E} _ {t} \left[ g _ {t} ^ {\top} (\Theta (t)) (\Theta (t) - \Theta^ {\star}) \right] \\ - 2 \eta \mathbb {E} \left[ g _ {t} (\Theta (t)) ^ {\top} (\Theta (t) - \Theta^ {*}) \mathbb {1} \left\{\| g _ {t} (\Theta (t)) \| _ {2} > b _ {t} \right\} \mid \mathcal {F} _ {t} \right] + \eta^ {2} u b _ {t} ^ {1 - p}, \tag {19} \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} [ \| g _ {t} (\Theta (t)) \| _ {2} ^ {2} \mathbb {1} \{\| g _ {t} (\Theta (t)) \| _ {2} \leq b _ {t} \} | \mathcal {F} _ {t} ] \leq \mathbb {E} [ \| g _ {t} (\Theta (t)) \| _ {2} ^ {1 + p} b _ {t} ^ {1 - p} | \mathcal {F} _ {t} ], \tag {20} \\ \leq u b _ {t} ^ {1 - p}, \\ \end{array}
$$
in the last term. Now, for  $\mathbb{E}_t[g_t^\top (\Theta (t))(\Theta (t) - \Theta^\star)]$ , we have the following inequality:
$$
\begin{array}{l} \mathbb {E} _ {t} [ g _ {t} ^ {\top} (\Theta (t)) (\Theta (t) - \Theta^ {\star}) ] = \mathbb {E} _ {t} [ (R _ {t} + \gamma f _ {\Theta (t)} (X _ {t} ^ {\prime}) - f _ {\Theta (t)} (X _ {t})) (f _ {\Theta (t)} (X _ {t}) - \mathcal {V} (X _ {t})) ], \\ = \mathbb {E} _ {t} \left[ \left(\left(\mathcal {T} f _ {\Theta (t)}\right) \left(X _ {t}\right) - f _ {\Theta (t)} \left(X _ {t}\right)\right) \left(f _ {\Theta (t)} \left(X _ {t}\right) - \mathcal {V} \left(X _ {t}\right)\right) \right], \\ \end{array}
$$
where  $\mathcal{T}$  is the Bellman operator (17). By using the fact that the value function  $\nu$  is the fixed point of the Bellman operator  $\mathcal{T}$ , we have the following:
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \left(\left(\mathcal {T} f _ {\Theta (t)}\right) \left(X _ {t}\right) - f _ {\Theta (t)} \left(X _ {t}\right)\right) \left(f _ {\Theta (t)} \left(X _ {t}\right) - \mathcal {V} \left(X _ {t}\right)\right) \right] \\ = \mathbb {E} _ {t} \left[ \left(\mathcal {T} f _ {\Theta (t)} \left(X _ {t}\right) - \mathcal {T V} \left(X _ {t}\right)\right) \left(f _ {\Theta (t)} \left(X _ {t}\right) - \mathcal {V} \left(X _ {t}\right)\right) \right] - \mathbb {E} _ {t} \left[ \left(f _ {\Theta (t)} \left(X _ {t}\right) - \mathcal {V} \left(X _ {t}\right)\right) ^ {2} \right]. \tag {21} \\ \end{array}
$$
By using Lemma 2, we conclude that:
$$
\mathbb {E} \left[ g _ {t} ^ {\top} (\Theta (t)) (\Theta (t) - \Theta^ {\star}) \right] \leq - (1 - \gamma) \sum_ {x \in \mathbb {X}} \mu (x) \left(f _ {\Theta (t)} (x) - \mathcal {V} (x)\right) ^ {2} = - (1 - \gamma) \| f _ {\Theta (t)} - \mathcal {V} \| _ {\mu} ^ {2}. \tag {22}
$$
Then, we can rewrite (19) as follows:
$$
\begin{array}{l} \mathbb {E} \left[ \mathcal {L} \left(\Theta (t + 1)\right) \mid \mathcal {F} _ {t} \right] \leq \mathcal {L} \left(\Theta (t)\right) - 2 (1 - \gamma) \eta \| f _ {\Theta (t)} - \mathcal {V} \| _ {\mu} ^ {2} \\ - 2 \eta \mathbb {E} \left[ g _ {t} (\Theta (t)) ^ {\top} (\Theta (t) - \Theta^ {*}) \mathbb {1} \left\{\left\| g _ {t} (\Theta (t)) \right\| _ {2} > b _ {t} \right\} \mid \mathcal {F} _ {t} \right] + \eta^ {2} u b _ {t} ^ {1 - p}, \tag {23} \\ \end{array}
$$
The bias introduced by using the gradient clipping can be bounded as follows: