SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ruida Zhou*
Texas A&M University ruida@tamu.edu
Tao Liu*
Texas A&M University tliu@amu.edu
Min Cheng
Texas A&M University minrara0404@tamu.edu
Dileep Kalathil
Texas A&M University dileep kalathil@tamu.edu
P. R. Kumar
Texas A&M University prk@amu.edu
Texas A&M University chao.tian@tamu.edu
We study robust reinforcement learning (RL) with the goal of determining a well-performing policy that is robust against model mismatch between the training simulator and the testing environment. Previous policy-based robust RL algorithms mainly focus on the tabular setting under uncertainty sets that facilitate robust policy evaluation, but are no longer tractable when the number of states scales up. To this end, we propose two novel uncertainty set formulations, one based on double sampling and the other on an integral probability metric. Both make large-scale robust RL tractable even when one only has access to a simulator. We propose a robust natural actor-critic (RNAC) approach that incorporates the new uncertainty sets and employs function approximation. We provide finite-time convergence guarantees for the proposed RNAC algorithm to the optimal robust policy within the function approximation error. Finally, we demonstrate the robust performance of the policy learned by our proposed RNAC approach in multiple MuJoCo environments and a real-world TurtleBot navigation task.
Training a reinforcement learning (RL) algorithm directly on a real-world system is expensive and potentially risky due to the large number of data samples required to learn a satisfactory policy. To overcome this issue, RL algorithms are typically trained on a simulator. However, in most real-world applications, the nominal model used in the simulator model may not faithfully represent the real-world system due to various factors, such as approximation errors in modeling or variations in real-world parameters over time. For example, the mass, friction, sensor/actuator noise, and floor terrain in a mobile robot simulator may differ from those in the real world. This mismatch, called simulation-to-reality-gap, can significantly degrade the performance of standard RL algorithms when deployed on real-world systems [36, 44, 55, 51]. The framework of robust Markov decision process (RMDP) [19, 40] is used to model this setting where the testing environment is uncertain and comes from an uncertainty set around the nominal model. The optimal robust policy is defined as the one which achieves the optimal worst-case performance over all possible models in the uncertainty set. The goal of robust reinforcement learning is to learn such an optimal robust policy using only the data sampled from the simulator (nominal) model.
The RMDP planning problem has been well studied in the tabular setting [63, 62, 35, 46, 16]. Also, many works have developed model-based robust RL algorithms in the tabular setting [66, 69, 42,
64, 49], focusing on sample complexity. Many robust Q-learning algorithms [45, 33, 39, 59, 30] and policy gradient methods for robust RL [61, 27, 29, 58, 17] have been developed for the tabular setting. Different from all these works, the main goal of this paper is to develop a computationally tractable robust RL algorithm with provable convergence guarantees for RMDPs with large state spaces, using linear and nonlinear function approximation for robust value and policy.
One of the main challenges of robust RL with a large state space is the design of an effective uncertainty set that is amenable to computationally tractable learning with function approximation. Robust RL algorithms, both in their implementation and technical analysis, require robust Bellman operator evaluations [19, 40] which involve an inner optimization problem over the uncertainty set. Performing this inner optimization problem and/or getting an unbiased estimate of the robust Bellman operator evaluation using only the samples from the nominal model can be intractable for commonly considered uncertainty sets when the state space is large. For example, for  $f$ -divergence-based uncertainty sets [66, 69, 64, 49, 33, 30], the robust Bellman operator estimate requires solving for a dual variable associated with each state, which is prohibitive for large state spaces. For  $R$ -contamination [61], the estimate requires calculating the minimum of the value function over state space, which is impractical for large state spaces. It is also infeasible for  $\ell_p$  norm-based uncertainty sets [27], where the estimates require calculating the median, mean, or average peak of the value function depending on the choice of  $p$ . Robust RL with function approximation has been explored in a few works [53, 41, 45, 43, 60, 6]. However, these works explicitly or implicitly assume an oracle that approximately computes the robust Bellman operator estimate, and the uncertainty set design that facilitates computation and learning is largely ignored. We overcome this challenge by introducing two novel uncertainty set formulations, one based on double sampling (DS) and the other on an integral probability metric (IPM). Both are compatible with large-scale robust MDP, with the robust Bellman operators being amenable to unbiased estimation from the data sampled from the nominal model, enabling effective practical robust RL algorithms.
Policy-based RL algorithms [47, 31, 18, 15], which optimize the policy directly, have been extremely successful in learning policies for continuous control tasks. Value-based RL approaches, such as Q-learning, cannot be directly applied to continuous action problems. Moreover, recent advances in policy-based approaches can establish finite time convergence guarantees and also offer insights into the practical implementation [3, 37, 4, 8]. However, most of the existing works on robust RL with function approximation use value-based approaches [53, 41, 45, 43, 60, 34] that are not scalable to continuous control problems. On the other hand, the existing works that use policy-based methods for robust RL with convergence guarantees are limited to the tabular setting [61, 27, 29, 58, 17]. We close this important gap in the literature by developing a novel robust natural actor-critic (RNAC) approach that leverages our newly designed uncertainty sets for scalable learning.
Summary of Contributions:  $(i)$ . We propose two novel uncertainty sets, one using double sampling (Section 3.1) and the other an integral probability metric (Section 3.2), which are both compatible for robust RL with large state space and function approximation. Though robust Bellman operators require the unknown worst-case models, we provide unbiased empirical robust Bellman operators that are computationally easy to utilize and only based on samples from the nominal model;
(ii). We propose a novel RNAC algorithm (Section 4) which to the best of our knowledge is the first policy-based approach for robust RL under function approximation, with provable convergence guarantees. We consider both linear and general function approximation for theoretical study, with the latter relegated to Appendix E due to space constraints. Under linear function approximation, the RNAC with a robust critic performing "robust linear-TD" (Section 5) and a robust natural actor performing "robust Q-NPG" (Section 6) is proved to converge to the optimal robust policy within the function approximation error. Specifically,  $\tilde{O}(1/\varepsilon^2)$  sample complexity can be achieved, or  $\tilde{O}(1/\varepsilon^3)$  for policy update with a constant step size. For the robust linear-TD, we study the contraction behavior of the projected robust Bellman operator for RMDPs with the proposed uncertainty sets and the well-known  $f$ -divergence uncertainty sets, which we believe is of independent interest;
(iii). We implement the proposed RNAC in multiple MuJoCo environments (Hopper-v3, Walker2d-v3, and HalfCheetah-v3), and demonstrate that RNAC with the proposed uncertainty sets results in robust behavior while canonical policy-based approaches suffer significant performance degradation. We also test RNAC on TurtleBot [5], a real-world mobile robot, performing a navigation task. We show that the TurtleBot with RNAC successfully reaches its destination while canonical non
robust approaches fail under adversarial perturbations. A video of the demonstration on TurtleBot is available at [Video Link] and the RNAC code is provided in the supplementary material.
Due to the page limit, a detailed literature review and comparison of our work with the existing robust RL algorithms are deferred to Appendix G.
Notations: For any set  $\mathcal{X}$ , denote by  $|\mathcal{X}|$  its cardinality, by  $\Delta_{\mathcal{X}}$  a  $(|\mathcal{X}| - 1)$ -dimensional probability simplex, and by  $\mathrm{Unif}(\mathcal{X})$  a uniform distribution over  $\mathcal{X}$ . Let  $[m] := \{1, \dots, m\}$ .
A Markov decision process (MDP) is represented by a tuple  $(\mathcal{S},\mathcal{A},\kappa ,r,\gamma)$ , where  $\mathcal{S}$  is the state space,  $\mathcal{A}$  is the action space,  $\kappa = (\kappa_0,\kappa_1,\ldots)$  is a possibly non-stationary transition kernel sequence with  $\kappa_t:\mathcal{S}\times \mathcal{A}\to \Delta_{\mathcal{S}}$ ,  $r:\mathcal{S}\times \mathcal{A}\to [0,1]$  is the reward function, and  $\gamma \in (0,1)$  is the discount factor. For a stationary policy  $\pi :\mathcal{S}\rightarrow \Delta_{\mathcal{A}}$ , its value function is  $V_{\kappa}^{\pi}(s)\coloneqq \mathbb{E}_{\kappa ,\pi}[\sum_{t = 0}^{\infty}\gamma^{t}r(s_{t},a_{t})|s_{0} = s]$ , where the expectation is taken w.r.t. the trajectory  $(s_0,a_0,s_1,a_1,\dots)$  with  $a_{t}|s_{t}\sim \pi_{s_{t}}$  and  $s_{t + 1}|(s_t,a_t)\sim \kappa_{t,s_t,a_t}$ . We can similarly define the state visitation distribution under initial distribution  $\rho \in \Delta_{\mathcal{S}}$ ,  $d_{\rho}^{\pi ,\kappa}(s)\coloneqq (1 - \gamma)\mathbb{E}_{\kappa ,\pi}[\sum_{t = 0}^{\infty}\gamma^{t}\mathbb{1}(s_{t} = s)|s_{0}\sim \rho ]$ ; the state-action value function (Q function),  $Q_{\kappa}^{\pi}(s,a)\coloneqq \mathbb{E}_{\kappa ,\pi}[\sum_{t = 0}^{\infty}\gamma^{t}r(s_{t},a_{t})|s_{0} = s,a_{0} = a]$ , and the advantage function  $A_{\kappa}^{\pi}(s,a)\coloneqq Q_{\kappa}^{\pi}(s,a) - V_{\kappa}^{\pi}(s)$ .
Robust MDP: A RMDP is represented by a tuple  $(S, \mathcal{A}, \mathcal{P}, r, \gamma)$ , where  $\mathcal{P}$  is a set of transition kernels known as the uncertainty set that captures the perturbations around the nominal stationary kernel  $p^{\circ}: S \times \mathcal{A} \to \Delta_{\mathcal{S}}$ , and its robust value function is defined as the corresponding worst case:
$$
V _ {\mathcal {P}} ^ {\pi} (s) := \inf  _ {\kappa \in \bigotimes_ {t \geq 0} \mathcal {P}} V _ {\kappa} ^ {\pi} (s). \tag {1}
$$
We will assume the following key  $(s, a)$ -rectangularity condition that is commonly assumed to facilitate dynamic programming ever since the introduction of RMDPs [19, 40]:
Definition 1.  $\mathcal{P}$  is  $(s,a)$ -rectangular, if  $\mathcal{P} = \bigotimes_{s,a} \mathcal{P}_{s,a}$ , for some  $\mathcal{P}_{s,a} \subseteq \Delta_{\mathcal{S}}$ .
The corresponding robust Bellman operator  $\mathcal{T}_{\mathcal{P}}^{\pi}:\mathbb{R}^{S}\to \mathbb{R}^{S}$  is
$$
\left(\mathcal {T} _ {\mathcal {P}} ^ {\pi} V\right) (s) = \mathbb {E} _ {a \sim \pi (\cdot | s)} \left[ r (s, a) + \gamma \inf  _ {p \in \mathcal {P} _ {s, a}} p ^ {\top} V \right], \quad \forall s \in \mathcal {S}, V \in \mathbb {R} ^ {\mathcal {S}}, \tag {2}
$$
and the Bellman equation for RMDPs is  $V = \mathcal{T}_{\mathcal{P}}^{\pi}V$ , where  $V = V^{\pi}$  is its unique solution from the Banach fixed-point theorem. Typically,  $\mathcal{P}_{s,a}$  is taken as a ball  $\{\nu \subseteq \Delta_{S} : \mathrm{d}(\nu, p_{s,a}^{\circ}) \leq \delta\}$  around a nominal model  $p^{\circ}$  of the training environment, where  $\mathrm{d}(\cdot, \cdot)$  is some divergence measure between probability distributions, and  $\delta > 0$  controls the level of robustness.
There is a stationary optimal policy  $\pi^{*}$  that uniformly maximizes the robust value function, i.e.,  $V_{\mathcal{P}}^{\pi^{*}}(s) = \sup \{V_{\mathcal{P}}^{\pi}(s)$ : history dependent  $\pi$ ,  $\forall s$  [19, 40]. Thus, without loss of generality, we only need to optimize within stationary policies. Moreover, for any stationary policy  $\pi$  there exists a stationary worst-case kernel  $\kappa_{\pi}$  with  $V_{\kappa_{\pi}}^{\pi}(s) = V_{\mathcal{P}}^{\pi}(s), \forall s$ . We can define the robust Q-function and robust advantage function as  $Q_{\mathcal{P}}^{\pi}(s,a) \coloneqq Q_{\kappa_{\pi}}^{\pi}(s,a)$  and  $A_{\mathcal{P}}^{\pi}(s,a) \coloneqq A_{\kappa_{\pi}}^{\pi}(s,a)$ , respectively. When the uncertainty set is clear from the context, we will omit the subscript  $\mathcal{P}$  in  $V_{\mathcal{P}}^{\pi}, Q_{\mathcal{P}}^{\pi}$ , and  $A_{\mathcal{P}}^{\pi}$ .
To summarize, the motivation in the robust RL framework is to learn the optimal robust policy by only training on a simulator implementing an (unknown) nominal model  $p^{\circ}$  [53, 45, 41], with the robust RL algorithms only having access to data generated from  $p^{\circ}$ , and not from any other model in the uncertainty set  $\mathcal{P}$ .
3 Uncertainty Sets for Large State Spaces
Evaluating the robust Bellman operator  $\mathcal{T}_{\mathcal{P}}^{\pi}$  requires solving the optimization  $\inf_{p\in \mathcal{P}_{s,a}}p^\top V$ , which is challenging for arbitrary uncertainty sets. Moreover, since we only have access to data generated by the nominal model  $p^{\circ}$ , estimating  $\mathcal{T}_{\mathcal{P}}^{\pi}$  is not straightforward. Previously studied uncertainty sets, such as  $R$ -contamination,  $f$ -divergence, and  $\ell_p$  norm, are intractable for RMDPs with large state spaces for these reasons (see Appendix B for detailed explanation). We therefore design two uncertainty sets for RMDPs where the robust Bellman operator has a tractable form and can be unbiasedly
estimated by the data sampled from the nominal model, thus making effective learning possible. We will also use function approximation to tractably parameterize policy and value functions.
3.1 Double-Sampling (DS) Uncertainty Set
The central difficulty in employing the robust Bellman operator (2) is how to evaluate  $\inf_{p\in \mathcal{P}_{s,a}}p^{\top}V$ . Our first method is based on the following key idea for drawing samples that produce an unbiased estimate for it. Let  $\{s_1',s_2',\ldots ,s_m'\}$  be  $m$  samples drawn i.i.d. according to the nominal model  $p_{s,a}^{\circ}$ . Then, for any given divergence measure  $\mathrm{d}(\cdot ,\cdot)$  and radius  $\delta >0$ , there exists an uncertainty set  $\mathcal{P} = \otimes_{s,a}\mathcal{P}_{s,a}$  such that  $\inf_{p\in \mathcal{P}_{s,a}}p^{\top}V = \mathbb{E}_{s_{1:m}^{\prime}}\underset {\sim}{\overset{i.i.d.}{\sim}}p_{s,a}^{\circ}\left[\inf_{\alpha \in \Delta_{[m]}\colon \mathrm{d}(\alpha ,\mathrm{Unif}([m]))\leq \delta}\sum_{i = 1}^{m}\alpha_{i}V(s_{i}^{\prime})\right]$ . (See Appendix B.1 for a brief explanation.) We therefore define an empirical robust Bellman operator  $\hat{T}_{\mathcal{P}}^{\pi}$  corresponding to (2) by
$$
\left(\hat {\mathcal {T}} _ {\mathcal {P}} ^ {\pi} V\right) \left(s, a, s _ {1: m} ^ {\prime}\right) := r (s, a) + \gamma_ {\alpha \in \Delta_ {[ m ]}: \mathrm {d} (\alpha , \operatorname {U n i f} ([ m ])) \leq \delta} \sum_ {i = 1} ^ {m} \alpha_ {i} V \left(s _ {i} ^ {\prime}\right). \tag {3}
$$
Since  $(\mathcal{T}_{\mathcal{P}}^{\pi}V)(s) = \mathbb{E}_{a\sim \pi (\cdot |s), s_{1:m}^{\prime}\stackrel {i.i.d.}{\sim}p_{s,a}^{\circ}}\left[(\hat{\mathcal{T}}_{\mathcal{P}}^{\pi}V)(s,a,s_{1:m}^{\prime})\right], \hat{\mathcal{T}}_{\mathcal{P}}^{\pi}$  gives an unbiased estimate, which is one key property we use in our robust RL algorithm. We refer to this as "double sampling" since  $\sum_{i=1}^{m}\alpha_{i}V(s_{i}^{\prime})$  is the expected value when one further chooses one sample  $s^{\prime}$  from  $\{s_1', s_2', \ldots, s_m'\}$  according to the distribution  $\alpha$  on  $[m]$  and evaluates  $V(s^{\prime})$ . Above,  $\alpha$  is a perturbation of  $\mathrm{Unif}([m])$  and when  $\delta = 0$ ,  $\alpha = \mathrm{Unif}([m])$  and  $s^{\prime}\sim p_{s,a}^{\circ}$ . The uncertainty set  $\mathcal{P}$  corresponding to the double sampling is implicitly defined by specifying the choices of  $m$ ,  $\mathrm{d}(\cdot ,\cdot)$ , and  $\delta$ . Its key advantage is that samples can only be drawn according to the nominal model  $p^{\circ}$ .
Double sampling requires sampling multiple next states for a given state-action pair, which can be implemented if the simulator is allowed to set to any state. (All MuJoCo environments [56] support DS.) Since the calculation of  $\alpha$  is within  $\Delta_{[m]}$ , the empirical robust Bellman operator is tractable for moderate values of  $m$ . We use  $m = 2$  in experiments for training efficiency, where for almost all divergences  $\mathrm{d}(\cdot ,\cdot)$  we can explicitly write
$$
\left(\tilde {\mathcal {T}} _ {\mathcal {P}} ^ {\pi} V\right) \left(s, a, s _ {1: 2} ^ {\prime}\right) = r (s, a) + \gamma \left(0. 5 \left(V \left(s _ {1} ^ {\prime}\right) + V \left(s _ {2} ^ {\prime}\right)\right) - \delta \left| V \left(s _ {1} ^ {\prime}\right) - V \left(s _ {2} ^ {\prime}\right) \right|\right). \tag {4}
$$
Bellman completeness (value function class closed under the Bellman operator) is a key property for efficient reinforcement learning, whereas RMDP with canonical uncertainty sets may violate it. We show in Appendix B.1 that for RMDP with DS uncertainty sets, the linear function approximation class satisfies Bellman completeness if the nominal model is a linear MDP [22].
3.2 Integral Probability Metric (IPM) Uncertainty Set
Given some function class  $\mathcal{F} \subset \mathbb{R}^S$  including the zero function, the integral probability metric (IPM) is defined by  $\mathrm{d}_{\mathcal{F}}(p,q) \coloneqq \sup_{f \in \mathcal{F}} \{ p^\top f - q^\top f \} \geq 0$  [38]. Many metrics such as Kantorovich metric, total variation, etc., are special cases of IPM under different function classes [38].
The robust Bellman operator  $\mathcal{T}_{\mathcal{P}}^{\pi}V$  (2) requires solving the optimization  $\inf_{q\in \mathcal{P}_{s,a}}q^{\top}V$ . For an IPM-based uncertainty set  $\mathcal{P}$  with  $\mathcal{P}_{s,a} = \{q:\mathrm{d}_{\mathcal{F}}(q,p_{s,a}^{\circ})\leq \delta \}$ , we relax the domain  $q\in \Delta_{\mathcal{S}}$  to  $\sum_{s}q(s) = 1$  as done in [28, 27], which incurs no relaxation error for small  $\delta$  if  $\min_{s'}p_{s,a}^{\circ}(s') > 0$ .
One should choose  $\mathcal{F}$  so that it properly encodes information of the MDP and its value functions. We start by considering linear function approximation. Denote by  $\Psi \in \mathbb{R}^{S\times d}$  the feature matrix with rows  $\psi^{\top}(s),\forall s\in S$ . The value function approximation is  $V_{w} = \Psi w\in \mathbb{R}^{S}$ . A good choice of feature vectors encodes information about the state and transition. For example, vectors  $\psi (s),\psi (s^{\prime})$  should be "close" when states  $s,s^{\prime}$  are "similar" and  $V^{\pi}(s),V^{\pi}(s^{\prime})$  have small differences. We propose the following function class (with  $\ell_2$  as the preferred norm but any other can be used):
$$
\mathcal {F} := \{s \mapsto \psi (s) ^ {\top} \xi : \xi \in \mathbb {R} ^ {d}, \| \xi \| \leq 1 \}. \tag {5}
$$
Without loss of generality, assume  $\Psi$  has full column rank since  $d\ll |\mathcal{S}|$ , and let the first coordinate of  $\psi (s)$  be 1 for any  $s$ , which corresponds to the bias term of the linear regressor.
Proposition 1. For the IPM with  $\mathcal{F}$  in (5), we have  $\inf_{q\in \mathcal{P}_{s,a}}q^{\top}V_{w} = (p_{s,a}^{\circ})^{\top}V_{w} - \delta \| w_{2:d}\|$ .
Algorithm 1: Robust Natural Actor-Critic
Input:  $T$ ,  $\eta^{0:T-1}$ ,  $K$ ,  $N$
Initialize:  $\theta^0$  for policy parameterization and  $w_{init}$  for value function approximation
for  $t = 0,1,\ldots ,T - 1$  do
Robust critic updates  $w^t$ ; //E.g.,  $w^t = \text{RLTD}(\pi_{\theta^t}, K)$  Algorithm 2 Robust natural actor updates  $\theta^{t+1}$ ; //E.g.,  $\theta^{t+1} = \text{RQNPG}(\theta^t, \eta^t, w^t, N)$  Algorithm 3
We can thus design the empirical robust Bellman operator, which is an unbiased estimator of  $\mathcal{T}_{\mathcal{P}}^{\pi}$  with sample  $s^\prime$  drawn from the nominal model:
$$
\left(\hat {\mathcal {T}} _ {\mathcal {P}} ^ {\pi} V _ {w}\right) \left(s, a, s ^ {\prime}\right) := r (s, a) + \gamma V _ {w} \left(s ^ {\prime}\right) - \gamma \delta \| w _ {2: d} \|. \tag {6}
$$
Guided by the last regularization term of the empirical robust Bellman operator (6), when considering value function approximation by neural networks we add a similar negative regularization term for all the neural network parameters except for the bias parameter in the last layer.
4 Robust Natural Actor-Critic
We propose a robust natural actor-critic (RNAC) approach in Algorithm 1 for the robust RL problem. As its name suggests, there are two components - a robust critic and a robust actor, which update alternately for  $T$  steps. At each step  $t$ , there is a policy  $\pi^t$  determined by parameter  $\theta^t$ . The robust critic updates the value function approximation parameter  $w^t$  based on on-policy trajectory data sampled by executing  $\pi^t$  on the nominal model with length  $K$ . The robust actor then updates the policy with step size  $\eta^t$ , and the critic returns  $w^t$  by on-policy trajectory data with length  $N$ . In practice, a batch of on-policy data can be sampled and used for both critic and actor updates.
We now give the main (informal) convergence results for our RNAC algorithm with linear function approximation and DS or IPM uncertainty sets, where the robust critic performs robust linear-TD and the robust natural actor performs robust-QNPG (as the comments in Algorithm 1). The formal statements, proofs, and generalization to general function approximation are given in Appendix E.
Theorem 1 (Informal linear convergence of RNAC). Under linear function approximation, RNAC in Algorithm 1 with DS or IPM uncertainty sets using an RLTD robust critic update and an RQNPG robust natural actor update, with appropriate geometrically increasing step sizes  $\eta^t$ , achieves  $\mathbb{E}[V^{\pi^{*}}(\rho) - V^{\pi^{T}}(\rho)] = O(e^{-T}) + O(\epsilon_{stat}) + O(\epsilon_{bias})$  and an  $\tilde{O}(1 / \varepsilon^{2})$  sample complexity.
The optimality gap is bounded in this theorem via three terms, where the first term, related to the number of time steps  $T$ , is the optimization rate (linear convergence since  $O(e^{-T})$ ), the second term  $\epsilon_{stat} = \tilde{O}\left(\frac{1}{\sqrt{N}} + \frac{1}{\sqrt{K}}\right)$  is a statistical error that depends on the number of samples  $K$ ,  $N$  in the robust critic and robust actor updates, and the last term  $\epsilon_{bias}$  is the approximation error due to the limited representation power of value function approximation and the parameterized policy class. Omitting the approximation error  $\epsilon_{bias}$ , the sample complexities for achieving  $\varepsilon$  robust optimal value are  $\tilde{O}(1 / \varepsilon^2)$ , which achieves the optimal sample complexity in tabular setting [64]. However, RNAC with geometrically increasing step sizes induces a larger multiplicative constant factor (not shown in big- $O$  notation) and does not generalize well to general function approximation. We then analyze RNAC with a constant step size.
Theorem 2 (Informal sublinear convergence of RNAC). RNAC under the same specification as in Theorem 1 but with constant step size  $\eta^t = \eta$  has  $\mathbb{E}[V^{\pi^*}(\rho) - \frac{1}{T}\sum_{t=0}^{T-1} V^{\pi^t}(\rho)] = O\left(\frac{1}{T}\right) + O(\epsilon_{stat}) + O(\epsilon_{bias})$ , implying an  $\tilde{O}(1/\varepsilon^3)$  sample complexity.
Although the theorem shows a slower optimization rate of RNAC with constant step size, this nonincreasing step size is preferred in practice. Moreover, the analysis can be generalized to a general policy class with optimization rate  $O(1 / \sqrt{T})$ , and an  $\tilde{O}(1 / \varepsilon^4)$  sample complexity.