SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 2: Robust Linear Temporal Difference (RLTD)
Input:  $\pi, K$
Initialize:  $w_0, s_0$
for  $k = 0,1,\ldots ,K - 1$  do
$$
\text {S a m p l e} a _ {k} \sim \pi (\cdot | s _ {k}), y _ {k + 1} \text {a c c o r d i n g t o} p _ {s _ {k}, a _ {k}} ^ {\circ}, \text {a n d} s _ {k + 1} \text {f r o m} y _ {k + 1}
$$
$$
\text {U p d a t e} w _ {k + 1} = w _ {k} + \alpha_ {k} \psi (s _ {k}) \left[ \left(\hat {\mathcal {T}} _ {\mathcal {P}} ^ {\pi} V _ {w _ {k}}\right) \left(s _ {k}, a _ {k}, y _ {k + 1}\right) - \psi \left(s _ {k}\right) ^ {\top} w _ {k} \right]
$$
$$
/ / \text {F o r} y _ {k + 1} = s _ {1: m} ^ {\prime} \stackrel {i. i. d.} {\sim} p _ {s _ {k}, a _ {k}} ^ {\circ}, s _ {k + 1} \sim \operatorname {U n i f} \left(y _ {k + 1}\right) \text {a n d} \hat {\mathcal {T}} _ {\mathcal {P}} ^ {\pi} \text {i n (3)}
$$
$$
/ / \text {F o r} y _ {k + 1} = s _ {k + 1} \sim p _ {s _ {k}, a _ {k}} ^ {\circ} \text {a n d} \tilde {\mathcal {T}} _ {\mathcal {P}} ^ {\pi} \text {i n} \tag {6}
$$
Return:  $w_{K}$
5 Robust Critic
The robust critic estimates the robust value function with access to samples from the nominal model. One may note that in many previous actor-critic analyses for canonical RL [61, 29, 9], the critic learns the  $Q$  function, while realistic implementations in on-policy algorithms (e.g., proximal policy optimization (PPO)) treat the  $V$  function as the target of the critic for training efficiency. We consider a robust critic that learns the robust  $V$  function to align with such realistic implementations.
We present the robust linear temporal difference (RLTD) Algorithm 2, which is similar to the canonical linear-TD algorithm, but with an empirical robust Bellman operator. It iteratively performs the sampling and updating procedures. The sampling procedure differs for the uncertainty set by double sampling and IPM as shown in the comments in Algorithm 2. Using the samples, the parameter for the linear value function approximation  $V_{w_k} = \Psi w_k$  is updated with step size  $\alpha_k$ . The following assumption is common [10, 9, 29]:
Assumption 1 (Geometric mixing). For any policy  $\pi$ , the Markov chain  $\{s_k\}$  induced by applying  $\pi$  in the nominal model  $p^{\circ}$  is geometrically ergodic with a unique stationary distribution  $\nu^{\pi}$ .
The update procedure essentially minimizes the Mean Square Projected Robust Bellman Error  $\mathrm{MSPRBE}^{\pi}(w) = \| \Pi^{\pi}(\mathcal{T}_{\mathcal{P}}^{\pi}V_{w}) - V_{w}\|_{\nu^{\pi}}^{2}$ , where  $\| V\|_{\nu^{\pi}} = \sqrt{\sum_s\nu^{\pi}(s)V(s)^2}$  is the  $\nu^{\pi}$  weighted norm, and  $\Pi^{\pi} = \Psi (\Psi^{\top}D^{\pi}\Psi)^{-1}\Psi^{\top}D^{\pi}$  is the weighted projection matrix with  $D^{\pi} = \mathrm{diag}(\nu^{\pi})\in$ $\mathbb{R}^{S\times S}$ . The minimizer of  $\mathrm{MSPRBE}^{\pi}(w)$ , denoted by  $w^{\pi}$ , is the unique solution of the projected robust Bellman equation  $V_{w} = \Pi^{\pi}\mathcal{T}_{\mathcal{P}}^{\pi}V_{w}$ , which is equivalent to  $0 = \Psi^{\top}D^{\pi}(\mathcal{T}_{\mathcal{P}}^{\pi}\Psi w - \Psi w)$ . RLTD is thus a stochastic approximation algorithm since the empirical operator  $\hat{T}_{\mathcal{P}}^{\pi}$  ((3) or (6)) is unbiased with  $\mathbb{E}_{(s,a,y')\sim \nu^{\pi}\circ \pi \circ p^{\circ}}\left[\psi (s)\left[(\hat{T}_{\mathcal{P}}^{\pi}V_{w})(s,a,y') - \psi (s)^{\top}w\right]\right] = \Psi^{\top}D^{\pi}(\mathcal{T}_{\mathcal{P}}^{\pi}V_{w} - V_{w})$ .
To ensure that RLTD converges to the optimal linear approximation  $V_{w^{\pi}}$ , it is crucial that the projected robust Bellman operator  $\Pi^{\pi}\mathcal{T}_{\mathcal{P}}^{\pi}$  be a contraction map with some  $\beta < 1$ .
Definition 2.  $\Pi^{\pi}\mathcal{T}_{\mathcal{P}}^{\pi}$  is a  $\beta$  -contraction w.r.t.  $\| \cdot \|_{\nu^{\pi}}$  if  $\| \Pi^{\pi}\mathcal{T}_{\mathcal{P}}^{\pi}V - \Pi^{\pi}\mathcal{T}_{\mathcal{P}}^{\pi}V^{\prime}\|_{\nu^{\pi}}\leq \beta \| V - V^{\prime}\|_{\nu^{\pi}}$
Unlike in linear TD for MDP [57], Tamar et al. [53] make an additional assumption (Assumption 2) to establish the contraction property of  $\Pi^{\pi}\mathcal{T}_{\mathcal{P}}^{\pi}$  (Proposition 3) for RMDP.
Assumption 2. There exists  $\beta \in (0,1)$  with  $\gamma p_{s,a}(s') \leq \beta p_{s,a}^{\circ}(s') \forall s, s' \in \mathcal{S}, \forall a \in \mathcal{A}$ , and  $\forall p \in \mathcal{P}$ .
Proposition 2 (Prop.3 in [53]). Under Assumption 2,  $\Pi^{\pi}\mathcal{T}_{\mathcal{P}}^{\pi}$  is a  $\beta$ -contraction w.r.t.  $\| \cdot \|_{v^{\pi}}$ .
The implicit uncertainty set  $\mathcal{P}$  of double-sampling satisfies Assumption 2 for small  $\delta$ , and thus guarantees contraction of  $\Pi^{\pi}T_{\mathcal{P}}^{\pi}$ . For example, for  $m = 2$  as in (4), a  $\delta < \frac{1 - \gamma}{2\gamma}$  is sufficient. However, simply taking a small radius  $\delta$  is not a panacea for all uncertainty sets:
Proposition 3. For any  $f$ -divergence and radius  $\delta > 0$ , there exists a geometrically mixing nominal model such that the  $f$ -divergence defined uncertainty set violates Assumption 2.
On the other hand, Assumption 2, though well-accepted [53, 45, 41], may not be necessary. The proposed IPM uncertainty set relates robustness and regularization with an explicit formula for robust Bellman operator as in (6). The contraction behavior of  $\Pi^{\pi}\mathcal{T}_{\mathcal{P}}^{\pi}$  for IPM uncertainty set can be established without Assumption 2:
Lemma 1. For IPM uncertainty set with radius  $\delta < \lambda_{\mathrm{min}}(\Psi^\top D^\pi \Psi)\frac{1 - \gamma}{\gamma}$ , there exists  $\beta < 1$  that  $\Pi^{\pi}\mathcal{T}_{\mathcal{P}}^{\pi}$  is a  $\beta$ -contraction mapping w.r.t. norm  $\| \cdot \|_{v^{\pi}}$ .
Since the contraction of  $\Pi^{\pi}\mathcal{T}_{\mathcal{P}}^{\pi}$  is obtained by RMDP under DS or IPM uncertainty sets with small radius  $\delta$ , we have the first finite sample guarantee of RLTD by recent advances in Markovian stochastic approximation [10]:
Theorem 3 (Informal convergence of robust critic: Details in Appendix C). RLTD with step sizes  $\alpha_{k} = \Theta(1/k)$  satisfies  $\mathbb{E}[\|w_{K} - w^{\pi}\|^{2}] = \tilde{O}\left(\frac{1}{K}\right)$ .
6 Robust Natural Actor
The robust natural actor updates the policy parameter  $\theta$  along an ascent direction that improves the value via preconditioning through the KL-divergence  $\mathrm{KL}(p,q)\coloneqq \langle p,\log (p / q)\rangle$ . It has been well explored for natural policy gradient (NPG)-like algorithms, such as TRPO and PPO in canonical RL. The ascent direction is obtained by the policy gradient theorem in canonical MDP. We therefore first discuss the policy gradient for RMDP, where policy  $\pi_{\theta}$  is differentiably parameterized by  $\theta$ .
The robust value  $V_{\mathcal{P}}^{\pi_{\theta}}(\rho) \eqqcolon J(\theta)$  is typically Lipschitz under proper parameterization [61], and is therefore differentiable a.e. by Rademacher's theorem [14]. Where it is not differentiable, a Fréchet supergradient  $\nabla J$  of  $J$  exists if  $\lim \sup_{\theta' \to 0} \frac{J(\theta') - J(\theta) - \langle \nabla J(\theta), \theta' - \theta \rangle}{\|\theta' - \theta\|} \leq 0$  [25]. The following contains the policy gradient theorem for canonical RL as a special case:
Lemma 2 (Policy supergradient). For a policy  $\pi = \pi_{\theta}$  that is differentiable w.r.t. parameter  $\theta$
$$
\nabla_ {\theta} V ^ {\pi} (\rho) = \frac {\mathbb {E} _ {s \sim d _ {\rho} ^ {\pi , \kappa_ {\pi}}} \mathbb {E} _ {a \sim \pi_ {s}} [ Q ^ {\pi} (s , a) \nabla_ {\theta} \log \pi (a | s) ]}{1 - \gamma} = \frac {\mathbb {E} _ {s \sim d _ {\rho} ^ {\pi , \kappa_ {\pi}}} \mathbb {E} _ {a \sim \pi_ {s}} [ A ^ {\pi} (s , a) \nabla_ {\theta} \log \pi (a | s) ]}{1 - \gamma}
$$
is a Fréchet supergradient of  $V^{\pi}(\rho)$ , where  $\kappa_{\pi}$  is the worst-case transition kernel w.r.t.  $\pi$ .
We consider log-linear policies  $\pi_{\theta}(a|s) = \frac{\exp\bigl(\phi(s,a)^{\top}\theta\bigr)}{\sum_{a^{\prime}}\exp\bigl(\phi(s,a^{\prime})^{\top}\theta\bigr)}$ , where  $\phi (s,a)\in \mathbb{R}^{d}$  is the feature vector and  $\theta \in \mathbb{R}^d$  is the policy parameter. (The general policy class is treated in Appendix D).
In canonical RL, the training and testing environments follow the same nominal transition  $p^\circ$ . NPG updates the policy by  $\theta \gets \theta + \eta \mathcal{F}_{\rho}(\theta)^\dagger \nabla_{\theta} V_{p^\circ}^{\pi_\theta}(\rho)$ , where  $\mathcal{F}_{\rho}(\theta)^\dagger$  is the Moore-Penrose inverse of the Fisher information matrix  $\mathcal{F}_{\rho}(\theta) := \mathbb{E}_{(s,a) \sim d_{\rho}^{\pi_\theta, p^\circ} \circ \pi_\theta} [\nabla_{\theta} \log \pi_{\theta}(a|s) (\nabla_{\theta} \log \pi_{\theta}(a|s))^{\top}]$ . An "equivalent" Q-NPG was proposed in [3] to update the policy by  $\theta \gets \theta + \eta u'$ , where  $u' = \arg \min_u \mathbb{E}_{(s,a) \sim d_{\rho}^{\pi, p^\circ} \circ \pi} [(Q_{p^\circ}^\pi(s,a) - u^\top \phi(s,a))^2]$ . Note that  $u$  determines a Q value function approximation  $Q^u(s,a) := \phi(s,a)^\top u$ , which is compatible with the log-linear policy class [52]. Since  $Q^\pi$  contains the information on the ascent direction as suggested by Lemma 2, the Q-NPG update can be viewed as inserting the best compatible Q-approximation  $Q^{u'}$  for the policy.
We adopt the Q-NPG to robust RL and propose the Robust Q-Natural Policy Gradient (RQNPG) (Algorithm 3). Note that unlike canonical RL, where  $Q_{p^{\circ}}^{\pi}$  can be estimated directly from a sample trajectory of executing  $\pi$  in model  $p^{\circ}$ , the robust  $Q^{\pi}$  is hard to estimate with samples from  $p^{\circ}$ . A value function approximation  $V_{w}$  from the critic comes to help, as we can approximate the robust Q function via  $Q_{w}(s,a) = r(s,a) + \inf_{p\in \mathcal{P}_{s,a}}p^{\top}V_{w}$ , which exactly matches  $Q^{\pi}$  if  $V_{w} = V^{\pi}$ . The RQNPG obtains information on  $Q^{\pi}$  by first approximating it via a critic value  $V_{w}$ -guided function  $Q_{w}$ , and then estimating  $Q_{w}$  by a policy-compatible robust Q-approximation  $Q^{u}$ .
As shown in Algorithm 3, RQNPG estimates a compatible Q-approximation  $Q^{u_N}$  by iteratively performing sampling and updating procedures, where the sampling procedure follows that of the RLTD (Algorithm 2). The update procedure essentially approaches the minimizer  $u_w^\pi \coloneqq \arg \min_u\mathbb{E}_{(s,a)\sim \nu^\pi \circ \pi}\left[(Q_w(s,a) - u^\top \phi (s,a))^2\right]$  by stochastic approximation with step size  $\zeta_{n}$  (stochastic gradient descent with Markovian data), since conditioned on  $(s,a)$ ,  $\hat{T}_{\mathcal{P}}^{\pi}V_{w}$  ((3) or (6)) is an unbiased estimator for the Q function approximation  $Q_{w}$ .
Now we look at a specific update  $\theta^{t + 1} = \mathsf{RQNPG}(\theta^t,\eta^t,w^t,N)$  with  $\zeta_{n} = \Theta (1 / n)$ , where  $w^{t} = \mathrm{RLTD}(\pi_{\theta^{t}},K)$ . The following theorem shows an approximate policy improvement property of the RQNPG update:
Algorithm 3: Robust Q-Natural Policy Gradient (RQNPG)
Input:  $\theta, \eta, w, N$
Initialize:  $u_0, s_0$
for  $n = 0,1,\ldots ,N - 1$  do
$$
\text {S a m p l e} a _ {n} \sim \pi_ {\theta} (\cdot | s _ {n}), y _ {n + 1} \text {a c c o r d i n g} p _ {s _ {k}, a _ {k}} ^ {\circ} \text {a n d d e t e r m i n e} s _ {n + 1} \text {f r o m} y _ {n + 1} ^ {\prime}
$$
$$
\text {U p d a t e} u _ {n + 1} = u _ {n} + \zeta_ {n} \phi \left(s _ {n}, a _ {n}\right) \left[ \left(\hat {\mathcal {T}} _ {\mathcal {P}} ^ {\pi} V _ {w}\right) \left(s _ {n}, a _ {n}, y _ {n + 1}\right) - \phi \left(s _ {n}, a _ {n}\right) ^ {\top} u _ {n} \right]
$$
$$
/ / \text {F o r} y _ {n + 1} = s _ {1: m} ^ {\prime} \stackrel {i. i. d.} {\sim} p _ {s _ {n}, a _ {n}} ^ {\circ}, s _ {n + 1} \sim \operatorname {U n i f} \left(y _ {n + 1}\right) \text {a n d} \hat {\mathcal {T}} _ {\mathcal {P}} ^ {\pi} \text {i n (3)}
$$
$$
/ / \text {F o r} y _ {n + 1} = s _ {n + 1} \sim p _ {s _ {n}, a _ {n}} ^ {\circ} \text {a n d} \hat {\mathcal {T}} _ {\mathcal {P}} ^ {\pi} \text {i n (6)}
$$
Return:  $\theta +\eta u_{N}$
Theorem 4 (Approximate policy improvement). For any  $t \geq 0$ , we know
$$
V ^ {\pi^ {t + 1}} (\rho) \geq V ^ {\pi^ {t}} (\rho) + \frac {\mathrm {K L} _ {d _ {\rho} ^ {\pi^ {t + 1} , \kappa_ {\pi^ {t + 1}}} (\pi^ {t} , \pi^ {t + 1})} + \mathrm {K L} _ {d _ {\rho} ^ {\pi^ {t + 1} , \kappa_ {\pi^ {t + 1}}} (\pi^ {t + 1} , \pi^ {t})}}{(1 - \gamma) \eta^ {t}} - \frac {\epsilon_ {t}}{1 - \gamma}, \tag {7}
$$
where  $\mathrm{KL}_{\nu}(\pi, \pi') \coloneqq \sum_{s} \nu(s) \mathrm{KL}(\pi(\cdot|s), \pi'(\cdot|s)) \geq 0$  and  $\mathbb{E}[\epsilon_t] = \tilde{O}\left(\frac{1}{\sqrt{N}} + \frac{1}{\sqrt{K}}\right) + O(\epsilon_{bias})$ .
We demonstrate the robustness of our RNAC approach (Algorithm 1) with Double-Sampling (DS) and IPM uncertainty sets on MuJoCo simulation environments [56]. We also perform real-world evaluations using TurtleBot [5], a mobile robot, on navigation tasks under action/policy perturbation. We implement a practical version RNAC using neural network function approximation, with the robust critic minimizing squared robust TD-error and the robust natural actor performing a robust proximal policy optimization (PPO) (see Algorithm 4 in Appendix A for details). We call this RNAC algorithm as RNAC-PPO and compare it with the canonical PPO algorithm [48]. Additional experimental results and details are deferred to Appendix A. We provide code with detailed instructions at https://github.com/tliu1997/RNAC.
7.1 MuJoCo Environments
We present the experimental results for perturbed MuJoCo Envs (Hopper-v3, Walker2d-v3 and HalfCheetah-v3) by changing their physical parameters (leg_joint_stiffness, foot_joint_stiffness and
back(actuator_range). We compare the performance of RNAC-PPO with that of the canonical PPO algorithm in Fig. 1, where the curves are averaged over 30 different seeded runs and the shaded region indicates the mean  $\pm$  standard deviation. RNAC-PPO and PPO are trained with data sampled from the nominal models (e.g., leg_joint_stiffness = 0.0, foot_joint_stiffness = 0.0, and back(actuator_range = 1.0).
DS Uncertainty Set: MuJoCo environments have deterministic models. However, most uncertainty sets are only reasonable for stochastic models, e.g.,  $f$ -divergence uncertainty sets. So, we add a uniform actuation noise  $\sim$  Unif[-5e-3, 5e-3] in constructing stochastic MuJoCo environments as in [65]. We use  $m = 2$  and radius  $\delta = 1/6$  for RNAC-PPO in (4). The choice of  $m = 2$  makes the training time of RNAC-PPO with DS uncertainty set and PPO comparable. Fig. 1a-1c demonstrate the robust performance of RNAC-PPO with Double-Sampling (DS) uncertainty sets in stochastic MuJoCo environments. Compared to PPO, the cumulative rewards of RNAC-PPO decay much slower as perturbations increase though they are slightly lower at the beginning (i.e., for the nominal model). The slight drop in initial performance may stem from optimizing the robust value function (1) under the worst-case transition models instead of the nominal models.
IPM Uncertainty Set: Since IPM with robust Bellman operator in (6) establishes robustness by negative regularization, it applies to environments with deterministic transition kernels. We select  $\delta = 10^{-5}$  in (6) and evaluate RNAC-PPO with IPM uncertainty sets on deterministic MuJoCo environments in Fig. 1d-1f. RNAC-PPO has more robust behaviors with slow cumulative reward decay as perturbations increase. Notably, RNAC-PPO enjoys similar and sometimes even better initial performance on the nominal model compared to PPO, which we believe is due to the regularization of neural network parameters suggested by IPM (6) that can potentially improve neural network training.
7.2 TurtleBot Experiments
We demonstrate the robustness of the policy learned by RNAC-PPO on a real-world mobile robot (Fig. 3). We consider a navigation task as illustrated in Fig. 2, where the goal of the policy is to navigate the TurtleBot from the origin  $(0,0)$  to a target region centered at  $(1, -1)$ .
DS Uncertainty Set: We train RNAC-PPO and PPO on a stochastic nominal model with balanced actuation noise [54], and test the learned policies in the nominal model (Fig. 2a) and an unbalanced perturbed model (Fig. 2b). The policies learned by RNAC-PPO can reach the target region in both the nominal model and the perturbed model, while policies learned by PPO are fragile to perturbation and may not reach the target, as shown in Fig. 2b.
IPM Uncertainty Set: The robustness of the policies learned by RNAC-PPO trained on a deterministic nominal model is demonstrated in Fig. 2c and 2d, where the RNAC-PPO learned policies drive the robot to the target under perturbation, while the PPO-learned policies fail.
A video of this real-world demonstration TurtleBot is available at [Video Link].
8 Conclusion and Future Works
We have proposed two novel uncertainty sets based on double sampling and an integral probability metric, respectively, that are compatible with function approximation for large-scale robust RL. We propose a robust natural actor-critic algorithm, which to the best of our knowledge is the first policy-based approach for robust RL under function approximation with provable guarantees on learning the optimal robust policy. We demonstrate the robust performance of the proposed algorithm in multiple perturbed MuJoCo environments and a real-world TurtleBot navigation task.
Although several new theoretical and empirical results about large-scale robust RL are presented in this paper, there are still many open questions that need to be addressed. Current work focuses on the  $(s,a)$ -rectangular RMDP (Def. 1). We leave extensions to more general  $s$ -rectangular and non-rectangular RMDP for future works. Some theoretical analysis in this paper partly relies on Assumption 2. Though it is commonly made and accepted in theoretical works as discussed in paragraphs after Assumption 2, it is not a necessary condition. Further exploration of this can potentially lead to more theoretical advances. Another natural future direction is to extend current results to more complex settings such as robust constrained RL and robust multi-agent RL, following recent developments of policy gradient-based approaches in safe RL [68, 32] and multi-agent RL [67, 50].
Dileep Kalathil's work is partially supported by the funding from the U.S. National Science Foundation (NSF) grant NSF-CAREER-EPCN-2045783.
P. R. Kumar's work is partially supported by the US Army Contracting Command under W911NF-22-1-0151 and W911NF2120064, US National Science Foundation under CMMI-2038625, and US Office of Naval Research under N00014-21-1-2385. The views expressed herein and conclusions contained in this document are those of the authors and should not be interpreted as representing the views or official policies, either expressed or implied, of the U.S. NSF, ONR, ARO, or the United States Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
Portions of this research were conducted with the advanced computing resources provided by Texas A&M High Performance Research Computing.
[1] Mohammed Amin Abdullah, Hang Ren, Haitham Bou Ammar, Vladimir Milenkovic, Rui Luo, Mingtian Zhang, and Jun Wang. Wasserstein robust reinforcement learning. arXiv preprint arXiv:1907.13196, 2019.
[2] Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, pages 10-4, 2019.
[3] Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. The Journal of Machine Learning Research, 22(1):4431-4506, 2021.
[4] Carlo Alfano and Patrick Rebeschini. Linear convergence for natural policy gradient with log-linear policy parametrization. arXiv preprint arXiv:2209.15382, 2022.
[5] Robin Amsters and Peter Slaets. Turtlebot 3 as a Robotics Education Platform, pages 170-181. 01 2020.
[6] Jose Blanchet, Miao Lu, Tong Zhang, and Han Zhong. Double pessimism is provably efficient for distributionally robust offline reinforcement learning: Generic algorithm and robust partial coverage. arXiv preprint arXiv:2305.09659, 2023.
[7] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
[8] Shicong Cen, Chen Cheng, Yuxin Chen, Yuting Wei, and Yuejie Chi. Fast global convergence of natural policy gradient methods with entropy regularization. Operations Research, 70(4):2563-2578, 2022.
[9] Zaiwei Chen, Sajad Khodadadian, and Siva Theja Maguluri. Finite-sample analysis of off-policy natural actor-critic with linear function approximation. IEEE Control Systems Letters, 6:2611-2616, 2022.
[10] Zaiwei Chen, Sheng Zhang, Thinh T Doan, John-Paul Clarke, and Siva Theja Maguluri. Finite-sample analysis of nonlinear stochastic approximation with applications in reinforcement learning. Automatica, 146:110623, 2022.
[11] Esther Derman, Daniel J Mankowitz, Timothy A Mann, and Shie Mannor. Soft-robust actor-critic policy-gradient. arXiv preprint arXiv:1803.04848, 2018.
[12] John C Duchi and Hongseok Namkoong. Learning models with uniform performance via distributionally robust optimization. The Annals of Statistics, 49(3):1378-1406, 2021.
[13] Benjamin Eysenbach and Sergey Levine. Maximum entropy rl (provably) solves some robust rl problems. In International Conference on Learning Representations.
[14] Herbert Federer. Geometric measure theory. Springer, 2014.
[15] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[16] Vineet Goyal and Julien Grand-Clement. Robust markov decision processes: Beyond rectangularity. Mathematics of Operations Research, 48(1):203-226, 2023.
[17] Julien Grand-Clement and Christian Kroer. Scalable first-order methods for robust mdps. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 12086-12094, 2021.
[18] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[19] Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2):257-280, 2005.
[20] Nick Jakobi, Phil Husbands, and Inman Harvey. Noise and the reality gap: The use of simulation in evolutionary robotics. In Advances in Artificial Life: Third European Conference on Artificial Life Granada, Spain, June 4–6, 1995 Proceedings 3, pages 704–720. Springer, 1995.
[21] Bai Jiang, Qiang Sun, and Jianqing Fan. Bernstein's inequality for general markov chains. arXiv preprint arXiv:1805.10721, 2018.
[22] Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory, pages 2137-2143. PMLR, 2020.
[23] Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, pages 267-274, 2002.
[24] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[25] A Ya Kruger. On fréchet subdifferentials. Journal of Mathematical Sciences, 116(3):3325-3358, 2003.