Title: Natural Actor-Critic for Robust Reinforcement Learning with Function Approximation
Venue: NeurIPS 2023
Accepted: yes
Keywords: robust reinforcement learning

policy-based approach

function approximation

actor-critic
Abstract: We study robust reinforcement learning (RL) with the goal of determining a well-performing policy that is robust against model mismatch between the training simulator and the testing environment. Previous policy-based robust RL algorithms mainly focus on the tabular setting under uncertainty sets that facilitate robust policy evaluation, but are no longer tractable when the number of states scales up. To this end, we propose two novel uncertainty set formulations, one based on double sampling and the other on an integral probability metric. Both make large-scale robust RL tractable even when one only has access to a simulator. We propose a robust natural actor-critic (RNAC) approach that incorporates the new uncertainty sets and employs function approximation. We provide finite-time convergence guarantees for the proposed RNAC algorithm to the optimal robust policy within the function approximation error. Finally, we demonstrate the robust performance of the policy learned by our proposed RNAC approach in multiple  MuJoCo environments and a real-world TurtleBot navigation task.

Natural Actor-Critic for Robust Reinforcement Learning with Function Approximation

Abstract

We study robust reinforcement learning (RL) with the goal of determining a wellperforming policy that is robust against model mismatch between the training simulator and the testing environment. Previous policy-based robust RL algorithms mainly focus on the tabular setting under uncertainty sets that facilitate robust policy evaluation, but are no longer tractable when the number of states scales up. To this end, we propose two novel uncertainty set formulations, one based on double sampling and the other on an integral probability metric. Both make large-scale robust RL tractable even when one only has access to a simulator. We propose a robust natural actor-critic (RNAC) approach that incorporates the new uncertainty sets and employs function approximation. We provide finite-time convergence guarantees for the proposed RNAC algorithm to the optimal robust policy within the function approximation error. Finally, we demonstrate the robust performance of the policy learned by our proposed RNAC approach in multiple MuJoCo environments and a real-world TurtleBot navigation task.

1 Introduction

Training a reinforcement learning (RL) algorithm directly on a real-world system is expensive and potentially risky due to the large number of data samples required to learn a satisfactory policy. To overcome this issue, RL algorithms are typically trained on a simulator. However, in most realworld applications, the nominal model used in the simulator model may not faithfully represent the real-world system due to various factors, such as approximation errors in modeling or variations in real-world parameters over time. For example, the mass, friction, sensor/actuator noise, and floor terrain in a mobile robot simulator may differ from those in the real world. This mismatch, called simulation-to-reality-gap, can significantly degrade the performance of standard RL algorithms when deployed on real-world systems [36, 44, 55, 51]. The framework of robust Markov decision process (RMDP) [19, 40] is used to model this setting where the testing environment is uncertain and comes from an uncertainty set around the nominal model. The optimal robust policy is defined as the one which achieves the optimal worst-case performance over all possible models in the uncertainty set. The goal of robust reinforcement learning is to learn such an optimal robust policy using only the data sampled from the simulator (nominal) model.

The RMDP planning problem has been well studied in the tabular setting [63, 62, 35, 46, 16]. Also, many works have developed model-based robust RL algorithms in the tabular setting [66, 69, 42,

64, 49], focusing on sample complexity. Many robust Q-learning algorithms [45, 33, 39, 59, 30] and policy gradient methods for robust RL [61, 27, 29, 58, 17] have been developed for the tabular setting . Different from all these works, the main goal of this paper is to develop a computationally tractable robust RL algorithm with provable convergence guarantees for RMDPs with large state spaces, using linear and nonlinear function approximation for robust value and policy.

One of the main challenges of robust RL with a large state space is the design of an effective uncertainty set that is amenable to computationally tractable learning with function approximation. Robust RL algorithms, both in their implementation and technical analysis, require robust Bellman operator evaluations [19, 40] which involve an inner optimization problem over the uncertainty set. Performing this inner optimization problem and/or getting an unbiased estimate of the robust Bellman operator evaluation using only the samples from the nominal model can be intractable for commonly considered uncertainty sets when the state space is large. For example, for f -divergencebased uncertainty sets [66, 69, 64, 49, 33, 30], the robust Bellman operator estimate requires solving for a dual variable associated with each state, which is prohibitive for large state spaces. For R -contamination [61], the estimate requires calculating the minimum of the value function over state space, which is impractical for large state spaces. It is also infeasible for ℓ p norm-based uncertainty sets [27], where the estimates require calculating the median, mean, or average peak of the value function depending on the choice of p . Robust RL with function approximation has been explored in a few works [53, 41, 45, 43, 60, 6]. However, these works explicitly or implicitly assume an oracle that approximately computes the robust Bellman operator estimate, and the uncertainty set design that facilitates computation and learning is largely ignored. We overcome this challenge by introducing two novel uncertainty set formulations, one based on double sampling (DS) and the other on an integral probability metric (IPM). Both are compatible with large-scale robust MDP, with the robust Bellman operators being amenable to unbiased estimation from the data sampled from the nominal model, enabling effective practical robust RL algorithms.

Policy-based RL algorithms [47, 31, 18, 15], which optimize the policy directly, have been extremely successful in learning policies for continuous control tasks. Value-based RL approaches, such as Qlearning, cannot be directly applied to continuous action problems. Moreover, recent advances in policy-based approaches can establish finite time convergence guarantees and also offer insights into the practical implementation [3, 37, 4, 8]. However, most of the existing works on robust RL with function approximation use value-based approaches [53, 41, 45, 43, 60, 34] that are not scalable to continuous control problems. On the other hand, the existing works that use policy-based methods for robust RL with convergence guarantees are limited to the tabular setting [61, 27, 29, 58, 17]. We close this important gap in the literature by developing a novel robust natural actor-critic (RNAC) approach that leverages our newly designed uncertainty sets for scalable learning.

SummaryofContributions: ( i ) . Wepropose two novel uncertainty sets, one using double sampling (Section 3.1) and the other an integral probability metric (Section 3.2), which are both compatible for robust RL with large state space and function approximation. Though robust Bellman operators require the unknown worst-case models, we provide unbiased empirical robust Bellman operators that are computationally easy to utilize and only based on samples from the nominal model;

( ii ) . We propose a novel RNAC algorithm (Section 4) which to the best of our knowledge is the first policy-based approach for robust RL under function approximation, with provable convergence guarantees. We consider both linear and general function approximation for theoretical study, with the latter relegated to Appendix E due to space constraints. Under linear function approximation, the RNAC with a robust critic performing 'robust linear-TD' (Section 5) and a robust natural actor performing 'robust Q-NPG' (Section 6) is proved to converge to the optimal robust policy within the function approximation error. Specifically, ˜ O (1 /ε 2 ) sample complexity can be achieved, or ˜ O (1 /ε 3 ) for policy update with a constant step size. For the robust linear-TD, we study the contraction behavior of the projected robust Bellman operator for RMDPs with the proposed uncertainty sets and the well-known f -divergence uncertainty sets, which we believe is of independent interest;

( iii ) . We implement the proposed RNAC in multiple MuJoCo environments (Hopper-v3, Walker2dv3, and HalfCheetah-v3), and demonstrate that RNAC with the proposed uncertainty sets results in robust behavior while canonical policy-based approaches suffer significant performance degradation. We also test RNAC on TurtleBot [5], a real-world mobile robot, performing a navigation task. We show that the TurtleBot with RNAC successfully reaches its destination while canonical non-

robust approaches fail under adversarial perturbations. A video of the demonstration on TurtleBot is available at [Video Link] and the RNAC code is provided in the supplementary material.

Due to the page limit, a detailed literature review and comparison of our work with the existing robust RL algorithms are deferred to Appendix G.

2 Preliminaries

Notations: For any set X , denote by |X| its cardinality, by ∆ X a ( |X| 1) -dimensional probability simplex, and by Unif ( X ) a uniform distribution over X . Let [ m ] := { 1 , . . . , m } .

A Markov decision process (MDP) is represented by a tuple ( S , A , κ, r, γ ) , where S is the state space, A is the action space, κ = ( κ 0 , κ 1 , . . . ) is a possibly non-stationary transition kernel sequence with κ t : S × A → ∆ S , r : S × A → [0 , 1] is the reward function, and γ ∈ (0 , 1) is the discount factor. For a stationary policy π : S → ∆ A , its value function is V π κ ( s ) := E κ,π [ ∑ ∞ t =0 γ t r ( s t , a t ) | s 0 = s ] , where the expectation is taken w.r.t. the trajectory ( s 0 , a 0 , s 1 , a 1 , . . . ) with a t | s t ∼ π s t and s t +1 | ( s t , a t ) ∼ κ t,s t ,a t . We can similarly define the state visitation distribution under initial distribution ρ ∈ ∆ S , d π,κ ρ ( s ) := (1 -γ ) E κ,π [ ∑ ∞ t =0 γ t ✶ ( s t = s ) | s 0 ∼ ρ ] ; the state-action value function (Q function), Q π κ ( s, a ) := E κ,π [ ∑ ∞ t =0 γ t r ( s t , a t ) | s 0 = s, a 0 = a ] , and the advantage function A π κ ( s, a ) := Q π κ ( s, a ) -V π κ ( s ) .

Robust MDP: A RMDP is represented by a tuple ( S , A , P , r, γ ) , where P is a set of transition kernels known as the uncertainty set that captures the perturbations around the nominal stationary kernel p ◦ : S × A → ∆ S , and its robust value function is defined as the corresponding worst case:

We will assume the following key ( s, a ) -rectangularity condition that is commonly assumed to facilitate dynamic programming ever since the introduction of RMDPs [19, 40]:

Definition 1. P is ( s, a ) -rectangular, if P = ⊗ s,a P s,a , for some P s,a ⊆ ∆ S .

The corresponding robust Bellman operator T π P : R S → R S is

and the Bellman equation for RMDPs is V = T π P V , where V = V π is its unique solution from the Banach fixed-point theorem. Typically, P s,a is taken as a ball { ν ⊆ ∆ S : d( ν, p ◦ s,a ) ≤ δ } around a nominal model p ◦ of the training environment, where d( · , · ) is some divergence measure between probability distributions, and δ > 0 controls the level of robustness.

There is a stationary optimal policy π ∗ that uniformly maximizes the robust value function, i.e., V π ∗ P ( s ) = sup { V π P ( s ) : history dependent π } , ∀ s [19, 40]. Thus, without loss of generality, we only need to optimize within stationary policies. Moreover, for any stationary policy π there exists a stationary worst-case kernel κ π with V π κ π ( s ) = V π P ( s ) , ∀ s . We can define the robust Q-function and robust advantage function as Q π P ( s, a ) := Q π κ π ( s, a ) and A π P ( s, a ) := A π κ π ( s, a ) , respectively. When the uncertainty set is clear from the context, we will omit the subscript P in V π P , Q π P , and A π P .

To summarize, the motivation in the robust RL framework is to learn the optimal robust policy by only training on a simulator implementing an (unknown) nominal model p ◦ [53, 45, 41], with the robust RL algorithms only having access to data generated from p ◦ , and not from any other model in the uncertainty set P .

3 Uncertainty Sets for Large State Spaces

Evaluating the robust Bellman operator T π P requires solving the optimization inf p ∈P s,a p ⊤ V , which is challenging for arbitrary uncertainty sets. Moreover, since we only have access to data generated by the nominal model p ◦ , estimating T π P is not straightforward. Previously studied uncertainty sets, such as R -contamination, f -divergence, and ℓ p norm, are intractable for RMDPs with large state spaces for these reasons (see Appendix B for detailed explanation). We therefore design two uncertainty sets for RMDPs where the robust Bellman operator has a tractable form and can be unbiasedly

estimated by the data sampled from the nominal model, thus making effective learning possible. We will also use function approximation to tractably parameterize policy and value functions.

3.1 Double-Sampling (DS) Uncertainty Set

The central difficulty in employing the robust Bellman operator (2) is how to evaluate inf p ∈P s,a p ⊤ V . Our first method is based on the following key idea for drawing samples that produce an unbiased estimate for it. Let { s ′ 1 , s ′ 2 , . . . , s ′ m } be m samples drawn i.i.d. according to the nominal model p ◦ s,a . Then, for any given divergence measure d( · , · ) and radius δ > 0 , there exists an uncertainty set P = ⊗ s,a P s,a such that inf p ∈P s,a p ⊤ V = E s ′ 1: m i.i.d. ∼ p ◦ s,a [ inf α ∈ ∆ [ m ] :d( α, Unif ([ m ])) ≤ δ ∑ m i =1 α i V ( s ′ i ) ] . (See Appendix B.1 for a brief explana- tion.) We therefore define an empirical robust Bellman operator ˆ T π P corresponding to (2) by

Since ( T π P V )( s ) = E a ∼ π ( ·| s ) , s ′ 1: m i.i.d. ∼ p ◦ s,a [ ( ˆ T π P V )( s, a, s ′ 1: m ) ] , ˆ T π P gives an unbiased estimate, which is one key property we use in our robust RL algorithm. We refer to this as 'double sampling' since ∑ m i =1 α i V ( s ′ i ) is the expected value when one further chooses one sample s ′ from { s ′ 1 , s ′ 2 , . . . , s ′ m } according to the distribution α on [ m ] and evaluates V ( s ′ ) . Above, α is a perturbation of Unif ([ m ]) and when δ = 0 , α = Unif ([ m ]) and s ′ ∼ p ◦ s,a . The uncertainty set P corresponding to the double sampling is implicitly defined by specifying the choices of m , d( · , · ) , and δ . Its key advantage is that samples can only be drawn according to the nominal model p ◦ .

Double sampling requires sampling multiple next states for a given state-action pair, which can be implemented if the simulator is allowed to set to any state. (All MuJoCo environments [56] support DS.) Since the calculation of α is within ∆ [ m ] , the empirical robust Bellman operator is tractable for moderate values of m . We use m = 2 in experiments for training efficiency, where for almost all divergences d( · , · ) we can explicitly write

Bellman completeness (value function class closed under the Bellman operator) is a key property for efficient reinforcement learning, whereas RMDP with canonical uncertainty sets may violate it. We show in Appendix B.1 that for RMDP with DS uncertainty sets, the linear function approximation class satisfies Bellman completeness if the nominal model is a linear MDP [22].

3.2 Integral Probability Metric (IPM) Uncertainty Set

Given some function class F ⊂ R S including the zero function, the integral probability metric (IPM) is defined by d F ( p, q ) := sup f ∈F { p ⊤ f -q ⊤ f } ≥ 0 [38]. Many metrics such as Kantorovich metric, total variation, etc., are special cases of IPM under different function classes [38].

The robust Bellman operator T π P V (2) requires solving the optimization inf q ∈P s,a q ⊤ V . For an IPM-based uncertainty set P with P s,a = { q : d F ( q, p ◦ s,a ) ≤ δ } , we relax the domain q ∈ ∆ S to ∑ s q ( s ) = 1 as done in [28, 27], which incurs no relaxation error for small δ if min s ′ p ◦ s,a ( s ′ ) > 0 .

One should choose F so that it properly encodes information of the MDP and its value functions. We start by considering linear function approximation. Denote by Ψ ∈ R S× d the feature matrix with rows ψ ⊤ ( s ) , ∀ s ∈ S . The value function approximation is V w = Ψ w ∈ R S . A good choice of feature vectors encodes information about the state and transition. For example, vectors ψ ( s ) , ψ ( s ′ ) should be 'close' when states s, s ′ are 'similar' and V π ( s ) , V π ( s ′ ) have small differences. We propose the following function class (with ℓ 2 as the preferred norm but any other can be used):

Without loss of generality, assume Ψ has full column rank since d ≪|S| , and let the first coordinate of ψ ( s ) be 1 for any s , which corresponds to the bias term of the linear regressor.

Proposition 1. For the IPM with F in (5), we have inf q ∈P s,a q ⊤ V w = ( p ◦ s,a ) ⊤ V w -δ ∥ w 2: d ∥ .

We can thus design the empirical robust Bellman operator, which is an unbiased estimator of T π P with sample s ′ drawn from the nominal model:

Guided by the last regularization term of the empirical robust Bellman operator (6), when considering value function approximation by neural networks we add a similar negative regularization term for all the neural network parameters except for the bias parameter in the last layer.

4 Robust Natural Actor-Critic

Wepropose a robust natural actor-critic (RNAC) approach in Algorithm 1 for the robust RL problem. As its name suggests, there are two components - a robust critic and a robust actor, which update alternately for T steps. At each step t , there is a policy π t determined by parameter θ t . The robust critic updates the value function approximation parameter w t based on on-policy trajectory data sampled by executing π t on the nominal model with length K . The robust actor then updates the policy with step size η t , and the critic returns w t by on-policy trajectory data with length N . In practice, a batch of on-policy data can be sampled and used for both critic and actor updates.

We now give the main (informal) convergence results for our RNAC algorithm with linear function approximation and DS or IPM uncertainty sets, where the robust critic performs robust linear-TD and the robust natural actor performs robust-QNPG (as the comments in Algorithm 1). The formal statements, proofs, and generalization to general function approximation are given in Appendix E.

Theorem 1 (Informal linear convergence of RNAC) . Under linear function approximation, RNAC in Algorithm 1 with DS or IPM uncertainty sets using an RLTD robust critic update and an RQNPG robust natural actor update, with appropriate geometrically increasing step sizes η t , achieves E [ V π ∗ ( ρ ) -V π T ( ρ )] = O ( e -T ) + O ( ϵ stat ) + O ( ϵ bias ) and an ˜ O (1 /ε 2 ) sample complexity.

The optimality gap is bounded in this theorem via three terms, where the first term, related to the number of time steps T , is the optimization rate (linear convergence since O ( e -T ) ), the second term ϵ stat = ˜ O ( 1 √ N + 1 √ K ) is a statistical error that depends on the number of samples K,N in the robust critic and robust actor updates, and the last term ϵ bias is the approximation error due to the limited representation power of value function approximation and the parameterized policy class. Omitting the approximation error ϵ bias , the sample complexities for achieving ε robust optimal value are ˜ O (1 /ε 2 ) , which achieves the optimal sample complexity in tabular setting [64]. However, RNAC with geometrically increasing step sizes induces a larger multiplicative constant factor (not shown in bigO notation) and does not generalize well to general function approximation. We then analyze RNAC with a constant step size.

Theorem 2 (Informal sublinear convergence of RNAC) . RNAC under the same specification as in Theorem 1 but with constant step size η t = η has E [ V π ∗ ( ρ ) -1 T ∑ T -1 t =0 V π t ( ρ )] = O ( 1 T ) + O ( ϵ stat ) + O ( ϵ bias ) , implying an ˜ O (1 /ε 3 ) sample complexity.

Although the theorem shows a slower optimization rate of RNAC with constant step size, this nonincreasing step size is preferred in practice. Moreover, the analysis can be generalized to a general policy class with optimization rate O (1 / √ T ) , and an ˜ O (1 /ε 4 ) sample complexity.

Algorithm 2: Robust Linear Temporal Difference (RLTD)

5 Robust Critic

The robust critic estimates the robust value function with access to samples from the nominal model. One may note that in many previous actor-critic analyses for canonical RL [61, 29, 9], the critic learns the Q function, while realistic implementations in on-policy algorithms (e.g., proximal policy optimization (PPO)) treat the V function as the target of the critic for training efficiency. We consider a robust critic that learns the robust V function to align with such realistic implementations.

We present the robust linear temporal difference (RLTD) Algorithm 2, which is similar to the canonical linear-TD algorithm, but with an empirical robust Bellman operator. It iteratively performs the sampling and updating procedures. The sampling procedure differs for the uncertainty set by double sampling and IPM as shown in the comments in Algorithm 2. Using the samples, the parameter for the linear value function approximation V w k = Ψ w k is updated with step size α k . The following assumption is common [10, 9, 29]:

Assumption 1 (Geometric mixing) . For any policy π , the Markov chain { s k } induced by applying π in the nominal model p ◦ is geometrically ergodic with a unique stationary distribution ν π .

The update procedure essentially minimizes the Mean Square Projected Robust Bellman Error MSPRBE π ( w ) = ∥ Π π ( T π P V w ) -V w ∥ 2 ν π , where ∥ V ∥ ν π = √∑ s ν π ( s ) V ( s ) 2 is the ν π weighted norm, and Π π = Ψ(Ψ ⊤ D π Ψ) -1 Ψ ⊤ D π is the weighted projection matrix with D π = diag ( ν π ) ∈ R S×S . The minimizer of MSPRBE π ( w ) , denoted by w π , is the unique solution of the projected robust Bellman equation V w = Π π T π P V w , which is equivalent to 0 = Ψ ⊤ D π ( T π P Ψ w -Ψ w ) . RLTD is thus a stochastic approximation algorithm since the empirical operator ˆ T π P ((3) or (6)) is unbiased with E ( s,a,y ′ ) ∼ ν π ◦ π ◦ p ◦ [ ψ ( s ) [ ( ˆ T π P V w )( s, a, y ′ ) -ψ ( s ) ⊤ w ]] = Ψ ⊤ D π ( T π P V w -V w ) .

To ensure that RLTD converges to the optimal linear approximation V w π , it is crucial that the projected robust Bellman operator Π π T π P be a contraction map with some β < 1 .

Definition 2. Π π T π P is a β -contraction w.r.t. ∥ · ∥ ν π if ∥ Π π T π P V -Π π T π P V ′ ∥ ν π ≤ β ∥ V -V ′ ∥ ν π .

Unlike in linear TD for MDP [57], Tamar et al. [53] make an additional assumption (Assumption 2) to establish the contraction property of Π π T π P (Proposition 3) for RMDP.

′ ◦ ′ ′

Assumption 2. There exists β ∈ (0 , 1) with γp s,a ( s ) ≤ βp s,a ( s ) ∀ s, s ∈ S , ∀ a ∈ A , and ∀ p ∈ P . Proposition 2 (Prop.3 in [53]) . Under Assumption 2, Π π T π P is a β -contraction w.r.t. ∥ · ∥ v π .

The implicit uncertainty set P of double-sampling satisfies Assumption 2 for small δ , and thus guarantees contraction of Π π T π P . For example, for m = 2 as in (4), a δ < 1 -γ 2 γ is sufficient. However, simply taking a small radius δ is not a panacea for all uncertainty sets:

Proposition 3. For any f -divergence and radius δ > 0 , there exists a geometrically mixing nominal model such that the f -divergence defined uncertainty set violates Assumption 2.

On the other hand, Assumption 2, though well-accepted [53, 45, 41], may not be necessary. The proposed IPM uncertainty set relates robustness and regularization with an explicit formula for robust Bellman operator as in (6). The contraction behavior of Π π T π P for IPM uncertainty set can be established without Assumption 2:

Lemma 1. For IPM uncertainty set with radius δ < λ min (Ψ ⊤ D π Ψ) 1 -γ γ , there exists β < 1 that Π π T π P is a β -contraction mapping w.r.t. norm ∥ · ∥ v π .

Since the contraction of Π π T π P is obtained by RMDP under DS or IPM uncertainty sets with small radius δ , we have the first finite sample guarantee of RLTD by recent advances in Markovian stochastic approximation [10]:

Theorem 3 (Informal convergence of robust critic: Details in Appendix C) . RLTD with step sizes α k = Θ(1 /k ) satisfies E [ ∥ w K -w π ∥ 2 ] = ˜ O ( 1 K ) .

6 Robust Natural Actor

The robust natural actor updates the policy parameter θ along an ascent direction that improves the value via preconditioning through the KL-divergence KL( p, q ) := ⟨ p, log( p/q ) ⟩ . It has been well explored for natural policy gradient (NPG)-like algorithms, such as TRPO and PPO in canonical RL. The ascent direction is obtained by the policy gradient theorem in canonical MDP. We therefore first discuss the policy gradient for RMDP, where policy π θ is differentiably parameterized by θ .

The robust value V π θ P ( ρ ) =: J ( θ ) is typically Lipschitz under proper parameterization [61], and is therefore differentiable a.e. by Rademacher's theorem [14]. Where it is not differentiable, a Fr´ echet supergradient ∇ J of J exists if lim sup θ ′ → 0 J ( θ ′ ) -J ( θ ) -⟨∇ J ( θ ) ,θ ′ -θ ⟩ ∥ θ ′ -θ ∥ ≤ 0 [25]. The following contains the policy gradient theorem for canonical RL as a special case:

Lemma 2 (Policy supergradient) . For a policy π = π θ that is differentiable w.r.t. parameter θ ,

is a Fr´ echet supergradient of V π ( ρ ) , where κ π is the worst-case transition kernel w.r.t. π .

We consider log-linear policies π θ ( a | s ) = exp ( ϕ ( s,a ) ⊤ θ ) ∑ a ′ exp( ϕ ( s,a ′ ) ⊤ θ ) , where ϕ ( s, a ) ∈ R d is the feature vector and θ ∈ R d is the policy parameter. (The general policy class is treated in Appendix D).

In canonical RL, the training and testing environments follow the same nominal transition p ◦ . NPG updates the policy by θ ← θ + η F ρ ( θ ) † ∇ θ V π θ p ◦ ( ρ ) , where F ρ ( θ ) † is the Moore-Penrose inverse of the Fisher information matrix F ρ ( θ ) := E ( s,a ) ∼ d π θ ,p ◦ ρ ◦ π θ [ ∇ θ log π θ ( a | s ) ( ∇ θ log π θ ( a | s )) ⊤ ] . An 'equivalent' Q-NPG was proposed in [3] to update the policy by θ ← θ + ηu ′ , where u ′ = arg min u E ( s,a ) ∼ d π,p ◦ ρ ◦ π [( Q π p ◦ ( s, a ) -u ⊤ ϕ ( s, a )) 2 ] . Note that u determines a Q value function approximation Q u ( s, a ) := ϕ ( s, a ) ⊤ u , which is compatible with the log-linear policy class [52]. Since Q π contains the information on the ascent direction as suggested by Lemma 2, the Q-NPG update can be viewed as inserting the best compatible Q-approximation Q u ′ for the policy.

We adopt the Q-NPG to robust RL and propose the Robust Q-Natural Policy Gradient (RQNPG) (Algorithm 3). Note that unlike canonical RL, where Q π p ◦ can be estimated directly from a sample trajectory of executing π in model p ◦ , the robust Q π is hard to estimate with samples from p ◦ . A value function approximation V w from the critic comes to help, as we can approximate the robust Q function via Q w ( s, a ) = r ( s, a ) + inf p ∈P s,a p ⊤ V w , which exactly matches Q π if V w = V π . The RQNPG obtains information on Q π by first approximating it via a critic value V w -guided function Q w , and then estimating Q w by a policy-compatible robust Q-approximation Q u .

As shown in Algorithm 3, RQNPG estimates a compatible Q-approximation Q u N by iteratively performing sampling and updating procedures, where the sampling procedure follows that of the RLTD (Algorithm 2). The update procedure essentially approaches the minimizer u π w := arg min u E ( s,a ) ∼ ν π ◦ π [ ( Q w ( s, a ) -u ⊤ ϕ ( s, a )) 2 ] by stochastic approximation with step size ζ n (stochastic gradient descent with Markovian data), since conditioned on ( s, a ) , ˆ T π P V w ((3) or (6)) is an unbiased estimator for the Q function approximation Q w .

Now we look at a specific update θ t +1 = RQNPG ( θ t , η t , w t , N ) with ζ n = Θ(1 /n ) , where w t = RLTD ( π θ t , K ) . The following theorem shows an approximate policy improvement property of the RQNPG update:

Algorithm 3: Robust Q-Natural Policy Gradient (RQNPG)

Return: θ + ηu N

0

10

20

30

40

50

'leg_joint_stiffness' values (default=0.0)

500

1000

1500

2000

2500

3000

3500

4000

Cumulative reward

RNAC-PPO(DS)

PPO

(a) Double sampling, Hopper

0

10

20

30

40

'foot_joint_stiffness' values (default=0.0)

2500

3000

3500

4000

4500

5000

Cumulative reward

RNAC-PPO(DS)

PPO

(b) Double sampling, Walker

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Bound on back actuator range (default=1.0)

500

0

500

1000

1500

2000

Cumulative reward

RNAC-PPO(DS)

PPO

(c) Double sampling, HalfCheetah

0

10

20

30

40

50

'leg_joint_stiffness' values (default=0.0)

1000

1500

2000

2500

3000

3500

Cumulative reward

RNAC-PPO(IPM)

PPO

(d) IPM, Hopper

0

10

20

30

40

'foot_joint_stiffness' values (default=0.0)

2500

3000

3500

4000

4500

5000

5500

Cumulative reward

RNAC-PPO(IPM)

PPO

(e) IPM, Walker

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Bound on back actuator range (default=1.0)

500

0

500

1000

1500

2000

2500

Cumulative reward

RNAC-PPO(IPM)

PPO

(f) IPM, HalfCheetah

Theorem 4 (Approximate policy improvement) . For any t ≥ 0 , we know

7 Experimental Results

We demonstrate the robustness of our RNAC approach (Algorithm 1) with Double-Sampling (DS) and IPM uncertainty sets on MuJoCo simulation environments [56]. We also perform real-world evaluations using TurtleBot [5], a mobile robot, on navigation tasks under action/policy perturbation. We implement a practical version RNAC using neural network function approximation, with the robust critic minimizing squared robust TD-error and the robust natural actor performing a robust proximal policy optimization (PPO) (see Algorithm 4 in Appendix A for details). We call this RNAC algorithm as RNAC-PPO and compare it with the canonical PPO algorithm [48]. Additional experimental results and details are deferred to Appendix A. We provide code with detailed instructions at https://github.com/tliu1997/RNAC.

7.1 MuJoCo Environments

We present the experimental results for perturbed MuJoCo Envs (Hopper-v3, Walker2d-v3 and HalfCheetah-v3) by changing their physical parameters ( leg joint stiffness , foot joint stiffness and

0.0

0.5

1.0

1.5

2.0

TurtleBot position x

0.0

0.5

1.0

1.5

2.0

TurtleBot position y

RNAC-PPO(DS)

PPO

target

(

-

0

.

5

,

0

.

5)

0.0

0.5

1.0

1.5

2.0

TurtleBot position x

0.0

0.5

1.0

1.5

2.0

TurtleBot position y

RNAC-PPO(DS)

PPO

target

(1

,

1

.

5)

0.0

0.5

1.0

1.5

2.0

TurtleBot position x

0.0

0.5

1.0

1.5

2.0

TurtleBot position y

RNAC-PPO(IPM)

PPO

target

(c) IPM, deterministic

0.0

0.5

1.0

1.5

2.0

TurtleBot position x

0.0

0.5

1.0

1.5

2.0

TurtleBot position y

RNAC-PPO(IPM)

PPO

target

(1

,

1

.

5)

back actuator range ). We compare the performance of RNAC-PPO with that of the canonical PPO algorithm in Fig. 1, where the curves are averaged over 30 different seeded runs and the shaded region indicates the mean ± standard deviation. RNAC-PPO and PPO are trained with data sampled from the nominal models (e.g., leg joint stiffness = 0 . 0 , foot joint stiffness = 0 . 0 , and back actuator range = 1 . 0 ).

DS Uncertainty Set: MuJoCo environments have deterministic models. However, most uncertainty sets are only reasonable for stochastic models, e.g., f -divergence uncertainty sets. So, we add a uniform actuation noise ∼ Unif[-5e-3, 5e-3] in constructing stochastic MuJoCo environments as in [65]. We use m = 2 and radius δ = 1 / 6 for RNAC-PPO in (4). The choice of m = 2 makes the training time of RNAC-PPO with DS uncertainty set and PPO comparable. Fig. 1a-1c demonstrate the robust performance of RNAC-PPO with Double-Sampling (DS) uncertainty sets in stochastic MuJoCo environments. Compared to PPO, the cumulative rewards of RNAC-PPO decay much slower as perturbations increase though they are slightly lower at the beginning (i.e., for the nominal model). The slight drop in initial performance may stem from optimizing the robust value function (1) under the worst-case transition models instead of the nominal models.

IPM Uncertainty Set: Since IPM with robust Bellman operator in (6) establishes robustness by negative regularization, it applies to environments with deterministic transition kernels. We select δ = 10 -5 in (6) and evaluate RNAC-PPO with IPM uncertainty sets on deterministic MuJoCo environments in Fig. 1d-1f. RNAC-PPO has more robust behaviors with slow cumulative reward decay as perturbations increase. Notably, RNAC-PPO enjoys similar and sometimes even better initial performance on the nominal model compared to PPO, which we believe is due to the regularization of neural network parameters suggested by IPM (6) that can potentially improve neural network training.

7.2 TurtleBot Experiments

We demonstrate the robustness of the policy learned by RNAC-PPO on a real-world mobile robot (Fig. 3). We consider a navigation task as illustrated in Fig. 2, where the goal of the policy is to navigate the TurtleBot from the origin (0 , 0) to a target region centered at (1 , -1) .

DS Uncertainty Set: We train RNAC-PPO and PPO on a stochastic nominal model with balanced actuation noise [54], and test the learned policies in the nominal model (Fig. 2a) and an unbalanced perturbed model (Fig. 2b). The policies learned by RNAC-PPO can reach the target region in both the nominal model and the perturbed model, while policies learned by PPO are fragile to perturbation and may not reach the target, as shown in Fig. 2b.

IPM Uncertainty Set: The robustness of the policies learned by RNACPPO trained on a deterministic nominal model is demonstrated in Fig. 2c and 2d, where the RNAC-PPO learned policies drive the robot to the target under perturbation, while the PPO-learned policies fail.

A video of this real-world demonstration TurtleBot is available at [Video Link] .

8 Conclusion and Future Works

We have proposed two novel uncertainty sets based on double sampling and an integral probability metric, respectively, that are compatible with function approximation for large-scale robust RL. We propose a robust natural actor-critic algorithm, which to the best of our knowledge is the first policybased approach for robust RL under function approximation with provable guarantees on learning the optimal robust policy. We demonstrate the robust performance of the proposed algorithm in multiple perturbed MuJoCo environments and a real-world TurtleBot navigation task.

Although several new theoretical and empirical results about large-scale robust RL are presented in this paper, there are still many open questions that need to be addressed. Current work focuses on the ( s, a ) -rectangular RMDP (Def. 1). We leave extensions to more general s -rectangular and non-rectangular RMDP for future works. Some theoretical analysis in this paper partly relies on Assumption 2. Though it is commonly made and accepted in theoretical works as discussed in paragraphs after Assumption 2, it is not a necessary condition. Further exploration of this can potentially lead to more theoretical advances. Another natural future direction is to extend current results to more complex settings such as robust constrained RL and robust multi-agent RL, following recent developments of policy gradient-based approaches in safe RL [68, 32] and multi-agent RL [67, 50].

References

Mohammed Amin Abdullah, Hang Ren, Haitham Bou Ammar, Vladimir Milenkovic, Rui Luo, Mingtian Zhang, and Jun Wang. Wasserstein robust reinforcement learning. arXiv preprint arXiv:1907.13196 , 2019.

Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep , pages 10-4, 2019.

Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. The Journal of Machine Learning Research , 22(1):4431-4506, 2021.

Carlo Alfano and Patrick Rebeschini. Linear convergence for natural policy gradient with log-linear policy parametrization. arXiv preprint arXiv:2209.15382 , 2022.

Robin Amsters and Peter Slaets. Turtlebot 3 as a Robotics Education Platform , pages 170-181. 01 2020.

Jose Blanchet, Miao Lu, Tong Zhang, and Han Zhong. Double pessimism is provably efficient for distributionally robust offline reinforcement learning: Generic algorithm and robust partial coverage. arXiv preprint arXiv:2305.09659 , 2023.

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 , 2016.

Shicong Cen, Chen Cheng, Yuxin Chen, Yuting Wei, and Yuejie Chi. Fast global convergence of natural policy gradient methods with entropy regularization. Operations Research , 70(4):2563-2578, 2022.

Zaiwei Chen, Sajad Khodadadian, and Siva Theja Maguluri. Finite-sample analysis of offpolicy natural actor-critic with linear function approximation. IEEE Control Systems Letters , 6:2611-2616, 2022.

Zaiwei Chen, Sheng Zhang, Thinh T Doan, John-Paul Clarke, and Siva Theja Maguluri. Finite-sample analysis of nonlinear stochastic approximation with applications in reinforcement learning. Automatica , 146:110623, 2022.

Esther Derman, Daniel J Mankowitz, Timothy A Mann, and Shie Mannor. Soft-robust actorcritic policy-gradient. arXiv preprint arXiv:1803.04848 , 2018.

John C Duchi and Hongseok Namkoong. Learning models with uniform performance via distributionally robust optimization. The Annals of Statistics , 49(3):1378-1406, 2021.

Benjamin Eysenbach and Sergey Levine. Maximum entropy rl (provably) solves some robust rl problems. In International Conference on Learning Representations .

Herbert Federer. Geometric measure theory . Springer, 2014.

Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning , pages 1587-1596. PMLR, 2018.

Vineet Goyal and Julien Grand-Clement. Robust markov decision processes: Beyond rectangularity. Mathematics of Operations Research , 48(1):203-226, 2023.

Julien Grand-Cl´ ement and Christian Kroer. Scalable first-order methods for robust mdps. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 12086-12094, 2021.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research , 30(2):257-280, 2005.

Nick Jakobi, Phil Husbands, and Inman Harvey. Noise and the reality gap: The use of simulation in evolutionary robotics. In Advances in Artificial Life: Third European Conference on Artificial Life Granada, Spain, June 4-6, 1995 Proceedings 3 , pages 704-720. Springer, 1995.

Bai Jiang, Qiang Sun, and Jianqing Fan. Bernstein's inequality for general markov chains. arXiv preprint arXiv:1805.10721 , 2018.

Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory , pages 21372143. PMLR, 2020.

Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning , pages 267-274, 2002.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

A Ya Kruger. On fr´ echet subdifferentials. Journal of Mathematical Sciences , 116(3):33253358, 2003.

Yufei Kuang, Miao Lu, Jie Wang, Qi Zhou, Bin Li, and Houqiang Li. Learning robust policy against disturbance in transition dynamics via state-conservative policy optimization. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 7247-7254, 2022.

Navdeep Kumar, Esther Derman, Matthieu Geist, Kfir Levy, and Shie Mannor. Policy gradient for s-rectangular robust markov decision processes. arXiv preprint arXiv:2301.13589 , 2023.

Navdeep Kumar, Kfir Levy, Kaixin Wang, and Shie Mannor. Efficient policy iteration for robust markov decision processes via regularization. arXiv preprint arXiv:2205.14327 , 2022.

Yan Li, Tuo Zhao, and Guanghui Lan. First-order policy optimization for robust markov decision process. arXiv preprint arXiv:2209.10579 , 2022.

Zhipeng Liang, Xiaoteng Ma, Jose Blanchet, Jiheng Zhang, and Zhengyuan Zhou. Singletrajectory distributionally robust reinforcement learning. arXiv preprint arXiv:2301.11721 , 2023.

Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 , 2015.

Tao Liu, Ruida Zhou, Dileep Kalathil, PR Kumar, and Chao Tian. Policy optimization for constrained mdps with provable fast global convergence. arXiv preprint arXiv:2111.00552 , 2021.

Zijian Liu, Qinxun Bai, Jose Blanchet, Perry Dong, Wei Xu, Zhengqing Zhou, and Zhengyuan Zhou. Distributionally robust q -learning. In International Conference on Machine Learning , pages 13623-13643. PMLR, 2022.

Xiaoteng Ma, Zhipeng Liang, Li Xia, Jiheng Zhang, Jose Blanchet, Mingwen Liu, Qianchuan Zhao, and Zhengyuan Zhou. Distributionally robust offline reinforcement learning with linear function approximation. arXiv preprint arXiv:2209.06620 , 2022.

Shie Mannor, Ofir Mebel, and Huan Xu. Robust mdps with k-rectangular uncertainty. Mathematics of Operations Research , 41(4):1484-1509, 2016.

Shie Mannor, Duncan Simester, Peng Sun, and John N Tsitsiklis. Bias and variance approximation in value function estimates. Management Science , 53(2):308-322, 2007.

Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. In International Conference on Machine Learning , pages 6820-6829. PMLR, 2020.

Alfred M¨ uller. Integral probability metrics and their generating classes of functions. Advances in applied probability , 29(2):429-443, 1997.

Ariel Neufeld and Julian Sester. Robust q -learning algorithm for markov decision processes under wasserstein uncertainty. arXiv preprint arXiv:2210.00898 , 2022.

Arnab Nilim and Laurent El Ghaoui. Robust control of markov decision processes with uncertain transition matrices. Operations Research , 53(5):780-798, 2005.

Kishan Panaganti and Dileep Kalathil. Robust reinforcement learning using least squares policy iteration with provable performance guarantees. In International Conference on Machine Learning , pages 511-520. PMLR, 2021.

Kishan Panaganti and Dileep Kalathil. Sample complexity of robust reinforcement learning with a generative model. In International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 9582-9602, 2022.

Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, and Mohammad Ghavamzadeh. Robust reinforcement learning using offline data. In Advances in Neural Information Processing Systems , pages 32211-32224, 2022.

Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA) , pages 3803-3810. IEEE, 2018.

Aurko Roy, Huan Xu, and Sebastian Pokutta. Reinforcement learning under model mismatch. Advances in neural information processing systems , 30, 2017.

Reazul Hasan Russel and Marek Petrik. Beyond confidence regions: Tight bayesian ambiguity sets for robust mdps. Advances in Neural Information Processing Systems , 2019.

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning , pages 18891897. PMLR, 2015.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Laixi Shi and Yuejie Chi. Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity. arXiv preprint arXiv:2208.05767 , 2022.

Youbang Sun, Tao Liu, Ruida Zhou, PR Kumar, and Shahin Shahrampour. Provably fast convergence of independent natural policy gradient for markov potential games. arXiv preprint arXiv:2310.09727 , 2023.

Niko S¨ underhauf, Oliver Brock, Walter Scheirer, Raia Hadsell, Dieter Fox, J¨ urgen Leitner, Ben Upcroft, Pieter Abbeel, Wolfram Burgard, Michael Milford, et al. The limits and potentials of deep learning for robotics. The International journal of robotics research , 37(4-5):405-420, 2018.

Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems , 12, 1999.

Aviv Tamar, Shie Mannor, and Huan Xu. Scaling up robust mdps using function approximation. In International conference on machine learning , pages 181-189. PMLR, 2014.

Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applications in continuous control. In International Conference on Machine Learning , pages 6215-6224. PMLR, 2019.

Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS) , pages 23-30. IEEE, 2017.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems , pages 5026-5033. IEEE, 2012.

John Tsitsiklis and Benjamin Van Roy. Analysis of temporal-diffference learning with function approximation. Advances in neural information processing systems , 9, 1996.

Qiuhao Wang, Chin Pang Ho, and Marek Petrik. On the convergence of policy gradient in robust mdps. arXiv preprint arXiv:2212.10439 , 2022.

Shengbo Wang, Nian Si, Jose Blanchet, and Zhengyuan Zhou. A finite sample complexity bound for distributionally robust q-learning. In International Conference on Artificial Intelligence and Statistics , pages 3370-3398. PMLR, 2023.

Yue Wang and Shaofeng Zou. Online robust reinforcement learning with model uncertainty. Advances in Neural Information Processing Systems , 34:7193-7206, 2021.

Yue Wang and Shaofeng Zou. Policy gradient method for robust reinforcement learning. In International Conference on Machine Learning , pages 23484-23526. PMLR, 2022.

Wolfram Wiesemann, Daniel Kuhn, and Berc ¸ Rustem. Robust Markov decision processes. Mathematics of Operations Research , 38(1):153-183, 2013.

Huan Xu and Shie Mannor. Distributionally robust Markov decision processes. In Advances in Neural Information Processing Systems , pages 2505-2513, 2010.

Zaiyan Xu, Kishan Panaganti, and Dileep Kalathil. Improved sample complexity bounds for distributionally robust reinforcement learning. In International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 9728-9754, 2023.

Mengjiao Yang, Dale Schuurmans, Pieter Abbeel, and Ofir Nachum. Dichotomy of control: Separating what you can control from what you cannot. arXiv preprint arXiv:2210.13435 , 2022.

Wenhao Yang, Liangyu Zhang, and Zhihua Zhang. Toward theoretical understandings of robust markov decision processes: Sample complexity and asymptotics. The Annals of Statistics , 50(6):3223-3248, 2022.

Runyu Zhang, Jincheng Mei, Bo Dai, Dale Schuurmans, and Na Li. On the global convergence rates of decentralized softmax gradient play in markov potential games. Advances in Neural Information Processing Systems , 35:1923-1935, 2022.

Ruida Zhou, Tao Liu, Dileep Kalathil, PR Kumar, and Chao Tian. Anchor-changing regularized natural policy gradient for multi-objective reinforcement learning. Advances in Neural Information Processing Systems , 2022.

Zhengqing Zhou, Qinxun Bai, Zhengyuan Zhou, Linhai Qiu, Jose Blanchet, and Peter Glynn. Finite-sample regret bound for distributionally robust offline tabular reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pages 3331-3339, 2021.

A Experimental Details and Additional Experimental Results

In this section, we provide details of the RNAC algorithm (Algorithm 1) implemented in the experiments - robust natural actor-critic proximal policy optimization (RNAC-PPO) algorithm (Algorithm 4). We also demonstrate further experimental results evaluated on different perturbations of physical hyperparameters in Hopper-v3, Walker2d-v3, and HalfCheetah-v3 from OpenAI Gym [7] compared with soft actor-critic (SAC) [18] and soft-robust [11] PPO (SRPPO). Finally, we introduce experimental details of the TurtleBot navigation task.

A.1 RNAC-PPO Algorithm

We provide the RNAC-PPO algorithm in Algorithm 4, where the robust critic is minimizing squared robust TD error (MSRTDE) and the robust natural actor is performing the clipped version of robust PPO(RPPO) for computational efficiency, and we name it RNAC-PPO for simplicity. We implement RNAC-PPO employing neural network (NN) function approximation, where the robust natural actor is utilizing a neural Gaussian policy [48] with two hidden layers of width 64, and the value function in the robust critic is also parameterized by an NN with two hidden layers of width 64. Compared with the canonical empirical Bellman operator in the PPO algorithm, we adopt the robust empirical Bellman operator in the Robust PPO algorithm based on the double-sampling (DS) uncertainty set and the integral probability metric (IPM) uncertainty set, which can be efficiently computed. For a better comparison with SAC in the next subsection, we adopt several modifications (e.g., state normalization, reward scaling, gradient clip, etc) in the implementation of PPO-based algorithms to improve their performance under the nominal model. Note that for fairness we employ the same modification for both PPO and RNAC-PPO algorithms.

We use the same hyperparameters across different MuJoCo environments. Specifically, we select γ = 0 . 99 for the discount factor, η t = α t = 3 × 10 -4 for learning rates of both actor and critic updates implemented by ADAM [24], T = 3 × 10 6 for the maximum training steps, and B = 2048 for the batch size.

Algorithm 4: RNAC-PPO - MSRTDE + RPPO

0: T -1 0: T -1

Initialize:

θ 0 for policy parameterization, w -1 for value function approximation

Collect set of trajectories D t until |D t | = B by running policy π θ t under p ◦ Update value function by minimizing mean-squared TD error with learning rate α t

Update policy via maximizing robust PPO objective with learning rate η t

Output: θ T

0

20

40

60

80

100

Percentage change in 'leg_joint_damping'

2000

2250

2500

2750

3000

3250

3500

3750

Cumulative reward

RNAC-PPO(DS)

PPO

0

10

20

30

40

50

'leg_joint_stiffness' values (default=0.0)

2000

2500

3000

3500

4000

4500

5000

Cumulative reward

RNAC-PPO(DS)

PPO

(b) DS, Walker-v3

0

10

20

30

40

50

60

70

Percentage change in front 'joint_stiffness'

1000

1500

2000

2500

3000

3500

Cumulative reward

RNAC-PPO(DS)

PPO

(c) DS, HalfCheetah-v3

0

20

40

60

80

100

Percentage change in 'leg_joint_damping'

1500

2000

2500

3000

3500

Cumulative reward

RNAC-PPO(IPM)

PPO

0

10

20

30

40

50

'leg_joint_stiffness' values (default=0.0)

3000

3500

4000

4500

5000

5500

Cumulative reward

RNAC-PPO(IPM)

PPO

0

10

20

30

40

50

60

70

Percentage change in front 'joint_stiffness'

0

500

1000

1500

2000

2500

Cumulative reward

RNAC-PPO(IPM)

PPO

(f) IPM, HalfCheetah-v3

A.2 Perturbed Mujoco Environments with Other Hyperparameters

In this subsection, we provide more experimental results on different perturbations of physical hyperparameters (e.g., leg joint damping , leg joint stiffness , and front joint stiffness ) in Hopper-v3, Walker2d-v3, and HalfCheetah-v3. Figure 4 shows that the RNAC-PPO algorithm is consistently robust compared to the PPO algorithm.

A.3 Comparison with Soft Actor-Critic and Soft-Robust PPO

In Section 7, we demonstrate the robust behavior of the RNAC-PPO algorithm compared with the PPO algorithm in Figure 1. In this subsection, we add two more baselines for robust algorithms: soft actor-critic (SAC) [18] and soft-robust [11] PPO (SRPPO). SAC is regarded as one of the robust baselines since maximum entropy RL (e.g. SAC) was shown to solve some robust RL problems by maximizing the lower bound on a robust RL objective [13]. Soft-robust RL learns an optimal policy based on a distribution over an uncertainty set instead of considering the worst-case scenario [11]. For a fair comparison, we implement the idea of soft robustness into the framework of PPO. Specifically, we build an uncertainty set wrapping up four environments, including one nominal environment and three perturbed environments with leg joint stiffness =5.0 (default=0.0), gravity =-9.50 (default=-9.81), and actuator ctrlrange =(-0.95, 0.95) (default=(-1.0, 1.0)), respectively. Additionally, we select [0.85, 0.05, 0.05, 0.05] as the distribution over the above uncertainty set for the SRPPO algorithm.

Figures 5c and 5d show that the cumulative rewards of RNAC-PPO decay much slower compared with those of PPO, but similar to those of SAC under the perturbation of gravity . This verifies the claim that SAC can solve some robust RL problems [13]. However, as shown in Figures 5a and 5b, SAC is not a panacea for all robust RL problems. Under the perturbation of leg joint stiffness , SAC suffers faster cumulative rewards decay compared with RNAC-PPO. Since SRPPO considers a distribution over an uncertainty set instead of only sampling from the nominal model, it doesn't perform well under the nominal model, but it shows a fairly robust behavior in Figures 5a and 5b when leg joint stiffness values are perturbed.

Since the cumulative rewards of SAC ( > 14000 ) are much higher than those of PPO-based algorithms ( < 3000 ) in HalfCheetah-v3, we only report the results of SAC in this subsection for Hopper-v3 to prevent the information of robustness from being blurred. Additionally, the training time of RNAC-PPO, PPO, and SRPPO is similar, which is at least 5 times less than that of SAC to

0

10

20

30

40

50

'leg_joint_stiffness' values (default=0.0)

500

1000

1500

2000

2500

3000

3500

4000

Cumulative reward

RNAC-PPO(DS)

PPO

SAC

SRPPO

0

10

20

30

40

50

'leg_joint_stiffness' values (default=0.0)

1000

1500

2000

2500

3000

3500

4000

Cumulative reward

RNAC-PPO(IPM)

PPO

SAC

SRPPO

(b) IPM, deterministic Hopper-v3

50

40

30

20

10

0

Percentage change in 'gravity' (default=0.0)

500

1000

1500

2000

2500

3000

3500

4000

Cumulative reward

RNAC-PPO(DS)

PPO

SAC

SRPPO

50

40

30

20

10

0

Percentage change in 'gravity' (default=0.0)

0

1000

2000

3000

4000

Cumulative reward

RNAC-PPO(IPM)

PPO

SAC

SRPPO

(d) IPM, deterministic Hopper-v3

end the training of 3 million steps. This is due to the fact that PPO-based algorithms require fewer updates for critic V , while SAC requires more updates for critic Q.

A.4 TurtleBot Experiment Details

The goal of this TurtleBot navigation task is to guide the robot to any desired target within a 2-meter range. The state space is a 2-dimensional continuous space, comprising the distance and relative orientation between the robot and the target. The robot is fixed to move towards its front with a linear velocity of 15 cm/s, while its angular velocity is controlled by the algorithm. Action space is 1-dimensional and continuous, ranging from -2 to 2 . The action signal is then linearly scaled into [ -1 . 5 rad/s , 1 . 5 rad/s ] . The reward function is designed to be proportional to the product of distance and action-scaled orientation between the robot and the target. Hitting the boundary or reaching the target would cause a reward of -200 or 200 respectively. One trajectory would end when the robot hits the boundary, reaches the goal, or the elapsed time is more than 150 seconds.

We train both PPO and RNAC-PPO(IPM) under the simulator Gazebo. To introduce stochasticity into the originally deterministic Gazebo environment, we apply uniform action noise perturbation. PPO and RNAC-PPO(DS) are trained under a balanced Unif [ -0 . 5 , 0 . 5] noise-perturbed environment. All algorithms undergo 400 epochs of training, and the policy with the highest speed is saved. Subsequently, we evaluate all policies in an unbalanced Unif [1 . 0 , 1 . 5] noise-perturbed environment. We employ such a high noise since all learned algorithms are aggressive at fast turning, with action close to the limit of -2 or 2 . The trajectories of the robot under Gazebo simulator and a video for the real-world experiment are demonstrated in Figure 2 and [Video Link] .

Specifically, both PPO and RNAC-PPO are implemented by neural networks. The actor network is defined by the Gaussian policy, with one hidden layer and tanh activation function, projecting the

0.0

0.5

1.0

1.5

2.0

TurtleBot position x

0.0

0.5

1.0

1.5

2.0

TurtleBot position y

RNAC-PPO(DS)

PPO

Action Noise

Dynamic Random

Target

(a) DS, Unif(-0.5,0.5)

0.0

0.5

1.0

1.5

2.0

TurtleBot position x

0.0

0.5

1.0

1.5

2.0

TurtleBot position y

RNAC-PPO(DS)

PPO

Action Noise

Dynamic Random

Target

(b) DS, Unif(1,1.5)

0.0

0.5

1.0

1.5

2.0

TurtleBot position x

0.0

0.5

1.0

1.5

2.0

TurtleBot position y

RNAC-PPO(IPM)

PPO

Action Noise

Dynamic Random

Target

(c) IPM, deterministic

0.0

0.5

1.0

1.5

2.0

TurtleBot position x

0.0

0.5

1.0

1.5

2.0

TurtleBot position y

RNAC-PPO(IPM)

PPO

Action Noise

Dynamic Random

Target

(d) IPM, Unif(1,1.5)

input state into a feature with dimension 100, then into the 1-dimensional action. Critic network consists of one hidden layer, with a width 100, and a ReLU activation function. We choose discount factor as 0.99, learning rate as 2 . 5 × 10 -4 , batch size as 64, and optimizer ADAM.

Additionally, we also add more baselines (i.e., dynamics randomization [44] and action noise envelope [20]) to demonstrate the robustness of the proposed methods in the real-world TurtleBot environment. As shown in Figure 6, the proposed algorithm RNAC-PPO (DS / IPM) enjoys higher target reaching rates (100% / 100%), compared with action noise envelope (67.5% / 13%), dynamic randomization (9% / 2%), and PPO (0% / 0%), under perturbed testing environments, which illustrates the robustness of the RNAC-PPO algorithm.

We end this section by illustrating our hardware configurations. All experimental results are carried out on a Linux server with 48-core RTX 6000 GPUs, 48-core Intel Xeon 6248R CPUs, and 384 GB DDR4 RAM.

B Discussions on Uncertainty Sets

R -contamination For R -contamination [60], the uncertainty set is P = ⊗ s,a P s,a , where P s,a = { Rq +(1 -R ) p o s,a : q ∈ ∆ S } . It can be shown that

Its corresponding robust Bellman operator requires searching the entire state space S to calculate the minimum value in V , and thus intractable for large state space.

ℓ p norm [28] For any nominal model p ◦ , the uncertainty set is P = ⊗ s,a P s,a with P s,a = { p : ∥ q -p ◦ s,a ∥ p ≤ δ, ∑ s ′ q ( s ′ ) = 1 } , where ∥ · ∥ p is an ℓ p norm with its dual norm ∥ · ∥ q satisfying 1 p + 1 q = 1 , and the domain of q ∈ ∆ S is relaxed to hyperplane ∑ s ′ q ( s ′ ) = 1 due to the difficulty in handling the boundary of ∆ S . It was shown [28] that

which requires solving a minimization problem via binary search. The minimum value may have a closed-form representation (c.f. Table 1 in [27]). For example, the minimum value is the average of V for ℓ 2 norm, the median of V for ℓ 1 norm, and the average peak of V (i.e., the average of the maximum and minimum value of V ) for ℓ ∞ norm. ℓ p norm-based uncertainty set is thus intractable in the large-scale scenario.

f -divergence Given a continuous strict convex function f : R + → R with f (1) = 0 , f -divergence is defined by d f ( q, p ) = ∑ s p ( s ) f ( q ( s ) p ( s ) ) . By distributionally robust optimization literature [12], we have

where f ∗ is the Fenchel conjugate of f , i.e., f ∗ ( y ) = sup x> 0 ( yx -f ( x )) . Given such a formula, the robust Bellman operator can be estimated given samples from the nominal model but requires the optimal dual variables λ, η for each ( s, a ) pair.

Wasserstein distance Given some metric d ( · , · ) on state space S , Wassersteinσ distance is defined for any p, q ∈ ∆ S that

where U ( p, q ) is the set of all couplings of p and q . Wasserstein RMDP has uncertainty set P = ⊗ s,a P s,a with P s,a = { q : W σ ( q, p ◦ s,a ) ≤ δ } and we have

The Wasserstein RMDP has been previously studied [1, 26]. Kuang et al. [26] gives a state disturbance view of Wasserstein RMDP, where the robust Bellman operator requires searching the worst-case state in the vicinity of the next state sample, i.e., inf ˜ s : ∥ ˜ s -s ′ ∥≤ δ V (˜ s ) . Although this approach can be applied to RMDP with large state space, the theoretical guarantee provided in [26] is only for policy iteration (planning problem) in the tabular setting (i.e., the robust Bellman operator is a contraction mapping under ∥ · ∥ ∞ ).

B.1 Double-Sampling Uncertainty Sets

When action a is taken at state s , the nominal model transits to the next state s ′ ∼ p 0 s,a . It can be viewed as a double-sampling process that the next state s ′ is generated by uniformly sampling one from m states s ′ 1 , s ′ 2 , . . . , s ′ m sampled i.i.d. according to p 0 s,a . The transition kernel in the DS uncertainty set at ( s, a ) -pair can be viewed as selecting the next state s ′ in { s ′ 1 , . . . , s ′ m } according to a distribution α ∈ ∆( m ) that is perturbed from a uniform distribution Unif ( m ) and potentially depends on the samples { s ′ 1 , . . . , s ′ m } . The DS uncertainty set is implicitly defined by a choice of m , divergence measure d( · , · ) and radius δ > 0 .

Bellman completeness for robust linear MDP under DS uncertainty set: A Q function class G ⊂ R S×A is said to be Bellman complete if it is closed, and for all g ∈ G , T ∗ g also lies in G , where

Bellman completeness is critical for reinforcement learning. For RMDP with the Double-Sampling (DS) uncertainty set, the linear function approximation can satisfy Bellman completeness if the nominal model is a linear MDP. In linear MDP [22], we have

where µ is a vector of (signed) measures over S . Then for any V ∈ R S

For the DS uncertainty set P = ⊗ s,a P s,a determined by m, d( · , · ) , δ , let f V ( s ′ 1: m ) := inf α ∈ ∆ [ m ] : d ( α, Unif ([ m ])) ≤ δ ∑ m i =1 α i V ( s ′ i ) , and ¯ f V ( s ′ 1 ) := E s ′ 2: m i.i.d. ∼ p ◦ s,a [ f V ( s ′ 1: m )] . It then follows that for any V

The Bellman completeness for DS-based RMDP with linear MDP nominal model is thus proved.

B.2 Integral Probability Metric Uncertainty Sets

Under the linear value function approximation, V w ( s ) = ψ ( s ) ⊤ w . Denote Ψ ∈ R S× d as the feature matrix by stacking up ψ ( s ) ⊤ , and the value function approximation is a linear regressor V w = Ψ w ∈ R S . Let Ψ = [ ψ 1 , . . . ψ d ] with ψ i ∈ R S , without loss of generality, assume ψ 1 is an all-one vector, which corresponds to the bias term of the linear regressor, and Ψ is full column rank.

We propose the IPM d F ( p, q ) = sup f ∈F { p ⊤ f -q ⊤ f } determined by function class

The robust Bellman operator T π P V (2) requires solving the optimization inf q ∈P s,a q ⊤ V , which is equivalent to

We relax the domain constraint q ∈ ∆ S to ∑ s q ( s ) = 1 and define P s,a = { q : d F ( q, p ◦ s,a ) ≤ δ, ∑ s q ( s ) = 1 } . This relaxation omits the boundary effect of ∆ S , which facilitates the following analysis. The relaxation is also made for ℓ p norm-based uncertainty set [28], and one can argue that it does not introduce any relaxation error for p ◦ s,a ( s ′ ) > 0 , ∀ s, a, s ′ and small δ .

Restatement of Proposition 1. For the IPM with F in (5), we have inf q ∈P s,a q ⊤ V w = ( p ◦ s,a ) ⊤ V w -δ ∥ w 2: d ∥ .

Proof of Proposition 1. Denote u = q -p ◦ s,a . The inf q ∈P s,a q ⊤ V under the relaxation is

Since Ψ is full column rank and ψ 1 is an all 1 vector, we know { Ψ ⊤ u : u ⊤ 1 = 0 , u ∈ R S } = { y ∈ R d : y 1 = 0 } . The optimization problem can then be written as

The constraint is equivalent to ∥ y 2: d ∥ ∗ ≤ δ and the optimal value can then be written as ( p ◦ s,a ) ⊤ V w -δ ∥ w 2: d ∥ , which concludes the proof of Proposition 1.

The IPM has many merits due to its capability to take advantage of the geometry of the domain (state space) through the function class.

C Robust Critic Analysis

We analyze the robust critic component in the RNAC algorithm (Algorithm 1). This section would also be of independent interest for the robust policy evaluation problem, where one aims to estimate the robust value of some policy given Markovian data.

The first two subsections focus on the linear function approximation and prove the main theorems as stated in Section 5 of the main paper. The last subsection focuses on the general function approximation.

C.1 Linear Robust Value Function Approximation

Setting: We aim to approximate the robust value function V π thorough linear function class

The 'optimal' linear value approximation V w π is the solution of the projected robust Bellman equation i.e.,

where the projection matrix is Π π = Ψ(Ψ ⊤ D π Ψ) -1 Ψ ⊤ D π with D π = diag ( ν π ) and ν π is the state stationary distribution of executing policy π on the nominal model p ◦ .

Definition 3 (Linear value approximation error) . ϵ V,bias := sup π ∥ Π π V π -V π ∥ ν π .

w π is the solution of the projected robust Bellman equation Π π T π P V w = V w . When Π π T π P is β -contraction w.r.t. ∥ · ∥ ν π , we have (according to Corollary 4 in [53])

C.1.1 Contraction of Projected Robust Bellman Operator

The contraction property of the projected robust Bellman operator is the key to guaranteeing the convergence of linear TD algorithms. The contraction property can be guaranteed when Assumption 2 is satisfied, and we show that the DS uncertainty set with a small δ indeed satisfies this assumption.

We refer to a divergence d( · , · ) as C -continuous at uniform distribution Unif ([ m ]) , if { α ∈ ∆ [ m ] : d( α, Unif ([ m ])) ≤ ϵ } ⊆ { α ∈ ∆ [ m ] : ∥ α -Unif ([ m ]) ∥ ∞ ≤ Cϵ } , ∀ ϵ > 0 .

Proposition 4. Uncertainty set implicitly defined by double sampling with divergence d( · , · ) that is C -continuous at uniform distribution Unif ([ m ]) and δ < 1 -γ γmC satisfies Assumption 2.

Proof. For the uncertainty set P implicitly defined by double-sampling with robust Bellman operator 3

Since d( · , · ) is C -continuous at uniform distribution, take arbitrary ( s, a, s ′ ) , we know

To guarantee γ sup q ∈P s,a q ( s ′ ) ≤ βp ◦ s,a with some β < 1 , we know need (1 + mCδ ) < 1 γ , which can be achieved by δ < 1 -γ γmC .

The following theorem shows that choosing a small δ is not a panacea for the well-known f -divergence uncertainty set. Note that many well-known metrics, such as KL-divergence, total variation, and χ 2 -divergence are special cases of f -divergence.

Proposition 5 (Restatement of Proposition 3) . For any f -divergence and radius δ > 0 , there exists a geometrically mixing nominal model such that the f -divergence defined uncertainty set violates Assumption 2.

Proof of Proposition 3. Given a continuous strict convex function f : R + → R with f (1) = 0 , the f -divergence is defined as d f ( q, p ) = ∑ s p ( s ) f ( q ( s ) p ( s ) ) .

Given a nominal model p ◦ and a radius δ > 0 , the f -divergence-based uncertainty set is P with P s,a = { q : d f ( q, p ◦ s,a ) ≤ δ } . For any f and δ > 0 , consider a nominal model p ◦ , which has a uniform transition probability, i.e., transits to a uniformly and randomly selected next state at any state-action pair, except for ( s, a ) . The transition probability at ( s, a ) is p ◦ s,a = ( α, 1 -α |S|1 , 1 -α |S|1 , . . . , 1 -α |S|1 ) , where α is some parameter to be determined later. It is clear that this nominal model p ◦ is well mixed for any α ∈ [0 , 1] . Consider another model q which coincides with p ◦ except for states ( s, a ) with q s,a = ( α/γ, 1 -α/γ |S|1 , 1 -α/γ |S|1 , . . . , 1 -α/γ |S|1 ) . Note that q s,a (1) p ◦ s,a (1) = 1 γ , thus if q ∈ P , the Assumption 2 of β < 1 is violated. Since as α → 0 , by the continuity of f and f (1) = 0 , we have

Therefore, with a sufficiently small α , we have q s,a ∈ P s,a and clearly q ∈ P . Thus there does not exist a universal choice of δ for f -divergence-based uncertainty set to guarantee Assumption 2.

We next prove Lemma 1, which shows the contraction of robust Bellman operator for IPM-based RMDP without Assumption 2.

Proof of Lemma 1. Let P π ( s ′ | s ) := ∑ a p ◦ s,π ( a | s ) ( s ′ ) be the state transition kernel of executing policy π on the nominal model p ◦ . We can view P π ∈ R S×S as a matrix. Recall D π = diag ( ν π ) , we have

Since Π π is a non-expansion mapping w.r.t. ∥ · ∥ ν π , Π π T π P is a contraction mapping if γ (1 + δ λ min (Ψ ⊤ D π Ψ) ) < 1 , which is equivalent to the condition δ < λ min (Ψ ⊤ D π Ψ) 1 -γ γ .

C.1.2 Convergence of Robust Linear TD

As discussed in the paragraph under Assumption 1, robust linear TD (RLTD) is a stochastic approximation (c.f. Section F.3 for a brief overview of stochastic approximation) since empirical operator ˆ T π P ((3) or (6)) is unbiased with

We can then let F ( w,x ) = ψ ( s ) [ ( ˆ T π P V w )( s, a, y ′ ) -ψ ( s ) ⊤ w ] with x = ( s, a, y ′ ) and RLTD is solving the following equation

through stochastic approximation:

It is clear that { x k } is also a Markov chain with domain X = S × A × Y , Y = S m for the double-sampling uncertainty set with m samples and Y = S for the IPM uncertainty set.

Theorem 5 (Formal statement of Theorem 3) . Take δ as in Lemma 1 and in Proposition 4, for the DS uncertain set or the IPM uncertainty set, respectively. Under Assumption 1, RLTD with step sizes α k = Θ(1 /k ) guarantees E [ ∥ w K -w π ∥ 2 ] = ˜ O ( 1 K ) and E [ ∥ V w K -V w π ∥ 2 ν π ] = ˜ O ( 1 K ) .

Proof of Theorem 5. By Proposition 4 and Lemma 1, DS or IPM with small δ can guarantee Π π T π P is a β -contraction w.r.t. ∥ · ∥ ν π for some β < 1 . The theorem can be proved by applying Lemma 12. For this purpose, we only need to show that Assumption 8 is satisfied for RLTD. We check the three conditions in Assumption 8 as follows.

The geometric mixing property of { x k = ( s k , a k , y k +1 ) } k ≥ 0 is straightforward given geometrically mixed { s k } k ≥ 0 .

Since r ∈ [0 , 1] and ∥ ψ ( s ) ∥ is bounded ∀ s ∈ S , we have ∥ F (0 , x ) ∥ = ∥ ψ ( s ) r ( s, a ) ∥ ≤ ∥ ψ ( s ) ∥ is also bounded for any x = ( s, a, y ′ ) .

For double-sampling (3) and any x = ( s, a, s ′ 1: m ) ,

Let Ψ s ′ 1: m ∈ R m × d be the matrix by stacking up ψ ( s ′ 1 ) ⊤ , . . . , ψ ( s ′ m ) ⊤ , we have

where sup α , inf α are taking in { α ∈ ∆ [ m ] : d( α, Unif ([ m ])) ≤ δ } . Thus there exists L 1 = max(max s ∥ ψ ( s ) ∥ , 2 γ max s ∥ ψ ( s ) ∥ 2 ) that guarantee ∥ F (0 , x ) ∥ ≤ L 1 and ∥ F ( w,x ) -F ( w ′ , x ) ∥ ≤ L 1 ∥ w -w ′ ∥ for the double-sampling RMDP.

For IPM (6) and any x = ( s, a, s ′ ) ,

We have

which also satisfies the second item of Assumption 8 by setting L 1 = max(max s ∥ ψ ( s ) ∥ , max s ( ( γ +1) ∥ ψ ( s ) ∥ 2 + γδ ∥ ψ ( s ) ∥ ) ) .

Note that for small δ , Π π T π P is a β -contraction w.r.t. ∥ · ∥ ν π . Since there exists w π with ¯ F ( w π ) = 0 , let D := D π = diag ( ν π ) , we have

where the first inequality is due to the Cauchy-Schwarz inequality and the second inequality is due to the β -contraction property of Π π T π P .

In addition, we have

C.2 Extension of General Value Function Approximation

Setting: In the general function approximation setting, we consider a known finite and bounded function class G ⊂ R S to fit the robust value function V π , i.e.,

Note that IPM-based RMDP with empirical robust Bellman operator (6) only applies to linear function approximation. We implement it similarly by adding a negative regularization with neural network approximating the robust values, which also induces robust behavior of the learned policy as illustrated in the experiments (Section 7). However, it may not directly match any specific uncertainty set. We thus only consider the DS-based RMDP in this general value function approximation setting.

We define the robust Bellman error as follows.

Definition 4 (Robust Bellman Error) . ϵ g,bias := max π max g ∈G min g ′ ∈G ∥T π P g -g ′ ∥ ν π .

If ϵ g,bias = 0 , then the robust value is realizable within the function class, i.e., V π ∈ G for any π ∈ G .

C.2.1 Fitted Robust Value Evaluation Algorithm

We propose the fitted robust value evaluation (FRVE) in Algorithm 5, a robust version of the fitted value evaluation commonly used in offline RL. This algorithm first samples a batch of data from the nominal model, then select the last half as training data for analytical purpose without losing the order while the Markovian data are close to the stationary distribution due to geometric mixing Assumption 1, and iteratively solve for a better robust value approximation based on the current approximation g k and its robust value estimate ˆ T π P ( g k ) .

Algorithm 5: Fitted Robust Value Evaluation (FRVE)

Theorem 6 (Convergence of FRVE) . For DS RMDP with δ suggested by Proposition 4, T π P is a β -contraction mapping for some β < 1 . Under Assumption 7, the return of FRVE Algorithm 5 has

where G max := max g ∈G (( ˆ T π P g )( s, a, y ′ ) , ∥T π P g ∥ ν π , ∥ g ∥ ν π ) and ϵ g,stat = ˜ O (1 / √ K ) . Thus E [ ∥ V π -g K ∥ ν π ] = ˜ O (1 / √ K ) + O ( ϵ g,bias ) .

Proof. By the contraction mapping of T π P

Note that T π P g k -1 is a target that g k is an approximation through MSE with Markovian data. If the data is stationary, i.e., s K/ 2 ∼ ν π , applying Lemma 11 and the union bound over G (taking T π P g as a target for each g ∈ G ), we know with probability at least 1 -δ ,

Let P µ be a distribution on D with x K/ 2 ∼ µ , and let P ν π be a distribution on D with x K/ 2 ∼ ν π . Take µ as the true distribution of x K/ 2 according to the data sampling process as in Algorithm 5. We know ∥ µ -ν π ∥ TV = O ( e -K ) by Assumption 1, and thus

The proof of the theorem is thus concluded.

D Robust Natural Actor Analysis

We analyze the robust natural component in the RNAC algorithm (Algorithm 1).

We first discuss the Fr´ echet supergradient of the robust value function in the first subsection. The second subsection focuses on the linear function approximation and proves the main theorems as stated in Section 5 of the main paper. The last subsection focuses on the general function approximation.

D.1 Policy Gradient and Performance Difference

The robust value function J ( θ ) = V π θ ( ρ ) is in general not differentiable. But since it is Lipschitz (w.r.t. to Lipschitz policy parameterization), it is differentiable almost everywhere according to Rademacher's theorem [14]. At the place not differentiable, Fr´ echet supergradient ∇ J of J is then defined as

When function J is differentiable, J at any point θ has a unique Fr´ echet supergradient, which is the gradient of J at θ .

Proof of Lemma 2. The Fr´ echet supergradient exists for tabular RMDP [29], which is

where [ ∇ π V π ( ρ )] s,a indicates the ( s, a ) coordinate of vector ∇ π V π ( ρ ) . π = π θ is a policy parameterized by θ .

where the first relation is by the chain rule of supergradient.

D.2 Linear Function Policy Approximation

Setting: This subsection considers the log-linear policy with

where ϕ ( s, a ) ∈ R d is some known feature vector and θ ∈ R d is the policy parameter. Let Φ ∈ R |S||A|× d be the feature matrix by stacking up the feature vectors ϕ ( s, a ) ⊤ .

Recall the discussion of the proposed RQNPG in Section 6. We approximate the robust Q function Q π via Q w ( s, a ) = r ( s, a ) + inf p ∈P s,a p ⊤ V w given a value function approximation V w from the robust critic, and then approximate Q w by a policy-compatible [52] robust Qapproximation Q u = Φ u . In other words, we project Q w onto span (Φ) . Denote by Π π Φ := Φ(Φ ⊤ diag ( ν π ◦ π )Φ) -1 Φ ⊤ diag ( ν π ◦ π ) ∈ R |S||A|×|S||A| the projection matrix onto space span (Φ) w.r.t. norm ∥ · ∥ ν π ◦ π . We then define the approximation error ϵ Q,bias below. When realizable, i.e., Q π ∈ span (Φ) , the approximation error ϵ Q,bias = 0 .

We assume a finite relative condition number (Assumption 3) (similar to that in [3]). The relative condition number is not necessarily related to the size of the state space (details are shown in Remark 6.3 of [3]).

Now we look at a specific update θ t +1 = RQNPG ( θ t , η t , w t , N ) , where w t = RLTD ( π θ t , K ) .

D.2.1 RQNPG One-Step Analysis - Robust Q Function Approximation

In this update θ t +1 = RQNPG ( θ t , η t , w t , N ) , where w t = RLTD ( π θ t , K ) . RQNPGfirst approximates Q w t ( s, a ) = r ( s, a ) + γ inf p ∈P s,a p ⊤ V w t = E [( ˆ T π P V w t )( s, a, y ′ ) | s, a ] by Q u ( s, a ) = ϕ ( s, a ) ⊤ u , as the caculation of u N in Algorithm 3. Let

be the optimal approximation for the target Q w t . u t ∗ is approximated by stochastic approximation (c.f. Section F.3 for a brief overview of stochastic approximation) with a mean squared error loss

We know u t ∗ is the unique solution of

Let F ( u, x ) be the function inside the expectation with x = ( s, a, y ′ ) , and ¯ F ( u ) be the negative gradient -∇ u L ( u ; V, π ) . We then solve this stochastic zero point problem by stochastic approximation as in

where x n = ( s n , a n , y n +1 ) .

Lemma 3 (Convergence of compatible Q function approximation (SGD with Markovian data)) . Under Assumption 1, RQNPG (Algorithm 3) with step sizes ζ n = Θ(1 /n ) guarantees E [ ∥ u N -u t ∗ ∥ 2 ] = ˜ O ( 1 N ) and E [ ∥ Q u N -Q u t ∗ ∥ 2 ν t ◦ π t ] = ˜ O ( 1 N ) .

Proof of Lemma 3. The lemma is implied by Lemma 12. To see this, we only need to show that Assumption 8 is satisfied. We check the three conditions in Assumption 8 as follows.

The geometric mixing property of { x k } k ≥ 0 is straightforward given geometrically mixed { s k } k ≥ 0 .

Since ∥ ϕ ( s, a ) ∥ is bounded ∀ ( s, a ) ∈ S ∈ A , similar proof follows as in the proof of 5.

Since there exists u π with ¯ F ( u π ) = 0 and L ( u ; V, π ) is λ min ( E s,a [ ϕ s,a ϕ ⊤ s,a ]) -strongly convex, we have

Denote by Π π Φ := Φ(Φ ⊤ diag ( ν π ◦ π )Φ) -1 Φ ⊤ diag ( ν π ◦ π ) ∈ R |S||A|×|S||A| the projection matrix of function Q ∈ R |S||A| onto matrix Φ ∈ R |S||A|× d under norm ∥ · ∥ ν π ◦ π . We know Q u t ∗ = Π π t Φ Q w t . We have

For the update at step t , θ t +1 = RQNPG ( θ t , η t , w t , N ) and w t = RLTD ( π θ t , K ) , let π t = π θ t be the policy at step t . Let u t = u N ( u N in Algorithm 3) and Lemma 3 above shows that E [ ∥ Q u t -Q u t ∗ ∥ 2 ν t ◦ π t ] = ˜ O ( 1 N ) . This does not necessarily implies that Q u t and Q π t are close since the property of the critic returned w t is required. We measure the difference between Q u t and Q π t in the following lemma.

Let w t ∗ = w π t and w t = RLTD ( π θ t , K ) is the output of RLTD at step t . Define ϵ V,stat := max t =0 , 1 ,...,T -1 { √ E [ ∥ V w t ∗ -V w t ∥ 2 ν t ] } , and ϵ V,stat = ˜ O (1 / √ K ) by Theorem 3.

Lemma 4. Under the conditions in Theorem 5 and Assumption 3, there is some ϵ = ϵ stat + ϵ bias that for any π, π ′ and any d = d π ′ ,κ π ′ ρ , d π ∗ ,κ π t ρ or ν π ∗ ,

where the outside expectation is taken w.r.t. the randomness of the data collected in w t = RLTD ( π θ t , K ) and θ t +1 = RQNPG ( θ t , η t , w t , N ) . Moreover, ϵ stat = √ |A| ( ξβϵ V,stat + ξϵ Q,stat ) with ϵ V,stat = ˜ O ( 1 √ K ) and ϵ Q,stat = ˜ O ( 1 √ N ) ; ϵ bias = √ |A| ( ξβ 1 -β ϵ V,bias + ϵ Q,bias ) with ϵ V,bias and ϵ Q,bias in Definitions 3 and 5, respectively.

Proof of Lemma 4. For any d = d π t +1 ,κ t +1 s , d π ∗ ,κ t ρ or ν π ∗ and for any π , we know for any Q,Q ′

To quantify the error between Q u t -Q π t , recall Q u t ∗ = Π π t Φ Q w t and we decompose it into

and bound each term respectively. We can transfer the norm within the space spanned by Φ via the assumption that ∥ Φ u ∥ d ◦ Unif ≤ ξ ∥ Φ u ∥ ν π ◦ π . Note that the first three terms all lie in the span (Φ) , we have the first term bounded by

for some ϵ Q,stat = ˜ O ( 1 √ N ) as in Lemma 3. The second term is bounded by

for some ϵ V,stat = ˜ O ( 1 √ K ) as in Theorem 5. The third term is bounded by

where the last inequality is by Definition 3 and inequality (13). The last term is then bounded by Definition 5 E [ ∥ Π π t Φ Q π t -Q π t ∥ d ◦ Unif ] ≤ ϵ Q,bias . We thus have

which concludes the proof.

D.2.2 RQNPG One-step Analysis - Mirror Ascent Update

Now we look at the policy improvement of the update θ t +1 = RQNPG ( θ t , η t , w t , N ) with ζ n = Θ(1 /n ) (Algorithm 3), where w t = RLTD ( π θ t , K ) . Let u t = u N ( u N in Algorithm 3), and we know the RQNPG update is θ t +1 = θ t + η t u t .

This RQNPG update is equivalent to a certain mirror ascent update. Specifically, recall Q u t ( s, a ) = ϕ ( s, a ) ⊤ u t is the approximated robust Q function, the RQNPG update θ t +1 = θ t + η t u t is equivalent to [4]

where we let π s := π ( ·| s ) and Q s := Q ( s, · ) for simplicity. Note that this update can be viewed as a mirror descent step with KL-divergence as Bregman divergence. Given this mirror descent formulation of policy update in Eq (17), the pushback property indicates that for any policy π (Eq (2) in [68]),

which is equivalent to the following fundamental inequality

Restatement of Theorem 4 (Approximate policy improvement) For any t ≥ 0 , we know

where KL ν ( π, π ′ ) := ∑ s ν ( s )KL( π ( ·| s ) , π ′ ( ·| s )) ≥ 0 and E [ ϵ t ] = ˜ O ( 1 √ N + 1 √ K ) + O ( ϵ bias ) .

Proof of Theorem 4. Let κ t +1 = κ π t +1 be the worst-case transition kernel w.r.t. policy π t +1 , and let ϵ t = E s ∼ d π t +1 ,κ t +1 ρ [ ⟨ Q π t s -Q u t s , π t +1 s -π t s ⟩ ] . We have

Algorithm 6: Robust Natural Policy Gradient (RNPG)

Return: θ + ηu

N

where the first inequality is by Lemma 8, and the last inequality is by taking π = π t in the fundamental inequality Eq (18), which implies

ϵ t can then be bounded by Lemma 4 with E [ ϵ t ] ≤ 2 ϵ = 2 ϵ stat + 2 ϵ bias , where ϵ stat , ϵ bias are specified in Lemma 4 with ϵ stat = ˜ O ( 1 √ N + 1 √ K ) .

D.3 Extension of General Function Approximation of Policy

Setting: In this subsection, we consider a general policy class of form

where f θ is a differentiable function. This general policy class contains the log-linear policy class as a special case by f θ ( s, a ) = ϕ ( s, a ) ⊤ θ .

D.3.1 Robust Natural Policy Gradient with General Function Approximation

For the general policy class in Eq (20), we propose a Robust NPG (RNPG) algorithm.

This algorithm can be applied to RNAC (Algorithm 1) for the robust natural actor update. Now we look at a specific update θ t +1 = RNPG ( θ t , η t , w t , N ) , where w t is the output of the robust critic at step t . Note that for the critic with general function approximation Eq. (14), we slightly abuse the notation by V w t = g t , where g t is the output of the FRVE (Algorithm 5) at step t of RNAC. We can view g ∈ G is parameterized by some w , as indicated in the RNAC algorithm Algorithm 1.

Denote by ϕ θ ( s, a ) := ∇ θ log π θ ( a n | s n ) and Φ θ ∈ R |S||A|× d as the feature matrix stacking up feature vector ϕ θ . For each t = 0 , 1 , . . . , T -1 , we let ϕ t := ϕ θ t and Φ t := Φ θ t for simplicity. The RNPG update is θ t +1 ← θ t + η t u t , where u t = u N as the output of the stochastic gradient descent in Algorithm 6,

RNPG approximates the value approximated advantage function A w t ( s, a ) := r ( s, a ) + γ inf p ∈P s,a p ⊤ V w t -V w t ( s ) by A u t := Φ t u . Note that ( ˆ T π t P V w t )( s n , a n , y n +1 ) -V w t ( s n ) is an unbiased estimate of A w t ( s n , a n ) , i.e.,

RNPG thus is iteratively solving the following optimization

by stochastic approximation (stochastic gradient descent with Markovian data). Denote by u π t ∗ the optimal value of the optimization, which is also a solution of the equation

Theorem 7 (Convergence of compatible advantage function approximation (SGD with Markovian data)) . Under assumption 1, RNPG ( θ t , η t , w t , N ) (Algorithm 6) with step sizes ζ n = Θ(1 /n ) guarantees E [ ∥ u N -u t ∗ ∥ 2 ] = ˜ O ( 1 N ) and E [ ∥ A u N t -A u t ∗ t ∥ 2 ν t ◦ π t ] = ˜ O ( 1 N ) .

Proof of Theorem 7. The theorem can be proved in the same manner as that for Lemma 3, since the objective function in Eq (21) is strongly convex.

With slight abuse of notation, denote by Σ t ν,π := E ( s,a ) ∼ ν ◦ π [ ϕ t ( s, a ) ϕ t ( s, a ) ⊤ ] . The optimal value u t ∗ = ( λI + (Φ t ) ⊤ diag ( ν t ◦ π t )Φ t ) -1 ((Φ t ) ⊤ diag ( ν t ◦ π t ) A w t ) satisfies ∥ u t ∗ ∥ ≤ ∥ A wt ∥ ν t ◦ π t √∑ d i =1 ∥ ϕ t i ∥ 2 ν t ◦ π t λ + λ min (Σ t ν t ◦ π t ) . Let u t = u N , Theorem 7 gives E [ ∥ u t ∥ 2 ] ≤ 2 E [ ∥ u t ∗ ∥ 2 ] + ˜ O ( 1 N ) . We then have E [ ∥ u t ∥ 2 ] ≤ U under the following assumption.

Assumption 4 (Bounded feature) . Assume sup θ ∑ d i =1 ∥ ϕ θ i ∥ 2 ν π θ ◦ π θ < ∞ . Since G (14) is bounded, ∥ A w t ∥ v t ◦ π t < ∞ and there exists 0 < U < ∞ that max t E [ ∥ u t ∥ 2 ] ≤ U .

D.3.2 Robust General Advantage Function Approximation

Denote by Π t Φ t the projection mapping to the space { Φ t u : ∥ u ∥ ≤ ∥ u t ∗ ∥} under metric ∥ · ∥ ν t ◦ π t .

Note that ϵ A,bias also implicitly depends on the choice of λ since u t ∗ depends on λ . If the realizable case, i.e., A π t ∈ span (Φ t ) , ϵ A,bias = 0 if λ = 0 .

Assumption 5. There is some 0 < ξ ′ < ∞ that for any θ , sup u ∥ Φ θ u ∥ ν π ∗ ◦ π ∗ ∥ Φ θ u ∥ ν π θ ◦ π θ ≤ ξ ′ .

Lemma 5. Under the conditions in Theorems 7 and 6 and Assumption 5, for any t ,

where the outside expectation is taken w.r.t. the randomness of the data collected the robust natural actor and robust critic update. Moreover, ϵ ′ stat = ˜ O ( 1 √ K + 1 √ N ) ; ϵ ′ bias = O ( ϵ A,bias + ϵ g,bias ) or O ( ϵ A,bias + ϵ V,bias ) for linear robust critic.

Proof of Lemma 5. To quantify the error between A u t -A t , we decompose it into

and bound each term respectively. The proof follows similarly to that in Lemma 4 by applying Theorem 7 and Theorem 6.

E Robust Natural Actor-Critic Analysis

In this section, we first state and prove the formal versions of the main theorems Theorem 1 and Theorem 2 in Theorem 8 and Theorem 9, respectively. We then give the convergence of RNAC employing general function approximation in Theorem 10.

E.1 Linear Function Approximation

We introduce the detailed setup of the RNAC algorithm with linear function approximation, based on which the theorems are stated and proved.

RNAC-Linear Setting: We study the RNAC (Algorithm 1) with robust critic performing RLTD (Algorithm 2 with ξ n = Θ(1 /k ) as in Theorem 5) and robust natural actor performing RQNPG (Algorithm 3 with ζ n = Θ(1 /n ) as in Lemma 3) under linear value function approximation as in Eq (12) and log-linear policy as in Eq (15), for DS or IPM uncertainty sets taking δ suggested by Proposition 4 or Lemma 1, respectively. We assume Assumption 1 and Assumption 3 hold.

For each time t = 0 , 1 , . . . , T -1 , the value approximation and policy in RNAC are updated as w t = RLTD ( π θ t , K ) and θ t +1 = RQNPG ( θ t , η t , w t , N ) , respectively. Denote by u t as u N in RQNPG ( θ t , η t , w t , N ) (Algorithm 3), which defines a robust Q-function approximation Q u t . Let π t = π θ t , κ t = κ π t , V t = V π t for simplicity. We have the following lemma.

Lemma 6. Under the RNAC-Linear Setting , for any t = 0 , 1 , . . . , T -1 , we have ⟨ Q u t s , π t +1 s -π t s ⟩ ≥ 0 , ∀ s ∈ S and

where the expectation E is taken w.r.t. the data sampled by RNAC, and ϵ = ϵ stat + ϵ bias is the same as that in Lemma 4.

Proof of Lemma 6. Taking π = π t in the fundamental inequality Eq (18) gives

which implies π t +1 is indeed improving π t along Q u t direction. We then have

where the second inequality is by Lemma 8. The proof of the lemma is then concluded by taking expectation E [ E s ∼ ρ [ · ]] on both sides and applying Lemma 4.

∗ ,κ

Assumption 6. For initial state distribution ρ , there exists M that sup κ ∈P ∥ d ρ ρ ∥ ∞ ≤ M < ∞ . Theorem 8 (Formal statement of Theorem 1) . Under RNAC-Linear Setting and Assumption 6, RNAC with geometrically increasing step sizes η t ≥ M 1 -γ 1 -1 -γ M η t -1 for each t = 1 , 2 , . . . , T , satisfies

where ϵ = ϵ stat + ϵ bias as in Lemma 4 with ϵ stat = ˜ O ( 1 √ K + 1 √ N ) . Omitting the approximation ϵ bias , the sample complexity for achieving ε robust optimal value (i.e., V ∗ ( ρ ) -E [ V T ( ρ )] ≤ ε ) is ˜ O (1 /ε 2 ) by taking N = K = ˜ Θ(1 /ε 2 ) and T = Θ(log(1 /ε )) .

Proof of Theorem 8. Taking π = π ∗ in inequality (18), we have

We then take the expectation E [ E s ∼ d ∗ ,κ t ρ [ · ]] on both sides. Note that

where the inequality is by Lemma 8 and Lemma 4. Then by Lemma 6 and Assumption 6 that M = sup κ ∥ d ∗ ,κ ρ ρ ∥ ∞ , we have

since ⟨ Q u t s , π t s -π t +1 s ⟩ ≤ 0 . The outside expectation E is taken w.r.t. the data sampled by RNAC, and since all the following statements are under expectation E , we omit E in the proof for simplicity and bring it back at the end. Combining the inequality above and Eq (23), we have

It follows that

which implies

Take geometrically increasing step sizes η t ≥ M 1 -γ 1 -1 -γ M η t -1 for each t . Since M = sup κ ∥ d ∗ ,κ ρ ρ ∥ ∞ ≥

We bring back the expectation E over the data and finally arrive at

where KL d ∗ κ 0 ( π ∗ , π 0 ) ≤ log |A| when π 0 is a uniform policy.

ρ

Theorem 9 (Formal statement of Theorem 2) . Under RNAC-Linear Setting and Assumption 6 when IPM uncertainty set is considered, take ρ = ν π ∗ and η t = η = (1 -γ ) log( |A| ) , RNAC satisfies

where β is in Assumption 6 (guaranteed by Proposition 4 for DS uncertainty set and assumed for IPM uncertainty set), and ϵ = ϵ stat + ϵ bias as in Lemma 4 with ϵ stat = ˜ O ( 1 √ K + 1 √ N ) . Omitting the approximation ϵ bias , the sample complexity for achieving ε robust optimal value on average (i.e., V ∗ ( ρ ) -1 T ∑ T t =1 E [ V t ( ρ )] ≤ ε ) is ˜ O (1 /ε 3 ) by N = K = ˜ Θ(1 /ε 2 ) and T = Θ(1 /ε ) .

Proof of Theorem 9. Taking π = π ∗ in inequality (18), we have

Since all the following statements are under expectation E over the data, we omit E in the proof for simplicity and bring it back at the end. According to Lemma 9 and Lemma 4, we have

Based on Lemma 6, we have

It then follows that

Taking summation from t = 0 to T -1 from both sides we have

We then conclude the theorem by choosing η = (1 -γ ) log( |A| ) and bringing back the expectation E over data,

Discussion: The linear convergence for NPG with linear function approximation has been previously studied in canonical MDP [4]. We present the linear convergence of RNAC (Algorithm 1) in Theorem 8. Compared to Theorem 8, Theorem 9 utilizing constant step sizes and leads to sublinear convergence, and it is proved only for initial state distribution ρ = ν π ∗ . Moreover, Theorem 9 does not require Assumption 6 though may need Assumption 2 for IPM uncertainty set.

E.2 General Function Approximation

RNAC-General Setting. We study the RNAC (Algorithm 1) with robust critic performing FRVE (Algorithm 5) and robust natural actor performing RNPG (Algorithm 6 with ζ n = Θ(1 /n ) as in Theorem 7) under general value function approximation as in Eq (14) and general policy class as in Eq (20), for DS uncertainty set taking δ suggested by Proposition 4. We assume Assumption 1, Assumption 5 and Assumption 4 ( ∥ u t ∗ ∥ ≤ U ), hold.

Theorem 10. Under RNAC-General Setting , suppose log π θ ( a | s ) is an L -smooth function of θ . Take ρ = ν π ∗ and η t = η = √ 2KL ρ ( π ∗ ,π 0 ) 2 for each t , RNAC satisfies

where β is in Assumption 6 (guaranteed by Proposition 4 for DS uncertainty set), and ϵ ′ = ϵ ′ stat + ϵ ′ bias as in Lemma 5 with ϵ ′ stat = ˜ O ( 1 √ K + 1 √ N ) . Omitting the approximation ϵ ′ bias , the sample complexity for achieving ε robust optimal value is ˜ O (1 /ε 4 ) by N = K = ˜ Θ(1 /ε 2 ) and T = Θ(1 /ε 2 ) .

Proof. We have

where the first inequality is by Lemma 9, the first equality is by the definition of A u t ( s, a ) = ∇ log π t ( a | s ) ⊤ u t , and the second inequality is by the L -smoothness of log π θ ( a | s ) and Lemma 5.

Note that in RNAC-General Setting , the critic is employing general function approximation, where the IPM uncertainty set is not defined (c.f. Section C.2 for more discussion). It is not hard to show a similar result as in Theorem 10 for the RNAC with robust critic performing RLTD (Algorithm 2) and robust natural actor performing RNPG (Algorithm 6) for both DS and IPM uncertainty sets.

F Supporting Lemmas

We introduce some supporting lemmas for proving the results of this paper.

F.1 Performance Difference Lemmas

Akey supporting lemma in the proof of convergence of policy gradient-based methods is the performance difference lemma. In the canonical RL with a fixed transition kernel κ , we have the following performance lemma.

Lemma 7 (Performance difference [23]) . For any policy π, π ′ , and transition κ , we have

However, due to the non-singleton uncertainty set P , the worst-case transition kernel is a function of policy π , and we have the following performance difference inequality lemma.

Lemma 8 (Robust performance difference) . For any policy π, π ′ , denote κ ′ = κ π ′ and κ = κ π be their worst-case transition kernels, respectively. Then

Proof of Lemma 8. The LHS is by

The RHS is by

Moreover, the optimality gap between the optimal policy π ∗ and any policy π is upper bounded as in the lemma below.

Lemma 9 (Robust optimality gap lemma) . Under Assumption 2, for any π and any ρ ,

Proof of Lemma 9.

F.2 Concentration Lemmas with Markov samples

Assumption 7 (Uniformly geometric mixing, quantitative restatement of Assumption 1) . There exists some 0 < λ < 1 and C > 0 such that for any policy π , the Markov chain { s k } induced by applying π in the nominal model p ◦ is geometrically ergodic with unique stationary distribution ν π , i.e., max s ∥ P ( s k ∈ ·| s 0 = s ) -ν π ∥ TV ≤ Cλ k , ∀ k . The uniform convergence to unique stationary distribution implies that for any policy π the spectral gap (i.e., the difference between the largest and the second largest eigenvalues) of the state transition kernel has a strictly positive lower bound.

Lemma 10 (Bernstein's inequality for stationary Markovian data, Theorem 2 in [21]) . Suppose Z 1 , Z 2 , . . . , Z n follow some stationary Markovian data in space Z with stationary distribution ν ∈ ∆ Z and spectral gap of the Markov chain strictly greater than 0 . For any function f : Z → [ -c, c ] that, E Z ∼ ν [ f ( Z )] = 0 and E Z ∼ ν [ f ( Z ) 2 ] = σ 2 , with probability at least 1 -δ ,

for some spectral gap-dependent multiplicative factors C 1 , C 2 .

Since we assume Assumption 7 throughout the paper during the analysis, the spectral gap is uniformly lower bounded by some constant great than zero, and we view C 1 , C 2 as some constant that can be omitted in the big-O notation. We refer to [21] for further details of Bernstein's inequality for stationary Markovian data.

Lemma 11 (Lemma A.11 in [2] with Markovian data) . Consider a stationary Markov chain on domain X × Z that is geometrically mixing with transition p ( x ′ , z ′ | x, z ) = p X ′ | Z ( x ′ | z ) p Z | X ( z ′ | x ) and stationary distribution ν ◦ p Z | X . Let D = { ( X 1 , Z 1 ) , ( X 2 , Z 2 ) , . . . , ( X n , Z n ) } be stationary Markovian data sampled with X 1 ∼ ν . Given a function class G ⊂ R X and a function G : X × Z → R that E Z ∼ p Z | X ( ·| x ) [ G ( x, Z )] = g ∗ ( x ) and | G ( x, z ) | ≤ G max . Let ˆ g = arg min g ∈G 1 2 ∑ ( x,z ) ∈D ( G ( x, z ) -g ( x )) 2 be the MSE estimate of g ∗ within G , then with probability at least 1 -δ ,

where ϵ bias = min g ∈G ∥ g -g ∗ ∥ ν .

Proof. The proof follows the same as [2]. The only difference is replacing the canonical Bernstein's inequality with i.i.d. data by Bernstein's inequality for stationary Markovian data in Lemma 10.

F.3 Convergence of Stochastic Approximation

F : R d × Z → R d is a stochastic operator. The stochastic approximation algorithm updates the parameter by w k +1 = w k + α k F ( w,Z k ) , where ( Z 0 , Z 1 , . . . ) is a Markov chain, that is geometrically ergodic for some C > 0 , 1 > λ > 0 with stationary distribution ν , i.e., max z ∥ P ( Z k ∈ ·| Z 0 = z ) -ν ∥ TV ≤ Cλ k , ∀ k . (c.f. Assumption 1). The stochastic approximation algorithm solves the following equation

under the following assumption and appropriate step sizes ( α 1 , α 2 , . . . ) .

Assumption 8. 1. ( Z 0 , Z 1 , . . . ) are geometrically mixed to the stationary distribution.

∃ L 1 > 0 that ∥ F ( w,z ) -F ( w ′ , z ) ∥ ≤ L 1 ∥ w -w ′ ∥ , ∥ F (0 , z ) ∥ ≤ L 1 for any w,w ′ ∈ R d , z ∈ Z .

¯ F ( w ) = 0 has unique solution w ∗ and there exists c 0 > 0 that ( w -w ∗ ) ⊤ ¯ F ( w ) ≤ -c 0 ∥ w -w ∗ ∥ 2 for any w ∈ R d .

The second condition of Assumption 8 is a Lipschitz condition of the operator F , and the third condition is the 'strongly-concavity' structure of the averaged operator ¯ F .

Lemma 12 (Big-O version of Corollary 2.1.2 in [10]) . Under Assumption 8, taking appropriate α k = Θ(1 /k ) gives E [ ∥ w K -w ∗ ∥ 2 ] = ˜ O (1 /K ) .

To apply the lemma above, we need to show that the proposed operator F satisfies Assumption 8.

G More Related Works

The framework of the robust Markov decision process (RMDP) is proposed by [19, 40] to learn the optimal robust policy that achieves the optimal worst-case performance over all possible models in the uncertainty set. For the planning problem of RMDP, Iyengar [19] and Nilim et al. [40] show that value iteration achieves linear convergence to the optimal robust values.

When the transition model is unknown, several model-based and value-based approaches have been proposed and studied for robust RL in both the tabular setting and the function approximation setting. Under the tabular setting, Xu et al. [64] design the model-based robust phase value learning algorithm and demonstrates an O (1 /ϵ 2 ) sample complexity bound. Under the function approximation setting, Panaganti et al. [43] develop a robust fitted Q-iteration with the total variation uncertainty set and show an O (1 /ϵ 2 ) sample complexity to achieve approximate optimality and [6] develop a model-based method with the same sample complexity under certain coverage assumption. However, these methods are not scalable to continuous control problems. Instead, policy-based approaches directly parameterize policy and have more representation power in modeling stochastic policies, more flexibility for policy manipulation, and a better ability to solve robotics control problems with large action space.

The existing policy-based methods with convergence guarantees mainly focus on the tabular setting. Wang et al. [61] show an O (1 /ϵ 7 ) sample complexity for robust actor-critic with R -contamination uncertainty sets. Li et al. [29] prove an ˜ O (1 /ϵ 2 ) sample complexity for robust natural actor-critic assuming the existence of an oracle to solve the inner optimization problem. Kumar et al. [27] develop a robust policy gradient method based on ℓ p norm uncertainty sets and extends its formulation under the s -rectangular assumption. All works mentioned above are limited to the tabular case with critic Q and adopt computationally infeasible uncertainty sets in the case of the large state space. Kuang et al. [26] illustrate a state disturbance view of Wasserstein metric-based RMDP, which can be generalized to the large state space. However, they only provide a policy iteration guarantee in the tabular setting (contraction under ∥ · ∥ ∞ ). Instead, we focus on large-scale robust RL, propose

two computationally efficient uncertainty sets, and demonstrate an ˜ O (1 /ϵ 2 ) (resp., ˜ O (1 /ϵ 4 ) ) sample complexity under linear (resp., general) function approximation. Additionally, we regard V as a critic instead of Q, which is closer to the actual implementation of on-policy natural actor-critic algorithms.

Peer reviews:
comments: SUMMARY:
This paper studies the actor-critic approach for robust RL. Especially, a Double-Sampling Uncertainty Set and an Integral Probability Metric Uncertainty Set are developed to overcome the curse of problem scale. A robust natural Actor-Critic algorithm is then proposed with convergence results. A significant number of experiments are designed.

STRENGTHS:
The paper is well-written and clear in general. The design of the two uncertainty sets is new and novel, which shows advantages under large-scale problems.

WEAKNESSES:
1. The motivation for designing the two uncertainty sets is somehow unclear to me. I understand there is uncertainty in the sets designed, but don't understand the motivation of this uncertainty set. In lines 640-646, the authors explain the uncertainty set contains transition kernels that are perturbed from the uniform distribution, this explanation seems unclear to me, and I can't understand the motivation for such a definition. 
2. One of the critical problems in studying robust RL with function approximation is the contraction of the approximated Bellman operator. The approach used in this paper is similar to the previous ones, i.e., use conditions on the radius of the uncertainty set and discount factor. This hence reduces the novelty and contribution of the paper.

QUESTIONS:
Can you explain the motivation for the designing of the uncertainty sets? In what sense do they imply robustness and uncertainty?

LIMITATIONS:
See the parts above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles robust reinforcement learning in large state spaces, where the transition kernel is accessible only in a nominal setting. The authors demonstrate that the $f$-divergence, R-contamination, and the $l_{p}$ norm are computationally infeasible in the context of robust RL for large state spaces. To overcome this limitation, the authors propose two new tractable uncertainty set formulations suitable for large dimensions: double sampling (DS) and the integral probability metric (IPM).\nDS involves independently and identically distributed state sampling following a transition kernel $p^o_{s,a}$. The IPM corresponds to the robust Bellman operator, but with a regularization term on the norm of weights. Both approaches are enabling Robust RL in scenarios previously hindered by computational complexity.\nThe paper introduces a new algorithm, the Robust Natural Actor Critic (RNAC), for training both a critic and an actor for the proposed IPM and double sampling uncertainty sets. The authors provide convergence guarantees for the RNAC algorithm.\nFinally, the authors demonstrate the efficency of their approach via two applications: one involving the suite of MuJoCo environments and the other, a real-world robotics application. These practical applications lend credence to the theoretical contributions and the robustness of the proposed approach.\n\nSTRENGTHS:\n- The authors provide two straightforward and computationally feasible uncertainty set formulations for large state spaces.\n- The paper offers substantial theoretical contributions.\n- The real-world robotics application lends credibility to the paper's robustness claims.\n\nWEAKNESSES:\n- The convergence guarantees are valid for $(s,a)$-rectangular uncertainty sets, which is rather limiting. However, it is commendable that the authors have acknowledged this limitation in their work and suggested it as an avenue for future research.\n- The paper seems incomplete without the appendix. The need to constantly refer to the appendix disrupts the flow of reading.\n\nQUESTIONS:\n- In lines 175-177, why did you not apply regularization to the bias parameter in the last layer? Moreover, this seems inconsistent with the provided implementation (lines 201-206 in the `RNAC-ppo/ppo_continious.py` file), where regularization is applied to all the critic's layers.\n\n- Doesn't the IPM's proposed regularization ultimately reduce the Lipschitz constant of the learned function, making the agent less sensitive to state variations?\n\nLIMITATIONS:\nThe authors mention their limitations as future research directions. We appreciate the authors for being upfront.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the sim-to-real transfer problem. It extends the learning of a robust policy by using the framework of robust Markov decision processes (RMDPs). It extends this paradigm to large state and action spaces using two uncertainty set formulations: double sampling, and integral probability metric.\nThese formulations are then used in the proposed algorithm robust natural actor critic (RNAC). RNAC is tested in MuJoCo as well as on a real robot.\n\nSTRENGTHS:\n* Proposed uncertainty sets as well as the RNAC algorithm seem like practical steps forward in sim-to-real transfer with robustness approaches.\n* The theoretical analysis seems interesting\n\nWEAKNESSES:\n* The experiments compare only to PPO. They should compare to other sim-to-real methods such as dynamics randomization [1] or action noise envelope [2].\n* The related works being relegated to the appendix seems like a red flag. The authors should better organize the paper to include related works and comparisons to the main paper.\n\n### References\n[1] Peng, X.B., Andrychowicz, M., Zaremba, W. and Abbeel, P., 2018, May. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA) (pp. 3803-3810). IEEE.\n\n[2] Jakobi, N., Husbands, P. and Harvey, I., 1995. Noise and the reality gap: The use of simulation in evolutionary robotics. In Advances in Artificial Life: Third European Conference on Artificial Life Granada, Spain, June 4–6, 1995 Proceedings 3 (pp. 704-720). Springer Berlin Heidelberg.\n\n\n================================\n\nThe additional experiments alleviate my concerns about the experimental analysis.\n\nQUESTIONS:\n* In the second to last paragraph of Section 2, when describing the single worst-case kernel, is this kernel constant for the entire state space or varying?\n* In the double sampling setup, does $\\sum \\alpha = 1$ in Equation (3)?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
RL methods trained on simulators suffered from generalization problems because of the "simulation-to-reality-gap". Previous works proposed robust RL methods in a tabular setting, with limited search spaces. The paper aims to develop a computationally tractable robust RL algorithm with large search spaces. To this end, the paper proposed two novel uncertainty sets and the first policy-based approach for robust RL with provable convergence guarantees.

STRENGTHS:
The paper studies a critical problem. Several technical contributions are proposed to devise a robust policy-based RL method with a large search space. The theoretical analysis seems solid.

WEAKNESSES:
The paper aims to devise a robust RL method. More real-world experiments are expected to demonstrate the robustness of the proposed methods.

---------------------
After reading the rebuttal, my main concerns were addressed.

QUESTIONS:
- Can previous tabulated-based robust RL methods be applied to the experiments in Sec. 7? If so, what about the comparison results? On the other hand, can the author provide some experiments to directly compare the proposed method with large search space to the previous one with limited search space to demonstrate the benefits of the proposed method?

LIMITATIONS:
The paper has discussed its limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-08-01