Title: Policy Gradient for Rectangular Robust Markov Decision Processes
Venue: NeurIPS 2023
Accepted: yes
Keywords: robust Markov decision process

policy gradient
Abstract: Policy gradient methods have become a standard for training reinforcement learning agents in a scalable and efficient manner. However, they do not account for transition uncertainty, whereas learning robust policies can be computationally expensive. In this paper, we introduce robust policy gradient (RPG), a policy-based method that efficiently solves rectangular robust Markov decision processes (MDPs). We provide a closed-form expression for the worst occupation measure. Incidentally, we find that the worst kernel is a rank-one perturbation of the nominal. Combining the worst occupation measure with a robust Q-value estimation yields an explicit form of the robust gradient. Our resulting RPG can be estimated from data with the same time complexity as its non-robust equivalent. Hence, it relieves the computational burden of convex optimization problems required for training robust policies by current policy gradient approaches.

Policy Gradient for Rectangular Robust Markov Decision Processes

Abstract

Policy gradient methods have become a standard for training reinforcement learning agents in a scalable and efficient manner. However, they do not account for transition uncertainty, whereas learning robust policies can be computationally expensive. In this paper, we introduce robust policy gradient (RPG), a policybased method that efficiently solves rectangular robust Markov decision processes (MDPs). We provide a closed-form expression for the worst occupation measure. Incidentally, we find that the worst kernel is a rank-one perturbation of the nominal. Combining the worst occupation measure with a robust Q-value estimation yields an explicit form of the robust gradient. Our resulting RPG can be estimated from data with the same time complexity as its non-robust equivalent. Hence, it relieves the computational burden of convex optimization problems required for training robust policies by current policy gradient approaches.

1 Introduction

Markov decision processes (MDPs) provide an analytical framework to solve sequential decisionmaking problems and seek the best performance in a fixed environment. Since the resulting policy can be highly sensitive to parameter values [18], the robust MDP setting alternatively maximizes return under the worst scenario, thus yielding robustness to uncertain environments [20, 12]. In practice, the robust MDP paradigm quantifies the level of uncertainty through a set U determining the possible range of model perturbations. Then, a policy is said to be robust-optimal if it reaches maximal performance under the most adversarial model within the uncertainty set. Developing efficient solvers for robust MDPs is of great interest, as it can lead to behavior policies with generalization guarantees [34].

If not computationally expensive, robust MDPs can be strongly NP-hard [33]. Thus, to preserve tractability, we commonly assume that U is convex and s -rectangular, i.e., U = × s ∈S U s [20, 12, 33]. Well established in the robust reinforcement learning (RL) literature [8, 3, 9, 10, 28], the latter assumption means that the overall uncertainty should be designed independently for each state. Further simplification may consider ( s, a ) -rectangular uncertainty sets of the form U = × ( s,a ) ∈X U ( s,a ) , albeit this naturally leads to more conservative strategies. In any case, planning in robust MDPs can be computationally costly, as it involves successive max-min problems [9, 1, 33]. To address this issue, the works [3, 14] have established an equivalence between robustness and regularization in RL in order to derive efficient robust planning methods for s and ( s, a ) -rectangular robust MDPs. Indeed, it appears that resorting to proper regularization instead of solving a minimization problem can yield robust behavior without requiring the polynomial time complexity of convex optimization problems [3].

Alternatively to planning, policy gradient algorithms (PG) directly learn an optimal policy by applying gradient steps towards better performance [25]. Due to its scalability, ease of implementation, and adaptability to many different settings such as model-free and continuous state-action spaces [13, 24], PG has become the workhorse of RL. Although regularization techniques such as max-entropy [7] or Tsallis [15] have shown robust behavior without impairing computational cost, they only account for adversarial reward [2, 6, 3]. Differently, robust PG formulations (RPG) formulations aim to address uncertainty to reward and transition functions.

Despite their ability to propel robust behavior, RPG methods that target robust optimal policies are still rare in the RL literature. The global convergence of RPG established in [16, 30] already ensures global convergence of our proposed algorithm, but motivates a practical method for estimating the gradients. Indeed, [16, 30] occult the estimation part, as they assume full access to the policy gradient. Alternatively, the inner loop solution proposed in [30][Sec. 4.1] requires solving convex optimization problems to find the worst model, which represents a larger time complexity of O ( S 4 A log ϵ -1 ) for ( s, a ) -rectangular, or O ( S 4 A 3 log ϵ -1 ) for s -rectangular uncertainty sets. These worst kernel and reward models are needed to compute RPG using the policy gradient theorem [26]. Other approaches that elicit an expression for RPG rely on a specific type of uncertainty set such as reward uncertainty with known kernel [3, 5], r -contaminated kernel with known reward [32], or ( s, a ) -rectangular uncertainty [16], whereas we aim to tackle more general robust MDPs.

In this work, we introduce an RPG method for both s and ( s, a ) -rectangular ball-constrained uncertainty sets, with similar complexity as non-robust PG. Our approach provides a closed-form expression of RPG without relying on an oracle while applying to the most common robust MDPs. To this end, we derive the worst reward and transition functions, thus revealing the adversarial nature of the corresponding uncertainty set. Surprisingly, we also find that the worst kernel is a rank-one perturbation of the nominal kernel. Leveraging this rank-one perturbation enables us to derive a robust occupation measure. We concurrently propose an alternative definition of the robust Q-value together with an efficient way to estimate it. Combining these results enables us to obtain RPG in closed form. Our resulting RPG update requires O ( S 2 A log ϵ -1 ) computations, thus showing similar time complexity as non-robust PG.

To summarize our contributions: (i) We establish the worst reward and transition models in closedform; (ii) We show that the worst-case transition function is a rank-one perturbation of the nominal; (iii) Weintroduce alternative robust Q-values that can be evaluated through efficient Bellman recursion while retrieving the robust value function; (iv) We establish an expression of RPG that can be estimated with similar time complexity as non-robust PG. Experiments show that our RPG speeds up state-of-the-art robust PG updates by 2 orders of magnitude.

2 Related work

Although some previous works use gradient methods to learn robust policies, they seek empirical robustness to adversarial behavior rather than robust MDP solutions [21, 29, 4]. In that sense, our study differs from adversarial RL as we explicitly optimize the max-min objective to find a robust optimal policy. Accordingly, the risk-averse approach focuses on the internal uncertainty due to the stochasticity of the system, whereas robust RL addresses the external uncertainty of the system's dynamics. As a result, common risk-averse objectives can be reformulated as robust problems with specific uncertainty sets [27].

Previous studies that did aim to derive robust policy-based methods are [3, 32, 30]. These are summarized in Table 1, which also displays the complexity of existing approaches. [3] established RPG for s -rectangular reward-robust MDPs, i.e., robust MDPs with uncertain reward but given kernel. Although it applies to general norms, their result does not account for transition perturbation. Differently, in [32], the authors introduced RPG for r -contaminated MDPs, i.e., robust MDPs with uncertainty set U := { R 0 }× [(1 -r ) P 0 + r ∆ S ×A S ] . Although it has similar complexity as non-robust PG, by construction, their setting is limited to ( s, a ) -rectangularity with known reward and mixed transition. As such, the proof techniques in [32] are tailor-made to the r -contamination framework and do not apply to more general robust MDPs. In fact, we remark that the r -contamination setting is equivalent to the action robustness approach introduced in [29], which emphasizes its limitation to action perturbation. Differently, our RPG holds whenever the worst kernel is a rank-one perturbation of the nominal transition function (see Lemma 4.4).

The work [16] provides a convergence proof of robust policy mirror-descent in the ( s, a ) -rectangular case, whereas we study robust policy optimization for s -rectangular uncertainty sets. In fact, its restriction to the ( s, a ) -case prevents us from transposing the analysis to our setting. This is due to the fact that the standard robust Bellman operator on Q -functions can no longer be applied on s -rectangular sets. To address generic robust MDPs, [30] recently introduced RPG for general uncertainty sets. Their gradient update has a complexity of O ( S 6 A 4 ϵ -4 ) , which is more expensive than non-robust PG by a factor of S 4 A 3 ϵ -4 . Both works [16, 30] additionally assume access to an oracle gradient of the robust return with respect to the transition model. Avoiding this oracle assumption naturally leads to even higher time complexity in [30] which is not scalable. At the same time, the two works [16, 30] guarantee global convergence of projected robust gradient iterates, thus establishing the potential promise of RPG. In fact, equipped with RPG convergence, the remaining challenge in making it practical is to efficiently estimate the gradient. This represents the main focus of our study: We aim to explicit an RPG method that generalizes existing results on specific uncertainty sets [3, 32] while holding for s -rectangular robust MDPs.

3 Preliminaries

Notation: We denote the cardinal of an arbitrary finite set Z by |Z| . Given two real functions a , b : Z → R , their inner product is ⟨ a , b ⟩ Z := ∑ z ∈Z a ( z ) b ( z ) , which induces the ℓ 2 -norm ∥ a ∥ 2 := √ ⟨ a , a ⟩ Z . More generally, the ℓ p -norm of a is denoted by ∥ a ∥ p whose conjugate norm is ∥ a ∥ q := max ∥ b ∥ p ≤ 1 ⟨ a , b ⟩ Z with q -1 = 1 -p -1 , by Hölder's inequality. The vector of all zeros (resp. all ones) with appropriate dimensions is denoted by 0 (resp. 1 ); the probability simplex over Z by ∆ Z := { a : Z → R + |⟨ a , 1 ⟩ Z = 1 } , and I Z designates the identity matrix in R Z ×Z . Given v ∈ R Z , we finally define the variance function as κ q ( v ) = min w ∈ R ∥ v -w 1 ∥ q and the mean function as ω q ( v ) ∈ arg min w ∈ R ∥ v -w 1 ∥ q (see Tab. 2 for their closed-form expression when q ∈ { 1 , 2 , ∞} ).

3.1 Markov Decision Processes

A Markov decision process (MDP) is a tuple ( S , A , γ, µ, P, R ) such that S and A are finite state and action spaces of cardinal S and A respectively, γ ∈ [0 , 1) is a discount factor and µ ∈ ∆ S the initial state distribution. Denoting X := S ×A , the couple ( P, R ) corresponds to the MDP model with P : X → ∆ S being a transition kernel and R : X → R a reward function. A policy π : S → ∆ A maps each state to a probability distribution over A , and we denote by Π the set of such functions. For any policy π ∈ Π , R π ∈ R S is the expected immediate reward defined as R π ( s ) := ⟨ π s , R ( s, · ) ⟩ A , ∀ s ∈ S , where π s is a shorthand for π ( ·| s ) . We similarly define the stochastic matrix induced by π as P π ( s ′ | s ) := ⟨ π s , P ( s ′ | s, · ) ⟩ A , ∀ s, s ′ ∈ S , and extend the occupation measure to an arbitrary initial vector ν ∈ R S by defining

Note that the initial vector here is not necessarily a probability measure: it can be the initial state distribution, but also the balanced value function introduced in Sec. 4[Eq. (2)]. The performance measure we aim to maximize is the value function v π ( P,R ) := ( I S -γP π ) -1 R π , or alternatively, the return ρ π ( P,R ) := ⟨ µ, v π ( P,R ) ⟩ S . We denote the optimal value function (resp. optimal return) by v ∗ ( P,R ) = max π ∈ Π v π ( P,R ) (resp. ρ ∗ ( P,R ) = ⟨ µ, v ∗ ( P,R ) ⟩ ). It can be obtained using Bellman operators, which are defined as T π ( P,R ) v := R π + γP π v and T ∗ ( P,R ) v := max π ∈ Π T π ( P,R ) v, ∀ v ∈ R S , respectively [22]. For any vector v ∈ R S , we associate its Q-function Q ∈ R X such that

With a slight abuse of notation, we can similarly define a Bellman operator over Q-values as

3.2 Robust Markov Decision Processes

In a robust MDP setting, we assume that ( P, R ) ∈ U and aim to maximize return under the worst model from the set. We denote the robust performance of a policy π ∈ Π by ρ π U := min ( P,R ) ∈U ρ π ( P,R ) . It is maximal when it reaches ρ ∗ U := max π ∈ Π ρ π U at an optimal robust policy π ∗ U ∈ arg max π ∈ Π ρ π U . When considering the robust value function v π U := min ( P,R ) ∈U v π ( P,R ) , we further need to assume that U is convex and rectangular so that an optimal robust policy realizing v ∗ U := max π v π U can be computed in polynomial time [33]. We thus assume U to be convex and rectangular in the remainder of this work. Specifically, we denote an ( s, a ) -rectangular uncertainty set by U sa := × ( s,a ) ∈X ( P ( s,a ) , R ( s,a ) ) . It represents a particular case of s -rectangular uncertainty which we similarly denote by U s := × s ∈S ( P s , R s ) . In both cases, there exists an optimal robust policy that is stationary, although all optimal ones may be stochastic [33].

Similarly to non-robust MDPs, robust MDPs can be solved through Bellman recursion. Indeed, the robust value function v π U (resp., optimal robust value function v ∗ U ) is known to be the unique fixed point of the robust Bellman operator T π U v := min ( P,R ) ∈U T π ( P,R ) v (resp., the optimal robust Bellman operator T ∗ U v := max π ∈ Π T π U v ), ∀ v ∈ R S , both being γ -contractions for the sup-norm. Although this ensures linear convergence of robust value iteration, the evaluation of each Bellman operator can still be prohibitive for practical use.

3.2.1 Ball Constrained Uncertainty set

To facilitate the computation of robust Bellman updates, we consider uncertainty sets that are centered around a nominal model ( P 0 , R 0 ) , i.e., of the form U = ( P 0 , R 0 )+( P , R ) , and constrained according

to ℓ p -norm balls [3, 14, 9, 1]. In the ( s, a ) -rectangular case, the corresponding uncertainty set is denoted by U sa p := R sa p ×P sa p = × ( s,a ) ∈X ( P ( s,a ) , R ( s,a ) ) where for any ( s, a ) ∈ X ,

Similarly, an s -rectangular norm-constrained uncertainty is denoted by U s p := × s ∈S ( P s , R s ) where for any s ∈ S ,

In both cases, the kernel uncertainty set conceals linear constraints ensuring all entries in P 0 + P are non-negative. Indeed, we generally ignore P 0 to satisfy these constraints in practice [23]. Although it may include absurd models and unnecessarily lead to conservative policies, this proxy region is appropriate for model-free robust learning. Moreover, the norm-ball structure on uncertainty sets above enables us to compute robust Bellman updates with similar time complexity as non-robust ones using regularization [3, 14]. We formalize this below.

Proposition 3.1. ([14, Thm. 2-3].) For any policy π ∈ Π and any rectangular ℓ p -ball-constraint uncertainty set, the robust Bellman operator is equivalent to its regularized form:

where Ω q ( α, β, v ) := -⟨ π s , α s, · + γκ q ( v ) β P s, · ⟩ A for ( s, a ) -rectangular uncertainty U sa p , and Ω q ( α, β, v ) := -( α s + γβ s κ q ( v )) ∥ π s ∥ q for s -rectangular uncertainty U s p .

In the following, we leverage the regularized formulation of robust value functions to explicitly derive RPG for rectangular ℓ p -ball uncertainty sets.

3.2.2 Robust Gradient Method

Since the robust return can be non-differentiable, we need to follow the projected sub-gradient ascent rule in order to optimize the robust return, namely, update π k +1 := proj Π ( π k + η∂ π ρ π k U ) where

η is the learning rate, proj Π denotes the orthogonal projection on Π , and ( P π U , R π U ) is the worst model associated with π ∈ Π and U , i.e., ( P π U , R π U ) ∈ arg inf ( P,R ) ∈U ρ π ( P,R ) .

Given oracle access to sub-gradient ∂ρ π U , projected gradient ascent converges to an ϵ -optimal policy π ∗ U . Moreover, under similar conditions as in the non-robust setting, projected gradient ascent holds an iteration complexity of O ( S 4 A 2 ϵ -4 ) [30]. Yet, the sub-gradient in (1) is generally intractable, particularly because general convex uncertainty sets may yield NP-hard complexity. Instead, we propose to focus on ball-constrained uncertainty sets in order to efficiently compute RPG updates.

4 Towards RPG: Expressing the worst quantities

In this section, we provide all the ingredients needed for deriving RPG. Before diving into the gradient expression, we first settle on the general framework of policy gradient. Secondly, in Sec. 4.1, we focus on expressing the worst model according to the nominal explicitly. Surprisingly, we find that the worst transition kernel is a rank-one perturbation of the nominal. This finding enables us to derive the robust occupancy measure, i.e., the visitation frequency of the worst kernel in Sec. 4.2. As a last piece, in Sec. 4.3, we propose an alternative definition of robust Q-value and show that it can be estimated from a specific Bellman recursion.

Consider again the projected gradient ascent rule:

By definition of the sub-gradient in (1) and applying the standard PG theorem [26], it holds that:

where Q π U := Q π ( P π U ,R π U ) is the Q-value associated with the worst-case model, and d π U := d π P π U the occupation measure of the worst transition kernel. In fact, for the uncertainty sets we focus on in this work, the worst Q-value Q π U retrieves the common definition of robust Q-value [20, 28] (see the appendix for a detailed discussion). Therefore, for conciseness and with a slight abuse, we shall designate Q π U by the robust Q-value, and d π U by the robust visitation frequency. The remaining question is how to compute these quantities and in particular, can we efficiently find the worst parameters ( P π U , R π U ) ? The following part of our study aims to address these questions.

Given an uncertainty set U , let first define the normalized and balanced robust value function as:

By construction, it has zero mean and unit norm, i.e., ⟨ u π U , 1 ⟩ S = 0 and ∥ u π U ∥ p = 1 . In fact, as stated in the result below, u π U is the gradient of the q -variance function, and correlates with the (unnormalized, unbalanced) robust value function according to the same q -variance.

Proposition 4.1. For any policy π ∈ Π and ℓ p -ball rectangular uncertainty set, the following holds:

4.1 Worst Kernel and Reward

In the following results, we explicit the relationship between the nominal and the worst-case model for ( s, a ) and s -rectangular ℓ p -balls. We will then leverage this relationship to compute the robust Q-values and the robust occupation measure, both necessary for RPG.

Theorem 4.2 ( ( s, a ) -rectangular case) . Given uncertainty set U = U sa p and any policy π ∈ Π , the worst model is related to the nominal one through:

Based on Thm. 4.2, it follows that in the ( s, a ) -rectangular case, the worst reward function is independent of the employed policy. As we establish in Thm. 4.3 below, this no longer applies under s -rectangularity. In either case, the worst kernel is policy-dependent, discouraging the system to move toward high-rewarding states and directing it to low-rewarding ones instead. Surprisingly, the vector penalty u π U ∈ R S additionally illustrates that the worst kernel is a rank-one perturbation of the nominal. Indeed, considering the stochastic matrix induced by any policy π ∈ Π , we have

so that the perturbation matrix P π U -P π 0 is of rank one. In the sequel, we will leverage this finding to compute the robust visitation frequency.

Theorem 4.3 ( s -rectangular case) . Given uncertainty set U = U s p and any policy π ∈ Π , the worst model is related to the nominal one through:

Similarly to the ( s, a ) -case, the adversarial kernel is a rank-one perturbation of the nominal. Yet, an extra dependence on the policy through the coefficient ( π s ( a ) ∥ π s ∥ q ) q -1 appears in the s -case, affecting both the worst reward and the worst kernel. Intuitively, it means that the worst model cannot be chosen independently for each action, but must instead depend on the agent's policy. This further explains why optimal policies can all be stochastic in s -rectangular robust MDPs [33].

Thms. 4.2 and 4.3 enable us to derive the worst MDP model in closed form with time complexity O ( S 2 A log ϵ -1 ) , up to logarithmic factors (please see the appendix for a detailed discussion). It thus holds the same complexity as non-robust value iteration, since we additionally need to compute the value function to derive its corresponding regularizer [3, 14]. On the other hand, if we employ convex optimization using value methods instead, obtaining the worst model requires a time complexity of O ( S 4 A log ϵ -1 ) in the ( s, a ) -rectangular case, and O ( S 4 A 3 log ϵ -1 ) in the s -rectangular case [30][Sec. 4.1].

4.2 Robust Occupation Measure

We finally derive the robust occupation measure using nominal values, which will lead to an explicit RPG. Although intractable in general, we show that focusing on ball-constrained uncertainty enables deriving the robust occupation matrix efficiently from the (nominal) occupation measure. We first establish the lemma below, which leverages the fact that the worst transition function is a rank-one perturbation of the nominal and represents our core contribution.

Lemma 4.4. Let b, k ∈ R S and P 0 , P 1 ∈ (∆ S ) S two transition matrices. If P 1 = P 0 -bk ⊤ , i.e., P 1 is a rank-one perturbation of P 0 , then their occupation matrices D i := ( I -γP i ) -1 , i = 0 , 1 are related through:

Combining Thms. 4.2 and 4.3 with the above lemma, we obtain the robust occupation in terms of the nominal, as stated in Thm. 4.6 below. Prior to this, we introduce the notion of expected transition uncertainty below.

Definition 4.5. Let U a rectangular ℓ p -ball-constrained uncertainty set of transition radius β . For any policy π ∈ Π , the expected transition uncertainty at any state s ∈ S is given by β π s := ∑ a ∈A π s ( a ) β s,a if U = U sa p , and β π s := β s ∥ π s ∥ q if U = U s p .

Theorem 4.6. For any rectangular ℓ p -ball-constrained uncertainty and π ∈ Π , it holds that:

Thm. 4.6 explicitly highlights the relationship between the robust visitation frequency and the nominal one. Thus, according to Eq. (3), the standard non-robust occupation measure in the first term needs to be penalized by another one, d π P 0 ,u π U = ( u π U ) ⊤ ( I S -γP π 0 ) -1 , to obtain the robust occupation measure. Recall that u π U is the balanced-scaled value function determined by π ∈ Π and uncertainty set U . Thus, the penalty term d π P 0 ,u π U tends to zero if all coordinates of the robust value function vector converge to the same value.

Nonetheless, our expression (3) does present some challenges. First, the visitation frequency appearing in the correction term indicates that instead of taking a fixed initial state distribution, we should start from a varying and signed measure represented by the balanced value function. Although it suggests putting more weight on worst-performing states, obtaining a non-biased estimator for this occupancy measure remains unclear in model-free learning. One may use importance sampling, but as any off-policy approach, both variance and bias would need to be controlled then. Such statistical analysis goes beyond the scope of this work.

4.3 Robust Q-values

In this section, we focus on the last element needed for RPG and aim to estimate the robust Qvalue denoted previously by Q π := Q π π π . Define its associated value function as v π ( s ) =

⟨ π s , Q π U ( s, · ) ⟩ , ∀ s ∈ S , π ∈ Π . Based on standard Bellman recursion, it thus holds that:

U ( P U ,R U ) U

while Q π U is the unique fixed point of the γ -contracting operator

The relations above hold for general uncertainty sets, provided that we have access to the worst model. The s -rectangularity assumption additionally enables us to retrieve the robust value function using the Bellman operator above [33]. Concretely, we have: v π U = min ( P,R ) ∈U v π ( P,R ) = v π ( P π U ,R π U ) .

The following result derives a regularized operator equivalent to L π U , which results in an efficient iteration method to compute the robust Q-value.

Proposition 4.7. The Bellman operator L π U defined in Eq. (4) is equivalent to:

5 Robust Policy Gradient

We are now able to derive an RPG by combining our previous results. Notably, unlike previous works that need to sample next-state transitions based on all models from the uncertainty set [21, 17, 4], here, we only need the nominal kernel to get the occupation measures.

Theorem 5.1 (RPG) . For any rectangular ℓ p -ball-constrained uncertainty, the robust policy gradient is given by:

where

The implementation of RPG directly follows and can be found in Alg. 1. Thm. 5.1 is a straightforward application of non-robust PG, as its proof simply consists in plugging Eq. (3) into the standard PG expression ∂ π ρ π U = ∑ ( s,a ) ∈X d π U ,µ ( s ) Q π U ( s, a ) ∇ π s ( a ) . We obtain a regular PG in the first term, with the robust Q-value instead of the non-robust one, plus a correction term c π resulting from taking the visitation frequency of the worst kernel instead of the nominal. Unlike previous work that uses policy regularization to achieve empirical robustness in PG methods [2, 11], Thm. 5.1 establishes an RPG that accounts for transition uncertainty and targets a robust optimal policy. It crucially relies on the rank-one-perturbation structure of the worst transition kernel (see Lem. 4.4). As established in Thm. 4.6, ℓ p -ball uncertainty implies such property, but the converse as to whether any convex set leads to the worst transition kernel being a rank-one perturbation of the nominal remains an open question. For example, it would be interesting to investigate the structural properties needed on the uncertainty set for the rank-one perturbation to hold.

5.1 Complexity Analysis

A major concern in solving robust MDPs is time complexity [33]. Similarly, it is of major importance to assess the additional time required for computing an RPG update, compared to its non-robust variant. Although previous work has analyzed the convergence rate of RPG to a global optimum [30], it assumes access to an oracle gradient, thus occulting the computational concerns raised from gradient estimation. In fact, the NP-hardness of non-rectangular and/or non-convex robust MDPs [33] already indicates that their resulting RPG can be intractable.

To compute RPG in Thm. 5.1, we first need to evaluate the robust Q-value. Based on Lemma 4.7 and the Bellman operators introduced there, our evaluation method involves an additional estimation of the variance function κ p . According to [14], this takes logarithmic time at most, using binary search. As to the compensation term c π in Eq. (11), it requires computing occupancy measures with respect to two different initial vectors, namely the balanced value function and the initial distribution. Thus, the computational cost for estimating c π is the same as estimating a non-robust occupancy measure. Tab. 1 summarizes the complexity of different approaches while a detailed discussion can be found in the appendix. We refer to [30][Sec. 4.1] for the complexity of RPG based on convex optimization.

Generalization to arbitrary norms. Until now, we have focused on ℓ p -norm for concreteness. However, the above results apply to any norm ∥·∥ , at least if the uncertainty set is ( s, a ) -rectangular, in which case the variance function changes to κ ( v ) := min ∥ c ∥≤ 1 , 1 ⊤ c =0 ⟨ c, v ⟩ and the balanced value to arg min ∥ c ∥≤ 1 , 1 ⊤ c =0 ⟨ c, v ⟩ . The rank-one perturbation structure of the worst kernel is preserved, so the robust occupation measure can be obtained similarly using Lemma 4.4. The s -rectangular is more involved. We defer its discussion to the appendix and leave its complete derivation for future work.

6 Experiments

In order to test the effectiveness of our RPG update, we evaluate its increased time complexity relative to non-robust PG. In the following experiments, we randomly generate nominal models for arbitrary state-action space sizes. Each experiment was averaged over 100 runs. We refer the reader to the appendix for more details on the radius levels and other implementation choices.

We first focus on ℓ 1 -robust MDPs to compare our RPG with a convex optimization approach. Specifically, we consider a robust PG with an optimization solver, which we designate by LP-RPG. Indeed, recall that ℓ 1 -ball-constraints induce a linear program (LP) rather than a more general convex optimization problem. Therefore, to compute the robust value function for a given policy, we iteratively evaluate the robust Bellman operator using LP [30, Section 4.1]. Using this approximated value function, we can compute the worst value parameters to apply PG theorem by [26] and deduce an LP-based robust PG update. Differently, our RPG method relies on the regularized formulation of robust value iteration proposed in [3, 14], from which we deduce the normalized-balanced value function as in Eq. (2). We finally apply Thm. 4.6 to compute the robust occupation measure, and Prop. 4.7 to obtain the robust Q-value.

Tab. 3 displays the results obtained for the two alternative methods described above. In all experiments, the standard deviation was typically 2-10% so we omitted it for brevity. As can be seen in Tab. 3, LP-RPG does not scale well compared to RPG, whereas RPG has similar time complexity as PG. Notably, the running time of s -rectangular LP-RPG scales much better with the space size than its ( s, a ) -rectangular equivalent, which confirms the theoretical complexities from Tab. 1. Yet, since these methods were time-consuming, we repeated these for a few runs only. In fact, LP-RPG is more expensive than RPG by 1-3 orders of magnitude, which illustrates its inefficiency. We emphasize that here, we only focused on ℓ 1 -robust MDPs to leverage LP solvers in robust policy evaluation. We expect the computational cost of LP-RPG to scale even more poorly for other ℓ p -robust MDPs that involve polynomial time-consuming convex programs.

We further compare our RPG to non-robust PG on different ℓ p -balls. Tab. 4 confirms the comparable time complexity of RPG to non-robust PG, thus demonstrating the effectiveness of our method. We note that for p ∈ { 1 , 2 , ∞} , the corresponding regularization quantities can be computed in closed form, whereas they involve a binary search for other values [14]. We thus get a slight running-time increase for p ∈ { 5 , 10 } .

7 Discussion

This paper introduced an explicit expression of RPG for rectangular robust MDPs. Our approach involved auxiliary results such as deriving the worst model in closed form and showing that it is a rank-one perturbation of the nominal kernel. The resulting RPG extends vanilla PG with additional correction terms that can be derived in closed form as well. Thus, the computational time of RPG is similar to its non-robust variant.

A key assumption that would be interesting to relax is the normed-ball structure of the uncertainty sets considered in this study. Indeed, since the proofs of our technical results rely on norm properties, it is still unclear if and how RPG can generalize to metric-based or f -divergence uncertainty sets. The latter type of uncertainty can be particularly useful for data-driven settings, as the radius can be chosen according to cross-validation or statistical bounds [10]. Another compelling direction would be to explore other variants of RPG using mirror descent or natural policy gradient and examine their compatibility with deep architectures, which would further demonstrate the practical efficiency of our RPG method.

References

Bahram Behzadian, Marek Petrik, and Chin Pang Ho. Fast algorithms for ℓ ∞ -constrained s-rectangular robust MDPs. Advances in Neural Information Processing Systems , 34:2598225992, 2021.

Rob Brekelmans, Tim Genewein, Jordi Grau-Moya, Grégoire Delétang, Markus Kunesch, Shane Legg, and Pedro Ortega. Your policy regularizer is secretly an adversary. Transactions on Machine Learning Research (TMLR) , 2022.

Esther Derman, Matthieu Geist, and Shie Mannor. Twice regularized MDPs and the equivalence between robustness and regularization. Advances in Neural Information Processing Systems , 34:22274-22287, 2021.

Esther Derman, Daniel J. Mankowitz, Timothy A. Mann, and Shie Mannor. Soft-robust actorcritic policy-gradient. AUAI press for Association for Uncertainty in Artificial Intelligence , pages 208-218, 2018.

Esther Derman, Yevgeniy Men, Matthieu Geist, and Shie Mannor. Twice regularized Markov decision processes: The equivalence between robustness and regularization. arXiv preprint arXiv:2303.06654 , 2023.

Benjamin Eysenbach and Sergey Levine. Maximum entropy RL (provably) solves some robust RL problems. International Conference on Learning Representations , 2022.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning , pages 1861-1870. PMLR, 2018.

Chin Pang Ho, Marek Petrik, and Wolfram Wiesemann. Fast Bellman updates for robust MDPs. In International Conference on Machine Learning , pages 1979-1988. PMLR, 2018.

Chin Pang Ho, Marek Petrik, and Wolfram Wiesemann. Partial policy iteration for l 1 -robust Markov decision processes. J. Mach. Learn. Res. , 22:275-1, 2021.

Chin Pang Ho, Marek Petrik, and Wolfram Wiesemann. Robust ϕ -divergence MDPs. arXiv preprint arXiv:2205.14202 , 2022.

Hisham Husain, Kamil Ciosek, and Ryota Tomioka. Regularized policies are reward robust. In International Conference on Artificial Intelligence and Statistics , pages 64-72. PMLR, 2021.

Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research , 30(2):257-280, 2005.

Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In International Conference on Machine Learning . Citeseer, 2002.

Navdeep Kumar, Kfir Levy, Kaixin Wang, and Shie Mannor. Efficient policy iteration for robust markov decision processes via regularization, 2022.

Kyungjae Lee, Sungjoon Choi, and Songhwai Oh. Sparse Markov decision processes with causal sparse Tsallis entropy regularization for reinforcement learning. IEEE Robotics and Automation Letters , 3(3):1466-1473, 2018.

Yan Li, Tuo Zhao, and Guanghui Lan. First-order policy optimization for robust Markov decision process. arXiv preprint arXiv:2209.10579 , 2022.

Daniel Mankowitz, Timothy Mann, Pierre-Luc Bacon, Doina Precup, and Shie Mannor. Learning robust options. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Shie Mannor, Duncan Simester, Peng Sun, and John N Tsitsiklis. Bias and variance approximation in value function estimates. Management Science , 53(2):308-322, 2007.

Paul Milgrom and Ilya Segal. Envelope theorems for arbitrary choice sets. Econometrica , 70:583-601, 02 2002.

Arnab Nilim and Laurent El Ghaoui. Robust control of Markov decision processes with uncertain transition matrices. Operations Research , 53(5):780-798, 2005.

Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In International Conference on Machine Learning , pages 2817-2826. PMLR, 2017.

Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons, 2014.

Aurko Roy, Huan Xu, and Sebastian Pokutta. Reinforcement learning under model mismatch. Advances in Neural Information Processing Systems , 2017.

John Schulman, Xi Chen, and Pieter Abbeel. Equivalence between policy gradients and soft q-learning, 2017.

Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction . The MIT Press, second edition, 2018.

Richard S Sutton, David A McAllester, Satinder P Singh, Yishay Mansour, et al. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems , volume 99, pages 1057-1063. Citeseer, 1999.

Aviv Tamar, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. Policy gradient for coherent risk measures. Advances in neural information processing systems , 28, 2015.

Aviv Tamar, Shie Mannor, and Huan Xu. Scaling up robust MDPs using function approximation. In International Conference on Machine Learning , pages 181-189. PMLR, 2014.

Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applications in continuous control. In International Conference on Machine Learning , pages 6215-6224. PMLR, 2019.

Qiuhao Wang, Chin Pang Ho, and Marek Petrik. Policy gradient in robust MDPs with global convergence guarantee. In International Conference on Machine Learning , pages 35763-35797. PMLR, 2023.

Yue Wang, Fei Miao, and Shaofeng Zou. Robust constrained reinforcement learning. arXiv preprint arXiv:2209.06866 , 2022.

Yue Wang and Shaofeng Zou. Policy gradient method for robust reinforcement learning. In International Conference on Machine Learning , pages 23484-23526. PMLR, 2022.

Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem. Robust Markov decision processes. Mathematics of Operations Research , 38(1):153-183, 2013.

Huan Xu and Shie Mannor. Robustness and generalization. Machine learning , 86:391-423, 2012.

Contents

A Balanced and Normed Vectors

In this section, we lay down some basic properties of p -normalized-balanced vectors.

First recall the p -variance and the p -mean defined as:

Given any v ∈ R S , let also the p -balanced-normalized function:

According to [14][Sec. 16.1, Lemma 1], the following holds:

namely, the p -variance function is the optimal value of a linear optimization under kernel noise constraint. The result below further characterizes the solution to the above problem.

Lemma A.1. The vector defined as c ∗ := -ϵu q ( v ) is an optimal solution to the optimization problem

Proof. It suffices to show that c ∗ satisfies both constraints ∥ c ∗ ∥ p ≤ ϵ and ⟨ c ∗ , 1 ⟩ S = 0 , and that it reaches optimal value, i.e., -1 ϵ ⟨ c ∗ , v ⟩ = κ q ( v ) . We thus compute:

(By definition, κ q ( v ) = ∥ v -ω q 1 ∥ q )

so the norm constraint is satisfied. We check the noise constraint by computing:

Now, considering the real function φ : w → ∥ v -w 1 ∥ q and taking its derivative, we remark the proportional relation:

where C ∈ R is the proportionality coefficient. By construction, ω q ( v ) is a minimizer of φ , so we must have φ ′ ( ω q ( v )) = 0 and c ∗ satisfies the noise constraint.

We finally show that c ∗ reaches the optimal value:

A.1 Proof of Proposition 4.1

Proposition. For any policy π ∈ Π and ℓ p -ball rectangular uncertainty set, the following holds:

Proof. The second claim directly follows from Lemma A.1 applied to v := v π U , so that by optimality, κ q ( v π U ) = ⟨ u π U , v π U ⟩ . For the first claim, we take the gradient of κ p ( v ) := min w ∈ R ∥ v -w 1 ∥ p w.r.t. v using the envelope theorem [19]. Then, the p -balanced-normalized vector u p ( v ) is a sub-gradient of κ p ( v ) , that is,

which we apply to v := v

We have the additional properties below:

The variance function κ q is translation-invariant in all-ones directions, i.e., for all ω ∈ R , κ q ( v ) = κ q ( v + ω 1 ) . As a result, ⟨∇ κ q ( v ) , 1 ⟩ S = 0 .

The balanced-normalized vector u p ( v ) has unit norm, i.e., ∥ u p ( v ) ∥ p = 1 by Lemma A.1.

B Worst Kernel and Reward

Here we present the proofs for the worst/adversarial kernel and reward function characterization.

B.1 Proof of Theorem 4.2

Theorem ( ( s, a ) -rectangular case) . Given uncertainty set U = U sa p and any policy π ∈ Π , the worst model is related to the nominal one through:

Proof. By definition, where

Additionally, since U sa p = ( R 0 + R ) × ( P 0 + P ) , it results that:

By the ( s, a ) -rectangularity assumption, we get that for all ( s, a ) ∈ X ,

It is clear from the above that the worst reward is independent of policy π . Thus, by the ball constraint, it is given by

Differently, the worst kernel depends on the value function which itself depends on the policy. It is given by

The optimization is of the form so by Lemma A.1,

As a result, we proved that for all ( s, a ) ∈ X , R π U sa p ( s, a ) = R 0 ( s, a ) -α s,a and

B.2 Proof of Theorem 4.3

Theorem ( s -rectangular case) . Given uncertainty set U = U s p and any policy π ∈ Π , the worst model is related to the nominal one through:

Proof. By definition, where

and since U s p = ( R 0 + R ) × ( P 0 + P ) , we have

By the s -rectangularity assumption, we get that for all s ∈ S

Here, the worst reward does depend on policy π and is given by

As for the worst kernel, it depends both on the value function and the policy. It is given by

The optimization of interest is of the form

which is equivalent to the following two-fold minimization:

Thus, rewriting the problem in our context,

Computing the optimal β above, the optimization is now the same as in the ( s, a ) -rectangular case. Hence, we have

π .

which ends the proof by definition of the balanced value function u U

C Occupation Matrix

C.1 Proof of Lemma 4.4

Lemma. Let b, k ∈ R S and P 0 , P 1 ∈ (∆ S ) S two transition matrices. If P 1 = P 0 -bk ⊤ , i.e., P 1 is a rank-one perturbation of P 0 , then their occupation matrices D i := ( I -γP i ) -1 , i = 0 , 1 are related through:

Proof. By definition, D 1 = ( I S -γP 1 ) -1 so it follows that:

Now, multiplying both sides by k and noticing that k ⊤ D 0 b is a scalar we get

Combining Eqs. (7) and (8) thus yields:

which concludes the proof.

C.2 Proof of Theorem 4.6

Theorem. For any rectangular ℓ p -ball-constrained uncertainty and π ∈ Π , it holds that:

Proof. From Thms. 4.2 and 4.3, it holds that:

Therefore, setting P 1 := P π U , P 0 := P π 0 , b := β π and k := u π U , we can apply Lemma 4.4 and relate the corresponding occupation matrices. Additionally multiplying both sides of the relation by µ ⊤ ∈ R 1 ×S yields the desired result.

D Robust Q-value

D.1 Basic Properties

In the literature, robust Q-values are defined in various ways that turn out to be conflicting for s but non-( s, a ) rectangular uncertainty sets. In this section, we propose to define the robust Q-value solely based on the worst model. Define the robust Q-value, the robust value function, and the robust occupation respectively as:

For s -rectangular uncertainty sets (in particular, for ( s, a ) -rectangular), the above definition of robust value function coincides with the common one, i.e., v π ( P π U ,R π U ) = min ( P,R ) ∈U v π ( P,R ) [33]. If the uncertainty set is additionally ( s, a ) -rectangular (as in [31] or [3, 14]), the above definition of robust Q-value also coincides with the common one because then,

Getting back to our own definition, robust Q-value and value functions are related through:

as both quantities are defined based on worst kernel and reward, i.e., Q π U := Q π ( P π U ,R π U ) and v π U := v π ( P π U ,R π U ) .

Given an optimal robust policy π ∗ U , we further use P ∗ U , R ∗ U , v ∗ U , Q ∗ U , d ∗ U as a shorthand for P π ∗ U U , R π ∗ U U , v π ∗ U U , Q π ∗ U U , d π ∗ U U respectively. For ( s, a ) -rectangular uncertainty set U sa , the optimal value function is the best optimal Q-value, that is

because an optimal policy deterministically takes the action with the highest Q-value [20, 12]. This does no longer hold for s -rectangular or coupled uncertainty sets, as there, an optimal policy may be stochastic [33]. Still, based on Thms. 4.2 and 4.3, we get the Bellman recursion below.

Proposition D.1. Let an ℓ p -ball constrained uncertainty set. Then, for all ( s, a ) ∈ X and π ∈ Π , the robust Q-value satisfies the following recursion in the ( s, a ) and s -rectangular case respectively:

Proof. We give proof for the ( s, a ) -rectangular case only. The s -rectangular case follows the exact same lines except that it uses Thm. 4.3 instead of Thm. 4.2. We have:

The above recursion applies the standard Bellman operator on robust Q-values. We can similarly apply it on the robust value function (itself can be computed efficiently based on [3, 14]).

Corollary D.2. Let an ℓ p -ball constrained uncertainty set. Then, for all ( s, a ) ∈ X and π ∈ Π , the robust Q-value satisfies the following recursion in the ( s, a ) and s -rectangular case respectively:

D.2 Evaluation

Based on the Bellman recursion above, we now derive robust Q-learning equations to learn a robust Q-value. Precisely, we investigate if the linear operator below is contracting and can be evaluated efficiently:

where ( P π U ,v , R π U ,v ) ∈ arg min ( P,R ) ∈U T π ( P,R ) v and v ( s ) = ⟨ π s , Q ( s, · ) ⟩ A , ∀ s ∈ S .

Proposition D.3. Consider an ℓ p -ball constrained uncertainty set. Then, for all Q ∈ R X and π ∈ Π , the operator L π can be evaluated as:

where for all Q ∈ R X , its corresponding value is v ( s ) := ⟨ π s , Q ( s, · ) ⟩ , ∀ s ∈ S .

Proof. We give proof for the ( s, a ) -rectangular case only. The s -rectangular case follows the exact same lines except that we take the worst model for s -rectangular balls. By definition,

D.3 Convergence

In the remainder of this section, we focus on ℓ p -ball constrained uncertainty sets of the form U sa p or U s p . Let our Q-value iteration Q n +1 := L π U Q n , and denote v n ( s ) = ⟨ π s , Q n ( s, · ) ⟩ A , ∀ s ∈ S , n ∈ N .

Proposition D.4. For all Q ∈ R X , denote v ( s ) := ⟨ π s , Q ( s, · ) ⟩ A , ∀ s ∈ S . Then, for any policy π ∈ Π , the Q-value iteration defined according to Q n +1 = L π U Q n induces

Proof. By construction, for all s ∈ S we have

where the last equality holds because ( P π U ,v n , R π U ,v n ) ∈ arg min ( P,R ) ∈U T π ( P,R ) v n .

As a result of the above proposition, the value iteration induced by our Q-value iteration rule converges linearly to the robust value function, i.e., ∥ v n -v π U ∥ ∞ ≤ γ n ∥ v 0 ∥ ∞ . Therefore, Q-value iterates converge to a fixed point. Precisely, v n → n v π U implies that ( P π U ,v n , R π U ,v n ) → n ( P π U , R π U ) , which in turn implies that Q n → n Q π U . The result below further characterizes the convergence rate.

Proposition D.5. For any policy π ∈ Π , the recursion Q n +1 := L π U Q n , n ∈ N converges linearly to Q π U .

Proof. Thanks to Prop. D.4, we have:

where by the worst kernel characterization, B π ( s, a ) := β s,a for U = U sa p and B π ( s, a ) := β s ( π s ( a ) ∥ π s ∥ q ) q -1 for U = U sa p . This implies

Here, u n , u π U is the balanced-normalized vector associated with vector v n and v π U , respectively. Recall that the p -balanced normalized vector u associated with vector v is given by

where κ p ( v ) = min w ∥ v -w 1 ∥ p and ω p ( v ) = w ∥ v -w 1 ∥ p . It is easy to see that ω p , κ are Lipschitz in v . Hence, ∥ u n -u π U ∥ ∞ ≤ C · Pol ( ∥ v n -v π U ∥ ∞ ) · ψ ( κ ( v π U ) , S, A ) , where Pol is a polynomial and ψ a real function. This implies that

which concludes our proof.

E Robust Policy Gradient

Theorem (RPG) . For any rectangular ℓ p -ball-constrained uncertainty, the robust policy gradient is given by:

where

Proof. The proof directly follows from plugging the robust occupation measure of Thm. 4.6 and the robust Q-value into standard policy-gradient theorem [26].

F Complexity Analysis

In this section, we aim to derive the complexity of one RPG iteration for different uncertainty sets. We first focus on non-robust MDPs, to then see how the complexity increases with the uncertainty structure.

Non-robust MDPs. Computing non-robust Q-values and occupation measure takes O ( S 2 A log( ϵ -1 )) each, which represent the most expensive computations in PG. The product of d π , Q π and ∇ π in PG requires O ( SA ) operations, which is insignificant. Hence, the total cost of PG corresponds to the computational cost of Q-values. More precisely, approximate the Q-value by Q and the occupation measure by d with ϵ SA tolerance, that is, ∥ Q -Q π ∥ ∞ ≤ ϵ SA and ∥ d -d π ∥ ∞ ≤ ϵ SA . This involves O ( S 2 A log( SAϵ -1 )) operations for each. Then, we have

so the exact complexity of policy gradient for non-robust MDP is O ( S 2 A log( SAϵ -1 ) .

Convex non-rectangular uncertainty set. Robust policy improvement is strongly NP-Hard for non-rectangular uncertainty sets, even if convex [33]. The PG method finds global optimal given oracle access to policy gradient in polynomial time [30]. This implies that the computation of RPG must be NP-Hard.

F.1 Helper results for Robust MDPs

Variance and mean functions. Computing κ p ( v ) (resp. ω p ( v ) ) with ϵ -tolerance requires O ( S log( Sϵ -1 )) (resp. O ( S log( ϵ -1 )) ) computations if we use binary search [14].

Occupation measure. Let k ∈ R S be any vector. By definition,

and

Since P π is a stochastic matrix, ∥ P π ∥ ≤ 1 and

This implies that ∑ N -1 n =0 γ n k ⊤ ( P π ) n is an O ( γ N ) approximation of d π P,k . Now, take u 0 = k and

then ∑ N -1 n =0 γ n k ⊤ ( P π ) n = ∑ N -1 n =0 u n . Each iteration takes O ( S 2 ) computations, leading to total cost O ( S 2 N ) for N iterations. Computing P π from P is O ( S 2 A ) . We conclude that computing the occupation measure has a complexity of O ( S 2 A + S 2 log( ϵ -1 )) .

Lemma F.1. We can approximate d π P,k by ∑ N -1 n =0 γ n ( k ′ ) ⊤ ( P π ) n with complexity O ( S 2 A + S 2 log( ϵ -1 )) and tolerance

Proof.

Computing Q-value given value function. Let v be an ϵ 1 approximation of robust value function v π U , that is ∥ v -v π U ∥ ∞ ≤ ϵ 1 . We want to compute the Q-value using the relation:

where Ω U s p ( v π U s p , π ) = π s ( a ) q -1 ∥ π s ∥ q -1 q ( α s + γβ s κ q ( v π U s p ) ) and Ω U sa p ( v π U sa p , π ) = α sa + γβ sa κ q ( v π U sa p ) . Let Q be approximated from v by

We thus have

so that ∥ Q -Q π U ∥ ∞ ≤ O ( S 1 q ϵ 1 ) .

Lemma F.2. The variance function κ p is Lipschitz. More precisely,

Proof. Let v i ∈ R S and w i ∈ arg min w ∈ R ∥ v i -w 1 ∥ p for i = 1 , 2 . Without loss of generality, further assume that κ p ( v 1 ) ≥ κ p ( v 2 ) . Then,

Lemma F.3. Q π U sa p can be approximated to ϵ tolerance with the same complexity as computing v π U sa p to S -1 q ϵ .

Proof. We can compute the value function with S -1 q ϵ tolerance. The rest of the operations are insignificant. The result follows from above.

F.2 RPG Complexity

Let O sa p ( ϵ ) be the complexity of computing v π U sa p with ϵ -tolerance (see [14] for more details). Calculating Q-value up to ϵ 1 requires O sa p ( S -1 q ϵ 1 ) computations according to Lemma F.3. Letting d 1 and d 2 be ϵ 2 -approximations of d π P 0 ,µ and d π P 0 ,k respectively, their complexity is insignificant compared to O sa p ( S -1 q ϵ 2 ) . Now, approximating the gradient using d 1 , d 2 , Q, ∇ π as in Thm. 5.1 has a complexity

of O ( SA ) . Since the uncertainty set U is compact, all quantities are bounded. There are O ( SA ) operations in Thm. 5.1, so taking ϵ 1 , ϵ 2 = O ( ϵ SA ) , we get O ( ϵ ) for the gradient. Hence, the total complexity is O sa p ( S -1 q -1 A -1 ϵ ) which is ˜ O ( S 2 A log( ϵ -1 )) by hiding log factors. A similar analysis follows for the s -rectangular case.

G Generalization to arbitrary norms

In this section, we analyze how to generalize our result to general norms that are not ℓ p .

G.1 ( s, a ) -rectangular robust MDPs

Consider an ( s, a ) -rectangular uncertainty set U = U sa ∥·∥ constrained by:

The robust Bellman operator T π U can be evaluated as

where the variance function is defined as

This can be used to compute the robust value function. Then the worst values can found using robust Bellman operator T π U and robust value function v π U as

It is easy to see that the worst values are given as

where normalized-balanced value function u π U is a solution to

Observe that the worst kernel is still a rank-one perturbation of the nominal kernel. Hence, the robust occupation measure can be obtained using Lemma 4.4 as

where β π ( s ) = ∑ a π s ( a ) β s,a . The last ingredient to compute RPG is the robust Q-value which can be computed using robust value function and worst values. However, it can be computed directly using the following iterates:

as Q n converges to robust Q-value Q π U linearly.

The proofs of the above claims are very similar to the ℓ p case so they are omitted. Finally, computing the variance function κ ∥·∥ and the normalized-value function u π U can be done using numerical convex optimization methods for general norms. For the ℓ p case, these can be obtained in concrete form, so we choose to focus on ℓ p in our main study.

G.2 s -rectangular case

Generalization to s -rectangular balls of a general norm is not straightforward and may not be possible for all types of norms. The crucial property of the ℓ p -norm exploited in our rank-one perturbation proof is 'decoupling', that is, for all x ∈ R A×S , there must exist k, l, m ∈ R + such that

This holds in the ℓ p case with k = l = m = p . Further analysis of this setting is left for future work.

G.3 Generalization to metrics and divergences

Generalizing RPG to distance or divergence-based uncertainty sets is also not obvious. Our RPG method, in particular the robust occupation measure, crucially relies on the rank-one perturbation characterization of the worst kernel, which might not apply for example with KL-ball constraints. We leave this analysis for future work.

H Experiments

Parameters. All the nominal transition kernels and reward functions are generated randomly. The number of states and the number of actions are varied. Discount factor γ = 0 . 9 , reward noise radius α s,a , α s = 0 . 1 , transition noise kernel β s,a , β s = 0 . 01 SA .

Hardware Experiments are done on the machine with the following configuration: Intel(R) Core(TM) i7-6700 CPU @3.40GHZ, size:3598MHz, capacity 4GHz, width 64 bits, memory size 64 GiB.

Software and codes All the experiments were done in Python using numpy, matplotlib. All codes and results are available at https://github.com/navdtech/rpg.

Procedure and Results. All the experiments were repeated 100 times, except for Linear Programming (LP) cases as LP methods were very time-consuming. In LP methods, experiments were repeated 5 times except for the case ( S = 500 , A = 100 ) which was done only once. As this case was prohibitively expensive. Standard deviation in all cases was less than 10% , and typically 1 -2% . This conveniently illustrates the superiority of our methods over LP methods.

Observations

Scalability of our methods. Note that our methods scale very well with large state-action space. It takes a (small) constant times the time required by non-robust MDPs. On the other hand, LP methods explode. Both observations confirm the theoretical time complexity.

sa -case vs s case in LP methods. We see s -case outperforms sa -case for small state-action spaces via LP methods. This is opposite to the theoretical time complexity of s -case which expensive than sa -case. We believe this is due to the internal implementation issues. Note that computing the robust value function is the most expensive step which requires evaluation of the robust Bellman operator. In sa case, one evaluation requires solving SA LP programs with S variables each, while for s -case, it is S LP programs with SA variables each. To solve LP, scipy.linprog is used, we believe it does some parallelization for large LPs. Hence, we observe less cost for sa -case. However, we observe that the cost of s -case increases much faster than s -case, and eventually under-performing than sa -case.

H.1 RPG by LP

We compute RPG using LP in the following steps:

(Robust Value Iteration) Approximately compute the robust value function v π U using the iterates v n +1 := T π U v n . Evaluation of robust Bellman operator T π U is done via LPs as described below. This is the most expensive step as it requires evaluating robust Bellman operators O (log( ϵ -1 )) times, and each evaluation requires many LPs.

(Adversarial Values) Compute the worst values ( P π U , R π U ) using the robust value function from the following relation:

This is also solved by LP.

(Policy Gradient Theorem) We now compute the RPG using policy gradient Theorem [26] w.r.t. the adversarial values computed above, as

Observe that d π P can be approximated as ∑ n n =0 ( γP π ) n for large n enough, and Q π P,R can be approximated by dynamic programming [25]. Notably, this step and the second step are negligible as compared to the first step.

Robust Value Iteration by LP

sa -rectangular robust MDPs. We first consider sa -rectangular L 1 constrained uncertainty set U sa p = P × R . Robust Bellman operator is given by

Note that the above can be solved by A LPs as uncertainty set U sa p = P ×R induces linear constraint and the objective is also linear with S variables. Hence, evaluation of T π U sa p v requires solving SA LPs with S variable each.

s -rectangular robust MDPs. We now consider s -rectangular L 1 constrained uncertainty set U s p = P × R . Robust Bellman operator is given by

Note that the above can be solved by one LP as uncertainty set U s p = P ×R induces linear constraint and the objective is also linear with SA variables. Hence, evaluation of T π U s p v requires solving S LPs with SA variable each.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates policy gradient methods in the context of unknown transition kernels in Markov Decision Processes (MDPs). The authors focus on a robust approach, where the uncertain transition kernel is assumed to lie within a specific uncertainty set, known as a rectangular uncertainty set. The authors address the computation of robust policy gradients in this setting, aiming to achieve the same time complexity as in the non-robust case.\n\nSTRENGTHS:\nThe combination of policy gradients with robust MDPs is an interesting and practical approach with promising initial results. The paper is well-written and accessible, making it relatively easy to understand.\n\nWEAKNESSES:\nThere are a couple of issues regarding the modelling of the uncertainty set:\n1) In Section 3.2.1 it is written that the radius $\\beta$ of the uncertainty set should be “small enough” such that transition kernels of the form $P_0 + \\mathcal{P}$ are well defined. This, however, requires the nominal transitions kernel $P_0$ to be strictly positive everywhere. Moreover, small positive values of $P_0$ are also not enough. Therefore, this modelling seems highly restrictive - particularly for high-dimensional settings.\n2) The motivation for using rectangular ambiguity sets is not clear. In practice, ambiguity sets are often non-rectangular and derived from principles like maximum likelihood estimation. Could the method handle more general uncertainty sets?\n\nQUESTIONS:\n1) See the two questions raised as weaknesses?\n2) Uncertainty sets are often chosen with the goal to achieve some statistical properties of the learned solution (here policy gradient). Is there a statistical motivation for choosing the specific uncertainty set you choose? Or in other words, why not choosing another uncertainty set than the one you chose?\n3) In Section 3.1, what is the relation between $k$ and $\\mu$?\n4) How do you choose the radius $\\beta$?\n5) What is the motivation/justification for $\\mathcal{U}$ being convex?\n6) Line 106 has a typo. Should be $(P,R)$\n\nLIMITATIONS:\n1) Uncertainty set modelling is a) not well motivated and b) it seems rather restrictive. Please see the questions and points raised above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the robust policy gradient (RPG) algorithm as an efficient approach to solve rectangular RMDPs. The results are based on an explicit characterization of $\\ell_p$-ball-constrained RMDPs, where the worst-case kernel can be viewed as a rank-one perturbation of the nominal kernel and the worst-case occupation frequency can also be written in closed form, so that the well-known policy gradient theorem can still be applied to calculate the sub-gradient for the RMDP. It is further pointed out in the appendix that these results are all closely related to a fundamental characterization of a constrained inner product minimization problem. The paper also redefines the robust value functions using the worst-case kernel, and shows its equivalence with the common definition in literature, so that it comes up with a more efficient way to calculate the value function of RMDPs, which constitutes as the last piece for evaluating the sub-gradient. Numerical simulations are presented to show the efficiency of the proposed algorithm.\n\nSTRENGTHS:\n1. The flow of this paper is very good. It provides the reader with enough background information to understand the framework, and the final algorithm naturally builds upon the building blocks established in the first few sections so that the readers can easily acquire the intuition.\n2. The paper consists of a thorough literature review that justifies the place of this result in the spectrum of related works. The reviewer particularly appreciates the table presented in Section 2 that compares the contributions a full range of related works.\n3. The contribution of this work is significant as viewed by the reviewer, in that it is among the first to apply policy gradient methods to solve RMDPs. The algorithm is presented with theoretical complexity guarantees, and the paper also discusses more efficient LP-based algorithms  in the $\\ell_1$-constrained special case, which adds to the applicability of the algorithm.\n4. The mathematical proofs seem correct to the reviewer in the form they are presented in the paper. (Note: since the proof cites quite a few theorems from e.g. [12], the reviewer took those published results for granted and didn't bother to check them.)\n\nWEAKNESSES:\n1. The numerical experiments seem relatively crude and incomplete, and the results are not convincing as viewed by the reviewer. The authors do not release the code for the simulations, and nor do they describe the settings in detail so that the results can be replicated. The presentation of results only focuses on running time but says nothing about the policy found by the algorithm, nor are they compared to any SOTA baselines. Experiments seem to be repeated with different random seeds, yet the deviation is only roughly reported in the appendix. Finally, figures cannot be found anywhere! The reviewer will be more positive if the results are more concrete, visualized, and reproducible.\n2. This paper cites a lot of results from previous work like [12]. Readers unfamiliar with it (like me) may sometimes find it very uneasy to handle these cited results which basically scatter across the whole paper. It is advisable to have an independent preliminaries section that explains what is done in [12] and why we need them here.\n\nQUESTIONS:\n1. The reviewer urges the authors to design better experiments to show both the running time and the performance of the proposed RPG algorithm against a few other baseline algorithms. Results should ideally be presented in the form of figures.\n2. In the introduction, [14] seems to be closely relevant to this paper, yet it is not discussed in details in the related works section, and nor does it appear in the table. How does this work compare with [14]? And how about the efficiency of the algorithm proposed in [14]?\n\nLIMITATIONS:\nThe paper discusses about its limitations and future work in a satisfactory way.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a policy gradient method for solving rectangular-robust MDPs. They propose closed forms for the worst-case reward and transition models. They also present robust Q-value forms that can be efficiently updated by a proper Bellman operator. Furthermore, they show an expression for the robust policy gradient that matches the non-robust PG complexity, and is therefore very fast.\n\nSTRENGTHS:\nThe paper targets a specific problem, and is very well-written. I applaud the authors for their valuable work, and enjoyed reading through the paper without any significant issues. The notation used for the theory is standard and appropriately compact, and the theory itself is clear and insightful. The computational efficiency obtained by the proposed method is both supported by theory, and also shown to be significant-enough in practice. This proves, once again, that a well-tailored algorithm can produce significant speed-ups over generic solvers (e.g., linear programming or convex optimizers).\n\nWEAKNESSES:\nWhile this is a minor point, I think the authors could slightly increase external references and elaborate more to help (an otherwise unfamiliar) reader follow through the content. For instance, (1) in Line 100, it would only take the authors a few words to reference this inequality as the Holder inequality, (2) provide a second common name for terms (such as the credal set for the probability simplex, or the visitation frequency for the occupation measure), (3) briefly derive simple results with the proposed notation (e.g., the gamma-contraction property in Line 118). These are only a few examples, and will mostly help the reader warm-up and get comfortable with the derivations before jumping into the core part of the paper.\n\nAlso, while I understand that this is out of the scope of the paper, but due to its importance I should mention it. The authors could possibly comment deeper about the resiliency of their proposed approach to stochastic approximation. While there is some limited discussion in the paper (e.g., Lines 218) about this point, I would like this point to be discussed in more length. If in future, this method is adapted to a more realistic or model-free setting, it would be valuable to have the authors comment on the statistical aspects of this approach.\n\nQUESTIONS:\nI would like the authors to address the points I raised in the Weaknesses section, even though they don't point to any fundamental short-comings in the paper.\n\nLIMITATIONS:\nThe paper portrays a fair description of its limitations. This includes, but isn't limited to, (1) the second paragraph of discussion (Lines 304-311), (2) Lines 262-267, and (3) Lines 218-222.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops policy gradients for robust MDPs, and proves several interesting properties of the worst-case kernel and occupancy measure along the way. The paper also shows that the robust Bellman operator is just the vanilla Bellman operator, with regularization of the vector's generalized variance function. The key result is a robust gradient expression that only depends on nominal samples.\n\nSTRENGTHS:\n1. Except the large amount of newly introduced notation, this paper seems polished and well written.\n2. Good discussion on time complexity, which is often absent from many prior works but crucially important for robust MDPs. \n3. The proposed method only requires samples from the nominal data distribution, and appears more computationally efficient than prior convex optimization approaches.\n\nWEAKNESSES:\n1. Notation is a bit hard to follow and is defined all over the place throughout section 3. It would be helpful to have a centralized place where all notation is defined.\n2. There is no \"convergence to local/global optima\" result.\n3. The experimental results only show running time, but does not evaluate the robust MDP performance of the learned policies. Do all methods end up learning the same robust optimal policies?\n\nQUESTIONS:\n1. Why is there a p-norm on a scalar in def. of R_{(s,a)}? \n2. In Sec 3.2.1., what if the perturbation has a negative entry, which could possibly make P_0 + P have negative entries as well? This is not so unrealistic if P_0 has small entries somewhere.\n3. Can this result generalize to convex uncertainty sets, or does it crucially rely on the uncertainty set being a lp ball? \n4. Looking at Table 1, is the main benefit of this work a better time complexity, rather than the ability to learn better policies? Convex optimization approaches that guarantee global optimum seem to only be worse in terms of time complexity.\n\nLIMITATIONS:\nPlease see questions/weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-21"}

comments: SUMMARY:
This present studied robust policy gradient to solve rectangular robust Markov decision process, where the authors provide a closed-form expression for the worst case of reward model and transition kernel. The provided explicit form of the robust policy gradient can be estimated from data with the same time complexity as its non-robust equivalent.

STRENGTHS:
- Instead of previously performing max-min optimization approaches, this paper presents a closed-form expression for the worst case kernel and reward estimation; thus we can achieve similar time complexity. 

- Empirical experiments demonstrate that the new proposed method can achieve similar complexity as non-robust settings, make the algorithm more practical than previous ones. 

- The main part of the paper is easy to follow and understand the main idea.

WEAKNESSES:
- The techniques mainly replies on the properties of the normed-ball structure of the uncertainty set, I am not sure how practical it is in realistic settings such as continuous policy optimization?

- it is still unclear for real-world scenarios, how the method or the idea can be leveraged to derive a computational efficient and practical algorithm. To me, it seems that to get the optimal reward and kernel, it is quite difficult to estimate from empirical samples (continuous case).

QUESTIONS:
- Can authors give me a concrete example or explanation of the difference of the risk-averse approach and robust RL? the uncertainty set you mentioned?

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-08-01