Title: Risk-Averse Model Uncertainty for Distributionally Robust Safe Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: deep reinforcement learning

model uncertainty

safety

risk-averse

distributionally robust
Abstract: Many real-world domains require safe decision making in uncertain environments. In this work, we introduce a deep reinforcement learning framework for approaching this important problem. We consider a distribution over transition models, and apply a risk-averse perspective towards model uncertainty through the use of coherent distortion risk measures. We provide robustness guarantees for this framework by showing it is equivalent to a specific class of distributionally robust safe reinforcement learning problems. Unlike existing approaches to robustness in deep reinforcement learning, however, our formulation does not involve minimax optimization. This leads to an efficient, model-free implementation of our approach that only requires standard data collection from a single training environment. In experiments on continuous control tasks with safety constraints, we demonstrate that our framework produces robust performance and safety at deployment time across a range of perturbed test environments.

Risk-Averse Model Uncertainty for Distributionally Robust Safe Reinforcement Learning

Abstract

Many real-world domains require safe decision making in uncertain environments. In this work, we introduce a deep reinforcement learning framework for approaching this important problem. We consider a distribution over transition models, and apply a risk-averse perspective towards model uncertainty through the use of coherent distortion risk measures. We provide robustness guarantees for this framework by showing it is equivalent to a specific class of distributionally robust safe reinforcement learning problems. Unlike existing approaches to robustness in deep reinforcement learning, however, our formulation does not involve minimax optimization. This leads to an efficient, model-free implementation of our approach that only requires standard data collection from a single training environment. In experiments on continuous control tasks with safety constraints, we demonstrate that our framework produces robust performance and safety at deployment time across a range of perturbed test environments.

1 Introduction

In many real-world decision making applications, it is important to satisfy safety requirements while achieving a desired goal. In addition, real-world environments often involve uncertain or changing conditions. Therefore, in order to reliably deploy data-driven decision making methods such as deep reinforcement learning (RL) in these settings, they must deliver robust performance and safety even in the presence of uncertainty. Recently, techniques have been developed to handle safety constraints within the deep RL framework [3, 27, 28, 41, 44, 47, 55], but these safe RL algorithms only focus on performance and safety in the training environment. They do not consider uncertainty about the true environment at deployment time due to unknown disturbances or irreducible modeling errors, which we refer to as model uncertainty . In this work, we introduce a framework that incorporates model uncertainty into safe RL. In order for our framework to be useful, we emphasize the importance of (i) an efficient deep RL implementation during training and (ii) robustness guarantees on performance and safety upon deployment.

Existing robust RL methods address the issue of model uncertainty, but they can be difficult to implement and are not always suitable in real-world decision making settings. Robust RL focuses on worst-case environments in an uncertainty set, which requires solving complex minimax optimization problems throughout training. This is typically approximated in a deep RL setting through direct interventions with a learned adversary [38, 46, 49], or through the use of parametric uncertainty with multiple simulated training environments [31, 32, 39]. However, we do not always have access to fast, high-fidelity simulators for training [10, 33, 54]. In these cases, we must be able to incorporate robustness to model uncertainty without relying on multiple training environments or potentially dangerous adversarial interventions, as real-world data collection may be necessary.

A more informative way to represent model uncertainty is to instead consider a distribution over potential environments. Domain randomization [37] collects training data from a range of environments by randomizing across parameter values in a simulator, and optimizes for average performance. This approach to model uncertainty avoids minimax formulations and works well in practice [5], but lacks robustness guarantees. In addition, domain randomization focuses on parametric uncertainty, which still requires detailed simulator access and domain knowledge to define the training distribution.

In this work, we introduce a general approach to safe RL in the presence of model uncertainty that addresses the main shortcomings of existing methods. In particular, we consider a distribution over potential environments, and apply a risk-averse perspective towards model uncertainty . Through the use of coherent distortion risk measures, this leads to a safe RL framework with robustness guarantees that does not involve difficult minimax formulations. Using this framework, we show how we can learn safe policies that are robust to model uncertainty, without the need for detailed simulator access or adversarial interventions during training. Our main contributions are as follows:

We reformulate the safe RL problem to incorporate a risk-averse perspective towards model uncertainty through the use of coherent distortion risk measures, and we introduce the corresponding Bellman operators.

From a theoretical standpoint, we provide robustness guarantees for our framework by showing it is equivalent to a specific class of distributionally robust safe RL problems.

We propose an efficient deep RL implementation that avoids the difficult minimax formulation present in robust RL and only uses data collected from a single training environment.

We demonstrate the robust performance and safety of our framework through experiments on continuous control tasks with safety constraints in the Real-World RL Suite [18, 19].

2 Preliminaries

Safe reinforcement learning In this work, we consider RL in the presence of safety constraints. We model this sequential decision making problem as an infinite-horizon, discounted Constrained Markov Decision Process (CMDP) [4] defined by the tuple ( S , A , p, r, c, d 0 , γ ) , where S is the set of states, A is the set of actions, p : S × A → P ( S ) is the transition model where P ( S ) represents the space of probability measures over S , r, c : S × A → R are the reward function and cost function used to define the objective and constraint, respectively, d 0 ∈ P ( S ) is the initial state distribution, and γ is the discount rate. We focus on the setting with a single constraint, but all results can be extended to the case of multiple constraints.

We model the agent's decisions as a stationary policy π : S → P ( A ) . For a given CMDP and policy π , we write the expected total discounted rewards and costs as J p,r ( π ) = E τ ∼ ( π,p ) [ ∑ ∞ t =0 γ t r ( s t , a t )] and J p,c ( π ) = E τ ∼ ( π,p ) [ ∑ ∞ t =0 γ t c ( s t , a t )] , respectively, where τ ∼ ( π, p ) represents a trajectory sampled according to s 0 ∼ d 0 , a t ∼ π ( · | s t ) , and s t +1 ∼ p ( · | s t , a t ) . The goal of safe RL is to find a policy π that maximizes the constrained optimization problem

where B is a safety budget on expected total discounted costs.

We write the corresponding state-action value functions (i.e., Q functions) for a given transition model p and policy π as Q π p,r ( s, a ) and Q π p,c ( s, a ) , respectively. Off-policy optimization techniques [28, 55] find a policy that maximizes (1) by solving at each iteration the related optimization problem

where π k is the current policy and D is a replay buffer containing data collected in the training environment. Note that Q π p,r ( s, a ) and Q π p,c ( s, a ) are the respective fixed points of the Bellman operators

Model uncertainty in reinforcement learning Rather than focusing on a single CMDP with transition model p , we incorporate uncertainty about the transition model by considering a distribution µ over models. We focus on distributions of the form µ = ∏ ( s,a ) ∈S×A µ s,a , where µ s,a represents a distribution over transition models p s,a = p ( · | s, a ) ∈ P ( S ) at a given state-action pair and µ is the product over all µ s,a . This is known as rectangularity, and is a common assumption in the literature [11, 15, 16, 53, 56]. Note that µ s,a ∈ P ( M ) , where we write M = P ( S ) to denote model space. Compared to robust RL methods that apply uncertainty sets over transition models, the use of a distribution µ over transition models is a more informative way to represent model uncertainty that does not require solving for worst-case environments (i.e., does not introduce a minimax formulation ).

In order to incorporate robustness to the choice of µ , distributionally robust MDPs [53, 56] consider an ambiguity set U = ⊗ ( s,a ) ∈S×A U s,a of distributions over transition models, where µ s,a ∈ U s,a ⊆ P ( M ) . The goal of distributionally robust RL is to optimize the worst-case average performance across all distributions contained in U . In this work, we will show that a risk-averse perspective towards model uncertainty defined by µ is equivalent to distributionally robust RL for appropriate choices of ambiguity sets in the objective and constraint of a CMDP. However, our use of risk measures avoids the need to solve for worst-case distributions in U throughout training .

Risk measures Consider the probability space ( M , F , µ s,a ) , where F is a σ -algebra on M and µ s,a ∈ P ( M ) defines a probability measure over M . Let Z be a space of random variables defined on this probability space, and let Z ∗ be its corresponding dual space. A real-valued risk measure ρ : Z → R summarizes a random variable as a value on the real line. In this section, we consider cost random variables Z ∈ Z where a lower value of ρ ( Z ) is better. We can define a corresponding risk measure ρ + for reward random variables through an appropriate change in sign, where ρ + ( Z ) = -ρ ( -Z ) . Risk-sensitive methods typically focus on classes of risk measures with desirable properties [30], such as coherent risk measures [6] and distortion risk measures [17, 50].

Definition 1 (Coherent risk measure) . A risk measure ρ is a coherent risk measure if it satisfies monotonicity, translation invariance, positive homogeneity, and convexity.

Definition 2 (Distortion risk measure) . Let g : [0 , 1] → [0 , 1] be a non-decreasing, left-continuous function with g (0) = 0 and g (1) = 1 . A distortion risk measure with respect to g is defined as

where F -1 Z is the inverse cumulative distribution function of Z and ˜ g ( u ) = 1 -g (1 -u ) .

A distortion risk measure is coherent if and only if g is concave [52]. In this work, we focus on the class of coherent distortion risk measures . We will leverage properties of coherent risk measures to provide robustness guarantees for our framework, and we will leverage properties of distortion risk measures to propose an efficient, model-free implementation that does not involve minimax optimization. See the Appendix for additional details on the properties of coherent distortion risk measures. Many commonly used risk measures belong to this class, including expectation, conditional value-at-risk (CVaR), and the Wang transform [51] for η ≥ 0 which is defined by the distortion function g η ( u ) = Φ(Φ -1 ( u ) + η ) , where Φ is the standard Normal cumulative distribution function.

3 Related work

Safe reinforcement learning The CMDP framework is the most popular approach to safety in RL, and several deep RL algorithms have been developed to solve the constrained optimization problem in (1). These include primal-dual methods that consider the Lagrangian relaxation of (1) [41, 44, 47], algorithms that compute closed-form solutions to related or approximate versions of (1) [3, 28], and direct methods for constraint satisfaction such as the use of barriers [27] or immediate switching between the objective and constraint [55]. All of these approaches are designed to satisfy expected cost constraints for a single CMDP observed during training. In our work, on the other hand, we consider a distribution over possible transition models.

Uncertainty in reinforcement learning Our work focuses on irreducible uncertainty about the true environment at deployment time, which we refer to as model uncertainty and represent using a

distribution µ over transition models. The most popular approach that incorporates model uncertainty in this way is domain randomization [37, 48], which randomizes across parameter values in a simulator and trains a policy to maximize average performance over this training distribution. This represents a risk-neutral attitude towards model uncertainty, which has been referred to as a softrobust approach [16]. Distributionally robust MDPs incorporate robustness to the choice of µ by instead considering a set of distributions [11, 15, 53, 56], but application of this distributionally robust framework has remained limited in deep RL as it leads to a difficult minimax formulation that requires solving for worst-case distributions over transition models.

Robust RL represents an alternative approach to model uncertainty that considers uncertainty sets of transition models [21, 34]. A major drawback of robust RL is the need to calculate worstcase environments during training, which is typically approximated through the use of parametric uncertainty with multiple training environments [31, 32, 39] or a trained adversary that directly intervenes during trajectory rollouts [38, 46, 49]. Unlike these methods, we propose a robust approach to model uncertainty based on a distribution µ over models, which does not require access to a range of simulated training environments, does not impact data collection during training, and does not involve minimax optimization problems.

In contrast to irreducible model uncertainty, epistemic uncertainty captures estimation error that can be reduced during training through data collection. Epistemic uncertainty has been considered in the estimation of Q functions [9, 35, 36] and learned transition models [7, 13, 22, 25, 40], and has been applied to promote both exploration and safety in a fixed MDP. Finally, risk-sensitive methods typically focus on the aleatoric uncertainty in RL, which refers to the range of stochastic outcomes within a single MDP. Rather than considering the standard expected value objective, they learn risk-sensitive policies over this distribution of possible outcomes in a fixed MDP [12, 24, 26, 43, 45]. Distributional RL [8] trains critics that estimate the full distribution of future returns due to aleatoric uncertainty, and risk measures can be applied to these distributional critics for risk-sensitive learning [14, 29]. We also consider the use of risk measures in our work, but different from standard risksensitive RL methods we apply a risk measure over model uncertainty instead of aleatoric uncertainty.

4 Risk-averse model uncertainty for safe reinforcement learning

The standard safe RL problem in (1) focuses on performance and safety in a single environment with fixed transition model p . In this work, however, we are interested in a distribution of possible transition models p ∼ µ rather than a fixed transition model. The distribution µ provides a natural way to capture our uncertainty about the unknown transition model at deployment time. Next, we must incorporate this model uncertainty into our problem formulation. Prior methods have done this by applying the expectation operator over µ s,a at every transition [16]. Instead, we adopt a risk-averse view towards model uncertainty in order to learn policies with robust performance and safety. We accomplish this by applying a coherent distortion risk measure ρ with respect to model uncertainty at every transition.

We consider the risk-averse model uncertainty (RAMU) safe RL problem

where we use ρ + and ρ to account for reward and cost random variables, respectively, and we apply these coherent distortion risk measures over p s,a ∼ µ s,a at every transition to define

The notation ρ + p s,a ∼ µ s,a ( · ) and ρ p s,a ∼ µ s,a ( · ) emphasize that the stochasticity of the random variables are with respect to the transition models sampled from µ s,a . Note that we still apply expectations over the aleatoric uncertainty of the CMDP (i.e., the randomness associated with a stochastic transition model and stochastic policy), while being risk-averse with respect to model uncertainty. Because we are interested in learning policies that achieve robust performance and robust safety at deployment

time, we apply this risk-averse perspective to model uncertainty in both the objective and constraint of (3).

We write the corresponding RAMU reward and cost Q functions as Q π ρ + ,r ( s, a ) and Q π ρ,c ( s, a ) , respectively. Similar to the standard safe RL setting, we can apply off-policy techniques to solve the RAMU safe RL problem in (3) by iteratively optimizing

Therefore, we have replaced the standard Q functions for a fixed transition model p in (2) with our RAMU Q functions in (4).

We can write the RAMU Q functions recursively as

These recursive definitions motivate corresponding RAMU Bellman operators.

Definition 3 (RAMU Bellman operators) . For a given policy π , the RAMU Bellman operators are defined as

Note that the RAMU Bellman operators can also be interpreted as applying a coherent distortion risk measure over standard Bellman targets, which are random variables with respect to the transition model p s,a ∼ µ s,a for a given state-action pair.

Lemma 1. The RAMU Bellman operators can be written in terms of standard Bellman operators as

Proof. The results follow from the definitions of T π p,r and T π p,c , along with the translation invariance and positive homogeneity of coherent distortion risk measures. See the Appendix for details.

In the next section, we show that T π ρ + ,r and T π ρ,c are contraction operators, so we can apply standard temporal difference learning techniques to learn the RAMU Q functions Q π ρ + ,r ( s, a ) and Q π ρ,c ( s, a ) that are needed for our RAMU policy update in (4).

5 Robustness guarantees

Intuitively, our risk-averse perspective places more emphasis on potential transition models that result in higher costs or lower rewards under the current policy, which should result in learning safe policies that are robust to model uncertainty. Next, we formalize the robustness guarantees of our RAMU framework by showing it is equivalent to a distributionally robust safe RL problem for appropriate choices of ambiguity sets.

Theorem 1. The RAMU safe RL problem in (3) is equivalent to the distributionally robust safe RL problem

with ambiguity sets U + = ⊗ ( s,a ) ∈S×A U + s,a and U = ⊗ ( s,a ) ∈S×A U s,a , where

are sets of feasible reweightings of µ s,a with ξ s,a that depend on the choice of ρ + and ρ , respectively.

Proof. Using duality results for coherent risk measures [42], we show that the RAMU Bellman operators T π ρ + ,r and T π ρ,c are equivalent to distributionally robust Bellman operators [53, 56] with ambiguity sets of distributions U + and U , respectively. The RAMU Q functions are the respective fixed points of these Bellman operators, so they can be written as distributionally robust Q functions. Finally, by averaging over initial states and actions, we see that (3) is equivalent to (6). See the Appendix for details.

Theorem 1 shows that the application of ρ + and ρ at every timestep are equivalent to solving distributionally robust optimization problems over the ambiguity sets of distributions U + and U , respectively. This can be interpreted as adversarially reweighting µ s,a with ξ s,a at every state-action pair. Note that worst-case distributions appear in both the objective and constraint of (6), so any policy trained with our RAMU framework is guaranteed to deliver robust performance and robust safety. The level of robustness depends on the choice of ρ + and ρ , which determine the structure and size of the corresponding ambiguity sets based on their dual representations [42].

In addition, because (3) is equivalent to a distributionally robust safe RL problem according to Theorem 1, we can leverage existing results for distributionally robust MDPs [53, 56] to show that T π ρ + ,r and T π ρ,c are contraction operators.

Corollary 1. The RAMU Bellman operators T π ρ + ,r and T π ρ,c are γ -contractions in the sup-norm.

Proof. Apply results from Xu and Mannor [53] and Yu and Xu [56]. See the Appendix for details.

Therefore, we have that Q π ρ + ,r ( s, a ) and Q π ρ,c ( s, a ) can be interpreted as distributionally robust Q functions by Theorem 1, and we can apply standard temporal difference methods to learn these RAMU Q functions as a result of Corollary 1. Importantly, Theorem 1 demonstrates the robustness properties of our RAMU framework, but it is not used to implement our approach . Directly implementing (6) would require solving for adversarial distributions over transition models throughout training. Instead, our framework provides the same robustness, but the use of risk measures leads to an efficient deep RL implementation as we describe in the following section.

6 Model-free implementation with a single training environment

The RAMU policy update in (4) takes the same form as the standard safe RL update in (2), except for the use of Q π ρ + ,r ( s, a ) and Q π ρ,c ( s, a ) . Because our RAMU Bellman operators are contractions, we can learn these RAMU Q functions by applying standard temporal difference loss functions that are used throughout deep RL. In particular, we consider parameterized critics Q θ r and Q θ c , and we optimize their parameters during training to minimize the loss functions

where ˆ T π ρ + ,r and ˆ T π ρ,c represent sample-based estimates of the RAMU Bellman operators applied to target Q functions denoted by ¯ Q . Therefore, we must be able to efficiently estimate the RAMU Bellman targets, which involve calculating coherent distortion risk measures that depend on the distribution µ s,a .

Sample-based estimation of risk measures Using the formulation of our RAMU Bellman operators from Lemma 1, we can leverage properties of distortion risk measures to efficiently estimate the results in (5) using sample-based weighted averages of standard Bellman targets. For n transition models p ( i ) s,a , i = 1 , . . . , n , sampled independently from µ s,a and sorted according to their corresponding Bellman targets, consider the weights

0.0

0.2

0.4

0.6

0.8

1.0

u

0.00

0.25

0.50

0.75

1.00

g

(

u

)

Expectation

Wang (

η

=0.75

)

w

(1)

ρ

w

(2)

ρ

w

(3)

ρ

Expectation

Wang (

η

=0.75

w

(4)

ρ

w

(5)

ρ

where g defines the distortion risk measure ρ according to Definition 2. See Figure 1 for the distortion functions and weights associated with the risk measures used in our experiments. Then, from Jones and Zitikis [23] we have that

are consistent estimators of the results in (5), where T π p ( i ) ,r Q ( s, a ) are sorted in ascending order and T π p ( i ) ,c Q ( s, a ) are sorted in descending order. Finally, we can replace T π p ( i ) ,r Q ( s, a ) and T π p ( i ) ,c Q ( s, a ) with the standard unbiased sample-based estimates

where s ′ ∼ p ( i ) s,a and a ′ ∼ π ( · | s ′ ) . This leads to the sample-based estimates

which we use to train our RAMU Q functions. Note that the estimates in (7) can be computed very efficiently, which is a major benefit of our RAMU framework compared to robust RL methods. Next, we describe how we can sample models p ( i ) s,a , i = 1 , . . . , n , from µ s,a , and generate state transitions from these models to use in the calculation of our sample-based Bellman targets in (7).

Generative distribution of transition models Note that our RAMU framework can be applied using any choice of distribution µ , provided we can sample transition models p ( i ) s,a ∼ µ s,a and corresponding next states s ′ ∼ p ( i ) s,a . In this work, we define the distribution µ over perturbed versions of a single training environment p train , and we propose a generative approach to sampling transition models and corresponding next states that only requires data collected from p train . By doing so, our RAMU framework achieves robust performance and safety with minimal assumptions on the training process, and can even be applied to settings that require real-world data collection for training.

We consider a latent variable x ∼ X , and we define a transition model p s,a ( x ) for every x ∼ X that shifts the probability of s ′ under p train s,a according to a perturbation function f x : S × S → S . This perturbation function takes as input a state transition ( s, s ′ ) , and outputs a perturbed next state ˜ s ′ that depends on the latent variable x ∼ X . Therefore, a distribution over latent space implicitly defines a distribution µ s,a over perturbed versions of p train s,a . In order to obtain the next state samples needed to compute the Bellman target estimates in (7), we sample latent variables x ∼ X and apply f x to the state transition observed in the training environment. We have that s ′ ∼ p train s,a for data collected in the training environment, so ˜ s ′ = f x ( s, s ′ ) represents the corresponding sample from the perturbed transition model p s,a ( x ) .

In our experiments, we consider a simple implementation for the common case where S = R d . We use uniformly distributed latent variables x ∼ U ([ -2 ϵ, 2 ϵ ] d ) , and we define the perturbation function as

Weight

0.5

0.4

0.3

0.2

0.1

0.0

)

where all operations are performed per-coordinate. Therefore, the latent variable x ∼ U ([ -2 ϵ, 2 ϵ ] d ) can be interpreted as the percentage change in each dimension of a state transition observed in the training environment, where the average magnitude of the percentage change is ϵ . The hyperparameter ϵ determines the distribution µ s,a over transition models, where a larger value of ϵ leads to transition models that vary more significantly from the training environment. The structure of f x provides an intuitive, scale-invariant meaning for the hyperparameter ϵ , which makes it easy to tune in practice. This choice of distribution µ s,a captures general uncertainty in the training environment, without requiring specific domain knowledge of potential disturbances.

Algorithm We summarize the implementation of our RAMU framework in Algorithm 1. Given data collected in a single training environment, we can efficiently calculate the sample-based RAMU Bellman targets in (7) by (i) sampling from a latent variable x ∼ X , (ii) computing the corresponding next state samples f x ( s, s ′ ) , and (iii) sorting the standard Bellman estimates that correspond to these sampled transition models. Given the sample-based RAMU Bellman targets, updates of the critics and policy have the same form as in standard deep safe RL algorithms. Therefore, our RAMU framework can be easily combined with many popular safe RL algorithms to incorporate model uncertainty with robustness guarantees , using only a minor change to the estimation of Bellman targets that is efficient to implement in practice.

7 Experiments

In order to evaluate the performance and safety of our RAMU framework, we conduct experiments on 5 continuous control tasks with safety constraints from the Real-World RL Suite [18, 19]: Cartpole Swingup, Walker Walk, Walker Run, Quadruped Walk, and Quadruped Run. Each task has a horizon length of 1 , 000 with r ( s, a ) ∈ [0 , 1] and c ( s, a ) ∈ { 0 , 1 } , and we consider a safety budget of B = 100 . Unless noted otherwise, we train these tasks on a single training environment for 1 million steps across 5 random seeds, and we evaluate performance of the learned policies across a range of perturbed test environments via 10 trajectory rollouts. See the Appendix for information on the safety constraints and environment perturbations that we consider.

Our RAMU framework can be combined with several choices of safe RL algorithms. We consider the safe RL algorithm Constraint-Rectified Policy Optimization (CRPO) [55], and we use Maximum a Posteriori Policy Optimization (MPO) [1] as the unconstrained policy optimization algorithm in CRPO. For a fair comparison, we apply this choice of safe RL policy update in every method we consider in our experiments. We use a multivariate Gaussian policy with learned mean and diagonal covariance at each state, along with separate reward and cost critics. We parameterize our policy and critics using neural networks. See the Appendix for implementation details. 1

We summarize the performance and safety of our RAMU framework in Table 1 and Figure 2, compared to several baseline algorithms that we discuss next. We include detailed experimental results across all perturbed test environments in the Appendix. We apply our RAMU framework using the Wang transform with η = 0 . 75 as the risk measure in both the objective and constraint. In order

† Percentage of policies that satisfy the safety constraint across all tasks and test environments.

‡ Normalized relative to the average performance of standard safe RL for each task and test environment.

∗ Denotes need for adversary or simulator during data collection (i.e., trajectory rollouts) for training.

750

800

850

Total Reward

819

830

796

849

802

820

750

800

850

900

789

876

805

898

783

890

Walker Walk

350

400

450

405

424

411

454

425

377

850

900

950

1000

902

988

974

999

947

999

750

800

850

851

824

828

833

835

795

40

60

80

100

% Safe

69%

79%

72%

83%

64%

70%

40

60

80

100

53%

76%

60%

74%

59%

72%

40

60

80

100

56%

76% 79% 76%

59%

70%

RAMU (Wang 0.75)

RAMU (Expectation)

Domain Rand.

Domain Rand. (OOD)

40

60

80

100

42%

71% 70% 71%

51%

100%

40

60

80

100

34%

98%

88%

76%

43%

100%

Figure 2: Performance summary by task, aggregated across perturbed test environments. Performance of adversarial RL is evaluated without adversarial interventions. Top: Total rewards averaged across test environments. Bottom: Percentage of policies across test environments that satisfy the safety constraint.

to understand the impact of being risk-averse to model uncertainty, we also consider the risk-neutral special case of our framework where expectations are applied to the objective and constraint. For our RAMU results in Table 1 and Figure 2, we specify the risk measure in parentheses. Finally, we consider n = 5 samples of transition models with latent variable hyperparameter ϵ = 0 . 10 in order to calculate Bellman targets in our RAMU framework.

Comparison to safe reinforcement learning First, we analyze the impact of our RAMU framework compared to standard safe RL. In both cases, we train policies using data collected from a single training environment, so the only difference comes from our use of risk-averse model uncertainty to learn RAMU Q functions. By evaluating the learned policies in perturbed test environments different from the training environment, we see that our RAMU framework provides robustness in terms of both total rewards and safety. In particular, the risk-averse implementation of our algorithm leads to safety constraint satisfaction in 80% of test environments, compared to only 51% with standard safe RL. In addition, this implementation results in higher total rewards (1.08x) and lower total costs (0.51x), on average. We see in Table 1 that the use of expectations over model uncertainty (i.e., a risk-neutral approach) also improves robustness in both the objective and constraint, on average, compared to standard safe RL. However, we further improve upon the benefits observed in the risk-neutral case by instead applying a risk-averse perspective.

Comparison to domain randomization Next, we compare our RAMU framework to domain randomization, a popular approach that also represents model uncertainty using a distribution µ over models. Note that domain randomization considers parametric uncertainty and has the benefit of training on a range of simulated environments, while our method only collects data from a single training environment. In order to evaluate the importance of domain knowledge for defining the training distribution in domain randomization, we consider two different cases: an in-distribution

version that trains on a subset of the perturbed test environments, and an out-of-distribution (OOD) version that randomizes over a different perturbation parameter than the one varied at test time.

The results in Table 1 and Figure 2 show the importance of domain knowledge: in-distribution domain randomization leads to improved robustness compared to standard safe RL and the highest normalized average rewards (1.14x), while the out-of-distribution version provides little benefit. In both cases, however, domain randomization achieves lower levels of safety, on average, than our risk-averse formulation. In fact, we see in Figure 2 that the safety constraint satisfaction of our risk-averse formulation is at least as strong as both versions of domain randomization in 4 out of 5 tasks, despite only training on a single environment with no specific knowledge about the disturbances at test time . This demonstrates the key benefit of our risk-averse approach to model uncertainty.

Comparison to adversarial reinforcement learning Finally, we compare our approach to adversarial RL using the action-robust PR-MDP framework [46], which randomly applies worst-case actions a percentage of the time during data collection. Although adversarial RL only collects data from a single training environment, it requires potentially dangerous adversarial interventions during training in order to provide robustness at test time. In order to apply this method to the safe RL setting, we train an adversary to maximize costs and consider a 5% probability of intervention during training. The performance of adversarial RL is typically evaluated without adversarial interventions, which requires a clear distinction between training and testing.

We see in Figure 2 that adversarial RL learns policies that achieve robust safety constraint satisfaction at test time in the Quadruped tasks. Our risk-averse formulation, on the other hand, achieves higher levels of safety in the remaining 3 out of 5 tasks, and similar levels of safety on average. Unlike adversarial RL, our RAMU framework achieves robust safety in a way that (i) does not alter the data collection process, (ii) does not require training an adversary in a minimax formulation, and (iii) does not require different implementations during training and testing. In addition, our use of a distribution over models represents a less conservative approach than adversarial RL, resulting in higher normalized average rewards as shown in Table 1.

8 Conclusion

We have presented a framework for safe RL in the presence of model uncertainty, an important setting for many real-world decision making applications. Compared to existing approaches to model uncertainty in deep RL, our formulation applies a risk-averse perspective through the use of coherent distortion risk measures. We show that this results in robustness guarantees, while still leading to an efficient deep RL implementation that does not involve minimax optimization problems. Importantly, our method only requires data collected from a single training environment, so it can be applied to real-world domains where high-fidelity simulators are not readily available or are computationally expensive. Therefore, our framework represents an attractive approach to safe decision making under model uncertainty that can be deployed across a range of applications.

Prior to potential deployment, it is important to understand the limitations of our proposed methodology. The robustness and safety of our RAMU framework depend on the user-defined choices of model distribution µ and risk measure ρ . The distribution µ defines the uncertainty over transition models, and the risk measure ρ defines the level of robustness to this choice of µ . In addition, our approach only considers robustness with respect to model uncertainty and safety as defined by expected total cost constraints. It would be interesting to extend our techniques to address other forms of uncertainty and other definitions of safety, including epistemic uncertainty in model-based RL, observational uncertainty, and safety-critical formulations based on sets of unsafe states.

References

A. Abdolmaleki, J. T. Springenberg, Y. Tassa, R. Munos, N. Heess, and M. Riedmiller. Maximum a posteriori policy optimisation. In Sixth International Conference on Learning Representations , 2018.

A. Abdolmaleki, S. Huang, L. Hasenclever, M. Neunert, F. Song, M. Zambelli, M. Martins, N. Heess, R. Hadsell, and M. Riedmiller. A distributional view on multi-objective policy

optimization. In Proceedings of the 37th International Conference on Machine Learning , volume 119, pages 11-22. PMLR, 2020.

J. Achiam, D. Held, A. Tamar, and P. Abbeel. Constrained policy optimization. In Proceedings of the 34th International Conference on Machine Learning , volume 70, pages 22-31. PMLR, 2017.

E. Altman. Constrained Markov Decision Processes . CRC Press, 1999.

M. Andrychowicz, B. Baker, M. Chociej, R. Józefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba. Learning dexterous in-hand manipulation. The International Journal of Robotics Research , 39(1):3-20, 2020. doi: 10.1177/0278364919887447.

P. Artzner, F. Delbaen, J.-M. Eber, and D. Heath. Coherent measures of risk. Mathematical Finance , 9(3):203-228, 1999. doi: 10.1111/1467-9965.00068.

Y. As, I. Usmanova, S. Curi, and A. Krause. Constrained policy optimization via Bayesian world models. In Tenth International Conference on Learning Representations , 2022.

M. G. Bellemare, W. Dabney, and R. Munos. A distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning , volume 70, pages 449-458. PMLR, 2017.

H. Bharadhwaj, A. Kumar, N. Rhinehart, S. Levine, F. Shkurti, and A. Garg. Conservative safety critics for exploration. In Ninth International Conference on Learning Representations , 2021.

W. Cao, M. Benosman, and R. Ma. Domain knowledge-based automated analog circuit design with deep reinforcement learning. In The 59th ACM/IEEE Design Automation Conference , 2022.

R. Chen and I. C. Paschalidis. Distributionally robust learning. Foundations and Trends® in Optimization , 4(1-2):1-243, 2020. doi: 10.1561/2400000026.

Y. Chow, A. Tamar, S. Mannor, and M. Pavone. Risk-sensitive and robust decision-making: a CVaR optimization approach. In Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015.

K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

W. Dabney, G. Ostrovski, D. Silver, and R. Munos. Implicit quantile networks for distributional reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning , volume 80, pages 1096-1105. PMLR, 2018.

E. Derman and S. Mannor. Distributional robustness and regularization in reinforcement learning. arXiv preprint, 2020. arXiv:2003.02894.

E. Derman, D. J. Mankowitz, T. A. Mann, and S. Mannor. Soft-robust actor-critic policy-gradient. arXiv preprint, 2018. arXiv:1803.04848.

J. Dhaene, A. Kukush, D. Linders, and Q. Tang. Remarks on quantiles and distortion risk measures. European Actuarial Journal , 2:319-328, 2012. doi: 10.1007/s13385-012-0058-0.

G. Dulac-Arnold, N. Levine, D. J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester. An empirical investigation of the challenges of real-world reinforcement learning. arXiv preprint, 2020. arXiv:2003.11881.

G. Dulac-Arnold, N. Levine, D. J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning , 110:2419-2468, 2021. doi: 10.1007/s10994-021-05961-4.

M. W. Hoffman, B. Shahriari, J. Aslanides, G. Barth-Maron, N. Momchev, D. Sinopalnikov, P. Sta´ nczyk, S. Ramos, A. Raichuk, D. Vincent, L. Hussenot, R. Dadashi, G. Dulac-Arnold, M. Orsini, A. Jacq, J. Ferret, N. Vieillard, S. K. S. Ghasemipour, S. Girgin, O. Pietquin, F. Behbahani, T. Norman, A. Abdolmaleki, A. Cassirer, F. Yang, K. Baumli, S. Henderson, A. Friesen, R. Haroun, A. Novikov, S. G. Colmenarejo, S. Cabi, C. Gulcehre, T. L. Paine, S. Srinivasan, A. Cowie, Z. Wang, B. Piot, and N. de Freitas. Acme: A research framework for distributed reinforcement learning. arXiv preprint, 2020. arXiv:2006.00979.

G. N. Iyengar. Robust dynamic programming. Mathematics of Operations Research , 30(2): 257-280, 2005. doi: 10.1287/moor.1040.0129.

M. Janner, J. Fu, M. Zhang, and S. Levine. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

B. L. Jones and R. Zitikis. Empirical estimation of risk measures and related quantities. North American Actuarial Journal , 7(4):44-54, 2003. doi: 10.1080/10920277.2003.10596117.

R. Keramati, C. Dann, A. Tamkin, and E. Brunskill. Being optimistic to be conservative: Quickly learning a CVaR policy. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 4436-4443, 2020.

T. Kurutach, I. Clavera, Y. Duan, A. Tamar, and P. Abbeel. Model-ensemble trust-region policy optimization. In Sixth International Conference on Learning Representations , 2018.

P. L.A. and M. C. Fu. Risk-sensitive reinforcement learning via policy gradient search. Foundations and Trends® in Machine Learning , 15(5):537-693, 2022. ISSN 1935-8237. doi: 10.1561/2200000091.

Y. Liu, J. Ding, and X. Liu. IPO: Interior-point policy optimization under constraints. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 4940-4947, 2020.

Z. Liu, Z. Cen, V. Isenbaev, W. Liu, S. Wu, B. Li, and D. Zhao. Constrained variational policy optimization for safe reinforcement learning. In Proceedings of the 39th International Conference on Machine Learning , pages 13644-13668. PMLR, 2022.

X. Ma, L. Xia, Z. Zhou, J. Yang, and Q. Zhao. DSAC: Distributional soft actor critic for risk-sensitive reinforcement learning. arXiv preprint, 2020. arXiv:2004.14547.

A. Majumdar and M. Pavone. How should a robot assess risk? Towards an axiomatic theory of risk in robotics. In Robotics Research , pages 75-84. Springer International Publishing, 2020. ISBN 978-3-030-28619-4.

D. J. Mankowitz, N. Levine, R. Jeong, A. Abdolmaleki, J. T. Springenberg, Y. Shi, J. Kay, T. Hester, T. Mann, and M. Riedmiller. Robust reinforcement learning for continuous control with model misspecification. In Eighth International Conference on Learning Representations , 2020.

D. J. Mankowitz, D. A. Calian, R. Jeong, C. Paduraru, N. Heess, S. Dathathri, M. Riedmiller, and T. Mann. Robust constrained reinforcement learning for continuous control with model misspecification. arXiv preprint, 2021. arXiv:2010.10644.

S. Mowlavi, M. Benosman, and S. Nabi. Reinforcement learning state estimation for highdimensional nonlinear systems. In Tenth International Conference on Learning Representations , 2022.

A. Nilim and L. E. Ghaoui. Robust control of Markov decision processes with uncertain transition matrices. Operations Research , 53(5):780-798, 2005. doi: 10.1287/opre.1050.0216.

I. Osband, C. Blundell, A. Pritzel, and B. Van Roy. Deep exploration via bootstrapped DQN. In Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016.

I. Osband, J. Aslanides, and A. Cassirer. Randomized prior functions for deep reinforcement learning. In Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE International Conference on Robotics and Automation (ICRA) , pages 3803-3810, 2018. doi: 10.1109/ICRA.2018.8460528.

L. Pinto, J. Davidson, R. Sukthankar, and A. Gupta. Robust adversarial reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning , volume 70, pages 2817-2826. PMLR, 2017.

A. Rajeswaran, S. Ghotra, B. Ravindran, and S. Levine. EPOpt: Learning robust neural network policies using model ensembles. In 5th International Conference on Learning Representations , 2017.

A. Rajeswaran, I. Mordatch, and V. Kumar. A game theoretic framework for model based reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning , volume 119, pages 7953-7963. PMLR, 2020.

A. Ray, J. Achiam, and D. Amodei. Benchmarking safe exploration in deep reinforcement learning, 2019.

A. Shapiro, D. Dentcheva, and A. Ruszczy´ nski. Lectures on Stochastic Programming: Modeling and Theory, Second Edition . Society for Industrial and Applied Mathematics, 2014. ISBN 1611973422.

Y. Shen, M. J. Tobia, T. Sommer, and K. Obermayer. Risk-sensitive reinforcement learning. Neural Computation , 26(7):1298-1328, 2014. doi: 10.1162/NECO_a_00600.

A. Stooke, J. Achiam, and P. Abbeel. Responsive safety in reinforcement learning by PID Lagrangian methods. In Proceedings of the 37th International Conference on Machine Learning , volume 119, pages 9133-9143. PMLR, 2020.

A. Tamar, Y. Chow, M. Ghavamzadeh, and S. Mannor. Policy gradient for coherent risk measures. In Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015.

C. Tessler, Y. Efroni, and S. Mannor. Action robust reinforcement learning and applications in continuous control. In Proceedings of the 36th International Conference on Machine Learning , volume 97, pages 6215-6224. PMLR, 2019.

C. Tessler, D. J. Mankowitz, and S. Mannor. Reward constrained policy optimization. In Seventh International Conference on Learning Representations , 2019.

J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 23-30, 2017. doi: 10.1109/IROS.2017.8202133.

E. Vinitsky, Y. Du, K. Parvate, K. Jang, P. Abbeel, and A. Bayen. Robust reinforcement learning using adversarial populations. arXiv preprint, 2020. arXiv:2008.01825.

S. Wang. Premium calculation by transforming the layer premium density. ASTIN Bulletin , 26 (1):71-92, 1996. doi: 10.2143/AST.26.1.563234.

S. S. Wang. A class of distortion operators for pricing financial and insurance risks. The Journal of Risk and Insurance , 67(1):15-36, 2000. ISSN 00224367, 15396975. doi: 10.2307/253675.

J. L. Wirch and M. R. Hardy. Distortion risk measures: Coherence and stochastic dominance. Insurance Mathematics and Economics , 32:168-168, 2003.

H. Xu and S. Mannor. Distributionally robust Markov decision processes. In Advances in Neural Information Processing Systems , volume 23. Curran Associates, Inc., 2010.

J. Xu, S. Kim, T. Chen, A. Rodriguez, P. Agrawal, W. Matusik, and S. Sueda. Efficient tactile simulation with differentiability for robotic manipulation. In The Conference on Robot Learning (CoRL) , 2022.

T. Xu, Y. Liang, and G. Lan. CRPO: A new approach for safe reinforcement learning with convergence guarantee. In Proceedings of the 38th International Conference on Machine Learning , pages 11480-11491. PMLR, 2021.

P. Yu and H. Xu. Distributionally robust counterpart in Markov decision processes. IEEE Transactions on Automatic Control , 61(9):2538-2543, 2016. doi: 10.1109/TAC.2015.2495174.

A Properties of coherent distortion risk measures

Majumdar and Pavone [30] proposed a set of six axioms to characterize desirable properties of risk measures in the context of robotics.

A1. Monotonicity: If Z, Z ′ ∈ Z and Z ≤ Z ′ almost everywhere, then ρ ( Z ) ≤ ρ ( Z ′ ) .

A2. Translation invariance: If α ∈ R and Z ∈ Z , then ρ ( Z + α ) = ρ ( Z ) + α .

A3. Positive homogeneity: If τ ≥ 0 and Z ∈ Z , then ρ ( τZ ) = τρ ( Z ) .

A4. Convexity: If λ ∈ [0 , 1] and Z, Z ′ ∈ Z , then ρ ( λZ +(1 -λ ) Z ′ ) ≤ λρ ( Z ) + (1 -λ ) ρ ( Z ′ ) .

A5. Comonotonic additivity: If Z, Z ′ ∈ Z are comonotonic, then ρ ( Z + Z ′ ) = ρ ( Z ) + ρ ( Z ′ ) .

A6. Law invariance: If Z, Z ′ ∈ Z are identically distributed, then ρ ( Z ) = ρ ( Z ′ ) .

See Majumdar and Pavone [30] for a discussion on the intuition behind these axioms. Note that coherent risk measures [6] satisfy Axioms A1-A4, distortion risk measures [17, 50] satisfy Axioms A1-A3 and Axioms A5-A6, and coherent distortion risk measures satisfy all six axioms.

The properties of coherent risk measures also lead to a useful dual representation.

Lemma 2 (Shapiro et al. [42]) . Let ρ be a proper, real-valued coherent risk measure. Then, for any Z ∈ Z we have that

where E β s,a [ · ] represents expectation with respect to the probability measure β s,a ∈ P ( M ) , and

is a convex, bounded, and weakly* closed set that depends on ρ .

See Shapiro et al. [42] for a general treatment of this result.

B Proofs

In this section, we prove all results related to the RAMU cost Bellman operator T π ρ,c . Using the fact that ρ + ( Z ) = -ρ ( -Z ) for a coherent distortion risk measure ρ on a cost random variable, all results related to the RAMU reward Bellman operator follow by an appropriate change in sign.

B.1 Proof of Lemma 1

Proof. Starting from the definition of T π ρ,c in Definition 3, we have that

which proves the result. Note that the second equality follows from the positive homogeneity of ρ (Axiom A3), the third equality follows from the translation invariance of ρ (Axiom A2), and the fourth equality follows from the definition of the standard cost Bellman operator T π p,c .

B.2 Proof of Theorem 1

Proof. First, we show that T π ρ,c is equivalent to a distributionally robust Bellman operator. For a given state-action pair, we apply Lemma 2 to the risk measure that appears in the formulation of T π ρ,c

given by Lemma 1. By doing so, we have that

where U s,a is defined in Lemma 2. Therefore, T π ρ,c has the same form as a distributionally robust Bellman operator [53, 56] with the ambiguity set U = ⊗ ( s,a ) ∈S×A U s,a . The RAMU cost Q function Q π ρ,c ( s, a ) is the fixed point of T π ρ,c , so it is equivalent to a distributionally robust Q function with ambiguity set U . Using the rectangularity of U , we can write this succinctly as

Then, using the definition of J ρ,c ( π ) we have that

where we can move the optimization over U outside of the expectation operators due to rectangularity.

We can use similar techniques to show that T π ρ + ,r has the same form as a distributionally robust Bellman operator with the ambiguity set U + = ⊗ ( s,a ) ∈S×A U + s,a , and

Therefore, we have that the RAMU safe RL problem in (3) is equivalent to (6).

B.3 Proof of Corollary 1

Given the equivalence of T π ρ + ,r and T π ρ,c to distributionally robust Bellman operators as shown in Theorem 1, Corollary 1 follows from results in Xu and Mannor [53] and Yu and Xu [56]. We include a proof for completeness.

Proof. Due to the linearity of the expectation operator, for a given β s,a ∈ U s,a we have that

where ¯ p β s,a = E p s,a ∼ β s,a [ p s,a ] ∈ P ( S ) represents a mixture transition model determined by β s,a . Therefore, starting from the result in Theorem 1, we can write

where

0.0

0.5

1.0

Steps (M)

0

250

500

750

Total Reward

0.0

0.5

1.0

Steps (M)

0

500

1000

Walker Walk

0.0

0.5

1.0

Steps (M)

0

200

400

600

Unconstrained RL

0.0

0.5

1.0

Steps (M)

600

800

1000

0.0

0.5

1.0

Steps (M)

600

800

As a result, T π ρ,c has the same form as a robust Bellman operator [21, 34] with the uncertainty set P = ⊗ ( s,a ) ∈S×A P s,a .

Consider Q functions Q (1) and Q (2) , and denote the sup-norm by

Fix ϵ > 0 and consider ( s, a ) ∈ S × A . Then, there exists ¯ p (1) s,a ∈ P s,a such that

We have that

A similar argument can be used to show that

so we have that

By applying a supremum over state-action pairs on the left-hand side, we obtain

Finally, since ϵ > 0 was arbitrary, we have shown that T π ρ,c is a γ -contraction in the sup-norm.

C Implementation details and additional experimental results

Safety constraints and environment perturbations In all of our experiments, we consider the problem of optimizing a task objective while satisfying a safety constraint. We focus on a single safety constraint corresponding to a cost function defined in the Real-World RL Suite for each task,

and we consider a safety budget of B = 100 . The safety constraints used for each task are described in Table 2. In the Cartpole domain, costs are applied when the slider is outside of a specified range. In the Walker domain, costs are applied for large joint velocities. In the Quadruped domain, costs are applied for large joint angles. See Dulac-Arnold et al. [19] for detailed definitions of each safety constraint.

The definitions of these cost functions depend on a safety coefficient in [0 , 1] , which determines the range of outcomes that lead to constraint violations and therefore controls how difficult it will be to satisfy safety constraints corresponding to these cost functions. As the safety coefficient decreases, the range of safe outcomes also decreases and the safety constraint becomes more difficult to satisfy. In order to consider safe RL tasks with difficult safety constraints where strong performance is still possible, we selected the value of this safety constraint in the range of [0 . 15 , 0 . 20 , 0 . 25 , 0 . 30] for each task based on the performance of the baseline safe RL algorithm CRPO compared to the unconstrained algorithm MPO. Figure 3 shows total rewards throughout training for each task across this range of safety coefficients. We selected the most difficult cost definition in this range (i.e., lowest safety coefficient value) where CRPO is still able to achieve the same total rewards as MPO (or the value that leads to the smallest gap between the two in the case of Walker Run and Quadruped Run). The resulting safety coefficients used for our experiments are listed in Table 2.

In order to evaluate the robustness of our learned policies, we generate a range of test environments for each task based on perturbing a simulator parameter in the Real-World RL Suite. See Table 3 for the perturbation parameters and corresponding ranges considered in our experiments. The test range for each domain is centered around the nominal parameter value that defines the single training environment used for all experiments except domain randomization. See Figure 4 for detailed results of the risk-averse and risk-neutral versions of our RAMU framework across all tasks and environment perturbations.

Domain randomization Domain randomization requires a training distribution over a range of environments, which is typically defined by considering a range of simulator parameters. For the in-distribution version of domain randomization considered in our experiments, we apply a uniform distribution over a subset of the test environments defined in Table 3. In particular, we consider the middle 50% of test environment parameter values centered around the nominal environment value for training. In the out-of-distribution version of domain randomization, on the other hand, we consider a different perturbation parameter from the one varied at test time. We apply a uniform distribution over a range of values for this alternate parameter centered around the value in the nominal environment. Therefore, the only environment shared between the set of test environments and the set of training environments used for out-of-distribution domain randomization is the nominal environment. See Table 4 for details on the parameters and corresponding ranges used for training in domain randomization.

700

800

Total Reward

400

600

800

1000

Walker Walk

200

400

600

600

800

1000

800

850

0.8

1.0

1.2

Pole Length

0

100

200

Total Cost

0.1

0.2

0.3

0.4

0.5

0

100

200

0.1

0.2

0.3

0.4

0.5

0

100

200

RAMU (Wang 0.75)

RAMU (Expectation)

500

750

1000

1250

1500

0

100

200

500

750

1000

1250

1500

0

100

200

We include the results for domain randomization across all tasks and environment perturbations in Figure 5. Across all tasks, we observe that our RAMU framework leads to similar or improved constraint satisfaction compared to in-distribution domain randomization, while only using one training environment. In addition, our framework consistently outperforms out-of-distribution domain randomization, which provides little benefit compared to standard safe RL due to its misspecified training distribution.

Adversarial reinforcement learning In order to implement the action-robust PR-MDP framework, we must train an adversarial policy. We represent the adversarial policy using the same structure and neural network architecture as our main policy, and we train the adversarial policy to maximize total costs using MPO. Using the default setting in Tessler et al. [46], we apply one adversary update for every 10 policy updates.

We include the results for adversarial RL across all tasks and environment perturbations in Figure 6, where adversarial RL is evaluated without adversarial interventions. We see that adversarial RL leads to robust safety in some cases, such as the two Quadruped tasks. However, in other tasks such as Cartpole Swingup, safety constraint satisfaction is not as robust. Safety also comes at the cost of conservative performance in some tasks, as evidenced by the total rewards achieved by adversarial RL in Walker Run and Quadruped Run. Overall, our RAMU framework achieves similar performance to adversarial RL, without the drawbacks associated with adversarial methods that preclude their use in some real-world settings.

700

800

Total Reward

400

600

800

1000

Walker Walk

200

400

600

600

800

1000

800

850

0.8

1.0

1.2

Pole Length

0

100

200

Total Cost

0.1

0.2

0.3

0.4

0.5

0

100

200

0.1

0.2

0.3

0.4

0.5

0

100

200

RAMU (Wang 0.75)

Domain Rand.

Domain Rand. (OOD)

500

750

1000

1250

1500

0

100

200

500

750

1000

1250

1500

0

100

200

700

800

Total Reward

400

600

800

1000

Walker Walk

200

400

600

600

800

1000

800

850

0.8

1.0

1.2

Pole Length

0

100

200

Total Cost

0.1

0.2

0.3

0.4

0.5

0

100

200

0.1

0.2

0.3

0.4

0.5

0

100

200

RAMU (Wang 0.75)

500

750

1000

1250

1500

0

100

200

500

750

1000

1250

1500

0

100

200

Network architectures and algorithm hyperparameters In our experiments, we consider neural network representations of the policy and critics. Each of these neural networks contains 3 hidden layers of 256 units with ELU activations. In addition, we apply layer normalization followed by a tanh activation after the first layer of these networks as proposed in Abdolmaleki et al. [2]. We consider a multivariate Gaussian policy, where at a given state we have π ( a | s ) = N ( µ ( s ) , Σ( s )) where µ ( s ) and Σ( s ) represent outputs of the policy network. Σ( s ) is a diagonal covariance matrix, whose diagonal elements are calculated by applying the softplus operator to the outputs of the neural network. We parameterize the reward and cost critics with separate neural networks. In addition, we consider target networks that are updated as an exponential moving average with parameter τ = 5 e3 .

We consider CRPO [55] as the baseline safe RL algorithm in all of our experiments, which immediately switches between maximizing rewards and minimizing costs at every update based on the value of the safety constraint. If the sample-average estimate of the safety constraint for the current batch of data satisfies the safety budget, we update the policy to maximize rewards. Otherwise, we update the policy to minimize costs.

After CRPO determines the appropriate objective for the current batch of data, we apply MPO [1] to calculate policy updates. MPO calculates a non-parametric policy update based on the KL divergence parameter ϵ KL, and then takes a step towards this non-parametric policy while constraining the KL divergence from updating the mean by β µ and the KL divergence from updating the covariance matrix

by β Σ . We consider per-dimension KL divergence constraints by dividing these parameter values by the number of action dimensions, and we penalize actions outside of the feasible action limits using the multi-objective MPO framework [2] as suggested in Hoffman et al. [20]. In order to avoid potential issues related to the immediate switching between reward and cost objectives throughout training, we completely solve for the temperature parameter of the non-parametric target policy in MPO at every update as done in Liu et al. [28]. See Table 5 for the default hyperparameter values used in our experiments, which are based on default values considered in Hoffman et al. [20].

For our RAMU framework, the latent variable hyperparameter ϵ controls the definition of the distribution µ s,a over transition models. Figure 7 shows the performance of our RAMU framework in Walker Run and Quadruped Run for ϵ ∈ [0 . 05 , 0 . 10 , 0 . 15 , 0 . 20] . A larger value of ϵ leads to a distribution over a wider range of transition models, which results in a more robust approach when combined with a risk-averse perspective on model uncertainty. We see in Figure 7 that our algorithm more robustly satisfies safety constraints as ϵ increases, but this robustness also leads to a decrease in total rewards. We consider ϵ = 0 . 10 in our experiments, as it achieves strong constraint satisfaction without a meaningful decrease in rewards. Finally, for computational efficiency we consider n = 5 samples of transition models per data point to calculate sample-based Bellman targets in our RAMU framework, as we did not observe meaningful improvements in performance from considering a larger number of samples.

Computational resources All experiments were run on a Linux cluster with 2.9 GHz Intel Gold processors and NVIDIA A40 and A100 GPUs. The Real-World RL Suite is available under the Apache License 2.0. We trained policies for 1 million steps across 5 random seeds, which required

200

400

600

Total Reward

750

800

850

0.1

0.2

0.3

0.4

0.5

0

100

200

Total Cost

600

800

1000

1200

1400

0

100

200

RAMU (

ε

=0.05

)

RAMU (

ε

=0.10

)

RAMU (

ε

=0.15

)

RAMU (

ε

=0.20

)

approximately one day of wall-clock time on a single GPU for each combination of algorithm and task using code that has not been optimized for execution speed.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider a distribution over transition models and tackle the safe RL problem by applying a risk-averse perspective towards model uncertainty through coherent distortion risk measures. The proposed formulation can ease the burden of solving a min-max problem, which is often encountered in many worst-case safe RL algorithms. \n\nThe authors also theoretically show that their formulation is equivalent to a specific class of distributionally robust safe RL problems.\n\nSTRENGTHS:\n- This work proposes a new formulation of safe RL, by considering a distribution of transition models and applying the distortion risk measure toward the model uncertainty, which circumvents the burden of solving min-max problems.\n- This work theoretically proposes that the reformulated problem is equivalent to a specific class of distributionally robust safe RL problems.\n\nWEAKNESSES:\n- It is better to include the proof of Lemma 2 in Appendix B.2 for self-contained. Also, explaining the results of Lines 225-226 is better.\n- It seems that the performance of the proposed method can not surpass existing methods, especially adversarial RL. Also, the experiments presented are not sufficient.\n- Although applying the distortion risk measure $\\rho$ seems to be promising, I'm wondering about the necessity of doing so since it introduces more complex computation processes and seems to contribute little to the experiment results.\n\nQUESTIONS:\n- What's the choice of the function $g$ (line 224-225) in your experiments?\n- In lines 229-230, is there only one Q function for calculating all sampled transitions?\n- In Table 1, what’s the meaning of the bold numbers? I think you should highlight the best results instead of yours.\n- Also, please explain the necessity and merits of adopting the formulation proposed in this paper.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a methodology for distributionally robust RL via the use of risk measures and leveraging risk (Fenchel) duality, dealing with what they call model uncertainty. The paper introduces the RAMU Q function and Bellman operators respectively, which are based on modifying the standard risk-neutral operators by inducing risk-awareness over the choice of the transition model. The RAMU Q function and respective Bellman operators exhibit nice properties, due to the class of risk measures considered (distortion risk measures).\n\nThen, the paper describes a model-free implementation of the proposed approach with a single training environment, leveraging interesting results for statistical estimation of risk functionals (specifically distortion risk measures). Lastly, the proposed approach is verified on a number of numerical benchmarks.\n\nSTRENGTHS:\nOverall, I think this is a paper with potential, and it was a pleasure reading it. The RAMU framework is indeed interesting and seems to be effective, although I have some concerns (see below).\n\nThe estimation of the Bellman operators in Section 6 within the RAMU framework can indeed be computed efficiently, as the authors point out, which is an advantage of the approach. The application of the results of Jones and Zitikis is quite interesting.\n\nThe experimental section is quite detailed and the comparisons with existing methods is well-thought, especially with a large number of similar alternative approaches. Also the experiments bring out the effect and efficacy of risk-averse RL well.\n\nWEAKNESSES:\nRisk duality and equivalence to minimax (distributionally robust) optimization (under certain assumptions) is very well known and established (in essence it is just Fenchel duality). Therefore, I do not think it can be claimed as novelty or contribution of this paper by itself (e.g., in lines 5 to 7 in the abstract). I would suggest that the authors rephrase to highlight more specifically their contributions.\n\nIn the discussion about risk measures, I do not see any explanation on *why* one should choose to work with distortion risk measures or coherent risk measures, and why the axioms proposed by Majumdar and Pavone are appropriate. Axioms are subjective, so in principle there is no universal reason to use one set of axioms over another set of axioms. Of course distortion risk measures exhibit amazing analytical properties and probably this is among the main reasons justifying the authors chosing this class, but the discussion and point needs to be made.\n\nThe RAMU cost function as defined in line 156 is introduced somewhat arbitrarily, in the sense that it is unclear if this is somewhat related to a base problem such as (2). While I understand the motivation and indeed it *seems* to make sense, usually one would start from such a base problem, possibly risk-averse in a certain way that makes intuitive and operational sense, and then build their way towards Q factors and Bellman optimality conditions.\n\nTheorem 1 follows directly from the application of risk duality. I am not sure if this result \"deserves\" to be a theorem. I would think that a stating as a proposition would be more suiting. As a general comment, the theory in this paper is somewhat limited, however I have enjoyed reading the paper still.\n\nThe discussion under \"Generative distribution of transition models\", line 236 onwards is quite convoluted and involved without particular reason in my opinion. While I finally understood what happens (I think!), I believe that the authors should make an effort to write this part in a much clearer manner.\n\nQUESTIONS:\nWhat is a \"distribution over transition models\" in lines 41-42? This is very ambiguous at this point. \n\nAlso, related in Lines 72-74: Defining this product is somewhat obscure. For instance, does this make sense when S and A are uncountable sets (or you implicitly assume finite states and actions throughout)?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a Temporal Difference (TD) learning method for addressing the ``Risk-Averse Model Uncertainty for Distributionally Robust Safe Reinforcement Learning'' problem. Specifically, the authors consider a Constrained Markov Decision Process (CMDP) combined with Bayesian uncertainty sets. They are trying to attain a policy which is both robust and safe; Meaning that it optimizes a nested risk measure of discounted return, while satisfying certain guarantees over discounted cost. The proposed method is implemented and evaluated by comparing its performance against adversarial RL, Safe RL, and domain randomization approaches across multiple environments.\n\nSTRENGTHS:\n- This paper addresses a noteworthy problem. As a general matter, I believe that risk-aware policy selection methods for Bayesian MDP, as also studied in [1, 2, 3] is an interesting and open area of research.\n- Even though the problem is notation-heavy, the paper is well presented and maintains a high level of readability.\n- The authors have proven their implementation's effectiveness across five different tasks. Furthermore, the authors' utilization of a large number of training samples adds credibility to the algorithm's performance evaluation.\n\n\n[1] Giorgio Angelotti, Nicolas Drougard, & Caroline Ponzoni Carvalho Chanel. (2023). An Offline Risk-aware Policy Selection Method for Bayesian Markov Decision Processes. arXiv preprint arXiv:2105.13431.\n\n[2] Lobo, E. A., Ghavamzadeh, M., & Petrik, M. (2020). Soft-robust algorithms for batch reinforcement learning. arXiv preprint arXiv:2011.14495.\n\n[3] Petrik, M., & Russel, R. H. (2019). Beyond confidence regions: Tight Bayesian ambiguity sets for robust MDPs. Advances in neural information processing systems, 32.\n\nWEAKNESSES:\n- **Theoretical results**: The primary focus of this paper is the implementation and experimental evaluation; there are limited theoretical results in this paper.\n\n- **Comparisons**: In the ``strengths'' section of my review, I mentioned three risk-aware policy selection methods for Bayesian MDP methods that I am aware of [1, 2, 3]. These methods have similar objectives, albeit having two key differences (i) Instead of CMDPs the objective is for normal MDP and does not require safety guarantees (ii) the risk measure is (might be) applied over entire stochasticy $\\mu$ rather than recursively over each $\\mu_{s, a}$. As both algorithms use samples of models to have a Monto Carlo estimation of the risk measure, the $\\theta$ update in Algorithm 1 is particularly similar to the $\\operatorname{RiskEvaluation}$ operation introduced by [1].\n\n    - Firstly, I would suggest that the authors include these papers in the ``uncertainty in reinforcement learning'' part of the related works.\n\n    -  Secondly, it would have been more informative if the paper compared its proposed method to one of these existing methods, instead of focusing on comparisons with adversarial RL and domain randomization.\n\n- **Presentation**:  Theorem 1 requires a more concrete and explicit formulation. $\\zeta_{a, s}$ is introduced as ``depending on $\\rho^+$ and $\\rho$''; without properly defining the dual presentation of coherent risk measures beforehand. This lack of clarity makes it very challenging for non-expert readers to understand the paper.\n\n\nIn conclusion, while the paper proposes an interesting problem and provides valuable experimental results, there is a lot of room for improvement. Thus, I rate this paper as a borderline accept.\n\nQUESTIONS:\n- **The Objective**: It would be interesting to observe the performance of the proposed algorithm by varying the learning rates for $\\theta_r$ and $\\theta_c$ and exploring different risk measures for $\\rho^+$ and $\\rho$. For instance imagine we set $\\rho^+$ as $CVaR_{\\alpha_r}^+$, and $\\rho$ as $CVaR_{\\alpha_c}$. As $\\alpha_c$ decreases, the algorithm becomes more strict on violating safety conditions, and as $\\alpha_r$ decreases, the algorithm becomes more robust. I am curious to see the differences in experimental outcomes under such settings.\n- **Evaluation**: In the experiments, different algorithms are being compared based on the safety percentage, and total rewards. I would like to see their comparison based on (an estimated version of) the Lagrangian form of the actual objective (4) (while fixing the set of hyperparameters for all methods). Since the proposed algorithm directly aims to optimize this objective, it is reasonable to expect it to outperform other methods in terms of this specific evaluation metric. The conduct of such an evaluation can serve as a sanity check.\n\nLIMITATIONS:\nno limitations", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a deep reinforcement learning framework for safe decision-making in uncertain environments. The authors propose a risk-averse approach towards model uncertainty using coherent distortion risk measures. They provide robustness guarantees for the framework by showing its equivalence to a distributionally robust safe reinforcement learning problem. The framework is efficient and model-free, utilizing standard data collection from a single training environment. Experiments on continuous control tasks with safety constraints demonstrate the framework's robust and safe performance across a range of perturbed test environments.\n\nSTRENGTHS:\n[+] The paper is logically clear and well-written, making it easy to follow.\n\n[+] The paper starts from theory, first defining a new Q-function and a new Bellman operator, then proving the equivalence between the new Bellman operator and distributionally robust Bellman operators with respective ambiguity sets, thereby verifying the rationality of the definition and the contraction property of the new Bellman operator. This provides a theoretical basis for the algorithm design.\n\n[?] What is the relationship between safety and robustness? The paper theoretically proves that the newly proposed RAMU Bellman operator has robustness guarantees. However, it seems not to mention much about its relationship with Constrained Markov Decision Process (CMDP) and how to ensure that the point converged to by the RAMU Bellman operator is safe.\n\n[?] Related to the previous question, why consider both safety and robustness here? It seems that considering both safety and robustness is crucial for the Theorem 1. Is it necessary to clarify this point more explicitly in the abstract? For example, the current abstract's first sentence emphasizes the importance of safety, the line 5 separately emphasizes robustness guarantees, and the line 7 re-emphasizes robustness, which can be somewhat confusing.\n\nWEAKNESSES:\nsee above\n\nQUESTIONS:\nsee above\n\nLIMITATIONS:\nsee above", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the Risk-Averse Model Uncertainty (RAMU) framework for safe reinforcement learning in uncertain environments. RAMU incorporates a distribution of transition models and applies a risk-averse perspective using coherent distortion risk measures. The framework offers an efficient, model-free implementation through one single training environment. Experimental results demonstrate the framework's ability to produce robust, safe performance in perturbed test environments. Unlike existing distributional robust (DR) approaches, RAMU eliminates the need for minimax optimization.\n\nSTRENGTHS:\n- The proposed perturbation function models the distribution of environment transition models, providing a foundation for practical implementation of distributionally robust algorithms.\n-  Compared to existing methods, the RAMU framework is implemented efficiently. It avoids complex minimax optimization, which is the major obstacle to applying Distributional Robustness methods to DRL.\n\nWEAKNESSES:\n- The problem addressed in this paper is the handling of model uncertainty in safe reinforcement learning (RL) scenarios. However, it appears that the issue of model uncertainty in safe RL is similar to the problem in standard RL. Since the proposed RAMU method is not specifically designed for safe RL, it would be more convincing if it were compared to existing distributionally robust methods such as those discussed in [1] and [2].\n- The formulation of the distributionally robust safety problem in the article lacks clarity. The problem definition starts with directly modifying the Q function to the DR Q function in Eq4, which is a shortcut approach. It would have been more appropriate to first define the problem and then derive the suitable form of the Q function. Although solving the Eq4 definition is straightforward by plugging in the safety RL method with DR RL, it is important to note that the worst-case transitions associated with the reward and cost (the $\\beta$ in Eq5 and Eq6) are not the same. However, in reality, there is only one transition, which makes the proposed method more conservative in practice.\n-  The transition function f(s, s') mentioned in line 257 may only be effective for tasks involving robot control, where dynamics follow linear patterns and such perturbations are effective. For tasks involving image inputs, it remains unclear what kind of function f should be used.\n\nQUESTIONS:\n- I understand that the contribution of the paper lies in proposing a distributionally robust (DR) safe reinforcement learning method that does not require a minmax operation. However, given that there is no direct connection between DR algorithm design and safety, it is essential to compare it with existing DR methods such as [1] and [2].\n- The proposed algorithms does have similarities to policy smoothing algorithms, as mentioned in [3] and [4]. It is plausible to consider that combining policy smoothing with adversarial RL could potentially yield better results compared to the RAMU framework. In other words, exploring the combination of observational robustness and action robustness may provide improvements over predefined model robustness?\n\nRef.\n\n[1] Robust Reinforcement Learning using Offline Data\n\n[2] Distributionally Robust Q-Learning\n\n[3] Deep Reinforcement Learning with Robust and Smooth Policy\n\n[4] Policy Smoothing for Provably Robust reinforcement learning.\n\nLIMITATIONS:\nAs mentioned in the Conclusion section of the paper, the choice of the model distribution µ and risk measure ρ in the RAMU framework is user-defined, and RAMU framework only addresses robustness with respect to model uncertainty and safety defined by expected total cost constraints. Crruently, this paper only consider the dynamics of model transition follow linear patterns, which may not to be suitable for tasks that involve image inputs.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-19"}