Risk-Averse Model Uncertainty for Distributionally Robust Safe Reinforcement Learning
Authors: James Queeney
Mouhacine Benosman
Venue: NeurIPS 2023 poster (2023)
Keywords: deep reinforcement learning
model uncertainty
safety
risk-averse
distributionally robust
TL;DR: We introduce a deep reinforcement learning framework that incorporates risk-averse model uncertainty into safe decision making, which provides robustness and can be implemented efficiently.

Abstract: Many real-world domains require safe decision making in uncertain environments. In this work, we introduce a deep reinforcement learning framework for approaching this important problem. We consider a distribution over transition models, and apply a risk-averse perspective towards model uncertainty through the use of coherent distortion risk measures. We provide robustness guarantees for this framework by showing it is equivalent to a specific class of distributionally robust safe reinforcement learning problems. Unlike existing approaches to robustness in deep reinforcement learning, however, our formulation does not involve minimax optimization. This leads to an efficient, model-free implementation of our approach that only requires standard data collection from a single training environment. In experiments on continuous control tasks with safety constraints, we demonstrate that our framework produces robust performance and safety at deployment time across a range of perturbed test environments.

Risk-Averse Model Uncertainty for Distributionally Robust Safe Reinforcement Learning
James Queeney" Division of Systems Engineering Boston University jqueeneyObu edu
Mouhacine Benosman
Mitsubishi Electric Research Laboratories benosman@merl.. com
Abstract
Many real-world domains require safe decision making in uncertain environments. In this work, introduce deep reinforcement learning framework for approach- ing this important problem. We consider a distribution over transition models, and apply risk-averse perspective towards model uncertainty through the of coherent distortion risk measures. We provide robustness guarantees for this framework by showing is equivalent a specific class of distributionally robust safe reinforcement learning problems. Unlike existing approaches to robustness in deep reinforcement learning, however, our formulation does not involve minimax optimization. This leads to an efficient, model-free implementation of our approach that only requires standard data collection from a single training environment. experiments on continuous control tasks with safety constraints, we demonstrate that our framework produces robust performance and safety deployment time across range of perturbed test environments.
Introduction
In many real-world decision making applications, is important to satisfy safety requirements while achieving desired goal. In addition, real-world environments often involve uncertain changing conditions. Therefore, in order to reliably deploy data-driven decision making methods such as deep reinforcement learning (RL) in these settings, they must deliver robust performance and safety even in the presence of uncertainty. Recently, techniques have been developed to handle safety constraints within the deep RL framework performance and safety in the training environment. They do not consider uncertainty about the true environment deployment time due to unknown disturbances or irreducible modeling errors, which we refer to as model uncertainty. In this work, we introduce framework that incorporates model uncertainty into safe RI. In order for our framework to useful, we emphasize the importance of (i) an efficient deep RI. implementation during training and (ii) robustness guarantees on performance and safety upon deployment.
27
55 but these safe RL algorithms only focus on
Existing robust RL methods address the issue of model uncertainty, but they can be difficult to implement and are not always suitable in real-world decision making settings. Robust RL focuses on worst-case environments in an uncertainty set, which requires solving complex minimax optimization problems throughout training. This is typically approximated
deep RL setting through direct interventions with a learned adversary [38, 46 49] or through the use of parametric uncertainty with
39 However, we do not always have access to In these cases, we must be able to incorporate robustness model uncertainty without relying on multiple training environments or potentially dangerous adversarial interventions, as real-world data collection may be necessary.
multiple simulated training environments [3
32
fast, high- high-fidelity simulators for training [0]
33 54
Work partly done during an internship at Mitsubishi Electric Research Laboratorics.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

A more informative way to represent model uncertainty is to instead consider a distribution over po- tential environments. Domain randomization [37] collects training data from range of environments by randomizing across parameter values in simulator, and optimizes for average performance. This approach to model uncertainty avoids minimax formulations and works well in practice [5], but lacks robustness guarantees. In addition, domain randomization focuses on parametric uncertainty, which still requires detailed simulator access and domain knowledge to define the training distribution.
In this work, we introduce a general approach to safe RL in the presence of model uncertainty that addresses the main shortcomings of existing methods. particular, we consider a distribution over potential environments, and apply risk-averse perspective towards model uncertainty. Through the use of coherent distortion risk measures, this leads a safe RL framework with robustness guarantees that does not involve difficult minimax formulations. Using this framework, we show how we can learn safe policies that are robust to model uncertainty, without the need for detailed simulator access or adversarial interventions during training. Our main contributions are as follows:
1. We reformulate the safe RL problem incorporate risk-averse perspective towards model uncertainty through the use of coherent distortion risk measures, and we introduce the corresponding Bellman operators.
2. From a theoretical standpoint, we provide robustness guarantees for our framework by showing is equivalent
specific class of distributionally robust safe RL problems.
3. We propose an efficient deep RL implementation that avoids the difficult minimax formula- tion present in robust RL and only uses data collected from a single training environment.
4. We demonstrate the robust performance and safety of our framework through experiments on continuous control tasks with safety constraints in the Real- World RL Suite [18] 19
2 Preliminaries
Safe reinforcement learning In this work, we consider RL in the presence of safety constraints. We model this sequential decision making problem as an infinite-horizon, discounted Constrained Markov Decision Process (CMDP) [4] defined by the tuple AS,A, p, c, do 7), where S is the set of states, A is the set of actions, p: S x A + P(S) is the transition model where P(S) represents the space of probability measures over S, r. x A A R are the reward function and cost function used to define the objective and constraint, respectively, do E P(S) is the initial state distribution, and 7 is the discount rate. We focus on the setting with single constraint, but all results can be extended to the case of multiple constraints.
We model the agent's decisions stationary policy π S + P(A). For given CMDP and policy т, we write the expected total discounted rewards and costs as (m) = (x-((xp) 7 ('r( at)] and (x) )= = (p,() [Er= y'c(st, dt)], respectively, where T ~ (T, p) represents a trajectory sampled according to 8g do, at ~ x st), and 8t-1 ~ p( - st, az). The goal of safe RL is to find a policy that maximizes the constrained optimization problem
max (T) s.t. Jp.c(T) S B,
where B is a safety budget on expected total discounted costs.
We write the corresponding state-action value functions (i.c., Q functions) for a given transition model p and policy π as Qp.r (s, a) and Q (s, (s, respectively. Off-policy optimization techniques
28 55
find a policy that maximizes ① by solving each iteration the related optimization problem
ERE max
E E (a~x(-1s)
[075
s.t.
E
E
[o (s, a)]
s BB
TT
AND
aND
(()T(()s)
where Th is the current policy and D is a replay buffer containing data collected in the training environment. Note that Q5 (8,a) and Q (%,( a) are the respective fixed points of the Bellman
operators
Q(s,() = r(s,a) +7 E
E La ('
1Q(s' a')]
s'rppsa
((, = c(s,a) + +
(Q(5/
8¹ Papa (a'nus(-lo")

---

Model uncertainty in reinforcement learning Rather than focusing on single CMDP with transition model p. we incorporate uncertainty about the transition model by considering distribution over models. We focus on distributions of the form j [1(s.))SSxA Ms,as where Ms,. represents distribution over transition models Ps,e p(- s, s,a) P(S) at a given state-actin aitteacciinn pair and j is the product over all [4x.0- This is known as rectangularity, and is a common assumption in the
literature [1 15
16 53 [56]. Note that Ns,a E P(M). where we write M = MP)(S) to denote model space. Compared to robust RL methods that apply uncertainty sets over transition models, the use of a distribution μ over transition models is more informative way to represent model uncertainty that does not require solving for worst-case environments (i.e., does not introduce a minimax formulation).
In order to incorporate robustness to the choice of μ. distributionally robust MDPs [3] [56] consider
an ambiguity set = X @ S aa of distributions over transition models, where Ms,a E Uss,a S
P(M). The goal of distributionally robust RL is to optimize the worst-case average performance across all distributions contained in 24. this work, we will show that risk-averse perspective towards model uncertainty defined by ju. is equivalent to distributionally robust RL for appropriate choices of ambiguity sets in the objective and constraint of CMDP. However, our use of risk measures avoids the need to solve for worst-case distributions in U throughout training.
Risk measures Consider the probability space (M, Fe,e,,), where F is a o-algebra on M and Me.. P(M) defines a probability measure over M. Let Z be a space of random variables defined on this probability space, and let 2 be its corresponding dual space. A real-valued risk measure p= : Z . summarizes random variable as value the real line. In this section, we consider cost random variables Z E 2 where a lower value of p(Z) is better. We can define a corresponding risk measure p² for reward random variables through an appropriate change in sign, where p² (2) = pp(-2p -)). Risk-sensitive methods typically focus on classes of risk measures with desirable properties 30 such coherent risk measures 66 and distortion risk measures 17 50].
Definition 1 (Coherent risk measure). A risk measure is coherent risk measure it satisfies monotonicity, translation invariance, positive homogeneity, and convexity.
Definition 2 (Distortion risk measure). Let g [0, 1] + [0, 1] be a non-decreasing, left-continuous function with g(0) = 0 and g() = 1. A distortion risk measure with respect g is defined
p(Z) - ( Fz (u)dg(u),
where Fz' is the inverse cumulative distribution function of Z and g()) = g(1 - u).
distortion risk measure is coherent if and only if g is concave 152 In this work, we focus on the class of coherent distortion risk measures. We will leverage properties of coherent risk measures to provide robustness guarantees for our framework, and we will leverage properties of distortion risk measures to propose an efficient, model-free implementation that does not involve minimax optimization. See the Appendix for additional details on the properties of coherent distortion risk measures. Many commonly used risk measures belong to this class, including expectation, conditional value-at-risk (CVaR), and the Wang transform [5]] for 7) 2 O which is defined by the distortion function (u) 4 (
(w) + 17). where the standard Normal cumulative distribution function.
Related work
Safe reinforcement learning The CMDP framework is the most popular approach to safety in RL, and several deep RL algorithms have been developed to solve the constrained optimization problem
in (]. These include primal-dual methods that consider the Lagrangian relaxation of (T) 41 44 47).
algorithms that compute closed-fore solutions to related or approximate versions of ( 3 28 and
direct methods for constraint satisfaction such as the use of barriers [27] or immediate switching
between the objective and constraint (55). All of these approaches are designed to satisfy expected
cost constraints for a single CMDP observed during training. In our work, on the other hand, we
consider a distribution over possible transition models.
Uncertainty in reinforcement learning Our work focuses on irreducible uncertainty about the true environment deployment time, which we refer to model uncertainty and represent using
3

---

distribution je over transition models. The most popular approach that incorporates model uncertainty in this way is domain randomization [77 48 which randomizes across parameter values in a simulator and trains policy to maximize average performance over this training distribution. This represents risk-neutral attitude towards model uncertainty, which has been referred to as a soft- robust approach [16]. Distributionally robust MDPs incorporate robustness to the choice of by instead considering a set of distributions [U D5 53 [6]. but application of this distributionally robust framework has remained limited in deep RL it leads to difficult minimax formulation that requires solving for worst-case distributions over transition models.
Robust RL represents an alternative approach to model uncertainty that considers uncertainty sets of transition models [21 34]. A major drawback of robust RL is the need to calculate worst- case environments during training, which is typically approximated through the use of parametric uncertainty with multiple training environments (31, [2 39 or a trained adversary that directly intervenes during trajectory rollouts [8 46 49). Unlike these methods, we propose robust approach to model uncertainty based on a distribution j over models, which does not require access to range of simulated training environments, does not impact data collection during training, and does not involve minimax optimization problems.
In contrast to irreducible model uncertainty, epistemic uncertainty captures estimation error that can be reduced during training through data collection. Epistemic uncertainty has been considered in the estimation of Q functions [9] 35 36) and learned transition models 77 13 22 25 40]. and has been applied to promote both exploration and safety fixed MDP. Finally, sensitenieevve methods typically focus on the aleatoric uncertainty in RL, which refers to the range of stochastic outcomes within single MDP. Rather than considering the standard expected value objective, they learn risk-sensitive policies over this distribution of possible outcomes in fixed MDP [12 24
26 43 45
Distributional RL (8] trains critics that estimate the full distribution of future returns due to aleatoric uncertainty, and risk measures can be applied to these distributional critics for risk-sensitive learning We also consider the use of risk measures in our work, but different from standard risk- sensitive RL methods we apply risk measure over model uncertainty instead of aleatoric uncertainty.
14 29
4 Risk-averse model uncertainty for safe reinforcement learning
The standard safe RL problem in (I) focuses on performance and safety in single environment with fixed transition model In this work, however, we are interested in a distribution of possible transition models p ~ μ rather than fixed transition model. The distribution μ provides a natural way to capture our uncertainty about the unknown transition model at deployment time. Next, we must incorporate this model uncertainty into our problem formulation. Prior methods have done this by applying the expectation operator over Hs,a at every transition [16]. Instead, we adopt risk-averse view towards model uncertainty order to learn policies with robust performance and safety. We accomplish this by applying coherent distortion risk measure with respect to model uncertainty at every transition.
We consider the risk-averse model uncertainty (RAMU) safe RL problem
max Jpt.
s.t. Jp.e(T) S B.
(3)
where we use pt and to account for reward and cost random variables, respectively, and we apply these coherent distortion risk measures over Ps.a Mssa at every transition to define
(r(s',a') + +..11]]
Jpt = E enndo
E cowor(-la)
r(s, a) + pt
E 8 s'mips.a
E La (a'
[o(s', a') + DDI
Jp.(TT = E enddo
E Lawot(-ls)
c(s, a) +y
P Pa
E s'rupsia
E (a(=a(-11)))
The notation Pr Np... (-) and P ~ emphasize that the stochasticity of the random vari- ables are with respect to the transition models sampled from Ps.a- Note that we still apply expectations over the aleatoric uncertainty of the CMDP (i.e., the randomness associated with a stochastic transi- tion model and stochastic policy), while being avsk-averse with respect to model uncertainty. Because we are interested in learning policies that achieve robust performance and robust safety at deployment

---

time, we apply this risk-averse perspective to model uncertainty in both the objective and constraint of 3
(s, a) and Q (8, a), respectively. Similar to the standard safe RL setting, we can apply off-policy techniques to solve the RAMU safe RL problem (3) by iteratively optimizing
We write the corresponding RAMU reward and cost Q functions as Qz --
(2t (r. (s.
(s, ai]
max
E
E
s.t.
E
E Lamm(-is))
- B.
La--)(1ss)
AND
Therefore, have replaced the standard Q functions for fixed transition model p in with RAMU Q functions in (4).
We can write the RAMU Q functions recursively
(),,(s, ((ssa)) +7
IE p'rppe,a Pan
p"
aa'
(p = c(s, a) +
P Peeampee
E
E (nrr(-1ss))
sloppeo
L
These recursive definitions motivate corresponding RAMU Bellman operators.
Definition 3 (RAMU Bellman operators). For given policy æ,, the RAMU Bellman operators are defined as
= (=),Q((sa)===(((
+7
o+
E
E La'wa(-1a")
QQ(''
Pa.a para
s''ppa,a
(p,((s, = = +y
1Q(5' 2((s',
(a'not(:(s")
Note that the RAMU Bellman operators can also interpreted as applying coherent distortion risk measure standard Bellman targets, which random variables with respect to the transition model Ps.a ~ Ha.a for a given state-action pair.
Lemma 1. The RAMU Bellman operators can be written in terms of standard Bellman operators as T (()
p" Pa.arMposj
(Tr,Q(s,a)), (2,oQss,a)=
P Pu.a-pas.a
(Tro((s,)). (5)
Proof. The results follow from the definitions T and Tp.e along with the translation invariance and positive homogeneity of coherent distortion risk measures. See the Appendix for details.
In the next section, show that T and Tp are contraction operators, we can apply standard temporal difference learning techniques to learn the RAMU Q functions Q O7+, (s, a) and Q7. a) that are needed for our RAMU policy update in (4).
Robustness guarantees
Intuitively, our risk-averse perspective places more emphasis potential transition models that result in higher costs or lower rewards under the current policy, which should result learning safe policies that are robust model uncertainty. Next, we formalize the robustness guarantees of our RAMU framework by showing it is equivalent to a distributionally robust safe RL problem for appropriate choices of ambiguity sets.
Theorem 1. The RAMU safe RL problem in (3) is equivalent to the distributionally robust safe RL problem
max inf E Bell
[J)((T)] s.t.
sup E Bell pmu
[Jpo(TT)]
S
(6)
pag
with ambiguity sets u+ (s,0)SSxA Ut and ( Us where
u Us C (Ba P(M) Ba.a Ex Es,a Z*)
sets of feasible reweightings of ps with Essa that depend the choice of pt and p. respectively.
5

---

Proof. Using duality results for coherent risk measures 42 we show that the RAMU Bellman operators p and Tpe are equivalent to distributionally robust Bellman operators [53 56 with
ambiguity sets of distributions ut and U. respectively. The RAMU Q functions are the respective fixed points of these Bellman operators, they can be written as distributionally robust functions. Finally, by averaging over initial states and actions, we see that (3 is equivalent to ( See the Appendix for details.
Theorem 1 shows that the application of pt and p at every timestep are equivalent to solving distributionally robust optimization problems respectively. This can be interpreted as adversarially reweighting Hs.a with Çe,a at every state-action pair. Note that worst-case distributions appear in both the objective and constraint of (6). any policy trained with our RAMU framework is guaranteed to deliver robust performance and robust safety. The level of robustness depends the choice of pt and p. which determine the structure and size of the corresponding ambiguity sets based on their dual representations [42
the ambiguity sets of distributions U and U,
In addition, because (3) is equivalent to distributionally robust safe RL problem according to Theorem [] we can leverage existing results for distributionally robust MDPs 5
56
to show that
T and Tnd are contraction operators.
Corollary 1. The RAMU Bellman operators p+ and Tp. are "-contractions the sup-norm.
Proof. Apply results from Xu and Mannor 13 and Yu and Xu 56 See the Appendix for details.
Therefore, we have that (-, a) and (7, a) can be interpreted as distributionally robust Q functions by Theorem | and we can apply standard temporal difference methods to learn these RAMU Q functions as a result of Corollary of our RAMU framework, but it is not used to implement our approach. Directly implementing (6) would require solving for adversarial distributions over transition models throughout training. Instead, our framework provides the same robustness, but the use of risk measures leads to an efficient deep RL implementation as we describe in the following section.
Importantly, Theorem demonsttatee the robustness properties
6 Model-free implementation with single training environment
The RAMU policy update in 44 takes the same form as the standard safe RL update in except for the use Q pttt -tt..((s (s, a) and (s, (s,a)) a). Because our RAMU Bellman operators are contractions, we can learn these RAMU Q functions by applying standard temporal difference loss functions that are used throughout deep RL. In particular, we consider parameterized critics Qo, and Qa and we optimize their parameters during training to minimize the loss functions
C+ = E. (s,a)> [90. (0.(s,) Tn 27
(Q0 - -0
[(O.)
E (s,a)~D
where t , and fr represent sample-based estimates of the RAMU Bellman operators applied to target Q functions denoted by Q. Therefore, we must be able to efficiently estimate the RAMU Bellman targets, which involve caleulating coherent distortion risk measures that depend the distribution 1s,.a
Sample- based estimation of risk measures Using the formulation our RAMU Bellman oper- ators from Lemma [ we can leverage properties of distortion risk measures to efficiently estimate the results in (5) using sample-based weighted averages of standard Bellman targets. For n tran- sition models ps.a. i = 1,... sampled independently from Hs,a and sorted according to their corresponding Bellman targets, consider the weights
1
ml()) p
- g
71

---

0.5
0.75
Wang 0.75)
g
Waight
0.3
0.50
0.2
0.25
Expectation Wang 0.75)
0.1
0.00
0.0
0.2
0.4
0.6
0.8
Figure 1: Coherent distortion risk measures used in RAMU experiments. Left: Distortion function Right: Weights for sample-bas -baeed estimates in when n
where y defines the distortion risk measure / according Definition 2 See Figure for the distortion functions and weights associated with the risk measures used in our experiments. Then, from Jones and Zitikis [23] we have that
w T ass
E T Tia, Q(s,a),
Q(s,a)
are consistent estimators of the results in (5). where Ti Ci,,Q(ssss a) are sorted ascending order and pio (s, are sorted in descending order. Finally, we can replace p (1s,)) a) and T p Q with the standard unbiased sample-based based estimates
Th pi Qs. a) = r(s,a) + 7Q(s',a', T,,,000s,00 = c(s,a) +(((sa))(ss)))
where s ps.a and a' (() This leads to the sample- basele estimates
7 Σ 1 10 ((s, Q(s
(2, = w (p,,
(7)
which we use to train our RAMU Q functions. Note that the estimates in 7 can be computed very efficiently, which is a major benefit of our RAMU framework compared to robust RL methods. Next, we describe how we can sample models ps.., i = 1,..., from Nssaa and generate state transitions from these models to use the calculation of our sample-based Bellman targets 7
Generative distribution of transition models Note that our RAMU framework can be applied using any choice of distribution μt, provided we sample transition models ps.a Ms,a and corresponding next states s' ~ Ps.a. In this work, we define the distribution over perturbed versions of single training environment p train and we propose a generative approach sampling transition models and corresponding next states that only requires data collected from p pirain By doing so, our RAMU framework achieves robust performance and safety with minimal assumptions on the training process, and can even be applied to settings that require real-world data collection for training.
(6)
We consider latent variable x X, and we define transition model Ps.a (x)) for every X that shifts the probability of s' under train according perturbation function takes input state transition ss, and outputs perturbed next state 3' that depends on the latent variable x ~ X. Therefore, a distribution over latent space implicitly defines a distribution Ms.o over perturbed versions of prrain In order obtain the next state samples needed to compute the Bellman target estimates in (7). we sample latent variables r ~ X and apply fa to the state transition observed in the training environment. We have that s' y for data collected in the training environment, 8' (s represents the corresponding sample from the perturbed transition model (s.a(x).
perturbation function S - S. This
In our experiments, we consider simple implementation for the common case where S Rd. We use uniformly distributed latent variables ~ U[[ 2e, 2x] 2,)d), and we define the perturbation function
as
fa(s, (s' s) 2),

---

Algorithm Risk-Averse Model Uncertainty for Safe RL
Input: policy no. crities Qu Qs., risk measures pt , latent random variable X for k = 0, .,. do Collect data 5 (TT. plmii) and store it in D
for K updates do
Sample batch of data (s, C, D
Sample n latent variables 2, ~ X per data point, and compute next state samples fz, (s, s')
Calculate Bellman targets in 7 and update crities Qo Qv to minimize (0,,), ((0.)
Update policy क according
end for end for
where all operations are performed per-coordinate. Therefore, the latent variable ([[-2e, [e, 2 can be interpreted as the percentage change in each dimension of a state transition observed in the training environment, where the average magnitude of the percentage change is e. The hyperparameter determines the distribution MMs,e over transition models, where a larger value of leads to transition models that vary more significantly from the training environment. The structure of fa provides an intuitive, scale-invariant meaning for the hyperparameter c, which makes it easy to tune in practice. This choice of distribution Ma.a captures general uncertainty in the training environment, without requiring specific domain knowledge of potential disturbances.
Algorithm We summarize the implementation of our RAMU framework in Algorithm | Given data collected in single training environment,
efficiently calculate the sample-based RAMU Bellman targets in (7) by (i) sampling from a latent variable x ~ X, (ii) computing the corresponding next state samples fa (s, s'), and (iii) sorting the standard Bellman estimates that correspond to these sampled transition models. Given the sample- based RAMU Bellman targets, updates of the critics and policy have the same form as in standard deep safe RL algorithms. Therefore, our RAMU frumework can be easily combined with many popular safe RI. algorithms to incorporate model uncertainty with robusiness guarantees, using only minor change the estimation of Bellman targets that is efficient to implement in practice.
7 Experiments
In order to evaluate the performance and safety of our RAMU framework, we conduct experiments on 5 continuous control tasks with safety constraints from the Real-World RL Suite [18 19] Cartpole Swingup. Walker Walk, Walker Run, Quadruped Walk, and Quadruped Run. Each task has a horizon length of 1,000 with r(s, a) [0, and c(s,a) E {0, 1}, and we consider safety budget of B = 100. Unless noted otherwise, we train these tasks on a single training environment for million steps across random seeds, and we evaluate performance of the learned policies across range of perturbed test environments via 10 trajectory rollouts. See the Appendix for information on the safety constraints and environment perturbations that we consider.
Our RAMU framework can be combined with several choices of safe RL algorithms. We consider the safe RL algorithm Conntraint-Rectifiie Policy Optimization (CRPO) 55 and a Posteriori Policy Optimization (MPO) I as the unconstrained policy optimization algorithm in CRPO. For fair comparison, we apply this choice of safe RL policy update in every method consider in our experiments. We use multivariate Gaussian policy with learned mean and diagonal covariance at each state, along with separate reward and cost critics. We parameterize our policy and critics using neural networks. See the Appendix for implementation details
Maximum
We summarize the performance and safety of RAMU framework in Table and Figure 2 compared to several bascline algorithms that we discuss next. We include detailed experimental results across all perturbed test environments in the Appendix. We apply our RAMU framework using the Wang transform with 7) = 0.75 as the risk measure both the objective and constraint. In order
Code is publicly available httppss: hitttss//ggtthtt.ccmm/////e////e

---

Table 1: Aggregate performance summary
Normalized Ave.
Rollouts Require*
Algorithm
% Safe
Reward
Cost
Adversary
Simulator
Safe RL
51%
1.00
1.00
No
No
RAMU (Wang 0.75)
80 %
1.08
0.51
No
No
RAMU (Expectation)
74%
1.05
0.67
No
No
Domain Randomization
76%
1.14
0.72
No
Yes
Domain Randomization (OOD)
55%
1.02
1.02
No
Yes
Adversarial RL
82%
1.05
0.48
Yes
No
1 Percentage of policies that satisfy the safety constraint across all tasks and test environments.
Normalized relative the average performance of standard safe RL each task and environment.
Denotes need for adversary or simulator during data collection (i.c., trajectory rollouts) for training.
Swingup
Quaddruped
Quadruped
Tool (beooed
850
676
608
300
6SC
425
604
850
520
800
352
7.8
665
400
.0.
377
750
750
250
Safe
5.1%
54%
70%
135
70%
50%
20%
79% 79%
50%
79%
60
71% 2i/5s 21%
60
70%
34%
Figure Performance summary by task, aggregated across perturbed environments. Performance of adversarial RL is evaluated without adversarial interventions. Top: Total rewards averaged across test environments. Bottom: Percentage of policies across test environments that satisfy the safety constraint.
understand the impact of being risk-averse to model uncertainty, we also consider the neutrll special case of our framework where expectations are applied to the objective and constraint. For our RAMU results Table and Figure we specify the risk measure in parentheses. Finally, we consider = 5 samples of transition models with latent variable hyperparameter 0.10 in order to calculate Bellman targets in our RAMU framework.
Comparison to safe reinforcement learning First, we analyze the impact of our RAMU framework compared to standard safe RL. In both cases, we train policies using data collected from a single training environment, the only difference comes from use of risk-averse model uncertainty to learn RAMU Q functions. By evaluating the learned policies in perturbed test environments different from the training environment, both total rewards and safety. particular, the risk-averse implementation of our algorithm leads to safety constraint satisfaction in 80% of test environments, compared to only 51% with standard safe RL. In addition, this implementation results in higher total rewards (1.08x) and lower total costs model uncertainty (i.e.,
that our RAMU framework provides robustness in terms of
(0.5Ix), on average. We see in Table that the use of expectations risk-neutral approach) also improves robustness in both the objective and constraint, on average, compared standard safe RL. However, further improve upon the benefits observed in the risk-neutral case by instead applying a risk-averse perspective.
Comparison domain randomization Next, we compare our RAMU framework to domain randomization, a popular approach that also represents model uncertainty using distribution μ over models. Note that domain randomization considers parametric uncertainty and has the benefit of training on range of simulated environments, while our method only collects data from a single training environment. In order to evaluate the importance domain knowledge for defining the training distribution in domain randomization, consider two different cases: an in-distribution

---

version that trains on a subset of the perturbed test environments, and an out-of of--isitibbutiinn (OOD) version that randomizes over a different perturbation parameter than the one varied at test time.
The results in Tan Figure 2 show the importance of domain knowledge: in-distribution domain randomization leads to improved robustness compared to standard safe RL. and the highest normalized average rewards (1. 14x), while the out-of-distribttion version provides little benefit. In both cases, however, domain randomization achieves lower levels of safety, on average, than our risk-averse formulation. In fact, we see in Figure 2 that the safety constraint satisfaction of our risk-averse formulation is at least as strong as both versions of domain randomization in 4 out of 5 tasks, despite only training on single environment with no specific knowledge about the disturbances at test time. This demonstrates the key benefit of our risk-averse approach model uncertainty.
Comparison to adversarial reinforcement learning Finally, we compare our approach to ad- versarial RL using the action-robust PR-MDP framework [46]. which randomly applies worst-case actions a percentage of the time during data collection. Although adversarial RL only collects data from single training enviromment, it requires potentially dangerous adversarial interventions during training in order to provide robustness at test time. In order to apply this method to the safe RL setting, train an adversary maximize costs and consider 5% probability of intervention during training. The performance of adversarial RL is typically evaluated without adversarial interventions, which requires a clear distinction between training and testing.
We see in Figure that adversarial RL learns policies that achieve robust safety constraint satisfaction at test time in the Quadruped tasks. Our risk-averse formulation, on the other hand, achieves higher levels of safety in the remaining 3 out of 5 tasks, and similar levels of safety on average. Unlike adversarial RL, our RAMU framework achieves robust safety in way that (i) does not alter the data collection process, (ii) does not require training an adversary in a minimax formulation, and of
(iii) does not require different implementations during training and testing. In addition, distribution over models represents a less conservative approach than adversarial RL, resulting in higher normalized average rewards shown in Table [
8 Conclusion
We have presented framework for safe RL in the presence of model uncertainty, an important setting for many real-world decision making applications. Compared to existing approaches model uncertainty in deep RI., our formulation applies risk-averse perspective through the use of coherent distortion risk measures. We show that this results in robustness guarantees, while still leading to an efficient deep RL implementation that does not involve minimax optimization problems. Importantly, our method only requires data collected from a single training environment, so can be applied to real-world domains where high-fidelity simulators are not readily available expensive. Therefore, our framework represents an attractive approach to safe decision making under model uncertainty that can be deployed across range of applications.
computationally
Prior to potential deployment, it is important to understand the limitations of our proposed methodol- ogy. The robustness and safety of our RAMU framework depend on the user-defined choices of model distribution μ and risk measure p. The distribution js defines the uncertainty over transition models, approach
and the risk measure p defines the level of robustness this choice of μ. In addition, only considers robustness with respect to model uncertainty and safety as defined by expected total cost constraints. It would be interesting to extend our techniques to address other forms of uncertainty and other definitions of safety, including epistemic uncertainty in model- RL, observational uncertainty, and safety- crritccll formulations based on sets of unsafe states.
References
[1] A. Abdolmaleki, J. T. Springenberg, Y. Tassa, R. Munos, N. Heess, and M. Riedmiller. Maxi- mum posteriori policy optimisation. In Sixth International Conference on Learning Represen- tations, 2018.
[2] A. Abdolmaleki, S. Huang, I.. Hasenclever, M. Neunert, F. Song, M. Zambelli, M. Martins, N. Heess, R. Hadsell, and M. Riedmiller. A distributional view multi-objective policy
10

---

optimization. In Proceedings of the 37th International Conference on Machine Learning. volume 119, pages 11-22. PMLR, 2020.
[3] J. Achiam, D. Held, A. Tamar, and P. Abbeel. Constrained policy optimization. In Proceedings of the 34th International Conference Machine Learning, volume 70, pages 22-31. PMLR, 2017.
[4] E. Altman. Constrained Markov Decision Processes. CRC Press, 1999.
[5] M. Andrychowiez, B. Baker, M. Chociej, R. Józefowicz, McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and
W. Zaremba. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39)1)33-,,, 2020. doi: 10. 10.11770077336491188777
[6] Artzner, F. Delbaen, J.-M. Eber, and D. Heath. Coherent measures of risk. Mathemattiaa Finance, 93):203--223 1999. doi: 10.
[7] Y. As, Usmanova, S. Curi, and A. Krause. Constrained policy optimization via Bayesian world models. In Tenth International Conference on Learning Representations, 2022.
[8] M. G. Bellemare, W. Dabney, and R. Munos. A distributional perspective reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 449-458. PMLR, 2017.
[9] H. Bharadhwaj. A. Kumar, N. Rhinehart, Levine, Shkurti, and Garg. Conservative safety critics for exploration. In Ninth International Conference on Learning Representations, 2021.
[10] W. Cao, M. Benosmma, and R. Ma. Domain knowledge-based automated analog circuit design with deep reinforcement learning. In The 59th ACM/IEEE Design Automation Conference, 2022.
[11] R. Chen and I. C. Paschalidis. Distributionally robust learning. Foundations and Trends in Optimization, 4(1-2):1-243, 2020. doi: 10. 10.1561/2400000026.
[12) Y. Chow, A. Tamar, S. Mannor, and M. Pavone. Risk-sensitive and robust decision-making: a CVaR optimization approach. In Advances in Neural Information Processing Systems, vol- ume 28. Curran Associates, Inc., 2015.
[13] K. Chua, R. Calandra, R. McAllister, and Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[14] W. Dabney, G. Ostrovski, D. Silver, and R. Munos. Implicit quantile networks for distributional reinforcement learning. In Proceedings of the 35th International Conference Machine Learning, volume 80, pages 1096-1105, PMLR, 2018.
[15] E. Derman and S. Mannor. Distributional robustness and regularization in reinforcement learning. arXiv preprint, 2020. EXXi::2003.028..
[16] E. Derman, J. Mankowitz, T. A. Mann, and S. Mannor. Soft-robust actor-critic volicy-gradient arXiv preprint, 2018, 1rXiv:1803.00845
[17] J. Dhaene, A. Kukush, D. Linders, and Q. Tang. Remarks on quantiles and distortion risk measures. European Actuarial Journal, 2:319-328, 2012. doi: 10.1007/11385-0120055--0
[18] G. Dulac-Amold, N. Levine, D. J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester. An empirical investigation of the challenges of real-world reinforcement learning, arXiv preprint, 2020. arXiv:2003.1 1881.
[19] G. Dulac-Amold, N. Levine, D. J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester. Challenges real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning, 10:2441-2446 2021. doi: 10. 10.10077510994--11--5966-44

---

[20] M. W. Hoffman, B. Shahriari, J. Aslanides, G. Barth-Maroo N. Momchev, D. Sinopalnikov, Stanczyk, Ramos, A. Raichuk, D. Vincent, L. Hussenot, Dadashi, G. Dulac-Amold M. Orsini, A. Jacq, J. Ferret, N. Vieillard, S. K. S. Ghasemipour, S. Girgin, O. Pietquin, Behbahani, T. Norman, A. Abdolmaleki, A. Cassirer, Yang, K. Baumli, S. Henderson, A. Friesen, R. Haroun, A. Novikov, S. G. Colmenarejo, S. Cabi, C. Gulcehre, T. L. Paine,
S. Srinivasan, A. Cowie, Z. Wang, B. Piot, and N. de Freitas. Acme: A research framework for distributed reinforcement learning. arXiv preprint, 2020. arXiv:2006.00979.
[21] G. N. Iyengar, Robust dynamic programming. 257-280, 2005, doi: 10. 00.12877/mor..0400017.
Mathematics of Operations Research, 30(2):
[22] M. Janner, J. Fu, M. Zhang, and S. Levine. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[23] B. Jones and R. Zitikis. Empirical estimation of risk measures and related quantities. North American Actuarial Journal, 7(4):44-54, 2003, doi: 10. 10.1080/002202..205..0550
[24] R. Keramati, C. Dann, A. Tamkin, and E. Brunskill. Being optimistic to be conservative: Quickly learning a CVaR policy. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 34, pages 4436-4443, 2020.
[25] Kurutach, I. Clavera, Duan, Tamar, and P. Abbeel. Model-ensemble trust-region policy optimization. In Sixth International Conference on Learning Representations, 2018.
[26] L.A. and M. C. Fu. Risk-sensitive reinforcement learning via policy gradient search. Foun- dations and Trends in Machine Learning, 15(5):537-693, 2022. ISSN 1935-8237, doi: 10. 00.561/2200000091.
[27] Y. Liu, J. Ding, and X. Liu. IPO: Interior-point policy optimization under constraints. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4940-4947, 2020.
[28] Z. Liu, Z. Cen, V. Isenbaev, W. Liu, S. Wu, B. Li, and D. Zhao. Constrained variational policy optimization for safe reinforcement learning. In Proceedings of the 39th International Conference on Muchine Learning, pages 13644-13668. PMLR, 2022.
[29] X. Ma. L. Xia, Z. Zhou, J. Yang. and Q. Zhao. DSAC: Distributional soft actor critic for risk-sensitive reinforcement learning. arXiv preprint, 2020. 145447.
[30] A. Majumdar and M. Pavone. How should robot assess risk? Towards an axiomatic theory of risk robotics. In Robotics Research, pages 75-84. Springer International Publishing, 2020. ISBN 978-3-030-28619--1
[31] D. Mankowitz, N. Levine, R. Jeong, A. Abdolmaleki, J. T. Springenberg, Y. Shi, Kay, T. Hester, T. Mann, and M. Riedmiller. Robust reinforcement learning for continuous control with model misspecification. Eighth International Conference on Learning Representations, 2020.
[32] D. J. Mankowitz, D. A. Calian, R. Jeong, C. Paduraru, N. Heess, Dathathri, Riedmiller, and T. Mann. Robust constrained reinforcement learning for continuous control with model misspecification. arXiv preprint, 2021. arXiv:2010.106: 10644.
[33] S. Mowlavi, M. Benosman, and S. Nabi. Reinforcement learning state estimation for high- dimensional nonlinear systems. In Tenth International Conference on Learning Representations, 2022.
[34] A. Nilim and I.. F. Ghaoui. Robust control of Markov decision processes with uncertain transition matrices. Operations Research, 53(5):780-798) 2005. doi: 10. 10.1287/opre.10500 1650.02.6.
[35] I. Oshand, C. Blundell, A. Pritzel, and B. Van Roy. Deep exploration via bootstrapped DQN. In Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
12

---

[36] I. Osband, J. Aslanides, and A. Cassirer. Randomized prior functions for deep reinforcement learning. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[37] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3803-3810, 2018. doi: 10.1109/ICRA.2218.8.888 84660528..
[38] L.. Pinto, J. Davidson, R. Sukthankar, and Gupta. Robust adversarial reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 2817-2826. PMLR, 2017.
[39] A. Rajeswaran, Ghotra, Ravindran, and S. Levine, EPOpt: policies using model ensembles. In 5th International Conference on Learning Representations, 2017.
Learning robust neural network
[40] A. Rajeswaran, I. Mordatch, and V. Kumar. A game theoretic framework for model based reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, volume I 119, pages 7953-7963, PMLR, 2020.
[41] A. Ray, J. Achiam, and D. Amodei. Benchmarking safe exploration in deep reinforcement learning, 2019.
[42] A. Shapiro, D. Dentcheva, and A. Ruszczyúski. Lectures on Stochastic Programming: Modeling and Theory, Second Edition. Society for Industrial and Applied Mathematics, 2014. ISBN 1611973422.
[43] Y. Shen, M. J. Tobia, T. Sommer, and K. Obermayer. Risk-sensitive reinforcement learning. Neural Computation,
1291-1328888 2014. doi: 10. 1221166/EOO_a000600
[44] A. Stooke, J. Achiam, and P. Abbeel. Responsive safety in reinforcement learning by PID Lagrangian methods. In Proceedings of the 37th International Conference on Machine Learning. volume 119, pages 9133-9143. PMLR, 2020.
[45] A. Tamar, Y. Chow, M. Ghavamzadeh, and S. Mannor. Policy gradient for coherent risk measures. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
[46] C. Tessler, Y. Efroni, and S. Mannor. Action robust reinforcement learning and applications in continuous control. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pages 6215-6224. PMLR, 2019.
[47] C. Tessler, D. J. Mankowitz, and S. Mannor. Reward constrained policy optimization. In Seventh International Conference Learning Representations, 2019.
[48] J. Tobin, R. Fong, A. Ray, Schneider, W. Zaremba, and Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 23-30, 2017. doi: 10.1 100.0//0..000..200000
[49] E. Vinitsky, Y. Du, K. Parvate, K. Jang, P. Abbeel, and A. Bayen. Robust reinforcement learning using adversarial populations. arXiv preprint, 2020. arXiv:2008.018 00125.....
[50] S. Wang. Premium calculation by transforming the layer premium density. ASTIN Bulletin, 26 (1):71-92, 1996. doi: 10.2 10.2143/AST.226..53322.
[51] S. S. Wang. A class of distortion operators for pricing financial and insurance risks. The Journal of Risk and Insurance, 67(1):15-36, 2000. ISSN 00224367, 15396975, doi: 10.2307/253675.
[52] J. L. Wirch and M. R. Hardy. Distortion risk measures: Coherence and stochastic dominance. Insurance Mathematics and Economics, 32:168-166 168, 2003.
[53] H. Xu and S. Mannor. Distributionally robust Markov decision processes. In Advances in Neural Information Processing Systems, volume 23. Curran Associates, Inc., 2010.
13

---

[54] J. Xu, S. Kim, T. Chen, A. Rodriguez, P. Agrawal, W. Matusik, and S. Sueda. Efficient tactile simulation with differentiability for robotic manipulation. In The Conference on Robot Learning (CoRI.), 2022.
[55] T. Xu, Y. Liang, and G. Lan. CRPO: A new approach for safe reinforcement learning with convergence guarantee. In Proceedings of the 38th International Conference on Machine Learning, pages 11480-11491, PMLR, 2021.
[56] P. Yu and H. Xu. Distributionally robust counterpart in Markov decision processes. IEEE Transactions on Automatic Control, 61(9):538-2544 2016. doi: 10. 10.1009/AAC2205522295144
14

---

A Properties of coherent distortion risk measures
Majumdar and Pavone 30 proposed a set of six axioms to characterize desirable properties of risk measures in the context of robotics.
Al. Monotonicity: Z, Z' and I almost everywhere, then p(z) S p(z)).
A2. Translation invariance:
and Z 2, then p + = = +
A3. Positive homogeneity: If 2 and Z E Z, then p(r2) =Tp(Z).
^) S Ap( + (1 ())(() A5. Comonotonic additivity: If Z, Z' Z are comonotonic, then p Z') p(2) + p(Z)).
A4. Convexity: If A [0, 1] and Z, Z' E 2, then p(AZ
A6. Law invariance: If Z, Z' E 2 are identically distributed, then pl 7) p(Z))
See Majumdar and Pavone [3] for a discussion on the intuition behind these axioms. Note that coher- ent risk measures 6 satisfy Axioms A4 distortion risk measures [77 [5] satisfy Axioms Al A3
and Axioms A6 and coherent distortion risk measures satisfy all six axioms.
The properties of coherent risk measures also lead to useful dual representation.
Lemma (Shapiro et al. 422 Let be a proper, real-valued coherent risk measure. Then, for any Z E Z we have that
p(Z)
sup
BB [Z],
where Ep,, [.] represents expectation with respect to the probability measure Ba.a E P(M). and
Us.a S {Ba,a P(M) Ba.a Exaals,a Ex,a
}*}
is a convex, bounded, and weakly* closed set that depends on
See Shapiro et al. 422 for general treatment of this result.
Proofs
In this section, we prove all results related to the RAMU cost Bellman operator T Using the fact that p+ (Z) = -(- for coherent distortion risk measure p on cost random variable, all results related to the RAMU reward Bellman operator follow by an appropriate change in sign.
B.1 Proof of Lemma I
Proof. Starting from the definition of Tp. in Definition we have that
(B
Th (T.Q(s,) = a) +?
E [')s
[Q(s',
La La'no(--k)))
c(s +
E 1Q(s', a')]
s'oppai La'nNT(-1o")
P Pooa
((s,,)++
+7 R
E (a'NT(-1(s))
[Q('',
d'appone
P PoparMosa
T ((s,a))
which proves the result. Note that the second equality follows from the positive homogeneity of (Axiom A3), the third equality follows from the translation invariance of p (Axiom A2), and the fourth equality follows from the definition of the standard cost Bellman operator Tp...
B.2 Proof of Theorem 1
Proof. First, we show that Txx equivalent distributionally robust Bellman operator. For given bate-action pair, apply Lemma to the risk measure that appears in the formulation of Tx
15

---

given by Lemmal 1 By doing so, we have that To (2,Q(s,)))
Praapea (T% ((s,a))))
sup
E
IT ((,)())))))))
El Paa 3aa
as, +7 Bal Pa.ar-Baia
sup
E
(Q(s', a d'rops.a (a'mx()1e")
where as is defined Lemma 3 Therefore, Tp.c has the same form as distributionally robust
Bellman operator [53 56 with the ambiguity set (s.a)ESxA4 The RAMU cost Q function Qp a) the fixed point of Tp.c so is equivalent to a distributionally robust Q function with ambiguity set U. Using the rectangularity of U, we can write this succinctly as
(p.(ss a)
sup E BEMl p~B [9%.o(s.
Then, using the definition of (T))(T) we have that
(T) = E
E Lawi(--as) (-1a)
[Q%c(saa]]
ando
sup I BEll pag
[Q [8)
sndo
(1a)()
[0 (8,)(
sup E Bell prp Lawda
Lawa((1s)
sup E [T), seul pru
where we can move the optimization over l outside of the expectation operators due to rectangularity.
We can use similar techniques to show that pt., has the same form as distributionally robust Bellman operator with the ambiguity set U (x,,))SXXAAA U and
Jp (
inf [x), seu pr13
Therefore, have that the RAMU safe RL problem in
3 is equivalent to (
B.3 Proof of Corollary 1
Given the equivalence of T and Tp to distributionally robust Bellman operators as shown in
Theorem [ Corollary follows from results in Xu and Mannor [53 and Yu and Xu [56). We include proof for completeness.
Proof. Due the linearity of the expectation operator, for a given Bs.a E Us.a we have that
E
E La (amoT(-1s')
(0(s','')]
E
E [Q(s'
PrrariB Le'ropora
('
where Ph, = Epscad NBn [Ppaa P(S) represents mixture transition model determined by Ba.a. Therefore, starting from the result in Theorem we can write
Te. ((s,a) +
sup E Pra s'mups,a
E
E
E La (a'=x)--1e))
(((s',d'))
= =((s,a) t
E PheEP simph
sup
1Q(s's
(a'nz(-(er)
where
Paa
pe P(S) 1
[Poaa: Baa Usaa
Po Baia
16

---

Carppole Swingup
Quadruped
Quadruppd
1030
750
600
800
500
400
800
Tolal
250
200
800
Stops (M)
Stops (M)
Steps (M]
Steps
Stops
Unconstrained
Safa
Sufe
Figure Hyperparameter sweep of safety coefficient. Value in parentheses represents safety coefficient used for training in safe RL. Shading denotes half of one standard error across policies.
As a result, The has the same form as a robust Bellman operator 21 p (2,0)E8xA Pa.a.
with the uncertainty set
Consider Q functions Q(1) and Q(2), and denote the sup-norm by
I1Q(2) 11
(Q (s,
sup (a,0)ESXA
Fix 6 > O and consider (8,a) a) E x A. Then, there exists Psça E Pa.a such that
[0()((
-c. [01]
E
E (
sup P..a
E
E La c')o7)(-(o))
s'mppen (1)
La
s'opph
We have that
Te,
(T
( (8)) (s' a d'il))
sup
E
E La ('NoT(-1o')
sup
E
E
Ph EP.. s'mpha
pe PP s'mpha La'nosl 20)
(Q")(s'
(Q13)(6
S
E 8' s'ruph La'mot(-(a"")
+
E Lalnat(-lo)
E
d'mppll
(Q1)(s
-
tyc
5² s
< >11 (22))
llo0 7E.
similar argument can be used to show that
-2 - Q(2) loc - -C (TTT000))ss - -Tx
so we have that
Te. (s T a) s 11) 1(2) 11o + ye.
By applying a supremum over state-actin acteonction pairs on the left-hand side, we obtain
IT, - (2) S (IQ - (12) + yc.
Finally, since e was arbitrary, we have shown that Tht is c-contrcctoon in the sup-norm.
C
Implementation details and additional experimental results
Safety constraints and environment perturbations In all of our experiments, we consider the problem of optimizing a task objective while satisfying a safety constraint. We focus on a single safety constraint corresponding to cost function defined in the Real-World RL Suite for each task,
17

---

Table 2: Safety constraints for all tasks
Safety
Task
Safety Constraint
Coefficient
Cartpole Swingup
Slider Position
0.30
Walker Walk
Joint Velocity
0.25
Walker Run
Joint Velocity
0.30
Quadruped Walk
Joint Angle
0.15
Quadruped Run
Joint Angle
0.30
Table 3: Perturbation ranges for test environments across domains
Perturbation Parameter
Nominal Value
Domain
Test Range
Cartpole
Pole Length
1.00
[0.75, 1.25]
Walker
Torso Length
0.30
[0.10, 0.50]
Quadruped
Torso Density
1,000
5000, 1,500]
and we consider a safety budget B = 100. The safety constraints used for each task are described in Table In the Cartpole domain, costs are applied when the slider is outside of a specified range. In the Walker domain, costs are applied for large joint velocities. In the Quadruped domain, costs applied for large joint angles. See Dulac-Amold et al. [19] for detailed definitions of each safety constraint.
The definitions of these cost functions depend on safety coefficient in [0, 1J. which determines the range of outcomes that lead to constraint violations and therefore controls how difficult will to satisfy safety constraints corresponding to these cost functions. As the safety coefficient decreases, the range safe outcomes also decreases and the safety constraint becomes more difficult to satisfy. In order consider safe RL tasks with difficult safety constraints where strong performance is still possible, we selected the value of this safety constraint in the range of [0.15, 0.20, 0.25, 0.30] for each task based on the performance of the baseline safe RL algorithm CRPO compared to the unconstrained algorithm MPO. Figure3showw shows total rewards throughout training for each task across this range of safety coefficients. We selected the most difficult cost definition in this range (i.e., lowest safety coefficient value) where CRPO is still able to achieve the same total rewards as MPO (or the value that leads the smallest gap between the two in the case of Walker Run and Quadruped Run). The resulting safety coefficients used for our experiments are listed in Table 2
In order to evaluate the robustness of our learned policies, generate range of test environments for each task based on perturbing a simulator parameter in the KorldWorld RL Suite. See Table 3 for the perturbation parameters and corresponding ranges considered in our experiments. The test range for each domain is centered around the nominal parameter value that defines the single training environment used for all experiments except domain randomization. See Figure 4 for detailed results of the risk-averse and nest-ll versions of our RAMU framework across all tasks and environment perturbations.
Domain randonization Domain randomization requires a training distribution range of environments, which is typically defined by considering a range of simulator parameters. For the in-distribution version of domain randomization considered in our experiments, we apply uniform distribution over a subset of the test environments defined in Table In particular, we consider the middle 50% of test environment parameter values centered around the nominal environment value for training. In the out-of- -iitt---dittibttionn version of domain randomization, on the other hand, we consider different perturbation parameter from the one varied at test time. We apply uniform distribution over range of values for this alternate parameter centered around the value in the nominal environment. Therefore, the only environment shared between the set of test environments and the set of training environments used for out-of-distribution domain randomization is the nominal environment. See Table for details on the parameters and corresponding ranges used for training in domain randomization.
18

---

Carpole Swingup
Quadhupod
Guadhupod
Tota Bexeert
1030
600
1000
800
800
050
630
400
700
400
803
200
230
200
200
203
Toxe Coet
wength
Length
Langth
Derettyy
RAMU (Vaag
RAMU (Expcctaiio)
Figure 4: Comparison with standard safe RL across tasks and test environments. RAMU algorithms use the risk measure in parentheses applied to both the objective and constraint. Shading denotes half of one standard error across policies. Vertical dotted lines represent nominal training environment. Top: Total reward. Bottom: Total cost, where horizontal dotted lines represent safety budget.
Table Perturbation parameters and ranges for domain randomization across domains
Perturbation Parameter
Nominal Value
Training Range
Domain
In-Distribution
Cartpole
Pole Length 1.00
[0.875, 1.125]
Walker
Torso Length
0.30
[0.20, 0.40)
Quadruped
Torso Density
1,000
[750, 1,250]
Out-of- Distsiiiiiutto
Cartpole
Pole Mass 0.10
0.055, 0.15
Walker
Contact Friction
0.70
0.40, 1.00
Quadruped
Contact Friction
1.50
[1.00,
We include the results for domain randomization across all tasks and environment perturbations in Figure Across all tasks, we observe that our RAMU framework leads to similar or improved constraint satisfaction compared to in-distribution domain randomization, while only using one training environment. In addition, our framework consistently outperforms out-of-distribution domain randomization, which provides little benefit compared to standard safe RL due to its misspecified training distribution.
Adversarial reinforcement learning In order to implement the action-robust PR-MDP framework, we must train an adversarial policy. We represent the adversarial policy using the same structure and neural network architecture our main policy, and we train the adversarial policy to maximize total costs using MPO. Using the default setting in Tessler et [46], we apply one adversary update for every 10 policy updates.
We include the results for adversarial RL across all tasks and environment perturbations in Figure O where adversarial RL is evaluated without adversarial interventions. We see that adversarial RL leads robust safety in some cases, such as the two Quadruped tasks. However, other tasks such as Cartpole Swingup, safety constraint satisfaction is not as robust. Safety also comes at the cost of conservative performance in some tasks, as evidenced by the total rewards achieved by adversarial RL in Walker Run and Quadruped Run. Overall, our RAMU framework achieves similar performance to adversarial RL, without the drawbacks associated with adversarial methods that preclude their use in some real-world settings.
19

---

Carpole Swingup
Quadhupod
Guadhuped
1030
1000
Tota Boooent
ocu
800
050
700
430
803
200
230
200
200
203
Tone Coet
wingth
Length
Langth
Dersily
Deratly
RAMU
Domain Rand
(000)
Figure 5: Comparison with domain randomization across tasks and test environments. Grey shaded area denotes the training range for in-distribution domain randomization. Shading denotes half of one standard error across policies. Vertical dotted lines represent nominal training environment. Top: Total reward. Bottom: Total cost, where horizontal dotted lines represent safety budget.
Carbolee Swingup
Quadtuped
Quadhuped
Todel Reword
800
1000
800
8900
083
200
803
200
200
200
200
203
Toael Coet
weggh
length
Langth
Dersrly
Denatly
(Whwng
Figure Comparison with adversarial RL across tasks and test environments. Performance of adversarial RL is evaluated without adversarial interventions. Shading denotes half of one standard еrror across policies. Vertical dotted lines represent nominal training environment. Top: Total reward. Bottom: Total cost, where horizontal dotted lines represent safety budget.
Network architectures and algorithm hyperparameters In our experiments, we consider neural network representations of the policy and critics. Each of these neural networks contains 3 hidden layers of 256 units with ELU activations. In addition, we apply layer normalization followed by tanh activation after the first layer of these networks as proposed in Abdolmaleki et al. 2 We consider a multivariate Gaussian policy, where at given state we have xa s) = (N((s)) E(s) where uu( and E(s) represent outputs of the policy network. Σ(s) is diagonal covariance matrix, whose diagonal elements are calculated by applying the softplus operator to the outputs of the neural network. We parameterize the reward and cost critics with separate neural networks. In addition, consider target networks that are updated as an exponential moving average with parameter 7 = 5e-3.
We consider CRPO [55] the baseline safe RL algorithm in all of our experiments, which immedi- ately switches between maximizing rewards and minimizing costs at every update based on the value of the safety constraint. If the sample-average estimate of the safety constraint for the current batch of data satisfies the safety budget, we update the policy to maximize rewards. Otherwise, we update the policy to minimize costs.
After CRPO determines the appropriate objective for the current batch of data, we apply MPO 1 to calculate policy updates. MPO calculates a non- can-meraettric policy update based on the KL divergence parameter éxI. and then takes step towards this non-parametrie policy while constraining the KI. divergence from updating the mean by 8, and the KL divergence from updating the covariance matrix
20

---

Table 5; Network architectures and algorithm hyperparameters used in experiments
General
Batch size per update
256
Updates per environment step
1
Discount rate (?)
0,99
Target network exponential moving average (r)
5e-3
Policy
Layer sizes
256, 256, 256
Layer activations
ELU
Layer norm + tanh on first layer
Yes
Initial standard deviation
0.3
Learning rate
le-4
Non-parametric KL (EKL.)
0.10
Action penalty KL
le-3
Action samples per update Parametric mean KL (B₂)
20
0.01
Parametric covariance KL (3g)
le-5
Parametric KI. dual learning rate
0.01
Critics
Layer sizes
256, 256, 256
Layer activations
ELU
Layer norm + tanh first layer Learning rate
Yes
le-4
RAMU
Transition model samples per data point (n)
5 0.10
Latent variable hyperparameter (e)
by Sg. We consider per-dimension KL divergence constraints by dividing these parameter values by the number of action dimensions, and we penalize actions outside of the feasible action limits using the multi-objective MPO framework [2] suggested in Hoffman et 20] In order avoid potential issues related to the immediate switching between reward and cost objectives throughout training, we completely solve for the temperature parameter of the non-pan parameeett target policy in MPO at every update as done in Liu et al. (28). See Table 5 for the default hyperparameter values used our experiments, which are based on default values considered in Hoffman al. (00
For our RAMU framework, the latent variable hyperparameter controls the definition of the distribution Ps,a over transition models. 7 shows the performance of our RAMU framework in Walker Run and Quadruped Run for e [0.05, 0.10, 0.15, 0.20). A larger value of leads to distribution over wider range of transition models, which results in more robust approach when combined with a risk-averse perspective on model uncertainty. We see in Figure 7 that our algorithm more robustly satisfies safety constraints as increases, but this robustness also leads to a decrease in total rewards. We consider = 0.10 in our experiments, as it achieves strong constraint satisfaction without meaningful decrease in rewards. Finally, for computational efficiency we consider samples of transition models per data point to calculate sanple-based Bellman targets in our RAMU framework, as we did not observe meaningful improvements performance from considering larger number of samples.
Computational resources All experiments were run on a Linux cluster with 2.9 GHz Intel Gold processors and NVIDIA A40 and 100 GPUs. The Real-World RI Suite is available under the Apache License 2.0. We trained policies for I million steps across 5 random seeds, which required
21

---

Walker Run
Quadruped Run
Total Reward
850
200
200
Total Cost
0.1
0.2
0.3
0.4
0.5
600
800
1000
1200
1400
Length
Densily
0.05)
(c 0.10)
0.15)
0.201
Figure 7: Hyperparameter sweep of latent variable hyperparameter e on Walker Run and Quadruped Run. RAMU algorithms use the Wang transform with 7 0.75 applied to both the objective and constraint. Shading denotes half of one standard error across policies. Vertical dotted lines represent nominal training environment. Top: Total reward. Bottom: Total cost, where horizontal dotted lines represent safety budget.
approximately one day wall-clock time on single GPU for each combination of algorithm and task using code that has not been optimized for execution speed.
22


Reviews:
[{"review_id": "dUCjd4SKpC", "reviewer": "Reviewer_F3rP", "summary": "The paper introduces the Risk-Averse Model Uncertainty (RAMU) framework for safe reinforcement learning in uncertain environments. RAMU incorporates a distribution of transition models and applies a risk-averse perspective using coherent distortion risk measures. The framework offers an efficient, model-free implementation through one single training environment. Experimental results demonstrate the framework's ability to produce robust, safe performance in perturbed test environments. Unlike existing distributional robust (DR) approaches, RAMU eliminates the need for minimax optimization. ", "questions": "- I understand that the contribution of the paper lies in proposing a distributionally robust (DR) safe reinforcement learning method that does not require a minmax operation. However, given that there is no direct connection between DR algorithm design and safety, it is essential to compare it with existing DR methods such as [1] and [2].\n- The proposed algorithms does have similarities to policy smoothing algorithms, as mentioned in [3] and [4]. It is plausible to consider that combining policy smoothing with adversarial RL could potentially yield better results compared to the RAMU framework. In other words, exploring the combination of observational robustness and action robustness may provide improvements over predefined model robustness?\n\nRef.\n\n[1] Robust Reinforcement Learning using Offline Data\n\n[2] Distributionally Robust Q-Learning\n\n[3] Deep Reinforcement Learning with Robust and Smooth Policy\n\n[4] Policy Smoothing for Provably Robust reinforcement learning.", "limitations": "As mentioned in the Conclusion section of the paper, the choice of the model distribution µ and risk measure ρ in the RAMU framework is user-defined, and RAMU framework only addresses robustness with respect to model uncertainty and safety defined by expected total cost constraints. Crruently, this paper only consider the dynamics of model transition follow linear patterns, which may not to be suitable for tasks that involve image inputs. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposed perturbation function models the distribution of environment transition models, providing a foundation for practical implementation of distributionally robust algorithms.\n-  Compared to existing methods, the RAMU framework is implemented efficiently. It avoids complex minimax optimization, which is the major obstacle to applying Distributional Robustness methods to DRL.\n", "weaknesses": "- The problem addressed in this paper is the handling of model uncertainty in safe reinforcement learning (RL) scenarios. However, it appears that the issue of model uncertainty in safe RL is similar to the problem in standard RL. Since the proposed RAMU method is not specifically designed for safe RL, it would be more convincing if it were compared to existing distributionally robust methods such as those discussed in [1] and [2].\n- The formulation of the distributionally robust safety problem in the article lacks clarity. The problem definition starts with directly modifying the Q function to the DR Q function in Eq4, which is a shortcut approach. It would have been more appropriate to first define the problem and then derive the suitable form of the Q function. Although solving the Eq4 definition is straightforward by plugging in the safety RL method with DR RL, it is important to note that the worst-case transitions associated with the reward and cost (the $\\beta$ in Eq5 and Eq6) are not the same. However, in reality, there is only one transition, which makes the proposed method more conservative in practice.\n-  The transition function f(s, s') mentioned in line 257 may only be effective for tasks involving robot control, where dynamics follow linear patterns and such perturbations are effective. For tasks involving image inputs, it remains unclear what kind of function f should be used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. Please see below for clarifications and responses to your questions. In particular, we clarify the definition of “distributionally robust RL” [a, b] considered in our theoretical results, and how this differs from other definitions that have appeared in the literature more recently. We hope that our responses address your main concerns, and we ask that you please consider updating your review scores to reflect these clarifications.\n\n### [W1, Q1, Q2] Distributionally robust RL and comparison to other methods\n\n* **Definition of distributionally robust RL:** We consider a distribution $\\mu$ over transition models in our work, and show in Theorem 1 that our RAMU framework is equivalent to a class of distributionally robust RL problems as defined by [a, b]. Distributionally robust RL [a, b] considers ambiguity sets of *distributions over transition models* in $P(\\mathcal{M})$ (see lines 79-81), and is different from robust RL [c, d] that applies uncertainty sets directly over transition models in $\\mathcal{M}$.\n* **[W1, Q1] Comparison to [1] and [2]:** Please note that both [1] and [2] consider the typical robust RL setting [c, d] based on uncertainty sets of transition models. This is different from our approach, which considers a distribution over transition models and is equivalent to a class of distributionally robust RL problems as defined by [a, b]. In recent years, researchers have started using “robust” and “distributionally robust” interchangeably to mean robust RL [c, d], which has caused some confusion. Also different from our approach, [1] considers the offline RL setting and [2] focuses on the tabular RL case. \n* **[Q2] Comparison to [3] and [4]:** We agree that the idea of observational robustness considered in [3] and [4] is an important area of research, but it is not the focus of this work. We focus on being robust to uncertainty in the transition model (i.e., dynamics), and these two sources of uncertainty require different analysis and algorithms. Please note that [3] and [4] do not make any direct connections to the definition of distributionally robust RL [a, b] considered in this work. Action robustness is more closely related to uncertainty in dynamics because it leads to changes in state transitions, which has led to its use as a robust RL [c, d] method in deep RL settings.\n* **Experiment baselines:** We compare against the most popular methods for robustness to model uncertainty in deep RL. Adversarial RL represents a common implementation of robust RL [c, d] in the deep RL setting, so we use a popular action-robust adversarial RL method as a baseline in our experiments. Domain randomization is the most popular implementation based on distributions of transition models, so we also consider this as a baseline.\n\n### [W2] Problem formulation\n\n* We will update Section 4 to start from the formal problem definition that corresponds to the RAMU update in (4). The objective and constraint in this formulation are very similar to the corresponding Q functions defined in line 156, just averaged over initial states and actions. We will also make clear the distributionally robust safe RL problem definition to which our RAMU problem is equivalent, which involves ambiguity sets of distributions over transition models (see lines 79-81).\n* As you pointed out, the worst-case distributions $\\beta$ over transition models will be different for rewards and costs in our formulation. Because we do not know the true environment at test time, it makes sense to take this conservative approach in order to guarantee robustness in both the rewards and costs at deployment time. This is a common approach when considering robustness in a safe RL setting [e].\n\n### [W3] Extension to image inputs\n\nAs an example that makes sense for our experiments, we consider an intuitive perturbation function based on percentage changes in each dimension of state transitions. However, our methodology works with any choice of distribution $\\mu$ over transition models (or equivalently, any choice of perturbation function $f_x$), which is defined by the user to best suit the application. In RL from images, it is common to consider an MDP in a latent representation space, and we can apply our methodology in this latent space. Alternatively, in scenarios where detailed simulator access is available, it would also be possible to generate next state images from multiple transition models by leveraging this simulator.\n\n**References:**\n\n[a] H. Xu and S. Mannor. Distributionally robust Markov decision processes. In Advances in Neural Information Processing Systems, volume 23. Curran Associates, Inc., 2010.\n\n[b] P. Yu and H. Xu. Distributionally robust counterpart in Markov decision processes. IEEE Transactions on Automatic Control, 61(9):2538–2543, 2016.\n\n[c] G. N. Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2): 257–280, 2005.\n\n[d] A. Nilim and L. E. Ghaoui. Robust control of Markov decision processes with uncertain transition matrices. Operations Research, 53(5):780–798, 2005.\n\n[e] D. J. Mankowitz, D. A. Calian, R. Jeong, C. Paduraru, N. Heess, S. Dathathri, M. Riedmiller, and T. Mann. Robust constrained reinforcement learning for continuous control with model misspecification. arXiv preprint, 2021. arXiv:2010.10644."}, {"review_id": "xyTfe7SPSB", "reviewer": "Reviewer_ZX7b", "summary": "The paper introduces a deep reinforcement learning framework for safe decision-making in uncertain environments. The authors propose a risk-averse approach towards model uncertainty using coherent distortion risk measures. They provide robustness guarantees for the framework by showing its equivalence to a distributionally robust safe reinforcement learning problem. The framework is efficient and model-free, utilizing standard data collection from a single training environment. Experiments on continuous control tasks with safety constraints demonstrate the framework's robust and safe performance across a range of perturbed test environments.", "questions": "see above", "limitations": "see above", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[+] The paper is logically clear and well-written, making it easy to follow.\n\n[+] The paper starts from theory, first defining a new Q-function and a new Bellman operator, then proving the equivalence between the new Bellman operator and distributionally robust Bellman operators with respective ambiguity sets, thereby verifying the rationality of the definition and the contraction property of the new Bellman operator. This provides a theoretical basis for the algorithm design.\n\n[?] What is the relationship between safety and robustness? The paper theoretically proves that the newly proposed RAMU Bellman operator has robustness guarantees. However, it seems not to mention much about its relationship with Constrained Markov Decision Process (CMDP) and how to ensure that the point converged to by the RAMU Bellman operator is safe.\n\n[?] Related to the previous question, why consider both safety and robustness here? It seems that considering both safety and robustness is crucial for the Theorem 1. Is it necessary to clarify this point more explicitly in the abstract? For example, the current abstract's first sentence emphasizes the importance of safety, the line 5 separately emphasizes robustness guarantees, and the line 7 re-emphasizes robustness, which can be somewhat confusing.\n\n", "weaknesses": "see above", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. We are glad you found the paper to be clear and well-written, and appreciated the theoretical support for our proposed framework. Please see below for responses to your questions, which clarify the importance of both safety and robustness. If these clarifications address your concerns, we ask that you please consider updating your overall review score to reflect this.\n\n### [Q1, Q2] Importance of safety and robustness\n\n * Safety is often a prerequisite for real-world decision making applications, so we consider a safe RL setting with a Constrained MDP (CMDP) as our starting point. However, safe RL with a CMDP finds a policy that is only safe in a single training environment, with no robustness guarantees related to performance and safety in other environments. In many real-world scenarios, the environment at deployment time may be different from the training environment due to factors such as modeling errors or unknown disturbances.\n* We incorporate robustness to model uncertainty in both the objective (rewards) and safety constraint (costs) of a CMDP. We accomplish this by learning separate RAMU Q functions for the reward and cost, which both appear in our RAMU update in (4). As demonstrated in our experimental results, this update leads to policies that (i) achieve robust performance (due to the use of our RAMU reward Q function in the objective) and (ii) remain safe across a range of test environments (due to the use of our RAMU cost Q function in the safety constraint). Our RAMU framework significantly outperforms the standard safe RL baseline that applies the update in (2), which uses standard Q functions that only consider a single training environment $p$ and do not incorporate robustness."}, {"review_id": "c04ssl4idu", "reviewer": "Reviewer_mRCw", "summary": "This paper presents a Temporal Difference (TD) learning method for addressing the ``Risk-Averse Model Uncertainty for Distributionally Robust Safe Reinforcement Learning'' problem. Specifically, the authors consider a Constrained Markov Decision Process (CMDP) combined with Bayesian uncertainty sets. They are trying to attain a policy which is both robust and safe; Meaning that it optimizes a nested risk measure of discounted return, while satisfying certain guarantees over discounted cost. The proposed method is implemented and evaluated by comparing its performance against adversarial RL, Safe RL, and domain randomization approaches across multiple environments.", "questions": "- **The Objective**: It would be interesting to observe the performance of the proposed algorithm by varying the learning rates for $\\theta_r$ and $\\theta_c$ and exploring different risk measures for $\\rho^+$ and $\\rho$. For instance imagine we set $\\rho^+$ as $CVaR_{\\alpha_r}^+$, and $\\rho$ as $CVaR_{\\alpha_c}$. As $\\alpha_c$ decreases, the algorithm becomes more strict on violating safety conditions, and as $\\alpha_r$ decreases, the algorithm becomes more robust. I am curious to see the differences in experimental outcomes under such settings.\n- **Evaluation**: In the experiments, different algorithms are being compared based on the safety percentage, and total rewards. I would like to see their comparison based on (an estimated version of) the Lagrangian form of the actual objective (4) (while fixing the set of hyperparameters for all methods). Since the proposed algorithm directly aims to optimize this objective, it is reasonable to expect it to outperform other methods in terms of this specific evaluation metric. The conduct of such an evaluation can serve as a sanity check.", "limitations": "no limitations", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- This paper addresses a noteworthy problem. As a general matter, I believe that risk-aware policy selection methods for Bayesian MDP, as also studied in [1, 2, 3] is an interesting and open area of research.\n- Even though the problem is notation-heavy, the paper is well presented and maintains a high level of readability.\n- The authors have proven their implementation's effectiveness across five different tasks. Furthermore, the authors' utilization of a large number of training samples adds credibility to the algorithm's performance evaluation.\n\n\n[1] Giorgio Angelotti, Nicolas Drougard, & Caroline Ponzoni Carvalho Chanel. (2023). An Offline Risk-aware Policy Selection Method for Bayesian Markov Decision Processes. arXiv preprint arXiv:2105.13431.\n\n[2] Lobo, E. A., Ghavamzadeh, M., & Petrik, M. (2020). Soft-robust algorithms for batch reinforcement learning. arXiv preprint arXiv:2011.14495.\n\n[3] Petrik, M., & Russel, R. H. (2019). Beyond confidence regions: Tight Bayesian ambiguity sets for robust MDPs. Advances in neural information processing systems, 32.", "weaknesses": "- **Theoretical results**: The primary focus of this paper is the implementation and experimental evaluation; there are limited theoretical results in this paper.\n\n- **Comparisons**: In the ``strengths'' section of my review, I mentioned three risk-aware policy selection methods for Bayesian MDP methods that I am aware of [1, 2, 3]. These methods have similar objectives, albeit having two key differences (i) Instead of CMDPs the objective is for normal MDP and does not require safety guarantees (ii) the risk measure is (might be) applied over entire stochasticy $\\mu$ rather than recursively over each $\\mu_{s, a}$. As both algorithms use samples of models to have a Monto Carlo estimation of the risk measure, the $\\theta$ update in Algorithm 1 is particularly similar to the $\\operatorname{RiskEvaluation}$ operation introduced by [1].\n\n    - Firstly, I would suggest that the authors include these papers in the ``uncertainty in reinforcement learning'' part of the related works.\n\n    -  Secondly, it would have been more informative if the paper compared its proposed method to one of these existing methods, instead of focusing on comparisons with adversarial RL and domain randomization.\n\n- **Presentation**:  Theorem 1 requires a more concrete and explicit formulation. $\\zeta_{a, s}$ is introduced as ``depending on $\\rho^+$ and $\\rho$''; without properly defining the dual presentation of coherent risk measures beforehand. This lack of clarity makes it very challenging for non-expert readers to understand the paper.\n\n\nIn conclusion, while the paper proposes an interesting problem and provides valuable experimental results, there is a lot of room for improvement. Thus, I rate this paper as a borderline accept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. We are glad you agree this is an important problem, and we appreciate your comments on our paper’s clear presentation and strong experimental results. Please see below for responses to your questions. We hope that these clarifications address your main concerns. If so, we ask that you please consider updating your review scores to reflect our responses.\n\n### [W1, W3] Theoretical results / presentation\n\n* **[W1] Theoretical results:** Our RAMU framework represents a novel contribution to the RL literature, and we provide important theoretical results for the corresponding RAMU Bellman operators and Q functions that support the use of this framework. We provide theoretical connections to distributionally robust RL (Theorem 1), and we prove contraction properties for the RAMU Bellman operators (Corollary 2) that provide theoretical support for training the RAMU Q functions via standard temporal difference methods. It was not trivial to construct a novel RL framework for addressing model uncertainty with these theoretical properties and an efficient implementation.\n* **[W3] Presentation:** Due to space constraints, we present Theorem 1 with the necessary details to provide an intuitive understanding of the result, and we defer the full formal treatment of the result to the Appendix. Appendix B.2 formally defines the appropriate probability space and provides the dual representation result that we use (including reference to the appropriate dual space). We will update Theorem 1 in the main text for additional clarity.\n\n### [W2] Comparison of RAMU to other methods\n\n* **Experiment baselines:** In our experiments, we compare against the most popular methods for robustness to model uncertainty in deep RL. Adversarial RL represents a common implementation of robust RL in the deep RL setting, so we use a popular action-robust adversarial RL method as a baseline in our experiments. Domain randomization is the most popular implementation based on distributions of transition models, so we also consider this as a baseline.\n* **Comparison to [1, 2, 3]:** We agree that [1, 2, 3] are interesting works that consider model uncertainty in RL. However, they are not well suited as baselines in the deep RL setting that is the focus of this paper. The methods in [1, 2, 3] have several key differences compared to the setting that we consider in this paper. Most importantly, they are offline RL methods (vs. online RL in this paper) that are designed for tabular settings (vs. deep RL in this paper). Please also note that we do not focus on a Bayesian setting in this work (however, by choosing $\\mu$ to represent a posterior, our framework can be applied in a Bayesian setting).\n\n### [Q1, Q2] Experimental analysis\n\n* **[Q1]:** The comparison between RAMU (Wang 0.75) and RAMU (Expectation) represents an example of the analysis you suggest, as the expectation operator is equivalent to setting the Wang hyperparameter to zero. An alternative way to change the level of robustness is to vary the hyperparameter $\\epsilon$ that defines our distribution $\\mu$. We include these results in Appendix C (Figure 6), which demonstrates the trends you have mentioned. We expect that varying a risk measure hyperparameter would lead to similar trends (see comparison between RAMU with Wang 0.75 vs. Expectation).\n* **[Q2]:** The goal of safe RL is to maximize rewards while satisfying the safety constraint. Therefore, we directly measure these two key quantities as our metrics of interest in our experiments. We have also included the total rewards and total costs for every algorithm and test environment in Appendix C (Figures 3, 4, 5). Note that a Lagrangian relaxation would be one way to solve the safe RL problem, but we do not consider this approach in our experiments (we use CRPO; see lines 654-658 for details).\n"}, {"review_id": "cB7KFERkrE", "reviewer": "Reviewer_22QQ", "summary": "This paper proposes a methodology for distributionally robust RL via the use of risk measures and leveraging risk (Fenchel) duality, dealing with what they call model uncertainty. The paper introduces the RAMU Q function and Bellman operators respectively, which are based on modifying the standard risk-neutral operators by inducing risk-awareness over the choice of the transition model. The RAMU Q function and respective Bellman operators exhibit nice properties, due to the class of risk measures considered (distortion risk measures).\n\nThen, the paper describes a model-free implementation of the proposed approach with a single training environment, leveraging interesting results for statistical estimation of risk functionals (specifically distortion risk measures). Lastly, the proposed approach is verified on a number of numerical benchmarks.", "questions": "What is a \"distribution over transition models\" in lines 41-42? This is very ambiguous at this point. \n\nAlso, related in Lines 72-74: Defining this product is somewhat obscure. For instance, does this make sense when S and A are uncountable sets (or you implicitly assume finite states and actions throughout)?", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Overall, I think this is a paper with potential, and it was a pleasure reading it. The RAMU framework is indeed interesting and seems to be effective, although I have some concerns (see below).\n\nThe estimation of the Bellman operators in Section 6 within the RAMU framework can indeed be computed efficiently, as the authors point out, which is an advantage of the approach. The application of the results of Jones and Zitikis is quite interesting.\n\nThe experimental section is quite detailed and the comparisons with existing methods is well-thought, especially with a large number of similar alternative approaches. Also the experiments bring out the effect and efficacy of risk-averse RL well.\n\n", "weaknesses": "Risk duality and equivalence to minimax (distributionally robust) optimization (under certain assumptions) is very well known and established (in essence it is just Fenchel duality). Therefore, I do not think it can be claimed as novelty or contribution of this paper by itself (e.g., in lines 5 to 7 in the abstract). I would suggest that the authors rephrase to highlight more specifically their contributions.\n\nIn the discussion about risk measures, I do not see any explanation on *why* one should choose to work with distortion risk measures or coherent risk measures, and why the axioms proposed by Majumdar and Pavone are appropriate. Axioms are subjective, so in principle there is no universal reason to use one set of axioms over another set of axioms. Of course distortion risk measures exhibit amazing analytical properties and probably this is among the main reasons justifying the authors chosing this class, but the discussion and point needs to be made.\n\nThe RAMU cost function as defined in line 156 is introduced somewhat arbitrarily, in the sense that it is unclear if this is somewhat related to a base problem such as (2). While I understand the motivation and indeed it *seems* to make sense, usually one would start from such a base problem, possibly risk-averse in a certain way that makes intuitive and operational sense, and then build their way towards Q factors and Bellman optimality conditions.\n\nTheorem 1 follows directly from the application of risk duality. I am not sure if this result \"deserves\" to be a theorem. I would think that a stating as a proposition would be more suiting. As a general comment, the theory in this paper is somewhat limited, however I have enjoyed reading the paper still.\n\nThe discussion under \"Generative distribution of transition models\", line 236 onwards is quite convoluted and involved without particular reason in my opinion. While I finally understood what happens (I think!), I believe that the authors should make an effort to write this part in a much clearer manner.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. We are glad that you enjoyed reading it, and found our RAMU framework to be interesting and effective. We also appreciate your thoughtful suggestions, which will help to improve our paper. Please see below for responses to all of your questions and comments, which we hope address your main concerns. If so, we ask that you please consider updating your review scores to reflect our responses.\n\n### [W1, W4] Theoretical contributions / novelty\n\n* Our RAMU framework represents a novel contribution to the RL literature, and we provide important theoretical results for the corresponding RAMU Bellman operators and Q functions that support the use of this framework. We provide theoretical connections to distributionally robust RL (Theorem 1), and we prove contraction properties for the RAMU Bellman operators (Corollary 2) that provide theoretical support for training the RAMU Q functions via standard temporal difference methods. It was not trivial to construct a novel RL framework for addressing model uncertainty with these theoretical properties and an efficient implementation.\n* We agree that the general connection between coherent risk measures and robustness is known (we include this known result in Appendix B.2). However, we apply this known result to establish an equivalence between our RAMU framework and a class of distributionally robust safe RL problems, which is a novel and important result. By establishing this connection, we are able to efficiently address model uncertainty in a deep RL context through the use of risk measures. We will update the paper to make this contribution clear, and we will include the distributionally robust safe RL problem definition to which our RAMU problem is equivalent.\n\n### [W2] Choice of coherent distortion risk measures\n\n* In our RAMU framework, we leverage properties of **coherent** risk measures to provide robustness guarantees, and we leverage properties of **distortion** risk measures to provide an efficient, model-free implementation based on weighted sample averages that does not involve minimax optimization. We will add this commentary to the paper to make our choice of coherent distortion risk measures more clear.\n* Please see Majumdar and Pavone (2020) for a detailed discussion on why each of the properties of coherent distortion risk measures is important in the context of robotics. In general, coherent risk measures and distortion risk measures are two of the most popular classes of risk measures used in the literature, so there is a consensus that the characteristics of these classes are desirable.\n\n### [W3, W5, Q1, Q2] Other clarifications\n\n> [W3] The RAMU cost function as defined in line 156 is introduced somewhat arbitrarily…usually one would start from a base problem…\n\nWe will update Section 4 to start from the formal problem definition that corresponds to the RAMU update in (4). The objective and constraint in this formulation are very similar to the corresponding Q functions defined in line 156, just averaged over initial states and actions. \n\n> [W5] The discussion under \"Generative distribution of transition models\", line 236 onwards is quite convoluted and involved…\n\nThe main goal of this section is to describe how we can define a distribution $\\mu$ over transition models in a way that can be implemented using only data collected from a single training environment. We accomplish this by constructing samples from perturbed versions of the training environment. We will update this section to make this goal clear.\n\n> [Q1] What is a “distribution over transition models” in lines 41-42? This is very ambiguous at this point.\n\n“Distribution over transition models” in lines 41-42 refers to $\\mu$, the same concept of a distribution over potential environments that was introduced in the previous paragraph (lines 34-35). We will update the language to make this more clear at this stage of the paper.\n\n> [Q2] …in Lines 72-74: Defining this product is somewhat obscure…\n\nThis structure in lines 72-74 is known as rectangularity. It is a very common assumption throughout the robust RL literature (see line 75 for references), and we use standard notation from the literature in lines 72-74 to describe this structure. Rectangularity simply implies that the model uncertainty $\\mu_{s,a}$ at every state-action pair is independent from other state-action pairs. In general, this is a conservative assumption, but it allows for recursive definitions of Q functions."}, {"review_id": "crp3YzmGrN", "reviewer": "Reviewer_JpRL", "summary": "The authors consider a distribution over transition models and tackle the safe RL problem by applying a risk-averse perspective towards model uncertainty through coherent distortion risk measures. The proposed formulation can ease the burden of solving a min-max problem, which is often encountered in many worst-case safe RL algorithms. \n\nThe authors also theoretically show that their formulation is equivalent to a specific class of distributionally robust safe RL problems.", "questions": "- What's the choice of the function $g$ (line 224-225) in your experiments?\n- In lines 229-230, is there only one Q function for calculating all sampled transitions?\n- In Table 1, what’s the meaning of the bold numbers? I think you should highlight the best results instead of yours.\n- Also, please explain the necessity and merits of adopting the formulation proposed in this paper.", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This work proposes a new formulation of safe RL, by considering a distribution of transition models and applying the distortion risk measure toward the model uncertainty, which circumvents the burden of solving min-max problems.\n- This work theoretically proposes that the reformulated problem is equivalent to a specific class of distributionally robust safe RL problems.", "weaknesses": "- It is better to include the proof of Lemma 2 in Appendix B.2 for self-contained. Also, explaining the results of Lines 225-226 is better.\n- It seems that the performance of the proposed method can not surpass existing methods, especially adversarial RL. Also, the experiments presented are not sufficient.\n- Although applying the distortion risk measure $\\rho$ seems to be promising, I'm wondering about the necessity of doing so since it introduces more complex computation processes and seems to contribute little to the experiment results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. Please see below for detailed responses to your questions. In particular, we highlight the main benefits of our RAMU framework and the key takeaways of our experimental results. We hope that these responses address your main concerns, and we ask that you please consider updating your review scores to reflect these clarifications.\n\n### [W3, Q4] Main benefits of our RAMU framework\n\nThere are several benefits of our proposed RAMU framework compared to existing methods for addressing model uncertainty in deep RL (such as adversarial RL and domain randomization). The main benefits of our RAMU framework include:\n\n1. Our RAMU framework has robustness guarantees (see Theorem 1). This is not true of popular methods such as domain randomization that apply the expectation operator over distributions of transition models. \n2. Our RAMU framework achieves this robustness without requiring complex minimax optimization. This is not true of robust RL (and distributionally robust RL) methods, which must solve for worst-case transition models (or distributions over transition models) throughout training. Our approach, on the other hand, only requires weighted sample averages that can be computed very efficiently (see lines 220-235).\n3. Our RAMU framework can be implemented using standard data collection from a single training environment (see lines 236-263), without requiring potentially dangerous adversarial interventions (as in adversarial RL) or detailed simulator access (as in domain randomization). Therefore, unlike these existing methods, our approach can be applied in settings that require real-world data collection for training.\n\n### [W2, Q3] Key takeaways from experimental results\n\n* **[W2] Key takeaways:** We believe that our experimental results provide strong support for the benefits of our RAMU framework, and we include thorough comparisons against the most popular methods for addressing model uncertainty in deep RL. The experiments demonstrate the following key points:\n\n    1. Our RAMU framework achieves significant robustness and safety benefits compared to standard safe RL, while using exactly the same data collection process from a single training environment.\n    2. Our use of coherent distortion risk measures leads to robustness and safety benefits compared to a risk-neutral approach based on expectations, as shown by the comparison of RAMU (Wang 0.75) to the special case of RAMU (Expectation). Our risk-averse implementation achieves higher average rewards (1.08 vs. 1.05) and better safety constraint satisfaction (80% vs. 74%).\n    3. Our RAMU framework achieves similar or improved performance compared to the most popular baselines for incorporating model uncertainty in deep RL (domain randomization and adversarial RL). We accomplish this without requiring additional assumptions on the training process, such as detailed simulator access (as in domain randomization) or potentially dangerous adversarial interventions (as in adversarial RL), that are not always suitable in real-world settings.\n\n* **[W2] Comparison to adversarial RL:** Our RAMU framework achieves similar results to adversarial RL, and has significant benefits compared to adversarial RL in terms of implementation. RAMU achieves higher average rewards at test time in 3 out of 5 tasks (and higher average rewards in aggregate: 1.08 vs. 1.05), and better safety constraint satisfaction at test time in 3 out of 5 tasks (and similar safety constraint satisfaction in aggregate: 80% vs. 82%). Unlike adversarial RL, RAMU accomplishes this in a way that (i) does not alter the data collection process, (ii) does not require training an adversary in a minimax formulation, and (iii) does not require different implementations during training and testing. These all represent meaningful drawbacks of adversarial RL, which make adversarial RL unsuitable for training in many real-world tasks.\n* **[Q3] Table 1:** We bold the risk-averse implementation of our RAMU framework to highlight the relevant version of our approach for the reader to focus on.\n\n### [Q1, Q2] Implementation details\n\n> [Q1] What's the choice of the function $g$ (line 224-225) in your experiments?\n\nIn our experiments, we consider the Wang transform with $\\eta = 0.75$ as well as the special (risk-neutral) case of the expectation operator (see lines 302-306). The function $g$ corresponding to these are shown in Figure 7 in Appendix C. The form of $g$ for the Wang transform is also in line 103, and the expectation operator corresponds to a linear $g$.\n\n> [Q2] In lines 229-230, is there only one Q function for calculating all sampled transitions?\n\nYes, there is one RAMU cost Q function (and one RAMU reward Q function). As shown in the critic loss functions (lines 215-216), we are interested in calculating the estimates in (7) with the current target Q function. Therefore, for each sampled transition model, we apply the standard cost Bellman target in lines 229-230 using this single Q function. Then, we combine these estimates using the weighted sample average in (7)."}]

Author rebuttal:
Thank you to all of the reviewers for their thoughtful feedback. We are excited to see the reviewers agree that the paper is clear and well-written (ZX7b, mRCw, 22QQ), proposes a novel framework with a practical and efficient implementation (F3rP, 22QQ, JpRL), and provides strong theoretical (ZX7b, JpRL) and experimental (mRCw, 22QQ) support for this framework. We have replied directly to each reviewer with detailed responses, and we will update the paper to incorporate clarifications based on reviewer suggestions. If we have addressed your main concerns, we ask that you please consider updating your review scores to reflect our responses. Thank you for helping us to improve our paper!

Meta-review: The paper introduces the RAMU framework for safe reinforcement learning in uncertain environments. It uses a distribution of transition models and a risk-averse approach with coherent distortion risk measures. The authors aim to create a policy that's both robust and safe by combining Constrained Markov Decision Processes (CMDP) with distribution over uncertainty sets, optimizing a nested risk measure of discounted return while meeting cost-related guarantees. RAMU is efficient and model-free, working within a single training environment.

All reviewers are positive about this paper. We have two Borderline Accepts and three Weak Accepts. Among the positive aspects of this paper are
- Novelty of the framework for safe RL.
- Theoretical results
- Efficient implementation that avoids complex minimax optimization.

There are some concerns too, including (not comprehensive list):
- Reviewers request clarification on the connection between safety and robustness in the article.
- Missing some relevant work.
- Complicated discussion at some points.

Overall, I believe this is a good paper. **Given the unanimous support of all reviewers, I recommend its acceptance.**
Decision: Accept (poster)