Optimal Transport Perturbations for Safe Reinforcement Learning with Robustness Guarantees
Authors: James Queeney
Erhan Can Ozcan
Ioannis Paschalidis
Christos Cassandras
Venue: Accepted by TMLR ()

Abstract: Robustness and safety are critical for the trustworthy deployment of deep reinforcement learning. Real-world decision making applications require algorithms that can guarantee robust performance and safety in the presence of general environment disturbances, while making limited assumptions on the data collection process during training. In order to accomplish this goal, we introduce a safe reinforcement learning framework that incorporates robustness through the use of an optimal transport cost uncertainty set. We provide an efficient implementation based on applying Optimal Transport Perturbations to construct worst-case virtual state transitions, which does not impact data collection during training and does not require detailed simulator access. In experiments on continuous control tasks with safety constraints, our approach demonstrates robust performance while significantly improving safety at deployment time compared to standard safe reinforcement learning.

Published in Transactions on Machine Learning Research (03/2024)
Optimal Transport Perturbations for Safe Reinforcement Learning with Robustness Guarantees
James Queeney
queeney@merl.com
Mitsubishi Electric Research Laboratories
Erhan Can Ozcan
cozcon Wbu. edu
Division of Systems Engineering
Boston Universiby
Joannis Ch. Paschalidis
yannisp Øbu. edu ege @bu. edu
Christos G. Cassandras
Department of Electricol and Computer Engineering and Division of Systems Engineering Boston University
Reviewed on OpenReview: https: // openrevvee.. net/ forum? id- egSXpAR4G1
Abstract
Robustness and safety are critical for the trustworthy deployment of deep reinforcement learning. Real-world decision making applications require algorithms that can guarantee robust performance and safety in the presence of general environment disturbances, while making limited assumptions accomplish this goal, we introduce safe reinforcement learning framework that incorporates robustness through the use of an optimal transport cost uncertainty set. We provide an efficient implementation based on applying Optimal Transport Perturbations to construct worst-casc virtual state transitions, which does not impact data collection during training and does not require detailed simulator access. In experiments on continuous control tasks with safety constraints, our approach demonstrates robust performance while significantly improving safety at deployment time compared to standard safe reinforcement learning.
the data collection process during training. In order to
1 Introduction
Deep reinforcement learning (RL) is data-driven framework for sequential decision making that has demonstrated the ability to solve complex tasks, and represents promising approach for improving real-world decision making (Dulac-Arnold et al. 2021 In order for deep RL to be trusted for deployment in real-world decision making settings, however, robustness and safety are of the utmost importance (Xu et al. 2022). As
result, techniques have been developed to incorporate both robustness and safety into deep RL. Robust RL methods protect against worst-case transition models in an uncertainty set (Nilim & Ghaoui 2005 Iyengar 2005 while safe RT. methods incorporate safety constraints into the training process (Altman) 1999).
In real-world applications, disturbances in the environment can take many forms that are difficult to model in advance. Therefore, require methods that can guarantee robust performance and safety under general forms of environment uncertainty. Unfortumately, popular approaches to robustness in deep RL consider very structured forms of uncertainty in order to facilitate efficient implementations. Adversarial methods implement a specific type of perturbation, such as the application of a physical force (Pinto et al. 2017) or a change in the action that is deployed (Tessler et al. 2019a). Parametric approaches, on the other hand, consider robustness with respect environment characteristics that can be altered simulator (Rajeswaran) et al.] 2017 Peng al. 2018 Mankowitz et al.) 2020). When we lack domain knowledge on the structure of potential disturbances, these techniques may not. guarantee robust performance and safety.

---

Published in Transactions on Machine Learning Research (03/2024)
Worst-case transition models
Optimal Transport Perturbations
P(S)
s
L (8')
Ps.a
F2 (8')
P.r
Tp.
Figure 1: Illustration of Optimal Transport Perturbations. Left: Worst-case transition models in the optimal transport uncertainty set Ps,a P(S) that correspond to the robust Bellman operators in ( and 6 Right: Tractable reformulation by applying Optimal Transport Perturbations directly in S to given next state sample 8' ~ is.a. The black arrow denoles the state transition observed in the nominal environment, and the dashed arrows denote virtual state transitions used only to calculate robust Bellman operators.
Another drawback of existing approaches their need directly modify the environment during training. Parametric methods assume the ability to generate range of training environments with a detailed simulator, while adversarial methods directly influence the data collection process by attempting to negatively impact performance. In applications where simulators are inaccurate or unavailable, however, parametric methods cannot be applied and real-world data collection may be required for training. In this context, it is also undesirable to implement adversarial perturbations while interacting in the environment. Therefore, in many world domains, we must consider alternative methods for learning safe policies with robustness guarantces.
In this work, we introduce safe RL framework that provides robustness to general forms of environment disturbances using standard data collection in nominal training environment.. We consider robustness over general uncertainty set defined using the optimal transport cost between transition models, and we show how our framework can be efficiently implemented by applying Optimal Transport Perturbations after data collection to construct worst-case virtual state transitions. See Figure [ for an illustration. These perturbations can be added to the training process of many popular safe RL algorithms to incorporate robustness to unknown disturbances, without harming performance during training or requiring access to a range of simulated training environments.
We summarize contributions as follows:
We formulate a safe RL framework that incorporates robustness to general disturbances using the optimal transport cost between transition models.
We show in Theorem I that the resulting worst-case optimization problems over transition models can be reformulated as adversarial perturbations to state transitions in the training environment.
We propose an efficient deep RL implementation of our Optimal Transport Perturbations, which we use to construct worst-case virtual state transitions without impacting data collection during braining.
We demonstrate that the use of Optimal Transport Perturbations leads to robust performance and safety both during training and in the presence disturbances through experiments on continuous control tasks with safety constraints in the Real-World RL Suite (Dulac-Arnodd et. al. 2020 2021
2

---

Published in Transactions on Machine Learning Research (03/2024)
2 Related work
2.1 Safe reinforcement learning
The most common approach to modeling safety in RL is to incorporate constraints on expected lotal costs in Constrained Markov Decision Process (Altman 1999), which is the definition of safety consider in this work. In recent years, several deep RL algorithms have been developed for this framework. A popular approach is to solve the Lagrangian relaxation of the constrained problem (Tessler et al.) 2019b Ray et al., 2019; Stooke et al., 2020 which is supported by theoretical results establishing that constrained RL has zero duality gap (Paternain et al. 2019).
Other approaches to safe RL construct closed-forms solutions to guide
policy updates (Achiam et al.
2017 Liu et al., 2022). or consider immediate switching between the objective
and constraints to better satisfy salety during training (Xu et al.. 2021p.
A separate line of work focuses the issue of safe exploration during data collection (Brunke et al. 2022 Cheorel- approaches focus on avoiding sets of unsafe states through the use of control barrier functions 2021 or other safety filters (Dalal et al. 2018 but often (Cheng et al., 2019 Emam et al. 2021 Ma et al.)
require additional assumptions on system dynamics. Recently, action correction mechanisms have been applied with unknown dynamics through the use learned safety critics (Srinivasan et al. 2020 Bharadhwai lt al.) 2021) and recovery policies (Thananjeyan al., 2021 Waggneer et al.) 2021]. We do not directly consider the issue of safe exploration in this work, but it possible to combine our robust training approach with correction mechanisms for safe exploration.
2.2 Robust reinforcement learning
Robust RL methods account for uncertainty in the environment by considering worst-case transition models from an uncertainty set (Nilim & Ghaoui 2005 Iyengar 2005). In order to scale the robust RL framework to the deep RL setting, most techniques have focused on (i) parametric uncertainty or (ii) adversarial training.
Parametric uncertainty methods generate multiple training environments by modifying parameters in simulator that are often determined based domain knowlodge. Domain randomization (Tobin et al. 2017 Peng et al." 2018) maximizes average performance over range of training environments, which has been referred to as a sobt-robust approach (Derman et al.) 2018). Other methods directly impose a robust perspective towards parametric uncertainty by focusing on the worst-case training environments generated over a range of simulator parameters (Rajeswaran et al. 2017 Abdullah et al. 2019 Mankowitz et al. 2020). All of these approaches assume access to a simulated version of the real environment, as well as the ability to modify parameters of this simulator.
Adversarial RL methods represent alternative approach to robustness that introduce perturbations directly into the training process. In order to learn policies that perform well under worst-case disturbances, these perturbations are trained to minimize performance. Deep RL approaches to adversarial training have introduced perturbations in the form of physical forces in the environment (Pinto et al.] 2017), as well as adversarial corruptions to actions (Tessler et al. 2019a; Vinitsky et al. 2020) and state observations Mandlekar et al. 2017 Zhang et al., 2020 Kuang et al. 2022 Liu al.J [2023a b). In this work, we learn adversarial perturbations on state transitions, but different from most adversarial RL methods we apply these perturbations to construct virtual transitions without impacting the data collection process.
Finally, safety and robustness have recently been considered together in unified RL framework. Mankowitz et al. 2021 and Russel et al. (2021) propose a formulation that incorporates robustness into both the objective and constraints in safe RL. We consider this general framework as a starting point for our work.
3 Preliminaries
3.1 Safe reinforcement learning
Consider an infinitohhorioon, discounted Constrained Markov Decision Process (C-MDP) Altman 1999 defined by the tuple (S p. r; c, Po, 7), where S is the set of states, A is the set of actions, p: S x A P(S)
3

---

Published in Transactions on Machine Learning Research (03/2024)
is the transition model where P(S) denotes the space of probability measures over S, r : A R is the reward function, e x A IR is the cost. function, Po is the initial state distribution, and 7 is the discount. rate.
We model the agent's decisions as a stationary policy π S + P(A). For a given C-MDP and policy n, the expected total discounted rewards and costs are given by Jp.((T) (x-(s.)) 7 at)] and (n.(TT) p))(= - 7'c(st, respectively, where (T. represents a trajectory sampled according to So ~ Po. a, .(. 82), and 8441 p(- az). The goal of safe RL is to find policy that solves the constrained optimization problem
max (T) s.t. ppc(T) s B,
(1)
where is a safety budget on expected total discounted costs.
We denote the state-action value functions (i.c., Q functions) of for given C-MDP as (s, and Liu et al. 2022 iteratively optimize by
(p.c(s, a). Off-policy optimization techniques (Xu al. 2021
considering the related optimization problem
max
E
E
1 (8,
a)]
s.t.
E
E Lavx(-1l)
(p)c(s,)))
< B,
(2)
(a-x(-18)
where Ty is the current policy and is a replay buffer containing data collected during training.
3.2 Robust and safe reinforcement learning
We are often interested in finding policy that achieves strong, safe performance across range of related environments. In order to accomplish this, Mankowilz et. al. (2021) and Russel et al. 2021)) propose a Robust. Constrained MDP (RC-MDP) framework defined by the tuple (S,A, A.P.Pr Po: 7), where P represents an uncertainty set of transition models. We assume = X (s.a)ExxA Ps.a, where Ps.a a set of transition
models Ps.a = p( 8, a) E P(S) al a given stale-action pair and P is the product of these sets. This structure is referred to as rectangularity, and is a common assumption the literature (Nilim & Ghaoui 2005 2005
[yengar
In order to learn policy with robust performance and safety across all environments in P, the RC-MDP framework considers robust version of ( given by
max inf (x)( s.t. sup Jp.c(T) < B. PEP
33)
PEP
Remark 1. A policy n that solves 3 satisfies the safety constraint across all transition models in P (i.e., achieves robust safety). Within this set of feasible policies, a policy that solves (3) marimizes the worst-case performance across all transition models in P (i.e., achieves robust performance).
We are interested in solving (3) to find policy with robust performance and safety as described in Remark 1 This is in contrast to solving the standard safe RT problem in (1]. which only considers performance and safety in a single environment. As in the standard safe RL setting, we can apply off-policy optimization 3 by considering the related optimization problem
techniques to itcratively optimize
O OP,, (s.0))
O (07a(0.a)) [a>x(:1s)
E aND
S.L.
E
E
$ B,
(-))(1)s))
anD
where Q7, a) and (5, a) represent robust Q functions. In this work, we focus on how to efficiently learn the robust Q functions that are needed for the robust and safe policy update in 4
3.3 Robust Bellman operators
Compared the standard safe RL update in 2 the only difference in the robust and safe RL update of 4 comes from the use of robust Q functions. Therefore, in order to incorporate robustness into existing deep
4

---

Published in Transactions on Machine Learning Research (03/2024)
Ps.a
Pa.a
ds,a(8'
Figure 2: Illustration optimal transport cost between transition models ao Pa,a E P(S). Informally, OTCd, Pa.a) represents the cost of transporting the probability mass of Ps.a Pa,a using the optimal (i.e., minimum cost) transport plan v*, where the transport cost is determined by dg.a-
safe RL algorithms, we must be able to efficiently learn (8, a) and (7,c(s, a). These robust Q functions represent the unique fixed points of the corresponding robust Bellman operators NNilim & Ghaoui Tyengar 2005
2005
T (F.0r(s.)):=(((11) +7
inf P.aEPa. s'rpp,
E
VV7(s))
(5)
T= := c(s, +y sup
E Plrca
[s'()
(6)
Pr.eP
where we write (s') En' (ss') Qr(s' d')] and V
Ea Qe(s', Note that Tp and T5 are contraction operators (Nilim & Ghaoui 2005 Lyengar 2005), we can apply standard temporal difference learning techniques to learn Q (s, a) and Q (, a). In order to do so, must be able to calculate the Bellman targets in 5 and (6), which involve worst-case optimization problems over transition models that depend on the choice of uncertainty set Pe.a at every state-action pair. In order to estimate these Bellman targets, popular choices of Ps.a in the literature require the ability to change physical parameters of the environment (Peng et al. 2018) or directly apply adversarial perturbations during trajectory rollouts (Tessler eet al. 2019a) lo calculate worst-case transitions. However, because these implementations rely on multiple simulated training environments or potentially dangerous adversarial interventions, they are not compatible with settings that require real-world data collection for training.
4 Optimal transport uncertainty set
In this work, we consider an uncertainty set based on optimal transport cost, and show that this choice of
uncertainty set leads to efficient implementation of the worst-case optimization problems in (5) and (6).
In order to do so, we assume that S is Polish space (i.e., separable, completely metrizable topological space). Note that the Euclidean space R" is Polish, so this is not very restrictive. Next, we define Pa.a using the optimal transport cost between transition models.
Definition (Optimal transport cost). Let S be Polish space, and let de,a : S S + R4 be non-negative, lower semicontinuous function satisfying ds'als,, 8')= O for all E S. Then, the optimal transport cost.
between two transition models Pa.a: E P(S) is defined as
inf / OTCa,.. (Po.a vEI(p sxs ( sd)
where T(ps,as Ps.a) the set of all couplings of Ps.a and Ps.a-
See Figure for an illustration of Definition 1 For a given transport cost function dg OTC (P..a represents the minimum cost required transport the probability mass of Ps.a Ps.as and the optimal coupling D* describes the transport plan that achieves this minimum cost. If dy is chosen to be a metric

---

Published in Transactions on Machine Learning Research (03/2024)
raised to some power p 2 recover the p- W-sssessttin distance raised to the power p special case
[Chen & Paschalidis, 2020). If we let (3.'(8''')) styest, recover the total variation distance special case [Villani 2008).
By considering the optimal transport cost from some nominal transition model Ps.as we define the optimal transport uncertainty set as follows.
Definition 2 (Optimal transport uncertainty set). For a given nominal transition model Pa transport cost function da and radius esaa at (s, a) E x A, the optimal transport uncertainty set defined as
Ps.a (Po.a E P(S) OTC (Pe. Po.a) S c,,a)
This uncertainty set has previously been considered in robust RL for the special case of the Wasserstein distance 4Abdullah et al., 2019; Hou et al. 2020 Kuang et al.) 2022). As we will show in the following sections, the use of an optimal transport uncertainty set leads to an efficient, model-free implementation of robust and safe RL that only requires the ability to collect data in a nominal environment.
Note that the optimal transport uncertainty set in Definition can be applied across a variety of domains. Optimal transport cost remains valid for distributions that do not share the same support, unlike other popular measures between distributions such the Kullback-Leibler divergence. As result, the optimal transport uncertainly set is very general and can be applied to both stochastic and deterministic transition models. Optimal transport cost also accounts for the relationship between states in S through the function da and allows significant flexibility in the choice of d, This includes threshold-based binary comparisons PPyd & Jog, 2020) and percentage-based comparisons between states that are not metrics or pseudo-metrics. To demonstrate the flexibility of our approach for the common setting where S R,, we consider in our experiments a percentage- based comparison of state transitions given by
1
8 8
2
a
(8', 80)
-
with the convention that 0/0 so that a (s', 8 0 for all s' S. Note that 7 satisfies all requirements in Definition | and is not metric pscudo-metric. In more general scttings, choose dz,a based the application to reflect the geomelry of S in meaningful way.
5 Reformulation as worst-case virtual state transitions
We are interested solving the robust and safe RL problem in 3 for an optimal transport uncertainty set, which we can accomplish by applying the policy updates given by In order to learn the robust Q functions that appear in 44 for an optimal transport uncertainty set, focus on how to efficiently caleulate the robust Bellman operators in and G6. We consider the following main assumptions.
Assumption 1. For any and Qr's', d') in (5') Ee'=a((-ee) (s' a')] is lower semicontinuous and Ey'np. V (s') < co. For any x and Qe(s', a') in 6 V En'nx(1)s)) [Qe(s', d') is upper semicontinuous
and E p.... V.2 (8') <o.
Assumption 2. Optimal transport plans exist for the worst-case oplimization problems over transition models in and 6
Note that Assumptions 1 correspond assumptions in Blanchet & Murthy (019 applied to our setting. In practice, the use of neural network representations results in continuous value functions, which are bounded for the common case when rewards and costs are bounded, respectively. A sufficient condition for Assumption 2 to hold is if S is compact, or if we restrict our attention to a compact subset of next states in our definition of Ps,a which is reasonable in practice. Blanchet & Murthy (2019) also provide other sufficient. conditions for Assumption to hold.
Under these assumptions, we can reformulate the robust Bellman operators in and to arrive at a tractable result. that can be efficiently implemented in deep RL. setting.

---

Published in Transactions on Machine Learning Research (03/2024)
Theorem 1. Let Assumptions hold, and let g be the set of all functions from S to S. Then, we have
Tr, r(s, +
E
(V
(8)
a)) = =c(,, +
E (o
(9)
B'mpe.a
where for given state-action pair (8,a) E S A we have
Ia arg min gEG
E
(v(('())
s.t.
E
[daa(3' (((')) S 62,01
(10)
8f g'mpe
92.0 E arg max
E gEG 3
(g(*(())
s.t.
E sl-Pan
[3)] S Cz.a.
(11)
Proof. First, we leverage results from Blanchet & Murthy 2019 to show that optimal transport strong duality
holds for the worst-case optimization problems over transition models in (5) and (6) under Assumption 1
Next, we establish that the optimization problems in (5) and (6) share the same dual problems as (10) and
(I), respectively. Finally, we use Assumption 2 to show that strong duality also holds for (10) and
which proves the result. See the Appendix for details.
For the optimization problems in (5) and 66) note that the optimal transport plan D' from the nominal
transition model fs.a to the worst-case transition model in Ps.a can be characterized by a deterministic perturbation function in state space (Blanchet & Murthy 2019). Utilizing this structure, Theorem [ demonstrates that we can calculate the robust Bellman operators Tø. and T by using samples collected from fr..: and adversarially perturbing the next state samples according (10) and [11. respectively. We refer to the resulting changes in state transitions Optimal Transport Perturbations (OTP). As result, we have replaced difficult optimization problems over transition models in (5) and (6) with the tractable problems of computing Optimal Transport Perturbations in state space. See Figure
[ for an illustration.
Theorem represents the main theoretical contribution of our work, which directly motivates an efficient. deep RL implementation of robust and safe RL.
Finally, note that these perturbed state transitions are only needed to calculate the Bellman targets in (8) and 9 which we use train the robust Q functions (5, a) and Qp (s,a). Therefore, unlike other adversarial approaches to robust RL (Pinto et al.) 2017 Tessler et al.] 2019a Vinitsky et al., 2020). our Optimal Transport Perturbations ions have no impact on trajectory rollouts in the nominal training environment. Instead, these perturbations bations are applied after data collection to construct worst-case virtual state transitions.
6 Perturbation networks for deep reinforcement learning
From Theorem we can calculate Bellman targets for our robust Q functions Q7, 27, a) and Qs, a) by considering adversarially perturbed versions of next states sampled from p..a- We can construct these adversarial perturbations by solving (10) and [11 respectively. Note that we can combine the perturbation functions I2.a: E G from Theorem across state-action pairs by including (s,a) as input for context, in addition to the next state &' to be perturbed. This leads to the combined perturbation functions gf : S x A + S, where g", a, 8') = (3") and ge a, 8¹) = (8)) We let F be the set of all functions from
A x S to S, with g².g" F.
In the context of deep RL, consider class of perturbation functions F, C F parameterized by neural network d: S x Ax x S + S. In our experiments, consider tasks where S = R and we apply multiplicative perturbations to state transitions. In particular, we consider perturbation functions of the form
go(s, a, 8') + (8' - s) 5(s,a,8))
(12)
where 6(s, a, 8') E R and all operations are performed per-coordinate. By defining Fs in this way, we obtain plausible adversarial transitions that are interpretable, where ö(s, a, 8') represents the percentage change the nominal state transition in each coordinate.
7

---

Published in Transactions on Machine Learning Research (03/2024)
Algorithm 1: Safe RL with Optimal Transport Porturbations Input: initial policy no: critics Qe Qe OTP networks 6,, de. for = 0,,,2..... do
Collect data T ~ (TT. p) and store it in D.
for K updates do
Sample batch of data (s,a,r,c,53 D.
Update OTP networks 6,, d according to (13 and 14
Calculate Bellman targets (0,), ((.(())
15 and [66 and update critics Qe Qo. minimize
Update policy according
end
end
Using da,a from
7
we have that
1
",a(5',g6(s,,,33 -10(s, - 821
Then, following common practice in off-policy deep RL, we combine the perturbation function updates in 10 and 111 across state-action pairs by averaging over samples from the replay buffer. By doing so, we can efficiently update our perturbation networks a deep RL setting according to
6, E arg min
E (s
V (ga(s, a, 8'))
s.t.
[10((,, a, 8') S ne
(13)
8e arg max à
E VV 8'))] s.t. (s
(s,0,8") [10((s, a 8') S ne
(14)
where as) D are transitions collected in the nominal environment and 65 represents the average per-coordinate magnitude of 6(s, a. 8') with E(,,)) (c,al = c.. In practice, we apply gradient-based updates to the Lagrangian relaxations of (13 and (1], and we update the corresponding dual variables throughout training. It also possible to satisfy perturbation function constraints at every state-action pair through the use of clipping mechanisms, desired. Note that any violation of the perturbation function constraints practice will only lead additional robustness and will not negatively impact safety.
We train separate reward and cost perturbation networks 8, and de, and apply the resulting Optimal Transport Perturbations to calculate the Bellman targets in (8) and (9) for training the robust Q functions Q7, 27, (s, a) and (s, a). For (5,0,8) ~ D, we consider the sample-basee estimates
+ + (96. (s, a'))), tF.0c(s,a) c(s,a) + vV. (9s. (s, a, a
= -
(15)
(16)
7 Algorithm
We summarize our approach to robust and safe RL Algorithm| [ At every update, we sample previously collected data from a replay buffer D. We update our reward and cost perturbation networks 8, and 5, according to (33 and (14), respectively. Then, we estimate Bellman targets according (15) and {16].
which to update critics via standard temporal difference learning loss functions. We consider

---

Published in Transactions on Machine Learning Research (03/2024)
Table 1: Aggregate performance summary
Normalized Ave.
Rollouts Require
Algorithm
% Safet
Reward
Cost
Adversary Simulator
Safe RL
51%"
1.00*
1.00
No
No
ОТР
87%
1.06
0.34
No
No
Adversarial RL (10%)
88%
0.95*
0.28
Yes
No
Adversarial RL (5%)
82%*
1.05
0.48*
Yes
No
Domain Randomization
76%
1.14"
0.72"
No
Yes
Domain Randomization (OOD)
55%
1.02
1.02"
No
Yes
Risk- Averse Model Uncertainty
80%*
1.08
0.51
No
No
Percentage policies that satisfy the safoty constraint across tasks and test environments.
Normalized relative the average performance of standard safe RT for each and environment.
Denotes need for adversary or simulator during data collection (i.., trajectory rollouts) for training.
Statistically significant difference (p < 0.05) compared to OTP using paired t-test.
parameterized critics Qo, and Qo.: and we optimize their parameters to minimize the loss functions
(p,r
E
( (0. [
27
(s,a,))
F0, [
[90
fp.(
E (0
where Qo, and Qo, represent target critic networks. Finally, we use these critic estimates update our policy according to (4..
Compared to standard safe RL methods, the only additional components of our approach are the perturbation networks used to apply Optimal Transport Perturbations, which we train alongside the critics and the policy using standard gradient-basee methods. Otherwise, the computations for updating the critics and policy remain unchanged. Therefore, is simple to incorporate our OTP framework into existing deep safe RL algorithms in order to provide robustness guarantees on performance and safety.
8 Experiments
We analyze the use of Optimal Transport Perturbations for robust and sale RL on continuous control lasks with safety constraints in the Real- World RLL Suite (Dulac-Arood al.] 2020 2021). We follow the same experimental design used in Queeney & Benosman (2023). In particular, we consider 5 constrained tasks over domains (Cartpole Swingup, Walker Walk, Walker Run, Quadruped Walk, and Quadruped Run), which all have horizons of 1, 1,000 with r(s, a) E [0, 1] and c(s,a) 1). In all tasks, consider safety budget of = 100. We train policies in a nominal training environment for million steps over 5 random seeds, and we evaluate the robustness the learned policies in terms of both performance and safety across a range of perturbed test environments. See the Appendix for details on the safety coints and environment. perturbations considered for cach task.
Our Optimal Transport Perturbations can be combined with many popular safe RL algorithms that perform policy updates according to (2) by replacing standard Q functions with robust Q functions in 4 which is benefit of our methodology. In our experiments, we consider the safe RL algorithm Constraint-Rectifed Policy Optinization (CRPO) (Xu et al.] 2021), and we use the unconstrained deep RL algorithm Maximum a Posteriori Policy Optimization (MPO) [Abdolmaleki et al. 2018) to calculate policy updates in CRPO. For fair comparison, we apply CRPO with MPO policy updates as the bascline safe RI. algorithm in every method we consider in our experiments. See the Appendix for additional results using a Lagrangian- safe RL update method (Tessler et al. 2019b Ray et al. 2019 We train multivariate Gaussian policy,

---

Published in Transactions on Machine Learning Research (03/2024)
Caripole Swinguu
Walker Walk
Walker Run
Quadruped Walk
Quadruuped Run
1.10
1.40
1.20
Nomn Reward
1.00
1:5gg00
1.04
0.0
1,20
1.29
1.16
1.12
1:17
1.13133
1.00
1.00
6.M
01900000
1.00
1.0
1.06
0.90
0.80
0.80
0.80
100
100
66%
100
100
106/810%
100
92%
40%
10/260095
% Sala
80
4mh
00
80
10%
80
50
60
60
M5%
60
40
40
3.5%
(101))
RL (916)
Randomization
(OOD)
Figure 3: Performance summary by task, aggregated across test environments. Performance of adversarial RL is evaluated without adversarial interventions. Algorithms to the right of the dotted lines require additional assumptions during data collection compared to standard safe RL (see Table | for details). Top: Average total reward, normalized relative to the average performance of standard safe RL for each test environment. Bottom: Percentage of policies that satisfy the safety constraint across all test environments.
where the mean and diagonal covariance at given state are parameterized by a neural network. We also consider separate neural network parameterizations for the reward and cost critics. See the Appendix for additional implementation details, including network architectures and values of all hyperparameters
We incorporate robustness into the baseline safe RL algorithm in four ways: (i) Optimal Transport Perturba-
tions, (ii) adversarial RL using the action-robust PR-MDP framework from Tessler et al. (2019a) applied to
the safety constraint, (iii) the soft-robust approach of domain randomization tion (Peng et al. 2018), and (iv) the
distributionally robust approach of Risk-Averse Model Uncertainty (RAMU) from Queeney & Benosman (023). For our Optimal Transport Perturbations, we consider еạ 0.02 (i.e., 2% perturbations on average). Figure and Table summarize the performance of all algorithms at deployment time, where performance metrics are aggregated across the range of perturbed test environments. See the Appendix for detailed results across all test environments. Note that the training process of each algorithm only considers safety for a specific set of environments, so we do not expect the resulting policies to remain safc across all possible test cases.
8.1 Comparison to safe reinforcement learning
Figure compares our OTP framework to standard safe RL across all tasks and perturbed test environments. By applying Optimal Transport Perturbations to the objective and constraint in safe RL, we achieve meaningful test-time improvements compared to the standard non-robust version of safe RL. While in most cases we observe a decrease in total rewards in the nominal environment in order to achieve robustness, average our framework leads to an increase in total rewards of 1.06x relative to safe RL across the range of test environments. Most importantly, we see a siguificant improvement in safety, with our algorithm satislying constraints in 87% of test cases (compared to 51% for safe RL) and incurring 0.34x the costs of safe RL, on average. Note that we achieve this robustness while collecting data from the same training environment considered in standard safe RL, without requiring adversarial interventions in the environment or domain knowledge on the structure of the perturbed test. environments.
expected,
8.2 Comparison to adversarial reinforcement learning
Next, we compare our approach to the PR-MDP framework Tessler et al. 2019a), ) adversarial RL method that randomly applies adversarial actions a percentage of the time during training. In order to apply this method to the safe RT setting, we train the adversary to maximize costs. We apply the default probability of
Code publicly available http:::/ihhtt.cccc/////e//rettet
10

---

Published in Transactions on Machine Learning Research (03/2024)
Cartpole Swingup
Walker Walk
Walker Run
Quadruped Walk
Quadruped Run
1000
Total Reward
800
800
800
850
700
600
400
600
800
400
200
600
200
200
200
Total Cost
100
100
100
100
0
0.8
1.0
12
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.6
500
750
1000
1250
1600
500
750
1000
1250
1600
Pole Langth
Torso Length
Torso Length
Torso Density
Torso Density
Safe RL
OTP
Figure 4: Comparison of OTP with standard safe RL across tasks and test environments. Shading denotes one standard error across policies. Vertical dotted lines represent nominal training environment. Top: Total reward. Bottom: Total cost, where horizontal dotted lines represent the safety budget and values below these lines represent safcty constraint satisfaction.
intervention of 10% considered in Tessler el. al. (2019a). As shown in Figure 3 this adversarial approach leads to robust constraint satisfaction at test time in the Quadruped tasks. Our OTP framework, on the other hand, leads to improved constraint satisfaction the remaining 3 out of 5 tasks. However, the robust safety demonstrated by adversarial RL also results in lower total rewards on average, and is the only robust approach in Table T that. underperforms standard safe RT in terms of reward. In order improve performance with respect to reward,
also considered adversarial RL with lower probability of intervention of 5%. While this less adversarial implementation is comparable to our OTP framework in terms of total rewards, it leads to a decrease in safety constraint satisfaction to 82%. Therefore, our OTP formulation demonstrates the safety benefits of the more adversarial setting and the reward benefits of the less adversarial setting.
In addition, an important drawback of adversarial RL is that it requires the intervention of an adversary the training environment. Therefore, in order to achieve robust safety at deployment time, adversarial RL incurs additional cost during training due to the presence of adversary. Even in the Quadruped tasks where adversarial RL results in near-zero cost at deployment time, Figure shows that adversarial RL leads to the highest total cost during training due to adversarial interventions. In many real-world situations, this additional cost during training is undesirable. Our OTP framework, on the other hand, achieves the lowest total cost during training, while also resulting in robust safety when deployed in perturbed environments. This is due to the fact. that our Optimal Transport Perturbations do not impact the data collection process during training.
8.3
Comparison to domain randomization
We now compare our ОТP framework to the soft-robust approach of domain randomization (Peng et al, 2018). which assumes access to a range of environments during training through the use of a simulator. By training across half of the test environments, domain randomization achieves strong performance across test cases in terms of reward (1.14x compared safe RL, on average), which was the motivation for its development in the setting of sim-to-real transfer. However, domain randomization only satisfies safety constraints in 76% of test cases, which is lower than both OTP and adversarial RL which explicitly consider robust. formulations. This is likely due to its soft-robust approach that focuses average performance across the training distribution.
It is important to note that domain randomization not only requires access to a range training environments, it also requires prior knowledge on the structure of potential disturbances define its training distribution. order evaluate the case where we lack domain knowledge, we consider an out-of-distribution (OOD) version of domain randomization that. is trained on distribution over different parameter than the one varied at.
11

---

Published in Transactions on Machine Learning Research (03/2024)
Total
50
Cartpole Swingup
Walker Walk
Walker Run
Quadruped Walk
Quadruped Run
Safe RL
OTP
Adversarial RL (10%)
Adversarial RL (5%)
Figure 5: Average final training cost in the nominal training environment. Training cost of adversarial RL includes impact of adversarial interventions. Black bars denole one slandard deviation across policies. Horizontal dotted line represents safety budget.
test time. When the training distribution is not appropriately selected, we see that domain randomization provides little benefit compared to standard safe RL. Our OTP framework, on the other hand, guarantecs robust and safe performance under general forms environment uncertainty while only collecting data from a single training environment.
8.4 Comparison to distributionally robust safe reinforcement learning
Finally, we compare our robust and safe RL method to a distributionally robust safe RL algorithm from Queeney Benosman (2023) called Risk-Averse Model Uncertainty (RAMU). Distributionally robust RL [uu Mannor 2010 Yu & Xu 2016) considers a distribution over transition models p ~ μ4, and provides robustness to worst-case distributions over transition models in an ambiguity set centered around μ. RAMU applics coherent distortion risk measure over μ to achieve distributionally robust guarantecs in safe RL setting. For specific choices of μr, this approach can be implemented without requiring detailed simulator access. Because distributionally robust RL provides robustness to worst-case distributions over transition models instead of worst-case transition models, RAMU results in a softer notion of robustness compared to the robust and safe RL problem in 3
We see Figure 3 and Table that RAMU slightly improves performance terms total rewards compared to our OTP framework due its less conservative notion of robustness, but this comes at the expense of safety constraint satisfaction. The stronger robustness guarantees of our problem formulation translate to statistically significant improvement in safety at deployment time compared to distributionally robust safe RI. Our OTP framework remains safe in 87% of test environments, compared to 80% for RAMU. Overall, these experimental results are in line with the different robustness guarantees associated with these frameworks.
9 Conclusion and broader impacts
In this work, we have developed general, efficient framework for robust and safe RL. Through the use of optimal transport theory, we demonstrated that we can guarantee robustness to general forms of environment disturbances by applying adversarial perturbations to observed state transitions. These Optimal Transport Perturbations can be efficiently implemented by constructing virtual transitions without impacting data collection during training, and can be casily combined with existing techniques for safe RL provide protection against unknown disturbances. Because our framework makes limited assumptions on the data collection process during training and does not require directly modifying the environment, it should be compatible with many real-world decision making applications. As result, we hope that our work represents promising step towards trustworthy deep RL algorithms that can be reliably deployed to improve real-world decision making.
Despite this progress, there are limitations our approach that should be considered prior potential deployment. Our use of the RC-MDP framework guarantees robust performance and safety for all environments in the uncertainty set P, which requires specifying the transport cost function d and radius 6s.a to achieve
12

---

Published in Transactions on Machine Learning Research (03/2024)
the desired robustness without being overly conservative. In addition, consider the C-MDP definition of safety given by a constraint on expected total costs. In order to develop robust control methods for safety-critical scemarios that instead require avoiding sets of unsafe states, the combination of our OTP framework with control-theorctic approaches to safety represents an interosting avenue for future work. Finally, our experimental analysis focused on a set of simulated continuous control lasks with deterministic dynamics from the Real-World RL Suite. Given the flexibility of our framework, we believe there are exciting opportunities to apply our methodology across a variety of additional tasks and problem settings in future work, including complex robotics applications where accurate simulators not available and real-world data collection is required for training.
Acknowledgments
This research was partially supported by the NSF under grants CCF-2200052, CNS-1645681, CNS-2149511, DMS-1664644 ECCS-1931600 and 119-1914792, by the ONR under grants N00014-19-1-257 and N00014-21-1 2844, by the NIH under grants RO1 GM135930 and UL54 TR004130, by AFOSR under grant FA9550-19- 19-1-0,58, by ARPA-E under grant DE-ARO001283 282, by the MathWorks, and by the Boston University Kilachand Fund for Integrated Life Science and Engineering. The majority of this work was done while James Queeney was with the Division of Systems Engineering, Boston University. He is currently with and exclusively supported by Mitsubishi Electric Research Laboratories.
References
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. In Sirth International Conference on Learning Representations, 2018.
Abbas Abdolmaleki, Sandy Huang, Leonard Hasenclever, Michael Neunert, Francis Song, Martina Zambelli, Murilo Martins, Nicolas Heess, Raia Hadsell, and Martin Riedmiller. A distributional view on multi- objective policy optimization. In Proceedings of the 37th International Conference Machine Learning, volume 119, pp. 11-22. PMLR, 2020.
Mohammed Amin Abdullah, Hang Ren, Haitham Bou Ammar, Vladimir Milenkovic, Rui Luo, Mingtian Zhang, and Jun Wang. Wasserstein robust reinforcement learning. arXiv preprint, 2019. arXiv:1907.13196
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Proceedings of the 94th International Conference on Machine Learning, volume 70, pp. 22-31. PMTRR 2017.
Fitan Altman. Constrained Markov Decision Processes. CRC Press, 1999.
Homanga Bharadhwaj, Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, and Animesh Garg. Conservative safety critics for exploration. In Ninth International Conference on Learning Representations, 2021.
Jose Blanchet and Karthyek Murthy. Quantifying distributional model risk via optimal transport. Mathematics of Operations Research, 44(2):565-60) -00, 2019. doi: 10. 111222/070.../ 2atooo..018.000 8.0366.
Lukas Brunke, Melissa Greeff, Adam W. Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and An- gela P. Schoellig. Safe learning in robotics: From learning-based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 5(1):411-44) 2022. doi: /atp://ucyooooootto-02-------0-0
Ruidi Chen and Toannis Ch. Paschalidis. Distributionally robust learning. Foundations and Trends6 in Optimization, 4(1-2):1 243, 2020. doi: 10. 11166/22400000066
Richard Cheng, Gábor Orosz, Richard M. Murray, and Joel W. Burdick. End-to-end safe reinforcement learning through barrier functions for safety-critich continuous control tasks. In Proceedings of the AAAI Conference Artificial Intellggnce, volume 33, pp. 3387- 3395. AAAT Press, 2019.
13

---

Published in Transactions on Machine Learning Research (03/2024)
Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint, 2018. arXiv:1801.08777
Esther Derman, Danicl J. Mankowitz, Timothy A. Mann, and Shic Mannor. Soft-robust actor-critic policy- gradient. arXiv preprint, 2018. arX.v:1803.04848.
Gabriel Dulac-Arolld, Nir Levine, Daniel J. Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. An empirical investigation of the challenges of real-world reinforcement learning. arXiv preprint, 2020. arXiv:2003.11881.
Gabriel Dulac-Arnold, Nir Levine, Daniel J. Mankowilz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-woot weal-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning, 110:2419 2468, 2021. doi:
Yousef Emam, Paul Glotfelter, 7solt Kira, and Magnus Egenstedt. Safe model-based reinforcement learning using robust control barrier functions, arXiv preprint, 2021. arXiv:2110.05415,
Matthew W. Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Nikola Momchev, Danila Sinopalnikov, Piotr Stañczyk, Sabela Ramos, Anton Raichuk, Damien Vincent, Léonard Hussenot, Robert Dadashi, Gabriel Dulao-Amold, Manu Orsini, Alexis Jacq, Johan Ferret, Nino Vieillard, Seyed Kamyar Seyed Ghasemipour, Sertan Girgin, Olivier Pietquin, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, Sarah Henderson, Abe Friesen, Ruba Haroun, Alex Novikov, Sergio Gómez Colmenarejo, Serkan Cabi, Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Andrew Cowie, Ziyu Wang, Bilal Piot, and Nando de Freitas. Acme: research framework for distributed reinforcement learning. arXiv preprint, 2020. arXiv:2006.00999.
Linfang Hou, Liang Pang, Xin Hong, Yanyan Lan, Zhiming Ma, and Dawei Yin. Robust reinforcement learning with Wasserstein constraint. arXiv preprint, 2020. arXiv:2006.0094
Garud N. Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2):257-20) 2005.
doi: 10.1287 moor. 1040.0122.
Yufei Kuang, Miao Lu, Jie Wang, Qi Zhou, Bin Li, and Houqiang Li. Learning robust policy against disturbance in transition dynamics via conserconsevv policy optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 7247-7254 2022.
Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Steven Wu, Bo Li, and Ding Zhao. Constrained variational policy optimization for safe reinforcement lcarning. In Proceedings of the 39th International Conference on Machine Learning, pp. 13644 13668. PMLR, 2022.
Zuxin Liu, Zijian Guo, Zhepeng Cen, Huan Zhang, Jie Tan, Bo Li, and Ding Zhao. On the robustness of safe reinforcement learning under observational perturbations. In Eleventh International Conference on Learning Representations, 2023a.
Zuxin Liu, Zijian Guo, Zhepeng Cen, Huan Zhang, Yihang Yao, Hanjiang Hu, and Ding Zhao. Towards robust and safe reinforcement learning with benign off-policy data. In Proceedings of the 40th International Conference on Machine Learning, pp. 21586 21610. PMLR, 2023b.
Haitong Ma, Jianyu Chen, Shengbo Eben, Ziyu Lin, Yang Guan, Yangang Ren, and Sifa Zheng. Model- based constrained reinforcement learning using generalized control barrier function. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4552-4559, 2021. doi: 10.1109/ TROS51 000...222...666600
Ajay Mandlekar, Yuke Zhu, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Adversarially robust policy learning: Active construction of physically-plausibb perturbations. In 2017 IEEE/RA REEE/SSJ/A International Conference on Intelligent Robols and Systems (IROS), pp. 3932 3939, 2017. doi: 10.1 10.1009/ROOS.201....22
14

---

Published in Transactions on Machine Learning Research (03/2024)
Daniel J. Mankowitz, Nir Levine, Rae Jeong, Abbas Abdolmaleki, Jost Tobias Springenberg, Yuanyuan Shi, Jackie Kay, Todd Hester, Timothy Mann, and Martin Riedmiller. Robust reinforcement learning for continuous control with model misspecification. In Eighth International Conference on Learning Representations, 2020.
Daniel J. Mankowitz, Dan A. Calian, Rae Jeong, Cosmin Paduraru, Nicolas Heess, Sumanth Dathathri, Martin Riedmiller, and Timothy Mann. Robust constrained reinforcement learning for continuous control with model misspecifictiinn. arXiv preprint, 2021. arXiv:2010.106:
Arnab Nilim and Laurent El Ghaoui. Robust control of Markov decision processes with uncertain transition matrices. Operalions Research, 53(5):780 798, 2005. doi: 11228//ppre.0500002
Santiago Paternain, Luiz Chamon, Miguel Calvo-Fullana, and Alejandro Ribeiro. Constrained reinforcement learning has zero duality gap. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 3803 3810, 2018. doi: 10.1 MAL1109/CCRA.01.1 201.855550.0.80505555...2...
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning- Tn Proceedings of the 34th International Conference on Machine Learning, volume 70, pp. 2817 2826. PMLR, 2017.
Muni Sreenivas Pydi and Varun Jog. Adversarial risk via optimal transport and optimal couplings. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pp. 7814-7823. PMLR, 2020.
James Queeney and Mouhacine Benosman. Risk-averse model uncertainty for distributionally robust safe reinforcement learning. In Advances in Neural Information Processing Systems, volume 36. Curran Associates, Inc., 2023.
Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. EPOpt: Learning robust neural network policies using model ensembles. In 5th International Conference on Learning Representations, 2017.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning, 2019.
Reazul Hasan Russel, Mouhacine Benosman, Jeroen Van Baar, and Radu Corcodel. Lyapunov robust constrained-MDDP: Soft-constrained robustly stable policy optimization under model uncertainty. arXiv preprint, 2021. arXiv:2108.0277
Krishnan Srinivasan, Benjamin Eysenbach, Sehoon Ha, Jie Tan, and Chelsea Finn. Learning to be safe: Deep RL. with safety critic. arXiv preprint, 2020. 2:X.v:2010.14603.
Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by PID Lagrangian methods. In Proceedings of the 37th International Conference Machine Learning, volume 119, pp. 9133-9143. PMLR, 2020.
Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applications in continuous control. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 6215 6224. PMLR, 2019a.
Chen Tessler, Daniel J. Mankowitz, and Shie Mannor. Reward constrained policy optimization. In Seventh International Conference on Learning Representations, 2019b.
Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E. Gonzalez, Julian Ibarz, Chelsca. Finn, and Ken Goldberg. Recovery RL: Safe reinforcement learning with learned recovery zones. IEEE Robotics and Automation Letters, 63)44415544 2021. doi: 10.1109/T.RA .307025..
15

---

Published in Transactions on Machine Learning Research (03/2024)
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 FEEE/RS International Conference on Intelligent Robots and Systems (IROS), pp. 23 30, 2017. doi: 10.1109/IROS,
0017.8202133.
Cédric Villani. Optimal transport, old and new. Springer, 2008. doi: 10. 1111007777-3-540--777
Eugene Vinitsky, Yuqing Du, Kanaad Parvate, Kathy Jang, Picter Abbeel, and Alexandre Bayen. Robust reinforcement learning using adversarial populations. arXiv preprint, 2020. arXiv:2008.01825.
Nolan Wagener, Byron Boots, and Ching-An Cheng, Safe reinforcement learning using advantage- baseedd intervention. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pp- 10630-10640. PMLR, 2021.
Huan Xu and Shie Mannor. Distributionally robust Markov decision processes. In Advances in Neural Information Processing Systems, volume 23. Curran Associates, Inc., 2010.
Mengdi Xu, Zuxin Liu, Peide Huang, Wenhao Ding, Zhepeng Cen, Bo Li, and Ding Zhao. Trustworthy reinforcement learning against intrinsic vulnerabilities: Robustness, safety, and generalizability, arXiv preprint, 2022. #:Xiv:2209000022
Tengyu Xu, Yingbin Liang, and Guanghui Lan. CRPO: A new approach for safe reinforcement learning with convergence guarantee. In Proceedinggs of the 38th International Conference on Machine Learning, pp. 11480 11491. PMLR, 2021.
Pengqian Yu and Huan Xu. Distributionally robust counterpart in Markov decision processes. Transactions on Automatic Control, (1()): :2538)258---2 -2538-2543, 2016. doi: 10.1109/TAC.2011 201.2/11....2.24220 2495174.
IEEE
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations state observations. In Advances in Newral Information Processing Systems, volume 33, pp. 21024 21037. Curran Associates, Inc., 2020.
A Proofs
In order to prove the tractable reformulation in Theorem will make of the following result.
Lemma Let Assumption hold. Then, we have
((((s,a)r(ss,p))) + sup E inf Vi((") + (ds,a -cs,a)
(17)
A s'oppnn
S'ES
F. = c(s, +
inf
E
sup
(=T - > (do. (8', -ESaa)
(18)
X
8'aDa.a
s'ES
Note that
inf Pr na s'mpae
E [V](6']]
=-
sup Ps.. EP
E [-V)
of PPoaa
Therefore, in this section we only prove results for the robust cost Bellman operator T Results related the robust reward Bellman operator Tp follow immediately by applying the same proofs after appropriate change in signs.
A.1 Proof of Lemma 1
Proof. Under Assumption note that Assumption (A1) and Assumption (A2) of Blanchet & Murthy (2019) are satisfied for the worst-case optimization problem over transition models in (6). Assumption (A1) is satisfied by our definition of optimal transport cost, and Assumption (A2) is satisfied by our Assumption
16

---

Published in Transactions on Machine Learning Research (03/2024)
Then, according to Theorem in Blanchet & Murthy optimization problem in 66 Therefore, we have that
2019
optimal transport strong duality holds for the
V (5)) (d -3)-ccaa)
sup E.P. p'pps
V [
inf X
sup sEES
By substituting this result into 6 arrive at the result in
18
A.2 Proof of Theorem 1
Proof. First, we show that 11 and the worst-case optimization problem over transition models in share the same dual problem. We write the dual problem to (11) as
(
inf sup E (g(((())) X geg s'mpasa
IE Samper
,8'''''))) 6s,a
inf sup A20 gEg 8--Pa.a
(o( A (d..a(''.g(8)) cs.a)l
Using the definition of G, we can rewrite this as
Ve (= (da,a(3', coa) -
inf E g'ap
sup dEES
(19)
X20
which appcars in the right-hand side of 18 from Lemma As shown in Lemma 19 is also the dual to the optimization problem in ( and optimal transport strong duality holds.
Next, we show that strong duality holds between and 19 Let A be the optimal dual variable in (19 and let
-h (8') E arg max SES
(2) - (do.a (5',')) c2,a)
We have that X and (8) exist according to Theorem I(b) in Blanchet Murthy 2019 along with Assumption and Iü characterizes the optimal transport plan v" that moves the probability of & under Ps.a to 2 J. By the complementary slackness results of Theorem 1(b) in Blanchet & Murthy (2019). also have that
X
E
6,,a 0.
81 8'mopon
Therefore,
inf
E
sup (s') - (de,,(('', csa)
E s'rppsc
(d - cc.a)]
X 8'mppsa
s'ES
E stopp
[VT
( (3'((()))) (3'))]
Moreover, by the primal feasibility of the optimal transport plan v", we have that
E StaPaae
dd (8')) S 6g,a
Sh is feasible solution to 11 with the same objective value the value of 19 Therefore, strong
duality holds between {Il) and 19 and Ih is optimal solution to 11 (i.e., optimal solution to [1]
exists). Then, for any optimal solution Lh to [1 we have that
E glaupsa
V.. ( (8')))
E
VV (
and the right-hand side of 18 is equivalent. to the right-hand side of
17

---

Published in Transactions on Machine Learning Research (03/2024)
Table 2: Safety constraints for all tasks
Safety
Task
Safety Constraint
Coefficient
Cartpole Swingup
Slider Position
0.30
Walker Walk
Joint Velocity
0.25
Walker Run
Joint Velocity
0.30
Quadruped Walk
Joint Angle
0.15
Quadruped Run
Joint Angle
0.30
Table Perturbation ranges for test environments across domains
Perturbation Parameter
Nominal Value
Domain
Test Range
Cartpole
Pole Length
1.00
(0.75, 1.25
Walker
Torso Length
0.30
[0.10, 0.50
Quadruped
Torso Density
1,000
[500, 1,500]
B Implementation details
B.1 Safety constraints and environment perturbations
We consider the same experimental design used in Queeney & Benosman (2023 to define our training and test environments. For each task, we consider single salety constraint defined in the Real-World RL Suile, which we summarize in Table 2] A policy incurs cost in the Cartpole domain when the slider moves outside of specified range, in the Walker domain when the velocity of any joint exceeds a threshold, and in the Quadruped domain when joint angles are outside of an acceptable range. The specific ranges that result in cost violations are determined by the safety coefficients in Table 2 which can take values in [0, 1] where lower values make cost violations more likely. See DuulacArnoll et al. (2021) for detailed definitions of the safety constraints we consider.
We evaluate the performance of learned policies across a range of test environments different from the training environment. We define these test environments by varying a simulator parameter in each domain across a range of values, which are listed in Table 3 We vary the length of the pole in the Cartpole domain, the length of the torso in the Walker domain, and the density of the torso in the Quadruped domain. Note that the parameter value associated with the nominal training environment in the center of the range of parameter values considered at test time. For each test environment, evaluate performance of the learned policies via 10 trajectory rollouts.
Finally, note that the domain randomization baselines consider range of environments during training. We consider the same training distributions for domain randomization as in Queeney & Benosman 2023 As summarized in Table in-distribution domain randomization applies a uniform distribution over the middle 50% of the parameter values considered at test time. For the out-of-distribution variant of domain randomization, we instcad consider uniform distribution over range of values for different simulator parameter than the varied at Lest time. See Table 4 for details.
B.2 Network architectures
In our experiments, we consider neural network representations the policy and critics. We consider networks with 3 hidden layers of 256 units and ELU activations, and we apply layer normalization followed by a tanh activation after the first layer as in Abdolmaleki al. (2020). We represent the policy as multivariate Gaussian distribution with diagonal covariance, where at given state the policy network outputs the mean
18

---

Published in Transactions on Machine Learning Research (03/2024)
Table 4: Perturbation paraneters and ranges for domain randomization across domains
Perturbation Parameter
Nominal Value
Training
Domain
Range
In- In-Distribution
Cartpole
Pole Length
1.00
(0.875, 1.125]
Walker
Torso Length
0.30
[0.20, 0.40]
Quadruped
Torso Density
1,000
[750, 1,250]
Distribtittiittiinn
Cartpole
Pole Mass
0.10
[0.05, 0.15]
Walker
Contact Friction
0.70
[0.40, 1.00]
Quadruped
Contact Friction
1.50
[1.00, 2.00]
pls) and diagonal covariance Σ(s) of the action distribution. The diagonal of ((s) is calculated by applying the softplus operator to the outputs the neural network corresponding to the covariance. In addition to the policy network, we consider separate networks for the reward and cost critics. We maintain target versions of the policy and critic nelworks using exponential moving average of the weights with T 5e-3. Finally, also consider neural networks for our perturbation networks ó, and de. In this work, we consider small networks with 2 hidden layers of 64 units and ELU activations. We clip the outputs in the range [-2cs, 2cs] for additional stability.
B.3 Algorithm hyperparameters
All of the experimental results in Section build upon the baseline safe RL algorithm CRPO (Xu et al., 2021 At every update, CRPO calculates the current value of the safety constraint based on batch of sampled data. If the safety constraint is satisfied for the current batch, it applies policy update to maximize rewards. Otherwise, it applics a policy update to minimize costs. In both cascs, we use the unconstrained RL algorithm MPO (Abdolmaleki et al. 2018) to calculate policy updates. MPO calculates a non-parametric target policy with KL divergence EKL from the current policy, and updates the current policy towards this target while constraining soparate KL divergence contributions from the mcan and covariance by 3n and fg, respectively. We apply per- per-dimension KL divergence constraints and action penalization using the multi-objective MPO framework (Abdolmaleki et al.) 2020) as in Hoffman et al. (2020). and we consider closed-form updates of the temperature parameter used in the non-parametric target policy as in Liu et al. (2022) to account for the immediate switching between objectives in CRPO. See Table 5 for all important hyperparameter values associated with the implementation of policy updates using MPO, and see Abdolmaleki et al. (0088 for additional details.
For our OTP framework, we update the perturbation networks alongside the policy and crities. We combine the perturbation function updates in (10) and [11 across state-action pairs by averaging over samples from the replay buffer, and we apply radius of ez. This leads to updates of the form
g6- arg min J. (s
E VV7 (gs(s,,, a, s'))]
s.t.
[d (6'...5(ss,,) a, 3'))] S Eg (s.a,8')<
E
g5. E arg max gf
IV (gs(s, a, (e.D
E
8')]] s.b.
(8', go(s, a, 8'))] S E
(6,0,3')
where Fs represents the class of perturbation functions with the form in 12 We consider da,a given by 77 Lo arrive at the perturbation network updates in (13) and [14). where es determines the average per-coordinate magnitude of the outputs of &, and 6,. We apply gradient-based updates on the Lagrangian relaxations of (33 and (14), and we also update the corresponding dual variables throughout training. See Table for hyperparameter values associated with our OTP framework.
19

---

Published in Transactions on Machine Learning Research (03/2024)
Table 5: Network architectures and algorithm hyperparameters used in experiments
General
Batch size per update
256
Updates per environment step
1
Discount rate (7)
0.99
Target network exponential moving average (r)
5e-3
Policy
Layer sizes
256, 256, 256
Layer activations
ELU
Layer norm + tanh on first layer
Yes
Initial standard deviation
0.3
Learning rate Non-parametric KL (<KL) Action penalty KL
1e-4
0.10
le-3
Action samples per update Parametric mean KL (3p,
20
0.01
Parametric covariance KL (8z)
le-5
Parametric KT. dual learning rate
0.01
Critics
Layer sizes Layer activations
256, 256, 256
ELU
Layer norm + tanh on first layer Learning rate
Yes
le-4
Optimal Transport Perturbations
Layer sizes
64, 64 ELU
Layer activations
Layer norm tanh on first layer
No
Output clipping
2cg,, 2cg]
Learning rate
1e-4
Dual learning rate
0.01
Per-coordinate perturbation magnitude (ex)
0.02
Finally, also implement adversarial RL, domain randomization, and RAMU using CRPO with MPO policy updates. The adversarial policy used in the PR-MDP framework is updated using MPO with the goal of maximizing costs. Using the default settings from Tessler et al. (0019a). we apply one adversary update for every 10 policy updates. Domain randomization considers the same updates as the CRPO baseline, but collects data from the range of training environments summarized in Table 4 RAMU applies the distribution over transition models and risk measure used in Queeney Benosman
2023
B.4 Computational resources
All experiments were run on a Limux cluster with 2.9 GHz Intel Gold processors and NVIDIA A40 and A100 GPUs. The Real- World RL Suite (Dulac-Arood al. 2020 2021) is available under the Apache License 2.0. Using code that has not been optimized for execution speed, each combination of algorithm and task
20

---

Published in Transactions on Machine Learning Research (03/2024)
Cartpole Swingup
Walker Walk
Walker Run
Quadruped Walk
Quadruped Run
1000
Total Reward
800
780
800
850
400
700
500
200
750
600
250
200
200
200
Total Cost
100
100
100
100
0.8
1.0
12
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.6
500
750
1000
1250
1600
500
750
1000
1250
1600
Pole Langth
Torso Length
Torso Length
Torso Density
Torso Density
Sale Rd
OTP
Adversorial RL. (10%)
Adverserial Rl. (5%)
Figure 6: Comparison with adversarial RL. Performance of adversarial RL is evaluated without adversarial interventions. Shading denotes one standard error across policies. Vertical dotted lines represent nominal training environment. Top: Total reward. Bottom: Total cost, where horizontal dotted lines represent the safety budget and values below these lines represent safety constraint satisfaction.
required approximately one day of wall-clock time on single GPU to train policies for 1 million steps across 5 random seeds.
C Additional experimental results
C.1 Detailed experimental results across test environments
In this section, we include detailed results across tasks and test environments for all algorithms listed in Table F compares compares safe RL and OTP to both variations of adversarial RL using different probabilities of adversary intervention. In general, adversarial RL with 5% intervention probability achieves similar total reward to OTP across all lasks, while the more adversarial 10% implementation leads to overly conservative performance. Both versions of adversarial RL generate strong, safe performance in the Quadruped tasks, while our OTP framework results in more consistent constraint satisfaction in the other 3 out of tasks. Importantly, our OTP framework accomplishes this without requiring adversarial interventions during data collection.
Figure shows the performance of domain randomization across tasks and environment perturbations. The grey shaded areas represent the ranges of the training distributions for the in-distribution implementation of domain randomization. We see that domain randomization leads to strong, robust performance in terms of rewards across all test cases, as well as improved constraint satisfaction in perturbed environments compared to standard safe RL with a single training environment. However, in tasks such as Walker Run and Quadruped Run, domain randomization does not. robustly satisfy safety constraints for test environments that were not seen during training. This issuc amplified in the case of out-of-distribution domain randomization. which does not demonstrale consistent robustness benefits compared to standard safe RL. In fact, it even leads to increase in musstraint-violatin test cases in Cartpole Swingup compared to safe RL. This demonstrates that training on multiple environments does not necessarily load to robust performance. Instoad, domain knowledge is critical in order for domain randomization to work well in practice.
Finally, Figure g shows the performance RAMU across tasks and test environments. The distributionally robust approach of RAMU leads to slight improvements in terms of rewards compared to OTP across most tasks and test environments, while our OTP framework delivers more robust safety constraint satisfaction in out of tasks. This highlights the different robustness guarantees associated with these frameworks.
21

---

Published in Transactions on Machine Learning Research (03/2024)
Cartpole Swingup
Walker Walk
Walker Run
Quadruped Walk
Quadruped Run
1000
Total Reward
600
800
800
850
700
600
400
200
600
200
200
200
Total Cost
100
100
100
100
0.8
1.0
12
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.6
500
750
1000
1250
1600
500
750
1000
1250
1600
Pole Langth
Torso Length
Torso Longth
Torso Density
Torso Density
Safe RL
OTP
Domain Randomization
Domain Randomization (ODD)
Figure 7: Comparison with domain randomization. Shading denotes standard error across policics. Grey shaded areas represent ranges of training distributions for in-distribution version of domain randomization. Vertical dotted lines represent nominal training environment. Top: Total reward. Bottom: Total cost, where horizontal dottod lines represent the safety budget and values below these lines represent safcty constraint satislaction.
Cartpole Swingup
Walker Walk
Walker Run
Quadruped Walk
Quadruped Run
1000
Tobal Roward
600
800
800
850
700
600
400
800
800
400
600
400
Total Cost
100
100
100
100
100
0.6
1250
1250
Polc Langth
Torso Length
Torsa Length
Torso Density
Torso Density
Safe RL
OTP
RAMU
Figure 8: Comparison with distributionally robust safe RL. Shading denotes one standard error across policies. Vertical dotted lines represent nominal training environment. Top: Total reward. Bottom: Total cost, where horizontal dotted lines represent the safety budget and values below these lines represent safety constraint satisfaction.
C.2 Comparison of safe reinforcement learning update methods
Our Optimal Transport Perturbations can be combined with many popular safe RL algorithms that perform policy updates according to ( by replacing standard Q functions with robust Q functions as (4) In our main experimental results, we combined our methodology with CRPO [Xu et al. 2021), which immediately switches between maximizing total reward and minimizing total cost depending on safety constraint satisfaction. To demonstrate the flexibility of our approach, in this section we also combine our OTP framework with a Lagrangian- based safe RL update method (Tessler et al. 2019b: Ray et al. 2019 Lagrangian- brasad methods
update the policy based on a Lagrangian relaxation of (2) given by
Q0th (s.) -B). -
min max E
E (a~
(7) - >
E Lawa((1ss)
(20)
A20
AND
SnD
where the Lagrange dual parameter is updated at slower scale throughout training. Similarly, this approach can be applied to the robust and safe policy update in 44. We apply this update method with
22

---

Published in Transactions on Machine Learning Research (03/2024)
Table 6: Comparison of safe RL update methods
Training % Safe
Deploy
Normalized Ave.#
Algorithm
% Safet
Reward
Cost
CRPO
Safe RL
97%* 98%
51%* 87%
1.00* 1.06
1.00*
ОТР
0.34
Lagrange
Safe RL
85% *
62% 92%
1.02 1.05
0.83*
ОТР
91 %
0.25
Percentage of time throughout training that policies satisfy the safety constraint in the nominal training environment, aggregated across all tasks.
Percentage of policies that satisfy the safety constraint across all tasks and test enviromments.
Normalized relative to the average performance of standard safe RL using CRPO for each task and environment.
Statistically significant difference (p < 0.05) compared OTP using paired t-test.
Cartpole Swingup
Walker Walk
Walker Run
Quadruped Walk
Quadruped Run
1.10
1.20
1.05
Norm, Reward
104
1.10
1.20
1.17
1.13
115
115
1.00
1.00
1.00
1.05
1.00
1.10
1.02
1.10
1.00
1.00
0.95
1,00
0.54
0.90
0.90
0.90
0.89
0.90
0.85
100
85%
100
16%
89%
100
92%
100
90%
100
65%
% Safe
80
69%
80
69%
80
80
80
55%
48%
47%
37%
Safe RL (CRPO)
OTP (CRPO)
Safe RI, (Lagrange)
OTP (Lagrange)
Figure 9: Performance summary by task for safe RL and OTP using different safety methods (CRPO and Lagrange), aggregated across lest, environments. Top: Average total reward, normalized relative to the average performance of standard safe RL using CRPO for each test environment. Bottom: Percentage of policies that satisfy the safety constraint across all test environments.
dual learning rate of 5e-6 to both standard safe RT and our OTP framework. Tn Table ① and Figure we compare these experimental results to the same aigorithms using CRPO. We also provide detailed results for the Lagrangian-barn based implementations across tasks and test environments in Figure 10
Importantly, we see the benefits of our OTP framework across both choices of update method (CRPO and Lagrange) in terms of robust performance and safety. In both cases, our OTP framework increases the average total reward across unseen test environments at deployment time compared to standard safe RL, and leads to significant improvement in safety at deployment time. As shown in Figure g the performance difference between standard safe RI and OTP is very similar across update methods. The only mcaningful difference occurs in Quadruped Walk, where the Lagrangian-based approach demonstrrtes stronger performance and safety. However, the slight improvements shown by the Langrangian- based updates at deployment time may be the result of incrcased unsafe exploration during training. Because the Lagrange dual parameter X is updated at slow scale throughout training, the Lagrangian relaxation in (20) can result in safety constraint violations during training. We in Table t that CRPO achieves higher levels of safety during
23

---

Published in Transactions on Machine Learning Research (03/2024)
Cartpole Swingup
Walker Walk
Walker Run
Quadruped Walk
Quadruped Run
Total Reward
1000
600
1000
650
800
980
700
600
400
960
600
200
200
200
Total Cost
100
100
100
100
1.0
12
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.6
500
750
1000
1250
1600
500
750
1000
1250
1600
Pole Langth
Length
Torso Longth
Torso Density
Torso Density
Safe RL. (Lagrange)
OTP (lagrang))
Figure 10: Comparison of OTP with standard safe RL across tasks and test environments using Lagrangian- based update method. Shading denotes one slandard error across policies. Vertical dotted lines represent. nominal training environment. Top: Total reward. Bottom: Total cost, where horizontal dotted lines represent the safety budget and values bolow these lincs represent safety constraint satisfaction.
the training process compared to a Lagrangian-based update method, as CRPO quickly reacts to safety constraint violations during training by directly minimizing total cost. Therefore, in settings where safety is also important throughout the training process, CRPO may be preferred over Langrangian Langrangian-based approach.
24


Reviews:
[{"review_id": "oUv3f9Bm67", "reviewer": "Reviewer_vzU7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n1.\tUsing optimal transport as uncertainty set in safe RL seems novel.\n2.\tThe writing flow of the paper is good, which provides the necessary content to help me understand the high-level idea and details.\n3.\tThe empirical implementation seems to be highly related to the theoretical analysis.\n\n**Weaknesses:**\n1. It seems that the proposed method uses different hyperparameters for different tasks. This might be a signal of weak generalizability or stability of the proposed method. An empirical analysis of the influence of the hyper-parameters (e.g., safety coefficient) could be more convincing.\n2. Missing related works and baselines: Liu, Zuxin, et al. \"Towards robust and safe reinforcement learning with benign off-policy data.\" International Conference on Machine Learning. PMLR, 2023.\n3. Since this paper focuses on safe RL, I assume that the experiments should be run on tasks that are highly related to risks, such as safety-gym or self-driving. The safety constraints used in this paper are mainly about joint angle and joint velocity, which may not be consistent with the binary cost setting in the formulation and are not very critical in real-world settings.\n4. This paper seems to share very similar content and structure with [1] in related work, problem formulation, empirical implementation, and experiments. Since these two papers propose fundamentally different methods, I am worried that there are too many overlaps between them. In addition, I don’t find [1] as a baseline in the experiments.\n\n[1] Risk-Averse Model Uncertainty for Distributionally Robust Safe Reinforcement Learning", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper tackles the robustness problem in safe reinforcement learning. The main contribution is that the authors propose to use the optimal transport uncertainty set, leading to an efficient implementation of the worst-case optimization problems. This paper provides both a theoretical analysis of the uncertainty set and an empirical implementation of a deep RL algorithm. Specifically, the empirical implementation introduces an additional component named Optimal Transport Perturbations to generate the perturbations.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Add more experiments on other benchmarks that are designed for Safe RL.\n2. Add more recent baselines as discussed in the weakness section.\n3. The authors need to explain the reason for sharing a similar structure and content with [1].\n\n[1] Risk-Averse Model Uncertainty for Distributionally Robust Safe Reinforcement Learning", "broader_impact_concerns": "There is no ethical concern as far as I know.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Bnryh2Fx0H", "reviewer": "Reviewer_qng5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- the proposed approach for uncertainty set quantification is simple to implement, and the paper shows how it can be added on top of one safe RL algorithm\n\n- the details in the paper and appendix are clear, and provide sufficient understanding of the implementation details in the experiments. \n\n- a major strength of the approach is that it doesn't require a simulator (unlike domain randomization) for improving robustness, and so in principle can be incorporated for hard to simulate real-world control problems\n\n- the paper is overall easy to follow, and the theorems introduced in the paper have clear assumptions, and detailed proof in the appendix. \n\nWeaknesses\n\n- the paper's experiment section claims \"Our Optimal Transport Perturbations can be combined with many popular safe RL algorithms, which is a benefit of our methodology\" however the results are shown only with respect to ONE existing safe Rl algorithm. So this claim is not fully substantiated. \n\n- the DM control simulation environments are very simple control problems and it is unclear whether the approach can be applied for improving safety (or robustness) in more complex and high dimensional problems (for example robot manipulation). In particular, it is unclear whether the approach significantly improves upon the baseline in terms of behavioral changes (e.g. the baseline is not able to solve a task at all i.e. ~0% success rate due to safety failures, while addition of the approach improves success by > 20%) \n\n- there isn't enough discussion about safety guarantees for the proposed approach - how much is worst case safety or average case safety improved compared to a baseline?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel way of quantifying uncertainty sets in deep RL, for improved robustness and safety. The key contribution of the paper is to define uncertainty sets at each time-step of a control problem, based on an optimal transport cost from a nominal state-transition model. The benefit of this uncertainty set quantification that data only needs to be collected in a nominal environment. Results across simulated mujoco environments show that the proposed approach can be incorporated with an existing safe RL algorithm to obtain lower costs compared to the baseline, and improved robustness compared to standard domain randomization.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Changes necessary for acceptance\n\n- please provide a detailed discussion (ideally with theorems) of quantifying safety / robustness of the approach. Currently, there isn't enough discussion about safety guarantees for the proposed approach - how much is worst case safety or average case safety improved compared to a baseline? \n\n- please provide experimental justifications for \"Our Optimal Transport Perturbations can be combined with many popular safe RL algorithms, which is a benefit of our methodology\" or quantify for what type of safe RL algorithms (for example those discussed in the related works) it will be non-trivial to apply the proposed approach \n\n\nChanges not necessary for acceptance\n\n- it will be good to experiment with slightly more complex control problems (ref to weakness above)", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IIFaLAWhqp", "reviewer": "Reviewer_dhZ4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n* The paper is carefully prepared\n* The approach of using optimal transport perturbations is interesting.\n\n**Weaknesses**\n* Only deterministic benchmarks are used without mentioning this limitation.\n* It remains unclear what a real application of the technique would look like.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper deals with safe and robust RL. However, safety is not understood here as in safe policy improvement, but in the sense of avoiding constraints.\nContinuous state and action spaces are considered and experiments are conducted on five deterministic benchmarks. The method is compared with a reference method and four other variants and shows good results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It bothers me that there are several statements in the introduction without a reference. In my opinion, a reference should be given for each statement as far as possible, or if it is an own finding, it should be pointed out where in the paper this finding is presented. I am referring to the sentences:\n\n> Deep reinforcement learning (RL) is a data-driven framework for sequential decision making that has demonstrated the ability to solve complex tasks, and represents a promising approach for improving real-world decision making.\n\n> As a result, techniques have been developed to incorporate both robustness and safety into deep RL.\n\n> Robust RL methods protect against worst-case transition models in an uncertainty set, while safe RL methods incorporate safety constraints into the training process.\n\n\nLikewise, I would expect `Parametric methods assume the ability to generate a range of training environments with a detailed simulator, while adversarial methods directly influence the data collection process by attempting to negatively impact performance` to provide references to methods that are meant by these statements.\n\n\nI find statement `These perturbations can be added to the training process of any safe RL algorithm to incorporate robustness to unknown disturbances, without harming performance during training or requiring access to a range of simulated training environments.` very daring. Can this really be claimed for EVERY method?\nLater it is written more harmlessly: `Our Optimal Transport Perturbations can be combined with many popular safe RL algorithms`\n\nShouldn't a reference to safe exploration be given at `A separate line of work focuses on the issue of safe exploration during data collection`, e.g. [1]?\n\nI find it unusual and confusing that only half a standard error is shown in Figure 4. Why?\n\n[1] A. Hans el al., Safe Exploration in reinforcement learning, 2008", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: qng5: "The paper has some interesting ideas like proposing a novel way of quantifying uncertainty sets in deep RL. After revision, most of the claims are well-quantified in the paper and I do not see any major issues with the work. The experimental results are not as exciting (simple DM control environments) and there is no experimental validation of combining the proposed optimal transport perturbations approach with multiple safe RL algortihms (this claim is only justified through arguments, not through experiments). However, despite these limitations, I think the paper meets the TMLR bar for acceptance."

vzU7: "All my concerns have been resolved in the response of the authors."

dhZ4: "My objections were minor or stylistic in nature and have now all been addressed by the authors, so that it is now a solid paper on an interesting topic."
Decision: Accept (poster)