Wasserstein Robust Reinforcement Learning
Authors: Mohammed Amin Abdullah
Hang Ren
Haitham Bou-Ammar
Vladimir Milenkovic
Rui Luo
Mingtian Zhang
Jun Wang
Venue:  (2020)
Keywords: Reinforcement Learning
Robustness
Wasserstein distance

Abstract: Reinforcement learning algorithms, though successful, tend to over-fit to training environments, thereby hampering their application to the real-world. This paper proposes $\text{W}\text{R}^{2}\text{L}$ -- a robust reinforcement learning algorithm with significant robust performance on low and high-dimensional control tasks. Our method formalises robust reinforcement learning as a novel min-max game with a Wasserstein constraint for a correct and convergent solver. Apart from the formulation, we also propose an efficient and scalable solver following a novel zero-order optimisation method that we believe can be useful to numerical optimisation in general. 
We empirically demonstrate significant gains compared to standard and robust state-of-the-art algorithms on high-dimensional MuJuCo environments

Under review as a conference paper at ICLR 2020
WASSERSTEIN ROBUST REINFORCEMENT LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Reinforcement learning algorithms, though successful, tend to over-fit to train- ing environments, thereby hampering their application to the real-world. This paper proposes WR - a robust reinforcement learning algorithm with signifi- cant robust performance on low and nigh-dimensional control tasks. Our method formalises robust reinforcement learning as max-min game with a constraint derived from Wasserstein distance, and we propose an efficient and scalable al- gorithm to solve it. Our algorithm applies zero-order optimisation method that we believe can be useful to numerical optimisation in general. Apart from the formulation, we also propose an efficient and scalable solver following a novel zero-order optimisation method that we believe can be useful to numerical opti- misation in general. We empirically demonstrate significant gains compared to standard and robust statc-of-the-ard algorithms on high- digennimnnsional MuJuCo en- vironments.
INTRODUCTION
Reinforcement learning (RL) has become standard tool for solving decision-making problems with feedback, and though significant progress has been made, algorithms often over-fit to train- ing environments and fail to generalise across even slight variations of transition dynamics (Packer) (et al. 2018 Zhao et al. 2019). Robustness to changes in transition dynamics is a crucial component for adaptive and safe RL in real-world environments. Motivated by real-world applications, recent literature has focused on the above problems. proposing a plethora of algorithms for robust decision- making (Morimoto & Doya, 2005 Pinto al. 2017 Tessler et al. 2019). Most of these techniques borrow from game theory to analyse, typically in a discrete state and actions spaces, worst-case de- viations of agents' policies and/or environments, see (Sargent & Hansen (200T); Nilim & El Ghaoui (005):[[yegaar )2005); Namkoong & Duchi (2016) and references therein. These methods have also been extended linear function approximators (Chow et al. 2015), and deep neural networks (Peng et al. 2017) showing (modest) improvements in performance gain across variety of disturbances, e.g., action uncertainties, or dynamical model variations. this paper, we propose a generic frame- work for robust reinforcement learning that can cope with both discrete and continuous state and actions spaces. Our algorithm, termed Wasserstein Robust Reinforcement Learning (WR*L., aims to find the best policy, where any given policy is judged by the worst-case dynamics amongst all candidate dynamics in a certain set. This set is essentially the average Wasserstein ball around a ref- erence dynamics Po. The constraints makes the problem well-defined, as searching dynamics can only result in non non-performing system. The measure of performance is the standard RL objective, the expected return. Both the policy and the dynamics are parameterised; the policy parameters O₂ may be the weights of a deep neural network, and the dynamics parameters Φ, the parameters of simulator differential equation solver. The algorithm performs estimated descent steps in Φ space and - after (almost) convergence performs an update of policy parameters, i.e., in space. Since Φ may be digh-dimensional, we adapt zero' 'th order sampling method based ex- tending [Slliaans et al. (2017) to make estimations of gradients, and in order to define the constraint set which Φ is bounded by, we generalise the technique to estimate Hessians (Proposition 2 We emphasise that although access to simulator with parameterisable dynamics is required, the actual reference dynamics Po need not be known explicitly nor learnt by our algorithm. Put another way, we are in the "RL setting", not the "MDP setting" where the transition probability matrix is known a priori. The difference is made obvious, for example, in the fact that we cannot perform dynamic programming, and the determination of a particular probability transition can only be estimated from sampling, not retrieved explicitly. Hence, our algorithm is not model- bbaed in the traditional sense of learning a model to perform planning.
arbitrary

---

Under review as a conference paper at ICLR 2020
We believe contribution is useful and novel for two main reasons. Firstly, framing of the robust learning problem is in terms of dynamics uncertainty sets defined by Wasserstein distance. Whilst not the first to introduce the Wasserstein distance into the context of MDPs (see, e.g., Yang 2017) or Lecarpentier & Rachelson (2019), we believe our formulation is amongst the first suitable for application to the demanding application-space we desire, that being, high-dimensional, continuous state and action spaces. Secondly, we believe our solution approach is both novel and effective (as evidenced by experiments below, see Section [5) and does not place a great demand on model or domain knowledge, merely access to simulator or differentiable equation solver that allows for the parameterisation of dynamics. Furthermore, it is not computationally demanding, in particular, because it does not attempt to build model of the dynamics, and operations involving matrices are efficiently executable using the Jacobian- product facility of automatic differen- tiation engines.
2
BACKGROUND
A Markov decision process (MDP is denoted by M A.. where S C Rd denotes the state space, A S Rd the action space. P S A describing the system's i dynamics, performance, and y [0, 1) specifies the degree which rewards are discounted over time.
1] is state transition probability S x A + R is the reward function measuring the agent's
At each time step f, the agent in state s; E S and must choose an action a, E .A, transitioning it to new state $ P (s,+ 1 | a:), and yielding reward R a,). A policy S x A - [0, 1] is defined as a probability distribution over state-action pairs, where π (ar (8))) represents the density of selecting action a, in state 8,. Upon subsequent interactions with the environment, the agent collects a trajectory T of state-action pairs. The goal is to determine optimal policy x* by solving:
arg max Ey-p, (T) [R)aaa(T]]
where Pr denotes the trajectory density function, and Rrom(T) the return, that is, the total aceu- mulated reward:
T-1
Pa (T( = pr (ao)s) II P(8 (1841188, (n) and RToul (T) = Σ R(s,, ar); (2)
with (-(-) denoting the initial state distribution.
We make use of the Wasserstein distance to quantify variations from reference transition density Po(.). The latter being a probability distribution,
may consider other divergences, such as Bellback-Leibler (KL) or total variation (TV). Our main intuition for choosing Wasserstein distance explained below, but we note that it has number of desirable properties: Firstly, it is symmetric well-defined for measures with
(W, = Wp(v, (v. (1), a property K-L lacks). Secondly, different supports (which K-L also lacks). Indeed, the Wasserstein distance is flexible in the forms of the measures that can be compared discrete, continuous or a mixture. Finally, it takes into account the underlying geometry of the space the distributions are defined on, which can encode valuable information. is defined as follows: Let A' be a metric space with metric d(.,:). Let C(X) be the space of continuous functions on X' and let (MX)) be the set of probability measures on X. Let μı, E MM')). Let K(U, v) be the set of couplings between μa, V: K(H,v) v) = {n E M(X x X); ((A, B) C x X, K(A X) = ((A), x - =B)((B)) (3)
That is, the set of joint distributions E M(X x X) whose marginals agree with μ and spectively. Given a metric (serving as a cost function) d(.,.) for X, the p'th Wasserstein distance Wp(,, v) for p 2 1 between μ and v/ is defined as:
W Wp(,,)) v) = min <KK((a))) Xxy
/
d(z,y) dr( v)
1/P
(4)
(in this paper, and mostly for computational convenience, we use = 2, though other values of applicable).
Please note that we present reinforcement learning with continuous states and actions. This allows us to easily draw similarities to optimal control as detailed later. Extending these notions to discrete settings is relatively forwir..
2

---

Under review as a conference paper at ICLR 2020
3 WASSERSTEIN ROBUST REINFORCEMENT LEARNING
The desirable properties of Wasserstein distance aside, our main intuition for choosing it is described thus: Per the definition, constraining the possible dynamics to be within an e-Wasserstein ball of a reference dynamics Po(-) means constraining it in a certain way. Wasserstein distance has the form mass x distance. If this quantity is constrained to be less than a constant e, then if the mass is large, the distance is small, and if the distance is large, the mass is small. Intuitively, when modelling the dynamics of system, it may be reasonable concede that there could be a systemic error or bias in the model, but that bias should not be too large. It is also reasonable to suppose that occasionally, the behaviour of the system may wildly different to the model, but that this should be a low low-probability event. If the model is frequently wrong by large amount, then there is in it. In a sense, the Wasserstein ball formalises these assumptions.
3.1 PROBLEM DEFINITION: ROBUST OBJECTIVES AND CONSTRAINTS
Due to the continuous nature of the state and action spaces considered in this work, we resort to deep neural networks to parameterise policies, which we write as (a(a)));), where 0 E Rd is set of tunable syper-parameters to optimise. For instance, these policies can correspond to multi- layer perceptrons for MuJoCo environments, d
to convolutional neural networks in case of high- dimensional states depicted as images. Exact policy details are ultimately application dependent and, consequently, provided in the relevant experiment sections.
In principle, one can similarly parameterise dynamics models using deep networks (e.g.. LSTM- type models) to provide one or more netion-conditionel future state predictions. Though appealing, going down this path led us agents that discover worst-case transition models which minimise training data but lack any valid physical meaning. For instance, original experiments we conducted on CartPole ended up involving transitions that alter angles without any change angular veloci- ties. More importantly, these effects became more apparent in high- ligh-dimenssonal settings where the number of potential minimisers increases significantly. It is worth noting that we are not the first to realise such an artifact when attempting to model physics-based dynamics using deep networks. Au- thors in (Lutter et al. 2019) remedy these problems by introducing Lagrangian mechanics to deep networks, while others (Koller et al., 2018 Chen al. 2018) argue the need to model dynamics given by differential equation structures directly.
Though incorporating physics- based priors to deep networks is an important and challenging task that holds the promise of scaling model-based reinforcement learning for efficient solvers, this paper we rather study an alternative direction focusing on perturbing differential equation solvers and/or simulators with respect to the dynamic specification parameters & E Rd Not only would such a consideration reduce the dimension of parameter spaces representing transition models, but would also guarantee valid dynamics due to the nature the simulator. Though tackling some the above problems, such direction arrives with new set of challenges related to computing gradients and Hessians of black-box solvers. Section we develop an efficient and scalable zero-order method for valid and accurate model updates.
Unconstrained Loss Function: Having equipped agents with the capability of representing poli- cies and perturbing transition models, we are now ready to present an unconstrained version of WR'L's loss function. Borrowing from robust optimal control, we define robust reinforcement learning as an algorithm that learns best-case policies under worst-case transitions:
max min E
(T) (Rrocal (T)
where pg (T) is a trajectory density function parameterised by both policies and transition models, ie.. 0 and Φb respectively:
(T) (ao()))
II
P at)
Ta(a
spees vector and diff, solver deep network
3

---

Under review as a conference paper at ICLR 2020
At this stage, it should be clear that our formulation, though inspired from robust optimal control, is, truthfully, more generic as it allows for parameterised classes of transition models without incor- porating additional restrictions the structure or the scope by which variations are executed
Constraints & Complete Problem Definition: Clearly, the problem in Equation 5 is ill-defined due the arbitrary class of parameterised transitions. To ensure well-behaved optimisation ob- jectives, we next introduce constraints to bound search spaces and ensure convergence to feasible transition models. For a valid constraint set, our method assumes access to samples from a reference dynamics model Po(-1 a),, and bounds learnt transitions in an e-Wasserstein ball around Po Po(-|s, a.), i.., the set defined as:
(6) where R is hyperparameter used to specify the "degree of robustness" in similar spirit to maximum norm bounds in robust optimal control. It is worth noting, that though we have ac- cess to samples from reference simulator, our setting is by means restricted model-based reinforcement learning in an MDP setting. That is, our algorithm operates successfully given only traces from Po accompanied with its specification parameters, e.g., pole lengths, torso masses, etc. more flexible framework that does not require full model learners.
We (P),), Po()) = (P&()) : w2 (Po(-1s, a), Po(-)s, S e, ((saa E x A},
Though defining a better behaved optimisation objective, the set in Equation 6 introduces infinite number of constraints when considering continuous state and/or actions spaces. To remedy this problem, we target relaxed version that considers constraint of average Wasserstein distance bounded by hyperparameter
- {Pp()): f(a0) (anaaW/W(P((s...,,(,(c()))))))d,
(alaeegge) (Po(), -
= (Po(()
a))] {c}
The sampling (s, a) in the expectation is done as follows: We sample trajectories using reference dynamics Po and policy ı that chooses actions uniformly random (uar). Then (s, a) pairs are sampled uar from those collected trajectories. For given pair (s, a), W2 (-s(-|8, a), Po (-18, a)) approximated through the empirical distribution: we use the state that followed (s, a) the col- lected trajectories
data point. Estimating Wasserstein distance using empirical data is standard, sec, c.g., Peyré et al. (2019), One approach which worked well in our experiments, was to assume given by deterministic functions plus Gaussian noise with diagonal convari- ance matrices. This makes estimation easier in high dimensions since sampling in each dimension is independent of others, and the total samples needed is constant factor of the number of dimensions. Gaussian distributions also have closed-form expressions for Wasserstein distance, given in terms of mean and covariance.
that the dynamics
We thus arrive at WR'L's optimisation problem allowing for best policies under worst-case yet bounded transition models:
Wasserstein Robust Reinforcement Learning Objective:
max min &
T (T) |Roeal (T)
s.t. E(s,a) [M) sc
(8)
3.2 SOLUTION METHODOLOGY
Our solution alternates between updates of θ and Φ, keeping fixed when updating the other. Fixing dynamics parameters Φ, policy parameters can be updated by solving maxg By-ng(r) (T)), which is the formulation of standard RL problem. Consequently, one can easily adapt any policy search method for updating policies under fixed dynamical models. As
'f course, allowed perturbations are ultimately constrained by the hypothesis space. Even then, our model is more general compared to robust optimal control that assumes additive, multiplicative, or other forms of disturbances.
4

---

Under review as a conference paper at ICLR 2020
described later in Section 4 make of proximal policy optimisation (Schulman et al. 2017 When updating Φ given a set of fixed policy parameters 8, the Wasserstein constraints must be re- spected. Unfortunately, even with the simplification introduced in Section 3.1 the constraint is still difficult to compute.
To alleviate this problem, we propose to approximate the constraint in (8) by its Taylor expansion up to second order. That is, defining (()) := (e [W (-1s, a), Po(-Is, a))]. the above can be approximated around Φo by a second-order Taylor as:
1 (())= = ((oo) W (Ø Φo) + ( Φo) (do) (Φ Φo).
0 (the distance between the same probability densities), and V (áb)) = 0 since Φy minimises (Φ)), we can simplify the Hessian approximation by writ- ing: (Ø)) ~ (6 ) (@))(() Φu). Substituting our approximation back in the original problem in Equation 8 we reach the following optimisation problem for determining model param- eter given fixed policies:
Recognising that (
min E (T) (Rooa((T) s.t. 2(0 Φ0) T Ho( - 60) S.
(9)
is the Hessian of the expected squared sb 2- Wasserstein distance evaluated Φo. Optimisation problems with quadratic constraints can be efficiently solved using interior-point methods. To do so, one typically approximates the loss with a first-order expansion and determines closed-form solution. Consider a pair of parameters O(k) and اbو (which will correspond to parameters of the j "tth inner loop of the k'th outer loop in the algorithm we present). To find ali+1), we solve:
where Ho 22 E (2,a)
Pa)
min V& (T) [Rroaa (T)]
(Ø وااه s.t. = To) -00)= S ce
1
014 (bbb))
It is easy to show that minimiser the above equation can derived in closed-form as:
ع Φ0
2c Ho '9" KJl
(10)
g (kJlT Ho
ik.
with V Vo EE E RRoaa T) p
[Koo] denoting the gradien
evaluated
0 and Lbl, i... gاaا
Having described the two main steps needed for updating policies and mod- els, we now summarise these findings in the pseudo-code in Algorithm As the Hessiar of the Wasserstein distance is evaluated based reference dynamics and any policy æ, pass it, along with and Φ as inputs. Then Algorithms opperates in a aasscent-ascent fashion in two main phases. In the first, lines 5 to 10 in Algorithm dynamics parameters solution in Equation [O] while ensuring that learning rates abide by a step size condition (we used the Wolfe conditions (Wolfe 1969), though it can be some other method). With this, the second phase (line 11) utilises any state-of-the-art reinforcement learning method to adapt policy parame- ters generating g g Regarding the termination condition for the inner loop, we leave this as a decision for the user. It could be, for example, large finite time-out, the norm of the gradient (k.j) being below threshold, or whichever happens first
Generic Algorithm:
updated using the closed-form
RReaark Although this looks superficially similar to an approximation made in TRPO (chulman et al, 2015a). the latter aims to optimise the policy parameter rather than dynamics. Furthermore, the constraint based the Kullhack-I.eihle divergence rather than the Wasserstein distance
'Please note our algorithm does not need to store the Hessian matrix. In line 7 of the algorithm. it is clear that require Hessian-vector products. These can be easily computed using computational graphs without having access the full Hessian matrix.
5

---

Under review as a conference paper at ICLR 2020
Algorithm 1: Wasserstein Robust Reinforcement Learning
Inputs: Wasserstein distance Hessian, Ho evaluated at do under any policy x, radius of the Wasserstein ball and the reference simulator specification parameters Φo
2: Initialise Φl with Φo and policy parameters (1") arbitrarily
3: for k ===07 0,
do
4: x=!] f- @ and +
5: Phase I: Update model parameter while fixing the policy:
6: while termination condition not met do
7:
Compute descent direction for the model parameters as given by Equation TO
pb1 t Ø0-
2e g (ksll H gl H g [hd)] 1 -x
8:
Update candidate solution, while satisfying step size conditions (see discussion below) on the learning rate a: xbi+|| <- xli + aplil and j + j++1
9:
end while
10:
Perform model update setting & |++1]
11:
Phase I: Update policy given new model parameters:
12:
Use any standard reinforcement learning algorithm for ascending in the gradient direction,
e.g., g[++1] + ( + 3 Vo (op((T)
[R) lewn) s00+1, with g(M) a learning rate.
13: end for
4 ZERO' TH ORDER WASSERSTHIN ROBUST REINFORCEMENT LEARNING
Consider simulator (or differential equation solver) S. for which the dynamics are parameterised by aeea vector Φ and for which we can execute steps of trajectory (i.e., the simulator takes as input an action and gives back successor state and reward). For generating novel transitions, one can simply alter & and execute the instruction in S,, from some a state 8 E S, while applying action E A. Not only does this ensure valid (under mechanics) transitions, but also promises scalability as specification parameters typically reside lower dimensional spaces compared to the number of tuneable weights when using deep networks as transition models.
Gradient Estimation: Recalling the update rule Phase I of Algorithm I we realise the need for, estimating the gradient of the loss function with respect to the vector specifying the dynamics of
environment, i.e., g V V (T (T))
at each iteration of the inner-loop j.
Handling simulators as black-box models, we estimate the gradients by sampling from a Gaussian distribution with mean ( and σ² co-variance matrix. Our choice for such estimates is not arbitrary but rather theoretically grounded as one can easily prove the following proposition: Proposition 1 (Zero-Order Gradient Estimate). For a fixed O and the gradient can be computed as:
V V (T) [Rioar(T)] = o² EE-.voo. E a+ (T)Reuua(
( (dT
Hessian Estimation: Having derived a zero-order gradient estimator, we now generalise these notions to form allowing us to estimate the Hessian. It is also worth reminding the reader that such a Hessian estimator needs to be performed one time only before executing the instructions in Algorithm (i.e., H, is passed as an input). Precisely, we prove the following proposition:
Proposition 2 (Zero-Order Hessian Estimate). The hessian of the Wasserstein dis- tance around Do can be estimated based on function evaluations.
Recalling that
Ho V3, E (s.))a(()) () [W2 (Po(-ls. Po(-ls, x=
and defining ((na)(

---

Under review as a conference paper at ICLR 2020
W3 (Po(1ss,a) (a(lsss,),PP(lss,..o we prove:
Ho oz 2
g2 ( (E(e.))77))))3
MMx.a) (0o + +1)1 ET
B ((aa))a()))00 (Me. + E) I
Proofs of these propositions are given in Appendix A They allow for a procedure where gradi- ent and Hessian estimates can be simply based on simulator value evaluations while perturbing ø and Φu- It important to note that in order to apply the above, we are required be able to (Vee,a)(oo) under random & perturbations sampled from N( o² I). An empirical estimate of the p- p-Wasserstein distance between two measures μ and can be per- formed by computing the p-Wasserstein distance between the empirical distributions evaluated at sampled data. That is, one can approximation μ by Unn = 1 and independently distributed according to μ.. Approximating in similarly, we then realise tha W2(,,1) se W2(pn, Vn).
evaluate (a)a))=(t)))
n Σ 5z, where æ; are identically
EXPERIMENTS & RESULTS
We evaluate W on a variety of continuous control benchmarks from the MuJoCo environment. Dynamics in our benchmarks were parameterised by variables defining physical behaviour, e.g., density of the robot's torso, friction of the ground, and so on. We consider both low and high dimen- sional dynamics and demonstrate that algorithm outperforms state-of-the-arr from both standard and robust reinforcement learning. We are chiefly interested in policy generalisation across environ- ments with varying dynamics, which we measure using average test returns on novel systems. The comparison against standard reinforcement learning algorithms allows us to understand whether lack of robustmess is critical challenge for sequential decision making, while comparisons against robust algorithms test if we outperform state-of of-thee-rtt that considered similar setting to ours. From stan- dard algorithms, we compare against proximal policy optimisation (PPO) (Schulman et al., 2017) and trust region policy optimisation (TRPO) (Schulman et al.) 2015b); an algorithm based on natural actor-crtic (Peters& Schaal 2008 Pajarinen et al. 2019). From robust algorithms, we demonstrate how WR favours against robust adversarial reinforcement learning (RARL) (Pinto et al. 2017), and action-perturbed Markov decision processes (PR-MDP) proposed in (Tessler et al. (019). Due to space constraints, the results are presented in Appendix B.2 It is worth noting that we attempted to include deep deterministic policy gradients (DDPG) (Silver et al.) (014) in our comparisons. Results including DDPG were, however, omitted it failed show any significant robustness per- formance even on relatively simple systems, such as the inverted pendulum; see results reported in Appendix B.3 During initial trials, we also performed experiments parameterising models using deep neural networks. Results demonstrated that these models, though minimising training data er- ror, fail to provide valid models that vary pole angles without exerting any angular speeds. This problem became even more apparent in high-dimensional systems, c.g., Hopper, Walker, etc due to the increased number of pos- sible minima. As such, results presented in this section make use of our zero-order method that can be regarded as scalable alternative for robust solutions.
dynamics. For instance, we arrived at inverted pendula
5.1 MUJoCo BENCHMMRRSS
We evaluate our method both in low and high-dimenssonal MuJuCo tasks (Brockman et al. 2016 We consider a variety of systems including CartPole, Hopper, and Walker2D; all of which require direct joint-torque control. Keeping with the generality of our method, we utilise these frameworks as-is with no additional alterations, that is, we use the exact setting of these benchmarks as that shipped with OpenAl gym without any reward shaping, state-space augmentation, feature extraction, or any other modifications of-that-sort. Details are given in section B
'In case the dynamics are assumed to he Gaussian, similar procedure can be followed or a closed form be used, see Takatsu 2008).
7

---

Under review as a conference paper at ICLR 2020
(a) 0 (PPO)
(b) 0.003
(c) < 0.015
(d) PPO mcan
1922
3000
3400
2009
2009
30
17 21 24
12
33
2.4
11 23
(e) < 0 (PPO)
(f) 0.1
(g) 1.0
(h) PPO mean
von
400
400
4500
400
300
310
06
19
19
02
50 100 15W 2009 35W
5000
50
100h 15W 2000
35
5000
50 100 33 2001
35
500
50
1000 35 2009 25W
500
(i) c = 0.0 (PPO)
(i c == 0.005
(k) c = 0.03
(I) PPO mean
Figure 1: Dynamics variations in two dimensions and with different e, for Hopper (top row), Walker (middle row), and HalfCheetah (bottom row). Fourth column is PPO trained with dynamics sampled uar from the Wasserstein ball. Darker is better, with the numbers in the graded scale being the range of scores that the tasks can generate.
Due to space constraints, results for one-dimensional parameter variations are given in Appendix B.2 where it can be seen that LR outperforms both robust and non-robust algorithms when one- considered. Figure [ shows results for dynamics variations
dimensional simulator variations along two dimensions. Here again, our methods demonstrates considerable robustness. The fourth column, "PPO mean". refers to experiments where PPO trained on dynamics sampled uniformly at random from the Wasserstein constraint set. It displayes more robustness than when trained on just the reference dynamics, however, as can be seen from Fig. 2 our method performs noticably better in high dimensions, which is the main strength of our algorithm.
Results with High- Digg--iimnsionnl Model Variation: Though results above demonstrate robust- ness, an argument against a min-max objective can be made especially when only considering low- dimensional changes in the simulator. Namely, one can argue the need for such an objective as opposed to simply sampling a set of systems and determining policies performing-well on average similar to the approach proposed in (Rajeswaran et al. 2017).
A counter- arguumnt the above is that gradient- based optimisation scheme is more efficient than a sampling-based one when high-dimensional changes are considered. In other words, a sampling procedure hardly applicable when more than few parameters are altered, while WR*L can re- main suitable. To assess these claims, we conducted two additional experiments on the Hopper and HalfCheetah benchmarks. In the first, we trained robustly while changing friction and torso densi- ties, and tested on 1,000 systems generated by varying all 11 dimensions of the Hopper dynamics, and 21 dimensions of the HalfCheetah system. The histogram Figures b and (f) demonstrate that the empirical densities of the average test returns are mostly centered around 3,000 for the Hopper, and around 4,500 for the Cheetah, which improves that of PPO trained on reference (Figures a(a) and (e)) with return masses mostly accumulated at around 1,000 in the case of the Hopper and almost equally distributed when considering HalfCheetah. Such improvements, however, can be an artifact of the careful choice of the now-dimensional degrees of freedom allowed to be modified during Phase I of Algorithm [I To get further insights, Figures ((c) and (g) demonstrate the effectiveness of our method trained and tested while allowing to tune all 11 dimensional parameters of the Hopper sim-
8

---

Under review as a conference paper at ICLR 2020
1.00
A000
6.00
20 optede
(a) PPO Test High
(b) Train Low Test High
(c) Train High Test High
(d) PPO mean
(c) PPO Test High
(f) Train Low Test High
(g) Train High Test High
(h) PPO mean
Figure 2: Results evaluating performance when considering high dimensional variations
Hopper (top row) and HalfCheetah (bottom row) environment. All figures show the empirical distribution of returns on 1,000 testing systems. The first column demonstrates the robustness of PPO when trained on the reference dynamics and trained with high dimensional variations. The second column reports empirical test returns of WR-L's policy trained on only two parameter changes (e-g., friction and density) of the environment but tested on systems with all high-dimensional dynamical parameters modified. third column trains and tests WR'1 altering all dimensional parameters of the simulator. The fourth column reports results when PPO is trained on dynamics chosen uar and tested with high dimensional variation.
ulator, and the dimensions of the HalfCheetah. Indeed, our results are accordance with these of the previous experiment depicting that most of the test returns' mass remains around 3,000 for the Hopper, and improves to accumulate around 4,500 for the HalfCheetah. Interestingly, however, our algorithm is now capable of acquiring higher returns on all systemg since it is allowed to alter all parameters defining the simulator. As such, we conclude that WR outperforms others when high-dimensiooal simulator variations are considered. In Figures d)) and (h), PPO trained with dynamics sampled uar from the Wasserstein constraint set. We see that although in the two-dimensional variation case this training method worked well (see [igures[dd) (h), (l)) does not scale well to high dimensions, and our method does better.
the results for
6 RELATED WORK
Previous work on robust MDPs, e.g., Iyengar (2005; Nilim & El Ghaoui (2005); Wiesemann et al.) (2013); Tirinzoni al. (0018) Petrik & Russell (2019), whilst valuable in its own right, is not sufficient f'or the RL setting due to the need in the latter case to give efficient solutions for large state and action spaces, and the fact that the dynamics are not known a priori.
Closer to our own work, Rajeswaran et al. (2017) approaches the robustness problem by training on an ensemble of dynamics order to be deployed on a target environment. The algorithm introduced, Ensemble Policy Optimisation (EPOpt), alternates between two phases: (i) given distribution over dynamics for which simulators (or models) are available (the source domain), train policy that performs well for the whole distribution; (ii) gather data from the deployed environment (target domain) to adapt the distribution. The objective is not max-min, but a softer variation defined by conditional value-at-risk (CVaR). The algorithm samples set of dynamics {Φ+) ) from distribution over dynamics Pæ. and for each dynamics di. samples a trajectory using the current policy parameter 0,- It then selects the worst performing e-fraction of the trajectories to use to update the policy parameter. Clearly this process bears some resemblance to our algorithm, but there is crucial difference: when the dynamics parameters sit in a high-dimensional space, since in that case, optimisation- comm-smpling could demand considerable number samples. In any case, our experiments demonstrate our algorithm performs well even in these high dimensions. We note that we were
algorithm takes descent steps in the space. The difference if important
'Please note that we attempted to compare against Rajeswaran et al. 2017). Due to the lack of open-source code, we able to regenerate their results.

---

Under review as a conference paper at ICLR 2020
were unable to find the code for this paper, and did not attempt to implement it ourselves. The CVaR criterion is also adopted in Pinto et all (2017), in which, rather than sampling trajectories and finding quantile in terms of performance, two policies are trained simultaneously: "protagonist" which aims to optimise performance, and an adversary which aims to disrupt the protagonist. The protagonist and adversary train alternatively, with one being fixed whilst the other adapts. We made comparisons against this algorithm in our experiments. More recently. Tessler et al. (2019) studies robustness with respect to action perturbations. There are two forms of perturbation addressed: (i) Probabilistic Action Robust MDP (PR-MDP), and (ii) Noisy Action Robust MDP (NR-MDP). In PR-MDP, when an action taken by an agent, with probability a, a different, possibly adversarial action is taken instead. In NR-MDP, when action is taken, perturbation is added to the action itself. Like Rajeswaran et al. (2017) and Pinto et al. (017)) the algorithm is suitable for applying deep neural networks, and the paper reports experiments on InvertedPendulum. Hopper, Walker2d and Humanoid. We tested against PR-MDP in some of our experiments, and found it to be lacking in robustness (sce Section [5). In Lecarpentier & Rachelson (2019) non-stationary Markov Decision Process model is considered, where the dynamics can change from one time step to another. The constraint is based Wasserstein distance, specifically, the Wasserstein distance between dynamics at time and l' is bounded by Lit- i.e., is L-Lipschitz with respect to time, for some constant L. They approach the problem by treating nature as an adversary and implement a Minimax algorithm. The basis of their algorithm is that due to the fact that the dynamics changes slowly (due to the Lipschitz constraint), a planning algorithm can project into the future the scope of possible future dynamics and plan for the worst. The resulting algorithm, known as Risk Averse Tree Search, is as the name implies tree search algorithm. It operates on a sequence "snapshots" of the evolving MDP, which are instances of the MDP at points in time. The algorithm is tested on small grid world, and does not appear to be readily extendible to the continuous state and action scenarios our algorithm addresses. To summarise, our paper uses the Wasserstein distance for quantifying variations in possible dynamics, in common with Lecarpentier & Rachelson (2019), but suited to applying deep neural networks for continuous state and action spaces. Our algorithm does not require a full dynamics available to it, merely a parameterisable dynamics. It competes well with the above papers, and operates well for high dimensional problems, as evidenced by the experiments.
7 CONCLUSION & FUTURE WORK
In this paper, we proposed a robust reinforcement learning algorithm capable of outperforming oth- ers in terms of test returns on unseen dynamics. The algorithm makes use of Wasserstein constraints for policies generalising across varying domains, and considers a zero-order method for scalable solutions. Empirically, we demonstrated superior performance against state-of-the-art from both standard and robust reinforcement learning on low and high-dimensional MuJuCo environments. In future work, we aim to consider robustness in terms of other components of MDPs, e.g., state repre- sentations, reward functions, and others. Furthermore, we will implement L on real hardware, considering fim-to-real experiments.
REFERENCES
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In S. Bengio, H. Wallach, H. Larochelle. K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. Curran Associates, Inc., 2018. URL http:://apers: nips.cc/paper/ pdf
Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decision-making: a evar optimization approach. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Gamett (eds.), Advances in Neural Information Processing Systems 28, pp. 1522-1530. Curran Associates, Inc., 2015. URL http:://pappppp nips ctc.ppscccpppppe wtwpraiiiiiennnnrrtrtt--e-ee-e 6014-risk 1114-risk-sensitae and robust www.ddeiiiiiioo ott.aaaeee.aaaaaaaaaaaaaaaaaaaaa annppppocc.......chh.......
pdf
10

---

Under review as a conference paper at ICLR 2020
Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2): 257-280, 2005.
Torsten Koller, Felix Berkenkamp, Matteo Turchetta, and Andreas Krause. Learning-based model predictive control for safe exploration and reinforcement learning. CoRR, abs/ htss/1803.08887, 2018.
URL http::/arxxi... org/ abs/1803.... 08287
Erwan Lecarpentier and Emmanuel Rachelson. Non-stationary markov decision processes worst-
case approach using model-based reinforcement learning. arXiv preprint 19Xiv:9904.1000
2019.
Michael Lutter, Christian Ritter, and Jan Peters, Deep lagrangian networks: Using physics as model prior for deep learning, 07 2019.
Jun Morimoto and Kenji Doya. Robust reinforcement learning. Neural Comput., 17(2):335-359, February 2005. ISSN 0899-7667. doi: 10. 1000669977776601111222 URL http::/dx.do. org/ 10. /00.5//08390 0T....1116.209997751011111111110
Hongseok Namkoong and John C. Duchi. Stochastic gradient methods for distributionally robust optimization with f-divergences. I Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp.
USA, 2016. Curran Associates Inc.
ISBN 978-1-5108-3881-9 URL http: //dl: acm. wwt.://.ct.......cmmmmm...m
Yurii Nesterov. Random gradient-free minimization of convex functions. CORE Discussion Pa- pers 201 1001, Université catholique de Louvain, Center for Operations Research and Economet- rics (CORE), 2011. URL
aepesconPPpeer..om repec. wwt//eeE:::ooooooooooooooe cor: : louvco:
2011001
Arnab Nilim and Laurent El Ghaoui. Robust control of markov decision processes with uncertain transition matrices. Operations Research, 53(5):780-778, 2005.
Charles Packer, Katelyn Gao, Jemej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforcement learning. CoRR, abs/1810.120 12282, 2018. URL
http:://axivv: org/ abs/ /bbs/1810..m 12282
Joni Pajarinen, Hong Linh Thai, Riad Akrour, Jan Peters, and Gerhard Neumann. Compatible natural gradient policy search. CoRR, abs/1902.02823,, 2019. URL http: http://axxvvvorro.
abs/1902. 02823
Xue Bin Peng, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. CoRR, abs/1710.06537, 2017. URL http:
7/ arxiv. org/ wttp:o/o//bb//11.0003333
Jan Peters and Stefan Schaal. Natural actor-critic. Neurocomput. 7107-9):1180-11990 March 2008. ISSN 0925-2312. doi: 10. neccom..uuc.
URL httt:://dx..c httt:://dx.doi.ozg/111
101 1016/1. newcom. 2007 11.1111.. 026
Marek Petrik and Reazul Hasan Russell. Beyond confidence regions: Tight bayesian ambiguity sets for robust mdps. arXiv preprint
2019.
Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport. Foundations and Trendy in Machine Learning. 1((56))335-6677 2019.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learning Research, pp. 2817-2826, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. URL igg mm www.srrsssseo... v 7 / o 17a.oo77..... html
Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learning robust neural network policies using model ensembles. International Conference on Learning Representations (ICLR) 20/7, arXiv preprint arXiv: 160.0226 2017.
11

---

Under review as a conference paper at ICLR 2020
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies scalable alternative to reinforcement learning. arXiv preprint arXiv: 2017.
Thomas Sargent and Lars Hansen. Robust control and model uncertainty. American Economic Review, 91(2):60 66, 2001. URL https: apers rwwec. Brbpee..org/RPP..... aea: secreviv: 91 2001 :::::::::: 60-66
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1889-1897, Lille, France, 07-09 Jul 2015a, PMLR. URL htll:t::ppppcccednnng/ press/v37 press/v37/schulmaan 5tmml
John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Picter Abbeel. Trust region policy optimization. CoRR, abs/1502.054777, 2015b.
05477
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. Proceedings of the 31st International Confer- International Conference Machine Learning Volume 32, ICML'14, pp. 1-387- 1-395. JMLR.org, 2014. URL http:/ /dlp://dl... /alp:///dllcc.... wwtpciiaaiiinno.o////////////ott htt.:ii:....mmmmd000110e 3044850
Asuka Takatsu. Wasserstein geometry of gaussian measures, 2008.
Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applica- tions in continuous control. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Long Beach, California. USA, 09-15 Jun 2019. PMLR. URL Learming Research, pp. www. press / www.ttssseeell........ 2ttp://mm.....22222///2/////1110 html http://proceedingg mlr
Andrea Tirinzoni, Marek Petrik, Xiangli Chen, and Brian Ziebart. Policy condiiecod uncertainty sets for robust markov decision processes. In Advances in Neural Information Processing Systems, pp. 9939-8949, 2018.
Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem. Robust markov decision processes. Mathe- matics of Operations Research, 38(1):153-183) 2013.
Philip Wolfe. Convergence conditions for ascent methods. SIAM review, 11(2)226-2335 1969.
Insoon Yang. A convex optimization approach to distributionally robust markov decision processes with wasserstein distance. IEEE control systems letters, 1(I):164-169, 2017.
Chenyang Zhao, Olivier Sigaud, Freck Stulp, and Timothy M. Hospedales. Investigating gen- cralisation in continuous deep reinforcement learning. CoRR, abs/1902.07015, 2019. URL http://arxiv. wttp/ogggg//////eeeeee abs/1902.07015
A PROOFS
Proof of Proposition 7 The proof of the above proposition can easily be derived by combining the lines of reasoning in (Salimans al. 2017 Nesterov 2011). while extending the parameterisation of dynamical models. To commence, begin by defining To(Φ) ) B T [R)aa(TT] for fixed policy
parameters O. Given any perturbation vector, & _N(0, 2T), we can derive (through a Taylor expansion) the following:
To + E) Jo( (VV)))+"V
V VV)oo) +
+ O (higher-order terms).
12

---

Under review as a conference paper at ICLR 2020
Multiplying by &, and taking the expectation both sides of the above equation, get:
2g (TJo( + E) = (0 (Jo(o) + V 2 V5ETV300101
= =oV
Dividing by o², we derive the statement of the proposition as:
1 -/2 Be (EJo( E) 0²
VoJo(Ø)
Proof of Proposition Commencing with the right-hand-side of the above equation, we perform second-order Taylor expansions for each the terms under under the expectation of &. Namely, we write:
1 RE-N100021) 0² ((aa)))((0) (Ma,a) (Øo &)] E*
Hn oo
(11)
(s.a))a())) () [Maaaa E) I E
E (0. (Mo)))((O)) EET + EEE V Vol (s.) (Bo)) &T BE-N
(Mo) &I + c 22 (..))~())))
1 Be-N(O.002) σ²
B Blaapr [Maa)((o))]] I V
(6o))
(a)))))))))
+ 5 v
2 o
(sa)
( EI
Now, we analyse each of the above terms separately. For ease of notation, we define the following variables:
g VoB (A,)) [M))n))
VALE H (s.a)~=())) [Mn.a)(ou)]
A T g
B EE HEE
c < H..
Starting with A, we can easily see that any (i,j) j) component can be written as A -dd EiE Therefore, the expectation under & NV0 o² I) be derived
EE [E,E,n.g,l gn -.
if = j and otherwise.
Thus, we conclude that Ee~N( 21) [A] Odg
Continuing with the second term, i.e., B, we realise that any (i,j) component can be written as Bij Dde Id E.E Hm. Now, we consider two cases:
Diagonal Elements (i.e., when i j): The expectation under g ~ NN(0, can be fur- ther split in three sub-cases
Sub-Case I when i = j m m
We have Be-N(0.022 Em. Hm,]]
H,, (e,,(.0) 30 Hi,i-
-Su--uasse II when i = = n.
We have Ee-N(o,00)) (E,E, H Hm.l
H mn Bg
E3e2 d mm
13

---

Under review as a conference paper at ICLR 2020
Sub-Cast Case III when indices are all distinct: We have Eg-:V(,,02 E Hm,
Diagonal Elements Conclusion: Using the above results we conclude that Be-N(o,- 20 H, ortrace(H))
Off- Off-Diagonal Elements (i.e., when i j): The above analysis is now repeated for comput- ing the expectation of the off- off-diagonal elements of matrix B. Similarly, this can also be split into three sub-cases depending on indices:
Sub-Case when i m j n
We have Eg-vV(,,21) ( Hm.al
[T5) = =o²H., Hig.
Sub-Case II when i = n + m: We have BE-A(O.0=1) En H Hmml
H He, [<]]]] o H
Sub-Cass III when indices are all distinct: We have Ee- Em Hm,l 0.
Off- Off-Diagonal Elements Conclusion: Using the above results and due to the sym- metric properties of H, we conclude that (g,,N((,021) [B,y] 20 Hij
Finally, analysing c, one can realise that =e= = Ee-N((,,2)) H.).
Ede Egen EiE
Substituting the above conclusions back in the original approximation in Equation II and using the linearity of the expectation casily achieve the statement of the proposition.
B FURTHER DETAILS ON EXPERIMENTS
For clarity, we summarise variables parameterising dynamics in Table [ and detail specifies next.
The goal of this classic control benchmark is to balance a pole by driving a cart along rail. The state space is composed of the position æ and velocity è of the cart, well the angle 0 and angular velocities of the pole å We consider two termination conditions in experiments: 1) pole deviates from the upright position beyond a pre-specified threshold, or 2) cart deviates from its zeroth initial position beyond certain threshold. To conduct robustness experiments, we param- eterise the dynamics of the CartPole by the pole length Ip. and test by varying I, E [0.3,3).
CartPole:
this benchmark, the agent is required to control a hopper robot to move forward without falling. The state of the hopper is represented by positions, {z, 2). and linear velocities, {ē, ú, E}. of the torso in global coordinate, well angles, {0.) and angular speeds, {0:)²_ of the three joints. During training, we exploit an early-stopping scheme if "unhealthy" states of the robot were visited. Parameters characterising dynamics included densities {p. of the four links, armature {a=)== and damping {G of three joints, and the friction coefficient M4g To test for robustness, we varied both frictions and torso densities leading to significant variations dynamics. We further conducted additional experiments while varying all 11 dimensional specification parameters.
Hopper:
This benchmark is similar to Hopper except that the controlled system is a biped robot with seven bodies and six joints. Dimensions for its dynamics are extended accordingly reported in Table I Here, we again varied the torso density for performing robustness experiments in the range Po 3000].
Walker2D:
Halfcheetah: This benchmark is similar the above except that the controlled system is two- dimensional slice of a diree-dimessional cheetah robot. Parameters specifying the simulator consist of 21 dimensions, with 7 representing densities. In our two-dimensional experiments we varied the
14

---

Under review as a conference paper at ICLR 2020
ID experiment
2D experiment
High-dimensional experiment
Inverted Pendulum
lp
None
None
Hopper
Po
{po: Mg {po. Hg) {po, 14)
(pili fai) (Gi Ug (pi)i u far)is U (cilioo Uy
Walker2D
Po
HalfCheetah
None
Table Parameterisation of dynamics. See section 5.1 for the physical meaning of these parameters.
lorso-density and floor friction, while in high-dimensional ones, we allowed the algorithm to control all 21 variables.
B. EXPERIMENTAL PROTOCOL
Our experiments included training and testing phases. During the training phase we applied Algo- rithm for determining robust policies while updating transition model parameters according to the min-max formulation. Training was performed independently for each of the algorithms on the rel- evant benchmarks while ensuring best operating conditions using hyper-parameter values reported Tessler et al., 2019).
elsewhere (Schulman et al. 2017
Pinto al. 2017
For all benchmarks, policies were represented using parametrised Gaussian distributions with their means given by neural network and standard derivations by group of free parameters. The neural network consisted of two hidden layers with units and hyperbolic tangent activations in each of the layers. The final layer exploited linear activation actor-critic framework, we also trained a standalone critic network having the same structure as that of the policy.
to output a real number. Following the
For each policy update, we rolled-out in the current worst-case dynamics to collect a number of transitions. The number associated to these transitions was application-dependenn and varied be- tween benchmarks in the range of 5,000 to 10,000. The policy then optimised (i.e., Phase II of Algorithm D using proximal policy optimization with generalised advantage estimation. To solve the minimisation problem in the inner loop of Algorithm a diagonal Gaussian distribution that is centered at the current worst-case dynamics model. The number of sampled dynamics and the variance of the sampled distributions depended on both the benchmark itself, and well as the dimensions of the dynamics. Gradients needed for model updates estimated using the results in Propositions and 8. Finally, terminated training when the threshold.
sampled number of dynamics from
policy entropy dropped below an upplication-dependee
When testing, we evaluated policies on unseen dynamics that exhibited simulator variations as de- scribed earlier. We measured performance using returns averaged over 20 episodes with maximum length of 1,000 time steps on testing environments. We note that we used non-discounted mean episode rewards compute such averages.
B.2 RESULTS WITH ONE-DIMENSION MODEL VARIATION
Figure shows the robustness of policies on various taks. For fair comparison, we trained two standard policy gradient methods (TRPO (Schulman et al. (015b) and PPO (Schulman et 2017)), and two robust RL algorithms (RARL (Pinto al. 2017), PR-MDP (Tessler al. (019g) with the reference dynamics preset by our algorithm. The range of evaluation parameters was intentionally designed to include dynamics outside of the --Wassesstein ball. Clearly, WR²L. outperforms all baselines in this benchmark.
B.3 DEEP DETERMINISTIC POLICY GRADIENTS RESULTS
As mentioned in the experiments section of the main paper, refrained from presenting results involving deep deterministic policy gradients (DDPG) due to its lack in robustness even on simple systems, such the CartPole.
15

---

Under review as a conference paper at ICLR 2020
2976
W0.2
M.
506
100
1-4
(b) Hopper. Reference is Po 1750: PPOg uses the same implementation as PPO trained with po 3000.
(a) Inverted pendulum, reference is lp 1.65.
5300
MAA
1000
1000
FP0
3000
(c) Walker. Reference is pa 1750.
HalfCheetah. Reference μ 1.5.
Figure 3: Robustness tests for various tasks; dynamics varied along one dimension.
1000
800
Mean reward
600
400
WR
PPO
0B 17
TRPO
25
DDPG
Pole length
Figure 4: Robustness results on the inverted pendulum demonstrating that our method outperforms state-of-the--rt in terms of average test returns and that DDPG lacks in robustness performance.
Figure 4 depicts these results showing that DDPG lacks robustness even when minor variations in the pole length are introduced. TRPO and PPO, the other hand, demonstrate acceptable performance retaining test return of 1,000 across a broad range of pole lengths variations.
C DERIVATION OF THE CLOSED FORM SOLUTION
In Section 3.2 we presented closed form solution to the following optimisation problem:
1 = -p) Ho( Φb) se (o وااه) s.t.
min V E (7) (Rocaa (T)]
which took the form of:
2r
Ho '9 N.sl
gl:h H, 'l
16

---

Under review as a conference paper at ICLR 2020
In this section of the appendix, derive such update rule from first principles. We commence transforming the constraint optimisation problem into an unconstrained one using the Lagrangian:
L( A) - 1.. (0- (Ø-$1)) -b11 + x
( To))Hole 60) -
where À is a Lagrange multiplier, and g |&d] = V& rmp(() [Rooaa (T))
(bb)
Deriving the Lagrangian with respect to the primal parameters Φb we write:
VoL(d, A) " A( To)THo Ho-
(12)
Setting Equation 12 to zero and solving for primal parameters, we attain:
1 HD N
Φ Φ0
À
Plugging & back into the equation representing the constraints, we derive:
(0o 1 x Hy 1 g <b Ho -
Φ - H ( 200)
2c
x²
g
KJlT
H 1
1 [t.l.
2c
It is easy to see that with the positive solution for X, the are satisfied. Since the objective and constraint are both convex, the KKT conditions are sufficient and necessary for optimality, thus finalising our derivation.
(KKT) conditions
17


Reviews:
[{"review_id": "SklT0Tx0tS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a novel solution to a *restricted* reinforcement learning problem. I am not quite sure if I understand the restriction from the text or thus what relevance this has in applications. The method factorizes the transition dynamics (where they rely on a parametrized solver) and a policy modeled by a NN and trained with policy gradient with monte-carlo returns. The ultimate goal is robust control so they care about the maximin objective of highest return along the worst trajectory. The robustness is implemented as a constraint in the dynamics optimization: a wasserstein distance between the distribution of samples obtained applying the learnt policy and learnt dynamics parameters and those obtained by random policy and some reference dynamics model. They solve the maximin by alternating minimization on the dynamics and policy. The contribution is in the intricacies of computing the wasserstein efficiently in particular they obtain a perturbation based estimate of the gradient of the dynamics parameters. The experiments seem sensible but I  have some suggestions for improvement.\n\nI have published in RL but my familiarity with Wasserstein and robustness is limited. I have not checked the derivations in the appendix in detail. If there is a high score I'll have a closer look. \n\nMajor points:\n1) The restriction imposed is that only part of the transition model is really learnt. So it is more like a system identification setup ?! Why is that necessary ? \n2) The explanation in on page 4 about how the wasserstein distance is computed is a bit unclear. Is it sampling data from the P_\\phi and \\pi on one hand and P_0 and \\pi_{Uniform} ? Then there is a bucketing step (what does it mean ? how is it computed ?). Is it then a discrete wasserstein between the two empirical distributions ?! What is Monte-Carlo in this case ?\n3) All the approximations done in the method should be more clearly explained because they are relevant both to people who might apply the method in practice as well as other researchers that might want to build on/ improve on it.\n4) The experiments seem interesting but I am not sure how to interpret the results. If a method has access to a simulator and a representation of the relevant dynamics parameter space it seems clear that it will be more robust. This is impressive if one has a simulator and can infer some state components e.g. in pendulum starting with an empty simulator and adding a pendulum. But that seems as hard or maybe even harder than trying to learn an implicit transition model just from observations. \n5) Imagine the method only gets observations and an imperfect simulator, which will be the case in an application, what are the caveats we should have in mind ?\n6) Can you have a learnt model baseline and compare your learnt and simulation ? Maybe it would have to be a latent variable model so you can do the same inference for internal parameters and still do an adaptation in the same way as now. This seems easy since it is a supervised problem.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJlwW1TpYB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies the problem of training policies which maximize returns under worst case environmental perturbations. They propose to solve this max min problem using iterative optimization with the max step performed using standard RL. The min step relies on sample-based estimates of gradients and hessians with respect to environment parameters. The problem of robust control is an important one, and the paper presents interesting results, however it lacks a proper comparison to baselines.\n\n1. I think figure 1 should include two more columns. The first showing the performance of a policy trained to maximize returns averaged over the training environmental perturbations. This doesn't require the min step of the algorithm and is a natural baseline for the setup you consider. The second showing the same but trained with an LSTM policy. This way the policy can learn to do system identification. It might not be robust (i.e. not generalize to unseen perturbations) but this should be verified. If it does generalize, then there's no reason to complicate things with the min step.\n2. Maybe I'm missing something but neither RARL nor PR-MDP baselines are shown in Fig. 1 and 2. despite you claiming in intro to section 5 that you test against those. These baselines are included in Figure 3 in the Appendix but I find those numbers odd. E.g. RARL doesn't seem to match PPO even on the reference environment in Fig 3c. This suggest that it relies on a different algorithm and it doesn't make sense to compare it to WR2L which relies on PPO.\n3. In Related work you says \"we were unable to find the code for this paper, and did not attempt to implement it ourselves\". Even if this code was available, comparing to a different codebase is not a meaningful comparison (see e.g. [1]). There is no replacement for implementing relevant baselines.\n4. You estimate the Hessian using data from a random policy. I find this odd because random policy never visits the interesting states. Have you tried recalculating the Hessian in every min step?\n5. Could you clarify the use of Wasserstein distance? If I only use samples from the two distributions to estimate it then this is likely limited to low-dimensional settings, right?\n\n[1] Henderson et al., (2017). Deep Reinforcement Learning that Matters.\n\n------------------------------------------------------------------------------------------------------------------------------------------------------\nThanks for the clarifications and for including the \"average\" policy baseline. I still think the paper could use more work and hence keeping my score as is. I'd try to make the LSTM sys-id baseline to work--those have been used with domain randomizations previously, as well as \"meta-learners\" in the meta RL literature.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hkl0M_Rx9H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on the problem of robust reinforcement learning and proposed a generic framework that is able to deal with both discrete and continuous state and action spaces. They formalized the robust reinforcement learning problem as a min-max game with Wasserstein constraints and put forward an efficient and scalable solver that is helpful to numerical optimization. Overall, this paper is of well-written with a clear illustration of the methodology and comprehensive experimental results. I still have a few concerns below, that prevent me from giving a direct acceptance: \n1.\tHowever, the proposed method seems to be somewhat incremental: In Yang (2017), it is clearly stated that it is viable to treat the robust reinforcement learning objective as a min-max game (in eq.(1)) with support belonging to a Wasserstein ball (in eq.(2)). The statement in the submission “… a novel min-max game with a Wasserstein constraint …” seems overclaimed, namely, eq.(8) in the submission, is a combination of eq.(1) and (2) in Yang (2017), with the constraint in eq.(8) not exactly equal but just a relaxation of eq.(2) in Yang (2017).\n2.\tFurthermore, since a few papers have proposed methods to deal with both transition and reward dynamics, it would be nice and complete to also address (or hint on) the reward ambiguity problem.\n3.\tDoes the relaxation of the Wasserstein constraint in eq.(7) make the learned policy conservative? Some illustrations of this effect of \\epsilon would be interesting and complete.\n4.\tFor the experiments in section 5.1, it would be better to give an illustration of what the color means in Figure 1. \nInsoon Yang. A convex optimization approach to distributionally robust markov decision processes with Wasserstein distances. IEEE control systems letters, 1(1):164-169, 2017.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the reviewers agreed that the problem of learning robust policies is an important one, there were a number of major concerns raised about the paper, and as a result I would recommend that the paper not be accepted at this time. The important points are: (1) limited novelty in light of prior work in this area (see R2 and R3); (2) a number of missing comparisons (see R2). There is also a bit of confusion in the reviews, which I think stems from a somewhat unclear statement in the paper of the problem formulation. While there is nothing wrong with assuming access to a parameterized simulator and studying robustness under parametric variation, this is of course a much stronger assumption than some prior work on robust reinforcement learning. Clarity on this point is crucial, and there are a large number of prior methods that can likely do well in this setting (e.g., based on system ID, etc.).
Decision: Reject