Robust Reinforcement Learning for Continuous Control with Model Misspecification
Authors: Daniel J. Mankowitz
Nir Levine
Rae Jeong
Abbas Abdolmaleki
Jost Tobias Springenberg
Yuanyuan Shi
Jackie Kay
Todd Hester
Timothy Mann
Martin Riedmiller
Venue:  (2020)
Keywords: reinforcement learning
robustness

Abstract: We provide a framework for incorporating robustness -- to perturbations in the transition dynamics which we refer to as model misspecification -- into continuous control Reinforcement Learning (RL) algorithms. We specifically focus on incorporating robustness into a state-of-the-art continuous control RL algorithm called Maximum a-posteriori Policy Optimization (MPO). We achieve this by learning a policy that optimizes for a worst case, entropy-regularized, expected return objective and derive a corresponding robust entropy-regularized Bellman contraction operator. In addition, we introduce a less conservative, soft-robust, entropy-regularized objective with a corresponding Bellman operator. We show that both, robust and soft-robust policies, outperform their non-robust counterparts in nine Mujoco domains with environment perturbations. In addition, we show improved robust performance on a challenging, simulated, dexterous robotic hand. Finally, we present multiple investigative experiments that provide a deeper insight into the robustness framework; including an adaptation to another continuous control RL algorithm. Performance videos can be found online at https://sites.google.com/view/robust-rl.

Published as a conference paper at ICLR 2020
ROBUST REINFORCEMENT LEARNING FOR CONTINU- OUS CONTROL WITH MODEL MISSPECIFICATION
Daniel J. Mankowitz", Nir Levine', Rae Jeong, Abbas Abdolmaleki, Jost Tobias Springenberg, Yuanyuan Shi, Jackie Kay, Timothy Mann, Todd Hester, Martin Riedmiller DeepMind
[dmankowitz, nirlevine, raejeong, aabdolmaleki, springenberg yyshi, kayj, timothymann, toddhester, riedmiller (aaellleerrrrggellee @google. com.
ABSTRACT
We provide a framework for incorporating robustness to perturbations in the transition dynamics which we refer to as model misspecification into continuous control Reinforcement Learning (RL) algorithms. We specifically focus on incor- porating robustness into state-of- of- the-ahr continuous control RL algorithm called Maximum a-posteriori Policy Optimization (MPO). We achieve this by learning a policy that optimizes for a worst case expected return objective and derive a corresponding robust entropy-regularic tion, we introduce a less conservative, soft-robust, entropy-regularize objective with corresponding Bellman operator. We show that both, robust and soft-robust policies, outperform their non-robust counterparts nine Mujoco domains with environment perturbations. In addition, we show improved robust performance on a high-dimensional, simulated, dexterous robotic hand. Finally, we present multiple investigative experiments that provide a deeper insight into the robustness framework. This includes an adaptation to another continuous control RL algorithm as well as learning the uncertainty set from offline data. Performance videos can be found online at htttt:::s//.. :/ tes............ googloooooooooooooo.oooooooo.ooe le. com com/vieee ew robust-
Bellman contraction operator. In addi-
INTRODUCTION
Reinforcement Learning (RL) algorithms typically learn policy that optimizes for the expected
return (Sutton & Barto 1998). That is, the policy aims to maximize the sum of future expected
rewards that an agent accumulates in a particular task. This approach has yielded impressive results in recent years, including playing computer games with super human performance (Mnih et al. 2015
Tessler et al. 2016), multi-task RL (Rusu al. 2016 Devin al. 2017 Teh al. 2017 Mankowitz
et al. 201 8b Riedmiller et al. 2018) as well as solving complex continuous control robotic tasks (Duan et al. 2016 Abdolmaakkk et al., 20186 Kalashnikov al. 2018 Haarnoja et al.) 2018).
The current crop of RL agents are typically trained in a single environment (usually a simulator). As a consequence, an issue that faced by many these agents is the sensitivity of the agent's policy to environment perturbations. Perturbing the dynamics of the environment during test time, which may include executing the policy a real-world setting, can have a significant negative impact the performance of the agent (Andrychowicz et al.) 2018 Peng et al.; (2018) Derman et al.) 2018 Di Castro et al., 2012) Mankowitz et al., 2018a). This is because the training environment is necessarily very good model of the perturbations that an agent may actually face, leading to potentially unwanted, sub-optimal behaviour. There are many types of environment perturbations. These include changing lighting/weather conditions, sensor noise, actuator noise, action delays
(Dulac-Arood el al. 2019).
It is desirable train agents that agnostic environment perturbations. This especially erucial in the Sim2Real setting (Andrychowicz et al. 20188] Peng et al. 2018:) Wulfmeier et al. 2017 Rastogi al. 2018 Christiano et al. 2016) where policy is trained a simulator and then executed
"Equal contribution
Work done during an intersship Deepmind

---

Published as a conference paper at ICLR 2020
real-world domain. As example, consider robotic arm that executes control policy to perform specific task in a factory. If, for some reason, the arm needs to be replaced and the specifications do not exactly match, then the control policy still needs to be able to perform the task with the 'perturbed' robotic arm dynamics. In addition, sensor noise due to malfunctioning sensors, as well as actuator noise, may benefit from robust policy to deal with these noise-induced perturbations.
Model misspecification: For the purpose of this paper, we refer to an agent that is trained in one environment and performs in a different, perturbed version of the environment (as in the above examples) as model misspecification. By incorporating robustness into our agents, we correct for this misspecification yielding improved performance in the perturbed environment(s).
In this paper, we propose framework for incorporating robustness into continuous control RL algorithms. We specifically focus on robustness to model misspecification in the transition dynamics. Our main contributions are as follows:
(1) We incorporate robustness into a state-of- of-theeart continuous control RL algorithm called Maximum posseeiioii Policy Optimization (MPO) (Abdolmaleki et al.] 2018b) to yield Robust MPO (R-MPO). We also carry out an additional experiment, where we incorporate robustness into an additional continuous RL algorithm called Stochastic Value Gradients (SVG) (Heess et al. 2015b).
(2) Entropy regularization encourages exploration and helps prevent early convergence to sub-optimal policies (Nachum et al. 2017). To incorporate these advantages, we: (i) Extend the Robust mellman operator (Iyengar 2005) to robust and soft-robust entropy-regularin
versions, and show that these operators are contraction mappings. In addition, we (ii) extend MPO to Robust Entropy-regullrizee MPO MPEM and Soft RE-MPO (SRE-MPO) and show that they perform at least as well as R-MPO and in some cases significantly better. All the derivations have been deferred to Appendices C
B
and D
We want to emphasize that, while the theoretical contributions are novel, our most significant contribution is that of the extensive experimental analysis we have performed to analyze the robustness performance of our agent. Specifically:
(3) We present experimental results in nine Mujoco domains showing that RE-MPO, SRE-MPO and R-MPO, SR-MPO outperform both E-MPO and MPO respectively.
(4) To ensure that our method scales, we show robust performance on a high-dimensional, simulated, dexterous robotic hand called Shadow hand which outperforms the non-robust MPO baseline.
(5) Multiple investigative experiments to better understand the robustness framework. These include
(i) an analysis of modifying the uncertainty set; (ii) comparing our technique to data augmentation;
(iii) comparison to domain randomization; (iv) comparing with and without entropy regularization; (v) We also train the transition models from offline data and use them the uncertainty set to run R-MPO. We show that R-MPO with learned transition models as the uncertainty set can lead to improved performance over R-MPO.
BACKGROUND
A Markov Decision Process (MDP) is defined as the tuple (S, A,r, 7, P) where S' is the state space, A the action space, r: S x A -> is bounded reward function; Y E [o, 1] is the discount factor and P: S x A + A maps state-action pairs to probability distribution over next states. We use A S to denote the |S| simplex. The goal of Reinforcement Learning agent for the purpose of control is to learn policy x S A which maps state and action to a probability of executing the action from the given state so as to maximize the expected return J(T)) = E Ee >rre where r, is a random variable representing the reward received at time t (Sutton & Barto 2018). The value function is defined as V*) = E= [> 7r s] and the action value function as Q a) = r(s,a) + +.B (s')).
A Robust MDP (R-MDP) is defined tuple (S,, A, r, 7, P) where S, A,r and y are defined as above; P(s, a) Ç M(S) is an uncertainty set where M(S) is the set of probability measures over agent selecting state and action pair, and the next
next states s' S. This is interpreted state s' is determined by a conditional measure ((s'ss, a) E P(s. a) [Lyengar 2005). A robust policy
optimizes for the worst-case expected return objective: JR(Z) inf pep B
Di fra)]
2

---

Published as a conference paper at ICLR 2020
The robust value function is defined as V (s) inf pep EP. I> 2t=0 r and the robust action value function as QR a) = r(s, a) + inf Ex' (s RR (s')] Both the robust Bellman operator TR RIS) + RI8I for fixed policy and the optimal robust Bellman operator TRe(s) = max, vs)) have previously been shown to be contractions [Tyengar 2005). A rectangularity
assumption on the uncertainty set (Iyengar, 2005) ensures that "nature" can choose worst-case transition function independently for every state s and action a.
Maximum A A-Posteriori Policy Optimization (MPO) (Abdolmalcki et al. 201 8a by a continuous control RL algorithm that performs an expectation maximization form of policy iteration. There are two steps comprising policy evaluation and policy improvement. The policy evaluation step receives as input a policy Ty and evaluates an action-value function Q (s, a) by minimizing the squared TD error: ming (re + 70% ($t+1 ~ P(-|se. as), 4141 ~ ~TT(( (- - Q (st. a,))², where
0 denotes the parameters of a target network (Mnih et al. 2015) that are periodically updated from replay- replay-buffer of samples in order to perform the policy evaluation step. The second step comprises a policy improvement step. The policy improvement step consists of optimizing the objective Jss x) (Q, (s, a)] for states drawn from a state distribution ju(s). In practice the state distribution samples are drawn from an experience replay. By improving J in all states 8, we improve our objective. To do so, a two step procedure is performed.
8. In practice we
First, construct non-parametric estimate such that J(ss, q) J[ss, This is done by maximizing J(s, q) while ensuring that the solution, locally, stays close to the current policy This optimization has closed form solution given as ((ass) x TA (a(a)s) exp 27 (s.0l/n, where 17 a temperature parameter that can be computed by minimizing convex dual function (Abdolmaleki et al.] (2018b)). Second, we project this non- parametric representation back onto a parameterized policy by solving the optimization problem where Tg++1 is the new and improved policy and where
Th: i.e. Eu(=) (gL((()))) (T (+13))
Th+1 arg min, Ex(e) (LL()))KL(((() (a one typically employs additional regularization (Abdolmaleki et al. 2018a). Note that this amounts to supervised learning with samples drawn fron g(a ((a)s); see Abdolmaleki et al. (2018a) for details.
3 ROBUST MPO
To incorporate robustness into MPO, we focus on learning a worst-case value function in the policy evaluation step. Note that this policy evaluation step can be incorporated into any actor-critic algorithm. In particular, instead of optimizing the squared TD error, we optimize the worst-case squared TD error, which is defined as:
Q7 (1141 ~ p(- |8e, ae), aeti ~ TH (114+1)) - -Q (st, (st,ar))
min Ft +7 inf 0
(I)
PEP(s()))
where P(st, az) is an uncertainty set for the current state s; and action ae; Ti is the current network's policy, and 0 denotes the target network parameters. It is in this policy evaluation step (Line 3 in in Appendix [ that the Bellman operators in the previous sections are applied.
Algorithms 12 and
B
Relation to MPO: In MPO, this replaces the current policy evaluation step. The robust Bellman operator (Tyengar 2005) ensures that this process converges to unique fixed point for the policy Tkk This is achieved by repeated application of the robust Bellman operator during the policy evaluation step until convergence to the fixed point. Since the proposal policy q(als) (see Section 2 proportional to the robust action value estimate Q (s, a), it intuitively yields robust policy as the policy is being generated from worst-case value function. The fitting of the policy network to the proposal policy yields a robust network policy 1
rntropy-regularized MPO: Entropy regulaizrrggllrizzatiinn encourages exploration and helps prevent early convergence to sub-optimal policies (Nachum et al. 2017). To incorporate these advantages, we extended the Robust Bellman operator (Iyengar 2005) to robust and soft-robust
regulariee versions (See Appendix B and respectively for a detailed overview and the corresponding deriva- tions) and show that these operators contraction mappings (Theorem [ below and Theorem 2 Appendix E) and yield a
value-iteratio bound with respect to the max norm.
Theorem 1. The robust entropy-regularized Bellman operator TKxL for a fixed policy is a contrac- tion operator. Specifically: VU, V E RRS and 7 E (0, 1), we have, T U TR VII >IIU VIJ.
3

---

Published as a conference paper at ICLR 2020
In addition, we extended MPO to Robust regroiy-regullriized MPO (RE-MPO) and Soft RE-MPO (SRE-MPO) (see Appendix D for a detailed overview and derivations) and show that they perform at least well as R-MPO and in some cases significantly better. All the derivations have been deferred to the Appendix. The corresponding algorithms for R-MPO, RE-MPO and SRE-MPO can be found in Appendix
4 EXPERIMENTS
We now present experiments on nine different continuous control domains (four of which we show in the paper and the rest can be found in Appendix H.44) from the DeepMind control suite (Tassa et al. 2018). In addition, we present an experiment on high-dimensional dexterous, robotic hand called Shadow hand (ShadowRobol 2019). In our experiments, we found that the autropy-regularie version of Robust MPO had similar performance and in some cases, slightly better performance than the expected return version of Robust MPO withous cattooy-rreularizzan
We therefore decided to objective (non-robust, robust and
include experiments of our agent optimizing the entropy-regulanin soft-robust versions). This corresponds to (a) non-robust E-MPO baseline, (b) Robust E-MPO (RE- MPO) and (c) Soft-Robust E-MPO (SRE-MPO). From hereon in, it is assumed that the algorithms optimize for the entropy regulaieee objective unless otherwise stated.
Appendix:
In Appendix H.4 we present results of our agent optimizing for the expected return objective without entropy regularization (for the non-robust, robust and soft-robust versions). This corresponds to (a") non-robust MPO baseline, (b') R-MPO and (c') SR-MPO.
The experiments are divided into three sections. The first section details the setup for robust and soft-robust training. The next section compares robust and soft-robust performance to the non-robust MPO baseline in each of the domains. The final section is a set of investigative experiments to gain additional insights into the performance of the robust and soft-robust agents.
Sctup: For each domain, the robust agent is trained using a pre-defined uncertainty set consisting of three task perturbations Each of the three perturbations corresponds particular perturbation of the Mujoco domain. For example, in Cartpole, the uncertainty set consists of three different pole lengths. Both the robust and non-robust agents are evaluated on test set of three unseen task perturbations. In the Cartpole example, this would correspond to pole lengths that the agent has not seen during training. The chosen values of the uncertainty and evaluation set for each domain can be found in Appendix H.3 Note that it is common practice to manually select the pre-defined uncertainty set and the unseen test environments. Practitioners often have significant domain knowledge and can utilize this when choosing the uncertainty set (Derman & Mannor 2019
Denman et al. 2018 Di Castro et al. 2012 Mankowitz al. 2018a Tamar et al. 2014).
During training, the robust, soft-robust and non-robust agents act in an unperturbed environment which we refer to as the nominal environment. During the TD learning update, the robust agent calculates an infimum between Q values from each next state realization for each of the uncertainty set task perturbations (the soft-robust agent computes an average, which corresponds to a uniform distribution over P. instead of an infimum). Each transition model is different instantiation of the Mujoco task. The robust and soft-robust agents are exposed to more state realizations than the non-robust agent. However, of samples and the diversity of the samples for the non-robust agent still results in poor performance compared to the robust and soft-robust agents.
show in our ablation studies, significantly increasing the number
4.1 MAIN EXPERIMENTS
We compare the performance of non-robust MPO to the robust and soft-robust variants. Each training run consists of 30k episodes and the experiments are repeated 5 times. In the bar plots, the y-axis indicates the average reward (with standard deviation) and the x-axis indicates different unseen evaluation environment perturbations starting from the first perturbation (Env0) onwards. Increasing environment indices correspond to increasingly large perturbations. For example, in Figure T (top left), EnvO, Eny] and Eny2 for the Cartpole Balance task represents the pole perturbed
Mujoco Domains:
We did experiments on larger set with similar results, but settled three for computational efficiency.
4

---

Published as a conference paper at ICLR 2020
hast-hop
1000
800
252
000
600
600
400
203
52-
200
Figure Three domains showing RE-MPO (blue), SRE-MPO (green) and E-MPO (red). The addition six domains can be found in the appendix. In addition, the results for R-MPO, SR-MPO and MPO can be found in Appendix H.4 with similar results.
Deiiel
353
242
223
10-40
Figure 2: (1) The Shadow hand domain (left) and results for RE-MPO and E-MPO (middle left). (2) A larger test set: the figures show the performance of RE-MPO (blue), SRE-MPO (green) and E-MPO (red) for test set that extends from the nominal environment significant perturbations outside the training set for Cartpole Balance (middle right) and Pendulum Swingup (right).
lengths of 2.0, 2.2 and 2.3 meters respectively. Figure T shows the performance of three Mujoco domains (The remaining six domains are in Appendix H.4j. The bar plots indicate the performance of E-MPO (red), RE-MPO (blue) and SRE-MPO (green) on the held-out test perturbations. This color scheme is consistent throughtout the experiments unless otherwise stated. As can be seen in each of the figures, RE-MPO attains improved performance over E-MPO. This same trend holds true for all nine domains. SRE-MPO outperforms the non-robust baseline in all but the Cheetah domain, but not able to outperform RE-MPO. An interesting observation can be seen in the video for the Walker walk task (https: / google. cww/vieww robust- rl), where the RE-MPO agent learns to 'drag' its leg which a fundamentally different policy to that of the non-robust agent which learns regular gait movement.
Appendix:
The appendix contains additional experiments with the non entropy-regularize versions of the algorithms where again the robust (R-MPO) and soft robust (SR-MPO) versions of MPO outperform the non-robust version (MPO).
This domain consists of dexterous, simulated robotic hand called Shadow hand whose goal is to rotate a cube into pre-defined orientation (ShadowRobot, 2019). The space 79 dimensional vector and consisting of angular positions and velocities, the cube orientation and goal orientation. The action space is a 20 dimensional vector and consisting of the desired angular velocity of the hand actuators. The reward a function of the current orientation of the cube relative to the desired orientation. The uncertainty set consists of three models which correspond to increasingly smaller sizes of the cube that the agent needs to orientate. The agent evaluated on a different, unseen holdout set. The values can be found in Appendix H.3] We compare RE-MPO to E-MPO trained agents. Episodes are 200 steps long corresponding to approximately 10 seconds of interaction. Each experiment is run for 6k episodes and is repeated 5 times. As seen in Figure 2 RE-MPO outperforms E-MPO, especially as the size the cube decreases (from Env0 to Env2). This is especially challenging problem due the nigh-dimensionality of the task. As seen in the videos (https: /ttt:s::/sitss. google com/ www.e//ww////// robust-xl the RE-MPO agent able to manipulate significantly smaller cubes than had observed in the nominal simulator.
Shadow hand domain:
5

---

Published as a conference paper at ICLR 2020
1000
1010
09)
200
D1
292
Figure 3: (1) Domain Randomization (DR): Domain randomization performance for the Cartpole balance (left) and Pendulum swingup (middle left) tasks. (2) Stochastic Value Gradients (SVG): Two right images show the performance of Robust hatropy-regularizer SVG (RE-SVG) and SRE-SVG compared to E-SVG for Pendulum and Cartpole respectively.
4.2 INVESTIGATIVE EXPERIMENTS
This section aims to investigate and try answer various questions that may aid in explaining the performance of the robust and non-robust agents respectively. Each investigative experiment is conducted on the Cartpole Balance and Pendulum Swingup domains.
What if we increase the number of training samples? One argument is that the robust agent has access to more samples since it calculates the Bellman update using the infimum of three different environment realizations. To balance this is effect, the non-robust agent was trained for three times more episodes than the robust agents. Training with significantly more samples does not increase the performance of the non-robust agent and, can even decreases the performance, as result of overfitting to the nominal domain. See Appendix H.5 Figure 12 for the results.
What about Domain Randomization? A subsequent point would be that the robust agent sees more diverse examples compared to the non-robust agent from each of the perturbed environments. We therefore trained the non-robust agent in a domain randomization setting (Andrychowicz et al. 2018 Peng al. 2018). We compare our method to two variants of DR. The first variant Limited-DR uses the same perturbations as in the uncertainty set of RE-MPO. Here, we compare which method better utilizes a limited set of perturbations learn a robust policy. As seen in Figure (left and middle left for Carpole Balance and Pendulum Swingup respectively), RE-MPO yields significantly better performance given the limited set of perturbations. The second variant Full-DR performs regular DR on significantly larger set of 100 perturbations in the Pendulum Swingup task. In this setting, DR, which uses 30 times more perturbations, improves but still does not outperform RE-MPO (which still only uses three perturbations). This result can be seen in Figure [3] Appendix H.5
What is the intuitive difference between DR and RE-MPO/SRE-MPOT DR defines the loss to be the expectation of TD-errors over the uncertainty set. Each TD error is computed using state, action, reward, next state < 8, a, r; 8 > trajectory from a particular perturbed environment, (selected uniformly from the uncertainty set). These TD errors are then averaged together. This is a form of data augmentation and the resulting behaviour is the average across all of this data. RE-MPO/SRE- MPO: In the case of robustness, the TD error is computed such that the target action value function is computed as a worst case value function with respect to the uncertainty set. This means that the learned policy is explicitly searching for adversarial examples during training to account for worst-case performance. In the soft-robust case, the subtle yet important difference (as seen in the experiments) with DR is that the TD loss is computed with the average target action value function with respect to next states (as opposed to averaging the TD errors of each individual perturbed environment as in DR). This results in different gradient updates being used update the action value function compared to DR.
A larger test set: It is also useful to view the performance of the agent from the nominal envi- ronment to increasingly large perturbations in the unseen test set (see Appendix H.3 for values). These graphs can be seen in Figure 2 for Cartpole Balance and Pendulum Swingup respectively. As expected, the robust agent maintains a higher level of performance compared to the non-robust agent. Initially, the soft-robust agent outperforms the robust agent, but its performance degrades as the perturbations increase which is consistent with the results Derman et al. (2018). In addition, the robust and soft-robust agents are competitive with the non-robust agent in the nominal environment.

---

Published as a conference paper at ICLR 2020
640
140
200
150
Figure 4: Modifying the uncertainty set: Pendulum Swingup when modifying the third perturbation of the uncertainty set to values of 1.2 (left), 1.3 (middle) and 2.0 (right) meters respectively.
Modifyyng the uncertainty set: We now evaluate the performance of the agent for different uncertainty sets. For Pendulum Swingup, the original uncertainty set values of the pendulum arm are 1.0, 1.1 and 1.4 meters. We modified the final perturbation to values of 1.2, 1.3 and 2.0 meters respectively. The agent is evaluated on unseen lengths of 1.5, 1.6 and 1.7 meters. An increase in performance can be seen Figure A as the third perturbation approaches that of the unseen evaluation environments. Thus it appears that if the agent is able to approximately capture the dynamics of the unseen test environments within the training set, then the robust agent is able to adapt to the unseen test environments. The results for cartpole balance can be seen in Appendix H.S Figure 14
What about incorporating Robustness into other algorithms? To show the generalization of this robustness approach, we incorporate it into the critic of the Stochastic Value Gradient (SVG) con- tinuous control RL algorithm (See Appendix H.T. As seen in Figure 3 Robust Entropy-regullrizee SVG (RE-SVG) and Soft RE-SVG (SRE-SVGH significantly outperform the non-robust Entropy- regularized SVG (E-SVG) baseline in both Cartpole and Pendulum.
Robust entropy- return vs. robust expected return: When comparing the robust entropy-reguuarrized return performance to the robust expected return, we found that the entropy- regularized return appears to do no worse than the expected return. In some cases, c.g., Cheetah, the cntropy-regularizee objective performs significantly better (see Appendix H.5 Figure
Different Nominal Models: In this paper the nominal model was always chosen the smallest perturbation parameter value from the uncertainty set. This was done to highlight the strong perfor- mance of robust policies increasingly large environment perturbations. However, what if we set the nominal model as the median or largest perturbation with respect to the chosen uncertainty set for each agent? As seen in Appendix H.S) Figure [5] the closer (further) the nominal model is to (from) the holdout set, the better (worse) the performance of the rob-so agent. However, in all cases, the robust agent still performs at least as well as (and sometimes better than) the non-robust agent.
What about learning the uncertainty set from offfine data? In real-world settings, such as robotics and industrial control centers (Gao 2014), there may be nominal simulator available as well as offline data captured from the real-world system(s). These data could be used to train transition models to capture the dynamics of the task at hand. For example, set of robots in a factory might each be performing the same task, such as picking up a box. In industrial control cooling centers, there are number of cooling units in each center responsible for cooling the overall system. In both of these examples, each individual robot and cooling unit operate with slightly different dynamics due to slight fluctuations in the specifications of the designed system, wear-and-tear as well as sensor calibration errors. As result, an uncertainty set of transition models can be trained from data generated by each robot or cooling unit.
However, can we train a set of transition models from these data, utilize them as the uncertainty set in R-MPO and still yield robust performance when training on a nominal simulator? To answer this question, we mimicked the above scenarios by generating datasets for the Cartpole Swingup and the Pendulum swingup tasks. For Cartpole swingup, we varied the length of the pole and generated a dataset for each pole length. For Pendulum Swingup, we varied the mass of the pole and generated the corresponding datasets. We then trained transition models on increasingly large data batches ranging from 100 to million datapoints for each pole length and pole mass respectively. We then utilized each set of transition models for different data batch sizes as the uncertainty set and ran R-MPO on each task. We term this variant of R-MPO, Data-Driven Robust MPO (DDR-MPO). The results
7

---

Published as a conference paper at ICLR 2020
domansCarrpad taskrySonguup
domannPenndllumm
900
700
taskySwingup
800-
6.00
7:0-
600
5:00
500-
Aureege
400
200
300
200
200-
100
100
0- Ervo
Erivl
Erir
Ervo
Envl
Figure 5: Training uncertainty sets of transition models on different batch sizes of offline data. The performance of Data Driven R-MPO (DDR-MPO) can be seen in the figures above for Cartpole Swingup (left) and Pendulum Swingup (right) respectively.
can be seen in Figure [ There are a number of interesting observations from this analysis. (1) As expected, on small batches of data, the models are too inaccurate and result in poor performance. (2) An interesting insight is that as the data batch size increases, DDR-MPO starts to outperform R-MPO, especially for increasingly large perturbations. The hypothesis here is that due the transition models being more accurate, but not perfect, adversarial examples are generated in small region around the nominal next state observation, yielding an increasingly robust agent. size increases further, and the transition models get increasingly close to the ground truth models, DDR-MPO converges to the performance of R-MPO.
(3) As the batch
5 RELATED WORK
From theoretical perspective, Robust Bellman operators were introduced in (Iyengar] 2005 Nilim & Ghaoui 2005 Wiesemann et al.) 2013: Hansen & Sargent 2011: Tamar et al. [2014). Our theoretical
work extends this operator to the entropy regularized setting, for both the robust and soft-robust formulation, and modifics the MPO optimization formulation accordingly. A more closely related work from theoretical perspective is that of Grau-
al. (2016) who introduces formulation for robustness to model misspecification. This work is a special case robust MDPs where they introduce robust Bellman operator that regularizes the immediate reward with two KL terms; one entropy term capturing model uncertainty with respect to a base model, and the other term being entropy regularization with respect to a base policy. Our work differs from this work in number of respects: (1) Their uncertainty set is represented by KL constraint which has the effect of restricting the set of admissible transition models. Our setup does not have these restrictions. (2) The uncertainty
set elements from Grau-Moya et al. (2016) output probability distribution over model parameter space whereas the uncertainty set elements our formulation output distribution over next states.
Mankowitz et al. 20188 20) learn robust options, also known as temporally extended actions (Sutton ct al. | [999). using policy gradient. Robust solutions tend to be overly conservative. To combat this, Derman et al. (2018) extend the actor-critic two-timescale stochastic approximation algorithm to a "soft-robust" formulation to yield a less, conservative solution. Di Castro et al. (2012) introduce robust implementation of Deep Q Networks (Mnih et al. 2015). Domain Randomization (DR) (Andrychowicz et al., 2018 Peng et al. 2018) is a technique whereby an agent trains on different perturbations of the environment. The agent batch averages the learning error of these different perturbed trajectories together to yield an agent that is robust to environment perturbations. This can be viewed as data augmentation technique where the resulting behaviour is the average across all of the data. There are also works that look into robustness to action stochasticity (Fox et al. 2015
Braun al. 2011
Rubin et al. 2012).
6 CONCLUSION
We have presented a framework for incorporating robustness to perturbations the transition dynamics, which we refer to model misspecification into continuous control RL algorithms. This framework is suited to continuous control algorithms that learn a value function, such as an actor critic setup. We specifically focused on incorporating robustness into MPO well our entropy- regularized version of MPO (E-MPO). addition, we presented an experiment which incorporates robustness into the SVG algorithm. From theoretical standpoint, adapted MPO to entropy-
8

---

Published as a conference paper at ICLR 2020
regularized version (E-MPO): we then incorporated robustness into the policy evaluation step of both algorithms to yield Robust MPO (R-MPO) and Robust E-MPO (RE-MPO) as well as the soft-robust variants PRR-MPO/SRE-MPES robust intropy-regularized Bellman operators to ensure that the policy evaluation step converges in each case. We have extensive experiments showing that the robust versions outperform the non-robust counterparts on nine Mujoco domains as well as high-dimensional dexterous, simulated robotic hand called Shadow hand (ShadowRobol 2019). We also provide numerous investigative experiments to understand the robust and soft-robus, policy in more detail. This includes an experiment showing improved robust performance over R-MPO when using an uncertainty set of transition models learned from offline data.
This was achieved by deriving the corresponding robust and soft-
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg, Jonas Degrave, Steven Bohez, Yuval Tassa, Dan Belov, Nicolas Heess, and Martin A. Riedmiller. Relative entropy regularized policy iteration. CoRR, abs/1812.00226,, 201 8a.
02256
Abbas Abdolmaleki, Jost Tobias Springenberg. Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum posteriori policy optimisation. arXiv preprint arXiv: wew.:000660009000 2018b.
Marcin Andrychowicz. Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. arXiv preprint 17XXvv1800000177 2018.
Daniel A. Braun, Pedro A. Ortega, Evangelos Theodorou, and Stefan Schaal. Path integral control and bounded rationality, 2011 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL). pp. 202-209, 2011.
Paul F. Christiano, Zain Shah, Igor Mordatch, Jonas Schneider, Trevor Blackwell, Joshua Tobin, Pieter Abbeel, and Wojciech Zaremba. Transfer from simulation real world through learning deep inverse dynamics model. CoRR, abs/1610.03518, 2016.
1610. 03518
Esther Derman, Daniel J Mankowitz, Timothy A Mann, and Shie Mannor. Soft-robust actor-critic policy-gradient. arXiv preprint 1rXivv1803: 04848, 2018.
Mankowitz Daniel J Mann Timothy Derman, Esther and Shie Mannor. A bayesian approach to robust reinforcement learning. In Association for Uncertainty in Artificial Intelligence, 2019.
Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, and Sergey Levine. Learning modular neural network policies for multi-task and multi-robot transfer. 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 2169-2176, 2017.
Dotan Di Castro, Aviv Tamar, and Shie Mannor. Policy gradients with variance related risk criteria. arXiv preprint arXiv: 1206.6004 2012.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In JCML, 2016.
Gabriel Dulac-Amold, Daniel J. Mankowitz, and Todd Hester. Challenges of rennr//neeeeorraoaoodoeeeeerrree reinorremmen learning. CoRR, abs/1904. 12901, 2019. URR/tttp://rr//woo00/
noise Roy Fox, Ari Pakman, and Naftali Tishby. Segamingggggiiingtttemmmmiiieenn soft updates. CoRR, abs/ 111.01862.... 2015. /RL.atpp.//arxvv..rr////aaaep 08562
learning via
Jim Gao. Machine learning applications for data center optimization. 2014.
Jordi Grau-Moya, Felix Leibfried, Tim Genewein, and Daniel A. Braun. Planning with information- processing constraints and model uncertainty in markov decision processes. CoRR, abs/1 0ass/1604.0200 604.02080, 2016. URL/ttt:://arxtt www.eorrg/aao. 1604. C2080
Tuomas Haamoja,, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor- algorithms and applications. CoRR, abs/1 8 812.05905. 2018

---

Published as a conference paper at ICLR 2020
Lars Peter Hansen and Thomas Sargent. Robustess. Princeton University Press, 11 2011. ISBN 1180691114422
Nicolas Heess, Gregory Wayne, David Silver, Tim Lillierap, Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems 28 (NIPS). 2015a.
Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems, pp. 2944-2952, 2015b.
Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2): 257-280, 2005.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Scalable deep reinforcement learning for rision-based robotic manipulation. In CoRL, 2018.
Diederik P Kingma and Max Welling. arXiv:1312.6114 2013.
Auto-encoding variational bayes. arXiv preprint
Daniel Mankowitz, Timothy A Mann, Pierre-Lue Bacon, Doina Precup, and Shie Mannor. Learning robust options. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018a.
Daniel J. Mankowitz, Augustin Zidek, André Barreto, Dan Horgan, Matteo Hessel, John Quan, Junhyuk Oh, Hado van Hasselt, David Silver, and Tom Schaul. Unicorn: Continual learning with a universal, off-policy agent. CoRR, abs/1802.08294, 2018b. 1802.08294
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.
Ofrr Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. In Advances in Neural Information Processing Systems, pp. 2775-2785, 2017.
Arna Nilim and Laurent El Ghaoui. Robust control of markov decision processes with uncertain transition matrices. Operations Research, 53:780-798, 2005.
Xue Bin Peng. Marcin Andrychowicz, Wojeiech Zaremba, and Picter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1-8. IEEE, 2018.
Divyam Rastogi, Ivan Koryakovskiy. and Jens Kober. Sample-efficient reinforcement learning via difference models. In Machine Learning Planning and Control of Robot Motion Workshop at ICRA, 2018.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference on Machine Learning (ICML), 2014.
Martin A. Riedmiller, Roland Hafner, Thomas Lampe, Michael Neuncrt, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In ICML, 2018.
Jonathan Rubin, Ohad Shamir, and Naftali Tishby. Trading value and information in mdps. 2012.
Andrei A. Rusu, Sergio Gomez Colmenarejo, Çaglar Gülçehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukeuoglu, and Raia Hadsell. Policy distillation. CoRR, abs/1511 03s//15110062955 106295, 2016.
10

---

Published as a conference paper at ICLR 2020
John Schulman, Xi Chen, and Pieter Abbeel. Equivalence between policy gradients and soft q-learning. arXiv preprint wwX.vv1704.064// 2017.
ShadowRobot.
ShadowRobot Dexterous Hand.
https: htttps:/www.wc httpwwwwoshodoobbbttttttoooooott
com/ wwtp:wwwwooooo
wtt.e/onttppooassssssaaaaaaaaeaa dexterous. maan_techhiccal
#pecifiicatoon___01992222.
2019. [Online; accessed 25- 23-September-20119)
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.
URL http::t/wwwwarrttt cs umass.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2)1811-21, 1999.
Aviv Tamar, Shie Mannor, and Huan Xu. Scaling up robust mdps using function approximation. In International Conference on Machine Learning, pp. 181-189, 2014.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmalcki, Josh Merel, Andrew Lefrancq, Timothy P. Lillicrap, and Martin Riedmiller. Deepmind control suite. CoRR, abs/1801 100690, 2018. URL http://arxix www.wiiv..rr.ooooooooo..o org. /abs/
0801.006920
Yee Whye Teh, Victor Bapst, Wojciech Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In NIPS, 2017.
Chen Tessler, Shahar Givony, Tom Zahavy, Daniel J. Mankowitz, and Shie Mannor. A deep hierarchical approach to lifelong learning in minecraft. CoRR, abs/1604.07255, 2016. URL
http::/arxix: org/ abs/ 1604 07255
Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem. Robust markov decision processes. Math. February 2013. ISSN 0364-765X. doi: 10.1287/moor 120.05666. 11..///00.....066666 URL
Oper. Res., (81))313111338133
http://dx.doi: do1. wtt./............. wttp://w....gg/...../////oo/oo 1287/moor
1120. 0566
Markus Wulfmeier, Ingmar Posner, and Pieter Abbeel. Mutual alignment transfer learning. arXiv
preprint arXiv:1707.09007, 2017.

---

Published as a conference paper at ICLR 2020
A BACKGROUND
retropy-r Reinforcement Learning: Entropy regularization encourages exploration and helps prevent early convergence to sub-optimal policies (Nachum al. 2017). We make use the relative entropy-regularized RL objective defined as JKL (T;
E" 2i= ? (=-lss) (/ss)](((((1))) (-|xs)) where T is temperature parameter and KL ( ((-(sss)) is the Kullback- Leibler (KL) divergence between the current policy л and a reference policy म given a state value function is defined as VKi. (s; (s; #) E" 2 (r- 7 KL (T((-ss) (+(x))) So = s]. Intuitively, augmenting the rewards with the KL term regularizes the policy by forcing be "close" in some sense to the base policy.
(Schulman et al., 2017). The entropy-regulariae
B ROBUST ENTROPY-REGULARE
BELLMAN OPERATOR
(Relative- (Relative)Jnnrrppy regularization has been shown to encourage exploration and prevent early conver- gence to sub-optimal policies (Nachum et al.) 2017). To take advantage of this idea when developing a robust RL algorithm we extend the robust Bellman operator to robust entropy regularized Bellman operator and prove that it contraction We also show that well-know value iteration bounds can be attained using this operator. We first define the robust entropy-regularty
value function (- (r(s, a) log x(-22] + pnnnppp Eg/~p((lss.)) (s'; m)]]- For the re-
V RK-KL(s;
*(-la)
mainder of this section, we drop the sub-and superscripts, as well as the reference policy conditioning, from the value function kr (s; T), and simply represent it as Vss) for brevity. We define the robust entropy-regularize Bellman operator for fixed policy ı in Equation [2] and show is a max norm contraction (Theorem J.
TR-KL (s) = Ba a) - log (1-ss)
+ inf E (etpt)
(2)
(-11s) PEp
Theorem 1. The robust entroox--eeuularized Bellman operator xI a fixed policy is contrac- tion operator: Specifically: VU, V E RRSS and (0, 1), we have, U -R'x VII S >|l - Vil.
The proof can be found the (Appendix[E) Theorem Using the optimal robust entropy- regularized Bellman operator 7R.KI. sup, R-KI, which is shown to also be a contraction operator in Appendix standard value iteration error bound can derived (Appendix) Corollary I].
Theorem 2
C RoBt-ROUUST
BELLMAN OPERATOR
In this section, we derive a soft-robust entropy-regularizeed Bellman operator and show that it is also
a 7-contraction in the max norm. First, we define the average transition model as á = B [p) which corresponds to the average transition model distributed according to some distribution over the uncertainty set P. This average transition model induces an average stationary distri- bution (see Derman et al. (018)). The soft-robust entropp-regullrrized value function is defined as V
SR-KL (s: +t) = (-- (rss a) -log π[-(a]
T(-|=) 1 K (s) Again, for ease of
notation, we denote V/R KI. (s) = (s) for the remainder of the section. The soft-robust entropy-
regularized Bellman operator for fixed policy defined as:
T (.1s8) T V(s) = (1)r()))((ssaa) - log a(-ss)
+ + [Vs')]]
(3)
which is also contraction mapping (see Appendix
F Theorem 3) and yields the same bound as Corollary for the optimal soft-robust Bellman operator derived in Appendix Theorem
D ROBUST ENTROPY-REGUULRTTEE POLICY EVALUATION
To extend Robust policy evaluation to robust entropy-regulariized policy evaluation, two key steps need to be performed: (1) optimize for the entropy-regularicn expected return as op- posed to the regular expected return and modify the TD update accordingly; (2) Incorporate
'Note that while MPO already bounds the per step relative entropy we, in addition, want regularize the action-value function to obtain robust regularized algorithm.
12

---

Published as a conference paper at ICLR 2020
robustness into the entropy-regulanize expected return and modify the entropy- regplar-re TD update. To achieve (1), we define the entropy-regularized expected return as Qxi (s, a; Φ = r(s,a) a) (K (-)s))]((((ss))) Bx"=p((ls.a)) KL" (s'; #)], and show in Appendix that perform- ing policy evaluation with the entropy-regularized value function is equivalent to optimizing the ontropy-regularized squared TD error (same Eq. equation + only omitting the inf operator). To achieve (2), we optimize for the robust entropy regularized expected return objective defined QR (s, a; #) r(s, a) <KL(T* (+)](())) infpep Bx TT robust cntropy-regularee squared TD enror:
( (s'; #)], yielding the
Thi o ~ p(-1s at), at+1 ~ TT((11841);T)
min Cety inf p(.,
(4)
2
-KLL(TA(-18441 ~ (())s
((((ss,a))))
Q (0
where QR-KL remove the infimum from the TD update and replace the next state transition function p(-1st, ax) with the average next state transition function p(-(st,a)) az).
(x; RT) = TR (s, az #)) ( For the soft-robust setting, we
Relation to MPO: As in the previous section, this step replaces the policy evaluation step of MPO. Our robust regraizyee Bellman operator R-KI. and soft-robust pntropy-reguarrzer Bellman SR-KL ensures that this process converges to unique fixed point for the policy Ty for the robust and soft-robust cases respectively. We use πT the reference policy æ. The pseudo code for the R-MPO, RE-MPO and Soft-Robust Engropy-regullrizzed MPO (SRE-MPO) algorithms can be found in Appendix | (A Algoriithms 1 and respectively).
operator T T
E
PROOFS
Theorem 1.
Proof. We follow the proofs from Tamar al. 2014 Iyengar 2005 and adapt them to account for the additional entropy regularization for fixed policy æ. Let U, V R IiS and S arbitrary state. Assume T (s) 2 TK (s)). Let e > 0 be an arbitrary positive number.
By the definition of the inf operator, there exists Ps such that,
E (anr(119)[ -log T(-ss) *(-ss) + M., (V
inf E E log T-|s8) PEP
(-|ss) + (e + (5)
In addition, we have by definition that:
E (1)r(1))))) 7lo (-1ss) 7--1s) + 7Eg "1ss
> inf E Ea Eanr( log T(-18) pEP
x(-1s) + Ees (6)
Thus, have,
13

---

Published as a conference paper at ICLR 2020
0 S RKi U(s) - KI. (ss)
T(-ss)
(a)r(())(((ssa)) log
+ B (1..))
7(-
π (-18))
a) - log
+ y (le +e
x(-ls)
= (( - E Ba (l te
$ rlu +E
Applying a similar argument for the case KR. S TR. (8)) results in
T U - KR VI < 7 - V|| +
(8)
Since e an arbitrary positive number, we establish the result, i.e.,
| TR VI S > VII-
(9)
14

---

Published as a conference paper at ICLR 2020
Theorem 2.
Proof. We follow similar argument to the proof of Theorem | Let U, V E R and s E S an arbitrary state, Assume (s)) TR-KL Vss). Let > be arbitrary positive number. By definition of the sup operator, there exists ă E II such that,
#(-ls)
inf EE (- ((s, log (-(s) + 1eBe (U()]]] Uss) -e (10)
PEP
In addition, by the definition of the inf operator, there exists Pa P such that,
- #(-18) *(-1s) + >p [V('']]
E ((=A(())(((8aa)) - Flog
*(1ss)
inf (s log #-188) PEp
+ 7B (o.) te (1I)
Thus, we have,
0 s Us) - TR.XI (ss)
#(113)
< (inf E BewA(1))r(ss a) - - log PEp
+ 7B, (0 +c)
x(1s8)
- (inn (s) T log A (-1ss)
+ +EE, + l (V())]]
pEP
. (-|ss)
(B(ss(()(((s,a)) - log (-1ss)
(12)
ALE ( [U(' [[aa) +
(-1s))
(-1ss)
(Enns((s)(((ssa)) -log (--(s)
7 (1.00) [ss))] c)
(oo0 (U(()) B (((s)) 2
$ r Vl| + 2e
Applying a similar argument for the case TR-KI U(s) < TR-KLL Vss) results in
K U KR. V < < V 2c.
(13)
Since e is an arbitrary positive number, we establish the result, i.e.,
URRKUU TR V S S/U VI|-
(14)
15

---

Published as a conference paper at ICLR 2020
Corollary 1. Let N be the greedy policy after applying N value iteration steps. The bound between the optimal value function V and VEN, the value function that is induced by nN, is given by,
2ne 2- N+1
- - V** ||
(1-1)a (1-7) I Voll. where maNugkeN TR Vi Vk+ill is the
function approximation error, and Vo is the initial value function.
Proof. From Berteskas (1996), we have the following proposition:
Let V be the optimal value function, V some arbitrary value function, æ the greedy policy with respect to V, and V the value function that is induced by Thus,
27 (1
I vell S
11 VII
(15)
Next, define the maximum projected loss be:
|| Vi- - Va+1ll OSKEN
max
(16)
We can now derive a bound on the loss between the optimal value function V and the value function obtained after N updates of value iteration (denoted by VN) follows:
I Vxll S V TR-KL VN || +
Vxll
|R. V TR-KL V + || VN-I Vill
7 -ViN-1ll + TR.XL VN-1 -wll
$ >1lV* -VN-1l1+ +
(17)
(1 7 ... + -e 7 N M Voll
e
- Voll
11 g}
Then, using Lemma E get:
27
| - VTN || V
5
||V* Vvvll
(1 7)
27
6
27 ~ ~ -Vol
< (1 7) (1 - 7)
+ (1 (1-))²
(18)
246
NT1+
(1 7)2 +
(1 (1 V Voll
which establishes the result.
16

---

Published as a conference paper at ICLR 2020
F SOFT-ROBUST ENTROPY-REGULARIZE BELLMAN OPERATOR
Theorem 3.
Proof. For an arbitrary U, V E Ri and for fixed policy
|R. U() - KI. 6s
sup (a a) - log (113))
+ 7B 10..))
16.0) [U(6)]]
(-1s)
(a - T log (-1s))
+ / [V
x(-1s)
= 7su Σ p(s' s. ())((()) V(s)]]
$ suu Σ
v(s')l
S ssup Σ a6 v(s')llo
-
Theorem 4.
Proof. Let U. e Rl and E S an arbitrary state. Assume TSR (s) 2 TSR V(s). Let 0 be an arbitrary positive number. By definition of the sup operator, there exists E such that,
*(-ls) *--18)
E (r) r
+ (U))))) > U()) (19)
Thus, have,
0 x U(s) - TSR.KI. (x))
+(-13)
< (F - - log T(-[s)
+ 1 [U(s']] + +c)
(-18)
(sup (s r log 7 (-1s)
+ 7E, Be
nell
I (19 - -Flog ---13)
(20)
+ (s [U(']]]
(-|ss)
(-(ss)
(Eans())) a) log
+ + (V('))))
म (-1s)
=o= hu -
E
+
U Vll VIl+ +E
Applying a similar argument for the case TSR-KL (s) < TSR-KL (s) results
KL - KI. V < U V +
(21)
Since e is an arbitrary positive number, we establish the result, i.e..
|TRR-LU U - TSR-KL 5/>0/ - VI].
(22)
17

---

Published as a conference paper at ICLR 2020
G ENTROPY-REGULARZZE POLICY EVALUATION
This section describes: (1) modification to the TD update for the expected return to optimize for the entropy-regulanee expected return, (2) additional modification to account for robustness.
We start with (1).
The regroiy-re value function is defined as:
(s;
E 7 (()=- = =|
(23)
and the corresponding entropy-regularized action value function is given by:
(kL(,,
ET
Σ
(ln -8, 8,00 a)
(24)
1=0
r(s,a) (LL + E xp VV (5':
(25)
Next, we define:
(s, a; r(s, a) + ~p(la))))) (s'
(26)
thus,
Qkl (s, a) Qk (s, a,; ([((((())]]
(27)
Therefore, we have the following relationship:
ViL (s': = (,) (sL
(a (s, as, (KL(((s)]
(28)
We now retrieve the TD update for the entropy-regulariized action value function:
rt - ([(ss)))
QQt P(-(se, 0t), de+1 ~ a( ((((8441);;))
(1t,
= Tr - 7-L(TT(- (x (1)))) + (81+1 ~ P(- ur), aes1 ~ (()(044));
(se az5 (<)
re + 72K (8t41 P(11 at), a+t1 (-119111:: - QR, (st, 01;
(29)
= -r + क7 (8t41 P( P(- az) a+4 x-1(5(++1;;
- -T5L( P(. ar (l) P(.)se,))))) QRL ((
Note that in the above TD update we replaced Qk, with QR.-
Next, move to (2).
Before extending the TD update to the robust case, we first consider the robust entropy-regularized value function, which is defined as:
KI..I (8)
inf
( = =]
7 (
(30)
PEP
t=0
18

---

Published as a conference paper at ICLR 2020
Applying similar steps above yields the following TD update:
6g = - - (KL(((( + y inf
kL ~ p(11s ar). ar+1 ~ (+15((+));
pEP
(9t ae;
T (- ( + inf (z ~ p(-1s 0r41 ~r (-1344+);;
PEp
- -Rkk (0t, ag R) + (+++
(31)
s4, at), =r + inf QR (( att1 ~ ((1191+1); - (8,,
PEp
T p(( a++1 ae+1 ((1(+++))
- -r inf pEp
- p(1191 p(((ss)))))) ( (G;
19

---

Published as a conference paper at ICLR 2020
Hyperparameters
SVG
Policy net
200-200-2000
Q function net
500-500-500
Discount factor (r)
0.99
Adam learning rate
0.0003
Replay buffer size
1000000
Target network update period
200
Batch size
1024
Activation function
clu
Tanh on output of layer norm
Yes
Layer norm on first layer Tanh on Gaussian mean
Yes
Yes
Min variance
0.1
Max variance
unbounded
Table I: Hyperparameters for SVG
H EXPERIMENTS
H.1 ADDITIONAL DETAILS ON THE SVG BASELINE
For the stochastic value gradients SVG(0) baseline we use the same policy parameterization as for our algorithm, e.g. have
Tg = NN(no(s),
where / denotes the identity matrix and oo(s) is computed from the network output via softplus activation function.
To obtain baseline that is, in spirit, similar to our algorithm we used SVG in combination with Entropy regularization. That is, we optimize the policy via gradiend ascent, following the reparame- terized gradient for given state sampled from the replay:
V Volnatalal [Q(a, s)] all (a(als))
(32)
which can be computed, using the reparameteriizttion trick, as
(ec.vo.. () aV (T(al)))))
(33)
where go(s, <) = (s) + o * Ç is now a deterministic function of a sample from the standard multivariate normal distribution. See e.g. Heess al. (2015a) (for SVG) as well as Rezende et al. (2014): Kingma & Welling (2013) (for the reparameterization trick) for a detailed explanation.
H.2 EXPERIMENT DETAILS FOR MPO AND SVG
In this section we outline the details on the hyperparameters used for the MPO and SVG algorithms. All experiments use feed-forward two layer neural network with 50 neurons to map the current state of the network the mean and diagonal covariance of the Gaussian policy. The policy given by Gaussian distribution with a diagonal covariance matrix. The neural network outputs the mean μ (us) and diagonal Cholesky factors A A(s), such that Σ AAT The diagonal factor A has positive diagonal elements enforced by the softplus transform A <- log(1 + exp(AA;)) to ensure positive definiteness of the diagonal covariance matrix. Tables and show the hyperparameters
used for the MPO and SVG algorithms.
H.3 UNCERTAINTY SET PARAMETERS
Table 3 contains the chosen uncertainty set values for each of the domains and the corresponding holdout set perturbations. The final column of the table contains the parameter that perturbed.
20

---

Published as a conference paper at ICLR 2020
Hyperparameters
MPO
Policy net Number of actions sampled per state
200-200-200
15
Q function net
500-500-500
0.1
0.01
6x
0.00001
Discount factor (n)
0.99
Adam learning rate
0.0003
Replay buffer size
1000000
Target network update period
200
Batch size
1024
Activation function
elu
Layer norm on first layer
Yes
Tanh on output of layer norm
Yes
Tanh on Gaussian mean
No
Min variance
Zero
Max variance
unbounded
Table 2: Hyperparameters for MPO
Table 3: The parameters chosen for the uncertainty set perturbations as well as the holdout set perturbations. The final column contains the parameter that was perturbed.
Domain
Uncertainty Perturbations
Hold-out Test Perturbations
Parameter
Acrobot
1.0. 1.025, 1.05 meters
1.15. 1.2. 1.25 meters
First pole length
Cartpole Balance
0.5, 1.9, 2.1 meters
2.0, 2.2, 2.3 meters
Pole length
Cartpole Swingup
1.0, 1.4, meters
1,2, 1.5, meters
Pole Length
Cheetah Run
0.4, 0.45, 0.5 meters
0.3, 0.325, 0.35 meters
Torso Length
Hopper Hop
-0.32, -0.33, -0.34 meters
-0.4, -0.45, -0.5 meters
Calf Length
Hopper Stand
-0.32, -0.33, -0.34 meters
0.4, -0.475, -0.5 meters
Calf Length
Pendulum Swingup
1.0, 1.1, 1.4 Kg
1.5, 1.6, 1.7 Kg
Ball Mass
Walker Run
0,225, 0,2375, 0.25 meters
0.35, 0.375, 0.4 meters 0.35, 0.375, meters
Thigh Lengths
Walker Walk
0.225, 0,2375, 0.25 meters
Thigh Lengths
Shadow hand
0.025, 0.022, 0.02 meters
0.021, 0,018, 0.015 meters
Half-cube width
Cartpole Balance:
0.5, 1.9 .2.1 meters
0.5, 0.7, 0.9, I.I,
Pole Length
Larger Test Set
1.3, 1.5, 1.7, 1.9 meters
Pendulum Swingup:
1.0, 1.1, 1.4 meters
1.0, 1.1, 1.2,
Pole Length
Larger Test Set
1.3, 1.4, 1.5 meters
Pendulum Swingup- Offline datasets
1.0, 1.I, 1.2 Kg
1.5, 1.6, Kg7
Ball Mass
Cartpole Swingup- Offfine datasets
1.0, 1.4, 1.7 meters
1.2, 1.5, 1,8 meters
Pole Length
H.4 MAIN EXPERIMENTS
This section contains sets of plots. Figure contaains bar plots comparing the performance of RE-MPO (blue bars), SRE-MPO (green bars) and E-MPO (red bars) across nine Mujoco domains. The performance of the agents function of evaluation steps is shown in Figure 7 for all nine domains respectively. Figrue shows the bar plots for R-MPO, SR-MPO and MPO and Figure 9 shows the corresponding performance of the agents as function of evaluation steps.
H.5 INVESTIGATIVE EXPERIMENTS
This section contains additional investigative experiments that were mentioned in the main paper.
Figure II presents the difference in performance between the catropy-regularin agents and the entropy entropy-regularizer regularized agents agents. Although the performance comparable (left figure), the entropy-regularize, version performs no worse on average than the non non-entropy-oo antropy-regularizee agent. In addition, there are some tasks where there is large improvement in performance, such as the
21

---

Published as a conference paper at ICLR 2020
domein
doma D0
195tmdMingup
1000
010
7.00
600
500
400
200
F-MPO
RE MPO
200
100
5114
Erw2
EnV1
Env
Ehr2
hopper,
domain-we loe.
300
600
10000
fastmwal
250
5.0
Nowe Naeward
200
4.00
2.50
Noveape
300
400
1.0
200
100
200
Enw0
En92
EmV2
znv
Ehx2
Ene
Env1
Emi2
tia hoppor,
domain-wa kor.
Jaskmstand
pe Lamm,
Leskemun
800
teak
400
Avorge Rovard
Aveeage Reward
350.
Meorage Rowar
250
400-
200
200
100
tmv
Figure 6: All nine domains showing RE-MPO (blue), SRE-MPO (green) and E-MPO (red).
1000
200
250
700
200
600
230
200
400
Zealsaties
1:4
104
Enariation
144
000
1M4 moc.:ema-muulp-- S0nt, anghtt
cark 2004
210
500
200-
400
500
1.50
200
100
2.2
0.5
1.0
1.5
3.5
2.5
3.0
0.0
0.5
3.0
3.5
2.0
255
300
0.0
0.5
1.0
2.5
3.0
2.5
550
5c0
400
700
300
500
200.
0.5
0.5
1.0
1.5
3/3
2.5
3.0
0.9
0.5
2.2
Dralkartion Sto2e
Etatudtio Stane
Eralastion Stego
161
Figure 7: All nine domains showing RE-MPO (blue), SRE-MPO (green) and E-MPO (red) function of evaluation steps during training.
22

---

Published as a conference paper at ICLR 2020
deroenmeerppol tastnsmingup
Gisk-balance
(22k-2inngu
800
700
50-1PD
Roseage Remeac
600
500
400
400
200
200
100
Env
Ehx2
Emvo
Eni2
domain-wn lier
3.0
600
10000
tastmwalk
5.0
2.50
Nose Naeward
4:00
200
Noveape
500
130
400
100
200
100
200
Enw0
En92
Emv2
knv
Ehk2
Env1
Emi2
domal hopper.
done n-/ol
1ae em
U25kider rguup
300
Aowege
Wrmgge
150
100
Ervo
Env2
Ehvi
Eriz
Figure All nine domains showing R-MPO (blue), SR-MPO (green) and MPO (red).
1000
250
700 600
2.
100
150
50
400
Zealsaties
1:4
104
Enuriation
144
eettet Talk www.:rmmm-mtttt www.aamaaaaa...... Sont anght
400
$55-
200
1.0
2.2
0.5
1.0
1.5
3.5
2.5
3.0
0.0
0.5
3.0
3.5
2.0
255
300
0.0
0.5
1.0
2.5
3.0
2.5
300
400
2co
50-
200
300
0.5
0.5
3.0
1.5
3/3
2.5
0.9
0.5
2.2
Dralkartion Stoze
Statuation Stane
Eralastion Stego
Figure 9: All nine domains showing R-MPO (blue), SR-MPO (green) and MPO (red) as a function of evaluation steps during training.
23

---

Published as a conference paper at ICLR 2020
1002
1032
bwwege Soaare
902-
932
002
8020
200
702
732-
400
432
Evsluation
100
Steps
16t
Soman pemannn pardaau
www. Crassss
700
700
500
600
500
400
300
200
200
200
200
0.0
0.5
3.0
3.5
2.5
2.5
0.3
0.5
3.0
1.5
2.5
3.0
0.0
35
7.0
1.5
2.0
2.5
Figure 10: Increasing the range of the training uncertainty set for Cartpole balance (top row) and Pendulum swingup (bottom row).
Average
450
Cheetah
400
400
350
350
Average Reward
300
300
250
250
200
Non- Non-Robust without KL
Average
200
Non Robust with KL
150
Robust without KL
150
100
100
Soft-Robust without KL
50
50
Soft-Robust with KL
Comparing entropy-regularrzzed objective to the non-entropy regularized objective (left figure). The entropy--reuulrized version does no worse than the non entropy-regularized setup and in some cases, for example Cheetah, performs considerably better than the expected return objective (right figure).
Figure 11:
Cheetah task for the entropy-regularized agent variants non entropy-regularized agent variants (right figure).
Training with more samples: Adding three times more samples to the non-robust baseline still yields significantly inferior performance compared to that of the robust and soft-robust versions as seen in Figure 12 for Cartpole balance and Pendulum swingup respectively.
As can be seen in the figure, RE-MPO makes better use of limited number of perturbations compared to Limited-DR in Cartpole Balance (left) and Pendulum Swingup (middle) respectively. If the number of perturbations are increased 100 (right figure) for Pendulum Swingup, DR, which uses approximately 30 times more perturbations, improves but still does not outperform RE-MPO.
What about Domain Randomization? The DR results are shown in Figure 13
Modifying the uncertainty set: Figure 14 contains the performance for cartpole balance (top row) and pendulum swingup (bottom row) when modifying the uncertainty set. For the Cartpole Balance task, the original uncertainty set training values are 0.5, 1.4 and 2.1 meters for the cartpole arm length. We modified the third perturbation (2.1 meters) of the uncertainty set to pole lengths of 1.5, 2.5 and 3.5 meters respectively. The agent is evaluated on pole lengths of 2.0, 2.2 and 2.3 meters respectively. As seen in the top row of Figure [4] as the training perturbation is near the evaluation set, the performance of the robust and soft-robust agents are near optimal. However, as the perturbation increases further (i.e., 3.5 meters), there is drop in robustness performance. This
24

---

Published as a conference paper at ICLR 2020
domain=cartpole, task-balance
domain domain-pendulum
ta task=swinggp
1000
600
800
Average Reward
Average Reward
400
600
300
400
200
200
100
Envo
Env2
Envo
Env1
Env2
Figure 12: Additional Training Samples: Two plots show times more additional training samples for non-robust E-MPO (dark grey) in the Cartpole Balance and Pendulum Swingup tasks respectively.
1000
tescmsaingup
800
tpal salngup
600
Average Roverd
500
400
400
400
EMPO
DR
RE-4PD
SRE-MPO
100
Eric
Figure 13: Domain Randomization (DR): Domain randomization performance for the Cartpole balance (left) and Pendulum swingup (middle) tasks. As we increase the number of perturbations for DR to 100 (right figure), we that performance improves but still does outperform RE-MPO, which still only uses 3 perturbations.
probably due to the agent learning a policy that robust with respect to perturbations that are relatively far from the unseen evaluation set. However, the agent still performs significantly better than the non-robust baseline in each case. For Pendulum Swingup, the original uncertainty set values of the pendulum 1.2, 1.3 and 2.0 meters respectively. The agent is evaluated on unseen lengths of 1.5, 1.6 and 1.7 meters. A significant increase in performance can be seen in the bottom of Figure 14 the third perturbation approaches that of the unseen evaluation environments. Thus it appears that if the agent able to approximately capture the dynamics of the unseen set, then the robust agent is able to adapt to the unseen test environments. Figure I0 presents the evaluation curves for the corresponding Cartpole Balance (top row) and Pendulum swingup (bottom row) tasks as the third perturbation of the uncertainty set is modified.
are 1.0, 1.1 and 1.4 meters. We modified the final perturbation to values of
environments within the training
Different Nominal Models: Figure 15 indicates the effect of changing the nominal model to the median and largest perturbation from the uncertainty set for the Cartpole balance (top row) and Pendulum swingup (bottom row) tasks respectively. For Cartpole, since the median and largest perturbations are significantly closer to the evaluation set, performance of the non-robust, robust and soft-robust agents are comparable. However, Pendulum swingup, the middle actor still far from the evaluation set and here the robust agent significantly outperforms the non-robust agent.
25

---

Published as a conference paper at ICLR 2020
210
1503
8:0
Pestage Ramera
Nettgg Recare
592
600
600
602
402
200
230
0
(y54054 noup
C-460
Iascnon ngup
002
702
A2-100
546 MPO
003
130
102
6n40
Ens2 Enve
Enez
Ehva
Emi
Figure 14: Modifying the uncertainty set: The top row indicates the change in performance for Cartpole balance as the third perturbation of the uncertainty set is modified to 1.5, 2.5 and 3.5 meters respectively. The bottom row shows the performance for Pendulum Swingup for final perturbation changes of 1.2, 1.3 and 2.0 meters respectively.
domalneccrrpolo, task=balars
domainecarpodle task-balance
1200
1000
1000
Average Rewwrrd
800
400
400
200
Env0
Env1
Eri2
Ervo
Env1
Env2
domain-penduumm
500
tatkussinguup
600
tasks swingup
400
5.00
Avarage Boward
300
Avergge
300
200
Env1
Envo
Env2
Figure 15: Changing the nominal model: The top two figures indicate setting the nominal model as the median and largest perturbation of the uncertainty set for Cartpole Balance respectively. The right two figures the same setting but for the Pendulum swingup domain. Legend: E-MPO (red), RE-MPO (blue), SRE-MPO (green).
26

---

Published as a conference paper at ICLR 2020
Algorithm I Robust MPO (R-MPO) algorithm for single iteration
1: given batch-size (K), number of actions (N), old-policy Te and replay-buffer // Step 1: Perform policy evaluation on m. yield Q
07 (#o+1 ae), a0t1 Th m(((le6t1)) 20 (2))
min +
inf pEP(s)))))
repeat
Sample batch of size N from replay buffer
6:
// Step 2: sample based policy (weights)
7:
q{a;]s;) = fij. computed as:
8:
for j = I.....kdd do
for l.....N
10:
a, ~ mi(als,)
11:
Qi = Q** (s), Q4)
12:
Çb Compute Wui (See (Abdolmaleki et al.
2018b)
13:
// Step 3: update parametric policy
14:
Given the data-set {zjz (as, (t.))=t.... W j=
Update the Policy finding
16:
Mo+1 argmax, >5 D, qúz (aiss))));)
17:
(subject to additional (KL) regularization)
18:
until Fixed number of steps
19:
return T
ALGORITHM
The Robust MPO algorithm is defined as Algorithm [ The algorithm be divided into three steps: Step (1) perform policy evaluation on the policy from the action value function Q Step (3) update the policy by minimizing the KL divergence between the proposal distribution & and the policy æ. The corresponding robust entropy-regularized version can seen Algorithm 2 and the robu-robust entropy-regularized version in Algorithm)
Step (2) build proposal distribution g(a s)
27

---

Published as a conference paper at ICLR 2020
Algorithm 2 Robust Settropy--eeularizze MPO (RE-MPO) algorithm for a single iteration
I: given batch-size (K), number of actions (N), old-policy ne and replay-bufier
// Step I: Perform policy evaluation тy yield Q**
3:
O (8:42 ~ p(-1st, ar), 0441 ~ =T(+(((4)):T))
min rety
inf pe ((e,,ae)
- ~
~ p( -Kt
repeat
5:
Sample batch of size N from replay buffer
6:
// Step 2: sample based policy (weights)
7:
q(a](s;) [8;] 4sj. computed
for j = I.....K
9:
for i I.....N do
10:
~ xx(a(s;)
11:
Q. QTh (s;,
12:
Øi Compute Weights( Qu (see (Abdolmalcki et
2018b)
13:
// Step 3: update parametric policy
Given the data-set (sy, (as,
15:
Update the Policy by finding
16:
Th+1 argmax Σ3 EN 9t log (a.s()));))
17:
(subject to additional (KL) regularization)
18:
until Fixed number of steps
19:
return Tk++1
Algorithm 3 Soft-Robust Entrooy-RReglaaizzee MPO (SRE-MPO) algorithm for single iteration
I: given batch-size (K), number of actions (N), old-policy m. and replay-buffer // Step I: Perform policy evaluation on mi yield Q
min rett Sne G ~ Fl se, oz), a+t1 ~ (=((4)))))
0
-TKI(TT(-)O ~ (114 pl. (()))) -
4: repeat
5:
Sample batch of size N from replay buffer
6:
// Step 2: sample based policy (weights)
7:
q(a.(s;) = Qsj. computed
for j l.....K
9:
for i 1.....N do
10:
~ mı(als;)
11:
Q2 QTh (sj, 04)
12:
Øb Compute Weights((Qu
(see (Abdolmalcki al. 2018b)
13:
// Step 3: update parametric policy
Given the data-set (sys (as, 4:) 1..) 1= b Update the Policy by finding
15:
16:
Te++ argmax, Σ* EN 98 (a:l(););;
17:
(subject to additional (KL) regularization)
18:
until Fixed number of steps
19:
return mt++
28


Reviews:
[{"review_id": "ryePWBCTtB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n# Summary\nThe paper compares 3 ways to account for the variability of the dynamics model in the TD error computation:\n(i) be robust (take the lowest target value obtained when evaluating the value-function on the training distribution of models);\n(ii) be Bayesian (average over models);\n(iii) domain randomization (compute TD error as usual but randomize the dynamics to obtain a more diverse data set).\n\nThe comparison is carried out using the MPO algorithm. An entropy term is furthermore added to the MPO objective to yield a new E-MPO algorithm, but it didn't affect the performance much (cf. Figs. 5-6 and 7-8).\n\nThe paper is mainly an empirical study.\n\n# Decision\nThe engineering effort of running many experiments and ablation studies is appreciated. However, there are big questions to the evaluation methodology and the contribution of the paper, that preclude me from recommending it for publication in its current form.\n\n# Concerns\n1) First, my concern is the problem setting and evaluation methodology, and in particular Table 3 with varied parameters. The authors chose to have 3 values during training, and then evaluate the optimized policy on further 3 testing environments. There are many questions to this table. I give only a few.\n        - For most environments, training and test ranges are disjoint (e.g., train on 1.00–1.05m and test on 1.15–1.25). Why? Would your results hold if they are not disjoint?\n        - Fig. 14 shows that when trained on a single model with parameters closest to the evaluation range (right column), there is no difference between the algorithms on the Cartpole and a minor difference on the Pendulum. That seems to imply that it is sufficient to just train on one environment which is most similar to the evaluation environments to perform well in the experiments. Is it true? Do you have a counterexample?\n        - Natural question, of course, what do you propose to do when varying multiple parameters? Discretizing over several values will quickly yield exponential explosion.\n        - Did you try using distributions over ranges instead of fixed parameters? That would be a bit more solid than hand-picked values.\n\n2) Evaluation of domain randomization. The authors say that \"TD errors are then averaged together\" when doing domain randomization. That means, when updating the value function, gradients corresponding to various models get intermixed. It is usually better to fix the domain, then do a few gradient updates using the data from that domain, and after that collect data from another domain. Such procedure is similar to keeping a target network in Q-learning, which is absolutely necessary to stabilize the learning process. It would be interesting to see if domain randomization still performs poorly when applied in this way.\n\n3) It would be helpful to sharpen the main message.\n        - Right now there is the E-MPO introduced, but then it is not really compared to plain MPO (only plots are shown in the Appendix); it is only said that it doesn't hurt to have the entropy term. It seems orthogonal to the robustification scheme in the TD error, so maybe it would make sense to omit the entropy for clarity of exposition.\n        - What is the conclusion? The fact that robust works better than non-robust is obvious even without any experiments. So, the only question one could have is whether it is better to average or to take the worst-case model. I didn't get it from the paper whether there is any conclusion regarding that.\n        - In Conclusion, the authors appeal to some theoretical contribution of this paper, such as proof of contraction in Theorem 1. I would consider it a very minor contribution as it is mainly a recapitulation of known results as both the robust Bellman operator and the entropy-regularized one are known to be contractions. Furthermore, this theorem is not really used in the paper, so to me it seems detached.\n\nSmall: typo in Sec. 7.: you say \"to lengths of 2.0, 2.1 and 2.2\", but according to Table 3, these should be \"2.0, 2.2 and 2.3\"\n\n# AFTER REBUTTAL\nThe authors incorporated the feedback of the reviewers and *significantly* rewrote the paper. Now it is more focused and puts the work in context by discussing connections to Grau-Moya etc. I still think the contribution is minor and the method will not scale, but from the formal side, the paper fulfills the requirements to be accepted.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rkxmdUuG9r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\nRebutal Response: \nI keep my rating of weak accept due to the extensive empirical evaluation. However, I want to point out that the Area chair should assign more weight to the other reviews as the other reviewers have provided more extensive reviews and have more knowledge about the existing literature.\n\n\n###########\nSummary: \nThe paper proposes a robust variant of MPO and evaluates the performance on control tasks including the cartpole, hopper, walker & the shadow hand. The performance shows that the robust MPO outperforms vanilla MPO and MPO + Domain randomization. \n\nOpen Questions:\n- I am not convinced by the domain randomization performance as the performance increases only marginally over the vanilla MPO and DR has usually performed quite well for robust policies. Can you explain this marginal increase? \n\n- Could the authors please include a qualitative evaluation of the learned controllers? Especially as the Cartpole stabilization  is a linear system one could compare against the analyitc optimal controllers. \n\nConclusion: \nCurrently, I would rate the paper as weak accept, as the derivation is interesting and the approach seems to work quite well in  simulation. However, the work is only incremental and does not propose a new perspective to robust RL. It would be nice to show that the policy is also robust for a physical system. \n\nMinor Comments\n- What does the bar mean, e.g., \\bar{J} & \\bar{\\pi} mean? I cannot find a definition of the bar.\n- Typo in the Background section 'Kullback-Liebler  (KL)' \n\n(I can try to extend my review if the other reviewers disagree)", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1gRd9T3YH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "AFTER REBUTTAL:\nThe authors answered all my questions and (significantly) modified the paper according to all the comments from all reviewers. The paper has significant value given the extensive experimental results and I believe could provide insight to other researchers in the field. I still have (like other reviewers) some concerns about scalability regarding the space of possible perturbations, but, as it is, the paper is worth for publication. For these reasons, *I increase my score to 7*, since I believe it's better than a weak accept (ignore the system rating, since it doesn't allow to put 7). \n\n======================================================\nSummary:\n\nThe authors provide a framework to incorporate robustness against perturbed dynamics to RL agents. They build upon the MPO algorithm and incorporate a few modifications that enable such robustness and entropy regularization. They show in multiple experiments that the robust versions of MPO outperform the non-robust ones when testing on environments with novel perturbations.\n\n\nDecision:\n\nI am happy with the experiments conducted on this paper and I think they would be a nice contribution to the community. However, I have serious concerns about the novelty of the proposed approach. The authors claim many novel contributions, but most of them are special cases of existing literature (not cited), or are simple modifications to existing approaches. For this reason my score is a reject. If the authors clearly cite the existing literature, state their contributions accordingly, and redirect the paper more towards their good experimental results (which they have plenty) I would be willing to substantially upgrade my score. Find below my supporting arguments.\n\n\nAbout novelty\n=============\n\n\nThe authors claim to have a novel framework and novel MDP formulations  for robust MDPs disregarding a substantial volume of the literature on robustness in the context of planning, control, reinforcement learning and MDPs. Since the authors really consider two types of robustness (robustness to dynamics and robustness to action stochasticity) I will expose in the following the relevant literature that is missing on both types, and when combining both.\n\n\n\nRobustness to Dynamics:\n\nBart van den Broek, Wim Wiegerinck, and Hilbert J. Kappen.  Risk sensitive path integral control. In UAI, 2010.\n\nArnab Nilim and Laurent El Ghaoui.   Robust control of markov decision processes with uncertain transition matrices.Operations Research, 53(5):780–798, 2005\n\nWolfram Wiesemann, Daniel Kuhn, and Berc  Rustem.  Robust markov decision processes. Mathematics of Operations Research, 38(1):153–183, 2013.\n\nLars Peter Hansen and Thomas J Sargent.Robustness. Princeton university press, 2008\n\nYun Shen, Michael J Tobia, Tobias Sommer, and Klaus Obermayer. Risk-sensitive reinforcement learning. Neural computation, 26(7):1298–1328, 2014.\n\nYinlam  Chow,  Aviv  Tamar,  Shie  Mannor,  and  Marco  Pavone.   Risk-sensitive  and  robust decision-making: a cvar optimization approach.   In Advances in Neural Information Pro-cessing Systems, pages 1522–1530, 2015.\n\n\n\nRobustness to Action Stochasticity:\n\nRoy Fox, Ari Pakman, and Naftali Tishby.  G-learning: Taming the noise in reinforcement learning via soft updates.arXiv preprint arXiv:1512.08562, 2015.\n\nJonathan Rubin, Ohad Shamir, and Naftali Tishby.  Trading value and information in mdps.In Decision Making with Imperfect Decision Makers, pages 57–74. Springer, 2012.\n\nDaniel A Braun, Pedro A Ortega, Evangelos Theodorou, and Stefan Schaal.  Path integral control  and bounded  rationality.   In Adaptive  Dynamic  Programming  And  ReinforcementLearning (ADPRL), 2011 IEEE Symposium on, pages 202–209. IEEE, 2011.\n\n\n\nCombination of both:\n\n[1] Grau-Moya, Jordi, et al. \"Planning with information-processing constraints and model uncertainty in Markov decision processes.\" Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, Cham, 2016.\n\n\nImportantly, in [1] above, it is shown a very similar line of research as the authors are proposing. In [1] it is combined entropy regularization with robustness in dynamics. In particular, using the formulation in [1] one can see how the formulations in the present paper can be recovered by setting the parameters in various ways. For example,\n1- letting the entropy regularization coefficient ($\\alpha$ in [1]) to a finite value and the robustness coefficient ($\\beta$ in [1]) to $-\\infty$ one recovers the traditional Robust MDP formulation.\n2- Selecting a uniform prior over discrete theta (where each component in theta would correspond to a particular perturbation)  and setting $\\beta \\rightarrow 0$ one can also recover Soft-RE formulation (although a non-MPO version of it, i.e. just the MDP formulation).\n3- Similarly, one can get rid of the entropy regularization by setting $\\alpha \\rightarrow \\infty$.\n\nNote that in [1] the uncertainty set is determined by a KL constraint whereas here it is determined by the chosen perturbations. However, in [1] setting the proper prior and $\\beta$, one can obtain the same behaviour.\n\n\nImportantly, in all previous references multiple contraction proofs (for entropy regularization Bellman operators, for robust operators alone, and for robust and entropy regularized operators) have already been discovered. \n\nI hope all of the previous points can convince the authors about my decision above regarding novelty. Note, please, that I am aware that a robust version of MPO is slightly novel, however, a robust MDP formulation is more general than that and it could have perfectly been applied to any other RL optimization algorithm. Therefore, I don't think adding robustness to MPO alone is sufficient for enough novelty (other wise we would publish 1 paper per method converting one by one all known algorithms into their robust version of it). \n\n\nAbout scalability\n=================\n\nI liked the experimental section. However, I have a question about practicality of the approach. As the authors show in the paper, the proposed method scales well in terms of  high-dimensional state and action spaces. But what about the scalability in terms of possible perturbations? I can imagine a plausible situation where we are dealing with a high-dimensional robot, let's say a humanoid robot, which allows for many possible \"perturbations\" e.g. longer left leg, longer right leg, longer fingers, shorter thumbs, and so on. The space of possible perturbations can be potentially very big. Could the authors elaborate on scalability issues of this type? Have they already thought about how one would solve this? I imagine that perturbing, let's say in one dimension, might change the dynamics quite in a different way compared to perturbing in another dimension. It would be interesting to see how their method generalizes when testing on unseen \"dimension\" perturbations.\n\nAlthough this might present problems in terms of generalization, I understand that it does not invalidate the results presented here.\n\n\nGood experimental results:\n\nI have carefully checked all experimental results and I must say that they are very well executed e.g. multiple ablation studies and sufficient environments to support their approach.  Maybe a small comment: In figures 6 and 8 in the appendix there are multiple plots that only differ in the data (the titles are the same, e.g. two hopper figures with the same title). Maybe the authors can correct for this.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors provide a framework for improving robustness (if the model of the dynamics is perturbed) into the RL methods, and provide nice experimental results, especially in the updated version. I am happy to see that the discussion for this paper went in a totally positive and constructive way which lead to a) constructive criticism of the reviewers b) significant changes in the paper c) corresponding better scores by the reviewer. Good work and obvious accept.
Decision: Accept (Poster)