Monotonic Robust Policy Optimization with Model Discrepancy
Authors: Yuankun Jiang
Chenglin Li
Junni Zou
Wenrui Dai
Hongkai Xiong
Venue:  (2021)
Keywords: Reinforcement Learning
generalization

Abstract: State-of-the-art deep reinforcement learning (DRL) algorithms tend to overfit in some specific environments due to the lack of data diversity in training. To mitigate the model discrepancy between training and target (testing) environments, domain randomization (DR) can generate plenty of environments with a sufficient diversity by randomly sampling environment parameters in simulator. Though standard DR using a uniform distribution improves the average performance on the whole range of environments, the worst-case environment is usually neglected without any performance guarantee. Since the average and worst-case performance are equally important for the generalization in RL, in this paper, we propose a policy optimization approach for concurrently improving the policy's performance in the average case (i.e., over all possible environments) and the worst-case environment. We theoretically derive a lower bound for the worst-case performance of a given policy over all environments. Guided by this lower bound, we formulate an optimization problem which aims to optimize the policy and sampling distribution together, such that the constrained expected performance of all environments is maximized. We prove that the worst-case performance is monotonically improved by iteratively solving this optimization problem. Based on the proposed lower bound, we develop a practical algorithm, named monotonic robust policy optimization (MRPO), and validate MRPO on several robot control tasks. By modifying the environment parameters in simulation, we obtain environments for the same task but with different transition dynamics for training and testing. We demonstrate that MRPO can improve both the average and worst-case performance in the training environments, and facilitate the learned policy with a better generalization capability in unseen testing environments.

Under review as a conference paper at ICLR 2021
MONOTONIC ROBUST POLICY OPTIMIZATION WITH MODEL DISCREPANCY
Anonymous authors Paper under double-blind review
ABSTRACT
State-of-the-art deep reinforcement learning (DRL) algorithms tend overfit in some specific environments due to the lack of data diversity in training. To mit- igate the model discrepancy between training and target (testing) environments, domain randomization (DR) can generate plenty of environments with sufficient diversity by randomly sampling environment parameters simulator. Though standard DR using uniform distribution improves the average performance on the whole range environments, the worst-case environment is usually neglected without any performance guarantee. Since the average and worst-case perfor- mance are equally important for the generalization in RL,, in this paper, we propose policy optimization approach for concurrently improving the policy's perfor- mance in the average case (i.e., over all possible environments) and the worst-case environment. We theoretically derive lower bound for the worst-case perfor- mance of given policy over all environments. Guided by this lower bound, we formulate an optimization problem which aims to optimize the policy and sam- pling distribution together, such that the constrained expected performance of all environments maximized. We prove that the worst-case performance mono- tonically improved by iteratively solving this optimization problem. Based the proposed lower bound, we develop a practical algorithm, named monotonic robust policy optimization (MRPO), and validate MRPO several robot control tasks. By modifying the environment parameters in simulation, we obtain environments for the same task but with different transition dynamics for training and testing. We demonstrate that MRPO can improve both the average and worst-case perfor- mance in the training environments, and facilitate the learned policy with better generalization capability in unseen testing environments.
INTRODUCTION
With deep neural network approximation, deep reinforcement learning (DRL) has extended classical reinforcement learning (RL) algorithms to successfully solving complex control tasks, e.g., playing computer games with human-level performance (Mnih et al.) [013 Silver et al. 2018) and continu- robotic control (Schulman et al. (017). By random exploration, DRL often requires tremendous amounts data to train a reliable policy. It thus infeasible for many tasks, such as robotic control and autonomous driving, training the real world is not only time-consuming and expensive, but also dangerous. Therefore, training is often conducted on a very limited set of samples, resulting in overfitting and poor generalization capability. One alternative solution is to learn policy in simulator (i.e., source/training environment) and then transfer it to the real world (i.e., target/testing environment). Currently,
impossible to model the exact environment and physics of the real world. For instance, the physical effects like nonrigidity and fluid dynamics are quite difficult to be accurately modeled by simulation. How to mitigate the model discrepancy between the training and target environments remains challenging for the generalization in RL.
To simulate the dynamics of the environment, domain randomization (DR), a simple but effective method is proposed. It randomizes the simulator (e.g., by randomizing the distribution of environ- ment parameters) to generate a variety of environments for training the policy in the source domain. Compared with training in a single environment, recent researches have shown that policies learned through an ensemble of environment dynamics obtained by DR achieve better generalization per- formance with respect to the expected return. The expected return is referred to the average per-

---

Under review as a conference paper at ICLR 2021
formance across all the trajectories sampled from different environments. Since these trajectories, regardless of their performance, are uniformly sampled, the trajectories with the worst performance would severely degrade the overall performance.
In contrast, another line of research on the generalization in RL is from the perspective of control theory, i.e., learning policies that are robust to environment perturbations. Robust RL algorithms learn policies, also using model ensembles produced by perturbing the parameters of the nominal model. EPOpt (Rajeswaran et al. 2017), a representative of them, trains policy solely on the worst performing subset, i.e., trajectories with the worst a percentile of returns, while discarding all the higher performing trajectories. other words, seeks a higher worst-case performance the cost of degradation on the average performance. In general, robust RL algorithms may sacrifice perfor- mance on many environment variants and focus only on environments with the worst performance, such that the policy learned will not behave very badly in previously unseen environment.
In this paper, we focus on the generalization issue in RL. and aim to mitigate the model discrepancy of the transition dynamics between the training and target environments. Considering that both the average and worst-case performance are equally important for evaluating the generalization capabil- ity of the policy, we propose a policy optimization approach in which the distribution of the sampled trajectories are specifically designed for concurrently improving both the average and worst-case performance. Our main contributions are summarized as follows.
For given policy and a wide range of environments, we theoretically derive a lower bound for the worst-case expected return of that policy over all the environments, and prove that maximizing this lower bound (equivalent to maximizing the worst-case performance) can be achieved by solving an average performance maximization problem, subject to constraints that bound the update step in policy optimization and statistical distance between the worst and average To the best of our knowledge, this theoretical analysis of the relationship between the worst-case and average performance is reported for the first time, which provides a practical guidance for updating policies towards both the worst-case and average performance maximization.
environments.
Trajectories obtained from diverse environments may contribute differently to the generalization capacity of the policy. Therefore, in face of huge amount of trajectories, the problem that which types of trajectories are likely to mostly affect the generalization performance should be considered. Unlike traditional uniform sampling without the worst-case performance guarantee, and different from the worst percentile sampling in which the parameter is empirically preset, we propose a criterion for the sampling trajectory selection based on the proposed worst-case and average performance maximization, with which both the environment diversity and the worst-case environments are taken into account.
Based on the proposed theorem, we develop a monotonic robust policy optimization (MRPO) al- gorithm to learn the optimal policy with both the maximum worst-case and average performance. Specifically, MRPO carries out a two-step optimization to update the policy and the distribution of the sampled trajectories, respectively. We further prove that the policy optimization problem can be transformed to trust region policy optimization (TRPO) (Schulman et al. (2015) on all possible environments, such that the policy update can be implemented by the commonly used proximal policy optimization (PPO) algorithm (Schulman et al. (017)). Finally, we prove that by updating the policy with the MRPO, the worst-case expected return can be monotonically increased.
To greatly reduce the computational complexity, we impose Lipschitz continuity assumptions on the transition dynamics and propose a practical implementation of MRPO. We then conduct experiments on five robot control tasks with variable transition dynamics of environments, and show that MRPO can improve both the average and worst-case performance in the training envi- ronments compared to DR and Robust RL baselines, and significantly facilitate the learned policy with a better generalization capability in unseen testing environments.
2 BACKGROUND
Under the standard RL setting, the environment is modeled as a Markov decision process (MDP) defined by a tuple S,A, R >. S is the state space and A is the action space. For the con- venience of derivation, we assume they are finite. T S x A x S + [0, 1] is the transition dynamics determined by the environment parameter p E P. where P denotes the environment pa-
2

---

Under review as a conference paper at ICLR 2021
rameter space. For example in robot control, environment parameter could be physical coefficient that directly affect the control like friction of joints and torso mass. Throughout this paper, by en- vironment p. mean that environment has the transition dynamies determined by parameter R. S x A - IR is the reward function. At each time step i, the agent observes the state 8, E S and takes action a₂ A guided by policy (ar]se)) Then, the agent will receive a reward = R(5 at) and the environment shifts from current state 8, to the next state 5t+1 with probability ( (e; p). The goal of RL is search for policy ı that maximizes the expected cumula- de;
tive discounted reward (p)) E, [G(T[p)], G(Tlp) Σi= T'r.. T {st, dz, Pt, St41 1 denotes the trajectory generated by policy in environment p and y [0. is the discount fac- tor. We can then define the state value function as (s) E - 7 7* = s]. the tion value function as Q= a)
o r*rx*//s= = s, a = a] and the advantage function as (s, a) (8, - (s). We denote the state distribution under environment p and policy i as Px (slp) and that at time step t as (slp). During the policy optimization in RL, by updating the current policy ı to a new policy i. Schulman et al. (2015) prove that
2Xy (1 - 7)2 S L. m)(ppp) - E (())a))
i(aas)
7(+pp) 2 L (mlp)
A (s,a) TTaas)
where A max, [Baantala (ale) (8, is the maximum mean advantage following current policy and 3 = max, Drv(n (+((s) [[ (+1s) is the maximum total variation (TV) distance between ı and t. The policy's ss expected return after updating can be monotonically improved by maximizing the lower bound in (T) w.r.t. Based on this and with certain approximation, [Schulman al. (2015) then propose a algorithm named trust region policy optimization (TRPO) that optimizes т towards the direction of maximizing L (f). subject to the trust region constraint B S
In standard RL, environment parameter is fixed without any model discrepancy. While under the domain randomization (DR) settings, because of the existence of model discrepancy, environment parameter should actually be random variable following probability distribution P over P. By introducing DR, the goal of policy optimization to maximize the mean expected cumulative discounted reward over all possible environment parameters, i.e., max, Ep- [p)].
In face of model discrepancy, our goal is to provide a performance improvement guarantee for the worst-case environments, and meanwhile to improve the average performance over all environments.
Lemma 1. Guided by certain policy there exists non-negative constant C' O, such that the expected cumulative discounted reward in environment with the worst-case performance satisfies:
( Ep-p [n(x|p)] 2 -C,
(2)
where environment Pw corresponds to the worst-case performance, and C is related pw and n.
Proof. See Appendix A.I for details.
Theorem 1. In MDPs where reward function is bounded, for any distribution P over P, by updating the current policy a to a new policy it, the following bound holds:
7Bp-P [c(p [p))
4/r|maxa
n(Flp.)) 2 Ep-r [n(((pp)] 2/
(3)
(1 7)-
(1 71-7)2²
where c(pulpp) max; P' B Bawa(- Drv (T(s)s', Pw (s a. p)), environment
Pw corresponds to the worst-case performance under the current policy n, and a P
max, Ey Drv (ewiDir(aaaa)))) (I
Proof. See Appendix
A.2 for details, and Appendix A.7 for bounded reward function condition.
In B). e(p (p) specifies the model discrepancy between two environments pv and p in terms of the maximum expected TV distance of their transition dynamics of all time steps in trajectory sampled in environment Pw using policy म, and a denotes the maximum expected TV distance two policies along trajectory sampled in environment Pa using policy т. In general, the RHS of (3) provides a lower-bount for the expected return achieved in the worst-case environment p... where the first term denotes the mean expected cumulative discounted reward over all environments following the
3

---

Under review as a conference paper at ICLR 2021
Algorithm 1 Monotonic Robust Policy Optimization
Initialize policy To. uniform distribution of environment parameters U, number of environment
parameters sampled per iteration M, maximum number of iterations N and maximum episode length T.
2: for k = 0 to N do
3: Sample a set environment parameters { M- according to U.
4: for = 0 to M do
5:
Sample I. trajectories (Tr,) /--] in environment Pt using Tk-
6:
Determine ph arg min p. (==) M- Σj=0 G(Tr,)lPt)/
7: Compute E(p, m&) = Σ 6--1 G(T.ylp.) 2/r pe for environment p;-
(1-
8: end for
9: Select the trajectory T ( E(p, T&) 2 Ep - }
10: Use PPO for policy optimization on T to get the updated policy Tk+1-
end for
sampling distribution P. while the other two terms can be considered as penalization on a large TV distance between the worst-case environment Pw and the average case, and large update step from the current policy æ to the new policy म, respectively. Therefore, by maximizing this lower bound, we can improve the worst-case performance, which in practice is equivalent to the following constrained optimization problem with two constraints:
max E (#[p)) R.P
s.t. a 1S01, Epur (c(P [p)) S 62-
The optimization objective is to maximize the mean expected cumulative discounted reward over all possible environments, by updating not only the policy म. but the environment parameter's 's sampling distribution P. The first constraint imposes similar trust region to TRPO (Schulman et al. 2017) that constrains the update step in policy optimization. region constraint
addition, we further propose a new trust the sampling distribution P that the TV distance between the worst-case envi- ronment Pv and average case over P is bounded, such that by achieving the optimization objective in 44, the worst-case performance is also improved.
To solve the constrained optimization problem in (4). we need to seek for the optimal policy Ω and the distribution P of the sampled trajectories. In practice, we carry out a two-step optimization procedure to simplify the computational complexity. First, we fix the policy by letting т k, and this case, we no longer need to consider the
optimize the objective in (4) w.r.t. the distribution first constraint on the policy update, and thus can convert the second constraint on the sampling dis- tribution into the objective with the guidance Theorem formulating the following unconstrained optimization problem:
max E [E(p, #]]: P
(5)
The first term in E(p, i)) indicates policy #'s performance in environment p, while the second term measures the model discrepancy between environment p and Pw: Since the objective function in (5) linear to P. we can update P by assigning higher probability environment p with higher E(p, Ā). As consequence, sampling according to (pp #) would increase the sampling probability of environments with both poor and good-enough performance, and avoid being trapped in the worst-case environment. Specifically, propose to select samples from environment that meets Ep, т) > E(pw. т) for the training of policy which equivalent assigning zero probability the other samples.
where we denote Epp, #) & n((ppp) - 2| (I-y)2
In the second step, we target at optimizing the policy æ with the updated distribution fixed, i.e., the following optimization problem:
max Ep-p (n(ppp)
s.t.
5 61
(6)
Optimization in 66) can be transformed to trust region robust policy optimization similar to TRPO and solve it practically with PPO (refer to Appendix A.3 [Schulman et al. (2017) for more infor- mation). To summarize, we propose monotonic robust policy optimization (MRPO) in Algorithn
4

---

Under review as a conference paper at ICLR 2021
At each iteration k uniformly sample M environments and run trajectory for each sampled en- vironment, For each environment Pi, we sample trajectories {Tr.))f=1> approximate (7(xx(p)) with Σf=o G(Ta.llp))/ L, and determine the worst-case environment pw based on Σド= G(T)))Pi)) of given set of environments pii=o MM11 We then optimize the policy with PPO on the selected trajectory subset T according to (ps, T&) and (p TT).
We now formally show that by maximizing the lower bound provided in Theoren the worst-case performance within all the environments can be monotonically improved by MRPO.
Theorem 2.
The sequence of policy 1 T2:....,TN. generated by Algorithm 7 is guaranteed with the monotonic worst-case performance improvement, i.e.,
イ lww) S ((T2ll2)) < ... S (TTNNPEE)
where puu denotes the parameter of environment with the worst-case performance guided by the current policy Tik at iteration k.
Proof. See Appendix A.4 for details.
PRACTICAL IMPLEMENTATION USING SIMULATOR
Motivated by Theorem] we propose Algorithm| that provably promotes monotonic improvement for the policy's performance in the worst-case environment according Theorem[] However, The- orem [ imposes calculation of e(pw (pp) that requires the estimation of the expected total variation distance between the worst-case environment and every other sampled environment each time step. Estimation by sampling takes exponential complexity. Besides, in the model-free setting, we unfortunately have access to analytical equation of the environment's transition dynamics. Hence, computation for the total variation distance between two environments unavailable. Under the de- terministic state transition, environment will shift its state with probability of one. Hence, can set state of one environment's 's simulator to the state along the trajectory from the other environ- ment, and take the same action. We then can compare the next state and compute the total variation distance step by step. Though feasible, this method requires large computational consumption. In this section, we propose instead a practical implementation for Algorithm
According to Appendix A.9, we first make strong assumption that the transition dynamics model -ipshhpsc in terms of the environment parameter p:
[ a, Tssls' a, a.Pl)l Lpllp Pwll
Then, we can simplify the calculation of e( (p)) via:
(pwllp)) S max PlaapP(((pppnn Ba (-1s)) Explle Lppll Pwll = Σ Iip 11p Pur
be seen from the expression of e(por that measures the transition dynamics distance be- tween the worst-case environment Pw and a specific environment p. In simulator, the environment's s transition dynamics would vary with the environment parameters, such as the friction and mass in robot simulation. Hence the difference between environment parameters can reflect the distance be- tween the transition dynamics. In addition, if we use in practice the penalty coefficient of c(p.e recommended by Theorem the subset T would be very small. Therefore, we integrate it with Lp as a tunable hyperparameter K, and propose a practical version of MRPO in Algorithm in Appendix A.6
EXPERIMENTS
We now evaluate the proposed MRPO in five robot control benchmarks designed for evaluation of generalization under changeable dynamics. These five environments are modified by the open- source generalization benchmarks (Packer et al. 2018) based on the complex robot control tasks introduced in (Schulman et al.] 2017). We compare MRPO with two baselines, PPO-DR and PW- DR, respectively. In PPO-DR, PPO is applied for the policy optimization in DR. In PW-DR,
5

---

Under review as a conference paper at ICLR 2021
Table 1: Range of parameters for each environment.
Environment
Parameters
Training Range
Testing Range
Density
[750, 1250]
[1. 750]
Walker2dHHopprrHaaiihhhhhah
Friction
00.5, 1.1]
00.2, 0.3]
HalicheeethhBroadRange
Density
[750, 1250]
[,00]
Friction
[0.2, 2.25]
[0.05, 0.2]
Inverte@DoublePenduuuum
Polel length
00500, 1.00]
[1.00, 1.10]
Pole2 length
(0.50, 1.00]
[1.00,
Cartpole
Force magnitude
[1.00, 20.00]
10
Pole length
[0.05, 1.00
[1.00, 6.00
Pole mass
[0.01,
[1.00, 6.00)
purely trajectories from the 10% worst-case environments for training and still apply PPO for the policy optimization. Note that PW-DR is the implementation of EPOpt algorithm proposed in (Ra-) jeswaran et al., (017) without the value function baseline. Since this value function baseline in EPOpt is not proposed for the generalization improvement (in fact could incur performance im- provement for all the policy optimization algorithms), we do not adopt it in PW-DR. We utilize two 64-unit hidden layers construct the policy network and value function in PPO. For MRPO, the practical implementation as described in Algorithm 2 Further note that during the experiments, find that environments generating poor performance would be far away from each other in terms of the TV distance between their transition dynamics with the dimension of changeable parameters increasing. In other words, single worst-case environment usually may not represent all the en- vironments where the current policy performs very poorly. Hence, we choose the 10% worst-case environments to replace the calculation of E' Epht. T&) in Algorithm 2 That is, the trajectories we add to the subset T should have the E' (p.,TTk) greater than and equal to those of all the 10% worst-case environments. From the expression of Ed (", m Σ)= G(Tr.)lP.)/L4 x lpp pt.ll for environ- ment pi, it can be seen that the trajectory selection based on trade-off between the performance and the distance to the worst-case environment, as we described in detail in the paragraph under (5). Our experiments are designed to investigate the following two classes of questions:
Can MRPO effectively improve the policy's worst-case and average performance over the whole range of environments during training. Compared to DR, will the policy's s average performance degrade by using MRPO? Will MRPO outperform PW-DR the worst-case environments?
How does the performance of trained robust policy using MRPO degrade when employed in environments with unseen dynamies? And what determines the generalization performance to unseen environments during training?
4.1 TRAINING PERFORMANCE WITH DIFFERENT DYNAMICS
The five robot control tasks follows. (1) Walker2d: control a 2D bipedal robot to walk;
(2) Hopper: control a 2D one-legged robot to hop as fast as possible; (3) HalfCheetah: control 2D cheetah robot run (Brockman et al. 2016); (4) InvertedDoubleeenduuue
control a cart (attached to a two-link pendulum system by a joint) by applying force to prevent the two-link pendulum from falling over; and (5) Cartpole: control cart (attached to pole by joint) by ap- plying force to prevent the pole from falling over. In robot control, the environment dynamics are directly related the value of some physical coefficients. For example, if sliding friction of the joints is large, it will be more difficult for the agent to manipulate the robot compared to smaller sliding friction. Hence, a policy that performs well for a small friction may not by generalized to the environment with large friction due to the change of dynamics. In the simulator, by random- izing certain environment parameters, we can obtain set of environments with the same goal but different dynamics (Packer et al., 2018). The range of parameters that preset for training of each environment shown in Table
We run the training process for the same number of iterations N sampled from preset range of en- vironment parameters. At each iteration k. we generate trajectories from M = 100 environments sampled according uniform distribution U. The results obtained by running each algorithm

---

Under review as a conference paper at ICLR 2021
PM NEPP
HRRO
VIRO
2000
200
alloma
606
1000
200
3M
200 1900
1250
1999 2100
200
(a) Walker2d
(b) Hopper
(c) Halfcheetah
DA
PM
1909
10
atlons
itaradoon
1995
Itrratioss
(d) Walker2d
(c) Hopper
(f) Halfcheetah
PW-BR
FIN on
PM on VRRO
aresgerdtun
2809
T10 allons
2005
Rerations
1006
300
1901 1996
(g) InvertedDDubleeeenduuum
(h) Cartpole
(i) HaffchectahBroadRaage
ER
BA
DR PM 2011
10Rworrs coto retum
VRRO
vou
sorations
Iteradions
Heradens
(j)
InverteDDubleeenduuuum
(k) Cartpole
(1) HalfcheettahBrodRRangg
Figure 1: Training curves of average return and 10% worst-case return.
Table 2: Average and worst-case performance in training, where W, H, C, CP, I and CB refer to Walker2d, Hopper, Halfcheetah, Cartpole, InvertedDaubleeenddllum and HalfcheetahBrrodRRann
Algorithum
Average W
Worst W
Average HI
Worst II
Average C
Worst C
MRPO
1930.0 + 66.7
1702.5 + 185.9
2257.4 + 423.5
2048.0 + 246.8
2537.0 + 98.3 2970.0 + 30.0 1810.0 41.2
2340.0 + 61.2 2512.5 + 120.9 1677.5 72.3
DR
2170.0 + 57.9
1772.5 + 162.2
2250.0 + 80.1
1251.0 # 348.5 1440.0
PW-DR
+
Algorithm
Average
Worst
Average CP
Worst CP
Average
Worst CB
MRPO
9185.2 + 165.6
5137.0 + 1421.1 1354.5 + 381.2
998.7 1.7
988.4 15.4
2363.3 + 82.6
2166.7 + 70.4
DR
8252.6 = 139.8
995.5 + 5.6
961.6 = 18.3
26466.7 + 303.5
1883.3 = 103.7
PW-DR
9222.2 + 173.3
5450.4 + 1687.3
895.4 147.2
791.4 288.9
1456.7 + 301.3
1278.7 1: 215.9
with five different random seeds. The average return is computed over the returns of M sampled en- vironments at each iteration k. We show the training curves of Walker2d, Hopper and Halfcheetah, InvertedDoubleeendulni
Cartpole, and HalfchhettahBroadRangg. respectively, in Figs. [[a] and [gg I(i) In Figure 1, the solid curve is used to represent the average performance of each algo- rithm on all the five seeds, while the shaded-ares denotes the standard error. It is seen that DR can steadily improve the average performance on the whole training range expected, while MRPO does not significantly degrade the average performance in all the the tasks. PW-DR, on the other hand, focuses the worst-case performance optimization, leading obvious degradation of av-
7

---

Under review as a conference paper at ICLR 2021
(a) MRPO
(b) PWDR
(c) DR
12
204
201
dengi
density
densit
6044
friction
Friction
friction
(a) Walker2d
MRPO
(b) PWDR
(c) DR
304
density.
demsity
density
068
friction
friction
friction
(b) Hopper
(c) DR
1.4
density
311
friction
friction
friction
(c) HalfcheetahBrradRRngg
Figure 2: Heatmap of return in unseen environments on Waler2d, Hopper and Halfcheetah with policies trained by MRPO, PW-DR and DR the training environments.
erage performance on Hopper and Halfcheetah and failure on Walker2d. We measure the worst-case performance by computing the worst 10% performance in all the sampled environments at iteration and the corresponding training curves are illustrated in Figs. [[d) I(f) and [() I() respectively. It can be observed that MRPO presents the best worst-case performance
Hopper, Cartpole and HalichectahBrouRRangge while DR neglects the optimization on its worst-case performance and performs badly in these three tasks. PW-DR shows limited improvement of the worst-case perfor- mance compared to MRPO on Hopper, Halfcheetah, Cartphole and HallchectthhBroaRRngge and failure Walker2d. Comparing Figs. [(t and [(]] it can be concluded that the original parameter range we set for the Halfcheetah task (e.g.. the friction range [0.5, 1.1)) was too narrow to cause seriously poor performance on the 10% worst-case environments. By enlarging the friction range from [0.5, 1.1] to [0.2, 2.5] for HalfcheerahBroaRRange the training curves in Fig. 1()) can clearly demonstrate that MRPO outperforms the other baselines. The tabular comparison of the average and worst-case performance achieved during training by different algorithms in different tasks can be found in Table 2.
4.2 GENERALIZATION TO UNSERN ENVIRONMENTS
MRPO has been demonstrated in theoretical analysis to optimize both average and worst-case per- formance during training. Here, we carry out experiments to show that MRPO can generalize to broader range unseen environments in testing. To this end, we compare the testing performance on some unseen environments of Walker2d. Hopper and Halfcheetthh cies obtained by MRPO, DR and PW-DR from training, with the range of parameters set for testing showin in Table I It is observed that policies all degrade with the decrease of friction, while the impact of unseen density is not that obvious as the friction. The heatmap of return achieved all the testing Hopper environments by each al algorithm is shown and compared in Fig. 2. can be seen that
with the best poli-
8

---

Under review as a conference paper at ICLR 2021
MRPO has better generalization ability to the unseen environments, while DR can hardly generalize in testing. Compared to PW-DR, MRPO has a broader generalization range, from which We remark that both the worst-case and average performance during training are crucial for the generalization to an unseen environment.
RELATED WORK
With the success of RL in recent years, plenty of works have focused on how to improve the general- ization ability for RL. Learning a policy that is robust to the worst-case environment is one strategy. Based on theory of Ho control (Zhou et al.) 1996), robust RL takes into account the disturbance of environment parameters and model it as an adversary that is able to disturb transition dynamics in order to prevent the agent from achieving higher rewards (Morimoto & Doya 2005). The policy optimization is then formulated as a zero-sum game between the adversary and the RL agent. Pinto al. (2017) incorporate robust RL to DRL method, which improves robustness of DRL in complex robot control tasks. To solve robust RL problem, robust dynamic programming formulates a ro- bust value function and proposes accordingly a robust Bellman operator [Iyengar 2005 Mankowitz et al., 2020). The optimal robust policy can then be achieved by iteratively applying the robust Bellman operator in a similar way to the standard value iteration (Sutton & Barto 2018). Besides, Rajeswaran al. (2017) leverage data from the worst-case environments adversarial samples to train a robust policy. However, the aforementioned robust formulations will lead to an unstable learning. What's worse, the overall improvement of the average performance over the whole range of environments will also be stumbled by their focus on the worst-case environments. In contrast, in addition to the worst-case formulation, we also aim improve the average performance.
For generalization across different state spaces, an effective way is domain adaptation, which maps different state space common embedding space. The policy trained on this common space can then be easily generalized to a specific environment (Higgins et al. 701 James et al. 2019 Am- mar et 2015) through a learned mapping, with certain mapping methods, such as 3-VAE (Higgins
et al. 201 7a), cGAN (Isola et al. 2017), and manifold alignment (Wang & Mahadevan 2009).
Function approximation enables RL to solve complex tasks with high-dimensional state and action spaces, which also incurs inherent generalization issue under supervised learning. Deep neural net- work (DNN) suffers overfitting due to the distribution discrepancy between training and testing sets. Iz-regularization, dropout and dataset augmentation (Goodfellow al. (016) play an significant role for generalization in deep learning, which have also enabled improvement of policy's generalization on some specifically designed environments (Cobbe et al.) 2019; Farebrother et al. 2018).
In terms of the theoretical analysis. Murphy (005) provide a generalization error bound for Q- learning, where by generalization error they mean the distance between expected discounted reward achieved by converged Q-learning policy and the optimal policy. [Wang et al. (2019) analyze the generalization gap reparameterizable RL limited to the Lipschitz assumptions namics, policy and reward function. For monotonic policy optimization in RL, Schulman et al.) (2015) propose to optimize a constrained surrogate objective, which can guarantee the performance improvement of updated policy. In the context of model- RL, Janner et al. (2019); Luo et al. (2019) formulate the lower bound for certain policy's performance on true environment in terms of the performance on the learned model. It can therefore monotonically improve the performance on true environment by maximizing this lower bound. Different from this, the proposed MRPO in this work can guarantee the robustness of the policy in terms of the monotonically increased worst-case performance, and also improve the average performance.
transition dy-
6 CONCLUSION
In this paper, we have proposed a robust policy optimization approach, named MRPO, for improv- ing both the average and worst-case performance of policies. Specifically, we theoretically derived a lower bound for the worst-case performance of a given policy over all environments, and formulated an optimization problem to optimize the policy and sampling distribution together, subject to con- straints that bounded the update step in policy optimization and statistical distance between the worst and average case environments. We proved that the worst-case performance was monotonically im- proved by iteratively solving this optimization problem. We have validated MRPO several robot control tasks, demonstrating a performance improvement on both the worst and average case envi- ronments, well as a better generalization ability to wide range of unseen environments.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Haitham Bou Ammar, Eric Eaton, Paul Ruvolo, and Matthew E Taylor. Unsupervised cross-domain transfer in policy gradient reinforcement learning via manifold alignment. In Twenty-Ninth AAAI Conference on Artificial Intelligence. Citeseer, 2015.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 2016.
Ya-Chien Chang, Nima Roohi, and Sicun Gao. Neural Iyapunov control. In Advances in Neural Information Processing Systems, pp. 2245-3254, 2019.
Karl Cobbe, Oleg Klimov, Chris Hesse, Tachoon Kim, and John Schulman. Quantifying generaliza- tion in reinforcement learning, In International Conference on Machine Learning, pp. 1282-1289. PMLR, 2019.
Jesse Farebrother, Marlos C Machado, and Michael Bowling. Generalization and regularization in dqn. arXiv preprint arXiv:1810.00123, 2018.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT press Cambridge, 2016.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botviniek, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. 2017a.
Irina Higgins, Arka Pal, Andrei A Rusu, Loic Matthey, Christopher P Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot trans- fer in reinforcement learning, arXiv preprint
2017b.
Phillip Isola, Jun Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 1125-1134, 2017.
Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2): 257-280, 2005.
S James, P Wohlhart, M Kalakrishnan, D Kalashnikov, A Irpan, Ibarz, S Levine, R Hadsell, and K Bousmalis. Sim-to-real via sim-to-sim: Data-efficient robotic grasping via randomized-to- canonical adaptation networks. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12619-12629, 2019.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. In Advances in Neural Information Processing Systems, pp. 12519- 12530, 2019.
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. Interna- tional Conference on Learning Representations, 2019.
Daniel J Mankowitz, Nir Levine, Rac Jeong, Abbas Abdolmaleki, Jost Tobias Springenberg, Yuanyuan Shi, Jackie Kay, Todd Hester, Timothy Mann, and Martin Riedmiller. Robust reinforce- ment learning for continuous control with model misspecification. In International Conference on Learning Representations, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 13/2.v:312..6000 2013.
Jun Morimoto and Kenji Doya. Robust reinforcement learning. Neural computation, 17(2):335-359, 2005.
Susan A Murphy, A generalization error for q-learning. Journal of Machine Learning Research, 1079) 2005.
10

---

Under review as a conference paper at ICLR 2021
Charles Packer, Katelyn Gao, Jemej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song. Assessing generalization deep reinforcement learning. arXiv preprint 1TXX1::8100.2282.. 2018.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce-
ment learning. In International Conference on Machine Learning, pp. 2817-2826, 2017.
Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learning robust neural network policies using model ensembles. 2017.
John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1140- 1144, 2018.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Chang Wang and Sridhar Mahadevan. Manifold alignment without correspondence. In IJCAI, volume 2, pp. 3. Citeseer, 2009.
Huan Wang, Stephan Zheng, Caiming Xiong, and Richard Socher. On the generalization gap in reparameterizable reinforcement learning. In International Conference on Machine Learning, pp- 6658, 2019.
Kemin Zhou, John Comstock Doyle, Keith Glover, et al. Robust and optimal control, volume 40. Prentice hall New Jersey, 1996.
11

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 PROOF OF LEMMA II
Proof. First, we define nT(T)pp)) p 77(ppp) - -Cu. where Cy 0 depends on and P. Then, given a policy π and any environment E P, and for any non-negative constant C 2 Co 2 0, we thus have:
<TT(Pp)) Ppup (n([[))] 2 mT((pp))
max ()(pp) = Co 2 -C.
(10)
pEP
A.2 PROOF OF THEOREE 1
Lemma 2. For any two joint distribution Pi(x,y) = (()))) and P2(x,y) = Pa(x)P2(ylx) over and y, we can bound the total variation distance of them by:
Drv(P((ss)))) S (rv(P((x)))(())) + max DrV PPPP)(()))))))))))))))))))))))))
(11)
Proof.
Drv Drv((((s.))11(((x.)) -
Σ
(1 - P2(2,
(12)
P3(1)) ((()))
(13)
2 J.
Σ (CB))(()))---P()))))))))))))))))
- (a(x))
2
3.9
(14)
P1 (1 P+(yl=) + (P)(()) P2(2)
(15)
2
2 2
N.8
=E EE,.
+ Dr. (Xx(((())))
(16)
Lemma 3. Suppose the initial state distributions (s) and (s) are the same. Then the distance in the state marginal at time step t bounded as:
Dr (rv(P(s))I)((s))) mmx p: Drv P666
(17)
Proof.
-P)()))P((s)
Σ Pi(s s(s)) PPs(s Pa(st = s(s')
(18)
Pi(s s16'))(- Pa(st s16')
(19)
Σ Pi s|s') Pa(st ols')
(20)
+
= = (6') Ps(s: s(s')
(21)
SE SE - P2(3|s') + Σ
(22)
12

---

Under review as a conference paper at ICLR 2021
Dry (P((()())))((()) 2 Σ (P((s) P2(S)
(23)
1 Σ
( - P2(s|5)11+ Σ Ps(s)s") P2-
(5')
(24)
R pp- p
(r + DTV (P- (25)
P (ro)))))))))))))))22))))))))))))
(26)
1=1
St max B Pt (Ro6)))))
(27)
Theorem 3. (A modified version of Theorem 7 For any distribution P over P. by updating the current policy a to a new policy it, the following bound holds:
n(#lpw) - Ep=p (n(#pp)] 2 -2/r BBpp (c(pw)ll))
4/x/maxx
(28)
(1 7)-
(1 - 7))
where <(wllp) A max, E.' (' (o E (10) Drv a, Pew Il a p)), environment Pu
corresponds to the worst-case performance, and a max, B Drv ([
Proof. We can rewrite the LHS of (2 as:
( (pw) Ep-p [n(Flp)] = () (()) - (w))pw)) + (<((p/)) Ep-p [n(([p)]
For the last two term, we have
E Spurln(T|pun) (w:) n((pp)) = =Epor
Σ 7
Pd (s,alp.))) PP(,a))))())
t
< <ppmp
-2/r| Imax
[Drv PP' (s, alp P*
(29)
where = and
= -
By Lemma
we have
EpeP [Drv (s. (pp.
<l DTV )
+ Ep- (P
(30)
Note that
P(s|s',Pw) Σ T(sls' a, (T)ass
(31)
P(s)s' = Σ T a, ()
(32)
Similar to Lemma
we have
Drv Pw) PP(ssss'
(33)
EΣ
(34)
2
Σ
-T6SB + ΣΣΣ
#(als')
(35)
-Eawr(
(7s + DrV (Tals
(36)
13

---

Under review as a conference paper at ICLR 2021
By Lemma
have
Ep-p Drv (slpw) (s((p)
StPp P max Be Drv (P(s)s', Pw) Prr p))
StBp max - (-Iow))
(T((s Pon))ssss a,,))
t max -pr (n (
(37)
Since <(pllpw) max, (t(ppl))pp Drv (aass inequality, we have
max, E. E. (1pp.. ( (1)) Drv (s(ss' a, Pw (((ss'ss aasss',.,) a)) and a
combining 29 (30) and (37 and referring Jensen's
nn Pw) (7
E p pwo) (p)
2//r max Σ / 1) max (tt
Dry (T (5 + max Be (ann (
-2/r max
Σn [(Eppp (e(p [[)] + aa) + a
-2/ max
>1Epp (e(Pwllp)]
+ a 7)2
(38)
(1 71-
With policy i be updated to म, n/(l)Pe)) S Epur [n(#|p)). Then,
pRurr [e [P)
( - Epar (+(ppp -2 max
(39)
7)
(1 71-
Similar to the derivation of 38 and refer to Janner et al. (019)) we have
m <TTppw) nTTPpw)) 2 (1 7)2
(40)
Combining the above results, we end up with the proof
yEpur (c(pw [[)]
4/rlmaxe
n(F|pw) Ep (n(ppp)]
-2/r|
(41)
71-
(1 7)2
A.3 DERIVATION OF POLICY OPTIMIZATION STEP
In the policy optimization step, we aim to solve the following optimization problem:
max max [n(# p)]
s.t. a 61.
(42)
Referring to T we have:
2 (1 7)
Epur (n((1p) Eppr [Te (+lp))
82
(43)
We now turn to optimize the RHS of 43 to maximize the objective in 6 under constraints
81:
2Xy
max Ep-r [T (+|p))
82
s.t. a 5r
(44)
(1 7)2
Note that we have: a max (1) D (T)((a)))))))
max (aaass))| 3.
(45)
Following the approximation in Schulman et al.
2015)
can be equivalently transformed to:
(a|s)
EppP Eg, (-p1)as
A (s, a)
8 5 8,
(46)
π (a|s)
which can be solved using PPO (Schulman et al.
2017
14

---

Under review as a conference paper at ICLR 2021
A.4 PROOF OF THEOREM 2
Proof. Denote H(* | max; Eg (Tv (1*ts)))**
Updating T to
Th+1 at each iteration k and following Theorem
[ we have
((*++1lp*)) Ep- p(TRt1
m|rlmax> (pllpp (1 - 7)2
4/rl ||lmma H (
(47)
(1 7)-
Since pk+1 and T++1 obtained by maximizing the RHS of 33, we have
2/rl max (e(px>e(pppp
4/r man H (TT TT+11)
Ep Ep-
(T<+1lpP)
(48)
(1-7)²
(1 7)2
2r 2rman 2 (e(pl1pt))
4/r Imax H (T<]|TTR)
> E pt+1
7(TTxpp) (p)
(49)
(1 -7
- 7)
(TTTxlp)) 2rmax 7 (e)(p(l)))) (pl1pt
-
(50)
(1 - 7)²
From Line 9 in Algorithm the environment selected for training satisfies:
(T (p) 2/r Tepplll)) 2 (TTxIP=)) 2/r mmx ve(ptr lpt)
(51)
7)
(1 7)2
Therefore, combining
47
have:
7(TT+1 + Rs (/TTT Rp- [7( [p "TTTA l.))
(52)
where the approximation is made under the assumption that the expected returns of worst-case environment between two iterations are similar, which stems from the trust region constraint we impose on the update step between current and new policies, and can also validated from experi- ments.
A.5 EMPIRICAL VERIFICATION OF ASSUMPTION THBBREM
To verify the assumption made in Theorem in Fig. 3, we study how the parameters of environ- ments with poor performance scatter the parameter space with different dimensions. Specifically, we plot the heatmap of return for the range of Hopper environments used for training, achieved by using MRPO to update the policy between two iterations. It can be validated that the iteration k 300, the poorly performing environments of the two policies before and after the MRPO update concentrate in the same region, i.e., the area of small frictions. The same result can be observed for the iteration 1: 350.
For example, as shown in Figs. 3(a) and 3(b), at iteration = 300, 2 = (750, 0.5). the MC estimation of (T (000) is 487.6 and that of (T (p200) is 532.0. At iteration h 301, p000 (1027.8, 0.5) and the MC estimation of a 1p 301 is 517.6. As shown in Figs. 3(c) and 3(d), (8 0.5), the MC estimation of 17(3350) p550 is 385.9 and that of (T |p (2250 250) is 422.2. At iteration & = 351, 1 (750, 0.5) and the MC estimation of (133511 351 is 394.0. In both cases, the empirical results can support the assumption that we made
iteration k 350, 350
in (52), i.c., the expected returns of worst-case environment between two iterations are similar.
A.6 PRACTICAL IMPLEMENTATION OF MRPO
can be seen from the expression of c(parl1) IP)) that measures the transition dynamics distance be- tween the worst-case environment Pw and specific environment p. In simulator, the environment's transition dynamics would vary with the environment parameters, such as the friction and mass in robot simulation. Hence the difference between environment parameters tween the transition dynamics. In addition, we use in practice the penalty coefficient c(p.. lIp) recommended by Theorem the subset T would be very small. Therefore, we integrate it with Lp in (9) as a tunable hyperparameter s, and propose practical version of MRPO in Al Algorithm 2
reflect the distance be-
15

---

Under review as a conference paper at ICLR 2021
15.1
8511
denaity
density
0003
100.3
1339
13309
A5
14
56
21
friction
friction
(a) itcration k=300
(b) iteration k=301
00.5
6611
69L1
density
1198
11:0.
119.5
404
119.1
05
CO 80
14
36
20
friction
Friction
(c) itcration k=350
(d) iteration k=351
Figure 3: Heatmaps of return between policy update at iterations &: 300 and k 350, using MRPO on Hopper.
Algorithm 2 Practical Inplementation of Monotonic Robust Policy Optimization
Initialize policy To. uniform distribution of environment parameters number of environment parameters sampled per iteration M, maximum number of iterations N and maximum episode length T.
2: for k k= 0 to N - do
M-1 according to U. 3: Sample a set environment parameters {p
4: for = 0 to M 1 do
5: Sample L trajectories (Tr,) 1-11 in environment pt using Tk-
6: Determine p arg min p. (p: (M:) Σ/=O G(T,,) IP:)/L
7: Compute (p:, TA) = Σ1-0 G(T,)lpd)
xllpx - pt for environment pa.
8: end for
9: Select trajectory set T = (T: E',,, T&) 2 E'( Tb)}.
Use PPO for policy optimization T get the updated policy T&+1-
11: end for
A.7 BOUNDED REWARD FUNCTION CONDITION IN ROBOT CONTROL TASKS
In Theorem 1
state the condition that reward function is bounded. Referring the source code of OpenAl gym [Brockman et al. 2016), the reward function for the five robot control tasks evaluated in this paper are listed below.
Hopper and Walker2d:
R = I1+1 2, 0.001
Halfcheetah:
R = F44 - 0.001
16

---

Under review as a conference paper at ICLR 2021
2002
time elapsed (10%
average return
5000
2003 iterations
iterations
(a) Performance
(b) Time clapsed
Training curves of average return of MRPO Hopper with different L; elapsed versus number of iterations curves during training.
Figure 4: (a)
Time
Cartpole:
R R=1, 1, if the pole does not fall down;
InvereeDDoublePendulum:
R b- Cdist Cuel-
In Hopper, Walker2d and Halfcheetah, 2++1 and H, denote the positions of the robot at timestep ++1 and respectively. For Hopper and Walker2d, b E {0, 1}, and b equals 0 when the robot falls down or otherwise. The squared norm action represents the energy cost of the system. Since the maximum distance that the robot move in one timestep and the energy cost by taking action at each timestep are bounded, these three tasks all have the bounded reward function. In Cartpole, the reward is always 1. In InvertedDDoullePedddde 10 otherwise, Cdist is the distance between the rohot and the centre, and oue! is the weighted sum of the two pendulum's 's angular velocities. Since all the three parameters Cdiat and Cuel are physically bounded, the reward function, as a linear combination of them, is also bounded.
b equals 0 when the pendulum falls down
A.8 ANALYSIS OF THE MONTE CARLO ESTIMATION OF (]([pp)
In Theorem 1, the worst-case environment parameter Pee needs to selected according to the ex- pected cumulative discounted reward s)(=(p) of environment p. However, 1y( p)) is infeasible to get the practical implementation. Therefore, as a commonly used alternative approach as (Ra- jeswaran et al.] (211)), use the mean of the cumulative discounted reward of L sampled trajec- tories 2j=0 C(Tr.lp))) to approximate the expectation ry(mpp:) E, [G(Tlp:) of any environ- ment Ps. by using Monte Carlo method. We then determined the worst-case environment Pa based M In the following, will analyze the --1 G(T.. pp:/LL of given set of environments pi:-o on Dj=
impact of L on the estimation error.
Theoretical analysis of the impact L: Referring to Chebyshev's inequality, for any environment and any
2 O, with probability of at least Lez, we have
Σ == G(Trolps)
E/5 Er., (G(Ta)lp))
END G(Trolpi)
(TT(p))))))))))))))))(()) -
Ss,
(53)
L
where Var Var (G(T)l)) is the variance of trajectory T's return. From the above equation, we find out that the variance of the return does affect the MC estimation of r/(=lp) and larger L can guarantee higher probability for the convergence E/ G(Tr.ll:)/L to ( (ppp.)).
Empirical evaluation of the impact of L: In practice, conduct experiment of MRPO Hop- per with different choices of L. We find out that the a larger L would not greatly affect the perfor- mance in terms of average return shown in Fig. 4(a) but will significantly increase the training
17

---

Under review as a conference paper at ICLR 2021
2005
==
ours
10%worrst- cassst-ccaeeeeee return
average return
1502
x= 20
1200
20
1292
x=60 60
60
750
502
220
iterations
iterations
(a) Average return
(b) 10% worst-case return
Figure 5: (a) Training curves of (a) average return and (b) 10% worst-case return of MRPO Hopper with different K.
time shown in Fig. 4(b) In other words, for the same number of training iterations, a larger L would consume significantly longer running time than a smaller L, while the performance is similar. Therefore, set L = 1 in our practical implementation of MRPO to strike trade-off between the approximation accuracy and time complexity training.
A.9 ANALYSIS OF THE LIPSCHITZ ASSUMPTION
In robot control tasks, classical optimal control methods commonly utilize the differential equation formulate the dynamic model, which then indicates that the transition dynamics model Tpp Lipschitz and this formulated dynamic function can be used to estimate the Lipschitz constant Ly-
For example, the inverted double pendulum, one our test environments, can be viewed as a two- link pendulum system (Chang al. 2019). To simplify the analysis, we illustrate here single inverted pendulum, which the basic unit that forms the inverted double pendulum system. The single inverted pendulum has two state variables 0 and and one control input u, where and represent the angular position from the inverted position and the angular velocity, respectively, and u is the torque. The system dynamics can therefore be described as
mgl sin + u 0.10
(54)
ml2
where the mass, g is the Gravitational acceleration, and the length pendulum. our set- ting. we may choose the variable environment parameter p. Since the above system dynamics are differentiable w.r.t. m, it can be verified that the maximum value of the first derivative of the system dynamic model can be chosen as the Lipschitz constant Lp.
A. HYPERPARAMETER
In Algorithm 2 when we update the sampling distribution for policy optimization, is hyperpa- rameter that controls the trade-off between the expected cumulative discounted reward 17(TH (px) and distance IIp: - to the worst-case environment. Theoretically, larger means that the policy environments, while smaller к would par more attention
cares more about the poorly-performin to the average performance. As empirical evaluation, we conduct experiment
MRPO on Hopper with different choices of hyperparameter s. The training curves of both average return and the 10% worst-cas case return are shown in Figs. 5(a) and ((b) respectively. It can be verified that for the fixed value choice of the curve of
outperforms the curves of A 20, 40, 60 in terms of the aver- age return in Fig. ((a) while the curve of x 60 outperforms the curves x = 5, 20, 40 in terms
of the 10% worst-case return in Fig. 5(b) In practical implementation, gradually increase x
18

---

Under review as a conference paper at ICLR 2021
(a) MRPO
(b) PWDR
(c) DR
pole mass
pole mass
pole length
pole length
pole length
(a) Cartpole
(a) MRPO
PWDR
(c) DR
length22
F length2s
lengtt2e 25
length1s
length1s
length1s
(b) InvertedDoubllPenndluum
Figure 6: Heatmap of return in unseen environments Cartpole and InvertedDoubllePnndulum with policies trained by MRPO, PW-DR and DR in the training environments.
a fixed high value. It can therefore strike tradeoff between average return and 10% worst-case return, demonstrating the best performance both in Figs. ((a) and ((b)
A.11 GENERALIZATION TO UNSEEN ENVIRONMENTS OF CARTPOLE AND INVERTEDDOULLEENDLLLE
In Fig. 6, show the comparison results of MPRO, PR-DR and DR unseen environments for the other two benchmarks, Cartpole and InvereeDDoulleenndttum,, to provide empirical support for the generalization capability of MRPO.
19


Reviews:
[{"review_id": "hVyquA5GeqP", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "summary:\nThis paper introduces Monotonic Robust Policy Optimization (MRPO), an RL algorithm that aims to jointly optimize policy and domain sampling distribution, with the goal of improving policy performance for both average and worst-case scenarios and addressing the model discrepancy between the training and target environments. They derive a lower bound for the worst-case performance, which comprises the average performance, policy change, and the statistical distance between the worst and average case environments. A TRPO-like monotonic performance improvement guarantee is provided for the worst-case expected return. Finally, a practical approximation to MRPO is proposed, which imposes the assumption on Lipschitz continuity with respect to the environment parameters and circumvents the estimation of total variation distance between the worst-case environment and the sampled environment. Experiments are conducted on three control tasks with diverse transition dynamics parameters, where MRPO could improve both average and worst-case performance in the training environments, and it shows better generalization to the unseen test environments than baseline algorithms.\n\n\npros:\n- The theoretical analysis is provided, which shows the relationship between the worst-case and average performance for the first time.\n\n- The algorithm is backed by the theoretical guarantee of monotonic worst-case performance improvement.\n\n\ncons:\n- The assumption that the transition dynamics model is L-Lipschitz with respect to the environment parameter seems to be strong.\n\n- Some of the experimental results are not convincing. For example, in Figure 1f, MRPO underperforms DR, even if DR does not consider the worst-case performance during optimization at all.\n\n\ncomments and questions:\n- How natural is the model's Lipschitz assumption? Are many real-world problems satisfying this assumption?\n\n- In Figure 1, what does the shaded-area stand for? standard deviation? standard error? Also, it is not clear that MRPO outperforms other baselines statistically significantly.\n\n- It seems that two dense layers are used to construct the policy and value networks in the experiments. Why was the recurrent (e.g. LSTM) policy not used? Since the recurrent policy can implicitly embed system identification, I think the performance of the DR baseline could have been improved with the use of the recurrent policy. It would be great to see the performance comparison when the recurrent policy is used for MRPO and baselines.\n\n- For the experiments on generalization to unseen environments, only the results for Hopper is provided, which may not be sufficient to demonstrate the behavior of each algorithm. It would be great to provide the heatmap results for other domains, i.e. Walker and HalfCheetah.\n\n- In Theorem 1, is $p_w$ is the worst-case parameter for $\\pi$? or for $\\tilde \\pi$? It would be good if notation presents the dependence on the policy of $p_w$, e.g. $p_w^\\pi$.\n\n- In Algorithm 2, line 6: how can $p_w^k$ be found? (even before completing sampling the trajectories for each environment)\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "edlYnHc1j4a", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on the generalization issue in reinforcemetn leanring, specifically aims to address the problems of domain randomization(DR) technique. Different from standard DR which treats all the sample environment as equal, this paper proposed to improve the performance over all possible environments and the worst-case environment concurrently. This paper theoretically derives a lower bound for the worst-case performance of a given policy over all environment, and in practical, the proposed method, monotonic robust policy optimization(MRPO) carries out a two-step optimization to imporve the lower bound such as to maximize the averaged and worst-case policy perfomance. \n\n\nThis paper is well written and the key concept is clearly introduced. The Theorem.1 makes the connections between the averaged and the worst-case performance, such that maximizing the worst-case performance can be solved by maximizing the averaged performance problem with some trajectories from environments with both poor and good-enough performance. The emprical results also support the theorical analysis.\n\n1. For Lemma 1: The conclusion is based on the assumption that the the worst case $\\rho(\\pi|p_w) - \\max_p \\rho(\\pi|\\rho)$ is bounded (Proof A.1). However, such equation does not strictly holds without bounded reward function. The author should stated the condition.\n\n2. About the monotonic worst-case performance improvement theorem, the proof says \"... the approximation is made under the assumption that the worst-case environment between two iterations are similar, which stems from the trust region constraint we impose on the update step between current and new policies...\", however, the trust region constraint can only limit the difference between policy updates, the similarity between worst-case environments can not be promised.\n\n3. In theorem 2, the fomula (50) and (51) in the proof, is this approximation reasonable? Since the policy is updated, the worst-case environment may have changed a lot. Similarly, if the updated policy changes very little, can we make $\\pi_{new}=\\pi_{old}$ ? \n\n4. The experiments are slightly inadequate, the effects of tunable hyperparameter k should be further analyzed; In unseen environment, the MRPO algorithm is only tested on one environment.\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "RLBJHEg3WlB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Motivated by the domain transfer problem in RL where policies are trained on simulators that may not reflect perfectly the reality, this paper propose a new policy optimization algorithm named MRPO that is expected to be robust to changes in the environment's dynamic.\nThe formal setting and the notations are the same as in EPOpt (Rajewara 2017): each variant of the environment is an MDP parametrized by a parameter p and the trained policy is expected to be robust to (adversarial) changes on p.\nInstead of focusing on worst cases with a CEM-like procedure on p distribution like in EPOpt, the authors propose to divert the TRPO approximation bound into a safety bound.\nTheorem 1 gives a TRPLO-like lower bound, Theorem 2 show that optimizing for the LHS of Theorem 1 inequality may not degrade the wort-case reward.\nThe experiments study both the 10% word-case returns and the avarge returns. They show that MRPO improves clearly from EPOpt (renamed PW-DR for the occasion), the improvement against simple uniform domain randomization is less significant.\n\nProbably because this paper relies on notions gathered from both (Rajewara 2017) and (Schulman et al. 2017), I found the 8 pages of the main paper quite dense and hard to follow. The proofs in the appendix are however clearly detailed and easy to read. I checked integrally the proof of Theorem 1/3 without any difficulty.\n\nThis domain randomization model formally equivalent to a single (continuous) MDP where the the environment's dynamic is parametrized by the initial state distribution (for instance by enriching the MDP states by the p parameter).\nIt is therefore unclear to me that a specific algorithm is required for the specific case of parametrized MDPs.\nWhat would be the performance of a generic CVaR algorithm like \"Risk-constrained reinforcement learning with percentile risk criteria\" (Chow et al. 2017) on this setting ?\nI found the idea of diverting the TRPO approximation bound into a safety bound appealing. Applied to a single MDP it could lead to a CVaR variant of TRPO.\n\nMinor remarks:\np3 detalis -< details\nI found the \\rho notation for cumulative reward a bit confusing especially when p is involved in the equations, maybe a \\nu instead would improve readability ?\nExperiments on non-free systems like Mujoco are not easily reproducible. A few experiments on free-to-use environments would improve the reproducibility of the paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "BnlbUBTtMPs", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors proposed a more robust policy optimization method for domain randomization, by constraining the gap between the average performance of the whole range of environments and the performance of the worst-case environments. To achieve this, the author provide a lower bound for the worst-case performance, though the lower bound does not take the uncertainty of the finite samples into account. \n\nIn addition, the algorithm 1 proposed by authors requires to calculate a model discrepancy between $p_{w}$ and other environments $p_{i} \\sim P$, which is impractical to estimate by samples if the discrepancy is total variation distance. To achieve this, the authors assumes that the transition is lipschitz, with the requirement of tunning lipschitz constant. For empirical evaluation, the author compare with PPO with DR and PW-DR on three continuous benchmark mujoco task, which demonstrate that MRPO has some advantage over the other two algorithms.\n\nThe followings are my detailed comments and questions:\n- I feel that selecting the worst-case environment is one of the key challenging of the proposed algorithm. I did not find the description how to choose the $p_{w}$ given a set of environments $\\{ p_{i} \\}_{i=0}^{M-1}$. If the authors means that the expected return of the a single trajectory can be used to select the worst-case environment, then how do your algorithm can guarantee the expected return of the sampled trajectories is the exact performance of the environment? The author did not give finite sample high confidence upper bound for empirical mc estimation, and the selection of the worst case environment would be hard to implement in practical settings?\n\n- How do you choose or estimate the lipschitz constant? If the lipschitz constant is not right, then the bound will not given any practical guidence here.\n\n- It would be great if the authors can explain the gap between algorithm 2 and your practical implementation of using the 10% worst-case environments. If so, then the algorithm the authors use in the experiments can be viewed as directly select top performance trajectories to perform policy optimization, which I think the final algorithm is not consistent with your algorithm presented in the methodology part (please correct me if I am wrong about the final algorithm). \n\n- The experiments do not give strong empirical support for the new algorithm. The authors only evaluate on three environments, which I think is not enough, can the authors add more mujoco benchmarks? Also from the current results, I can not conclude that MRPO is better than PPO-DR since the evaluated domain is only three. Further, can the authors run more iterations to make sure the algorithms converge? The curves now presented in the paper did not converge. \n\n\nOverall I think there is a gap between the methodology presented in the paper and the final practical algorithm, and the  lower bound presented in the paper does not take the uncertainty caused by the finite samples into account, which will not give guidance to design empirical algorithms since the variance of the mc return of the policy is large.  Finally the evaluation of the algorithms have not been conducted thoroughly.\n\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper tackles the problem of mitigating the effect of model discrepancies between the learning and deployment environments. In particular, the author focus on the worst-case possible performance. The paper has both an empirical and theoretical flavor. The algorithm they derived is backed by theoretical guarantees. There exists a gap between the theory presented and the final practical algorithm, which generated some elements of concern from the reviewers. Some of these issues (choice and sensitivity of the Lipschitz constant, in what cases can we make that assumption, choice of p_w, discrepancy between the theoretical proposal and the practical algorithm) are well addressed in the rebuttal. However, after careful examination of the reviews, the meta-reviewer is still not convinced that the paper meets the minimum requirements for acceptance, as many of the reviewers' initial concerns still remain.
Decision: Reject