Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: In this paper, the authors proposed a more robust policy optimization method for domain randomization, by constraining the gap between the average performance of the whole range of environments and the performance of the worst-case environments. To achieve this, the author provide a lower bound for the worst-case performance, though the lower bound does not take the uncertainty of the finite samples into account. 

In addition, the algorithm 1 proposed by authors requires to calculate a model discrepancy between $p_{w}$ and other environments $p_{i} \sim P$, which is impractical to estimate by samples if the discrepancy is total variation distance. To achieve this, the authors assumes that the transition is lipschitz, with the requirement of tunning lipschitz constant. For empirical evaluation, the author compare with PPO with DR and PW-DR on three continuous benchmark mujoco task, which demonstrate that MRPO has some advantage over the other two algorithms.

The followings are my detailed comments and questions:
- I feel that selecting the worst-case environment is one of the key challenging of the proposed algorithm. I did not find the description how to choose the $p_{w}$ given a set of environments $\{ p_{i} \}_{i=0}^{M-1}$. If the authors means that the expected return of the a single trajectory can be used to select the worst-case environment, then how do your algorithm can guarantee the expected return of the sampled trajectories is the exact performance of the environment? The author did not give finite sample high confidence upper bound for empirical mc estimation, and the selection of the worst case environment would be hard to implement in practical settings?

- How do you choose or estimate the lipschitz constant? If the lipschitz constant is not right, then the bound will not given any practical guidence here.

- It would be great if the authors can explain the gap between algorithm 2 and your practical implementation of using the 10% worst-case environments. If so, then the algorithm the authors use in the experiments can be viewed as directly select top performance trajectories to perform policy optimization, which I think the final algorithm is not consistent with your algorithm presented in the methodology part (please correct me if I am wrong about the final algorithm). 

- The experiments do not give strong empirical support for the new algorithm. The authors only evaluate on three environments, which I think is not enough, can the authors add more mujoco benchmarks? Also from the current results, I can not conclude that MRPO is better than PPO-DR since the evaluated domain is only three. Further, can the authors run more iterations to make sure the algorithms converge? The curves now presented in the paper did not converge. 


Overall I think there is a gap between the methodology presented in the paper and the final practical algorithm, and the  lower bound presented in the paper does not take the uncertainty caused by the finite samples into account, which will not give guidance to design empirical algorithms since the variance of the mc return of the policy is large.  Finally the evaluation of the algorithms have not been conducted thoroughly.




review_title: Review for Monotonic Robust Policy Optimization

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary:
This paper introduces Monotonic Robust Policy Optimization (MRPO), an RL algorithm that aims to jointly optimize policy and domain sampling distribution, with the goal of improving policy performance for both average and worst-case scenarios and addressing the model discrepancy between the training and target environments. They derive a lower bound for the worst-case performance, which comprises the average performance, policy change, and the statistical distance between the worst and average case environments. A TRPO-like monotonic performance improvement guarantee is provided for the worst-case expected return. Finally, a practical approximation to MRPO is proposed, which imposes the assumption on Lipschitz continuity with respect to the environment parameters and circumvents the estimation of total variation distance between the worst-case environment and the sampled environment. Experiments are conducted on three control tasks with diverse transition dynamics parameters, where MRPO could improve both average and worst-case performance in the training environments, and it shows better generalization to the unseen test environments than baseline algorithms.


pros:
- The theoretical analysis is provided, which shows the relationship between the worst-case and average performance for the first time.

- The algorithm is backed by the theoretical guarantee of monotonic worst-case performance improvement.


cons:
- The assumption that the transition dynamics model is L-Lipschitz with respect to the environment parameter seems to be strong.

- Some of the experimental results are not convincing. For example, in Figure 1f, MRPO underperforms DR, even if DR does not consider the worst-case performance during optimization at all.


comments and questions:
- How natural is the model's Lipschitz assumption? Are many real-world problems satisfying this assumption?

- In Figure 1, what does the shaded-area stand for? standard deviation? standard error? Also, it is not clear that MRPO outperforms other baselines statistically significantly.

- It seems that two dense layers are used to construct the policy and value networks in the experiments. Why was the recurrent (e.g. LSTM) policy not used? Since the recurrent policy can implicitly embed system identification, I think the performance of the DR baseline could have been improved with the use of the recurrent policy. It would be great to see the performance comparison when the recurrent policy is used for MRPO and baselines.

- For the experiments on generalization to unseen environments, only the results for Hopper is provided, which may not be sufficient to demonstrate the behavior of each algorithm. It would be great to provide the heatmap results for other domains, i.e. Walker and HalfCheetah.

- In Theorem 1, is $p_w$ is the worst-case parameter for $\pi$? or for $\tilde \pi$? It would be good if notation presents the dependence on the policy of $p_w$, e.g. $p_w^\pi$.

- In Algorithm 2, line 6: how can $p_w^k$ be found? (even before completing sampling the trajectories for each environment)

review_title: interesting algorithm with theoretical support

final_score: rating: 6: Marginally above acceptance threshold
review_content: Motivated by the domain transfer problem in RL where policies are trained on simulators that may not reflect perfectly the reality, this paper propose a new policy optimization algorithm named MRPO that is expected to be robust to changes in the environment's dynamic.
The formal setting and the notations are the same as in EPOpt (Rajewara 2017): each variant of the environment is an MDP parametrized by a parameter p and the trained policy is expected to be robust to (adversarial) changes on p.
Instead of focusing on worst cases with a CEM-like procedure on p distribution like in EPOpt, the authors propose to divert the TRPO approximation bound into a safety bound.
Theorem 1 gives a TRPLO-like lower bound, Theorem 2 show that optimizing for the LHS of Theorem 1 inequality may not degrade the wort-case reward.
The experiments study both the 10% word-case returns and the avarge returns. They show that MRPO improves clearly from EPOpt (renamed PW-DR for the occasion), the improvement against simple uniform domain randomization is less significant.

Probably because this paper relies on notions gathered from both (Rajewara 2017) and (Schulman et al. 2017), I found the 8 pages of the main paper quite dense and hard to follow. The proofs in the appendix are however clearly detailed and easy to read. I checked integrally the proof of Theorem 1/3 without any difficulty.

This domain randomization model formally equivalent to a single (continuous) MDP where the the environment's dynamic is parametrized by the initial state distribution (for instance by enriching the MDP states by the p parameter).
It is therefore unclear to me that a specific algorithm is required for the specific case of parametrized MDPs.
What would be the performance of a generic CVaR algorithm like "Risk-constrained reinforcement learning with percentile risk criteria" (Chow et al. 2017) on this setting ?
I found the idea of diverting the TRPO approximation bound into a safety bound appealing. Applied to a single MDP it could lead to a CVaR variant of TRPO.

Minor remarks:
p3 detalis -< details
I found the \rho notation for cumulative reward a bit confusing especially when p is involved in the equations, maybe a \nu instead would improve readability ?
Experiments on non-free systems like Mujoco are not easily reproducible. A few experiments on free-to-use environments would improve the reproducibility of the paper.
review_title: An improvement of EPOpt based on a TRPO-like lower bound on worst-case cumulative policy reward

final_score: rating: 7: Good paper, accept
review_content: This paper focuses on the generalization issue in reinforcemetn leanring, specifically aims to address the problems of domain randomization(DR) technique. Different from standard DR which treats all the sample environment as equal, this paper proposed to improve the performance over all possible environments and the worst-case environment concurrently. This paper theoretically derives a lower bound for the worst-case performance of a given policy over all environment, and in practical, the proposed method, monotonic robust policy optimization(MRPO) carries out a two-step optimization to imporve the lower bound such as to maximize the averaged and worst-case policy perfomance. 


This paper is well written and the key concept is clearly introduced. The Theorem.1 makes the connections between the averaged and the worst-case performance, such that maximizing the worst-case performance can be solved by maximizing the averaged performance problem with some trajectories from environments with both poor and good-enough performance. The emprical results also support the theorical analysis.

1. For Lemma 1: The conclusion is based on the assumption that the the worst case $\rho(\pi|p_w) - \max_p \rho(\pi|\rho)$ is bounded (Proof A.1). However, such equation does not strictly holds without bounded reward function. The author should stated the condition.

2. About the monotonic worst-case performance improvement theorem, the proof says "... the approximation is made under the assumption that the worst-case environment between two iterations are similar, which stems from the trust region constraint we impose on the update step between current and new policies...", however, the trust region constraint can only limit the difference between policy updates, the similarity between worst-case environments can not be promised.

3. In theorem 2, the fomula (50) and (51) in the proof, is this approximation reasonable? Since the policy is updated, the worst-case environment may have changed a lot. Similarly, if the updated policy changes very little, can we make $\pi_{new}=\pi_{old}$ ? 

4. The experiments are slightly inadequate, the effects of tunable hyperparameter k should be further analyzed; In unseen environment, the MRPO algorithm is only tested on one environment.




review_title: Model discrepancy between enviornments plays a role in generalization 

Meta-review: The paper tackles the problem of mitigating the effect of model discrepancies between the learning and deployment environments. In particular, the author focus on the worst-case possible performance. The paper has both an empirical and theoretical flavor. The algorithm they derived is backed by theoretical guarantees. There exists a gap between the theory presented and the final practical algorithm, which generated some elements of concern from the reviewers. Some of these issues (choice and sensitivity of the Lipschitz constant, in what cases can we make that assumption, choice of p_w, discrepancy between the theoretical proposal and the practical algorithm) are well addressed in the rebuttal. However, after careful examination of the reviews, the meta-reviewer is still not convinced that the paper meets the minimum requirements for acceptance, as many of the reviewers' initial concerns still remain.
Decision: Reject