Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper focuses on the problem of robust reinforcement learning and proposed a generic framework that is able to deal with both discrete and continuous state and action spaces. They formalized the robust reinforcement learning problem as a min-max game with Wasserstein constraints and put forward an efficient and scalable solver that is helpful to numerical optimization. Overall, this paper is of well-written with a clear illustration of the methodology and comprehensive experimental results. I still have a few concerns below, that prevent me from giving a direct acceptance: 
1.	However, the proposed method seems to be somewhat incremental: In Yang (2017), it is clearly stated that it is viable to treat the robust reinforcement learning objective as a min-max game (in eq.(1)) with support belonging to a Wasserstein ball (in eq.(2)). The statement in the submission “… a novel min-max game with a Wasserstein constraint …” seems overclaimed, namely, eq.(8) in the submission, is a combination of eq.(1) and (2) in Yang (2017), with the constraint in eq.(8) not exactly equal but just a relaxation of eq.(2) in Yang (2017).
2.	Furthermore, since a few papers have proposed methods to deal with both transition and reward dynamics, it would be nice and complete to also address (or hint on) the reward ambiguity problem.
3.	Does the relaxation of the Wasserstein constraint in eq.(7) make the learned policy conservative? Some illustrations of this effect of \epsilon would be interesting and complete.
4.	For the experiments in section 5.1, it would be better to give an illustration of what the color means in Figure 1. 
Insoon Yang. A convex optimization approach to distributionally robust markov decision processes with Wasserstein distances. IEEE control systems letters, 1(1):164-169, 2017.

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper proposes a novel solution to a *restricted* reinforcement learning problem. I am not quite sure if I understand the restriction from the text or thus what relevance this has in applications. The method factorizes the transition dynamics (where they rely on a parametrized solver) and a policy modeled by a NN and trained with policy gradient with monte-carlo returns. The ultimate goal is robust control so they care about the maximin objective of highest return along the worst trajectory. The robustness is implemented as a constraint in the dynamics optimization: a wasserstein distance between the distribution of samples obtained applying the learnt policy and learnt dynamics parameters and those obtained by random policy and some reference dynamics model. They solve the maximin by alternating minimization on the dynamics and policy. The contribution is in the intricacies of computing the wasserstein efficiently in particular they obtain a perturbation based estimate of the gradient of the dynamics parameters. The experiments seem sensible but I  have some suggestions for improvement.

I have published in RL but my familiarity with Wasserstein and robustness is limited. I have not checked the derivations in the appendix in detail. If there is a high score I'll have a closer look. 

Major points:
1) The restriction imposed is that only part of the transition model is really learnt. So it is more like a system identification setup ?! Why is that necessary ? 
2) The explanation in on page 4 about how the wasserstein distance is computed is a bit unclear. Is it sampling data from the P_\phi and \pi on one hand and P_0 and \pi_{Uniform} ? Then there is a bucketing step (what does it mean ? how is it computed ?). Is it then a discrete wasserstein between the two empirical distributions ?! What is Monte-Carlo in this case ?
3) All the approximations done in the method should be more clearly explained because they are relevant both to people who might apply the method in practice as well as other researchers that might want to build on/ improve on it.
4) The experiments seem interesting but I am not sure how to interpret the results. If a method has access to a simulator and a representation of the relevant dynamics parameter space it seems clear that it will be more robust. This is impressive if one has a simulator and can infer some state components e.g. in pendulum starting with an empty simulator and adding a pendulum. But that seems as hard or maybe even harder than trying to learn an implicit transition model just from observations. 
5) Imagine the method only gets observations and an imperfect simulator, which will be the case in an application, what are the caveats we should have in mind ?
6) Can you have a learnt model baseline and compare your learnt and simulation ? Maybe it would have to be a latent variable model so you can do the same inference for internal parameters and still do an adaptation in the same way as now. This seems easy since it is a supervised problem.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The paper studies the problem of training policies which maximize returns under worst case environmental perturbations. They propose to solve this max min problem using iterative optimization with the max step performed using standard RL. The min step relies on sample-based estimates of gradients and hessians with respect to environment parameters. The problem of robust control is an important one, and the paper presents interesting results, however it lacks a proper comparison to baselines.

1. I think figure 1 should include two more columns. The first showing the performance of a policy trained to maximize returns averaged over the training environmental perturbations. This doesn't require the min step of the algorithm and is a natural baseline for the setup you consider. The second showing the same but trained with an LSTM policy. This way the policy can learn to do system identification. It might not be robust (i.e. not generalize to unseen perturbations) but this should be verified. If it does generalize, then there's no reason to complicate things with the min step.
2. Maybe I'm missing something but neither RARL nor PR-MDP baselines are shown in Fig. 1 and 2. despite you claiming in intro to section 5 that you test against those. These baselines are included in Figure 3 in the Appendix but I find those numbers odd. E.g. RARL doesn't seem to match PPO even on the reference environment in Fig 3c. This suggest that it relies on a different algorithm and it doesn't make sense to compare it to WR2L which relies on PPO.
3. In Related work you says "we were unable to find the code for this paper, and did not attempt to implement it ourselves". Even if this code was available, comparing to a different codebase is not a meaningful comparison (see e.g. [1]). There is no replacement for implementing relevant baselines.
4. You estimate the Hessian using data from a random policy. I find this odd because random policy never visits the interesting states. Have you tried recalculating the Hessian in every min step?
5. Could you clarify the use of Wasserstein distance? If I only use samples from the two distributions to estimate it then this is likely limited to low-dimensional settings, right?

[1] Henderson et al., (2017). Deep Reinforcement Learning that Matters.

------------------------------------------------------------------------------------------------------------------------------------------------------
Thanks for the clarifications and for including the "average" policy baseline. I still think the paper could use more work and hence keeping my score as is. I'd try to make the LSTM sys-id baseline to work--those have been used with domain randomizations previously, as well as "meta-learners" in the meta RL literature.
review_title: Official Blind Review #2

Meta-review: While the reviewers agreed that the problem of learning robust policies is an important one, there were a number of major concerns raised about the paper, and as a result I would recommend that the paper not be accepted at this time. The important points are: (1) limited novelty in light of prior work in this area (see R2 and R3); (2) a number of missing comparisons (see R2). There is also a bit of confusion in the reviews, which I think stems from a somewhat unclear statement in the paper of the problem formulation. While there is nothing wrong with assuming access to a parameterized simulator and studying robustness under parametric variation, this is of course a much stronger assumption than some prior work on robust reinforcement learning. Clarity on this point is crucial, and there are a large number of prior methods that can likely do well in this setting (e.g., based on system ID, etc.).
Decision: Reject