Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: 
This work aims to produce reinforcement learning methods that are ‘distributionally robust’. They approach this by assuming the transition function may vary between elements of the domain distribution and extend some recent results (Blanchet & Murthy, 2019) to give some theoretical results (contraction and optimal deterministic policy) and an actor-critic algorithm. This is an important research area and the work takes a very nice approach. However, the clarity of the work and the empirical results could both use some work.

There are a great many grammatical errors throughout the paper. The nature of the errors suggests a non-native speaker, which I really do not want to discourage, but this needs a proofread and edit. One exception, there is an actual typo just after assumption 1 “dfnied”, but actually better grammatically to just remove the word entirely.

In the intro, the citation of Mannor et al. 2004; 2007 for what is essentially a description of the gap between simulation to real transfer is a very strange choice. Is it possible this was a mistake?

The description leading up to the main result could be made much clearer. I went through this section, the main results, and the proofs in the appendix. Things look good, although I do feel like (as just said) it could be made much better/clearer. One snag I hit was in the proof of Lemma 1, everything is good except I don’t know why “u_1 \le u_2 + \gamma \| u_1 - u_2 \|_\infty” must hold, but I might have simply missed something obvious so please help me out.

The experimental results are very minimal and not very convincing. It is not immediately clear to me that Figures 1&2 actually show that the robustac algorithm is more robust than ac. The performance is worse than ac, which is completely understandable, but for most of the environments ac is still significantly better.

Lastly, a small point to notice that this paper bleeds into the 9th page, and considering the amount of contributions (largely down to the two theoretical results and a very small experiment) I do not think the extra space is warranted. This could be compressed and actually come out stronger as a result of being forced to be clearer and more concise.

Update:

Thank you for your response, especially the fix to the proof. I will keep my score, but do believe that further experiments and small adjustments to the writing will see a future version of this accepted.

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: In this work the authors studied the robust reinforcement learning problem in which the constraint on model uncertainty is captured by the Wasserstein distance. Inspired by the analysis in the distribution-ally robust setting, they derived several sensitivity conditions to study how the radius of the wasserstein ball and the order of the wasserstein distance affect the conservativeness in model uncertainty.  Similar to the robust MDP work, they also show that this robust RL problem has a robust optimal Bellman operator, and the optimal value function can be computed using  robust policy iteration, which can be extended to model-free actor critic algorithm when state/action spaces are large/continuous.

This work extends the wasserstein uncertainty modeling  to the robust MDP setting. However, I found the contribution rather limited/unclear. First, the proposed robust Bellman optimality result is standard and can be found in many robust MDP work when the uncertainty set is state-action rectangular. The major novel part here is therefore the sensitivity analysis that is specifically tied to wasserstein distance, which has limited novelty. Second, the robust actor critic algorithm is based on a multi-time scale minimax gradient approach, which is also quite standard in this field, and besides asymptotic convergence it is unclear how efficient this algorithm is. Third, the experiment in this paper is very limited (only based on cartpole, and is only compared with the non-robust AC algorithm) and its illustration on the effectiveness of this algorithm is rather limited. More comparisons with other robust MDP methods would be useful to understand how the value of the proposed wasserstein robust RL formulation.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: Summary:
In this paper, the authors study the problem of robust MDP (RMDP), where the feasibility set is defined as a Wasserstein ball around the reference transition probability. After formulating the problem, they first prove the contraction of the resulting operator and the existence of a deterministic stationary Markov optimal policy (Sec. 2.3). Then, they provide a sensitivity analysis of the optimal value function w.r.t. the radius and order of the Wasserstein ball (Sec. 2.4). Finally, they propose an actor-critic algorithm to solve the Wasserstein RMDP problem (Sec. 3) and evaluate its performance using simple experiments (Sec. 4). 

Comments:
- Section 2.3, which the authors call it Main Result, is not that novel. The Wasserstein RMDP is just a state-action-rectangular RMDP with convex ambiguity set, and Lemma 1 and Theorem 1 are known to be true for such RMDPs. 
- The sensitivity analysis section (Sec. 2.4) is interesting and useful, although it could have been written much better. 
- The algorithm is new, although its section (Sec. 3) is not well-written. There is no analysis for the convergence of the algorithm either. It is a multi-time-scale stochastic approximation algorithm. Similar policy gradient and actor-critic algorithms have been derived in risk-sensitive MDPs for mean-variance, mean-VaR, and mean-CVaR optimization (see the references below), but such algorithms for RMDPs is relatively new. 
- The experimental results are very simple and not very convincing. 
- I would suggest that the authors put their emphasis on the algorithm and try to explain it much better. And support it better with more comprehensive and convincing experimental results. This would definitely improve the quality of the paper.


References on risk-sensitive MDPs: 
1) A. Tamar, D. Di Castro, and S. Mannor. “Policy Gradients with Variance Related Risk Criteria”. ICML-2012.
2) Prashanth L.A. and M. Ghavamzadeh. “Actor-Critic Algorithms for Risk-Sensitive MDPs”. NIPS-2013.
3) Y. Chow and M. Ghavamzadeh. “Algorithms for CVaR Optimization in MDPs”. NIPS-2014.
4) A. Tamar, Y. Glassner, and S. Mannor. “Optimizing the CVaR via Sampling”. AAAI-2015.
5) A. Tamar, Y. Chow, M. Ghavamzadeh, and S. Mannor. “Policy Gradient for Coherent Risk Measures”. NIPS-2015.

review_title: Official Blind Review #2

Meta-review: This paper studies the robust reinforcement learning problem in which the constraint on model uncertainty is captured by the Wasserstein distance. The reviewers expressed concerns regarding novelty with respect to prior work, the presentation or the results, and unconvincing experiments. In its current form the paper is not ready for acceptance to ICLR-2020.
Decision: Reject