Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Summary:
The paper considers the task of learning robust policies. Specifically, they focus on the noisy-robust (NR) variant of the action robust framework proposed in [1]. As noted in [1], as strong duality does not hold in the NR variant it can not be solved using deterministic policies (pure strategies).
The authors propose to combine SGLD with DDPG, to maintain a distribution over deterministic policies (essentially a mixed strategy) - hence overcoming these issues.
The results look promising, outperforming [1] in all tested domains.

Review:
Overall this seems like the first approach of using Bayesian learning in order to reach mixed Nash equilibrium in RL. This is super-important, since many problems can be formulated as zero-sum games for which the solution is not necessarily a pure strategy.
As [2] have already shown the ability of this concept to find mixed equilibria in GANs, the main novelty is the introduction to RL.

As this work does not introduce new theory or a dramatic new concept, I feel that the acceptance of this work lies mainly on the empirical side. In my opinion, the experiments need to be more convincing - for instance, include additional domains (such as the Inverted Pendulum which was a failure case in [1]) and additional forms of robustness (e.g., the probabilistic-robust variant from [1] which is based on deterministic strategies and was shown to work better, and RARL [3] which test robustness to external disturbances).
An addition option is to build a low-dimensional toy problem in which the exact solution is known. This will enable you to show that while the naive solution in [1] either does not converge or converges to sub-optimal solutions, the SGLD approach is capable of finding superior solutions (hopefully the global optimum).

The idea is in the right direction. However, in my opinion, it is not there yet and is thus not ready for ICLR.

--- Post Rebuttal ---

I stand by my original assessment. I feel that such work needs to be more convincing. I for one would feel more confident had the authors provided simpler experiments in which they show that their approach indeed converges to the mixed Nash equilibria while the NR-DDPG approach from [1] does not.
I did overall like this direction and I believe Robustness in RL is very important.


[1] Action Robust Reinforcement Learning and Applications in Continuous Control - Tessler et al. 2019
[2] Finding mixed nash equilibria of generative adversarial networks - Hsieh et al. 2019
[3] Robust Adversarial Reinforcement Learning - Pinto et al. 2017
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: The paper proposes a new algorithm for solving Noisy Robust MDPs (NR-MDPs) based on computing an approximate gradient using  stochastic gradient langevin dynamics (SGLD). THE NR-MDPs can be thought of as an agent learning in an adversarial environment.
The paper is well written and the references pointed to give sufficient background to understand the problem undertaken.
Most of the theory comes from the "Finding mixed nash equilibria of generative adversarial networks." (ICML, 2019), and the papers main contribution lies in applying the theory to reinforcement learning.
It is not altogether unexpected though that Langevin Dynamics give better results than the more standard gradient-based approach considered for saddle-point problems, that's what they are supposed to do. But, this seems to be the first time SGLD has been applied to such a problem.
The authors compare on the MuJoCo benchmark with common but not identical instances to "Action robust reinforcement learning and applications in continuous control", which also provides the baseline algorithm used for comparison. Mentioned in this paper is the difficulty of solving the "inverted pendulum" instance by the algorithm proposed therein. It is infact mentioned as a failure case. Maybe, the authors can show results on the same. 
Additionally, maybe it would make sense to have an ablation study of the hyper-parameters from Table 1. 

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper approaches two-player zero-sum Markov game by using mixed Nash equilibrium by sampling from randomized policies. The authors propose that the optimization is done using Stochastic Gradient Langevin Dynamics iterations. 
In my opinion, the use of SGLD to this problem is potentially useful - as partially proved by this work. However, this application is obvious and does not have enough novelty merits to be accepted to this ICLR. The experiment make senses but is not rigorous enough to assure the practitioners on the improvement over baseline. 
I recommend the authors to further develop this idea in both theoretical improvement and experiment settings to further explore the direction.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Update: I have read the author responses. As mentioned before I don't have the background to carefully assess the experiments and will have to rely on my fellow reviewers here. However I stand by my opinion that the experimental results would need to be very strong to warrant an acceptance, since the conceptual contribution is relatively limited.
I was also disappointed by the authors unwillingness to back up their claims of "theoretically convergent" with a proof, or at least a theorem. Therefore I still tend towards rejecting the paper.
-----------------------------------------------
Summary
The present work proposes to use the recently developed Stochastic Gradient Langevin Dynamics (SGLD) to compute mixed approximate equilibria in two-player reinforcement learning, following the methodology proposed by hsieh et al for generative adversarial networks. The authors report practical improvements compared to pure strategies computed as proposed by Tessler et al.

Decision
The idea of using randomized strategies for two-player reinforcement learning is interesting and natural. However, as the authors note, mixed strategies are classical in game theory. Furthermore, the adaption of the methodology of hsieh et al is straightforward, limiting the strength of the theoretical contribution. 
Unfortunately, the theoretical contribution is not stated consistently, since in the introduction, the paper states "Our paper precisely bridges this gap between theory and practice in previous works, by proposing the ﬁrst theoretically convergent algorithm for robust RL", but this claim is missing in the abstract or conclusion, and there is no theorem that justifies this rather strong claim. 
In my opinion, this paper should only be accepted if it provides very convincing numerical experiments, which I am not qualified to assess. I happy to increase my score if the experimental results are deemed strong by the reviewers with more expertise in practical reinforcement learning.

Questions to authors
You write "Our paper precisely bridges this gap between theory and practice in previous works, by proposing the ﬁrst theoretically convergent algorithm for robust RL". What is the exact mathematical statement here? Does this refer to the algorithm that is used in the numerical experiments?
review_title: Official Blind Review #1

Meta-review: The authors address the problem of robust reinforcement learning. They propose an adversarial perspective on robustness. Improving the robustness can now be seen as two agent playing a competitive game, which means that in many cases the first agent needs to play a mixed strategy. The authors propose an algorithm for optimizing such mixed strategies. 

Although the reviewers are convinced of the relevance of the work (as a first approach of Bayesian learning to reach mixed Nash equilibria, which is useful not only for robustness but for any problem that can be formulated as zero-sum game requiring a mixed strategy), they are not completely convinced by the work in current state. Three of the reviewers commented on the experiments not being rigorous and convincing enough in current form, and thus not (yet!) being able to recommend acceptance to ICLR. 
Decision: Reject