Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper introduces methods for reinforcement learning based on a Wasserstein Natural Gradients (WNG), an approach to measuring similarity between policies. The methods are based on Policy Gradients and Evolutionary Strategies and add policy similarity term based on WNG instead of KL constraint as in TRPO.

The paper is well written and easy to follow (the conclusion section is missing though). Although the method is interesting, I think that the current experimental evaluation has significant flaws: the method does not demonstrate the state-of-the-art performance and significantly improves over the baselines on a small number of tasks. I believe that the paper will significantly benefit from comparisons  with stronger baselines such as PPO. Moreover, the experiments performed in the paper (figure 2) demonstrate that the approach only marginally outperforms the baselines on the harder HalfCheetah and Hopper tasks that raises concerns regarding the generality of the approach.

The paper proposed an interesting approach to policy constraints in RL but the experimental evaluation is not sufficient. 
review_title: An interesting approach to measuring policy similarity in reinforcement learning

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: 
Amari's Natural Gradient has been very successfully applied for policy optimization, e.g. in a recent line of work by Schulman et al. These benefit of using these natural gradients that restrict the change in KL divergence between successive policies was more stable and faster convergence. However two distributions may differ a lot in terms of KL divergence but because of the dynamics of the MDP they may still have almost the same behavior, therefore recent work has focused on using the Wasserstein distance to measure the divergence between successive policies which naturally leads to "Wasserstein Natural Descent". In this paper the authors build upon recent work on Kernelized Wasserstein NGD by making it more widely applicable and scalable. Moreover they present a good empirical comparison between KL-NGD and Wasserstein-NGD on a combination of pedagogical toy problems and some standard RL benchmarks from OpenAI gym. Overall this paper will be a good contribution to the conference and I recommend acceptance.


**Corrections and suggestion for improving presentation**

1. [Citations] Page 2, third paragraph from bottom cites Schulman 2015 instead of Schulman 2017 for PPO. Page 4 last paragraph has a citation missing and Li and Zhao "Wasserstein Information Matrix" paper is cited twice. 

2. I think it will be better to write down that equation (5) defines $f_u$. Also shouldn't the $1/2$ factor in equation (5) be actually $\beta / 2$ ? 

3. On page 3 it is said that "the penalty only accounts for global proximity in behavior .... " in reference to equation (4), but PPO is not implemented with a single value of $\beta$, i.e. the strength of the penalty typically varies as optimization goes on. Why can't the same be done for the weight of the $W_2$ penalty ?

review_title: Well reasoned extension of recent work to make Wasserstein NGD more scalable. Accept

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary
This paper proposes to use natural gradient instead of standard gradient to optimize a regularized objective with the regularization being the Wasserstein distance between the so-called behaviour distributions for the previous policy and new policy. It then combines this Wasserstein gradient descent with Policy Gradient and Evolutionary Strategies.  Experiments conducted in OpenAI and Roboschool show some promising results for this combination.

### Strong points: 
-	Clarity: The paper is well structured
-	Empirical significance: The empirical results seem promising where it shows that Wasserstein gradient descent could be a good choice to constraint the behaviour changes efficiently without sacrificing too much computational cost. I personally like that the authors used Roboschool (RS) as a benchmark for continuous control tasks as it is, as opposed to Mujuco, a free simulator software (thus it is more accessible and reproducible for everyone).   

### Weak points: 
-	Novelty: the work however has low technical novelty where it combines several known results into a new framework. In particular, the idea of constraining behaviour policy changes and the efficient way to estimate Wasserstein gradient descent are all known and off-the-shelf. Adopting Wasserstein gradient descent to RL constraint update seems straightforward that does not require any significant technical challenge. The interpretation of the framework also seems straightforward, e.g., it is of course that updating along the Wasserstein natural gradient would incorporate the local geometry of parameterization and help overcome some ill-conditioning issues where KL has. 
-	Empirical significance: The empirical results though promising are not strong given that this is mostly an empirical work. In particular, the present work presents the experiments for PG case in only 4 environments which I think insufficient to make a reliable conclusion about its empirical significance. 

###  Questions for the authors 
-	In Section 2: “Reusing trajectories can reduce the computational cost but drastically increases the variance of the gradient estimator”. Could the authors elaborate on why reusing trajectories drastically increases the variance of the gradient estimator?
-	Fig. 1 (c): Have all algorithms been initialized at the same initial point? Also, according to Fig.1 (c) that it seems that FNGD has a ‘right’ convergence when it converges to the point where \sigma=0 and \mu = midpoint, why in the last paragraph of page 4, the authors conclude that “FNG remains far away from optimum”? What do I miss here? 
-	What is the difference between W2-penality and WNGD in Figure 1? 
-	On page 5, “The Wasserstein penalty Equation (4) encourages global proximity between updates qθk”. What does globality here refer to while Eq (6) holds only locally?

### Minor comments 
-	The second term of Eq. (5): Shouldn’t it be f_u instead of f there?
-	Eq. (8): \argmax_{u}
-	The second last sentence at the end of page 4: cite -> \cite{sth}
-	It seems that \mu and \sigma in Fig. 1 (c) have not defined explicitly anywhere. In the capture of Fig.1., it writes \theta = (mu,v), so there is a chance of inconsistent notations here? 
-	On page 5, “To avoid slowing-down, there is an intricate balance between the step-size and penalty β that needs to be maintained Schulman et al.
(2017)”: \cite -> \citep  

###  My initial recommendation

Given the weak and strong points above, I vote for rejecting for this current form.

### My finial recommendation 

After the discussion and revision, the authors have presented more convincingly and more clearly the empirical significance and applicability of their method. I highly recommend the authors to highlight the lastest discussion in the final paper, especially the ill-conditioned argument, as it is highly relevant to the practitioners. I think this paper can be interesting for a moderate number of readers, especially the use of the open-sourced Roboschool could also increase its reproduciability. I agree to increas my score to 6. 
review_title: An encouraging empirical result but low technical novelty and seemly insufficient experiments to make a reliable conclusion given this work is mostly empirical

Meta-review: This paper explores the Wasserstein natural gradient in the context of reinforcement learning. R5 rated the paper marginally below the acceptance threshold, but is not very confident about the correctness of his/her assessment. His/her main criticism was the experimental evaluation. This concern was shared by a confident R1. R1 found the paper well structured and that it contains encouraging empirical results, but low technical novelty and (initially) insufficient experiments. His/her initial recommendation was reject, but following an extensive discussion and improvements of the manuscript by the authors, he/she was more convinced about the empirical significance and applicability of the method, and raised his/her score to 6, indicating that the interpretation and presentation improved but that the paper might be interesting only to a moderate number of readers. A confident R2 found this paper very good, although only providing a short review. Two other unfinished or not sufficiently confident reports were not taken into account. Weighing the reports by contents, confidence, and participation in the discussion, the paper scores marginally above the acceptance threshold. In view of the authors' responses, I am discounting R5's criticism about lack of comparison with the PPO baseline. I personally consider the paper very well written, that it presents a natural and potentially useful application of the Wasserstein natural gradient to the context of reinforcement learning, and enjoyed the discussion of behavioral geometry. I am recommending a borderline accept. However, I also appreciate the concern of the referees about the limited technical innovation and how some of the strengths of the method could be presented more convincingly. Please take these comments carefully into consideration when preparing the final version of the paper. 
Decision: Accept (Poster)