Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies policy optimization in reinforcement learning with Wasserstein and Sinkhorn trust regions. Compare to the standard TRPO which based on KL-divergence, the proposed WPO and SPO go beyond the parametric policy distribution class. The authors also derive closed form policy update, as well as theoretical performance guarantees for both problems.
main_review: The paper is overall well written, the results are well organized and reasonably clear to follow. In terms of novelty, I believe the closed form update parts (Theorem 1 & 3) are reasonably standard primal-dual arguments. However, Theorem 2 is an interesting and significant contribution since it provides theoretical guarantees even when the update $\beta_t$ is not the optimal solution in (6), which circumvent the difficulty in obtaining closed form solution of (6). Theorem 4 also seems important as it gives theoretical guarantee for us to approximate WPO by SPO with large $\lambda$. I would like to discuss a few more questions with the authors: 

The upper bound of $\beta^*_{\lambda}$ for SPO in Theorem 3 is of order $1/\delta$, which seems not very good because the bound would be trivial if there is no perturbation to the policy (i.e., $\delta = 0$). Also, the bound for $\beta^*$ for WPO in Theorem 1 seems to be independent of $\lambda$, I would like the authors to discuss some insights behind the different dependency on $\lambda$. Do we expect the bound for $\beta^*_{\lambda}$ is actually also independent of $\delta$. Moreover, it would be also interesting to understand the optimal dependency of $\beta^*_{\lambda}$ as a function of $\lambda$.
In Theorem 4, can we characterize the rate of convergence of $|F_{\lambda}(\beta) - F(\beta)|$? Seems like an upper bound could be $1/\lambda$, but is this correct (or optimal if it is correct)? Also, it would be good if we could understand the convergence of the optimizers in part 2 of Theorem 4.

Small typo in Abstract: ‘extensions of policy optimziation’ -> ‘extensions of policy optimization’.

summary_of_the_review: This paper presents two policy update frameworks (WPO and SPO) which relax the restriction to parametric policy distribution in the standard setting. Theoretical guarantees are provided. The numerical results also suggest that proposed policy optimization methods outperform the standard TRPO and PPO with better performance and faster convergence.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to use two extensions of the TRPO algorithm relying on the Wasserstein distance and the Sinkhorn divergence which dont require to explicitly specify a distribution for the policy. The authors provide a theoretical analysis giving a closed form policy update for their two methods and a performance improvement bound in the case of Wasserstein policy optimisation. They evaluate their methods empirically on tabular domains (Taxi, Chain and Cliff Walking) and on some discrete locomotion tasks (Cartpole, Acrobot). They find that their method outperform TRPO and PPO while being more sample-efficient, and converging faster.
main_review: The paper is well-organised and present a new way of incorporating the Wasserstein distance within policy optimisation algorithms. 
The theoretical results look correct at first glance although I admit didn't check them carefully in the Appendix.

It would be nice for theorem 1 to be self-contained: some of the variables are defined earlier in the text like M or beta but it would ease the reading to define them in the theorem. The presentation of the theoretical results are a bit hard to follow so adding a few explanatory sentences about the importance of each terms would also be helpful.

Could the authors also comment on the computational complexity of the method?

In terms of experiments, the domains considered seem to show the benefits of the method. I believe stronger tasks would make the paper stronger, in particular continuous control ones or at least showing results for a few more domains.

Related Work: "Wasserstein-like metrics have only been recently studied in the context of reinforcement learning." Please note that the Wasserstein metric has been used in RL since at least 2012 with the introduction of bisimulation metrics https://arxiv.org/pdf/1207.4114.pdf, and recent work havent only used it for imitation learning but also for generalization https://arxiv.org/pdf/2101.05265.pdf https://arxiv.org/abs/2102.01514 https://arxiv.org/abs/2006.10742.
I believe the Wasserstein and Sinkhorn metrics have also been used in the distributional RL literature so it might be nice to discuss the similarity of ideas in both areas.

Typos:
Wassersteim 
*the* Wasserstein metric, *etc.* : please dont use "etc" and add all necessary details.
summary_of_the_review: This paper provides a new way of optimising the policy distribution relying on the Wasserstein and Sinkhorn distances. The methods is theoretically grounded so I would recommend an accept but I believe the authors would need to evaluate their method on more domains to make the paper stronger.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper considers Wasserstein and Sinkhorn trust region policy optimization (WPO and SPO) for reinforcement learning. Unlike existing works on WPO, it does not assume a parametric policy class. Theoretically, it shows the performance improvement of WPO at every iteration, and that SPO converges to WPO. It also conducts some experiements to illustrate the advantages of the proposed methods.
main_review: Strength:

1. The paper provides a relatively complete analysis of the Wasserstein and Sinkhorn policy optimization, including closed-form policy updates, performance improvement bound, actor-critic algorithm, and experiments on popular instances.

Weakness:

1. The theoretical advances seem quite limited. The closed-form policy update in Theorem 1 and Theorem 3 should follow from existing duality results on Wasserstein and Sinkhorn optimization (for example, [1] and [2]). It is unfortunate that such connections are not recognized in the paper. Theorem 2 is a simple consequence of the well-known performance difference lemma. Theorem 4 is sort of expected, as many existing results on the relationship between Wasserstein distance and Sinkhorn distance. In addition, it is unclear whether SPO has a performance improvement at each iteration for any fixed entropic regularization parameter.

2. Computational-wise, I am wondering how the policy update in (WPO) and (SPO) can be implemented when the state/action space is large, and the computation of the parameters $\beta_t$ seems also time-consuming whereas I did not find a technically sound argument for the heuristic choices of $\beta$. 

3. The experiment results are not convincing enough to demonstrate clear advantages of WPO or SPO. 

(1) There is no comparison with existing Wasserstein policy optimization (with parametric classes), such as algorithms in Moskovitz et al. (2020) and Pacchinao et al. (2020). Therefore, it is unclear to me whether it is worth considering a non-parametric policy update, given that is the high computational cost for large spaces.

(2) In most results (for example, Figures 2,3,5), SPO is outperformed by WPO in terms of the reward at the end of the timesteps, although it converges to a suboptimal value faster. It might be helpful to consider a varying entropic regularization parameter.

References.
[1] Blanchet, Jose, and Karthyek Murthy. "Quantifying distributional model risk via optimal transport." Mathematics of Operations Research 44.2 (2019): 565-600.
[2] Wang, Jie, Rui Gao, and Yao Xie. "Sinkhorn Distributionally Robust Optimization." arXiv preprint arXiv:2109.11926 (2021).
summary_of_the_review: The main feature of this paper is considering a non-parametric policy class for Wasserstein policy optimization. However, the theoretical contribution adds marginal to the existing literature, and the numerical findings are not convincing to demonstrate the advantages of the new framework.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces Wasserstein and Sinkhorn policy optimization, with three main contributions. First, the paper derives closed form expressions for updates using Lagrange multipliers. Second, it proves monotonicity of performance improvement. Third, experiments show efficiency and effectiveness relative to TRPO and PPO. 

main_review: Review of EFFICIENT WASSERSTEIN AND SINKHORN POLICY OPTIMIZATION

In general, I found the paper to be a nice, if modest, extension of TRPO. In essence, the key idea is trading KL regularization for Wasserstein or Sinkhorn. Experiments show this is, in general, a win, particularly for the case of approximated advantage functions. I found the exposition to be mostly fine with some minor critiques. First, it would be better to have a more detailed treatment of the prior work (TRPO is properly introduced later). Second, separately discussing Wasserstein and Sinkhorn is unnecessary and redundant. They are very closely related and the text (and math) would benefit from making that clearer. 


Additional points: 
- The discussion of related work is inadequate. 
- It would be desirable to make a clear statement of what the reasons for and consequences of assumption 1 are. 
- "Compared to the performance bound when using KL-based trust region
(see, e.g., Schulman et al. (2015); Cen et al. (2020)), using the Wasserstein metric yields a tighter performance improvement bound and is more robust to the choice of parameters βt." It would be nice to show this explicitly. 
- Theorems 1 and 3 are closely related, as are Wasserstein and Sinkhorn. Is is really necessary to break them into separate sections? This choice makes the relation less clear. 
summary_of_the_review: Overall, this is not the most ambitious paper, but generally a nice contribution.
review_title: null

Meta-review: This paper proposes two extensions of the TRPO algorithm in which the trust region is defined using the Wasserstein distance and the Sinkhorn divergence. The proposed methods do not restrict the policy to belong to a parametric distribution class and the authors provide 
closed-form policy updates and a performance improvement bound for the Wasserstein policy optimization.
The authors provide an empirical evaluation of their approaches on tabular domains and some discrete locomotion tasks, comparing the performance with some state-of-the-art policy optimization approaches.

After reading the authors' feedback and interacting with the authors, the reviewers did not reach a consensus: one of the reviewers votes for rejection, while the other three reviewers are slightly positive.
In particular, the reviewer that voted for rejection raised a number of concerns that have been discussed at length with the authors, who were able to clarify some of the issues, but some of the answers did not satisfy the reviewer.
I went through the paper and I found the paper solid from a technical point of view, but I share some of the reviewers' concerns and I think that the authors should better position their contribution with respect to the state of the art. 
Overall, this paper is borderline and I feel it needs still some work to deserve clear acceptance (which I think will be soon).
Decision: Reject