Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies the trust region policy optimization (TRPO) problem, by replacing the KL divergence policy discrepancy constraint in TRPO with optimal transport discrepancies, such as Wasserstein distance. Considering the optimal transport discrepancy region, a dual form of the constrained problem has been developed, which contains a Lagrange multiplier \lambda. Find the optimal \lambda^* at each step only needs to solve a one-dimensional convex optimization problem, which can be solved using off-the-shelf solvers, instead of line-search in TRPO. Then, the optimal \pi can be computed using \lambda^* with closed-form solution. Experiments on both discrete and continuous settings are conducted.
strengths_and_weaknesses: Pros:

The paper is clearly written. It proposes a new dual form of trust region PO problem using optimal transport discrepancy, where the solution can be computed efficiently compared to line-search using KL divergence.


Cons:

The compared methods have included many closely related works in trust region PO, however, the benchmark environments are somehow limited and insufficient. Since the OT-TRPO is derived for both discrete and continuous actions, to give a thorough support for the effectiveness of OT-TRPO, it is recommended to evaluate on some standard RL benchmarks such as the Atari suite, and Mujoco families. The considered tasks, such as CliffWalking, Taxi, MountainCar, Swimmer, etc., are relatively simple environments, in which the learning steps are relatively small to obtain good performances for general algorithms.

questions: 1. Are there any explanations that for many environments, such as Taxi, Hopper, and Swimmer, TRPO learns much fasters at early stages compared to OT-TRPO? Does these environments share some similarity in dealing with the trust regions?

2. WPO can be extended to deal with continuous actions as studied in the original paper, while it is ignored here. For example, in Hopper, WPO can reach much higher scores if we count for the same amount of learning steps. In Figure 1, it seems TRPO can achieve very competitive performance, while this differs from the results reported in BGPG and WNPG. Would the implementation of TRPO has some particular treatment here? 

3. Would an efficiency comparison with the line-search procedure in TRPO be possible? Since one important benefit using the optimal transport discrepancy is that the optimal policy at each step has a closed-form solution to avoid the computational line-search.

4. Line 341: “it leads to superior asymptotic performance”. I do not think the sentence is rigorous, since when you refer to asymptotic performance you consider nearly infinite samples while the learning steps in the experiments are of very small scale considering current RL problems.

5. Minor typos:
Line 338: “not do” -> “do not”

limitations: Some experimental results of the compared methods are not clear enough. Benchmark environments should be enriched to thoroughly evaluate the empirical performance of OT-TRPO.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper under review studied the problem of trust region policy optimization(TRPO).
TRPO aims to optimize the expected return through stable policy update, where the 
stability is control by only allowing update within certain region around the current policy 
in each iteration. Hence one needs to pick a norm to measure distances between policies.
In this paper, the authors proposed to use OT distance.
Allowing the continuous setting for action and state spaces, 
they derive the dual of the proposed OT-TRPO and proved strong duality,
base on which, an update algorithm was proposed.
Moreover, by establishing eq (11), the authors demonstrated the monotonic improvement of the proposed method.
In addition, the authors illustrated the efficiency of OT-TRPO through simulations.


strengths_and_weaknesses: The paper is well written and easy to follow.
The idea of using optimal transport discrepancy is natural and interesting.
The proposed update algorithm is nicely stated and and properly motivated.
The fact that it bypass the actual computation of the forward OT has many potential applications.
The theoretical results are sounds besides some questions listed below.



questions: Q1: In section 4.3, the authors stated that eq(11) shows monotonicity of the update.
However, to guarantee $J(\tilde{\pi}^*) > J(\pi)$, one needs to show that the sum of the last two terms in eq(11) is positive.
It is not clear why the sum is always positive. Would the authors explain a bit further on it?

Q2: Regarding the experiments: what are the choices of costs and other parameters? Are the results sensitive to these choices?
What is the running time comparing to other methods? In particular, step 4 and step 5 in Algo1.

Q3: Is it possible to empirically validate that the proposed algorithm actually approximates the solution of problem (P)?
For example, check for a small discrete setting where OT solution can be computed?

Q4: Does duality proved in Thm 1 holds for Sinkhorn divergence (instead of classical OT without the entropy term)? 

limitations: The authors mentioned the convergence properties of the proposed algorithm is left open.
I agree with the authors that proving converges and calculation of convergence rate 
is an essential step that is currently missing.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes to use optimal transport discrepancies as an alternative to the Kullback-Leibler divergence for Trust Region Policy Optimisation (TRPO). The authors provide the intractable optimisation problem (P), and then propose a dual formulation (D) of the optimal transport problem which is tractable. They base the proposed algorithm upon this dual formulation and perform the analysis of the method. 

strengths_and_weaknesses: 
Strengths:
- The problem statement is interesting and gives insight into optimal transport formulation of the TRPO. The mathematical implementation is solid, and the idea of dual problem formulation to circumvent the intractability is theoretically well grounded.
- The work is well placed within the current literature, and the authors contrast it with the existing trust regions and optimal transport-based methods such as Song et al (2022).

Weaknesses:
- While the paper is sound, in the experimental analysis, it could be good to show the failure modes (see below) (see question 1 for the details below)
- It would be also good to show the impact of different policy parameterisations as described in Algorithm 1, Step 5, as well as different estimates of the advantage function (Step 5); it is also important to more clearly outline which experiment uses which policy parameterisation and the advantage function (see question 2)

questions: Questions:

**Question 1:** While the analysis in the paper gives substantial detail on experimental comparison against other methods, as well as on a choice of optimal transport distance, it is still important to give more details on the failure modes. The closest I could get to that were the results from Hopper (Figure 1), where the model performs comparably with standard TRPO. Would be interesting to see if there are any other examples when TRPO works pretty much as well or outperforms the model, and what could be the reason behind it? While seeing that the experimental results show convincing experimental advantage, it looks like covering more limitations could be helpful in understanding the potential downsides of the method.

**Question 2:**
In line 664 onwards, authors write:
 “In continuous action spaces, we also experimented with training a neural network that approximates  the advantage function of the current policy using Generalized Advantage Estimators. Our experimental results indicate that policy optimization with such a neural network approximate does not perform comparably to the single sample estimation of the objective. In general, we observed that (1) simultaneous training of an advantage network and the OT-TRPO policy update could be unstable, and (2) in the case of convergence, OT-TRPO with neural advantage function approximation converges  to a suboptimal policy. We hypothesize that the OT-TRPO objective is sensitive to biased value estimates due to the infimum over the action space in the neighborhood of the mean of the policy.  The observation that the optimal GAE hyperparameter in the single-sample advantage estimation  was very close to 1 (i.e., an unbiased Monte Carlo estimate) in our experiments supports this idea. “Would be really good to explicitly state, for the experiments in the main text and in the appendix, which policy parameterisations and which advantage function estimates correspond to which experiment. To support the claim, is it possible, for example, to plot the comparison graphs for different types of estimates? 

**Question 3: **
In section A.5, I haven’t found enough information on the details of implementation of the Monte-Carlo estimate, could the authors clarify on this to ensure reproducibility?
limitations: Societal impact limitations: I think the authors sufficiently addressed them, especially given that the problem statement of reinforcement learning in simulated environments, the datasets and similar methods are well-known as detailed in Section 2 of the paper.

Technical limitations: as outlined in questions, more experimental evidence on the failure modes could be given (see Question 1 for the details).
review_title: null

Meta-review: metareview: The paper studies trust region optimization but replace the typical KL divergence with optimal transport distance - which is a natural and meaningful generalization. The authors provided a tractable algorithm by using optimization duality, and provide experimental results on control tasks. All reviewers appreciate the main idea is novel and interesting, and further the paper is very written and easy to understand. Some reviewer have questions about the theory and experiments. The authors largely address the reviewers' comments during rebuttal, and all authors are in favor of acceptance.
confidence: Certain
Decision: Accept