Trust Region Policy Optimization with Optimal Transport Discrepancies: Duality and Algorithm for Continuous Actions
Authors: Antonio Terpin
Nicolas Lanzetti
Batuhan Yardim
Florian Dorfler
Giorgia Ramponi
Venue: NeurIPS 2022 Accept (2022)
Keywords: Trust region policy optimization
optimal transport

Abstract: Policy Optimization (PO) algorithms have been proven particularly suited to handle the high-dimensionality of real-world continuous control tasks. In this context, Trust Region Policy Optimization methods represent a popular approach to stabilize the policy updates. These usually rely on the Kullback-Leibler (KL) divergence to limit the change in the policy. The Wasserstein distance represents a natural alternative, in place of the KL divergence, to define trust regions or to regularize the objective function. However, state-of-the-art works either resort to its approximations or do not provide an algorithm for continuous state-action spaces, reducing the applicability of the method.
In this paper, we explore optimal transport discrepancies (which include the Wasserstein distance) to define trust regions, and we propose a novel algorithm - Optimal Transport Trust Region Policy Optimization (OT-TRPO) - for continuous state-action spaces. We circumvent the infinite-dimensional optimization problem for PO by providing a one-dimensional dual reformulation for which strong duality holds.
We then analytically derive the optimal policy update given the solution of the dual problem. This way, we bypass the computation of optimal transport costs and of optimal transport maps, which we implicitly characterize by solving the dual formulation.
Finally, we provide an experimental evaluation of our approach across various control tasks. Our results show that optimal transport discrepancies can offer an advantage over state-of-the-art approaches.

Trust Region Policy Optimization with Optimal Transport Discrepancies: Duality and Algorithm for Continuous Actions
Antonio Terpin* Automatic Control Laboratory ETH Zürich aterpin@ethz.ch ch
Nicolas Lanzetti" Automatic Control Laboratory
Batuhan Yardim
Dept. of Computer Science ETH Zürich
ETH Zürich
Inicolas@ethz.ch
alibatuhan..arddaaaa www..yyyarddmmmm..mmmmmm ch
Florian Dörfler
Giorgia Ramponi
Automatic Control Laboratory ETH Zürich dorflerGethz.ch ch
Dept. of Computer Science
ETH AI Center
giorgia.rampont @ainiiiiiiiai ethz. ch
Abstract
Policy Optimization (PO) algorithms have been proven particularly suited to handle the digeessinnns of real-world continuous control tasks. In this context, Trust Region Policy Optimization methods represent popular approach to stabilize the policy updates. These usually rely on the Kullback-Leibler (KL) divergence to limit the change in the policy. The Wasserstein distance represents a natural alternative, in place of the KI. divergence, to define trust regions or to regular- ize the objective function. However, state-of- -hht--aftteerrtttt works either resort to its approximations or do not provide an algorithm for continuous state-action spaces, reducing the applicability of the method. In this paper, we explore optimal trans- port discrepancies (which include the Wasserstein distance) to define trust regions, and we propose novel algorithm Optimal Transport Trust Region Policy Op- timization (OT-TRPO) for continuous state-action spaces. We circumvent the optimization problem for PO by providing dual reformulation for which strong duality holds. We then analytically derive the optimal policy update given the solution of the dual problem. This way, we bypass the computation of optimal transport costs and of optimal transport maps, which we implicitly characterize by solving the dual formulation. Finally, provide an experimental evaluation of our approach across various control tasks. Our results show that optimal transport discrepancies offer an advantage over state-of-the-ar approaches.
dinenniinii
1 Introduction
Reinforcement Learning (RL) has achieved outstanding results in numerous fields, from resource management [16], recommendation systems [42], and optimization of chemical reactions [44], to
video-games [18, 43, 10] and board games [39], without sparing the world's champion of GO [32]. Many of these successful applications rely Policy Optimization (PO) algorithms, family of RL methods that are particularly suited to handle the high-dimensionality real-world control tasks. PO algorithms approach the RL setting an optimization problem in the policy space. this context, the main challenge is to provide policy improvement guarantees. One remarkable option in this direction
Equal contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

is represented by Trust Region Policy Optimization (TRPO) [29]. which constrains the optimization problem to policies that are "close" to the current one, whereby the Kullback-Leibler (KL) divergence is used as similarity measure. Nevertheless, "closeness" in the policy space can also be quantified via other functions. Recent work [22, 33, 20] proposed to replace the KI. divergence with the Wasserstein distance, particular instance of optimal transport discrepancy (or cost). Besides being very natural and expressive, optimal transport discrepancies enjoy powerful topological, differential, geometrical, computational. and statistical features and guarantees [37, 3. 14J. In particular, (i) optimal transport discrepancies allow us to compare probability measures (and thus policies) not sharing the same support (for which the KL divergence is infinity); and (ii) they encapsulate the geometry encoded by the transport cost in the action space: the discrepancy between two actions coincides with the discrepancy between the corresponding deterministic policies (whereas the KL divergence is again infinity). These reasons make optimal transport discrepancies particularly attractive for RI.. However, the mere evaluation of optimal transport discrepancies entails solving a transportation problem (e.g., see (23, 34),, which poses significant computational challenges for its deployment. Most of the previous work on the topic [22, 20] overcomes the computational burden via approximation, effectively changing the original problem. Conversely, [33] proposes two algorithms to solve the PO problem exactly, studying the trust regions described via the Wasserstein distance and the Sinkhorn divergence. However, their analysis is limited to discrete (and finite) settings. In our work, we consider optimal transport discrepancies to construct the trust region in settings where actions and states take value in general compact Polish spaces. This allows tackling many applications of interest involving continuous domains such as physical control tasks. We derive and leverage dual reformulation of the PO problem to ensure optimal policy update within the trust region, without any additional need for line searches (conversely to [29]). We circumvent the computation of optimal transport discrepancies via an analytical expression of the transport maps, which are characterized thanks to the dual reformulation. Notably, our analysis enables a practical and efficient algorithm that encompasses both discrete and continuous settings.
Contributions. Our contributions are summarized follows:
1. We derive the dual of the optimal transport trust region policy optimization problem and we show that strong duality holds for general compact metric stat-acttio spaces. We further characterize the optimal policy update given the solution to the dual problem. We show that policy updates can result in monotonic improvement of the performance function.
2. We propose a novel PO algorithm for continuous spaces, Optimal Transport Trust Region Policy Optimization (OT-TRPO). Herein, we leverage the derived duality theory to provide policy updates that satisfy the optimal transport discrepancy constraint while circumventing its computation.
3. We conduct experiments in several RL benchmarks in both discrete and continuous state-action spaces, comparing our method to state-of-the-ar art approaches. Our results show the effectiveness of our approach for PO and the benefits of using optimal transport discrepancies.
2 Related Works
Optimal transport, and in particular the Wasserstein distance, has found various applications in RL and in particular in PO algorithms. In this section, we discuss the most relevant for our work; a broader overview is postponed to Appendix A. In [22], the authors propose Behavior Guided Policy Gradient (BGPG), whereby they replace the KL divergence trust region from TRPO [29] by a Wasserstein distance penalty in behavioral space. Although our alternating procedure in Section 5 may resemble the approach of [22] in spirit, our approach is fundamentally different; cfr. [22, Algorithms and 3] with Algorithm 1. In [20], the authors further suggested incorporating additional information about the local behavior of policies encapsulated in the so-called Wasserstein Information Matrix, in the attempt to speed up the PO using a Wasserstein Natural Policy Gradient (WNPG). However, these approaches are relatively slow, compared to the traditional Proximal Policy Optimization (PPO) and TRPO. Conversely to our work, they do not build on the idea of trust regions: we instead guarantee that the policy update is "close" to the previous one, where the "closeness" is defined via an optimal transport discrepancy (e.g., the Wasserstein distance). Accordingly, the closest related work to ours is the recent paper [33] which studied Wasserstein Policy Optimization (WPO) for discrete action spaces. In contrast to [33], the present work addresses the general setting of compact Polish spaces encompassing the cases of continuous and discrete state-action spaces as particular cases. While we also adopt a duality approach, our level of generality induces many challenges compared to the discrete action space setting (see Remark 2), and it is compatible even with non-direct policy
2

---

parametrizations. Finally, our work is closely connected with Wasserstein Distributionally Robust Optimization (DRO) [8, 19, 4, 25, 13]. Albeit our duality results are inspired from this literature. DRO is concerned with quantifying the worst-case risk of a cost functional over an ambiguity set of probability measures, which is a fundamentally different setting; see Remark 1. Conversely to all the previous work, we show how to perform exact optimal transport-based TRPO in continuous settings: we exploit optimal transport theory to circumvent the computational burden of evaluating optimal transport discrepancies while still performing exact policy updates within the trust regions. Accordingly, to the best of our knowledge, our practical algorithm completely novel.
3 Preliminaries
We briefly introduce useful background and notation for the remainder of the paper.
Notation. For every Polish space X' (i.e., completely metrizable separable topological space), the set of Borel probability measures on X is denoted by P()). The Dirac measure at some point E X' is denoted by 8z. Given two Polish spaces X.Y. a Borel probability measures E P()), and a Borel map T: X + y, the pushforward measure of ju, denoted by T4/4, is defined by (T=))(A) == (A)) for all A E B()), where (()) is the collection of Borel subsets of y. The set of probability measures be denoted by P(X). For a given function f X + R, the notation || refers to suPzex [f
finite set X coincides with the probability simplex and will also
Markov Decision Process. We consider an infinite-horizon discounted Markov Decision Process (MDP) [24] M
p. 7). where is the state space, is the action space, : SxAxS - R20 is the state transition probability kernel, r: A + R is the reward function, p is the initial state probability distribution, and 7 E [0.1) the discount factor. A randomized stationary Markovian policy, which we will simply call policy in the rest of the paper, is mapping π S + P(AA) specifying for each & E The set of all policies is denoted by II. Each policy æ E IT induces a discrete-time Markov reward process {(s,, r(st, (E.,, where 8t E S represents the state of the system at time t and r(st, ot) corresponds to the reward received when executing action de E A in state s, We denote by Pp,x the probability distribution of the Markov chain (st, az) issued from the MDP controlled by the policy æ with initial state distribution p. The associated expectation is denoted by Ep.r and the notation Eg is used whenever there is no dependence on p. The state-value function V= S > R and the action-value function Q" S x A + R are defined for all S,a A by (s) = Ex [E Y'r(s,, as) = s) and Q" (s, a) = E E ('r(s), az) su = 8, ag = a]. We also define the advantage function AT S x A - R by A ('(saa) a) (*((sa) V (s)). Given
probability measure over the set of actions A by (118)) E P(A).
an initial state probability distribution p. our goal is to find a policy x maximizing the expected long-term return
( = Ep p.x
Σ Crr(ss at) 1=
which is well-defined when, c.g., the reward function is bounded. To solve this PO problem, we only have access to the observed state, action, and reward St, Qt Tz at each time step i, whereas the state transition kernel P is unknown. When the state and action spaces (S and A) are finite, an optimal policy x is guaranteed to exist. When S and A are continuous, a (measurable) optimal policy is also guaranteed to exist (see [24, Theorem 6.1 11.. 1. p. 262]) under appropriate assumptions on the state and action spaces, the reward function and the transition kernel; we will explicit these later on. In this paper, we focus the continuous state-action space setting and comment the discrete (non necessarily finite) setting as a special case.
Optimal transport. Consider a Polish space X' and a continuous non-negative function e: X x X - R20, referred to as transport cost. Let μ1, v E P(X) and define the set of joint probability measures on X x X with marginals/ μ and v: I(4,,) v) = {7 E P(X x X): ((A x X) = ((A), x B) = v(B) VA e B(X)).
We define the optimal transport discrepancy on P(2) for every probability measures μ and by
C(p, = min [ cla, x ( (1,
(1)
gET(u,) xxx
Notice that this definition valid for both discrete and continuous measures. When e = dP, where d is a distance on X' and p > 1, then C(u, 2/2 reduces to the celebrated type-p Wasserstein
3

---

distance [37, 2]. In our PO context, we will use this discrepancy to compare two probability measures T(-(s) E (AA) and #(-ls) P(A) for every s S, where म, II two policies.
4 Optimal Transport for Trust Region Policy Optimization
In this section, we study the TRPO algorithm with a trust region defined using an optimal transport discrepancy as measure of closeness between policies. We prove that the arising optimization problem admits an amenable dual reformulation. Importantly, we show that, given the dual optimal solution, the primal solution has an analytical expression, which can lead to monotonic improvements of the performance index.
4.1 Policy iteration algorithm with optimal transport-based trust regions
By the policy difference lemma [11, Lemma 6.1], the difference between the expected returns of two policies n, II reads
- + / (s
where pa is the discounted state-occupancy measure [11]. The complex dependency of the discounted visitation frequency p= on the policy म hampers the direct optimization of (2); see [29, Sec. 2]. Following previous work, we consider instead a local approximation of the expected return J, defined by
L J / A (s, (s))
Observe that this approximation uses the discounted state-oceupancy measure Pr. (which can be estimated) instead of Pa (see (2)). In other words, the influence of a policy change the discounted state-occupancy measure is neglected. Moreover, this surrogate function coincides with the expected return J when मे л. Then, (3) motivates policy update rule maximizing at each time step the approximation L (T) over मं where т is the current policy that we want to improve upon (see also [33, Section 2, Eq. (1)]). To ensure stability of the update, we conservatively update the policy using a discrepancy constraint between the current and the new one. Unlike TRPO, we do not use the KL divergence define the trust region, but instead an optimal transport discrepancy. Then, at each time step, our method solves
sup ( (s, (ddd(s)d)d( #Ell
s.t. Ā E (T( = t # [
where > 0 is a parameter defining the radius of the trust region TT(T). Similarly [29, Eq. (12) and [33, Problem (4)], we consider the average optimal transport discrepancy over the state space as optimization constraint. Accordingly, the OT-TRPO policy optimization results from iteratively solving Problem (P).
4.2 Dual of the trust-region constrained problem (P)
Problem (P) is intractable for two main reasons. First, as soon as the state or action space is continuous, it is infinie-dimmensional optimization problem. Second, the mere evaluation of the trust-region constraint (e.g., for line search as in TRPO [29]) needs (possibly) infinitely many computations of the optimal transport discrepancy. which is itself already challenging estimate. However, inspired by prior works on Wasserstein DRO [8, 19, 41, 4], we show that problem (P) admits tractable one-dimensional convex dual reformulation. This duality theorem the cornerstone of the design of our algorithm. Before stating the result, we make the following assumptions.
The state space S is compact subset of an Euclidean space, the action space A a compact subset of a Polish space, the reward function is a continuous function and for every continuous function w on S, (N a) is continuous in both and a.
Assumption 1.

---

Under this assumption, there exists an optimal measurable (stationary) policy to the PO problem formulated in Section 3. We refer the reader to [24, Theorem 6.11.11, p. 262] for a statement of this result and milder assumptions. In particular, our duality result continues to hold if S is compact Polish space (i.e., not necessarily Euclidean).
Assumption 2. For every policy E II, the advantage function A" 8 A - R is continuous. Moreover, the transport cost e : x A + Rg is continuous and satisfies c(a, a) = 0 for all a E A.
In the next theorem, we show that under these assumptions Problem (P) admits a dual reformulation for which strong duality holds.
Theorem (Dual formulation). For every e >> 0 and for every policy E II, under Assumptions I and 2 the following strong duality result holds:
(1s
/
A" a) (ā(als)dp.(s) / ((f((-(s, SEf
max #En
(P)
d 1
min X20
A'E A max( (s, ac(a,aa))ddraaa a') (d)
(D)
Moreover, the primal and dual problems (P) and (D) admit maximizer and a minimizer, respectively,
Remarkably, Problem (D) -ieedmmesioonal and convex, and it only involves the current policy т. advantage function A=, and visitation frequency Px- The proof of Theorem is constructive. In particular, we derive a closed-form solution of problem (P) as a function of the optimal Lagrange multiplier X solving problem (D) and the policy defining the problem. Even if the closed-form policy part of Theorem I and its proof, we present separately for clarity and for later reference. To do so, we introduce some additional notation, which is instrumental to derive a practical algorithm (similarly to [33]).
Under Assumptions I and 2, define for every À 2 0 the X-regularized advantage Φ S x A R and its associated set of maximizers for every 8,a A as follows:
4 4x = max A (s, a') - Ac(a,
a'EA
(4)
D*(s, arg max A A Ac(a,
a'E.A
Corollary 2 (Optimal policy). Under the setting and assumptions of Theorem 1, for any policy π E II, let X3 2 O be the minimizer of the dual problem (D). Then, the following statements hold:
1. For every X 2 O, there exist two measurable selection maps Tx= S A and S such that for every s E S,a E A
A
ax) E arg min c(a,a) Tx(8,a) E arg max c(a,a') d'EDx(#.a)
(5)
d'EDx(s.a)
2. If X- > O, there exists t* E [0, such that
tt | Tx- (dd (1 (11 / cla, x =c.
(6)
3. There exists optimal feasible policy स for problem (P) defined for every E S #(-Is) (s. (-Is) (1- f')T (s.
where 1* results from (6) if X* 0 and
0.
Intuitively, Corollary 2 suggests that the optimal policy results from displacing the probability mass (als) to the maximizers of the X A--regularized advantage 1 x- a). where X 2 the optimal dual solution. Since maximizers are generally not unique (i.e., Dx- (s, a) is not a singleton), one needs to balance between the closest (i.e., Tx- (s, (s) and the furthest apart (i.e., Tx (s. a)) to satisfy the trust region constraint. In the special case (i.e.. the optimal policy lies within the trust region) or it does not affect the optimal trust region constraint (i.e.. the optimal policy would lie at the boundary the trust region even if the constraint is removed). In this case, it suffices to displace all probability mass to the closest maximizer of the advantage functions (i.e., To (s, a) E Do(s, a)). The complete proof of the results of this section deferred to Appendix B.
O, the trust region constraint is either not active
5

---

Remark 1. Similar results were previously established in the literature in the context of DRO (e.g.. see [8, Theorem 1] and [4. Theorem IJ). While these results closely inspire our proof, there major difference: in DRO, one seeks to evaluate the worst-case cost over an ambiguous set of probability distributions, expressed in terms of the optimal transport discrepancy. As such, the average optimal transport discrepancy in the trust region constraint is replaced by single optimal transport discrepancy. Thus, one does not need to ensure the regularity of the problem with respect to the state (e.g., measurability of L'x w.r.l. s). This is reflected in our assumption of joint continuity of the advantage in state and action and the state space being compact. To readily deploy existing results in DRO, one needs (i) to consider single state only (i.e., S {s)) (ii) to define trust region for each state (at the price of infinitely many constraints). To transform (P) into a DRO, one might alternatively identify policy as a probability measure over IL.s A, and hope deploy standard duality arguments DRO. However, the uncountable product of Polish spaces is not Polish space, which makes all results in DRO, and more generally in optimal transport [37], inapplicable.
Remark 2. A similar result in the discrete case was presented in [33]. We highlight four major differences. First, in the discrete setting, problem (P) is a finite-dimensional linear optimization problem, for which strong duality holds. Thus, linear programming arguments be used derive the dual reformulation. In the continuous setting, the same proof strategy would require to mobilize the abstract machinery of infinite-dimensionn dimensional linear programming [15]. Second, in the discrete setting, continuity and measurability of all functions are "for free". On the contrary, the continuous case imposes careful analysis of these issues. Third, the optimal policy update [33] implicitly assumes that the set Dxxs a) (see (4)) is a singleton, which rarely satisfied in practice. Fourth, as a byproduct of our proof, we show that (D) is (one-dimensiona)) convex optimization problem, which can be solved efficiently via off-the-shelf solvers. This way, we do not need to resort to approximation techniques [33, Section 6.1] for the optimal dual multiplier.
Discrete state-action spaces. In the remainder of this section, we specialize our results to discrete (finite) state-action spaces (which trivially satisfy Assumptions and 2). Without loss of generality, we represent the state and action spaces by S = (s1,..., 8M and A = (a1...., ..y) where M and N are two positive integers, and we describe any policy ı E IT and its corresponding state-occupancy measure Pa discrete measures:
M
<--18)) = Σ Ta 1=1
ViE (1,..,,,,M,
Pr =
Σ
PrOset
(8)
where Pi, Mi.j 2 0 for every (1, (1,..., M). (1, (I,..,, NJ. Di=1 Pi and E}= Tij 1 for
M
N
every i E {1....,M}, ? The analogous results to Theorem and Corollary 2 are as follows.
Corollary (Dual formulation discrete setting). Let > 0. For every policy 11. the following strong duality result holds:
M
N
max 11,0, JE EAA (E(L....M..EE (I....))
Σ
P
Tss
(1 () brs)):
(discrete-P)
M
N
Σ
Σ
(te(aj, bag) + (1-)))+(;;;; b(g)) se
P;
Tij
j=1
M
min
P:
(discrete-D)
X
j=1
In particular, let X 2 0 be a solution to (discrete-D), and given for every i E (l,...,M}, j e (I...., NN. select any
big E
arg min d'E (e),0))
claj,a'),
arg max ('EDx-(e...))
e(aj,a',,
and let
M
N The Tijcla, b,
c
P
P:
j=1
"This representation is also valid beyond the finite state-action space setting when the policies and the state-occupancy measures are empirical distributions with finitely many samples.

---

Then, an optimal policy क is given by
N
*(-1s4)
Σ The (f"bot, + +11 (1-19)05 EiE My
(10)
j=1
with t" = (2- s)/(2 - a) E [0, (and E (0.1) > 0 and to = 0 Of
The proof of this result stems from substituting the discrete measures defined in (8) in problems (P) and (D), and observing that the images of the mappings Tx. and x- have finite support in the current setting. Notably, Corollary directly provides an implementable algorithm for the policy update, circumventing the difficulty of the mixed-integer optimization problem (discrete-P): solving the one-dimensional convex program (discrete-D) provides the optimal Lagrange multiplier associated to the trust region constraint of the primal problem which can be directly used compute the actions , Di, via (9), and thus the policy á via (10).
Remark 3. The policy update suggested by (10) differs from the one in [33] (see [33, Theorem 1, (5)
where (i, E {0, 1} with their notation). Indeed, our policy update relies on "splitting the probability mass": the probability mass :t (-|ss) is displaced to by and Di,, with weights 1 and 1 t², respectively. This result is consistent with the Wasserstein DRO literature (e.g., see [4, Remark 2)).
The result provided in [33] corresponds to the particular case where bū = b which amounts to supposing that the set Dx (st, az) defined in (4) is singleton. We provide further comments and examples in Appendix A.2 to illustrate the importance of this "mass splitting".
4.3 Policy improvement
In the next result, we show that our policy update leads to a monotonic improvement of the perfor- mance function J up the advantage function estimation error.
Proposition 4 (Performance improvement). Let E IL Consider solutions #* II and X 0 of problems (P) and (D), respectively. If the true advantage function A is approximated by some estimated advantage function A such that ||AT - -A < co, then the following bound holds:
X 1 C(T(11s), ((1))))) (5) A A J( J(T) +
(11)
Proposition 4 indicates that optimal ransport-basedt trust region policy optimization leads to mono- tonic improvement of the performance function when have access to the true advantage function. The proof, postponed to Appendix B.2, of this result builds on the performance difference lemma (see (2)) and uses the closed-form expression of the optimal policy solving problem (P) as constructed in the proof of Theorem (see Corollary 2). The analog of this result for a finite action space was proved in [33, Theorem p. 5J. To the best of our knowledge, this result novel for the continuous state-action space setting.
5 Practical Optimal Transport Trust Region Policy Optimization Algorithm
In this section, we use the duality results on Section to derive a practical algorithm for OT- TRPO. Herein, we restrict the policy search set II to the set of policies no parametrized by vector E Rdd for some integer d > We require the policy parametrization to be continuously differentiable with respect to 0 (for every state and action). This way, simultaneously cover the direct parametrization (for which Corollary 3 directly provides policy update) as well as commonly used policy parametrizations (e.g., softmax and the Gaussian policies). Accordingly, the dual problem (D) can be reformulated as follows for every 0 E Rd::
min G(), 0) ds S (ATx a') Acla, a' (d)))ddr((((s)dd)) (8)- X20
(12)
7

---

Given a current policy represented by the vec-
Algorithm 1 OT-TRPO.
tor 0, we first solve the one-dimensional convex
1: Initialize Te,
problem (12) to obtain its solution Thee, we
2: for all t = 0, 1.... do
use the optimal dual multiplier X to derive the optimal policy update within the trust region. The procedure summarized in Algorithm 1. De- pending on the parametrization of the policy, the steps of Section are as follows:
3: Estimate A and Pro."
4: Compute X argminx20 G(),
5: Update de to 0.+1 using X"
6: end for
Algorithm step In the discrete setting, the visitation frequency is estimated via Monte Carlo methods. In the continuous case, we weight every visited state equally. We propose three ways to estimate the unknown advantage function via samples':
1. Monte Carlo methods or TD-learning (for discrete settings only);
General Advantage Estimation (GAE) [30], using neural network to approximate the value function like in standard actor-critic methods; and
3. Direct estimation via non-linear approximators (e.g., using directly a neural network for the advantage function).
A Igorithm step 4 (evaluation of G). Depending on the setting, we propose various ways to evaluate O). They all apply to both continuous and discrete states.
Finite actions: Since the maximization in (12) is over finitely many actions, we can directly evaluate (12) for any X 2 O.
2. Gaussian policy parametrization: With m(s) being the mean of the Gaussian policy (with fixed variance), we approximate G(),0) by
A ExE max {A (s, d') - Ac(ms (s), Pro if A via GAE, G(A,0) ~ A maxx' {A (s, d') - Ac(ms(s), a d') (s) if A via NN,
where S are the states visited in the trajectory and A(s) is a (possibly state-dependent) collection of samples from A.
3. Arbitrary policy parametrization: For neural network approximation of the advantage function, we approximate G(X, 0) by
G(A ~ AS + ExEs Encdi(s) max (17 (s, a') - Ac(a, (()ool)))) (s),
where S are the states visited in the trajectory and (s) are (possibly state-dependent) collections of samples from A.
Algorithm 1 step (solving for X*). Since (D) is one-dimensional convex optimization problem, x can be found using any solver for convex optimization problems.
Algorithm 1 step 5. Update the parameter vector 0.
Direct parametrization (finite spuces): Update according to (10) and (9).
Direct parametrization via policy network (continuous states, discrete actions): Use (10) and (9) to compute the optimal policy update at the visited states, denoted by गे. Then, update the policy
network by performing gradient descent on the loss L(0) = Σ.E$ Pmo, (s) (-(-ss) गद. (-1s)|| to steer Ty towards the optimal policy update тे, within the trust region.
3. Arbitrary policy parametrization: Since there are infinitely many actions, the computation of the maximization is computationally demanding, and so Corollary 2 cannot be directly utilized for the policy update. Yet, according to Corollary 2, the optimal policy update attains the maximum max,'EA ( i (s, a') X X"c(a, (()a,a)) at each state. Thus, can steer the policy my to maximize
update the policy via gradient ascent. The intuition is as follows:
Eves SA max,' (A (x, (x,a') X c(aa, af) d())dmo(ao) (aols)) (s).
In the particular case of a Gaussian policy with parametrized mean (and fixed variance), combined with GAE estimate of the advantage function, one can maximize
E.Es max ( Te, (s, a') X ((), a'). 03Pso, (s).
In the experiments reported in the main paper we used the first and the second method for discrete and continuous environments, respectively. In Appendix A.5 we further comment the different methods.
8

---

Intuitively, this update implicitly estimates the transport maps Tx- and y-. which are needed for the optimal policy update. This way, we steer the policy network towards the optimal policy update within the trust region. Among others, this policy update allows for the following interpretation: imposing an optimal mansport-based trust region constraints is, at least formally, equivalent to maximizing a regularized advantage function, where the value of the regularization A 2 0 is based on the transport cost c and the radius of the trust region E.
Experiments and Insights
In this section, we evaluate the performance of OT-TRPO across variety of environments [5, 36] of increasing complexity, ranging from discrete continuous settings. We compare to the classical TRPO [29, 9] and PPO [31, 9], with Advantage Actor Critic (A2C) [17, 9], with the recent approaches leveraging the Wasserstein distance, BGPG [22] and WNPG [20] (in continuous settings), and with WPO [33] (in discrete settings). The training curves are shown in Fig. I; see Appendix A.3 for implementation details, Appendix A.6 for further details on the experimental results, and Appendix A.4 for an ablation study on our algorithm.
CliffWaking-v0
MoumainnCr
100
-20
-2,000
-400
600
-100
-4,000
200
400
20
40
60
80
100
Learning steps (10")
Learning stops (10")
Learning steps (10?)
Hopper-v3 v3
Swimnner-v3
HallCheetah
400
6,000
4,000
2,000
200
2,000
1,000
0
0.2
0.4
0.6
0.8
0
0.4
0.6
0.8
0.2
0.8 Learning steps (10")
0.4
0.6
Learning steps (10")
Learning steps (10")
OT-TRPO TRPO
WPO
PPO
A2C
BGPG
WNPG
Figure I: Cumulative rewards during the training process in diffcrent environments. The shaded area represents the mean + the standard deviation across 10 independent runs. Every policy evaluation in each run is averaged over 10 sampled trajectories.
Our approach is shown to consistently improve over the other algorithms: OT-TRPO leads to larger final returns, with lower variance, and only in few cases the expense of slightly slower learning curve. Four remarks are in order. First, the performance gain of OT-TRPO compared to BGPG and WNPG confirms that trust regions help stabilize training, already observed in [29]. Second, optimal transport discrepancies induce a more natural notion of "closeness" between policies compared to the KL divergence (c.g., see [3, Example 2.1]). For instance, CliffWalking-vO, consider the optimal policy TT" and the candidate policy depicted below, which differ at one state only (see figure). The optimal transport discrepancy between and म is P (+()C((T()sss) x² (-(s)) = (*(s) ())()))(D(())))))))) (s) Right). When us- ing the KL divergence, instead, the discrepancy infinite, since the two policies do not share the same support. In particular, if
*": optimal policy, m: "close" policy
initialized with म, TRPO cannot converge to the optimal policy, regardless of the radius of the trust region. Third, OT-TRPO improves on WPO, in two ways: (i) leads to superior per- formances of the trained policies and (ii) it does not violate

---

the trust region constraint (which, e.g., in Taxi-v3 is the case for 72% of the updates of WPO). This performance improvement results from the "mass splitting" (see Remark 3), which is the only difference between the two algorithms (in discrete settings). Fourth, in continuous settings, the per- formance of OT-TRPO is aligned with the best-performing alternative approach. the environment Swimmer-v3, it even yields an improvement of more than 50% in the performance of the trained agent.
Conclusion and Future Work
We studied trust region policy optimization for continuous state-action spaces whereby the trust region is defined in terms of a general optimal transport discrepancy. Our analysis bases on one- dimensional convex dual reformulation of the optimization problem for the policy update which (i) enjoys strong duality and (ii) directly characterizes the optimal policy update, bypassing the computational burden of evaluating optimal transport discrepancies. Moreover, we show that the policy update can yield a monotonic improvement of the performance index. Empowered by our theoretic results, propose novel algorithm, OT-TRPO, for trust region policy optimization with optimal transport discrepancies. We evaluate its performance across several environments. Our results reveal that trust regions defined by optimal transport discrepancies can offer advantages the KI. divergence or non-trust region methods.
There are several research directions that merit further investigation. We highlight two. First, transport costs provide us actionable knobs to shape the geometry of the trust region, and can used to encode prior knowledge the environment preferred exploration strategies. Second, we would like to study the convergence properties of the proposed algorithm.
Acknowledgements
This project has received funding from Google Brain, Swiss National Science Foundation under the NCCR Automation (grant agreement 51NF40 180005)8 and was partially supported by the ETH AI Center.
References
[1] Charalambos D. Aliprantis and Kim C. Border. Infinite Dimensional Analysis: a Hitchhiker's Guide. Springer, Berlin; London, 2006.
[2] Luigi Ambrosio, Nicola Gigli, and Giuseppe Savaré. Gradient flows: In Metric Spaces and in the Space of Probability Measures. Springer, 2008.
[3] Liviu Aolaritei, Nicolas Lanzetti, Hongruyu Chen, and Florian Dörfler. Uncertainty propagation via optimal transport ambiguity sets. arXiv preprint arXiv:2205.00343. 2022.
[4] Jose Blanchet and Karthyek Murthy. Quantifying distributional model risk via optimal transport. Mathematics of Operations Research, 44(2):565-600, 2019.
[5] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. arXiv preprint /X06:/606.01.. 2016.
[6] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
[7] Rui Gao, Xi Chen, and Anton J. Kleywegt. Wasserstein distributionally robust optimization and variation regularization. arXiv preprint arXiv:1712.0660 2017.
[8] Rui Gao and Anton J. Kleywegt. Distributionaalyy robust stochastic optimization with Wasser- stein distance. arXiv preprint arXiv:1604.02199 2016.
[9] Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. https: //gi thub. com hoomhhlllea a/ stable wateeebbaeeeeen...ee 2018.
10

---

[10] Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Mitos, Blažej Osiński, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, al. Model based reinforcement learning for Atari. In International Conference Learning Representations, 2019.
[11] Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In International Conference on Machine Learning, pages 267-274. PMLR, 2002.
[12] Achim Klenke. Probability Theory; A Comprehemsive Course. Springer, 2008.
[13] Daniel Kuhn, Peyman Mohajerin Esfahani, Viet Anh Nguyen, and Soroosh Shaficezadeh- Abadeh. Wasserstein distributionally robust optimization: Theory and applications machine learning. In Operations research & management science in the age of analytics, pages 130-166. Informs, 2019.
[14] Nicolas Lanzetti, Saverio Bolognani, and Florian Dörfler. First-order conditions for optimization in the Wasserstein space. arXiv preprint arXiv:2209. 12/97, 2022.
[15] David G. Luenberger. Optimization by vector space methods. John Wiley & Sons, 1997.
[16] Hongzi Mao, Mohammad Alizadeh, Ishai Menache, and Srikanth Kandula. Resource manage- ment with deep reinforcement learning, In 15th ACM workshop on hot topics in networks, pages 50-56, 2016.
[17] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. In International Conference on Machine Learning, pages 1928 1938.193. PMLR, 2016.
(18) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: /3/2.50/2. 2013.
[19] Peyman Mohajerin Esfahani and Daniel Kuhn. Data-driven distributionally robust optimization using the Wasserstein metrie: performance guarantees and tractable reformulations, Mathemati- cal Programming. 171(1):115-166, 2018
[20] Ted Moskovitz, Michael Arbel, Ferenc Huszar, and Arthur Gretton. Efficient Wasserstein natural gradients for reinforcement learning. arXiv preprint arXiv:2010.05380, 2020.
[21] James R. Munkres. Topology. Featured Titles for Topology. Prentice Hall, Incorporated, 2000.
[22] Aldo Pacchiano, Jack Parker-Holder, Yunhao Tang, Anna Choromanska, Krzysztof Choroman- ski, and Michael Jordan. Learning score behaviors for guided policy optimization. In International Conference on Machine Learning, pages 7401-7410. PMLR, 2020.
[23] Gabriel Peyré and Marco Cuturi. Computational optimal transport. Foundations and Trends in Machine Learning, 11 11(5-))1--277, 2019.
[24] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2005.
[25] Hamed Rahimian and Sanjay Mehrotra. Distributionally robust optimization: A review. arXiv preprint arXiv: 05659, 2019.
[26] Pierre Richemond and Brendan Maginnis, On Wasserstein reinforcement learning and the Fokker-Plane equation. arXiv preprint arXiv:1712.07188, 2017.
[27] Ralph Tyrell Rockafellar. Convex Analysis. Princeton University Press, 2015.
[28] Walter Rudin. Real and Complex Analysis, 3rd Ed. McGraw-Hill, Inc., USA, 1987.
[29] John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust region policy optimization, International Conference on Machine Learning, pages 1889-1897. PMLR, 2015.
11

---

[30] John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. International Conference Learning Representations, 2016.
[31] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017.
[32] D. Silver, A. Huang, and C. et al. Maddison. Mastering the game of go with deep neural networks and tree search. Nature 529, 484 489, 2016.
[33] Jun Song, Chaoyue Zhao, and Niao He. Efficient Wasserstein and Sinkhorn policy optimization, 2022.
[34] Bahar Taskesen, Soroosh Shafieczadeh-Abete Abadeh, and Daniel Kuhn. Semi- remi-discrete optimal transport: Hardness, regularization and numerical solution. arXiv preprint arXiv:2 (tdi:::003306266. 2021.
[35] Dávid Terjék and Diego and generalized Sinkhorn algorithm. International Conference on Artificial Intelligence and Statistics, pages 5135-5165. PMLR, 2022.
Optimal transport with f-divergence regularization
[36] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012.
[37] Cédric Villani. Optimal Transport: Old and New. Springer-Verlas Berlin Heidelberg, 2008
[38] Patrick Nadeem Ward, Ariella Smofsky, and Avishek Joey Bose. Improving exploration in soft-actor-critic with normalizing flows policies. arXiv preprint arXiv: 1rxiv::900.00277,, 2019.
[39] Konstantia Xenou, Georgios Chalkiadakis, and Stergos Afantenos. Deep reinforcement learning in strategic board game environments. Springer International Publishing, 2019.
[40] Ruiyi Zhang, Changyou Chen, Chunyuan Li, and Lawrence Carin. Policy optimization as wasserstein gradient flows. In International Conference Machine Learning, pages 5737- 5746. PMLR, 2018.
[41] Chaoyue Zhao and Yongpei Guan. Data-driven risk-averse stochastic optimization with Wasser- stein metric. Operations Research Letters, 46(2):262-25 267, 2018.
[42] Guanjie Zheng, Fuzheng Zhang, Zihan Zheng, Yang Xiang, Nicholas Jing Yuan, Xing Xie, and Zhenhui Li. DRN: A deep reinforcement learning framework for news recommendation. In World Wide Web Conference, pages 167-176, 2018.
[43] Yue Zheng. Reinforcement learning and vidco games. arXiv preprint arXiv:1909.04777 2019.
[44] Zhenpeng Zhou, Xiaocheng Li, and Richard N Zare. Optimizing chemical reactions with deep reinforcement learning. ACS Central Science, 3(12):1337-1344, 2017.
12


Reviews:
[{"review_id": "zSyH6EZwr87", "reviewer": "Reviewer_71r1", "summary": "The paper under review studied the problem of trust region policy optimization(TRPO).\nTRPO aims to optimize the expected return through stable policy update, where the \nstability is control by only allowing update within certain region around the current policy \nin each iteration. Hence one needs to pick a norm to measure distances between policies.\nIn this paper, the authors proposed to use OT distance.\nAllowing the continuous setting for action and state spaces, \nthey derive the dual of the proposed OT-TRPO and proved strong duality,\nbase on which, an update algorithm was proposed.\nMoreover, by establishing eq (11), the authors demonstrated the monotonic improvement of the proposed method.\nIn addition, the authors illustrated the efficiency of OT-TRPO through simulations.\n\n", "questions": "Q1: In section 4.3, the authors stated that eq(11) shows monotonicity of the update.\nHowever, to guarantee $J(\\tilde{\\pi}^*) > J(\\pi)$, one needs to show that the sum of the last two terms in eq(11) is positive.\nIt is not clear why the sum is always positive. Would the authors explain a bit further on it?\n\nQ2: Regarding the experiments: what are the choices of costs and other parameters? Are the results sensitive to these choices?\nWhat is the running time comparing to other methods? In particular, step 4 and step 5 in Algo1.\n\nQ3: Is it possible to empirically validate that the proposed algorithm actually approximates the solution of problem (P)?\nFor example, check for a small discrete setting where OT solution can be computed?\n\nQ4: Does duality proved in Thm 1 holds for Sinkhorn divergence (instead of classical OT without the entropy term)? \n", "limitations": "The authors mentioned the convergence properties of the proposed algorithm is left open.\nI agree with the authors that proving converges and calculation of convergence rate \nis an essential step that is currently missing.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written and easy to follow.\nThe idea of using optimal transport discrepancy is natural and interesting.\nThe proposed update algorithm is nicely stated and and properly motivated.\nThe fact that it bypass the actual computation of the forward OT has many potential applications.\nThe theoretical results are sounds besides some questions listed below.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "oTtyCU6rbb", "reviewer": "Reviewer_hVuN", "summary": "This paper studies the trust region policy optimization (TRPO) problem, by replacing the KL divergence policy discrepancy constraint in TRPO with optimal transport discrepancies, such as Wasserstein distance. Considering the optimal transport discrepancy region, a dual form of the constrained problem has been developed, which contains a Lagrange multiplier \\lambda. Find the optimal \\lambda^* at each step only needs to solve a one-dimensional convex optimization problem, which can be solved using off-the-shelf solvers, instead of line-search in TRPO. Then, the optimal \\pi can be computed using \\lambda^* with closed-form solution. Experiments on both discrete and continuous settings are conducted.", "questions": "1. Are there any explanations that for many environments, such as Taxi, Hopper, and Swimmer, TRPO learns much fasters at early stages compared to OT-TRPO? Does these environments share some similarity in dealing with the trust regions?\n\n2. WPO can be extended to deal with continuous actions as studied in the original paper, while it is ignored here. For example, in Hopper, WPO can reach much higher scores if we count for the same amount of learning steps. In Figure 1, it seems TRPO can achieve very competitive performance, while this differs from the results reported in BGPG and WNPG. Would the implementation of TRPO has some particular treatment here? \n\n3. Would an efficiency comparison with the line-search procedure in TRPO be possible? Since one important benefit using the optimal transport discrepancy is that the optimal policy at each step has a closed-form solution to avoid the computational line-search.\n\n4. Line 341: “it leads to superior asymptotic performance”. I do not think the sentence is rigorous, since when you refer to asymptotic performance you consider nearly infinite samples while the learning steps in the experiments are of very small scale considering current RL problems.\n\n5. Minor typos:\nLine 338: “not do” -> “do not”\n", "limitations": "Some experimental results of the compared methods are not clear enough. Benchmark environments should be enriched to thoroughly evaluate the empirical performance of OT-TRPO.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\nThe paper is clearly written. It proposes a new dual form of trust region PO problem using optimal transport discrepancy, where the solution can be computed efficiently compared to line-search using KL divergence.\n\n\nCons:\n\nThe compared methods have included many closely related works in trust region PO, however, the benchmark environments are somehow limited and insufficient. Since the OT-TRPO is derived for both discrete and continuous actions, to give a thorough support for the effectiveness of OT-TRPO, it is recommended to evaluate on some standard RL benchmarks such as the Atari suite, and Mujoco families. The considered tasks, such as CliffWalking, Taxi, MountainCar, Swimmer, etc., are relatively simple environments, in which the learning steps are relatively small to obtain good performances for general algorithms.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WdLp8pbDv0y", "reviewer": "Reviewer_JmUx", "summary": "The paper proposes to use optimal transport discrepancies as an alternative to the Kullback-Leibler divergence for Trust Region Policy Optimisation (TRPO). The authors provide the intractable optimisation problem (P), and then propose a dual formulation (D) of the optimal transport problem which is tractable. They base the proposed algorithm upon this dual formulation and perform the analysis of the method. \n", "questions": "Questions:\n\n**Question 1:** While the analysis in the paper gives substantial detail on experimental comparison against other methods, as well as on a choice of optimal transport distance, it is still important to give more details on the failure modes. The closest I could get to that were the results from Hopper (Figure 1), where the model performs comparably with standard TRPO. Would be interesting to see if there are any other examples when TRPO works pretty much as well or outperforms the model, and what could be the reason behind it? While seeing that the experimental results show convincing experimental advantage, it looks like covering more limitations could be helpful in understanding the potential downsides of the method.\n\n**Question 2:**\nIn line 664 onwards, authors write:\n “In continuous action spaces, we also experimented with training a neural network that approximates  the advantage function of the current policy using Generalized Advantage Estimators. Our experimental results indicate that policy optimization with such a neural network approximate does not perform comparably to the single sample estimation of the objective. In general, we observed that (1) simultaneous training of an advantage network and the OT-TRPO policy update could be unstable, and (2) in the case of convergence, OT-TRPO with neural advantage function approximation converges  to a suboptimal policy. We hypothesize that the OT-TRPO objective is sensitive to biased value estimates due to the infimum over the action space in the neighborhood of the mean of the policy.  The observation that the optimal GAE hyperparameter in the single-sample advantage estimation  was very close to 1 (i.e., an unbiased Monte Carlo estimate) in our experiments supports this idea. “Would be really good to explicitly state, for the experiments in the main text and in the appendix, which policy parameterisations and which advantage function estimates correspond to which experiment. To support the claim, is it possible, for example, to plot the comparison graphs for different types of estimates? \n\n**Question 3: **\nIn section A.5, I haven’t found enough information on the details of implementation of the Monte-Carlo estimate, could the authors clarify on this to ensure reproducibility?", "limitations": "Societal impact limitations: I think the authors sufficiently addressed them, especially given that the problem statement of reinforcement learning in simulated environments, the datasets and similar methods are well-known as detailed in Section 2 of the paper.\n\nTechnical limitations: as outlined in questions, more experimental evidence on the failure modes could be given (see Question 1 for the details). ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n- The problem statement is interesting and gives insight into optimal transport formulation of the TRPO. The mathematical implementation is solid, and the idea of dual problem formulation to circumvent the intractability is theoretically well grounded.\n- The work is well placed within the current literature, and the authors contrast it with the existing trust regions and optimal transport-based methods such as Song et al (2022).\n\nWeaknesses:\n- While the paper is sound, in the experimental analysis, it could be good to show the failure modes (see below) (see question 1 for the details below)\n- It would be also good to show the impact of different policy parameterisations as described in Algorithm 1, Step 5, as well as different estimates of the advantage function (Step 5); it is also important to more clearly outline which experiment uses which policy parameterisation and the advantage function (see question 2)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept