Truly Deterministic Policy Optimization
Authors: Ehsan Saleh
Saba Ghaffari
Matthew West
Tim Bretl
Venue:  (2021)
Keywords: Deterministic Policy Gradient
Deterministic Exploration
Reinforcement Learning

Abstract: In this paper, we present a policy gradient method that avoids exploratory noise injection and performs policy search over the deterministic landscape. By avoiding noise injection all sources of estimation variance can be eliminated in systems with deterministic dynamics (up to the initial state distribution). Since deterministic policy regularization is impossible using traditional non-metric measures such as the KL divergence, we derive a Wasserstein-based quadratic model for our purposes. We state conditions on the system model under which it is possible to establish a monotonic policy improvement guarantee, propose a surrogate function for policy gradient estimation, and show that it is possible to compute exact advantage estimates if both the state transition model and the policy are deterministic. Finally, we describe two novel robotic control environments---one with non-local rewards in the frequency domain and the other with a long horizon (8000 time-steps)---for which our policy gradient method (TDPO) significantly outperforms existing methods (PPO, TRPO, DDPG, and TD3).

Under review as a conference paper at ICLR 2021
TRULY DETERMINISTIC POLICY OPTIMIZATION
Anonymous authors
Paper under double-blind review
ABSTRACT
In this paper, we present a policy gradient method that avoids exploratory noise injection and performs policy search over the deterministic landscape. By avoiding noise injection all sources of estimation variance can be eliminated in systems with deterministic dynamics (up the initial state distribution). Since deterministic policy regularization is impossible using traditional non-metric measures such as the KL divergence, we derive a Wasserstein-basse quadratic model for our purposes. We state conditions on the system model under which it is possible to establish a monotonic policy improvement guarantee, propose surrogate function for policy gradient estimation, and show that it is possible to compute exact advantage estimates both the state transition model and the policy are deterministic. Finally, we deseribe two novel robotic control environments one with non-local rewards in the frequency domain and the other with a long horizon (8000 stee-tteps)- for which policy gradient method (TDPO) significantly outperforms existing methods (PPO, TRPO, DDPG, and TD3).
Policy Gradient (PG) methods can be broadly characterized by three defining elements: the policy gradient estimator, the regularization measures, and the exploration profile. For gradient estimation, episodic (Williams 1992), serportance-sapllnge
(Schulman et al. 2015a), and deterministic (Silver et al. 2014) gradients are some of the most common estimation oracles. As for regularization measures, either an Euclidean distance within the parameter space (Williams) 1992; Silver et al. al. 2015a Kakade & Langford 2002 Schuuman et al. 2017 Kakade 2002; Wu et al. 201 7) have been frequently adapted. Common exploration profiles include Gaussian (Schulman al.] (2155a) and stochastic processes (Lillicrap et al. 2015). These elements form the basis of many model-free and stochastic policy optimization methods successfully capable of learning high-dimensional policy
2014 Lillicap al., 2015), or dimensionally consistent non-metric measures (Schulman
parameters.
Both stochastic and deterministic policy search can be useful in applications. stochastic policy has the effect of smoothing or filtering the policy landscape, which is desirable for optimization. Searching through stochastic policies has enabled the effective control of challenging environments under general framework (Schulman et al. 2015a 2017). The same method could either learn robotic movements or play basic games (1) with minimal comainn-spccffe knowledge, (2) regardless of function approximation classes, and (3) with less human intervention (ignoring reward engineering and hyper-paramete tuning) (Duan et al. 2016). Using stochasticity for exploration, although it imposes approximations and variance, has provided a robust way to actively search for higher rewards. Despite many successes, there are practical environments which remain challenging for current policy gradient methods. For example, non-local rewards (e.g., those defined in the frequency domain), long time horizons, and cetturrall--esoonn environments all occur in realistic robotic systems (Kuo) & Golnaraghi] 2002, Meirovitch] 1975 Preumont & Seto 2008) but can present issues for policy gradient search.
To tackle challenging environments such as these, this paper considers policy gradient methods based on deterministic policies and deterministic gradient estimates, which could offer advantages by allowing the estimation of global reward gradients on long horizons without the need to inject noise into the system for exploration. To facilitate dimensionally consistent and low-variance deterministic policy search, a compatible policy gradient estimator and a metric measure for regularization should be employed. For gradient estimation focus on Vine estimators (Schulman et al.) 2015a), which can be easily applied to deterministic policies. As a metric measure we use the Wasserstein distance, which measure meaningful distances between deterministic policy functions that have non-

---

Under review as a conference paper at ICLR 2021
overlapping supports (in contrast to the Kullback-Liebler (KL) divergence and the Total Variation (TV) distance).
The Wasserstein metric has seen substantial recent application in variety of machine-learning domains, such as the successful stable learning of generative adversarial models (Arjovsky et al.
2017). Theoretically, this metric has been studied in the context of Lipschitz-continuon Markov decision processes in reinforcement learning (Hinderer 2005 Ferns et al. 2012). Pirotta et al.) (2015) defined policy gradient method using the Wasserestein distance by relying on Lipschitz continuity assumptions with respect to the policy gradient itself. Furthermore, for Lipschitz-continoous Markov decision processes, AAadi et al. (2018) and Rachelson & Lagoudakis| (2010) used the Wasserstein distance to derive model- based value-iteration and policy-iteration methods, respectively. On a more practical note, (aacchaao et al. (2019) utilized Wasserstein regularization for behavior-guided stochastic policy optimization. Moreover, Abdullah et al. (2019) has proposed another robust stochastic policy gradient formulation. Estimating the Wasserstein distance for general distributions is more complicated than typical KL-divergences (Villanil 2008). This fact constitutes and emphasizes the contributions offAbdullah et al. (2019) and Pacchiano et al. (2019). However, for our deterministic observation- conditionnnniiiinnnnll. policies, closed-form computation of Wasserstein distances is possible without any approximation.
Existing deterministic policy gradient methods (c.g.. DDPG and TD3) use deterministic policies (Silver et al.) 2014] Lillicrap et al. 2015 Fujimoto et al. 2018), meaning that they learn a deterministic policy function from states actions. However, such methods still use stochastic search (i.e., they add stochastic noise to their deterministic actions to force exploration during policy search). In contrast, we will be interested in a method which not only uses deterministic policies, but also uses deterministic search (i.e., without constant stochastic noise injection). We call this new method truly deterministic policy optimization (TDPO) and may have lower estimation variances and better scalability to long horizons, as we will show in numerical examples.
Scalability to long horizons is one of the most challenging aspects for policy gradient methods that use stochastic search. This issue is sometimes referred to as the curse of horizon in reinforcement learning (Liu et al., 2018). General worst-case analyses suggests that the sample complexity of
reinforcement learning is exponential with respect to the horizon length (Kakade et al. [2003] Kearns ct al. 2000 2002). Deriving polynomial lower-bounds for the sample complexity of reinforcement learning methods is still open problem (Jiang & Agarwal 2018). Llwer-bounding the sample complexity of reinforcement learning for long horizons under different settings and simplifying assumptions has been topic of theoretical research (Dann & Brunskill, 20155 Wang al.! 2020). Some recent work has examined the scalability of importance sampling gradient estimators to long horizons in terms of both theoretical and practical estimator variances (Liu et al., 2018 Kallus & Uchara 2019 2020). All in all, long horizons are challenging for all reinforcement learning methods, especially the ones suffering from excessive estimation variance due to the use of stochastic policies for exploration, and our truly deterministic method may have advantages in this respect.
In this paper we focus on continuous domann robotic environments with reset capability to previously visited states. The main contributions of this work are: (I) we introduce Deterministic Vine (DeVine) policy gradient estimator which avoids constant exploratory noise injection; (2) we derive a novel iittemmniistcalll-comppaiiiii
surrogate function and provide monotonic payoff improvement guarantees; (3) we show how to use the Vine policy gradient estimator with the Wasserstein- surrogate in a practical algorithm (TDPO: Truly Deterministic Policy Optimization); (4) we illustrate the robustness of the TDPO policy search process in robotic control environments with non-local rewards, long horizons, and/or resonant frequencies.
BACKGROUND
MDP preliminaries. An infinite-horizon discounted Markov decision process (MDP) is specified by (S, P. P, n,7), where S is the state space, AA is the action space, P P:S $ x A + (SS) is the transition dynamics, R S x A + [0, Rmaa] is the reward function, 7 E [0,1) is the discount factor, and p(s) is the initial state distribution of interest (where ((F) denotes the set of all probability distributions over F. otherwise known the Credal set of F). The transition dynamics P is defined as an operator which produces a distribution over the state space for the next state s' P(s,a) a). The transition dynamics can be easily generalized to take distributions of states or
2

---

Under review as a conference paper at ICLR 2021
actions as input (i.e., by having P defined P: ((S) x A
P: A(A) A(S).
We may abuse the notation and replace 6₂ and 62 by s and a. where 6N and 6, are the deterministic distributions concentrated at the state and action a, respectively. A policy π S + A(A) specifies a distribution over actions for each state, and induces trajectories from a given starting state & as
follows: $1 8, 01 =(81), r1 R( 82 P(s2, a2), 02 I (s2),, etc. We will denote trajectories as state-action tuples T = (51, a1, 52, a2,...). One can generalize the dynamics (1) by using policy instead of an action distribution P(Us, TT == Es-y, L (s) a)]l, and (2) by introducing the t-step transition dynamics recursively as (',., T) == P(P"-1(p4 (Hs,
(Ms..) R) T) with pd (Ps: TT) == Ms, where M4 is distribution over S. The visitation frequency can then be defined as Pz =1 - - pp--ppp-111 (1, 1T. Table 2f the appendix summarizes all MDP notation.
The value function of π is defined as V (s) := E| "" $1 = s; æ]. Similarly, one can define (" a) by conditioning on the first action. The advantage function can then be defined as their difference (i.e. A ()) := Q* of one policy with respect to another using A (s,
V (s)). Generally, one can define the advantage/value ([Q" a) (*))
x' r((ss) and Q" (s, r') == E[[" (s, a) a ~ T' (-[s)]. Finally, the payoff of policy T/ == E[V (s);s s ~ pd] is the average value over the initial states distribution of the MDP.
Probabilistic and mathematical notations. Sometimes refer to I f(x)g(x)dz integrals as (f. g), Hilbert space inner products. Assuming that Ç and V are two probabilistic densities, the Kulback-Liebler (KL) divergence is DRI((lP) == (C(x)) log( ((a))) the MariatVaiation (TV) distance is TV(Ç, =: {(((()) - -v(x),, 1)2, and the Wasserstein distance is W(C, v) = inf (I|=r - yll, (25, y))s where v) the set of couplings for Ç and We de- fine Lip(f (z, v):x) == sup, fz, v)ll and assume the existence of Lip(Q a); a) and (RK) duality
|| Lip((V. VV.Q"((s (s. a): constants. Under this notation, the Aebiintten-Kantrrvviie states that the (((xx) -=x),, f(=)) S W(S.v) Lip(f;=)) bound is tight for all f. For brevity, may abuse the notation and denote sup, W ((T (-))) (-((s) with W (T1, T2) (and similarly for other measures). For parameterized policies, we define V fT) == (af(T) where is parameterized by the vector 0 Table I of the appendix summarizes all these mathematical definitions.
Policy gradient preliminaries. The advantage decomposition lemma provides insight into the relationship between payoff improvements and advantages (Kakade & Langford 2002). That is,
Tm.
E Eap [A (s, T2)]. (s)
(1)
1
We will denote the current and the candidate next policy as क. and π2, respectively. Taking derivatives of both sides with respect to T2 at Ti yields
1 - 1-7
VV (v. ()) (T m1)) + (p% (+), V Vn A (2))
(2)
Since क. does not have any advantage over itself (i.e., A (., 1)) 0), the first term is zero. Thus, the Policy Gradient (PG) theorem is derived as
E
[V A A (s, x2)]
(3)
52=K1
For policy iteration with function approximation, we assume nं2 and Ti to be parameterized by 0₂ and f1, respectively. One can view the PG theorem as a Taylor expansion of the payoff at 0j.
Conservative Policy Iteration (CPI) (Kkaade & Langford 2002) was one of the early dimensionally consistent methods with a surrogate of the form Cr,
(T2) (T))
Tm. + T E - [AT (s, (2))
C TVV T2). The coefficient guarantees non-decreasing payoffs. However, CPI is limited to linear function approximation classes due to the update rule Tnew <- (1 T. ax'. This lead to the design of the Trust Region Policy Optimization (TRPO) (Schulman et al. 2015a) algorithm.
3
TRPO exchanged the bounded squared TV distance with the KL divergence by lower bounding it using the Pinsker inequality. This made TRPO closer to the Natural Policy Gradient algorithm(Kake (akaade 2002), and for Gaussian policies the modified terms had similar Taylor expansions within small trust
3

---

Under review as a conference paper at ICLR 2021
regions. Confined trust regions are stable way of making large updates and avoiding pessimistic coefficients. For gradient estimates, TRPO used Importance Sampling (IS) with a baseline shift:
@Th (s. (s. [2(a
Vo,B VepBonpyi [ (s, n2)]
Vo,
(4)
02=0;
smpph a
(I(as) 10-0
While empirical [= (s, T2]] and ([Q T2)] estimates yield identical variances in principle, the
importance sampling estimator in (4) imposes larger variances. Later, Proximal Policy Optimization (PPO) (Schulman et al., 2015b) proposed utilizing the Generalized Advantage Estimation (GAE) method for variance reduction and incorporated first-order smoothing like ADAM (Kingma & Baj (TD) learning (Bhatnagar et al., 2009) for variance reduction. Although TD-learning was not theoretically guaranteed to converge and could add bias,
2014). GAE employed
improved the estimation accuracy.
As an alternative to importance sampling, deterministic policy gradient estimators were also utilized in an actor-critic fashion. Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al. 2015)
generalized deterministic gradients by employing Approximate Dynamic Programming (ADP) (Mnih
et al.
2013) for variance reduction. Twin Delayed Deterministie Policy Gradient (TD3) (Fujimoto
et al. 2018) improved DDPG's approximation to build an even better policy optimization method.
Although both methods used deterministic policies, they still performed stochastic search by adding stochastic noise to the deterministic policies to force exploration.
Other lines of stochastic policy optimization were later proposed. Wu et al. (017 used a Kronecker- factored approximation for curvatures. HHarnoja et al. (2018) proposed a maximum entropy actor- critic method for stochastic policy optimization.
2 MONOTONIC POLICY IMPROVEMENT GUARANTEE
We use the Wasserstein metric because it allows the effective measurement of distances between probability distributions or functions with con-overlapping support, such as deterministic policies, unlike the KL divergence or TV distance which are either unbounded or maximal in this case. The physical transition model's smoothness enables the use of the Wasserstein distance to regularize deterministic policies. Therefore, we make the following two assumptions about the transition model:
(PP(p, 11), P(u, x2)) S L W(T1, (2),
(5)
W (PP(PL.T) +) < L, W (21)
(6)
Also, we make the dynamics stability assumption sup Σ [. 1 700
1p. < oo, with
the definitions of the new constants and further discussion of the implications deferred to Section of the appendix where we also discuss Assumptions 5 and and the existence of Lip(Q" (P"(Q(ss, a);
The advantage decomposition lemma can be rewritten as
TT. 1/ 1-7 E [A (s, (s, R2))
1-7 (p) P A (.,T)))
The
-p-- - 0 AT m2)) term has zero gradient at T2 = TI. which qualifies it to be crudely called "the second-order term". The theory behind lower-bounding this second-order term with all derivations left to the appendix. Next, we present the theoretical bottom line.
2.1 THE MONOTONIC PAYOFF IMPROVEMENT GUARANTEE
Combining the results of Inequality 30 and Theorems A.5 and A.4 of the appendix leads us to define the regularization terms and coefficients:
2 Lip(Q=)(s, C = sup
L
7) L4
22 L
C₂ sup
(1-7)(1- 7,,)
Lwa(mi, 82:8) = W( ग. (a|s))
Te(a|s") + + (als)) T2(a)s) + T (als'')
x V V. W
(8)
2
2
4

---

Under review as a conference paper at ICLR 2021
This gives the novel lower bound for payoff improvement:
cuu (x2) = m/ +
Barpi
[A (8, m2) - Ci sup (2;
C2 sup (MTz(a)s)
(9)
We have the inequalities mm Theorem 2.1 as an instance of Minorization iettrattoonMaaimizzae Maximization algorithms (Hunter & Lange 2004).
C (m2) and (T)) - mm. This facilitates the application of
Theorem 2.1. Successive maximization of L yields non non-decreasing policy payoffs.
Proof. With arg max, C we have ( 2 C (TT)) Thus,
1/2 z Lx (22) and 1)r = L (TI) 1)52 1752 2 Loup ( (T22 L (TI) 2 0.
(10)
SURROGATE OPTIMIZATION AND PRACTICAL ALGORITHM
Successive optimization of L (T)) generates non-decreasing payoffs. However, it is impractical due to the large number of constraints and statistical estimation maximums. To mitigate this, we take similar approach TRPO and optimize for the surrogate
(T2) 1/1
1-7 Benpj [A ( (2) C1 Bxupp (wo((m)22is))
C2 BaNpj [(Ta(als)
(1I)
Although first order stochastic optimization methods can be directly applied to the surrogate defined in Equation (11), second order methods can be more efficient. Since ( #2; 8) is the geometric mean of two Functions of quadratic order, it is also of quadratic order. However, (wa(T 725 s) may not be twice continuously differentiable. For this, lower bound (wa(T1, T2द s) further using the AM-GM inequality and replace it with
T2 (2((ass) + (als)) Tz(aas) mi (alss)
LGa (T1, T2; 8) V.
(12)
e'=el12
to form quadratic regularization term with definable Hessian matrix (see Section A.8 of the appendix for detailed derivations). This induces our final surrogate which is used for second order optimization:
1
Cx, (T2) Ta. +
E Banpp
[AT (s,
C Banpf [oa(TI,
1 1-
C2 Bann [(Te(als)
(13)
The coefficients C₁ and C2 are dynamics-dependent For simplicity, we used constant coefficients and a trust region. This yields the Truly Deterministic Policy Optimization (TDPO) as given in Algorithm T See the appendix for practical notes on the choice of C₁ and Cz. Alternatively, one could adopt processes similar to SSShullman et al. (2015a) where the IS -S-baee advantage estimator used a line search for proper step size selection, or the adaptive penalty coefficient setting in Schulman al. 2017). We plan consider such approaches future work.
THE INTERPRETATION OF THE SURROGATE FUNCTION
For deterministic policies, the squared Wasserstein distance W W(Tz(als), Ti (?) degenerates to the Euclidean distance over the action space. Any policy defines a sensitivity matrix at a given state which is the Jacobian matrix of the policy output with respect The policy sensitivity
5

---

Under review as a conference paper at ICLR 2021
Algorithm Truly Deterministic Policy Optimization (TDPO)
Require: The squared Wasserestein regularization coefficient C22 The secondary regularization coefficient ratio C//C2, and a trust region radius δmax-
Require: Initial policy wo-
Require: Advantage estimator and sample collector oracle A" for == .........
2: Collect trajectories and construct the advantage estimator oracle ATh,
3: Compute the policy gradient at 0g 9 + V ( |='=T.
4: Construct surrogate Hessian vector product oracle 7 H uu such that for 0' =0g 80,
E g
(al'(a)s), ma(als))]] +
Brnpo (1' mx: C2
Ci
50 H50 + h.o.t., (14)
where h.o.t. denotes higher order terms 60.
5: Find the optimal update direction 80 H 1-1g "g using the Conjugate Gradient algorithm.
6: Determine the best step size a" within the trust region:
C2 (a50*)) H(a00))
= arg max of (1550)
2
a
s.t.
2 (a' (("") 80") 6 6
(15)
7: Update the policy parameters: 0k+1 t- 0g 58"
8: end for
term (TI T2: s) is essentially the squared Euclidean distance over the Jacobian matrix elements. In other words, our surrogate prefers to step in directions where the action-to-observation sensitivity is preserved within updates.
Although surrogate
metric distance instead of the traditional non-metric measures for regu- larization, we do not consider this sole replacement a major contribution. The squared Wasserestein distance and the KL divergence of two identically-scales Gaussian distributions to constant (i.e., DKI. (m,, [ o)) ([N(m,, R), (mm,,, o))² /2o²). On the other hand, our surrogate's 's compatibility with deterministic policies makes it valuable asset for policy gradient algorithm; both deterministic policies и and 72 numerically without resorting any approximations overcome singularities.
the same up
Tı (als)) and ( s2 can be evaluated for two
4 MODEL-FREE ESTIMATION OF POLICY GRADIENT
The DeVine advantage estimator is formally defined in Algorithm 2 Unlike DDPG and TD3, the DeVine estimator allows method perform deterministic search by not consistently injecting noise in actions for exploration. Essentially, De Vine rolls out trajectory and computes the values of each state. Since the transition dynamics and the policy deterministic, these values exact. Then, it picks perturbation state according to the visitation frequencies using importance sampling. A state-reset s; is made, a o-pertureed action is applied for single time-step, followed by Ti policy. This exactly produces Q (s,, a, + o). Then, A (St, + or) can be computed by subtracting the and A (St, ar ar) define two-point linear A (8t, a)
value baseline. Finally, AT A 0z) model with respect to the action. Parallelization can be used to have as many states of the first roll-out included in the estimator as desired. The parameter a acts as an exploration parameter and a finite difference to establish derivatives. While ~ 0 can produce exact gradients, larger a can build stabler interpolations.
Under deterministic dynamics and policies, if the DeVine oracle samples each dimension at each time-step exactly once then in the limit small a it can produce exact advantages, as stated in Theorem 4.I whose proof is deferred to the appendix.
Theorem 4.1. Assume a finite horizon MDP with both deterministic transition dynamics P and initial distribution μ. with maximal horizon length of H. Define K H - dim((A), uniform v, and

---

Under review as a conference paper at ICLR 2021
Algorithm 2 Deterministic Vine (DeVine) Policy Advantage Estimator
Require: The number of parallel workers K
Require:
policy и, an exploration policy q, discrete time-step distribution ((),, initial state distribution j(s)), and the discount factor
1: Sample an initial state So from μ., and then roll out a trajectory T = (so, do, $1 01 ... using n
2: for k 2. K do
3: Sample the integer number t lx from V.
4: Compute the value V** = Di= " ai).
5: Reset the initial state to 9t, sample the first action of according to (-11s)), and use ı for the of the trajectory. This will create T' (st, ag 6/41 "441""..)
6: Compute the value (s,, d.) Ei=,"" " a()
7: Compute the advantage A (St, a,) (st, a,) - (s)).
8: end for
K dim(AA) . 77th (T2(s) U2e (a/z atc)) A (Siz) d(.). (a/2 a (a ane)
9: Define AT =
K
v(tx)
k=1
10: Return AT (T2) and V Vx,A A (T2) as unbiased estimators for Esupji [A (s, T2)] and PG, re- spectively.
((s;)) = qi (s) σe, in Algorithm with e, being the j basis element for A. If the (j,tz) pairs sampled to exactly cover {1, (i) x (1, (1.... HJ. then we have
lim V m. A (T2)
V Ura mm=m1
(16)
Theorem 4.1 provides guarantee for recovering the exact policy gradient if the initial state distri- bution was deterministic and all time-steps of the trajectory were used to branch vine trajectories. Although this theorem sets the stage for computing a fully deterministic gradient, stochastic ap- proximation can be used in Algorithmp 2 by randomly sampling a small set of states for advantage estimation. In other words, Theorem 4. would use v to deterministically sample all trajectory states, whereas this is not practical requirement for A and the gradients are still unbiased if random set of vine branches used.
The Vine estimator can be advantageous in at least two scenarios. First, in the case of rewards that cannot be decomposed into summations of immediate rewards. For example, overshoot penalizations frequency-bee based rewards as used in robotic systems are non-local. DeVine can be robust to non-local rewards as it is insensitive to whether the rewards were applied immediately or after a long period. Second, De Vine can be an appropriate choice for systems that sensitive to the injection of noise, such as high-bandwidth robots with natural resonant frequencies. In such cases, using white (or colored) noise for exploration can excite these resonant frequencies and cause instability, making learning difficult. DeVine avoids the need for constant noise injection.
5 EXPERIMENTS
The next two subsections show challenging robotic control tasks including frequency-base based non-local rewards, long horizons, and sensitivity to resonant frequencies. See Section AA.1 of the appendix for comparison on traditional gym environments. TDPO works similar or slightly worse on these traditional gym environments as they seem to be well-suited for stochastic exploration.
5.1 AN ENVIRONMENT WITH NON-LOCAL REWARDS
The first environment that we consider simple pendulum. The transition function is standard the states are joint angle and joint velocity, and the action is joint torque. The reward function is non- standard- rather than define a local reward in the time domain with the goal of making the pendulum
Non- rewards are reward functions of the entire trajectory whose payoff's cannot be decomposed into the sum of terms such as 17 E, f(,,, at), where functions only depend on nearby states and actions. An example non-local reward is one that depends the Fourier transform of the complete trajectory signal.
7

---

Under review as a conference paper at ICLR 2021
TDPO (ours)
Disc. Payoff
-10-
-10
-10²
TRPO
PPO
DDR
-10a
100M
200M
300M
400M
500M
Sample Count
TRPO
DDPG
TD3
10
10
10
5
10
10
Time (s)
Time
Ampituzte (tao)
Desired Offset
Desired Offset
d-- Desired Offset
+- Desired Offset
Desired Offset
0.4
Desired
Desired
Frequency
Frequency Amplitude
Frequency Amplitude
Frequency
Frequency
0.2
Amplitudo
Amplitude
Amplitude
10
5
10
Frequency (H2)
Frequency (H2)
Frequency (Hz)
Frequency (H2)
Frequency (H2)
Figure 1: Results for the simple pendulum with non-local rewards. Upper panel: training curves with empirical discounted payoff's. Lower panels: trajectories both the time domain and frequency domain, showing target values of oscillation frequency, amplitude, and offset.
stand upright (for example), we define a non-local reward in the frequency domain with the goal of making the pendulum oscillate with desired frequency and amplitude about desired offset. In particular, we compute this non-local reward by taking the Fourier transform of the joint angle signal over the entire trajectory and by penalizing differences between the resulting power spectrum and desired power spectrum. We apply this non-local reward at the last time step of the trajectory. Implementation details and similar results for more pendulum variants are left to the appendix.
Figure T shows training curves for TDPO (our method) compared to TRPO, PPO, DDPG, and TD3. These results were averaged over 25 experiments which the desired oscillation frequency was 1.7 Hz (different from the pendulum's 's natural frequency of 0.5 Hz), the desired oscillation amplitude was 0.28 rad, and the desired offset was 0.52 rad. Figure alloo shows trajectories obtained by the best agents from each method. TDPO (our method) was able to learn high-reward behavior and to achieve the desired frequency, amplitude, and offset. TRPO was able to learn the correct offset but did not produce any oscillatory behavior. TD3 also learned the correct offset, but could not produce desirable oscillations. PPO and DDPG failed to learn any desired behavior.
5.2 AN ENVIRONMENT WITH LONG HORIZON AND RESONANT FREQUENCIES
The second environment that we consider is single leg from quadruped robot (Park et al. 2017). This leg has two joints, a "hip" and a "knee," about which is possible to exert torques. The hip is attached slider that confines motion to a vertical line above flat ground. We assume the leg is dropped from some height above the ground and the task is to recover from this drop and to stand upright rest after impact. States given the agent are the angle and velocity of each joint (slider position and velocity are hidden), and actions are the joint torques. The reward function penalizes difference from upright posture, slipping chattering at the contact between the foot and the
concept from control theory. In the frequency domain, signals of certain frequencies excited more than others when applied to system. This captured by the frequency-domain transfer function the system, which may have peak of magnitude greater than one. The resonant frequeney the frequency at which the frequency-domen of systems with a resonant frequency include the undamped pendulum, which oscillates at its natural frequency. and RLC circuits which have characteristic frequencies at which they most excitable. See Chapter 8 of Kuo & Golnaraghi (2002) for information.
*Resonant frequencies
transfer function has the highest amplitude. Common examples
8

---

Under review as a conference paper at ICLR 2021
-3x10>
TDPO (ours)
Payoft
TRPO
10
3x10
TD3
DDPG
0
100M
200M
300M
400M
Sample Count
TDPO
TRPO
PPO
DDPG
TD3
Velocibies (radis)
Hip
Hip
Hip
0-
-20
Knee
en
5
Torques (N m)
my Knee
Knee
Knee
Knee
0-
Hip
Hip
Hip
Hip
0,2
0.3
0.2
0.3
0.2
0.3
D.2
0.3
0.2
0.3
Time
Time (s)
Time
Time
Figure 2: Results for the leg environment with long horizon and resonant frequencies due to ground compliance. Upper panel: training curves with empirical discounted payoffs.. Lower panel: partial trajectories, restricted to times shortly before and after impact with the ground. Note the oscillations about 100 Hz that appear just after the impact at 0.2 s these oscillations evidence of resonant frequency.
ground, non-zero joint velocities, and steady-state joint torque deviations. We use MuJoCo for simulation (Todorov et al. 20122), with high-fidelity models of ground compliance, motor nonlinearity, and joint friction. The control loop rate is 4000 Hz and the rollout length 2 s, resulting horizon of 8000 steps. Implementation details are left to the appendix.
Figure 2 shows training curves for TDPO (our method) as compared to TRPO, PPO, DDPG and TD3. These results were averaged over 75 experiments. A discount factor of 7 0.99975 was chosen for all methods, where (1 7) is half the trajectory length. Similarly, the GAE factors for PPO and TRPO were scaled up to 0. 0.99875 and 0.9995, respectively, in proportion to the trajectory length. Figure 2 shows trajectories obtained by the hest agents from each method. TDPO (our method) was able learn high-reward behavior. TRPO, PPO, DDPG, and TD3 were not.
We hypothesize that the reason for this difference performance is that TDPO better handles the combination of challenges presented by the leg environment-ai unusually long time horizon (8000 steps) and the existence of resonant frequency that results from compliance between the foot and the ground (note the oscillations a frequency of about 100 Hz that appear in the trajectories after impact). Both high-speed control loops and resonance due to ground compliance are common features of real-world legged robots which TDPO seems be more resilient.
DISCUSSION
This article proposed a deterministic policy gradient method (TDPO: Truly Deterministic Policy Op- timization) based on the use of a deterministic Vine (DeVine) gradient estimator and the Wasserstein metric. We proved monotonic payoff guarantees for our method, and proposed novel surrogate for policy optimization. We showed numerical evidence for superior performance with non-local rewards defined in the frequency domain and a realistic long-horizon resonant environment. This method enables applications of policy gradient to customize frequency response characteristics of agents. Future work should include the addition a regularization coefficient adaptation process for the C1 and C2 parameters in the TDPO algorithm.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Mohammed Amin Abdullah, Hang Ren, Haitham Bou Ammar, Vladimir Milenkovic, Rui Luo, Mingtian Zhang, and Jun Wang- Wasserstein robust reinforcement learning. arXiv preprint 19Xi::/907.13196, 2019.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan. arXiv preprint arXiv:1701.07875 2017.
Kavosh Asadi, Dipendra Misra, and Michael L. Littman. Lipschitz continuity in model-based reinforcement learning. arXiv preprint arXiv:1804.0711 2018.
Shalabh Bhatnagar, Doina Precup. David Silver, Richard S Sutton, Hamid R Maei, and Csaba Szepesvári. Convergent temporal-different learning with arbitrary smooth function approximation. In Advances in neural information processing systems, pp. 2204-1212, 2009.
Christoph Dann and Emma Brunskill. Sample complexity of episodic
reinforcement
learning. In Advances in Neural Information Processing Systems, pp. 2818-2826, 2015.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Marthias Plappert, Alec Radford,
John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai basclines. https
2017.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning. pp. 1329-1338, 2016.
Norman Ferns, Prakash Panangaden, and Doina Precup. Metrics for markov decision processes with infinite state spaces. arXiv preprint 12xx.v/207... 2012.
Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint
2018.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, 2010.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maxi- mum entropy deep reinforcement learning with stochastic actor. arXiv preprint 1xXiv:80010111 2018.
Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. https://github...
hill-al a/ aaw-aa/sbbbbl stable baselines: 2018.
Karl Hinderer. Lipschitz continuity of value functions in markovian decision processes. Mathematical Methods of Operations Research, 62(1):3-22, 2005.
David R Hunter and Kenneth Lange. A tutorial on mm algorithms. The American Statistician, 58(1): 30-37, 2004.
Nan Jiang and Alekh Agarwal. Open problem: The dependence of sample complexity lower bounds on planning horizon. In Conference Learning Theory, pp. 3395-3398, 2018.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning, In ICML, volume 2, pp. 267-274, 2002.
Sham M Kakade. A natural policy gradient. In Advances in neural information processing systems, pp. 1531-1538, 2002.
Sham Machandranath Kakade et al. On the sample complexity of reinforcement learning. PhD thesis, University of London London, England, 2003.
10

---

Under review as a conference paper at ICLR 2021
Nathan Kallus and Masatoshi Uehara. Efficiently breaking the curse of horizon in off- policy evaluation with double reinforcement learning. arXiv, pp. arXiv-1909, 2019.
Nathan Kallus and Masatoshi Uehara. Statistically efficient off-policy policy gradients. arXiv preprint arXiv:2002.040/1 2020.
Michael Kearns, Yishay Mansour, and Andrew Y Ng. A sparse sampling algorithm for near-optimal planning in large markov decision processes. Machine learning, 49(2-3):193-200) 2002.
Michael J Kearns, Yishay Mansour, and Andrew Y Ng. Approximate planning in large pomdps via reusable trajectories. Advances in Neural Information Processing Systems, pp. 1001-1007, 2000.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 6980, 2014.
Benjamin C. Kuo and Farid Golnaraghi. Automatic Control Systems. John Wiley & Sons, Inc., USA, 8th edition, 2002. ISBN 0471134767.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.029)) 02971, 2015.
Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the curse of horizon: Infinite- horizon off-policy estimation. Advances in Neural Information Processing Systems, pp. 5356- 5366, 2018.
Leonard Meirovitch. Elements of vibration analysis. McGraw-Hill Science, Engineering & Mathe- matics, 1975.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 51 518(7540):529-55 -339995331111 2015.
Aldo Pacchiano, Jack Parker- Holder, Yunhao Tang, Anna Choromanska, Krzysztof Choromanski. and Michael Jordan. Learning to score behaviors for guided policy optimization. arXiv preprint DrXiv:/9066 04349, 2019.
Hae-Won Park, Patrick M Wensing, and Sangbae Kim. High-speed bounding with the mit cheetah 2: Control design and experiments. The International Journal of Robotics Research, 3672):167-192. 2017.
Matteo Pirotta, Marcello Restelli, and Luca Bascetta. Policy gradient in lipschitz markov decision processes. Machine Learning. 100(2-3):25-2285 2015.
André Preumont and Kazuto Seto. Active control of structures. John Wiley & Sons, 2008
Emmanuel Rachelson and Michail G. Lagoudakis. On the locality action domination in sequential decision making. In / Ith International Symposium on Artificial Intelligence and Mathematics (ISIAM 2010), pp. 1-8, Fort Lauderdale, US, 2010. Utthttts:: //tt:p:://aato.. oo univetoulouse Ex Ex/179777
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897,, 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.022. 02438, 2015b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1rX1v:1707.06347, 2017.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In ICMI., 2014.
11

---

Under review as a conference paper at ICLR 2021
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033 IEEE, 2012.
Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008.
Ruosong Wang, Simon S Du, Lin F Yang, and Sham M Kakade. Is long horizon reinforcement learning more difficult than short horizon reinforcement learning? arXiv preprint arXiv:2005.00527, 2020.
Ronald J Williams. Simple statistical gradicnt-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-2566 1992.
Yuhuai Wu, Elman Mansimov, Roger B Grosse, Shun Liao, and Jimmy Ba. Scalable trust-region method for deep reinforcement learning using cronecker-factoree approximation. In Advances in neural information processing systems, pp. 5279-5288, 2017.
12

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A. TABLES OF NOTATION
The same mathematical definitions and notations used in the paper were re-introduced and summarized in two tables; Table [ describes the mathematical functions and operators used throughout the paper, and Table describes the notations needed define the Markov Decision Process (MDP). The tables consist of two columns; one showing or defining the notation, and the other includes the name in which the same notation was called in the paper.
Name
Mathematical Definition or Description
VT(():= 1 E scrppj (R(st,a;) 4:)] 1-7 a(r))(s1)
Value function
-E[E 70 a1) 31 8, at (T))) ~ P()
Q-Value function
(s( = R(s, a) + Eg PBy (V*())
Advantage function
AT a) == (7, (s).
Advantage function
(s, Es) ()) (s
Arbitrary functions
I and are arbitrary functions used next.
Arbitrary distributions
and Ç are arbitrary probability distributions used next.
Hilbert inner product
(f.9)= == (z)
Kulback-Liebles (KL) divergence
DKL(Clv) = (C(x), log( ((t))) = [(xx) log( ((x) dr
u(ar)
Total Variation (TV) distance
TV(,,u) = [(((())) - ((x) 1),- = 2/5 ((x)) (x))ddr.
Coupling set
r(C,v) 1/) is the set of couplings for Ç and v.
Wasserstein distance
W(5, inf inf
Policy Wasserstein dis- tance Lipschitz Constant
W(m, == sup.es W (1 (-((s))
Li)(f(z,)):z) = sup, 1
Rubinstein- Kantrovich
- ((()))>(z), W(),v) Li)(f:z).
(RK) duality
Table The mathematical notations used throughout the paper.
13

---

Under review as a conference paper at ICLR 2021
Mathematical Notation
Name and Description
S
This the state space of the MDP.
A
This is the action space of the MDP.
Y
This is the discount factor of the MDP.
R: S X A - R
This is the reward function of the MDP.
1
This is the initial state distribution of the MDP over the state space.
A
A(F) the set of all probability distributions over the arbitrary set F (otherwise known the Credal set of F).
In general, π denotes the policy of the MDP. However, the output argument type could vary in the text. See the next lines.
Given state S, (T)) and (-1ss) denote the action distribution suggested by the policy n
S: + A(A)
In other words, a ~ (s) and T--ss).
For a deterministic policy Tda. the unique action a suggested by the policy given the state can be denoted by T()) specially. In other words, a (s).
Tiee : A
IT
IT is the set of all policies (i.e., Vn : क E TT).
P
In general, P denotes the transition dynamics model of the MDP. However, the input argument types could vary throughout the text. See the next lines for more clarification.
P:SxA+A00
Given a particular state s and action a, P(s, a) will be the next state distribution of the transition dynamics (i.e. s P(s, a) where s' denotes the next state after applying 5, a to the transition P).
P: A() X A A(8)
This is generalization of the transition dynamics to accept state distributions as input. In other words, P(us, a) := Ex [P(s, a)).
P:Sx ((A) + ASS)
This is generalization of the transition dynamics to accept action distributions as input. In other words, P(s, va) Eowr, [P(s, a)].
This is a generalization of the transition dynamics to accept state distribution and a policy as input. Given an arbitrary state distribution v, and policy π. and P(v,, x) will be the next state distribution given that the state is sampled from V4 and the action is sampled from the r(s) distribution.
P: A(S) I1 + A(S)
In other words, we have PV,,, T) = srup, am>m(s)
(P(s,
Pr: A(S) x II + A(S)
This is the (-step transition dynamics generalization. Given an arbitrary state distribution v, and a policy æ and non-negative in- teger t, one can define pd recursively as (V,, т) := V, and 1pt (V,, T) : (Fpt (v., T).
PT
The discounted visitation frequency Pi is a distribution over S, and can be defined as PM == (1 7) D (P,
Table 2: The MDP notations used throughout the paper.
14

---

Under review as a conference paper at ICLR 2021
A.2 BOUNDING W W(P (1, IPt. (p, m2))
To review, the dynamical smoothness assumptions were
( (1) =2)) S Lx- W ((T,)))
W ((P(H1, P(12, m)) S I.p W(11; P2).
The following lemma states that these two assumptions are equivalent to more concise assumption. This will be used bound the t-step visitation distance and prove Lemma A.2
Lemma A.I. Assumptions and (0) are equivalent to having
MI) m2)) S Ly. (Wu,, Lu (TI, T2).
(17)
Proof.
To prove the
5 6
direction, the triangle inequality for the Wasserstein distance
gives
W(P(u, m), P(22, m2)) s (P) m)) + m1)
(18)
and using 5 6 and 18 then implies
W (F
& L, W W(11 + La W W(T1,
(19)
The other direction is trivial.
Lemma A.2. Under Assumptions and o we have the bound
W(P' (n, II,, (H, (2)) La (1+ Ly 1. W W(T1, (TI1TT)) T2).
(20)
where TP (p, x) denotes the state distribution after running the MDP for time-steps with the initial state distribution μ. and policy
Proof. For / 1, the lemma is equivalent to Assumption (5) This paves the way for the lemma to be proved using induction. The hypothesis is
W W(F- (1, T Pt- (M, T2)) S La (1+ + Ly + .... ......44 +11 W (2),
(U,
(21)
and for the induction step write
W(P (H, mT), P (H. R2)) (PP(P--1(H (H, TT), P(P" 1 (1,TT) T2)).
(22)
Using Assumption 17). which according to Lemma A.I is equivalent to Assumptions 5 and (). we
can combine 21 and 22 into
W (H, PP x2)) L, W(T,, m2) + L, ((P''l(M, T1,, p"- (2, T2)). (23)
Thus, by applying the induction Hypothesis 21 we have
W (p, F T2)) S L W(Ti, T2) + Ly Lx (1 + Ly +...+ 11 (TI, (24)
which can be simplified into the lemma statement (i.c., Incquality (0)).
A.3 BOUNDING W (PT
Lemma A.2 suggests making the Tip < assumption and paves the way for Theorem A.4 The L, assumption is overly restrictive and unnecessary, but makes the rest of the proof casier to follow. This assumption can be relaxed by general transition dynamics stability assumption which is discussed in more detail later at section A.5.3 and an equivalent II assumption is introduced
to replace L < 1.
First, we need to introduce Lemma A.3 first, which will be used in the proof of Theorem A.4
Lemma A.3. The Wasserstein distance between linear combinations of distributions can be bounded as W μ1 (1 3) 21, u2 + - 33) 12) < 3 W WWi,,1) (1 3) W W(1,, v2).
Proof. Plugging 7 3- (H) + (1- B) 1) (on.))) in the Wasserstein definition yields the result. O
15

---

Under review as a conference paper at ICLR 2021
Theorem A.4. Assuming S 6 and yLp have the inequality
7 74ب
W :p
(TT1, T2).
(25)
Proof. Using Lemma A.3 and the definition Pj we can write
( P 3 (1-m)
( TI) (H, π2)).
(26)
t=
Using Lemma A.2 we can take another step relax the inequality (26) and write
W(P Pp-) S L -
( (TI, 22)
((>L)) - 7¹).
(27)
(Lμ -
Due to the y assumption, the right-hand summation in 27 is convergent. This leads us to
W La (1 - (1) (TI, T2)
(28)
(Jy -
Inequality (28) can then be simplified to give the result.
A.4
STEPS TO BOUND THE SECOND-ORDER TERM
The RK duality yields the following bound:
10 PT A (1. m2)) S W PPP) sup /V,A" (s, 12)122
(29)
To facilitate the further application of the RK duality, any advantage can be rewritten as the following inner product: AT (s, π2) (Te (a) mi (als), Q (s, a)).. Taking derivatives of both sides with respect to the state variable and applying the triangle inequality produces the bound
sup A.A** (s, 22) I sup [| (T2 mi(as))), Q" (s, (s.a))all
+ sup ([]((]]) Ti (a(ss) V.QT (s.
(30)
The second term of the RHS in 30 is compatible with the RK duality, However, the form of the first term does not warrant an easy application of RK. For this, we introduce Theorem A.5 Theorem A.S. Assuming the existence of Lip(Q"= (s, a);a) we have the bound
( (Ve(Te(a)s) - (a)s))) (=)
(31)
V. Te(aas') + (1(ss) x2(a s) + Ti (a calbs')
2 Lipp(("
2
2
Proof. By definition, we have
[VV. mi (als)))
(
(s, (s, a)),ll
(S)
(2 - (a)s)), Q (s, (s,a)))
(32)
For better insight, we will write the derivative using finite differences:
Osti (alss) QT
(s. (s.a))
lim 1
((2g(als + 8s ee))
-a(as + 8s e,)
)s)=))
5s
(((2((a)s)
(als))
(33)
16

---

Under review as a conference paper at ICLR 2021
We rearrange the finite difference terms to get
Ogli) TT (1)s))
(alls)))[([(() (s,
Qs, (s.
lim
(m12(aas ez))
(al(s
) ),)))),)))))
6--0 58
(T2((a)))
tal + 0,)
).
(34)
Equivalently, we can divide and multiply the inner products by factor of 2, to make the inner product arguments resemble mixture distributions:
Os(i T2 (a(a))) (als)), (s,
TT(aas + òs e₁) + क (als) Qm
lim 68-+0 5s
2
(Ta(als) + (a 8s ej)
QT (mT(
(35)
2
The RK duality can now be used to bound this difference as
(m((ass) ग. (als)), Q (s. ())
(36)
Os(i)
Ta(aas+ + 5s 05) TI (als) ma(als) aI malss + ōs ..)))) Lip(Q
lim 8a-40
W
which can be simplified as
K ( ( (Ta(als) (i(ass)) Q
(2(ass') + Tı (als) =2(a|s) + n (als')
2 2- Lip(Q" Os'Gs
W
(37)
2
2
Combining Incquality 337 and Equation (32) we obtain the bound in the theorem.
A.5 THE LIPSCHITZ CONTINUITY AND TRANSITION STABILITY ASSUMPTIONS
There three key groups of assumptions made in the derivation of our policy improvement lower bound. First is the existence of Qr-function Lipschitz constants. Second is the transition dynamics Lipschitz-continuity assumptions. Finally, make assumption about the stability of the transition dynamics. Next, we will discuss the meaning and the necessity of these assumptions in the same order.
A.5.1 ON THE EXISTENCE OF THE LIP(Q". a) CONSTANT
The Lip(Q*, a) constant may be undefined when either the reward function or the transition dynamics are discontinuous. Examples of known environments with undefined Lip(Q" a) constants include those with grazing contacts which define discontinuous transition dynamics. In practice, even for environments that do not satisfy Lipschitz continuity assumptions, there are mitigating factors; practical Qa functions reasonably harrow-bounde in small trust region neighborhood, and since we use non-vanishing exploration scales and trust regions, a bounded interpolation slope can still model the Q-function variation effectively. We should also note that a slightly stronger version of this assumption is frequently used in the context of Lipschitz MDPs (Pirotta et al.) 2015 Rachelson Lagoudakis 2010 Asadi et al. 2018). In practice, we have not found this be substantial limitation.
A.5.2 THE TRANSITION DYNAMICS LIPSCHITZ CONTINUITY ASSUMPTION
Assumptions 5 and of the main paper essentially represent the Lipschitz continuity assumptions of the transition dynamics with respect to actions and states, respectively. If the transition dynamics
17

---

Under review as a conference paper at ICLR 2021
and the policy are deterministic, then these assumptions are exactly equivalent the Lipschitz continuity assumptions. Assumptions and only generalize the Lipschitz continuity assumptions in distributional sense.
The necessity of these assumptions is a consequence of using metric measures for bounding errors. Traditional non-metric bounds force the use of full-support stochastic policies where all actions have non-zero probabilities (e.g., for the KL-divergence of two policies to be defined, TRPO needs to operate on full-support policies such as the Gaussian policies). In those analyses, since all policies share the same support, the next state distribution automatically becomes smooth and Lipschitz continuous with respect to the policy measure even if the transition dynamics was not originally smooth with respect to its imput actions. However, metric measures are also defined for policies of non-overlapping support. To be able to provide closeness bounds for future state visitations of two similar policies with non-overlapping support, it becomes necessary to assume that close-enough actions or states must be yielding close-enough next states. In fact, this is very common assumption in the framework of Lipschitz MDPs (See Section 2.2 of Rachelson & Lagoudakis (2010), Section 3 of Asadi et al. (2018), and Assumption of Pirotta et al. (2015)).
A.5.3 THE TRANSITION DYNAMICS STABILITY ASSUMPTION
Before moving to relax the 7h, assumption, we will make few definitions and restate the previous lemmas and theorems under these definitions. We define L pa. to be the infi- mum non-negative value that makes the equation ((P(p1, P(P,, T))
L W p2) hold. Similarly, 12p.. is defined the infimum non-negative value that makes the equation T. W (mz(-1)) (2(=(1²) hold. For notation brevity, we will also de- P(4,, x)) (P)(M,
note (. and (), by (() mm and (t) mp. respectively.
Under these definitions, Lemma A.I evolves into
W(P()), (1), 72)) S Lp W (11, P2) + Ly W(T1,
(38)
We can apply a time-point recursion to this lemma and have W(P(P' (1, m1), m1), P(P (H, #2), x2))
< (1.))))), W (MI, + (a)m W (PP (I), IP,, x2))
(39)
which can be notationally simplified to
W W (1, (I (2) <
M, ((1, +
()-1)
((t-1) p.mii W m1), (4,)) (p,)T)))
(40)
These modifications lead Lemma A.2 to be updated accordingly into
W(P' (H, (H, m.), IP' (p, T2)) S Lapaari W(T1,TT2)
(
(41)
where we have
(
II
(t--)) "patia
(42)
Lap.s
1=1
s=1
By a simple change of variables, we can have the equivalent definition of
(())
II s=k+1
i00
(43)
Now, we would replace the rL, < assumption with the following assumption.
The Transition Dynamics Stability Assumption: A transition dynamics P is called stable if and only if the induced {Lamm Jizo and ( sequences satisfy
Cr. = sup F2.1
(A) Liu.TiiT
sup MA 22
1)-1)
II
70 MaRiNN <
00.
(44)
1=k+1
To help understand which (L ma Jozo and ( Fezo sequences can satisfy this assumption, will provide some examples:
18

---

Under review as a conference paper at ICLR 2021
The vt ((0) assumption.
,2 eι and 1I i L m. 02 sequences violate the dynamics stability
The vt: : I Lam < 1 and vt: L La O sequences satisfy the dynamics stability
assumption.
- m guarantees the dynamics stability assumption.
sup;
V 2 to I mm guarantees the dynamics stability assumption no matter (1) how big
(8)
to is (as long as it is finite), or (2) how big the members of the finite set { [(()) { It < to are.
If the dynamics stability assumption holds with constant CL. one can define a L constant such that C2 T. Σt=g Ei=g( (TT,), Then, can replace all the T,, instances in the rest of the proof with the corresponding I constant, and the results will remain the same without any change of format.
The mm and i 1, constants can be thought as tighter versions of I and Lg, but with sufficient yet
dependency on TI, T22 μ and the time-point of application. Having د,μ, < unnecessary condition for this dynamics stability assumption to hold. Vaguely speaking, L, is an expansion rate for the state distribution distance; it tells you how much a divergence in the state distribution will expand after single application of the transition dynamics. Having effective expansion rates that are larger than one throughout an infinite horizon trajectory sign of the system instability; some change in the initial state's 's distribution could cause the observations to diverge exponentially. While controlling unstable systems an important and practical challenge, none of the existing reinforcement learning methods is capable of learning effective policies on such environments. Roughly speaking, having the dynamics stability assumption guarantees that the expansion rates cannot be consistently larger than one for infinite time steps.
A.6 CHOICE OF C2 AND C2
Since the TDPO algorithm operates using the metric Wasserstein distance, thinking about how normalizing actions and rewards affect the corresponding optimization objective builds insight into how to set these coefficients properly. Say we use the same dynamics, only the new actions are scaled up by factor of B and the rewards are scaled up by factor of a::
Onew B Gold
Fnew
Fold
(45)
If the policy function approximation class remained the same, the policy gradient would be scaled by factor of 3 (i.e., One regularization coefficient and trust region sizes can be obtained by
3 Do.e
Therefore, one can easily show that the corresponding new
Cnew
Cold
(46)
B²
and
gnew
3 8oll mmax.
(47)
We used equal regularization coefficients (i.c., C = C2 = C). and the process to choose them can be summarized follows: (1) Define C = 3600 a B-2 6max = 3//000 and og 3/60 (where og is the action disturbance parameter used for Vine)., (2) using prior knowledge or by trial and error determine appropriate action and reward normalization coefficients. The reward normalization coefficient a was sought to be approximately the average per-step discounted reward difference of a null policy and an optimal policy. We used reward scaling value of = 5 and an action scaling value of 3 = 5 for the non-locally rewarded pendulum and / 1.5 for the long-horizon legged robot. Both environments had per-step discounted reward of approximately -5 for a null policy and non-positive rewards, justifying the choice of a.
A.7 PROOF OF THROREM
We restate Theorem 4. below for reference and now prove it.
19

---

Under review as a conference paper at ICLR 2021
Theorem 4.1. Assume finite horizon MDP with both deterministic transition dynamics P and initial distribution Ja, with maximal horizon length of H. Define K H dim(,A), a uniform v, and q(s; o)) Ti (s) + de, in Algorithm 2 with e; being the jth basis element for A. the (j. tx) pairs are sampled to exactly cover (1,...,,iime (lim(AA) x (1....,H] H), then we have
lim Vn (T2) o
Vaa ma=mi
(48)
Proof. According to the advantage decomposition lemma, we have
m2 Tm
E smpp
V A (s, T2) m==!"
(49)
m
Due to the fact that the transition dynamics, policies лı and mι, and initial state distribution all deterministic, we can simplify Equation 449
to
#-1
m2
Uma m2
Σ
A (8t, T2) m2==11
(50)
t=
where 8t the state after applying the policy ni for time-steps. We can use the chain rule write
V TT 1 (8t, (2),T))
V A (st, 0t)
mm
x2=x)
dim(A)
0
n (5ts az) a:=a((.))))
(51)
aa,
To recap, Equations
50
50
and 51
can be summarized as
# 1
dim(AA)
V m2
A (st
(a) 82=81
(52)
daf
Under the assumption that the (j. pairs are sampled to exactly cover {1..... dim(A)) xf1,, f1...., H). we can simplify the Vine oracle to
1 H-1 dim((A)
(di((A) 7)
(T2(8)) - ==():)) =m((s))
A T1
(TT)
Σ 1=0
K
v(t) (q(st;io)) ((ss)) aI (st))
( (($()())))
(53)
From the definition, we have o) 1 (8t) = de, and (g(*;; (s:))) ((( i (s,)) d². Since is uniform (i.e., v(t) = #) and K = H dim(A), we can take the policy
gradient of Equation (3) and simplify it into
H-1 dim(A)
VZZ 7((0))) A (s,
V A T1 (22) m=
(54)
Since, AT (St, (st)) = O, we can write A (st, q(sri, o))
A (st, (8x) + toe,) - A (st, T1
lim a=0
lim
a-1
à
A (St, ar) aelaammeee
(55)
da
Also, by the definition of the gradient, can write
V V (TT - ग (st)) e, o.
(56)
Combining Equations 55 and (66). and applying them to Equation 54 yields
H (A
lim
T2
A s (st, at)
()))))))))
(57)
0 +0
1=1
Da
m2=81
Finally, the theorem can be obtained by comparing Equations (57 and 52
20

---

Under review as a conference paper at ICLR 2021
A.8 QUADRATIC MODELING OF POLICY SENSITIVITY REGULARIZATION
First, we will build insight into the nature of the
( N2: s) =W (T2(ss)), TI
(als)))
V.W W TT((als)) m (als) *2(als) x (a)s'')
(58)
2
2
term. is fairly obvious that
((2((a)s), TI (a(ss))) m =.
(59)
22 Mi. then the two distributions mu(als (ala)) and =a will be the same no matter what s' is. In other words.
T = 22 Vo' W Tz(als") + (als) T2(a[s) + x
O.
(60)
2
This means that
V W ma(als)) + (als) T2(a m (als')
0.
(61)
m2=m
The Taylor expansion the squared Wasserestein distance can be written as
W (1) 10,=0,+60 = 50 Hz00 + h.o.t..
(62)
2
Considering
60) and similar to the previous point, one can write the following Taylor expansion
(2(ass') + (als) T2(als) +
V W
500 H2 + h.o.t. (63)
2
2
s'=s 02201++60
According to above, LwG is the geometric mean of two functions of quadratic order. Although this makes LwG of quadratic order (i.e., limase-0 Ewalas8)
(wo((0) a² holds for any constant a), this does not guarantee that LwG is twice continuously differentiable w.r.t the policy parameters, and may have defined Hessian matrix (e.g., f(1 x22) |æ1_2| is of quadratic order, yet is twice differentiable). To avoid this issue, we compromise on the local model. Using the AM-GM inequality and for any arbitrary positive a, one can bound the LWG term into quadratic terms:
1 1 2
LWG(TI. 82:
WTIz(als)
TI (alss))
a²
a²
Ta(als') + mI (als) [2(ass) + क. (a(ss') V./V
V
W
(64)
2
Therefore, by defining
(0, T2is) = v W. W T2(a|s') + (a(ss) (2(ass) + (a(sss)
(65)
2
Cf a and C222 == (C2 + C
the new surrogate will have the twice-differentiabll form
C (T2) = 1-7
EE
(s,
Cy E Eanpy
(T1,
- -C
E. Eampi! (TT(aa))).m(al))))))
(66)
C₁ and C% will be the corresponding regularization coefficients for the surrogate defined in (13). Due to the arbitrary a used in bounding, no constrain governs the Cý and C/ coefficients. Therefore, C and C/ can be chosen without constraining each other.
21

---

Under review as a conference paper at ICLR 2021
A.9 IMPLEMENTATION DETAILS FOR THE ENVIRONMENT WITH NON-LOCAL REWARDS
We used the stable-baseling implementation Hiill et al. 2018), which has the same structure as the original OpenAl baselines (Dhariwal al. 2017) implementation. We used the "ppol" variant since no hardware acceleration was necessary for automatic differentiation and MPI parallelization was practically efficient. TDPO, TRPO, and PPO used the same function approximation architecture with two hidden layers, 64 units in each layer, and the tanh activation. TRPO, PPO, DDPG, and TD3 used their default TDPO, and multiplied the outputs of the network by factor of 0.001 that the initial actions were small and nearly zero. We also confirmed that TDPO does not decrease in performance when using an identical network initialization PPO/TRPO. TD3's baseline implementation was amended to support MPI parallelization just like TRPO, PPO, and DDPG. To produce the results for DDPG and TD3, we used hyperparameter optimization both with and without the lanh final activation function that is common for DDPG and TD3 (this causes the difference in initial payoff in the figures). However, under no conditions were DDPG and TD3 able to solve these environments effectively, suggesting that the deterministic search used by TDPO is operating in a qualitatively different way than the stochastic policy optimization used by DDPG and TD3. Note that we made a thorough attempt to compare DDPG and TD3 fairly, including trying different initializations, different final layer scalings/activations. different network architectures, and performing hyperparameter optimization. Mini-batch selection was unnecessary for TDPO since optimization for samples generated by De Vine was fully tractable. The confidence intervals in all figures were generated using 1000 samples of the statistics of interest.
settings. We used Xavier initialization (Glorot & Bengio 2010) for
For designing the environment, we used Dhariwal et al. 20177's pendulum dynamics and relaxed the torque thresholds to be large as 40 N m. The environment also had the same episode length of 200 time-steps. We used the reward function described by the following equations:
R(se, a+) = Cn R(T)- [ = 200)
R(T) Riree (7) Romsee + RAmp(? (T)
fanax
Rireg = 0.1
Σ f
std (f)²
Smin
( 0)
-
-1(210 200
199
Rottse(F)) (T)
Baager Offset
0 Parget Offset
200
AC
RAMp (T) hptocowise Orarget Amp.
(67)
where
0 the pendulum angle signal in the time domain.
+ the magnitude of the Fourier transform of 6.
the same as only for the positive frequency components.
Ac is the normalized oscillatory spectrum of O:
Vo+To+
A
(68)
200
hpiecewise is piece-wise linear error penalization function:
Npiecewise (x) = x 1(x 2 0} + 10- xr 1(-x 0).
(69)
+ is the standardized positive amplitudes vector:
(70)
std
V 10-6
10-6
22

---

Under review as a conference paper at ICLR 2021
CR x 10 is reward normalization coefficient, and was chosen to yield approximately the same payoff as null policy would yield in the typical pendulum environment of Dhariwal ct al. (2017).
Target Offict is the target offset, Target Amp. is the target amplitude, and [fuin fmax] is the target frequency range of the environment.
All methods used 48 parallel workers. The machines used Xeon E5-2690-v3 processors and 256 GB of memory. Each experiment was repeated 25 times for each method, and each run was given 6 hours or 500 million samples to finish.
A.10 IMPLEMENTATION DETAILS FOR THE ENVIRONMENT WITH LONG HORIZON AND RESONANT FREQUENCIES
For the robotic leg, we used exactly the same algorithms with the same parameters described in Section above.
We used the reward function described by the following equations:
R
Rposture
+ Ryelocity + Rioot offset + Rioot height + Rground force + Rkane beight + Roa-air torques
(71)
with
Rposture - Oxnce +
+ Onip
Riyeocity 0.08 ||wknood + |whipp]
Ricct offset
x [Irinal 1 {{Enee < 0.23]
Reround force
- [lfe - -mgl 1lfz < mg} lochhowwn
Rioor height
- [|2tool] 1rouchdown]
Pxnoe height - 15 x [| [/ "knee
target
1souchdown
Ron-air torques -10 [[Tince + Thip) (1 touchdown)
(72)
where
Unce and Onip are the knee and hip angles in radians, respectively.
Wince and Whip are the knee and hip angular velocities in radians per second, respectively.
Ticat and Aiou the horizontal and vertical foot offsets in meters from the desired standing point on the ground respectively.
Tkace and Znne are the horizontal and vertical knee offsets meters from the desired standing point on the ground, respectively.
fa is the vertical ground reaction force on the robot in Newtons.
is the robot weight in kilograms (i.e., 0.76 kg).
is the gravitational acceleration in meters per second squared.
1 wuchhown is the indicator function of whether the robot has ever touched the ground.
rarget is target knee height of 0.1
Anee
Knne and Thip are the knee and hip torques in Newton meters, respectively.
All methods used 72 full trajectories between each policy update, and each run was given 16 hours of wall time, which corresponded to almost 500 million samples. This experiment was repeated 75 times for each method. The empirical mean of the discounted payoff values were reported without any performance or seed filtration. The same hardware as the non-local rewards experiments (i.e., Xeon E5--269-vv processors and 256 GB of memory) used.
23

---

Under review as a conference paper at ICLR 2021
A.11 GYM SUITE BENCHMARKS
While it is clear that our deterministic policy gradient performs well on the new control environments consider, one may naturally wonder about its performance existing RL control benchmarks. We ran our method on a suite of Gym environments and include four representative examples in Figure 3 Broadly speaking, our method (TDPO) performs slightly worse on average than others, but occasionally performs much better as seen in the Swimmer-v3 environment. We speculate that these gym environments are reasonably robust to injected noise, and this may mean that stochastic policy gradients can more rapidly and efficiently explore the policy space, or there may be other algorithmic enhancements that are needed for fully deterministic policy gradients these cases. The experiments granted each method 72 parallel MPI workers for about 144 million steps (i.e., million sequential steps), and the returns were averaged over 100 different seeds for each method. Since the computational cost of running both DDPG and TD3 were high, we only included TD3 since it was shown to outperform DDPG in earlier benchmarks.
InverteedDoubleeenddlunnyy
Swimmer-v3
200
8000
Episode Return
150
6000
100
4000
2000
0-
HalfCheetah-v3
Reacher-v2
88
6000
Episode Return
24
4000
332
40
TDPO (Ours)
2000
448
TRPO
56
TD3
64
25M
50M
75M
100M
125M
150M
25M
50M
75M
100M
125M 150M
Sample Count
Sample Count
Figure 3: Results for the gym suite benchmarks.
A.12 RUNNING TIME COMPARISON
Figure depicts a comparison of each method's running time per million steps. These plots show the combination of both the simulation (i.e., environment sampling) and the optimization (i.e., computing the policy gradient and running the conjugate gradient solver) time. It is clear that our method (TDPO) generally faster than the other algorithms. This mainly due the computational efficiency the Vine gradient estimator, which summarizes two full trajectories in a single state-action action-advantage advantage tuple which can significantly reduce the optimization time. That being said, these relative comparisons could vary to large extent (1) under different processor architectures, (2) with more (or less) efficient implementations, or (3) when running environments whose simulation time constitutes significantly larger (or smaller) portion of the total running time.
A.1 OTHER SWINGING PENDULUM VARIANTS
Multiple variants of the pendulum with non-local rewards were used, each with different frequency targets and the same reward structure. Table summarizes the target characteristics of each variant.
24

---

Under review as a conference paper at ICLR 2021
Non-local Pendulum
Robotic Leg
HalfCheetah-v3
Seconds Millonn Steps
200
300
300
150
200
200
100
100
TDPO
TRPO
PPO
DDPG
TD3
TDPO
TRPD
PPO
DDPG
TD3
TDPO
TRPO
PPO
TD3
Inverte@Doubleeendduuuw
Swimmer-v3
Reacher-v2
Seconds Million Steps
200
200
200
150
100
100
100
50
50
50
TDPO
TRPO
PPO
TD3
TDPO
TRPO
PPO
TD3
TDPO
TRPO
PPO
TD3
Figure 4: Training time comparison in different environments. The lower the bar, the faster the method. The vertical axis shows the time in seconds needed to consume million state-action pairs for training. Each environment was shown separately in a different subplot.
The main variant was shown in the paper. Figures 5 6 for the second to ninth variants. To focus on our method's ability to solve all these variants efficiently, we only show the performance of our method (TDPO) on all variants in Figure 13 Overall, we found TRPO, PPO, DDPG, and TD3 to occasionally find the correct offset. They either excited the natural the maximum (not the desired) frequency of the pendulum, but they were not able to drive the desired frequency and amplitude. TDPO was able to achieve the desired oscillations (and thus high rewards) in all variants.
and 12 show similar results
Pendulum Variant
Main Desired Frequency
Desired Offset Desired Amplitude
1.7-2 Hz
0.524 rad
0.28 rad
Second
0.5.-0.7 Hz
1.571 rad
1.11 rad
Third
2.5-3 Hz
0.524 rad
0.28 rad
Fourth
2.-2.4 Hz
0.785 rad
0.28 rad
Fifth
2.-2.4 Hz
1.571 rad
0.74 rad
Sixth
2-2.4 Hz
0.524 rad
0.28 rad
Seventh
2.-2.4 Hz
1.047 rad
0.28 rad
Eighth
2-2.4 Hz
0.785 rad
0.74 rad
Ninth
2.-2.4 Hz
1.309 rad
0.28 rad
Table The target oscillation characteristics used define different pendulum swinging environ- ments.
A.14 NOTES ON How TO IMPLEMENT TDPO
In short, our method (TDPO) structured the same way TRPO was structured; both TDPO and TRPO use policy gradient estimation, and conjugate-gradion product machinery. On the other hand, there are some algorithmic differences that distinguish TDPO from TRPO. First of all, TRPO uses line-search heuristics to adaptively find the update scale; no such heuristics are applied in TDPO. Second, TDPO uses the De advantage estimator, which requires storing and reloading pseudo-random generator states. Finally, the Hessian-vece product machinery used in TDPO computes Wasserstein vector products, which is slightly different from those used in TRPO. The hyper-parameter settings and notes how choose them discussed
solver utilizing a Hessian-vector
25

---

Under review as a conference paper at ICLR 2021
TDPO (ours)
Disc. Payoff
-10
TD3
-00
TRPO
DDPG
-104
PPO
100M
200M
300M
400M
500M
Sample Count
TDPO
TRPO
PPO
DDPG
TD3
Angle (rao)
2.0
1.0
10
Time (s)
Time (5)
Time (9)
Time (s)
Time (9)
Amplitude (rao)
1.5
Desired Offset
Desired Offiet
Desired Offset
Desired Offset
Desired Offset
1.0
Desired
Desired Frequency
0.5
Frequency
Frequency Amplitudo
Frequency Amplitude
Frequency
Amplitude
Amplltude
Amplitudo
0.0
Frequency (H2)
Frequency
(Hz)
Frequency (H2)
Frequency (H2)
Frequency (H2)
Figure 5: Results for the second variant of the simple pendulum with non-local rewards. Upper panel: training curves with empirical discounted payoffs. Lower panels: trajectories both the time domain and frequency domain, showing target values of oscillation frequency, amplitude, and offset.
-101
(ours)
Disc Payoff
-10
TD3
TRPO
DDPG
-104
PPO
100M
200M
300M
400M
500M
Sample Count
TDPO
TRPO
PPO
DDPG
TD3
(tad)
www.ae
10
10
5 Time (s)
5
10
Time
Time (s)
Time (5)
Time
Amplituse (rad)
Desired Offect
Desired Officet
Desired Offscct
+ Desired Offset
x Desired Offsct
0.4
Desired Frequency
Desiredi
Desired Frequency
Desired Frequency
Desired Frequency & Amplitude
Frequency
0.2
& Amplitude
Amplitude
Amplitude
& Amplitude
0.0
10
Frequency (H2)
Frequency
(H2)
Frequency (H2)
Frequency (H2)
Frequency (H2)
Figure 6: Results for the third variant of the simple pendulum with non-local rewards. Upper panel: training curves with empirical discounted payoffs. Lower panels: trajectories both the time domain and frequency domain, showing target values of oscillation frequency, amplitude, and offset.
26

---

Under review as a conference paper at ICLR 2021
-10-
TDPO (ours)
Disc. Payoff
-10
TD3
-10
TRPO
DDPG
-10
PPO
100M
200M
300M
400M
500M
Sample Count
TDPO
TRPO
PPO
DDPG
TD3
1.0
(rad)
0.5
0.0
-0.5
Time (5)
Time (s)
Time (5)
Time
Time (s)
Ampltude (rad)
0.8
Desired Offset
Desired Offset
* Desired Offset
# Desired Offistt
oe Desired Offset
0.5
Desired Amplitudo
Desired
Desired
Desired Frequency & Amplitude
Desired Frequency & Ampiltudp
0.2
& Amplitudoo
& Ampiitude
0.0
Frequency (Hz)
Frequency
(Hz)
Frequancy (Hz)
Frequency (Hz)
Frequency (Hz)
Figure 7: Results for the fourth variant of the simple pendulum with non-local rewards. Upper panel: training curves with empirical discounted payoffs. Lower panels: trajectories both the time domain and frequency domain, showing target values of oscillation frequency, amplitude, and offset.
-10-
TDPO (ours)
Disc. Payoff
-10?
TD3
-10
-10
DDPG
PPO
100M
200M
300M
400M
500M
Sample Count
TDPO
TRPO
PPO
DDPG
TD3
(rad)
2.0
10
10
10
0
5
10
100
Time
Time (5)
Time (5)
Time (s)
Time (S)
Destred
Desired Olfset
+ Destred Office
+ Destred affset
Destred
Ampltude
Desired Frequency
Destred
Desired Frequency
Desired
Desired
0.5
Frequency
Frequency
Frequency
Amplitude
Amplitude
Amplitude
Amplltude
Amplitude
0.0
10
Frequency (H2)
Frequency
(Hz)
Frequency (H2)
Frequency (H2)
Frequency (H2)
Figure 8: Results for the fifth variant of the simple pendulum with non-local rewards. Upper panel: training curves with empirical discounted payoffs. Lower panels: trajectories both the time domain and frequency domain, showing target values of oscillation frequency, amplitude, and offset.
27

---

Under review as a conference paper at ICLR 2021
-10
(ours)
Disc. Payoff
-10
TD3
-10
TRPO
DDPG
-10"
PPO
100M
200M
300M
400M
500M
Sample Count
TDPO
TRPO
PPO
DDPG
TD3
10
Time (s)
Time (s)
Time (s)
Time (s)
Timo (5)
Desired Frequency
Destrod
Desired
Desired Frequency & Amplltude
Desirad
Amglitudo
Fraquency & Amplitude
Frequency Amplitude
Frequency
0.2
& Amplitude
& Amplitude
0.0
10
Frequency (Hz)
Frequency
(Hz)
Frequency (Hz)
Frequency (H2)
Frequency (Hz)
Figure 9: Results for the sixth variant of the simple pendulum with non-local rewards. Upper panel: training curves with empirical discounted payoffs. Lower panels: trajectories both the time domain and frequency domain, showing target values of oscillation frequency, amplitude, and offset.
TDPO (ours)
-102
Disc. Payoff
TD3
-10
TRPO
-104
DDPG
PPO
100M
200M
300M
400M
500M
Sample Count
TDPO
TRPO
PPO
DDPG
TD3
Angle (rad)
WWW
0.0
10
10
10
10
10
(s)
Ampitudd (rad]
Destred Offset
+ Desired Gifsst
+ Desired Offset
- Desired Offset
Desired Offset
Desired Frequency & Amplitude
iredi Frequency
Desired
Desired Frequency
Desired Frequency & Amplitude
Frequency
Amplitude
Amplitude
Amplitude
10
Frequency (Hz)
Frequency (Hz)
Frequency (Hz)
Frequency (Hz)
Frequency (Hz)
Figure 10: Results for the seventh variant of the simple pendulum with non-local rewards. Upper panel: training curves with empirical discounted payoff's. Lower panels: trajectories both the time domain and frequency domain, showing target values of oscillation frequency, amplitude, and offset.
28

---

Under review as a conference paper at ICLR 2021
-10
Payoff
-10
(ours)
-00
TD3
-10
TRPO
DDPG
-10-
PPO
-10
100M
200M
300M
400M
500M
Sample Count
TDPO
TRPO
PPO
DDPG
TD3
(rad)
10
Time
Time (5)
Time (3)
Time (s)
Time (9)
Ampliuue (rao)
0.8
Desired Offset
Desired Offset
Desired Orfset
Desired Offset
Desired Offsct
0.5
Desired Frequency
Des iredi
Desired
Desired Frequency & Amplitude
Desired Frequency & Amplltudo
Frequency
Frequency Amplitude
0.2
& Ampiliude
Amplitude
0.0
5
Frequency (H2)
Frequency (H2)
Frequency (H2)
Frequency (H2)
Frequency (H2)
Figure 11: Results for the eighth variant of the simple pendulum with non-local rewards. Upper panel: training curves with empirical discounted payoffs. Lower panels: trajectories both the time domain and frequency domain, showing target values of oscillation frequency, amplitude, and offset.
TDPO (ours)
-10
Disc. Payoff
TD3
-10
PPO
100M
200M
300M
400M
Sample Count
TDPO
TRPO
PPO
DDPG
TD3
Angle (tad)
1.0
0.5
10
10
5
10
Time
Time (5)
Time (5)
Time (5)
Time
Amplitude (rad)
Desired Offset
Desired Oifset
+ Desired Offset
+ Desired Offset
+
Desired Offset
1.0
Desired Frequency
Destred
Desired
Desired
Desired Frequency
0.5
Fraquency
Frequency
Amplitude
Amplitude
Amplitude
Amplitudi
0.0
10
Frequency (H2)
Frequency
(Hz)
Frequency (H2)
Frequency (H2)
Frequency (H2)
Figure 12: Results for the ninth variant of the simple pendulum with non-local rewards. Upper panel: training curves with empirical discounted payoffs. Lower panels: trajectories both the time domain and frequency domain, showing target values of oscillation frequency, amplitude, and offset.
29

---

Under review as a conference paper at ICLR 2021
A.6 and A.10 We will describe how to implement TDPO, and focus the subtle differences between TDPO and TRPO next.
in Sections A.9
As for the state-reset capability, our algorithm does not require access to reset function arbitrary states. Instead, we only require to be able to start from the prior trajectory's ' initial state. Many environments, including the Gym environments, instantiate their pseudo-random generators and only utilize that pseudo-random generator for all randomized operations. This facilitates a straightforward implementation of the DeVine oracle; in such environments, implementing an arbitrary state-reset functionality is unnecessary, and only reloading the pseudo-randon generator to its configuration prior to the trajectory would suffice. In other words, the DeVine oracle can store the initial configuration of the eseudo-random generator before asking for a trajectory reset, and then start sampling. Once the main trajectory is finished, the pseudo-random generator can be reloaded, thus producing the same initial state upon a reset request. Other time-step states can then be recovered by applying the same preceeding action sequence.
To optimize the quadratic surrogate, the conjugate gradient solver was used. Implementing the conjugate gradient algorithm is fairly straightforward, and is already included in many common automatic differentiation libraries. The conjugate gradient solver is perfect for situations where (1) the Hessian matrix is larger than can efficiently be stored in the memory, and (2) the Hessian matrix includes many nearly identical eigenvalues. Both of these conditions apply well for TDPO, well as for TRPO. Instead of requiring the full Hessian matrix to be stored, the conjugate gradient solver only requires Hessian-vector product machinery for TDPO. Our surrogate function can be viewed as
HU, which must be specifically implemented
C22 (() = --" 80 + 56 H80
2
where the Hessian matrix can be defined
Ci
H = H2 + C2
H1,
H1 = Vi siptk
Cou(T' m):s)
H2 = V R snp
((r (())))))
(73)
In order to construct a Hessian-vector product machinery e - Hv, one can design an automatic- differentiation procedure that returns the Hessian-vector product. Many nutomatie-diffrrentitton packages already include functionalities that can provide a Hessian-vector product machinery of a given scalar loss function without computing the Hessian matrix. This can be used to implement the Hessian-vecto product machinery in straightforward manner; one only needs to provide the scalar quadratic terms of our surrogate, and would obtain the Hessian-vector product machinery in return. On the other hand, this may not be the most computationally efficient approach, as our problem exhibits a more specific structure. Alternatively, one can implement a more elaborate and specifically designed Hessian- vector product machinery by following these three steps:
Compute the Wasserstein-veetee product e - H2v according to Algorithm 3
Compute the Sensitivity-veetto product - Hiv according Algorithm
Return the weighted sum of Hi and H2u as the final Hessian-vector product Hv.
One may also need to add a conjugate gradient damping to the conjugate gradient solver (i.e., return Be + He for some small / as opposed to returning He purely), which is also done in the TRPO method. This may be important when the number of policy parameters is much larger than the sample size. Setting 3 0 may yield poor numerical stability if H had small cigenvalues, and setting large will cause the conjugate gradient optimizer to mimic the gradient descent optimizer by making updates in the same direction as the gradient. The optimal conjugate gradient damping may depend such as the sample size. However, it can easily be picked
on the problem and other to be small value that ensures numerical stability.
Once the conjugate gradient solver returned the optimal update direction H 9. it must be scaled down by factor of C222 (i.e., 50" gC)). If 50* satisfied the trust region criterion (i.e.,
30

---

Under review as a conference paper at ICLR 2021
500" H88* I Smax), then to the next iteration. Otherwise, the proposed update 50" must be scaled down further, namely by a, such that the trust region condition would be satisfied (i.e., {(000)) (a800) 5max) before making the update new = fota + a50"
make the parameter update (i.e., Inew = 6 58") and proceed
Algorithm 3 Wasserstein- ProtuutProduct Machinery
Require: Current Policy л1 with parameters 01. Require: The vector with the same dimensions as 01.
Require:
An observation 8.
1: Compute the action for the observation with |A| elements.
(8)
d|
(74)
(IAD
This vector should be capable of propagating gradients back to the policy parameters when used in automatic differentiation software.
2: Define / to have be constant vector with the same shape as It could be populated with any values such as ones.
3: Define the scalar ă := t.'t.
4: Using back prock-rrppgg,onn, find the gradient
IA
Σ
00;
Σ
Be 891
Do. 88
(75)
5: Compute the following dot-product:
[A]
|
da, 00,
Das
(Váā, r))
01 .....
v-el-
(76)
(0)
1=1
1=1
6: Using automatic differentiation, take the gradient w.r.. the t vector.
361
01 + ... +
00
elel
da. 301
89
āø,u V. v)
(77)
80 Al
80
du Al
091
U1 +
a6
Viel
00:
09
7: Compute the dot product (āg, a).
8: Using pockppropagation, take the gradient w.r.t 0, and return it the gain-vector-producct
Das
Oas 1e
De: 30,
Da
39
V Va a)
(78)
dayal 30,
0u D9
0ajai '86,
0u A1 De
31

---

Under review as a conference paper at ICLR 2021
Algorithm 4 Sensitivity- Vectirr Producc Machinery
Require: Current Policy with parameters 01.
Require: The vector with the same dimensions as 01- An observation s.
Require:
Compute the action to observation Jacobian matrix
(x)))()
(s)
JAAX(S =
(79)
(x (s)
On (r)
Ba(3)
8al
This either be done using finite-differences in the observation using
(x)
x (1 + ds d))))))) (s)
(80)
as())
ds
(which may be a bit numerically inaccurate). or using automatic differentiation. In any case, this matrix should be parameter tensor capable of propagating gradients back to the parameters when used automatic differentiation software.
2: Define J to be the vectorized (i.e., reshaped into column) J matrix, with |SS|
|A|
|S]
rows and one column.
3:
Define & have be constant vector with the same shape as J. It could be populated with any values such as all ones.
4: Define the scalar J, := Jtt.
5: Using back pack-propagation, find the gradient
|AI |s|
|
S
VJJ.
VoJr.
tJ
OJii 801
0J.
(81)
i=1==1
j=1
6: Compute the following dot-product.
|
IS
aJi 00,
|4|
SS|
OJi.
(VoJ,,
tis
01
+
x vie
(82)
00
1=1
7: Using automatic differentiation, take the gradient w.r.t. the vector.
8J1.1 08,
8J..1
+ .....
86 esl
vle)
(VoJ)c = u)
BJ 1J 001
aJ AL
U1
08
(L.)
(1.)
091
88 m
(83)
(A
801
89
8: Reshape (VoJ)e into column vector and name it Jo..-
9: Compute the dot product JJ.
10:
Using back nackppropagatio, take the gradient w.r.t. 6, and return it as the gain-vector-
aj(i. 86,
T
aJa..
09,
00
J)
(84)
0. A1,8D
88,
89
09:
00
32

---

Under review as a conference paper at ICLR 2021
Main Variant
Second Variant
Third Variant
2.0
Angle
1.0
V
000000000000
0.0
10
10
Time
Time
Time (s)
1.5
Desired Offset
Desired Offset
Desired Offset
Ampltude
1.0
Desired
Desired
0.5
Frequency
Frequency
Frequency Amplitude
& Amplitude
Amplitude
0.0
10
10
10
Frequency (Hz)
Frequency (Hz)
Frequency (Hz)
(a)
Fourth Variant
Fifth Variant
Sixth Variant
MMM00000
2.0
Angle
1.0
VVVVMMMV
10
10
2
Time (s)
Time (s)
Time (s)
1.5
Desired Offset
Desired Offset
Desired Offset
Amplttude
1.0
Desired
Desired
Desired
0.5
Frequency
Frequency
Frequency
& Amplitude
Amplitude
Amplitude
0.0
10
10
10
Frequency (Hz)
Frequency (H2)
Frequency (Hz)
(b)
Seventh Variant
Eighth Variant
Ninth Variant
1.5
1.0
WW000000
0.5
0.0
100
10
10
Time (s)
Time (4)
Time (s)
Amplitude (rad)
Desired Offset
Desired Offset
Desired Offset
1.0
Desired
Desired Frequency & Amplitude
Desired
Frequency
Frequency
Amplitude
Amplitude
0.0
2
10
2
10
10
Frequency (Hz)
Frequency (Hz)
Frequency (Hz)
(c)
Figure 13: Time and frequency domain trajectories for method (TDPO) multiple variants of the simple pendulum with non-local rewards. (a) The high-reward trajectories for the first group of variants, (b) the high-reward trajectories for the second group of variants, (c) the high-reward trajectories for the third group of variants. Target values of oscillation frequency, amplitude, and offset were annotated in the frequency domain plots.
33


Reviews:
[{"review_id": "bEE-dtUa3X8", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper builds on the template of TRPO and proposes an algorithm that searches for deterministic policy. A novel policy improvement lemma is derived for Wasserstein distance and thus a new algorithm is proposed based on iteratively optimizing policy performance lower bound.\n\nI feel overall this paper may not be quite well motivated.  When describing the disadvantage of stochastic policies, it is better to make it clear. It is hard to get why stochastic policies are not able to handle non-local rewards, long time horizons, and naturally-resonant environments, in what aspects? What exactly means non-local rewards? Why TRPO can not handle long time horizons? There is no citation for \"exploratory noise injection\"? Where does the noise be injected? Many terms used here may lack of precise definitions. Overall I suggest the authors to make a clear comparison with TRPO that in what aspect you can improve TRPO. This is better to present the novelty of current works. Some improvements on MuJoCo over TRPO might not be sufficiently enough since RL algorithms involve too much tricks to make the comparison completely fair.\n\nIn (9), the theoretical lower bound to be optimized involves C_1 and C_2 that has a very complicated form. For the practical algorithm, I feel you just treat them as a tuning parameter when translating the regularized form to constrained form. Will it be problematic? In Section 5.1, the modification of reward function for the pendulum is a bit artificial. Anyone did this before or what's the reason to do that? Again, this may correspond the motivation of the proposed algorithm.\n\nThe presentation of Section 2.1 may not be in an optimal way. This should be a main result but several key derivations (Inequality (31) and Theorems A.5 and A.4) are deferred to the appendix. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "PNigW_6JAe", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new deterministic policy gradient method (TDPO). The main idea and its derivations are based on the use of a deterministic Vine (DeVine) gradient estimator and the Wasserstein metric. The paper shows that a closed-form computation of Wasserstein distance can be derived without any approximation on the deterministic policies. A monotonic policy improvement guarantee is provided. Deterministic Vine (DeVine) is proposed as an optimization method for the proposed surrogate loss function, which is also inspired by the TRPO's surrogate loss. The proposed method is evaluated on two new robotic control domains that are proposed by the authors. The comparisons are against PPO, TRPO, DDPG, and TD3. Additional experiment results on Open AI Gym environments are provided in Appendix.\n\nOverall, the studied idea is interesting. The finding of a closed-form Wasserstein distance on deterministic policies is useful. The new algorithm might be of interest to the RL community. Here are some of my main concerns regarding to this idea and the quality of the paper.\n\n- The motivation of using the first experiment domain with non-local rewards is not clear on why this can other approaches have challenges? Why it could be a showcase of the use of a deterministic policy gradient method. In addition, as DDPG is an off-policy and deterministic policy gradient method, I was wondering why it does not perform well. More detailed experiment settings will be helpful.\n\n- The second domain is basically a Mujoco task with a long-horizon. The results show it favors a deterministic method like TDPO. The same question on why DDPG and TD3 do performs well. They are off-policy methods that can be considered more sample-efficient than on-policy counter-parts like PPO/TRPO. On the other hand, much recent work show TRPO and PPO would perform comparably [1]. Therefore, a detailed experiment setting and a discussion on how each method's hypeparameters get tuned would be important.\n\n- It is also questionable on why the TDPO does not perform well in comparisons to the baseline on Gym suite environments. Because those Gym domains are also based on Mujoco and programmed with deterministic dynamics.\n\n\n[1] Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, Aleksander Madry:\nImplementation Matters in Deep RL: A Case Study on PPO and TRPO. ICLR 2020", "extra_scores": {}, "extra_text": {}}, {"review_id": "ACGXxCPKtbT", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a policy gradient method based on deterministic policies and deterministic gradient estimates. By assuming such a deterministic setting, the authors show that the proposed technique can estimate gradients on long-horizon tasks without the need to inject noise into the system for exploration. The authors propose using a Wasserstein-based model to perform regularization in this particular setting involving deterministic policies. They also establish the conditions under which it is possible to guarantee monotonic policy improvement.\n\nOverall, this is a well-written paper with sound mathematical arguments. The authors present a convincing review of related work and argue that the proposed method is better-suited than existing techniques when dealing with problems where stochastic exploration is not efficient or feasible. The authors formally show that it is possible to lower bound the payoff improvement, which is a non-trivial result. All update equations are carefully derived and discussed in the appendix. \n\nThe experiments seem to show that the method works well in a possibly restricted set of settings, involving either control tasks with resonant frequencies or problems with rewards associated with achieving oscillatory behaviors. The experiments on these particular handcrafted problems are well-designed and demonstrate that the proposed method (TDPO) achieves better performance than its competitors. However, the proposed method generally performs worse (on average) than others when evaluated on standard control benchmarks. Even though the experiments may suggest that the method is not widely applicable, I believe that the underlying theory is, on its own, an interesting contribution. \n\nI believe that this paper introduces a relevant contribution to the RL community that is concerned with scaling up learning to long-horizon problems. It presents a principled method and introduces non-trivial bounds. In my opinion, this conference's community would benefit from having this paper accepted to its proceedings.\n\n\nI have a few questions for the authors:\n\n1) in the introduction, the authors discuss rewards defined in the frequency domain. Could you please clarify what those are, in terms of the corresponding structure that they may imply in the reward function?\n\n2) the authors also argue that the proposed method is robust in control environments with resonant frequencies. Could you formally define what this means, and what are the implications of this assumption when deploying this method in more general MDPs which may not be modeling robotics problems?\n\n3) in the Background section, P is defined as DeltaS x DeltaA -> DeltaS. Shouldn't it be SxA -> DeltaS, instead?\n\n4) in the Background section, there is an issue with the sentence that begins with \"We may abuse notation and replace (...)\". I believe that the terms mentioned here should have been delta_s, delta_a, s, a.\n\n5) notation: P is sometimes used to refer to the transition dynamics, but sometimes it refers to the probability of an event. This isn't very clear. See, e.g., the reuse of this symbol when defining P(mu_s, pi).\n\n6) what is P(mu_s, pi(. | s~mu_s))? Is this the joint distribution over mu_s and pi? If \"P\" denotes a probability, I'm confused: the terms used at this point do not refer to random variables or events, but to distributions. Could you please improve the notation used when defining such a generalized dynamics?\n\n7) you assume the existence of Lip(Q^pi; a). Could you please discuss a few practical problems (with continuous actions) where the Q-function does not vary smoothly with actions? A discussion on the practical implications of this assumption would be interesting.\n\n8) what is the intuition behind Eqs. 5 and 6? What do they mean, intuitively, and how do they relate to the goal of regularizing deterministic policies, and to the goal of ensuring monotonic policy improvement? These seem to be key assumptions to the method, but (in my opinion) they were not well motivated or discussed.\n\n9) in this paper, your focus is on robotic environments with state-reset capabilities. This might be an unrealistic assumption, particularly in robotics problems where the goal is precisely to autonomously learn how to reach particular states or configuration. This assumption might also be reflecting deeper requirements; e.g., that the tasks being tackled are not goal-achieving tasks. Could you please discuss this assumption and its implications in more detail?\n\n10) in practice, how do you pick the coefficients C_1 and C_2 (Eq.13)? How sensitive is your method to this choice?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4q5owdtWwG", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a deterministic policy gradient method using the Wasserstein distance to quantify the difference of deterministic policies. It also introduces a new deterministic estimator for the policy gradient with theoretical justification. The paper uses a similar path as TRPO to first introduce a surrogate objective then replace the complex term by a tractable term for practical performance. The method has been shown to work well under long-horizon and non-local reward environments whereas existing methods struggle to solve.\n\nStrength: \n- Being able to deal with non-traditional RL settings can be beneficial to the RL community.\n- The adaptation of Wasserstein distance into the surrogate objective of TRPO is new.\n- Theoretical justification for the main results.\n- Numerical experiments show advantage of the proposed method in non-local or long-horizon settings.\n\nWeakness:\n- The presentation of background section can be improved as it is too dense with different notations. It may be a good idea to summarize these notations into a table.\n- Only consider 2 environments: 1 with non-local reward and 1 with long horizon and resonant frequencies.\n\nI have the following questions and comments:\nQ1. What is the motivation of changing frequency of pendulum to 1.7? instead of 0.5Hz? Have you tried different settings of the pendulum environment (use the original frequency of 0.5Hz)? If yes I am curious how other methods perform compare to TDPO.\n\nC1. It would be nice to have a discussion to compare the computation cost of each iteration of TDPO and other methods like TRPO and TD3.\n\nC2. As shown in the experiments, TDPO does not work well in the common gym environments, is there any changes to algorithm design possible to improve the current algorithm for these setting?\n\nC3. From the way the method is presented, it seems that it is not simple to implement the method? It is better to have a discussion on the implementation aspect of the method.\n\nSmall comments: \n- Secion 1, first paragraph, line 6, there are two \\delta_s\n- The definition of KL divergence in terms of Hilbert space inner product missing a subscript?\n- Introduce the definition of the term h.o.t. before use although it can be inferred from the context\n\nOverall, I suggest a weak-accept decision based on the following reason:\n- Novelty in algorithm development with theoretical justification for the monotonic improvement.\n- Although experiments indeed show the clear advantage of the proposed method over existing ones, more environments or more setting in the same environment should be presented to better evaluate its performance.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a deterministic policy gradient method that does not require to inject noise in the action selection.
Although the reviewers acknowledge that this paper has merits (novel and interesting idea, well written, technically sound), they have some doubts about the motivations for the proposed approach and about its empirical performance: a deeper analysis is requested.
The paper is borderline and needs to be revised before being ready for publication.
Decision: Reject