Truly Deterministic Policy Optimization
Authors: Ehsan Saleh
Saba Ghaffari
Tim Bretl
Matthew West
Venue: NeurIPS 2022 Accept (2022)
Keywords: Reinforcement Learning
Policy Gradient
Deterministic Policy Gradients

Abstract: In this paper, we present a policy gradient method that avoids exploratory noise injection and performs policy search over the deterministic landscape, with the goal of improving learning with long horizons and non-local rewards. By avoiding noise injection all sources of estimation variance can be eliminated in systems with deterministic dynamics (up to the initial state distribution). Since deterministic policy regularization is impossible using traditional non-metric measures such as the KL divergence, we derive a Wasserstein-based quadratic model for our purposes. We state conditions on the system model under which it is possible to establish a monotonic policy improvement guarantee, propose a surrogate function for policy gradient estimation, and show that it is possible to compute exact advantage estimates if both the state transition model and the policy are deterministic. Finally, we describe two novel robotic control environments---one with non-local rewards in the frequency domain and the other with a long horizon (8000 time-steps)---for which our policy gradient method (TDPO) significantly outperforms existing methods (PPO, TRPO, DDPG, and TD3). Our implementation with all the experimental settings and a video of the physical hardware test is available at https://github.com/ehsansaleh/tdpo .

Truly Deterministic Policy Optimization
Ehsan Saleh', Saba Ghaffari', Timothy Bret Matthew West
Department of Computer Science
*Department of Aerospace Engineering
3 Department of Mechanical Science and Engineering
University of Illinois Urbana-Champaign ehsans2, sabag2, tbretl, mwest@illinois edu
Abstract
In this paper, we present a policy gradient method that avoids exploratory noise injection and performs policy search the deterministic landscape, with the goal of improving learning with long horizons and non-local rewards. By avoiding noise injection all sources of estimation variance can be eliminated in systems with deterministic dynamics (up to the initial state distribution). Since deterministic policy regularization impossible using traditional non-metric measures such as the KL. divergence, we derive a Wasserstein-basse purposes. We state conditions on the system model under which it is possible establish monotonic policy improvement guarantee, propose a surrogate function for policy gradient estimation, and show that is possible compute exact advan- tage estimates if both the state transition model and the policy are deterministic. Finally, we describe two novel robotic control environments- rewards in the frequency domain and the other with long horizon (8000 time- steps)-for which our policy gradient method (TDPO) significantly outperforms existing methods (PPO, TRPO, DDPG, and TD3). Our implementation with all the experimental settings and video the physical hardware test is available at https://gthhub.oo//hhsssssslllh
quadratic model for our
with non-local
Policy Gradient (PG) methods can be broadly characterized by three defining elements: the policy gradient estimator, the regularization measures, and the exploration profile. For gradient estimation, episodic [58), importance-sampling based [50]. and deterministic [54]] gradients are some of the most
common estimation oracles. As for regularization measures, either an Euclidean distance within the parameter space [58
5z! 35 or dimensionally consistent non non-metric measures 550 255
52
26 59
have been frequently adapted. Common exploration profiles include Gaussian 50 and stochastic
processes 35 These elements form the basis of many model-free and stochastic policy optimization methods successfully capable of learning high-dimensional policy parameters.
Both stochastic and deterministic policy search can be useful in applications. A stochastic policy has the effect of smoothing or filtering the policy landscape, which is desirable for optimization. Searching through stochastic policies has enabled the effective control of challenging environments under general framework 500 [22) The same method could either learn robotic movements or play
basic games (1) with minimal domain- specific knowledge, (2) regardless of function approximation classes, and (3) with less human intervention (ignoring reward engineering and ayper-parameter tuning) 13 Using stochasticity for exploration, although it imposes approximations and variance, has provided robust way actively search for higher rewards. Despite many successes, there are practical environments which remain challenging for current policy gradient methods. For example, non-local rewards (e.g., those defined the frequency domain), long time horizons, and naturally- resonant environments all occur in realistic robotic systems policy gradient search.
34
38 4
but present issues for
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

To tackle challenging environments such as these, this paper considers policy gradient methods based on deterministic policies and deterministic gradient estimates, which could offer advantages by allowing the estimation of glohal reward gradients on long horizons without the need to inject noise into the system for exploration. To facilitate dimensionally consistent and low-variance deterministic policy search, compatible policy gradient estimator and a metric measure for regularization should be employed. For gradient estimation we focus on Vine estimators [50], which can be easily applied to deterministic policies. As metric measure, we use the Wasserstein distance, which can measure meaningful distances between deterministic policy functions that have non-overlapping supports (in contrast to the
(KL) divergence and the Total Variation (TV) distance).
The Wasserstein metric has seen substantial recent application in a variety of machine-learning domains, such as the successful stable learning of generative adversarial models [4] Theoretically. this metric has been studied in the context of Lipschitz-continung
Markov decision processes in reinforcement learning (21 [4]] Pirotta et al. 45 defined a policy gradient method using the Wasserestein distance by relying on Lipschitz continuity assumptions with respect to the policy gradient itself. Asadi et al. [5] and Rachelson and Lagoudakis
48 used the Wasserstein distance to derive model-based value-iteration and policy-iteration methods, respectively. On a more practical introduced a method which in the deterministic mode optimized policies using the Wasserstein distance. Pacchiano et al. [4] utilized Wasserstein regularization for behavior -uided stochastic policy optimization. Moreover, Abdullah al. [I] has proposed another robust stochastic policy gradient formulation. Estimating the Wasserstein distance for general distributions is more complicated than typical KL-divergences [56]. This fact constitutes and emphasizes the contributions
note, Ciosek et al. ?
of Abdullah et al. [1 and Pacchiano et al. 4 However, for our deterministic observation-condiioona policies, closed-form computation of Wasserstein distances is possible without any approximation.
Existing deterministic policy gradient methods (e.g., DDPG and TD3) use deterministic policies meaning that they learn deterministic policy function from states to actions. However, such methods still use stochastic search (i.e., they add stochastic noise to their deterministic actions to force exploration during policy search). In contrast, we will be interested in a method which not only uses deterministic policies, but also uses deterministic search (i.c., without constant stochastic noise injection). We call this method truly deterministic policy optimization (TDPO) and it may have lower estimation variances and better scalability to long horizons, as we will show in numerical examples.
54 35 15
Scalability to long horizons is one of the most challenging aspects of policy gradient methods that use stochastic search. This issue is sometimes referred to as the curse of horizon in reinforcement learning 36 General worst-case analyses suggest that the sample complexity of reinforcement learning is exponential with respect to the horizon length 27 B 30 Deriving polynomial lower-bounds for the sample complexity of reinforcement learning methods is still an open problem 24 Lower-bounding the sample complexity of reinforcement learning for long horizons under different settings and simplifying assumptions has been a topic of theoretical research [00 57 Some recent work has examined the scalability of importance sampling gradicnt estimators to long horizons in terms of both theoretical and practical estimator variances [36 28 29 All in all, long horizons are challenging for all reinforcement learning methods, especially the ones suffering from excessive estimation variance due the of stochastic policies for exploration, and our truly deterministic method may have advantages in this respect.
In this paper, we focus on continuous-domain robotic environments with reset capability to previously visited states. The main contributions of this work are: (1) we introduce a Deterministic Vine (DeVine) policy gradient estimator which avoids constant exploratory noise injection; (2) we derive a novel intermmiiistiallyycompplll
surrogate function and provide monotonic payoff improvement guarantees; (3) we show how to use the De Vine policy gradient estimator with the Wasserstein- surrogate in a practical algorithm (TDPO: Truly Deterministic Policy Optimization); (4) we illustrate the robustness of the TDPO policy search process in robotic control environments with non-local rewards, long horizons, and/or resonant frequencies.
1 Background
MDP preliminaries. An infinite-horizon discounted Markov decision process (MDP) is specified by A, P. R,,47), where S is the state space, A is the action space, P: A + ASS) the transition dynamics, R: S x A + [0, Rmax] is the reward function, 7 E [0,1) is the discount factor,
2

---

and pas) is the initial state distribution of interest (where A(F) denotes the set of all probability distributions over F, otherwise known as the Credal set of F). The transition dynamics P is defined as an operator which produces a distribution over the state space for the next state s' ~ P(s, a). The transition dynamics can be easily generalized to take distributions of states or actions as input (i.e., by having P defined as P (SS) x A - A(S) or P P x ((A) + A(S).. We may abuse the notation and replace 6, and 6, by s and a, where 6₃ and da are the deterministic distributions concentrated at the state and action a, respectively. A policy over actions for each state, and induces trajectories from a given starting state & as follows: $1 = s, a1 ~ IT (s11,, 71 = R(s1, 01), $2 P(s2, 02), d2 ($2), etc. We will denote trajectories state- action tuples т = ($1. 01, $2, 02, .... One can generalize the dynamics (1) by using a policy instead of an action distribution P(us, T) E (a-z(s)) (P(s, and (2) by introducing the t-step transition dynamics recursively TP' (1,, TT) (PP(TT- (1ys m), x) with P0 (1g
- A(A) specifies distribution
Ms. where μ4, is a distribution over S. The visitation frequency can be defined as P := (1-7) 1 1-1 Etm (U, T)
Table 2 of the Supplemenaary Material summarizes all MDP notation.
The value function of π is defined as (s) == E (لاس=r---
81 = 8; w|. Similarly, one can define (s, a) by conditioning on the first action. The advantage function can then be defined as their difference (i.e. A (s, a) := Q= a) - V (s)). Generally, one can define the advantage/value of one policy with respect to another using A (s)
T' r'([ss) and Q" (s, )') := E[Q"(s, a) a ~ T (-])s) Finally, the payoff of a policy 1/= == ElV* (8); ~ 14] is the average value over the initial states distribution of the MDP.
EsQ" a) (*(s)
Probabilistic and mathematical notations. Sometimes we refer to J f()) do integrals as (f. g) Hilbert space inner products. Assuming that Ç and are two probabilistic densities, the Kulback-Liebler (KL) divergence is DKI(Clv) == (C(x), log( ((2))) the Total-Variation ((((x)) =(x)), 1)2, and the Wasserstein distance is W(C,v) = infzerrç,,) <||= - yll, (22 V)) where v) is the set of couplings for C and v. We de- fine Li (x, v);x) == sup, f y)l and assume the existence of Lip(Q" (s. a); a) and Lip(V, "OV,,"((sss)) (s. constants. Under this notation, the Rubinstein-Kamnn Kantrovich (RK) duality states that the [(((z) - (z))x1 < WC, abuse the notation and denote sup, (-ss, (+1)), (a(- with W( T2) (and similarly for other measures). For parameterized policies, we define (=)(T)= == Vos(z) where π is parameterized by the vector 0. Table D of the Supplementary Material summarizes all these mathematical definitions.
(TV) distance is VV
Lip(/;x) bound is tight for all f. For brevity, we may
Policy gradient preliminaries. The advantage decomposition lemma provides insight into the relationship between payoff improvements and advantages (25) That is,
1m
E 2
[A (s, T2)]-
We will denote the current and the candidate next policy as Tı and n2, respectively. Taking derivatives of both sides with respect TT2 at T yields
A T1)) + (p%) ()), V AT (T (2)) V Na - VV (V.. (7,
1
Since ni does not have any advantage over itself (i.e., A (:, #1)= 0), the first term is zero. Thus, the Policy Gradient (PG) theorem is derived as
n2 Tu
E Banp
/V m2 T (s, (2,
(3)
m2=
For policy iteration with function approximation, we assume T2 and Ti to be parameterized by 02 and 01, respectively. One can view the PG theorem as a Taylor expansion of the payoff at 0z.
brief introduction of the Conservative Policy Iteration (CPI)
the Trust Region Policy Opti-
mization (TRPO) [50], the Proximal Policy Optimization (PPO)
51. the Deep Deterministic Policy
Gradient (DDPG) 35 and the Twin-Delayed Deterninistic Policy Gradient (TD3) [5] policy gradient methods is left to the Supplementary Material. Whether using deterministic policy gradients (e.g., DDPG and TD3) or stochastic policy gradients (e.g., TRPO and PPO), all these methods still perform stochastic search by adding stochastic noise to the deterministic policies to force exploration.
3

---

2 Monotonic Policy Improvement Guarantee
We use the Wasserstein metric because it allows the effective measurement of distances between probability distributions or functions with non-overlapping support, such as deterministic policies, unlike the KL divergence TV distance which are either unbounded maximal in this case. The physical transition model's smoothness enables the use of the Wasserstein distance to regularize deterministic policies. Therefore, we make the following two assumptions about the transition model:
T2)) S L ((11,7)) W (PP(p1, T), P(2, +)) - Ln W (2).
(4)
(5)
Also, we make the dynamics stability assumption sup -i= L, 11i-+11 Up. < <0, with the definitions of the new constants and further discussion of the implications deferred to the Supplementary Material where we also discuss Assumptions 4 and S and the existence of other
7
Lipschitz constants which appear as coefficients the final lower bound.
The advantage decomposition lemma can be rewritten as
1//
1//1 7 E Egnpp [A (8, m2)]
1 1-7
(p 2
P A (+,72))
The 1pp A (-, T2)) term has zero gradient at TI, which qualifies it to be crudely called "the second-order term". We dedicate a full section of our Supplementary Material to the theoretical derivations and proofs necessary to lower-bound this second-orde term into objective in form well-suited for practical optimization. Next, we present the final bound:
4 (T2) 1//
E Baupy
[ATI (8, T2)] C2 sup
W ((
T22 (als'')
+ ग. (als) T2(als) + क (als')
-C1- sup
V. W
2
2
22= expression as ( T2; 8) in the rest of the paper. We have 7/ 2 CauP (T2) and Ca (m)) = 7T, This facilitates the application of Theorem 2.1 as an algorithms (2..
For brevity, we denote the | s ((..
instance of Aamnoriattio-Maxiiiaae
Theorem 2.1. Successive maximization of Loup yields non-decreasing policy payoffs.
Proof. With 222 = arg max, Caup(TT we have Caup (TT2 2 Caup (TI). Thus,
Mra 2 (puu((z)) and 7m, = C (m1) = Mm Mr 2 L (T2) C ( 2 0.
Successive optimization of C (M2) generates non deneeeeerringgggg payoffs. However, is impractical due to the large number of "sup" terms that need to be expensively statistically estimated and implemented as constraints [50] To mitigate this, we take similar approach to TRPO and replace the maximums with expectations over the observations.
The Truly Deterministic Policy Optimization (TDPO) method is given in Algorithm T In the basic variant, the coefficients C1 and C22 are constant and a trust region is used. The coefficients C₁ and C2 are dynamics-dependent and the Supplementary Material provides practical notes on their choice. In the advanced variant, we use line search similar to Schulman et al. [50] and an adaptive tuning of the exploration scale using an importance sampling derivative estimate, as described Algorithm
We note that, for deterministic policies, the squared Wasserstein distance (Fz((a)s), T (als)) degenerates to the Euclidean distance over the action space. Any policy defines sensitivity matrix at given state 8, which is the Jacobian matrix of the policy output with respect to 8. The policy sensitivity term LGa (m,, n2;88 is essentially the squared Fuclidean distance over the action-to- observation Jacobian matrix elements. In other words, our surrogate prefers to step in directions where the action-to-observation sensitivity is preserved within updates.
3 Model-Free Estimation of Policy Gradient
The DeVine advantage estimator formally defined in Algorithm 2 Unlike DDPG and TD3, the Vine estimator allows our method to perform deterministic search by not consistently injecting

---

Algorithm Truly Deterministic Policy Optimization (TDPO)
Require: Initial policy To and exploration scale a.
Require: Advantage estimator and sample collector oracle A of Algorithm 2
1: for i = 0, 1, 2.... do
2: Collect trajectories and construct A using Algorithm 2
3: Compute the policy gradient at 8, : g t A a
4: Construct a surrogate Hessian vector product oracle
H such that for 0' 0; -8,
B
((r'(als)
C C2
Banpj
Co (()()
000 H60 + h.o.t. (9)
where h.o.t. denotes higher order terms in 80.
5: Find the optimal update direction 50" = H- 9 using the Conjugate Gradient algorithm.
6: (Basic Variant) Determine the best step size ax within the trust region:
C2 (ad0")" H(a8t")
O'' arg max o" (a88")
2
S.C.
2 (aaa (o 80") S 62 max
(10)
7: (Advanced Variant) Determine the best step size a using a line-search procedure and pick the
best one; each coefficient's performance can be evaluated by sampling from the environment. 8: (Advanced Variant) Update the exploration scale A." using the collected samples.
9: Update the policy parameters: 0,+1 +- 0, + 50"
10: end for
Algorithm 2 Deterministic Vine (DeVine) Policy Advantage Estimator
Require: The number of workers K, policy æ, initial state distribution μ(()), and discount factor y
Require: An exploration index set distribution exploration scale and maximal horizon H. Sample an initial state so from μs and then roll out a trajectory (sp. ag $1 ".,".-) using π. from v.
2: Sample the exploration indices set Axx == {(11,
3: for = 1. 1.2.... K do
4: Compute the value V (*t() -i=
5: Reset the initial state Sty set d/e := T(t,,) + eju. and use for the rest of the trajectory,
with ej being the jth basis element for A. This will create 7' (Stur aey s "/a1::'''"
6: Compute the value QT (Stu ob22) = -imry 2 af).
7: Compute the advantage A (st2 a/.) = Q (S o4,.) V (T
8: end for
K dim(A) H H. v(Xx)
(T2 arp ( - acs)) A (s d.,))
9: Define (TT2)
K
(a/, Ore [a/// dcee
10: Return A (T²) and V
A Ti (T2) unbiased estimators for E Exmp [A (s, T2)] and the PG.
noise in actions for exploration. Algorithm uses an exploration index sampler v. which samples set of time-ster stees and action dimensions for exploration perturbation. The truly deterministic version of TDPO uses the deterministic Vaa which always returns the complete covering of {1.... dim (AA} x
(1..... H Using Mits, DeVine produces exact policy gradients in the limit of small stated in
Theorem 3.T whose proof is deferred to the Supplementary Material.
Theorem 3.1. Assume finite horizon MDP with both deterministic transition dynamics and initial distribution p. with maximal horizon length of H. Define K H dim(A) and v := Van where Vie always returns the complete covering of [1,.... dim(AA) {1...., "L,...,H..... Then we have
H}.
lim V 010
T1 (T2)
V m==m
(11)
Although this theorem sets the stage for computing a fully deterministic gradient, stochastic approxi- mation can be used in Algorithm 2 by randomly sampling small set of states from for advantage
5

---

TDPO (ours)
DSc. Payoff
-10--
TD3
100
TRPO
-00
PPO
DDPG
-109
100M
200M
300M
400M
500M
Sample Count
TDPO
TRPO
PPO
DDPG
TD3
0.8
0.5
0.2
Time (5)
Time (s)
Time (s)
Time (s)
Time (s)
Desired Offset
se- Desired Offset
* Desired Offset
- Desired
e- Desired Offset
0.4
Desired Frequency & Amplitude
resired Frequoncy & Amplitude
Dasirad Frequency & Amplitude
Desired Frequency
Desired Frequency & Amplitude
Amplitude
0.2
& Amplitude
0.0
Frequency (H1z)
Frequency (H2))
Frequency (H2)
Frequency (H2)
Frequency (Hzz
Figure 1: Results for the simple pendulum with non-local rewards. Upper panel: training curves with empirical discounted payoffs. Lower panels: trajectories both the time domain and frequency domain, showing target values of oscillation frequency, amplitude, and offset. The basic variant of our method inon-adaptive exploration scales and update coefficients) was used this experiment. The initial agent payoff's indicate the performance after the first epoch.
estimation. In other words, Theorem 3.1 would use v to deterministically sample all trajectory states, whereas this is not practical requirement for Algorithm 2 and the gradients are still unbiased if random set of vine branches used.
The Vine estimator can be advantageous in at least two scenarios. First, in the case of rewards that cannot be decomposed into summations of immediate rewards. example, overshoot penalizations or frequency-based rewards as used in robotic systems are non-local. DeVine can be robust to non-local rewards as it is insensitive to whether the rewards were applied immediately or after a long period. Second, De Vine can be appropriate choice for systems that sensitive to the injection of noise, such as high-bandwidth robots with natural resonant frequencies. In such cases, using white (or colored) noise for exploration can excite these resonant frequencies and cause instability, making learning difficult. DeVine avoids the need for constant noise injection.
Experiments
The next three subsections show challenging robotic control tasks including frequency-based non- local rewards, long horizons, and sensitivity to resonant frequencies. In Sections 4.1 and we use the basic variant of our method (i.e., fixed exploration scale and update coefficient hyper-parameters throughout the training). This will facilitate a better understanding of our core method's capabilities without any additional tweaks. See the Supplementary Material for comparison environments, where the basic variant of TDPO works similarly to existing methods. Section 4.3 includes the most difficult setting in paper, where use the advanced variant of our method (i.e., with line-search the update coefficient and adaptive exploration scales).
traditional gym
4.1 An Environment with Non-Local Rewards
The first environment that consider is simple pendulum. The transition function is standard the states are joint angle and joint velocity, and the action is joint torque. The reward function is non-
local rewards are reward functions of the entire trajectory whose payoff's cannot be decomposed into the sum of terms such as E. (St, a.), where functions fi only depend on nearby states and actions. An example noc-local reward is one that depends the Fourier transform of the complete trajectory signal.

---

-3x10*
TDPO (ours)
Disc. Payoff
TRPO
-104
PPO
-3x10*
DDPG
200M
300M
400M
Sample Count
TDPO
TRPO
DDPG
TD3
Velocitees (radis)
Hip
Hip
Hip
-20
Knee
Knee
Knee
Knee
Torques (N m)
mh Knee
Knee
Knee
Knee
Hip
Hip
Hip
Hip
0.2
0.3
0.2
0.3
0.2
0.3
0.2
0.3
0.2
0.3
Time (s)
Time (5)
Time (s)
Time (s)
Time (s)
Figure 2: Results for the leg environment with long horizon and resonant frequencies due to ground compliance. Upper panel: training curves with empirical discounted payoffs. Lower panel: partial trajectories, restricted to times shortly before and after impact with the ground. Note the oscillations about 100 Hz that appear just after the impact at 0.2 s these oscillations are evidence of resonant frequency. The basic variant of our method (non-adaptive exploration scales and update coefficients) was used in this experiment.
standard-rather than define local reward in the time domain with the goal of making the pendulum stand upright (for example), we define non-local reward in the frequency domain with the goal of making the pendulum oscillate with desired frequency and amplitude about a desired offset. In particular, we compute this non-local reward by taking the Fourier transform of the joint angle signal the entire trajectory and by penalizing differences between the resulting power spectrum and desired power spectrum. We apply this non-local reward at the last time-step of the trajectory. All methods here only used the current state of the systems, despite the fact that these environments in all cases partially observable or tistory-dependent Nevertheless, the agents are able to achieve high-reward behaviors, since these environments have weak history dependence. Note that this environment is toy-problem to illustrate frequency dependence, and may as such be solved using Wavelet short-term Fourier transformations in conjunction with the existing PG methods. However, our focus is on the representative features of this example, rather than this particular problem itself. Implementation details and similar results for more variants are left to the Supplementary Material.
Figure shows training curves for TDPO (our method) as compared to TRPO, PPO, DDPG, and TD3. These results were averaged over 25 experiments in which the desired oscillation frequency was 1.7 Hz (different from the pendulum's 's natural frequency of 0.5 Hz), the desired oscillation amplitude was 0.28 rad, and the desired offset was 0.52 rad. Figure T also shows trajectories obtained by the best agents from each method. TDPO (our method) was able to learn high-reward behavior and to achieve the desired frequency, amplitude, and offset. TRPO was able to learn the correct offset but did not produce any oscillatory behavior. TD3 also learned the correct offset, but not the desired oscillation. PPO and DDPG failed learn any desired behavior. We hypothesize that our method was able to learn good behaviors here because using a deterministic policy for exploration avoids excitation of spurious frequencies and the Vinine estimator is accurate even for the non-local reward.
4.2
An Environment with Long Horizon and Resonant Frequencies
The second environment that we consider a single leg from a quadruped robot This leg has two joints, "hip" and "knee," about which it is possible to exert torques. The hip is attached to
7

---

TRPO
PPO
TD3
Optuna
ProSRS
Dayesian@pumiccaion
PraSRS
ProSRS
Payoff
10-
SKOpt
Payoft
-10
Bayolandddmmiatin
Payoff
10
Optune
SKOpt
-15
GPyOpt
-15
-15
GPyOpt
-20
-20
HPO Iteration
HPO Iteration
HPO Iteration
Figure 3: The best payoff vs. the Hyper- Parameter Optimization (HPO) iteration on short-horizon variant of the legged robotic environment. The HPOs are performed for each of the TRPO, PPO, and TD3 methods in a separate panel. DDPG is a special case of TD3 with HPO. Since TD3 was considerably more expensive, only show Optuna and ProSRS for it.
-102
TDPO (ours)
Payoff
TRPO
-103
PPO
-104
0
1B
2B
3B
4B
5B
Sample Count
Optimization (HPO) training curves with the best settings found for TRPO and PPO compared to the advanced variant of method (TDPO with adaptive exploration scales and line search). TD3 had a significantly poor performance in the initial parameter sweeps. Due to resource limitations and poor initial performance, excluded TD3 from this experiment.
Figure 4: Post
slider that confines motion vertical line above flat ground. We assume the leg is dropped from some height above the ground and the task is to recover from this drop and to stand upright at rest after impact. States given to the agent are the angle and velocity of each joint (slider position and velocity are hidden), and actions are the joint torques, The reward function penalizes difference from an upright posture, slipping chattering at the contact between the foot and the ground, non-zero joint velocities, and steady-state joint torque deviations. We use the open- seen-source MuJoCo software simulation (55), with high-fidelity models of ground compliance, motor nonlinearity, and joint friction. The control loop rate is 4000 Hz and the rollout length is 2 s, resulting in horizon of 8000 steps. Implementation details are left to the Supplementary Material.
Figure shows training curves for TDPO (our method) as compared to TRPO, PPO, DDPG and TD3. These results were averaged
75 experiments. A discount factor of y 0.99975 was chosen for all methods, where (1 -1 is half the trajectory length. Similarly, the GAE factors for PPO and TRPO were scaled up to 0.99875 and 0.9995, respectively, in proportion the trajectory length. Figure 2 also shows trajectories obtained by the best agents from each method. TDPO (our method) was able learn high-reward behavior. TRPO, PPO, DDPG, and TD3 were not.
We hypothesize that the reason for this difference performance is that TDPO better handles the combination of two challenges presented by the leg environment (8000 steps) and the existence of resonant frequency that results from compliance between the foot and the ground (note the oscillations frequency of about 100 Hz that appear in the trajectories after impact). Both high-speed control loops and resonance due to ground compliance are common features of real-world legged robots which TDPO seems be more resilient.
unusually long time horizon
Ressoantt frequencies are a concept from control theory. In the frequency domain, signals of certain frequencies are excited more than others when applied to system. This is captured by the frequency-domain transfer function of the system, which may have peak of magnitude greater than The resonant frequency the frequency which frequency-doman of systems with a resonant frequency include the undamped pendulum, which oscillates at its natural frequency, and RLC circuits which have characteristic frequencies at which they are most excitable. See Chapter 8 of Kuo and Golnaraghi [4] for more information.
transfer function the highest amplitude. Common examples

---

4.3 Practical Training and Hardware Implementation
For the most realistic setting, we take the environment from the previous section and make it highly stochastic by (a) injecting physical modeling noise into the transition dynamies P. and (b) making the initial state distribution ll random as physically possible. We also systematically perform Hyper- Parameter Optimization (HPO) on all methods allow the most fair comparison.
The choice of the HPO method can have a significant impact on the RI. agent's performance. We consider list of five off-the-shelf HPO implementations and run them in their default settings: Optuna 2 BayesianOp- Scikit-Optimize To] GPyOpt [T7], and ProSRS 153 These implementations include range of HPO methods, including Gaussian processes and tree Parzen estimators. For better performance, HPO methods need reasonable set of initial guesses. For this, we perform a one- aheevariable-ate at-a-at---ltee atemmmmm parameter sweep along every typer-parameter near the RL method's default syper-parameters. These param- eter sweep results are then input to each HPO method for full optimization. Using all HPO algorithms for all RL methods in the long-horizon environment (where each full training run takes 5 billion samples) is com- putationally infeasible. To pick the best HPO method, we benchmark a short-horizon environment with only 200 time-steps in trajectory. The result is shown in Figure 3) (see the Supplementary Material for full details on the HPO methods). Overall, we found that Optuna and ProSRS are the best HPO methods on the test problem. Since Optuna is widely-tested and arguably the most popular HPO library, we pick it as the main HPO method for our long-horizon environment.
timization 42
Figure 5: The simulation-to-reet transfer of the best TDPO agent to perform a suc- cessful drop test at 4 kHz control rate.
We repeat the same HPO procedure on the long-horizer environment using Optuna, and pick the best hypee-parameters found in the course of HPO for final training. Figure 4 shows this final training. TDPO shows superior performance in this highly stochastic environment, and such benefits cannot be obtained by merely performing HPO on other methods. To showcase the practicality of our method, we picked the best TDPO trained agent, and implemented it on the physical hardware. The transferred agent was able to successfully perform drop-and-catch tests on the robot system at 4 kHz, with both global control and suppression of high-frequency transients. Figure shows a glimpse of this test, and short video is also included in the code repository.
5 Discussion
We proposed a deterministic policy gradient method (TDPO: Truly Deterministic Policy Optimization) based on the use of a deterministic Vine VDeVine) gradient estimator and the Wasserstein metric. We proved monotonic payoff guarantees for our method, and defined a novel surrogate for policy optimization. We believe that using deterministic policies for exploration and avoiding the need for consistent noise injection results in lower gradient estimation variances, enabling our method to solve tasks with longer horizons and non-local rewards. We introduced several realistic robotic control tasks that have such features and we showed that TDPO performs well on them, contrast to existing policy gradient methods.
There are number of limitations of this paper. First, we assumed continuous environments and required a state reset capability of the environment, which is commonly available in simulators but would prevent learning on hardware. Second, TDPO relies on local gradients and thus may not be able to learn effectively in environments with payoffs that are sparse in either action or state. Such environments typically need substantial exploration which TDPO may not be able achieve. Third, the analysis in this paper does not specifically pinpoint the reasons why existing methods fail on the long-horizon con-loral-reward robotic control environments. For instance, such failures might be more specifically attributed to either (1) the frequency- based nature of the reward, (2) the non-locality of the reward signal, or (3) the long horizon. Understanding this an important prerequisite for future work.

---

6 Acknowledgement
We sincerely thank Chenzhuang Li for his valuable help, feedback, and expertise with the physical robotic system. We also thank Hae- Won Park for sending the robotic platform to facilitate the experiments in this article, and the valuable advice for system identification. This research is part of the Blue Waters sustained- Foundation (awards OCI-0725070 and ACI- ACI-1238993) the State Illinois, and as of December, 2019, the National Geeospatia-IItelliennne Agency. Blue Waters is joint effort of the University of Illinois at Chammaign and its National Center for Supercomputing Applications [8] 331 This work also utilizes resourees supported by the National Science Foundation's 's Major Research Instrumentation (MRI) program, grant Number 1725729.
computing project, which is supported by the National Science
References
[1] Mohammed Amin Abdullah, Hang Ren, Haitham Bou Ammar, Vladimir Milenkovic, Rui Luo, Mingtian Zhang, and Jun Wang. Wasserstein robust reinforcement learning. arXiv preprint arXiv:1907.13196, 2019.
[2] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25rd ACM SIGKDD) International Conference on Knowledge Discovery and Data Mining, 2019.
[3] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Conerete problems in ai safety. arXiv preprint arXiv:1606.0m (60.06660.. 2016.
[4] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan. arXiv preprint 17Xi::7701.07875. 2017.
[5] Kavosh Asadi, Dipendra Misra, and Michael L Littman. Lipschitz continuity in model- reinforcement learning. arXiv preprint 0rXiv:/804.07/990 2018.
[6] Vahid Behzadan and Arslan Munir. Vulnerability of deep reinforcement learning to policy induction attacks. In International Conference on Machine Learning and Data Mining in Pattern Recognition, pages 262-275. Springer, 2017.
[7] Shalabh Bhatnagar, Doina Precup, David Silver, Richard S Sutton, Hamid R Maei, and Csaba Szepesvári. Convergent cemporal-difference learning with arbitrary smooth function approxi- mation. In Advances in neural information processing systems, pages 1204-1212, 2009. 2009.
[8] Brett Bode, Michelle Butler, Thom Dumning, Torsten Hoefler, William Kramer, William Gropp, and Wen-mei Hwu. The Blue Waters super-system for super-science. In Contemporary High Performance Computing, Chapman & Hall/CRC Computational Science, pages 339 366. Chapman and Hall/CRC, April 2013. ISBN 978-1-4665-68344 URL httpt:://///:t
taylorfrancis. httmm//ooo/e////7156555555
[9] Kamil Ciosek, Quan Vuong. Robert Loftin, and Katja Hofmann. Better exploration with optimistic actor critic. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, F. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https::/proceeddngss www.igg.............o wtt..erriii.scccppe////22.... file/a /tep://aaa2333322200020088822228 Paper. pdf
[10] Christoph Dann and Emma Brunskill. Sample complexity of episodic fixed-horizon reinforce- ment learning. In Advances in Neural Information Processing Systems, pages 888-2826. 2015.
[11] Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https: https://githur wwt.uu.cccmo/peeeeeeeeeeee wwwp.oooooo//oaaaelianon..
2017.
[12] Barzin Doroodgar, Yugang Liu, and Goldie Nejat. A learning-based semi-autonomous con- troller for robotic exploration of unknown disaster scenes while searching for victims. IEEE Transactions Cybernetics, 44(12):2719-2333 2014.
10

---

[13] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning, pages 1329-13333 2016.
[14] Norman Ferns, Prakash Panangaden, and Doina Precup. Metries for markov decision processes with infinite state spaces. arXiv preprint arXiv: /386, 2012.
[15] Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv:1802.09477, 2018.
[16] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pages 249-256, 2010.
[17] Javier González, Zhenwen Dai, Philipp Hennig, and Neil Lawrence. Batch bayesian optimization via local penalization. In Artificial intelligence and statistics, pages 648 038-6. PMLR, 2016.
[18] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290 2018.
[19] Tim Head, MechCoder, Gilles Louppe, Iaroslav Shcherbatyi, fcharras, Zé Vinícius, cmmalone, Christopher Schröder, ne1215, Nuno Campos, Todd Young, Stefano Cereda, Thomas Fan, rene rex, Kejia (KJ) Shi, Justus Schwabedal, carlosdanielcsantos, Hvass-Labs, Mikhail Pak, SoManyUsernames Taken, Fred Callaway, Loic Estève, Lilian Besson, Mehdi Cherti, Karlson Pfannschmidt, Fabian Linzberger, Christophe Cauet, Anna Gut, Andreas Mueller, and Alexander v0.5.2, March 2018. URL httpss//doii org/10
Fabisch. 1207017
5281/zenodo.
[20] Ashley Hill, Antonin Raffin, Maximilian Emestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert. Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. https
//github. www.cooooooaaaaaceececceeeceaa www.aaattaaaaaaee
saaaaa/aabaaee-ealleeee baselines, 2018.
[21] Karl Hinderer. Lipschitz continuity value functions in markovian decision processes. Mathe- matical Methods of Operations Research, 62(1):3-22, 2005.
[22] David R Hunter and Kenneth Lange. A tutorial on mm algorithms. The American Statistician, 58(1):30-37, 2004.
[23] Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, 4(26):eaau5872, 2019.
[24] Nan Jiang and Alekh Agarwal. Open problem: The dependence of sample complexity lower bounds on planning horizon. In Conference On Learning Theory, pages 3395-3398, 2018.
[25] Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In ICML, volume 2. pages 267-274, 2002.
[26] Sham M Kakade. A natural policy gradient. In Advances in neural information processing systems, pages 1531-1538, 2002.
[27] Sham Machandranath Kakade et al. On the sample complexity of reinforcement learning. PhD) thesis, University of London London, England, 2003.
[28] Nathan Kallus and Masatoshi Uehara. Efficiently breaking the curse of horizon in off-policy evaluation with double reinforcement learning. arXiv, pages arXiv-1909, 2019.
[29] Nathan Kallus and Masatoshi Uchara. Statistically efficient off off-policy policy gradients. arXiv preprint arXiv:2002.04011 2020.
[30] Michael Kearns, Yishay Mansour, and Andrew Y Ng. A sparse sampling algorithm for near- optimal planning in large markov decision processes. Machine learning, 49(2-3):193-208, 2002.
11

---

[31] Michael J Kearns, Yishay Mansour, and Andrew Y Ng. Approximate planning in large pomdps via reusable trajectories. In Advances in Neural Information Processing Systems, pages 1001- 1007, 2000.
[32] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 6980, 2014.
[33] William Kramer, Michelle Butler, Gregory Bauer. Kalyana Chadalavada, and Celso Mendes. Blue Waters Parallel I/O Storage Sub-system. In Prabhat and Quincey Koziol, editors, High Performance Parallel I/O, pages 17-32. CRC Publications, Taylor and Francis Group, 2015.
ISBN 978-1-4665- -824--7
[34] Benjamin C. Kuo and Farid Golnaraghi. Automatic Control Systems. John Wiley & Sons, Inc., USA, 8th edition, 2002. ISBN 0471134767.
[35] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 00..509.9997/ 2015.
[36] Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the curse of horizon: off-policy estimation. In Advances in Neural Information Processing Systems, pages 2356-5366 2018.
[37] Hongzi Mao, Mohammad Alizadeh, Ishai Menache, and Srikanth Kandula. Resource manage- ment with deep reinforcement learning. In Proceedings of the 15th ACM Workshop on Hot Topics in Nenworks, pages 50-56, 2016.
[38] Leonard Meirovitch. Elements of vibration analysis. McGraw-HHe Hill Science, Engineering & Mathematics, 1975.
[39] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-538 2015.
[40] Ngoc Duy Nguyen. Thanh Nguyen, Saeid Nahavandi, Asim Bhatti, and Glenn Guest. Manip- ulating soft tissues by deep reinforcement learning for autonomous robotic surgery. In 2019 IEEE International Systems Conference (SysCon), pages 1-7. IEEE, 2019.
[41] Farzad Niroui, Kaicheng Zhang, Zendai Kashino, and Goldie Nejat. Deep reinforcement learn- ing robot for search and rescue applications: Exploration in unknown cluttered environments. IEEE Robotics and Automation Letters, 4(2):610-617, 2019.
[42] Fernando Nogueira. Bayesian Optimization: Open source constrained global optimization tool for Python, 2014-. URL htl.tttss/gttabbb.ooo///////sasp
[43] Aldo Pacchiano, Jack Parker-Holder, Yunhao Tang, Anna Choromanska, Krzysztof Choroman- ski, and Michael Jordan. Learning to score behaviors for guided policy optimization. arXiv preprint (t06:/99000444444 2019.
[44] Hae- Won Park, Patrick M Wensing, and Sangbae Kim. High-speed bounding with the mit cheetah 2: Control design and experiments. The International Journal of Robotics Research, 36 (2):167-192, 2017.
[45] Matteo Pirotta, Marcello Restelli, and Luca Bascetta. Policy gradient in lipschitz markov decision processes. Machine Learning, (0002-3):255-2883 2015.
[46] Mariya Popova, Olexandr Isayev, and Alexander Tropsha. Deep reinforcement learning for de novo drug design. Science advances, 4(7))eaap7885, 2018.
[47] André Preumont and Kazuto Seto. Active control of structures. John Wiley & Sons, 2008
[48] Emmanuel Rachelson and Michail G. Lagoudakis. On the locality of action domination in sequential decision making. In / /1th International Symposium on Artificial Intelligence and Mathematics (ISIAM 2010), pages 1-8, Fort Lauderdale, US, 2010. URL https::/ //ootoo fr/17977/
12

---

[49] David Rohde, Stephen Bonner, Travis Dunlop, Flavian Vasile, and Alexandros Karatzoglou. Recogym: A reinforcement learning environment for the problem of product recommendation in online advertising. arXiv preprint arXiv: 808.00020, 2018.
[50] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. International conference on machine learning, pages 1889-1897, 2015.
[51] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.022430 2015.
[52] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford. and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
[53] Chenchao Shou and Matthew West. A tree-based radial basis function method for noisy parallel surrogate optimization. arXiv preprint arXiv: 070900 2019.
[54] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In ICML, 2014.
[55] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012.
[56] Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008.
[57] Ruosong Wang, Simon S Du, Lin F Yang, and Sham M Kakade. Is long horizon reinforce- ment learning more difficult than short horizon reinforcement learning? arXiv preprint arXiv:2005.00527, 2020.
[58] Ronald J Williams.
Simple statistical followng algorithms for connectionist reinforce- ment learning. Machine learning, 8(3-4)222--555 1992.
[59] Yuhuai Wu, Elman Mansimov, Roger B Grosse, Shun Liao, and Jimmy Ba. Scalable trust-region method for deep reinforcement learning using froneecker-fctorred approximation. In Advances in neural information processing systems, pages 5279-5288, 2017.
[60] Sangdoo Yun, Jongwon Choi, Youngjoon Yoo, Kimin Yun, and Jin Young Choi. Action-decision networks for visual tracking with deep reinforcement learning. In Proceedings of the IEEE conference on computer vision and patiern recognition, pages 2711-2720, 2017.
[61] Zhempeng Zhou, Xiaocheng Li, and Richard N Zare. Optimizing chemical reactions with deep reinforcement learning. ACS central science, 3(12):1337-1344, 2017.
Checklist
1. For all authors...
(a) Do the main claims made the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] The claims are well-supported with both theoretical derivations and numerical examples.
(b) Did you describe the limitations of your work? [Yes] The last sentence of the discussion section talks about the current limitation of our paper non-discrete environments.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We dedicated section of the appendix to discussing potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] All theoretical results and mathematical notations are clearly stated in the form of lemmas and theorems, both the main paper and the appendix.
13

---

(b) Did you include complete proofs of all theoretical results? [Yes] All the theoretical theorems and lemmas are proven either in the main paper or in the appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] We have included detailed instructions to reproduce all experiments in the appendix, and anonymized code repository includes all the code we used to produce the paper and the appendix results.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Yes, training details and experiment specifications in the appendix. For best reproducibility, we also released the code along with the specific (in JSON format) our anonymized code repository. Our code repository includes all software/library version details that we used to produce the results.
discussed at length
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] All our experiments include 95% confidence intervals, and are ran for 100 independent random seeds.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] This was left to the appendix.
4. you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cited the scholar article for the Mujoco physical simulator used in our paper.
(b) Did you mention the license of the assets? [Yes] After de-anonymization, will open-source our code with an MIT license. We used the open-source Mujoco simulator which is available under the Apache 2.0 lisence.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable?? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "xlWTf23m8Og", "reviewer": "Reviewer_KtyC", "summary": "The authors present a (mostly) deterministic policy gradient method. They derive a theoretical algorithm showing it recovers the same policy gradient estimator.\n\nThis is then evaluated in practice showing that it performs similarly to other on-policy methods when tackling standard mujoco tasks. In addition, they show impressive performance gains when tackling long-horizon non-markovian rewards that rely on frequency domain matching.", "questions": "- DDPG and TD3 are shown to fail. These tasks are by design partially observable or history dependent. DDPG (and variants) are by design off-policy history agnostic methods. Are PPO/TRPO/TDPO utilizing memory modules or are they only observing the current state?\n\n- What happens when the problem is shorter term with more frequent rewards? For instance, the frequency can be estimated using STFT (for instance) and combining multiple per-step rewards. This might not be the exact same objective, but it's a denser task that is interesting to investigate.\n\n- How do the various methods compare on sparse, non-markovian, rewards that are not necessarily frequency based?\n\nOverall, while the experiments are impressive, I feel they do not cover the full analysis expected of this method. If the point is fully deterministic, the question is where does this help the agent perform better? How does this improve performance. Is it only in sparse reward tasks? Does it demand non-markovian nature or not? Why do previous on-policy methods fail in non-markovian returns (TRPO/PPO)?", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- They present a (mostly) deterministic method that is capable of solving long-horizon problems in the frequency domain. A task that prior works are shown to be incapable of tackling.\n- Results in the frequency domain are impressive.\n\nWeaknesses:\n\n- The premise of the paper is a truly deterministic policy gradient method, yet Algorithm 2 is not truly deterministic. Even though q is sampled from less often than off-policy exploration schemes, this still results in additional noise injection.\n- Overall the work is presented as if the major benefit is true determinism whereas I feel the major benefit here is in the ability to tackle long horizon sparse reward tasks.\n\nMinor points:\n\n- I argue the authors to go over the paper with a spelling/grammar correction tool (such as grammarly). There are many mistakes such as \"appeare\" which would be trivially solved with minimal effort.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ldEy8LbiP0X", "reviewer": "Reviewer_c8Yt", "summary": "A solid paper, which I think represents a valuable contribution to the NeurIPS community because it tackles an important problem (improving actor-critic methods) in a novel way. The major weakness is in presentation, especially of issues around exploration. With deterministic policies, we need to be very careful to explore enough, but the paper is remarkably silent about what policy is used for exploration. \n\nThe literature review section is missing OAC [1], whose deterministic version explores using deterministic policies, solving an optimisation problem that involves the Wasserstein divergence. \n\nTypos:\n- line 152: well-suited form => in a form well-suited\n- line 287: widely-test => widely tested\n\nReferences:\n[1] Ciosek, K., Vuong, Q., Loftin, R., & Hofmann, K. (2019). Better exploration with optimistic actor critic. Advances in Neural Information Processing Systems", "questions": "1. Please provide more details about how the exploration policy is chosen. Moreover, can you clarify the pseudocode of Algorithm 1 to make it clear where the exploration policy comes form?\n2. Can you explain what you mean by constraints in line 160? The optimisation objective in equation (7) does not seem to come with any constraints.\n3. In line 116, you take the 2-norm of a Lipschitz constant. What does that mean given the Lipschitz constant is defined as a scalar?\n4. In practice, can you explain how injecting noise into the transition dynamics you describe in the beginning of section 4.3 is different from injecting noise into the action distribution (which is what vanilla TD3 and DDPG do).\n5. In the top part of Figure 1, can you explain why the different algorithms have vastly different performance at zero samples. ", "limitations": "The authors do not explicitly discuss the limitations of the work, except for the last sentence in the Discussion section. In my view, the major limitations are:\n1. Empirically, while the evaluation is statistically excellent (the authors use 100 random seeds), the authors only test on relatively simple environments. Moreover, other than pendulum, the used environments are non-standard and seem to have been introduced with the proposed algorithm in mind. This presents a risk that the good performance of the proposed method is due to overfitting a cherry-picked benchmark. However, the authors do alleviate these concerns to a certain extent by testing on a real robot.\n2. Theoretically, the authors do not show, even in the bandit case, why their method explores better than alternative approaches. You do not have to provide a regret bound (although that would be ideal), but providing some intuitions about how the proposed method is able to explore would be nice.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper addresses an important issue of improving policy-gradient-based RL agents. This has huge practical significance since most RL agents used in practice are based on policy gradients.\n2. The empirical evaluation is limited in scope, but very solid, with 100 independent runs.\n3. As far as I can tell, the proposed update to the policy is novel.\n4. There is an experiment on a real robot.\n\nWeaknesses:\n1. The presentation could be improved, particularly concerning:\n    - how the algorithm explores\n    - the pseudocode needs to be made more specific (make it clear that algorithm 2 is invoked in line 2 of algorithm 1).\n2.  The simulated benchmarks seem a bit artificial.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "B40eiIaStoK", "reviewer": "Reviewer_3Zyz", "summary": "This work proposes a gradient-based algorithm for updating a deterministic policy’s parameters in a reinforcement learning setting. While RL algorithms for training deterministic policies exist, they require the use of a stochastic exploration policy that can introduce variance into policy updates. This variance is especially problematic in environments with non-local rewards and long horizons. The TDPO algorithm uses a deterministic version of the Vine estimator to compute exact advantage estimates, and regularizes the policy updates using the Wasserstein metric, which is uniquely suited for deterministic policies. The algorithm is benchmarked against state-of-the-art baselines on robotic domains with non-local rewards and long horizons, and clearly shows a benefit in achieved performance.", "questions": "My primary question is: in what environments do you expect the algorithm to perform poorly? My main concern is that it would have inadequate exploration. Though $\\epsilon$-greedy is a notoriously inefficient strategy, would this algorithm fair better than, say, Thompson sampling or UCB-exploration, if those algorithms could be tractably applied to the same problem setting?", "limitations": "The discussion of limitations in this work is restricted its scope, i.e. that the algorithm is designed for continuous state and action spaces only. These is no discussion of negative societal impact, though I think such a discussion is unnecessary for this work. Regarding scope, and as stated in the Strengths & Weaknesses section, I would add a discussion on how the algorithm would fare in settings where exploration is required, and when to avoid using it in favor of another algorithm.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThis paper is clearly written, and derives its approach with appropriate rigor. Moreover, it identifies an important problem, proposes a sensible solution, and thoroughly demonstrates the benefits of the work through experiments. \n\nWeaknesses:\nThough this is a very strong paper, I think there is inadequate discussion of its limitations. Namely, it seems to be the cases that any exploration can only result from perturbing the initial condition. It seems that some exploration is necessary in many domains, and it would be worth spelling out how TDPO would fare in such domains. \n\nThere are also a few typos, such as “Since Optuna is widely-test[ed] and…“.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept