Truly Deterministic Policy Optimization

In this paper , we present a policy gradient method that avoids exploratory noise injection and performs policy search over the deterministic landscape . By avoiding noise injection all sources of estimation variance can be eliminated in systems with deterministic dynamics ( up to the initial state distribution ) . Since deterministic policy regularization is impossible using traditional non-metric measures such as the KL divergence , we derive a Wasserstein-based quadratic model for our purposes . We state conditions on the system model under which it is possible to establish a monotonic policy improvement guarantee , propose a surrogate function for policy gradient estimation , and show that it is possible to compute exact advantage estimates if both the state transition model and the policy are deterministic . Finally , we describe two novel robotic control environments—one with non-local rewards in the frequency domain and the other with a long horizon ( 8000 time-steps ) — for which our policy gradient method ( TDPO ) significantly outperforms existing methods ( PPO , TRPO , DDPG , and TD3 ) . Policy Gradient ( PG ) methods can be broadly characterized by three defining elements : the policy gradient estimator , the regularization measures , and the exploration profile . For gradient estimation , episodic ( Williams , 1992 ) , importance-sampling-based ( Schulman et al. , 2015a ) , and deterministic ( Silver et al. , 2014 ) gradients are some of the most common estimation oracles . As for regularization measures , either an Euclidean distance within the parameter space ( Williams , 1992 ; Silver et al. , 2014 ; Lillicrap et al. , 2015 ) , or dimensionally consistent non-metric measures ( Schulman et al. , 2015a ; Kakade & Langford , 2002 ; Schulman et al. , 2017 ; Kakade , 2002 ; Wu et al. , 2017 ) have been frequently adapted . Common exploration profiles include Gaussian ( Schulman et al. , 2015a ) and stochastic processes ( Lillicrap et al. , 2015 ) . These elements form the basis of many model-free and stochastic policy optimization methods successfully capable of learning high-dimensional policy parameters . Both stochastic and deterministic policy search can be useful in applications . A stochastic policy has the effect of smoothing or filtering the policy landscape , which is desirable for optimization . Searching through stochastic policies has enabled the effective control of challenging environments under a general framework ( Schulman et al. , 2015a ; 2017 ) . The same method could either learn robotic movements or play basic games ( 1 ) with minimal domain-specific knowledge , ( 2 ) regardless of function approximation classes , and ( 3 ) with less human intervention ( ignoring reward engineering and hyper-parameter tuning ) ( Duan et al. , 2016 ) . Using stochasticity for exploration , although it imposes approximations and variance , has provided a robust way to actively search for higher rewards . Despite many successes , there are practical environments which remain challenging for current policy gradient methods . For example , non-local rewards ( e.g. , those defined in the frequency domain ) , long time horizons , and naturally-resonant environments all occur in realistic robotic systems ( Kuo & Golnaraghi , 2002 ; Meirovitch , 1975 ; Preumont & Seto , 2008 ) but can present issues for policy gradient search . To tackle challenging environments such as these , this paper considers policy gradient methods based on deterministic policies and deterministic gradient estimates , which could offer advantages by allowing the estimation of global reward gradients on long horizons without the need to inject noise into the system for exploration . To facilitate a dimensionally consistent and low-variance deterministic policy search , a compatible policy gradient estimator and a metric measure for regularization should be employed . For gradient estimation we focus on Vine estimators ( Schulman et al. , 2015a ) , which can be easily applied to deterministic policies . As a metric measure we use the Wasserstein distance , which can measure meaningful distances between deterministic policy functions that have non- overlapping supports ( in contrast to the Kullback-Liebler ( KL ) divergence and the Total Variation ( TV ) distance ) . The Wasserstein metric has seen substantial recent application in a variety of machine-learning domains , such as the successful stable learning of generative adversarial models ( Arjovsky et al. , 2017 ) . Theoretically , this metric has been studied in the context of Lipschitz-continuous Markov decision processes in reinforcement learning ( Hinderer , 2005 ; Ferns et al. , 2012 ) . Pirotta et al . ( 2015 ) defined a policy gradient method using the Wasserestein distance by relying on Lipschitz continuity assumptions with respect to the policy gradient itself . Furthermore , for Lipschitz-continuous Markov decision processes , Asadi et al . ( 2018 ) and Rachelson & Lagoudakis ( 2010 ) used the Wasserstein distance to derive model-based value-iteration and policy-iteration methods , respectively . On a more practical note , Pacchiano et al . ( 2019 ) utilized Wasserstein regularization for behavior-guided stochastic policy optimization . Moreover , Abdullah et al . ( 2019 ) has proposed another robust stochastic policy gradient formulation . Estimating the Wasserstein distance for general distributions is more complicated than typical KL-divergences ( Villani , 2008 ) . This fact constitutes and emphasizes the contributions of Abdullah et al . ( 2019 ) and Pacchiano et al . ( 2019 ) . However , for our deterministic observation-conditional policies , closed-form computation of Wasserstein distances is possible without any approximation . Existing deterministic policy gradient methods ( e.g. , DDPG and TD3 ) use deterministic policies ( Silver et al. , 2014 ; Lillicrap et al. , 2015 ; Fujimoto et al. , 2018 ) , meaning that they learn a deterministic policy function from states to actions . However , such methods still use stochastic search ( i.e. , they add stochastic noise to their deterministic actions to force exploration during policy search ) . In contrast , we will be interested in a method which not only uses deterministic policies , but also uses deterministic search ( i.e. , without constant stochastic noise injection ) . We call this new method truly deterministic policy optimization ( TDPO ) and it may have lower estimation variances and better scalability to long horizons , as we will show in numerical examples . Scalability to long horizons is one of the most challenging aspects for policy gradient methods that use stochastic search . This issue is sometimes referred to as the curse of horizon in reinforcement learning ( Liu et al. , 2018 ) . General worst-case analyses suggests that the sample complexity of reinforcement learning is exponential with respect to the horizon length ( Kakade et al. , 2003 ; Kearns et al. , 2000 ; 2002 ) . Deriving polynomial lower-bounds for the sample complexity of reinforcement learning methods is still an open problem ( Jiang & Agarwal , 2018 ) . Lower-bounding the sample complexity of reinforcement learning for long horizons under different settings and simplifying assumptions has been a topic of theoretical research ( Dann & Brunskill , 2015 ; Wang et al. , 2020 ) . Some recent work has examined the scalability of importance sampling gradient estimators to long horizons in terms of both theoretical and practical estimator variances ( Liu et al. , 2018 ; Kallus & Uehara , 2019 ; 2020 ) . All in all , long horizons are challenging for all reinforcement learning methods , especially the ones suffering from excessive estimation variance due to the use of stochastic policies for exploration , and our truly deterministic method may have advantages in this respect . In this paper we focus on continuous-domain robotic environments with reset capability to previously visited states . The main contributions of this work are : ( 1 ) we introduce a Deterministic Vine ( DeVine ) policy gradient estimator which avoids constant exploratory noise injection ; ( 2 ) we derive a novel deterministically-compatible surrogate function and provide monotonic payoff improvement guarantees ; ( 3 ) we show how to use the DeVine policy gradient estimator with the Wasserstein-based surrogate in a practical algorithm ( TDPO : Truly Deterministic Policy Optimization ) ; ( 4 ) we illustrate the robustness of the TDPO policy search process in robotic control environments with non-local rewards , long horizons , and/or resonant frequencies . 1 BACKGROUND . MDP preliminaries . An infinite-horizon discounted Markov decision process ( MDP ) is specified by ( S , A , P , R , µ , γ ) , where S is the state space , A is the action space , P : S × A → ∆ ( S ) is the transition dynamics , R : S × A → [ 0 , Rmax ] is the reward function , γ ∈ [ 0 , 1 ) is the discount factor , and µ ( s ) is the initial state distribution of interest ( where ∆ ( F ) denotes the set of all probability distributions over F , otherwise known as the Credal set of F ) . The transition dynamics P is defined as an operator which produces a distribution over the state space for the next state s′ ∼ P ( s , a ) . The transition dynamics can be easily generalized to take distributions of states or actions as input ( i.e. , by having P defined as P : ∆ ( S ) ×A → ∆ ( S ) or P : S ×∆ ( A ) → ∆ ( S ) ) . We may abuse the notation and replace δs and δa by s and a , where δs and δa are the deterministic distributions concentrated at the state s and action a , respectively . A policy π : S → ∆ ( A ) specifies a distribution over actions for each state , and induces trajectories from a given starting state s as follows : s1 = s , a1 ∼ π ( s1 ) , r1 = R ( s1 , a1 ) , s2 ∼ P ( s2 , a2 ) , a2 ∼ π ( s2 ) , etc . We will denote trajectories as state-action tuples τ = ( s1 , a1 , s2 , a2 , . . . ) . One can generalize the dynamics ( 1 ) by using a policy instead of an action distribution P ( µs , π ) : = Es∼µs [ Ea∼π ( s ) [ P ( s , a ) ] ] , and ( 2 ) by introducing the t-step transition dynamics recursively as Pt ( µs , π ) : = P ( Pt−1 ( µs , π ) , π ) with P0 ( µs , π ) : = µs , where µs is a distribution over S. The visitation frequency can then be defined as ρπµ : = ( 1− γ ) ∑∞ t=1 γ t−1Pt−1 ( µ , π ) . Table 2 of the appendix summarizes all MDP notation . The value function of π is defined as V π ( s ) : = E [ ∑∞ t=1 γ t−1rt | s1 = s ; π ] . Similarly , one can define Qπ ( s , a ) by conditioning on the first action . The advantage function can then be defined as their difference ( i.e . Aπ ( s , a ) : = Qπ ( s , a ) − V π ( s ) ) . Generally , one can define the advantage/value of one policy with respect to another using Aπ ( s , π′ ) : = E [ Qπ ( s , a ) − V π ( s ) | a ∼ π′ ( ·|s ) ] and Qπ ( s , π′ ) : = E [ Qπ ( s , a ) | a ∼ π′ ( ·|s ) ] . Finally , the payoff of a policy ηπ : = E [ V π ( s ) ; s ∼ µ ] is the average value over the initial states distribution of the MDP . Probabilistic and mathematical notations . Sometimes we refer to ∫ f ( x ) g ( x ) dx integrals as 〈f , g〉x Hilbert space inner products . Assuming that ζ and ν are two probabilistic densities , the Kulback-Liebler ( KL ) divergence is DKL ( ζ|ν ) : = 〈ζ ( x ) , log ( ζ ( x ) ν ( x ) ) 〉x , the Total-Variation ( TV ) distance is TV ( ζ , ν ) = : 12 〈|ζ ( x ) − ν ( x ) | , 1〉x , and the Wasserstein distance is W ( ζ , ν ) = infγ∈Γ ( ζ , ν ) 〈‖x − y‖ , γ ( x , y ) 〉x , y where Γ ( ζ , ν ) is the set of couplings for ζ and ν . We define Lip ( f ( x , y ) ; x ) : = supx ‖∇xf ( x , y ) ‖2 and assume the existence of Lip ( Qπ ( s , a ) ; a ) and ‖Lip ( ∇sQπ ( s , a ) ; a ) ‖2 constants . Under this notation , the Rubinstein-Kantrovich ( RK ) duality states that the |〈ζ ( x ) − ν ( x ) , f ( x ) 〉x| ≤W ( ζ , ν ) · Lip ( f ; x ) bound is tight for all f . For brevity , we may abuse the notation and denote supsW ( π1 ( ·|s ) , π2 ( ·|s ) ) with W ( π1 , π2 ) ( and similarly for other measures ) . For parameterized policies , we define∇πf ( π ) : = ∇θf ( π ) where π is parameterized by the vector θ . Table 1 of the appendix summarizes all these mathematical definitions . Policy gradient preliminaries . The advantage decomposition lemma provides insight into the relationship between payoff improvements and advantages ( Kakade & Langford , 2002 ) . That is , ηπ2 − ηπ1 = 1 1− γ · Es∼ρπ2µ [ A π1 ( s , π2 ) ] . ( 1 ) We will denote the current and the candidate next policy as π1 and π2 , respectively . Taking derivatives of both sides with respect to π2 at π1 yields ∇π2ηπ2 = 1 1− γ [ 〈∇π2ρπ2µ ( · ) , Aπ1 ( · , π1 ) 〉+ 〈ρπ1µ ( · ) , ∇π2Aπ1 ( · , π2 ) 〉 ] . ( 2 ) Since π1 does not have any advantage over itself ( i.e. , Aπ1 ( · , π1 ) = 0 ) , the first term is zero . Thus , the Policy Gradient ( PG ) theorem is derived as ∇π2ηπ2 ∣∣∣ π2=π1 = 1 1− γ · Es∼ρπ1µ [ ∇π2A π1 ( s , π2 ) ] ∣∣∣ π2=π1 . ( 3 ) For policy iteration with function approximation , we assume π2 and π1 to be parameterized by θ2 and θ1 , respectively . One can view the PG theorem as a Taylor expansion of the payoff at θ1 . Conservative Policy Iteration ( CPI ) ( Kakade & Langford , 2002 ) was one of the early dimensionally consistent methods with a surrogate of the form Lπ1 ( π2 ) = ηπ1 + 11−γ · Es∼ρπ1µ [ A π1 ( s , π2 ) ] − C 2 TV 2 ( π1 , π2 ) . The C coefficient guarantees non-decreasing payoffs . However , CPI is limited to linear function approximation classes due to the update rule πnew ← ( 1− α ) πold + απ′ . This lead to the design of the Trust Region Policy Optimization ( TRPO ) ( Schulman et al. , 2015a ) algorithm . TRPO exchanged the bounded squared TV distance with the KL divergence by lower bounding it using the Pinsker inequality . This made TRPO closer to the Natural Policy Gradient algorithm ( Kakade , 2002 ) , and for Gaussian policies the modified terms had similar Taylor expansions within small trust regions . Confined trust regions are a stable way of making large updates and avoiding pessimistic coefficients . For gradient estimates , TRPO used Importance Sampling ( IS ) with a baseline shift : ∇θ2Es∼ρπ1µ [ A π1 ( s , π2 ) ] ∣∣∣ θ2=θ1 = ∇θ2E s∼ρπ1µ a∼π1 ( ·|s ) [ Qπ1 ( s , a ) π2 ( a|s ) π1 ( a|s ) ] ∣∣∣ θ2=θ1 . ( 4 ) While empirical E [ Aπ1 ( s , π2 ) ] and E [ Qπ1 ( s , π2 ) ] estimates yield identical variances in principle , the importance sampling estimator in ( 4 ) imposes larger variances . Later , Proximal Policy Optimization ( PPO ) ( Schulman et al. , 2015b ) proposed utilizing the Generalized Advantage Estimation ( GAE ) method for variance reduction and incorporated first-order smoothing like ADAM ( Kingma & Ba , 2014 ) . GAE employed Temporal-Difference ( TD ) learning ( Bhatnagar et al. , 2009 ) for variance reduction . Although TD-learning was not theoretically guaranteed to converge and could add bias , it improved the estimation accuracy . As an alternative to importance sampling , deterministic policy gradient estimators were also utilized in an actor-critic fashion . Deep Deterministic Policy Gradient ( DDPG ) ( Lillicrap et al. , 2015 ) generalized deterministic gradients by employing Approximate Dynamic Programming ( ADP ) ( Mnih et al. , 2015 ) for variance reduction . Twin Delayed Deterministic Policy Gradient ( TD3 ) ( Fujimoto et al. , 2018 ) improved DDPG ’ s approximation to build an even better policy optimization method . Although both methods used deterministic policies , they still performed stochastic search by adding stochastic noise to the deterministic policies to force exploration . Other lines of stochastic policy optimization were later proposed . Wu et al . ( 2017 ) used a Kroneckerfactored approximation for curvatures . Haarnoja et al . ( 2018 ) proposed a maximum entropy actorcritic method for stochastic policy optimization .

Summary: This paper proposes a new deterministic policy gradient method (TDPO). The main idea and its derivations are based on the use of a deterministic Vine (DeVine) gradient estimator and the Wasserstein metric. The paper shows that a closed-form computation of Wasserstein distance can be derived without any approximation on the deterministic policies. A monotonic policy improvement guarantee is provided. Deterministic Vine (DeVine) is proposed as an optimization method for the proposed surrogate loss function, which is also inspired by the TRPO's surrogate loss. The proposed method is evaluated on two new robotic control domains that are proposed by the authors. The comparisons are against PPO, TRPO, DDPG, and TD3. Additional experiment results on Open AI Gym environments are provided in Appendix.