Deep Coherent Exploration For Continuous Control

1 INTRODUCTION . The balance of exploration and exploitation ( Kearns & Singh , 2002 ; Jaksch et al. , 2010 ) is a longstanding challenge in reinforcement learning ( RL ) . With insufficient exploration , states and actions with high rewards can be missed , resulting in policies prematurely converging to bad local optima . In contrast , with too much exploration , agents could waste their resources trying suboptimal states and actions , without leveraging their experiences efficiently . To learn successful strategies , this trade-off between exploration and exploitation must be balanced well , and this is known as the exploration vs. exploitation dilemma . At a high level , exploration can be divided into directed strategies and undirected strategies ( Thrun , 1992 ; Plappert et al. , 2018 ) . While directed strategies aim to extract useful information from existing experiences for better exploration , undirected strategies rely on injecting randomness into the agent ’ s decision-making . Over the years , many sophisticated directed exploration strategies have been proposed ( Tang et al. , 2016 ; Ostrovski et al. , 2017 ; Houthooft et al. , 2016 ; Pathak et al. , 2017 ) . However , since these strategies still require lower-level exploration to collect the experiences , or are either complicated or computationally intensive , undirected exploration strategies are still commonly used in RL literature in practice , where some well-known examples are -greedy ( Sutton , 1995 ) for discrete action space and additive Gaussian noise for continuous action space ( Williams , 1992 ) . Such strategies explore by randomly perturbing agents ’ actions at different steps independently and hence are referred to as performing step-based exploration in action space ( Deisenroth et al. , 2013 ) . As an alternative to those exploration strategies in action space , exploration by perturbing the weights of linear policies has been proposed ( Rückstieß et al. , 2010 ; Sehnke et al. , 2010 ; Kober & Peters , 2008 ) . Since these strategies in parameter space naturally explore conditioned on the states and are usually trajectory-based ( only perturb the weights at the beginning of each trajectory ) ( Deisenroth et al. , 2013 ) , they have the advantages of being more consistent , structured , and global ( Deisenroth et al. , 2013 ) . Later , van Hoof et al . ( 2017 ) proposed a generalized exploration ( GE ) scheme , bridging the gap between step-based and trajectory-based exploration in parameter space . With the advance of deep RL , NoisyNet ( Fortunato et al. , 2018 ) and Parameter Space Noise for Exploration ( PSNE ) ( Plappert et al. , 2018 ) were introduced , extending parameter-space exploration strategies for policies using deep neural networks . Although GE , NoisyNet , and PSNE improved over the vanilla exploration strategies in parameter space and were shown leading to more global and consistent exploration , they still suffer from several limitations . Given this , we propose a new exploration scheme with the following characteristics . 1 . Generalizing Step-based and Trajectory-based Exploration ( van Hoof et al. , 2017 ) Since both NoisyNet and PSNE are trajectory-based exploration strategies , they are considered relatively inefficient and bring insufficient stochasticity ( Deisenroth et al. , 2013 ) . Following van Hoof et al . ( 2017 ) , our method improves by interpolating between step-based and trajectory-based exploration in parameter space , where a more balanced trade-off between stability and stochasticity can be achieved . 2 . Recursive Analytical Integration of Latent Exploring Policies NoisyNet and PSNE address the uncertainty from sampling exploring policies using Monte Carlo integration , while GE uses analytical integration on full trajectories , which scales poorly in the number of time steps . In contrast , we apply analytical and recurrent integration after each step , which leads to low-variance and scalable updates . 3 . Perturbing Last Layers of Policy Networks Both NoisyNet and PSNE perturb all layers of the policy network . However , in general , only the uncertainty in parameters of the last ( linear ) layer can be integrated analytically . Furthermore , it is not clear that deep neural networks can be perturbed in meaningful ways for exploration ( Plappert et al. , 2018 ) . We thus propose and evaluate an architecture where perturbation is only applied on the parameters of the last layer . These characteristics define our contribution , which we will refer to as Deep Coherent Exploration . We evaluate the coherent versions of A2C ( Mnih et al. , 2016 ) , PPO ( Schulman et al. , 2017 ) , and SAC ( Haarnoja et al. , 2018 ) , where the experiments on OpenAI MuJoCo ( Todorov et al. , 2012 ; Brockman et al. , 2016 ) tasks show that Deep Coherent Exploration outperforms other exploration strategies in terms of both learning speed and stability . 2 RELATED WORK . As discussed , exploration can broadly be classified into directed and undirected strategies ( Thrun , 1992 ; Plappert et al. , 2018 ) , with undirected strategies being commonly used in practice because of their simplicity . Well known methods such as -greedy ( Sutton , 1995 ) or additive Gaussian noise ( Williams , 1992 ) randomly perturb the action at each time step independently . These high-frequency perturbations , however , can result in poor coverage of the state-action space due to random-walk behavior ( Rückstieß et al. , 2010 ; Deisenroth et al. , 2013 ) , washing-out of exploration by the environment dynamics ( Kober & Peters , 2008 ; Rückstieß et al. , 2010 ; Deisenroth et al. , 2013 ) , and to potential damage to mechanical systems ( Koryakovskiy et al. , 2017 ) . One alternative is to instead perturb the policy in parameter space , with the perturbation held constant for the duration of a trajectory . Rückstieß et al . ( 2010 ) and Sehnke et al . ( 2010 ) showed that such parameter-space methods could bring improved exploration behaviors because of reduced variance and faster convergence , when combined with REINFORCE ( Williams , 1992 ) or Natural Actor-Critic ( Peters et al. , 2005 ) . Another alternative to independent action-space perturbation , is to correlate the noise applied at subsequent actions ( Morimoto & Doya , 2000 ; Wawrzynski , 2015 ; Lillicrap et al. , 2016 ) , for example by generating perturbations from an Ornstein-Uhlenbeck ( OU ) process ( Uhlenbeck & Ornstein , 1930 ) . Later , van Hoof et al . ( 2017 ) used the same stochastic process but in the parameter space of the policy . This approach uses a temporally coherent exploring policy , which unifies step-based and trajectory-based exploration . Moreover , the author showed that , with linear policies , a more delicate balance between these two extreme strategies could have better performance . However , this approach was derived in a batch mode setting and requires storing the full trajectory history and the inversion of a matrix growing with the number of time step . Thus , it does not scale well to long trajectories or complex models . Although these methods pioneered the research of exploration in parameter space , their applicability is limited . More precisely , these methods were only evaluated with extremely shallow ( often linear ) policies and relatively simple tasks with low-dimensional state spaces and action spaces . Given this , NoisyNet ( Fortunato et al. , 2018 ) , PSNE ( Plappert et al. , 2018 ) and Stochastic A3C ( SA3C ) ( Shang et al. , 2019 ) were proposed , introducing more general and scalable methods for deep RL algorithms . All three of these methods can be seen as learning a distribution over policies for trajectory-based exploration in parameter space . These exploring policies are sampled by perturbing the weights across all layers of a deep neural network , with the uncertainty from sampling being addressed by Monte Carlo integration . Whereas NoisyNet learns the magnitudes of the noise for each parameter , PSNE heuristically adapts a single magnitude for all parameters . While showing good performance in practice ( Fortunato et al. , 2018 ; Plappert et al. , 2018 ) , these methods suffer from two potential limitations . Firstly , trajectory-based strategies can be inefficient as only one strategy can be evaluated for a potentially long trajectory ( Deisenroth et al. , 2013 ) , which could result in a failure to escape local optima . Secondly , Monte Carlo integration results in high-variance gradient estimates which could lead to oscillating updates . 3 BACKGROUND . This section provides background for reinforcement learning and related deep RL algorithms . 3.1 REINFORCEMENT LEARNING . Reinforcement learning is a sub-field of machine learning that studies how an agent learns strategies with high returns through trial-and-error by interacting with an environment . This interaction between an agent and an environment is described using Markov Decision Processes ( MDPs ) . A MDP is a tuple ( S , A , r , P , γ ) , where S is the state space , A is the action space , r : S × A × S → R is the reward function with rt = r ( st , at , st+1 ) , P : S × A × S → R+ is the transition probability function , and γ is a discount factor indicating the preference of short-term rewards . In RL with continuous action space , an agent aims to learn a parametrized ( e.g . Gaussian ) policy πθ ( a|s ) : S ×A → R+ , with parameters θ , that maximizes the expected return over trajectories : J ( θ ) = Eτ∼p ( τ |πθ ) [ R ( τ ) ] , ( 1 ) where τ = ( s0 , a0 , ... , aT−1 , sT ) is a trajectory and R ( τ ) = ∑T t=0 γ trt is the discounted return . 3.2 DEEP REINFORCEMENT LEANING ALGORITHMS . Deep reinforcement learning combines deep learning and reinforcement learning , where policies and value functions are represented by deep neural networks for more sophisticated and powerful function approximation . In our experiments , we consider the following three deep RL algorithms . Advantage Actor-Critic ( A2C ) Closely related to REINFORCE ( Williams , 1992 ) , A2C is an on-policy algorithm proposed as the synchronous version of the original Asynchronous Advantage Actor-Critic ( A3C ) ( Mnih et al. , 2016 ) . The gradient of A2C can be written as : ∇θJ ( θ ) = Eτ∼p ( τ |θ ) [ T−1∑ t=0 ∇θ log πθ ( at|st ) Aπθ ( st , at ) ] , ( 2 ) where Aπθ ( st , at ) is the estimated advantage following policy πθ . Proximal Policy Optimization ( PPO ) PPO is an on-policy algorithm developed to determine the largest step for update while still keeping the updated policy close to the old policy in terms of Kullback–Leibler ( KL ) divergence . Instead of using a second-order method as in Trust Region Policy Optimization ( TRPO ) ( Schulman et al. , 2015 ) , PPO applies a first-order method and combines several tricks to relieve the complexity . We consider the primary variant of PPO-Clip with the following surrogate objective : LCLIPθk ( θ ) = Eτ∼p ( τ |θk ) [ T−1∑ t=0 [ min ( rt ( θ ) , clip ( rt ( θ ) , 1− , 1 + ) ) A πθk t ] ] , ( 3 ) where rt ( θ ) = πθ ( at|st ) πθk ( at|st ) and is a small threshold that approximately restricts the distance between the new policy and the old policy . In practice , to prevent the new policy from changing too fast , the KL divergence from the new policy to the old policy approximated on a sampled batch is often used as a further constraint . Soft Actor-Critic ( SAC ) As an entropy-regularized ( Ziebart et al. , 2008 ) off-policy actor-critic method ( Lillicrap et al. , 2016 ; Fujimoto et al. , 2018 ) with a stochastic policy , SAC ( Haarnoja et al. , 2018 ) learns the optimal entropy-regularized Q-function through ‘ soft ’ Bellman back-ups with offpolicy data : Qπ ( s , a ) = Es′∼p ( s′|s , a ) , ã′∼π ( ã′|s′ ) [ r + γ ( Qπ ( s′ , ã′ ) + αH ( π ( ã′|s′ ) ) ) ] , ( 4 ) whereH is the entropy and α is the temperature parameter . The policy is then learned by maximizing the expected maximum entropy V -function via the reparameterization trick ( Kingma et al. , 2015 ) .

Summary: This paper focuses on undirected exploration strategies in reinforcement learning. Following the prior work, this paper proposes an exploration method unifying the step-based and trajectory-based exploration. The authors propose to perturb only the last(linear) layer of the policy for exploration, instead of perturbing all layers of the policy network. Also, the authors use analytical and recurrent integration for policy updates. Experiments show that the proposed exploration strategy mostly helps A2C, PPO and SAC in three Mujoco environments.