NADPEx: An on-policy temporally consistent exploration method for deep reinforcement learning

1 INTRODUCTION . Exploration remains a challenge in reinforcement learning , in spite of its recent successes in robotic manipulation and locomotion ( Schulman et al. , 2015b ; Mnih et al. , 2016 ; Duan et al. , 2016 ; Schulman et al. , 2017b ) . Most reinforcement learning algorithms explore with stochasticity in stepwise action space and suffer from low learning efficiency in environments with sparse rewards ( Florensa et al. , 2017 ) or long information chain ( Osband et al. , 2017 ) . In these environments , temporally consistent exploration is needed to acquire useful learning signals . Though in off-policy methods , autocorrelated noises ( Lillicrap et al. , 2015 ) or separate samplers ( Xu et al. , 2018 ) could be designed for consistent exploration , on-policy exploration strategies tend to be learnt alongside with the optimal policy , which is non-trivial . A complementary objective term should be constructed because the purpose of exploration to optimize informational value of possible trajectories ( Osband et al. , 2016 ) is not contained directly in the reward of underlying Markov Decision Process ( MDP ) . This complementary objective is known as reward shaping e.g . optimistic towards uncertainty heuristic and intrinsic rewards . However , most of them require complex additional structures and strong human priors when state and action spaces are intractable , and introduce unadjustable bias in end-to-end learning ( Bellemare et al. , 2016 ; Ostrovski et al. , 2017 ; Tang et al. , 2017 ; Fu et al. , 2017 ; Houthooft et al. , 2016 ; Pathak et al. , 2017 ) . It would not be necessary though , to teach agents to explore consistently through reward shaping , if the policy model inherits it as a prior . This inspires ones to disentangle policy stochasticity , with a structure of time scales . One possible solution is policy parameter perturbation in a large time scale . Though previous attempts were restricted to linear function approximators ( Rückstieß et al. , 2008 ; Osband et al. , 2014 ) , progress has been made with neural networks , through either network section duplication ( Osband et al. , 2016 ) or adaptive-scale parameter noise injection ( Plappert et al. , 2017 ; Fortunato et al. , 2017 ) . However , in Osband et al . ( 2016 ) the episode-wise stochasticity is unadjustable , and the duplicated modules do not cooperate with each other . Directly optimizing the mean of distribution of all parameters , Plappert et al . ( 2017 ) adjusts the stochasticity to the learning progress heristically . Besides , as all sampled parameters need to be stored in on-policy exploration , it is not directly salable to large neural networks , where the trade-off between large memory for multiple networks and high variance with sinlge network is non-trivial to make . This work proposes Neural Adaptive Dropout Policy Exploration ( NADPEx ) as a simple , scalable and improvement-guaranteed method for on-policy temporally consistent exploration , which generalizes Osband et al . ( 2016 ) and Plappert et al . ( 2017 ) . Policy stochasticity is disentangled into two time scales , step-wise action noise and episode-wise stochasticity modeled with a distribution of subnetworks . With one single subnetwork in an episode , it achieves inherent temporal consistency with only a little computational overhead and no crafted reward , even in environments with sparse rewards . And with a set of subnetworks in one sampling batch , agents experience diverse behavioral patterns . This sampling of subnetworks is executed through dropout ( Hinton et al. , 2012 ; Srivastava et al. , 2014 ) , which encourages composability of constituents and facilitates the emergence of diverse maneuvers . As dropout could be excerted on connections , neurons , modules and paths , NADPEx naturally extends to neural networks with various sizes , exploiting modularity at various levels . To align separately parametrized stochasticity to each other , this sampling is made differentiable for all possible dropout candidates . We further discuss the effect of a first-order KL regularizer on dropout policies to improve stability and guarantee policy improvement . Our experiments demonstrate that NAPDEx solves challenging exploration tasks with sparse rewards while achieving as efficient or even faster convergence in the standard mujoco benchmark for state-of-theart PPO agents , which we believe can be generalized to any deep reinforcement learning agents . 2 PRELIMINARIES . We consider a standard discrete-time finite-horizon discounted Markov Decision Process ( MDP ) setting for reinforcement learning , represented by a tupleM = ( S , A , P , r , ρ0 , γ , T ) , with a state set S , an action setA , a transitional probability distribution P : S×A×S → R+ , a bounded reward function r : S × A → R+ , an initial state distribution ρ0 : S → R+ , a discount factor γ ∈ [ 0 , 1 ] and horizon T . We denote a policy parametrized by θ as πθ : S×A → R+ . The optimization objective of this policy is to maximize the expected discounted return η ( πθ ) = Eτ [ ∑T t=0 γ tr ( st , at ) ] , where τ = ( s0 , a0 , ... ) denotes the whole trajectory , s0 ∼ ρ0 ( s0 ) , at ∼ πθ ( at|st ) and st+1 ∼ P ( st+1|st , at ) . In experimental evaluation , we follow normal setting and use undiscounted return Eτ [ ∑T t=0 r ( st , at ) ] . 2.1 PROXIMAL POLICY OPTIMIZATION ( PPO ) . Theoretically speaking , NADPEx works with any policy gradient based algorithms . In this work , we consider the recent advance in on-policy policy gradient method , proximal policy optimization ( PPO ) ( Schulman et al. , 2017b ) . Policy gradient methods were first proposed by Williams ( 1992 ) in REINFORCE algorithm to maximize the aforementioned objective in a gradient ascent manner . The gradient of an expectation ∇θη ( πθ ) is approximated with a Monte Carlo average of policy gradient : ∇θη ( πθ ) ≈ 1 NT N∑ i=1 T∑ t=o ∇θ log πθ ( ait|sit ) ( Rit − bit ) , ( 1 ) where N is the number of episodes in this batch , Rit = ∑T t′=t γ t′−trit′ and b i t is a baseline for variance reduction . Schulman et al . ( 2015b ) proposes a policy iteration algorithm and proves its monotonicity in improvement . With the πθ ’ s corresponding discounted state-visitation frequency ρθ , the sampling policy πθold and the updated policy πθ , it solves the following constrained optimization problem with a line search for an appropriate step size within certain bound δKL , called trust region : argmaxθ Es∼ρ θold , a∼π θold [ πθ ( a|s ) πθold ( a|s ) Aθold ( s , a ) ] s.t . Es∼ρ θold [ DKL ( π old θ ( ·|s ) |πθ ( ·|s ) ) ] ≤ δKL ( 2 ) whereDKL ( ·||· ) is the Kullback-Leibler ( KL ) divergence , Aθold ( s , a ) is calculated with Generalized Advantage Estimation ( GAE ) ( Schulman et al. , 2015c ) . Proximal Policy Optimization ( PPO ) transforms ( 2 ) to a unconstrained optimization and solves it with first-order derivatives , embracing established stochastic gradient-based optimizer like Adam ( Kingma & Ba , 2014 ) . Noting that DKL ( πoldθ ( ·|s ) |πθ ( ·|s ) ) is actually a relaxation of total variational divergence according to Schulman et al . ( 2015b ) , whose first-order derivative is 0 when θ is close to θold , DKL ( πθ ( ·|s ) ||πoldθ ( ·|s ) ) is a natural replacement . Combining the first-order derivative of DKL ( πθ ( ·|s ) ||πoldθ ( ·|s ) ) ( Schulman et al. , 2017a ) ( check Appendix A for a proof ) : ∇θDKL ( πθ ( ·|s ) |πθold ( ·|s ) ) ] = Ea∼πθ [ ∇θ log πθ ( a|s ) ( log πθ ( a|s ) − log πθold ( a|s ) ) ] , ( 3 ) PPO optimizes the following unconstrained objective , called KL PPO loss1 : LKL = Es∼ρ θold , a∼π θold [ rθ ( s , a ) Aθold ( s , a ) − β 2 ( log πθ ( a|s ) − log πθold ( a|s ) ) 2 ] , ( 4 ) where rθ ( s , a ) = πθ ( a|s ) π θold ( a|s ) . Schulman et al . ( 2017b ) also proposes a clipping version PPO , as a lower bound to ( 4 ) : Lclip = Es∼ρ θold , a∼π θold [ min ( rθ ( s , a ) , clip ( rθ ( s , a ) , 1− , 1 + ) ) Aθold ( s , a ) ] ( 5 ) In this work , we try both KL PPO and clipping PPO . 2.2 DROPOUT . Dropout is a technique used in deep learning to prevent features from co-adaptation and parameters from overfitting , by randomly dropping some hidden neuron units in each round of feed-forward and back-propagation ( Hinton et al. , 2012 ; Srivastava et al. , 2014 ) . This is modeled by multiplying a Bernoulli random variable zkj to each hidden unit , i.e . neuron activation h k j , for j = 1 ... m , where m is the number of hidden units in kth layer . Then the neuron activation of the k + 1th layer is hk+1 = σ ( W ( k+1 ) TDkzh k + b ( k+1 ) ) , ( 6 ) where Dz = diag ( z ) ∈ Rm×m , W ( k+1 ) and b ( k+1 ) are weights and biases at k + 1th layer respectively and we simply denote them with θ .= < W , b > . σ ( x ) is the nonlinear neuron activation function . The parameter of this Bernoulli random variable is pkj . = P ( zkj = 0 ) , aka the dropout rate of the jth hidden unit at kth layer . In supervised learning , these dropout parameters are normally fixed during training in some successful practice ( Hinton et al. , 2012 ; Wan et al. , 2013 ) . And there are some variants to dropout connections , modules or paths ( Wan et al. , 2013 ) . 1Though the concrete form is not provided in Schulman et al . ( 2017b ) , it is given in Dhariwal et al . ( 2017 ) publicly released by the authors 3 METHODOLOGY . 3.1 NEURAL ADAPTIVE DROPOUT POLICY EXPLORATION ( NADPEX ) . Designing an exploration strategy is to introduce a kind of stochasticity during reinforcement learning agents ’ interaction with the environment to help them get rid of some local optima . While action space noise ( parametrized as a stochastic policy πθ ) might be sufficient in environments where stepwise rewards are provided , they have limited effectiveness in more complex environments ( Florensa et al. , 2017 ; Plappert et al. , 2017 ) . As their complement , an exploration strategy would be especially beneficial if it can help either sparse reward signals ( Florensa et al. , 2017 ) or significant long-term information ( Osband et al. , 2016 ) to be acquired and back-propagated through time ( BPTT ) ( Sutton & Barto , 1998 ) . This motivates us to introduce a hierarchy of stochasticity and capture temporal consistency with a separate parametrization . Our algorithm , Neural Adaptive Dropout Policy Exploration ( NADPEx ) , models stochasticity at large time scales with a distribution of plausible subnetworks . For each episode , one specific subnetworks is drawn from this distribution . Inspected from a large time scale , this encourages exploration towards different directions among different episodes in one batch of sampling . And from a small time scale , temporal correlation is enforced for consistent exploration . Policies with a hierarchy of stochasticity is believed to represent a complex action distribution and larger-scale behavioral patterns ( Florensa et al. , 2017 ) . We achieve the drawing of subnetwork through dropout . As introduced in Section 2.2 , originally , dropout is modeled through multiplying a binary random variable zkj ∼ Bernoulli ( 1− pkj ) to each neuron activation hkj . In Srivastava et al . ( 2014 ) , this binary dropout is softened as continuous random variable dropout , modeled with a Gaussian random variable z ∼ N ( I , σ2 ) , normally referred to as Gaussian multiplicative dropout . Here we denote both distributions with qφ ( z ) . Later we will introduce how both of them could be made differentibale and thus adaptative to learning progress . During the sampling process , at the beginning of every episode τ i , a vector of dropout random variables zi is sampled from qφ ( z ) , giving us a dropout policy πθ|zi for step-wise action distribution . zi is fed to the stochastic computation graph ( Schulman et al. , 2015a ) for the whole episode until termination , when a new round of drawing initiates . Similar as observations , actions and rewards , this random variable is stored as sampled data , which will be fed back to the stochastic computation graph during training . Policies with this hierarchy of stochasticity , i.e . NADPEx policies , can be represented as a joint distribution : πθ , φ ( · , z ) = qφ ( z ) πθ|z ( · ) , ( 7 ) making the the objective : η ( πθ , φ ) = Ez∼q ( z ) [ Eτ |z [ T∑ t=0 γtr ( st , at ) ] ] . ( 8 ) If we only use the stochasticity of network architecture as a bootstrap , as in BootstrapDQN ( Osband et al. , 2016 ) , the bootstrapped policy gradient training is to update the network parameters θ with the following gradients : ∇θη ( πθ , φ ) = ∇θEz∼qφ ( z ) [ η ( πθ|z ) ] = Ez∼qφ ( z ) [ ∇θη ( πθ|z ) ] ≈ 1 N N∑ i=1 ∇θL ( θ , zi ) ( 9 ) L ( θ , zi ) is the surrogate loss of dropout policy πθ , zi , variates according to the specific type of reinforcement learning algorithm . In next subsection , we discuss some pitfalls of bootstrap and provide and our solution to it .

Summary: The authors introduce a  novel  on-policy  temporally  consistent  exploration  strategy, named Neural  AdaptiveDropout Policy Exploration (NADPEx), for deep reinforcement learning agents. The main idea is to sample from a distribution of plausible subnetworks modeling the temporally consistent exploration. For this, the authors use the ideas of the standard dropout for deep networks. Using the proposed  dropout transformation that is differentiable, the authors show that the KL regularizers on policy-space play an important role in stabilizing its learning. The experimental validation is performed on continuous control learning tasks, showing the benefits of the proposed.