NADPEx: An on-policy temporally consistent exploration method for deep reinforcement learning

1 INTRODUCTION . Exploration remains a challenge in reinforcement learning , in spite of its recent successes in robotic manipulation and locomotion ( Schulman et al. , 2015b ; Mnih et al. , 2016 ; Duan et al. , 2016 ; Schulman et al. , 2017b ) . Most reinforcement learning algorithms explore with stochasticity in stepwise action space and suffer from low learning efficiency in environments with sparse rewards ( Florensa et al. , 2017 ) or long information chain ( Osband et al. , 2017 ) . In these environments , temporally consistent exploration is needed to acquire useful learning signals . Though in off-policy methods , autocorrelated noises ( Lillicrap et al. , 2015 ) or separate samplers ( Xu et al. , 2018 ) could be designed for consistent exploration , on-policy exploration strategies tend to be learnt alongside with the optimal policy , which is non-trivial . A complementary objective term should be constructed because the purpose of exploration to optimize informational value of possible trajectories ( Osband et al. , 2016 ) is not contained directly in the reward of underlying Markov Decision Process ( MDP ) . This complementary objective is known as reward shaping e.g . optimistic towards uncertainty heuristic and intrinsic rewards . However , most of them require complex additional structures and strong human priors when state and action spaces are intractable , and introduce unadjustable bias in end-to-end learning ( Bellemare et al. , 2016 ; Ostrovski et al. , 2017 ; Tang et al. , 2017 ; Fu et al. , 2017 ; Houthooft et al. , 2016 ; Pathak et al. , 2017 ) . It would not be necessary though , to teach agents to explore consistently through reward shaping , if the policy model inherits it as a prior . This inspires ones to disentangle policy stochasticity , with a structure of time scales . One possible solution is policy parameter perturbation in a large time scale . Though previous attempts were restricted to linear function approximators ( Rückstieß et al. , 2008 ; Osband et al. , 2014 ) , progress has been made with neural networks , through either network section duplication ( Osband et al. , 2016 ) or adaptive-scale parameter noise injection ( Plappert et al. , 2017 ; Fortunato et al. , 2017 ) . However , in Osband et al . ( 2016 ) the episode-wise stochasticity is unadjustable , and the duplicated modules do not cooperate with each other . Directly optimizing the mean of distribution of all parameters , Plappert et al . ( 2017 ) adjusts the stochasticity to the learning progress heristically . Besides , as all sampled parameters need to be stored in on-policy exploration , it is not directly salable to large neural networks , where the trade-off between large memory for multiple networks and high variance with sinlge network is non-trivial to make . This work proposes Neural Adaptive Dropout Policy Exploration ( NADPEx ) as a simple , scalable and improvement-guaranteed method for on-policy temporally consistent exploration , which generalizes Osband et al . ( 2016 ) and Plappert et al . ( 2017 ) . Policy stochasticity is disentangled into two time scales , step-wise action noise and episode-wise stochasticity modeled with a distribution of subnetworks . With one single subnetwork in an episode , it achieves inherent temporal consistency with only a little computational overhead and no crafted reward , even in environments with sparse rewards . And with a set of subnetworks in one sampling batch , agents experience diverse behavioral patterns . This sampling of subnetworks is executed through dropout ( Hinton et al. , 2012 ; Srivastava et al. , 2014 ) , which encourages composability of constituents and facilitates the emergence of diverse maneuvers . As dropout could be excerted on connections , neurons , modules and paths , NADPEx naturally extends to neural networks with various sizes , exploiting modularity at various levels . To align separately parametrized stochasticity to each other , this sampling is made differentiable for all possible dropout candidates . We further discuss the effect of a first-order KL regularizer on dropout policies to improve stability and guarantee policy improvement . Our experiments demonstrate that NAPDEx solves challenging exploration tasks with sparse rewards while achieving as efficient or even faster convergence in the standard mujoco benchmark for state-of-theart PPO agents , which we believe can be generalized to any deep reinforcement learning agents . 2 PRELIMINARIES . We consider a standard discrete-time finite-horizon discounted Markov Decision Process ( MDP ) setting for reinforcement learning , represented by a tupleM = ( S , A , P , r , ρ0 , γ , T ) , with a state set S , an action setA , a transitional probability distribution P : S×A×S → R+ , a bounded reward function r : S × A → R+ , an initial state distribution ρ0 : S → R+ , a discount factor γ ∈ [ 0 , 1 ] and horizon T . We denote a policy parametrized by θ as πθ : S×A → R+ . The optimization objective of this policy is to maximize the expected discounted return η ( πθ ) = Eτ [ ∑T t=0 γ tr ( st , at ) ] , where τ = ( s0 , a0 , ... ) denotes the whole trajectory , s0 ∼ ρ0 ( s0 ) , at ∼ πθ ( at|st ) and st+1 ∼ P ( st+1|st , at ) . In experimental evaluation , we follow normal setting and use undiscounted return Eτ [ ∑T t=0 r ( st , at ) ] . 2.1 PROXIMAL POLICY OPTIMIZATION ( PPO ) . Theoretically speaking , NADPEx works with any policy gradient based algorithms . In this work , we consider the recent advance in on-policy policy gradient method , proximal policy optimization ( PPO ) ( Schulman et al. , 2017b ) . Policy gradient methods were first proposed by Williams ( 1992 ) in REINFORCE algorithm to maximize the aforementioned objective in a gradient ascent manner . The gradient of an expectation ∇θη ( πθ ) is approximated with a Monte Carlo average of policy gradient : ∇θη ( πθ ) ≈ 1 NT N∑ i=1 T∑ t=o ∇θ log πθ ( ait|sit ) ( Rit − bit ) , ( 1 ) where N is the number of episodes in this batch , Rit = ∑T t′=t γ t′−trit′ and b i t is a baseline for variance reduction . Schulman et al . ( 2015b ) proposes a policy iteration algorithm and proves its monotonicity in improvement . With the πθ ’ s corresponding discounted state-visitation frequency ρθ , the sampling policy πθold and the updated policy πθ , it solves the following constrained optimization problem with a line search for an appropriate step size within certain bound δKL , called trust region : argmaxθ Es∼ρ θold , a∼π θold [ πθ ( a|s ) πθold ( a|s ) Aθold ( s , a ) ] s.t . Es∼ρ θold [ DKL ( π old θ ( ·|s ) |πθ ( ·|s ) ) ] ≤ δKL ( 2 ) whereDKL ( ·||· ) is the Kullback-Leibler ( KL ) divergence , Aθold ( s , a ) is calculated with Generalized Advantage Estimation ( GAE ) ( Schulman et al. , 2015c ) . Proximal Policy Optimization ( PPO ) transforms ( 2 ) to a unconstrained optimization and solves it with first-order derivatives , embracing established stochastic gradient-based optimizer like Adam ( Kingma & Ba , 2014 ) . Noting that DKL ( πoldθ ( ·|s ) |πθ ( ·|s ) ) is actually a relaxation of total variational divergence according to Schulman et al . ( 2015b ) , whose first-order derivative is 0 when θ is close to θold , DKL ( πθ ( ·|s ) ||πoldθ ( ·|s ) ) is a natural replacement . Combining the first-order derivative of DKL ( πθ ( ·|s ) ||πoldθ ( ·|s ) ) ( Schulman et al. , 2017a ) ( check Appendix A for a proof ) : ∇θDKL ( πθ ( ·|s ) |πθold ( ·|s ) ) ] = Ea∼πθ [ ∇θ log πθ ( a|s ) ( log πθ ( a|s ) − log πθold ( a|s ) ) ] , ( 3 ) PPO optimizes the following unconstrained objective , called KL PPO loss1 : LKL = Es∼ρ θold , a∼π θold [ rθ ( s , a ) Aθold ( s , a ) − β 2 ( log πθ ( a|s ) − log πθold ( a|s ) ) 2 ] , ( 4 ) where rθ ( s , a ) = πθ ( a|s ) π θold ( a|s ) . Schulman et al . ( 2017b ) also proposes a clipping version PPO , as a lower bound to ( 4 ) : Lclip = Es∼ρ θold , a∼π θold [ min ( rθ ( s , a ) , clip ( rθ ( s , a ) , 1− , 1 + ) ) Aθold ( s , a ) ] ( 5 ) In this work , we try both KL PPO and clipping PPO . 2.2 DROPOUT . Dropout is a technique used in deep learning to prevent features from co-adaptation and parameters from overfitting , by randomly dropping some hidden neuron units in each round of feed-forward and back-propagation ( Hinton et al. , 2012 ; Srivastava et al. , 2014 ) . This is modeled by multiplying a Bernoulli random variable zkj to each hidden unit , i.e . neuron activation h k j , for j = 1 ... m , where m is the number of hidden units in kth layer . Then the neuron activation of the k + 1th layer is hk+1 = σ ( W ( k+1 ) TDkzh k + b ( k+1 ) ) , ( 6 ) where Dz = diag ( z ) ∈ Rm×m , W ( k+1 ) and b ( k+1 ) are weights and biases at k + 1th layer respectively and we simply denote them with θ .= < W , b > . σ ( x ) is the nonlinear neuron activation function . The parameter of this Bernoulli random variable is pkj . = P ( zkj = 0 ) , aka the dropout rate of the jth hidden unit at kth layer . In supervised learning , these dropout parameters are normally fixed during training in some successful practice ( Hinton et al. , 2012 ; Wan et al. , 2013 ) . And there are some variants to dropout connections , modules or paths ( Wan et al. , 2013 ) . 1Though the concrete form is not provided in Schulman et al . ( 2017b ) , it is given in Dhariwal et al . ( 2017 ) publicly released by the authors 3 METHODOLOGY . 3.1 NEURAL ADAPTIVE DROPOUT POLICY EXPLORATION ( NADPEX ) . Designing an exploration strategy is to introduce a kind of stochasticity during reinforcement learning agents ’ interaction with the environment to help them get rid of some local optima . While action space noise ( parametrized as a stochastic policy πθ ) might be sufficient in environments where stepwise rewards are provided , they have limited effectiveness in more complex environments ( Florensa et al. , 2017 ; Plappert et al. , 2017 ) . As their complement , an exploration strategy would be especially beneficial if it can help either sparse reward signals ( Florensa et al. , 2017 ) or significant long-term information ( Osband et al. , 2016 ) to be acquired and back-propagated through time ( BPTT ) ( Sutton & Barto , 1998 ) . This motivates us to introduce a hierarchy of stochasticity and capture temporal consistency with a separate parametrization . Our algorithm , Neural Adaptive Dropout Policy Exploration ( NADPEx ) , models stochasticity at large time scales with a distribution of plausible subnetworks . For each episode , one specific subnetworks is drawn from this distribution . Inspected from a large time scale , this encourages exploration towards different directions among different episodes in one batch of sampling . And from a small time scale , temporal correlation is enforced for consistent exploration . Policies with a hierarchy of stochasticity is believed to represent a complex action distribution and larger-scale behavioral patterns ( Florensa et al. , 2017 ) . We achieve the drawing of subnetwork through dropout . As introduced in Section 2.2 , originally , dropout is modeled through multiplying a binary random variable zkj ∼ Bernoulli ( 1− pkj ) to each neuron activation hkj . In Srivastava et al . ( 2014 ) , this binary dropout is softened as continuous random variable dropout , modeled with a Gaussian random variable z ∼ N ( I , σ2 ) , normally referred to as Gaussian multiplicative dropout . Here we denote both distributions with qφ ( z ) . Later we will introduce how both of them could be made differentibale and thus adaptative to learning progress . During the sampling process , at the beginning of every episode τ i , a vector of dropout random variables zi is sampled from qφ ( z ) , giving us a dropout policy πθ|zi for step-wise action distribution . zi is fed to the stochastic computation graph ( Schulman et al. , 2015a ) for the whole episode until termination , when a new round of drawing initiates . Similar as observations , actions and rewards , this random variable is stored as sampled data , which will be fed back to the stochastic computation graph during training . Policies with this hierarchy of stochasticity , i.e . NADPEx policies , can be represented as a joint distribution : πθ , φ ( · , z ) = qφ ( z ) πθ|z ( · ) , ( 7 ) making the the objective : η ( πθ , φ ) = Ez∼q ( z ) [ Eτ |z [ T∑ t=0 γtr ( st , at ) ] ] . ( 8 ) If we only use the stochasticity of network architecture as a bootstrap , as in BootstrapDQN ( Osband et al. , 2016 ) , the bootstrapped policy gradient training is to update the network parameters θ with the following gradients : ∇θη ( πθ , φ ) = ∇θEz∼qφ ( z ) [ η ( πθ|z ) ] = Ez∼qφ ( z ) [ ∇θη ( πθ|z ) ] ≈ 1 N N∑ i=1 ∇θL ( θ , zi ) ( 9 ) L ( θ , zi ) is the surrogate loss of dropout policy πθ , zi , variates according to the specific type of reinforcement learning algorithm . In next subsection , we discuss some pitfalls of bootstrap and provide and our solution to it .

Summary: This paper proposed to use dropout to randomly choose only a subset of neural network as a potential way to perform exploration. The dropout happens at the beginning of each episode, and thus leads to a temporally consistent exploration. The paper shows that with small amount of Gaussian multiplicative dropout, the algorithm can achieve the state-of-the-art results on benchmark environments. And it can significantly outperform vanilla PPO for environments with sparse rewards.