NADPEX: AN ON-POLICY TEMPORALLY CONSISTENT EXPLORATION METHOD FOR DEEP REINFORCEMENT LEARNING
Year: 2019

Abstract:
Reinforcement learning agents need exploratory behaviors to escape from local optima. These behaviors may include both immediate dithering perturbation and temporally consistent exploration. To achieve these, a stochastic policy model that is inherently consistent through a period of time is in desire, especially for tasks with either sparse rewards or long term information. In this work, we introduce a novel on-policy temporally consistent exploration strategy Neural Adaptive Dropout Policy Exploration (NADPEx) for deep reinforcement learning agents. Modeled as a global random variable for conditional distribution, dropout is incorporated to reinforcement learning policies, equipping them with inherent temporal consistency, even when the reward signals are sparse. Two factors, gradients’ alignment with the objective and KL constraint in policy space, are discussed to guarantee NADPEx policy’s stable improvement. Our experiments demonstrate that NADPEx solves tasks with sparse reward while naive exploration and parameter noise fail. It yields as well or even faster convergence in the standard mujoco benchmark for continuous control.

heading: 1 INTRODUCTION
text: Exploration remains a challenge in reinforcement learning, in spite of its recent successes in robotic manipulation and locomotion (Schulman et al., 2015b; Mnih et al., 2016; Duan et al., 2016; Schulman et al., 2017b). Most reinforcement learning algorithms explore with stochasticity in stepwise action space and suffer from low learning efficiency in environments with sparse rewards (Florensa et al., 2017) or long information chain (Osband et al., 2017). In these environments, temporally consistent exploration is needed to acquire useful learning signals. Though in off-policy methods, autocorrelated noises (Lillicrap et al., 2015) or separate samplers (Xu et al., 2018) could be designed for consistent exploration, on-policy exploration strategies tend to be learnt alongside with the optimal policy, which is non-trivial. A complementary objective term should be constructed because the purpose of exploration to optimize informational value of possible trajectories (Osband et al., 2016) is not contained directly in the reward of underlying Markov Decision Process (MDP). This complementary objective is known as reward shaping e.g. optimistic towards uncertainty heuristic and intrinsic rewards. However, most of them require complex additional structures and strong human priors when state and action spaces are intractable, and introduce unadjustable bias in end-to-end learning (Bellemare et al., 2016; Ostrovski et al., 2017; Tang et al., 2017; Fu et al., 2017; Houthooft et al., 2016; Pathak et al., 2017).
It would not be necessary though, to teach agents to explore consistently through reward shaping, if the policy model inherits it as a prior. This inspires ones to disentangle policy stochasticity, with a structure of time scales. One possible solution is policy parameter perturbation in a large time scale. Though previous attempts were restricted to linear function approximators (Rückstieß et al., 2008; Osband et al., 2014), progress has been made with neural networks, through either network section duplication (Osband et al., 2016) or adaptive-scale parameter noise injection (Plappert et al., 2017; Fortunato et al., 2017). However, in Osband et al. (2016) the episode-wise stochasticity is
unadjustable, and the duplicated modules do not cooperate with each other. Directly optimizing the mean of distribution of all parameters, Plappert et al. (2017) adjusts the stochasticity to the learning progress heristically. Besides, as all sampled parameters need to be stored in on-policy exploration, it is not directly salable to large neural networks, where the trade-off between large memory for multiple networks and high variance with sinlge network is non-trivial to make.
This work proposes Neural Adaptive Dropout Policy Exploration (NADPEx) as a simple, scalable and improvement-guaranteed method for on-policy temporally consistent exploration, which generalizes Osband et al. (2016) and Plappert et al. (2017). Policy stochasticity is disentangled into two time scales, step-wise action noise and episode-wise stochasticity modeled with a distribution of subnetworks. With one single subnetwork in an episode, it achieves inherent temporal consistency with only a little computational overhead and no crafted reward, even in environments with sparse rewards. And with a set of subnetworks in one sampling batch, agents experience diverse behavioral patterns. This sampling of subnetworks is executed through dropout (Hinton et al., 2012; Srivastava et al., 2014), which encourages composability of constituents and facilitates the emergence of diverse maneuvers. As dropout could be excerted on connections, neurons, modules and paths, NADPEx naturally extends to neural networks with various sizes, exploiting modularity at various levels. To align separately parametrized stochasticity to each other, this sampling is made differentiable for all possible dropout candidates. We further discuss the effect of a first-order KL regularizer on dropout policies to improve stability and guarantee policy improvement. Our experiments demonstrate that NAPDEx solves challenging exploration tasks with sparse rewards while achieving as efficient or even faster convergence in the standard mujoco benchmark for state-of-theart PPO agents, which we believe can be generalized to any deep reinforcement learning agents.

heading: 2 PRELIMINARIES
text: We consider a standard discrete-time finite-horizon discounted Markov Decision Process (MDP) setting for reinforcement learning, represented by a tupleM = (S,A,P, r, ρ0, γ, T ), with a state set S, an action setA, a transitional probability distribution P : S×A×S → R+, a bounded reward function r : S × A → R+, an initial state distribution ρ0 : S → R+, a discount factor γ ∈ [0, 1] and horizon T . We denote a policy parametrized by θ as πθ : S×A → R+. The optimization objective of this policy is to maximize the expected discounted return η(πθ) = Eτ [ ∑T t=0 γ
tr(st, at)], where τ = (s0, a0, ...) denotes the whole trajectory, s0 ∼ ρ0(s0), at ∼ πθ(at|st) and st+1 ∼ P(st+1|st, at). In experimental evaluation, we follow normal setting and use undiscounted return Eτ [ ∑T t=0 r(st, at)].

heading: 2.1 PROXIMAL POLICY OPTIMIZATION (PPO)
text: Theoretically speaking, NADPEx works with any policy gradient based algorithms. In this work, we consider the recent advance in on-policy policy gradient method, proximal policy optimization (PPO) (Schulman et al., 2017b).
Policy gradient methods were first proposed by Williams (1992) in REINFORCE algorithm to maximize the aforementioned objective in a gradient ascent manner. The gradient of an expectation ∇θη(πθ) is approximated with a Monte Carlo average of policy gradient:
∇θη(πθ) ≈ 1
NT N∑ i=1 T∑ t=o ∇θ log πθ(ait|sit)(Rit − bit), (1)
where N is the number of episodes in this batch, Rit = ∑T t′=t γ t′−trit′ and b i t is a baseline for variance reduction.
Schulman et al. (2015b) proposes a policy iteration algorithm and proves its monotonicity in improvement. With the πθ’s corresponding discounted state-visitation frequency ρθ, the sampling policy πθold and the updated policy πθ, it solves the following constrained optimization problem with a line search for an appropriate step size within certain bound δKL, called trust region:
argmaxθ Es∼ρ θold ,a∼π θold [ πθ(a|s) πθold(a|s) Aθold(s, a)]
s.t. Es∼ρ θold [DKL(π old θ (·|s)|πθ(·|s))] ≤ δKL
(2)
whereDKL(·||·) is the Kullback-Leibler (KL) divergence,Aθold(s, a) is calculated with Generalized Advantage Estimation (GAE) (Schulman et al., 2015c).
Proximal Policy Optimization (PPO) transforms (2) to a unconstrained optimization and solves it with first-order derivatives, embracing established stochastic gradient-based optimizer like Adam (Kingma & Ba, 2014). Noting that DKL(πoldθ (·|s)|πθ(·|s)) is actually a relaxation of total variational divergence according to Schulman et al. (2015b), whose first-order derivative is 0 when θ is close to θold,DKL(πθ(·|s)||πoldθ (·|s)) is a natural replacement. Combining the first-order derivative of DKL(πθ(·|s)||πoldθ (·|s)) (Schulman et al., 2017a) (check Appendix A for a proof):
∇θDKL(πθ(·|s)|πθold(·|s))] = Ea∼πθ [∇θ log πθ(a|s)(log πθ(a|s)− log πθold(a|s))], (3)
PPO optimizes the following unconstrained objective, called KL PPO loss1:
LKL = Es∼ρ θold ,a∼π θold [rθ(s, a)Aθold(s, a)− β 2 (log πθ(a|s)− log πθold(a|s))2], (4)
where rθ(s, a) = πθ(a|s) π θold
(a|s) . Schulman et al. (2017b) also proposes a clipping version PPO, as a lower bound to (4):
Lclip = Es∼ρ θold ,a∼π θold [min(rθ(s, a), clip(rθ(s, a), 1− , 1 + ))Aθold(s, a)] (5)
In this work, we try both KL PPO and clipping PPO.

heading: 2.2 DROPOUT
text: Dropout is a technique used in deep learning to prevent features from co-adaptation and parameters from overfitting, by randomly dropping some hidden neuron units in each round of feed-forward and back-propagation (Hinton et al., 2012; Srivastava et al., 2014). This is modeled by multiplying a Bernoulli random variable zkj to each hidden unit, i.e. neuron activation h k j , for j = 1...m, where m is the number of hidden units in kth layer. Then the neuron activation of the k + 1th layer is
hk+1 = σ(W(k+1)TDkzh k + b(k+1)), (6)
where Dz = diag(z) ∈ Rm×m, W(k+1) and b(k+1) are weights and biases at k + 1th layer respectively and we simply denote them with θ .=<W,b >. σ(x) is the nonlinear neuron activation function. The parameter of this Bernoulli random variable is pkj . = P(zkj = 0), aka the dropout rate of the jth hidden unit at kth layer. In supervised learning, these dropout parameters are normally fixed during training in some successful practice (Hinton et al., 2012; Wan et al., 2013). And there are some variants to dropout connections, modules or paths (Wan et al., 2013).
1Though the concrete form is not provided in Schulman et al. (2017b), it is given in Dhariwal et al. (2017) publicly released by the authors

heading: 3 METHODOLOGY

heading: 3.1 NEURAL ADAPTIVE DROPOUT POLICY EXPLORATION (NADPEX)
text: Designing an exploration strategy is to introduce a kind of stochasticity during reinforcement learning agents’ interaction with the environment to help them get rid of some local optima. While action space noise (parametrized as a stochastic policy πθ) might be sufficient in environments where stepwise rewards are provided, they have limited effectiveness in more complex environments (Florensa et al., 2017; Plappert et al., 2017). As their complement, an exploration strategy would be especially beneficial if it can help either sparse reward signals (Florensa et al., 2017) or significant long-term information (Osband et al., 2016) to be acquired and back-propagated through time (BPTT) (Sutton & Barto, 1998). This motivates us to introduce a hierarchy of stochasticity and capture temporal consistency with a separate parametrization.
Our algorithm, Neural Adaptive Dropout Policy Exploration (NADPEx), models stochasticity at large time scales with a distribution of plausible subnetworks. For each episode, one specific subnetworks is drawn from this distribution. Inspected from a large time scale, this encourages exploration towards different directions among different episodes in one batch of sampling. And from a small time scale, temporal correlation is enforced for consistent exploration. Policies with a hierarchy of stochasticity is believed to represent a complex action distribution and larger-scale behavioral patterns (Florensa et al., 2017).
We achieve the drawing of subnetwork through dropout. As introduced in Section 2.2, originally, dropout is modeled through multiplying a binary random variable zkj ∼ Bernoulli(1− pkj ) to each neuron activation hkj . In Srivastava et al. (2014), this binary dropout is softened as continuous random variable dropout, modeled with a Gaussian random variable z ∼ N (I,σ2), normally referred to as Gaussian multiplicative dropout. Here we denote both distributions with qφ(z). Later we will introduce how both of them could be made differentibale and thus adaptative to learning progress.
During the sampling process, at the beginning of every episode τ i, a vector of dropout random variables zi is sampled from qφ(z), giving us a dropout policy πθ|zi for step-wise action distribution. zi is fed to the stochastic computation graph (Schulman et al., 2015a) for the whole episode until termination, when a new round of drawing initiates. Similar as observations, actions and rewards, this random variable is stored as sampled data, which will be fed back to the stochastic computation graph during training. Policies with this hierarchy of stochasticity, i.e. NADPEx policies, can be represented as a joint distribution:
πθ,φ(·, z) = qφ(z)πθ|z(·), (7) making the the objective:
η(πθ,φ) = Ez∼q(z)[Eτ |z[ T∑ t=0 γtr(st, at)]]. (8)
If we only use the stochasticity of network architecture as a bootstrap, as in BootstrapDQN (Osband et al., 2016), the bootstrapped policy gradient training is to update the network parameters θ with the following gradients:
∇θη(πθ,φ) = ∇θEz∼qφ(z)[η(πθ|z)] = Ez∼qφ(z)[∇θη(πθ|z)] ≈ 1
N N∑ i=1 ∇θL(θ, zi) (9)
L(θ, zi) is the surrogate loss of dropout policy πθ,zi , variates according to the specific type of reinforcement learning algorithm. In next subsection, we discuss some pitfalls of bootstrap and provide and our solution to it.

heading: 3.2 TRAINING NADPEX
text: STOCHASTICITY ALIGNMENT
One concern in separating the parametrization of stochasticity in different levels is whether they can adapt to each other elegantly to guarantee policy improvement in terms of objective (8). Policy
gradients in (9) alway reduces πθ|z’s entropy (i.e. stochasticity at small time scale) as the policy improves, which corresponds to the increase in agent’s certainty. But this information is not propagated to qφ for stochasticity at large time scale in bootstrap. As in this work qφ is a distribution of subnetworks, this concern could also be intuitively understood as component space composition may not guarantee the performance improvement in the resulting policy space, due to the complex non-linearity of reward function. Gangwani & Peng (2017) observe that a naive crossover in the parameter space is expected to yield a low-performance composition, for which an extra policy distillation stage is designed.
We investigate likelihood ratio trick (Ranganath et al., 2014; Schulman et al., 2015a) for gradient back-propragation from the same objective (8) through discrete distribution e.g. binary dropout and reparametrization trick (Kingma & Welling, 2013; Rezende et al., 2014) for continuous distribution e.g. Gaussian multiplicative dropout, thus covering all possible dropout candidates (The proof is provided in Appendix B):
∇θ,φη(πθ,φ) ≈ 1
N N∑ i=1 (∇θL(θ, zi) +∇φ log qφ(zi)A(si0, ai0)), (10)
∇θ,φη(πθ,φ) ≈ 1
N N∑ i=1 ∇θ,φL(θ, I + φ i). (11)
In (10)φ is the parameters for Bernoulli distributions,A(si0, a i 0) is the GAE (Schulman et al., 2015c) for the ith trajectory from the beginning, i.e. (si0, a i 0). In (11) φ = σ thus z = I + φ i, is an element-wise multiplication, and i is the dummy random variable i ∼ N (0, I).
POLICY SPACE CONSTRAINT
However, simply making the dropout ditribution differentiable may not guarantee the policy improment. As introduced in Section 2.1, Schulman et al. (2015b) reduce the monotonic policy improvement theory to a contraint on KL divergence in policy space. In this work, the analytical form of NADPEx policy πθ,φ is obtainable as qφ is designed to be fully factorizable. Thus ones can leverage the full-fledged KL constraint to stablize the training of NADPEx.
Here we give an example of PPO-NADPEx. Similar as (3), we leverage a first-order approximated KL divergence for NADPEx policies. As credit assignment with likelihood ratio trick in (3) may suffer from curse of dimensionality for qφ(z), we stop gradients w.r.t. φ from the KL divergence to reduce variance with a little bias. We further replace πθ,φ with a representative, the mean policy πθ = πθ|z , where z is the mean of dropout random variable vectors. Then the objective is:
LNADPExKL = E[rθ|z(s, a)Aθold(s, a)− β
2 (log πθ(a|s)− log πθold|z(a|s))2] (12)
where rθ|z(s, a) = πθ|z(a|s) π θold|z(a|s) . A proof for the second term is in Appendix C. Intuitively, KL divergence between dropout polices πθ|z and mean policies πθ is added to remedy the omission of ∇φDKL. Optimizing the lower bound of (12), clipping PPO could adjust the clip ratio accordingly.

heading: 3.3 RELATIONS TO PARAMETER NOISE
text: Episode-wise parameter noise injection (Plappert et al., 2017) is another way to introduce the hierarchy of stochasticity, which could be regarded as a special case of NADPEx, with Gaussian mulitplicative dropout on connection and a heurstic adaptation of variance. That dropout at neurons is a local reparametrization of noisy networks is proved in Kingma et al. (2015). A replication of the proof is provided in Appendix D. They also prove this local reparametrization trick has a lower variance in gradients. And this reduction in variance could be enhanced if ones choose to drop modules or paths in NADPEx.
More importantly, as we scope our discussion down to on-policy exploration strategy, where z from qφ(z) need to be stored for training, NADPEx is more scalable to much larger neural networks with possibly larger mini-batch size for stochastic gradient-based optimizers. As a base case, to dropout neurons rather than connections, NADPEx has a complexity of O(Nm) for both sampling and memory, comparing to O(Nm2) of parameter noise, with m denoting the number of neurons in one layer. This reduction could also be enhanced if ones choose to drop modules or paths.

heading: 4 EXPERIMENTS
text: In this section we evaluate NADPEx by answering the following questions through experiments.
(i) Can NADPEx achieve state-of-the-art result in standard benchmarks?
(ii) Does NADPEx drive temporally consistent exploration in sparse reward environments?
(iii) Is the end-to-end training of the hierarchical stochasticity effective?
(iv) Is KL regularization effective in preventing divergence between dropout policies and the mean policy?
We developed NADPEx based on openai/baselines (Dhariwal et al., 2017). We run all experiments with PPO as reinforcement learning algorithm. Especially, we developed NADAPEx based on the GPU optimized version PPO. Details about network architecture and other hyper-parameters are available in Appendix E. During our implemented parameter noise in the same framework as NADPEx, we encountered out of memory error for a mini-batch size 64 and training batch size 2048. This proves our claim of sampling/memory complexity reduction with NADPEx. We had to make some trade-off between GPU parallelism and more iterations to survive.
For all experiment with NADPEx, we test the following configurations for the dropout hyperparameter, the initial dropout rate: (a) pkj = 0.01, (b) p k j = 0.1, (c) p k j = 0.3. For Gaussiasn dropout, we estimate the dropout rate as pkj = σ k j /(1 + σ k j ). And for experiments with parameter noise, we set the initial variance accordingly: (a) σkj = 0.001, (b) σ k j = 0.01, (c) σ k j = 0.05, to ensure same level of stochasticity. For experiments with fixed KL version PPO, we run with β = 0.0005. All figures below show statistics of experiments with 5 randomly generated seeds.

heading: 4.1 NADPEX IN STANDARD ENVIRONMENTS
text: First, we evaluate NADPEx’s overall performance in standard environments on the continuous control environments implemented in OpenAI Gym (Brockman et al., 2016). From Figure 2 we can see that they show similar pattern with Gaussian dropout and binary dropout, given identical dropout rates. Between them, Gaussian dropout is slightly better at scores and consistency among initial dropout rates. With the three initial dropout rates listed above, we find pkj = 0.1 shows constistent advantage over others. On the one hand, when the initial dropout rate is small (pkj = 0.01), NADPEx agents learn to earn reward faster than agents with only action space noise. It is even possible that these agents learn faster than ones with pkj = 0.1 in the beginning. However, their higher variance between random seeds indicates that some of them are not exploring as efficiently and the NADPEx policies may not be optimal, therefore normally they will be surpassed by ones with pki = 0.1 in the middle stage of training. On the other hand, large initial dropout rate (pkj = 0.3) tends to converge slowly, possibly due to the claim in Molchanov et al. (2017) that a large dropout rate could induce large varaince in gradients, making a stable convergence more difficult.

heading: 4.2 TEMPORALLY CONSISTENT EXPLORATION
text: We then evaluate how NADPEx with Gaussian dropout performs in environments where reward signals are sparse. Comparing with environments with stepwise rewards, these environments are more challenging as they only yield non-zero reward signals after milestone states are reached. Temporally consistent exploration therefore plays a crucial role in these environments. As in Plappert et al. (2017) we run experiments in rllab (Duan et al., 2016), modified according to Houthooft et al. (2016). Specifically, we have: (a) SparseDoublePendulum, which only yields a reward if the agent reaches the upright position, and (b) SparseHalfCheetah, which only yields a reward if the agent crosses a target distance, (c) SparseMountainCar, which only yields a reward if the agent drives up the hill. We use the same time horizon of T = 500 steps before resetting and gradually increment the difficulty during repetition until the performance of NADPEx and parameter noise differentiates. We would like to refer readers to Appendix G for more details about the sparse reward environments.
As shown in Figure 3, we witness success with temporally consistent exploration through NADPEx, while action perturbation fails. In all enviroments we examed, larger initial dropout rates can achieve faster or better convergence, revealing a stronger exploration capability at large time scales. Comparing to parameter noise, NADPEx earns higher score in shorter time, possibly indicating a higher exploration efficiency.

heading: 4.3 EFFECTIVENESS OF STOCHASTICITY ADAPTATION
text: One hypothesis NADPEx builds upon is end-to-end training of separately parametrized stochasticity can appropriately adapt the temporally-extended exploration with the current learning progress. In Figure 4 we show our comparison between NADPEx and bootstrap, as introduced in Section 3. And we can see that though the difference is subtle in simple task like Hopper, the advantage NADPEx has over bootstrap is obvious in Walker2d, which is a task with higher complexity. In Humanoid, the task with highest dimension of actions, the empirical result is interesting. Though bootstrap policy learns almost as fast as NADPEx in the begining, that the dropout is not adaptative drives it to over-explore when some dropout policies are close to converge. Being trapped at a local optimum, bootstrap policy earns 500 scores less than NADPEx policy.

heading: 4.4 KL DIVERGENCE BETWEEN DROPOUT POLICIES
text: To evaluate the effect of the KL regularizer, we also run experiments with KL PPO. Though in the original paper of Schulman et al. (2017b) clipping PPO empirically performs better than KL PPO, we believe including this KL term explicitly in the objective makes our validation self-explanatory. Figure 5 left shows the experiment result in Walker2d. Different from clipping PPO, NADPEx with small initial dropout rate performs best, earning much higher score than action noise. As shown in Figure 5 right, the KL divergence between dropout polices and mean policies is bounded.

heading: 5 RELATED WORKS
text: On-policy reinforcement learning methods have gained attention in recent years (Schulman et al., 2015b; Mnih et al., 2016; Schulman et al., 2017b), mainly due to their elegance with theoretical grounding and stability in policy iteration (Henderson et al., 2017b). Despite of the effectiveness, to improve their data efficiency remains an active research topic.
In this work, we consider the exploration strategies for on-policy reinforcement learning methods. Most of the aforementioned works employ naive exploration, with stochasticity only in action space. However, they fail to tackle some tasks with either sparse rewards (Florensa et al., 2017) or longterm information (Osband et al., 2016), where temporally consistent exploration is needed.
One solution to this challenge is to shape the reward to encourage more directed exploration. The specific direction has various foundations, including but not restricted to state visitation count (Jaksch et al., 2010; Tang et al., 2017), state density (Bellemare et al., 2016; Ostrovski et al., 2017; Fu et al., 2017), self-supervised prediction error (Pathak et al., 2017) etc. Some of them share the Probably Approximately Correct (PAC) with discrete and tractable state space (Jaksch et al., 2010). But when state space and action space are intractable, all of them need additional computational structures, which take non-trivial efforts to implement and non-negligible resources to execute.
Orthogonal to them, methods involving a hierarchy of stochasticity are proposed. Based on hierarchical reinforcement learning, Florensa et al. (2017) models the stochasticity at large time scales with a random variable - option - and model low-level policies with Stochastic Neural Networks. However, authors employ human designed proxy reward and staged training. Almost concurrently, Osband et al. (2016) and Plappert et al. (2017); Fortunato et al. (2017) propose network section ensemble and parameter noise injection respetively to disentangle stochasticity. Under the banner of Stochastic Neural Networks, NADPEx generalize them all. BootstrapDQN is a special case of NADPEx without stochasticity adapation and parameter noise is a special case with high variance and complexlity, as well as some heursitic approximation. Details are discussed in Section 3.
In NADPEx, this stochasticity at large time scales is captured with a distribution of plausible neural subnetworks from the same complete network. We achieve this through dropout (Srivastava et al., 2014). In spite of its success in supervised deep learning literature and Bayesian deep learning literature, it is the first time to combine dropout to reinforcement learning policies for exploration. The closest ones are Gal & Ghahramani (2016); Henderson et al. (2017a), which use dropout in value network to capture agents’ uncertainty about the environment. According to Osband et al. (2016), Gal & Ghahramani (2016) even fails in environment requiring temporally consistent exploration.
There are also some attempts from the Evolutionary Strategies and Genetic Algorithms literature (Salimans et al., 2017; Such et al., 2017; Gangwani & Peng, 2017) to continuous control tasks. Though they model the problem much differently from ours, the relation could be an interesting topic for future research.

heading: 6 CONCLUSION
text: Building a hierarchy of stochasticity for reinforcement learning policy is the first step towards more structured exploration. We presented a method, NADPEx, that models stochasticity at large time scale with a distribution of plausible subnetworks from the same complete network to achieve onpolicy temporally consistent exploration. These subnetworks are sampled through dropout at the beginning of episodes, used to explore the environment with diverse and consistent behavioral patterns and updated through simultaneous gradient back-propagation. A learning objective is provided such that this distribution is also updated in an end-to-end manner to adapt to the action-space policy. Thanks to the fact that this dropout transformation is differentiable, KL regularizers on policy space can help to further stabilize it. Our experiments exhibit that NADPEx successfully solves continuous control tasks, even with strong sparsity in rewards.

heading: A KL DIVERGENCE FIRST ORDER APPROXIMATION
text: We derive the first order derivatitve of KL divergence from a refernce policy π to a parametrized policy πθ to :
∇θDKL(πθ(·|s)||π(·|s)) = ∇θ ∫ πθ(a|s)(log πθ(a|s)− log π(a|s))da
= ∫ ∇θ(πθ(a|s)(log πθ(a|s)− log π(a|s)))da
= ∫ ∇θπθ(a|s)(log πθ(a|s)− log π(a|s) + 1)da
= ∫ πθ(a|s)∇θ log πθ(a|s)(log πθ(a|s)− log π(a|s) + 1)da
= Ea∼πθ [∇θ log πθ(a|s)(log πθ(a|s)− log π(a|s))]
(13)
Note that line 3 derives line 4 with likelihood ratio trick, line 4 derives line 5 as∫ πθ(a|s)∇θ log πθ(a|s)da = 0

heading: B GRADIENTS OF NADPEX
text: Here we provide a full derivation of NADPEx’s gradients. Specifically, gradients for two types of dropout are discussed.
As shown in (7), a NADPEx policy πθ,z is a joint distribution, which could be factorized with a dropout distribution qφ(z) and a conditional distribution, i.e. the dropout policy, πθ|z , z ∼ qφ(z) is the dropout random variable vector.
In reinforcement learning with NADPEx, we use gradient based optimization to maximize the objective (8).
∇η(πθ,φ) = ∇Ez∼q(z)[Eτ |z[ T∑ t=0 γtr(st, at)]]. (14)
B.1 DISCRETE DROPOUT
Normally the dropout distribution qφ(z) is a discrete distribution, for example, z ∼ Bernoulli(φ), ones can use likelihood ratio trick to calculate (14):
∇θ,φη(πθ,φ) = ∇θ,φEz∼qφ [Eτ |z[ T∑ t=0 γtr(st, at)]]
= ∇θ,φ ∫ qφ(z) ∫ pτ |z,θRτdτdz
= ∫ qφ(z)∇θ ∫ pτ |z,θRτdτdz +∇φ ∫ qφ(z) ∫ pτ |z,θRτdτdz
= ∫ qφ(z)∇θ ∫ pτ |z,θRτdτdz +∇φ ∫ qφ(z)Eτ |z[R(s0|z, a0|z)]dz
= Ez∼qφ [∇θL(θ, z) +∇φ log qφ(z)A(s0|z, a0|z)]
≈ 1 N N∑ i=1 (∇θL(θ, zi) +∇φ log qφ(zi)A(si0, ai0)),
(15)
where L(·) is the surrogate loss, varying from reinforcement learning algorihtms, A(si0, ai0) is the GAE (Schulman et al., 2015c) for the ith trajectory from the beginning i.e. (si0, a i 0), such that the gradient has low variance.
B.2 GAUSSIAN MULTIPLICATIVE DROPOUT
In Srivastava et al. (2014), Gaussian multiplicative dropout is proposed, where z ∼ N (I,σ2) is the multiplicative dropout random variable vector, with the same expectation of discrete dropout P (z = 1) = 11+φ . Reparametirzed with a dummy random variable ∼ N (0, I) (Kingma & Welling, 2013; Rezende et al., 2014), we have z = I + φ , where φ is used to denote σ for consistency of notation, is an element-wise multiplication, such that (14) could be calculated as:
∇θ,φη(πθ,φ) = ∇θ,φEz∼N (I,φ2)[Eτ |z[ T∑ t=0 γtr(st, at)]]
= ∇θ,φEz∼N (I,φ2)[η(πθ|z)] = ∇θ,φE ∼N (0,I)[η(πθ|I+φ )] = E ∼N (0,I)[∇θ,φη(πθ|I+φ )] = E ∼N (0,I)[∇θ,φL(θ, I + φ )]
≈ 1 N N∑ i=1 ∇θ,φL(θ, I + φ i)
(16)

heading: C NADPEX KL DIVERGENCE FIRST ORDER APPROXIMATION
text: In NADPEx, the dropout distribution is designed to be fully factorizable, thus the gradient of KL divergence in the NADPEx policy space is obtainable:
∇θ,φDKL(πθ,φ(·|s)||πθold,φold(·|s)) =∇θ,φ ∫ ∫ πθ,φ(a, z|s) log πθ,φ(a, z|s)
πθold,φold(a, z|s) dadz =∇θ,φ ∫ ∫ qφ(z)πθ|z(a|s) log qφ(z)πθ|z(a|s)
qφold(z)πθold|z(a|s) dadz
= ∫ qφ(z)∇θ ∫ πθ|z(a|s) log
qφ(z)πθ|z(a|s) qφold(z)πθold|z(a|s) dadz
+∇φ ∫ qφ(z) ∫ πθ|z(a|s) log
qφ(z)πθ|z(a|s) qφold(z)πθold|z(a|s) dadz.
(17)
And thus the same first-order approximation could be done just as in Appendix A. However, note that likelihood ratio trick is used during the derivation, which should be applied to the random variable vectors a and z here.
C.1 REDUCING VARIANCE IN MONTE CARLO ESTIMATE WITH MUPROP
As normally z could have a much higher dimension than a, making this likelihood ratio trick suffer from curse of dimensionality, we inspect into the second term with MuProp (Gu et al., 2016) for a low-variance estimate. Let
f(z) = ∫ πθ|z(a|s) log
qφ(z)πθ|z(a|s) qφold(z)πθold|z(a|s) da, (18)
then we have
gφ = ∇φ ∫ qφ(z)f(z)dz = ∫ qφ∇φ log qφ(z)f(z)dz
= Ez[∇φ log qφ(z)f(z)] ĝφ = ∇φ log qφ(z)f(z) where z ∼ qφ.
(19)
The idea of MuProp is to use a control variate that corresponds to the first-order Taylor expansion of f around some fixed value z̄, i.e. h(z) = f(z̄) + f ′(z̄)(z − z̄), such that
gMφ = ∇φEz[(f(z)− h(z)) + h(z)] = ∇φEz[f(z)− f(z̄)− f ′(z̄)(z − z̄)] + (f(z̄) + f ′(z̄)(z − z̄))] = Ez[∇φ log qφ(z)[f(z)− h(z)]] + f ′(z̄)∇φEz[z]
ĝMφ = ∇φ log qφ(z)[f(z)− h(z)] + f ′(z̄)∇φEz[z] where z ∼ qφ.
(20)
To further reduce the variance, the first term is sometimes omitted with acceptable biased introduced (Raiko et al., 2015). And empirically, we find it could be almost negligible comparing with the gradient from the reinforcement learning objective. For Gaussian dropout, ∇φEz[z] = ∇φ1 = 0, gφ is thus eliminated. With binary dropout, ∇φEz[z] = ∇φφ = 1, we have
ĝMφ = f ′(z)|z=z̄
= ∂
∂z log
qφ(z)
qφold(z) +
∂
∂z
∫ πθ|z(a|s) log
πθ|z(a|s) πθold|z(a|s) da
= ∂
∂z log
∏m−1 i=0 φ
zi(1− φi)1−zi∏m−1 i=0 φ old i zi(1− φoldi ) 1−zi + ∂ ∂z
∫ πθ|z(a|s) log
πθ|z(a|s) πθold|z(a|s) da
= m−1∑ i=0 log φi(1− φoldi ) (1− φi)φoldi + ∂ ∂z ∫ πθ|z(a|s) log πθ|z(a|s) πθold|z(a|s) da.
(21)
Note that in line 3 we assume the probabilistic distribution function to be continuous in an infinitesimal regions around {z|zi ∈ {0, 1} for ∀zi} to allow the existence of the first order derivative for the first term. The first term is then close to zero asφ is close toφold. Another possible relaxation method is to use Concrete distribution (Maddison et al., 2016):
qφ,λ(z) = m−1∏ i=0 λφiz −λi−1 i (1− zi)−λ−1 (φiz −λ i + (1− zi)−λ)2 , (22)
where λ is the temperature of this relaxation. Substituting it back to (21), we have
ĝMφ = f ′(z)|z=z̄
= ∂
∂z log
qφ,λ(z)
qφold,λ(z) +
∂
∂z
∫ πθ|z(a|s) log
πθ|z(a|s) πθold|z(a|s) da
= ∂
∂z m−1∑ i=0 (log λφi λφoldi − 2 log( φiz −λ i + (1− zi) −λ φoldi z −λ i + (1− zi) −λ )) + ∂ ∂z ∫ πθ|z(a|s) log πθ|z(a|s) πθold|z(a|s) da
= −2 m−1∑ i=0 ∂ ∂z (log( φiz −λ i + (1− zi) −λ φoldi z −λ i + (1− zi) −λ )) + ∂ ∂z ∫ πθ|z(a|s) log πθ|z(a|s) πθold|z(a|s) da
= 2λ m−1∑ i=0 ( φiz −λ−1 i + (1− zi) −λ−1 φiz −λ i + (1− zi) −λ − φoldi z −λ−1 i + (1− zi) −λ−1 φoldi z −λ i + (1− zi) −λ )
+ ∂
∂z
∫ πθ|z(a|s) log
πθ|z(a|s) πθold|z(a|s) da
(23) We can reach the same claim as above that the first term could be removed if we set z = z̄ → 0. And the second term could also be eliminated as first order derivative of KL divergence between two different distributions parametrized with almost but not entirely null networks if z̄ → 0.
C.2 RELAXING ANALYTICAL KL DIVERGENCE WITH TRUST REGION
Ones may argue that the second term in (17) is decomposable to ∇φ ∫ qφ(z)(log qφ(z)− log qφold(z))dz = ∇φDKL(qφi(zi)||qφoldi (zi)), (24)
∇φ ∫ qφ(z) ∫ πθ|z(a|s)(log πθ|z(a|s)− log πθold|z(a|s))dadz (25)
such that (24) has analytical form as q is either diagonal Gaussian or Bernoulli and thus the Monte Carlo gradient estimate and the MuProp approximation above need only to be exerted on (25). However, it can be proved that∇φDKL(qφi(zi)||qφoldi (zi)) is still omittable with trust region (Schulman et al., 2015b). When first proposed in Schulman et al. (2015b), trust region is a relaxation on DKL to encourage efficient learning, with the idea that step size will only be adapted if the trust region constraint is violated, i.e. DKL > δ. The adaptation of step size is replaced in Wang et al. (2016) with a mechanism to clip gradients from surrogate loss adaptively only if DKL > δ to further boost the efficiency.
We prove below that Dkl will almost never violate the constraint δ = 1 proposed in Wang et al. (2016). For Gaussian dropout we have:
DKL(qφi(zi)||qφoldi (zi)) = log φoldi φi + φ2i 2φoldi 2 − 1 2 = log φoldi φoldi + ∆φi + (φoldi + ∆φi)
2
2φoldi 2 −
1
2
= log 1
1 + ∆φi φoldi
+ ∆φi φoldi + ∆φi
2
φoldi 2 = − log(1 + ∆φi φoldi ) + ∆φi φoldi + ∆φi
2
φoldi 2
= − log(1 + x) + x+ x2, (26)
where we let x = ∆φi φoldi . According to our experiment, as well as some conclusions in the literature for dropout (Kingma et al., 2015), φi ∈ (0.005, 0.5). Obviously x ∈ (0, 1), DKL(qφi(zi)||qφoldi (zi)) ∈ (0, 1).
For binary dropout we have: DKL(qφi(zi)||qφoldi (zi))
=φi(logφi − logφoldi ) + (1− φi)(log(1− φi)− log(1− φoldi ))
=[log(1− φi)− log(1− φoldi )] + φi[log φi 1− φi − log φ
old i
1− φoldi ]
=[log(1− φoldi −∆φi)− log(1− φoldi )] + φi[log φoldi + ∆φi 1− φoldi −∆φi − log φ old i 1− φoldi ].
(27)
We can thus discuss how DKL changes with ∆φi and φoldi : ∂
∂∆φi DKL(qφi(zi)||qφoldi (zi))
=− 1 1− φoldi −∆φi + (φoldi + ∆φi)( 1 φoldi + ∆φi +
1
1− φoldi −∆φi )
+ (log φoldi + ∆φi 1− φoldi −∆φi − log φ old i 1− φoldi )
= log φoldi + ∆φi 1− φoldi −∆φi − log φ old i 1− φoldi = h(φoldi + ∆φi)− h(φoldi ).
(28)
∂
∂φoldi DKL(qφi(zi)||qφoldi (zi))
=− 1 1− φoldi −∆φi + 1 1− φoldi + (φoldi + ∆φi)(
1
φoldi + ∆φi +
1
1− φoldi −∆φi )
+ (φoldi + ∆φi)( 1
φoldi +
1
1− φoldi ) + (log φoldi + ∆φi 1− φoldi −∆φi − log φ old i 1− φoldi )
=−∆φi( 1
1− φoldi +
1
φoldi ) + log φoldi + ∆φi 1− φoldi −∆φi − log φ old i 1− φoldi
= 1
2! h′′(φoldi )∆φi +
1 3! h′′′(φoldi )∆φ 2 i + ...
(29)
With h(x) = log x1−x , x ∈ (0.005, 0.5), it is easy to see that ∂DKL ∂∆φi = 0 when ∆φi = 0, ∂DKL∂∆φi > 0 when ∆φi > 0 and ∂DKL∂∆φi < 0 when ∆φi < 0. Similarly, when φ old = 0.5, ∂DKL ∂φold
= 0; when φold > 0.5, ∂DKL
∂φold > 0; when φold < 0.5, ∂DKL ∂φold < 0. Hence DKL(qφi(zi)||qφoldi (zi))
reaches its maximum at |∆φi|max and |φoldi − 0.5|max. It is reasonable to set |∆φi|max = 0.1, leading us to DKL(qφi(zi)||qφoldi (zi))max ≈ 0.225 < δ = 1. We reach the same claim that ∇φDKL(πθ,φ(·|s)||πθold,φold(·|s)) could be stopped for the sake of learning efficiency at a cost of acceptable bias.
C.3 REMEDY BIAS WITH MEAN POLICY
As derived above, the regularization term only back-propagates gradients to θ: ∇θDKL(πθ,φ(·|s)||πθold,φold(·|s))
= ∫ qφ(z)∇θ ∫ πθ|z(a|s) log
qφ(z)πθ|z(a|s) qφold(z)πθold|z(a|s) dadz
= ∫ qφ(z)∇θ ∫ πθ|z(a|s)(log
πθ|z(a|s) πθold|z(a|s) + log qφ(z) qφold(z) )dadz
= ∫ qφ(z)∇θ ∫ πθ|z(a|s)(log πθ|z(a|s)− log πθold|z(a|s))dadz
=Ez∼qφ [∇θDKL(πθ|z(·|s)||πθold|z(·|s))]
≈ 1 N N−1∑ i=0 ∇θDKL(πθ|zi(·|s)||πθold|zi(·|s)),
(30)
where N is the number of dropout policies in this batch. Note that from line 3 to line 4 we simply remove log qφ(z)q
φold (z) , which can be regarded as substracting it as a baseline b(z) to reduce variance.
Obviously, (30) has similar level of variance as (9), closing our discussion on variance reduction even though some other techniques could possibly go further.
Seeing the lack of regularization on φ due to the gradient omission, we further enforce the idea that dropout policy had better to be close to each other, which is the supposed role of regularizer on φ. This could be intuitively understood as a remedy from θ to φ:
1
N N−1∑ i=0 N−1∑ j=0 ∇θDKL(πθ|zi(·|s)||πθold|zj (·|s)). (31)
Noticing that this term has a complexity of O(N2), we replace πθ,φ with a mean policy πθ = πθ|z . In the deep learning literature, it is fairly prevalent to use mean network for dropout training with some adjustment in gradient back-propagation, due to stability and effiency concern. For instance, it is proved by Wang & Manning (2013) that the sampling process for dropout network with sigmoid or ReLU activators could be avoided by integrating a Gaussian approximation at each layer of a neural network. Therefore, we have
1
N N−1∑ i=0 ∇θDKL(πθ(·|s)||πθold|zi(·|s))
= 1
N N−1∑ i=0 ∇θ ∫ πθ(a|s)(log πθ(a|s)− log πθold|zi(a|s))da
= 1
N N−1∑ i=0 ∫ πθ(a|s)∇θ log πθ(a|s)(log πθ(a|s)− log πθold|zi(a|s))da
= 1
N N−1∑ i=0 Ea∼πθ|z [∇θ log πθ(a|s)(log πθ(a|s)− log πθold|zi(a|s))]
= 1
N N−1∑ i=0 Ea∼πθ|z [ 1 2 ∇θ(log πθ(a|s)− log πθold|zi(a|s)) 2].
(32)
Empirically, we found little difference between fast dropout back-propagation and normal backpropagation when (32) is combined into (12).
D VARITAIONAL DROPOUT AND LOCAL REPARAMETRIZATION
As introduced in Section 2.1, multiplicative dropout at neuron activation of kth layer can also be viewed as a multiplicative noise for the input at k + 1th layer:
hk+1 = σ(W(k+1)TDkzh k + b(k+1)). (33)
After a simple rearrangement W̃(k+1)T = W(k+1)TDkz , we have:
hk+1 = σ(W̃(k+1)Thk + b(k+1)). (34)
That is, the stochastic neuron activation in networks with Gaussian multiplicatice dropout can also be regarded as stochastic neuron activation in Noisy Networks with correlated Gaussian parameter noise, whose means equal the corresponding ones in networks with dropout.

heading: E HYPERPARAMETERS
text: For most of the hyperparamters, we follow the setting in original PPO paper. Though, to emphasis the scalability of our method, we use 2 parallel enviroments and only 1 minibatch in each epoch.

heading: F EXPERIMENT RESULTS IN STANDARD ENVIRONMENTS

heading: G ENVIRONMENTS WITH SPARSE REWARDS
text: We use the same sparse reward environments from rllab Duan et al. (2016), modified by Houthooft et al. (2016):
• SparseHalfCheetah (S ⊂ R17,A ⊂ R6), which only yields a reward if the agent crosses a distance threshold,
• SparseMountainCar (S ⊂ R2,A ⊂ R), which only yields a reward if the agent drives up the hill,
• SparseDoublePendulum (S ⊂ R6,A ⊂ R), which only yields a reward if the agent reaches the upright position.

Review summaries:
The authors introduce a  novel  on-policy  temporally  consistent  exploration  strategy, named Neural  AdaptiveDropout Policy Exploration (NADPEx), for deep reinforcement learning agents. The main idea is to sample from a distribution of plausible subnetworks modeling the temporally consistent exploration. For this, the authors use the ideas of the standard dropout for deep networks. Using the proposed  dropout transformation that is differentiable, the authors show that the KL regularizers on policy-space play an important role in stabilizing its learning. The experimental validation is performed on continuous control learning tasks, showing the benefits of the proposed. 

This paper proposed to use dropout to randomly choose only a subset of neural network as a potential way to perform exploration. The dropout happens at the beginning of each episode, and thus leads to a temporally consistent exploration. The paper shows that with small amount of Gaussian multiplicative dropout, the algorithm can achieve the state-of-the-art results on benchmark environments. And it can significantly outperform vanilla PPO for environments with sparse rewards.