Title: NADPEX: AN ON-POLICY TEMPORALLY CONSISTENT EXPLORATION METHOD FOR DEEP REINFORCEMENT LEARNING

Abstract: Reinforcement learning agents need exploratory behaviors to escape from local optima. These behaviors may include both immediate dithering perturbation and temporally consistent exploration. To achieve these, a stochastic policy model that is inherently consistent through a period of time is in desire, especially for tasks with either sparse rewards or long term information. In this work, we introduce a novel on-policy temporally consistent exploration strategy -Neural Adaptive Dropout Policy Exploration (NADPEx) -for deep reinforcement learning agents. Modeled as a global random variable for conditional distribution, dropout is incorporated to reinforcement learning policies, equipping them with inherent temporal consistency, even when the reward signals are sparse. Two factors, gradients' alignment with the objective and KL constraint in policy space, are discussed to guarantee NADPEx policy's stable improvement. Our experiments demonstrate that NADPEx solves tasks with sparse reward while naive exploration and parameter noise fail. It yields as well or even faster convergence in the standard mujoco benchmark for continuous control.

1. INTRODUCTION:

Exploration remains a challenge in reinforcement learning, in spite of its recent successes in robotic manipulation and locomotion (Schulman et al., 2015b; Mnih et al., 2016; Duan et al., 2016; Schulman et al., 2017b) . Most reinforcement learning algorithms explore with stochasticity in stepwise action space and suffer from low learning efficiency in environments with sparse rewards (Florensa et al., 2017) or long information chain (Osband et al., 2017) . In these environments, temporally consistent exploration is needed to acquire useful learning signals. Though in off-policy methods, autocorrelated noises (Lillicrap et al., 2015) or separate samplers (Xu et al., 2018) could be designed for consistent exploration, on-policy exploration strategies tend to be learnt alongside with the optimal policy, which is non-trivial. A complementary objective term should be constructed because the purpose of exploration to optimize informational value of possible trajectories (Osband et al., 2016) is not contained directly in the reward of underlying Markov Decision Process (MDP). This complementary objective is known as reward shaping e.g. optimistic towards uncertainty heuristic and intrinsic rewards. However, most of them require complex additional structures and strong human priors when state and action spaces are intractable, and introduce unadjustable bias in end-to-end learning (Bellemare et al., 2016; Ostrovski et al., 2017; Tang et al., 2017; Fu et al., 2017; Houthooft et al., 2016; Pathak et al., 2017) .

It would not be necessary though, to teach agents to explore consistently through reward shaping, if the policy model inherits it as a prior. This inspires ones to disentangle policy stochasticity, with a structure of time scales. One possible solution is policy parameter perturbation in a large time scale. Though previous attempts were restricted to linear function approximators (Rückstieß et al., 2008; Osband et al., 2014) , progress has been made with neural networks, through either network section duplication (Osband et al., 2016) or adaptive-scale parameter noise injection (Plappert et al., 2017; Fortunato et al., 2017) . However, in Osband et al. (2016) the episode-wise stochasticity is unadjustable, and the duplicated modules do not cooperate with each other. Directly optimizing the mean of distribution of all parameters, Plappert et al. (2017) adjusts the stochasticity to the learning progress heristically. Besides, as all sampled parameters need to be stored in on-policy exploration, it is not directly salable to large neural networks, where the trade-off between large memory for multiple networks and high variance with sinlge network is non-trivial to make. This work proposes Neural Adaptive Dropout Policy Exploration (NADPEx) as a simple, scalable and improvement-guaranteed method for on-policy temporally consistent exploration, which generalizes Osband et al. (2016) and Plappert et al. (2017) . Policy stochasticity is disentangled into two time scales, step-wise action noise and episode-wise stochasticity modeled with a distribution of subnetworks. With one single subnetwork in an episode, it achieves inherent temporal consistency with only a little computational overhead and no crafted reward, even in environments with sparse rewards. And with a set of subnetworks in one sampling batch, agents experience diverse behavioral patterns. This sampling of subnetworks is executed through dropout (Hinton et al., 2012; Srivastava et al., 2014) , which encourages composability of constituents and facilitates the emergence of diverse maneuvers. As dropout could be excerted on connections, neurons, modules and paths, NADPEx naturally extends to neural networks with various sizes, exploiting modularity at various levels. To align separately parametrized stochasticity to each other, this sampling is made differentiable for all possible dropout candidates. We further discuss the effect of a first-order KL regularizer on dropout policies to improve stability and guarantee policy improvement. Our experiments demonstrate that NAPDEx solves challenging exploration tasks with sparse rewards while achieving as efficient or even faster convergence in the standard mujoco benchmark for state-of-theart PPO agents, which we believe can be generalized to any deep reinforcement learning agents. 

2. PRELIMINARIES:

We consider a standard discrete-time finite-horizon discounted Markov Decision Process (MDP) setting for reinforcement learning, represented by a tuple M = (S, A, P, r, ρ 0 , γ, T ), with a state set S, an action set A, a transitional probability distribution P : S × A × S → R + , a bounded reward function r : S × A → R + , an initial state distribution ρ 0 : S → R + , a discount factor γ ∈ [0, 1] and horizon T . We denote a policy parametrized by θ as π θ : S×A → R + . The optimization objective of this policy is to maximize the expected discounted return η(π θ ) = E τ [ T t=0 γ t r(s t , a t )], where τ = (s 0 , a 0 , ...) denotes the whole trajectory, s 0 ∼ ρ 0 (s 0 ), a t ∼ π θ (a t |s t ) and s t+1 ∼ P(s t+1 |s t , a t ). In experimental evaluation, we follow normal setting and use undiscounted return E τ [ T t=0 r(s t , a t )].

2.1. PROXIMAL POLICY OPTIMIZATION (PPO):

Theoretically speaking, NADPEx works with any policy gradient based algorithms. In this work, we consider the recent advance in on-policy policy gradient method, proximal policy optimization (PPO) (Schulman et al., 2017b) .

Policy gradient methods were first proposed by Williams (1992) in REINFORCE algorithm to maximize the aforementioned objective in a gradient ascent manner. The gradient of an expectation ∇ θ η(π θ ) is approximated with a Monte Carlo average of policy gradient:

EQUATION

where N is the number of episodes in this batch, R i t = T t =t γ t -t r i t and b i t is a baseline for variance reduction. Schulman et al. (2015b) proposes a policy iteration algorithm and proves its monotonicity in improvement. With the π θ 's corresponding discounted state-visitation frequency ρ θ , the sampling policy π θ old and the updated policy π θ , it solves the following constrained optimization problem with a line search for an appropriate step size within certain bound δ KL , called trust region:

argmax θ E s∼ρ θ old ,a∼π θ old [ π θ (a|s) π θ old (a|s) A θ old (s, a)] s.t. E s∼ρ θ old [D KL (π old θ (•|s)|π θ (•|s))] ≤ δ KL (2)

where (Schulman et al., 2015c 

D KL (•||•) is the Kullback-Leibler (KL) divergence, A θ old (s, a) is calculated with Generalized Advantage Estimation (GAE)

EQUATION

PPO optimizes the following unconstrained objective, called KL PPO loss1 :

EQUATION

where r θ (s, a) = π θ (a|s) π θ old (a|s) . Schulman et al. (2017b) also proposes a clipping version PPO, as a lower bound to (4):

EQUATION

In this work, we try both KL PPO and clipping PPO.

2.2. DROPOUT:

Dropout is a technique used in deep learning to prevent features from co-adaptation and parameters from overfitting, by randomly dropping some hidden neuron units in each round of feed-forward and back-propagation (Hinton et al., 2012; Srivastava et al., 2014) . This is modeled by multiplying a Bernoulli random variable z k j to each hidden unit, i.e. neuron activation h k j , for j = 1...m, where m is the number of hidden units in kth layer. Then the neuron activation of the k + 1th layer is

EQUATION

where k+1) and b (k+1) are weights and biases at k + 1th layer respectively and we simply denote them with θ . =< W, b >. σ(x) is the nonlinear neuron activation function. The parameter of this Bernoulli random variable is p k j . = P(z k j = 0), aka the dropout rate of the jth hidden unit at kth layer. In supervised learning, these dropout parameters are normally fixed during training in some successful practice (Hinton et al., 2012; Wan et al., 2013) . And there are some variants to dropout connections, modules or paths (Wan et al., 2013) .

EQUATION

3. METHODOLOGY:



3.1. NEURAL ADAPTIVE DROPOUT POLICY EXPLORATION (NADPEX):

Designing an exploration strategy is to introduce a kind of stochasticity during reinforcement learning agents' interaction with the environment to help them get rid of some local optima. While action space noise (parametrized as a stochastic policy π θ ) might be sufficient in environments where stepwise rewards are provided, they have limited effectiveness in more complex environments (Florensa et al., 2017; Plappert et al., 2017) . As their complement, an exploration strategy would be especially beneficial if it can help either sparse reward signals (Florensa et al., 2017) or significant long-term information (Osband et al., 2016) to be acquired and back-propagated through time (BPTT) (Sutton & Barto, 1998) . This motivates us to introduce a hierarchy of stochasticity and capture temporal consistency with a separate parametrization.

Our algorithm, Neural Adaptive Dropout Policy Exploration (NADPEx), models stochasticity at large time scales with a distribution of plausible subnetworks. For each episode, one specific subnetworks is drawn from this distribution. Inspected from a large time scale, this encourages exploration towards different directions among different episodes in one batch of sampling. And from a small time scale, temporal correlation is enforced for consistent exploration. Policies with a hierarchy of stochasticity is believed to represent a complex action distribution and larger-scale behavioral patterns (Florensa et al., 2017) .

We achieve the drawing of subnetwork through dropout. As introduced in Section 2.2, originally, dropout is modeled through multiplying a binary random variable z k j ∼ Bernoulli(1 -p k j ) to each neuron activation h k j . In Srivastava et al. (2014) , this binary dropout is softened as continuous random variable dropout, modeled with a Gaussian random variable z ∼ N (I, σ 2 ), normally referred to as Gaussian multiplicative dropout. Here we denote both distributions with q φ (z). Later we will introduce how both of them could be made differentibale and thus adaptative to learning progress.

During the sampling process, at the beginning of every episode τ i , a vector of dropout random variables z i is sampled from q φ (z), giving us a dropout policy π θ|z i for step-wise action distribution. z i is fed to the stochastic computation graph (Schulman et al., 2015a) for the whole episode until termination, when a new round of drawing initiates. Similar as observations, actions and rewards, this random variable is stored as sampled data, which will be fed back to the stochastic computation graph during training. Policies with this hierarchy of stochasticity, i.e. NADPEx policies, can be represented as a joint distribution:

EQUATION

making the the objective:

EQUATION

If we only use the stochasticity of network architecture as a bootstrap, as in BootstrapDQN (Osband et al., 2016) , the bootstrapped policy gradient training is to update the network parameters θ with the following gradients:

∇ θ η(π θ,φ ) = ∇ θ E z∼q φ (z) [η(π θ|z )] = E z∼q φ (z) [∇ θ η(π θ|z )] ≈ 1 N N i=1 ∇ θ L(θ, z i ) (9) L(θ, z i )

is the surrogate loss of dropout policy π θ,z i , variates according to the specific type of reinforcement learning algorithm. In next subsection, we discuss some pitfalls of bootstrap and provide and our solution to it.

3.2. TRAINING NADPEX STOCHASTICITY ALIGNMENT:

One concern in separating the parametrization of stochasticity in different levels is whether they can adapt to each other elegantly to guarantee policy improvement in terms of objective (8). Policy gradients in (9) alway reduces π θ|z 's entropy (i.e. stochasticity at small time scale) as the policy improves, which corresponds to the increase in agent's certainty. But this information is not propagated to q φ for stochasticity at large time scale in bootstrap. As in this work q φ is a distribution of subnetworks, this concern could also be intuitively understood as component space composition may not guarantee the performance improvement in the resulting policy space, due to the complex non-linearity of reward function. Gangwani & Peng (2017) observe that a naive crossover in the parameter space is expected to yield a low-performance composition, for which an extra policy distillation stage is designed.

We investigate likelihood ratio trick (Ranganath et al., 2014; Schulman et al., 2015a) for gradient back-propragation from the same objective (8) through discrete distribution e.g. binary dropout and reparametrization trick (Kingma & Welling, 2013; Rezende et al., 2014) for continuous distribution e.g. Gaussian multiplicative dropout, thus covering all possible dropout candidates (The proof is provided in Appendix B):

EQUATION

EQUATION

In ( 10) φ is the parameters for Bernoulli distributions, A(s i 0 , a i 0 ) is the GAE (Schulman et al., 2015c ) for the ith trajectory from the beginning, i.e. (s i 0 , a i 0 ). In (11) φ = σ thus z = I + φ i , is an element-wise multiplication, and i is the dummy random variable i ∼ N (0, I).

POLICY SPACE CONSTRAINT:

However, simply making the dropout ditribution differentiable may not guarantee the policy improment. As introduced in Section 2.1, Schulman et al. (2015b) reduce the monotonic policy improvement theory to a contraint on KL divergence in policy space. In this work, the analytical form of NADPEx policy π θ,φ is obtainable as q φ is designed to be fully factorizable. Thus ones can leverage the full-fledged KL constraint to stablize the training of NADPEx.

Here we give an example of PPO-NADPEx. Similar as (3), we leverage a first-order approximated KL divergence for NADPEx policies. As credit assignment with likelihood ratio trick in (3) may suffer from curse of dimensionality for q φ (z), we stop gradients w.r.t. φ from the KL divergence to reduce variance with a little bias. We further replace π θ,φ with a representative, the mean policy π θ = π θ|z , where z is the mean of dropout random variable vectors. Then the objective is:

EQUATION

)

where r θ|z (s, a) = π θ|z (a|s) π θ old |z (a|s) . A proof for the second term is in Appendix C. Intuitively, KL divergence between dropout polices π θ|z and mean policies π θ is added to remedy the omission of ∇ φ D KL . Optimizing the lower bound of (12), clipping PPO could adjust the clip ratio accordingly.

3.3. RELATIONS TO PARAMETER NOISE:

Episode-wise parameter noise injection (Plappert et al., 2017) is another way to introduce the hierarchy of stochasticity, which could be regarded as a special case of NADPEx, with Gaussian mulitplicative dropout on connection and a heurstic adaptation of variance. That dropout at neurons is a local reparametrization of noisy networks is proved in Kingma et al. (2015) . A replication of the proof is provided in Appendix D. They also prove this local reparametrization trick has a lower variance in gradients. And this reduction in variance could be enhanced if ones choose to drop modules or paths in NADPEx.

More importantly, as we scope our discussion down to on-policy exploration strategy, where z from q φ (z) need to be stored for training, NADPEx is more scalable to much larger neural networks with possibly larger mini-batch size for stochastic gradient-based optimizers. As a base case, to dropout neurons rather than connections, NADPEx has a complexity of O(N m) for both sampling and memory, comparing to O(N m 2 ) of parameter noise, with m denoting the number of neurons in one layer. This reduction could also be enhanced if ones choose to drop modules or paths.

4. EXPERIMENTS:

In this section we evaluate NADPEx by answering the following questions through experiments.

(i) Can NADPEx achieve state-of-the-art result in standard benchmarks? (ii) Does NADPEx drive temporally consistent exploration in sparse reward environments? (iii) Is the end-to-end training of the hierarchical stochasticity effective? (iv) Is KL regularization effective in preventing divergence between dropout policies and the mean policy?

We developed NADPEx based on openai/baselines (Dhariwal et al., 2017) . We run all experiments with PPO as reinforcement learning algorithm. Especially, we developed NADAPEx based on the GPU optimized version PPO. Details about network architecture and other hyper-parameters are available in Appendix E. During our implemented parameter noise in the same framework as NADPEx, we encountered out of memory error for a mini-batch size 64 and training batch size 2048. This proves our claim of sampling/memory complexity reduction with NADPEx. We had to make some trade-off between GPU parallelism and more iterations to survive.

For all experiment with NADPEx, we test the following configurations for the dropout hyperparameter, the initial dropout rate: (a) p k j = 0.01, (b) p k j = 0.1, (c) p k j = 0.3. For Gaussiasn dropout, we estimate the dropout rate as p k j = σ k j /(1 + σ k j ). And for experiments with parameter noise, we set the initial variance accordingly: (a) σ k j = 0.001, (b) σ k j = 0.01, (c) σ k j = 0.05, to ensure same level of stochasticity. For experiments with fixed KL version PPO, we run with β = 0.0005. All figures below show statistics of experiments with 5 randomly generated seeds.

4.1. NADPEX IN STANDARD ENVIRONMENTS:

First, we evaluate NADPEx's overall performance in standard environments on the continuous control environments implemented in OpenAI Gym (Brockman et al., 2016) . From Figure 2 we can see that they show similar pattern with Gaussian dropout and binary dropout, given identical dropout rates. Between them, Gaussian dropout is slightly better at scores and consistency among initial dropout rates. With the three initial dropout rates listed above, we find p k j = 0.1 shows constistent advantage over others. On the one hand, when the initial dropout rate is small (p k j = 0.01), NADPEx agents learn to earn reward faster than agents with only action space noise. It is even possible that these agents learn faster than ones with p k j = 0.1 in the beginning. However, their higher variance between random seeds indicates that some of them are not exploring as efficiently and the NADPEx policies may not be optimal, therefore normally they will be surpassed by ones with p k i = 0.1 in the middle stage of training. On the other hand, large initial dropout rate (p k j = 0.3) tends to converge slowly, possibly due to the claim in Molchanov et al. (2017) that a large dropout rate could induce large varaince in gradients, making a stable convergence more difficult.

Figure 2 : Experiments with NADPEx in standard envs, three p k j are presented for Gaussian dropout, as well as the best of binary dropout. Extensive comparison is given in Appendix F.

4.2. TEMPORALLY CONSISTENT EXPLORATION:

We then evaluate how NADPEx with Gaussian dropout performs in environments where reward signals are sparse. Comparing with environments with stepwise rewards, these environments are more challenging as they only yield non-zero reward signals after milestone states are reached. Temporally consistent exploration therefore plays a crucial role in these environments. As in Plappert et al. (2017) we run experiments in rllab (Duan et al., 2016) , modified according to Houthooft et al. (2016) . Specifically, we have: (a) SparseDoublePendulum, which only yields a reward if the agent reaches the upright position, and (b) SparseHalfCheetah, which only yields a reward if the agent crosses a target distance, (c) SparseMountainCar, which only yields a reward if the agent drives up the hill. We use the same time horizon of T = 500 steps before resetting and gradually increment the difficulty during repetition until the performance of NADPEx and parameter noise differentiates. We would like to refer readers to Appendix G for more details about the sparse reward environments.

As shown in Figure 3 , we witness success with temporally consistent exploration through NAD-PEx, while action perturbation fails. In all enviroments we examed, larger initial dropout rates can achieve faster or better convergence, revealing a stronger exploration capability at large time scales. Comparing to parameter noise, NADPEx earns higher score in shorter time, possibly indicating a higher exploration efficiency. 

4.3. EFFECTIVENESS OF STOCHASTICITY ADAPTATION:

One hypothesis NADPEx builds upon is end-to-end training of separately parametrized stochasticity can appropriately adapt the temporally-extended exploration with the current learning progress. In Figure 4 we show our comparison between NADPEx and bootstrap, as introduced in Section 3. And we can see that though the difference is subtle in simple task like Hopper, the advantage NADPEx has over bootstrap is obvious in Walker2d, which is a task with higher complexity. In Humanoid, the task with highest dimension of actions, the empirical result is interesting. Though bootstrap policy learns almost as fast as NADPEx in the begining, that the dropout is not adaptative drives it to over-explore when some dropout policies are close to converge. Being trapped at a local optimum, bootstrap policy earns 500 scores less than NADPEx policy. To evaluate the effect of the KL regularizer, we also run experiments with KL PPO. Though in the original paper of Schulman et al. (2017b) clipping PPO empirically performs better than KL PPO, we believe including this KL term explicitly in the objective makes our validation self-explanatory. Figure 5 left shows the experiment result in Walker2d. Different from clipping PPO, NADPEx with small initial dropout rate performs best, earning much higher score than action noise. As shown in Figure 5 right, the KL divergence between dropout polices and mean policies is bounded. 

5. RELATED WORKS:

On-policy reinforcement learning methods have gained attention in recent years (Schulman et al., 2015b; Mnih et al., 2016; Schulman et al., 2017b) , mainly due to their elegance with theoretical grounding and stability in policy iteration (Henderson et al., 2017b) . Despite of the effectiveness, to improve their data efficiency remains an active research topic.

In this work, we consider the exploration strategies for on-policy reinforcement learning methods. Most of the aforementioned works employ naive exploration, with stochasticity only in action space. However, they fail to tackle some tasks with either sparse rewards (Florensa et al., 2017) or longterm information (Osband et al., 2016) , where temporally consistent exploration is needed.

One solution to this challenge is to shape the reward to encourage more directed exploration. The specific direction has various foundations, including but not restricted to state visitation count (Jaksch et al., 2010; Tang et al., 2017) , state density (Bellemare et al., 2016; Ostrovski et al., 2017; Fu et al., 2017) , self-supervised prediction error (Pathak et al., 2017) etc. Some of them share the Probably Approximately Correct (PAC) with discrete and tractable state space (Jaksch et al., 2010) . But when state space and action space are intractable, all of them need additional computational structures, which take non-trivial efforts to implement and non-negligible resources to execute.

Orthogonal to them, methods involving a hierarchy of stochasticity are proposed. Based on hierarchical reinforcement learning, Florensa et al. (2017) models the stochasticity at large time scales with a random variable -option -and model low-level policies with Stochastic Neural Networks. However, authors employ human designed proxy reward and staged training. Almost concurrently, Osband et al. (2016) and Plappert et al. (2017); Fortunato et al. (2017) propose network section ensemble and parameter noise injection respetively to disentangle stochasticity. Under the banner of Stochastic Neural Networks, NADPEx generalize them all. BootstrapDQN is a special case of NADPEx without stochasticity adapation and parameter noise is a special case with high variance and complexlity, as well as some heursitic approximation. Details are discussed in Section 3.

In NADPEx, this stochasticity at large time scales is captured with a distribution of plausible neural subnetworks from the same complete network. We achieve this through dropout (Srivastava et al., 2014) . In spite of its success in supervised deep learning literature and Bayesian deep learning literature, it is the first time to combine dropout to reinforcement learning policies for exploration. The closest ones are Gal & Ghahramani (2016) ; Henderson et al. (2017a) , which use dropout in value network to capture agents' uncertainty about the environment. According to Osband et al. (2016) , Gal & Ghahramani (2016) even fails in environment requiring temporally consistent exploration.

There are also some attempts from the Evolutionary Strategies and Genetic Algorithms literature (Salimans et al., 2017; Such et al., 2017; Gangwani & Peng, 2017) to continuous control tasks. Though they model the problem much differently from ours, the relation could be an interesting topic for future research.

6. CONCLUSION:

Building a hierarchy of stochasticity for reinforcement learning policy is the first step towards more structured exploration. We presented a method, NADPEx, that models stochasticity at large time scale with a distribution of plausible subnetworks from the same complete network to achieve onpolicy temporally consistent exploration. These subnetworks are sampled through dropout at the beginning of episodes, used to explore the environment with diverse and consistent behavioral patterns and updated through simultaneous gradient back-propagation. A learning objective is provided such that this distribution is also updated in an end-to-end manner to adapt to the action-space policy. Thanks to the fact that this dropout transformation is differentiable, KL regularizers on policy space can help to further stabilize it. Our experiments exhibit that NADPEx successfully solves continuous control tasks, even with strong sparsity in rewards.

A KL DIVERGENCE FIRST ORDER APPROXIMATION:

We derive the first order derivatitve of KL divergence from a refernce policy π to a parametrized policy π θ to :

∇ θ D KL (π θ (•|s)||π(•|s)) = ∇ θ π θ (a|s)(log π θ (a|s) -log π(a|s))da = ∇ θ (π θ (a|s)(log π θ (a|s) -log π(a|s)))da = ∇ θ π θ (a|s)(log π θ (a|s) -log π(a|s) + 1)da = π θ (a|s)∇ θ log π θ (a|s)(log π θ (a|s) -log π(a|s) + 1)da = E a∼π θ [∇ θ log π θ (a|s)(log π θ (a|s) -log π(a|s))] (13)

Note that line 3 derives line 4 with likelihood ratio trick, line 4 derives line 5 as π θ (a|s)∇ θ log π θ (a|s)da = 0

B GRADIENTS OF NADPEX:

Here we provide a full derivation of NADPEx's gradients. Specifically, gradients for two types of dropout are discussed.

As shown in ( 7), a NADPEx policy π θ,z is a joint distribution, which could be factorized with a dropout distribution q φ (z) and a conditional distribution, i.e. the dropout policy, π θ|z , z ∼ q φ (z) is the dropout random variable vector.

In reinforcement learning with NADPEx, we use gradient based optimization to maximize the objective (8).

EQUATION

B.1 DISCRETE DROPOUT Normally the dropout distribution q φ (z) is a discrete distribution, for example, z ∼ Bernoulli(φ), ones can use likelihood ratio trick to calculate (14):

EQUATION

where L(•) is the surrogate loss, varying from reinforcement learning algorihtms, A(s i 0 , a i 0 ) is the GAE (Schulman et al., 2015c) for the ith trajectory from the beginning i.e. (s i 0 , a i 0 ), such that the gradient has low variance.

B.2 GAUSSIAN MULTIPLICATIVE DROPOUT:

In Srivastava et al. (2014) , Gaussian multiplicative dropout is proposed, where z ∼ N (I, σ 2 ) is the multiplicative dropout random variable vector, with the same expectation of discrete dropout P (z = 1) = 1 1+φ . Reparametirzed with a dummy random variable ∼ N (0, I) (Kingma & Welling, 2013; Rezende et al., 2014) , we have z = I + φ

, where φ is used to denote σ for consistency of notation, is an element-wise multiplication, such that ( 14) could be calculated as:

EQUATION

C NADPEX KL DIVERGENCE FIRST ORDER APPROXIMATION

In NADPEx, the dropout distribution is designed to be fully factorizable, thus the gradient of KL divergence in the NADPEx policy space is obtainable:

∇ θ,φ D KL (π θ,φ (•|s)||π θ old ,φ old (•|s))

=∇ θ,φ π θ,φ (a, z|s) log π θ,φ (a, z|s) π θ old ,φ old (a, z|s) dadz =∇ θ,φ q φ (z)π θ|z (a|s) log q φ (z)π θ|z (a|s) q φ old (z)π θ old |z (a|s) dadz = q φ (z)∇ θ π θ|z (a|s) log q φ (z)π θ|z (a|s) q φ old (z)π θ old |z (a|s) dadz

EQUATION

And thus the same first-order approximation could be done just as in Appendix A. However, note that likelihood ratio trick is used during the derivation, which should be applied to the random variable vectors a and z here.

C.1 REDUCING VARIANCE IN MONTE CARLO ESTIMATE WITH MUPROP:

As normally z could have a much higher dimension than a, making this likelihood ratio trick suffer from curse of dimensionality, we inspect into the second term with MuProp (Gu et al., 2016) for a low-variance estimate. Let

EQUATION

then we have

EQUATION

Published as a conference paper at ICLR 2019

The idea of MuProp is to use a control variate that corresponds to the first-order Taylor expansion of f around some fixed value z, i.e. h(z) = f ( z) + f ( z)(zz), such that

EQUATION

To further reduce the variance, the first term is sometimes omitted with acceptable biased introduced (Raiko et al., 2015) . And empirically, we find it could be almost negligible comparing with the gradient from the reinforcement learning objective. For Gaussian dropout,

EQUATION

Note that in line 3 we assume the probabilistic distribution function to be continuous in an infinitesimal regions around {z|z i ∈ {0, 1} f or ∀z i } to allow the existence of the first order derivative for the first term. The first term is then close to zero as φ is close to φ old . Another possible relaxation method is to use Concrete distribution (Maddison et al., 2016) :

EQUATION

)

where λ is the temperature of this relaxation. Substituting it back to (21), we have

ĝM φ = f (z)| z= z = ∂ ∂z log q φ,λ (z) q φ old ,λ (z) + ∂ ∂z π θ|z (a|s) log π θ|z (a|s) π θ old |z (a|s) da = ∂ ∂z m-1 i=0 (log λφ i λφ old i -2 log( φ i z -λ i + (1 -z i ) -λ φ old i z -λ i + (1 -z i ) -λ )) + ∂ ∂z π θ|z (a|s) log π θ|z (a|s) π θ old |z (a|s) da = -2 m-1 i=0 ∂ ∂z (log( φ i z -λ i + (1 -z i ) -λ φ old i z -λ i + (1 -z i ) -λ )) + ∂ ∂z π θ|z (a|s) log π θ|z (a|s) π θ old |z (a|s) da = 2λ m-1 i=0 ( φ i z -λ-1 i + (1 -z i ) -λ-1 φ i z -λ i + (1 -z i ) -λ - φ old i z -λ-1 i + (1 -z i ) -λ-1 φ old i z -λ i + (1 -z i ) -λ ) + ∂ ∂z π θ|z (a|s) log π θ|z (a|s) π θ old |z (a|s) da (23)

We can reach the same claim as above that the first term could be removed if we set z = z → 0. And the second term could also be eliminated as first order derivative of KL divergence between two different distributions parametrized with almost but not entirely null networks if z → 0.

C.2 RELAXING ANALYTICAL KL DIVERGENCE WITH TRUST REGION:

Ones may argue that the second term in ( 17) is decomposable to

EQUATION

∇ φ q φ (z) π θ|z (a|s)(log π θ|z (a|s) -log π θ old |z (a|s))dadz (25) such that (24) has analytical form as q is either diagonal Gaussian or Bernoulli and thus the Monte Carlo gradient estimate and the MuProp approximation above need only to be exerted on (25). However, it can be proved that ∇ φ D KL (q φi (z i )||q φ old i (z i )) is still omittable with trust region (Schulman et al., 2015b) . When first proposed in Schulman et al. (2015b) , trust region is a relaxation on D KL to encourage efficient learning, with the idea that step size will only be adapted if the trust region constraint is violated, i.e. D KL > δ. The adaptation of step size is replaced in Wang et al. (2016) with a mechanism to clip gradients from surrogate loss adaptively only if D KL > δ to further boost the efficiency.

We prove below that D kl will almost never violate the constraint δ = 1 proposed in Wang et al. (2016) . For Gaussian dropout we have:

EQUATION

where we let x = ∆φi φ old i . According to our experiment, as well as some conclusions in the literature for dropout (Kingma et al., 2015) , φ i ∈ (0.005, 0.5). Obviously x ∈ (0, 1),

D KL (q φi (z i )||q φ old i (z i )) ∈ (0, 1).

For binary dropout we have:

EQUATION

We can thus discuss how D KL changes with ∆φ i and φ old i :

EQUATION

EQUATION

With h(x) = log x 1-x , x ∈ (0.005, 0.5), it is easy to see that ∂D KL ∂∆φi = 0 when ∆φ i = 0, ∂D KL ∂∆φi > 0 when ∆φ i > 0 and ∂D KL ∂∆φi < 0 when ∆φ i < 0. Similarly, when φ old = 0.5, ∂D KL ∂φ old = 0; when φ old > 0.5, ∂D KL ∂φ old > 0; when φ old < 0.5, ∂D KL ∂φ old < 0. Hence D KL (q φi (z i )||q φ old i (z i )) reaches its maximum at |∆φ i | max and |φ old i -0.5| max . It is reasonable to set |∆φ i | max = 0.1, leading us to D KL (q φi (z i )||q φ old i (z i )) max ≈ 0.225 < δ = 1. We reach the same claim that ∇ φ D KL (π θ,φ (•|s)||π θ old ,φ old (•|s)) could be stopped for the sake of learning efficiency at a cost of acceptable bias.

C.3 REMEDY BIAS WITH MEAN POLICY:

As derived above, the regularization term only back-propagates gradients to θ:

EQUATION

where N is the number of dropout policies in this batch. Note that from line 3 to line 4 we simply remove log q φ (z) q φ old (z) , which can be regarded as substracting it as a baseline b(z) to reduce variance. Obviously, (30) has similar level of variance as (9), closing our discussion on variance reduction even though some other techniques could possibly go further.

Seeing the lack of regularization on φ due to the gradient omission, we further enforce the idea that dropout policy had better to be close to each other, which is the supposed role of regularizer on φ. This could be intuitively understood as a remedy from θ to φ:

EQUATION

Noticing that this term has a complexity of O(N 2 ), we replace π θ,φ with a mean policy π θ = π θ|z .

In the deep learning literature, it is fairly prevalent to use mean network for dropout training with some adjustment in gradient back-propagation, due to stability and effiency concern. For instance, it is proved by Wang & Manning (2013) that the sampling process for dropout network with sigmoid or ReLU activators could be avoided by integrating a Gaussian approximation at each layer of a neural network. Therefore, we have (32)

Empirically, we found little difference between fast dropout back-propagation and normal backpropagation when (32) is combined into (12).

D VARITAIONAL DROPOUT AND LOCAL REPARAMETRIZATION:

As introduced in Section 2.1, multiplicative dropout at neuron activation of kth layer can also be viewed as a multiplicative noise for the input at k + 1th layer:

EQUATION

)

After a simple rearrangement W(k+1)T = W (k+1)T D k z , we have:

EQUATION

)

That is, the stochastic neuron activation in networks with Gaussian multiplicatice dropout can also be regarded as stochastic neuron activation in Noisy Networks with correlated Gaussian parameter noise, whose means equal the corresponding ones in networks with dropout.

E HYPERPARAMETERS:

For most of the hyperparamters, we follow the setting in original PPO paper. Though, to emphasis the scalability of our method, we use 2 parallel enviroments and only 1 minibatch in each epoch. • SparseHalfCheetah (S ⊂ R 17 , A ⊂ R 6 ), which only yields a reward if the agent crosses a distance threshold, • SparseMountainCar (S ⊂ R 2 , A ⊂ R), which only yields a reward if the agent drives up the hill, • SparseDoublePendulum (S ⊂ R 6 , A ⊂ R), which only yields a reward if the agent reaches the upright position.

:

Though the concrete form is not provided inSchulman et al. (2017b), it is given inDhariwal et al. (2017) publicly released by the authors

Figures and Tables:

Figure 1: (a) Conceptual graph to illustrate the hierarchical stochasticity in NADPEx. Agent spontaneously explores towards different directions with different dropouts. (b) Graphical model for an MDP with NADPEx. z is a global random variable in one episode τ .

Figure 3: Experiments with NADPEx and parameter noise in sparse reward envs.

Figure 4: Comparison between NADPEx and bootstrap with Gaussian dropout.

Figure 5: NADPEx KL PPO in Walker2d. Left: learning curves; right: true D KL (π θ old |z ||π θ|z ).

θ D KL (π θ (•|s)||π θ old |zi (•|s)) θ π θ (a|s)(log π θ (a|s) -log π θ old |zi (a|s)a|s)∇ θ log π θ (a|s)(log π θ (a|s) -log π θ old |zi (a|s)a∼π θ|z [∇ θ log π θ (a|s)(log π θ (a|s) -log π θ old |zi (a|s))] a∼π θ|z [ 1 2 ∇ θ (log π θ (a|s) -log π θ old |zi (a|s)) 2 ].

Figure 6: NADPEx in standard envs where Gaussian dropout is used

Figure 7: NADPEx in standard envs where binary dropout is used

KL (π old θ (•|s)|π θ (•|s)) is actually a relaxation of total variational divergence according to Schulman et al. (2015b), whose first-order derivative is 0 when θ is close to θ old , D KL (π θ (•|s)||π old θ (•|s)) is a natural replacement. Combining the first-order derivative of D KL (π θ (•|s)||π old θ (•|s)) (Schulman et al., 2017a) (check Appendix A for a proof):

Table content: <table/>

Hyperparamters for PPO HyperparametersValue

Table content: <table><tr><td>Horizon</td><td>2048</td></tr><tr><td>Adam stepsize</td><td>3 × 10 -4</td></tr><tr><td>Num. epochs</td><td>10</td></tr><tr><td>Num. minibatch</td><td>1</td></tr><tr><td>Discount (γ)</td><td>0.99</td></tr><tr><td>GAE λ</td><td>0.95</td></tr><tr><td>PPO clip</td><td>0.2</td></tr><tr><td>Num. layers</td><td>2</td></tr><tr><td colspan="2">Num. hidden units 64</td></tr></table>

Peer review (iclr2019):
 The authors propose a new on-policy exploration strategy by using a policy with a hierarchy of stochasticity. The authors use a two-level hierarchical distribution as a policy, where the global variable is used for dropout. This work is interesting since the authors use dropout for policy learning and exploration. The authors show that parameter noise exploration is a particular case of the proposed policy. The main concern is the gap between the problem formulation and the actual optimization problem in Eq 12. I am very happy to give a higher rating if the authors address the following points. Detailed Comments (1) The authors give the derivation for Eq 10. However, it is not obvious that how to move from line 3 to line 4 at Eq 15. Minor: Since the action is denoted by "a", it will be more clear if the authors use another symbol to denote the parameter of q(z) instead of ".alpha" at Eq 10 and 15. (2) Due to the use of the likelihood ratio trick, the authors use the mean policy as an approximation at Eq 12. Does such approximation guarantee the policy improvement? Any justification? (3) Instead of using the mean policy approximation in Eq 12, the authors should consider existing Monte Carlo techniques to reduce the variance of the gradient estimation. For example, [1] could be used to reduce the variance of gradient w.r.t. .phi. Note that the gradient is biased if the mean policy approximation is used. (4) Are .theta and .phi jointly and simultaneously optimized at Eq 12? The authors should clarify this point. (5) Due to the mean policy approximation, does the mean policy depend on .phi? The authors should clearly explain how to update .phi when optimizing Eq 12. (6) If the authors jointly and simultaneously optimize .theta and .phi, why a regularization term about q_{.phi}(z) is missing in Eq 12 while a regularization term about .pi_{.theta|z} does appear in Eq 12? (7) The authors give the derivations about .theta such as the gradient and the regularization term about .theta (see, Eq 18-19). However, the derivations about .phi are missing. For example, how to compute the gradient w.r.t. .phi? Since the mean policy is used, it is not apparent that how to compute the gradient w.r.t. .phi. Minor, 1/2 is missing in the last line of Eq 19. Reference: [1] AUEB, Michalis Titsias RC, and Miguel Lázaro-Gredilla. "Local expectation gradients for black box variational inference." In Advances in neural information processing systems, pp. 2638-2646. 2015.