Year: 2020

Abstract:
In policy search methods for reinforcement learning (RL), exploration is often performed by injecting noise either in action space at each step independently or in parameter space over each full trajectory. In prior work, it has been shown that with linear policies, a more balanced trade-off between these two exploration strategies is beneficial. However, that method did not scale to policies using deep neural networks. In this paper, we introduce Deep Coherent Exploration, a general and scalable exploration framework for deep RL algorithms on continuous control, that generalizes step-based and trajectory-based exploration. This framework models the last layer parameters of the policy network as latent variables and uses a recursive inference step within the policy update to handle these latent variables in a scalable manner. We find that Deep Coherent Exploration improves the speed and stability of learning of A2C, PPO, and SAC on several continuous control tasks.

heading: 1 INTRODUCTION
text: The balance of exploration and exploitation (Kearns & Singh, 2002; Jaksch et al., 2010) is a longstanding challenge in reinforcement learning (RL). With insufficient exploration, states and actions with high rewards can be missed, resulting in policies prematurely converging to bad local optima. In contrast, with too much exploration, agents could waste their resources trying suboptimal states and actions, without leveraging their experiences efficiently. To learn successful strategies, this trade-off between exploration and exploitation must be balanced well, and this is known as the exploration vs. exploitation dilemma.
At a high level, exploration can be divided into directed strategies and undirected strategies (Thrun, 1992; Plappert et al., 2018). While directed strategies aim to extract useful information from existing experiences for better exploration, undirected strategies rely on injecting randomness into the agent’s decision-making. Over the years, many sophisticated directed exploration strategies have been proposed (Tang et al., 2016; Ostrovski et al., 2017; Houthooft et al., 2016; Pathak et al., 2017). However, since these strategies still require lower-level exploration to collect the experiences, or are either complicated or computationally intensive, undirected exploration strategies are still commonly used in RL literature in practice, where some well-known examples are -greedy (Sutton, 1995) for discrete action space and additive Gaussian noise for continuous action space (Williams, 1992). Such strategies explore by randomly perturbing agents’ actions at different steps independently and hence are referred to as performing step-based exploration in action space (Deisenroth et al., 2013).
As an alternative to those exploration strategies in action space, exploration by perturbing the weights of linear policies has been proposed (Rückstieß et al., 2010; Sehnke et al., 2010; Kober & Peters, 2008). Since these strategies in parameter space naturally explore conditioned on the states and are usually trajectory-based (only perturb the weights at the beginning of each trajectory) (Deisenroth et al., 2013), they have the advantages of being more consistent, structured, and global (Deisenroth et al., 2013). Later, van Hoof et al. (2017) proposed a generalized exploration (GE) scheme, bridging the gap between step-based and trajectory-based exploration in parameter space. With the advance of deep RL, NoisyNet (Fortunato et al., 2018) and Parameter Space Noise for Exploration (PSNE) (Plappert et al., 2018) were introduced, extending parameter-space exploration strategies for policies using deep neural networks.
Although GE, NoisyNet, and PSNE improved over the vanilla exploration strategies in parameter space and were shown leading to more global and consistent exploration, they still suffer from several limitations. Given this, we propose a new exploration scheme with the following characteristics.
1. Generalizing Step-based and Trajectory-based Exploration (van Hoof et al., 2017) Since both NoisyNet and PSNE are trajectory-based exploration strategies, they are considered relatively inefficient and bring insufficient stochasticity (Deisenroth et al., 2013). Following van Hoof et al. (2017), our method improves by interpolating between step-based and trajectory-based exploration in parameter space, where a more balanced trade-off between stability and stochasticity can be achieved.
2. Recursive Analytical Integration of Latent Exploring Policies NoisyNet and PSNE address the uncertainty from sampling exploring policies using Monte Carlo integration, while GE uses analytical integration on full trajectories, which scales poorly in the number of time steps. In contrast, we apply analytical and recurrent integration after each step, which leads to low-variance and scalable updates.
3. Perturbing Last Layers of Policy Networks Both NoisyNet and PSNE perturb all layers of the policy network. However, in general, only the uncertainty in parameters of the last (linear) layer can be integrated analytically. Furthermore, it is not clear that deep neural networks can be perturbed in meaningful ways for exploration (Plappert et al., 2018). We thus propose and evaluate an architecture where perturbation is only applied on the parameters of the last layer.
These characteristics define our contribution, which we will refer to as Deep Coherent Exploration. We evaluate the coherent versions of A2C (Mnih et al., 2016), PPO (Schulman et al., 2017), and SAC (Haarnoja et al., 2018), where the experiments on OpenAI MuJoCo (Todorov et al., 2012; Brockman et al., 2016) tasks show that Deep Coherent Exploration outperforms other exploration strategies in terms of both learning speed and stability.

heading: 2 RELATED WORK
text: As discussed, exploration can broadly be classified into directed and undirected strategies (Thrun, 1992; Plappert et al., 2018), with undirected strategies being commonly used in practice because of their simplicity. Well known methods such as -greedy (Sutton, 1995) or additive Gaussian noise (Williams, 1992) randomly perturb the action at each time step independently. These high-frequency perturbations, however, can result in poor coverage of the state-action space due to random-walk behavior (Rückstieß et al., 2010; Deisenroth et al., 2013), washing-out of exploration by the environment dynamics (Kober & Peters, 2008; Rückstieß et al., 2010; Deisenroth et al., 2013), and to potential damage to mechanical systems (Koryakovskiy et al., 2017).
One alternative is to instead perturb the policy in parameter space, with the perturbation held constant for the duration of a trajectory. Rückstieß et al. (2010) and Sehnke et al. (2010) showed that such parameter-space methods could bring improved exploration behaviors because of reduced variance and faster convergence, when combined with REINFORCE (Williams, 1992) or Natural Actor-Critic (Peters et al., 2005).
Another alternative to independent action-space perturbation, is to correlate the noise applied at subsequent actions (Morimoto & Doya, 2000; Wawrzynski, 2015; Lillicrap et al., 2016), for example by generating perturbations from an Ornstein-Uhlenbeck (OU) process (Uhlenbeck & Ornstein, 1930). Later, van Hoof et al. (2017) used the same stochastic process but in the parameter space of the policy. This approach uses a temporally coherent exploring policy, which unifies step-based and trajectory-based exploration. Moreover, the author showed that, with linear policies, a more delicate balance between these two extreme strategies could have better performance. However, this approach was derived in a batch mode setting and requires storing the full trajectory history and the inversion of a matrix growing with the number of time step. Thus, it does not scale well to long trajectories or complex models.
Although these methods pioneered the research of exploration in parameter space, their applicability is limited. More precisely, these methods were only evaluated with extremely shallow (often linear) policies and relatively simple tasks with low-dimensional state spaces and action spaces. Given this,
NoisyNet (Fortunato et al., 2018), PSNE (Plappert et al., 2018) and Stochastic A3C (SA3C) (Shang et al., 2019) were proposed, introducing more general and scalable methods for deep RL algorithms.
All three of these methods can be seen as learning a distribution over policies for trajectory-based exploration in parameter space. These exploring policies are sampled by perturbing the weights across all layers of a deep neural network, with the uncertainty from sampling being addressed by Monte Carlo integration. Whereas NoisyNet learns the magnitudes of the noise for each parameter, PSNE heuristically adapts a single magnitude for all parameters.
While showing good performance in practice (Fortunato et al., 2018; Plappert et al., 2018), these methods suffer from two potential limitations. Firstly, trajectory-based strategies can be inefficient as only one strategy can be evaluated for a potentially long trajectory (Deisenroth et al., 2013), which could result in a failure to escape local optima. Secondly, Monte Carlo integration results in high-variance gradient estimates which could lead to oscillating updates.

heading: 3 BACKGROUND
text: This section provides background for reinforcement learning and related deep RL algorithms.

heading: 3.1 REINFORCEMENT LEARNING
text: Reinforcement learning is a sub-field of machine learning that studies how an agent learns strategies with high returns through trial-and-error by interacting with an environment. This interaction between an agent and an environment is described using Markov Decision Processes (MDPs). A MDP is a tuple (S,A, r, P, γ), where S is the state space, A is the action space, r : S × A × S → R is the reward function with rt = r (st,at, st+1), P : S × A × S → R+ is the transition probability function, and γ is a discount factor indicating the preference of short-term rewards.
In RL with continuous action space, an agent aims to learn a parametrized (e.g. Gaussian) policy πθ (a|s) : S ×A → R+, with parameters θ, that maximizes the expected return over trajectories:
J(θ) = Eτ∼p(τ |πθ)[R(τ)], (1)
where τ = (s0,a0, ...,aT−1, sT ) is a trajectory and R(τ) = ∑T t=0 γ trt is the discounted return.

heading: 3.2 DEEP REINFORCEMENT LEANING ALGORITHMS
text: Deep reinforcement learning combines deep learning and reinforcement learning, where policies and value functions are represented by deep neural networks for more sophisticated and powerful function approximation. In our experiments, we consider the following three deep RL algorithms.
Advantage Actor-Critic (A2C) Closely related to REINFORCE (Williams, 1992), A2C is an on-policy algorithm proposed as the synchronous version of the original Asynchronous Advantage Actor-Critic (A3C) (Mnih et al., 2016). The gradient of A2C can be written as:
∇θJ(θ) = Eτ∼p(τ |θ) [ T−1∑ t=0 ∇θ log πθ (at|st)Aπθ (st,at) ] , (2)
where Aπθ (st,at) is the estimated advantage following policy πθ.
Proximal Policy Optimization (PPO) PPO is an on-policy algorithm developed to determine the largest step for update while still keeping the updated policy close to the old policy in terms of Kullback–Leibler (KL) divergence. Instead of using a second-order method as in Trust Region Policy Optimization (TRPO) (Schulman et al., 2015), PPO applies a first-order method and combines several tricks to relieve the complexity. We consider the primary variant of PPO-Clip with the following surrogate objective:
LCLIPθk (θ) = Eτ∼p(τ |θk) [ T−1∑ t=0 [ min (rt(θ), clip (rt(θ), 1− , 1 + ))A πθk t ]] , (3)
where rt(θ) = πθ (at|st) πθk (at|st) and is a small threshold that approximately restricts the distance between the new policy and the old policy. In practice, to prevent the new policy from changing too fast, the KL divergence from the new policy to the old policy approximated on a sampled batch is often used as a further constraint.
Soft Actor-Critic (SAC) As an entropy-regularized (Ziebart et al., 2008) off-policy actor-critic method (Lillicrap et al., 2016; Fujimoto et al., 2018) with a stochastic policy, SAC (Haarnoja et al., 2018) learns the optimal entropy-regularized Q-function through ‘soft’ Bellman back-ups with offpolicy data:
Qπ(s,a) = Es′∼p(s′|s,a),ã′∼π(ã′|s′) [r + γ (Qπ (s′, ã′) + αH (π (ã′|s′)))] , (4)
whereH is the entropy and α is the temperature parameter. The policy is then learned by maximizing the expected maximum entropy V -function via the reparameterization trick (Kingma et al., 2015).

heading: 4 DEEP COHERENT EXPLORATION FOR CONTINUOUS CONTROL
text: To achieve the desiderata in Section 1, we propose Deep Coherent Exploration, a method that models the policy as a generative model with latent variables. This policy is represented as πwt,θ(at|st) = N (Wtfθ(st) + bt,Λ−1a ). Here wt denotes all last layer parameters of policy network at step t by combining Wt and bt, θ denotes the parameters of the policy network except for the last layer, and Λa is a fixed and diagonal precision matrix. Our method treats the last layer parameters wt as latent variables with marginal distribution wt ∼ N ( µt,Λ −1 t ) , where µt and Λt are functions of learnable parameters µ and Λ respectively. In this model, all learnable parameters can be denoted as ζ = (µ,Λ,θ). We provide a graphical model of Deep Coherent Exploration in Appendix A.
As in van Hoof et al. (2017), Deep Coherent Exploration generalizes step-based and trajectory-based exploration by constructing a Markov chain of wt. This Markov chain specifies joint probabilities through an initial distribution p0(w0) = N ( µ,Λ−1 ) and the conditional distribution p(wt|wt−1). This latter term explicitly expresses temporal coherence between subsequent parameter vectors. In this setting, step-based exploration corresponds to the extreme case when p(wt|wt−1) = p0(wt), and trajectory-based exploration corresponds to another extreme case when p(wt|wt−1) = δ(wt − wt−1), where δ is the Dirac delta function. To ensure the marginal distribution of wt will be equal to the initial distribution p0 at any step t, we directly follow van Hoof et al. (2017) with the following transition distribution for wt:
p(wt|wt−1) = N ( (1− β)wt−1 + βµ, (2β − β2)Λ−1 ) , (5)
where β is a hyperparameter that controls the temporal coherency of wt and wt−1. Then, the two extreme cases corresponds to β = 0 for trajectory-based exploration and β = 1 for step-based exploration, while the intermediate exploration corresponds to β ∈ (0, 1). For intermediate values of β, we obtain smoothly changing policies that sufficiently explore, while reducing high-frequency perturbations.

heading: 4.1 ON-POLICY DEEP COHERENT EXPLORATION
text: Our method can be combined with all on-policy policy gradient methods and here we present this adaptation with REINFORCE (Williams, 1992). Starting from the RL objective in Equation 1:
∇ζJ(ζ) = Eτ∼p(τ |ζ) [∇ζ log p(τ |ζ)R(τ)] , (6)
the gradients w.r.t the sampled trajectory can be obtained using standard chain rule:
∇ζ log p(τ |ζ) = T−1∑ t=0 ( ∇ζ log p(at|s[0:t],a[0:t−1], ζ) ) . (7)
Here, since information can still flow through the unobserved latent variable wt, our policy is not Markov anymore. To simplify this dependency, we introduce wt into p(at|s[0:t],a[0:t−1], ζ):
p(at|s[0:t],a[0:t−1], ζ) = ∫ p(at,wt|s[0:t],a[0:t−1], ζ)dwt (8)
= ∫ p(at|st;wt,θ)︸ ︷︷ ︸
Gaussian policy πwt,θ(at|st) p(wt|s[0:t−1],a[0:t−1],µ,Λ,θ)︸ ︷︷ ︸ forward message α(wt) dwt, (9)
where the first factor is the Gaussian action probability and the second factor can be interpreted as the forward message α(wt) along the chain. We decompose α(wt) by introducing wt−1 :
α(wt) = ∫ p(wt,wt−1|s[0:t−1],a[0:t−1], ζ)dwt−1 (10)
= ∫ p(wt|wt−1;µ,Λ)︸ ︷︷ ︸
transition probability of wt
p(at−1|st−1;wt−1,θ)α(wt−1) Zt−1 dwt−1, (11)
where the first factor is the transition probability of wt (Equation 5) and Zt−1 in the factor term is a normalizing constant. Since Gaussians are closed under marginalization and condition, the second factor can be obtained analytically without the need of computing the normalizing constant Zt−1. Moreover, α(wt−1) is a Gaussian by mathematical induction from the initial step. As a result, we arrive at an efficient recursive expression for exact inference of wt. Again, with the property of Gaussians, all integrals appearing above can be solved analytically, where the marginal action probability given the history at each step t can be obtained and used for policy updates.
Summarizing, non-Markov policies require substituting the regular p(at|st;θ) term in the update equation with p(at|s[0:t],a[0:t−1], ζ) (Equation 7). Equations 8-11 show how this expression can be efficiently calculated recursively. Except for this substitution, learning algorithms like A2C and PPO can proceed as normal. For detailed mathematical derivation, please refer to Appendix B.

heading: 4.2 OFF-POLICY DEEP COHERENT EXPLORATION
text: Combining our method with off-policy methods (Lillicrap et al., 2016; Fujimoto et al., 2018; Haarnoja et al., 2018) requires defining both the behavior policy and the update equation. The behavior policy is the same as in the on-policy methods discussed earlier (Equation 5). The policy update procedure may require adjustments for specific algorithms. Here, we show how to adapt our method for SAC (Haarnoja et al., 2018). In the SAC policy update, the target policy is adapted towards the exponential of the new Q-function. The target policy here is the marginal policy, rather than the policy conditioned on the sampled w, as this second option would ignore the dependence on µ and Λ. This consideration leads to the following objective for policy update:
J(ζ) = Est∼D [ KL ( p (at|st, ζ) ∥∥∥exp (Qφ (st,at)) Zφ (st) )] (12)
= Est∼D,at∼p(at|st,ζ) [log p(at|st, ζ)−Qφ (st,at)] , (13)
where φ denotes the parameters of the Q-function and p (at|st, ζ) is the marginal policy which can again be obtained analytically (Equation 22):
p (at|st, ζ) = ∫
p (at|st;w0,θ)︸ ︷︷ ︸ Gaussian policy πw0,θ(at|st) p(w0|µ,Λ)︸ ︷︷ ︸ marginal probability of w0 dw0, (14)
where all parameters can be learned via the reparameterization trick (Kingma et al., 2015).

heading: 5 EXPERIMENTS
text: For the experiments, we compare our method with NoisyNet (Fortunato et al., 2018), PSNE (Plappert et al., 2018) and standard action noise. This comparison is evaluated in combination of A2C (Mnih et al., 2016), PPO (Schulman et al., 2017) and SAC (Haarnoja et al., 2018) on OpenAI Gym MuJoCo (Todorov et al., 2012; Brockman et al., 2016) continuous control tasks.
For exploration in parameter space, we use a fixed action noise with a standard deviation of 0.1. For A2C and PPO, their standard deviations of parameter noise are all initialized at 0.017, as suggested in Fortunato et al. (2018). For SAC, we initialize the standard deviation of parameter noise at 0.034 for both our method and PSNE as it gave better results in practice. Besides, Deep Coherent Exploration learns the logarithm of parameter noise, while NoisyNet learns the parameter noise directly, and PSNE adapts the parameter noise. To protect the policies from changing too dramatically, we consider three small values of β (0.0, 0.01, and 0.1) for Deep Coherent Exploration, where we use β = 0.01 for comparative evaluation with other exploration strategies. Our implementation of NoisyNet is based on the code from Kaixhin1. For PSNE, we refer to author’s implementation in OpenAI Baselines2 (Dhariwal et al., 2017) and the original paper, where we set the KL threshold for A2C and PPO to 0.01 and the MSE threshold for SAC to 0.1. On the other hand, exploration with action noise uses the default setting proposed by Achiam (2018), where the standard deviation of action noise is initialized at around 0.6 for A2C and PPO. For SAC, in the baseline setting the standard deviation of action noise is output by the policy network.
In all experiments, agents are trained with a total of 106 environmental steps, where they are updated after each epoch. A2C and PPO use four parallel workers, where each worker collects a trajectory of 1000 steps for each epoch, resulting in epochs with 4000 steps in total. After each epoch, both A2C and PPO update their value functions for 80 gradient steps. At the same time, A2C updates its policy for one gradient step, while PPO updates its policy for up to 80 gradient steps until the KL constraint is satisfied. SAC uses a single worker, with a step size of 4000 for each epoch. After every 50 environmental steps, both policy and value functions are updated for 50 gradient steps. To make the parameter noise stable, we adapt the standard deviation of parameter noise after each epoch.
Our implementation of A2C, PPO, and SAC with different exploration strategies are adapted based on OpenAI Spinning Up (Achiam, 2018) with default settings. All three algorithms use two-layer feedforward neural networks with the same network architectures for both policy and value function. To be more precise, A2C and PPO use network architectures with 64 and 64 hidden nodes activated by tanh units. In comparison, SAC uses a network architecture of 256 and 256 hidden nodes activated by rectified linear units (ReLU). Parameters of policies and value functions in all three algorithms are updated using Adam (Kingma & Ba, 2015). A2C and PPO use a learning rate of 3 · 10−4 for the policies and a learning rate of 10−3 for the value functions. SAC uses a single learning rate of 10−3 for both policy and value function.
For each task, we evaluate the performance of agents after every 5 epochs, with no exploration noise. Additionally, each evaluation reports the average reward of 10 episodes for each worker. To mitigate the randomness within environments and policies, we report our results as the average over 10 random seeds. All settings not explicitly explained in this section, are set to the code base defaults. For more details, please refer to documents and source code from OpenAI Spinning Up (Achiam, 2018) and our implementation3.

heading: 5.1 COMPARATIVE EVALUATION
text: In this section, we present the results for A2C (Mnih et al., 2016), PPO (Schulman et al., 2017), and SAC (Haarnoja et al., 2018) on three control tasks, with the additional results shown in Appendix D. Figure 1 shows that, overall, Coherent-A2C outperforms all other A2C-based methods in terms of learning speed, final performance, and algorithm stability. In particular, given that Antv2 is considered a challenging task, Deep Coherent Exploration considerably accelerates learning speed. For PPO-based method, our method still outperforms NoisyNet and PSNE significantly in all tasks. However, our method’s advantage compared to standard action noise is smaller. Particularly, Coherent-PPO underperforms PPO in Walker2d-v2. Two reasons might explain this. Firstly, some environments might be more unstable and require a larger degree of exploration, which favors PPO as it initializes its action noise with much greater value. Secondly, because of having extra parameters, policies of Coherent-PPO, NoisyNet-PPO, and PSNE-PPO tend to satisfy the KL constraint in fewer update steps, which leads to slower learning. For SAC, the advantages of our method are smaller compared to A2C and PPO. More specifically, Coherent-SAC learns slightly
1https://github.com/Kaixhin/NoisyNet-A3C 2https://github.com/openai/baselines/tree/master/baselines/ddpg 3Source code to be released after review
faster than SAC in HalfCheetah-v2 and achieves the highest average returns in Walker2d-v2 and Ant-v2. Furthermore, Coherent-SAC shows variance lower than PSNE-SAC but higher than the baseline SAC.

heading: 5.2 ABLATION STUDIES
text: In this section, we present three separate ablation studies to clarify the effect of each characteristic discussed in Section 1. These ablation studies are performed with A2C, to ensure that all characteristics are applicable and because the fixed number of gradient steps puts the methods on equal footing. Here we show the results for HalfCheetah-v2, where the full results can be found in Appendix D.
Generalizing Step-based and Trajectory-based Exploration As shown in Figure 2a, we evaluate three different values 0.0, 0.01, 0.1 of β for Coherent-A2C. Here both two intermediate strategies (β = 0.01 and β = 0.1) outperforms the trajectory-based strategy (β = 0.0). Coherent-A2C with β = 0.01 seems to achieve the best balance between randomness and stability, with a considerably higher return than the other two.
Analytical Integration of Latent Exploring Policies We introduce OurNoisyNet for comparison. OurNoisyNet equips a noisy linear layer for only its last layer, and this layer learns the logarithm of standard deviation, as in Deep Coherent Exploration. We compare Coherent-A2C using β = 0.0 and OurNoisyNet-A2C, with the only difference thus being whether we integrate analytically or use the reparameterization trick (Kingma et al., 2015).
We first measure the variance of gradient estimates in both methods. This variance is measured by computing the trace of the covariance matrix using 10 gradient samples. We report this measure in six stages during training, as shown in Figure 2c. We can observe that analytical integration leads to lower-variance gradient estimates across all training stages for HalfCheetah-v2. We further present the learning curves of both methods in Figure 2b, where Coherent-A2C with β = 0.0 shows higher return than OurNoisyNet-A2C. It is interesting that Coherent-A2C displays a much lower standard deviation across different random seeds. Furthermore, the lower-variance gradient estimates of Coherent-A2C could enable a larger learning rate for faster training without making policy updates unstable.
Perturbing Last Layers of Policy Networks In this part, we compare OurNoisyNet-A2C perturbed over all layers, and OurNoisyNet-A2C perturbed over only the last layer. The result is shown in Figure 2b. Somewhat to our surprise, the latter seems to perform much better. There are several possible reasons. Firstly, since it is unknown how the parameter noise (especially in lower layers) is realized in action noise, perturbing all layers of the policy network may lead to uncontrollable perturbations. Such excess exploratory noise could inhibit exploitation. Secondly, perturbing all layers might disturb the representation learning of states, which is undesirable for learning a good policy. Thirdly, perturbing only the last layer could also lead to fewer parameters for NoisyNet.

heading: 6 CONCLUSION
text: In this paper, we have presented a general and scalable exploration framework that extends the generalized exploration scheme (van Hoof et al., 2017) for continuous deep RL algorithms. In particular, recursive calculation of marginal action probabilities allows handling long trajectories and high-dimensional parameter vectors. Compared with NoisyNet (Fortunato et al., 2018) and PSNE (Plappert et al., 2018), our method has three improvements. Firstly, Deep Coherent Exploration generalizes step-based and trajectory-based exploration in parameter space, which allows a more balanced trade-off between stochasticity and coherence. Secondly, Deep Coherent Exploration analytically marginalizes the latent policy parameters, yielding lower-variance gradient estimates that stabilize and accelerate learning. Thirdly, by perturbing only the last layer of the policy network, Deep Coherent Exploration provides better control of the injected noise.
When combining with A2C (Mnih et al., 2016), PPO (Schulman et al., 2017), and SAC (Haarnoja et al., 2018), we empirically show that Deep Coherent Exploration outperforms other exploration strategies on most of the MuJoCo continuous control tasks tested. Furthermore, the ablation studies show that, while each of the improvements is beneficial, combining them leads to even faster and more stable learning. For future work, since Deep Coherent Exploration uses a fixed and small action noise, we believe one interesting direction is to study whether the learnable perturbations in action space can be combined with our method in a meaningful way for even more effective exploration.

heading: A GRAPHICAL MODEL OF DEEP COHERENT EXPLORATION
text: In this appendix, we provide a graphical model representation of Deep Coherent Exploration, shown in Figure 3. This graphical model uses the same conventions as in Bishop (2007), where empty circles denote latent random variables, shaded circles denote observed random variables, and dots denote deterministic variables.

heading: B MARGINAL ACTION PROBABILITY FOR ON-POLICY COHERENT EXPLORATION
text: As discussed in Section 4.1, forward message α(wt) is used to compute the marginal action probability given the history at step t for the final learning objective. Suppose we have the Gaussian policy as:
πwt,θ(at|st) = N (Wtxt + bt,Λ−1a ), (15)
where at ∈ Rp, xt = fθ(st) ∈ Rq , Wt ∈ Rp×q is the coefficient matrix, bt ∈ Rp is the bias vector and Λa is a constant precision matrix for the Gaussian policy. It’s helpful to represent wt ∈ Rpq+p by flattening Wt and combining bt:
wt =  w11 ... w1q ... wp1 ...
wpq b1 ... bp
 , (16)
such that the parameters could still be sampled using multivariate Gaussians. Moreover, we stack xt into Xt ∈ Rp×(pq+p):
Xt =  xTt 0 T q,1 . . . 0 T q,1 0 T q,1 1 0 . . . 0 0 0Tq,1 x T t . . . 0 T q,1 0 T q,1 0 1 . . . 0 0 ... ... . . . ... ... ... ... . . . ... ...
0Tq,1 0 T q,1 . . . x T t 0 T q,1 0 0 . . . 1 0 0Tq,1 0 T q,1 . . . 0 T q,1 x T t 0 0 . . . 0 1  , (17) where 0q,1 is a q-dimension zero column vector. After this transformation, the Gaussian policy is represented equivalently as:
πwt,θ(at|st) = N (Xtwt,Λ−1a ). (18)
B.1 BASE CASE
For the base case t = 0, forward message α(w0) and the initial transition probability of w0 is identical by definition:
α(w0) = p0(w0;µ,Λ) = N ( µ,Λ−1 ) . (19)
Additionally, the action probability is given by: πw0,θ(a0|s0) = N (X0w0,Λ−1a ). (20)
With the property of multivariate Gaussians, we obtain the marginal action probability given the history at t = 0:
log p(a0|s0,µ,Λ,θ) = log ∫ πw0,θ(a0|s0)α(w0)dw0 (21)
= logN (X0µ,Λ−1a + X0Λ−1XT0 ). (22)
B.2 GENERAL CASE
For the general case of step t > 0, we need the state st−1, action at−1 as well as mean and covariance of forward message α(wt−1) stored from previous step. Suppose α(wt−1) is written as: α(wt−1) = N (vt−1,L−1t−1), (23) and the action probability from the previous step is given by:
πwt−1,θ(at−1|st−1) = N ( Xt−1wt−1,Λ −1 a ) . (24)
We have directly: p(wt−1|s[0:t−1],a[0:t−1],µ,Λ,θ) = N (ut−1,Σt−1) , (25)
with ut−1 = Σt−1 ( XTt−1Λaat−1 + Lt−1vt−1 ) (26)
Σt−1 = ( Lt−1 + X T t−1ΛaXt−1 )−1 . (27)
Combining the transition probability of wt: p(wt|wt−1;µ,Λ) = N ( (1− β)wt−1 + βµ, (2β − β2)Λ−1 ) , (28)
we obtain the forward message α(wt): α (wt) = N ( vt,L −1 t ) , (29)
where vt = (1− β)ut−1 + βµ (30)
L−1t = (2β − β2)Λ−1 + (1− β)2Σt−1. (31) Here, vt and L−1t should be stored and used for exact inference of α(wt+1) at the next step. Finally, the marginal action probability given the history at step t > 0 is given by:
log p(at|s[0:t],a[0:t−1],µ,Λ,θ) = log ∫ πwt,θ(at|st)α(wt)dwt (32)
= logN (Xtvt,Λ−1a + XtL−1t XTt ). (33)

heading: C DEEP COHERENT REINFORCEMENT LEARNING
text: Here, we provide a brief introduction of adapting Deep Coherent Exploration for A2C (Mnih et al., 2016), PPO (Schulman et al., 2017) and SAC (Haarnoja et al., 2018). Respectively, we call them Coherent-A2C, Coherent-PPO and Coherent-SAC.
C.1 COHERENT ADVANTAGE ACTOR-CRITIC (COHERENT-A2C)
Coherent-A2C is straightforward to implement. To do that, one could just replace the original A2C gradient estimates ∇̂θJ(θ) with the on-policy coherent gradient estimates ∇̂µ,Λ,θJ(µ,Λ,θ). The pseudo-code of single-worker Coherent-A2C is shown in Algorithm 1.
Algorithm 1: Coherent-A2C Input: initial policy parameters µ0,Λ0,θ0, initial value function parameters φ0.
1 for k=0,1,2,...,K do 2 Create a buffer Dk for collecting a trajectory τk with T steps. 3 for t=0,...,T do 4 if t=0 then 5 Sample last layer parameters of policy network wt ∼ p0(wt;µk,Λk) and store wt. 6 else 7 Sample last layer parameters of policy network wt ∼ p(wt|wt−1;µk,Λk) and store wt. 8 Observe state st and select action at ∼ πwt,θk(at|st). 9 Execute at in the environment.
10 Observe next state st+1, reward rt, done signal d, and store (st,at, rt, st+1, d) in buffer Dk. 11 If st+1 is terminal, reset environment state. 12 Infer forward message α(wt) using previous state st−1, previous action at−1 as well as mean vt−1 and covariance L−1t−1 of previous forward message α(wt−1). 13 Store mean vt and covariance L−1t of current forward message α(wt). 14 Compute marginal action probability p(at|s[0:t],a[0:t−1],µk,Λk,θk). 15 Compute rewards-to-go Rt and any kind of advantage estimates Ât based on current value function Vφk for all steps t. 16 Estimate gradient of the policy:
∇̂µ,Λ,θJ(µ,Λ,θ) = T−1∑ t=0 ∇µ,Λ,θ log p(at|s[0:t],a[0:t−1],µk,Λk,θk)Ât,
and update the policy by performing a gradient step:
µk+1 ← µk + αµ∇̂µJ(µ,Λ,θ) Λk+1 ← Λk + αΛ∇̂ΛJ(µ,Λ,θ) θk+1 ← θk + αθ∇̂θJ(µ,Λ,θ).
17 Learn value function by minimizing the regression mean-squared error:
L(φ) = 1
T T∑ t=0 (Vφk (st)−Rt) 2 ,
and update the value function by performing a gradient step:
φk+1 ← φk + αφ∇̂φL(φ).
C.2 COHERENT PROXIMAL POLICY OPTIMIZATION (COHERENT-PPO)
Coherent-PPO can be implemented in a similar way. As in Coherent-A2C, we substitute the original objective LCLIPθk (θ) with L CLIP µk,Λk,θk (µ,Λ,θ), which is given by:
LCLIPµk,Λk,θk(µ,Λ,θ) (34)
= Eτ∼p(τ |µk,Λk,θk) T−1∑ t=0 [ min (rt(µ,Λ,θ), clip (rt(µ,Λ,θ), 1− , 1 + ))A πµ,Λ,θ t ] , (35)
where rt(µ,Λ,θ) = p(at|s[0:t],a[0:t−1],µ,Λ,θ)
p(at|s[0:t],a[0:t−1],µk,Λk,θk) .
Here, after each step of policy update, p(at|s[0:t],a[0:t−1],µ,Λ,θ) from the new policy should be evaluated on the most recent trajectory τk for both next update and approximated KL divergence. However, this quantity can not be calculated directly, but only through sampling wt and then integrating wt out. Since wt is integrated out in the end, it does not matter what specific wt is sampled. So one could sample a new set of wt, or use a fixed w along the recent trajectory τk. The second way is often faster because sampling is avoided. The pseudo-code of single-worker Coherent-PPO is shown in Algorithm 2.
C.3 COHERENT SOFT ACTOR-CRITIC (COHERENT-SAC)
For Coherent-SAC, only two changes are needed. Firstly, we sample the last layer parameters of the policy network wt in each step t for exploration. Secondly, we improve the marginal policy instead of the actual policy performing exploration after each epoch. The pseudo-code of single-worker Coherent-SAC is shown in Algorithm 3.

heading: D ADDITIONAL RESULTS
text: In this appendix, we provide additional results for both comparative evaluation and ablation studies.
D.1 COMPARATIVE EVALUATION
The results of comparative evaluation for A2C (Mnih et al., 2016) and PPO (Schulman et al., 2017) on Reacher-v2, InvertedDoublePendulum-v2 and Hopper-v2 are shown in Figure 4. For SAC (Haarnoja et al., 2018), since it is a state-of-the-art deep RL algorithm, we only test it with three of our six OpenAI MuJoCo continuous control tasks with highest state and action dimensions, as shown in Figure 1.
D.2 ABLATION STUDIES
The full results of all three ablation studies on all six OpenAI MuJoCo continuous control tasks are shown in Figure 5, Figure 6, Figure 7 and Figure 8.
Algorithm 2: Coherent-PPO Input: initial policy parameters µ0,Λ0,θ0, initial value function parameters φ0.
1 for k=0,1,2,...,K do 2 Create a buffer Dk for collecting a trajectory τk with T steps. 3 for t=0,...,T do 4 if t=0 then 5 Sample last layer parameters of policy network wt ∼ p0(wt;µk,Λk) and store wt. 6 else 7 Sample last layer parameters of policy network wt ∼ p(wt|wt−1;µk,Λk) and store wt. 8 Observe state st and select action at ∼ πwt,θk(at|st). 9 Execute at in the environment.
10 Observe next state st+1, reward rt, done signal d, and store (st,at, rt, st+1, d) in buffer Dk. 11 If st+1 is terminal, reset environment state. 12 Infer forward message α(wt) using previous state st−1, previous action at−1 as well as mean vt−1 and covariance L−1t−1 of previous forward message α(wt−1). 13 Store mean vt and covariance L−1t of current forward message α(wt). 14 Compute marginal action probability p(at|s[0:t],a[0:t−1],µk,Λk,θk). 15 Compute rewards-to-go Rt and any kind of advantage estimates Ât based on current value function Vφk for all steps t. 16 Learn policy by maximizing the PPO-Clip objective:
LCLIPµk,Λk,θk(µ,Λ,θ) = T−1∑ t=0 [ min (rt(µ,Λ,θ), clip (rt(µ,Λ,θ), 1− , 1 + )) Ât ] ,
and update the policy by performing multiple gradient steps until the constraint of approximated KL divergence being satisfied:
µk+1 ← µk + αµ∇̂µLCLIPµk,Λk,θk(µ,Λ,θ)
Λk+1 ← Λk + αΛ∇̂ΛLCLIPµk,Λk,θk(µ,Λ,θ)
θk+1 ← θk + αθ∇̂θLCLIPµk,Λk,θk(µ,Λ,θ).
17 Learn value function by minimizing the regression mean-squared error:
L(φ) = 1
T T∑ t=0 (Vφk (st)−Rt) 2 ,
and update the value function by performing a gradient step:
φk+1 ← φk + αφ∇̂φL(φ).
Algorithm 3: Coherent-SAC Input: initial policy parameters µ,Λ,θ, initial Q-function parameters φ1,φ2, empty replay
buffer D. 1 Set target parameters equal to main parameters φtarg,1 ← φ1,φtarg,2 ← φ2. 2 for each step do 3 if just updated then 4 Sample last layer parameters of policy network w ∼ p0(w;µ,Λ) and store w as wprev. 5 else 6 Sample last layer parameters of policy network w ∼ p(w|wprev;µ,Λ) and store w as wprev. 7 Observe state s and select action a ∼ πw,θ(a|s). 8 Execute a in the environment. 9 Observe next state s′, reward r, done signal d, and store (s,a, r, s′, d) in replay buffer D.
10 If s′ is terminal, reset environment state. 11 if it’s time to update then 12 for j in range(number of updates) do 13 Randomly sample a batch of transitions B = {(s,a, r, s′, d)}. 14 Compute targets for Q-functions:
y (r, s′, d) = r + γ(1− d) ( min i=1,2 Qφtarg,i (s ′, ã′)− α log πw,θ (ã′|s′) ) ,
where ã′ ∼ πw,θ(ã′|s′). 15 Update Q-functions by one step of gradient descent using:
∇φi 1 |B| ∑
(s,a,r,s′,d)∈B
(Qφi(s,a)− y (r, s′, d)) 2 , for i = 1, 2.
16 Update policy by one step of gradient ascent using:
∇µ,Λ,θ 1 |B| ∑ s∈B [ min i=1,2 Qφi (s, ã)− α log p (ã|s,µ,Λ,θ) ] ,
where ã is a sample from p (ã|s,µ,Λ,θ) which is differentiable w.r.t µ,Λ,θ via the reparameterization trick.
17 Update target networks with:
φtarg,i ← ρφtarg,i + (1− ρ)φi for i = 1, 2.
18 else 19 Continue.

Review summaries:
This paper proposes Deep Coherent Exploration that unifies step-based exploration and trajectory-based exploration on continuous control. There exists a prior work that bridges a gap between the two exploration methods for linear policies, and this paper generalizes the prior work for various deep RL methods: on-policy (A2C, PPO) and off-policy (SAC). Finally, Deep Coherent Exploration enhances the performance of baseline algorithms and has better performance than prior works (NoisyNet, PNSE) on Mujoco tasks.

This paper focuses on undirected exploration strategies in reinforcement learning. Following the prior work, this paper proposes an exploration method unifying the step-based and trajectory-based exploration. The authors propose to perturb only the last(linear) layer of the policy for exploration, instead of perturbing all layers of the policy network. Also, the authors use analytical and recurrent integration for policy updates. Experiments show that the proposed exploration strategy mostly helps A2C, PPO and SAC in three Mujoco environments.