Deep Coherent Exploration For Continuous Control
Authors: Yijie Zhang
Herke van Hoof
Venue:  (2021)
Keywords: reinforcement learning
exploration
latent variable models

Abstract: In policy search methods for reinforcement learning (RL), exploration is often performed by injecting noise either in action space at each step independently or in parameter space over each full trajectory. In prior work, it has been shown that with linear policies, a more balanced trade-off between these two exploration strategies is beneficial. However, that method did not scale to policies using deep neural networks. In this paper, we introduce Deep Coherent Exploration, a general and scalable exploration framework for deep RL algorithms on continuous control, that generalizes step-based and trajectory-based exploration. This framework models the last layer parameters of the policy network as latent variables and uses a recursive inference step within the policy update to handle these latent variables in a scalable manner. We find that Deep Coherent Exploration improves the speed and stability of learning of A2C, PPO, and SAC on several continuous control tasks.

Under review as a conference paper at ICLR 2021
DEEP COHERENT EXPLORATION FOR CONTINUOUS CONTROL
Anonymous authors Paper under double-blind review
ABSTRACT
In policy search methods for reinforcement learning (RL). exploration is often performed by injecting noise either in action space at each step independently or in parameter space over each full trajectory. In prior work, it has been shown that with linear policies, more balanced trade-off between these two exploration strategies is beneficial. However, that method did not scale to policies using deep neural networks. this paper, we introduce Deep Coherent Exploration, a gen- eral and scalable exploration framework for deep RL algorithms continuous control, that generalizes step-based and trajectory- based exploration. This frame- work models the last layer parameters of the policy network latent variables and uses a recursive inference step within the policy update to handle these latent variables in scalable manner. We find that Deep Coherent Exploration improves the speed and stability of learning of A2C, PPO, and SAC on several continuous control tasks.
INTRODUCTION
The balance of exploration and exploitation (Kearns & Singh 2002 Jaksch et al. 2010) is a long- standing challenge in reinforcement learning (RL). With insufficient exploration, states and actions with high rewards can be missed, resulting in policies prematurely converging to bad local optima. In contrast, with too much exploration, agents could waste their resources trying suboptimal states and actions, without leveraging their experiences efficiently. To learn successful strategies, this trade-off between exploration and exploitation must be balanced well, and this is known as the exploration vs. exploitation dilemma.
At a high level, exploration can be divided into directed strategies and undirected strategies Thrun 1992 [Plappert al., 2018). While directed strategies aim extract useful information from exist- ing experiences for better exploration, undirected strategies rely on injecting randomness into the agent's decision-making. Over the years, many sophisticated directed exploration strategies have been proposed (Tang et al. 2016 Ostrovski et al. 2017 Houthooft et al.) 2016 Pathak al.) (017)- However, since these strategies still require lever-level exploration to collect the experiences, or either complicated
computationally intensive, undirected exploration strategies are still com- monly used in RL literature in practice, where some well-known examples are 6-greedy (Sutton 1995) for discrete action space and additive Gaussian noise for continuous action space (Williams) 1992 Such strategies explore by randomly perturbing agents' actions at different steps indepen- dently and hence are referred to performing step-based exploration in action space (Deisenroth et al. 2013).
As an alternative to those exploration strategies action space, exploration by perturbing the weights of linear policies has been proposed (Rückstiell et al. 2010: [Shnnke et al. 2010; Kober Peters 2008). Since these strategies in parameter space naturally explore conditioned on the states and are usually trajectory-based (only perturb the weights at the beginning of each trajectory) [Deisenroth et al.| 22013), they have the advantages of being more consistent, structured, and global DDiiennothh et al. 2013). Later, |van Hoof et al. (2017) proposed generalized exploration (GE) exploration in parameter space. With the advance of deep RL, NoisyNet (Fortunato et al. 2018) and Parameter Space Noise for Exploration (PSNE) (Plappert et al., (2188) were introduced, extending parameter-spen space exploration strategies for policies using deep neural networks.
scheme, bridging the gap between step-based and trajectory-

---

Under review as a conference paper at ICLR 2021
Although GE, NoisyNet, and PSNE improved over the vanilla exploration strategies in parameter space and were shown leading to more global and consistent exploration, they still suffer from sev- eral limitations. Given this, we propose a new exploration scheme with the following characteristies.
1. Generalizing Step-based and Trajectory- Exploration (van Hoof et al., 2017) Since both NoisyNet and PSNE are trajectory-based exploration strategies, they are con- sidered relatively inefficient and bring insufficient stochasticity (Deisenroth et al. 2013). Following van Hoof et al. (2017), our method improves by interpolating between siep-based and trajectory- based exploration in parameter space, where more balanced trade-off be- tween stability and stochasticity can be achieved.
2. Recursive Analytical Integration of Latent Exploring Policies NoisyNet and PSNE address the uncertainty from sampling exploring policies using Monte Carlo integration, while GE uses analytical integration on full trajectories, which scales poorly in the number of time steps. In contrast, we apply analytical and recurrent integration after each step, which leads to low-variance and scalable updates.
3. Perturbing Last Layers of Policy Networks Both NoisyNet and PSNE perturb all lay- ers of the policy network. However, in general, only the uncertainty parameters of the last (linear) layer can be integrated analytically. Furthermore, it is not clear that deep neu- ral networks can be perturbed in meaningful ways for exploration (Plappert et al. 2018). We thus propose and evaluate architecture where perturbation is only applied on the parameters of the last layer.
These characteristics define our contribution, which we will refer to as Deep Coherent Exploration. We evaluate the coherent versions of A2C (Mnih al. 2016). PPO (Schulman et al.] (017), and SAC (Haarnoja et al. 2018), where the experiments on OpenAl MuJoCo (Todorov et al. 2012 Brockman et al. 2016) tasks show that Deep Coherent Exploration outperforms other exploration strategies in terms of both learning speed and stability.
2 RELATED WORK
As discussed, exploration can broadly be classified into directed and undirected strategies (Thrun 1992 Plappert et al. (018), with undirected strategies being commonly used in practice because of their simplicity. Well known methods such as e-greedy (Sutton 1995) or additive Gaussian noise (Williams 1992) randomly perturb the action at each time step independently. These nigh-frequency result in poor coverage of the state-action space due to random-wall behavior BRückstieB et al.] 2010 Deisenroth et al., 2013). washing-out of exploration by the envi- ronment dynamics (Kober & Peters, 2008; Rückstieß et al., 2010; Deisenroth et al., 2013), and to potential damage to mechanical systems (Koryakovskiy et al. (0117).
perturbations. however,
One altermative is instead perturb the policy in parameter space, with the perturbation held con- stant for the duration of a trajectory. RückktießB et al.] (2010) and [Sehnke et al.] (2010) showed that such parameter-space methods could bring improved exploration behaviors because of reduced variance and faster convergence, when combined with REINFORCE (Williams 1992) or Natural Actor-Critic (Peters et al. 2005).
Another alternative to independent action-space perturbation, is to correlate the noise applied at subsequent actions (Morimoto & Doya 2000; Wawrzynski 2015 Lillicrpp et al. 2016). for exam- ple by generating perturbations from an Ornstein- Uhlenbeck (OU) process (Uhlenbeck & Ornstein 1930). Later, vvan Hoof al. (2017) used the same stochastic process but in the parameter space of the policy. This approach uses temporally coherent exploring policy, which unifies step-based and trajectory-based exploration. Moreover, the author showed that, with linear policies, a more delicate balance between these two extreme strategies could have better performance. However, this approach was derived in a batch mode setting and requires storing the full trajectory history and the inversion of matrix growing with the number of time step. Thus, it does not scale well to long trajectories or complex models.
Although these methods pioneered the research of exploration in parameter space, their applicability is limited. More precisely, these methods were only evaluated with extremely shallow (often linear) policies and relatively simple tasks with low-dimensional state spaces and action spaces. Given this
2

---

Under review as a conference paper at ICLR 2021
NetsyNet (Fortunato et al. 2018). PSNE (laappert et al. 2018) and Stochastic A3C (SA3C) (Shang ct al. 2019) were proposed, introducing more general and scalable methods for deep RL algorithms.
All three of these methods can be seen learning a distribution over policies for trajectory- -bseed exploration in parameter space. These exploring policies are sampled by perturbing the weights across all layers of deep neural network, with the uncertainty from sampling being addressed by Monte Carlo integration. Whereas NoisyNet learns the magnitudes of the noise for each parameter, PSNE heuristically adapts single magnitude for all parameters.
While showing good performance in practice (Fortunato et al. 2018 Plappert et al. 2018). these methods suffer from two potential limitations. Firstly, trajectory- strategies be inefficient as only one strategy can be evaluated for potentially long trajectory (Deisenroth et al., 2013), which could result failure escape local optima. Secondly, Monte Carlo integration results high-variance gradient estimates which could lead to oscillating updates.
BACKGROUND
This section provides background for reinforcement learning and related deep RL algorithms.
3.1 REINFORCEMENT LEARNING
Reinforcement learning is a sub-field of machine learning that studies how an agent learns strategies with high returns through trial-and-error by interacting with environment. This interaction be- tween an agent and an environment is described using Markov Decision Processes (MDPs). A MDP tuple (S, A. r, 7) where S is the state space. A is the action space, the reward function with function, and y is discount factor indicating the preference of short-term rewards.
A - ER is (S,, S(41), P S x A x S - itt is the transition probability
In RL with continuous action space, agent aims to learn parametrized (e.g. Gaussian) policy Te (a|s) S x E with parameters that maximizes the expected return over trajectories:
J(0) = Eryp(r ma) [R(T]],
where T = (8o, a., - 8T) is a trajectory and R(T) = Ei rhr is the discounted return.
3.2 DEEP REINFORCEMENT LEANING ALGORITHMS
Deep reinforcement learning combines deep learning and reinforcement learning, where policies and value functions are represented by deep neural networks for more sophisticated and powerful function approximation. In our experiments, consider the following three deep RL algorithms.
Advantage Actor-Critic (A2C) Closely related to REINFORCE (Williams) 1992). A2C is an on-policy algorithm proposed as the synchronous version of the original Asynchronous Advantage Actor-Crite (A3C) (Mnih et al. 2016). The gradient A2C can be written as:
VoJ(O) (1700==E,,aa)) 1E-=p(r(0)
Ve log no (a, (a) A (St, an)
where A as) the estimated advantage following policy no-
Proximal Policy Optimization (PPO) PPO is an on-policy algorithm developed to determine the largest step for update while still keeping the updated policy close to the old policy in terms of Kullback-Leibler (KL) divergence. Instead of using second-order method in Trust Region Policy Optimization (TRPO) (Schulman et al., 2015), PPO applies a first-order method and com- bines several tricks to relieve the complexity. We consider the primary variant of PPO-Clip with the following surrogate objective:
CLIP (0) ET (l)) L
min (r.(0), (r:(0), c, 11116 c)) 1
(3)
3

---

Under review as a conference paper at ICLR 2021
where r0) The (ar(st) To. (a (s,)
and small threshold that approximately restricts the distance
between the new policy and the old policy. In practice, to prevent the new policy from changing too fast, the KL divergence from the new policy to the old policy approximated on a sampled batch is often used as further constraint.
Soft Actor-Critic (SAC) As an entropy-regularized (Ziebart et al. 2008 off-policy actor-critic method (Lillicrap et al. 2016 Fujimoto et al. 2018) with stochastic policy, SAC (Haarnoja al. 2018) learns the optimal cntropy-regularizer Q Q-fuctionn through 'soft' Bellman back-ups with off- policy data:
Q" a) = (x))) (s
|r + (Q* ( (s', + aH ( (a)ls
(4)
where H is the entropy and a is the temperature parameter. The policy is then learned by maximizing the expected maximum entropy V V-function via the reparameterization trick (Kingma al., (013).
DEEP COHERENT EXPLORATION FOR CONTINUOUS CONTROL
To achieve the desiderata in Section 1 propose Deep Coherent Exploration, a method that models the policy as a generative model with latent variables. This policy is represented as Tw. (ar = N (W) + b,, A Here w, denotes all last layer parameters of policy network at step t by combining W, and b 0 denotes the parameters of the policy network except for the last layer, and A, is a fixed and diagonal precision matrix. Our method treats the last layer parameters w, as latent variables with marginal distribution w, . (H,,,," A,,". )), where f4, and A, are functions of learnable parameters and A respectively. In this model, all learnable parameters can be denoted as ( (J,, A, 0). We provide a graphical model of Deep Coherent Exploration in Appendix A
As in van Hoof et al. (2017). Deep Coherent Exploration generalizes step-based and trajectory-based exploration by constructing a Markov chain of w. This Markov chain specifies joint probabilities through an initial distribution Po (wo) = =N (μ, 1) and the conditional distribution p( w, w:-1). This latter term explicitly expresses temporal coherence between subsequent parameter vectors. In this setting, step-based exploration corresponds to the extreme case when p(w; W:-1) = Po(w.) and trajectory-basee
d(w, wz-1), where d is the Dirac delta function. To ensure the marginal distribution of w, will be equal to the initial distribution Po at any step I, we directly follow yan Hoof et al. (2017 7) with the following transition distribution for W.
exploration corresponds to another extreme case when p(W:) (////
p( w. W. W (3) w + (28 A))
(5)
where 3 is a hyperparameter that controls the temporal coherency of w, and wr-1- Then, the two extreme cases corresponds to 0 for trajectory- based exploration and exploration, while the intermediate exploration corresponds to 3 E (0, 1). For intermediate values of 8, we obtain smoothly changing policies that sufficiently explore, while reducing high-frequency perturbations.
for step-based
4.1 ON- IN-POLICY DEEP COHERENT EXPLORATION
Our method can be combined with all on-policy policy gradient methods and here present this adaptation with REINFORCE (Williams) 1992). Starting from the RL objective in Equation]
VCJ(C) Expp(ri)) [V. Pcllop(((()R(()
(6)
the gradients w.r.t the sampled trajectory can be obtained using standard chain rule:
Velogp(T(S)
Σ
V (lgg((a)) -0o ())
(7)
4

---

Under review as a conference paper at ICLR 2021
Here, since information still flow through the unobserved latent variable w., our policy is nor
Markov anymore. To simplify this dependency, we introduce w, into p(ay p(ax//to.)) 11, <):
p S|o:t] a|o:t-11: () / p(ar, Wx (jo:)) - ()w
p(ay/s;: We, 0)
p( p(we|storr-1) 11) H4, 0) dwi.
Gaussian policy Sw, ((../s.)
Torward message a<W.))
where the first factor is the Gaussian action probability and the second factor can be interpreted as the forward message a(w,) along the chain. We decompose a(,, by introducing w,-1
al a(we)= / p(we, Wt We-1 alo S)dwi-1
(10)
f p(we/w(-1:11AA) p(a, St W 1 ()((w()) dw:-1:
(11)
Zc-1
transition probability W.
where the first factor is the transition probability of w, (Equation 5 and Z in the factor term is normalizing constant. Since Gaussians are closed under marginalization and condition, the second factor can be obtained analytically without the need of computing the normalizing constant Z- Moreover, a(W,-1) is Gaussian by mathematical induction from the initial step. As result, we arrive at an efficient recursive expression for exact inference of wy. Again, with the property of Gaussians, all integrals appearing above be solved analytically, where the marginal action probability given the history at each step t can be obtained and used for policy updates.
Summarizing, non-Markov policies require substituting the regular p(ay 0) term in the update equation with p(a, a100-=))))) (Equation 7 Equations 81 show how this expression can be efficiently calculated recursively. Except for this substitution, learning algorithms like A2C and PPO can proceed as normal. For detailed mathematical derivation, please refer to Appendix
4.2 OFF-POLICY DEEP COHERENT EXPLORATION
Combining our method with off-policy methods (Lillicrap et al.] 2016 Fujimoto et al.) 2018 Haarnoja et al. 2018) requires defining both the behavior policy and the update equation. The behavior policy the same as in the on-policy methods discussed earlier (Equation 55. The policy update procedure may require adjustments for specific algorithms. Here, our method for SAC (Haarnoja et al. 20188) In the SAC policy update, the target policy is adapted towards the exponential of the new Quunctioo.. The target policy here is the marginal policy, rather than the policy conditioned on the sampled w, as this second option would ignore the dependence on a and A. This consideration leads to the following objective for policy update:
show how to adapt
J(C)= = EN KL (aaulss. C) exp (QΦ (se, ar))
(12)
Zs (s:)
D.s.s.Da..sp(aalls... [log pegp(au ar (s. <) Q (s., at)l
(13)
where & denotes the parameters of the Q-function and (a. |st, () the marginal policy which can again be obtained analytically (Equation 22
/
(a. St, st. 9)
p (ar S1; Wo. 0)
p( pr, A)
dwo-
(14)
Gaussian policy Two o (ar)ex) marginal probahility we
where all parameters can be learned via the reparameterization trick (Kingma et al. 2015).
5 EXPERIMENTS
For the experiments, we compare our method with NoisyNet (Fortunato et al. 2018). PSNE Plap- pert et al. 2018) and standard action noise. This comparison is evaluated in combination of A2C Mnih et al. (016)). PPO (Schulman et al. (017)) and SAC (Haarnoj et al. 2018) on OpenAl Gym MuloCo (Todorov et al. 2012 Brockman al. 2016) continuous control tasks.
5

---

Under review as a conference paper at ICLR 2021
For exploration in parameter space, we fixed action noise with a standard deviation of 0.1. For A2C and PPO, their standard deviations of parameter noise are all initialized at 0.017, as sug- gested in Fortunato et al. 20188. For SAC, we initialize the standard deviation of parameter noise at 0.034 for both our method and PSNE as it gave better results in practice. Besides, Deep Coherent Exploration learns the logarithm of parameter noise, while NoisyNet learns the parameter noise di- rectly, and PSNE adapts the parameter noise. To protect the policies from changing too dramatically, we consider three small values of 3 (0.0, 0.01, and 0.1) for Deep Coherent Exploration, where we use 3 = 0.01 for comparative evaluation with other exploration strategies. Our implementation of NoisyNet is based on the code from Kaixhin For PSNE, we refer to author's implementation in OpenAl Baselines (Dhariwal et al. 2017) and the original paper, where we set the KL threshold for A2C and PPO to 0.01 and the MSE threshold for SAC to 0.1, On the other hand, exploration with action noise uses the default setting proposed by Acciam (2018), where the standard deviation of action noise is initialized at around 0.6 for A2C and PPO. For SAC, in the baseline setting the standard deviation of action noise is output by the policy network.
In all experiments, agents are trained with a total of 10 environmental steps, where they are updated after each epoch. A2C and PPO use four parallel workers, where each worker collects trajectory of 1000 steps for each epoch, resulting in epochs with 4000 steps in total. After each epoch, both A2C and PPO update their value functions for 80 gradient steps. At the same time, A2C updates its policy for one gradient step, while PPO updates its policy for up to 80 gradient steps until the KI. constraint is satisfied. SAC uses a single worker, with a step size of 4000 for each epoch. After every 50 environmental steps, both policy and value functions are updated for 50 gradient steps. To make the parameter noise stable, we adapt the standard deviation of parameter noise after each epoch.
Our implementation of A2C, PPO, and SAC with different exploration strategies are adapted based on OpenAl Spinning Up (Achiam 2018) with default settings. All three algorithms use two-layer feedforward neural networks with the same network architectures for both policy and value func- tion. To be more precise, A2C and PPO use network architectures with 64 and 64 hidden nodes activated by tanh units. In comparison, SAC uses network architecture of 256 and 256 hidden nodes activated by rectified linear units (ReLU), Parameters of policies and value functions in all three algorithms are updated using Adam (Kingma & Bal 2015). A2C and PPO use learning rate of 3 10-4 4 for the policies and a learning rate of 10- for the value functions. SAC uses a single learning rate of 10 for both policy and value function.
For each task, we evaluate the performance of agents after every 5 epochs, with no exploration noise. Additionally, each evaluation reports the average reward of 10 episodes for each worker. To mitigate the randomness within environments and policies, we report our results as the average over 10 random seeds. All settings not explicitly explained in this section, are set to the code base defaults. For more details, please refer to documents and source code from OpenAl Spinning Up (Achiam 2018) and our implementation
5.1 COMPARATIVE EVALUATION
In this section, we present the results for A2C (Mnih et al. 2016). PPO (Schulman et al. 2017), and SAC (Haarnoja et al. 2018) on three control tasks, with the additional results shown in Ap- pendix D Figure 1 shows that, overall, Coherent-A2C outperforms all other A2C-based methods in terms of learning speed, final performance, and algorithm stability. In particular, given that Ant- v2 is considered challenging task, Deep Coherent Exploration considerably accelerates learning speed. For PPO-based method, our method still outperforms NoisyNet and PSNE significantly in all tasks. However, our method's advantage compared to standard action noise is smaller. Particu- larly, Coherent-PPo underperforms PPO in Walker2d-v2. Two reasons might explain this. Firstly, some environments might be more unstable and require a larger degree of exploration, which fa- vors PPO as it initializes its action noise with much greater value. Secondly, because of having extra parameters, policies of Coherent-PPO, NoisyNet-PPO, and PSNE-PPO tend to satisfy the KL constraint in fewer update steps, which leads to slower learning. For SAC, the advantages of our method are smaller compared to A2C and PPO. More specifically, Coherent-SAC learns slightly
Source code to be released after review

---

Under review as a conference paper at ICLR 2021
Coherent A2C
NoisyNet AZC
PSNE-AZC
HallCheeteh-v2
Welker2d-v2
Ant-v2
Atcorage retum
Total environmental steps (1eb)
Tutal enviromennall steps (leb)
Tutal environmentll steps [1:65
Coherent: PPO (beta=0.01)
NoisyNet--PP PPO
PSNE-PPO
PPO
HallCheet ah-r2
Walker2dbu2
Antov2
SoC steps
staps
HellCheeta--e2
Walker20-12
Antov2
Total whwllcnwitommtttlll stops (1e6)
Total environmental stops (106)
Total onvironmental stops (136)
Figure 1: Learning curves for deep RL algorithms with different exploration strategies on OpenAI MuloCo continuous control tasks, where the top, middle and bottom row corresponds to results of A2C, PPO, and SAC respectively. The solid curves correspond to the mean, and the shaped region represents half standard deviation of the average return over 10 random seeds.
faster than SAC in HalfCheetah-v2 and achieves the highest average returns Walker2d-v2 and Ant-v2. Furthermore, Coherent-SAC: shows variance lower than PSNE-SAC but higher than the baseline SAC.
5.2 ABLATION STUDIES
In this section, we present three separate ablation studies clarify the effect of each characteristic discussed in Section These ablation studies are performed with A2C, to ensure that all characteris- ties are applicable and because the fixed number of gradient steps puts the methods on equal footing. Here we show the results for HalfCheetah-v2, where the full results can be found in Appendix D
Generalizing Step-based and Trajectory-based Exploration As shown in Figure 2a we evalu- ate three different values 0.0, 0.01, 0. of /3 for CohherentA2C: Here both two intermediate strategies (3 = 0.01 and = 0.1) outperforms the trajectory-base strategy (B = 0.0). Coherent-A2C: with = 0.01 seems achieve the best balance between randomness and stability, with considerably higher return than the other two.
Analytical Integration of Latent Exploring Policies We introduce OurNoisyNet for compari- son. OurNoisyNet equips noisy linear layer for only its last layer, and this layer learns the logarithm of standard deviation, in Deep Coherent Exploration. We compare Coherent-A2C using 8 0.0 and OurNoisyNet-A2G with the only difference thus being whether we integrate analytically or use the reparameterization trick (Kingma al. 2015).

---

Under review as a conference paper at ICLR 2021
HallCheetah-v2 v2
HaltCheetah-v2
HellCheetah--22
Avarage natum
Coherent: AZC (beta-0.00
Concrent A2C (beta-0.01
Coherent- (beta=0.0)
Ourtioisyfeet layer
Coherent-AIC (betam0
Coherent-A2C (beta 1)
OurNolsyyeer-ACC (ell layers)
Ourhisiyyhat-AO.
Total environmental 1leb)
Si enstronmental steps (1eb)
62
envitonmental steps
(c)
Figure 2:
Results of Coherent-A2e with different settings for HalfCheetah-v2, where Figure 2a and
Figure 2b the learning curves and (guure2cJshows the average log variance of gradients during six stages in learning. The solid curves correspond the mean, and the shaped region represents half a standard deviation of the average return over 10 random seeds.
We first measure the variance of gradient estimates in both methods. This variance is measured by computing the trace of the covariance matrix using 10 gradient samples. We report this measure in six stages during training, leads to lower-variance gradient estimates across all training stages for HalfCheetah-v2. We further present the learning curves of both methods in Figure 2b where Coherent-A2C with B 0.0 shows higher return than Our-oisyNet-A2C. It is interesting that Coherent-A2C displays much lower standard deviation across different random seeds. Furthermore, the lower-variance gradient estimates of Coherent-A2C could enable a larger learning rate for faster training without making policy updates unstable.
shown in Figure 2c We can observe that analytical integration
Perturbing Last Layers of Policy Networks In this part, we compare OurNoisyNet-A2C per- turbed over all layers, and OurNoisyNet-A2C pertubbrd over only the last layer. The result is shown in Figure 2b] Somewhat to our surprise, the latter seems to perform much better. There are several possible reasons. Firstly, since is unknown how the parameter noise (especially in lower layers) is realized in action noise, perturbing all layers of the policy network may lead to uncontrollable perturbations. Such excess exploratory noise could inhibit exploitation. Secondly, perturbing all layers might disturb the representation learning of states, which is undesirable for learning good policy. Thirdly, perturbing only the last layer could also lead to fewer parameters for NoisyNet.
6 CONCLUSION
In this paper, we have presented a general and scalable exploration framework that extends the generalized exploration scheme (van Hoof al. (017) for continuous deep RI. algorithms. In par- ticular, recursive calculation of marginal action probabilities allows handling long trajectories and high-dimensional parameter vectors. Compared with NoisyNet (Fortunato al. 2018) and PSNE (Plappert et al., 22018), our method has three improvements. Firstly, Deep Coherent Exploration generalizes baspd and trajectory-based exploration in parameter space, which allows a more balanced trade-off between stochasticity and coherence. Secondly, Deep Coherent Exploration an- alytically marginalizes the latent policy parameters, yielding lower-variance gradient estimates that stabilize and accelerate learning. TThirdly, by perturbing only the last layer of the policy network, Deep Coherent Exploration provides better control of the injected noise.
When combining with A2C (Mnih et al. 2016), PPO (Schulman et al., (017), and SAC (Haarnoja itt al. 2018), we empirically show that Deep Coherent Exploration outperforms other exploration strategies on most of the MuJoCo continuous control tasks tested. Furthermore, the ablation studies show that, while each of the improvements is beneficial, combining them leads to even faster and more stable learning. For future work, since Deep Coherent Exploration uses fixed and small action noise, we believe one interesting direction is to study whether the learnable perturbations in action space be combined with method in meaningful way for even more effective exploration.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Joshua Achiam. Spinning Up in Deep Reinforcement Learning.
2018.
URL https://
Christopher M. Bishop. Pattern recognition and machine learning, 5th Edition. Information science
and statistics. Springer, 2007. ISBN 9780387310732. URL http:/ httt:://www... http://wwwlld.atac .orgg
oc1c/71008143
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Mare Peter Deisenroth, Gerhard Neumann, and Jan Peters. A survey on policy search for robotics. Foundations and Trends in Robotics, 2(1-2):1-142 2013. doi: 10.1561//300000021.. URL
https://doi. http://oo.ooggg////// http://...1111//0000000000
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alce Radford,
John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https: 2017.
//github. wwwp/////wp..aaaa..oaoooaaaaaaee
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Mcnick, Matteo Hessel, Ian Os- band, Alex Graves, Volodymyr Mnih, Rémi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL httpp:://ppenreveee..eee forum?id So-umidddryHCCKKa.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th Inter- national Conference Machine Leurning, ICML 2018, Stockholmsmüsssan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1582-1591. PMLR, 2018. URL htRK.tttt://prrceediiii
ml r. press / v80/ www.imooolllaaa 8a html
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning. ICML 2018, Stockholmsmässaan Stockholm, Sweden, July 10-15, 2018, volume 80 Proceedings of Machine Learning Research, pp. 1856-1865, PMLR, 2018. htR:: http::::///ccceeingg....
mlr.press A...ppees/vv8J httesss/80//aaznoo1 hwwrnaaaanaaaaa.aaa.aa html
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. VIME: variational information maximizing exploration. Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 1109-1111 2016. URL http: htttt::/pappzs: nips:rsssipppooomm www..iisp....pp.....eepppppppppe 6591-vine-variatioon
Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal regret bounds for reinforcement learning. Mach. Learn. Res., 11:1563-1600, 2010. citation
URL http://portal. acm org/
Michael J. Kearns and Satinder P. Singh. Near-optimal reinforcement learning in polynomial time. Mach. Learn, 49(2-3):209-233 2002, doi: 10.10033/A:011198441133 URL h/tps://doo. org/10 02 1ee111222111119441333300
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua
Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URI http:
6980
Diederik Kingma, Tim Salimans, and Max Welling, Variational dropout and the local reparameter- ization trick. CoRR, 0bs/1550.025. 2015. URL WtRLhhtt:///axxt wwtp:/wwwwww....ooop..o.otoo.s 6..6. 02557

---

Under review as a conference paper at ICLR 2021
Jens Kober and Jan Peters. Policy search for motor primitives in robotics. In Daphne Koller, Dale Schuurmans, Yoshua Bengio, and Léon Bottou (eds.), Advances in Neural Information Processing Systems 21, Proceedings of the Twenty-Seconel Annual Conference formation Processing Systems, Vancouver, British Columbia, Canada, December 8-11, 2008, pp. 849 856. Curran Associates, Inc., 2008 URL http::/paperrse ni www...ipppcoo........m.... httpps.cccpper//pp/ppppppppppppt 3545 powpcy search for- motor primitives in robotics
Neural In-
Ivan Koryakovskiy, Heike Vallery, Robert Babuška, and Wouter Caarls, Evaluation of physical dam- age associated with action selection strategies in reinforcement learning. IFAC-PapersOnll.ngg 50 doi: 10. 101 (f..o.ac.l.20177 1218. URL htttpt: 20th World Congress of the International Federation of Automatic Control (IFAC), 2017, IFAC 2017: Conference date: 09-07-2017 Through 14-07-2017.
(1):6928-6933) 2017. ISSN 1474-6670.
//www. wff.c0111101 org
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
//arxiv.org/a/t abcgg///b///i/ 02971
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Maria-Florina Balcan and Kilian Q. Weinberger (eds.). Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19- 24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 1737.1937.25 JMLR.or, 2016. httt:::ppppopeeeteeiaai http://annnnmmm...erereeeeeeeeee wwwppp/wwwwwp..pppppppppeppppaae
mlr
html
Jun Morimoto and Kenji Doya. Acquisition of stand-up behavior by real robot using hierarchical reinforcement learning. In Pat Langley (ed.), Proceedings of the Seventeenth International Con- ference on Machine Learning (ICML 2000), Stanford University, Stanford, CA, USA, June 29 Morgan Kaufmann, 2000.
July 2, 2000, pp. 623-6300
Georg Ostrovski, Marc G. Bellemare, Aäron van den Oord, and Rémi Munos. Count-based explo- ration with neural density models. In Doina Precup and Yee Whye Teh (eds.). Proceedings of the 34th International Conference Machine Leurning, ICML 2017, Sydney, NSW, Australia, 6-11 Augus! 2017, volume 70 Proceedings of Machine Learming Research, pp. 2721-2730, PMLR, 2017.
mlr. www.sprsssss..... /w/ wttp:///wwo/ottttss11111
html
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity exploration by self-supervised prediction. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 2778-2787. PMLR, 2017. URL httL:ttt//oocceennnge mlr ppress//700ppthakl//
Jan Peters, Sethu Vijayakumar, and Stefan Schaal. Natural actor-critic. In João Gama, Rui Ca- macho, Pavel Brazdil, Alípio Jorge, and Luís Torgo (eds.), Machine Learning: ECML 2005, 16th European Conference on Machine Learning, Porto, Portugal, October 3-7, 2005, Proceed- ings, volume 3720 of Lecture Notes in Computer Science, pp. 280-291, Springer, 2005, doi: 10.1007/11564096 29. URL URL hattps:://ai..oo
1007 11tpg//10...001155455555
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y. Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for explo- ration. In 6th International Conference on Learning Represemaations, ICLR 2018, Vancouver, BC, Canada, April 30- May 2018, Conference Track Proceedings. OpenReview.nee, 2018. URL https: http:://oppnreveee
net Eorum? id Byruuiid ByBA12eAZ
Thomas Rückstießß Frank Sehnke, Tom Schaul, Daan Wierstra, Yi Sun, and Jürgen Schmidhu- ber. Exploring parameter space in reinforcement learning. Puladyn J. Behav. Robotics, (1):14244, 2010. doi: 10.2778/113230-0000000 01000000100000-----0000000000000
URL pestps://dot wtt....orr1..240...
John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust re- gion policy optimization. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd
10

---

Under review as a conference paper at ICLR 2021
International Conference Machine Learning. ICML 2015, Lille, France, 6-11 July 2015, vol- ume 37 of JMLR Workshop and Conference Proceedings, pp. 1889-1897. JMLR.org, 2015. URL
http://proceedingg mwr. www.m.t.......... pwwss/v31////e/// hwt./////scumaana// 1htmml
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford. and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707. 1077706677, 2017. URL http:// /arxiv/rxxvv..o org/ abs/
1707. 06347
Frank Sehnke, Christian Osendorfer, Thomas RückktieBß Alex Graves, Jan Peters, and Jürgen Schmidhuber. Parameter-exploring policy gradients. Neural Networks, 23(4):551-559. 2010. doi: 10.101 10.1016/j.neumet.2000 www.eeett....122..000... 12. 004
URL
wtt.0///......eee. 2009.
Wenling Shang, Douwe van der Wal, Herke van Hoof, and Max Welling. Stochastic activation actor critic methods. In Ulf Brefeld, Élisa Fromont, Andreas Hotho, Arno J. Knobbe, Mar- loes H. Maathuis, and Céline Robardet (eds.), Machine Learning and Knowledge Discovery in Databases European Conference, ECML PKDD 2019, Würzzurg, Germany, September 16-20, 2019, Proceedings, Part III, volume 11908 of Lecture Notes in Computer Science, pp. 103-117. Springer, 2019, doi: 10. 978 3- 030 46133-1-111
URL https://ddo. org/ 11111011000 1007/
Richard S. Sutton.
Generalization in reinforcement learning: Successful examples using sparse coarse coding. In David S. Touretzky, Michael Mozer, and Michael E. Hasselmo (eds.), NIPS, pp. 1038-1044. MIT Press, 1995. ISBN 0-262-20107-D URL httthhttp//dbl/// http://blp...iiiii....eo.. trreereeddd db/con htt.conf/ips//niinec 999. 995 html #Sutton95
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel, #exploration: A study of count-based exploration for deep re- inforcement learning. CoRR, abs/1611 1,04477 2016.
04717
Sebastian Thrun. The role of exploration in learning control. In Handbook for Intelligent Control: Neural, Fuzzy and Adaptive Approaches. Van Nostrand Reinhold, Florence, Kentucky, January 1992.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based con- trol. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, IROS 2012, Vilamoura, Algarve, Portugal, October 7-12, 2012, pp. 5026-5033. IEEE, 2012. doi: 10. 109/ 1ROS.2012.6386109. BtLLhtts::/// //op:::///dddoo /oi otreioorg//000 11011111111111111 IROS. 2012. 6386109
George. E. Uhlenbeck and Leonard S. Ornstein. On the Theory of the Brownian Motion. Physical Review, 36(5):823-844 841, September 1930.
Herke van Hoof, Daniel Tanneberg, and Jan Peters. Generalized exploratio in policy search. Mach. Learn.,
2017, doi: 10. 10110777119944---5555711 | httpp:///ooi.
org/ htcp/...1077/1108--00--5555
Pawel Wawrzynski. Control policy with autocorrelated noise reinforcement learning for robotics. International Journal of Machine Learning and Computing. 5:91-95, 04 2015. doi: 10.7763/ WMMCC.015.V5.448
Ronald Williams. Simple statistical gradient-following a for connectionisttreifforeeeen learning. Mach. Learn., 8:229-256 1992. doi: 10.1007/BFF00992666 URL https::/doi.
2x1/1001007/870099555
Brian D. Ziebart, Andrew L. Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Dieter Fox and Carla P. Gomes (eds.), Proceedings of the Thenty-Third AAAI Conference on Artificial Intelligence, AAAI 2008, Chicago, Illinois, USA, July 13-17, 2008, pp. AAAI /000 2ttp:/2000/aaaai--2220
AAAI Press, 2008. URL http: / www aaai. oorii..or//// orggLibraryy

---

Under review as a conference paper at ICLR 2021
A GRAPHICAL MODEL OF DEEP COHERENT EXPLORATION
In this appendix, provide graphical model representation of Deep Coherent Exploration, shown in Figure 3 This graphical model uses the same conventions as in Bishop (2007), where empty circles denote latent random variables, shaded circles denote observed random variables, and dots denote deterministic variables.
A
Wo
W
30
a1
So
S
Figure 3: Graphical model of Deep Coherent Exploration.
B MARGINAL ACTION PROBABILITY FOR ON-POLICY COHERENT EXPLORATION
As discussed in Section 4.1 forward message aww;) is used to compute the marginal action prob- ability given the history at step for the final learning objective. Suppose we have the Gaussian policy as:
Tw. (( = NW,,,, A,,))
(15)
where E R. fo(s:) E R7 W, R is the coefficient matrix, b, R is the bias vector and Ag is constant precision matrix for the Gaussian policy. It's helpful to represent W; E Brttp by flattening W, and combining b,
W11
Wi9
W;
Wp
(16)
Wp b
bp
12

---

Under review as a conference paper at ICLR 2021
such that the parameters could still be sampled using multivariate Gaussians, Moreover, stack X, into X, E (Px(()+));
X
9.1
07
%
0
0
4,1
X
0.1
9,1
X
(17)
07
07
0 9.1
9:1
9.1
X;
where 0y.1 is g-dimension column vector. After this transformation, the Gaussian policy is
represented equivalently as:
Tw. N(( A.t).
(18)
B. BASE CASE
For the base case = O, forward message ae a(w)) and the initial transition probability of wo is identical by definition:
a(wo) = Po(wu: nt, A) N (H.Aa)).
(19)
Additionally, the action probability is given by:
T (So) (Xoo A-
(20)
With the property of multivariate Gaussians, obtain the marginal action probability given the history at f 0:
log / Tw., (awls)
log Dogp(an/so: μe, A,
(21)
loog Looo, A + XoA 'x,).
(22)
B.2
GENERAL CASE
For the general case of step >> 0, we need the state 5;-1, action a;-1 as well as mean and covariance of forward message aW).-1) stored from previous step. Suppose a(W,-1) is written as:
o(W,-1)
(23)
and the action probability from the previous step is given by:
Tw, 1.0(ar- N (X.-Iw--,,
(24)
We have directly:
p( 1/ 810:- A, 0) = (N Ex-1),
(25)
with
u = E1-1 X A a-11 1I-1V(-1)
(26)
= X T
AaXi-1)
(27)
Combining the transition probability of Wy
p(w:/wo1:: 44, = V ((1 3) W 8r, (23 A1)
(28)
we obtain the forward message a(w.):
(w,) N Lv),
(29)
where
V. = (1 (1- -1-1188)--1 + 344
(30)
L, = --
(31)
Here, v, and L, should be stored and used for exact inference of a o(we+) 1) at the step. Finally, the marginal action probability given the history at step / > is given by:
log p(a Sto: - 1): nt, A, 0) log
Tw, ( (a, 8s )a (aoaa(()ddc w.)dw.
(32)
log.V(X.v.,, A- X,L,
(33)
13

---

Under review as a conference paper at ICLR 2021
C DEEP COHERENT REINFORCEMENT LEARNING
Here, we provide brief introduction of adapting Deep Coherent Exploration for A2C (Mnih et al., 2016). PPO (Schulman al. 2017) and SAC (Haarnoja al. 2018 Respectively, call them Coh,rent-A2C, Coherent-PPO and Coherent-SAC.
C.I COHERENT ADVANTAGE ACTOR-CRITE (COHERENT-A2C)
Coherent-A2C is straightforward to implement. To do that, could just replace the original A2C gradient estimates (o) with the on-policy coherent gradient estimates N A, 0). The pseudo-code of single-worker Coherent-A2C is shown in Algorithm]
Algorithm 1: Coherent-A2C
Input: initial policy parameters μoo Ao, 0g, initial value function parameters Φo.
for k=0,,,,,,.... 1,2,..,K do
Create buffer Dx for collecting a trajectory 7. with T steps.
for p= do
1=0 then
Sample last layer parameters of policy network we Po Mk, A&) and store wi-
else
Sample last layer parameters of policy network w. p( (w,, Wz- pkx As) and store W,.
Observe state S, and select action Tw, 02 (ar)s:). Execute a, in the environment.
Observe next state $1+1, reward rt, done signal d, and store (st, a,, Pe, S1+1, d) in buffer D₁-
If 1 is terminal, reset environment state.
12
Infer forward message ax( w,) using previous state S,-1, previous action a;-1 as well as mean V.-1 and covariance - of previous forward message a (w:-1). Store mean V, and covariance L, of current forward message ax w,).
14
Compute marginal action probability p(a, [8)o,), [[0rr-1]: AAk, Ag, 0z).
Compute rewards-to-ge RG and any kind of advantage estimates A, based current value function Vo. for all steps t. Estimate gradient of the policy:
16
v.A..ee(,, 0)
Σ
VnA. log p(a alo:t-1] Mkx AR, On Ar,
1=
and update the policy by performing gradient step:
14++ < ap v J A. 0)
Ac+1 A + as V
Oc+1 + 0g + ag v (14, A, 0)
17
Learn value function by minimizing the regression mean equarer error:
1 L( T
Σ
(Vo. (s:) R₂)²
t=0
and update the value function by performing gradient step:
Φk+1 Φb + as & (())..
14

---

Under review as a conference paper at ICLR 2021
C.2 COHERENT PROXIMAL POLICY OPTIMIZATION (COHERENT-PPO)
Coherent- PPO can be implemented in a similar way. As in Coherent-A2C, we substitute the original objective ICLIP (0) with LELIP 0.,
",, (Ja, A, 0), which is given by:
L CLIP
Op (M, A,
(34)
= -Brap(r/ueA
min (re(p, A,
(35)
where ",
plae Bjous (1.
Here, after each step of policy update, |(or)s ((0r=)))) μ. A, 0) from the new policy should be evaluated on the most recent trajectory тı for both next update and approximated KL divergence. However, this quantity can not be calculated directly, but only through sampling w, and then inte- grating Wi out. Since W, integrated out in the end, does not matter what specific W; is sampled. So one could sample new set of w,, or use fixed w along the recent trajectory тк. The second way often faster because sampling is avoided. The pseudo-code of single-worker Coherent-PPO is shown in Algorithm 2
C.3 COHERENT SOFT ACTOR-CRITIC (COHERENT-SACC
For Coherent-SAC. only two changes are needed. Firstly, we sample the last layer parameters of the policy network w, in each step t for exploration. Secondly, improve the marginal policy instead of the actual policy performing exploration after each epoch. The pseudo-code of single-worker Coherent-SAC is shown Algorithm
D ADDITIONAL RESULTS
In this appendix, provide additional results for both comparative evaluation and ablation studies.
D.1 COMPARATIVE EVALUATION
The results of comparative evaluation for A2C (Mnih et al. (016) and PPO (Schulman et al. 2017) on Reacher-v2, InvereedDobbePanduuu2222 and Hopper-v2 are shown Figure For SAC (Haarnoja al. 2018), since it is state-of-the-arr deep RL algorithm, we only test it with three of our six OpenAl MuJoCo continuous control tasks with highest state and action dimensions, as shown in Figure
D.2 ABLATION STUDIES
The full results of all three ablation studies on all six OpenAl MuJoCo continuous control tasks are Figure 7 and Figure
shown in Figure Figure
15

---

Under review as a conference paper at ICLR 2021
Algorithm/ 2: Coherent-PPO
Input: initial policy parameters noo Ao, Oo. initial value function parameters do. for k=0, I...., do
Create buffer D& for collecting trajectory т with T steps.
for r= do
if 1=0 then
Sample last layer parameters of policy network we ~ Po [4, A1. and store w
else
Sample last layer parameters of policy network w. ~ p(w, W2 As) and store W..
Observe state S, and select action a, ~ Tw. (s:). Execute a, in the environment.
10
Observe next state S1+1, reward rt, done signal d, and store (st, ag, Te, S141, d) in buffer DA.
11
s is terminal, reset environment state.
Infer forward message ar( w.) using previous state S,-1, previous action a as well
mean V.-1 and covariance L, of previous forward message (W:-1)) Store mean V. and covariance L, of current forward message a (.).
13
Compute marginal action probability p(a, [$)o.) A[o:r-1]: MAk, Ag, 0x)-
Compute rewards-to-go RG and any kind of advantage estimates ÅŞ based on current value function Vo, for all steps t. Learn policy by maximizing the PPO-Clip objective:
15
16
0), CLIP O, (u, (, = Σ min (rr(u, A, 0). clip (re (r,),, A,
and update the policy by performing multiple gradient steps until the constraint of approximated KL divergence being satisfied:
M4++1 M au & CLIP
u 1z (n, A, 0)
A + A + AA V VA ME. (H, A,
CLIP
0 0₂ aa A VOL A
CLIP (μ. A, 0).
17 Learn value function by minimizing the regression mean mean-squared error:
1 T 1=0
I(Φ)
Σ (Vo. (s,) R2)²
and update the value function by performing a gradient step:
Φk+1 Φb + as v L(D).
16

---

Under review as a conference paper at ICLR 2021
Algorithm 3: Coherent-SAC
Input: initial policy parameters μ., A, 0, initial Q-function parameters Φ1.02, empty replay buffer D.
Set target parameters equal to main parameters Drarg,1 < Φ1, Prarg, + Φ2.
for each step do
if just updated then
Sample last layer parameters of policy network w (w: μe, A) and store as Wprev
else
Sample last layer parameters of policy network w ~ p(w H//W/Wpee prev p4, A) and store w as prev.
Observe state s and select action a ~ Tw.o(als). Execute in the environment.
Observe next state s', reward r, done signal d, and store (s, r,,,',, in replay buffer D. If terminal, reset environment state.
10
11
if it's time to update then
12
for in range(number of updates) do Randomly sample a batch of transitions B = {(s, a, r,,,'s,)))
Compute targets for Q-functions:
y (r, d) = r + d)
min O (s', - ar log Tw. (ā')s') 1=1,2
where a'
(ā'ls'.
15
Update Q-functions by one step of gradient descent using:
Σ
(Q. (r) for i 1,2.
(s...T." JIEB
Update policy by step of gradient ascent using:
min QΦ. (s, a) a (ag. u, 1.2
P.A.0
B
where à is a sample from p (ä s, u, A, 0) which is differentiable w.r.t A, O via the reparameterization trick.
Update target networks with:
Dtarg, + porargit + (1- p)pp for = 1,2.
18
else
19
Continue.
17

---

Under review as a conference paper at ICLR 2021
A2C (beta=0.01)
Nosyyet-A2C
PSNE A2C
A2C
Invere@DoudiFaanduuuwee
Hopper-v2
Total envirommental steps (166)
Total environmental Stops 1166)
Total environmental Stops 11:6)
Coherent:PPO (beta=6.01)
NobsyNet-FRO
PSNE-PO PPO
PPO
Reacher-v2
Happer 12
Average return
Niee rettrm
steps
enstronmenta
stops
Figure 4: Learning curves for deep RL algorithms with different exploration strategies, where the top and bottom row corresponds to results A2C and PPO respectively. The solid curves correspond to the mean, and the shaped region represents half standard deviation of the average return over 10 random seeds.
beta=0.01
beta-0.0
beta-0.1
HallCheetahw22
Walker26w2
Ant-22
Avorage IOUun
Total emdiromental steps (1ab)
Tolal environmentaa steps (1a6)
Total envirummental steps (146) Hopper v2
Reacher V2
inveredDuuliceendlllm v2
Averoge rotuun m
Averag retur
onvironmental steps
steps
steps
Figure 5: Learning curves for Coherent-A2C OpenAl MuJoCo continuous control tasks. The
solid curves correspond to the mean, and the shaped region represents half standard deviation of the average return over 10 random seeds.
18

---

Under review as a conference paper at ICLR 2021
(beta=0.0) 0)
OurNoisyhet-A2C
HailChetth-v22
Walker2d
Ant-va
Oge log vorlancc of grodercc
Sapoo 20 1o
Neenge 5g
Toral environmenaal steps (106)
Total envronmental stope (106)
Total environmenta stops (165) Happer v2
Reacher V2
InverseDDouloPenduuuu v2
Noeeage brg vatte gradless
Rooer agga varte varitance spaadiee
Total enverormentall stups (1e6)
Total envioomental staps (166)
Total environmenia steps (105)
Figure 6: Log variance of gradient estimates for Coherent-A2C and OurNoisyNet-A2C on OpenAI MuloCo continuous control tasks. The solid curves correspond to the mean, and the shaped region represents half standard deviation of the average log variance over random seeds.
Coheront-A2C (bela=0.0)
OurNeessyNett ABC (jast loyer)
HallChoetah
bcerage returrn
80
Total envirnnenncal steps (166)
O anviromental staps (1eb)
Total iivironmenta saass (146) Hopper-V2
Reacherv2
invereetbauleFenntttiiz
Averoge roturn
Averaga edturn
Avarage raturn
stops (1o6)
endironmental staps (Le5)
Figure 7: Learning curves for Coherent-A2C and OurNoisyNet-A2C on OpenAl MuJoCo continu-
ous control tasks. The solid curves correspond to the mean, and the shaped region represents half standard deviation of the average return over 10 random seeds.
19

---

Under review as a conference paper at ICLR 2021
D Dest layert
Our lepers)
HalfCheetah-v2
Walker2d-v2
Total enviromeenaal steps (1eb) Reachar-v2
Total envizomeetall steas (leb) mwertoddooubieeennnlllo
Total environmental stege (lef) Hopper-v2
Avrergee metu
Auserooe
Figure 8: Learning curves for OurNoisyNet-A2C on OpenAI MuJoCo continuous control tasks. The solid curves correspond the mean, and the shaped region represents half standard deviation of the average return over 10 random seeds.
20


Reviews:
[{"review_id": "nscZAmpeQGQ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper focuses on undirected exploration strategies in reinforcement learning. Following the prior work, this paper proposes an exploration method unifying the step-based and trajectory-based exploration. The authors propose to perturb only the last(linear) layer of the policy for exploration, instead of perturbing all layers of the policy network. Also, the authors use analytical and recurrent integration for policy updates. Experiments show that the proposed exploration strategy mostly helps A2C, PPO and SAC in three Mujoco environments.\n\nClarity:\nThis paper is generally written clearly. Some details need more clarification as pointed out in 'Cons'.\n\nOriginality:\nAs far as I know, the proposed technique is novel in the literature of undirected exploration. But for the three bullet points in section 1, the first point of \"Generalizing Step-based and Trajectory-based Exploration\" should not be one of the main contributions of this paper, because this paper follows the formulation of policy in van Hoof et al. (2017) and the latter proposed the generalized exploration connecting step-based and trajectory-based exploration. The work can be viewed as an extension of van Hoof et al. (2017)  with a deep policy network.\n\nSignificance:\nThe proposed method is mathematically solid, but the main concern lies in empirical performance. Nowadays SAC is the state-of-the-art and generally used method for continuous control tasks and it is more advanced than A2C and PPO. But the proposed method does not obviously improve the performance of SAC while inducing much more complexity in policy learning. Therefore the significance of the proposed approach in practice might be limited.\n\nPros:\n*The authors provide detailed mathematical derivation (in the main text and the appendix) to support the proposed method.\n*The proposed method significantly outperforms the baselines when investigating the on-policy methods A2C and PPO.\n*The authors provide ablative studies about hyper-parameter values and components of the proposed method with A2C.\n\nCons:\n*In section 4.2, \"we maintain and adapt a single magnitude σ for the parameter noise\". What's the motivation of this setting different from the formulation in section 4.1?\n*In section 5, why the advantage of the proposed method is poor with SAC? What's the value of hyper-parameters α and δ? Is the proposed method sensitive to these hyper-parameter choices? \n*In section 5, apart from the comparison of the performance of the learned policy, the comparison of the complexity (which might be measured by wall time to learn the policy?) of different exploration strategies can also be interesting. \n*In the first two rows of Figure 1, why the baseline methods NoisyNet-A2C(PPO) and PSNE-A2C(PPO) even significantly underperform the vanilla A2C(PPO)? The intuition is that introducing exploration strategies will mostly help the agent learns more quickly. Is it possible that the baselines are not tuned well?\n*The experiments on a single domain (Mujoco) seems not convincing enough. It will be better if there are experiments on other more complicated domains.", "extra_scores": {}, "extra_text": {}}, {"review_id": "jeF5t42RPFd", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I would like to thank the authors of \"Deep Coherent Exploration For Continuous Control\" for their valuable and interesting submission. \n\nSummary of the paper\n--\n\nThe basis of this work is van Hoof et al., 2017; there, “Generalized Exploration” views policy parameters as being drawn from a per-trajectory Markov chain. Experience is collected with a different set of parameters at each timestep, corresponding to steps along the chain.\nThe authors of this work introduce “Deep Coherent Exploration”, which scales to deep reinforcement learning methods.\nThe main contributions are:\n1. Simplifying the setting by modeling just the parameters in the last layer.\n2. A recursive, analytic expression for marginalizing over the last-layer parameters, useful for obtaining low-variance gradients with on-policy methods.\n3. Detailed recipes for incorporating the method into on-policy methods (A2C and PPO) as well as an off-policy method (SAC).\n\nAssessment\n--\nThis work explores an important problem in RL and proposes a promising method that would be of interest to many in the community, and I think it would be a valuable contribution to ICLR.\n\n-- The positives -- \n\nThe paper is well written, and does a great job of introducing the reader to the relevant concepts and situating itself in the literature.\nThe empirical results for the on-policy methods are really strong and clearly demonstrate the value of this approach. Additionally, the ablation experiments were very insightful.\nThe detailed appendix makes me confident that readers would be able to easily reproduce the method.\n\n-- The concerns --\n\nThe story for off-policy methods seems almost unrelated: the generative model is much more restrictive (isotropic noise), the optimization method is based on a heuristic (that subsequent policy parameters should be separated by a fixed distance in the action distribution), detailed balance isn't maintained within the Markov chain, and the experimental results aren’t as strong as those of the on-policy settings.\n\nSuggestions\n--\nIt might make more sense to reframe this as an on-policy method and explicitly address the off-policy case as a limitation. Would the authors consider this alternative?\n\nI tentatively score this paper as accept, and looking forward to the rebuttal to calibrate with the other reviewers.", "extra_scores": {}, "extra_text": {}}, {"review_id": "fD09cgHo5Zx", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method to combine step-based exploration with trajectory-based exploration (in the form of action-space noise and parameters-space noise) in continuous MDPs, which is scalable to deep RL methods.\n\nThe paper is overall well-written and easy to follow. The Introduction and Related-work sections are good and clear.\nSection 3 could benefit from some proof-reading. In particular, Section 3.2 is quite dense. I think it would be unhelpful for the reader not already familiar with the discussed algorithms, and on the other hand redundant for those familiar with them. I would consider moving it to the appendix, and instead provide a more high-level description of policy-gradient methods, without getting into the specific details of PPO vs SAC vs A2C. Also consider that this section uses terms which are not explicitly defined (Q-function and Advantage function) again, making it less approachable or clear for readers less familiar with RL.\nOne other minor (and technical) issue is that the font used in the figures (legend, axis titles, etc) is very small, barely readable even in 150%.\n\nWhile the underlying theoretical ideas are not novel (as the authors mention, the basic approach here is following Hoof et al. 2017), there is an important contribution in the scalability of the method, as well as in its evaluation on \"standard\" benchmark for continuous RL against some other strong baselines. Another important advantage of the approach is that while the policy is non-markov (due to the \"global\" trajectory-based exploration or coherence), the policy gradients can still be estimated in a more-or-less standard, step-based, way, thanks to analytical integration of the \"latent\" variables (basically the parameters of the last layer), hereby overcoming the challenge of high variance in PG estimate for non-markov policies.\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "QEkFnN7VACb", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \n\nThis paper proposes Deep Coherent Exploration that unifies step-based exploration and trajectory-based exploration on continuous control. There exists a prior work that bridges a gap between the two exploration methods for linear policies, and this paper generalizes the prior work for various deep RL methods: on-policy (A2C, PPO) and off-policy (SAC). Finally, Deep Coherent Exploration enhances the performance of baseline algorithms and has better performance than prior works (NoisyNet, PNSE) on Mujoco tasks.\n\nPros:\n\n+ For combining the proposed method with on-policy learning, this paper derives the log-likelihood of whole trajectory recursively.\n+ For on-policy methods (A2C, PPO), the proposed method has large performance gain on Mujoco tasks.\n\nCons:\n\n- The idea of this paper directly follows GE [van Hoof et al., 2017] and is not much different from GE.\n- For SAC, the proposed method is not much effective and it even degrades the performance of the HalfCheetah task.\n- The paper focuses on exploration, but the experiments only focus on the return performance of simple Mujoco tasks.\n- In order to show the superiority of the proposed method, additional experiments on pure exploration or sparse rewarded tasks are needed.\n\nMinor concerns:\n\n* In background, there is no explanation about step-based and trajectory-based exploration.\n* For the off-policy case, there is insufficient explanation for why they use single sigma and the connection point of the proposed method and eq (5).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Unfortunately some of the reviewers' reactions to the author feedback won't be visible to the authors.
The reviewers highly appreciated the replies and revision of the paper

Pros:
- The paper renders Generalized Exploration tractable for deep RL.
- The idea is applicable to many DRL methods and is potentially very valuable to deal with the headaches associated to DRL.

Cons:
- R2 and R4 are still concerned about whether 'smart' exploration will always be advantageous, and whether the added complexity is a good trade-off for the (potentially) better performance. A comparison to 'pure' exploration would still be insightful. 
- the new 'SAC with Deep Coherent Exploration' only partially addresses the concerns of R2 and R4, especially in terms of performance

While the paper has improved drastically during the reviewing process, there are still a few too many doubts.
Decision: Reject