Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: I would like to thank the authors of "Deep Coherent Exploration For Continuous Control" for their valuable and interesting submission. 

Summary of the paper
--

The basis of this work is van Hoof et al., 2017; there, “Generalized Exploration” views policy parameters as being drawn from a per-trajectory Markov chain. Experience is collected with a different set of parameters at each timestep, corresponding to steps along the chain.
The authors of this work introduce “Deep Coherent Exploration”, which scales to deep reinforcement learning methods.
The main contributions are:
1. Simplifying the setting by modeling just the parameters in the last layer.
2. A recursive, analytic expression for marginalizing over the last-layer parameters, useful for obtaining low-variance gradients with on-policy methods.
3. Detailed recipes for incorporating the method into on-policy methods (A2C and PPO) as well as an off-policy method (SAC).

Assessment
--
This work explores an important problem in RL and proposes a promising method that would be of interest to many in the community, and I think it would be a valuable contribution to ICLR.

-- The positives -- 

The paper is well written, and does a great job of introducing the reader to the relevant concepts and situating itself in the literature.
The empirical results for the on-policy methods are really strong and clearly demonstrate the value of this approach. Additionally, the ablation experiments were very insightful.
The detailed appendix makes me confident that readers would be able to easily reproduce the method.

-- The concerns --

The story for off-policy methods seems almost unrelated: the generative model is much more restrictive (isotropic noise), the optimization method is based on a heuristic (that subsequent policy parameters should be separated by a fixed distance in the action distribution), detailed balance isn't maintained within the Markov chain, and the experimental results aren’t as strong as those of the on-policy settings.

Suggestions
--
It might make more sense to reframe this as an on-policy method and explicitly address the off-policy case as a limitation. Would the authors consider this alternative?

I tentatively score this paper as accept, and looking forward to the rebuttal to calibrate with the other reviewers.
review_title: A promising exploration method that would be of interest to many in the community.

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: This paper focuses on undirected exploration strategies in reinforcement learning. Following the prior work, this paper proposes an exploration method unifying the step-based and trajectory-based exploration. The authors propose to perturb only the last(linear) layer of the policy for exploration, instead of perturbing all layers of the policy network. Also, the authors use analytical and recurrent integration for policy updates. Experiments show that the proposed exploration strategy mostly helps A2C, PPO and SAC in three Mujoco environments.

Clarity:
This paper is generally written clearly. Some details need more clarification as pointed out in 'Cons'.

Originality:
As far as I know, the proposed technique is novel in the literature of undirected exploration. But for the three bullet points in section 1, the first point of "Generalizing Step-based and Trajectory-based Exploration" should not be one of the main contributions of this paper, because this paper follows the formulation of policy in van Hoof et al. (2017) and the latter proposed the generalized exploration connecting step-based and trajectory-based exploration. The work can be viewed as an extension of van Hoof et al. (2017)  with a deep policy network.

Significance:
The proposed method is mathematically solid, but the main concern lies in empirical performance. Nowadays SAC is the state-of-the-art and generally used method for continuous control tasks and it is more advanced than A2C and PPO. But the proposed method does not obviously improve the performance of SAC while inducing much more complexity in policy learning. Therefore the significance of the proposed approach in practice might be limited.

Pros:
*The authors provide detailed mathematical derivation (in the main text and the appendix) to support the proposed method.
*The proposed method significantly outperforms the baselines when investigating the on-policy methods A2C and PPO.
*The authors provide ablative studies about hyper-parameter values and components of the proposed method with A2C.

Cons:
*In section 4.2, "we maintain and adapt a single magnitude σ for the parameter noise". What's the motivation of this setting different from the formulation in section 4.1?
*In section 5, why the advantage of the proposed method is poor with SAC? What's the value of hyper-parameters α and δ? Is the proposed method sensitive to these hyper-parameter choices? 
*In section 5, apart from the comparison of the performance of the learned policy, the comparison of the complexity (which might be measured by wall time to learn the policy?) of different exploration strategies can also be interesting. 
*In the first two rows of Figure 1, why the baseline methods NoisyNet-A2C(PPO) and PSNE-A2C(PPO) even significantly underperform the vanilla A2C(PPO)? The intuition is that introducing exploration strategies will mostly help the agent learns more quickly. Is it possible that the baselines are not tuned well?
*The experiments on a single domain (Mujoco) seems not convincing enough. It will be better if there are experiments on other more complicated domains.
review_title: Complex approach with little improvement

final_score: rating: 7: Good paper, accept
review_content: This paper presents a method to combine step-based exploration with trajectory-based exploration (in the form of action-space noise and parameters-space noise) in continuous MDPs, which is scalable to deep RL methods.

The paper is overall well-written and easy to follow. The Introduction and Related-work sections are good and clear.
Section 3 could benefit from some proof-reading. In particular, Section 3.2 is quite dense. I think it would be unhelpful for the reader not already familiar with the discussed algorithms, and on the other hand redundant for those familiar with them. I would consider moving it to the appendix, and instead provide a more high-level description of policy-gradient methods, without getting into the specific details of PPO vs SAC vs A2C. Also consider that this section uses terms which are not explicitly defined (Q-function and Advantage function) again, making it less approachable or clear for readers less familiar with RL.
One other minor (and technical) issue is that the font used in the figures (legend, axis titles, etc) is very small, barely readable even in 150%.

While the underlying theoretical ideas are not novel (as the authors mention, the basic approach here is following Hoof et al. 2017), there is an important contribution in the scalability of the method, as well as in its evaluation on "standard" benchmark for continuous RL against some other strong baselines. Another important advantage of the approach is that while the policy is non-markov (due to the "global" trajectory-based exploration or coherence), the policy gradients can still be estimated in a more-or-less standard, step-based, way, thanks to analytical integration of the "latent" variables (basically the parameters of the last layer), hereby overcoming the challenge of high variance in PG estimate for non-markov policies.




review_title: Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: 

This paper proposes Deep Coherent Exploration that unifies step-based exploration and trajectory-based exploration on continuous control. There exists a prior work that bridges a gap between the two exploration methods for linear policies, and this paper generalizes the prior work for various deep RL methods: on-policy (A2C, PPO) and off-policy (SAC). Finally, Deep Coherent Exploration enhances the performance of baseline algorithms and has better performance than prior works (NoisyNet, PNSE) on Mujoco tasks.

Pros:

+ For combining the proposed method with on-policy learning, this paper derives the log-likelihood of whole trajectory recursively.
+ For on-policy methods (A2C, PPO), the proposed method has large performance gain on Mujoco tasks.

Cons:

- The idea of this paper directly follows GE [van Hoof et al., 2017] and is not much different from GE.
- For SAC, the proposed method is not much effective and it even degrades the performance of the HalfCheetah task.
- The paper focuses on exploration, but the experiments only focus on the return performance of simple Mujoco tasks.
- In order to show the superiority of the proposed method, additional experiments on pure exploration or sparse rewarded tasks are needed.

Minor concerns:

* In background, there is no explanation about step-based and trajectory-based exploration.
* For the off-policy case, there is insufficient explanation for why they use single sigma and the connection point of the proposed method and eq (5).

review_title: Deep Coherent Exploration For Continuous Control

Meta-review: Unfortunately some of the reviewers' reactions to the author feedback won't be visible to the authors.
The reviewers highly appreciated the replies and revision of the paper

Pros:
- The paper renders Generalized Exploration tractable for deep RL.
- The idea is applicable to many DRL methods and is potentially very valuable to deal with the headaches associated to DRL.

Cons:
- R2 and R4 are still concerned about whether 'smart' exploration will always be advantageous, and whether the added complexity is a good trade-off for the (potentially) better performance. A comparison to 'pure' exploration would still be insightful. 
- the new 'SAC with Deep Coherent Exploration' only partially addresses the concerns of R2 and R4, especially in terms of performance

While the paper has improved drastically during the reviewing process, there are still a few too many doubts.
Decision: Reject