Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper investigates how to achieve smooth exploration during deep reinforcement learning (deep RL), which is particularly an important problem when we want to apply deep RL to real-world environments. Basically, the author wants to improve unstructured exploration that shows noisy exploration at the early stage (Figure 1). The key idea is to improve the idea of State-Dependent Exploration (SDE) which defines action noises as a function of the state and an additional episodic parameter theta. The authors identified four disadvantages of SDE that limit exploration, and propose to 1) sample the episodic parameters every n steps instead of every n episodes and 2) use other feature values to generate noises. 

The authors demonstrated that this approach can mitigate the known trade-off issue between safe exploration and learning performance. The authors also have shown that the proposed method can learn motor skills in three different real-world robotic environments, including a tendon-driven robot, a quadruped, and an RC car.

strengths_and_weaknesses: + The paper discusses a very important problem to deploy deep RL on real-world environments.
+ The paper demonstrates a good intellectual discussion about SDE, including its strengths and weaknesses.
+ The authors tested many interesting scenarios, including standard benchmark problems and real-robot learning.
- The design of gSDE sounds a little bit ad hoc.
- Some experiments are not complete: please refer to the text below.

summary_of_recommendation: In summary,my support is lukewarm. I like the problem itself and discussion about SDE, but gSDE seems to be a minor extension of SDE without much technical novelty.

First of all, I am very interested in real-robot learning, and this paper discusses one of the most important problems, how can we achieve safe exploration at the early stage of the learning process. And the paper describes pros and cons of SDE, which seem reasonable. I agree that SDE is based on a few design choices that can limit its exploration capability.

On the other hand, I’m not sure gSDE provides good solutions to it. The first idea is to resample the episodic randomization parameter theta in the middle of episodes, per two to 64 steps. I agree that they can be a good strategy for periodic locomotion tasks that are compared in Table I, but it cannot be a universal solution for general MDPs. I guess that this simple trick may not be very effective for other non-periodic (or episodic) tasks, such as manipulation. The example of a tendon-driven robot also seems a bit in favor of gSDE because the target trajectories have been periodically changed. I believe this can be a major drawback of the proposed approach.

The second innovation is to use the policy features, which are defined as the last layer values before the deterministic outputs. Yes, they may work better than the baselines. But I am a little bit concerned that the main reason for choosing this feature is simply because “we can in fact use any features”. I do not see any scientific conjectures here. What other features can we use? What if they do not work? 

This lack of scientific justification requires the authors to provide more through experiments. Please see the below comments for more details.
- In general, I believe a much wider set of experiments is needed for supporting the paper’s statements.
- Figure 2 does not include SDE experiments. Is this intended? Not sure gSDE-Episodic is identical to SDE.
- I’m not sure Table 2 suggests that gSDE is the best. Low-pass filter achieves lower positional errors. Continuity cost does not say anything, if it is acceptable (= it is possible to learn a policy on real robots).
- Experiments on a quadrupedal robot and an RC car do not seem informative due to lack of statistics.

issues: First, I would suggest the authors discuss more scientific justification of gSDE.
I also suggest the authors conduct more extended experiments.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This work addresses the ubiquitous problem of shaky controllers trained using model free deep RL by proposing the exploration technique, generalized State Dependent Exploration (gSDE). gSDE builds on prior work SDE and substitutes the per-step random Gaussian action noise in many deep RL algorithms with state-dependent noise that only varies every n-steps. The effect is smoother and more consistent exploration trajectories which reduce wear and tear without substantially compromising performance. gSDE is assessed in simulation as well as on three different physical robotic systems.  

strengths_and_weaknesses: **Strengths**

* Effectively addresses a ubiquitous problem in Deep RL for robotics - shaky controllers.
* The proposed method, gSDE, is widely applicable. It can be applied to any continuous control problem that is solved using either a pure policy gradient method (e.g. REINFORCE) or a hybrid policy and value based method (e.g. PPO, A2C, TD3, SAC).
    1. The only approaches for which this does not appear applicable are value-based methods such as DQN, Rainbow, or QT-OPT
* It is very simple to implement.
* The effectiveness of gSDE compared to other exploration noise structures is explored through rigorous and extensive experiments in simulation using a range of relevant algorithms (SAC, TD3, PPO, A2C) on multiple environments.
    1. The ablation studies (e.g. feature selection, noise re-sampling frequency) are relevant and illuminating.
    2. Figures 2 and 3 are especially interesting.
    3. All simulation results are reported with the mean and std error over 10 runs.
* The approach is further validated with the training of successful policies on three very different physical robots.
* The clarity of presentation and technical quality are excellent throughout.
    1. The explanations provided in both the main paper and appendix strike a good balance between detail and concision and I am confident that the results in the paper would be reproducible based on the information provided.

**Weaknesses**

* Low originality: gSDE is a minor modification to an existing method, SDE
* Lines 100 - 126: The claim that gSDE addresses all four of the identified limitations with SDE is not fully convincing
    1. (ii) Now the variance of the policy depends on the policy features dimension so needs to be tuned for each network.
    2. (iv) For similar reasons to those given in (iv) (lines 108 -109) shouldn’t the policy features now be normalized?
* Figure 2: Given the apparent linear relationship between the frequency of noise parameter re-sampling and continuity cost, the sudden drop-off to the Unstructured noise is surprising. It is achieving returns equivalent to gSDE-64 but with almost 2x normalized cost
    1. There is a similar puzzling relationship occurring in Figure 3b)
    2. To better understand this it would be helpful to add gSDE-4 and gSDE-2 to Figure 2 and gSDE 2 to Figure 3b). 
    3. I would expect a somewhat smooth transition between gSDE-2 and unstructured noise. Regardless, adding these points would be illuminating.
* Grammar
    1. Line 13: “permits to have” → facilitates / enables?
    2. Line 54: “without the need of a” -> “without need of a” / “without the need for”
    3. Line 94: “allows to adapt” → “enables adaptation of”
    4. Line 184: “allows to have” → “permits a” / “makes it possible to”

**Comments and Questions**
* Lines 42 - 44: There is significant interest in and efforts made to apply deep RL to real robotic problems so it is not clear to me that a focus on simulation is the main reason for SDE not being combined with deep RL. Perhaps up to now other issues such as sample efficiency and generalizing to less structured environments or multiple tasks have occupied the attention of the community.
* Line 66: When actions are discrete exploration is also typically done in the action space.
* Table 2: Is there just one evaluation trajectory?
* It would be interesting to see the learning curves for the tendon-driven task
* Walking task: It would be interesting to compare the learning curve (Figure 5b) with SAC trained with a low pass filter on this task
* Driving task: It seems that the standard SAC exploration would not break this type of robot. It would be very interesting to see a comparison SAC, SAC w/ low pass filter, SAC + gSDE

summary_of_recommendation: This work is highly relevant to robotic learning using model free deep RL and the effectiveness of gSDE is convincingly demonstrated. Additionally the method is very simple and easy to implement. Experiments are well selected and thorough. The paper is lucid, easy to understand, and overall a pleasure to read. To further strengthen the paper I raised a few minor issues and questions.

Comment on impact 4:
The paper convincingly demonstrates the effectiveness of existing ideas on reducing shakiness of deep RL policies. The simplicity of the idea and ease of implementation increase the potential impact of this work.
issues: Please address the following comments and questions:
* Lines 100 - 126: The claim that gSDE addresses all four of the identified limitations with SDE is not fully convincing
    1. (ii) Now the variance of the policy depends on the policy features dimension so needs to be tuned for each network.
    2. (iv) For similar reasons to those given in (iv) (lines 108 -109) shouldn’t the policy features now be normalized?
* Figure 2: Given the apparent linear relationship between the frequency of noise parameter re-sampling and continuity cost, the sudden drop-off to the Unstructured noise is surprising. It is achieving returns equivalent to gSDE-64 but with almost 2x normalized cost
    1. There is a similar puzzling relationship occurring in Figure 3b)
    2. To better understand this it would be helpful to add gSDE-4 and gSDE-2 to Figure 2 and gSDE 2 to Figure 3b). 
    3. I would expect a somewhat smooth transition between gSDE-2 and unstructured noise. Regardless, adding these points would be illuminating.
* Table 2: Is there just one evaluation trajectory?
* Grammar
    1. Line 13: “permits to have” → facilitates / enables?
    2. Line 54: “without the need of a” -> “without need of a” / “without the need for”
    3. Line 94: “allows to adapt” → “enables adaptation of”
    4. Line 184: “allows to have” → “permits a” / “makes it possible to”
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a new method (gSDE) for generating exploration noise for RL, conditioned on the last hidden layer of the policy network and re-generated at a frequency tunable between every timestep and every episode. This is a modification of state-dependent exploration (SDE), with the differences being 1) SDE conditions on the environment state instead of the network, and 2) SDE always re-generates once per episode. gSDE is evaluated on 4 MuJoCo tasks in simulation and 3 real-world systems and found to generate smoother action sequences than standard unstructured Gaussian noise with similar returns. The update frequency can be adjusted to modulate the trade-off between smoothness and performance.

strengths_and_weaknesses: Strengths:
* The problem is very relevant. We now have RL algorithms that are data-efficient enough to allow real-world training, which means hardware damage from noisy actions has become a blocker.
* The MuJoCo tasks, head actuation, and quadruped locomotion results are convincing.

Weaknesses:
* The changes to SDE are very slight.
* In the video of head actuation, the comparison between exploration with per-step noise and with gSDE is useful. It would be nice to have the same comparison for the quadruped.

summary_of_recommendation: The work is a relatively simple extension of SDE without a huge novelty value. However, the authors correctly point out that SDE is *not* used in RL, and gSDE solves the problems that make SDE less effective. I believe it is worthwhile to demonstrate that this simple method can make real-world training feasible for systems where it otherwise wouldn't be.

issues: * Please clarify the reasons for using policy features as exploration function input. I understand that it eliminates the need for normalization if the features are already normalized. I do not understand why SDE cannot use pixel input--wouldn't it simply require a random sample (or 3) per pixel? Is the concern the quantity of samples and multiplications?
review_title: null

Meta-review: This paper proposes a simple and easy-to-implement idea for generating exploration noise for RL. This approach is especially making RL more effective when used on real-world robotic systems, i.e. it addresses the shaky behavior problem in practice. Though the proposed idea is just minor modifications to a existing method, i.e. SDE, and lacks of scientific justification or has very little technical novelty, the reviewers believed it can have great potential applicability, e.g. any continuous control problem. Though it is evaluated on a good set of tasks: on 4 MuJoCo tasks in simulation and 3 real-world systems. However it's encouraged to provide more results to support the paper’s statements, e.g. additional statistical information of experiments on a quadrupedal robot and an RC car.

The authors have made a great response that have addressed most concerns. Additional experiment results also support the paper's claims stronger. The proposed idea can be simple but it's very practically useful and worth being shared with the robotic community.
Decision: Accept (Poster)