Smooth Exploration for Robotic Reinforcement Learning
Authors: Antonin Raffin
Jens Kober
Freek Stulp
Venue: CoRL2021 Poster (2021)
Keywords: Robotics
Reinforcement Learning
Exploration
Real World

Abstract: Reinforcement learning (RL) enables robots to learn skills from interactions with the real world.
In practice, the unstructured step-based exploration used in Deep RL -- often very successful in simulation -- leads to jerky motion patterns on real robots.
Consequences of the resulting shaky behavior are poor exploration, or even damage to the robot.
We address these issues by adapting state-dependent exploration (SDE) to current Deep RL algorithms.
To enable this adaptation, we propose two extensions to the original SDE, using more general features and re-sampling the noise periodically, which leads to a new exploration method generalized state-dependent exploration (gSDE).
We evaluate gSDE both in simulation, on PyBullet continuous control tasks, and directly on three different real robots: a tendon-driven elastic robot, a quadruped and an RC car.
The noise sampling interval of gSDE enables a compromise between performance and smoothness, which allows training directly on the real robots without loss of performance.

Smooth Exploration for Robotic Reinforcement Learning
Antonin Raffin
Jens Kober
Robotics and Mechatronics Center (RMC) German Aerospace Center (DLR) Germany antonin raffin@dlr.de
Cognitive Robotics Department
Delft University of Technology The Netherlands
j .koberttuellft....
Freek Stulp
Robotics and Mechatronics Center (RMC)
German Aerospace Center (DLR) Germany freek. wtw.ppdlr.lddrdee
Abstract:
Reinforcement learning (RL) enables robots learn skills from interactions with the real world. In practice, the unstructured step-bau step-based exploration used in Deep - very successful in simulation - leads to jerky motion patterns on real robots. Consequences of the resulting shaky behavior are poor exploration, or even damage the robot. We address these issues by adapting dete-dependent ex- ploration (SDE) [1] to current Deep RL algorithms. To enable this adaptation, we propose two extensions the original SDE, using more general features and re- sampling the noise periodically, which leads to a new exploration method general- ized state-dependent exploration (gSDE). We evaluate gSDE both in simulation, PyBullet continuous control tasks, and directly on three different real robots: a tendon-drine elastic robot, quadruped and an RC car. The noise sampling interval of gSDE enables compromise between performance and smoothness, which allows training directly on the real robots without loss performance.
Introduction
One the first robots that used artificial intelligence methods was called "Shakey", because would shake lot during operation [2]. Shaking has again become quite prevalent in robotics, but for different reason. When learning robotic skills with deep reinforcement learning (DeepRL), the de facto standard for exploration is to sample noise vector é, from Gaussian distribution independently at each time step f, and then adding to the policy output. This approach leads to the type of noise illustrated to the left in Fig. 1, and it can be very effective in simulation [3, 4, 5, 6, 7].
dtanmi sik
Sdtant B80i action
coplorato
trploratin ndus
mayy adier
malpe action
Acton
0.09
-13
-025
-033
-033
100
230
202
30
230
100
230
230
Timestaps
Timestaps
Figure 1: Left: unstructured exploration, typically used in simulated RL. Right: sSDE provides smooth and consistent exploration.
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Unstructured exploration has also been applied to robotics [8, 9]. But for experiments on real robots, has many drawbacks, which have been repeatedly pointed out [1. 10, 11, 12. 13]: 1) Sampling independently at each step leads to shaky behavior [14], and noisy, jittery trajectories. 2) The jerky motion patterns can damage the motors on a real robot, and lead to increased wear-and-tear. 3) In the real world, the system acts low pass filter. Thus, consecutive perturbations may cancel each other, leading to poor exploration. This is particularly true for high control frequency [15]. 4) It causes large variance which grows with the number of time-steps [10, 11, 12]
In practice, we have observed all of these drawbacks on three real robots, including the tendon- driven robot David, depicted in Fig. 4a, which is the main experimental platform used in this work. For all practical purposes, Deep RL with unstructured noise cannot be applied to David.
In robotics, multiple solutions have been proposed to counteract the inefficiency of unstructured noise. These include correlated noise [8, 15]. low-pass filters [16, 17], action repeat [18] or lower level controllers [16, 9]. more principled solution is to perform exploration in parameter space, rather than in action space [19, 20J. This approach usually requires fundamental changes in the algorithm, and is harder to tune when the number of parameters is high.
State- Dependent Exploration (SDE) [1, 11] was proposed as a compromise between exploring in parameter and action space. SDE replaces the sampled noise with a state-dependent exploration function, which during an episode returns the same action for given state. This results in smoother exploration and less variance per episode.
To the best of our knowledge, no Deep RI. algorithm has yer been successfully combined with SDE. We surmise that this because the problem that it solves - solles--haky,, jerky movement is not as noticeable in simulation, which is the current focus of the community.
In this paper, we aim at reviving interest in SDE as effective method for addressing exploration issues that arise from using independently sampled Gaussian noise on real robots. Our concrete contributions, which also determine the structure of the paper, are:
1. Highlighting the issues with unstructured Gaussian exploration (Sect. 1).
2. Adapting SDE to recent Deep RL algorithms, and addressing some issues of the original formulation (Sects. 2.2 and 3).
3. Evaluate the different approaches with respect to the compromise between smoothness and performance, and show the impact of the noise sampling interval (Sects. 4.1 and 4.2).
4. Successfully applying RI. directly on three real robots: a tendon-driven robot, a quadruped and an RC car, without need of simulator or filters (Sect. 4.3).
2 Background
In reinforcement learning, an agent interacts with its environment, usually modeled as Markov Decision Process (MDP) (S, A, p,r) where S is the state space, A the action space and p(s'|s (s, a) the transition function. At every step t, the agent performs an action in state s following its policy S + A. It then receives a feedback signal in the next state s': the reward r(s, a). The objective of the agent is maximize the long-term reward. More formally, the goal to maximize the expectation of the sum of discounted reward, over the trajectories Per generated using its policy m:
Σ E(er [r'r(st, ar)]
where 7 E 1) is the discount factor and represents a trade-off between maximizing short-term and long-term rewards. The agent-environment interactions are often broken down into sequences called episodes, that end when the agent reaches a terminal state.
2.1 Exploration in Action or Policy Parameter Space
In the case of continuous actions, the exploration is commonly done in the action space [21, 22, 23, 24, 25, 5]. At each time-step, a noise vector e, is independently sampled from a Gaussian distribution and then added to the controller output:
pas;; 0,)) + Co:
G ~=(00
(2)

---

where ss(s:) is the deterministic policy and <(alls)) ~ (Nu(s)), ) is the resulting stochastic policy, used for exploration. 82 denotes the parameters of the deterministic policy.
For simplicity, throughout the paper, we will only consider Gaussian distributions with diagonal covariance matrices. Hence, here, a vector with the same dimension as the action space A.
Alternatively, the exploration can also be done in the parameter space [11, 19, 20]:
a. - (( O, e),
E E~ N0, o²)
the beginning of an episode, the perturbation e is sampled and added to the policy parameters pr This usually results in more consistent exploration but becomes challenging with an increasing number of parameters [19].
2.2 State- Dependent Exploration
Dependent Exploration (SDE) [1, 11] is an intermediate solution that consists in adding noise function of the state St, to the deterministic action p(ss). At the beginning of episode, the parameters Oe of that exploration function are drawn from a Gaussian distribution. The resulting action a, is as follows:
a, [(s;; 0,)) + c(s;; 0.),
0. ~~ ~~0,0
(4)
This episodc-based exploration is smoother and more consistent than the unstructured step-based exploration. Thus, during one episode, instead of oscillating around a mean value, the action a for given state will be the same.
SDE should not be confused with unstructured noise where the variance be state-dependent but the noise is still sampled at every step, as it is the case for SAC.
In the remainder of this paper, to avoid overloading notation, we drop the time subscript 4, i. e. we now write s instead of st. 8, or a, now refer to an element of the state or action vector.
In the case of a linear exploration function E(s; 0.) = 0.s. by operation on Gaussian distributions, Rückstieß al. [1] show that the action element a, normally distributed:
Ty(ayls) (s) ~ N (1)(ss),
(5)
where â is diagonal matrix with elements di Vz, (or)))))
Because we know the policy distribution, we can obtain the derivative of the log-likelihood log (aas)) with respect to the variance a::
log π (ais) da:j
aa, n1) o3 st
(6)
63
6
This can be casily plugged into the likelihood ratio gradient estimator [26], which enables adaptation of a during training. SDE is therefore compatible with standard policy gradient methods, while addressing most shortcomings of the unstructured exploration.
For non-linear exploration function, the resulting distribution x (als) is most of the time unknown. Thus, computing the exact derivative w.r.t. the variance not trivial and may require approximate inference. As focus simplicity, we leave this extension for future work.
3 Generalized State-Dependent Exploration
Considering Eqs. (5) and (6), some limitations of the original formulation are apparent:
i The noise does not change during one episode, which is problematic if the episode length is long, because the exploration will be limited [27].
The variance of the policy o; VE, (a depends the state space dimension (it grows with it), which means that the initial a must be tuned for each problem.
ii There only a linear dependency between the state and the exploration noise, which limits the possibilities.

---

iv The state must be normalized, as the gradient and the noise magnitude depend on the state magnitude.
To mitigate the mentioned issues and adapt to Deep RL algorithms, we propose two improvements:
1. We sample the parameters 0g of the exploration function every steps instead of every episode.
2. Instead of the state can in fact any features. We chose policy features z Ozy)) (last layer before the deterministic output (ss) On (s; Oz,,)) )) as input to the noise func- tion e(s; 0.) (s).
Sampling the parameters 9g every n steps tackles the issue i. and yields a unifying framework [27] which encompasses both unstructured exploration (n. 1) and original SDE (n lenghd)..ength). Although this formulation follows the description of Deep RL algorithms that update their parame- ters every m steps, the influence of this crucial parameter on smoothness and performance was until now overlooked.
Using policy features allows mitigating issues ii, iii and iv: the relationship between the state and the noise e is non-linear and the variance of the policy only depends on the network architecture. This makes gSDE more task tusk-independent (as the network architecture is usually kept constant) and saves a lot of parameters and computation when working with large state space such as images: the number of parameters and operations is only function of the last layer size and action dimension and no more of the state space size. This formulation is therefore more general and includes the original SDE description, when using state input the noise function when the policy is linear.
We call the resulting approach generalized State-Dependen Exploration (gSDE).
Deep RL algorithms Integrating this updated version of SDE into recent Deep RL algorithms, such as those listed in the appendix, straightforward. For those that rely on a probability distribu- tion, such SAC PPO, can replace the original Gaussian distribution by the one from Eq. (5), where the analytical form of the log-likelihood known (cf. Eq. (6)).
4 Experiments
In this section, we study gSDE to answer the following questions:
How does gSDE compares the original SDE? What is the impact of each proposed modification?
How does gSDE compares other type of exploration noise in terms of compromise be- tween smoothness and performance?
How does gSDE performs real system?
4.1 Compromise Between Smoothness and Performance
Experiment setup In order to compare SDE to other type of exploration in terms compromise between performance and smoothness, we chose 4 locomotion tasks from the PyBullet [28] envi- ronments: HALFCHEETAI, ANT, HOPPER and WALKER2D. They are similar to the one found in OpenAl Gym [29] but the simulator is open source and they are harder to solve!. In this section, focus on the SAC algorithm as it will be the one used on the real robot, although we report results for additional algorithms such PPO in the appendix.
To evaluate smoothness, we define a continuity cost C = 100 x E, (~س which yiclds values between 0 (constant output) and 100 (action jumping from one limit to another at every step). The continuity cost of the training Cirain is proxy for the wear-and-tear of the robot.
hhttps://trrmm..innk/Pppepp harder- tain--than-Mooooo

---

We compare the performance of the following configurations: (a) no exploration noise, (b) unstruc- tured Gaussian noise (original SAC implementation). (c) correlated noise Crrstein-Uhllnbed pro- cess [30] with a=0.2, OU noise in the figure), (d) adaptive parameter noise [19] (a=0.2), (e) g DE. To decorrelate the exploration noise from the one due to parameter update, and to be closer to a real robot setting, we apply the gradient updates only at the end of each episode.
million steps and report the average score over 10 runs together with the average continuity cost during training and their standard error. For each run, we test the learned policy on 20 evaluation episodes every 10000 steps, using the deterministic controller (u(s)). Re- garding the implementation, we use modified version of Stable-Baseling Baselines3 [31] together with the RL Zoo training framework [32]. The methodology we follow to tune the hyperparameters and their details can be found in the appendix. The code we used to run the experiments and tune the
We fix the budget
hyperparameters can be found in the supplementary material.
Algorithm
HALFCHEETAH
ANT
HOPPER
WALKER2D
SAC
Return f
Cusin
Return f
Cerain
Retur f
Crram
Return 1
Crrain
w/o noise
2562 +/ 102
2.6
2600 +/ 364
2.0 0.2
1661 +/
+/ 0.1
2216 +/ 40
1/ 0.1
w/ unstrrictured
2994 +/- 89
4.8 el 0.2
3394 w/- 64
5.1 --1
2434 +/- 190
3.6 +/- 0.1
2225 -/- 35
3/6 0.1
OU
2692 +/
2.9
2849 +/
2200 +/ 53
2/ 0.1
2089 25
2.0 0.0
w/ param noise
2834 +/- 54
2.9 et 0.1
3294 w/- 55
+/-
1685 +/- 279
2/2
2294 -/- 40
1.8 +/-
2
2987 +/ 85
4.1 0.2
3366 +/ 50
4.7 0.1
2532 +/ 70
2/8 0.1
2237
+/- 55
2.8 +/ 0.1
w/ 2SDE-
2798 +/- 41
4.1+// 0.2
3227 +/- 182
1.8++/0.2
2541 +/- 49
2.6+//0. 0.1
2322
+/- 69
2.6 0.1
gSDE 8
2850 +/ 73
4,1 0.2
3459 +/ 52
3.9 +1-02
2646 +/ 45
2/4 0.1
2341
+/ 45
2.5 +/ 0.1
gSDE-664
2970 +/- 132
3,5
+/- 0.1
3160
+/ 184
3.5 +/10.1
2476
+/- 99
2.0-//0. +/-0.1
2324
+/- 39
2.3 +/- 0.1
gSDE Episodic
2741 +/ 115
3.1 0.2
3044 +/ 106
2.6 0.1
2503 +/ 80
1.8 +/
0.1
2267
+/ 34
2.2 +/2 0.1
Table 1: Detailed return and continuity cost results for SAC with different type of exploration on PyBullet environments. We report the mean and standard error over 10 runs of 1 million steps. For each benchmark, we highlight the results the method(s) with the best mean when the difference is statistically significant.
Continuily Performance PyBullet Environments
Results Table 1 and Fig. 2 shows the results on the PyBullet tasks and the compromise be- tween continuity and performance. Without any noise ("No Noise" in the figure), SAC is still able to solve partially those tasks thanks to a shaped reward, but it has the highest variance in the results. Although the correlated and pa- rameter noise yield lower continuity cost dur- ing training, it comes at cost of performance. gSDE is able to achieve a good compromise be- tween unstructured exploration and correlated noise by making use of the noise repeat param- eter. gSD- (sampling the noise every 8 steps) even achieves better performance with a lower continuity cost at train time. Such behavior is what is desirable for training on a real robot: we must minimize wear-and-tear at training while still obtaining good performance at test time.
1.00
gSDE.8 @SDE-32
gSDE-2
0.95
BSDE
Unstructured
Normalized Return
-SDE-Epiiooic
0.90
OU Noise
0.85
Parameter Noise
0.80
No Noise
0.5
0.6
0.9
Normalized Continuity Cost
Normalized retur and continuity cost of SAC on 4 PyBullet tasks with different type of explo- ration. gSDE provides a compromise between perfor- mance and smoothness.
Figure 2:
4.2 Comparison to the Original SDE
In this section, investigate the contribution of the proposed modifications to the original SDE: sampling the exploration function parameters every n steps and using policy features as input to the noise function.
Sampling Interval gSDE is a n-step version of SDE, where n allows interpolating between the unstructured exploration n = 1 and the original SDE per-episode formulation. This interpolation permits a compromise between performance and smoothness at train time (cf. Table and Fig. 2). Fig. 3b shows the importance of that parameter for PPO on the WALKER2D task. If the sampling interval is too large, the agent would not explore enough during long episodes. On the other hand, with high sampling frequency n R 1, the issues mentioned in Sect. arise.
5

---

Influence of the exploration function input
Influence of the sampling intenval on Walker2D
2.00
5000
2540
360
Score
Nettod
Ho IChwetsh
Art
Hopper
Wallacd
Fpsecic
Environment
Sampling interval
(a) Exploration function input
(b) Sampling interval (PPO on WALKER2D)
Figure 3: Impact of gSDE modifications over the original SDE on PyBullet tasks. (a) Influence of the input to the exploration function c(s; 0.) for SAC and PPO: using latent features from the policy z, (Latent) usually better than using state (Original). (b) The frequency sampling noise function parameters crucial for PPO with gSDE.
Policy features as input Fig. 3a shows the effect of changing the exploration function input for SAC and PPO. Although it varies from task to task, using policy features ("latent" in the figure) is usually beneficial, especially for PPO. It also requires less tuning and no normalization only on the policy network architecture. Here, the PyBullet tasks are low dimensional and the state space size is of the same order, so no careful per-task tuning needed. Relying on features also allows learning directly from pixels, which is not possible in the original formulation.
depends
Compared to the original SDE, the two proposed modifications with the noise sampling interval n having the most impact. Fortunately, as shown in Table and Fig. it can be chosen quite freely for SAC. In the appendix, provide additional ab- lation study that shows SDE is robust to the choice of the initial exploration variance.
beneficial to the performance,
4.3 Learning to Control a Tendon-Driven Elastic Robot
0, (0eg)
(dmg)
(deg)
Desired Pass
Modal
45DE
Low-2ass fiber
10
14
Time
Tendon-driven clastic
humanoid robot
Model-Based controller controller on the
Figure 4: (a) The lendon-driven robot [33] used for experiment. tendons are highlighted orange. (b)
The model-based controller and the RL agent performs similarly on an evaluation trajectory.
Experiment setup To assess the usefulness of gSDE, we apply it on a real system. The task is to control a tendon-driven elastic continuum neck [33] (see Fig. 4a) to given target pose. Con- trolling such soft robot is challenging because of the nonlinear tendon coupling, together with the deformation of the structure that needs to be modeled accurately. This modeling computationally expensive [34, 35] and requires assumptions that may not hold in the physical system.

---

The system is under-actuated actuated (there are only 4 tendons), hence, the desired pose is a 4D vector: angles for the rotation 0z. By 8₂ and one for the position æ. The input is 16D vector composed of: the measured tendon lengths (4D), the current tendon forces (4D), the current pose (4D) and the target pose (4D). The reward a weighted sum between the negative geodesic distance the desired orientation and the negative Euclidean distance to the desired position. The weights are chosen such that the two components have the same magnitude. oscillations in the final policy, The action space consists in desired delta in tendon forces, limited to N. For safety reasons, the tendon forces are clipped below 10 N and above 40 N. An episode terminates either when the agent reaches the desired pose or after timeout of s.. The episode is considered successful if the desired pose is reached within threshold of 10 mm for the position and 5 for the orientation. The agent controls the tendons forces at 30 Hz, while PD controller monitors the motor current at 3 kHz on the robot. Gradient updates are directly done on 4-core laptop, after each episode.
also add a small continuity cost to reduce the
Results We first ran the unstructured exploration on the robot, but had to stop the experiment early: high-frequency noise in the command was damaging the tendons and would have broken them due to their friction the bearings. Therefore, hand-crafted action smoothing (2 Hz cutoff Butterworth low-pass filter) for two hours.
baseline, trained a policy using SAC with
Then, we trained a controller using SAC with gSDE for the same duration. We compare both learned controllers to an existing model-based controller (passivity-basee
approach) presented in [34, 35] using pre-defined trajectory (ef. Fig. 4b). On the evaluation trajectory, the controllers are equally precise (cf. Table 2): the mean error orientation below 3 and the one in position below 3 mm. However, the policy trained with the low-pass filter much more jittery than the two others. We quantify this jitter as the mean absolute difference between two timesteps, denoted as continuity cost in Table
Unstructured
gSDE
Low-pass filter
Model-Based
noise
Position error (mm)
N/A
2.65 +/- 1.6
1,98 +/ 1.7
1.32 +/: 1.2
Orientation error (deg)
N/A
2.85 +/- 3.53 4/4.00
2.90 +/- 2.8
Continuity (deg)
N/A
0.20 +/- 0.04
0.38 0.07
0.16 1/16+/-0.00 0.04
Table 2: Comparison of the mean emror in position, orientation and continuity trajectory. We highlight best approaches when difference significant. The model-based and learned controllers yield comparable results but the policy trained with low-pass filter has much higher continuity cost.
the evaluation
Learning to walk with the quadruped robot bert
w/ 550E
Training Episodic Reward
28
26
211
1.0
1.5
2.0
2.5
3.5
Walltime (in hours)
(n) Quadruped and RC car robots
(b) Learning curve the walking task for the quadruped robot
Figure 5: Additional robots successfully trained using SAC with gSDE directly in the real world.
Additional Real Robot Experiments We also successfully applied SAC with gSDE on two ad- ditional real robotics tasks (see Fig. 5a): training an elastic quadruped robot to walk (learning curve in Fig. 5b) and learning to drive around track using RC car. Both experiments fully done
7

---

the real robot, without the use of simulation nor filter. We provide in the supplementary material the videos of the trained controllers.
5 Related Work
Exploration is a key topic in reinforcement learning [36]. It has been extensively studied in the discrete case and most recent papers still focus on discrete actions [37, 38).
Several works tackle the issues of unstructured exploration for continuous control by replacing it with correlated noise. Korenkevych et al. [15] use an autoregressive process and introduce two variables that allows to control the smoothness of the exploration. In the same vein, van Hoof al. [27] rely on a temporal coherence parameter interpolate between the step- or episode-based exploration, making use of a Markov chain to correlate the noise. This smoothed noise comes at cost: requires an history, which changes the problem definition.
Exploring in parameter space [10, 39, 11, 12, 40] is an orthogonal approach that also solves some issues of the unstructured exploration. It successfully applied to real robot but relied motor primitives [41, 12], which requires expert knowledge. Plappert et al. [19] adapt parameter explo- ration Deep RL by defining distance in the action space and applying layer normalization to handle high-dimensional space.
Population based algorithms, such Evolution strategies (ES) Genetic Algorithms (GA), also plore in parameter space. Thanks to massive parallelization, they were shown be competitive [42] with RL in terms of training time, at the cost of being sample inefficient. To address this problem, recent works [20] proposed to combine ES exploration with RL gradient update. This combination, although powerful, unfortunately adds numerous hyperparameters and non-negligible computa- tional overhead.
Obtaining smooth control is essential for real robot, but it is usually overlooked by the DeepRL community. Recently, Mysore et al. [14] integrated a continuity and smoothing loss inside RL algorithms. Their approach is effective to obtain a smooth controller that reduces the energy used at test time on the real robot. However, does not solve the issue of smooth exploration at train time, limiting their training to simulation only.
Conclusion
In this work, we highlighted several issues that arise from the unstructured exploration Deep RL algorithms for continuous control. Due to those issues, these algorithms cannot be directly applied to learning on real-world robots.
To address these issues, we adapt State-Dependent Exploration to Deep RL algorithms by extending noise every n steps and replace the exploration function put by learned features. This generalized version (gSDH), provides simple and efficient alternative to unstructured Gaussian exploration.
the original formulation: we sample the
gSD achieves very competitive results on several continuous control benchmarks, while reducing wear-and-tear train time. We also investigate the contribution of each modification by performing an ablation study: the noise sampling interval has the most impact and allows a compromise between performance and smoothness. Our proposed exploration strategy. combined with SAC, is robust to hyperparameter choice, which makes it suitable for robotics applications. To demonstrate it, we successfully apply SAC with gSDE directly on three different robots.
Although much progress is being made in sim2real approaches, we believe more effort should be invested in learning directly real systems, duration of learning. This paper meant as a step towards this goal, and we hope that it will revive interest in developing exploration methods that can be directly applied to real robots.
if this poses challenges in terms of safety and
Acknowledgments
The work described in this paper was partially funded by the project "Reduced Complexity Models" from the "Helmmoliz-Geemeinschhl Deutscher Forschungszentren" and by the EU H2020 project

---

"VERtical Innovation in the Domain of Robotics Enabled by Artificial intelligence Methods". We would like to thank all the people that gave us valuable feedback on an early version of this paper.
References
[1] T. Rückstieß, M. Felder, and J. Schmidhuber. State-dependent exploration for policy gradient methods. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 234-249 Springer, 2008.
[2] N. J. Nilsson. Shakey the robot. Technical Report 323, Artificial Intelligence Center, SRI International, Menlo Park, CA, USA, 1984. URL http://wttr w.w.waiiii.. ai..ri. com/ commshakey/...
[3] Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel. Benchmarking deep reinforce- ment learning for continuous control. In International Conference on Machine Learning, pages 1339-1338, 2016.
[4] M. Andrychowicz, B. Baker, Chociej, Jozefowicz, McGrew, Pachocki, Petron, M. Plappert, G. Powell, A. Ray, et al. Learning dexterous in-hand manipulation. arXiv preprint arXiv:1808.0010 00177, 2018.
[5] S. Fujimoto, H. van Hoof, and D. Meger. Addressing function approximation error in actor- critic methods. arXiv preprint 1802.08800.9944 2018.
[6] X. B. Peng, P. Abbeel, S. Levine, and M. van de Panne. Deepmimic: Example-guided deep re- inforcement learning of physics-based character skills. ACM Transactions on Graphics (TOG), 37(4):143, 2018.
[7] Hwangho, J. Lee, A. Dosovitskiy, Bellicoso, V. Tsounis, V. Koltun, and Hutter. Learn- ing agile and dynamic motor skills for legged robots. arXiv preprint arXiv:/901 1901.08652, 2019.
[8] Haarnoja, A. Zhou, Hartikainen, G. Tucker, Ha, J. Tan, V. Kumar, H. Zhu. A. Gupta, P. Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint 2018.
[9] A. Kendall, J. Hawke, D. Janz, P. Mazur, D. Reda, J.-M. Allen, V.-D. Lam, A. Bewley, and A. Shah. Learning to drive in day. In 2019 International Conference on Robotics and Automation (ICRA), pages 8248 8254. IEEE, 2019.
[10] Kober and J. Peters. Policy search for motor primitives in robotics. In Advances in neural information processing systems, pages 849-856, 2009.
[11] T. Rückstiess, F. Sehnke, T. Schaul, D. Wierstra, Y. Sun, and J. Schmidhuber. Exploring parameter space in reinforcement learning. Paladyn, Journal of Behavioral Robotics, 1(1): 14-24, 2010.
[12] Stulp and O. Sigaud. Robot skill learning: From reinforcement learning to evolution strate- gies. Paladyn, Journal of Behavioral Robotics, 4(1):49-61, 2013.
[13] M. P. Deisenroth, G. Neumann, J. Peters, et al. A survey on policy search for robotics. Foun- dations and Trends in Robotics, 2(1 11122):1-142, 2013.
[14] S. Mysore, B. Mabsout, R. Mancuso, and K. Saenko. Regularizing action policies for smooth control with reinforcement learning. 2021.
[15] D. Korenkevych. A. R. Mahmood, G. Vasan, and J. Bergstra. Autoregressive policies for continuous control deep reinforcement learning. arXiv preprint arXiv:1903.11524 2019.
[16] T. Haarnoja, S. Ha, A. Zhou, J. Tan, G. Tucker, and S. Levine. Learning to walk via deep 2018.
reinforcement learning. arXiv preprint
[17] Ha, Xu, Z. Tan, S. Levine, and J. Tan. Learning to walk in the real world with minimal human effort, arXiv preprint 0855000002.88550, 02 2020.

---

[18] M. Neunert, A. Abdolmaleki, M. Wulfmeier, T. Lampe, J. T. Springenberg, R. Hafner, F. Ro- mano, J. Buchli, N. Heess, and M. Riedmiller. Continuous-diserete reinforcement learning for hybrid control in robotics. arXiv preprint arXiv:2001.0044) 2020.
[19] M. Plappert, R. Houthooft, P. Dhariwal, S. Sidor, R. Y. Chen, X. Chen, T. Asfour, Abbeel, and M. Andrychowiez. Parameter space noise for exploration. arXiv preprint arXiv: (0fGv.700.000 1706.009000000 2017.
[20] A. Pourchot and O. Sigaud. Cem-rl: Combining evolutionary and gradient-based methods for policy search. arXiv preprint arXiv:1810.01222 2018.
[21] J. Schulman, S. Levine, Abbeel, M. Jordan, and Moritz. Trust region policy optimization.
In International conference on machine learning, pages 1889-1897, 2015.
[22] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint /rXiv: 309.02971, 2015.
[23] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928-1937, 2016.
[24] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1 1eeiii10770.3330 2017.
[25] Haarnoja, H. Tang, P. Abbeel, and S. Levine. Reinforcement learning with deep energy- based policies. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 1352-1361. JMLR. org, 2017.
[26] R. Williams. Simple statistical gradient-followingg algorithms for connectionist reinforce- ment learning. Machine learning, 8(3-4):229-23) 229-256, 1992.
[27] H. van Hoof, D. Tanneberg, and J. Peters. Generalized exploration in policy search. Machine Learning, 10609-10):705-1124, oct 2017. Special Issue of the ECML PKDD 2017 Journal Track.
[28] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. http://pybbull org, 2016-2019.
[29] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint
04400, 2016.
[30] G. E. Uhlenbeck and L. S. Omstten. On the theory of the brownian motion. Physical review, 36(5):823, 1930.
[31] A. Raffin, A. Hill, M. Ernestus, A. Gleave, A. Kanervisto, and N. Dormann. Stable baselines3. https://g/
2019.
[32] A. Raffin. RI baselines3 zoo. httts::/ggttub..com wwwp:hhuuu....ooRotteteoett www.///aaaaelaeeeeeaeaaeaaeeaaae www.aaaeeeeneeeeeeaoae 2020.
[33] Reinecke, B. Deutschmann, and D. Fehrenbach. structurally flexible humanoid spine hased on a tendon-driven elastic continuum. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 4714-4721 IEEE, 2016.
[34] B. Deutschmann, A. Dietrich, and C. Ott. Position control of an underactuated continuum mechanism using reduced nonlinear model. In 2017 IEEE 56th Anmual Conference on Deci- sion and Control (CDC), pages 5223-5230 IEEE, 2017.
[35] B. Deutschmann, M. Chalon, Reinecke, M. Maier, and C. Ott. Six-dof pose estimation for tendon-driven continuum mechanism without a deformation model. IEEE Robotics and Automation Letters, 4(4):3425-3432, 2019.
[36] R. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[37] Osband, C. Blundell, Pritzel, and B. Van Roy. Deep exploration via bootstrapped dqn, In Advances in neural information processing systems, pages 4026-4034, 2016.
10

---

[38] I. Osband. J. Aslanides, and A. Cassirer. Randomized prior functions for deep reinforcement learning. In Advances in Neural Information Processing Systems, pages 8617-8629, 2018.
[39] Sehnke, C. Osendorfer, T. Rückstieß, A. Graves, J. Peters, and J. Schmidhuber, Parameter- exploring policy gradients. Neural Networks, 23(4):551-559, 2010.
[40] O. Sigaud and F. Stulp. Policy search in continuous action domains: overview. Neural Networks, 2019.
[41] Peters and S. Schaal. Reinforcement learning of motor skills with policy gradients. Neural networks, 21(4):682-697, 2008.
[42] Such, V. Madhavan, E. Conti, Lehman, K. Stanley, and J. Clune. Deep neuroevolution: Genetic algorithms are competitive alternative for training deep neural networks for rein- forcement learning. arXiv preprint arXiv: 12 2017.


Reviews:
[{"review_id": "tHgitJxC30u", "reviewer": "Reviewer_bar8", "summary": "The paper investigates how to achieve smooth exploration during deep reinforcement learning (deep RL), which is particularly an important problem when we want to apply deep RL to real-world environments. Basically, the author wants to improve unstructured exploration that shows noisy exploration at the early stage (Figure 1). The key idea is to improve the idea of State-Dependent Exploration (SDE) which defines action noises as a function of the state and an additional episodic parameter theta. The authors identified four disadvantages of SDE that limit exploration, and propose to 1) sample the episodic parameters every n steps instead of every n episodes and 2) use other feature values to generate noises. \n\nThe authors demonstrated that this approach can mitigate the known trade-off issue between safe exploration and learning performance. The authors also have shown that the proposed method can learn motor skills in three different real-world robotic environments, including a tendon-driven robot, a quadruped, and an RC car.\n", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The paper discusses a very important problem to deploy deep RL on real-world environments.\n+ The paper demonstrates a good intellectual discussion about SDE, including its strengths and weaknesses.\n+ The authors tested many interesting scenarios, including standard benchmark problems and real-robot learning.\n- The design of gSDE sounds a little bit ad hoc.\n- Some experiments are not complete: please refer to the text below.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "In summary,my support is lukewarm. I like the problem itself and discussion about SDE, but gSDE seems to be a minor extension of SDE without much technical novelty.\n\nFirst of all, I am very interested in real-robot learning, and this paper discusses one of the most important problems, how can we achieve safe exploration at the early stage of the learning process. And the paper describes pros and cons of SDE, which seem reasonable. I agree that SDE is based on a few design choices that can limit its exploration capability.\n\nOn the other hand, I’m not sure gSDE provides good solutions to it. The first idea is to resample the episodic randomization parameter theta in the middle of episodes, per two to 64 steps. I agree that they can be a good strategy for periodic locomotion tasks that are compared in Table I, but it cannot be a universal solution for general MDPs. I guess that this simple trick may not be very effective for other non-periodic (or episodic) tasks, such as manipulation. The example of a tendon-driven robot also seems a bit in favor of gSDE because the target trajectories have been periodically changed. I believe this can be a major drawback of the proposed approach.\n\nThe second innovation is to use the policy features, which are defined as the last layer values before the deterministic outputs. Yes, they may work better than the baselines. But I am a little bit concerned that the main reason for choosing this feature is simply because “we can in fact use any features”. I do not see any scientific conjectures here. What other features can we use? What if they do not work? \n\nThis lack of scientific justification requires the authors to provide more through experiments. Please see the below comments for more details.\n- In general, I believe a much wider set of experiments is needed for supporting the paper’s statements.\n- Figure 2 does not include SDE experiments. Is this intended? Not sure gSDE-Episodic is identical to SDE.\n- I’m not sure Table 2 suggests that gSDE is the best. Low-pass filter achieves lower positional errors. Continuity cost does not say anything, if it is acceptable (= it is possible to learn a policy on real robots).\n- Experiments on a quadrupedal robot and an RC car do not seem informative due to lack of statistics.\n", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "First, I would suggest the authors discuss more scientific justification of gSDE.\nI also suggest the authors conduct more extended experiments.\n", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {}}, {"review_id": "I1dM-F2PGB", "reviewer": "Reviewer_8waC", "summary": "This work addresses the ubiquitous problem of shaky controllers trained using model free deep RL by proposing the exploration technique, generalized State Dependent Exploration (gSDE). gSDE builds on prior work SDE and substitutes the per-step random Gaussian action noise in many deep RL algorithms with state-dependent noise that only varies every n-steps. The effect is smoother and more consistent exploration trajectories which reduce wear and tear without substantially compromising performance. gSDE is assessed in simulation as well as on three different physical robotic systems.  \n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n* Effectively addresses a ubiquitous problem in Deep RL for robotics - shaky controllers.\n* The proposed method, gSDE, is widely applicable. It can be applied to any continuous control problem that is solved using either a pure policy gradient method (e.g. REINFORCE) or a hybrid policy and value based method (e.g. PPO, A2C, TD3, SAC).\n    1. The only approaches for which this does not appear applicable are value-based methods such as DQN, Rainbow, or QT-OPT\n* It is very simple to implement.\n* The effectiveness of gSDE compared to other exploration noise structures is explored through rigorous and extensive experiments in simulation using a range of relevant algorithms (SAC, TD3, PPO, A2C) on multiple environments.\n    1. The ablation studies (e.g. feature selection, noise re-sampling frequency) are relevant and illuminating.\n    2. Figures 2 and 3 are especially interesting.\n    3. All simulation results are reported with the mean and std error over 10 runs.\n* The approach is further validated with the training of successful policies on three very different physical robots.\n* The clarity of presentation and technical quality are excellent throughout.\n    1. The explanations provided in both the main paper and appendix strike a good balance between detail and concision and I am confident that the results in the paper would be reproducible based on the information provided.\n\n**Weaknesses**\n\n* Low originality: gSDE is a minor modification to an existing method, SDE\n* Lines 100 - 126: The claim that gSDE addresses all four of the identified limitations with SDE is not fully convincing\n    1. (ii) Now the variance of the policy depends on the policy features dimension so needs to be tuned for each network.\n    2. (iv) For similar reasons to those given in (iv) (lines 108 -109) shouldn’t the policy features now be normalized?\n* Figure 2: Given the apparent linear relationship between the frequency of noise parameter re-sampling and continuity cost, the sudden drop-off to the Unstructured noise is surprising. It is achieving returns equivalent to gSDE-64 but with almost 2x normalized cost\n    1. There is a similar puzzling relationship occurring in Figure 3b)\n    2. To better understand this it would be helpful to add gSDE-4 and gSDE-2 to Figure 2 and gSDE 2 to Figure 3b). \n    3. I would expect a somewhat smooth transition between gSDE-2 and unstructured noise. Regardless, adding these points would be illuminating.\n* Grammar\n    1. Line 13: “permits to have” → facilitates / enables?\n    2. Line 54: “without the need of a” -> “without need of a” / “without the need for”\n    3. Line 94: “allows to adapt” → “enables adaptation of”\n    4. Line 184: “allows to have” → “permits a” / “makes it possible to”\n\n**Comments and Questions**\n* Lines 42 - 44: There is significant interest in and efforts made to apply deep RL to real robotic problems so it is not clear to me that a focus on simulation is the main reason for SDE not being combined with deep RL. Perhaps up to now other issues such as sample efficiency and generalizing to less structured environments or multiple tasks have occupied the attention of the community.\n* Line 66: When actions are discrete exploration is also typically done in the action space.\n* Table 2: Is there just one evaluation trajectory?\n* It would be interesting to see the learning curves for the tendon-driven task\n* Walking task: It would be interesting to compare the learning curve (Figure 5b) with SAC trained with a low pass filter on this task\n* Driving task: It seems that the standard SAC exploration would not break this type of robot. It would be very interesting to see a comparison SAC, SAC w/ low pass filter, SAC + gSDE\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This work is highly relevant to robotic learning using model free deep RL and the effectiveness of gSDE is convincingly demonstrated. Additionally the method is very simple and easy to implement. Experiments are well selected and thorough. The paper is lucid, easy to understand, and overall a pleasure to read. To further strengthen the paper I raised a few minor issues and questions.\n\nComment on impact 4:\nThe paper convincingly demonstrates the effectiveness of existing ideas on reducing shakiness of deep RL policies. The simplicity of the idea and ease of implementation increase the potential impact of this work.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "Please address the following comments and questions:\n* Lines 100 - 126: The claim that gSDE addresses all four of the identified limitations with SDE is not fully convincing\n    1. (ii) Now the variance of the policy depends on the policy features dimension so needs to be tuned for each network.\n    2. (iv) For similar reasons to those given in (iv) (lines 108 -109) shouldn’t the policy features now be normalized?\n* Figure 2: Given the apparent linear relationship between the frequency of noise parameter re-sampling and continuity cost, the sudden drop-off to the Unstructured noise is surprising. It is achieving returns equivalent to gSDE-64 but with almost 2x normalized cost\n    1. There is a similar puzzling relationship occurring in Figure 3b)\n    2. To better understand this it would be helpful to add gSDE-4 and gSDE-2 to Figure 2 and gSDE 2 to Figure 3b). \n    3. I would expect a somewhat smooth transition between gSDE-2 and unstructured noise. Regardless, adding these points would be illuminating.\n* Table 2: Is there just one evaluation trajectory?\n* Grammar\n    1. Line 13: “permits to have” → facilitates / enables?\n    2. Line 54: “without the need of a” -> “without need of a” / “without the need for”\n    3. Line 94: “allows to adapt” → “enables adaptation of”\n    4. Line 184: “allows to have” → “permits a” / “makes it possible to”"}}, {"review_id": "BHRywTjPH9w", "reviewer": "Reviewer_uHC8", "summary": "This paper proposes a new method (gSDE) for generating exploration noise for RL, conditioned on the last hidden layer of the policy network and re-generated at a frequency tunable between every timestep and every episode. This is a modification of state-dependent exploration (SDE), with the differences being 1) SDE conditions on the environment state instead of the network, and 2) SDE always re-generates once per episode. gSDE is evaluated on 4 MuJoCo tasks in simulation and 3 real-world systems and found to generate smoother action sequences than standard unstructured Gaussian noise with similar returns. The update frequency can be adjusted to modulate the trade-off between smoothness and performance.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The problem is very relevant. We now have RL algorithms that are data-efficient enough to allow real-world training, which means hardware damage from noisy actions has become a blocker.\n* The MuJoCo tasks, head actuation, and quadruped locomotion results are convincing.\n\nWeaknesses:\n* The changes to SDE are very slight.\n* In the video of head actuation, the comparison between exploration with per-step noise and with gSDE is useful. It would be nice to have the same comparison for the quadruped.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The work is a relatively simple extension of SDE without a huge novelty value. However, the authors correctly point out that SDE is *not* used in RL, and gSDE solves the problems that make SDE less effective. I believe it is worthwhile to demonstrate that this simple method can make real-world training feasible for systems where it otherwise wouldn't be.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "* Please clarify the reasons for using policy features as exploration function input. I understand that it eliminates the need for normalization if the features are already normalized. I do not understand why SDE cannot use pixel input--wouldn't it simply require a random sample (or 3) per pixel? Is the concern the quantity of samples and multiplications?"}}]

Author rebuttal:

Meta-review: This paper proposes a simple and easy-to-implement idea for generating exploration noise for RL. This approach is especially making RL more effective when used on real-world robotic systems, i.e. it addresses the shaky behavior problem in practice. Though the proposed idea is just minor modifications to a existing method, i.e. SDE, and lacks of scientific justification or has very little technical novelty, the reviewers believed it can have great potential applicability, e.g. any continuous control problem. Though it is evaluated on a good set of tasks: on 4 MuJoCo tasks in simulation and 3 real-world systems. However it's encouraged to provide more results to support the paper’s statements, e.g. additional statistical information of experiments on a quadrupedal robot and an RC car.

The authors have made a great response that have addressed most concerns. Additional experiment results also support the paper's claims stronger. The proposed idea can be simple but it's very practically useful and worth being shared with the robotic community.
Decision: Accept (Poster)