Action Noise in Off-Policy Deep Reinforcement Learning: Impact on Exploration and Performance
Authors: Jakob Hollenstein
Sayantan Auddy
Matteo Saveriano
Erwan Renaudo
Justus Piater
Venue: Accepted by TMLR ()

Abstract: Many Deep Reinforcement Learning (D-RL) algorithms rely on simple forms of exploration
such as the additive action noise often used in continuous control domains. Typically,
the scaling factor of this action noise is chosen as a hyper-parameter and is kept constant
during training. In this paper, we focus on action noise in off-policy deep reinforcement
learning for continuous control. We analyze how the learned policy is impacted by the noise
type, noise scale, and impact scaling factor reduction schedule. We consider the two most
prominent types of action noise, Gaussian and Ornstein-Uhlenbeck noise, and perform a vast
experimental campaign by systematically varying the noise type and scale parameter, and
by measuring variables of interest like the expected return of the policy and the state-space
coverage during exploration. For the latter, we propose a novel state-space coverage measure
$\operatorname{X}_{\mathcal{U}\text{rel}}$ that is more robust to estimation artifacts caused by points close to the
state-space boundary than previously-proposed measures. Larger
noise scales generally increase state-space coverage. However, we found that increasing the
space coverage using a larger noise scale is often not beneficial. On the contrary, reducing
the noise scale over the training process reduces the variance and generally improves the
learning performance. We conclude that the best noise type and scale are environment
dependent, and based on our observations derive heuristic rules for guiding the choice of the
action noise as a starting point for further optimization.


Published in Transactions on Machine Learning Research (11/2022)
Action Noise in Off-Policy Deep Reinforcement Learning: Impact on Exploration and Performance
Jakob Hollenstein Deportment of Computer Science, University of Innsbruck
hakkohhhlllensteei@@bbb..co
Sayantan Auddy
aayantan.onuddyyyaiaa
Department of Computer Science, University of Trnnbruck
Matteo Saveriano
matteo. wwweriavveeaaaaao........ @reeritnno@mmtt.....
Department of Industrial Engineering, University of Trento
Erwan Renaudo
eruen.renoxdo@uibb.cc
Department of Computer Science, University of Innsbruck
Justus Piater
mnstus.piaterruibk..c ac.
Department of Computer Science, University of Innsbruck
Reviewed on OpenReview: https: openreviee. net/ hor/mform? 18-MNNBB266
Abstract
Many Deep Reinforcement Learning (D-RL) algorithms rely on simple forms of exploration such the additive action noise often used in continuous control domains. Typically, the scaling factor of this action noise is chosen a hyper-parameter and is kept constant during training. In this paper, we focus on action noise in off-policy deep reinforcement learning for continuous control. We analyze how the learned policy is impacted by the noise type, noise scale, and impact scaling factor reduction schedule. We consider the two most prominent types of action noise, Gaussian and Urhsteen-Uhhennecck noise, and perform a vast experimental campaign by systematically varying the noise type and scale parameter, and by measuring variables of interest like the expected return of the policy and the state- space coverage during exploration. For the latter, propose novel state-space coverage measure Xurel that is more robust to estimation artifacts caused by points close to the state- space boundary than previously-proposed measures. Larger noise scales generally increase state-space coverage. However, we found that increasing the space coverage using a larger noise scale is often not beneficial. On the contrary, reducing the noise scale over the training process reduces the variance and generally improves the learning performance. We conclude that the best noise type and scale environment dependent, and based on our observations derive heuristic rules for guiding the choice of the action noise a starting point for further optimization. https:// // thub. commjkbjh/ee cttp//coeoooononnooo action noise in off ceff-polcy_d-x
1 Introduction
In (deep) reinforcement learning agent aims to learn policy to act optimally based on data it collects by interacting with the environment. In order to learn a well performing policy, data (i.e. slale-action--ewward sequences) of sufficiently good behavior need to collected. A simple and very common method to discover better data induce variation in the data collection by adding noise the action selection process. Through this variation, the agent will try wide range of action sequences and eventually discover useful information.

---

Published in Transactions on Machine Learning Research (11/2022)
A motivating example: Mountain-Car
DDPG
SAC
TD3
noise Type
Ornstein- Uhlenbeck
100
100
100
Gaussian
Return
Scale
0.6
0.5
0
Return
-30.2
-30.4
0.1
1.3
0
50000
50000
0
50000
Training Steps
Training Steps
Training Steps
Gaussian
Orustein- Orasteia-Uhlenbeet
Table Untrained random policies, Gaussian (ar 0.6) and Ornstein- Orustein-Uhlenbeck (o 0.5) achieve similar returns and appear interchange- able.
Figure 1: Training with the action noises (Table 1) shows the impact of noise type; Ornstein-Uhlenbecc solves the task, but Gaussian does not. Other algorithm parameters are taken from the tuned param- eters found by Rafinn (2020). The lines indicate the medians, the shaded areas the quartiles of ten independent runs.
Action Noise In off-policy reinforcement learning algorithms applied to continuous control domains, a go-to approach is to add a randomly-sampled action noise to the action chosen by the policy. Typically the action noise is sampled from a Gaussian distribution or an Ornstein- Uhlenbeck process, either because algorithms are proposed using these noise types (Fujimoto ct al., 2018; Lillicrap al., 2016), or bocause these two types are provided by reinforcement learning implementations (Liang et. al., 2018; Raffin et al., 2021a; Fujila et al., 2021; Seno & Imai, 2021). While adding action noise is simple, widely used, and surprisingly effective, the impact of action noise type or scale does not feature very prominently in the reinforcement learning literature. However, the action noise can have a huge impact on the learning performance as the following example shows.
A motivating example: Consider the case of the Mountain-Car Car (Brockman et al., 2016; Moore, 1990) environment. In this task, a car starts in a valley between mountains on the left and right and does not have sufficient power to simply drive up the mountain. Tt needs repetitive swings to increase its potential and kinetic energy to finally make it up to the top of the mountain the right side. The actions apply force to the car and incur cost that is quadratic to the amount of force, while reaching the goal yields a final reward of 100. This parameterization implies a local optimum: not performing any action and achieving a return of zero.
Driving the environment with purely random policies based on the two noise types (Gaussian, o 0.6, Orlstein-UUllebeeck 0.5, see Table 1), yields similar returns. However, when we apply the algorithms DDPG, TD3 and SAC (Lillierap et al., 2016; Fujimoto et al., 2018; Haarnoja al., 2019) to this task, the resulting learning curves (Figure 1) very clearly depict the huge impact the noise configuration has. While returns of the purely random noise-only policies were similar, we achieve substantially different learning results. Learning either fails (Gaussian) or leads to success (Ornstein-Ullenbeel;; This shows the huge importance of the action noise configuration. See Section A for further details.
Exploration Schedule very common strategy Q-learning algorithms applied to discrete control is select random action with certain probability E. In this posilon-greedl strategy, the probability e is often chosen higher in the beginning of the training process and reduced to smaller value over course of the training. Although very common in Q-learning, comparable strategy has not received lot of attention for action noise in continuous control. The descriptions of the most prominent algorithms using action noise, namely DDPG (Lillicrap et. al., 2016) and TD3 (Fujimolo et. al., 2018), do not mention changing the noise over the training process. Another prominent algorithm, SAC (Haarnoja et al., 2019), adapts the noise to an entropy target. The entropy target, however, is kept constant over the training process. In many cases the optimal policy would be deterministic, but the agent has to behave with similar average action-entropy matter whether the optimal policy has been found or not.
2

---

Published in Transactions on Machine Learning Research (11/2022)
An indication that reducing the randomness over the training process has received little attention is that only very few reinforcement learning implementations, e-g-; RLlib (Liang et al., 2018), implement reducing the impact of action noise over the training progress. Some libraries, like coach (Caspi et al., 2017), only implement a form of contimuous epsilon greedy: sampling the action noise from a uniform distribution with probability 8. The majority of available implementations, including stable-baselines (Raffin al., 2021a), PFRL (Fujita al., 2021), acme (Hoffman al., 2020), and d3rlpy (Seno & Imai, 2021), do not implement any strategies to reduce the impact of action noise over the training progress.
Exploration schedules for action noise are also not mentioned in several recent surveys (Yang et al., 2022; Ladosz et al., 2022; Amin et al., 2021)
Contributions
In this paper we analyze the impact of Gaussian and Censtein-Uhlenbeck noise on the learning process of DDPG, TD3, SAC and a deterministic SAC variant. Evaluation is performed on multiple popular environments (Table E.1): Mountain-Car (Brockman al., 2016) environment from the OpenAl Gym, Inverted-Pendulume Pendulum-Swingupp Reacher, Hopper, Walker2D and Half-Cheetah environments implemented us- ing PyBullct (Coumans & Bai, 2016-2021; Ellenberger, 2018).
We investigate the relation between exploratory state-space coverage X, returns collected by the exploratory policy and learned policy performance P.
We propose to assess the state-space coverage using our novel measure Xurel that is more robust to approximation artifacts on bounded spaces compared to previously proposed measures.
We perform vast experimental study and investigate the question whether of the two noise types is generally preferable (Q1), whether specific scale should be used (Q2), whether there is any benefit to reducing the scale over the training progress (lincarly, logistically) compared to keeping it constant (Q3), and which of the parameters noise type, noise scale and scheduler is most important (Q4).
We provide set of heuristics derived from our results to guide the selection of initial action noise configurations.
Findings We found that the noise coufiguration, noise type and noise scale, have an important impact and can be necessary for learning (e.g. Mountain-Cast Car) or can break learning (e.g. Hopper). Larger noise scales tend to increase state-space coverage, but for the majority of our investigated environments increasing the state-space coverage is not beneficial: increased state-space coverage was associated with reduction in performance. This indicates that in these environments, local exploration, which is associated with smaller state-space coverage, tends to be favorable. We recommend to select and tune action noise based on the reward and dynamics structure on a per-eneironment basis.
We found that across noise configurations, decaying the impact of action noise tends to work beiter than keeping the impact constant, in both reducing the variance across seeds and improving the learned policy performance and can thus make the algorithms more robust to the action noise hyper-parameters scale and type. We recommend to reduce the action noise scaling factor over the training time.
We found that for all environments investigated in this study noise scale is the most important parameter, and some environments (e.g. Mountain-Caa Caatt benefit from larger noise scales, while other environments require very small scales (e.g. Walker2D). We recommend to assess an environment's 's action noise scale preference first.
2 Related Work
By combining Deep Learning with Reinforcement Learning their DQN method, Mnih et al. (2015) achieved substantial improvements on the Atari Games RL benchmarks (Bellemare et al., 2013) and sparked lasting interest in Deep Reinforcement learning (D-RT)..
3

---

Published in Transactions on Machine Learning Research (11/2022)
Robotic environments: In robotics, the interest in Deep Reinforcement Learning has also been rising and common benchmarks are provided by OpenAT Gym (Brockman et al., 2016), which includes control classics such as the Mountain-Car environment (Moore, 1990) as well as more complicated (robotics) tasks based on the Mujoco simulator (Todorov et al., 2012). Another common benchmark is the DM Control Suite (Tassa et al., 2018); also based on Mujoco. While Mujoco has seen widespread adoption it was, until recently, not freely available. A second popular simulation engine, that has been freely available, is the Bullet simulation engine (Coumans & Bai, 2016 2021) and very similar benchmark environments are also available for the Bullet engine (Coumans & Bai, 2016-2021; Ellenberger, 2018).
While the Atari games feature large and (approximately) continuous observation spaces, their action spaces are discrete and relatively small, making Q-learning a viable option. In contrast, typical robotics tasks require continuous action spaces, implying uncountably many different actions.
Continuous Control:
A tabular Q-lcarning approach or a discrete Q-uunction output for each action are therefore not possible and maximizing the action over a learned function approximator for ((s,a) is computationally expensive (although not impossible Kalashnikov et al. (2018) have shown). Therefore, in continuous action spaces, policy seareh is employed, to directly optimize function approximator policy, mapping from state to best performing action (Williams, 1992). To still reap the benefits of redueed sample complexity of ID-methods, policy search is often combined with learning a value function, a critic, leading to an actor-crilic approach (Sutton et al., 1999).
On- and Off-policy: Current state of the art D-RL algorithms consist of on-policy methods, such as TRPO (Schulman et al., 2015) or PPO (Schulman et al., 2017), and off-policy methods, such as DDPG (Lil- licrap et al., 2016), TD3 (Fujimoto et al., 2018) and SAC (Haarnoja et al., 2019). While the on-policy methods optimize the next ileration of the policy with respect to the data collected by the current iteration, off-po methods are, apart from stability issues and requirements on the samples, able to improve policy performance based on data collected by any arbitrary policy and thus can also re-use older samples.
To improve the policy, variation (eaploration) in the collected data is necessary. The most comminn form of exploration is based on randomness: in on-policy methods this comes from a stochastic policy (TRPO, PPO), while in the off-policy case it is possible policy (Silver et al., 2014) with added action noise (DDPG, TD3). Since off- off-policy algorithms can learn from data collected by other policies, it is also possible to combine stochastic policies (e.g. SAC) with action noise.
stochastic policy (SAC) or, to use deterministic
State-Space Coverage: Often, the reward is associated with reaching certain areas in the state-space. Thus, in many cases, explovation is related to state-space coverage. An intuitive method to calculate state space coverage is based on binning the state-space and counting the percentage of non-empty bins. Since this requires exponentially many points the dimensionality increases, other mcasures necessary. Zhan et al. (2019) propose to measure state coverage by drawing a bounding box around the collected data and mcasuring the mcans of the side-lengths, or by mcasuring the sum of the cigenvalues of the estimated covariance matrix of the collected data. However, so far, there is no common and widely adopted approach.
The architecture for the stochastic policy in SAC (Haarnoja et al., 2019) consists of neural network parameterizing a Gaussian distribution, which is used to sample actions and estimate action-likelihoods. A similar stochastie policy architecture is also used in TRPO (Schulman et al., 2015) and PPO (Schulman et al., 2017). While this is the most commonly used type of distribution, more complicated parameterized stochastic policy distributions based normalizing flows have been proposed (Mazoure et al., 2020: Ward et al., 2019). In the case of action noise, the noise processes are not limited to uncorrelated Gaussian (e.g. TD3) and temporally correlated Ornstein-Uhlenbeen of action noise types is available under the name of colored noise, which has been successfully used to improve the Cross Entropy- (Pinneri et al., 2020). A quite different type of random exploration are the parameter space exploration methods (Mania et al., 2018; Plappert et al., 2018), where noise is not applied to the resulting action, but instead, the parameters of the policy are varied. As a somewhat intermediate method, state dependent exploration (Raffin et al., 2021b) has been proposed, where action noise is deterministically generated by a function based on the state. Here, the function parameters are changed randomly for each episode, leading to different deterministic "action noise" for each episode. Presumably
Methods of Exploration:
noise (e.g. DDPG): a whole family
4

---

Published in Transactions on Machine Learning Research (11/2022)
among the most intricate methods to generate exploration are the methods that train a policy to achieve exploratory behavior by rewarding exploratory actions (Burda et al., 2019; Tang et al., 2017; Mutti et al., 2020; Hong et al., 2018; Pong et al., 2020). Another alternative can be a two-step approach, where in the first stage cetiinsicaall-moivvaatt the information in the bulfer exploited to learn a policy (Colas et al., 2018).
exploration is used to populate the replay buffer, and in the second stage
It is however, not clear yet, which exploration method is most beneficial, and when a more complicated method is actually worth the additional computational cost and complexity. In this work we aim to reduce this gap, by investigating the most widely used baseline method in more detail: exploration by action noise.
Studies of Random Exploration Exploration in Deep Reinforcement Learning is also the subject of multiple surveys. However, the topic of action noise is only covered very sparsely. Yang et al. (2022) only briefly mention action noise as being used in DDPG (Lillicrap et al., 2016) and TD3 (Fujimoto et al., 2018) but do not provide further discussion. A section on randomness-based methods that focuses mostly on discrete action spaces, for continuous action spaces on parameter noise, is provided by Ladosz et al. (2022). Amin et al. (2021) provide section on randomized action selection in policy search, nicely divided into action-space and parameter-space methods, and discuss temporally correlated or uncorrelated perturbations. However, they also do not point to any empirical study specifically comparing the effects of random exploration.
Generally, however, it appears that most work focuses on proposing modifications of the action noise, rather than investigating the effects of the bascline parameters. For example, for stochastic policics, Rao et al. (2020) show that the weight initialisation procedure can lead to different initial action distributions of the stochastic policies. Chou et al. (2017) propose stochastic policies based the 3-distribution. Nobakht & Liu (2022) use gathered experience to tune the action noise model.
In contrast to these works, in our previous work (Hollenstein et al., 2021), we investigated action noise as the immediate means to control the environments, i.c. adding action noise to a constant-zero policy. Not surprisingly, we found that there are dependencies between environment dynamics, reward structure and action noise. However, this study did not investigate the infiuence of action noise on learning progress or results. In this work, we investigate the impact of action noise in the context of learning.
3 Methods
In this section, we describe the action noise types, the schedulers to reduce the scaling factor of the action noise over time and the evaluation process in more detail. We briefly list the analyzed benchmark cnviron- ments and their most important properties. We chose environments of increasing complexity that model widely used benchmark tasks. We list the used algorithms and then describe how we gather evaluation data and how it is aggregated. Last, we describe the methods we use for analyzing state-space coverage.
3.1 Noise types: Gaussian and Ornstein-Uhlenbedd
The action noise ear is added to the action drawn from the policy:
a: - Gaiin Araa
āt + clip(eadl amax Gmin
+ amax
amin
(1)
2
2
where āj ~ TT(-Ss)) for stochastic policies or ā, = T0(s)) for deterministie policies. We introduce an additional impact scaling factor B, which is typically kept constant al. the value one. In Section 3.2 we describe how we change / over time to create noise scheduler. The action noise Ea, is drawn from either a Gaussian distribution or an Ubheebbck (OU) process. The noisc distributions are factorized, i.c. noise samples are drawn independently for each action dimension. For the generation of action noise samples, the action space is assumed to be limited to [-1,1] but then rescaled to the actual limits defined by the environment.
Gaussian noise is temporally uncorrelated and is typically applied on symmetric action spaces (Hill al., 2018; Raffin, 2020) with commonly used values of μ = 0 and = 0.1 with Σ = I.g. Action noise is sampled
5

---

Published in Transactions on Machine Learning Research (11/2022)
according to
Sa, ~ (N, Σ)
(2)
In this setup, Gaussian action noise is sampled and clipped to the action limits as needed.
Ornstein- CeattennUUhlebbekk noise is sampled from the following temporal process, with each action dimension calculated independently of the other dimensions:
Eas = Ear-1 + +04 - Ea..)) )- dt + ovdt c
(3)
Eao
4 ~
(4)
The process was originally described by Uhlenbeck & Ornstein (1930) and applied to reinforcement learning in DDPG by Lillicrap al. (2016). The parameters we use for the -uletbbb- noise are taken from widely used RL-algorithm implementation (Hill et al., 2018):
0.15, dt 0.01, / 0, 0.1
Due to the huge number of possible combinations of environments, algorithms, noise type, noise scale and the necessary repetition with different sceds, we had to limit the number of investigated scales. We set out with two noise scales a encountered in pre-tuned ayper-parameeteriatiion (Raffin, 2020), 0.1,0.5, and continued with a linear increase, 0.9, 1.3, 1.7. Much smaller noise scales vanish in the variations induced by learning and much larger scales lead to Bernoulli trials of the min-max actions without much difference.
Because the action noise clipped to [-1, before being scaled the actual action limits, very large scale, such 1.7, implies larger percentage of on-the-boundary action noise samples and is thus more similar to bang-bace bang control actions, the latter having been found surprisingly effective in many RL benchmarks (Seyde et al., 2021).
3.2 Scheduling strategies to reduce action noise
Noise Schedulers
1.0
Scaling Ratio
0.4
Coustant
0.2
Logistic
Linear
0.0
0%
20%
40%
60%
80%
100%
Training Steps
Figure 2: Action noise used for exploration. The agent should favor exploration in the beginning but later favor exploitation. Similar to e-greedy strategies in discrete-action Q-learning, the logistic and linear schedulers reduce the impact of noise (scaling ratio, ,9 in (1)) over the course of the training progress.
In (1) we introduce the action noise scaling-ratio 3. In this work we compare a constant-, linear- and logistic- scheduler for the value of The effective scaling of the action noise by the noise schedulers is illustrated in Figure 2. The noise types are described in more detail in Section 3.1.
Changing the a (see (3) and (2)) instead of 3 could result in a different shape of the distribution, for example when values are clipped, or when the a indirectly affects the result as in the Drnstein-Uhlenbaee process. To keep the action noise distribution shape constant, the action noise schedulers do not change the o parameter the noise process but instead scale down the resulting sampled action noise values by changing
An altemaatve way of sampling Gaussian action noise would be use truncated Gaussian distribution. investigate non-truncated Gaussian distributions together with clipping, they are more common practice.

---

Published in Transactions on Machine Learning Research (11/2022)
the B parameter: this means that the effective range of the action noise, before scaling and adjusting to the environment limits, changes over time from [-1, 1], the maximum range, to 0 for the linear and logistic schedulers.
3.3 Environments
For evaluation we use various environments of increasing complexity: Mounlain-Car, Inverted-Pendullme Swingup, Reacher, Hopper, Walker2D, Chlff-hhetahh Observation dimensions range from 2 to 26, and action dimensions range from to 6. See Table E.1 for details, including a rough sketch of the reward. The table indicates whether the reward is sparse or dense with respect to goal state, goal region, or a change of the distance to the goal region. Many environments feature linear quadratic (energy) penalties on the actions (e.g. Hopper). Penalties the state can be sparse (such as joint limits), dense (such force or required power induced by joint states). Brockman et al. (2016), Coumans & Bai (2016-2021), and Ellenberger (2018) provide further details.
3.4 Performed experiments
We evaluated the effects of action noise on the popular and widcly-used algorithms: TD3 (Fujimoto et al., 2018), DDPG (Lillicrap et al., 2016), SAC (Haarnoja et al., 2019), and a deterministic version of SAC (DetSAC, Algorithm D.1). Originally SAC was proposed with only exploration from its stochastic policy. However, since SAC is an off-policy algorithm, is possible to add additional action noise, a common solution for environments such the Mountain Car. The stochastic policy in SAC typically is a parameterized Gaussian and combining the action noise with the stochasticity of actions sampled from the stochastic policy could impact the results. Thus, we also compared to our DetSAC version, where action noise is added to the mean action of the DetSAC policy (Algorithm D.1).
We used the implementations provided by Raffin et al. (2021a), following the papeee-prrmmeteiizaionn pro- vided by Raffin (2020), but adapting the action noise settings.
The experiments consisted of testing 6 environments, 4 algorithms, 5 noise scales, 3 schedulers and 2 noise types. Each experiment was repeated with 20 different seeds, amounting to 14400 experiments in total. On a single node, AMD Ryzen 2950X equipped with four GeForce RTX 2070 SUPER, 8 GB, running about twenty experiments in parallel this would amount to a runtime of approximately 244 node-days (which accounted for about 6 weeks on our cluster).
Section I lists further details such as the returns averaged across seeds for each experimental configuration.
3.5 Measuring Performance
For each experiment (i.e. single seed), we divided the learning process into 100 segments and evaluated the exploration and learned policy performance for each of those segments. At the end of cach segment, performed evaluation rollouts for 100 episodes 10000 steps, whichever was reached first, using only complete episodes. This ensures sufficient. data points when the episode length varies greatly (e.g. for the Hopper). This procedure was performed for both the deterministic exploitation policy as well as the exploratory (action noise) policy. The two resulting datasets of evaluation rollouts are used to calculate state-space coverages and returns. These evaluation rollouts, both exploring and exploiting, were not used for training and thus do not change the amount of training data soen between training steps. We took the mean over these 100 measurements aggregate them into single value. This is equivalent to measuring the area under the learning curves. For the evaluation returns, this is called the Performance P and is our main measure for learning performance. Similarily, aggregated evaluation returns measured in this fashion are denoted by R.
The learning algorithm uses a noisy (exploratory) policy to collect data and exploratory return and state- space coverage could be asscssed based on the replay buffer data. However, to get statistically more robust estimates of the quality of the exploratory policy (returns and state-space coverage), we performed the above mentioned exploratory evaluation rollouts and used these rollouts for assessing state-space coverage
7

---

Published in Transactions on Machine Learning Research (11/2022)
moch
good
(a) Xiin divides the space into bins and mea- sures the ratio non-empty bins.
(b) XBBM measures the spread by mean the side-lengths of the bounding box.
(c) XNN measures the spread of the data by the sum the eigenvalues of the covariance the data.
(d) Xuen (ours) measures the symmetric KT-diver- gence between prior over the state space and the col- lected state-space data.
Figure 3: Illustrations of the state-space coverage measures, Xenee scales to high dimensions (unlike Xbin) and is not susceptible to estimation artifacts due to points close to the support boundary (unlike XBBM, XNN and kNN based estimators).
and exploratory returns instead of the data in the replay buffer. Again, these 100 measurements were aggregated by taking the mean and denoted as the exploratory state-space coverage X and the evaluation state-space coverage E.
3.6 State-Space Coverage
We assess exploration in terms of state-space coverage. We assume that the environment states s E Rd have finite upper and lower limits: low S 8 < high, low, high E Rd. We investigate four measures: Xbin Xttel, XBBM, which are illustrated Figure 3.
The most intuitive measure for state-space coverage is histogram- based approach Xpin which divides the state space into equally many bins along each dimension and measures the ratio of non-empty bins to the total number of bins:
# of non-empty bins # number of bins
Xbin
(5)
The number of bins, the product of divisions along each dimension, grows exponentially with the dimen- sionality. This means that either the number of bins has to be chosen very low, or, if there more bins than data points, the ratio has to be adjusted. We chose to limit the number of bins. For sample of size and dimensionality d the divisions k along each dimension are chosen to allow for at least e points per bin
m
k=
[+]
(6)
However, for nigh-dimensional data, the number bins becomes very small and the measure easily reaches 100% and becomes meaningless, or, the required number of data points becomes prohibitively large very quickly. Thus, alternatives are necessary.
Zhan et al. (2019) proposed two state-space coverage measures that also work well in high-dimensional spaces: the bounding boz mean XBBM, and the nuclear norm XNN. XBBM measures the spread of the data by d dimensional bounding box around the collected data D = {...,s()) ....} and measuring the mean of the side lengths of this bounding box:
XBBM = Σ max 8 min (
(7)
NNN, the muclcar norm estimates the covariance matrix C of the data and measures data sprcad by the trace, the sum of the eigenvalues of the estimated covariance:
XNN (D) trace (C(D))
(8)
8

---

Published in Transactions on Machine Learning Research (11/2022)
As shown below in Section 3.6.1, extreme values values close to the state-space boundaries can lead to over-estimation of the state-space coverage by these two measures. We therefore propose measure more closely related to Xbin but more suitable to higher dimensions: (D). The Unfoora-relative-entooy measure Xyreel assesses the uniformity of the collected data, by measuring the state-space coverage as the symmetric divergence between uniform prior over the stale space and the data distribution QD:
(D) = DKL (UI((D) - DKL (QDIUU)
(9)
The inspiration for this measure comes from the observation that the exploration reward for count-based methods without task reward would be maximized by a uniform distribution. We assume that for robotics lasks reasonable bounds the state space can be found. In bounded state space, the uniform distri- bution is the least presumptive (maximum-entroppy distribution. The addition of the (UIIQQ) term helps to reduce under-estimation of the divergence in areas with low density in QD. Note that QD is only available through estimation, and the support for QD is never zero as the density estimate never goes to zero. To estimate the relative uniform entropy we evaluated two divergence estimators, a kNN-based (k- Nearest-Neighbor) estimator and a Nearest-NNigghbo--aatie (NNR) estimator (Noshad et al., 2017). Density estimators based on kNN are susceptible over- / under- estimation artifacts close the boundaries (sup- port) of the state space (see Figure B.1 for an illustration). In contrast, the NNR estimator does not suffer from these artifacts. If not specified explicitly, Xurel refers to the NNR-based variant.
ANN Xuvet eslimator: Xutee can be estimated using kNN densily estimale (8()), ) described in (Bishop, 2006), where Va denotes the unit volume of a d-dimensional sphere, Rx(x)) is the Euclidean distance to the k-th neighbor of v, and n is the total number of samples in D:
Vd T/2
(10)
T( 1)
k
k 1
(x) Vala()))
(11)
naa Ra (x)d
where I denotes the gamma function.
NNR Xutet estimator: Alternatively, Xutel can be estimated using NNR, an f-divergence estimator, based on the ratio of the ncarest neighbors around a query point.
For the general case of estimating DKL. (PI[Q), we take samples from X Q and Y ~ P. Let R& denote the set of the k-nearest neighbors of Y; in the set Z = X UY. N, is the number of points from X n Rx (Yi), M, is the number of points from Y n (Y;), M is the number of points in Y and N is the number of points in X, 1 M The NNR measure requires the density of P and Q to be bounded with the lower limit Cz2 > O, and measures the ratio of points from two different distributions around a query point. Assuming all points of sample of size n are concentrated around single point, we lower-bound the density to Cr. = 4. To limit the peaks around single point we upper-bound the densities to Cy- 1"
(kl(P rD,(X,Y)
(12)
1
N/N:
.0)
D,(X,Y) max M
9
Ms
MM+11
(13)
where ((x:) max g((x))((r)/Cv))
(14)
g(p) log
(15)
3.6.1 Evaluation of Measures on Synthetic Data
To compare the different exploration mcasures, we assumed a d dimensional state space, generated data from two different types of distributions, and compared the exploration measures on these data. The experiments repeated 20 times, and the mean and min-max values are plotted in Figure 4. Fach
9

---

Published in Transactions on Machine Learning Research (11/2022)
sampled dataset consists of 5000 points. For most measures the variance is surprisingly small. While the data are d-dimensional, they come from factorial distributions, similarly distributed along each dimension. Thus, we can gain intuition about the distribution from scatter plots of the first vs. second dimension. This is depicted at the top of each of the two parts. The bottom part of each comparison shows the different exploration measures, where the scale parameter is depicted on the a axis and the exploration measure on the y axis.
(a) Growing Uniform: Figure 4(a) depicts data generated by a uniform distribution, centered around the middle of the state space; with minimal and maximal values growing relatively to the full state space
Scale:
Scale: 17%
Scale: 34%
Scale: 50%
Scale:
Scale:
Scale:
Scale:
Scale:
Scale:
%
67%
83%
100%
116%
133%
150%
-10
-10
100
1
-10 1
-10
1
-10 1
-10 1
-10
110
-10
5pt. Bin
Bounding Box
U-RelEntropy (NNR) (ours)
U-RelEntropy (kNN) (ours)
Nuclear Norm
Coverage
Mean
0.5
0.0
BBMM
Xbin
100
NNN
100
Xirel
100
Xurel
100
Scale
Scale
Scale
Scale
Scale
larger scale values more points are clipped to the state-space boundaries, leading to an expected decrease in state-space coverage for scales larger than 100%. This behavior is only captured by Xteei (NNR).
(a) Growing Uniform distribution: evaluation of the state-space coverage measures on synthetic data
Scale:
Scale:
Scale: 34%
Scale: 50%
Scale: 67%
Scale: 83%
Scale: 100%
Scale: 116%
Scale: 133%
Scale:
1%
17%
150%
110
-10
10
110
-10
5pt. Bin
Bounding Box
U-RelEntropy (NNR) (ours)
U-RelEntropy (kNN) (ours)
Nuclear Norm
Coverage
Mean
2.0
20
70
0.5
80
1.0
-90
bin
0
100
XBBM
100
XNN
100
Xurol
100
Xurel
100
Scale
Scale
Scale
Scale
Scale
(b) Growing Distance of Modes of 2-Mixture of Truncated Normal: evaluation of the state-space coverage measures on synthetic data. For larger scale values, the location of the mixture components is closer to the boundary leading to an expected reduction in coverage for larger scale values. Xain, XBRM, Xxy fail to capture this behavior.
Figure 4: state-space coverage measures may not accurately represent the real coverage. Each comparison (a-b) shows the different exploration measures X XBBM, XNN and X (ours) synthetic 25 dimensional data. Xpia becomes constant and XBBM and XNN suffer from estimation artifacts for points close to the support boundary. The different. scale parameter. The distributions are factorial and similarly distributed
data generating distributions depend along each dimension. The scatter plots in (a-b) depict first vs, second dimension. The mean and min-mat max variation is shown. In the majority of cases the variance surprisingly small.
10

---

Published in Transactions on Machine Learning Research (11/2022)
according to the scale parameter from 1% to 150%. Since in the latter case, many points would lie outside the allowed state space; these values are clipped to the state-space boundaries. This loosely corresponds to an undirectedly exploring agent that overshoots and hits the state-space limits, sliding along the state- space boundaries. Note how the estimation (kNN vs. NNR) has a great impact on the Xutel measure's performance here: We would expect a maximum around scale of 100% and smaller values before and after (due to clipping). Here the Xurel (NNR) mcasure most closely follows this expectation. The ground-truth value of the divergence would follow a similar shape. However, since the densities are limited for the NNR estimator, the ground-truth divergence would show more extreme values.
(b) Bi-Modal Truncated Normal moving locations: Figure 4(b) shows a mixture of two truncated Gaussian distributions, with equal standard deviations but located further and further apart (depending the scale parameter). In this case, the state-space coverage should increase until both distributions are sulficiently far aparl, should then stay the same, and finally begin to drop because the proximily to the saate-space-boundar limits the points to an ever smaller volume. The inspiration for this example distribution is an agent setting off in two opposite directions and getting stuck at these two opposing limits. While somewhat contrived and more extreme than the inspiring example, it highlights difficulties in the exploration measures. Both the bounding-box mean XBBM and the nuclear norm XNN completely fail to account for vastly unexplored areas between the extreme points.
Since the Xurel NNR measure is clipped (by definition of NNR) the measure reaches its limits when the density ratios become extreme, which presumably happens for very small and large scale parameters in this setting. The Xurel kNN approximator is better able to capture the extreme divergence values, however, as pointed out before, this comes at the cost of under-estimating the divergence for points close to the support boundary.
The exporiments on synthetic data showed that the histogram based mcasure not useful in dineesiinnni spaces. The alternatives XлоM and XNN are susceptible to artifacts on bounded support. This susceptibility to boundary artifacts is also present in the kNN-based Xetel estimator, because of these results we employ the NNR-ostinator based Xutel in the rest of this paper and refer to it as X
4 Results: What action noise to use?
In this section we analyze the data collected in the experiments described in Section 3.4. We first look at the experiments performed under a constant scale scheduler since this is the most common case in the literature. In this setting we will look al two aspects: first, is one of the two action noise types generally superior to the other (Q1)? And secondly, is there a generally preferable action noise scale (Q2)? Then, we compare across constant, linear and logistic schedulers to see if reducing the noise impact over the training process is a reasonable thing to do (Q8). Finally we compare the relative importance of the scheduler, noise type and scale (Q4). See Section F.1 for a brief description of the statistical methods used in this paper and the verification of their assumptions.
4.1 (Q1) Which action noise type to use? (and what are the impacts)
To compare the impact of the action noise type, we look at the constant == case, group the aggregated performance and exploration results (see Section 3.5) by the factors algorithm, environment, and action noise scale and standardize the results to control for their influence. These standardized results are then combined for each noise type. Figure illustrates the results. The comparisons are performed by Welch-t- test, symmetric v-values are listed.
Figure 5 (c) shows that Ornstein- Uhlenbeck noise leads to increased state-space coverage under the ex- ploratory policy X as measured by Xttret- For completeness Figure 5 (d) shows the state-space coverage of the evaluation policy. Here Uhllebee increases coverage which might indicate slightly longer trajectorics for policies trained under Drnstein-Uhlenbeee noise, however whether this preferable or not is task dependant. Exploration likely incurs additional costs, e.g. through action penalties, but also by moving the agent away from high-reward-trajecoories Since Ornstein-Uhlenbeek noise is temporally correlated, it
11

---

Published in Transactions on Machine Learning Research (11/2022)
G ou
G > OU <<< 10
G < OU < 10
G < OU
P (Eval-Reesee (ll-Retuuss))
p = 0.40
R (Expl, (Expl-Reeture -Returus)
X (Lixpl. Urell))
(Evval-XX (Eival-XUrel) -Urrl)
p << 10
2.5
2.3
2.5
0.0
0
0.0
0.0
2.5
2.5
-25
OU
OU
OU
(a)
(b)
(c)
(d)
Figure 5: Comparison of standardized measures (P, R, X, E), for Gaussian (G) and Maastein-Uhlenbeck (OU) noise types, (a-d). Values standardized to control for and combine algorithm, environment. and noise scale: (a) For learned performance P, measured by evaluation returns, neither of the two noise types is significantly better. (b) For Returns collected under the exploration policy Caussian noise collects data with slightly better returns (p < 100 (c) For State-space coverage of the exploratory policy X Ornstein-Uhlebeeck performs better. (d)
The State-space coverage of evaluation rollouts E slightly larger for Ornstein- Ornstein-Uhlenbeed noise without significantly affecting the evaluation returns P. Overall neither of the two noise lypes is superior.
Environment
P Pp
dp
R
pn
dn
X px
dx
pE
dE
Half-Cheetan
G 0.002
0.22
OU 0.004
0.21
0.20
Hopper
OU <10-3 0.27
G
<10-4
0.29
G <10-8
0.41
0.71
Inverted-
0.38
G <10-51 51
1.15 OU <10-56
1.22
G 0.002
0.22
Pendulum- Sonduuuu--Swigup
Mountain-Car
OU <10-10
0.47
OU <10-19 0.66
OU <10-5
0.34
OU <10-2 21
0.71
Reacher
G <10-3> 30
0.87
G <10- 0.80
OU <10-40 1.01
OU <10
0.84
Walker2D
0.039
G 0.010 0.18
OU <10-9 0.46
0.28
Table 2: Per enviromment the noise Lype is important: Comparison of Evaluation Returus P. Exploratory Returns Exploratory-Xxoe X, and Evaluation-Xzvvl E. Values are standardized to control for and aggregale over algorithm, and noise scale. The results are compared using a Welch-t-test. Significantly better noise type for each environment and measure is reported (p < 0.01), as well as two-tailed p-values P()) and Cohen-d effect size d()- While overall neither of the noise types leads to significantly better performance P (see Figure 5), per environment noise type difference is significant.
is more efficient in covering more state-space but also in moving the agent away from high-reward trajecto- ries. Thus exploratory relurns are larger for Gaussion noise and conversely smaller for Ornstein- -hheeebbbh noise, see Figure (b). The learning process is able to offset some differences in the data as shown in Figure 5 (a): the significant differences in exploratory returns R and exploratory stato-space coverage X do not trans- late into significantly-difeee differeet performance across encironments. When viewed on per-environment basis, Table 2 (column P) shows that, the preferable noise type depends on the environment: Ornstein-Uhlenineee is preferable for Hopper and Mountain-Cas Car, but Gaussian for the Reacher environment. Table (column X) shows that Ornstein-Uullnbeeck leads to larger state-space coverage, as before, and Gaussian noise leads larger exploratory returns (column R). The only exceptions this are the Hopper environment, where the Ornstein-Uhlenbeck is more likely topple the agent and the Mountain-Car environment, where the returns are very closely related to increasing the state-space coverage and thus exhibits an improvement of noise.
R by
These results show that the noise type is important and significantly impacts the performance for some environments. Neither of the two noise types leads to better performance, evaluation return P, in general. However Ornstein- Uhlenbeck generally increases state-space coverage. This likely due to the effect, that in many cases the environment acts as an integrator over the actions: many environments the action constitutes some type of velocity or force control, which by stepping forward, and thus integrating forward in time, amounts to changes in position, or respectively changes in velocity.
12

---

Published in Transactions on Machine Learning Research (11/2022)
p(P,o): -0.31
P(R, a): -0.55
o(X,o) 0.56
o(E,a): 0.05
R (Expl- Repl--Returns)
(Expl. (Expl--XUL) -XUrel)
(Eval--X Urell)
1 (Eval-Returns
2
2
X
0.1
0.5
0.9
0.1
0.5
0.9
1.7
0.1
0.5
13
1.7
0.1
0.9
1.5
0
(a)
(b)
(c)
(d)
Figure Across environments larger noise scales a are effective increasing state-space coverage (c), but reduce exploratory returns (b). Measures (P, X, H, B) are standardized to control for and aggregate over algorithm, environment and noise type. (a) Evaluation Performance P negatively correlated with action noise scale (p -0.31). (b) Larger noise scales correlate with smaller exploratory returns R. (c) Increasing the noise scale increases exploratory state space coverage X. (d) State-space coverage evaluation rollouts E: learned trajectories appear unaffected by larger noise scale.
Environment
p(P, R)
((P,))
p(P, Oscale)
p(R,
plR, Oscalo)
p(X, Oscale)
All
0.57
-0.03
-0.31
-0.30
-0.55
0.56
Half-Cheetall
0.22
-0.28
-0.35
-0.64
-0.74
0.75
Hopper
0.69
0.15
-0.87
0.27
-0.74
-0.17
-Swinupp
-0.15
0.23
0.27
-0.88
-0.83
0.77
Mountain-Car
0.94
0.87
0.58
0.76
0.37
0.75
Reacher
0.84
-0.88
-0.56
-0.96
-0.84
0.69
Walker2D
0.76
-0.44
-0.81
-0.52
-0.82
0.63
Table 3: Data quality, measured by exploratory returns R, does not completely determine performance, measured by evaluation returns P. denotes Spearman correlation coefficients. Generally R is positively, but surprisingly not always strongly, correlated with P. For some environments, exploratory state-space coverage X is beneficial, while generally it is associated with decreased evaluation performance P. Across environments and noise types, increasing the noise scale increases exploratory state-space coverage but reduces exploratory returns R.
4.2 (Q2) Which action noise scale to use?
To analyze the impact of action noise scale, we look the constant (3 = 1) case, and control for the impact of the factors algorithm, environment and noise type: by grouping the results according to these factors and standardizing the results. Then results for the same noise scale are combined.
An interesting observation shown in Figure 6 (c) is that state-space coverage of the exploratory policy X correlates positively with action noise scale a (p Spearman correlation coefficients). The takeaway from this is that instead of changing the noise type, one might increase state-space coverage by increasing o. This however leads reduction the exploratory returns R, see Figure 6 (c), (p(R, a) = -0.55). Subsequently, larger noise scales are associated with decreased learned performance, i.e. smaller evaluation returns P. Figure 6 (a), when viewed across environments. Note that for very small noises (a = 0.1) the variance of the results P becomes very large. It appears that, in many cases, less noise is actually better, but too little noise often does not work well. A good default for a appears to be > 0.1 but < 0.9. The scale does not appear to have a strong effect on the evaluation state-space coverage E, Figure (d). When viewed separately for each environment (Table 3), the association between X and consistent. The only exception is the Hopper task, where large noise is more likely to topple the agent, making it fail carlier, thereby reducing state-space coverage. The association between p(R, a) consistently negative, with the exception of the Mountain-Car where state-space coverage directly translates to higher returns, because the
13

---

Published in Transactions on Machine Learning Research (11/2022)
Chef-Cheetah
Hopper
Normalized
Performance
-2
Mountain- Car
Reacher
Walker2D
Normalized Porformance
2
2
Schedulers
0
Constant
-2
-2
Linear
Logistie
Figure majority of cases action noise schedulers improve performance. The figure shows the comparison of the learned policy performance, measured by evaluation returns P, for each environment and scheduler. Data is standardized control influence algorithm, environment, noise scale a and noise type. In the majority cases the linear and logistic schedulers perform better than comparably the constant scheduler.
var(P) Linear
P
Scheduler
Constant
Logistic
> Constant
> Linear
Logistic
Constant
0
0
0
0
1
Linear
4
0
4 0
2
Logistic
4
4
Table 4: In the majority of cases, using scheduler reduces variance of the performance (evaluation returns) var(P); and improves expected performance P. The evaluation returns P are standardized to control for the influence of algorithm, noise scale a and noise type. Levene's tests are used assess difference variance var(P) and multiple- comparison Games- Howell test indicates superior performance P. The table shows the number of environments which each scheduler (row) is significantly better than the other schedulers (column). See Table H.l for full per- environment results.
environment underactuated and energy needs be injected into the system. Offiine-Rl RL findings indicate that it is casier to learn from expert data than from data of mixed-quality (Fu et al., 2020). As such, we would expect a very strong correlation between exploratory relurns R as a measure of data quality and evaluation returns P as a measure of learned performance. Indeed, P.P R) shows that overall exploratory returns R and evaluation returns P are mostly positively correlated. However, the correlation is not always very strong and can even negative. This is interesting, because this means that exploratory returns are not the only determining factor for learned performance. For example, in the Invertci-eaddulln-Svinggg p(P, R) is slightly negative while p(P,X) is positive. The results indicate that, the noise scale has to be chosen to achieve trade-off between either increasing state-space coverage X or returns R as required for each specific environment.
4.3 (Q3) Should we scale down the noise over the training process?
The previous sections indicated that there is no unique solution for the best noise lype and that this choice is dependent on the environment. The analysis of the noise scale showed overall preference for smaller noise scales, but also showed that, in contrast, some environments require more noise to be solved successfully. In this section we analyze schedulers that reduce the influence of action noise (3) over the training progress.
Figure shows the performance for each environment and each scheduler. The data is normalized by environ- ment and algorithm before aggregation. The general tendency observed across environments that, when the environment reacts negatively larger action noise scale WHalf-Cheetah, Hopper, Reacher, Walker2D; shown in Table 3), reducing the noise impact over time consistently improves performance. The
14

---

Published in Transactions on Machine Learning Research (11/2022)
Spearman Correlation
vf Effect Size
Envname
p(P,X)
p(P.o) o)
p(P, R)
Trype
Scheduler
All
-0.503
-0.120
0.770
0.005
0.000
0.084
Mountain-Cae Car
0.662
0.442
0.959
0.032
0.060
0.261
PeveroedPenuulun--Swnngg
-0.003
0.123
0.163
0.005
0.009
0.115
Reacher
-0.872
-0.382
0.803
0.048
0.181
0.181
Hopper
-0.349
-0.599
0.651
0.045
0.022
0.660
Walker2D
-0.658
-0.494
0.677
0.014
0.017
0.607
Half-Cheotee
-0.581
-0.259
0.745
0.007
0.002
0.148
Table Spearman correlation coefficients and ANOVA 1/² effect sizes P for: scheduler, noise type and noise scale a. Action noise scale a is associated with the largest effect size for evaluation returns P. Results are shown across all enviromments (standardized and controlled for environment and algorithm, first row), and per environment. (standardized and controlled for algorithm). Generally, exploratory returns R and evaluation performance P are positively associated, while generally larger state-space coverage X appears impact performance P negatively.
verse effect appears to be less important: for environments benefiting from larger noise scales, the constant scheduler does not consistently outperform the linear and logistic schedulers.
Table shows summarized results indicating the number of environments where scheduler (1), indicated by row, is better than scheduler (2), indicated by column, in terms of variance var(P) and mean performance P. See Table H.1 for full results on the pairwise comparisons. Performance differences are assessed by a Games-Howell multiple comparisons test, while variance is compared using Levene's test.
The tests underlying Table show that the differences observed in Figure are indoed significant. Further- more, the schedulers (linear, logistic) reduce variance var(P) compared to the constant case in four out of six cases. Keeping the impact constant has no beneficial effect on variance any environment. This indicates that using a scheduler to reduce action noise impact increases consistency in terms of learned performance.
4.4 (Q4) How important are the different parameters?
In the previous sections we looked at each noise configuration parameter independently, first for the constant 3 case (Q1, Q2), secondly for scheduled reduction of / (Q3). However, the question remains whether all the parameters are equally important. We standardize results to control for environment and algorithm, and compare across all noise types, noise scales a and all three schedulers.
Table shows Spearman correlation coefficients p(P, p(P,.), Rt) across all three schedulers (compare to Table 3 which showed correlations for the constant B 1 case only). Across environments the schedulers reduce correlation p(P, o) between learned performance (measured by evaluation returns P) and noise scale 0: from p( a)) = -0.31 in the constant scheduler case to p( p(P.a) = -0.12 when compared across all three types of schedulers. This is a further indication that using a scheduler increases robustness to a. The correlations between p(P, R) are increased to 0.77 vs. 0.57, presumably because reducing & makes the exploratory policy more on-policy and thus P and become more similar. Interestingly, the schedulers also increase the negative correlation p(P,X) between the performance and the exploratory state-spaco coverage, from -0.03 in the constant case to -0.50 when viewed across all schedulers. This could be driven by the environments reacting positively to reduced state-space coverage, which under the schedulers achieve more runs high in R but low in X, and thus a stronger negative correlation.
The three columns on the right in Table 5 show of effect sizes of a three-way ANOVA on the evaluation returns P: 2 V[Scheuuler p n2 The 17 effect sizes measure the percentage-of variance explained by each factor. Only the Reacher environment, action noise type very important. Surprisingly, in all cases the most important factor action noise scale, while the requirement for large or small action noise scale varies for each environment.
15

---

Published in Transactions on Machine Learning Research (11/2022)
Enyname
Scheduler
Турe
Horizon
Recommendation
All
lin
0..5
OU
Mountain-Cas
log
1.7
OU
L
large 0, sched
Eiverled-Pendulum-Swnnng
con
0.5
Gauss
L
large o
Reacher
lin
0.1
Gauss
small 0, Gauss, sched
Hopper
lin/log
0.1
OU
S
small 0, sched, OU
Walker2D
lin
0.1
OU
S
small 0, OU, sched small
Half-Cheetah
lin/log
0.5
Gauss/ /OU
S
Table 6: Comparison of best-ranked noise type, scale and scheduler across all environments and for each enviromment individually. Scheduler, type and scale are investigated separately by standardizing the values to control for environ- ment, algorithm and the other two respective factors. Horizon indicates whether we expect a long (L) or short (S) effective planning horizon. Recommendation indicates action noise configuration choices in order of importance per Table 5, for options with effect sizes n/² > 0.01 (small effect).
5 Discussion & Recommendations
The experiments conducted in this paper showed that the action noise does, depending on the environment, have a significant impact on the evaluation performance of the learned policy (Q1). Which action noise type is best unfortunately depends on the environment. For the action noise scale (Q2), our results have shown that generally larger noise scale increases stale-space coverage. But since for many environments, learning performance is negatively associated with larger state-space coverage, a large noise scale does not generally have preferable impact. Similarly, very small scales also appear not to have preferable impact, as they appear to increase variance of the evaluation performance (Figure 6). However, overall, reducing the action noise scaling factor over time (Q3) mostly has positive effects. Finally we also looked at all factors concurrently (Q4) and found that for most environments noise scale is the most important factor.
It difficult to draw general conclusions from a limited set of environments and extending the evaluation is limited by the prohibitively large computational costs. However, we would like to provide heuristics derived from our observations that may guide the scarch for the right action noisc. Table shows the best- ranking scheduler, scale and lype configurations for each, and across environments. The ranking is based on the count of significantly better comparisons (pairwise GowellHowell test on difference, p < 0.01, positive test statistic). For each of schoduler, type and scale we standardize to control for the other two factors. Intuitively, the locomotion enviromments require only a short effective planning horizon: the reward in the environments based on the distance moved and is relevant as soon as the locomotion pattern is repeated; for example a 30-step horizon enough for similar locomotion benchmarks (Pinneri et al., 2020). In contrast, the Mountain- Car environment only provides informative reward at the end of successful episode and thus, the planning horizon needs to be long enough to span complete successful trajectory (e.g. closer to 100 steps). Similarly, the SeerttedPPnddulum-SSiiii local optima: to swing up and increase system energy, the distance to the goal has to be increased again. These observations are indicated in the column Horizon (Table 6). Finally, the recommendation column interprets the best-ranked results under the observed importance (Q4) reported Table Given these results, we provide the following intuitions a starting point for optimizing the action noise parameters (read as: to address this do that):
uses a shaped reward that does not account for spurious
Environment is under-actuated & increase state-space coverage We found that in the case of the both of which are underactuated tasks and require a
Mountain-Car and the Ewverted-Pendllum-SSee swinging up phase, larger state-space coverages or larger action noise scales appear beneficial (Table 3 and Table 5). Intuitively, under-actuation implies harder-to-reach state-space areas.
Reward shape is misleading > increase state-space coverage Actions are penalized in the Mountain Car by an action-energy penalty, which means not performing any action forms a local optimum. In the case of the Pendulud-ennuuuuun-S swinging up, increasing the distance to the goal is necessary. Thus, the shaped reward can be misleading:
the distance to the goal forms a shaped reward. However, when
16

---

Published in Transactions on Machine Learning Research (11/2022)
following the reward gradient to greedity leads the agent to a spurious local optimum. Optimizing for spurious local optimum implies not reaching areas of the state space where the actual goal would be found, thus the state-space coverage needs to be increased to find these areas.
Horizon is short >> reduce state-space coverage The environments Hopper, Reacher, Walker2D model locomotion lasks with repetitive movement sequences. In the Mountain-Car, positive reward is only achieved at the successful end of the episode, where as in the locomotion tasks positive reward is received after each successful cycle of the locomotion pattern. Thus effectively the required planning horizon is shorter compared to tasks such as the Mountain-Cant Consistently with the previous point, if the effective horizon is shorter, the rewards are shaped more efficiently, we see negative correlations with the state-space coverage and the noise scale: if the planning horizon is shorter, the reward can be optimized more greedily, meaning the state-space coverage can be more focused and thus smaller.
Need more state-space coverage >> increase scale Our analysis showed that, to increase state-space coverage, one way is to increase the scale of the action noise. This leads to higher probability of taking larger actions. In continuous control domains, actions are typically related to position-, velocity- or torque- control. In position-control, larger actions are directly related to more extreme positions in the state space. In velocity control, larger actions lead to moving away from the initial state more quickly. In torque control, larger torques lead to more energy the system and larger velocities. Currently most policies in D-RL. are either uni-modal stochastic policies, or deterministic policies. In both cases, larger action noise leads to broader selection of actions and, by the aforementioned mechanism, to a broader state-space coverage. Note that while this is the general effect we observed, it is also possible that too large action can have a detrimental effect, c.g. the Hopper falling, and the premature end of the episode will lead to a reduction of the state-space coverage.
Need more state-space coverage >> try Ornstein- Uhlenbeck Depending on the environment dynam- ies, correlated noise (Ornstein-Uhlenbeeki can increase the state-space coverage: for example, if the environ- ment shows integrative behavior over the actions, temporally uncorrelated noise (Gaussian) lcads to more actions that "undo" previous progress and thus less coverage. Thus correlated Orustein-Uhlenbert noise helps increase state-space coverage.
Need less state-space coverage or on-policy data i> reduce scale use scheduler to decrease 8 If the policy is alrcady sufficiently good, or the reward is shapod well enough, exploration should focus around good trajectories. This can be achieved using a small noise scale a. However, if the environment requires more exploration to find reward signal, it makes to sense to use a larger action noise scale a in the beginning while gradually reducing the impact of the noise (Q3). The collected data then gradually becomes "more on-policy".
In general use scheduler We found that using schedulers to reduce the impact of action noise over time, decreases variance of the performance, and thus makes the learning more robust, while also generally increasing the evaluation performance overall. Presumably because, once a trajectory to the goal is found, more fine grained exploration around the trajectory is better able to improve performance.
6 Conclusion
In this paper we present an extensive empirical study on the impact of action noise configurations. We compared the two most prominent action noise types: Gaussian and Uhhnebin-Uhlenneck, different scale parameters (0.1, 0.5, 0.9, 1.3, 1.7), proposed a schoduled reduction of the impact / of the action noisc over the training progress and proposed the state-space coverage measure Xurol assess the achieved exploration in terms of state-space coverage. We compared DDPG, TD3, SAC, and its deterministic variant detSAC on the Reacher, Hopper, Walker2D, and Half-Chectah.
benchnarks Mountain-Car,
We found that (Q1) neither of the two noise types (Gaussian, Orustein-Uhlenbecc, is generally superior across environments, but that the impact of noise type on learned performance can be significant when viewed separately for each environment: the noise type needs to be chosen to fit the environment. We found that (Q2) increasing action noise scale, across environments, increases state-space coverage but tends to reduce learned performance. Again, whether state-space coverage and performance are positively correlated,
17

---

Published in Transactions on Machine Learning Research (11/2022)
and thus larger scale is desired, depends the environment. The positive negative correlation should guide the selection of action noise. Reducing the impact (3) of action noise over training time (Q3), improves performance in the majority of cases and decreases variance in performance and thus increases robustness to the action noise choice. Surprisingly, we found (Q4) that the most important factor appears to be the action noise scale a: less state-space coverage required, the scale can be reduced. More stale-space coverage can be achieved by increasing the action noise scale. This approach is successful even for Gaussian noise the Mountain-Car. We synthesized our results into a set of heurishics on how to choose the action noise based on the properties of the environment. Finally we recommend scheduled reduction of the action noise impact factor 3 of over the training progress to improve robustness to the action noise configuration.
Acknowledgments
We would like thank Bart Keulen, David Peer, Onno Eberhard, Sebastian Blacs and the TMLR Reviewers for the useful discussion.
References
Susan Amin, Maziar Gomrokchi, Harsh Satija, Herke van Hoof, and Doina Precup. A Survey of Exploration Methods in Reinforcement Learning. v:Xiv:2199.00157 [es], September 2021.
Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An
evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253 279, 2013.
Christopher M. Bishop. Pattern recognition. Machine Learning, 128:1 58, 2006.
C. Alan Boncau. The effects of violations of assumptions underlying the test. Psychological Bulletin, 57: 49 64, 1960.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym. arXiv:1606.0155 [es). June 2016.
Morton B. Brown and Alan B. Forsythe. Robust tests for the equality of variances. Journal of the American Statistical Association, (9(346)3364333) 367, 1974.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distil- lation. In International Conference on Learning Representations, 2019.
Ttai Caspi, Gal Leibovich, Shadi Endrawis, and Gal Novik. Reinforcement Learning Coach. Zenodo, Decem- ber 2017. URL https://dol httpa::/doooooo/////// 10. /tt///00221//eooda 1134899.
Po- Wei Chou, Daniel Maturana, and Sebastian A. Scherer. Improving stochastic policy gradients in contin- nous control with deep reinforcement lcarning using the beta distribution. In International Conference on Machine Learning, volume 70, pp. 834-843, 2017.
Cédric Colas, Olivier Sigaud, and Pierre-Yve: Yves Oudeyer. GEP-PG: Decoupling exploration and exploitation in deep reinforcement learning algorithms. In International Conference on Machine Learning, volume 80, pp. 1038-1047, 2018.
Erwin Coumans and Yunfei Bai. PyBullet, a Python module for physics simulation for games, robotics and machine learning. 2016 2021. URL http://pybulles: org.
Benjamin Ellenberger. PyBullet gymperium. GitHub repository, 2018. URL httpst//////tthtt cotp://e/mbb//ee/////////o pybullet-gyr
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine, D4RL: Datasets for Deep
Data-Driven Reinforcement Learning. arXiv:2004.07219, 2020.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing Function Approximation Error in Actor- Critii Methods. In International Conference Machine Learning, pp. 1587-1596; PMLR, October 2018.
18

---

Published in Transactions on Machine Learning Research (11/2022)
Yasuhiro Fujita, Prabhat Nagarajan, Toshiki Kataoka, and Takahiro Ishikawa. ChainerRL: A Deep Rein- forcement Learning Library. Journal of Machine Learning Research, 22(77):1-14, 2021. ISSN 1533-7928.
Paul A. Games and John F. Howell. Pairwise Multiple Comparison Procedures with Unequal N's and/or Variances: A Monte Carlo Study, Journal of Educational Statistics, 1(2):113 125, 1976.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Ku- mar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic Algorithms and [cs, stat], January 2019.
Applications.
Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Rene Traore, Prafulla Dhariwal, Christo- pher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alee Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable Baselines. GitHub repository, 2018. URL https://github. com/hilla// stable-baselines
Matt Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Feryal Behbahani, Tamara Nor- man, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, Sarah Henderson, Alex Novikov, Sergio Gómez Colmenarejo, Serkan Cabi, Caglar Gulcehre, Tom Le Paine, Andrew Cowie, Ziyu Wang, Bilal Piot, and Nando de Freitas. Acme: A Research Framework for Distributed Reinforcement Learning. arXiv:2006.000: [cs], June 2020.
Jakob Hollenstein, Matteo Saveriano, Auddy Sayantan, Erwan Renaudo, and Justus Piater. How does the type of exploration-noise affect returns and exploration on Reinforcement Learning benchmarks? In Austrian Robotics Workshop, pp. 22 26, 2021.
Zhang-Wei Hong, Tuu-Yun Shann, Shih-Yang Su, Yi-Hsiang Chang, Tsu-Jui Fu, and Chun-Yi Lee. Diversity- driven exploration strategy for deep reinforcement learning. In Advances in Neural Information Processing Systems, pp. 10489 10500, 2018.
Eric Jones, Travis Oliphant, and Pearu Peterson. SciPy: Open source scientific tools for Python, 2001. URL http: http://wwwwwwcom scipy. org.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishman, Vincent Vanhoucke, and Sergey Levine. QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation. orXiv:1806. 10293, 2018.
Pawel Ladosz, Lilian Weng, Minwoo Kim, and Hyondong Oh. Exploration in deep reinforcement learning: A survey. Inf. Fusion, 85:1-22, 2022. doi: 2..1016/..nffuss22222.0.
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael I. Jordan, and Ion Stoica. RLlib: Abstractions for distributed reinforcement learning. In Jen- nifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan. Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 3059-3068. PMLR, 2018.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Proc. 4th Int. Conf. Learning Representations, (ICLR), 2016.
Thomas Lumley, Paula Diehr, Scott Emerson, and Lu Chen. The importance of the normality assumption large public health data sets. Annual review of public health, 23(1):151-169) 2002.
Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search provides a competitive approach reinforeement learning. arXiv:1803.07000 2018.
Bogdan Mazoure, Thang Doan, Audrey Durand, Joelle Pineau, and R. Devon Hjelm. Leveraging exploration in off-policy algorithms via normalizing flows. In Conference Robot Learning, pp. 430-444, 2020.
19

---

Published in Transactions on Machine Learning Research (11/2022)
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518 533, 2015.
Andrew William Moore. Efficient memory-based learning for robot control. Technical Report UCAM- CL-TR-2009 University of Cambridge, Computer Laboratory, 1990. URL https://wwws cl. cam. ac uk/ Lechreports/UCAN-CL--T TR-C-M---------...... 2.R.209.ppp. pdd.
Mirco Mutti, Lorenzo Pratissoli, and Marcello Restelli. A Policy Gradient Method for Tgno- Explo- ration. arXi::2007.04640, 2020.
Hesan Nobakht and Yong Liu, Action space noise optimization as exploration in deterministic policy gradient for locomotion tasks. Applied Intelligence, 52(12):14218-1422) 2022.
Morteza Noshad, Kevin R. Moon, Salimeh Yasaei Sekeh, and Alfred O. Hero. Direct estimation of information divergence using nearest neighbor ratios. In IEEE International Symposium on Information Theory, pp. 903 907, 2017.
Cristina Pinneri, Shambhuraj Sawant, Scbastian Blacs, Jan Achterhold, Joerg Stueckler, Michal Rolinck, and Georg Martius. Sample-efficient cross-entropy method for real-time planning. In Conference on Robol Learning, volume 155, pp. 1049-1065, 2020.
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y. Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. In International Conference on Leurning Representations, 2018.
Vitchyr Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State- covering self-supervised reinforcement learning. In International Conference on Machine Learning, volume 119, pp.
2020.
Antonin Raffin. RL baselines3 2oo. HilHub reposilory, 2020. URL https://araffin. http::ii.iiitubb.oooooeeeee rl-baselines-zo0.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines:: Reliable reinforcement learning implementations. Journal of Machine Learning Re- search, 22(268):1-8, 2021a.
Antonin Raffin, Jens Kober, and Freek Stulp. Smooth exploration for robotic reinforcement learning. In Conference Robot Learning, volume 164, pp. 1634-1644, 2021b.
Philip H. Ramsey, Kyrstle Barrera, Pri Hachimine Hachinine-Semppebon.. and Chang- Chia Liu. Pairwise comparisons of means under realistic nonnormality, unequal variances, outliers and equal sample sizes. Journal of Statistical Compulation and Simulation, 81 (2): 2011.
Gary C. Ramseyer and Tie-Kia Tcheng. The Robustness of the Studentized Range Statistic to Violations of the Normality and Homogeneity of Variance Assumptions. American Educational Research Journal, (3):235 240, 1973.
Nimai Rao, Elie Aljalbout, Axel Sauer, and Sami Haddadin. How to Make Deep RL Work in Practice. arXiv:2010.1300 2020.
Derek C. Sauder and Christine E. DeMars. An Updated Recommendation for Multiple Comparisons. Ad- vances in Methods and Practices in Psychological Science, 2(1):26 44, 2019.
John Schulman, Sergey Levine, Philipp Moritz, Michael Jordan, and Pieter Abbecl. Trust Region Policy 2015.
Optimization. In International Conference on Machine Learning, pp.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Opti- mization Algorithms. Xiv 1rxi::17070033. 707.06347, 2017.
20

---

Published in Transactions on Machine Learning Research (11/2022)
Takuma Seno and Michita Imai. D3rlpy: An offiine deep reinforcement library. In NeurIPS 2021 Offline Reinforcement Learning Workshop, December 2021.
Tim Seyde, Igor Gilitschenski, Wilko Schwarting, Bartolomeo Stellato, Martin A. Riedmiller, Markus Wulfmeier, and Daniela Rus. Is bang-bang control all you need? Solving continuous control with bernoulli policies. In Advances in Neural Information Processing Systems, pp. #7209-27221, 2021.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Determin- istic policy gradient algorithms. In International Conference on Machine Learning, pp. 387-395, 2014.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy Gradient Methods for Reinforcement Learning with Function Approxination. In Advances in Newral Information Processing Systems, 1999.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAT Xi Chen, Yan Duan, John Schulman, Filip Delurck, and Pieter Abbeel. #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning. In Advances in Newral Information Processing Systems, pp. 2753 2762. 2017.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancg, Timothy Lillierap, and Martin Riedmiller. DeepMind Control Suite. 1801.0099000 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. TEEE/RSJ International Conference Intelligent Robots and Systems, pp. 5026-5033, 2012.
George E. Uhlenbeck and Leonard S. Ornstein. On the theory of the Brownian motion. Physical review, 36
(5): 1930.
Raphael Vallat. Pingouin: Statistics in Python. Journal of Open Source Software, 3(31):1026, 2018.
Patrick Nadeem Ward, Ariella Smofsky, and Avishek Joey Bose. Improving Exploration in Seft-Actor-Crii with Normalizing Flows Policics. 19o:00966.,771,,, 2019.
Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learming, 8(3-4):229 256, 1992.
Tianpei Yang, Hongyao Tang, Chenjia Bai, Jinyi Liu, Jianye Hao, Zhaopeng Meng. Peng Liu, and Zhen Wang. Exploration in Deep Reinforcement Learning: A Comprehensive Survey. arXix:2109.0600 066686668888888 [cs). July 2022. doi: 10.48550/arXiv (4466550//rXi.221000.66666
Zeping Zhan, Batu Aytemiz, and Adam M Smith. Taking the scenic route: Automatic exploration for videogames. In CEUR Workshop Proceedings, pp. 26 26-34, 2019.
21

---

Published in Transactions on Machine Learning Research (11/2022)
Appendix
Appendices
A A motivating example
The action is generated as å (s:), 0r āz + Ea.: where Eo, denotes the action noise. We calibrate the noise scale to achieve similar returns for both noise types. To calibrate the action noise scale, assume constant-zero-action policy upon which the action noise is added and effectively use = Eay as the action sequence. We find that a scale of about 0.6 for Gaussian action noise and a scale of about 0.5 for Ornstein- Uhlenbeck noise lead to a mean return of about -30. This is shown in Table 1. A successful solution to the Mountain- Car environment yields a positive return 0 < En < 100. We then use these two noise configurations and perform learning with DDPG, SAC and TD3. The resulting learning curves are shown in Figure and very clearly depict the huge impact the noise configuration has: with similar returns of the noise-only policies, we achieve substantially different learning results, either leading to failure or success the task.
To achieve a swing-up, the actions must not change direction too rapidly but rather need to change direction with the right frequency. Orostein-Uhlenbeet noise is lemporally correlated and thus helps solving the environment successfully with a smaller scale o. In this environment, the algorithms tend to converge either to the successful solution of the environment by swinging up, or to a passive zero-action solution which incurs no penalty.
B Boundary Artifacts
query point- ttth neighbour distance
Figure B.1: ke-nearett-neighbor density estimators suffer from boundary artifacts when estimating densities with bounded support. The density around a query point is estimated by the volume required include the k nearest points. Top right, high density region: the volume to include k-points is smaller when the density is high. bottem left, low density: in lower densities a larger volume is required to include & points. This also illustrates the boundary artifacts: when querying the density close to the support boundary, part of the query volume is outside the support. Thus the volume required to contain & points is over- estimated. This problem is amplified in higher dimensional spaces as the boundary artifacts occur as soon any single dimension of the sphere protrudes outside the support.
high density
low density
volume outside the support
C Action Noise in SAC
SAC as defined by (Haarnoja et al., 2019) does not use action noise for exploration. Instead, actions are sampled from stochastic Gaussian policy However, since SAC an off-policy algorithm, additive action noise can additionally be used. The SAC algorithun uses larget entropy parameter. The entropy coefficient of SAC is trained such that the average entropy of the Gaussian policy matches this target. In the implementation we use (Raffin et al., 2021a), the entropy target can be automatically chosen based on the size of the action space. In the Mountain-Car this amounts to a target entropy of 1. The entropy of a Gaussian is defined as Hy(() In (ov2xe). A = 1.7 approximately translates to an entropy target of 1.95.
In SAC the value function V contains an additional entropy bonus term: V(s)) = Ee.~ [Q(st, at) + HH((--1)))) This term is weighted by the entropy coefficient a. Additionally, the SAC policy defined as softmax operation over the Q function:
Agoftmax(a (4. expí (+).,))) where Z is a normalizing term, chosen s.t. f (a)stmax(( da = 1. Here, the entropy cocfficient a plays double role, in both the entropy bonus and the softness of the softmax operation. Thus, increasing the scale of the Gaussian has a direct influence on the smoothness of the softmax and can thus change the learning performance. Using action noise is independent of the softmax and can be tuned indopendently. Furthermore, action noise allows for the use of correlated noise process, which in the case of the Mountain- Car has a large beneficial influence. This explains why using action noise can be beneficial even for stochastic policies.
2(s.)
22

---

Published in Transactions on Machine Learning Research (11/2022)
Appendix
X
R
E
Environment
Algorithm
Type
Scale
Entropy Target
1.95
33
-2.85
-34
-2.66
auto
-7
-4.28
-7
-4.29
0.1
auto
-5
-4.15
-6
-4.22
0.5
auto
3
-2.94
-18
-3.93
Gauss
0.9
auto
17
-2.27
-18
-3.55
Mountain-Cas SAC
1.3
auto
23
-2.06
-21
-3.38
1.7
auto
24
-1.97
-25
-3.34
0.1
auto
-1
-3.94
-3
-4.11
0.5
auto
51
-180
37
2.62
OU
0.9
auto
68
-1.49
53
-2.22
1.3
auto
72
-1.22
57
-2.21
1.7
auto
73
-1.39
57
-2.14
Table C.1: Comparison of SAC with Action noise against SAC relying the stochastic policy for exploration. Increasing the entropy larget increases the state space coverage.
D Deterministic SAC
Algorithm D.1 (Deterministic) Soft Actor-Critic
Initialize parameter vectors vi v., 0, d.
for each iteration do for each environment step do
Me, fa(st) Ex ~ A a: = 14 + Ta(-18)) -N(-1ee,0)) ai (-(())))
D. A... action noise process
D. DetSAC
SAC
a: di t
8+11 ~ p(8441186,00 ae)
D DU (8), de, ((s,, a.), 8(-1)
end for for each gradient step do
... original SAC update (Haarnoja et al., 2019)
end for end for
23

---

Published in Transactions on Machine Learning Research (11/2022)
Appendix
E
Benchmark Environments
ENVIRONMENT
ILLUSTRATION
dim Odlim (AREWARD
Mounlain-Car
2 1
1 (8))8G)
- (a 22 2
Inverted- Pondulum-Svinguup
1
((((s)) pal1
Reacher
2
V 18 - sal2 -1p(s;)1 1(p(()); PPimitt - lael1
Hopper
)
15
3
8t scl1
-lp(se)1 1(p((s)), (Plmit)) -larl1
Walker2D
2
22
6
18 V sal1
-1p(())1 (pp(s)); (Pimmt)) - -lae/
Half- Calf-Cheetahh
26
6
V- 1sc sal1 -lp(sc)1 1(p(s))) (Plmmt))
-la
Table E.1: Benchmarks environments used in our evaluation in increasing order of complexity. |O] denotes Observa- tion space dimensions. |A] denotes Action space dimensions. Explanation Reward components: 1 c) indicator function (sparse reward penalty) of b w.r.t. the set e; [b/m n-norm of b; so(s:) angular component of state; v b finiteddiflerence reduction of & between time-steps; Fmax joint limit; sg goal state; (p(()) denotes an angular- power-penally Factors in the reward omilted. Distances e.g. sclo may refer to subspace of the veclor s:. Section 3.3
24

---

Published in Transactions on Machine Learning Research (11/2022)
Appendix
F
Statistical Methods
Section
Where
Statistic
n/N
non-Normal
Variance
Section 4.1
Figure 5 Table
Welch t-Test
2400/4800
CLT
Robust
Welch t-Test
400/800
CLT
Robust
Section G
Table G.1
Mann- Whitney-U Test
400/800
Robust Robust
Robust
Section 4.3
Table 4
Levene's Test.
800/2400
Table 4
Games-Howell--T -owell-Test Test
800/2400
a 0.2
Robust
Section 5
Table 6 (All) Scheduler
Games Howell-TTess Test
4800 14400
a 0.2
Robust
Section 5
Table 6 (All) Scale
Games-Howell-
2880/ 14400
a 0.2
Robust
Section 5
Table 6 (All) Type
Games- Howell-
3600/ /14400
a 0..
Robust
Section 5
Table (Env) Scheduler
Games- Howell-Tess
800/2 2400
a 0.2
Robust
Section 5
Table (Env) Scale
Games- Howell-Tesl
480/2 /400
Robust
Section 5
Table 6 (Env) Type
Games-Howell Howell-Test
1200/ /200
a 0.2
Robust
Table F.1: Summary of applied tests, per group sample size and cumulative size across groups N, see Section F.1
about the a adjustment in the nowell test. For large sample sizes the t-statistic approaches a normal dis- tribution (CLT). Sample sizes of 30 (Boneau, 1960) usually assumed be large enough. (Lumley al., 2002) provide further evidence for the adequacy our sample sizes.
F.1 Statistical Methods Details
We use statistical methods implemented in (Jones al., 2001; Vallat, 2018) well as our own implemen- tations.
Welch t-test: does not assume equal variance. Reporting two-tailed p-value. Significant for one-tailed when <a.
Games- test Performing multiple comparisons with a t-test increases the risk of Type I errors. To control for Type I errors, the HameslHowell test (Games & Howell, 1976), a coltipte-comparison test applicable to cases with helerogeneity of variance, should be used (Sauder & DeMars, 2019). Sample sizes should be
in each group.
The test statistic t is distributed according to Tukey's studentized range (Games & Howell, 1976) describe that the Lest. has been found to be robust to non-normality by (Ramseyer & Tcheng, 1973), especially in the case of equal sample sizes. This holds our case. (Ramsey et al., 2011) have found the concurrent violation of homogeneity of variance and non-normality can increase type-I errors. Their results indicate that an error level of 0.05 can be achieved by applying a reduction of the significance level of a and find controlling for this error by reducing the significance level to 0.38a. Further evidence for reducing the significance threshold to 0.01 in order to achieve error rates < 0.05 is provided by (Ramseyer & Tcheng, 1973).
F; F.j
(16)
σ
2
b
(17)
71
+
df
(18)
22
2
10g
(19)
25

---

Published in Transactions on Machine Learning Research (11/2022)
Appendix
The p-value is then calculated for l sample-grous groups
0t./2.k.df
(20)
ANOVA We perform balanced N-way ANOVA, i.e. with N independent factors, each with multiple levels (categorical values). Since the study design is balanced this is equivalent to a type-I ANOVA in which the order terms does not matter (because the design is balanced).
Eta squared ny² The effect size eta squared 1/2 denotes the relative variance explained by a factor to the total variance observed: xf2
SSc(a)
SSToul
DF
Sum of Squares
F PR(>F)
Cly)
9.0
4167.583
478.576
0
C(x)
9.0
91.118
10.463
1.7e-15
C(y):C(x)
81.0
81.172
1.036
0.397
Residual
901.0
871.798
Total
5211.67
Table F.2: ANOVA example. The partial xy² for factor calculated the sum of squares, variance explained by that factor, divided by the sum of the variance explained plus the unexplained residual variance.
Effect sizes are interpreted as:
n 2 0.01
small effect
(21)
27² 2 0.06
medium effect
(22)
17² 2 0.14
large effect
(23)
(24)
Levene's Test assesses (unjequality of group variances.
2g Vij ū,1
(25)
F N Σ3 12(2, 8)2
(26)
p 1 Σ Σ 1h (zis 21)²
dj 1p-1 -
(27)
d2 =N
(28)
2
(29)
n
zij
(30)
where p is the number of groups, n, is the size of group and N is the total number of observations. Ú is the median of group 2ij denotes sample in group The F statistic follows the F-distribution with degrees of freedom d1, d2.
This variant of Levene's test, Dj median instead of mean, is also called Brown-Forsythe test (Brown & Alan B. Forsythe, 1974) and is more robust to non non-normal distributions.
Cohen-d effect size Cohen-d is illustrated in Figure F.1 and measures the distance of the means of sample groups normalized to the pooled variance:
26

---

Published in Transactions on Machine Learning Research (11/2022)
Appendix
d = 2.00
d = 1.40
d = 0.80 Large
0.4
A
PDF(x)
B
PDF(x)
PDF(x)
B
98%
92%
79% %
-2
4
-2
-2
2
4
X
X
X
d = 0.50 Medium
= 0.20 Small
d = 0.00
0.4
0.4
0.4
A
PDF(
PDF(X)
B
B
B
0.2
0.2
PDR
0.2
69%
58%
50%
0.0
-2
-2
-2
2
X
X
X
Figure F.1: Illustration of Cohen-d effect size: the Cohen-d mensures the standardized difference between the means of two groups, equivalent z-score. Effect sizes d 2 0.2 are called small, 2 0.5 medium, 0.8 large effects. Under equal-variance Gaussian assumption this can be interpreted repercent of group A below the mean of group B. Illustrated as the shaded area.
[Maan Group A] - [Mean Group Bj
Effect size
(31)
Pooled Std Deviation
d T1 22
(32)
(n1 1)s + (n2 - 1)s²
(33)
112
G
(Q1) Which action noise type to use? Mann-Whitney-U Test
Environment
P
Pp
dp
R
pn
dn
X
Px
dx
PE
dg
Half- Cheetah
0.08
0.86
OU 0.005
0.21
0.28
Hopper
OU <10-5 0.27
G <10 0.29
G <10-10
0.41
0.69
Inverted-
0.040
G <10 1.15
OU <10-47
1.22
0.32
Pendulum-Swinguu
Mountain- Car
OU <10-10 10 0.47
OU <10-15 0.66
OU <10-11 0.34
OU <10 21
0.71
Reacher
G <2 0.87
G <10-21
0.80
OU <10-35 1.01
OU <10- 26 0.84
Walker2D
OU < 5
0.15
G 0.006
0.18
OU 13 0.46
G <10 0.08
Table G.1: Comparison of noise types in Table 2, p-values of Mann- Mann-Whitney- U test reported instead of the Welch-t-test. Similar tendencies are shown.
27

---

Published in Transactions on Machine Learning Research (11/2022)
Appendix
H Impact of Scheduler on Variance and Learned Performance
var(
P
Constant
Linear
Logistie
Constant
Lineur
Logiatic
Scherluler
Envname
HalffChhettab
No
No
No
No
Hopper
No
No
No
No
Inverted Pendulunnmd Pendulum-Swinguu
No
No
No
Yes
10-5
Coustant
Mountain-Cas Car
No
No
Yes
10-
No
Reacher
No
No
No
No
Walker2D
No
No
No
No
Half-Cheetah
Yes p < 10-3
No
Yes p < 10-6
No
Нopper
Yes p < 10-5 No
No
Yes p < 10-6
No
Inverted Pendulum Pendulum-Swinguu Swingup
No
No
No
Linear
Mountain Car
Na
No
Reacher
Yes < 10-29 Yes p < 10-1
Yes p < 10-15
Yes p 1000
Yes p 10-6
Walker2D
Yes p < 10-0
Yes p 10-6
Half-Cheetall
Yes p 10-4
Yes ps 10-6
No
Hopper
Yes p 10-3 No
Yes p < 10-8
No
Inverted- Invertodi-eendulnn Pendulum-SSwnngeee
No
No
No
Logistic
Mountain Car
No
Yes 0.002
Yes 10-5
Yes 10-5
Reacher
Yes p 10
No
No
No
Walker2D
Yes p 10-6
No
Yes 10-4
No
Coustant
Sum
0
0
0
1
1
Linear
Sum
4
4
0
2
Logistic
Sum
4
4
0
Table H.1: In the majority of cases, using scheduler reduces variance var(P) of the performance (evaluation returns), and improves expected performance The rows shows whether "Scheduler" significantly better than the scheduler indicated in the columns var(P) and P. The evaluation returns P are standardized to control for the influence of algorithm, noise scale and noise type. Levene's lest used to assess difference variance var(P) and multiple- comparison Ganes- Howell test indicates superior performance P.
Performed experiments
This section lists the achieved final returns, calculated as the average of the evaluation returns of the last 5 out of our 100 training segments, for each noise setting, for the constant Table I.1, linear Table I.2, and logistic Table I.1 schedulers.
Each noise configuration is repeated with 20 random seods. Table L.4 lists the performance P for cach noise configuration as the mean across the 20 seeds.
Choosing fixed set of 20 different seeds could introduce randomness artifacts, making one algorithm appeor to perform better than the other. To combat these biases, each run performed from independently, randomly drawn seed. The seeds are sampled using urandom, which provides string of random byles suitable for cryptographic use. This should be sufficient to ensure independence between seeds.
28

---

Published in Transactions on Machine Learning Research (11/2022)
Appendix
Retur
Scale Type
0.1
0.5
0.9
1.3
1.7
0.1
0.5
0.9
1.3
1.7
Gauss
Gauss
Gauss
Gauss
Gauss
OU
OU
OU
QU
OU
Scheduler Environment
Algorithm
DDPG
179
269
392
417
341
165
214
279
293
289
DetSAC
1719
1741
867
587
590
1619
1578
1156
787
731
Half-Cheetal
SAC
1702
1856
531
389
292
1906
1550
1139
766
680
TD3
1891
1651
1158
856
701
1582
1470
975
940
722
DDPG
1113
541
348
344
276
1131
808
666
307
461
DeSACC
2170
1101
837
572
542
2159
1602
1345
906
Hopper
979
808
717
604
2298
1599
992
1833
1370
2258
1164
816
879
877
Inverted Pendulum Cenduluu-Swingup
DetSAC
880
886
SAC
881
887
883
888
888
889
887
Constant
TD3
868
883
881
883
873
879
882
884
886
DDPG
-0
67
-0
56
DetAAC
47
78
13
94
Mountain- Car
SAC
23
42
GO
52
92
94
94
TD3
94
65
74
74
-D
94
74
74
94
DDPG
17
18
18
18
17
16
15
14
13
13
DecSAC
19
19
19
19
19
18
18
17
16
15
Reacher
SAC
18
18
17
16
16
18
17
16
15
13
TD3
17
16
15
15
14
17
16
15
15
14
DDPG
787
519
391
263
321
849
759
443
471
465
DetSAC
1814
1211
421
305
311
1616
1139
740
661
641
Walker2D
SAC
1883
1233
448
392
383
1710
1068
712
691
641
TD3
2001
1878
1321
1228
1821
1870
1596
1353
1230
Table 1.1: This table shows the final evaluation return (mean over last five percent training) for each configuration under the Constant regime. The mean across all 20 runs reported.
Return
Scale
0.5
0.9
1.7
0.9
1.3
1.7
Type
Gauss
Gauss
Gauss
Gauss
Gauss
OU
OU
OU
OU
OU
Scheduler
Environment
Algorithm
DDPG
158
155
151
150
1-12
142
138
127
DetSAC
1826
2538
1131
888
924
1547
1989
2062
1564
1657
Cale-Cheetahh
SAC
1991
2471
1331
791
860
2249
2217
2062
1784
1735
TD3
1957
2366
2031
1658
1731
1976
2069
2168
1881
1339
DDPG
1039
1071
883
864
790
965
937
999
964
871
DecSAC
2276
1923
1634
15:00
1318
2308
1913
1702
1484
1297
Hopper
SAC
2288
1840
905
814
789
2208
1899
1697
1579
1341
TD3
2007
2107
1088
1581
1610
2099
2051
1943
1818
1633
DDPG
836
830
832
838
835
823
817
829
830
750
DetAAC
867
852
885
878
882
778
887
887
888
888
SAC
884
883
879
881
888
879
888
886
888
887
Linear
TD3
8G6
857
836
845
830
850
859
800
868
DDPG
-0
56
46
31
65
-0
67
26
45
DecACC
13
28
38
33
95
Mountain- Car
SAC
-1
37
71
95
DDPG
16
16
16
16
Reacher
DesSAC
19
19
SAC
18
18
TDS
17
18
19
19
17
17
DDPG
791
755
635
360
661
642
460
Walker2D
De
1602
1348
1048
970
894
1656
1276
1167
1014
876
SAC
1678
1630
845
637
652
1859
1334
1161
945
1028
TD3
1780
1660
1584
1254
1238
1820
1766
1765
1647
1622
Table I.2: This table shows the final evaluation return (mean over last five percent of training) for each configuration under the Linear scheduler regime. The mean across all 20 runs is reported.
29

---

Published in Transactions on Machine Learning Research (11/2022)
Appendix
Return
Scale
0.1
0.5
0.9
1.3
1.7
0.1
0.5
0.9
1.3
1.7
Type
Gauss
Gauss
Gauss
Gauss
Gauss
OU
OU
OU
OU
OU
Scheduler Environment
Algorithm
DDPG
158
156
143
136
124
180
153
129
128
108
DetSAC
1639
2168
1923
1539
1208
1680
2100
1921
1719
1235
Half-Cheetal
SAC
1995
2199
1984
1746
1413
2158
2217
1927
1613
1511
TD3
1694
2329
1864
1373
1363
2030
2047
1954
2002
1682
DDPG
902
1181
1103
1246
1147
1104
1057
1059
1207
1153
DeSAC
2237
1824
1691
1533
1338
2250
1822
1622
1482
1314
Hopper
2130
1888
1485
1385
2239
1855
1469
1257
1736
2037
1759
841
832
747
Inverted Pendulum Senduuuun-winnuu
DetSAC
887
SAC
888
888
889
888
888
Logistic
TD3
860
847
846
852
869
875
DDPG
36
45
-0
45
DetSAC
95
95
52
95
95
Mountain-CC Car
SAC
85
95
95
85
95
95
TD3
-0
5
52
94
94
-D
94
79
84
84
DDPG
16
16
15
15
15
16
16
15
15
15
DecSAC
18
17
17
17
17
19
17
17
17
Reacher
SAC
18
17
17
17
16
18
18
17
17
17
TD3
17
19
19
19
19
17
17
16
16
16
DDPG
905
792
659
426
341
792
885
676
457
324
DetSAC
1701
1371
986
891
789
1526
1201
923
914
767
Walker2D
SAC
1642
1274
1018
882
837
1676
1418
1148
850
805
TD3
1759
1622
1294
1196
879
1842
1093
1611
1607
1630
Table L.3: This table shows the final evaluation return (mean over last five percent training) for each configuration under the Logistic scheduler regime. The mean across all 20 runs is reported.
30

---

Published in Transactions on Machine Learning Research (11/2022)
Appendix
Seale
01.1
0.5
0.9
1.3
1.7
0.1
0.5
0.9
1.3
Type
Ganes
Gauss
Couss
Gonse
Garse
0U
Schedillrr Eavironment
Algorithm
DDPG
192
218
343
322
259
174
204
254
238
240
HalF-Cherala
SAC
1109
1413
1477
437
354
1158
1102
670
621
TD3
1284
1179
855
690
605
1053
1052
760
770
626
DDPG
222
521
Happer
DetAAC
1987
1100
814
671
632
1927
1458
1215
903
H1T
SAC
1976
1106
813
746
600
2011
1661
1915
006
813
1575
963
DDPG
738
751
746
748
740
738
700
790
153
760
Invertedi- Pww.lllom--Singgee
DetAAC
703
857
855
858
864
815
843
840
842
842
527
Coustant
T13
703
763
748
761
749
706
738
749
705
761
DDPG
444
62
50
-0
67
80
53
Mountain Car
SAC
17
23
24
-1
51
688
72
73
TD3
58
TO
-0
87
Reacher
DetSAC
SAC
17
18
17
17
16
15
16
10
DDPG
494
367
250
243
239
408
455
327
336
346
WalkeeD
DeSACC
1324
1103
1034
448
901
203
367
1176
1290
846
674
600
504
TD3
1504
1419
1098
842
730
1491
1440
11:77
885
7I7
DDPG
172
178
187
177
163
187
162
185
168
167
907
Hall-Cherals
SAC
1218
1840
900
654
122
TD3
1300
1727
1390
1099
1076
1288
DDPG
198
577
594
Hupper
DetSAC
SAC
2076
1966
1551
1654
1302
783
1174
713
701
977
2006
1977
1000
1007
1916
1326
1117
1121
920
1688
1315
1177
DDPG
713
749
707
756
748
729
753
750
751
671
Inverted- Luvertedd-enncllann Pww.eulnn-SStiieeeeee
DaSACC
818
861
857
857
850
774
838
840
840
Linear
TD3
606
748
746
745
740
720
746
745
735
747
DDPG
46
39
28
-0
61
40
Mountsin-Car
SAC
-3
16
7
-3
45
10
72
TD3
40
Reacher
DeIAAC
SAC
16
17
17
18
18
18
17
16
15
16
13
11
11
15
DDPG
484
467
572
203
266
500
406
371
931
272
WalkerDD
DetAAC
1233
976
742
705
1206
890
757
689
1206
1291
TD3
1477
1384
918
665
812
1457
1397
1273
1176
1072
DDPG
176
108
106
196
162
182
177
180
167
166
1350
196
Hall-Chera's
SAC
1269
1433
1084
943
1111
TD3
1224
1700
1202
945
15
1960
1221
1136
DDPG
Happer
DeIACC
SAC
2064
1911
1543
1633
1961
1506
1139
1006
1000
988
2027
2067
1880
1226
1522
1228
1073
1082
962
941
1827
1337
932
939
1803
1030
1104
DDPG
726
736
710
757
689
656
748
770
676
739
Inverted- Invertooi-Penccllaee www.eelln----iiieeeeeee
DeSAAC
704
841
838
842
843
846
845
843
640
Logistic
TD3
717
753
736
753
750
712
752
751
T66
740
DDPG
52
-0
66
42
Mountain-
SAC
-0
52
62
B8
-4
48
72
70
72
TD3
Reacher
DOISAC
16
14
13
12
16
16
15
11
SAC
16
12
15
10
15
17
13
DDPG
508
468
352
260
257
408
462
350
267
242
Walker21)
DeSAAC
1206
952
751
688
11:06
732
099
650
935
652
1207
TD3
1412
1313
672
608
476
1400
1326
1151
1000
982
Table L4:: This table shows the performance as the mean across the different seeds.
31

---

Published in Transactions on Machine Learning Research (11/2022)
Appendix
J Hyperparameters
Environment
Walker2D
Inverted- Pendulum- Swingup
Hlopper
Mountain-Col
Half-Choetah
Reacher
wrapper Femuuee
Temuueature
TimeFeature Wrapper
Wrapper
Wrapper
gamma
0.99
0.99
0.99
0.99
0.99
buller size 1000000
1000000
1000000
50000
1000000
1000000
learning starts
1000
1000
1000
0
10000
1000
gradient. steps
-1
-1
train freq
(1, 'eplsode")
(1, 'episode")
(1, 'episode")
(1, 'episode")
(1, 'episode")
leurning rate
0.0003
0.0003
0.0003
timesteps
2000000
1000000
2000000
60000
2000000
1000000
ID
Walker2DBuleet Env-vD
HopperPyBullet Env-vül
MountainCar Continsous-vi
HalfCheetab
ReacherPyBullet
Pendulum Suuliee Env-vD
PyBulleEEnv-vo
Env-v0
batch size
256
64
256
64
256
64
ent cooe
0
0
auto
auto
0
lau
0.005
0.005
0.005
0.005
0
0.005
SAC/DetSAC Hyperparamekers
Environment
Walker2D
Inverted- Pendulum-
Hopper
Mountain-Car
Half-Cheetah
Reacher
Swingup
env wrapper
TimeFeature
TimeFeature
TemeFeature
TimeFeature Wrapper
TimeFeature Wrapper
Wrapper
Wrapper
Wrapper
gamma
0.99
1
buffer 200000
200000
200000
1000000
200000
200000
learning starts
100
10000
10000
gradient steps
-1
urain freg
(1. 'episode')
(1, 'episode")
(1. 'episodee)
(1. 'episode')
(1. 'episode")
(1. 'episode')
learning rate
0.001
0.001
0
0.001
0.001
0.001
policy kwargs
[ ['net_an arch':
('net_arch's arch':
('ne arch':
None
{ ('net _accht
['net_ aach'
100, 300)} 1000000
4000 300]}
100, 300)} 1000000
400, 300]
400, 300)
timesteps
300000
300000
1000000
300000
ID
Waller2DBlllet
Inverted
HopperPy Bullet
MountainClar Continuous-v0
HalfCheatah
ReacherPyBulle
Env-vo
Pendulum
Env-v0
En
Env-v0
SwingupPyBullet
Env-vO
TD3 Hyperparameters
Environment
WDlker2D
Inverted- Pendulum- Swingup
Hopper
Mountain-Caa
Half-Cheetah
Reacher
env env__ wrapper
TimeFrature Wrapper
Fenture Wrapper
Feature Wrapper
Festure Wrapper
TimeFeature Wrapper
ganma
0.99
1
buffer size
1000000
200000
1000000 10000
1000000
200000
200000
learning starts
10000
10000
100
10000 -1
10000 -1
gradient__ster stops
-1
-1
-1
-1
trinn freg
(1, lepisodee)
(1, 'episode")
(1. 'episode')
(1, 'episode')
(1. 'episode")
1
learning rate
0.0007
0.001
0.0007
0.001
0.001
0.001
policy kwargs
nnet arch':
( ('net_ arch': [400, 300]
(nnt aarch::
None
('net arch':
peet'arch''
400, 300)} 1000000
400, 300] 1000000
400, 300]
400, 300]
timestops
300000
1000000
300000
ID
Waller2DBullet
Hopper Bullet Env-v0
MountainClr Consinuous-v0
PaacherryBiulledd
Env-v
Pendulum Swilge Env-v0
En
Env-v0
botch size
256
100
256
100
100
100
DDPG Hyperparameters
Table J.1: Hyperparameters for SAC, TD3 and DDPG taken from (Raffin, 2020) left default values defined in (Ralfin et al., 2021a).
32

---

Published in Transactions on Machine Learning Research (11/2022)
Appendix
K Environment Limits
Barinneunt
MoratoisCCrraniinn Can
HoppePyBellaease
Wollee2DBali
-1.20000.. 6.6000
-10005... 1.6011
-0.2000..0.33
0.2000
-1.3453...
0.5614
-1.25.6...0.D
-C.6512...
400700 0.000
.6.1296....2000 6.200
-0.2703 .027799....30 0.2700
-00500
(1.0000
0..00....00 0.0000
-0.0000)
(00000)
-1.000...1.00000 1.6000
-0 -0.1799 0.4158
-10.0
1.0000
-1.000
1..00..1..000
-1.000
1.0000
590
00,7799..
-5000
3.4273
-.512
-21900....
1.0000
4,0000..
1.0000...110000
-5.0000....
-3.400..
-.9658... 1.8548
-100003
-3.111.
0.0000
1414
0.000
11254... 1,2701
-1,0700
1.5342
115708
1.0025
1.0708
1.050
-1000000
10.0000
-1.3911
2.1082
22,277
1.5482
-3,0000
1.18
-5.0000... 5,0003
-1,0000
4.6218
3.6661.. 8.8614
-1.3917 1,2215
+1.566
1.8112
2.1550
20
-5,0000.... 5,000
-.222
550000.
4.361.....122
1.00000...
-00000
1,0000
22,370
1.2012
1,6122
-1.679
4.1316
-5,0000
4.0066
-1.2236
1.7002
-5,0000.. 8,2067
-3.235
4.2251
.5.000...5500 5.0000
g/t
4.066
8.3224
205
1.0000
4,0000 1.0000
-0,0000 ... 1.0000
-0.0000 1.00000
-0,0000 1.00000
(00)
1,0000
Table K.1: The calculation of Xusa requires defined state space limits for each environments. However, some environments define the limits as 00). In these cases we collected state space samples and defined the limits empirically.
33


Reviews:
[{"review_id": "nSF007gGqb", "reviewer": "Reviewer_jTC2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1. Well written and mostly clear. Very easy to read!\n1. Empirical investigation is thorough and seems principled.\n1. Design decisions are mostly explained and justified.\n\n**Weaknesses:**\n1. Is using action noise for exploration even a good idea? It's unlikely to be the best solution to the exploration problem and is more of a stand-in for principled exploration strategies. On the other hand, it is commonly used and strongly affects the performance of the algorithms that use it, so might be worth exploring.\n1. The findings of the paper are not necessarily surprising, and some of the recommendations seem to be common sense. Hyperparameters are often already set on a per-environment basis. Reducing exploration (in this case implemented as action noise) over time is common, although apparently not in the existing software packages cited in the paper. Likewise, it's unsurprising that the scale of the noise has the largest impact on performance of the factors considered. However, the paper's thorough empirical investigation gives concrete evidence to support this folk wisdom, and is a helpful contribution, especially for practitioners looking to use the studied algorithms on new environments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper conducts a thorough empirical investigation of the effects of various configurations of action noise (scale, type, reduction schedule) on performance for several deep RL algorithms and environments in the continuous control setting. The paper also proposes a measure of state space coverage with desirable properties, and gives two estimators that implement the proposed measure. In addition, the paper gives several helpful recommendations for choosing action noise parameters, that are rooted in the experiment results.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "**Critical changes:**\n1. The Related Work section doesn't actually mention any papers that empirically examine the effects of action noise on algorithm performance, even for different algorithms or settings (like epsilon-greedy in the finite action setting, which is instead discussed in the introduction). It actually seems more like a background section that introduces the concepts that the paper builds on. I recommend changing the name of the Related Work section to \"Background\", and including a Related Work section after the introduction that summarizes the works that are most closely related to the main point of the paper (i.e., any papers that study action noise, even in another setting or algorithm). This section should also compare and contrast the findings of related works to better position this paper's contributions within the literature. For example, do the findings of the current paper echo findings from the finite-action setting, or are there important/surprising differences?\n1. It wasn't clear to me what \"boundary artifacts\" meant until I read section 3.6. Then it wasn't clear to me how they are meaningfully different from \"outliers\"; whether the space is bounded or not, values far from the others can disproportionately affect certain estimators. Then after seeing Figure 4, I think I understood better what is meant by \"boundary artifacts\" and how they could be different from outliers (e.g., the clipping process causing samples to concentrate around the boundaries instead of the interior of the space). I would suggest either defining \"boundary artifacts\", giving examples, or coming up with a name that better communicates the cause of the phenomena (i.e., \"clipping artifacts\"). Earlier in the paper when it's first used would be ideal.\n1. The caption for figure 4 (and later figures) repeats details that were stated in the main text just before it. I would suggest moving all details for all figures to the main text, and having more succinct captions that include details related to the specific figure (e.g., \"best performance in bold\", \"shaded regions are standard error\", etc.), and a summary of the main findings for that figure (similar to the captions of figures 4(a) and 4(b), which I liked a lot).\n\n**Changes that would strengthen the work in my view:**\n1. In the paragraph following \"Findings\", the paper states \"Larger noise scales tend to increase state-space coverage, but for the majority of our investigated environments increasing the state-space coverage is not beneficial.\" It seems like increasing the state-space coverage specifically by increasing the action noise scale is not beneficial, but more principled ways of increasing the state-space coverage could still be beneficial. It would be good to clarify this statement.\n1. Should \"In case of action noise\" be \"In the case of action noise\"?\n1. It would be good to be very clear that the version of Mountain Car described in the paper is not the usual version from OpenAI Gym.\n1. The square root symbol in equation 3 only extends to the left parenthesis.\n1. Why sample noise from an unbounded Gaussian and clip values instead of sampling from a truncated Gaussian distribution directly? I understand that this setup is something others have done in the past, but it would be good to justify building on top of a bad legacy, perhaps by appealing to practitioners looking to use and understand existing methods.\n1. It's good to use the past tense to communicate to the reader what was actually done in the experiments, and save the present tense for statements that are generally true and conclusions that are being drawn from the experiments.\n1. The first sentence of section 3.5 is a little confusing as is. It says \"we...evaluate the exploration and learned policy performance once for each...segment\", which made me think one rollout was performed, but then in the next sentence the paper states that either 100 episodes or 10000 steps were performed. I would recommend rephrasing the first sentence to make it clearer, maybe by just removing \"once\".\n1. The discussion at the bottom of page 8 feels like it jumps back and forth a few times describing the kNN and NNR estimators. It might be better to introduce both estimators in one sentence, fully describe one estimator (1 paragraph), and then fully describe the other (another paragraph). That way, when comparisons between the estimators are made, the reader already knows what they need to know to understand the comparison.\n1. It would be good to explain why C_L and C_U were chosen how they were (1/n and n/1, respectively); as it is these choices don't make a ton of intuitive sense to me.\n1. It wasn't clear to me what was meant by \"the environment acts as an integrator over the actions.\" It could be good to elaborate.\n1. In the caption of Table 5, the first letter of the second sentence should be capitalized.\n\n**Questions:**\n1. The description of the experiments (top of page 7) says they \"would\" amount to 244 node-days with the specified hardware setup. It's always better to describe what was actually done. Did the experiments take 244 days to run, or were they run on a cluster?\n1. Why add action noise to SAC? It already randomizes its actions by maximizing the entropy of the policy in addition to maximizing value, which seems like a better way of achieving the same goal as adding action noise.\nMaximizing policy entropy as well as value allows the agent to act randomly when it makes sense, but act less randomly when it would be harmful. Adding action noise seems redundant; it would make more sense to increase the entropy hyperparameter to cause the agent to act more randomly.\n1. Why was the evaluation for each segment done the (seemingly complex) way it is (i.e., 100 episodes or 10000 steps, whichever is reached first, using only complete episodes)? Is it due to the nature of the environments (e.g., some episodic, some continuing)? It's always better to give exactly equal amounts of experience to each algorithm to aid comparisons; I can't tell if that is being done here or not. At the very least it would be good to justify or explain why this evaluation methodology was chosen.\n1. I like the symmetric KL-divergence-based measure of state space coverage; it seems much closer to what we actually want, with reasonable added complexity. Have you considered something like the earth mover's distance/Wasserstein metric (might be overly-complicated/overkill)?\n\n**Misc comments:**\n1. I greatly appreciated the explanation justifying the choice of noise scale parameter values.\n1. Thank you for explaining the reasoning behind using the rollouts instead of the replay buffer to measure returns and state space coverage; it was very helpful.\n1. Likewise, Figure 3 was very helpful for getting an intuitive understanding of each of the measures of state space coverage. Thank you for taking the time and effort to create and include it.\n1. Again, thank you for justifying/explaining the logic behind the KL-divergence based measure.\n1. I liked how the recommendations are stated (pages 15-16), and then explained. This should be really easy for readers to understand, and for practitioners to implement.\n", "broader_impact_concerns": "Doesn't seem applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cDJniLJpzh", "reviewer": "Reviewer_oEpB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Main Argument\nOverall, the paper provides a much needed analysis on the effects of action noise in deep actor-critic algorithms. The effects of both Gaussian and OU noise on deep actor-critic algorithms have not been thoroughly studied, and this paper provides a comprehensive analysis of this topic.\n\nUnfortunately, a number of empirical design choices cause me to question the overall results of the paper.\n\nFirst, the paper uses multiple different types of statistical significance tests. These tests each have their own assumptions on the form of the data to which they are applied. The paper does not comment on whether or not such assumptions are satisfied. For example, the t-test assumes data from a normal distribution (otherwise a normal sampling distribution for the mean, but this would be difficult to verify), yet the paper does not verify if this assumption is satisfied. Similar issues exist for the other tests which are utilized in the paper. Does the data satisfy the assumptions for each statistical significance test? If not, then the reported results of such tests may be overly optimistic. \n\nThe empirical study uses hyperparameters tuned for a specific action noise scale and schedule. Then re-uses these same hyperparameters when changing the action noise scale and schedule. For example in the Stable Baselines codebase, TD3 and DDPG use the hyperparameters for TD3 reported in [1], which were tuned for Gaussian noise with noise scale $\\sigma$ and a constant noise scheduler.  All other hyperparameters of TD3 have been tuned to work well with this noise type, scale, and schedule.\n\nIn the experiments outlined in this paper, hyperparameters are not re-tuned after changing the noise type, scale, or schedule. Instead, the default hyperparameters tuned for the original noise type, scale, and schedule are re-used, but these hyperparameters may no longer be appropriate for the altered action noise.  For example, since the step-sizes were tuned for TD3 using Gaussian noise for exploration, simply substituting OU noise for Gaussian noise likely results in a pessimistic estimate of performance for OU noise in TD3.\n\nThis could bias results, giving the noise type, scale, and schedule that was originally tuned for an unfair advantage. For example, in Figure 5b, Gaussian noise results in higher (exploratory) rewards than does OU noise. But since the hyperparameters used were tuned for Gaussian noise, it is likely the case that Gaussian noise had an advantage over OU noise in this experiment. Had the hyperparameters additionally been tuned for OU noise, perhaps different conclusions would have been drawn. I understand that we cannot perform a systematic sweep over all possible hyperparameters, as this would be prohibitively expensive.  Nevertheless, this is a significant limitation of this work, and the work would be greatly improved with a more thorough consideration of hyperparameter selection.\n\nSection 3.6.1, provides an experiment on each method for measuring state space coverage. A number of data points are sampled from a 25-dimensional distribution. Then, the scale of these data points are changed to vary the amount of state space coverage, including causing many data points to lie on the boundaries of the state space. Although this is a useful experiment, only 10 random seeds were used to generate the results. From my understanding, these are quite cheap experiments to run, and so the choice to use so few random seeds confused me. Furthermore, why was only the mean plotted and not a confidence interval? Perhaps there is some part of the algorithm that I am mis-understanding.\n\nIn appendix G, the figure states that the random seed used for each experiment was drawn independently and randomly. What were the bounds and distribution from which these random seeds were drawn? Were the seeds sampled with or without replacement? Why was the same random seed not used for each algorithm for each separate experimental trial? This would ensure each algorithm saw the same initial weights, same environmental starting states, etc.\n\n[1] Scott Fujimoto, Herke van Hoof, David Meger. Addressing Function Approximation Error in Actor-Critic Methods. 2018.\n\n## Small things which did not affect the scoring of the paper:\n\n- Appendix E: The text in the figure is incredibly small, making it hard to read. Furthermore, the section would benefit from some analysis of the figure.\n- Appendix G: Table G.1 is unnecessary and doesn't convey any additional information. It suffices to say \"each experiment used 20 runs\"\n- Section 3.3: \"See Brockman et al. (2016), Coumans and Bai (2016), and Ellenberger (2018) for further details.\". You cannot see the references, and the sentence would read better as \"Brockman et al. (2016), Coumans and Bai (2016), and Ellenberger (2018) provide further details.\"\n- Section 2 (Related Work) states that deterministic policies can **only** be used in the off-policy setting. This is incorrect. A deterministic policy can be used in the on-policy setting, but is discouraged due to a lack of exploration.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "## Summary\nThis paper provides an analysis of the effects of using action noise for exploration in deep actor-critic algorithms. In particular, the paper studies how adding Gaussian and OU noise to actions affects multiple facets of Soft Actor-Critic (SAC), Deep Deterministic Policy Gradient (DDPG), and Twin Delayed Deep Deterministic Policy Gradient (TD3). These facets particularly are the return achieved and the state space coverage of the exploitation policy (where action noise is not used) and the exploration policy (where action noise is used). The overall conclusions are that action noise does affect the performance and state-space coverage attained by these algorithms, but that the type of action noise and the scale of action noise should be tuned for each environment. Furthermore, reducing the action noise throughout training seems to improve performance.  The paper introduces a new method to measure state-space coverage which addresses issues with current methods of measuring state space coverage. In particular, this novel method is less sensitive to data points lying along the state space boundaries.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "The paper should comment on whether or not the assumptions of each statistical significance test is satisfied. If the assumptions are not satisfied, then the experiments should instead use a weaker statistical significance test for which the assumptions are satisfied. This detail is critical to securing my recommendation for acceptance. \n\nTuning the other hyperparameters for each noise type, scale, and scheduler would significantly strengthen the work and be beneficial to the RL research community as a whole. Even if only a subset of all other hyperparameters were tuned, this would strengthen the work greatly. \nThis could even be accomplished by first tuning **across all environments** over a small number of runs, then evaluating the tuned hyperparameters across all environments using a larger number of runs.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xXUgH1a8i2", "reviewer": "Reviewer_7knq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe authors motivate their investigation with a clear example in Mountain Car. After selecting the scope of their investigation (what metrics/domain/algorithms), they do an honest empirical work to analyze their findings.\n\n\nWeaknesses:\n\nThe paper puts a lot of emphasis on state-space coverage to measure exploration efficacy, with a particular metric based on the divergence of a state distribution from a uniform distribution. It’s not clear to me that increasing that coverage is desirable by itself, and it also ignores the distribution over actions which is important for driving policy improvements.\n \nA lot of space in the paper is given to description of standard methods (e.g. Eq 10-15, all of Appendix D) while comparatively less attention is given to details of the RL experimental setup. Section 3.5 describes what data is collected, but it’s not clear how that maps to the evaluation returns P and exploratory returns R for example.\n\nOverall the empirical results are not particularly enlightening. Fig 5 or 7 for example illustrates that there is really little difference in aggregate between the two noise processes across the different metrics that were chosen (even if the means are statistically different, the difference is minor). It’s not clear what to take away from the paper and the results. The finding that the best noise process depends on the domain and the algorithm is not surprising in itself, and the generality of the heuristics in Section 5 seem rather limited (it’s based on 6 toy domains and relies on vague conditions like “reward is misleading”). I have trouble seeing these results generalize to more complex settings, and for this reason I don’t think the community will benefit from these findings - even if they are valid.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors compare the role of two popular noise processes (Gaussian / O-U process) employed for exploration in DeepRL continuous control work. They evaluate different ways of setting the hyperparameters and schedule for these noise processes in 6 Mujoco domain and 4 different algorithms. They conclude that no single setting of the parameters leads to better state-space coverage or better performance across domains.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "The paper relies on published codebases to comment about what has been explored and studied in the past (e.g. end of page 2). Please take the published literature into account rather than only available online code for comments about existing work. Case in point, reducing randomness over the course of learning is common both in practical algorithms and in theoretical RL work. \n\nCan the returns be provided in Table 2, it would be more interpretable than the statistics to give a notion of the performance gaps. In general, I found the large number of statistical tests to be distracting.\n\n", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agreed on accept as is.
Decision: Accept (poster)