Pink Noise Is All You Need: Colored Noise Exploration in Deep Reinforcement Learning
Authors: Onno Eberhard
Jakob Hollenstein
Cristina Pinneri
Georg Martius
Venue: ICLR 2023 notable top 25% (2023)
Keywords: reinforcement learning
exploration
action noise
continuous control

Abstract: In off-policy deep reinforcement learning with continuous action spaces, exploration is often implemented by injecting action noise into the action selection process. Popular algorithms based on stochastic policies, such as SAC or MPO, inject white noise by sampling actions from uncorrelated Gaussian distributions. In many tasks, however, white noise does not provide sufficient exploration, and temporally correlated noise is used instead. A common choice is Ornstein-Uhlenbeck (OU) noise, which is closely related to Brownian motion (red noise). Both red noise and white noise belong to the broad family of colored noise. In this work, we perform a comprehensive experimental evaluation on MPO and SAC to explore the effectiveness of other colors of noise as action noise. We find that pink noise, which is halfway between white and red noise, significantly outperforms white noise, OU noise, and other alternatives on a wide range of environments. Thus, we recommend it as the default choice for action noise in continuous control.


Published as a conference paper at ICLR 2023
PINK NOISE Is ALL YOU NEED: COLORED NOISE EXPLORATION IN DEEP REINFORCEMENT LEARNING
Onno Eberhard' Jakob Hollenstein" 2,1
Cristina Pinneri
Georg Martius'
1 Max Planck Institute for Intelligent Systems, Tübingen, Germany 3 Max Planck ETH Center for Learning Systems
Universität Innsbruck
[firstname. Lastname ) e uebbli ingnnn de
ABSTRACT
In off-policy deep reinforcement learning with continuous action spaces, explo- ration is often implemented by injecting action noise into the action selection process. Popular algorithms based on stochastic policies, such as SAC or MPO, inject white noise by sampling actions from uncorrelated Gaussian distributions. many tasks, however, white noise does not provide sufficient exploration, and tem- porally correlated noise is used instead. A common choice is Ornstein- Uhleebbek (OU) noise, which is closely related to Brownian motion (red noise). Both red noise and white noise belong to the broad family colored noise. In this work, we perform comprehensive experimental evaluation MPO and SAC to explore the effectiveness of other colors noise as action noise. We find that pink noise, which is halfway between white and red noise, significantly outperforms white noise, OU noise, and other alternatives on a wide range of environments. Thus, we recommend it as the default choice for action noise continuous control.
1 INTRODUCTION
Exploration is vitally important reinforcement learning (RL) find unknown high reward regions in the state space. This is especially challenging in continuous control settings, such as robotics, because is often necessary to coordinate behavior over many steps to reach a sufficiently different state. The simplest exploration method is to use action noise, which adds small random perturbations to the policy's actions. off-policy algorithms, where the exploratory behavioral policy does not need to match the target policy, action noise may be drawn from any random process. If the policy deterministic, as in DDPG (Lillicrap et al., 2016) and TD3 (Fujimoto et al., 2018), action noise is typically white noise (drawn from temporally uncorrelated Gaussian distributions) Uhlenbeck (OU) noise, and is added to the policy's 's actions. In algorithms where the policy is stochastic, such as SAC (Haarnoja et al., 2018) or MPO (Abdolmaleki et al., 2018), the action sampling itself introduces randomness. As the sampling noise is typically uncorrelated over time, these algorithms effectively employ scale-modulated version of additive white noise, where the noise scale varies for different states.
Omstein-
White noise
Pink noise
OU noise
Figure 1: Trajectories of pure noise agents on bounded integrator environ- ment (Sec. 6). White action noise (left) does not reach far in this environment, and would not be able to col- lect a sparse reward at the edges: it explores locally. OU noise (right) only explores globally and gets stuck the edges. Pink noise (center) provides balance of local and global exploration, and covers the state space more uniformly than the other two.

---

Published as a conference paper at ICLR 2023
In many cases, white noise exploration is not sufficient reach relevant states. Both MPO and SAC have severe problems with certain simple tasks like MountainCar because of inadequate exploration. As in TD3 or DDPG, the off- offlpo nature of these algorithms makes it possible to replace the white noise process, which is implicitly used for action sampling, by a different random process. The effectiveness of temporal correlation in the action selection has been noted before (e.g. Osband et al., 2016) and is illustrated in Fig. I, where the exploration behavior of white noise (uncorrelated) is compared to that of noises with intermediate (pink noise) and strong (OU noise) temporal correlation on simple integrator environment (more on this in Sec. 6). Using highly correlated noise, such as OU noise, can yield sufficient exploration to deal with these hard cases, but it also introduces different problem: strongly off-policy trajectories. Too much exploration is not beneficial for learning good policy, as the on-policy state- visitation distribution must be covered during training to make statistical learning possible. Thus, a typical approach is to use white noise by default, and alternatives like OU noise only when necessary. In this work, our goal is to find a better strategy, by considering noises with intermediate temporal correlation, in the hope that these work well both on environments where white noise is enough, and on those which require increased exploration.
state-visitation
To this end, we investigate the effectiveness of colored noise as action noise in deep RL. Colored noise is a general family of temporally correlated noise processes with a parameter 3 to control the corre- lation strength. It generalizes white noise (3 = 0) and Brownian motion (red noise, 3 = 2), which is closely related to OU noise. We find that average performance across a broad range of environments can be increased significantly by using colored action noise with intermediate temporal correlation (0 < B < 2). In particular, we find pink noise (3 = 1) to be an excellent default choice. Interestingly, pink noise has also been observed in the movement of humans: the slight swaying of still-standing subjects, as well as the temporal deviations of musicians from the beat, have both been measured to ex- hibit temporal correlations in accord with pink noise (Duarte Zatsiorsky, 2001; Hennig et al., 2011) 1).
Our work contributes a comprehensive experimental evaluation of various action noise types on MPO and SAC. We find that pink noise has not only the best average performance across our selection of environments, but that in 80% of cases it is not outperformed by any other noise type. We also find that pink noise performs on par with an oracle that tunes the noise type to an environment, while white and OU noise perform at 50% and 25% between the worst noise type selection and the oracle, respectively. To investigate whether there are even better noise strategies, we test a scler-schedule that goes from globally exploring red noise to locally exploring white noise over the course of training, as well as a bandit method to automatically tune the noise color to maximize rollout returns. Both methods, though they significantly improve average performance when compared to white and OU noise, are nevertheless significantly outperformed by pink noise. In addition to the results of our experiments, we attempt to explain why pink noise works so well as a default choice. by constructing environments with simplified dynamics and analyzing the different behaviors of pink, white and OU noise. Our recommendation to switch from the current default of white noise to pink noise.
2 BACKGROUND & RELATED WORK
Reinforcement learning (RL) has achieved impressive results, particularly in the discrete control setting, such as achieving human-level performance in Atari games with DQN (Mnih et al., 2015) or mastering the game of Go (Silver al., 2016) by using deep networks as function approximators. In this paper, we are concerned with the continuous control setting, which is especially appropriate in robotics. In continuous action spaces, it is typically intractable to choose actions by optimizing a value function over the action space. This makes many deep RL methods designed for discrete control, such as DQN, not applicable. Instead, researchers have developed policy search methods (e.g. Williams, 1992; Silver et al., 2014), which directly parameterize policy. These methods can be divided into on-pol algorithms, such as TRPO (Schulman et al., 2015) and PPO (Schulman et al., 2017), and off off-policy algorithms such as DDPG, TD3, SAC and MPO.
All these algorithms have to address the problem of exploration, which is fundamental to RL: in order to improve policy performance, agents need to explore new behaviors while still learning to act optimally. One idea to address exploration is to add a novelty bonus to the reward (c.g. Thrun, 1992). In deep RL, this can be done by applying a bonus based on sample density (Tang et al., 2017) or prediction eиror (Burda et al., 2019). Another method to encourage exploration is to take inspiration from bandit methods like Thompson sampling (e.g. Russo et al., 201 2018), and act optimistically with
2

---

Published as a conference paper at ICLR 2023
respect the uncertainty in the Q-function (Osband al., 2016). The simplest strategy, however, is to randomly perturb either the policy parameters (Plappert et al., 2018; Mania et al., 2018), or the actions themselves. This can be done by randomly sampling a function for each episode that deterministically alters the action selection (Raffin & Stulp, 2020), by learning correlations between action dimensions and state space dimensions to induce increasing excitation in the environment (Schumacher et al., 2022), or by learning an action prior from task-agnostic data (Bagatella et al., 2022).
In this work, consider the simplest and most common form of exploration in continuous control: action noise. Action noise can be either explicitly added to the policy, or implicitly, by randomly sampling actions from stochastic policy. The most common form of action noise is white noise, which typically comes from sampling from independent Gaussian distributions at every time step. Apart from white action noise, Lillicrap et al. (2016) successfully used temporally correlated Ornstein- Uhlenbeck noise, and Pinneri et al. (2020) achieved improvements in model predictive control by utilizing colored noise. Inspired by this success, in this work we investigate the effectiveness of colored action noise in the context of model-free RL, specifically MPO and SAC.
METHOD
In this paper, investigate exploration using action noise. In algorithms like DDPG and TD3, where the learned policy μ is deterministic, action noise is simply added to the policy:
02 = ((())) + oe
(1)
where E1.T (EI...., ET) is sampled from random process, and o is a scale parameter. If E is sampled independently at every time step, e.g. from Gaussian distribution, then S1r is called white noise (WN). This is the prevailing choice of action noise, though it is also common to use time-correlated CeattinnUhheenbekk noise (E1:T OUT) (Uhlenbeck & Ornstein, 1930).
Algorithms which parameterize a stochastic policy, such as SAC and MPO, also use action noise. In continuous action spaces, the most common policy distribution is diagonal Gaussian, represented by the functions μ((;)) and (st): at ~ N (H((s)), diag(o This can equivalently be written as
a = (++)) + a (8s) 0
where E, (O, I). In this case, the action noise E1:T is again Gaussian white noise, which is modle-modulated by the function σ.
White noise not correlated over time (cov(E,, Ep] 0). In some environments, this leads to very slow exploration, which in turn leads to inadeuuate state space coverage, leaving high reward regions undiscovered. Thus, it is often beneficial to use action noise with temporal correlation (cov(Er,Ee:) Eer >> 0), like choice for DDPG, and has been shown to lead to a significant increase in state space coverage (Hollenstein et al., 2022). OU noise is defined by the stochastic differential equation (SDE)
(OU) noise. OU noise was recommended the default
é, 0c: + OT},
where is a white noise process. O = 0, then this equation defines integrated white noise, also called Brownian motion. Brownian motion is temporally correlated, but cannot be used as action noise if generated in this way, because its variance increases unboundedly over time, violating the action space limits. This problem is addressed by setting 0 > 0 (a typical choice 0 = 0.15), which bounds the variance. More details about OU noise and Brownian motion can be found in Sec. A.
broad family of temporally correlated noises is given by colored noise, which generalizes both white noise and Brownian motion (in this context called red noise).
Definition (Colored noise). A stochastic process is called colored noise with color parameter 3, if signals (t) drawn from it have the property [ x f- where é(f) F(s(t))(f) denotes the Fourier transform of et) (f is the frequency) and [(( is called the power spectral density (PSD).
The color parameter 3 controls the amount of temporal correlation in the signal. The PSDs of colored noise with different 3 are shown in Fig. A.2. If 8 0. then the signal is uncorrelated, and the PSD is flat, meaning that all frequencies are equally represented. This noise is called white noise in analogy to light, where signal with equal power all visible frequencies is perceived white.
3

---

Published as a conference paper at ICLR 2023
Figure 2: The environments we use: Pendulum, CartPole (balance + swingup tasks), Ball-In-Cup, MountainCar, Hopper, Walker, Reacher, Cheetah, Door. See Sec. for more details. Images partly taken from Tassa et al. (2018) with permission.
Red noise (B = 2) is named so, because it has more weight on lower frequencies, which in light corresponds to the red part the spectrum. Gaussian colored noise with constant variance can be efficiently generated, and the complete noise signal for episode can be sampled at once, to be used as action noise according to Equations (1) and (2). If generated like this, which we denote by E1:1 (3) (more details in Sec. A). white noise is identical independently sampling from a Gaussian distribution at every time step. Red noise (2))(2)) is very similar to OU noise with the default setting 0 0.15, both are essentially Brownian motion with bounded variance (see Fig. A.2). By setting 0 < 3 < 2, colored noise allows us to search for better default action noise type with intermediate temporal correlation between white and red noise. One special case is pink noise, which is defined by 3 = 1.
4 IS PINK NOISE ALL YOU NEED?
Fujimoto et al. (2018) found that the type of action noise (white or OU) in general does not influence the performance of TD3. In contrast to this, Hollenstein al. (2022) found that the noise type does have an influence, but that the impact of this choice, as well as which noise type is preferable, depends entirely on the enviroument. We start by confirming these latter results?, and compare white and OU action noise with selection of colored action noises (3 E [0, 2)) in terms of the achieved performance. all of our experiments we use MPO and SAC, relying on the implementations by Pardo (2020) and Raffin et al. (2021), respectively. We found that both algorithms significantly outperform TD3 across tasks, and thus only briefly discuss TD3 in Sec. B. Since the optimality the environment,
of action noise type depends different tasks taken from the DeepMind Control Suite (Tassa et al., 2018), OpenAl Gym (Brockman al., 2016), and the Adroit hand suite (Rajeswaran et al., 2018). These environments shown in Fig. 2 and are described in more detail in Sec. C. We report results on some additional tasks in Sec. G.
perform experiments
diverse set of 10
To evaluate the performance of training procedure (which always lasts 10 environment interactions), we run 5 evaluation rollouts every 10 interactions. We then report the performance as the mean return of all these evaluation rollouts. Since this performance is related the area under the learning curve, it is a measure combining both the final policy performance, and the sample efficiency of an algorithm. More detailed results, including learning curves and an analysis of the final policy performance, can be found in Sections B.2 and H.
4.1 DOES THE NOISE TYPE MATTER?
To assess the importance the choice of action noise, we evaluate the performances achieved by SAC and MPO when using white noise, OU noise and colored noise action noise (where
OOther settings of 0 for QU noise, which are less similar red noise, discussed in See. A. In the main 0.15.
text we only consider the default setting
"e also confirm the former results (see Sec. B.1), but find that colored noise (especially pink noise) outperforms both white and OU noise TD3.
4

---

Published as a conference paper at ICLR 2023
Environment
Best noise
p Pink?
0.5
Average Performance
Pendulum
2.0
0.01
Cartpole (b.)
1.0 (Pink)
0.0
Cartpole (s.)
1.0 (Pink)
Ball-In-Chn Cup
0.75
0.88
MountainCar
2.0
0.59
Hopper
1.0 (Pink)
Walker
0.5
0.36
Reacher
White noise
0.02
1.0
Cheetah 0.75
0.62
WN 0.1 0.2 0.35 0.5
0.75
1.0 1.5
Door
0.75
0.65
Table 1: A Welch f-test reveals that the per- formance difference between pink noise and the best noise is only significant in two out of ten environments. The rightmost column answers whether pink noise performs equally well as the best noise type.
Figure 3: Bootstrap distributions for the expected average performance of MPO and SAC using dif- ferent action noise types (details Sec. B.2). High- lighted are white noise (WN), pink noise (3 = 1), and Krhstein-Uhlenbecck noise (OU).
E {0.1, 0.35, 0.5, 0.75, 5, 1, 1.5, 2}). on the benchmark environments shown in Fig. We repeat all learning runs with 20 different seeds, resulting in total of 20 10 x 2 x 10 4000 experiments' each one reporting a single scalar performance. To control for the influence of the algorithm and environment the performance of particular noise type, we group all results by algorithm and task, and normalize them to zero mean and unit variance. We then calculate a noise type's average performance: the normalized performance of all runs using this noise type, averaged across algorithms and environments. In Fig. 3, bootstrap distributions for the expected average performances are shown, generated using the 20 random seeds available per task and algorithm (more details in Sec. B.2). It can be seen that the noise type indeed matters for performance. A clear preference for pink noise (3 = 1) becomes visible, which considerably outperforms white noise and OU noise across tasks. In Sec. B.2, this performance difference can be seen on the corresponding learning curves, where we compare white, OU and pink noise.
Achieving the best average performance across environments is not the same as being the best performing option on each individual environment. This begs the question of when pink noise (the best general option) also performs as good environment's best noise type. We perform Welch t-test for each environment to check whether the expected difference between the performances of pink noise and the task-specific best noise* is significant. The results are listed in Table 1. Although pink noise only achieves the highest mean across seeds three of the ten tasks, the statistical analysis reveals that the difference between pink noise and the best noise type is only significant in two out of ten cases, In other words, in the tested environments, pink noise performs on par with the best choice of noise type in 80% of cases! What about the two environments, Pendulum and Reacher, where pink noise is outperformed by other noise types? On Pendulum, pink noise, average, achieves 83% of the performance of red noise (B 2). In contrast, white noise only achieves 39% of the performance of red noise (OU performs similarly to red noise). On Reacher, pink noise achieves 99% of white noise's performance, while OU noise achieves only 76%. So, even on the few environments where default over white noise and OU
pink noise is outperformed significantly, it is clearly preferable noise. These results indicate that pink noise seems to be all you need.
4.2 Is PINK NOISE A GOOD DEFAULT?
The best performance on given environment is always achieved with the task-specific best action noise type. It would be nice to always use this best noise type, but it is often unpractical to run large hyperparameter search find especially when including many possible colors 3. It is common therefore, to stick to a "default" choice, which is typically white noise. In the previous section, we saw that pink noise better default choice than white noise, but is still unclear whether this
sceds tasks x algorithms x noise types (WN OU + 8 colors)
"To compute the best noise type, normalize out the contrihution of the algorithm (similarly to the average performance) and take the mean performance over random seeds on each environment.
5

---

Published as a conference paper at ICLR 2023
Average Performance
Figure 4: Bootstrap distributions for the expected average performances of all methods we discuss in this paper. High- lighted are again white noise and OU noise (the popular options), pink noise (our suggestion). While OU noise and white noise only achieve about 25% and 50% of the possible perfor- mance gain of an oracle method, pink noise performs equally to the oracle! Pink noise is also not outperformed by a color- scheduling method (Sec. 5.1), nor by bandit algorithm (Sec. 5.2).
1.0
0..
0.0
0.5
Orucle
Pink noise
well as
Bandit (Scc.
Random (Scc.
Schedule (Sec. 5.1)
White noise
Ornstein- Uhlenbeck
Anti-Oracle
0%
20%
40%
60%
80%
100%
Performance Gain
enough, or if a hyperparameter search might be needed for good performance. In this section, we will analyze how much performance is lost by sticking to a default value of white noise, pink noise, or OU noise, compared to using the task-specific best noise type.
We choose the task-specific best noise type via an "oracle" method, which can be thought of as very extensive grid search: environment's 's best noise type is selected by looking at the results of all noise types, and choosing the best performing option across 10 seeds. By also doing an kunti-orace" experiment, which selects the worst noise type each environment (i.e. the most unlucky pick of noise types possible), we can define a new "performance gain" measure, which might be easier to interpret than the average performance in the previous section. The performance gain of noise type specifies where its average performance falls between the anti-oracle's 's performance (0%) and the oracle's performance (100%). Figure 4 presents the performance gains of using white noise, Ornstein-Uhlenbeck noise and pink noise as a default for all environments.
By always sticking Crnstein-Uhlenbeee noise, only about 25% of the highest possible performance gain is achieved, and the resulting performance would be closer to using the anti-oracle. By using white noise instead, we already achieve performance gain over 50%. However, picking pink noise does not appear to sacrifice any performance compared to the oracle!' The gain achieved by switching from white noise as the default to pink noise is both considerable and significant, and we recommend switching to pink noise the default action noise.
5 ALL THE COLORS OF THE RAINBOW
We have found that pink noise is the best default action noise over broad range of environments. There are still some environments, however, where pink noise is outperformed by other noise types, specifically by white and red noise the Reacher and Pendulum tasks, respectively. This indicates that there may not exist a single noise type which performs best on all environments. However, this consideration is only valid if the noise is kept constant over the course of training. If we instead try different approach, and choose the noise type separately for each rollout, we may find a strategy that is outperformed nowhere. In this section we discuss two such non-constant methods, which differ in the way a rollout's noise type is selected: a solor-schedule going from / = 2 to 3 = O, and a bandit approach with the intention of finding the optimal color for an environment.
5.1
BETTER THAN PINK NOISE?
To find a more effective exploration method than pink noise, is helpful to understand why learning the Pendulum and Reacher environments works better with other noise types. The Pendulum environment is underactuated and requires gradual build-up of momentum by slowly swinging back and forth. Strongly correlated action noise, such as red noise, makes this behavior much more likely. The Reacher task, on the other hand, has neither particularly large state space, nor does it exhibit
'The (anti-Joracle is evaluated on the 10 seeds not used for noise type selection avoid sampling bias. We repeat this by selecting (evaluating) once on the first (latter) 10 sceds, and once on the latter (first) 10 seeds. 'It looks if pink noise is even exceeding the oracle's 's performance. This difference is not statistically significant and is due to the oracle only having access the 10 seeds not used for evaluation.

---

Published as a conference paper at ICLR 2023
Environment
P> S
P<S
Pink 2 Schedule?
P> B
P< B
Pink Bandit?
Pendulum
<0.01
X
0.73
Cartpole (b.)
<0.01
0,10
Cartpole (s.)
0.01
<0.01
Ball-In-Cup
<0.01
0.47
MountainCar
0.15
0.29
Hopper
0.05
0.15
Walker
<0.01
0.28
Reacher <0.01
0.50
Cheetah 0.05
0.64
Door
0.15
0.27
Table 2: How does pink noise compare schedule and bandit method? For each environment, we perform a Welch (-test to test for inequality the performances pink noise vs. the schedule/bandit method. The p-values significantly better than the schedule on most environments. Compared to the bandit algorithm, pink noise performs better overall, but on most environments the difference is not significant. In the "Pink 2 _..."" columns, (V) means that pink noise does not perform significantly worse than the alternative.
arranged to show which performance is higher. Pink noise performs
under-actuation, such that white noise is well suited to explore the space. Here, temporally correlated noise will only lead off- aff-po trajectory data, thereby inhibiting learning. In general, strongly correlated noise leads to more global exploration, while uncorrelated noise explores more locally. We return to these ideas Sec. 6, where we analyze the effects of high and low correlation on two simple environments.
Our method should work well on both of these environments, and on environments which require a mix of local and global exploration. Thus, we are looking for strategy which balances local and global exploration. simple idea to do this is a scheu-schedule: start with highly correlated red noise (8 2) and then slowly decrease 3 to white noise (B 0) over the course of training. The rationale behind this strategy is that high-reward regions can be quickly discovered at the beginning of training when B is large, while the trajectories get on-policy time, helping with environments like Reacher. Indeed, a similar approach that schedules the action noise scale, has been shown to work quite well (Hollenstein et al., 2022).
We implement a f-schedule, which linearly goes from 3 = 2 to 3 = 0, on MPO and SAC and repeat the experiment with 20 random seeds on all environments. Bootstrap distributions for the expected average performance across environments are shown in Fig. 4 (denoted by Schedule). The results indicate that the schedule is generally better than OU and white noise, but does not outperform pink noise. Indeed, pink noise is significantly better, the confidence intervals do not overlap. If we take a more detailed look at the individual environments (Table 2), we see that thanks to the additional highly correlated noise, the schedule does outperform pink noise on the Pendulum environment, as expected. However, in all other environments pink noise either significantly outperforms the schedule or they perform on par, our recommendation to use pink noise as a default remains.
5.2 Is BANDIT COLOR SELECTION BETTER THAN PINK NOISE?
The results in the previous section indicate that, while changing the noise type over the training process can improve performance, simply moving from globally exploring red noise more locally exploring white noise does not outperform pink noise. Instead of trying to find a different schedule to fit all environments. in this section we consider an adaptive approach. By using bandit algorithm to select the action noise color for each rollout on the basis of past rollout returns, it might be possible to find not only the general best noise for given environment, but even to automatically adapt the noise to different stages of training. The bandit algorithm we use is based on Thompson sampling, the details are explained Sec.
We use the rollout return itself as the bandit reward signal. The reasoning for this is that in environ- ments where strong exploration is necessary (such as Pendulum and MountainCar), high return will only be achieved by strongly correlated actions. On the other hand, if environments do not require correlated actions, or a capable policy has been learned, the highest return should be achieved by the action noise which least disturbs the policy, i.e. noise with low correlation.
7

---

Published as a conference paper at ICLR 2023
As additional baseline, we also perform an experiment where a color (3) is randomly selected for each rollout.' For both methods we use the same list of 3 values as in Sec. 4.1 (inel. 3 = 0), and repeat the experiments with 20 random seeds. The results on MPO and SAC are shown in Fig. (marked with Bandit and Random) and Table 2. It can be seen that the bandit method is again outperformed by pink noise. Indeed, bootstrapping test yields highly significant difference in expected average performance across environments (p = 0.005). Looking at the results on the individual tasks (Table 2). seems like the bandit method does outperform pink noise on the two problematic environments (Pendulum and Reacher), however, this difference is not significant. A detailed comparison of the bandit and its random baseline can be found in Table D. which shows that neither of the two methods significantly outperforms the other on any environment. This indicates that, while there may be merit in changing the noise type over the training process, the rollout return appears to contain too little information to effectively guide the noise type selection. Thus, our recommendation to use pink noise as a default remains unchanged.
6 HOW DO ACTION NOISE AND ENVIRONMENT DYNAMICS INTERACT?
Why is pink noise such a good default noise type? In Sec. 5.1, we briefly discussed the concepts of local and global exploration, and hypothesized that the best exploration behavior provides a balance of the two, such that high reward regions will be found, while trajectories are not too off-policy. To analyze how different noise types behave, we will look at simplified bounded integrator environment: a velocity-controlles 2D particle moving in a box (more details in Sec. F.2). If we control this particle purely by noise, we can analyze the exploration behavior in isolation of a policy. As first test, we run 20 episodes of 1000 steps in an environment of size 250 x 250 with white noise, pink noise, and OU noise (all with unit variance, x- and y- y-velocity controlled independently). The resulting trajectories are shown in Fig. 1. It can be seen that pink noise provides the best combination of local and global exploration: it reaches the edges (unlike white noise), but does not get stuck there (unlike OU noise).
A good mix of local and global exploration gives rise to more uniform state space coverage, as can be seen in Fig. 1. Thus, how well noise type explores depends highly on the size of the environment: if the environment was much smaller, white noise would be enough to cover the space and pink noise trajectories would look similar to the OU trajectories shown here. On the other hand, if the environment were bigger, then pink noise would not reach the edges and OU noise would explore better. The uniformity of the state space coverage is measured by the entropy of the state-vine vistiavisitttion distribution. We estimate the entropy induced by noise type using histogram density approximation: we partition the state space into a number of boxes (50 x 50 = 2500 boxes), sample 10s trajectories, and count the number of sampled points in each box.
Figure 5 shows the entropy achieved by white noise, OU noise and pink noise as a function of the environment size. The sizes are chosen to reflect the complete sensible range for episode lengths of 1000 steps, each with unit variance: from very small (50 x 50) to very large (2000 x 2000). Pink noise is not "special" in the sense that performs best on all environments, as we already saw in the previous sections. However, it performs best on "medium scales", as determined by the episode length, and does not suffer from severe degradation in performance over the whole spectrum of sensible environments. If we do not know where on this spectrum a given environment lies, then pink noise is clearly a better default choice than white noise or OU noise!
Besides integrating actions, another common aspect of environment dynamics is oscillation. Oscil- lation dynamics are dominant in the Pendulum and MountainCar environments but also relevant in other domains, like Ball-In-Cup. Cartpole, and Walker. To model these dynamics, we construct a second environment: a simple harmonic oscillator. This system is a frictionless I-dimensional phys- ical setup, in which mass m is attached to an ideal spring of stiffneess k. The state space consists of the mass's 's position and velocity, and the action describes a force that is applied to the mass. The goal is to maximize the energy in the oscillator system (which is equivalent to maximizing the amplitude), similar to the MountainCar and Pendulum tasks, where this is necessary to collect the sparse reward.
The oscillator environment is parameterized by the resonant frequency I of the system, which is fixed by setting the stiffness k- = 4x and the mass m = f/ (more details in Sec. F.I). Figure 5 shows
"This method would be roughly equivalent to the handit method if we provided no bandit reward signal. "See See. E. for a simple method exploiting this property to solve MountainCar.
8

---

Published as a conference paper at ICLR 2023
Bounded Integrator Environment
Oscillator Environment
Entropy [nats]
Energy [barm,]]
10-1
WN
10-2
Pink
OU
10
10
10
10
10
2
10
Area
Resonant frequency f
Figure 5: Pink noise strikes a favorable middle ground between white noise and Ornstein-Uhlenbeed noise on a wide range of environments. On both a bounded integrator environment parameterized by its size (left), and a simple harmonic oscillator environment parameterized by its resonant frequency (right), it is much more general in terms of the range of parameters which yield good results, and performs well on the complete range of reasonable parameterizzaions. We argue that this quality is what makes it good default.
the average energy in the oscillator system (over 1000 episodes of 1 1000 steps each) as a function of the resonant frequency f. which we vary from very low (f = 1000 episode length = 1 period) to very high (f 1. Nyquist frequency), when driven by white noise, pink noise, and OU noise. The energy is measured relative to the average energy achieved by a sinusoidal excitation at the resonant frequency, denoted harmy- Even though this is completely different setup the bounded integrator, and we are using a very different performance metric, the two plots look remarkably similar. Again, this shows the power of pink noise as default action noise: we do not know the resonant frequency of the given environment, pink noise is the best choice.
These two environments (bounded integrator and oscillator) rather simplistic. However, the dynamics of many real systems undoubtedly contain parts which resemble oscillations (when a spring or pendulum is present), single or double integration (when velocities/steps or forces/torques are translated into positions) or contact dynamics (such as the box in the bounded integrator). If an environment's dynamics are very complex, i.e. they contain many such individual parts, then the ideal action noise should score highly on each of these "sub-tasks". However, if all these individual parts have different parameters (like the environment size or resonant frequency above), it stands to reason that the best single action noise would be the one which is general enough to play well with all parameterizations, i.e. pink noise. On the flip side, the average performance in Fig. 3 over all environments may be interpreted as the performance sub-tasks being the "actual" environments. This might explain why we see this curve: all sub-tasks have very different parameters, and require different action noises (as in Table 1), but pink noise is general enough to work well on all sub-tasks, and thus easily outperforms noise types like white noise or OU noise, which are only good on very specific environments (see Fig. 5).
very complicated environment, with the
CONCLUSION
In this work performed a comprehensive experimental evaluation of colored noise as action noise in deep reinforcement learning for continuous control. We compared a variety of colored noises Omstein-Uhlenbeck with the standard choices of white noise and Ornstein Uhlenbeck noise, and found that pink noise
outperformed all other noise types when averaged across a selection of standard benchmarks. Pink noise only significantly outperformed by other noise types on two out of ten environments, and overall performs equally well to an oracle selection of the noise type. Additionally, we compared pink noise to more sophisticated methods that change the noise type over the course of training: a color-schedule, bandit method, and random selection scheme. No method outperforms pink noise, and our recommendation to use pink noise as the default action noise. Finally, we studied the behaviors of pure noise agents two simplified environments: bounded integrator and harmonic oscillator. The results showed that pink noise is much more general with respect to the environment parameterization than white noise and OU noise, which sheds some light on why performs well as the default choice.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We want thank Marco Bagatella, Sebastian Blaes, and Pierre Schumacher for helpful feedback on earlier revisions of this text, and the Max Planck ETH Center for Learning Systems for supporting Cristina Pinneri. Georg Martius a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 Project number 390727645. We acknowledge the support from the German Federal Ministry of Education and Research (BMBF) through the Tübingen Al Center (FKZ: 013318)39B)
REPRODUCIBILITY STATEMENT
We submitted our code as supplementary material, and provide a polished version of online (ht / //git higt.hu... com/ ma rt iittussiie laallaapplaa seink-nnise--ie took particular care in applying random seeds to all randomized parts of the algorithms, including the environments, algorithm initialization and training, as well as the action noise signals. Sec. C, we provide details on the exact algorithms and environments we use. The hyperparameters for our own methods are included in the code submission. Additionally, our main experiments were all repeated with 20 different random seeds to climinate statistical flukes.
In an effort to ensure reproducibility, we
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Rémi Munos, Nicolas Heess, and Martin A. Riedmiller. Maximum a posteriori policy optimisation. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL htwps: http:sse///pereviee... forum?i d==5 ANxQW0b.
Marco Bagatella, Sammy Joe Christen, and Otmar Hilliges, SFP: State-free priors for exploration in off-po reinforcement learning. Transactions on Machine Learning Research, 2022. URL
https:: http//ppperveeeeeeeeeeeee net do
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and org/
Wojciech Zaremba. Openai gym. CoRR, abs/1606.01 2016. URL abs/1606. 51500. 4, 20
Yuri Burda, Harrison Edwards, Amos J. Storkey, and Oleg Klimov. Exploration by random network distillation. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https: net/forumi 111H111111155 Ym. 2
James W. Cooley and John W. Tukey. An algorithm for the machine calculation of complex fourier series. Mathematics of Computation, 19:297-301, 1965. 13
Marcos Duarte and Vladimir M. Zatsiorsky. Long-range correlations in human standing. Physics Letters A, 283(1): 2001. ISSN 0375-9601. URL https: //www. sci www.scceeeeccce encedirect. com/ http://ceenneerrrraaaae/////////
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockhelmssässsan Stockholm, Sweden, July (0-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1582-1591. 102.11....MRRR 2018. URL http: htttp://procceedngss mlr. mlr.press/ v80 fu htt/://uiomooooooooooooooooooooo mimotooooo... html. 1,
Aurélien Garivier and Eric Moulines. On upper-confidence bound policies for non-stationary bandit problems. arxiv: 0805.3415, 2008. URL http:://: /aaxiv...aaivvv/aaa org/ abs/ 3415. 20
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference Machine Learning. ICML 2018, Stockholmsmääsann Stockholm, Swe- den, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1856-1865. PMLR, 2018. URL http::/t///... http////ooooooeoeoeeoooooo ngs ml w.prrss https////////aar//oaaaaaa 8b. html.
10

---

Published as a conference paper at ICLR 2023
Holger Hennig, Ragnar Fleischmann, Anneke Fredebohm, York Hagmayyr, Jan Nagler, Annette Witt, Fabian Theis, and Theo Geisel. The nature and perception of fluctuations in human musical rhythms. PLOS ONE, 6(10):1-7, 10 2011. URL httts:///ddo..rrg wtt.:/..rgg.... journaaa. pone. 0026457.
Jakob Hollenstein, Sayantan Auddy. Matteo Saveriano, Erwan Renaudo, and Justus Piater. Action noise off- off-policy deep reinforcement learning: Impact on exploration and performance. Transuc- lions Machine Learning Research, 2022. ISSN 3835-8856. URL https: /oppoppeervvieeeeee net/ Forum? finniidees id=N1 jB1 7 6hmG. Survey Certification. 3, 7,
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL htRL:tt::/ wtw.i...iiii.....a.aa......a...t /as/////////////// 15099. 02971. 1, 3, 15
Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search of static lin- car policies competitive for reinforcement learning. In Advances in Neural In- formation Processing Systems 31: Annual Conference on Neural Information Pro- cessing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp. 1805-1814, 2018. URL htttpa://proceedingg neurii neurips cttpcc//pppp/.e..c htppp/ppppp/e/////// 2010/hashh//////// /ash// Abstract rastrr html. 3
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nat., 518(7540):529-555 533, 2015. URL https: /tt.//dd.iio//////////.. org/10 /00. 1038/ matanee42336...
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN. In Advances in Neural Information Processing Systems 29: Anmual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 4026 4026-4034, 2016. URL htwps: /tt.t:pp:pppcccoceeeeee www.:eeeeeddinn.....ees.....e... oc (aaper/0166// html. 2,3
8d881 8c8e1
Fabio Pardo. Tonic: A deep reinforcement learning library for fast prototyping and benchmarking. CoRR, abs/201 1.07537, 2020. URL https: :tap:ps:///aaivvv////a http://awwwwwww../aaaaa www.///aaaaa...... at/aa/ass/2111 20t.//21111........11110110 4, 15, 20
Cristina Pinneri, Shambhuraj Sawant, Sebastian Blaes, Jan Achterhold, Joerg Stueckler, Michal Rolínek, and Georg Martius. Sample-efficient cross- tros--ntropy method for real-time planning. In 4th Conference on Robot Learning. CoRL 2020, 16- 18 November 2020, Virtual Event Cambridge, MA, USA, volume 155 of Proceedings of Machine Learning Research, pp. 1049-1065. PMLR, 2020. URL wwtptttt///otc httttp:://proceeeee pwwppeoooceeeeeeoooooooooooooo.e ngs mlr www.ppess/eee vvlvvv15..//
55/ /1155ppp// pwnneriiiii. 221 html.
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y. Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3. 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https: openreview net /tor/ooumm/////// Eorum?id-ByyAA2eAA 3
Antonin Raffin and Freek Stulp. Generalized state-dependent exploration for deep reinforcement learning in robotics. CoRR, abs/2005.05719, 2020. URL http://aaaaa...oa////////0//// 05719. 3
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-aseelinss: Reliable reinforcement learning implementationns. J. Mach. Learn. Res., 22:268:1-268:8, 2021. URL http::::://jirroooopaaaeeeeeeees 15, 20
html.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforce- ment learning and demonstrations. In Robotics: Science and Systems XIV, Carnegie Mellon

---

Published as a conference paper at ICLR 2023
University, Pittsburgh, Pennsylvania, USA, June 26-30, 2018, 2018. URL http: /www. roboticsproceeddngs wwt.://gg..ggg//s/////
html. 4,
Daniel Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, and Zheng Wen. A tutorial on thompson sampling. Found. Trends Mach. Learn., 11(1):1-96, 2018. URL httpt:t///ceee /ttttp:::/dd.... orgg
10.1561. 10.1151/2200000C0 /000//20000000000 2
John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning. ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 1889-1897. JMLR.org, 2015. URL http: http:///poceeedeeeeeeeooe https//tt/tttpppoooooooooog..gg mlr v37/ ht7/schhlmmnnn html.
pttp.eeedrngg....pprrsssss// press/
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06530 2017. URL htttp://arxiv. org/ abs/
1707. 06347.
Pierre Schumacher, Daniel Häufle, Dieter Büchler, Syn Schmitt, and Georg Martius. Dep-rl: Embod- ied exploration for reinforcement learning in overactuated and musculoskeletal systems. arXiv: 2206.00484, 2022. URL www.thtps///// arxia. org/ abs 220 6. 00484.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin A. Riedmiller. Deterministic policy gradient algorithms. In Proceedings of the 31th International Conference Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014, volume 32 of JMLR Workshop and Conference Proceedings, pp. 387-395. JMLR. 2014. URL http: /ttpp//roccee......o.//// m.lrprress v3/ silerr14 html. 2
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Panneershelvam, Marc Lanetot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nat., 529(7587):484488, 2016. URL https: doi. . www.orgg/ooooo 10. 1038 naaaaree6666..
Haoran Tang, Rein Houthooff, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. #exploration: A study of count-based exploration for deep reinforcement learning. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 2753-2762, 2017. URL https: 2017 hht.h/aaaahaaaaaaaaaaaaaaa 3a20 62 (
httpt::ss/pp.ccedingggg.... neuripg cc/ /ppppr///////////////////eee/aee html.
aa aawaaa5560bbbbo waw.aaalaaaaaaaaaaaaa22 wtw.aaaaaaa00000000000eeeeeeeeee
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy P. Lillicrap, and Martin A. Riedmiller. Deepmind control suite. CoRR, bbs//800.00690,, 2018. URL http:/ http://xxxx../g//...//// 1801. 000.0.6600000000 4, 20
Sebastian Thrun. Efficient exploration in reinforcement learning. Technical Report CMU-CS-92-102,
Camegie Mellon University, Pittsburgh, PA, January 1992. 2
Jens Timmer and Michel Koenig. On generating power law noise. Astronomy and Astrophysics, 300: 707, 1995. 13
G. E. Uhlenbeck and L. S. Omstten. On the theory the brownian motion. Phys. Rev. 36:823-841, Sep 1930. URL https: sttps::/Iie /ttp///inkk/////////// aps arg//doi/00 1103/PhysRev 823.
Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Mach. Learn., 8:229-256, 1992. URL https: http://oooooo...000000/1 BFC0992696.
12

---

Published as a conference paper at ICLR 2023
Appendix
Supplementary Material PINK NOISE IS ALL YOU NEED
COLORED NOISE AND BRNSTEIN-UHLENTEE NOISE
Colored noise has an interesting property that was not mentioned in the main text: integrating a colored noise signal with parameter B again yields colored noise signal, only with parameter 3 This stems from the property of the Fourier transform that an integration in the time domain corresponds multiplication with (i2mf) in the frequency domain. Let v(t) be the original colored noise signal with [&()))² x f f- Then the PSD of rr(t) = So u(T) dr is
= [ dT (f)
2
o(f)
f f=) f (9+2).
(4)
122
From this, and the definition of white noise as colored noise with , = 0, follows that Brownian motion (integrated white noise) is also colored noise with parameter B 2. In Fig. A.I, sampled
signals of most of the noise types we use in this paper are shown, and in Fig. A.2, we plot the power spectral densities of some of these.
White noise (3 0)
B = 0.5
Pink noise (3 = 1)
8 1.5
Red nosic (B 2)
OU noise (0 0.15)
250
500
750
1000
250
500
750
1000
250
500
750
1000
8
Figure A.1: Sampled signals from various action noise processes with noise scale 1. The exception is OU noise, whose noise scale is adjusted such that varle,] = (see discussion below).
We generate colored noise using the procedure deseribed by Timmer & Koenig (1995), based on the Fast Fourier Transform (FFT) (Cooley & Tukey, 1965). This method is very efficient, as it only requires sampling a Gaussian signal in the frequency space (where the PSD is shaped), and then transforming it to the time domain via the FFT. In particular, this procedure is faster than sampling an Amattin-Uhlenbee describe below). We use the co colorednoise Python package (https: /gittub. com/ se) to sample colored noise signals, and always sample signals of the complete episode length (which we denote by E1.T ~ (N)) The Python implementation contained a bug, which among other things made so the generated "white noise" was correlated, and our fix of this bug is included as of version 2.0 1. of the package. Colored noise sampled according to this procedure is stationary and Gaussian: the signals are marginally identical to standard Gaussian distributions, i.e. p(ε,) (E, 0. 1). The only difference white noise (independent Gaussian samples at every time step) is that they are temporally correlated: p(E,, Er) + p(E)))(EEr). This is shown empirically the example of pink noise in Fig. A.3.
signal (using the most common procedure, which we
fel expatzelle weeeet/collreedee
13

---

Published as a conference paper at ICLR 2023
Appendix
Colored Noise
Brownian Motion, OU Noise, Red Noise
White noise (8 0)
Iniegrated WN
Pink noise
OU mosse (0 0.15)
10
Red (3
Power Spectral Density
True fm
True f
10a
10a
10-2
10-2
10
10
10
Figure A.2: Left: The power law trends can be seen in the PSDs of sampled colored noise signals. Right: Brownian motion, here generated by integrating white noise sampled from (O, 1). is compared to two related stationary noises: Ornstein- CeieennnUhhebeekk noise (0 = 0.15), and red noise. The similarity between OU and red noise visible. All signals are of length 1000.
A. ORNSTEIN- DRNSTEIN-UHLENBEER UHLENBECK NOISE GENERATION AND VARIANCE CORRECTION
(OU) noise. be seen that OU noise starts out as non- non-stationary but quickly converges to the same marginal distribution p(e,) = [cc 0, 1) as the other noise types. Important to note is that these noise types are suitable for use as action noise only because they are (or quickly become) stationary, and hence their variance does not grow without bounds (contrary to that of Brownian motion). The property that all noise types have the same marginal distribution shows that our results only due change in the temporal correlation of the action noise, not in the scale or shape of the distribution, as this is the same as of regular Gaussian white noise. To make that OU noise converges standard Gaussian marginal distribution cannot use noise scale of a = 1, but have to correct it. Ornstein-Uhlenbeck noise can be defined by the stochastic differential equation
Also included in Fig. A.3 is
dæ, = -0x, dt + dw,,
(5)
White noise (E: ~ N(0, 1)) Pink noise (E1.7 CNr(1))
OU noise (E1:7 ~ OUr)
0.8
0.6
Density
0.4
500
Figure A.3: The colored noise we use as action noise same marginal distribution as indepen- dent Gaussian samples. We sampled 3 x 10 action noise signals of length T = 1000 from each of the following random processes: independent Gaussian samples (white noise, left), pink noise (center), Ornstein- Uhlenbeck noise (right). At every time step é we show histogram density estimate over action noise values Er. This shows that our results are only due to the increased temporal correlation of the action noise signals, the marginal distributions remain unchanged from white noise.
14

---

Published as a conference paper at ICLR 2023
Appendix
where wy is Wiener process (integrated white noise with the property that w(t)-w(t') ~ (O t0
for any 0 S < t). This definition of Omsteei- Ueein-euh noise is equivalent to the Langevin equation (3) in the main text, but is nicer work with, the white noise process t is ill-defined as the derivative of the Wiener process. We sample OU noise signals by discretizing the equation above:
alt + Ar] = x(t] Ox ovvte
(6)
where E (N 1). Denoting 2, = x|t (with __1 = 0) and &, NN0, 1) for all e No. it can be seen that
Zo o ov Ateu
21 = z0 - Ar + AtE
σV At(1 0 + ov
22 = OV At - 00 +
+ + Ats2
St oV A Σ (1 0A1)
Thus, as sum of zero-mean Gaussian distributions, the marginal distribution is:
= =v))= (. 0. ((1 - -)4) (1-0000)
= =N =N -N000.0'1 Σ(1
0)1)2*)
T=0
The variance of this distribution is geometric series which converges as 0o
which holds if 0 < 0At < 2. It is interesting to note that 02: = 1, then Eq. (6) yields white noise, as it reduces to = av Ats. On the other hand, if fAt = 0, the equation describes integrated white noise (Brownian motion), which is known to have unbounded variance. If AAt 2, then the signal exhibits negative temporal correlation, which follows from Eq. (6). If the geometric series converges, then the limiting variance given by
(1 0A/)2'
We can thus ensure standard Gaussian marginal distribution (in the limit) by setting the noise scale "corrected" value of
0
At
which is how we set the OU noise scale throughout the paper to make the comparison with white and colored noise fair. Fig. A.3, it can be seen that this limiting marginal distribution reached fairly quickly. In Sec. B, we also report Orntten-UUlenbbec. results with the more common choice of a 1, which we find to generally perform slightly worse (cf. Fig. B.
If the variance is corrected, then 0At is the only parameter of OU noise, such that e.g. (0 = 0.3, At 1) is equivalent to (0 30, A/ = 0.01). This immediately follows by plugging Eq. (7) into Eq. (6), yielding
x (1 0At)2, V1- (1-001ii)
which only contains the product AAI as a parameter. In this paper we thus set A/ = 0.01 without loss of generality. In the main text we also only consider OU noise replacement for strongly correlated Brownian motion and always set = 0.15, as this is the most common default setting used practice." However, noted in the discussion above, -hnnnben-Uhienbeck noise can also exhibit intermediate temporal correlation between white noise and Brownian motion, by setting 0 < < 100 (i.e. 0 A 1). This raises the question of whether there is a certain parameterization of OU noise which is as general as pink noise.
"We chose these values for At and 0 because these are the default choices the RL libraries we consider (Raffin et al., 2021; Pardo, 2020). Lillierap al. (2016) also recommend 0 0.15. If the variance is not corrected (we report these experiments in Sec. B), then the choice of At does make difference.
15

---

Published as a conference paper at ICLR 2023
Appendix
Power Spectral Density
Bounded Integrator (H /nats)
Oscillator (E/ harmy)
10a
Pink
Pink
10a
10
10
10
10
10-2
10-
10-3
Pink
10
10-2
10 10 10
10
109
10
10
f
Area
Resonant frequency
Figure A.4: Left: Power spectral densities of OU noise. OU noise interpolates between white noise and Brownian motion by changing the cutoff frequency of a low-pass filter which filters white noise. Center: Entropy achieved by OU noise of different 0 the bounded integrator environment. No 0 achieves a higher worst-case entropy than pink noise. Right: Energy achieved by OU noise of different l on the harmonic oscillator environment. No @ achieves worst-case energy that comes close to the one of pink noise.
A.2 GENERALITY OF ORNSTEIN- CHLTEBENUHLLNBECKK NOISE
The way in which OU noise interpolates between white noise and Brownian motion by choosing E (0, 100) is very different to colored noise with E (0, 2). We have shown (e.g. in Fig. A.2) that colored noise with intermediate temporal correlation has power-law power spectral density with intermediate exponent (or slope in the log-log plot). the other hand, noise be interpreted low-pass filter. How "leaky" this integrator is, is controlled by the parameter A: 8 = 0 then the integrator is ideal, resulting in integrated white noise (Brownian motion with diverging variance). If = 100 (with At 0.01), then the integrator is "completely leaky" (an all-pass filter) and the white noise passes through without being integrated. In terms of the power spectral density this change in 0 corresponds to shifting the cutoff' frequency of the low-pass filter. This is shown on the left in Fig. A.4 for E {0.01, 0.03, 0.1, 0.3, 1, 10, 30, 100).
"leaky integration" of white noise, i.c. white noise passed through
For an action noise type to be general (cf. Sec. 6), we want it to work well on all environments. In the power spectral density plots, it can already be seen that pink noise distributes power over the frequencies much more "generally" than Omstein-Uhleebeen f. pink noise exhibits higher power than most values of 6, and all values of ( have lower power than pink noise at most frequencies. Why this makes pink noise more general action noise can be made more concrete by revisiting the bounded integrator and harmonic oscillator environments introduced in Sec. 6. The generality of noise measures how robust it is to the choice or parameterization of the environment: The most general noise type is the one which performs best on the most adversarial environment parameterization. Thus, the most general 0 for an environment parameterized by parameter a solves the following optimization problem:
noise of any 0: At any given frequency
max min perf(a, ().
where the performance metric parf 0) should be normalized appropriately such that the maximum performance attainable for different values of is identical. This can be ensured by simply dividing by the performance attained by the best 0 for each value of a::
max min perf(a, 0) a. parf(a.0)
(8)
generality(A)
This gives the worst-case performance of the most general noise in terms of the best possible performance achievable by changing the noise type this worst-case environment. By replacing the expression (arf(a, 8) by perf(a, pink) and removing the maximization over 6, we can also calculate the generality of pink noise.
16

---

Published as a conference paper at ICLR 2023
Appendix
As discussed in Sec. 6, the performance of noise type the bounded integrator and oscillator environments is given by the achieved entropy and energy, respectively. This is shown for all values (as well as for pink noise) in Fig. A.4, where the parameterization parameter is the environment size for the bounded integrator and the resonant frequency for the harmonic oscillator. It can already be seen that both environments, for each choice of f there exists a parameter where the performance of 0 is worse than the worst-case performance of pink noise. This can be quantified by calculating the generality of each / and pink noise these environments according to Eq. (8). On the bounded integrator, the maximum generality of OU noise is 77%, and on the oscillator environment the maximum generality is 9.1%. On both environments, the maximum attained by noise achieves generalities of 79% and 22% on the bounded integrator and oscillator environments, respectively. This gives further evidence that pink noise is good default.
Pink
B ADDITIONAL RESULTS
B.1
TD3
In addition MPO and SAC, we also performed all experiments from the main text on TD3. MPO and SAC parameterize stochastic policy, meaning they learn the action noise scale as function (s) the state. TD3, on the other hand, uses a deterministic policy, and the action noise added independently of the state. Usually, the noise scale is kept fixed over the course of training, and this our experiments as well. However, a is an important hyperparameter, and there
how we handle no single value that works well on all environments. Thus, we repeat experiments with all of the values a E {0.05, 0.1, 0.3, 0.5, 1}, and different random seeds.
In Fig. B.I. the results of the TD3 experiments with constant noise type shown in the form of bootstrap distributions for the expected average performance, and compared to the same experiments on MPO and SAC, well out. As we have an additional hyperparameter (σ), we first average the TD3 performance over all a values, before computing the average performance across tasks. The beneficial effect of pink noise
Fig. 3-like plot where the influence of the agent has been normalized
MPO + SAC + TD3
MPO
1.0
Average Performance
0.5
0.0
00.
-1.0
SAC
TD3
Average Performance
0.3
0.0
-0.5
-1.0
-1.5
WN 0.1
0.2
0.35
0.5
0.75
1.0
1.5
2.0
0U
WN 0.1
0.2
0.35
0.5
0.75
1.0
1.5
2.0
OU
B
Figure B.1: All three algorithms (MPO. SAC, TD3) show clear preference for pink action noise as measured by the average performance over the environments of Fig. 2. The results the OU experiments with the uncorrected noise scale of are marked with dotted median.
17

---

Published as a conference paper at ICLR 2023
Appendix
Figure B.2: Average performances across environments are combined from all values (incl. WN). It can be seen that TD3 is consistently out- performed by both MPO and SAC. A closer look at the mean perfor- mance over all 3 values on each individual environment reveals that TD3 is outperformed on all environ- ments by both MPO and SAC.
TD3
SAC
MPO
Average Performance
can be clearly seen on TD3 as well. this figure we also show the results of Ornstein-Uhlenberd noise with noise scale of a 1 rather than the corrected noise scale of Eq. (7). Incidentally, these results also confirm Fujimoto et al. (2018)'s finding that, on TD3, white noise and OU noise (with 0 = 0.15) perform similarly.
The reason why we did not include TD3 into the analysis of the main text, that we found TD3 to be consistently outperformed by both MPO and SAC. In Fig. B.2, the average performances across environments are combined from all values (incl. white noise), and shown for MPO, SAC and TD3. It can be seen that TD3 generally performs much worse than MPO and SAC. Looking the mean performance over all 3 values on each individual environment, TD3 is outperformed on all environments by both MPO and SAC. We thus decided exclude TD3 from our main analysis.
B.2 MPO & SAC
In the majority of this work, we measure performance in terms of the mean evaluation return over training process. We use this method, because implicitly measures both the final policy performance, and the sample efficiency (how quickly does the algorithm reach high performance). Most the data present is additionally normalized, which is necessary to aggregate performances over different environments, and thus it often not very clear how exactly to interpret the results (other than recognizing statistical significance). In this section, familiar terms, namely learning curves and final policy performance.
present some of results in more
To validate the approach of using the (mean) performance instead of the performance the final policy, we have reproduced the results in Fig. 3 using the final policy performance (mean evaluation return in the last 5% of the training process), shown in Fig. B.3. In Fig. B.4, we show learning curves of white noise, pink noise, and OU noise on all environments for MPO and SAC. Both visualizations confirm our takeaway that pink noise is better default action noise than white noise or OU noise. More detailed results can be found in Sec. H.
The bootstrap distributions for the expected average performance (such as in Figures 3, 4, B. and B.3) are constructed by randomly choosing one seed for each environment, yielding one scalar
0.4
Figure B.3: The average final per- formance is like the average perfor- mance (see Sec. 4), but only uses the evaluation returns of the last 5% of training, thereby measuring the quality of the final learned policy. This figure shows the same anal- ysis MPO and SAC Fig. 3, and demonstrates that pink noise is preferable also in terms of final pol- icy performance.
Average Final Performance
0.2
0.0
-0.2
-0.4
-0.6
WN
0.1
0.2
0.35
0.5
0.75
1.0
1.5
2.0
OU
18

---

Published as a conference paper at ICLR 2023
Appendix
MPO
Pendulum
SAC
MPO
Cartpole (b.)
SAC
1000
800
750
Return
600
WN
400
500
250
OU
Pink
MPO
Cartpole (s.)
SAC
MPO
Ball-In-Cup
SAC
800
600
750
Return
500
200
250
MPO
MountainCar
SAC
MPO
Hopper
SAC
100
50
Return
50
50
MPO
Walker
SAC
MPO
Reacher
SAC
1000
750
Return
400
500
200
250
МPO
Cheetah
SAC
МPO
Door
SAC
3000
600
Return
100
1000
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Interactions x10"
Interactions 1
Interactions x10*
Interactions 100
Figure B.4: Learning curves (median and interquartile range of evaluation returns) of the two baseline action noise types white noise (WN) and Crmstein-Uhlenlee (OU) noise, as well as our suggestion of pink noise. It can be seen that pink noise, while being better than both on all environments, is the best default choice. It is never outperformed by both white noise and OU noise, and routinely outperforms white noise (e.g. MountainCar), OU noise (e.g. Door), or both (e.g. Hopper).
(normalized) performance per environment, assuming all other variables like algorithm and noise type fixed. Averaging these normalized performances (the reason that performances are normalized on each environment is so that this averaging is reasonable) gives an estimate for the average performance across environments the given variables (e.g. noise type and algorithm). As there are S different random seeds (typically S 20), we can repeat this procedure S times (with resampling) and take the mean of all S average performance estimates, giving us an estimate for the expected average performance of the given variables. Doing this N times (we N 10²), the N estimates for the expected average performance can be collected into a bootstrap distribution, as shown in these figures.
19

---

Published as a conference paper at ICLR 2023
Appendix
C ENVIRONMENTS & ALGORITHMS
We evaluate our method on 10 different tasks (see Fig. 2). Most of these are from the DeepMind Control Suite (DMC, Tassa et al., 2018), but we also use OpenAl Gym (Brockman et al., 2016) and the Adroit hand suite (Rajeswaran et al., 2018). The respective sources and exact IDs of all environments are compiled in Table C.I. See Sec. for results on additional tasks.
Environment
Source
ID
Pendulum
DMC
pendulum (swingup)
Cartpole (b.)
DMC
cartpole (balance_sparse)
Cartpole (s.)
DMC
cartpole
Ball-In-Cup
DMC
ball_in_cup (catch)
MountainCar
Gym
MountainCarContinoooo
Hopper
DMC
hopper (hop)
Walker
DMC
walker (run)
Reacher
DMC
reacher (hard)
Cheetah
DMC
cheetah (run)
Door
Adroit
door-v0
Table C.I: Environments used in this work (see also Fig. 2).
For our experiments, we relied on the TD3 and SAC implementations in Stable-Baselines! (Raffin the Tonic RL library (Pardo, 2020). We only
et al., 2021), 1), as well as the MPO implementation used the default hyperparameters of these algorithms, provided by the libraries. Our code for using colored noise with these libraries is made available online at https: htt///tttuu com/ ma perriiuss lab/p p noi ne-rr..
D BANDIT METHOD DETAILS
To use bandit algorithm to select the action noise color for rollout, it is necessary to define the bandit reward, which should score rollout in terms of the 3 that was chosen. In our case, we use the rollout return (sum of rewards) as the score, ex- plained in Sec. 5.2. Additionally, we have to se- lect a list of colors ("bandit arms") to search over: B = (31, 32....,Bxx (with 3& E [0, 2]. Vk in our case). If we assume that the bandit rewards (= roll- out scores) are Gaussian distributed with a known standard deviation o, we can Bayesian inference to estimate the means (μ E RK) ) of the reward dis- tributions. A simple bandit algorithm we can use end this context is Thompson sampling, shown in Algo-
Algorithm D.1: Thompson Sampling
Input: Arms B = (B1.... . 3x). Reward distributions std Initialize m E TR K E
for E N do
Sample q N(m, Σ)
a < arg max (1.....K Ih
T. < Run rollout with f3a
T f score of rollout T
Do Bayesian update of m, K using
{a,, (,)j,,a
rithm D.1 (SK denotes the set of positive semi-definite K x K matrices). The relationships between the random variables are shown in the Bayesian network in Figure D.la.
There is second strong assumption in the Thompson sampling algorithm shown in Algorithm D. (similarly for other algorithms like UCB): it assumes that the reward distributions are stationary, i.e. that they don't 't change over time. This is not the case in the context of reinforcement learning: the rollout score rÜ is defined as the return, then, if the reinforcement learning algorithm works, it should naturally be the case that the policy improves over time, and thus, on average, r > ry for i >>> j. This setting of non-stationary bandit distributions can be addressed by using wliding-window approach (e.g. Garivier & Moulines, 2008): instead of updating the belief parameters m, with respect to the whole history of observations, only keep window of the last N rollouts.
There remains one other problem: how do we choose the prior parameters m and Σ and the variance 22 of the reward distributions? For E, the easiest solution is to assume independent arms, i.e. make
20

---

Published as a conference paper at ICLR 2023
Appendix
Ha:
T1
Pas
ij
N
N
N
N
04
Ti
m, Σ
Σ
i N
1, 1,...
(a)
(b)
Figure D.1: (a) A Bayesian bandit with Gaussian reward distributions. The rewards from arm k are sampled from ((ue r). Thompson sampling (Alg. D.1) can infer μ while trading off exploration and exploitation. (b) By introducing the constants and the algorithm can be made scale invariant by performing Thompson sampling with respect to the normalized reward r; = (r; - b)/c.
E diagonal. This is not necessarily the most efficient solution, one can imagine that two similar values will also perform similarly their rollouts." For m, the non-stationarity becomes a use the rollout return as score, these scores will probably be much lower at the beginning of training than at the end. Additionally, we might not even know the scale of returns in task. To account for this, it would be necessary to make the prior variances Σns very large/uninformed. Similarly, a needs to be large, to account for the unknown scale of the bandit reward spread. However, this would mean that many more samples (rollouts) necessary tighten the belief distributions. This a problem, especially because we only have a small set of rollouts when using the sliding-window method.
problem: again assuming
The ideal would be a bandit method which is invariant with respect to affine transformations of the rewards, in the sense that it would make no difference if all rewards were transformed be br + for some constants 6 > 0 and с E IR for all arms. In Fig. D.Ib, this situation is shown in a Bayesian network. Here, the generative process is almost the same as before (see Fig, D.),), except that the reward i; is scaled and translated by b and e are independent of the chosen arm and stay constant within the window, it is possible to optimize them via maximum marginal likelihood, given the window of past observations of
bi; + before observation. If, as shown, the constants
The bandit inference task is to infer the distributional means μ = (color indices) the belief distributions to (m = 0), because we want the normalized reward distributions to be centered around O. For now, don't fix Σ, but let it be any positive semi-definite K x K matrix. The generative model for is defined via the following prior and likelihood function:
""(1,.... (Mi,...., (4k) from the actions
) and rewards (rollout scores)
(r:) We set the prior means of
p(u Σ) N( 0,
(9)
p(r μ. p. b, c, II M(ra bpo, + c, (bo)2)
(10)
These lead to the following evidence/marrinall likelihood function:
Il p(r: ai b,,,,,,o,
(11)
II / p(r: b, du i
(12)
II / NN(ri be 1 + c, (bo)2 0, du
(13)
II [(r [e 0 c, (bo)² + be Ebca,)
(14)
We also tried different approach by using modified RBF kernel matrix to account for covariance between the arms, but the results essentially the same as with independent arms.
21

---

Published as a conference paper at ICLR 2023
Appendix
II W(r (0 ? Emjo)));
(15)
where we used canonical basis vectors to represent Na; c n.. For maximization, it convenient to work with the log-evidence:
logp(r a,b, Σ) = log
II NNrr b +
(16)
(c - r:))
Σ log (e((2mb²( Emiai))
(17)
2b² + -aya,)
=: ((b,c)
(18)
We can now maximize the evidence by setting the partial derivatives to 0:
0.L(b,c) x Σ - r:) =0
(19)
(e-r))²
O,L(b, = Σ b + 1 + Emaar)
(20)
Solving these equations gives us
== N
Σ n
(21)
1
Σ - rc-r;)²
6
(22)
N
o² Da.a.
Using these values, we can "reconstruct" the unsealed/normalized reward
(23)
b
and perform Thompson sampling with respect to P.. This normalized Thompson sampling algorithm, including the sliding window modification, is presented in Algorithm D.2.
Next, we want to show that this method is indeed in- variant to affine transformations of the bandit reward.
Algorithm D.2: Normalized TS
Input: Arms = (B1,... (01.....f8.. .,,,),,
Proposition 1. The posterior distribution over μ in the normalized bandit algorithm (Alg. D.2) is iden- tical for the observations T (Ti...., TN) and r' U/r + for all b' > and e' R. In other words, the algorithm is invariant to a scaling and translation of the rewards.
Window size N
Initialize
+- 0 E _KK E SK SK
+-1
for I E N do
i-- mod N
M min N)
Sample q ~ N(m, E)
Proof. In this setting, the observed rewards T; normalized to
arg max [1.....' K)
+
Run rollout with S.
r; score of rollout Ti
F; r; - -((r))
(24)
b(r)
b
M EM
(0-r,)²
with
N
1; + rime b
c(r)
ri
(25)
Do Bayesian update of m, using {az, 1 11
M
end
(c(r) - -
b(r)
(26)
N
σ²
Ea,a.
1=1
To prove the invariance of the algorithm, will simply show that this normalized reward is the same for both sets of observations, i.c. that = F'. Then, clearly, the posteriors p(p1 f) and p(u f') will also be the same. Expanding Fl, we get:
22

---

Published as a conference paper at ICLR 2023
Appendix
r' c(r))
(27)
b(r')
b'r + -cbr+))
(28)
b(b++c)
Ur+ Ei=1 (Un(
(29)
L5 Zij1 + (U': t))s
0² Enja.
Ur+ d 8 Σ N
Ei=1 P7 cd
(30)
N 1i rj + -br; 2)3
1 >i
+ Maye.
b'(r ( (dr))
(31)
1/2 (c(r) r()3
0² + Dasa
c(r
(32)
b(r)
(33)
Thus, we can conclude that the reward normalization indeed guarantees invariance affine reward transformations in algorithms such Thompson sampling.
With this reward normalization, the prior parameters (of m m1) and (of Σ =²2 become redundant. We have already set m = 0, and we now also set the prior variances Σ& to 1. This en- courages the algorithm to keep the normalized mean estimates μ4 approximately A 1)-distributed. The "likelihood" parameter a remains to be tuned, but now not necessary to account for the large uncertainty in the reward scale, is only concerned with the normalized reward. In experiments we always set = 1.
D.1 BANDIT VS. RANDOM
Although we found the normalized bandit algorithm (Alg. D.2) to work well simple non-stationary tasks, in the RL setting (for choosing 3) the performance was just as that of a random 8 selection for every roll- out. In Table D.I, list the results of a Welch /-test, testing for inequal- ity of the performance distributions achieved by the bandit algorithm and random / selection on every environ- ment. It can be seen that the two meth- ods are statistically indistinguishable. This shows that the bandit method does not work as intended, as "ran- dom arm selection" should be easy baseline to outperform. The reason for this is probably due to the rollout return not being informative enough as bandit reward signal.
Environment
Bandit of Random
p
Pendulum
X
0.98
Cartpole (b.)
X
0.09
Cartpole (s.)
0.67
Ball-In-Cup
X
0.87
MountainCar
0.54
Hopper
X
0.09
Walker
0.15
Reacher
X
0.70
Cheetah
0.20
Door
X
0.59
Table D.1: Bandit vs. Random (Welch I-test)
23

---

Published as a conference paper at ICLR 2023
Appendix
E SOLVING MOUNTAINCAR BY FFT
MountainCar is very simple environment. Although its dynamics are almost those of harmonic oscillator, there a difference to the oscillator environment from Sec. MountainCar's oscillation dynamics non-linear. At the bottom of MountainCar's valley (see Fig. 2). the small-angle approximation of a non-lineat oscillator may be used, but for the motion to go up to the top, the behavior is different from simple harmonic motion. Nevertheless, very simple open-loop control algorithm to solve this environment, by running one rollout without applying any action (just letting the mountain make the car go back and forth bit), then analyzing the resulting trajectory and inferring the hill's (small-angle) resonant frequency (via the Fast Fourier Transform algorithm). Finally,
this insight develop
control the car by simply swinging back and forth the resonant frequency. This algorithm, which works very well on this task, is shown below.
import gym
import numpy as np from scipy. fft import rfft
Initialize envi ronment
env gym.make
make ( MMMootatannn innttaainCaCCoott monnttannaronnnnccucuuuou- inucus-vO)
T env episode steps
Run a single rollout with no force. Save x-coordinate to 'x'. 10 obs env.resettyy
[obs [0]]
for in rango (T):
obs, t_ step((0]p
x. x.append (cbs (0) [0])
15
Find resonant frequency = highest peak of FFT (excluding DC) (np.arze awogargmaxxto (abs (rfft (x)) [1:1) 1) (T 1)
19
Action plan (harmonic excitation - np.sin (2nn2pppppi * ar.anange )
Test 1000 rollouts 1000 solved = 0
for i in (N)go(N): env reset
for
range (T):
r, if x solved +-
env step ([a[t]])
break
33
print (E"Solved: (solved/N 100:.0f)8.") prints "Solved: 1008."
F Toy ENVIRONMENT DETAILS
F.1 OSCILLATOR ENVIRONMENT
The oscillator environment of Sec. 6, which we make available online as a gym environment (https:/ //ttttubb com//nnoobeerartte hwt.//otoolloooooooooooooooloooa
models the 1-dimensional motion of particle of mass m, attached to the origin by an ideal spring of stiffness k damped with friction coefficient b, and driven by force (the action) F. This motion described by the ordinary differential equation
mī F kr,
(34)
where x is the particle's position. In our experiments we set the friction coefficient to zero, ie. the system is undamped. This setup is then called simple harmonic oscillator. The energy of the
24

---

Published as a conference paper at ICLR 2023
Appendix
oscillator is the sum of kinetic and potential energy:
1
E mu 2 + - kx
(35)
2
The resonant frequency is:
(36)
2x
As we want to configure the oscillator to have a given resonant frequency f. we need to find m and accordingly. To get unique solution, we impose second constraint: the energy at x and should be 2x If we now solve the two equations (35) and (36) for m and k, imposing the constraint on E, we get the solution
h: 4m
(37)
(38)
f
the resonant frequency, In Fig, F.I, few pure-noise trajectories (akin to Fig, 1) are shown on the oscillator environment.
White noise
Pink noise
OU noise
0.04
10-3
0.02
0.00
0.02
2.5
0.0
2.5
225
0.0
2.5
22
0.0
2.5
f 2 x 10
0.2
0.0
-0.2
-0.4
-0.8
x 10
-10
-15
-10
0
10
10
Figure F.1: Trajectories on the oscillator environment. For each of the 3 resonance frequencies f {0.002, 0.02, 0.2), we sample 5 action noise signals of length J of white noise, pink noise and OU noise. We parameterization than white noise and OU noise, and always manages to excite the oscillator up to certain amplitude. White noise and OU noise only work well in the high- and low-frequency regime, respectively.
10
what already shown in Fig. 5: pink noise is much less sensitive to the
25

---

Published as a conference paper at ICLR 2023
Appendix
F.2 BOUNDED INTEGRATOR ENVIRONMENT
The bounded integrator environment of Sec. 6 has very simple dynamics:
8t+1 clip(s; + dz. -c, c),
(39)
where Sg and is the parameter determining the size of the environment. Thus, the "area" in Fig. 5 is given by c². In Fig. 1, this parameter is fixed at 250, and in Fig. F.2 these trajectories (center row) are compared to trajectories on smaller (top row) and larger environment (bottom row), in a similar spirit to Fig. F.I.
White noise
Pink noise
OU noise
25
250
c= 1000
Figure F.2: Trajectories on the bounded integrator environment. For each of the 3 environment sizes E (25, 250, 1000), we sample 20 action noise signals of length 1000 steps white noise, pink noise, and OU noise. We can see what was already shown in Fig. 5: pink noise is less sensitive the parameterization than white noise (which is too slow to explore the medium and large environments) and OU noise (which, the medium and small environments, gets stuck at the edges and fails to explore the interior).
G ADDITIONAL ENVIRONMENTS
In addition to the environments described in Sec. C, we also ran experiments several tasks from the "MuJoCo-Marz" suite (https: //gtttb.....//////////////..bb com/ kwgwyu/wwaa..c /mu maze). The results are shown in Fig. G.1 in the form of learning curves and the average performance (cf. Sec. B.2) of each corrected,
noise type over all six environments. We tested white noise, pink noise, and ef. Sec. A.1) Aasttin-UUhenbedd noise, and trained MPO and SAC all environments for 10 interactions using 20 seeds. Pink noise again outperforms white noise and Ornstein- Ornstein-Uhlenber Uhlenbeck noise default choice across environments. These experiments conducted verify method on
26

---

Published as a conference paper at ICLR 2023
Appendix
MPO
Point, U-Maze
SAC MPO
Point, 4 R-Roomm
SAC
Avg. performance
Return
WN
0.6
Pink
0.4
MPO
Swimmer, U-Maze
SAC MPO
Swimmer, 4- R-Roms
SAC
Return
20
0.0
0022
-0.1
MPO
Ant, U-Maze
SAC MPO
Ant, R-Rooms
SAC
20
-0.6
Return
-0.8
-1.0
x 100
x 10
x 10
x 10
Figure G.1: Performances white noise, pink noise, and OU noise on several "MuloCo-Maze" tasks (all with sparse rewards). It can be that pink noise is the best default choice of the three, it has the highest average performance (Sec. B.2).
different set of problems and are thus not included in the main analysis. The results provide further evidence in support of our main takeaway: pink noise makes very good default action noise.
H DETAILED RESULTS
Final Performance
Mean Performance
Environment
Agent
WN
OU
Pink
WN
OU
Pink
Oracle
Anti
Gain
Pendulum
МPO
311
702
574
247
651
558
670
239
430
SAC
224
350
446
158
283
294
361
158
202
Cartpole (b.)
MPO
999
1000
1000
928
940
967
967
928
39
SAC
960
908
958
939
890
941
950
890
59
Cartpole (s.)
MPO
703
784
788
535
499
666
666
489
177
SAC
377
608
730
226
459
532
533
159
374
Ball-In-Cup
MPO
974
973
978
926
909
948
948
909
39
SAC
976
975
979
930
901
933
941
901
39
MountainCar
MPO
13
56
92
13
52
91
92
13
78
SAC
0
90
94
89
93
93
0
93
Hopper
MPO
25
62
108
14
34
69
69
14
54
SAC
89
94
119
43
53
77
80
43
36
Walker
MPO
530
377
448
384
284
363
390
284
106
SAC
593
506
602
437
363
471
472
363
108
Reacher
MPO
956
856
966
864
600
871
888
581
306
SAC
955
914
940
776
653
776
653
122
Cheetah
MPO
666
612
678
481
440
543
543
440
103
SAC
631
577
6440
469
439
483
502
439
63
Door
MPO
2586
2492
2909
1830
1376
2207
2207
1376
830
SAC
2192
1535
2195
1332
546
1183
1332
546
785
Table H.1: Comparison of final policy performance (see Sec. B.2) and mean performance over the
training process (Sec. 4) on all environments. Results are averaged across seeds, and shown for white noise (WN), Constein-Uhlenbedd noise (OU), and pink noise (Pink) action noise on MPO and SAC. Additionally, the Oracle and Anti-Oracle ("Anti"") performances are shown. The gain between these (rightmost column) represents the difference achievable by changing the noise type, and the basis for the "performance gain" measure used in Sec. 4.2.
27


Reviews:
[{"review_id": "_FzkEQY0At", "reviewer": "Reviewer_9xrD", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the impact of the choice of colored noise in popular off-policy RL algorithms, TD3, SAC and MPO. They find that pink noise—an intermediary form of colored noise between totally, temporally uncorrelated white noise and temporally-correlated red noise (Brownian motion)—consistently results in the best performance, averaged over 10 different continuous control environments taken from several popular environment libraries. Pink noise tends either outperform other choices of noise, as well as a bandit baseline that adapts the key hyperparameter, $\\beta$, controlling for the \"color\" of the noise, as well baselines that randomize or anneal the value of $\\beta$ from red ($\\beta=2$) to white noise ($\\beta=0$).", "summary_of_the_review": "Overall, this paper is very good. The authors provide a generally lucid account of the mechanics of colored noise and how it plays a role in modern, off-policy RL algorithms like TD3, SAC, and MPO. Their extensive experiments on standard continuous control benchmarks and toy environments provide compelling empirical evidence that pink noise is an ideal, default noise setting for these methods. This finding is significant, as it shows that the standard choices of white and OU noise are suboptimal. Using pink noise by default then provides a free improvement to this entire class of algorithms. This finding thus has significant impact on future work, which, by following the recommendation to use pink noise by default, should see generally improved performance on the problems to which these algorithms are applied. In turn, this improved performance may allow methods like TD3/SAC/MPO to be more viably applied to more challenging domains.", "strength_and_weaknesses": "### Strengths\n- The writing is overall quite clear. In particular, the exposition on colored noise was easy to digest and seemed quite comprehensive, especially combined with the discussion in the appendix.\n- The experimental setup seems rigorous, covering a wide range of popular environments for continuous control, and making use of bootstrapped intervals and significance tests to assess the significance of their results.\n- The results are strong, with clear, compelling evidence that pink noise should be used as the default noise choice for TD3/SAC/MPO as it consistency outperforms other choices, while being close in performance to the optimal noise choice when it is not the best choice.\n- The toy environment experiments provide further compelling evidence that pink noise performs well across two isolated challenges that commonly occur in continuous control environments.\n\n### Weaknesses\n- The **major weakness** of this paper is the missing justification for the choice of $\\sigma$ used in the baselines. It seems that $\\sigma$ was set to a fixed value across all experiments, so it is possible that the advantages of pink noise disappear for other settings. To make the results fully convincing, it is necessary for the authors to show that the specific $\\sigma$ used is the optimal $\\sigma$ for the non-pink noise baselines, and show that replacing the noise type with pink noise for these values of $\\sigma$ results in improved performance. If the authors can show this empirically or otherwise provide a sound argument for why the chosen $\\sigma$ is the optimal choice for the baselines, I will raise my rating to an 8, as I believe the paper is otherwise very good.\n- The authors assess only the mean returns of each method averaged over all time steps of training, across both SAC and MPO. This is certainly an unconventional metric. While it does track the area under the training curve, it is generally possible to have a large area under the curve with performance crashing later in training. Thus, these results should be supplemented with at least a measure of final performance or the training curves. The latter results are currently reported in the Appendix. The main results section can benefit from their inclusion.\n- Results for the task-specific, best noise baseline (Section 4.2) is based on 10 seeds, while the pink noise baseline reports 20 seeds. Why is the oracle evaluation only based on the first 10 seeds? This seems like a potentially unfair comparison.\n- Several section of the paper can benefit from refactoring of the content presented, additional citations, or generally clearer exposition (see the Clarity section of this review for more details)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "BA_9J7QBEi", "reviewer": "Reviewer_D8U1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work presents a thorough investigation of the type of used action noise in reinforcement learning.\n\nThe typically used noise types are either Gaussian noise or Ornstein-Uhlenbeck noise (OU), which is temporally correlated.\nThese two noise types belong to a family of colored noise, where Gaussian noise is white noise and OU noise is red noise.\nThe family of noise has a frequency spectrum proportional to $f^{-\\beta}$, where $\\beta=0$ corresponds to white noise (a flat frequency spectrum) and $\\beta = 2$ corresponds to OU noise (the frequency power spectrum decays). In the work they consider other noise types with $0 < \\beta < 2$. In particular, they determined that $\\beta=1$ (pink noise) performs well as a default setting.\nThe noise can be generated using an FFT based algorithm prior to starting the episode, stored into a vector and then selected from the vector (this is a sensible procedure and is often also used for other noise types).\n\nThey performed experiments across 10 different tasks taken from DMControl Suite, OpenAI Gym and Adroit hand Suite. They tested different noise types for the SAC, MPO and TD3 algorithms. Experiments were run with 20 seeds. \nTo aggregate the performance across tasks, they normalized the performance in each task to have 0 mean and variance 1.\nPink noise had the best average performance across all tasks. Moreover, pink noise achieved the highest score in 3/10 tasks and the difference to the best was statistically insignificant in 5/10 of the remaining tasks. Meaning that pink noise was comparable to the best in 8/10 tasks. The difference in performance for pink noise in the remaining two tasks was also low, and the work included much discussion and further analysis of these results.\n\nIn addition the work tested scheduling the noise color schedule, as well as selecting the noise type using a bandit algorithm. Neither of these was significantly better than simply using a fixed $\\beta=1$ pink noise.\n\nSection 6 included an intuitive explanation of why pink noise might be better than other noise types, e.g., Figure 4 showed that in a random walk, Gaussian noise does not disperse much from the center, OU noise quickly leaves the center and gets stuck in the edges of the environment, while pink disperses from the center while traveling through a greater range of the intermediate distances.\n\nThe analysis indicated that pink noise is a better default setting for the noise compared to the previous Gaussian and OU noises.", "summary_of_the_review": "This is a well written paper on a topic that is relevant to many RL researches. The experimental work was thorough, and the discussion was interesting. Pink noise may become a default noise setting for future RL algorithms. I think this paper is a clear accept.\n\n**Update**\nI thank the authors for their response. I believe my assessment remains adequate, and I am keeping the score.", "strength_and_weaknesses": "**Strengths:**\n- The experimental work was solid and thorough doing a proper statistical analysis of the results. I believe the results are convincing.\n- Noise is added into most RL algorithms, so this work is applicable to a wide range of researches.\n- The paper was well written and included interesting explanations.\n\n**Weaknesses:**\n- If I have to name a weakness, I would say that ultimately the type of noise is an unalluring component of the RL system, and the specific type does not seem to make or break the performance of the algorithm (e.g., in Figure B.4 the other noise types also perform mostly fine). Even though pink noise statistically performs better, the improvement is not a conceptual advance in terms of performance.\n- I didn’t like the title and I would recommend changing it (this doesn't affect the review).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AhdYVq-gp3L", "reviewer": "Reviewer_EijA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper suggested that colored noise for motor exploration in RL, in particular Pink noise ($\\beta$ = 1) is a better choice than using Gaussian white noise and OU noise in many control tasks. The method is simple and versatile for online RL, and effective on the being tested environments. The main contribution is that we may improve online RL performance on most task with a very simple modification to the motor noise.\n\n", "summary_of_the_review": "I think the paper touched an important yet not fully addressed question in RL -- the choice of motor noise. The proposed method could have the potential to be a common design choice in RL if the authors can show the effectiveness of pink noise using more diverse tasks. Overall, I slightly lean toward rejection, but there is a large space to improve.\n\n\n-------------\n### Update\n\nSince the rebuttal has addressed my major concern about OU-noise, I increased my score and recommend acceptance.", "strength_and_weaknesses": "**[Strength]**\n- The idea is simple, intuitive and agnostic to algorithm and task.\n- Great experimental results on robotic control tasks\n- The writing is easy to follow.\n\n**[Liminations & Questions]**\n- One major concern is about the choice of hyper-parameters for OU noise ($\\theta$, $\\Delta t$ and $\\sigma$ in Equation 6). I have tried OU noise for motor exploration with myself in many tasks. The OU noise oftern outperforms white noise (My choice is $\\theta$=0.3, $\\Delta t$=1). And $\\sigma$ is selected so that the standard deviation (STD) is approximately the same as the original Gaussian white noise with $\\sigma = STD * sqrt(\\theta) * sqrt(2)$. Since the paper searched $\\beta$ for colored noise,  a search of $\\theta$  and $\\Delta t$ should also be conducted for OU noise.\n\n- Also, I think a fair comparison is to keep the STD of white noise, colored noise and OU noise the same in experiments, did the authors do so?\n\n- Another limitation of this work is that it is unclear how pink noise works on tasks rather than MuJoCo-like robotic control tasks. Since the method is simple, it should be straightforward to apply on more diverse benchmarks. \n\n- Have the authors considered applying temporally-correlated noise on discrete action space? \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%