Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The work presents a thorough investigation of the type of used action noise in reinforcement learning.

The typically used noise types are either Gaussian noise or Ornstein-Uhlenbeck noise (OU), which is temporally correlated.
These two noise types belong to a family of colored noise, where Gaussian noise is white noise and OU noise is red noise.
The family of noise has a frequency spectrum proportional to $f^{-\beta}$, where $\beta=0$ corresponds to white noise (a flat frequency spectrum) and $\beta = 2$ corresponds to OU noise (the frequency power spectrum decays). In the work they consider other noise types with $0 < \beta < 2$. In particular, they determined that $\beta=1$ (pink noise) performs well as a default setting.
The noise can be generated using an FFT based algorithm prior to starting the episode, stored into a vector and then selected from the vector (this is a sensible procedure and is often also used for other noise types).

They performed experiments across 10 different tasks taken from DMControl Suite, OpenAI Gym and Adroit hand Suite. They tested different noise types for the SAC, MPO and TD3 algorithms. Experiments were run with 20 seeds. 
To aggregate the performance across tasks, they normalized the performance in each task to have 0 mean and variance 1.
Pink noise had the best average performance across all tasks. Moreover, pink noise achieved the highest score in 3/10 tasks and the difference to the best was statistically insignificant in 5/10 of the remaining tasks. Meaning that pink noise was comparable to the best in 8/10 tasks. The difference in performance for pink noise in the remaining two tasks was also low, and the work included much discussion and further analysis of these results.

In addition the work tested scheduling the noise color schedule, as well as selecting the noise type using a bandit algorithm. Neither of these was significantly better than simply using a fixed $\beta=1$ pink noise.

Section 6 included an intuitive explanation of why pink noise might be better than other noise types, e.g., Figure 4 showed that in a random walk, Gaussian noise does not disperse much from the center, OU noise quickly leaves the center and gets stuck in the edges of the environment, while pink disperses from the center while traveling through a greater range of the intermediate distances.

The analysis indicated that pink noise is a better default setting for the noise compared to the previous Gaussian and OU noises.
strength_and_weaknesses: **Strengths:**
- The experimental work was solid and thorough doing a proper statistical analysis of the results. I believe the results are convincing.
- Noise is added into most RL algorithms, so this work is applicable to a wide range of researches.
- The paper was well written and included interesting explanations.

**Weaknesses:**
- If I have to name a weakness, I would say that ultimately the type of noise is an unalluring component of the RL system, and the specific type does not seem to make or break the performance of the algorithm (e.g., in Figure B.4 the other noise types also perform mostly fine). Even though pink noise statistically performs better, the improvement is not a conceptual advance in terms of performance.
- I didn’t like the title and I would recommend changing it (this doesn't affect the review).
clarity,_quality,_novelty_and_reproducibility: **Clarity**

Exceptionally good clarity. My only comment is that it may be good to say that $f$ is the frequency in Definition 1.

**Quality**

The quality of the experimental analysis was very high.

**Novelty**

The novelty is incremental.

**Reproducibility**

I believe the work is reproducible as sufficient details were provided. They also said that they will release the code.
summary_of_the_review: This is a well written paper on a topic that is relevant to many RL researches. The experimental work was thorough, and the discussion was interesting. Pink noise may become a default noise setting for future RL algorithms. I think this paper is a clear accept.

**Update**
I thank the authors for their response. I believe my assessment remains adequate, and I am keeping the score.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies the impact of the choice of colored noise in popular off-policy RL algorithms, TD3, SAC and MPO. They find that pink noise—an intermediary form of colored noise between totally, temporally uncorrelated white noise and temporally-correlated red noise (Brownian motion)—consistently results in the best performance, averaged over 10 different continuous control environments taken from several popular environment libraries. Pink noise tends either outperform other choices of noise, as well as a bandit baseline that adapts the key hyperparameter, $\beta$, controlling for the "color" of the noise, as well baselines that randomize or anneal the value of $\beta$ from red ($\beta=2$) to white noise ($\beta=0$).
strength_and_weaknesses: ### Strengths
- The writing is overall quite clear. In particular, the exposition on colored noise was easy to digest and seemed quite comprehensive, especially combined with the discussion in the appendix.
- The experimental setup seems rigorous, covering a wide range of popular environments for continuous control, and making use of bootstrapped intervals and significance tests to assess the significance of their results.
- The results are strong, with clear, compelling evidence that pink noise should be used as the default noise choice for TD3/SAC/MPO as it consistency outperforms other choices, while being close in performance to the optimal noise choice when it is not the best choice.
- The toy environment experiments provide further compelling evidence that pink noise performs well across two isolated challenges that commonly occur in continuous control environments.

### Weaknesses
- The **major weakness** of this paper is the missing justification for the choice of $\sigma$ used in the baselines. It seems that $\sigma$ was set to a fixed value across all experiments, so it is possible that the advantages of pink noise disappear for other settings. To make the results fully convincing, it is necessary for the authors to show that the specific $\sigma$ used is the optimal $\sigma$ for the non-pink noise baselines, and show that replacing the noise type with pink noise for these values of $\sigma$ results in improved performance. If the authors can show this empirically or otherwise provide a sound argument for why the chosen $\sigma$ is the optimal choice for the baselines, I will raise my rating to an 8, as I believe the paper is otherwise very good.
- The authors assess only the mean returns of each method averaged over all time steps of training, across both SAC and MPO. This is certainly an unconventional metric. While it does track the area under the training curve, it is generally possible to have a large area under the curve with performance crashing later in training. Thus, these results should be supplemented with at least a measure of final performance or the training curves. The latter results are currently reported in the Appendix. The main results section can benefit from their inclusion.
- Results for the task-specific, best noise baseline (Section 4.2) is based on 10 seeds, while the pink noise baseline reports 20 seeds. Why is the oracle evaluation only based on the first 10 seeds? This seems like a potentially unfair comparison.
- Several section of the paper can benefit from refactoring of the content presented, additional citations, or generally clearer exposition (see the Clarity section of this review for more details)

clarity,_quality,_novelty_and_reproducibility: ### Clarity
The writing is overall clear and easy to follow, but there are several parts that can benefit from clearer writing or additional details:
- The introductory discussion on colored noise is unclear. For example, the discussion at the bottom of page 1 refers to the hyperparameter $\beta$ before Definition 1, which defines $\beta$ is provided. I recommend giving a concise, high-level explanation for colored noise upfront in the introduction, with an intuitive explanation for what $\beta$ controls.
- Overall, the introductory discussion on the connections between red noise and OU noise is quite confusing. At points, these two noise types seem to be treated synonymously, while at other points, these two noise types are explicitly framed as being similar, but different. To avoid this confusion, I recommend providing an intuitive definition of colored noise and their connection to OU noise in the introduction.
- Regarding the previous two points, which center on providing an upfront intuitive definition of colored noise and OU noise in the introduction, I suggest making Figure 4 the main Figure 1 for the paper, right on the first or second page. This figure does a great job of communicating an intuitive difference between the different choices of noise. The authors could also consider adding a fourth image for Brownian motion (red noise) for completeness. 
- TD3/SAC/MPO are the main RL algorithms studied, but there is no detail about them provided anywhere in the paper. I suggest providing a high-level summary of these methods (highlighting how they are related) and pointing the reader to a more detailed synopsis of their implementation in the Appendix.
- The top-level column labels in Table 2 are confusing, and can be removed to benefit clarity.
- Figure 1 shows only 9 of 10 environments. Why not show all 10 environments for completeness?
- In Table 1.0, the authors could consider highlighting the "1.0 (Pink)" by changing the text color to pink for clarity.
- The authors do not define $\text{CN}_T(\beta)$ on Page 3. Though its definition can be guessed from context, its definition should be explicitly stated.
- The intuition for why temporally-correlated noise can improve exploration compared to white noise should be more clearly explained with the appropriate citation in the second paragraph (e.g. to Osband et al, 2016, which is cited elsewhere).
- The claim at the end of the second paragraph should include a citation to prior works that provide evidence for the claim: "Too much exploration is not beneficial for learning a good policy, as all algorithms require that the state-visitation distribution is approximately on-policy." This claim is not intuitively true, because the methods in question are all _off-policy_ methods.

### Quality
The presentation is polished with clearly labeled figures.

### Originality
This appears to be the first study of the impact of noise color in off-policy RL for continuous control.

### Reproducibility
The authors provide their implementation as a supplemental zip and clearly describe their experimental setup and hyperparameter choices. Thus the results in this study should be reproducible.
summary_of_the_review: Overall, this paper is very good. The authors provide a generally lucid account of the mechanics of colored noise and how it plays a role in modern, off-policy RL algorithms like TD3, SAC, and MPO. Their extensive experiments on standard continuous control benchmarks and toy environments provide compelling empirical evidence that pink noise is an ideal, default noise setting for these methods. This finding is significant, as it shows that the standard choices of white and OU noise are suboptimal. Using pink noise by default then provides a free improvement to this entire class of algorithms. This finding thus has significant impact on future work, which, by following the recommendation to use pink noise by default, should see generally improved performance on the problems to which these algorithms are applied. In turn, this improved performance may allow methods like TD3/SAC/MPO to be more viably applied to more challenging domains.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper suggested that colored noise for motor exploration in RL, in particular Pink noise ($\beta$ = 1) is a better choice than using Gaussian white noise and OU noise in many control tasks. The method is simple and versatile for online RL, and effective on the being tested environments. The main contribution is that we may improve online RL performance on most task with a very simple modification to the motor noise.


strength_and_weaknesses: **[Strength]**
- The idea is simple, intuitive and agnostic to algorithm and task.
- Great experimental results on robotic control tasks
- The writing is easy to follow.

**[Liminations & Questions]**
- One major concern is about the choice of hyper-parameters for OU noise ($\theta$, $\Delta t$ and $\sigma$ in Equation 6). I have tried OU noise for motor exploration with myself in many tasks. The OU noise oftern outperforms white noise (My choice is $\theta$=0.3, $\Delta t$=1). And $\sigma$ is selected so that the standard deviation (STD) is approximately the same as the original Gaussian white noise with $\sigma = STD * sqrt(\theta) * sqrt(2)$. Since the paper searched $\beta$ for colored noise,  a search of $\theta$  and $\Delta t$ should also be conducted for OU noise.

- Also, I think a fair comparison is to keep the STD of white noise, colored noise and OU noise the same in experiments, did the authors do so?

- Another limitation of this work is that it is unclear how pink noise works on tasks rather than MuJoCo-like robotic control tasks. Since the method is simple, it should be straightforward to apply on more diverse benchmarks. 

- Have the authors considered applying temporally-correlated noise on discrete action space? 


clarity,_quality,_novelty_and_reproducibility: **Clarity**

The paper is mostly clear and well-written. 

**Novelty**

The idea of using temporally-correlated noise is not totally new, while pink noise has not been comprehensively investigated. I would say the novelty is moderate, thus its effectiveness should be more focused.

**Quality**

The idea is most heurestic and the results are mostly empirical. The paper will be impactful in RL if the authors can consolidate their results with my suggestions above.

**Reproducibility**

It should be easy to reproduce though I have not checked the code.


summary_of_the_review: I think the paper touched an important yet not fully addressed question in RL -- the choice of motor noise. The proposed method could have the potential to be a common design choice in RL if the authors can show the effectiveness of pink noise using more diverse tasks. Overall, I slightly lean toward rejection, but there is a large space to improve.


-------------
### Update

Since the rebuttal has addressed my major concern about OU-noise, I increased my score and recommend acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies RL in control problems with continuous action spaces. A common default is to generate exploration by perturbing selected actions by random Gaussian noise that is chosen independently across time (White noise). This kind of noise fails to generate 'deep exploration' that is required in many problems. Lillicrap et. al [2016] inject action noise generated according to an Ornstein-Uhlenbeck process (red noise), which can force the agent to persistently perturb actions in a particular direction. The current paper studies a parameterized class of noise distributions that essentially interpolates between these cases. A particular choice of 'pink noise' is found empirically to offer better default across ten environments than either extreme. 

*Strengths*: While the use of temporally correlated noise is not new, the authors have udnerstood the importance of this design choice, evaluated it clearly, and offered intuition on when it is most important. It terms of new methodology, pink noise does, indeed, appear to be a nice default. Pragmatically speaking, when don't understand who to reach a scientific ideal (i.e. systematic exploration), it is helpful to have a few tuning parameters. The beta parameter that interpolates smoothly between white noise (beta=0), pink noise (beta =1) and colored noise (beta=2), could really help a practitioner. 

All reviewers felt the paper was clear accept. 

*Weaknesses:*  Adding temporally correlated noise is not, in general, sufficient to generate efficient exploration and overcome the deficiencies of white noise. It would be helpful if the paper offered some discussion of when temporally correlated noise might be enough, rather than simply tuning the degree of correlation. The paper focuses on an extremely narrow range of exploration methods. 
justification_for_why_not_higher_score: See weaknesses section. Interpolating between known, heuristic, exploration methods is useful. It can be enlightening. But there is a pretty high bar for oral presentations and reaching it likely requires more substantial new thinking. 
justification_for_why_not_lower_score: The paper is very well executed and offers some practical guidance. The reviewers are unanimous in feeling the paper deserves clear acceptance.
Decision: Accept: notable-top-25%