Perlin Noise for Exploration in Reinforcement Learning

Authors: Dominik Roth, Onur Celik, Fabian Otto, Gerhard Neumann
Primary area: reinforcement learning
Keywords: Reinforcement Learning, Exploration Strategies, Perlin Noise, Policy Optimization, Structured Exploration

Abstract:
Reinforcement Learning (RL) enables agents to solve tasks by autonomously acquiring policies by interacting with the environment receiving sparse or noisy feedback in the form of a reward. However, achieving successful optimization in RL requires efficient exploration, which remains a significant challenge, particularly in continuous action spaces. Existing exploration techniques often exhibit limited state-space reach and fail to overcome local optima, resulting in suboptimal policies. Additionally, these techniques can cause erratic movements, posing risks when applied to real-world robots.
In this work, we introduce a novel exploration strategy leveraging Perlin Noise, a gradient noise function that generates smooth, continuous disturbances, thus enhancing the agent's performance by promoting structured exploration and fluid motions. We quantitatively demonstrate the benefits of our approach compared to state-of-the-art methods, showing that it outperforms both unstructured and structured techniques in thorough experimental evaluations.

Decision: reject
Reviewer scores: 3, 6, 5

Reviews and discussion:
<thread>
creation date:
1730149854767

thread_type:
official_review

<thread_object>
creator: reviewer_nqa9

<content>
summary:
This paper proposes a new exploration strategy derived from Perlin Noise for Reinforcement Learning tasks. Instead of using classical Gaussian Noise, the authors found that Perlin noise could explore the hard-exploration task more effectively and smoothly. To allow Perlin noise to be used for training the RL algorithms, the authors conduct a theoretical work which shows the asymtotic normality of Perlin noise.  A normalization function is also proposed to map the Perlin noises to be gaussian-like distribution. Experiments are conducted in various task sets.

strengths:
1. The natural property of Perlin Noise such as smoothness can be particularly helpful for robotics tasks as smoother actions are usually perferred due to safety concern.
2. The authors have an extensive proof of the concepts mentioned in the paper.
3. Experiments are carried out in different tasks.

weaknesses:
1. Clarity is my biggest concern. Many math symbols, terms and performance criterion are undefined. Such as t and i in line 211 or the Mean squared Jerk are not properly defined. The procedure of the whole scheme is shown in Fig. 4. However, how normalize function is calculated exactly and what hyperparameters are there are not explicitly written.
2. The results produced by the authors are also different from existing work. Pink noise [1] shows that they are outperforming white noise in many tasks and the results are evaluated in a statistical way. But in this work, pink noise seems to be significantly worse than white noise. The overall performance of Perlin noise and white noise in terms of rewards is also very close to each other. 
3. The authors claim that Perlin noise can be used for hard-exploration tasks. But in the selected tasks, there aren't many tasks involving sparse rewards.
4. Smoothness would be the key benefits of using Perlin Noise in robotics. Thus, to fairly compare the benefits, it would be more interesting to compare white/pink noise with a simple PID as the filter.
5. Computational cost is not mentioned. How much extra compute cost would Perlin noise bring?
6. No ablation study has been carried out to show the robustness of Perlin noise

[1] Eberhard, Onno et al. “Pink Noise Is All You Need: Colored Noise Exploration in Deep Reinforcement Learning.” International Conference on Learning Representations (2023).

questions:
Some questiosn are mentioned in the weakness part. 
1. Could you please briefly describe the hyperparameter tuning for Perlin noise and other baselines? This would also help for understanding why Pink noise performs so badly.
2. Could you further elaborate how normalize function is in practice calculated?
3.  How much differences in terms of training wallclock time does perlin and other baselines have?
4. How robust is Perlin noise when different t_offset or t_speed are picked?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730304842082

thread_type:
official_review

<thread_object>
creator: reviewer_x2qd

<content>
summary:
This paper introduces Perlin noise to enhance stochastic policy distributions in reinforcement learning (RL) by encouraging smoother exploration. Perlin noise assigns random gradient vectors to points on a grid and interpolates between them, creating continuous transitions across space, a technique commonly used in computer graphics. The paper details the process of sampling Perlin noise and how it can be leveraged to improve RL policies.

strengths:
The paper leverages the Perlin noise, which is commonly used in computer graphics to generate smooth textures, effectively generate smooth and temporally related noise for RL policies. The Perlin is introduced mainly aiming to generate smoother and more coherent trajectories, thus leading to more nature movements, which is very important in robotics.

weaknesses:
1. The paper claims that using Perlin noise broadens exploration, but this aspect is neither theoretically justified nor sufficiently demonstrated experimentally.

2. While the paper compares several noise types, all approaches focus on adding noise to stochastic policy distributions. Many other exploration methods exist, such as entropy-regularized exploration [1], curiosity-driven exploration [2], intrinsic motivation-based approach [3], reward shaping [4], and novelty-guided exploration [5]. I believe comparisons with some other advanced exploration strategies besides the policy noise based approaches are necessary to fully demonstrate the advantages of Perlin noise in RL.

[1] Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pp. 1861–1870. PMLR, 2018.

[2] Pathak, Deepak, et al. "Curiosity-driven exploration by self-supervised prediction." International conference on machine learning. PMLR, 2017.

[3] Burda, Y., Edwards, H., Storkey, A., and Klimov, O. Exploration by random network distillation. In International Conference on Learning Representations, 2018.
 
[4] Devidze, R., Kamalaruban, P., and Singla, A. Exploration guided reward shaping for reinforcement learning under
 sparse rewards. Advances in Neural Information Processing Systems, 2022.

[5] Tang, H., Houthooft, R., Foote, D., Stooke, A., Xi Chen, O., Duan, Y., Schulman, J., DeTurck, F., and Abbeel, P. #
 exploration: A study of count-based exploration for deep reinforcement learning. Advances in Neural Information Processing Systems, 2017.

questions:
1. Regarding Figure 5, I understand that the sampled action trajectories are drawn from a fixed underlying distribution, and it is evident that Perlin noise generates much smoother trajectories. However, these smooth trajectories (e.g., over 100 actions) are based on a fixed underlying distribution. In real RL learning or inference, the stochastic policy distribution $\pi(\cdot | s_t)$ is conditioned on the current state, meaning the policy distribution changes at each step. In this scenario, does the sequence smoothness still hold? In other words, would generating Perlin noise for a changing policy trajectory still result in smooth action sequences?

2. Does the use of smooth action trajectories imply that action changes are relatively slow? Could this affect the speed of learning or inference? For instance, in scenarios that require rapid action adjustments (such as emergency braking in autonomous driving), would this smoothness impact the effectiveness of the policy?

3. Besides generating smooth action sequences, does Perlin noise effectively increase the diversity of the sampled actions? In model-free RL, efficient exploration, i.e., collecting more diverse samples, is critical for improving sample efficiency. How does the introduction of Perlin noise influence the breadth of exploration?

I would like to increase the scores if these concerns are addressed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730674016755

thread_type:
official_review

<thread_object>
creator: reviewer_qdjk

<content>
summary:
This paper proposes using Perlin noise as a drop-in replacement to other forms of noise for any RL algorithm, and demonstrates improvements in exploration in terms of both final performance and smoothness on various tasks in many domains.

strengths:
- The paper is well written with extensive evaluations on various tasks and in various domains.
-  Naively sampling perlin noise and applying it to the actions of the policy makes it impossible to use the reparameterization trick to compute gradients as in SAC. The authors propose a novel approach of first applying perlin noise to the actions and then modeling the resulting policy using a gaussian distribution using the “Normalize” trick. I agree with the authors that this approach is novel and makes it fairly straightforward for any RL practitioner to use it as a drop-in replacement for other forms of exploration noise. 
- The authors demonstrate good performance improvements on select difficult exploration tasks such as Ant maze (Fig. 7) and provide empirical evidence that Perlin noise leads to smoother action trajectories (Fig. 6).

weaknesses:
- Perhaps my main concern with this paper is that, at a high level, it simply proposes and investigates a different noise function for PPO. While simple solutions especially if they solve major problems are not necessarily bad, the novelty and technical contribution here is quite limited.

- Since the claim is that Perlin noise provides more smooth, structured exploration independent of the algorithm or task, it seems necessary to have at least one more algorithm beyond PPO to validate this claim. 

-  Figure 5 needs some context on what the task is, what kind of policy was used to collect the trajectory (is it a random policy? Trained policy? Hand-engineered one?), etc. For example, PPO has a learnable action standard deviation parameter that generally tends towards 0 i.e. a deterministic policy as the policy converges. I would then expect that the trajectory, regardless of what kind of noise is applied, to look relatively smooth. It is impossible to make any conclusions without further context here. 

- Final minor complaint: On lines 59-60 the authors contrast against intrinsic-motivation and trajectory-level noise methods by saying  *“However, these methods require additional treatment of the underlying optimization method by changing the objectives or introducing higher-level abstractions of the actions.”*.  Why is this a problem? This is similar to saying “PPO does better than REINFORCE, however it requires additional treatment of the underlying optimization method”. If that “underlying treatment” results in a fundamentally better algorithm, then the research community will adopt this new approach. This, in my opinion, is not a bad thing but rather the natural progression of a field. 

Overall, I appreciate the simplicity of the method, the extensive experiments on a variety of domains, and generally agree with the conclusions the authors draw. However, I don't believe that proposing a different noise function for just PPO is sufficient for publication as a conference paper. Accepting this paper sets a precedent for others to just try different noise functions on different algorithms on different tasks, and there are a countless number of ways to achieve "novel" results this way, none of which actually help the field progress. Perhaps a comprehensive study of many different noise functions on different algorithms and tasks, and a deeper analysis of why certain types of noise do well on certain algorithms / tasks and not others would warrant a full paper. The work the authors present here would do better in the blogpost or workshop track.

questions:
No questions at this time.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731943745247

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to Reviewer Feedback and Paper Changelog

comment:
We extend our sincere thanks to all reviewers for their detailed and constructive feedback. Your insights, particularly regarding the need for added context around certain figures and explanations, have highlighted areas where our work can be made clearer and more accessible. We hope that our response below addresses the collective concerns raised, and we look forward to further improving our paper based on these helpful observations.

Since multiple reviewers asked questions regarding Figure 5:
Figure 5 is referenced in Section 3.3, which does contain, what we believed to be a sufficient description. But since multiple reviewers pointed out a lack of context surrounding it, it seems like either the description is not clear or should be repeated in the subtitle of the Figure. A more elaborate explanation:

We can take two separate views of how we use Gaussian action sampling in RL:
 1. We define the policy as a Gaussian Policy. In Vanilla PPO, we span the mean via a NN, the variance is a parametric vector, which is also optimized via gradient descent, but is not dependent on the observation.
 2. We assume the policy NN to provide us with an deterministic action, which we then disturb via a random sample from Gaussian Noise with zero mean.

Mathematically, both views are equivalent; they differ only in where we draw the line of what is part of the policy definition. While the first view is common in the literature, we will adopt the second in our explanation.

In Section 3.3, we investigate the behavior of the tested noises initially in isolation from an RL setup using a static Gaussian Policy (N(0,1)). In an actual RL setup (as evaluated in section 4), this is the noise sample used to disturb the policy mean / deterministic action.
Switching from Colored noises to Perlin noise allows us to ensure, that these disturbances are smooth, slowly varying over time. This is independent of the mean of the action policy: If the NN does not provide smoothly varying means, the resulting action will also always be unsmooth. But following the proposed NN weight initialization of the original PPO implementation, these will be smooth in the beginning and we do not expect the network to learn to parameterize jerky means unless it is required to solve the task.

We worked this and other changes into the document:  
- **Added Definitions for \(i\) and \(t\) in Section 3.1 (Line 210):** We have now explicitly defined the indices \(i\) and \(t\) to avoid confusion, ensuring that the notation is clear and consistent throughout the text.  

- **Described the Computational Complexity of Perlin Noise Using Our Sampling Mechanism (Section 3.1, Line 230):** We have added an explanation of the computational complexity of our Perlin noise sampling mechanism to Section 3.1. This ensures a clear understanding of the trade-offs and efficiency of the method in terms of computational cost.

- **Expanded Figure 5 and Section 3.3 for Better Clarity:** In response to feedback, we have added additional context and details to Figure 5 and Section 3.3. This aims to make the figure’s purpose and content more transparent.  

- **Added Formalization and Calculation Description of 'Mean Squared Jerk' to Appendix (B.1):** To address concerns about clarity in the jerk calculation, we have formally defined the Mean Squared Jerk (MSJ) metric in Appendix B.1.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This works proposes to replace standard Gaussian exploration in on-policy RL methods usign Perlin noise based exploration, which is smooth and temporally correlated. The key innovation is to alter the noise structure and then propose a reparameterization with a new "normalize" function that enables the use of a standard likelihood or reparameterization based RL method. The methodology is investigated across several tasks in simulation. 

Strengths: 
The idea of Perlin noise is neat, seems really useful for most RL problems with temporally extended decision making. 
The methodology to enable likelihood/reparameterization based RL is neat!

Weaknesses: 
Really the major weakness of the paper is that the empirical results are not strong enough - the gap between Perlin and other methods is relatively small and not very marked. 
I think the authors' argument of discounting all intrinsic exploration methods is not a completely justified one. If the argument is that these are complementary methods, this should really be demonstrated in the paper. 

Overall the paper holds a lot of merit, but a stronger empirical comparison, more baselines that do exploration and a comparison on sparse reward tasks would add a lot of value.