Title: A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Reinforcement Learning

Hyperparameters

Empirical Methodology
Abstract: The performance of modern reinforcement learning algorithms critically relies
on tuning ever increasing numbers of hyperparameters. Often, small changes in
a hyperparameter can lead to drastic changes in performance, and different environments require very different hyperparameter settings to achieve state-of-the-art
performance reported in the literature. We currently lack a scalable and widely
accepted approach to characterizing these complex interactions. This work proposes a new empirical methodology for studying, comparing, and quantifying the
sensitivity of an algorithm’s performance to hyperparameter tuning for a given set
of environments. We then demonstrate the utility of this methodology by assessing
the hyperparameter sensitivity of several commonly used normalization variants of
PPO. The results suggest that several algorithmic performance improvements may,
in fact, be a result of an increased reliance on hyperparameter tuning.

A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning

Abstract

The performance of modern reinforcement learning algorithms critically relies on tuning ever increasing numbers of hyperparameters. Often, small changes in a hyperparameter can lead to drastic changes in performance, and different environments require very different hyperparameter settings to achieve state-of-the-art performance reported in the literature. We currently lack a scalable and widely accepted approach to characterizing these complex interactions. This work proposes a new empirical methodology for studying, comparing, and quantifying the sensitivity of an algorithm's performance to hyperparameter tuning for a given set of environments. We then demonstrate the utility of this methodology by assessing the hyperparameter sensitivity of several commonly used normalization variants of PPO. The results suggest that several algorithmic performance improvements may, in fact, be a result of an increased reliance on hyperparameter tuning.

1 Introduction

The performance of reinforcement learning algorithms critically relies on the tuning of numerous hyperparameters. With the introduction of each new algorithm, the number of these critical hyperparameters continues to grow. Consider the progression of value-based reinforcement learning algorithms, starting from DQN (Mnih et al., 2015), which has 16 hyperparameters that the practitioner must choose, to Rainbow (Hessel et al., 2018) with 25 hyperparameters. This increase can be observed in Figure 1. This proliferation is problematic because performance can vary drastically with respect to hyperparameters across environments. Often, small changes in a hyperparameter can lead to drastic changes in performance, and different environments require very different hyperparameter settings to achieve the reported good performances (Franke et al., 2021; Eimer et al., 2022, 2023; Patterson et al., 2024). Generally speaking, hyperparameter tuning requires a combinatorial search and thus many published results are based on a mix of default hyperparameter settings and informal hand-tuning of key hyperparameters like the learning rate. Our standard evaluation methodologies do not reflect the sensitivity of performance to hyperparameter choices, and this is compounded by a lack of suitable metrics to characterize said sensitivities.

There are many different ways one could characterize performance with respect to hyperparameter choices in reinforcement learning, but the community lacks an agreed standard. Hyperparameter

Number of

hyperparameters

41

34

Q-learning

Actor critic

R-learning

LSPI

NFQ

A3C

DQN

PER

PPO

SAC

Rainbow

DynaQ+

DynaQ

Actor Critic with

eligibility traces

Agent57

DreamerV2

DreamerV1

16

3

24

1983

1993

2023

2003

2013

Year

sensitivity curves, such as those found in the introductory textbook (Sutton & Barto, 2018), summarize performance with respect to several values of a key hyperparameter producing U-shaped curves. Sensitivity curves do not work well with many dimensions of hyperparameters, nor is there a wellestablished way to use them to summarize performance in multiple environments. If computation is of no object, then performance percentiles can be used to compute the likelihood that an algorithm will perform well if its hyperparameters are randomly sampled from some distribution (Jordan et al., 2020). Although very general, this approach does not reflect how practitioners tune their algorithms. The Cross-environment Hyperparameter Benchmark (Patterson et al., 2024) compares algorithms by a mean normalized performance score across environments but ultimately focuses on the possibility of finding a single good setting of an algorithm's hyperparameters that performs well rather than characterizing sensitivity. What performance-only metrics lack is a measurement of what proportion of realized performance is due to per-environment hyperparameter tuning.

We propose an empirical methodology to better understand the interplay between hyperparameter tuning and an reinforcement learning agent's performance. Our methodology consists of two metrics and graphical techniques for studying them. The first metric, called an algorithm's hyperparameter sensitivity , measures the degree to which an algorithm's peak reported performance relies upon perenvironment hyperparameter tuning. This metric captures the degree to which per-environment tuning improves performance relative to the performance of the best-fixed hyperparameter setting across a distribution of environments. The second metric, named effective hyperparameter dimensionality , measures how many hyperparameters must be tuned to achieve near-peak performance. It is often unclear how important specific hyperparameters are and if they should be included in the tuning process. These two metrics can help us better understand existing algorithms and drive research toward algorithmic improvements that reduce hyperparameter sensitivity.

We validate the utility of our methodology by studying several variants of PPO (Schulman et al., 2017) that have been purported to reduce hyperparameter sensitivity and increase performance. We performed a large-scale hyperparameter study over variants of the PPO algorithm consisting of over 4.3 million runs (13 trillion environment steps) in the Brax MuJoCo domains (Freeman et al., 2021). We investigate the relationship between performance and hyperparameter sensitivity with several commonly used normalization variants paired with PPO. We found that normalization variants, which increased PPO's tuned performance, also increased sensitivity. Other normalization variants had negligible effects on performance and marginal effects on hyperparameter sensitivity. This result contrasts the view that normalization makes reinforcement learning algorithms easier to tune and, as a consequence, results in improved performance.

2 Problem Setting and Notation

We formalize the agent-environment interaction as a Markov Decision Process (MDP) with finite state space S and action space A , bounded reward function R : S × A ×S → R ⊂ R , transition function P : S × A ×S → [0 , 1] , and discount factor γ ∈ [0 , 1] . At each timestep t , the agent observes the state S t , selects an action A t , the environment outputs a scalar reward R t +1 and transitions to a new state S t +1 . The agent's goal is to find a policy, π : A×S → [0 , 1] , that maximizes the expected return, G t . = R t +1 + γR t +2 + ... , in all states: E π [ G t | S t = s ] for all s ∈ S .

Most reinforcement learning agents learn and use approximate value functions in order to improve the policy through interaction with the world. The state-value function v π : S → R is the state conditioned expected return following policy π defined as v π ( s ) . = E π [ G t | S t = s ] . Similarly, the action-value function q π : S × A → R provides the state and action conditioned expected return following policy π defined by q π ( s, a ) . = E π [ G t | S t = s, A t = a ] . The advantage function A π ( s, a ) . = q π ( s, a ) -v π ( s ) describes how much better taking an action a in state s is rather than sampling an action according to π ( ·| s ) and following policy π afterward. Given an estimate of the value function ˆ v , an agent can update estimates of the value of states based on estimates of the values of successor states. An n -step return is defined as G t : t + n . = R t +1 + γR t +2 + · · · + γ n -1 R t + n + γ n ˆ v ( S t + n ) . A mixture of n -step returns, called a truncatedλ return can be created by weighting n -step returns by a factor λ ∈ [0 , 1] , G λ t : t + n . = (1 -λ ) ∑ n -1 j =1 λ j -1 G t : t + j + λ n -1 G t : t + n . Truncatedλ returns are often used in reinforcement learning algorithms such as PPO (Schulman et al., 2017) to estimate state values and state-action advantages, which are used to approximate the value function and improve the policy.

Of particular interest to this work is the setting where an empiricist is evaluating a set of algorithms Ω across a distribution of environments E . Each algorithm ω ∈ Ω is presumed to have some number of hyperparameters n ( ω ) . Each hyperparameter h i , 1 ≤ i ≤ n ( ω ) is chosen from some set of choices H ω i . The total hyperparameter space H ω is defined as a Cartesian product of those choices H ω . = H ω 1 × H ω 2 × ...H ω n ( ω ) . Once an algorithm ω ∈ Ω and a hyperparameter setting h ∈ H ω are chosen, the tuple ( ω, h ) specifies an agent .

3 Hyperparameter Sensitivity

First, we must define what we mean by sensitivity. A sensitive algorithm is an algorithm that requires a great deal of per-environment hyperparameter tuning to obtain high performance. Conversely, an insensitive algorithm is one where there exist hyperparameter settings such that the algorithm can obtain high-performance across a distribution of environments with fixed hyperparameters.

This section presents two contributions: a metric for assessing an algorithm's hyperparameter sensitivity, and a method of graphically analyzing the relationship between hyperparameter sensitivity and performance along a 2-dimensional plane. These tools may be used to develop a deeper understanding of existing algorithms and we hope will aid researchers in evaluating algorithms more holistically along dimensions other than just benchmark performance.

3.1 Sensitivity Metric

We want a performance metric that summarizes the learning of an online reinforcement learning agent. The natural choice of performance metric is to report the average return obtained during learning, which we call the area under the (learning) curve (AUC). The AUC on a run is denoted by p ( ω, e, h, κ ) where ω ∈ Ω is an algorithm, e ∈ E is an environment, h ∈ H ω is a hyperparameter setting, and κ ∈ K ⊂ N is the random number generator (RNG) seed. Performance observed during a run of a reinforcement learning agent depends on many factors: the reinforcement learning algorithm, the environment, the hyperparameter setting, and many forms of stochasticity. Even after fixing the algorithm, environment, and hyperparameter setting, performance distributions are often skewed and multi-modal (Patterson et al., 2023). Therefore, many runs are required to obtain accurate estimates of expected performance ˆ p ( ω, e, h ) . = 1 |K| ∑ κ ∈K p ( ω, e, h, κ ) where K ⊂ N is the set of RNG seeds used during the experiment. In the experiments presented in this paper, we perform 200 runs averaging performance over a subset of runs, after filtering (described later), and report 95% bootstrap confidence intervals around computed statistics.

It is crucial to capture performance across sets of environments in order to compute sensitivity. Recall, that our notion of sensitivity captures the degree to which an algorithm relies on per-environment hyperparameter tuning for its reported performance gains. Our choice of AUC as a performance metric does not allow for cross-environment performance comparisons directly because the magnitudes of returns vary greatly between environments. Consider the distributions of performance presented in the left plot in Figure 2. The performance realized by good hyperparameter settings in Halfcheetah is orders of magnitude greater than the performance of good hyperparameter settings in Swimmer. Nevertheless, just because the absolute magnitude is lower or the range of observed performances is tighter, that does not mean the differences are any less significant. Thus, in order to consider how hyperparameters perform across sets of environments, we need to normalize performance to a standardized score.

In this work, we use [5 , 95] percentile normalization. We choose percentile normalization as it has a lower variance than alternatives like min-max normalization. Other normalization methods, such as min-max or CDF normalization (Jordan et al., 2020), could also be used with our hyperparameter sensitivity formulation. After conducting a large number of runs across different algorithms, environments, and hyperparameter settings, for each environment e , we find the 5th percentile p 5 ( e ) and 95th percentile p 95 ( e ) of the distribution of observed performance in e . Then, for each algorithm, environment, and hyperparameter setting, the normalized environment score is obtained by squashing performance:

Note the right hand side of Figure 2, the distributions of normalized scores for hyperparameter settings in Swimmer and Halfcheetah now lie in a common range.

Swimmer

Swimmer

Halfcheetah

Halfcheetah

Performance (AUC)

Environment

Normalized Score

3000

2500

2000

1500

1000

500

0

-500

-1000

1.4

1.2

1.0

0.8

0.6

0.4

0.2

0.0

-0.2

Figure 2: Left : The distributions of performance (AUC) over 625 hyperparameter settings for the PPO algorithm in Swimmer and Halfcheetah Brax environments. Right : The same distributions after applying score normalization. Each data point is the mean AUC across runs. Each run consisted of 3M steps of agent-environment interaction.

Normalized scores allow practitioners to determine which fixed hyperparameter settings do well across multiple environments. That is, a practitioner can find the hyperparameter setting that maximizes the mean normalized score across a distribution of environments. Consider the performance of the hyperparameter setting denoted by the blue stars in Figure 3. This setting performs in the top quartile of hyperparameter settings in both Swimmer and Halfcheetah. In contrast, consider the hyperparameter setting denoted by the red stars. While this hyperparameter setting sits near the top of the distribution for Halfcheetah, it performs poorly in Swimmer.

Given an algorithm ω ∈ Ω , we define its hyperparameter sensitivity Φ as follows:

The hyperparameter sensitivity of an algorithm is the difference between its per-environment tuned score and its cross-environment tuned score. The per-environment tuned score is the average nor-

Setting that

maximizes mean

environment score

Swimmer

Halfcheetah

Normalized Environment Score

-0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Setting does well

in Halfcheetah but

poorly in Swimmer

malized environment score with hyperparameters tuned per environment. The cross-environment tuned score is the normalized environment score of the best fixed hyperparameter setting across the distribution of environments. We can use this notion of hyperparameter sensitivity to better understand new and existing algorithms. Reporting both hyperparameter sensitivity and the conventional performance-only evaluation metrics should provide a more complete picture of algorithm performance.

3.2 Sensitivity Analysis

Modern reinforcement learning algorithms are complex learning systems, and understanding them is a difficult task that requires multiple dimensions of analysis. Benchmark performance has been the primary metric (and often the only one) used for evaluating algorithms. However, this is only one dimension along which algorithms can be evaluated. Hyperparameter sensitivity is an important dimension to consider in the evaluation space, especially as practitioners begin to apply reinforcement learning algorithms to real-world applications. We propose the performance-sensitivity plane to aid in better understanding algorithms.

Consider the performance-sensitivity plane shown in Figure 4. To construct the plane, the center point is set to the hyperparameter sensitivity and per-environment tuned score of some reference point algorithm. We can consider how other algorithms relate to this reference point by considering which region of the plane they occupy. There are 5 regions of interest shaded by different colors and labeled numerically, which we will consider in turn.

An ideal algorithm would be both more performative and less sensitive. Therefore, algorithms that fall in Region 1 (the top left quadrant) of the plane would be a strict improvement over the reference point algorithm. For some applications, perhaps additional sensitivity can be tolerated if the gains in performance are large enough. Algorithms that fall in Region 2 are an example of this. The region represents algorithms whose increase in performance is greater than the corresponding increase in sensitivity. Conversely, for some applications sensitivity may matter a great deal and some performance loss can be endured. Algorithms that fall in Region 3 are an example of those whose decrease in sensitivity outmatches their corresponding decrease in performance. Regions 1-3 represent algorithms that have notable redeeming qualities either in terms of performance, hyperparameter sensitivity, or both. However, perhaps a practitioner does not care about sensitivity. For example, they want to maximize the score of a specific benchmark, and hyperparameter tuning is no issue. Algorithms in Region 4 may be adequate as they are algorithms that exhibit performance improvements and an even higher reliance upon per-environment hyperparameter tuning. Finally, those unfortunate algorithms that live in Region 5 are in a space with both lower performance and higher sensitivity, making them undesirable.

A natural application of this diagram is to set the reference (center) point to the hyperparameter sensitivity and performance of some base algorithm and study how proposed modifications (or ablations) affect both sensitivity and performance. Often, new algorithms are created by modifying existing algorithms, such as normalizing targets, adding a regularization term to the loss function, gradient clipping, etc. We illustrate an example of this using PPO as a reference point in the next section.

1

4

2

3

Per-environment Tuned Score

Hyperparameter sensitivity

4 Sensitivity Experiments

To illustrate the utility of the sensitivity analysis presented above, we performed an experiment to study the hyperparameter sensitivity and performance of several variants of the PPO algorithm, a widely used policy-gradient method in reinforcement learning (Schulman et al., 2017). We considered several normalization variants commonly used in PPO implementations (Andrychowicz et al., 2021; Huang et al., 2022) and some normalization variants, introduced in DreamerV3, that were purported to reduce hyperparameter sensitivity (Hafner et al., 2023; Sullivan et al., 2023).

4.1 Proximal Policy Optimization

PPO is an instance of an actor-critic method (Sutton & Barto, 2018) that maintains two neural networks: a policy network with parameters θ and a value network with parameters w . Given a policy π θ old , the agent performs a roll-out of T steps, ( S 1 , A 1 , R 1 , S 2 , A 2 , R 2 , ...S T , A T , R T ) , this rollout is then split into m batches of length k . The critic is then updated via ADAM (Kingma & Ba, 2015) to optimize MSE loss with the truncatedλ return as a target. The actor is updated via ADAM (Kingma & Ba, 2015) to optimize a clipped surrogate objective toward maximizing the expected return. An entropy regularizer is added to the actor loss to encourage exploration.

4.2 Normalization variants

Several normalization variants have been used in PPO implementations. We focus on three categories of normalization: observation normalization, value function normalization, and advantage normalization. An intuition behind value function or advantage normalization for hyperparameter sensitivity is that the scale and sparsity of rewards vary greatly across environments and that value function or advantage normalization should make actor-critic updates invariant to these factors possibly requiring less tuning of the step-size hyperparameters (van Hasselt et al., 2016). Another claimed benefit of advantage normalization variants is that by normalizing the advantage term, it is easier to find an appropriate value for the entropy regularizer coefficient τ across a distribution of environments (Hafner et al., 2023). Observation normalization standardizes the network inputs. This can mitigate large gradients, which may stabilize the learning system for hyperparameter tuning (Hafner et al., 2023; Andrychowicz et al., 2021), especially critic and actor step-size hyperparameters α w , α θ > 0 .

Advantage per-minibatch zero-mean normalization : A common implementation detail of PPO is per-minibatch advantage normalization (Huang et al., 2022). When performing an update, the advantage estimates used in the actor loss function are normalized by subtracting the mean of the advantage estimates in the sampled batch and dividing by the standard deviation of advantage estimates in the sampled batch.

Advantage percentile scaling : Another form of advantage normalization was introduced in the DreamerV3 (Hafner et al., 2023) ablations which divides the advantage estimate in the actor loss by a scaling factor. Exponential moving averages are maintained over the 95th and 5th percentiles of return estimates. The advantage term is divided by the difference of the two percentiles.

Figure 4: The performance-sensitivity plane for algorithmic evaluation. The center point indicates the hyperparameter sensitivity and performance of a reference point algorithm. The x-axis is the hyperparameter sensitivity metric as defined in equation 2. The y-axis is the per-environment tuned score (first term in equation 2). The diagonal line is the identity line shifted to intersect the reference point algorithm. The plane is then divided into 5 shaded regions that represent spaces of algorithms of varying qualities relative to the baseline.

Advantage lower bounded percentile scaling : An alternate variant of percentile scaling is used in the DreamerV3 algorithm. Lower bounded percentile scaling applies a max operation to the percentile scaling factor, preventing the estimated advantage term from blowing up if the percentile difference falls below a threshold.

Value target symlog : DreamerV3 introduced a method of scaling down the magnitudes of target values by the symlog function. The symlog function and its inverse symexp are defined as:

As in DreamerV3, symlog is applied to the target in the critic loss and symexp is applied to the output of the critic network. In a subsequent study that applied DreamerV3 tricks to PPO (Sullivan et al., 2023), it was reported that the symlog transformation of the value target was one of the most impactful tricks in environments without reward clipping when applied to PPO.

Observation zero-mean normalization : A very common procedure with PPO is to normalize observations by maintaining running estimates of the mean and standard deviation of observations.

Observation symlog : DreamerV3 proposed an alternative form of observation normalization by applying the symlog function, compressing the observations.

A handful of prior work has investigated the benefits of similar algorithmic modifications to PPO. Previous work has reported the performance impact of the normalization variants commonly used with PPO: per-minibatch zero-mean advantage normalization and zero-mean observation normalization (Andrychowicz et al., 2021). Other work (Sullivan et al., 2023) investigated how PPO's performance is affected by the normalization variants introduced in DreamerV3 (lower bounded percentile scaling, value target symlog, and symlog observation) and that symlog was especially helpful in Atari when reward clipping is disabled. In addition, this work did not perform any hyperparameter tuning for the variants of PPO they tested. In our results, hyperparameter tuning demonstrated a significant effect on the relative performance of these algorithms.

Given these normalization variants, a natural question that arises is how do they affect the hyperparameter sensitivity of PPO? To the best of our knowledge, a careful study of the effect these normalization variants have on the hyperparameter sensitivity of PPO has yet to be done and it provides a good test of our new methodology.

4.3 Sensitivity Experiment with PPO variants

We investigated the effect of each of the described normalization variants on PPO. To isolate these effects, we did not apply reward clipping, reward scaling, or observation normalization wrappers by default. We focused our attention on four critical hyperparameters of PPO: the step-size for the critic α w , the step-size for the actor α θ , the coefficient of the entropy regularizer τ , and the truncatedλ return mixing parameter λ . Weperformed a large grid search spanning five orders of magnitude across five Brax Mujoco domains. Near the extreme endpoints of the grid search, some hyperparameter configurations diverged. We ignored hyperparameter combinations that caused a particular algorithm to diverge over 10% of the time. We averaged the performance over the non-diverging runs.

Consider the performance-sensitivity plane in Figure 5. The reference point at the center is the hyperparameter sensitivity and performance found for PPO without normalization. The error bars displayed indicate 95% confidence intervals formed from a 10,000 sample bootstrap. First, note that none of the normalization variants resulted in an improvement that both raised performance and lowered sensitivity. All forms of advantage normalization increased performance. However, this performance gain comes with a trade-off: increased hyperparameter sensitivity. The marginal gain in performance per unit of increased sensitivity varied between advantage normalization methods. Advantage per-minibatch zero-mean normalization had a greater increase in performance than sensitivity (Region 2). Both percentile scaling-based variants of advantage normalization resulted in more significant sensitivity increases than performance increases (Region 4), indicating an enhanced reliance on hyperparameter optimization methods. Applying the symlog function to the value target lowered performance and may have slightly increased sensitivity (Region 5). It may, however, be the case that the choice of environment distribution did not have enough variation in reward magnitude for the utility of value target symlog to be demonstrated. It appears that observation normalization may slightly reduce sensitivity, although it is unclear due to the width of the confidence intervals.

Advantage

lower bounded

percentile scaling

Advantage

percentile

scaling

Advantage

per-minibatch

zero-mean

normalization

Observation

zero-mean

normalization

PPO

Value target

symlog

Observation

symlog

The performance-sensitivity plane provides insights into how algorithmic changes may alter an algorithm's reliance on per-environment hyperparameter tuning with no additional computational expense other than what is already required from a standard hyperparameter tuning procedure.

The sensitivity metric is intimately tied to the chosen environment distribution, and our findings are thus limited to the Brax Mujoco environments tested. We argue the restricted environment distribution is not just a practical choice, but was a feature of the study. Somewhat surprisingly we found significant sensitivity across a distribution of fairly similar environments. In Figure 7 displayed in Appendix D, we perform a leave-one-out study reporting how the performance-sensitivity plane changes when each of the five environments is dropped from the data. Observe that while the exact values of the points shift, their position relative to the reference point remains mostly unchanged.

The performance-sensitivity plane allows for a richer understanding of algorithms than performanceonly evaluation procedures, but it does not capture the full picture. Two algorithms can sit in the same location on the plane and yet have very different hyperparameter characteristics. Consider the case where there are two algorithms. The first algorithm is highly sensitive with respect to one hyperparameter, which needs to be carefully tuned per environment. The second algorithm has the same sensitivity but needs to be tuned per environment for dozens of hyperparameters with complex interactions. The hyperparameter sensitivity would not differentiate between these two algorithms. An additional metric is needed.

5 Effective Hyperparameter Dimensionality

There are many cases where a practitioner can tune some but not all of an algorithm's tunable hyperparameters. It may be the case that if a few key hyperparameters are tuned per environment, then a preponderance of an algorithm's potential performance can be gained. This motivates the definition of effective hyperparameter dimensionality , a metric that measures how many hyperparameters must be tuned in order to obtain near-peak performance.

Advantage

zm

normalization

Value target

symlog

Observation

symlog

PPO

Advantage

percentile

scaling

Advantage

LB percentile

scaling

Observation zm

normalization

Normalized Score (y-axis) vs. Number of Hyperparameters (x-axis)

For a given algorithm ω with hyperparameter space H ω , number of tunable hyperparameters n ( ω ) , and environment distribution E , let h ∗ . = arg max h ∈ H ω 1 |E| ∑ e ∈E Γ( ω, e, h ) be the hyperparameter setting which maximizes the cross-environment tuned normalized score. Effective hyperparameter dimensionality d ( ω ) is defined as

The effective hyperparameter dimensionality of an algorithm measures the minimal number of hyperparameters that must be tuned per-environment while retaining the majority of the performance that can be realized by tuning all hyperparameters per environment. The threshold of 95% peak performance can be changed at a practitioner's discretion to whatever meets their performance requirements. To compute effective hyperparameter dimensionality, one needs to consider subsets of hyperparameters to find a minimum subset that achieves the required performance threshold.

For the same algorithmic variants of PPO as studied above, Figure 6 displays normalized scores as a function of the number of hyperparameters tuned per environment, choosing the most performant subset to tune. Table 2 in Appendix C provides a listing of the most performant subsets of varying sizes observed during this experiment. The curve interpolates between the normalized scores. The vertical dashed line indicates the point along the curve that reaches 95% of the per-environment tuned score. In the case of advantage percentile scaling, modifying PPO with the normalization variant moves the point to the right, indicating this variant improves performance at the cost of increasing pressure on the number of hyperparameters necessary to tune. Also, note how the performance

ranking can shift based on the number of hyperparameters that have been tuned; such as with PPO and the advantage percentile scaling variant (top left plot). For some variants, performance flattens after tuning only three hyperparameters. Whereas for other variants, performance is almost linear in the number of hyperparameters tuned, suggesting sensitivity to all hyperparameters. On the performance-sensitivity plane (Figure 5 in the previous section), both the advantage percentile scaling and advantage lower-bounded percentile scaling variants fall in the same region (Region 4). Yet, in Figure 6, we can see that the advantage lower bounded percentile scaling variant can obtain higher performance levels than the advantage percentile scaling variant when tuned on smaller subsets of hyperparameters. This observation that algorithms can have similar hyperparameter sensitivities and vastly different effective hyperparameter dimensionalities indicates the power of using both metrics for studying algorithms.

6 Limitations and Future Work

The hyperparameter sensitivity and effective hyperparameter dimensionality metrics will depend heavily on several important empirical design choices. The premise of both metrics is that a practitioner is concerned with understanding sensitivity with respect to a distribution of environments that they care about. If the distribution of environments changes, the metrics will need to be evaluated with respect to the new distribution. This dependence on the environment distribution could be exploited, as someone could artificially make an algorithm appear less sensitive by including several easy environments that all hyperparameter settings will do well in, although score normalization will somewhat counteract this. In addition, the level of granularity with which hyperparameter sweeps are performed will have an effect on both metrics. Another factor that can impact the metrics is the choice of the score normalization method used. A practitioner could use other score normalization methods and the resulting sensitivity scores may be different.

Anext step is to apply the proposed sensitivity and dimensionality metrics to a larger set of algorithms and environments. Related to this work is the literature of AutoRL (Eimer et al., 2023). The goal of AutoRL is to tune hyperparameters via some hyperparameter optimization algorithm (which make use of their own hyperparameters-hyper-hyperparameters). Future work could use the definitions provided here to try to understand if the algorithms proposed in the AutoRL literature reduce sensitivity over the base algorithms that are being modified-measuing the sensitivity of the hyperhyperparameters. A study comparing the sensitivities and dimensionalities of AutoRL methods to the sensitivities and dimensionalities of the base learning algorithms they optimize would be prudent.

7 Conclusion

As learning systems become more complicated, careful empirical practice is critical. Modern reinforcment learning algorithms contain numerous hyperparameters whose interactions and sensitivities are not well understood. Common practice, which is focused on achieving state-of-the-art performance, risks overfitting to benchmark tasks and overly relying on hyperparameter optimization. Most empirical work in reinforcement learning has focused only on evaluating algorithms based on benchmark performance, leaving the effects of hyperparameters under-studied. In this work, we propose a new evaluation methodology based on two metrics that allow practitioners to better understand how an algorithm's performance relates to its hyperparameters. We show how this methodology is useful in evaluating methods purported to mitigate sensitivity. We identify that the studied advantage normalization methods, while improving performance, also increase hyperparameter sensitivity and can increase the number of sensitive hyperparameters.

8 Acknowledgments

The authors would like to thank Ana Paola Garcia Alonzo for assistance in formatting figures. The authors would like to thank David Sychrovský and Anna Hakhverdyan for useful discussions. This research was supported by grants from the Alberta Machine Intelligence Institute (Amii); Canada CIFAR AI Chairs, Amii; and NSERC. Compute was made available by the Digital Research Alliance of Canada.

References

Marcin Andrychowicz, Anton Raichuk, Piotr Stanczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. What Matters in On-Policy Reinforcement Learning? A Large-Scale Empirical Study. In International Conference on Learning Representations (ICLR) , 2021.

Theresa Eimer, Carolin Benjamins, and Marius Lindauer. Hyperparameters in Contextual RL are Highly Situational. CoRR , abs/2212.10876, 2022.

Theresa Eimer, Marius Lindauer, and Roberta Raileanu. Hyperparameters in Reinforcement Learning and How To Tune Them. In International Conference on Machine Learning (ICML) , 2023.

Jörg KH Franke, Gregor Köhler, André Biedenkapp, and Frank Hutter. Sample-Efficient Automated Deep Reinforcement Learning. In International Conference on Learning Representations (ICLR) , 2021.

Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax - A Differentiable Physics Engine for Large Scale Rigid Body Simulation. In Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS) , 2021.

Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy P. Lillicrap. Mastering Diverse Domains through World Models. CoRR , abs/2301.04104, 2023.

Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining Improvements in Deep Reinforcement Learning. In Conference on Artificial Intelligence (AAAI) , 2018.

Shengyi Huang, Rousslan Fernand Julien Dossa, Antonin; Raffin, Anssi, Anssi Kanervisto, and Weixun Wang. The 37 Implementation Details of Proximal Policy Optimization. In International Conference on Learning Representations Blog Track (ICLR) , 2022.

Scott M. Jordan, Yash Chandak, Daniel Cohen, Mengxue Zhang, and Philip S. Thomas. Evaluating the Performance of Reinforcement Learning Algorithms. In International Conference on Machine Learning (ICML) , 2020.

Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations (ICLR) , 2015.

Michail G. Lagoudakis and Ronald Parr. Least-squares policy iteration. Journal of Machine Learning Research (JMLR) , pp. 1107-1149, 2003.

Chris Lu, Jakub Kuba, Alistair Letcher, Luke Metz, Christian Schroeder de Witt, and Jakob Foerster. Discovered Policy Optimisation. In Neural Information Processing Systems (NeurIPS) , 2022.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-Level Control through Deep Reinforcement Learning. Nature , 518:529-533, 2015.

Andrew Patterson, Samuel Neumann, Martha White, and Adam White. Empirical Design in Reinforcement Learning. CoRR , abs/2304.01315, 2023.

Andrew Patterson, Samuel Neumann, Raksha Kumaraswamy, Martha White, and Adam M White. The Cross-Environment Hyperparameter Setting Benchmark for Reinforcement Learning. In Reinforcement Learning Conference (RLC) , 2024.

Martin Riedmiller. Neural fitted q iteration - first experiences with a data efficient neural reinforcement learning method. In European Conference on Machine Learning (ECML) , 2005.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. CoRR , abs/1707.06347, 2017.

Anton Schwartz. A reinforcement learning method for maximizing undiscounted rewards. In International Conference on Machine Learning (ICML) , 1993.

Ryan Sullivan, Akarsh Kumar, Shengyi Huang, John P. Dickerson, and Joseph Suarez. Reward Scale Robustness for Proximal Policy Optimization via DreamerV3 Tricks. In Advances in Neural Information Processing Systems (NeurIPS) , 2023.

Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction . MIT Press, 2018.

Hado van Hasselt, Arthur Guez, Matteo Hessel, Volodymyr Mnih, and David Silver. Learning Values Across Many Orders of Magnitude. In Neural Information Processing Systems (NeurIPS) , 2016.

Christopher John Cornish Hellaby Watkins. Learning from Delayed Rewards . PhD thesis, University of Cambridge, 1989.

A Broader Impact Statement

Hyperparameter sweeps and per-environment tuning are the most computationally expensive and environmentally impactful parts of reinforcement learning research. Our study ran for approximately 4.5 GPU years on NVIDIA 32GB V100s. While this is substantial, we believe that using compute to better understand the sensitivity of current algorithms is an essential step towards developing more environmentally friendly algorithms. This work investigated an empirical methodology for evaluating the hyperparameter sensitivity of reinforcement learning agents. The immediate societal impact is minimal. However, our methodology may aid in developing performative algorithms with low hyperparameter sensitivity. If this occurs, these algorithms will result in less need for hyperparameter tuning and, as a result, have a positive impact on lowering the carbon footprint of reinforcement learning experiments.

B Proliferation of Hyperparameters

There is a trend in which the current state-of-the-art algorithms often contain more hyperparameters than the previous state-of-the-art. Table 1 lists hyperparameter counts for representative algorithms from each of the three main categories of reinforcement learning methods: values-based, policygradient, and model-based.

C Hyperparameter Sweep Details

The PPO implementation used was heavily inspired by the PureJaxRL PPO implementation (Lu et al., 2022). The variants advantage per-minibatch zero-mean normalization and observation zero-mean normalization are the standard implementations provided within PureJaxRL. The variants: symlog observation, symlog value target, percentile scaling, and lower bounded percentile scaling closely follow the implementation of the DreamerV3 tricks applied to PPO shown in Sullivan et al. (2023) as well as referencing the original DreamerV3 repository Hafner et al. (2023).

The policy and critic networks were parametrized by fully connected MLP networks, each with two hidden layers of 256 units. The network used the tanh activation function. Separate ADAM optimizers (Kingma & Ba, 2015) were used for training the actor and critic networks. The environments used in the experiments were the Brax implementations of Ant, Halfcheetah, Hopper, Swimmer, and Walker2d.(Freeman et al., 2021). The hyperparameter sweeps were grid searches over eligibility trace λ ∈ { 0 . 1 , 0 . 3 , 0 . 5 , 0 . 7 , 0 . 9 } , entropy regularizer coefficient τ ∈ { 0 . 001 , 0 . 01 , 0 . 1 , 1 . 0 , 10 . 0 } , actor step-size α θ ∈ { 0 . 00001 , 0 . 0001 , 0 . 001 , 0 . 01 , 0 . 1 } , and critic step-size α w ∈ { 0 . 00001 , 0 . 0001 , 0 . 001 , 0 . 01 , 0 . 1 } . Each run lasted for 3M environment steps. 200 runs were performed for each of the algorithms, environments, and hyperparameter settings. Like the PureJaxRL PPO implementation, the entire training loop was implemented to run on GPU. We will release code and experiment data at https://github.com/jadkins99/hyperparameter_ sensitivity , promoting the further investigation of hyperparameter sensitivity in the field of reinforcement learning.

Environment

Tuned Performance

Sensitivity

Advantage

lower bounded

percentile scaling

Advantage

per-minibatch

zero-mean

normalization

Advantage

percentile scaling

Value target

symlog

PPO

Observation

zero-mean

normalization

Observation

symlog

Advantage

lower bounded

percentile scaling

Advantage

per-minibatch

zero-mean

normalization

Advantage

percentile scaling

Value target

symlog

PPO

Observation

zero-mean

normalization

Observation

symlog

Advantage

lower bounded

percentile scaling

Advantage

per-minibatch

zero-mean

normalization

Advantage

percentile scaling

Value target

symlog

PPO

Observation

zero-mean

normalization

Observation

symlog

Observation

zero-mean

normalization

PPO

Advantage

percentile scaling

Value target

symlog

Observation

symlog

Advantage

lower bounded

percentile scaling

Advantage

per-minibatch

zero-mean

normalization

Advantage

lower bounded

percentile scaling

Advantage

per-minibatch

zero-mean

normalization

Advantage

percentile scaling

Value target

symlog

PPO

Observation

zero-mean

normalization

Observation

symlog

D Additional Figures

The sensitivity metric is ultimately tied to a distribution of environments, and our findings are thus limited to Brax Mujoco environments tested. Claims cannot be about environments outside the evaluated environment set. One may wonder about the stability of the results if the environment distribution changes on a small scale. In Figure 7, we repeat the sensitivity metric for each of the five environment subsets that can be obtained by dropping a single environment. Leaving out environments does shift the reference point, and the position of the variants shifts somewhat relative to the reference point. However, the regional position of the variants relative to the reference point is mostly consistent.

In Figure 8, we also include results with final performance, i.e., the sum of rewards obtained over the final 1000 timesteps of learning (the truncation length). The performance of reinforcement learning agents does not monotonically increase. In many cases, with specific hyperparameters, it may collapse in the middle or end of learning. Therefore, this metric is noisy, and it is difficult to reason about where algorithms lie according to it on the performance sensitivity curve.

Advantage

percentile scaling

Value target

symlog

Advantage

per-minibatch

zero-mean

normalization

PPO

Advantage

lower bounded

percentile scaling

Observation

zero-mean

normalization

Observation

symlog

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The contribution of this paper is an empirical methodology for assessing hyperparameter sensitivity. The presented experiments provide evidence for the utility of the methodology, as stated in the introduction and abstract.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that the paper does not attain these goals.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See that there is a section titled "Limitations and Future Work" which discusses limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper provides definitions of metrics. It does not make theoretical claims. Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Details of implementations and hyperparameter sweeps are given in the appendix. The error bars that are presented are obtained by computing a 10000 sample 95% boostrap confidence intervals as described in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code is included in supplementary material. Instructions are in README file.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Experiments are described in the paper. Full experiment details are given in the appendix. Code is provided.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The error bars shown in the main paper are 95% bootstrap confidence intervals. The confidence interval around the learning curves in the appendix is a 95% Student tdistribution confidence interval. This is described in the main paper and appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: This information is provided in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer:[Yes]

Justification: We did not notice any ethical conflicts when reviewing the code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: This is discussed in a section of the appendix titled Broader Impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The work presented in this paper poses no such risk. It is an empirical methodology.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We wrote original code for this project. The PPO implementation was forked from PureJaxRL. They are cited throughout the paper, and their license is included in the supplementary material. We thank them and other contributors to the opensource community.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The contribution of this paper is an empirical methodology. Code for experiments will eventually be opensourced and made available to the community.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA] .

Justification: There were no human subjects used in this research.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This work does not involve crowdsourcing nor human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors define a environment-parameter sensitivity metric in terms of a Jensen gap of min-max scaled performance. The gap gives us the difference between the best average performance and the average best performance over a particular algorithms hyperparameters over a set of environments. The authors then provide a 2D visualization where they plot this sensitivity gap against the average best-performance. This gives an intuitive way to classify whether a change in parametersensitivity can be outweighed by its (positive) change in performance.\n\nThe authors run experiments for PPO on brax to illustrate their method.\n\nAt the end, the authors also use their performance metric to tune subsets of the total parameter set.\n\nSTRENGTHS:\nThe definition for the environmnent-dependent parameter sensitivity is intuitive through its use of Jensen's gap. I agree that this is a sound method to evaluate robustness of parameter-tuning. Although, care must be taken when choosing the environment-sets.\n\nFigure 3 and the discussion in 3.2 is excellently presented, and gives an intuitive way to think about what patterns we would like to see when performing parameter ablations.\n\nThe experimental setup is repeated over many repetitions and use appropriate statistical methods, when discussing the results, the authors discuss overlapping intervals and are decently careful about making conclusions.\n\nWEAKNESSES:\n### Summary\nAlthough this paper shows a nice way to visualize robustness of parameter optimization methods across sets of environments, the severe computation requirements needed to create these visualizations on top of the shallow experimental setup that the authors performed, makes it unlikely that the current results will prove useful to other practitioners (aside from the isolated case applying PPO to Brax). \n\nThe chosen metric for performance (AUC) is incomplete when considering how RL is often applied (when only caring about final performance), and arguably estimated using a non-robust bound estimator. Furthermore, the authors only test on 5 Brax mujoco environments when comparing extensions that originate from Atari, DMLab, Minecraft or elsewhere. The authors, do not take characteristics of the environments they test on into account when discussing results. As a consequence, the conclusions we can take from the presented results are quite limited.\n\nTherefore I vote for reject.\n\n### Major comments\n - The single choice of AUC metric is poorly motivated, although it is a relevant metric, in RL we are also often only interested in **final performance**. So, the AUC gives a proxy for cumulative regret, I also want to see a proxy for simple regret, the best-run within a algorithm's training cycle.\n - The authors argue that we need robust estimators of performance for hyperparameter sensitivity. So, I am worried about the choice for min-max normalization of the algorithms' performance, see Eq. 1). These can be extremely high-variance estimators for the performance bounds, and therefore, aren't really robust (perhaps explaining why the authors needed 200 seeds per ablation)... I understand that this always nicely scales results into $[0, 1]$, however, if we rerun the experiments (different seeds and parameter-sets), would Figures 1 and 2 give similar patterns? Would this change our conclusion about what choice of hyperparameter is better in terms of environment robustness? Why not choose for an interquantile range or even a $[p, 1-p]$ percentile range that has guaranteed lower asymptotic variance? \n - The authors don't discuss the sensitivity/ robustness to the choice of environment sets for Eq.2). I.e., what characteristics do the environments have that we test robustness for. For this reason, I also think that the discussion in 4.3 is a bit shallow considering the choice of Mujoco environments. The rewards in Brax are relatively well scaled and behaved (at least compared to e.g., a game like 2048 which has exponential rewards), so concluding that the value target symlog is not beneficial (on top of my comment about min-max scaling) should be nuanced and discussed appropriately. \n - It's not clear from Eq.5) (and the whole of section 5) how we decide what reference point to take when deciding which parameters to tune. I.e., when we choose to keep some parameters fixed, what should their values be? Are the defaults by definition robust or not?\n\n\n### Technical comments\n - Definition of normalized environment score $\\Gamma$ is wrong, why is the action-space $A$ used here? (see section 2). Why is the symbol for environment sets not consistent, why use $E$ instead of $\\mathcal{E}$. In section 5 suddenly $\\Omega$ is used again in place of $A$.\n\n\n### Minor comments\n - Eq. 2) Why not write the sum as an expectation, $\\mathbb{E}[\\max_h \\Gamma(...) | e \\sim {p(e)}] - \\max_h \\mathbb{E}_{p(e)} \\Gamma(...)$. I.e., the more recognizable Jensen gap.\n - Shortly discuss expected behaviour of Eq.2 and what it means, i.e., a value of 1 means a large gap, meaning not robust, and a value of 0 means no gap, meaning very robust.\n - Line 220, minibatch advantage normalization is not that important for PPO, even the blog-post that the authors cite here says so.\n - What ranges for the parameters were chosen in section 4.1 ? It's easy to break an algorithm through a bad or uninformed choice for the learning rates... It's also not really clear from the text how the authors created the algorithm configurations to test, i.e., random-search? an exhaustive grid? (Found this in Appendix D, the ranges for the values are OK when speaking from experience, and the authors used exhaustive grid-search; this **must** be included in the main text)\n - Line 291, \"It may ... be gained.\" This sentence is broken, maybe rephrase: \"By finetuning a few important parameters for each specific environment, you can unlock most of an algorithm's best performance.\"\n\nQUESTIONS:\n- Could the authors change the bound calculation (e.g., IQR or something similar), this should be an easy local change, and add the best-performing run metric (i.e., have the AUC and the best-run per algorithm side-by-side).\n- Could the authors improve on the discussion and conclusions of the results, discussing the characteristics of the environment and how the choice of parameters hypothetically covaries with this.\n\nLIMITATIONS:\nSection 6 shortly discuss limitations, however, the actual limitations are discussed in appendix D. The shallow experimental setup that was performed resulted in 7 GPU years on modern NVIDIA GPUs. Since most code was implemented in Jax and the algorithm used was PPO, one of the most light-weight RL algorithms currently out there, the current framework does not offer much headroom for testing other methods. \n\nThe severe computation costs are not included in the broader impact statement. The authors do state that reducing carbon footprint of DRL experiments are important, however, the presented methodology is not carbon friendly. If, the authors presented a better experimental setup, then this would alleviate the need for other practitioners from running these types of sensitivity analyses for PPO. However, this is currently not the case. Another way to improve the current setup is to use variance reduction methods in some way to reduce the immense compute requirements.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method to analyse how sensitive are RL methods with respect to hyperparameter tuning. The author argue that one method may perform well on average but require more HPO tuning per task which hides some computation and prevent from having comparable results. They introduce a sensitivity metric which measures the difference between the best hyperparameter tuned per task and the performance of the best hyperparameter on average. Then, they propose a quadrant analysis where both the sensitivity and the performance of a tuned score are displayed which allows to compare algorithms on those two dimensions. Experiments are then performed on PPO normalization variants where they show that current normalization have different trade-offs: some approaches improves the scores but increase hyperparameter sensitivity while other conversely lower scores and sensitivity but no approach allows to improves both scores and sensitivity currently. Finally, the authors study how many hyperparameters require tuning while having the other fix and while keeping 95% of the best performance.\n\nSTRENGTHS:\n* The paper is very well motivated. It tackles an important problem for RL (and in general on how to account for HPO sensitivity when reporting results.\n* The paper is very well written and easy to follow, the methods are well described and the experiment are very sound\n* The paper would provide a valuable contribution if the results of the runs are released (will they?, see my question)\n\nWEAKNESSES:\nThe paper is currently missing some analysis regarding the method stability (see questions). For instance how much the proposed method would be stable and reliable with respect to different environments (does adding one environment change the results completely?) or normalization (e.g. using CDF instead of min max).\n\nQUESTIONS:\nHere are points that would be important for me to raise my score:\n\n* In figure 4, are the results stable if you leave one out one environment? If you compute the plot 5 times each time leaving one of the environment, do you have similar plots? If not, then the analysis on the paper will be less warranted (that such and such method is more stable with hyperparameter).\n* Will the paper release the dataset of evaluation in addition to the code? (the code would have limited interest compared to the data) I would highly encourage to share the dataset, possibly with a script to reproduce some of the paper figures (for instance as was done by https://github.com/google-research/rliable) The dataset would be also very useful to simulate and compare HPO methods, in particular if it contains the evaluation per iteration (and per 100s of iterations for instance).\n\n* Figure 4: there are very large outliers for the obs zero mean normalization sensitivity and the distribution is very skewed, could you explain why they happen? \n\n* One popular method is f-Anova to study hyperparameter importance, could you include an analysis using it? It would be useful for practitioners to know which HP are most useful.\n\n* In the limitation section, you mention that the results may change under a different normalization (say CDF instead of min-max). This is indeed an important point, could you report the result for Fig4 as well with CDF? I assume it should be a minor change as it is just changing the metric. However, it seems important to assess how much the method would be impacted by different normalization (even if the results change, the paper would still be valuable)\n\n\nHere are points that are more details:\n- You have l110 p(w, e, h ,\\kappa) and then p(a, e, h), it would be nice to unify the notation\n- l151: I think it would be useful to precise that the quantity is always >= 0\n- \"The shaded region is a 95% Student t-distribution confidence interval around the mean return over 200 runs \"\n=> what is the CI not covering the mean in walker 2D?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new evaluation regime for reinforcement learning. As opposed to only taking into account benchmark performance (i.e., final return), as is ocmmon in previous literature, this work suggests considering an extra dimension of how sensitive algorithms are to hyperparameters in tuning based on a heuristic developed in the paper. They analyse PPO, and some of its variants, using this new evaluation regime, and find that performance increases often correspond to increased sensitivity to hyperparameters. Finally, the paper considers a top-k approach to hyperparameter tuning, adjusting only the most impactful hyperparameter, to see how the baseline algorithms compare under a more limited hyperparameter tuning regime.\n\nSTRENGTHS:\n- The paper clearly takes care to use an extensive number of experiments, possibly reaching into the realm of unnecessary, to provide some rigour to their results.\n- The domain being considered - considering the hyperparameter sensitivity of algorithms in addition to their performance - is an underexplored area and one which becomes increasingly important as the cost of experiments increase. \n- Approaching this problem in a visual setting seems reasonable.\n- I like the approach of distilling hyperparameter optimisation to optimising only a smaller set of more important values; this has real benefits in enabling tuning of the majority of hyperparameters in *cheap* environments and tuning the key values only in more expensive environments.\n\nWEAKNESSES:\n- Noting appendix A (the table of how many hyperparameters each algorithm has); the definition of 'hyperparameters' seems pretty weak, and a lot of those included seem to just be design decisions of the actual algorithms.\n- I find reading the plots quite confusing, exacerbated by all of the different colours marking each of the areas. I think this obfuscates the message of the plots and as a reader makes it hard to come to conclusions.\n- Given this work is based on PurejaxRL, which I think runs on Brax for 1e8 frames, these experiments seem very short (3e6 frames) and possibly doesn't give the algorithms the full opportunity to converge.\n- There is very limited discussion of preexisting literature in this space. While I appreciate this takes a subtly different tack (calling for us to measure how sensitive to hyperparameters RL algorithms are, rather than designing algorithms with few hyperparameters), I would expect to see significant more discussion of AutoRL literature. Framing this work better in related work would definitely strengthen the paper.\n- The formatting looks quite off with the figures. I think this is because a lot of the captions are to the side of the figures, rather than below.\n- Considering the actual hyperparameters being tuned, the results feels slightly disingenuous. If we assume the practitioner running the hyperparameter tuning, we would possibly expect them to select more reasonable values or focus the search in a significantly more targeted way. For instance, it is no surprise that the highly performant methods saw large performance decreases (i.e., hyperparameter sensitivity) when evaluated with entropy coefficients in the range [0.001, 10] or learning rates spanning 5 orders of magnitude. Instead, it would be much more sensible to explore how the performance changes with reasonable hyperparameters that one is more likely to practically tune over (i.e., we generally have a good idea of where to start with values, so are unlikely to try an entropy coefficient of 10).\n- It would be good to see some exploration of algorithms which have done something different than just considering the changes to normalisation in PPO returns or observations. It feels like some broad statements are made despite the fact that this analysis was only taken in a single setting.\n\nQUESTIONS:\n- In the plots, the authors state that they plot 95% confidence intervals over 200 seeds; but in the y-axis, all of the 'confidence intervals' are only negative. I am not particularly clear why. Is this a mistake, or am I misinterpreting results? At the same time, are the circles representing the mean score over runs, environments, or both; and therefore, are the confidence intervals defined over the runs, the enviroments or a big stack of (in this case) 1000 results per point? Is comparison for the hyperparameter sensitivity done per seed or for the averages over seeds? I think there's a lot of questions about how exactly these results are being presented which doesn't come through in the paper.\n- In some cases, increased hyperparameter sensitivity can be a positive - it can give us extra opportunities to boost performance. Do you think this should be mentioned as a limitation of this method, which effectively calls for less sensitivity?\n- Why did you use 200 seeds per hyperparameter tuning configuration, which is significantly greater than would be used in most other research? In practice, would it not have been more sensible to have used a much smaller number of seeds (eg 8), and focused on a much tighter range of hyperparameters?\n- As a brief aside, the authors have not removed the instructions of the Neurips checklist as requested, which should be done if the camera ready copy is released.\n\nLIMITATIONS:\nThe authors briefly discuss some limitations of their methods; in particular, that their proposed metrics are highly environment specific (even in this case, they are looking at only one suite of environments which may end up biasing results), and will need reevaluation every time you shift to a new set of domains. However, I think there are some other key limitations that are not discussed, many of which have been raised above. In particular:\n- In addition to being environment specific, the metrics proposed are going to be very specific to the range of values which are tested. I have mentioned above how the values chosen for hyperparameter tuning here are not particularly reasonable.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces an empirical framework for assessing the hyperparameter sensitivity of reinforcement learning algorithms.\nThe framework consists of two metrics: 1. hyperparameter sensitivity, which gives a normalized difference in performance between the per-task best hyperparameters and the across-task best hyperparameters. 2. effective hyperparameter dimensionality, which gives the number of hyperparameters that can be left the same as the across-task hyperparameters while tuning the rest and still obtain a threshold say (95%) of the per-task best hyperparameter performance.\nUsing these metrics in addition to performance metrics, practitioners and researchers can have a better idea of the benefits/downsides of a modification to an algorithm (how much better vs how much more sensitive)\nThe framework is used to compare several normalization variants of PPO introduced in the past years on the continuous control environments, giving a better picture of their contribution.\n\nSTRENGTHS:\nThe problem is well-motivated. Hyperparameter sensitivity is a well-known issue in deep RL with novel algorithms often providing better performance at the expense of a higher sensitivity. Such tradeoff must be made explicit.\nThe framework introduced in the paper provides an effective way to draw that tradeoff with a clear interpretation of it using Figure 3.\n\nThe metrics are simple and quite natural and provide a solid starting point for a hyperparameter sensitivity framework.\nAlthough computationally expensive and still to prove if its results transfer across domains, the framework is likely to have a high impact on the field.\nAt least on the continuous control with Brax and PPO variants where the framework has been used as an example.\nIn particular thanks to the thorough and accurate experimental protocol, with 200 seeds per experiment and confidence intervals.\n\nWEAKNESSES:\nMetric definition:\n- The effective hyperparameter dimensionality depends on the total number of hyperparameters of an algorithm and is likely to scale with it, so this makes it incomparable between algorithms with a different number of hyperparameters. Perhaps counting the number of parameters that changed instead would make it more comparable across algorithms, as is what would dictate the budget of practitioners eventually.\n\n\nHyperparameters:\n- I would consider the minibatch size and the number of epochs in PPO to be critical hyperparameters as well. It's not clear how the choice of hyperparameters to sweep over was made.\n- (minor) The epsilon in the denominator of the minibatch normalization may also play a big role.\n\nInconsistent/confusion notation:\nin line 91, the tuple (w,h) defines an agent (a,), but this is used in a confusing and inconsistent way throughout the paper.\n- line 116 $\\hat p(a, e, h): if $a$ is there then $h$ is redundant. It seems like $a$ there stands for $w$.\n- Equation 1 same. The $a$ should be a $w$.\n- Equation 2 and 5:  2 and 5: $\\Gamma(a, e, w)$ not it's even more confusing there is both an $a$ and an $w$ and there is an $h$ missing.\n\nClaims:\nLine 321 \"vastly difference effective hyperparameter dimensionalities\": the largest gap observed in the paper is from 2 to 4, so I would not extrapolate here, though arguably that's indeed an additional 2 dimensions to sweep over so scales exponentially.\n\nQUESTIONS:\nReporting the mean under the curve (AUC) with 95% bootstrapped confidence intervals over 200 runs is great, but I would expect some discussion on the use of a confidence interval, as it collapses the shaded area across the 200 runs to the statistic being computed (here the mean AUC).\nTo me, using a dispersion measure like the variance of the mean AUC would also be valid, as some hyperparameters would have more variance than others (although this could also be seen as an additional dimension of sensitivity).\nI would appreciate it if the authors could comment on this.\nAlso, at what point are the confidence intervals computed? When computing the expected performance or when computing the hyperparameter sensitivity, etc?\n\nLIMITATIONS:\nThe authors adequately state the limitations of their work and its broader impacts. In particular the potential limitation of the framework to a specific environment distribution.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-01"}