A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning
Authors: Jacob Adkins
Michael Bowling
Adam White
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement Learning
Hyperparameters
Empirical Methodology
Primary area: reinforcement_learning
TL;DR: An empirical methodology for evaluating hyperparameter sensitivity of reinforcement learning algorithms.

Abstract: The performance of modern reinforcement learning algorithms critically relies
on tuning ever increasing numbers of hyperparameters. Often, small changes in
a hyperparameter can lead to drastic changes in performance, and different environments require very different hyperparameter settings to achieve state-of-the-art
performance reported in the literature. We currently lack a scalable and widely
accepted approach to characterizing these complex interactions. This work proposes a new empirical methodology for studying, comparing, and quantifying the
sensitivity of an algorithm’s performance to hyperparameter tuning for a given set
of environments. We then demonstrate the utility of this methodology by assessing
the hyperparameter sensitivity of several commonly used normalization variants of
PPO. The results suggest that several algorithmic performance improvements may,
in fact, be a result of an increased reliance on hyperparameter tuning.

A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning
Jacob Adkins
Michael Bowling
Department of Computing Science
Department of Computing Science
University of Alberta; Amii
University of Alberta; Amii
Edmonton, Canada
Edmonton, Canada
jadkinss@ualberta..cm
mbonling@ualberrt.cco
Adam White
Department of Computing Science University of Alberta; Amii Edmonton, Canada amv8@ualberta.cm ca.
Abstract
The performance of modern reinforcement learning algorithms critically relies tuning ever increasing numbers of hyperparameters, Often, small changes in hyperparameter can lead drastic changes in performance, and different envi- ronments require very different hyperparameter settings to achieve state-of-the-art performance reported the literature. We currently lack a scalable and widely accepted approach to characterizing these complex interactions. This work pro- poses a new empirical methodology for studying, comparing, and quantifying the sensitivity of an algorithm's performance to hyperparameter tuning for given set environments. We then demonstrate the utility of this methodology by assessing the hyperparameter sensitivity of several commonly used normalization variants of PPO. The results suggest that several algorithmic performance improvements may, in fact, be result of an increased reliance hyperparameter tuning.
Introduction
The performance of reinforcement learning algorithms critically relies the tuning of numerous hyperparameters. With the introduction of each new algorithm, the number of these critical hy- perparameters continues grow. Consider the progression of value-based reinforcement learning algorithms, starting from DQN (Mnih et al. 2015). which has 16 hyperparameters that the practi- tioner must choose, Rainbow (Hessel al. 2018)) with 25 hyperparameters. This increase be observed in Figure [ This proliferation is problematic because performance can vary drastically with respect hyperparameters across environments. Often, small changes in hyperparameter can lead to drastic changes in performance, and different environments require very different hyperparameter settings achieve the reported good performances (Franke al. 2021 Eimer et al. 2022 2023
Patterson et al. 2024). Generally speaking, hyperparameter tuning requires a combinatorial search and thus many published results are based on a mix of default hyperparameter settings and informal hand-tuning of key hyperparameters like the learning rate. Our standard evaluation methodologies do reflect the sensitivity of performance to hyperparameter choices, and this is compounded by lack of suitable metrics to characterize said sensitivities.
There many different ways could characterize performance with respect to hyperparameter choices reinforcement learning, but the community lacks an agreed standard. Hyperparameter
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

41
DreamerV3
34
Agent57
DreamerV2
Number of hyperparaneters
Rainhow
DreamerV1)
* SAC
PER
16-
DON
A3C
Actor Oritic with
aligibilit tracos
Inear Dyna
O-learning @ilyna
NFO
Actor eritic
R-learning
1983
1993
2003
2013
2023
Year
Figure I: A count of hyperparameters for different reinforcement learning algorithms proposed over the last decade. We include value-based, volicy-gradiens and model-based methods. The counts do not include hyperparameters controlling the network architectures, such as number of layers, activation functions, etc. See Appendix B for details on how hyperparameters were counted.
sensitivity curves, such as those found in the introductory textbook (Sutton & Barto 2018), summarize performance with respect several values of key hyperparameter producing U-shaped curves. Sensitivity curves do not work well with many dimensions of hyperparameters, nor is there a well- established way to them summarize performance in multiple environments. If computation is of no object, then performance percentiles can be used to compute the likelihood that algorithm will perform well its hyperparameters are randomly sampled from some distribution (Jordan et al.) 2020). Although very general, this approach does not reflect how practitioners tune their algorithms. The Cross-cnvironment Hyperparameter Benchmark (Patterson et al., 2024) compares algorithms by mean normalized performance score across environments but ultimately focuses on the possibility of finding a single good setting of an algorithm's 's hy hyperparameters that performs well rather than characterizing sensitivity. What performance- -nly metrics lack is a measurement of what proportion of realized performance is due to per-environment hyperparameter tuning.
We propose an empirical methodology to better understand the interplay between hyperparameter tuning and an reinforcement learning agent's performance. Our methodology consists two metries and graphical techniques for studying them. The first metric, called an algorithm's 's hyperparameter sensitivity, measures the degree to which an algorithm's peak reported performance relies upon per- environment hyperparameter tuning. This metric captures the degree to which per-environment tuning improves performance relative to the performance of the fixt-fixed hyperparameter setting across a distribution of environments. The second metric, named effective hyperparameter dimensionality, measures how many hyperparameters must be tuned to achieve near-peak performance. It is often unclear how important specific hyperparameters are and if they should be included in the tuning process. These two metries can help us better understand existing algorithms and drive research toward algorithmic improvements that reduce hyperparameter sensitivity.
We validate the utility of our methodology by studying several variants of PPO (Schulman al. 2017) that have been purported to reduce hyperparameter sensitivity and increase performance. We performed large-scale hyperparameter study over variants of the PPO algorithm consisting of over 4.3 million runs (13 trillion environment steps) in the Brax MuJoCo domains (Freeman et al., 2021). We investigate the relationship between performance and hyperparameter sensitivity with several commonly used normalization variants paired with PPO. We found that normalization variants, which increased PPO's tuned performance, also increased sensitivity. Other normalization variants had negligible effects performance and marginal effects on hyperparameter sensitivity. This result contrasts the view that normalization makes reinforcement learning algorithms easier to tune and, as consequence, results in improved performance.

---

2 Problem Setting and Notation
We formalize the agent- envertenviro interaction a Markov Decision Process (MDP) with finite state space S and action space A, bounded reward function R: S x A x S - R C IR, transition function P x A S [0. 1]. and discount factor 7 [0, 1j. At each timestep the agent observes the state St, selects an action A,, the environment outputs a scalar reward R and transitions to a new state S1+1. The agent's goal is to find a policy, πT : A x S + [0, 1]. that maximizes the expected return, G2 = R&+1 + 7 + .... in all states: I,, [G s] for all s E S.
Most reinforcement learning agents learn and use approximate value functions in order to improve the policy through interaction with the world. The state-value function e*: S - R is the state conditioned expected return following policy и defined as (s) = [, s]. Similarly, the action-value function qr: S x + R provides the state and action conditioned expected return following policy π defined by qr (8, a) = E [G 8, A, a). The advantage function Ax a) = ( a) (*))) describes how much better taking an action a in state 8 is rather than sampling an action according to xt (-18) and following policy afterward. Given an estimate of the value function é, agent can update estimates of the value of states based on estimates of the values of successor states. An n-step return is defined as Gre+n = RR+1 + +2 +
17-1Rt+ + 7 (St=n). A mixture of n-step returns, called a truncated-A return can be created by weighting n-step returns by factor E [0, 1]. G = (1 - >) x Ge.t+ + A" Gra+n Truncated-X À returns are often used in reinforcement learning algorithms such as PPO (Schulman et al. 2017) to estimate state values and state-action advantages, which are used to approximate the value function and improve the policy.
Of particular interest to this work is the setting where an empiricist evaluating a set of algorithms Ω across a distribution of environments E. Each algorithm E Ω is presumed to have some number of hyperparameters n(w). Each hyperparameter h;, I I nww) is chosen from some set of Cartesian product of those choices
choices H.. - The total hyperparameter space H is defined H- Hy x Hy x ...H.. Once an algorithm w E Ω and hyperparameter setting h E HW are chosen, the tuple (w, specifies an agent.
Hy Hyperparameter Sensitivity
First, we must define what we mean by sensitivity. A sensitive algorithm is an algorithm that requires great deal of ger-ennironment hyperparameter tuning obtain high performance. Conversely, an insensitive algorithm is one where there exist hyperparameter settings such that the algorithm can obtain high-performance across a distribution of environments with fixed hyperparameters.
This section presents two contributions: a metric for assessing an algorithm's 's hyperparameter sensitivity, and a method of graphically analyzing the relationship between hyperparameter sensitivity and performance along a 2-dimensional plane. These tools may be used to develop a deeper understanding of existing algorithms and we hope will aid researchers evaluating algorithms more holistically along dimensions other than just benchmark performance.
3.1 Sensitivity Metric
We want performance metric that summarizes the learning of online reinforcement learning agent. The natural choice of performance metric is to report the average return obtained during learning, which we call the area under the (learning) curve (AUC). The AUC p(w, e, h, k) where w E setting, and K. C N is the random number generator (RNG) seed. Performance observed during run of a reinforcement learning agent depends on many factors: the reinforcement learning algorithm, the environment, the hyperparameter setting, and many forms of stochasticity. Even after fixing the algorithm, environment, and hyperparameter setting, performance distributions are often skewed and multi-modal (Patterson et al.) 2023). Therefore, many runs are required to obtain accurate estimates of expected performance p(w, h)) = K E>ex p(w, f, k) where KC C N is the set of RNG seeds used during the experiment. In the experiments presented in this paper, we perform 200 runs averaging performance over subset of runs, after filtering (described later), and report 95% bootstrap confidence intervals around computed statistics.
is denoted by an algorithm, e E E is an environment, h E H is a hyperparameter
3

---

It is crucial to capture performance across sets of environments in order to compute sensitivity. Recall, that our notion of sensitivity captures the degree to which an algorithm relies on per-environment hyperparameter tuning for its reported performance gains. Our choice of AUC as performance metric does not allow for cross-environment performance comparisons directly because the magnitudes of returns vary greatly between environments. Consider the distributions of performance presented in the left plot in Figure 2 The performance realized by good hyperparameter settings Halfcheetah orders of magnitude greater than the performance of good hyperparameter settings in Swimmer. Nevertheless, just because the absolute magnitude is lower or the range of observed performances is tighter, that does not mean the differences are any less significant. Thus, in order to consider how hyperparameters perform across sets environments, we need to normalize performance to a standardized score.
In this work, we use [5, 95] percentile normalization, We choose percentile normalization as it has lower variance than alternatives like min-max normalization. Other normalization methods, such as min-max or CDF normalization (Jordan et al.) 2020), could also be used with our hyperparameter sensitivity formulation. After conducting large number of runs across different algorithms,, envi- ronments, and hyperparameter settings, for each environment e, we find the 5th percentile (s(() and 95th percentile (e) of the distribution of observed performance in Then, for each algorithm, environment, and hyperparameter setting, the normalized environment score is obtained by squashing performance:
T(w, e, p(w, e, Ps(e)
Pas(e) (e)
Note the right hand side of Figure 2 the distributions of normalized scores for hyperparameter settings in Swimmer and Halfcheetah now lie in a common range.
Environment
Performance (AUC)
Normalized Score
3000
14
Figure 2: Left: The distributions perfor- mance (AUC) over 625 hyperparameter set- tings for the PPO algorithm in Swimmer and Halfcheetah Brax environments. Right: The same distributions after applying score nor- malization. Each data point is the mean AUC across runs. Each run consisted of 3M steps of agent--nvironmen interaction.
2500
1.0
1500
0.8
1000
0.6
500
-500
-00
-1000
.02
Swimmer Halfeheetah
Swimmer
Normalized scores allow practitioners determine which fixed hyperparameter settings do well across multiple environments. That is, practitioner can find the hyperparameter setting that maximizes the mean normalized score across a distribution of environments. Consider the performance of the hyperparameter setting denoted by the blue stars in Figure 3 This setting performs in the top quartile of hyperparameter settings in both Swimmer and Halfcheetah. contrast, consider the hyperparameter setting denoted by the red stars. While this hyperparameter setting sits near the top of the distribution for Halfcheetah, it performs poorly in Swimmer.
Given algorithm E Ω. we define its hyperparameter sentesttv & as follows:
4 (w)
max Σ
e(w,ehh)
max h
r(w,e,h)
(2)
hE
E
BEF
GEF
The hyperparameter sensitivity of an algorithm is the difference between its per-environment tuned score and its cross-enviromment tuned score. The per-enviromment tuned score is the average nor-

---

Normalized Environment Score
Figure 3: The distributions environment normalized scores for 625 hyperparameter set- tings the PPO algorithm the Swimmer and Halfcheetah environments. The red stars indicate the normalized environment scores of hyperparameter setting, which does well in Halfcheetah but poorly in Swimmer. The blue stars indicate the normalized scores of the hy- perparameter setting, which maximizes the mean of the normalized environment scores across both environments.
1.4
Setting
1.2
maximizes mean
1.0
environment score
0,4
Setting does well
0.2
Halfcheetah
poody Swimmer
0.0
-0.2
Swimmer
Halfcheetah
malized environment score with hyperparameters tuned per environment. The cross-environment tuned score is the normalized environment score of the best fixed hyperparameter setting across the distribution of environments. We can use this notion of hyperparameter sensitivity to better understand new and existing algorithms. Reporting both hyperparameter sensitivity and the conven- tional performance-only evaluation metrics should provide more complete picture algorithm performance.
3.2 Sensitivity Analysis
Modern reinforcement learning algorithms are complex learning systems, and understanding them difficult task that requires multiple dimensions of analysis. Benchmark performance has been the primary metric (and often the only one) used for evaluating algorithms. However, this is only one dimension along which algorithms can evaluated. Hyperparameter sensitivity an important dimension consider in the evaluation space, especially practitioners begin to apply reinforcement learning algorithms to real-world applications. We propose the performance-sene sensitiviiy plane to aid in better understanding algorithms.
Consider the performance- sertsmiance-sensiiiiitty plane shown in Figure To construct the plane, the center point is set to the hyperparameter sensitivity and per-environment tuned score of some reference point algorithm. We can consider how other algorithms relate this reference point by considering which region of the plane they occupy. There are regions of interest shaded by different colors and labeled numerically, which we will consider turn.
An ideal algorithm would be both more performative and less sensitive. Therefore, algorithms that fall in Region 1 (the top left quadrant) of the plane would be a strict improvement over the reference point algorithm. For some applications, perhaps additional sensitivity can be tolerated if the gains in performance are large enough. Algorithms that fall in Region 2 an example of this. The region represents algorithms whose increase in performance is greater than the corresponding increase sensitivity. Conversely, for some applications sensitivity may matter a great deal and some performance loss can be endured. Algorithms that fall in Region 3 are example of those whose decrease in sensitivity outmatches their corresponding decrease performance, Regions 1-3 represent algorithms that have notable redeeming qualities either in terms of performance, hyperparameter sensitivity, or both. However, perhaps a practitioner does not care about sensitivity. For example, they want to maximize the score of a specific benchmark, and hyperparameter tuning is no issue. Algorithms in Region 4 may be adequate as they are algorithms that exhibit performance improvements and an even higher reliance upon per-environment hyperparameter tuning. Finally, those unfortunate algorithms that live in Region 5 are space with both lower performance and higher sensitivity, making them undesirable.
natural application of this diagram is to set the reference (center) point the hyperparameter sensitivity and performance of some base algorithm and study how proposed modifications (or ablations) affect both sensitivity and performance. Often, new algorithms are created by modifying existing algorithms, such normalizing targets, adding regularization term to the loss function, gradient clipping, etc. We illustrate an example of this using PPO as a reference point in the next section.
5

---

Figure 4: The ceeeommance-sesstivvii plane for algorithmic evaluation. The center point indicates the hyperparame- ter sensitivity and performance of a ref- erence point algorithm. The x-axis is the hyperparameter sensitivity metric as defined in equation [ The y-axis is the per-environment tuned score (first term equation 2). The diagonal line is the identity line shifted to intersect the refer- ence point algorithm. The plane is then divided into 5 shaded regions that rep- resent spaces of algorithms of varying qualities relative to the baseline.
2
Per-uriinmmen Tuned Score
1
4
3
5
Hyperparameter seasiiivtty
Sensitivity Experiments
To illustrate the utility of the sensitivity analysis presented above, we performed an experiment to study the hyperparameter sensitivity and performance of several variants the PPO algorithm, widely used
gradient method in reinforcement learning (Schulman et al.] 2017)) We considered several normalization variants commonly used in PPO implementations (Andrychowicz et al. 2021 Huang et al., (2022) and some normalization variants, introduced in Dreamer V3, that were purported to reduce hyperparameter sensitivity (Hafner et al. 2023 Sullivan et al. 2023).
4.1 Proximal Policy Optimization
PPO is an instance of an actor-critic method (Sutton & Barto 2018) that maintains two neural networks:
policy network with parameters 9 and value network with parameters w. Given policy T the agent performs roll-out of T steps, (S1, A1, R1, S2, A2. RT. Ar, RT). this rollout is then split into m batches of length The critic is then updated via ADAM (Kingma Ba 2015) to optimize MSE loss with the truncated-A return target. The actor is updated via ADAM (Kngmma & Ba 2013) to optimize a clipped surrogate objective toward maximizing the expected return. An entropy regularizer is added to the actor loss to encourage exploration.
4.2 Normalization variants
Several normalization variants have been used in PPO implementations. We focus on three categories of normalization: observation normalization, value function normalization, and advantage normaliza- tion. An intuition behind value function or advantage normalization for hyperparameter sensitivity is that the scale and sparsity of rewards vary greatly across environments and that value function advantage normalization should make actor-critic updates invariant to these factors possibly requiring less tuning of the step-size hyperparameters (van Hasselt et al. 2016). Another claimed benefit of advantage normalization variants is that by normalizing the advantage term, it is easier to find an appropriate value for the entropy regularizer coefficient across distribution of environments (Hafner et al. 2023). Observation normalization standardizes the network inputs. This can mitigate large gradients, which may stabilize the learning system for hyperparameter tuning (Hafner al. 2023: Andrychowicz et al. 2021), ), especially critic and actor step-size hyperparameters Ow, Og >>
Ad Advantage per-minibatch zero-mean normalization: A common implementation detail of PPO per-minibatch advantage normalization (Huang et al. 2022). When performing an update, the advantage estimates used in the actor loss function are normalized by subtracting the mean of the advantage estimates in the sampled batch and dividing by the standard deviation of advantage estimates in the sampled batch.
Advantage percentile scaling: Another form of advantage normalization was introduced in the D3eamerV3 (Hafner et al., 2023) ablations which divides the advantage estimate in the actor loss by a scaling factor. Exponential moving averages are maintained over the 95th and 5th percentiles of return estimates. The advantage term is divided by the difference of the two percentiles.

---

Advantage lower bounded percentile scaling: An alternate variant of percentile scaling is used in the DreamerV3 algorithm. Lower bounded percentile scaling applies a max operation the percentile scaling factor, preventing the estimated advantage term from blowing up if the percentile difference falls below a threshold.
Value target symlog:
DreamerV3 introduced a method of scaling down the magnitudes of target values by the symlog function. The symlog function and its inverse symexp are defined as:
symlog(z) = sign(x)) 1)
symexp(#) = (exp(=) -
(3)
As in DreamerV3, symlog is applied to the target in the critic loss and symexp is applied to the output of the critic network. In a subsequent study that applied DreamerV3 tricks to PPO (Sullivan et al. 2023), it was reported that the symlog transformation of the value target was one of the most Me reward clipping when applicd to PPO.
impactful tricks in environments without
Observation zero-mean normalization: A very common procedure with PPO is to normalize
observations by maintaining running estimates of the mean and standard deviation of observations.
Observation symlog: DreamerV3 proposed an alternative form of observation normalization by applying the symlog function, compressing the observations.
handful of prior work has investigated the benefits of similar algorithmic modifications to PPO. Previous work has reported the performance impact of the normalization variants commonly used with PPO: per-minibatch zero-mean advantage normalization and zero-mean observation normalization (Andrychowicz et al. (02TT) Other work (Sullivan al. (023) investigated how PPO's 's performance is affected by the normalization variants introduced in Dreamer V3 (lower bounded percentile scaling, value target symlog, and symlog observation) and that symlog was especially helpful in Atari when reward clipping is disabled. In addition, this work did not perform any hyperparameter tuning for the variants of PPO they tested. In our results, hyperparameter tuning demonstrated significant effect on the relative performance of these algorithms.
Given these normalization variants, natural question that arises is how do they affect the hyperparam- eter sensitivity of PPO? To the best of our knowledge, a careful study of the effect these normalization variants have the hyperparameter sensitivity of PPO has yet to be done and it provides good test of our new methodology.
4.3 Sensitivity Experiment with PPO variants
We investigated the effect of each of the described normalization variants on PPO. To isolate these effects, we did not apply reward clipping, reward scaling, or observation normalization wrappers by default. We focused our attention on four critical hyperparameters of PPO: the step-size for the critic Ow, the step-size for the actor ag, the coefficient of the entropy regularizer т, and the truncated-X A return mixing parameter A. We performed large grid search spanning five orders of magnitude across five Brax Mujoco domains. Near the extreme endpoints of the grid search, some hyperparameter configurations diverged. We ignored hyperparameter combinations that caused a particular algorithm to diverge over 10% of the time. We averaged the performance over the non-diverging runs.
plane in Figure 5]] The reference point at the center is the hyperparameter sensitivity and performance found for PPO without normalization. The error bars displayed indicate 95% confidence intervals formed from a 10,000 sample bootstrap. First, note that none of the normalization variants resulted in an improvement that both raised performance and lowered sensitivity. All forms of advantage normalization increased performance. However, this performance gain comes with trade-off: increased hyperparameter sensitivity. The marginal gain in performance per unit of increased sensitivity varied between ad advantage normalization methods. Advantage per-minibatch zero-mean normalization had a greater increase in performance than sensitivity (Region 2). Both percentile scaling- based variants of advantage normalization resulted in more significant sensitivity increases than performance increases (Region 4), indicating an enhanced reliance on hyperparameter optimization methods. Applying the symlog function to the value target lowered performance and may have slightly increased sensitivity (Region 5). It may, however, be the case that the choice of environment distribution did not have enough variation in reward magnitude for the utility of value target symlog to be demonstrated. It appears that observation normalization may slightly reduce sensitivity, although it is unclear due to the width of the confidence intervals.
Consider the ieftormmnnce-enssitiieee
7

---

Environment Tuned Performance (y-axis)
1.5
Advantage
per-minihatch
zero-mean
normalization
Observation
Advantage
Advantage
percentile
normalization
PPO
lower bounded
scaling
perventile scaling
Observation symlog
1.2
Value target symlog
1.0 0.00
0.15
0.30
Sensitivity (x-axis)
Figure 5: Performance-sensitee plane with unnormalized PPO the center reference point. Variants of PPO plotted. The x-axis indicates hyperparameter sensitivity as defined in equation 2 The y-axis represents the per-environment tuned score (first term in the sensitivity calculation of equation 2 Hyperparameter sensitivity and per-environment tuned score metrics were computed from a 200 run sweep of 625 hyperparameter settings across 5 Brax Mujoco environments (Ant, Halfcheetah, Hopper, Swimmer, and Walker2d), Error bars show the endpoints of 10,000 sample 95% bootstrap confidence intervals around both the performance and hyperparameter sensitivity metrics (two dimensions).
The performance--eesittee plane provides insights into how algorithmic changes may alter an algorithm's reliance on per-environment hyperparameter tuning with no additional computational expense other than what is already required from standard hyperparameter tuning procedure.
The sensitivity metric intimately tied to the chosen environment distribution, and our findings are thus limited to the Brax Mujoco environments tested. We argue the restricted environment distribution not just practical choice, but was feature of the study. Somewhat surprisingly we found significant sensitivity across distribution of fairly similar environments. In Figure displayed in Appendix DD we perform a leave-one-out study reporting how the cerformance-sensitvve changes when each of the five environments is dropped from the data. Observe that while the exact values of the points shift, their position relative to the reference point remains mostly unchanged.
plane
The performance-senstte
plane allows for a richer understanding of algorithms than performance- only evaluation procedures, but it does not capture the full picture. Two algorithms can sit in the same location on the plane and yet have very different hyperparameter characteristics. Consider case where there are two algorithms. The first algorithm highly sensitive with respect to one hyperparameter, which needs to be carefully tuned per environment. The second algorithm has the same sensitivity but needs to tuned per environment for dozens of hyperparameters with complex interactions. The hyperparameter sensitivity would not differentiate between these two algorithms. An additional metric is needed.
5 Effective Hyperparameter Dimensionality
There are many cases where practitioner can tune some but not all of an algorithm's tunable hyperparameters. It may the case that if few key hyperparameters are tuned per environment, then a preponderance of an algorithm's 's potential performance can be gained. This motivates the definition of effective hyperparameter dimensionality, metric that measures how many hyperparameters must be tuned in order to obtain near-peak performance.

---

Normalized Score (y-axis) vs, Number Hyperparameters (x-axis)
1.3
Advantage percentile
1.3
1.3
Observation zm normalization
scaling
1.2
1.2
Advantage LB pereentile scaling
1.2
1.1
PPO
1.1
1.1
0
1
2
3
4
0
1
2
3
4
0
1
2
3
4
1.3
Observation symlog
1.2
1.3
Value target symlog
1.2
1.1
1.2
Advantage normalization
1.1
1.0
1.1
0
1
2
3
4
0
2 4
0
1
2
3
4
Figure 6: Normalized performance scores as a function of the number of hyperparameters tuned per environment. The subplots compare PPO to the PPO variants studied. The x-axis indicates the size of the subset of hyperparameters being tuned. The y-axis the average normalized score across the environment distribution. Each dot indicates the normalized score obtained by tuning the most performant subset of hyperparameters of each size. The curve is an interpolation between the dots. The dashed line indicates the point which the curve reaches 95% of peak performance. LB is abbreviation for lower bounded, zm is an abbreviation for zero-mean.
For given algorithm w with hyperparameter space H" number tunable hyperparameters n(w), and environment distribution ε, let h" = arg max |&l Eeeg [(w, e, h) be the hyperparameter
setting which maximizes the cross-environment tuned normalized score. Effective hyperparameter dimensionality d(w) defined as
d(w) n(w)
Σ
L h Ve E
E J.Ee
(4)
s.t.
l'l,e, 10) 0.95
max I'(w,e,
3
keH
33
333
The effective hyperparameter dimensionality of an algorithm measures the minimal number of hyperparameters that must be tuned per-environment while retaining the majority of the performance that can realized by tuning hyperparameters per environment. The threshold of 95% peak performance can be changed at practitioner's discretion to whatever meets their performance requirements. To compute effective hyperparameter dimensionality, one needs to consider subsets of hyperparameters to find minimum subset that achieves the required performance threshold.
For the same algorithmic variants PPO studied above, Figure (o displays normalized scores as function of the number of hyperparameters tuned per environment, choosing the most performant subset tune. Table Appendix [[rovidess listing of the most performant subsets of varying sizes observed during this experiment. The curve interpolates between the normalized scores. The vertical dashed line indicates point along the curve that reaches 95% of the per-environment tuned score. In the case of advantage percentile scaling, modifying PPO with the normalization variant moves the point to the right, indicating this variant improves performance the cost of increasing pressure on the number of hyperparameters necessary to tune. Also, note how the performance

---

ranking can shift based on the number of hyperparameters that have been tuned; such as with PPO and the advantage percentile scaling variant (top left plot). For some variants, performance flattens after tuning only three hyperparameters. Whereas for other variants, performance is almost linear in the number of hyperparameters tuned, suggesting sensitivity to all hyperparameters. On the performance-sensttivii plane (Figure[5jn the previous section), both the advantage percentile scaling and advantage lower-bounded percentile scaling variants fall in the same region (Region 4). Yet, in Figure [ we can see that the advantage lower bounded percentile scaling variant can obtain higher performance levels than the advantage percentile scaling variant when tuned on smaller subsets of hyperparameters. This observation that algorithms can have similar hyperparameter sensitivities and vastly different effective hyperparameter dimensionalities indicates the power of using both metrics for studying algorithms.
Limitations and Future Work
The hyperparameter sensitivity and effective hyperparameter dimensionality metrics will depend heavily on several important empirical design choices. The premise of both metries is that a practitioner is concerned with understanding sensitivity with respect to a distribution of environments that they care about. If the distribution of environments changes, the metrics will need to be evaluated with respect to the new distribution. This dependence exploited, as someone could artificially make an algorithm appear less sensitive by including several easy environments that all hyperparameter settings will do well in, although score normalization will somewhat counteract this. In addition, the level of granularity with which hyperparameter sweeps performed will have effect on both metrics. Another factor that impact the metries is the choice of the score normalization method used. A practitioner could use other score normalization methods and the resulting sensitivity scores may be different.
the environment distribution could be
next step is to apply the proposed sensitivity and dimensionality metrics to a larger set of algorithms and environments. Related to this work is the literature of AutoRL (Eimer et al. (023). The goal of AutoRL is to tune hyperparameters via some hyperparameter optimization algorithm (which make use of their own hyperparameters- -hapeneeeeeee--p---eeeeee (ypepartpperrrraeeetrr)))) Future work could use the definitions provided here to try to understand the algorithms proposed in the AutoRL literature reduce sensitivity over the base algorithms that are being modified-mmasuingg the sensitivity of the hyper- hyperparameters. A study comparing the sensitivities and dimensionalitics of AutoRL methods to the sensitivities and dimensionalities of the base learning algorithms they optimize would be prudent.
Conclusion
As learning systems become more complicated, careful empirical practice is critical. Modern reinfor- ement learning algorithms contain numerous hyperparameters whose interactions and sensitivities are not well understood. Common practice, which is focused on achieving state-of-the-ar performance, risks overfitting to benchmark tasks and overly relying on hyperparameter optimization. Most empiri- cal work in reinforcement learning has focused only evaluating algorithms based on benchmark performance, leaving the effects of hyperparameters under-studied. this work, we propose a new evaluation methodology based on two metrics that allow practitioners to better understand how an algorithm's performance relates to its hyperparameters. We show how this methodology is useful in evaluating methods purported to mitigate sensitivity. We identify that the studied advantage normalization methods, while improving performance, also increase hyperparameter sensitivity and can increase the number of sensitive hyperparameters.
Acknowledgments
The authors would like to thank Ana Paola Garcia Alonzo for assistance in formatting figures. The authors would like to thank David Sychrovský and Anna Hakhverdyan for useful discussions. This research was supported by grants from the Alberta Machine Intelligence Institute (Amii); Canada CIFAR AI Chairs, Amii; and NSERC. Compute was made available by the Digital Research Alliance of Canada.
10

---

References
Marcin Andrychowicz, Anton Raichuk, Piotr Stanczyk. Manu Orsini, Sertan Girgin, Raphaël Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. What Matters in On-Policy Reinforcement Learning? A Large-Scale Empirical Study. In International Conference Learning Representations (ICLR), 2021.
Theresa Eimer, Carolin Benjamins, and Marius Lindauer. Hyperparameters in Contextual RL are Highly Situational. CoRR, abs/2212. 10876, 2022.
Theresa Fimer, Marius Lindauer, and Roberta Raileanu. Hyperparameters in Reinforcement Learning and How To Tune Them. In International Conference on Machine Learning (ICML), 2023.
Jörg KH Franke, Gregor Köhler, André Biedenkapp, and Frank Hutter. Deep Reinforcement Learning. In International Conference Learning Representations (ICLR), 2021.
Automated
C. Daniel Freeman, Erik Frey. Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax A Differentiable Physies Engine for Large Scale Rigid Body Simulation. In Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS), 2021.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy P. Lillicrap. Mastering Diverse Domains through World Models. CoRR, abs/2301.04 2023.
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining Improvements in Deep Reinforcement Learning. In Conference on Artificial Intelligence (AAAI), 2018.
Shengyi Huang, Rousslan Fernand Julien Dossa, Antonin; Raffin, Anssi, Anssi Kanervisto, and Weixun Wang. The 37 Implementation Details of Proximal Policy Optimization. In International Conference on Learning Representations Blog Track (ICLR), 2022.
Scott M. Jordan, Yash Chandak, Daniel Cohen, Mengxue Zhang, and Philip S. Thomas. Evaluating the Performance of Reinforcement Learning Algorithms. In International Conference on Machine Learning (ICML), 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations (ICLR), 2015.
Michail Lagoudakis and Ronald Parr. Least-squares policy iteration. Journal of Machine Learning Research (JMLR), pp. 1107-1149 2003.
Chris Lu, Jakub Kuba, Alistair Letcher, Luke Metz, Christian Schroeder de Witt, and Jakob Foerster. Discovered Policy Optimisation. In Neural Information Processing Systems (NeurIPS), 2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human- Control through Deep Reinforcement Learning. Nature, 518:529-533, 2015.
Andrew Patterson, Samuel Neumann. Martha White, and Adam White. Empirical Design in Rein- forcement Learning. CoRR, abs/2304.013155 2023.
Andrew Patterson, Samuel Neumann, Raksha Kumaraswamy, Martha White, and Adam M White. The Cross Enviromett Hyperparameter Setting Benchmark for Reinforcement Learning. In Reinforcement Learning Conference (RLC), 2024.
Martin Riedmiller, Neural fitted iteration first experiences with a data efficient neural reinforcement learning method. In European Conference on Machine Learning (ECML), 2005.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. CoRR, abs/ 1ww.//0777767,,,0. 2017.

---

Anton Schwartz. A reinforcement learning method for maximizing undiscounted rewards. In International Conference on Machine Learning (ICML), 1993.
Ryan Sullivan, Akarsh Kumar, Shengyi Huang, John P. Dickerson, and Joseph Suarez. Reward Scale Robustness for Proximal Policy Optimization via DreamerV3 Tricks. In Advances in Neural Information Processing Systems (NeurIPS). 2023.
Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT Press, 2018.
Hado van Hasselt, Arthur Guez, Matteo Hessel, Volodymyr Mnih, and David Silver. Learning Values Across Many Orders Magnitude. In Neural Information Processing Systems (NeurIPS), 2016.
Christopher John Cornish Hellaby Watkins. Learning from Delayed Rewards. PhD thesis, University of Cambridge, 1989.
A Broader Impact Statement
Hyperparameter sweeps and per-environment tuning are the most computationally expensive and environmentally impactful parts of reinforcement learning research. Our study ran for approximately 4.5 GPU years NVIDIA 32GB V100s. While this is substantial, we believe that using compute to better understand the sensitivity of current algorithms is an essential step towards developing more environmentally friendly algorithms. This work investigated the hyperparameter sensitivity of reinforcement learning agents. The immediate societal impact is minimal. However, methodology may aid in developing performative algorithms with low hyperparameter sensitivity. this occurs, these algorithms will result in less need for hyperparameter tuning and, result, have positive impact on lowering the carbon footprint of reinforcement learning experiments.
empirical methodology for evaluating
B Proliferation of Hyperparameters
There is a trend in which the current state-of-the-arr algorithms often contain more hyperparameters than the previous state-of-the-art. Table[]list hyperparameter counts for representative algorithms from each of the three main categories of reinforcement learning methods; values-based, policy- gradient, and model-based.
C Hyperparameter Sweep Details
The PPO implementation used was heavily inspired by the PureJaxRL PPO implementation (Lu et al. 2022). The variants advantage per-minibatch zero-mean normalization and observation zero-mean normalization are the standard implementations provided within PureJaxRL. The variants: symlog observation, symlog value target, percentile scaling, and lower bounded percentile scaling closely follow the implementation of the DreamerV3 tricks applied to PPO shown in Sullivan et al. (2023) as well as referencing the original DreamerV3 repository Hafner et al. (2023).
The policy and critic networks were parametrized by fully connected MLP networks, each with two hidden layers of 256 units. The network used the tanh activation function. Separate ADAM optimizers (Kingma & Ba, 2015) were used for training the actor and critic networks.
The environments used in the experiments were the Brax implementations of Ant, Halfehee- tah, Hopper, Swimmer, and Walker2d. (Freeman et al.) 2021). The hyperparameter sweeps were grid searches over eligibility trace E {0.1, O.3, 0.5, 0.7, 0.9). entropy regularizer coefficient E f0.00001, 0.0001, 0.001, 0.01, 0.13, and
Waller2d..freeman
E {0.001, 0.01, 0.1. 1.0, 10.0). actor step-size critic step-size aw (0.00001, 0.0001, 0.001, 0.01, 0.1). Each run lasted for 3M environment steps. 200 runs were performed for each of the algorithms, environments, and hyperparameter settings. Like the PureJaxRL PPO implementation, the entire training loop was implemented to run on GPU. We will release code and experiment data at httpt:tt/// : gi thub. wwwp/aaaiis99/////.m.cmmm /hyperparameter. sensitivity) promoting the further investigation of hyperparameter sensitivity in the field of reinforcement learning.
12

---

Algorithm name
Year
Number of hyperparameters
Comments
Q-lcarning
1989
3
(+1) step size, (+1) e-greedy exploration, (+1) discount factor
(Watkins, 1989
R-Iearning
1993
3
(+2) size5, (+1) e-greedy exploration (Schwartz) 1993
LSPI
2003
4
(+1) discount factor (+1) stopping condition, e-geedy explo-
ration (+1) initialization parameter (Lagoudakis & Parr 2003
NFQ
2006
5
(+1) discount factor (+1) e-greedy exploration, (+2) U Use KProp,
(+1) number of inner-loop iterations (Riedmiller) 2005
DQN
2013
16
Hyperparameter tahle from paper.
PER
2016
20
(+16) inerited from DQN, (+4))added for PER.
Rainbow
2017
25
(+20) inherited from PER, (+5) (1 extra from changing optimizer
from RMSProp ADAM. step return, 3 from distributional
RL)
Agent57
2020
34
Hyperparameter tahle from paper.
Actor-critic
1983
(+1) actor slep size, (+1) critic step size, (+1) discount factor
Actor-critic with
1983
5
(+1) actor slep size, (+1) eritic slep size, (+1) discount lactor, (+2)
eligibility trace A3C
eligbiliiy traces
2016
(+3) Uses KMSProp, (+I) value function loss coefficient, (+1)
entropy coefficient, (+1) number of actors, (+1) batch size, (+1
gradient clipping. (+1) atari frame stacking, (+1) target network
update rate, (+1) discount factor
PPO-clip
2017
24
Hyperparameter table Irom paper Additional hyperparame-
ters discussed in ICLR blogpost (+11) inherited from A3C,
(+1) from switching to ADAM, (+1) returns, (+1) number of
epochs, (+1) actor loss clipping. (+1) value loss clipping, (+2)
Log stdev of action distribution I (imearAnnea(--0,7,1..00
(+1)
ADAM LR annealing. (+1) reward scaling, (+2) reward clip, (+2) nost-normalization observation clip.
SAC
2018
22
Stable Baselines documentation, (+10) inherited from A3C, (+1)
ADAM, (+1) number of epochs, (+1) reward scaling, (+2) reward
clip, (+2) post-normalization observation clip, (+1) entropy coef- ficient step-size, (+1) target entropy, (+2) action Gaussian noise, (+1) polyak update.
Tabular Dyna
1991
(+1) e-gredy, (+I) discount factor, (+1) number of planning steps,
(1) stepsize
Tabular Dyna+
1991
5
(+1) c-greedy, (+1) discount factor, (+1) number of planning steps,
(+1) stepsize (+1) planning exploration reward bonus
Linear Dyna
2012
(+1) e-greedy. (+1) discount factor, (+1) number planning
steps. (+-3) stepsizes (pupers uses one stepsize but there it is used
for transition matrix, expected reward vector, and value function estimate
DreamerV1
2020
26
Paper. (+1) number random seed episodes, (+1) number of update
steps, (+1) buffer size. (+1) batch size, (+1) batch length, (+1) imagination horizon, (+1) X. (+1) actor step-size, (+1) critic step- size, (+3) ADAM, (+1) RSSM units. (+1) discount factor, (+1) entropy coefficient, (+1) target network update, (+1) action model tanh scale factor, (+1) step-size for world model, (+1) gradient clipping, (+1) KL regularizer clipping. (+2) action Gaussian noise, (+1) action repeat, (+3) discrete action e-greedy linear decay over first 200k steps.
DreamerV2
2022
28
Modification summary. (+26) inherited from DreamerV1, (+1)
number of discrete latents, (+1) KI. balancing, (+1) actor gradient mixing, (+1) weight decay, (-2) removed action Gaussian noise. Hyperparameter table from paper Modification summary. (+28)
DreamerV3
2023
41
inherited from Dreamer V2, (+1) World model loss clipping, (+1) actor unimix random exploration, (+3) advantage percentile scal- ing (decay rate, lower bound. additional terms in world model loss, (+1) Critic EMA decay, (+1) Critic EMA regularizer coefficient, (+2) Adaptive Gradient Clipping, (+1) Critic replay loss scale, (+1) Critic loss scale, (+1) Actor loss scale, (+1) latent unimix, (-2) same step-size for actor, critic, and world model.
percentile scaling), (+2) two
Table 1: This table reports counts of hyperparameters (excluding those relating to neural network architecture) from a sampling of prominent algorithms proposed over the last decade. When possible. we tried to use hyperparameter tables listed in the original papers. Otherwise, we used documentation from popular implementations. The comments column contains links sources used.
13

---

Algorithm Variant
Size Subsets
Size Subsets
Size Subsets
PPO
A
T. A
T, A, Ow
Observation symlog
À
A
X,
Observation zero-mean normalization
À
A
),
Advantage percentile scaling
Gu
(ig Oww
Advantage lower bounded percentile scaling
Ou
Ou
Ge, Ctw
Advantage per-minibatch zero-mean normalization
Å, Ow
T, A, Ow
Value target symlog.
0g T,A À
T, A, Ow
Table 2:
The subsets of hyperparameters that were found be most impactful to tune per environment
as measured when creating Figure 6
Sigper
Environment
Tined Performance
Walerbd
Senvtivity
Figure 7: performance-sensiiivve planes shown are formed by leaving out each of the five environ- ments. Error bars are 95% confidence intervals obtained from 1000 sample bootstraps.
D Additional Figures
The sensitivity metric is ultimately tied to distribution of environments, and our findings are thus limited to Brax Mujoco environments tested. Claims cannot be about environments outside the evaluated environment set. One may wonder about the stability of the results if the environment distribution changes on small scale. In Figure 7 we repeat the sensitivity metric for each of the five environment subsets that can be obtained by dropping single enviromment. Leaving out environments does shift the reference point, and the position of the variants shifts somewhat relative to the reference point. However, the regional position of the variants relative to the reference point is mostly consistent.
In Figure 8, we also include results with final performance, i.c., the sum of rewards obtained over the final 1000 timesteps of learning (the truncation length). The performance reinforcement learning agents does not monotonically increase. In many cases, with specific hyperparameters, it may collapse in the middle or end of learning. Therefore, this metric is noisy, and it is difficult to reason about where algorithms lie according to it on the performance sensitivity curve.
14

---

Algorithm Variant
Environment
7
Cte
Ow
Mean of Returns
Standard
Deviation of Returns
Advantage percentile scaling
ant
0.9
10
10
10
34.64
8.54
Advantage percentile scaling
halfcheetah hopper
0.3
10
10
10
2220.81
179.74
Advantage percentile scaling
10
10
10
1005.
41.22
Advantage pereentile scaling
swimmer
0.9
10
10
10
35.34
5.83
Advantage percentile scaling
walker2d
0.9
10-2
10
10-3
846.77 72
94.99
Advantage lower bounded
ant
0.7
10
10
10
35.53
11.22
percentile scaling
Advantage lower bounded
halfcheetah
0.3
10
10-3
2216.12
176.56
percentile scaling
Advantage lower bounded
hopper
0.9
10
10
10-3
1003.62
46.69
percentile scaling
Advantage lower bounded
swimmer
0.9
10-3
10-2
10-4
41.24
23.59
percentile scaling
Advantage lower bounded
walker2d
0.9
10
10
10-3
831.81
140.81
percentile scaling
Advantage per-minibatch
0.7
10
10
36.98
10.37
zero-mean normalization
Advantage per-mine per-minibatch
halfcheetah
0.3
10
10
10
2369.19
169.13
mean normalization
Advantage per-minibatch
hopper
0.9
10-3
10-3
10-3
1006.39
30.88
zero-mean normalization
Advantage per-minibatch
swimmer
0.5
10
10
10-5
41.11
3.29
cero-mean normalization
Advantage per-minibatch
walker2d
0.7
10-2
10
10-3
824.87
96.69
meo-mean normalization
PPO
ant
0.7
10
10
10
38.95
9,94
PPO
halfcheetah
0.5
10-2
10
10-3
2311.89
172.22
PPO
hopper
0.9
10-3
10
10
1002.97
71.25
PPO
swimmer
0.9
10-3
10-4
10-4
33.41
9.41
PPO
walker2d
0.9
10
10
761.36
92.23
Observation zero-mean
ant
0.7
10
10
10
39.78
9.85
normalization
Observation zero-mean
halfcheetah
0.5
10 2
10
10
2306.22
161.09
normalization
Observation aro-mean
hopper
0.9
10-3
10 3
1
1004.47
68.99
normalization
Observation zero-mean normalization
swimmer
0.9
10-2
10
10-4
33.63
2.14
Observation zero-mean
walker2d
0.9
10
10
10-3
755.25
88.62
normalization
Value target symlog
ant
10 1
10
10
10
Value target symlog
halfcheetah
0.7
10
10
10
1731.16 16
149.47
Value target symlog
hopper
0.9
10
10
10
1067.40
57.11
Value target symlog
swimmer
0.5
10-3
10
10
34.67
1.00
Value target symlog
walker2d
0.9
10
10
10
698.09
85.97
Observation symlog
ant
0.7
10
10
10
39.57
11.15
Observation symlog
halfcheetah
0.5
10
10
10-3
2321.00
147.05
Observation symlog
hopper
0.9
10
10
10
1007.13
69.63
Observation symlog
swimmer
0.9
10-3
10
10
32.90
9.60
Observation symlog
walker2d
0.9
10-1
10
10
767.30
77.60
Table 3: This table reports means and standard deviations across seeds of the average return observed during learning for a subset the different hyperparameter settings, algorithm variants, and environ- ments.
15

---

Final Return Performance Metric
Advantage per-minhhutc
acto
Rocmallzztion
aato nocnelization
14
Advantage lowas bounded percentle scaling
persentie sciling
Obsrvatio
Value
190
synlog
0.2
Figure 8: Performance-sensitivee plane with final return as the performance metric, Variants of PPO plotted. The x-axis indicates hyperparameter sensitivity defined in equation The y-axis represents the per-environment tuned score (first term in the sensitivity calculation of equation 2 Error bars are 95% confidence intervals from 1000 sample bootstrap.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The contribution of this paper is an empirical methodology for assessing hyperparameter sensitivity. The presented experiments provide evidence for the utility of the methodology, as stated in the introduction and abstract.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as it is clear that the paper does not attain these goals.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
See that there is section titled "Limitations and Future Work" which discusses limitations.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
17

---

Answer:
[NA]
Justification:
This paper provides definitions of metrics. It does not make theoretical claims.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Details of implementations and hyperparameter sweeps are given in the appendix. The error bars that are presented are obtained by computing a 10000 sample 95% boostrap confidence intervals as described in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
18

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Code is included in supplementary material. Instructions are in README file.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ ccpppp.cc/pulll///ggge guwdes/ddeeeee CooeeSumeeieeeeeeeeeee ssi onyolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Experiments are described in the paper. Full experiment details are given in the appendix. Code is provided.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
The error bars shown in the main paper are 95% bootstrap confidence intervals. The confidence interval around the learning curves in the appendix is a 95% Student t- distribution confidence interval. This described in the main paper and appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This information is provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:[Yes
Justification:
We did notice any ethical conflicts when reviewing the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
This is discussed in a section of the appendix titled Broader Impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
20

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The work presented in this paper poses no such risk. It is an empirical methodology.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We wrote original code for this project. The PPO implementation was forked from PureJaxRI. They are cited throughout the paper, and their license is included in the supplementary material. We thank them and other contributors the opensource community.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The contribution of this paper is empirical methodology. Code for experi- ments will eventually be opensourced and made available to the community.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification:
There were no human subjects used this research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
22

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "spnizojjSa", "reviewer": "Reviewer_6Bue", "summary": "The paper introduces an empirical framework for assessing the hyperparameter sensitivity of reinforcement learning algorithms.\nThe framework consists of two metrics: 1. hyperparameter sensitivity, which gives a normalized difference in performance between the per-task best hyperparameters and the across-task best hyperparameters. 2. effective hyperparameter dimensionality, which gives the number of hyperparameters that can be left the same as the across-task hyperparameters while tuning the rest and still obtain a threshold say (95%) of the per-task best hyperparameter performance.\nUsing these metrics in addition to performance metrics, practitioners and researchers can have a better idea of the benefits/downsides of a modification to an algorithm (how much better vs how much more sensitive)\nThe framework is used to compare several normalization variants of PPO introduced in the past years on the continuous control environments, giving a better picture of their contribution.", "questions": "Reporting the mean under the curve (AUC) with 95% bootstrapped confidence intervals over 200 runs is great, but I would expect some discussion on the use of a confidence interval, as it collapses the shaded area across the 200 runs to the statistic being computed (here the mean AUC).\nTo me, using a dispersion measure like the variance of the mean AUC would also be valid, as some hyperparameters would have more variance than others (although this could also be seen as an additional dimension of sensitivity).\nI would appreciate it if the authors could comment on this.\nAlso, at what point are the confidence intervals computed? When computing the expected performance or when computing the hyperparameter sensitivity, etc?", "limitations": "The authors adequately state the limitations of their work and its broader impacts. In particular the potential limitation of the framework to a specific environment distribution.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The problem is well-motivated. Hyperparameter sensitivity is a well-known issue in deep RL with novel algorithms often providing better performance at the expense of a higher sensitivity. Such tradeoff must be made explicit.\nThe framework introduced in the paper provides an effective way to draw that tradeoff with a clear interpretation of it using Figure 3.\n\nThe metrics are simple and quite natural and provide a solid starting point for a hyperparameter sensitivity framework.\nAlthough computationally expensive and still to prove if its results transfer across domains, the framework is likely to have a high impact on the field.\nAt least on the continuous control with Brax and PPO variants where the framework has been used as an example.\nIn particular thanks to the thorough and accurate experimental protocol, with 200 seeds per experiment and confidence intervals.", "weaknesses": "Metric definition:\n- The effective hyperparameter dimensionality depends on the total number of hyperparameters of an algorithm and is likely to scale with it, so this makes it incomparable between algorithms with a different number of hyperparameters. Perhaps counting the number of parameters that changed instead would make it more comparable across algorithms, as is what would dictate the budget of practitioners eventually.\n\n\nHyperparameters:\n- I would consider the minibatch size and the number of epochs in PPO to be critical hyperparameters as well. It's not clear how the choice of hyperparameters to sweep over was made.\n- (minor) The epsilon in the denominator of the minibatch normalization may also play a big role.\n\nInconsistent/confusion notation:\nin line 91, the tuple (w,h) defines an agent (a,), but this is used in a confusing and inconsistent way throughout the paper.\n- line 116 $\\hat p(a, e, h): if $a$ is there then $h$ is redundant. It seems like $a$ there stands for $w$.\n- Equation 1 same. The $a$ should be a $w$.\n- Equation 2 and 5:  2 and 5: $\\Gamma(a, e, w)$ not it's even more confusing there is both an $a$ and an $w$ and there is an $h$ missing.\n\nClaims:\nLine 321 \"vastly difference effective hyperparameter dimensionalities\": the largest gap observed in the paper is from 2 to 4, so I would not extrapolate here, though arguably that's indeed an additional 2 dimensions to sweep over so scales exponentially.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review. Your comments are appreciated for improving this work. \n\n## Weaknesses \nMetric definition:\n\nThe effective hyperparameter dimensionality already does this by counting the number of the hyperparameters necessary to obtain a large fraction of peak performance. If algorithm x has two very sensitive hyperparameters that need to be tuned per environment, and algorithm y has 40 hyperparameters, but only one of them is sensitive, where the others are okay being left to default, then algorithm x will have an effective hyperparameter dimensionality of 2 while algorithm y will have an effective hyperparameter dimensionality of 1.\nOf course, as the number of hyperparameters grows, its number of sensitive hyperparameters may also grow, but this does not make it incomparable to other algorithms.\n\n## Hyperparameters\nThank you for the suggestions! Based on our review of the literature and previous experience, we chose hyperparameters that appeared to be important. The hyperparameter choices you suggested would be very interesting to investigate. If you believe it would add to the paper, we would be willing to run additional experiments with batch size, epochs, etc., before the camera-ready deadline.\n\n\n## Inconsistent notation\nThis resulted from a notation change for the algorithm set. Thank you for finding the inconsistency; it will be fixed for the camera-ready.\n\n## Questions\nThe confidence interval is computed by creating bootstrap datasets, i.e., sampling seeds with replacement from the original dataset, and then computing the performance and sensitivity metrics on the bootstrap datasets. This creates distributions of the algorithms' performance and sensitivity scores. The upper and lower endpoints of the confidence intervals are the 97.5th and 2.5th percentiles of the distributions. Please note that the collapse of the upper endpoint of the performance CI in Figure 5 resulted from a minor bug in the plotting code. After re-running experiments, we found that it did not impact the main message of our paper or the vast majority of the particular empirical outcomes. We have included the corrected Figure 5 in the PDF attached to the general rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wxs6fGLiFF", "reviewer": "Reviewer_yuTN", "summary": "This paper proposes a new evaluation regime for reinforcement learning. As opposed to only taking into account benchmark performance (i.e., final return), as is ocmmon in previous literature, this work suggests considering an extra dimension of how sensitive algorithms are to hyperparameters in tuning based on a heuristic developed in the paper. They analyse PPO, and some of its variants, using this new evaluation regime, and find that performance increases often correspond to increased sensitivity to hyperparameters. Finally, the paper considers a top-k approach to hyperparameter tuning, adjusting only the most impactful hyperparameter, to see how the baseline algorithms compare under a more limited hyperparameter tuning regime.", "questions": "- In the plots, the authors state that they plot 95% confidence intervals over 200 seeds; but in the y-axis, all of the 'confidence intervals' are only negative. I am not particularly clear why. Is this a mistake, or am I misinterpreting results? At the same time, are the circles representing the mean score over runs, environments, or both; and therefore, are the confidence intervals defined over the runs, the enviroments or a big stack of (in this case) 1000 results per point? Is comparison for the hyperparameter sensitivity done per seed or for the averages over seeds? I think there's a lot of questions about how exactly these results are being presented which doesn't come through in the paper.\n- In some cases, increased hyperparameter sensitivity can be a positive - it can give us extra opportunities to boost performance. Do you think this should be mentioned as a limitation of this method, which effectively calls for less sensitivity?\n- Why did you use 200 seeds per hyperparameter tuning configuration, which is significantly greater than would be used in most other research? In practice, would it not have been more sensible to have used a much smaller number of seeds (eg 8), and focused on a much tighter range of hyperparameters?\n- As a brief aside, the authors have not removed the instructions of the Neurips checklist as requested, which should be done if the camera ready copy is released.", "limitations": "The authors briefly discuss some limitations of their methods; in particular, that their proposed metrics are highly environment specific (even in this case, they are looking at only one suite of environments which may end up biasing results), and will need reevaluation every time you shift to a new set of domains. However, I think there are some other key limitations that are not discussed, many of which have been raised above. In particular:\n- In addition to being environment specific, the metrics proposed are going to be very specific to the range of values which are tested. I have mentioned above how the values chosen for hyperparameter tuning here are not particularly reasonable.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper clearly takes care to use an extensive number of experiments, possibly reaching into the realm of unnecessary, to provide some rigour to their results.\n- The domain being considered - considering the hyperparameter sensitivity of algorithms in addition to their performance - is an underexplored area and one which becomes increasingly important as the cost of experiments increase. \n- Approaching this problem in a visual setting seems reasonable.\n- I like the approach of distilling hyperparameter optimisation to optimising only a smaller set of more important values; this has real benefits in enabling tuning of the majority of hyperparameters in *cheap* environments and tuning the key values only in more expensive environments.", "weaknesses": "- Noting appendix A (the table of how many hyperparameters each algorithm has); the definition of 'hyperparameters' seems pretty weak, and a lot of those included seem to just be design decisions of the actual algorithms.\n- I find reading the plots quite confusing, exacerbated by all of the different colours marking each of the areas. I think this obfuscates the message of the plots and as a reader makes it hard to come to conclusions.\n- Given this work is based on PurejaxRL, which I think runs on Brax for 1e8 frames, these experiments seem very short (3e6 frames) and possibly doesn't give the algorithms the full opportunity to converge.\n- There is very limited discussion of preexisting literature in this space. While I appreciate this takes a subtly different tack (calling for us to measure how sensitive to hyperparameters RL algorithms are, rather than designing algorithms with few hyperparameters), I would expect to see significant more discussion of AutoRL literature. Framing this work better in related work would definitely strengthen the paper.\n- The formatting looks quite off with the figures. I think this is because a lot of the captions are to the side of the figures, rather than below.\n- Considering the actual hyperparameters being tuned, the results feels slightly disingenuous. If we assume the practitioner running the hyperparameter tuning, we would possibly expect them to select more reasonable values or focus the search in a significantly more targeted way. For instance, it is no surprise that the highly performant methods saw large performance decreases (i.e., hyperparameter sensitivity) when evaluated with entropy coefficients in the range [0.001, 10] or learning rates spanning 5 orders of magnitude. Instead, it would be much more sensible to explore how the performance changes with reasonable hyperparameters that one is more likely to practically tune over (i.e., we generally have a good idea of where to start with values, so are unlikely to try an entropy coefficient of 10).\n- It would be good to see some exploration of algorithms which have done something different than just considering the changes to normalisation in PPO returns or observations. It feels like some broad statements are made despite the fact that this analysis was only taken in a single setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We appreciate your feedback. \n\n ## Weaknesses \n\n1. We recognize that the delineation between hyperparameters and algorithm design can be fuzzy. We cite the sources used for our counts and, when possible, try to stick to hyperparameter lists identified by the original papers (e.g., the DQN Nature paper has a table of 16 hyperparameters, so the value in our table is 16).  When this was not possible, as a guiding principle, we tried to identify parameters of the algorithm that could easily be modified and may contribute to performance gains if tuned per environment. We did not count parameters related to network architecture (e.g. number of neurons in hidden layers). \n\n2. The two-dimensional plot is motivated by the fact that nuance is lost when algorithms are evaluated using one metric. Studying the relationship between sensitivity and performance can aid insight. Perhaps there is a better way to discriminate between regions other than colors. We would be happy to try out any suggestions the reviewer has. \n\n3.  The Mujoco experiments that we have observed in the literature ran for between 3e^6 - 5e^6 steps (e.g., OpenAI has published a 3e^6 step benchmark for the 5 environments that we tested: https://spinningup.openai.com/en/latest/spinningup/bench.html. Note we are evaluating based on AUC and not final performance; for final performance one might choose to run longer.\n\n4. You are correct that we are addressing a different problem. Often, algorithms in the AutoRL literature aim to tune hyperparameters via some HPO process (which sometimes contain additional hyperparameters of their own).  Future studies could use the definitions provided here to try to understand if the algorithms proposed in the AutoRL literature reduce sensitivity over the base algorithms that are being modified. We are proposing an evaluation methodology, not a solution to hyperparameter tuning.\n5. We will improve formatting.\n6. It is not uncommon to sweep stepsizes over multiple orders of magnitude see https://arxiv.org/pdf/2407.18840, which considers actor step size as low as 10^{-6} and as high as 10^{-2}). While it may seem a bit unusual to try an entropy coefficient of 10, consider that the proper order of magnitude of the entropy coefficient will depend heavily on the scale of the advantage term in the actor loss. This will be affected by the type of advantage normalization performed or the order of magnitude of the reward function if the advantage is left unnormalized. Existing literature (Table 4: https://arxiv.org/pdf/2306.01324 from the AutoRL literature) has already searched spaces of 4 orders of magnitude to find entropy coefficients for PPO in Brax. Considering that we are ablating various forms of normalization across different sets of environments, it does not seem unreasonable to try 5 different orders of magnitude. In our results, the entropy coefficient of 10 was not bad in every environment. In fact, there was a hyperparameter setting with an entropy coefficient of 10 for the observation normalization PPO variant that was in the 97th percentile of the settings we tried for Walker2d. Nevertheless, please note this paper's main contributions are the sensitivity and dimensionality metrics and the plots for visualizing them. The PPO study is used to demonstrate how these tools might be used. \n7. It would be great to apply the sensitivity and effective dimensionality metrics to other algorithms, environment distributions, etc. Note that claims will always be limited to some small set of agents and environments (as you noted above); general claims are never possible, especially considering many RL benchmark environments have been developed specifically to test current algorithms. We were very careful not to make claims beyond the methods tested. If there are specific statements in the paper that you feel are overreaching, let's discuss them.\n\n## Questions: \n1. The confidence intervals were computed by creating bootstrap datasets, i.e., sampling seeds with replacement from the original dataset and computing the performance and sensitivity metrics on the bootstrap datasets. This creates distributions of algorithms' performance and sensitivity scores. The upper and lower endpoints of the confidence intervals are the 97.5th and 2.5th percentiles of the distributions. Please note that the collapse of the upper endpoint of the performance CI in Figure 5 resulted from a minor bug in the plotting code. We have re-run the bootstrap and included the corrected figure 5 in the attached PDF.\n\n2. There are cases where a practitioner would be okay with increased sensitivity, especially if there are even larger increases in performance. The question highlights the benefit of studying the performance-sensitivity relationship of algorithms on a 2d plane, such as in Figure 4\n\n3. Because our proposed metrics are nonstandard, we wanted to avoid statistical tests that use distributional assumptions (e.g., student-t assumes normality). This is why we chose to use percentile bootstrap. However, percentile bootstrap CIs are often quite wide; therefore, we chose to run many seeds to obtain useful CI estimates. There is a long line of work suggesting the current practice of fewer than 10 seeds is not enough: \nhttps://arxiv.org/abs/2304.01315, \nhttps://arxiv.org/abs/1904.06979, \nhttps://openreview.net/pdf?id=Xe7n2ZqpBP, \nhttps://rlj.cs.umass.edu/2024/papers/RLJ_RLC_2024_330.pdf\n\n\n## Limitations:\n1. The computed sensitivity metric is intimately tied to the distribution of environments used, the process for choosing hyperparameter values, the number of seeds, etc.  An underlying assumption is that a practitioner cares about evaluating algorithms with respect to some distribution of environments. If the practitioner is consistent in their data collection process, these metrics can be used. As argued above, respectfully, we feel the ranges used were appropriate given the algorithmic variations under study.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W3kB372Fmu", "reviewer": "Reviewer_aSZK", "summary": "The paper proposes a method to analyse how sensitive are RL methods with respect to hyperparameter tuning. The author argue that one method may perform well on average but require more HPO tuning per task which hides some computation and prevent from having comparable results. They introduce a sensitivity metric which measures the difference between the best hyperparameter tuned per task and the performance of the best hyperparameter on average. Then, they propose a quadrant analysis where both the sensitivity and the performance of a tuned score are displayed which allows to compare algorithms on those two dimensions. Experiments are then performed on PPO normalization variants where they show that current normalization have different trade-offs: some approaches improves the scores but increase hyperparameter sensitivity while other conversely lower scores and sensitivity but no approach allows to improves both scores and sensitivity currently. Finally, the authors study how many hyperparameters require tuning while having the other fix and while keeping 95% of the best performance.", "questions": "Here are points that would be important for me to raise my score:\n\n* In figure 4, are the results stable if you leave one out one environment? If you compute the plot 5 times each time leaving one of the environment, do you have similar plots? If not, then the analysis on the paper will be less warranted (that such and such method is more stable with hyperparameter).\n* Will the paper release the dataset of evaluation in addition to the code? (the code would have limited interest compared to the data) I would highly encourage to share the dataset, possibly with a script to reproduce some of the paper figures (for instance as was done by https://github.com/google-research/rliable) The dataset would be also very useful to simulate and compare HPO methods, in particular if it contains the evaluation per iteration (and per 100s of iterations for instance).\n\n* Figure 4: there are very large outliers for the obs zero mean normalization sensitivity and the distribution is very skewed, could you explain why they happen? \n\n* One popular method is f-Anova to study hyperparameter importance, could you include an analysis using it? It would be useful for practitioners to know which HP are most useful.\n\n* In the limitation section, you mention that the results may change under a different normalization (say CDF instead of min-max). This is indeed an important point, could you report the result for Fig4 as well with CDF? I assume it should be a minor change as it is just changing the metric. However, it seems important to assess how much the method would be impacted by different normalization (even if the results change, the paper would still be valuable)\n\n\nHere are points that are more details:\n- You have l110 p(w, e, h ,\\kappa) and then p(a, e, h), it would be nice to unify the notation\n- l151: I think it would be useful to precise that the quantity is always >= 0\n- \"The shaded region is a 95% Student t-distribution confidence interval around the mean return over 200 runs \"\n=> what is the CI not covering the mean in walker 2D?", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* The paper is very well motivated. It tackles an important problem for RL (and in general on how to account for HPO sensitivity when reporting results.\n* The paper is very well written and easy to follow, the methods are well described and the experiment are very sound\n* The paper would provide a valuable contribution if the results of the runs are released (will they?, see my question)", "weaknesses": "The paper is currently missing some analysis regarding the method stability (see questions). For instance how much the proposed method would be stable and reliable with respect to different environments (does adding one environment change the results completely?) or normalization (e.g. using CDF instead of min max).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, thank you for your review and comments on improving our work. We will address your specific questions here and focus on general concerns shared between the reviews in our general rebuttal.\n\n\n## Weaknesses \nAs requested, we have included 5 repeats of the Figure 4 plots, leaving out an environment each time in the PDF attached to the general rebuttal. In addition, another reviewer asked us to demonstrate results with (p,1-p) percentile normalization instead of min-max; we will perform this and include these results. \n\n## Questions\n1. We have included a figure in the PDF attached to the general rebuttal that does this. Leaving out environments does shift the reference point, and the position of the variants shifts somewhat relative to the reference point. However, the regional position of the variants is mostly consistent (e.g., observation ZM normalization is still more performative and sensitive than the reference point).\n\n2. Yes, we will do this. The code for reproducing the plots was included in the supplementary material, but it is somewhat useless without the data. We will look into providing the data through Google Cloud or some other platform.\n\n3. Digging deeper into this, we found a minor error in the plotting code. After re-running experiments, we found that it did not impact the main message of our paper or the vast majority of the particular empirical outcomes. The corrected Figure 4 will be included in the PDF for the general rebuttal.\n\n4. The focus of our study has been characterizing algorithms’ performance across sets of environments with respect to their hyperparameters. The work we have seen that uses f-Anova has focused on characterizing the importance of hyperparameters within an environment, an important but slightly tangential goal. We could report the hyperparameters that were most impactful to tune per environment (measured when creating Figure 5) and report how the important hyperparameters and their values change when tuning pairs of hyperparameters jointly, etc. Do you believe this would be valuable?\n5. Another reviewer suggested normalizing by a (p,1-p) percentile range instead of min-max, which has a lower variance. We will include this normalization and compare it with min-max in the final version. \nAs for CDF normalization, after further consideration, we are unsure if it makes sense to visualize it in the style of Figure 5. Taking an expectation through a non-linear operator does not preserve orderings.  We normalize the AUCs of runs (seeds) and then average across runs to obtain the expected normalized performance. The interpretations of regions in Fig 4 may have very different meanings under CDF normalization. An algorithm could have a worse average return but be higher on the CDF normalized Figure 5 plot. E.g. some seeds perform slightly better in units of return, but are placed much higher in CDF from the nonlinearity. Nevertheless, it is a minor change and we can include results with CDF normalization in the appendix if requested.\n\n## Detailed comments\n1. Thank you for catching the notation error. It will be fixed for the camera-ready.\n2. We will clarify this. Another reviewer suggested noting the connection to Jesen’s inequality. This will be clarified for the camera-ready version.\n3. Thank you for catching this. After re-running the code and examining it, the confidence interval fully covers the mean. We suspect that a post-processing formatting issue introduced this error in the document. The corrected version is included in the general rebuttal PDF, and the figure will be corrected for the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7IZFuQLbRf", "reviewer": "Reviewer_ydma", "summary": "The authors define a environment-parameter sensitivity metric in terms of a Jensen gap of min-max scaled performance. The gap gives us the difference between the best average performance and the average best performance over a particular algorithms hyperparameters over a set of environments. The authors then provide a 2D visualization where they plot this sensitivity gap against the average best-performance. This gives an intuitive way to classify whether a change in parametersensitivity can be outweighed by its (positive) change in performance.\n\nThe authors run experiments for PPO on brax to illustrate their method.\n\nAt the end, the authors also use their performance metric to tune subsets of the total parameter set.", "questions": "- Could the authors change the bound calculation (e.g., IQR or something similar), this should be an easy local change, and add the best-performing run metric (i.e., have the AUC and the best-run per algorithm side-by-side).\n- Could the authors improve on the discussion and conclusions of the results, discussing the characteristics of the environment and how the choice of parameters hypothetically covaries with this.", "limitations": "Section 6 shortly discuss limitations, however, the actual limitations are discussed in appendix D. The shallow experimental setup that was performed resulted in 7 GPU years on modern NVIDIA GPUs. Since most code was implemented in Jax and the algorithm used was PPO, one of the most light-weight RL algorithms currently out there, the current framework does not offer much headroom for testing other methods. \n\nThe severe computation costs are not included in the broader impact statement. The authors do state that reducing carbon footprint of DRL experiments are important, however, the presented methodology is not carbon friendly. If, the authors presented a better experimental setup, then this would alleviate the need for other practitioners from running these types of sensitivity analyses for PPO. However, this is currently not the case. Another way to improve the current setup is to use variance reduction methods in some way to reduce the immense compute requirements.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The definition for the environmnent-dependent parameter sensitivity is intuitive through its use of Jensen's gap. I agree that this is a sound method to evaluate robustness of parameter-tuning. Although, care must be taken when choosing the environment-sets.\n\nFigure 3 and the discussion in 3.2 is excellently presented, and gives an intuitive way to think about what patterns we would like to see when performing parameter ablations.\n\nThe experimental setup is repeated over many repetitions and use appropriate statistical methods, when discussing the results, the authors discuss overlapping intervals and are decently careful about making conclusions.", "weaknesses": "### Summary\nAlthough this paper shows a nice way to visualize robustness of parameter optimization methods across sets of environments, the severe computation requirements needed to create these visualizations on top of the shallow experimental setup that the authors performed, makes it unlikely that the current results will prove useful to other practitioners (aside from the isolated case applying PPO to Brax). \n\nThe chosen metric for performance (AUC) is incomplete when considering how RL is often applied (when only caring about final performance), and arguably estimated using a non-robust bound estimator. Furthermore, the authors only test on 5 Brax mujoco environments when comparing extensions that originate from Atari, DMLab, Minecraft or elsewhere. The authors, do not take characteristics of the environments they test on into account when discussing results. As a consequence, the conclusions we can take from the presented results are quite limited.\n\nTherefore I vote for reject.\n\n### Major comments\n - The single choice of AUC metric is poorly motivated, although it is a relevant metric, in RL we are also often only interested in **final performance**. So, the AUC gives a proxy for cumulative regret, I also want to see a proxy for simple regret, the best-run within a algorithm's training cycle.\n - The authors argue that we need robust estimators of performance for hyperparameter sensitivity. So, I am worried about the choice for min-max normalization of the algorithms' performance, see Eq. 1). These can be extremely high-variance estimators for the performance bounds, and therefore, aren't really robust (perhaps explaining why the authors needed 200 seeds per ablation)... I understand that this always nicely scales results into $[0, 1]$, however, if we rerun the experiments (different seeds and parameter-sets), would Figures 1 and 2 give similar patterns? Would this change our conclusion about what choice of hyperparameter is better in terms of environment robustness? Why not choose for an interquantile range or even a $[p, 1-p]$ percentile range that has guaranteed lower asymptotic variance? \n - The authors don't discuss the sensitivity/ robustness to the choice of environment sets for Eq.2). I.e., what characteristics do the environments have that we test robustness for. For this reason, I also think that the discussion in 4.3 is a bit shallow considering the choice of Mujoco environments. The rewards in Brax are relatively well scaled and behaved (at least compared to e.g., a game like 2048 which has exponential rewards), so concluding that the value target symlog is not beneficial (on top of my comment about min-max scaling) should be nuanced and discussed appropriately. \n - It's not clear from Eq.5) (and the whole of section 5) how we decide what reference point to take when deciding which parameters to tune. I.e., when we choose to keep some parameters fixed, what should their values be? Are the defaults by definition robust or not?\n\n\n### Technical comments\n - Definition of normalized environment score $\\Gamma$ is wrong, why is the action-space $A$ used here? (see section 2). Why is the symbol for environment sets not consistent, why use $E$ instead of $\\mathcal{E}$. In section 5 suddenly $\\Omega$ is used again in place of $A$.\n\n\n### Minor comments\n - Eq. 2) Why not write the sum as an expectation, $\\mathbb{E}[\\max_h \\Gamma(...) | e \\sim {p(e)}] - \\max_h \\mathbb{E}_{p(e)} \\Gamma(...)$. I.e., the more recognizable Jensen gap.\n - Shortly discuss expected behaviour of Eq.2 and what it means, i.e., a value of 1 means a large gap, meaning not robust, and a value of 0 means no gap, meaning very robust.\n - Line 220, minibatch advantage normalization is not that important for PPO, even the blog-post that the authors cite here says so.\n - What ranges for the parameters were chosen in section 4.1 ? It's easy to break an algorithm through a bad or uninformed choice for the learning rates... It's also not really clear from the text how the authors created the algorithm configurations to test, i.e., random-search? an exhaustive grid? (Found this in Appendix D, the ranges for the values are OK when speaking from experience, and the authors used exhaustive grid-search; this **must** be included in the main text)\n - Line 291, \"It may ... be gained.\" This sentence is broken, maybe rephrase: \"By finetuning a few important parameters for each specific environment, you can unlock most of an algorithm's best performance.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review and comments on improving our work.\n\n\n## Major comments:\n\n1. Several performance metrics are common in reinforcement learning: best policy performance, performance during final k episodes, AUC, etc. We chose AUC because it informs us about the rate of learning. For example, in continual RL, the goal is not the final policy—there is none—and the main interest is learning, not the outcome of learning. Many applications of RL are similar, for example, in data-center cooling, learning happens in deployment. Another empiricist could care about final policy performance and measure sensitivity to that. We can add sensitivity analysis that uses final performance as a metric to the appendix for the final version. \n\n2. This is a valid question regarding the method's stability regarding the environment set and score normalization. We used the statistical bootstrap specifically to represent the variance in min-max estimation.  But, normalizing by a (p,1-p) percentile range may still be a better choice; we will add results with a (p,1-p) normalization. Thank you for the suggestion. \n\n3. Different environment sets may produce different results. Min-max normalization, in particular, may be inappropriate in environments with exponential rewards scaling. In such cases, another choice of normalization may be preferable. \nRegarding value target symlog’s benefit, our results and conclusions are indeed tied to the Mujoco environments we chose. We tried to avoid making general conclusions in the paper about the algorithms beyond the empirical setting that we tested in. If there is a sentence that is particularly problematic concerning this, please let us know. This is missing the important point that even in one common environment suite, sensitivity emerges and is a big deal. We chose PPO and Brax Mujoco to demonstrate that even with one of the most heavily studied algorithms in one of the most used environment suites, these definitions can provide insight. \nAs for “What characteristics do the environments have that we test robustness for” is a massive question—a paper all on its own—and we would argue that RL researchers have made little progress on this important but separate question.\n\n4. In section 5, When leaving hyperparameters fixed, we set them to the values that maximize cross-environment performance (i.e. the default). The intuition behind the definition of effective hyperparameter dimensionality (and thus all of section 5) is trying to measure how many hyperparameters are “robust” and able to be left to default. Indeed, another huge challenge in RL: how do we establish sensible defaults. \n\n## Technical comment\n1. You are correct; this is wrong. This typo resulted from a change of notation for the algorithm set. This will be fixed for the camera-ready version.\n\n## Minor Comments\n1.  Thank you for this suggestion. The notation change and noting the definition as a Jensen gap would be a nice improvement. We will change for the camera-ready version. \n2. We can add a sentence or two of extra interpretation of the ranges of the sensitivity gap in the discussion. Thank you for the suggestion. \n3. The blog post states that minibatch advantage normalization does not have much effect on performance. To the best of our knowledge, its relevance for sensitivity has not been well-studied. However, some intuition suggests that it may be important for making the entropy coefficient less sensitive. This was a stated [motivation](https://arxiv.org/pdf/2301.04104#page=6) for the (albeit different) form of advantage normalization performed in DreamerV3.  \n4. Thank you for letting us know this was a point of confusion. We will add to the main text that a grid search was performed.\n5. Thank you for your suggested edit!\n\n## Questions\n1. You are correct that using IQR to normalize performance will be a lower variance estimate. We agree it may be more appropriate, especially when practitioners do not wish to run 200 seed experiments. We will add results that use IQR normalization. \nWe have used the term “run” synonymously with seed. Given that the goal of RL is to maximize expected return, the best seed will not be an informative alternative to mean performance across seeds.  We assume that you meant something different with your use of the word run. Perhaps you meant final policy performance, as mentioned before? If so, we would happily include the results with the final performance in the appendix.  \n\n2. Yes, we can improve the discussion and conclusions. We will note in the discussion how the Mujoco environments have similar observations, reward scales, etc., and our results do not generalize to other classes of environments. Moreover, we will emphasize that PPO and Mujoco were chosen because they are heavily studied, and more importantly, they allowed us to demonstrate the metrics can still provide insight in these domains.\n\n## Limitations\nThe presented methodology is:\n1. Collect data about the performance of different algorithms concerning their hyperparameters across different environments\n2. Normalize performance (via min-max, CDF, or some other way)\n3. Using the data that has already been collected and normalized to compute sensitivity and effective hyperparameter dimensionality as we defined. \n\nWe are agnostic to how practitioners collect their data. Grid search will not scale to larger hyperparameter spaces, so something like a random search may be required and fits the proposed methodology.  At the same time, please note that RL is incredibly noisy.  There are many cases where studies with few seeds have greatly misled the community: https://arxiv.org/pdf/2304.01315\n Statistical estimates require many samples; as you correctly noted, we use min and max for normalization, which are high variance. It made sense to use the available computation that we had to run more seeds instead of additional environments, algorithms, etc.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you to the reviewers for your questions, comments, and suggestions on improving our work! We appreciate your feedback. We have addressed specific comments in the individual rebuttals and will provide a general response to comments shared among the reviewers here.

The main contributions of our paper are the hyperparameter sensitivity and effective hyperparameter dimensionality metrics and the plots for visualizing them. There is confusion that because our study performed many runs with a grid search this is what we are advocating for. Our methodology does not require one to do a grid search, 200 seeds, AUC performance metrics, min-max score normalization, or to perform a percentile bootstrap. The hyperparameter sensitivity and effective hyperparameter dimensionality metrics are agnostic to these choices. If a practitioner made a different empirical design choice than us on one of these decision points, then they may very well see different results; what is most important is that they are consistent and fair when comparing algorithms in their study. For example, another empiricist could do a random search with 10 seeds per hyper-parameter setting, consider final policy performance, use a different score normalization technique, report standard errors, and still use the definitions and plots that we provide. 

Several reviewers correctly noted that the sensitivity metric is intimately tied to the environment set and that our findings are thus limited to Brax Mujoco environments tested. We acknowledge this. No matter how large of an environment set is chosen, this will always be true; regardless of whether we include Atari, Minecraft, DMLab, or any other suite of environments, empirical results will not allow us to make claims about environments outside the evaluated environment distribution. What is being missed here is an important finding that even when there is only one common environment suite, sensitivity emerges and is a big deal. Limiting ourselves to Mujoco was a feature, not a bug of our study—notwithstanding the fact that these Mujoco environments are widely used to rank and evaluate new algorithms.

Several questions were raised regarding various empirical design choices. We addressed each question in the per-review rebuttals; however, we will also summarize our justifications for them here.

 The 200 seed bootstrap CI was performed for the following reasons: min-max estimation is high variance, the sensitivity metrics we propose are nonstandard, we did not wish to rely on confidence intervals that make distributional assumptions, there are many forms of stochasticity in RL, and existing literature has demonstrated that studies with small numbers of seeds can be misleading. We dug deeper based on the interesting questions from the reviewers. It revealed a minor error in the bootstrap CI code that did not impact the main messages of the paper or the vast majority of the particular empirical outcomes. The good news is that after running our experiments again with the bug fixed, two things that did change helped answer two of the questions from the reviewers regarding the size of the confidence intervals.  Please see the attached PDF for the corrected Figure 4. 

As for the grid search, we believe the values chosen are appropriate based on existing literature:  https://arxiv.org/pdf/2306.01324, https://arxiv.org/pdf/2407.18840, and the argument presented in our rebuttal to reviewer YuTN.

The performance metric AUC was chosen as it captures the agent’s rate of learning. Additional studies using other performance metrics may be valuable. We will add results with the final policy performance used as a metric to the appendix as requested by reviewer ydma. 

The choice of min-max normalization is indeed high variance. An alternative (suggested by ydma) would have been to normalize by a (p,1-p) percentile range (such as IQR). We will include results with this type of normalization.


Finally, reviewers noted some notation typos, sentence errors, and suggestions for added discussion. Thank you very much for pointing them out! We will ensure that all of these edits are made for the camera-ready submission. We appreciate your questions, comments, and criticisms for improving this research.

Meta-review: This paper tackles a very important problem in RL: how do we know when a newly proposed approach leads to generalizable improvements. The reviewers agree that it is very well-motivated and the proposed metrics are simple and intuitive. However, they have concerns about the robustness of the results due to the selection of environments only from Mujoco, a fixed set of hyperparameters, and the use of the min/max in the environment normalization score. Nevertheless, I feel that the proposed framework is valuable and can potentially impactfully improve evaluation in RL.
Decision: Accept (poster)