A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning

Authors: jacob  adkins
michael  bowling
adam  white
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 05:29:26

Abstract: The performance of modern reinforcement learning algorithms critically relies
on tuning ever increasing numbers of hyperparameters. Often, small changes in
a hyperparameter can lead to drastic changes in performance, and different environments require very different hyperparameter settings to achieve state-of-the-art
performance reported in the literature. We currently lack a scalable and widely
accepted approach to characterizing these complex interactions. This work proposes a new empirical methodology for studying, comparing, and quantifying the
sensitivity of an algorithm’s performance to hyperparameter tuning for a given set
of environments. We then demonstrate the utility of this methodology by assessing
the hyperparameter sensitivity of several commonly used normalization variants of
PPO. The results suggest that several algorithmic performance improvements may,
in fact, be a result of an increased reliance on hyperparameter tuning.

Main paper: 
 introduction: 
 The performance of reinforcement learning algorithms critically relies on the tuning of numerous hyperparameters. With the introduction of each new algorithm, the number of these critical hyperparameters continues to grow. Consider the progression of value-based reinforcement learning algorithms, starting from DQN (Mnih et al., 2015) , which has 16 hyperparameters that the practitioner must choose, to Rainbow (Hessel et al., 2018) with 25 hyperparameters. This increase can be observed in Figure 1 . This proliferation is problematic because performance can vary drastically with respect to hyperparameters across environments. Often, small changes in a hyperparameter can lead to drastic changes in performance, and different environments require very different hyperparameter settings to achieve the reported good performances (Franke et al., 2021; Eimer et al., 2022 Eimer et al., , 2023;; Patterson et al., 2024) . Generally speaking, hyperparameter tuning requires a combinatorial search and thus many published results are based on a mix of default hyperparameter settings and informal hand-tuning of key hyperparameters like the learning rate. Our standard evaluation methodologies do not reflect the sensitivity of performance to hyperparameter choices, and this is compounded by a lack of suitable metrics to characterize said sensitivities.There are many different ways one could characterize performance with respect to hyperparameter choices in reinforcement learning, but the community lacks an agreed standard. Hyperparameter 38th Conference on Neural Information Processing Systems (NeurIPS 2024) . Figure 1 : A count of hyperparameters for different reinforcement learning algorithms proposed over the last decade. We include value-based, policy-gradient, and model-based methods. The counts do not include hyperparameters controlling the network architectures, such as number of layers, activation functions, etc. See Appendix B for details on how hyperparameters were counted.sensitivity curves, such as those found in the introductory textbook (Sutton & Barto, 2018) , summarize performance with respect to several values of a key hyperparameter producing U-shaped curves. Sensitivity curves do not work well with many dimensions of hyperparameters, nor is there a wellestablished way to use them to summarize performance in multiple environments. If computation is of no object, then performance percentiles can be used to compute the likelihood that an algorithm will perform well if its hyperparameters are randomly sampled from some distribution (Jordan et al., 2020) . Although very general, this approach does not reflect how practitioners tune their algorithms. The Cross-environment Hyperparameter Benchmark (Patterson et al., 2024) compares algorithms by a mean normalized performance score across environments but ultimately focuses on the possibility of finding a single good setting of an algorithm's hyperparameters that performs well rather than characterizing sensitivity. What performance-only metrics lack is a measurement of what proportion of realized performance is due to per-environment hyperparameter tuning.We propose an empirical methodology to better understand the interplay between hyperparameter tuning and an reinforcement learning agent's performance. Our methodology consists of two metrics and graphical techniques for studying them. The first metric, called an algorithm's hyperparameter sensitivity, measures the degree to which an algorithm's peak reported performance relies upon perenvironment hyperparameter tuning. This metric captures the degree to which per-environment tuning improves performance relative to the performance of the best-fixed hyperparameter setting across a distribution of environments. The second metric, named effective hyperparameter dimensionality, measures how many hyperparameters must be tuned to achieve near-peak performance. It is often unclear how important specific hyperparameters are and if they should be included in the tuning process. These two metrics can help us better understand existing algorithms and drive research toward algorithmic improvements that reduce hyperparameter sensitivity.We validate the utility of our methodology by studying several variants of PPO (Schulman et al., 2017) that have been purported to reduce hyperparameter sensitivity and increase performance. We performed a large-scale hyperparameter study over variants of the PPO algorithm consisting of over 4.3 million runs (13 trillion environment steps) in the Brax MuJoCo domains (Freeman et al., 2021) . We investigate the relationship between performance and hyperparameter sensitivity with several commonly used normalization variants paired with PPO. We found that normalization variants, which increased PPO's tuned performance, also increased sensitivity. Other normalization variants had negligible effects on performance and marginal effects on hyperparameter sensitivity. This result contrasts the view that normalization makes reinforcement learning algorithms easier to tune and, as a consequence, results in improved performance.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The contribution of this paper is an empirical methodology for assessing hyperparameter sensitivity. The presented experiments provide evidence for the utility of the methodology, as stated in the introduction and abstract. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that the paper does not attain these goals. 
problem setting and notation: 
 We formalize the agent-environment interaction as a Markov Decision Process (MDP) with finite state space S and action space A, bounded reward function R: S × A × S → R ⊂ R, transition function P : S × A × S → [0, 1], and discount factor γ ∈ [0, 1]. At each timestep t, the agent observes the state S t , selects an action A t , the environment outputs a scalar reward R t+1 and transitions to a new state S t+1 . The agent's goal is to find a policy, π : A × S → [0, 1], that maximizes the expected return, G t . = R t+1 + γR t+2 + ..., in all states: E π [G t |S t = s] for all s ∈ S.Most reinforcement learning agents learn and use approximate value functions in order to improve the policy through interaction with the world. The state-value function v π : S → R is the state conditioned expected return following policy π defined as v π (s). = E π [G t |S t = s].Similarly, the action-value function q π : S × A → R provides the state and action conditioned expected return following policy π defined by q π (s, a).= E π [G t |S t = s, A t = a]. The advantage function A π (s, a) .= q π (s, a) -v π (s) describes how much better taking an action a in state s is rather than sampling an action according to π(•|s) and following policy π afterward. Given an estimate of the value function v, an agent can update estimates of the value of states based on estimates of the values of successor states. An n-step return is defined asG t:t+n . = R t+1 + γR t+2 + • • • + γ n-1 R t+n + γ n v (S t+n ).A mixture of n-step returns, called a truncated-λ return can be created by weighting n-step returns by a factor λ ∈[0, 1], G λ t:t+n . = (1 -λ) n-1 j=1 λ j-1 G t:t+j + λ n-1 G t:t+n .Truncated-λ returns are often used in reinforcement learning algorithms such as PPO (Schulman et al., 2017) to estimate state values and state-action advantages, which are used to approximate the value function and improve the policy.Of particular interest to this work is the setting where an empiricist is evaluating a set of algorithms Ω across a distribution of environments E. Each algorithm ω ∈ Ω is presumed to have some number of hyperparameters n(ω). Each hyperparameter h i , 1 ≤ i ≤ n(ω) is chosen from some set of choices H ω i . The total hyperparameter space H ω is defined as a Cartesian product of those choicesH ω . = H ω 1 × H ω 2 × ...H ω n(ω). Once an algorithm ω ∈ Ω and a hyperparameter setting h ∈ H ω are chosen, the tuple (ω, h) specifies an agent.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: See that there is a section titled "Limitations and Future Work" which discusses limitations.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
hyperparameter sensitivity: 
 First, we must define what we mean by sensitivity. A sensitive algorithm is an algorithm that requires a great deal of per-environment hyperparameter tuning to obtain high performance. Conversely, an insensitive algorithm is one where there exist hyperparameter settings such that the algorithm can obtain high-performance across a distribution of environments with fixed hyperparameters. This section presents two contributions: a metric for assessing an algorithm's hyperparameter sensitivity, and a method of graphically analyzing the relationship between hyperparameter sensitivity and performance along a 2-dimensional plane. These tools may be used to develop a deeper understanding of existing algorithms and we hope will aid researchers in evaluating algorithms more holistically along dimensions other than just benchmark performance.We want a performance metric that summarizes the learning of an online reinforcement learning agent. The natural choice of performance metric is to report the average return obtained during learning, which we call the area under the (learning) curve (AUC). The AUC on a run is denoted by p(ω, e, h, κ) where ω ∈ Ω is an algorithm, e ∈ E is an environment, h ∈ H ω is a hyperparameter setting, and κ ∈ K ⊂ N is the random number generator (RNG) seed. Performance observed during a run of a reinforcement learning agent depends on many factors: the reinforcement learning algorithm, the environment, the hyperparameter setting, and many forms of stochasticity. Even after fixing the algorithm, environment, and hyperparameter setting, performance distributions are often skewed and multi-modal (Patterson et al., 2023) . Therefore, many runs are required to obtain accurate estimates of expected performance p(ω, e, h) . = 1 |K| κ∈K p(ω, e, h, κ) where K ⊂ N is the set of RNG seeds used during the experiment. In the experiments presented in this paper, we perform 200 runs averaging performance over a subset of runs, after filtering (described later), and report 95% bootstrap confidence intervals around computed statistics.It is crucial to capture performance across sets of environments in order to compute sensitivity. Recall, that our notion of sensitivity captures the degree to which an algorithm relies on per-environment hyperparameter tuning for its reported performance gains. Our choice of AUC as a performance metric does not allow for cross-environment performance comparisons directly because the magnitudes of returns vary greatly between environments. Consider the distributions of performance presented in the left plot in Figure 2 . The performance realized by good hyperparameter settings in Halfcheetah is orders of magnitude greater than the performance of good hyperparameter settings in Swimmer. Nevertheless, just because the absolute magnitude is lower or the range of observed performances is tighter, that does not mean the differences are any less significant. Thus, in order to consider how hyperparameters perform across sets of environments, we need to normalize performance to a standardized score.In this work, we use [5, 95] percentile normalization. We choose percentile normalization as it has a lower variance than alternatives like min-max normalization. Other normalization methods, such as min-max or CDF normalization (Jordan et al., 2020) , could also be used with our hyperparameter sensitivity formulation. After conducting a large number of runs across different algorithms, environments, and hyperparameter settings, for each environment e, we find the 5th percentile p 5 (e) and 95th percentile p 95 (e) of the distribution of observed performance in e. Then, for each algorithm, environment, and hyperparameter setting, the normalized environment score is obtained by squashing performance:EQUATIONNote the right hand side of Figure 2 , the distributions of normalized scores for hyperparameter settings in Swimmer and Halfcheetah now lie in a common range. Normalized scores allow practitioners to determine which fixed hyperparameter settings do well across multiple environments. That is, a practitioner can find the hyperparameter setting that maximizes the mean normalized score across a distribution of environments. Consider the performance of the hyperparameter setting denoted by the blue stars in Figure 3 . This setting performs in the top quartile of hyperparameter settings in both Swimmer and Halfcheetah. In contrast, consider the hyperparameter setting denoted by the red stars. While this hyperparameter setting sits near the top of the distribution for Halfcheetah, it performs poorly in Swimmer.Given an algorithm ω ∈ Ω, we define its hyperparameter sensitivity Φ as follows: The hyperparameter sensitivity of an algorithm is the difference between its per-environment tuned score and its cross-environment tuned score. The per-environment tuned score is the average normalized environment score with hyperparameters tuned per environment. The cross-environment tuned score is the normalized environment score of the best fixed hyperparameter setting across the distribution of environments. We can use this notion of hyperparameter sensitivity to better understand new and existing algorithms. Reporting both hyperparameter sensitivity and the conventional performance-only evaluation metrics should provide a more complete picture of algorithm performance.EQUATIONModern reinforcement learning algorithms are complex learning systems, and understanding them is a difficult task that requires multiple dimensions of analysis. Benchmark performance has been the primary metric (and often the only one) used for evaluating algorithms. However, this is only one dimension along which algorithms can be evaluated. Hyperparameter sensitivity is an important dimension to consider in the evaluation space, especially as practitioners begin to apply reinforcement learning algorithms to real-world applications. We propose the performance-sensitivity plane to aid in better understanding algorithms.Consider the performance-sensitivity plane shown in Figure 4 . To construct the plane, the center point is set to the hyperparameter sensitivity and per-environment tuned score of some reference point algorithm. We can consider how other algorithms relate to this reference point by considering which region of the plane they occupy. There are 5 regions of interest shaded by different colors and labeled numerically, which we will consider in turn.An ideal algorithm would be both more performative and less sensitive. Therefore, algorithms that fall in Region 1 (the top left quadrant) of the plane would be a strict improvement over the reference point algorithm. For some applications, perhaps additional sensitivity can be tolerated if the gains in performance are large enough. Algorithms that fall in Region 2 are an example of this. The region represents algorithms whose increase in performance is greater than the corresponding increase in sensitivity. Conversely, for some applications sensitivity may matter a great deal and some performance loss can be endured. Algorithms that fall in Region 3 are an example of those whose decrease in sensitivity outmatches their corresponding decrease in performance. Regions 1-3 represent algorithms that have notable redeeming qualities either in terms of performance, hyperparameter sensitivity, or both. However, perhaps a practitioner does not care about sensitivity.For example, they want to maximize the score of a specific benchmark, and hyperparameter tuning is no issue. Algorithms in Region 4 may be adequate as they are algorithms that exhibit performance improvements and an even higher reliance upon per-environment hyperparameter tuning. Finally, those unfortunate algorithms that live in Region 5 are in a space with both lower performance and higher sensitivity, making them undesirable.A natural application of this diagram is to set the reference (center) point to the hyperparameter sensitivity and performance of some base algorithm and study how proposed modifications (or ablations) affect both sensitivity and performance. Often, new algorithms are created by modifying existing algorithms, such as normalizing targets, adding a regularization term to the loss function, gradient clipping, etc. We illustrate an example of this using PPO as a reference point in the next section. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: Code is included in supplementary material. Instructions are in README file.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
sensitivity experiments: 
 To illustrate the utility of the sensitivity analysis presented above, we performed an experiment to study the hyperparameter sensitivity and performance of several variants of the PPO algorithm, a widely used policy-gradient method in reinforcement learning (Schulman et al., 2017) . We considered several normalization variants commonly used in PPO implementations (Andrychowicz et al., 2020; Huang et al., 2022) and some normalization variants, introduced in DreamerV3, that were purported to reduce hyperparameter sensitivity (Hafner et al., 2023; Sullivan et al., 2023) .PPO is an instance of an actor-critic method (Sutton & Barto, 2018) that maintains two neural networks: a policy network with parameters θ and a value network with parameters w. Given a policy π θ old , the agent performs a roll-out of T steps, (S 1 , A 1 , R 1 , S 2 , A 2 , R 2 , ...S T , A T , R T ), this rollout is then split into m batches of length k. The critic is then updated via ADAM (Kingma & Ba, 2015) to optimize MSE loss with the truncated-λ return as a target. The actor is updated via ADAM (Kingma & Ba, 2015) to optimize a clipped surrogate objective toward maximizing the expected return. An entropy regularizer is added to the actor loss to encourage exploration.Several normalization variants have been used in PPO implementations. We focus on three categories of normalization: observation normalization, value function normalization, and advantage normalization. An intuition behind value function or advantage normalization for hyperparameter sensitivity is that the scale and sparsity of rewards vary greatly across environments and that value function or advantage normalization should make actor-critic updates invariant to these factors possibly requiring less tuning of the step-size hyperparameters (van Hasselt et al., 2016) . Another claimed benefit of advantage normalization variants is that by normalizing the advantage term, it is easier to find an appropriate value for the entropy regularizer coefficient τ across a distribution of environments (Hafner et al., 2023) . Observation normalization standardizes the network inputs. This can mitigate large gradients, which may stabilize the learning system for hyperparameter tuning (Hafner et al., 2023; Andrychowicz et al., 2020) , especially critic and actor step-size hyperparameters α w , α θ > 0.Advantage per-minibatch zero-mean normalization: A common implementation detail of PPO is per-minibatch advantage normalization (Huang et al., 2022) . When performing an update, the advantage estimates used in the actor loss function are normalized by subtracting the mean of the advantage estimates in the sampled batch and dividing by the standard deviation of advantage estimates in the sampled batch.Advantage percentile scaling : Another form of advantage normalization was introduced in the DreamerV3 (Hafner et al., 2023) ablations which divides the advantage estimate in the actor loss by a scaling factor. Exponential moving averages are maintained over the 95th and 5th percentiles of advantage estimates. The advantage term is divided by the difference of the two percentiles.Advantage lower bounded percentile scaling: An alternate variant of percentile scaling is used in the DreamerV3 algorithm. Lower bounded percentile scaling applies a max operation to the percentile scaling factor, preventing the estimated advantage term from blowing up if the percentile difference falls below a threshold.Value target symlog: DreamerV3 introduced a method of scaling down the magnitudes of target values by the symlog function. The symlog function and its inverse symexp are defined as:EQUATIONAs in DreamerV3, symlog is applied to the target in the critic loss and symexp is applied to the output of the critic network. In a subsequent study that applied DreamerV3 tricks to PPO (Sullivan et al., 2023) , it was reported that the symlog transformation of the value target was one of the most impactful tricks in environments without reward clipping when applied to PPO.Observation zero-mean normalization: A very common procedure with PPO is to normalize observations by maintaining running estimates of the mean and standard deviation of observations.Observation symlog: DreamerV3 proposed an alternative form of observation normalization by applying the symlog function, compressing the observations.A handful of prior work has investigated the benefits of similar algorithmic modifications to PPO. Previous work has reported the performance impact of the normalization variants commonly used with PPO: per-minibatch zero-mean advantage normalization and zero-mean observation normalization (Andrychowicz et al., 2020) . Other work (Sullivan et al., 2023) investigated how PPO's performance is affected by the normalization variants introduced in DreamerV3 (lower bounded percentile scaling, value target symlog, and symlog observation) and that symlog was especially helpful in Atari when reward clipping is disabled. In addition, this work did not perform any hyperparameter tuning for the variants of PPO they tested. In our results, hyperparameter tuning demonstrated a significant effect on the relative performance of these algorithms.Given these normalization variants, a natural question that arises is how do they affect the hyperparameter sensitivity of PPO? To the best of our knowledge, a careful study of the effect these normalization variants have on the hyperparameter sensitivity of PPO has yet to be done and it provides a good test of our new methodology.We investigated the effect of each of the described normalization variants on PPO. To isolate these effects, we did not apply reward clipping, reward scaling, or observation normalization wrappers by default. We focused our attention on four critical hyperparameters of PPO: the step-size for the critic α w , the step-size for the actor α θ , the coefficient of the entropy regularizer τ , and the truncated-λ return mixing parameter λ. We performed a large grid search spanning five orders of magnitude across five Brax Mujoco domains. Near the extreme endpoints of the grid search, some hyperparameter configurations diverged. We ignored hyperparameter combinations that caused a particular algorithm to diverge over 10% of the time. We averaged the performance over the non-diverging runs.Consider the performance-sensitivity plane in Figure 5 . The reference point at the center is the hyperparameter sensitivity and performance found for PPO without normalization. The error bars displayed indicate 95% confidence intervals formed from a 10,000 sample bootstrap. First, note that none of the normalization variants resulted in an improvement that both raised performance and lowered sensitivity. All forms of advantage normalization increased performance. However, this performance gain comes with a trade-off: increased hyperparameter sensitivity. The marginal gain in performance per unit of increased sensitivity varied between advantage normalization methods.Advantage per-minibatch zero-mean normalization had a greater increase in performance than sensitivity (Region 2). Both percentile scaling-based variants of advantage normalization resulted in more significant sensitivity increases than performance increases (Region 4), indicating an enhanced reliance on hyperparameter optimization methods. Applying the symlog function to the value target lowered performance and may have slightly increased sensitivity (Region 5). It may, however, be the case that the choice of environment distribution did not have enough variation in reward magnitude for the utility of value target symlog to be demonstrated. It appears that observation normalization may slightly reduce sensitivity, although it is unclear due to the width of the confidence intervals. The performance-sensitivity plane provides insights into how algorithmic changes may alter an algorithm's reliance on per-environment hyperparameter tuning with no additional computational expense other than what is already required from a standard hyperparameter tuning procedure.The sensitivity metric is intimately tied to the chosen environment distribution, and our findings are thus limited to the Brax Mujoco environments tested. We argue the restricted environment distribution is not just a practical choice, but was a feature of the study. Somewhat surprisingly we found significant sensitivity across a distribution of fairly similar environments. In Figure 7 displayed in Appendix D, we perform a leave-one-out study reporting how the performance-sensitivity plane changes when each of the five environments is dropped from the data. Observe that while the exact values of the points shift, their position relative to the reference point remains mostly unchanged.The performance-sensitivity plane allows for a richer understanding of algorithms than performanceonly evaluation procedures, but it does not capture the full picture. Two algorithms can sit in the same location on the plane and yet have very different hyperparameter characteristics. Consider the case where there are two algorithms. The first algorithm is highly sensitive with respect to one hyperparameter, which needs to be carefully tuned per environment. The second algorithm has the same sensitivity but needs to be tuned per environment for dozens of hyperparameters with complex interactions. The hyperparameter sensitivity would not differentiate between these two algorithms. An additional metric is needed. 
effective hyperparameter dimensionality: 
 There are many cases where a practitioner can tune some but not all of an algorithm's tunable hyperparameters. It may be the case that if a few key hyperparameters are tuned per environment, then a preponderance of an algorithm's potential performance can be gained. This motivates the definition of effective hyperparameter dimensionality, a metric that measures how many hyperparameters must be tuned in order to obtain near-peak performance. * , ρ(h) = n(ω) i=1 1[h i = h * i ].Effective hyperparameter dimensionality d(ω) is defined asEQUATIONThe effective hyperparameter dimensionality of an algorithm is the maximal number of hyperparameters that can be left to default (setting that maximizes cross-environment tuned performance) while retaining the majority of the performance that can be realized by tuning per environment. The threshold of 95% peak performance can be changed at a practitioner's discretion to whatever meets their performance requirements. To compute effective hyperparameter dimensionality, one needs to consider subsets of hyperparameters to find a minimum subset that achieves the required performance threshold.For the same algorithmic variants of PPO as studied above, Figure 6 displays normalized scores as a function of the number of hyperparameters tuned per environment, choosing the most performant subset to tune. Table 2 in Appendix C provides a listing of the most performant subsets of varying sizes observed during this experiment. The curve interpolates between the normalized scores. The vertical dashed line indicates the point along the curve that reaches 95% of the per-environment tuned score. In the case of advantage percentile scaling, modifying PPO with the normalization variant moves the point to the right, indicating this variant improves performance at the cost of increasing pressure on the number of hyperparameters necessary to tune. Also, note how the performance ranking can shift based on the number of hyperparameters that have been tuned; such as with PPO and the advantage percentile scaling variant (top left plot). For some variants, performance flattens after tuning only three hyperparameters. Whereas for other variants, performance is almost linear in the number of hyperparameters tuned, suggesting sensitivity to all hyperparameters. On the performance-sensitivity plane (Figure 5 in the previous section), both the advantage percentile scaling and advantage lower-bounded percentile scaling variants fall in the same region (Region 4). Yet, in Figure 6 , we can see that the advantage lower bounded percentile scaling variant can obtain higher performance levels than the advantage percentile scaling variant when tuned on smaller subsets of hyperparameters. This observation that algorithms can have similar hyperparameter sensitivities and vastly different effective hyperparameter dimensionalities indicates the power of using both metrics for studying algorithms.6 Limitations and Future WorkThe hyperparameter sensitivity and effective hyperparameter dimensionality metrics will depend heavily on several important empirical design choices. The premise of both metrics is that a practitioner is concerned with understanding sensitivity with respect to a distribution of environments that they care about. If the distribution of environments changes, the metrics will need to be evaluated with respect to the new distribution. This dependence on the environment distribution could be exploited, as someone could artificially make an algorithm appear less sensitive by including several easy environments that all hyperparameter settings will do well in, although score normalization will somewhat counteract this. In addition, the level of granularity with which hyperparameter sweeps are performed will have an effect on both metrics. Another factor that can impact the metrics is the choice of the score normalization method used. A practitioner could use other score normalization methods and the resulting sensitivity scores may be different.A next step is to apply the proposed sensitivity and dimensionality metrics to a larger set of algorithms and environments. Related to this work is the literature of AutoRL (Eimer et al., 2023) . The goal of AutoRL is to tune hyperparameters via some hyperparameter optimization algorithm (which make use of their own hyperparameters-hyper-hyperparameters). Future work could use the definitions provided here to try to understand if the algorithms proposed in the AutoRL literature reduce sensitivity over the base algorithms that are being modified-measuing the sensitivity of the hyperhyperparameters. A study comparing the sensitivities and dimensionalities of AutoRL methods to the sensitivities and dimensionalities of the base learning algorithms they optimize would be prudent. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Experiments are described in the paper. Full experiment details are given in the appendix. Code is provided.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 As learning systems become more complicated, careful empirical practice is critical. Modern reinforcment learning algorithms contain numerous hyperparameters whose interactions and sensitivities are not well understood. Common practice, which is focused on achieving state-of-the-art performance, risks overfitting to benchmark tasks and overly relying on hyperparameter optimization. Most empirical work in reinforcement learning has focused only on evaluating algorithms based on benchmark performance, leaving the effects of hyperparameters under-studied. In this work, we propose a new evaluation methodology based on two metrics that allow practitioners to better understand how an algorithm's performance relates to its hyperparameters. We show how this methodology is useful in evaluating methods purported to mitigate sensitivity. We identify that the studied advantage normalization methods, while improving performance, also increase hyperparameter sensitivity and can increase the number of sensitive hyperparameters.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: The error bars shown in the main paper are 95% bootstrap confidence intervals. The confidence interval around the learning curves in the appendix is a 95% Student tdistribution confidence interval. This is described in the main paper and appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: This is discussed in a section of the appendix titled Broader Impact. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The work presented in this paper poses no such risk. It is an empirical methodology.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We wrote original code for this project. The PPO implementation was forked from PureJaxRL. They are cited throughout the paper, and their license is included in the supplementary material. We thank them and other contributors to the opensource community.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: The contribution of this paper is an empirical methodology. Code for experiments will eventually be opensourced and made available to the community. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Hyperparameter sweeps and per-environment tuning are the most computationally expensive and environmentally impactful parts of reinforcement learning research. Our study ran for approximately 4.5 GPU years on NVIDIA 32GB V100s. While this is substantial, we believe that using compute to better understand the sensitivity of current algorithms is an essential step towards developing more environmentally friendly algorithms. This work investigated an empirical methodology for evaluating the hyperparameter sensitivity of reinforcement learning agents. The immediate societal impact is minimal. However, our methodology may aid in developing performative algorithms with low hyperparameter sensitivity. If this occurs, these algorithms will result in less need for hyperparameter tuning and, as a result, have a positive impact on lowering the carbon footprint of reinforcement learning experiments.There is a trend in which the current state-of-the-art algorithms often contain more hyperparameters than the previous state-of-the-art. The PPO implementation used was heavily inspired by the PureJaxRL PPO implementation (Lu et al., 2022) . The variants advantage per-minibatch zero-mean normalization and observation zero-mean normalization are the standard implementations provided within PureJaxRL. The variants: symlog observation, symlog value target, percentile scaling, and lower bounded percentile scaling closely follow the implementation of the DreamerV3 tricks applied to PPO shown in Sullivan et al. (2023) as well as referencing the original DreamerV3 repository Hafner et al. (2023) .The policy and critic networks were parametrized by fully connected MLP networks, each with two hidden layers of 256 units. The network used the tanh activation function. Separate ADAM optimizers (Kingma & Ba, 2015) were used for training the actor and critic networks. The environments used in the experiments were the Brax implementations of Ant, Halfcheetah, Hopper, Swimmer, and Walker2d.(Freeman et al., 2021) . The hyperparameter sweeps were grid searches over eligibility trace λ ∈ {0.1, 0.3, 0.5, 0.7, 0.9}, entropy regularizer coefficient τ ∈ {0.001, 0.01, 0.1, 1.0, 10.0}, actor step-size α θ ∈ {0.00001, 0.0001, 0.001, 0.01, 0.1}, and critic step-size α w ∈ {0.00001, 0.0001, 0.001, 0.01, 0.1}. Each run lasted for 3M environment steps. 200 runs were performed for each of the algorithms, environments, and hyperparameter settings. Like the PureJaxRL PPO implementation, the entire training loop was implemented to run on GPU. We will release code and experiment data at https://github.com/jadkins99/hyperparameter_ sensitivity, promoting the further investigation of hyperparameter sensitivity in the field of reinforcement learning.The sensitivity metric is ultimately tied to a distribution of environments, and our findings are thus limited to Brax Mujoco environments tested. Claims cannot be about environments outside the evaluated environment set. One may wonder about the stability of the results if the environment distribution changes on a small scale. In Figure 7 , we repeat the sensitivity metric for each of the five environment subsets that can be obtained by dropping a single environment. Leaving out environments does shift the reference point, and the position of the variants shifts somewhat relative to the reference point. However, the regional position of the variants relative to the reference point is mostly consistent.In Figure 8 , we also include results with final performance, i.e., the sum of rewards obtained over the final 1000 timesteps of learning (the truncation length). The performance of reinforcement learning agents does not monotonically increase. In many cases, with specific hyperparameters, it may collapse in the middle or end of learning. Therefore, this metric is noisy, and it is difficult to reason about where algorithms lie according to it on the performance sensitivity curve.  


Decision: This paper tackles a very important problem in RL: how do we know when a newly proposed approach leads to generalizable improvements. The reviewers agree that it is very well-motivated and the proposed metrics are simple and intuitive. However, they have concerns about the robustness of the results due to the selection of environments only from Mujoco, a fixed set of hyperparameters, and the use of the min/max in the environment normalization score. Nevertheless, I feel that the proposed framework is valuable and can potentially impactfully improve evaluation in RL.

Reviews: review: paper_summary: paper_summary: The paper introduces an empirical framework for assessing the hyperparameter sensitivity of reinforcement learning algorithms.
The framework consists of two metrics: 1. hyperparameter sensitivity, which gives a normalized difference in performance between the per-task best hyperparameters and the across-task best hyperparameters. 2. effective hyperparameter dimensionality, which gives the number of hyperparameters that can be left the same as the across-task hyperparameters while tuning the rest and still obtain a threshold say (95%) of the per-task best hyperparameter performance.
Using these metrics in addition to performance metrics, practitioners and researchers can have a better idea of the benefits/downsides of a modification to an algorithm (how much better vs how much more sensitive)
The framework is used to compare several normalization variants of PPO introduced in the past years on the continuous control environments, giving a better picture of their contribution.
strength_weakness: strength_weakness: Metric definition:
- The effective hyperparameter dimensionality depends on the total number of hyperparameters of an algorithm and is likely to scale with it, so this makes it incomparable between algorithms with a different number of hyperparameters. Perhaps counting the number of parameters that changed instead would make it more comparable across algorithms, as is what would dictate the budget of practitioners eventually.


Hyperparameters:
- I would consider the minibatch size and the number of epochs in PPO to be critical hyperparameters as well. It's not clear how the choice of hyperparameters to sweep over was made.
- (minor) The epsilon in the denominator of the minibatch normalization may also play a big role.

Inconsistent/confusion notation:
in line 91, the tuple (w,h) defines an agent (a,), but this is used in a confusing and inconsistent way throughout the paper.
- line 116 $\hat p(a, e, h): if $a$ is there then $h$ is redundant. It seems like $a$ there stands for $w$.
- Equation 1 same. The $a$ should be a $w$.
- Equation 2 and 5:  2 and 5: $\Gamma(a, e, w)$ not it's even more confusing there is both an $a$ and an $w$ and there is an $h$ missing.

Claims:
Line 321 "vastly difference effective hyperparameter dimensionalities": the largest gap observed in the paper is from 2 to 4, so I would not extrapolate here, though arguably that's indeed an additional 2 dimensions to sweep over so scales exponentially.
questions: questions: Reporting the mean under the curve (AUC) with 95% bootstrapped confidence intervals over 200 runs is great, but I would expect some discussion on the use of a confidence interval, as it collapses the shaded area across the 200 runs to the statistic being computed (here the mean AUC).
To me, using a dispersion measure like the variance of the mean AUC would also be valid, as some hyperparameters would have more variance than others (although this could also be seen as an additional dimension of sensitivity).
I would appreciate it if the authors could comment on this.
Also, at what point are the confidence intervals computed? When computing the expected performance or when computing the hyperparameter sensitivity, etc?
limitations: limitations: The authors adequately state the limitations of their work and its broader impacts. In particular the potential limitation of the framework to a specific environment distribution.
paper_summary: paper_summary: This paper proposes a new evaluation regime for reinforcement learning. As opposed to only taking into account benchmark performance (i.e., final return), as is ocmmon in previous literature, this work suggests considering an extra dimension of how sensitive algorithms are to hyperparameters in tuning based on a heuristic developed in the paper. They analyse PPO, and some of its variants, using this new evaluation regime, and find that performance increases often correspond to increased sensitivity to hyperparameters. Finally, the paper considers a top-k approach to hyperparameter tuning, adjusting only the most impactful hyperparameter, to see how the baseline algorithms compare under a more limited hyperparameter tuning regime.
strength_weakness: strength_weakness: - Noting appendix A (the table of how many hyperparameters each algorithm has); the definition of 'hyperparameters' seems pretty weak, and a lot of those included seem to just be design decisions of the actual algorithms.
- I find reading the plots quite confusing, exacerbated by all of the different colours marking each of the areas. I think this obfuscates the message of the plots and as a reader makes it hard to come to conclusions.
- Given this work is based on PurejaxRL, which I think runs on Brax for 1e8 frames, these experiments seem very short (3e6 frames) and possibly doesn't give the algorithms the full opportunity to converge.
- There is very limited discussion of preexisting literature in this space. While I appreciate this takes a subtly different tack (calling for us to measure how sensitive to hyperparameters RL algorithms are, rather than designing algorithms with few hyperparameters), I would expect to see significant more discussion of AutoRL literature. Framing this work better in related work would definitely strengthen the paper.
- The formatting looks quite off with the figures. I think this is because a lot of the captions are to the side of the figures, rather than below.
- Considering the actual hyperparameters being tuned, the results feels slightly disingenuous. If we assume the practitioner running the hyperparameter tuning, we would possibly expect them to select more reasonable values or focus the search in a significantly more targeted way. For instance, it is no surprise that the highly performant methods saw large performance decreases (i.e., hyperparameter sensitivity) when evaluated with entropy coefficients in the range [0.001, 10] or learning rates spanning 5 orders of magnitude. Instead, it would be much more sensible to explore how the performance changes with reasonable hyperparameters that one is more likely to practically tune over (i.e., we generally have a good idea of where to start with values, so are unlikely to try an entropy coefficient of 10).
- It would be good to see some exploration of algorithms which have done something different than just considering the changes to normalisation in PPO returns or observations. It feels like some broad statements are made despite the fact that this analysis was only taken in a single setting.
questions: questions: - In the plots, the authors state that they plot 95% confidence intervals over 200 seeds; but in the y-axis, all of the 'confidence intervals' are only negative. I am not particularly clear why. Is this a mistake, or am I misinterpreting results? At the same time, are the circles representing the mean score over runs, environments, or both; and therefore, are the confidence intervals defined over the runs, the enviroments or a big stack of (in this case) 1000 results per point? Is comparison for the hyperparameter sensitivity done per seed or for the averages over seeds? I think there's a lot of questions about how exactly these results are being presented which doesn't come through in the paper.
- In some cases, increased hyperparameter sensitivity can be a positive - it can give us extra opportunities to boost performance. Do you think this should be mentioned as a limitation of this method, which effectively calls for less sensitivity?
- Why did you use 200 seeds per hyperparameter tuning configuration, which is significantly greater than would be used in most other research? In practice, would it not have been more sensible to have used a much smaller number of seeds (eg 8), and focused on a much tighter range of hyperparameters?
- As a brief aside, the authors have not removed the instructions of the Neurips checklist as requested, which should be done if the camera ready copy is released.
limitations: limitations: The authors briefly discuss some limitations of their methods; in particular, that their proposed metrics are highly environment specific (even in this case, they are looking at only one suite of environments which may end up biasing results), and will need reevaluation every time you shift to a new set of domains. However, I think there are some other key limitations that are not discussed, many of which have been raised above. In particular:
- In addition to being environment specific, the metrics proposed are going to be very specific to the range of values which are tested. I have mentioned above how the values chosen for hyperparameter tuning here are not particularly reasonable.
paper_summary: paper_summary: The paper proposes a method to analyse how sensitive are RL methods with respect to hyperparameter tuning. The author argue that one method may perform well on average but require more HPO tuning per task which hides some computation and prevent from having comparable results. They introduce a sensitivity metric which measures the difference between the best hyperparameter tuned per task and the performance of the best hyperparameter on average. Then, they propose a quadrant analysis where both the sensitivity and the performance of a tuned score are displayed which allows to compare algorithms on those two dimensions. Experiments are then performed on PPO normalization variants where they show that current normalization have different trade-offs: some approaches improves the scores but increase hyperparameter sensitivity while other conversely lower scores and sensitivity but no approach allows to improves both scores and sensitivity currently. Finally, the authors study how many hyperparameters require tuning while having the other fix and while keeping 95% of the best performance.
strength_weakness: strength_weakness: The paper is currently missing some analysis regarding the method stability (see questions). For instance how much the proposed method would be stable and reliable with respect to different environments (does adding one environment change the results completely?) or normalization (e.g. using CDF instead of min max).
questions: questions: Here are points that would be important for me to raise my score:

* In figure 4, are the results stable if you leave one out one environment? If you compute the plot 5 times each time leaving one of the environment, do you have similar plots? If not, then the analysis on the paper will be less warranted (that such and such method is more stable with hyperparameter).
* Will the paper release the dataset of evaluation in addition to the code? (the code would have limited interest compared to the data) I would highly encourage to share the dataset, possibly with a script to reproduce some of the paper figures (for instance as was done by https://github.com/google-research/rliable) The dataset would be also very useful to simulate and compare HPO methods, in particular if it contains the evaluation per iteration (and per 100s of iterations for instance).

* Figure 4: there are very large outliers for the obs zero mean normalization sensitivity and the distribution is very skewed, could you explain why they happen? 

* One popular method is f-Anova to study hyperparameter importance, could you include an analysis using it? It would be useful for practitioners to know which HP are most useful.

* In the limitation section, you mention that the results may change under a different normalization (say CDF instead of min-max). This is indeed an important point, could you report the result for Fig4 as well with CDF? I assume it should be a minor change as it is just changing the metric. However, it seems important to assess how much the method would be impacted by different normalization (even if the results change, the paper would still be valuable)


Here are points that are more details:
- You have l110 p(w, e, h ,\kappa) and then p(a, e, h), it would be nice to unify the notation
- l151: I think it would be useful to precise that the quantity is always >= 0
- "The shaded region is a 95% Student t-distribution confidence interval around the mean return over 200 runs "
=> what is the CI not covering the mean in walker 2D?
limitations: limitations: Yes
paper_summary: paper_summary: The authors define a environment-parameter sensitivity metric in terms of a Jensen gap of min-max scaled performance. The gap gives us the difference between the best average performance and the average best performance over a particular algorithms hyperparameters over a set of environments. The authors then provide a 2D visualization where they plot this sensitivity gap against the average best-performance. This gives an intuitive way to classify whether a change in parametersensitivity can be outweighed by its (positive) change in performance.

The authors run experiments for PPO on brax to illustrate their method.

At the end, the authors also use their performance metric to tune subsets of the total parameter set.
strength_weakness: strength_weakness: ### Summary
Although this paper shows a nice way to visualize robustness of parameter optimization methods across sets of environments, the severe computation requirements needed to create these visualizations on top of the shallow experimental setup that the authors performed, makes it unlikely that the current results will prove useful to other practitioners (aside from the isolated case applying PPO to Brax). 

The chosen metric for performance (AUC) is incomplete when considering how RL is often applied (when only caring about final performance), and arguably estimated using a non-robust bound estimator. Furthermore, the authors only test on 5 Brax mujoco environments when comparing extensions that originate from Atari, DMLab, Minecraft or elsewhere. The authors, do not take characteristics of the environments they test on into account when discussing results. As a consequence, the conclusions we can take from the presented results are quite limited.

Therefore I vote for reject.

### Major comments
 - The single choice of AUC metric is poorly motivated, although it is a relevant metric, in RL we are also often only interested in **final performance**. So, the AUC gives a proxy for cumulative regret, I also want to see a proxy for simple regret, the best-run within a algorithm's training cycle.
 - The authors argue that we need robust estimators of performance for hyperparameter sensitivity. So, I am worried about the choice for min-max normalization of the algorithms' performance, see Eq. 1). These can be extremely high-variance estimators for the performance bounds, and therefore, aren't really robust (perhaps explaining why the authors needed 200 seeds per ablation)... I understand that this always nicely scales results into $[0, 1]$, however, if we rerun the experiments (different seeds and parameter-sets), would Figures 1 and 2 give similar patterns? Would this change our conclusion about what choice of hyperparameter is better in terms of environment robustness? Why not choose for an interquantile range or even a $[p, 1-p]$ percentile range that has guaranteed lower asymptotic variance? 
 - The authors don't discuss the sensitivity/ robustness to the choice of environment sets for Eq.2). I.e., what characteristics do the environments have that we test robustness for. For this reason, I also think that the discussion in 4.3 is a bit shallow considering the choice of Mujoco environments. The rewards in Brax are relatively well scaled and behaved (at least compared to e.g., a game like 2048 which has exponential rewards), so concluding that the value target symlog is not beneficial (on top of my comment about min-max scaling) should be nuanced and discussed appropriately. 
 - It's not clear from Eq.5) (and the whole of section 5) how we decide what reference point to take when deciding which parameters to tune. I.e., when we choose to keep some parameters fixed, what should their values be? Are the defaults by definition robust or not?


### Technical comments
 - Definition of normalized environment score $\Gamma$ is wrong, why is the action-space $A$ used here? (see section 2). Why is the symbol for environment sets not consistent, why use $E$ instead of $\mathcal{E}$. In section 5 suddenly $\Omega$ is used again in place of $A$.


### Minor comments
 - Eq. 2) Why not write the sum as an expectation, $\mathbb{E}[\max_h \Gamma(...) | e \sim {p(e)}] - \max_h \mathbb{E}_{p(e)} \Gamma(...)$. I.e., the more recognizable Jensen gap.
 - Shortly discuss expected behaviour of Eq.2 and what it means, i.e., a value of 1 means a large gap, meaning not robust, and a value of 0 means no gap, meaning very robust.
 - Line 220, minibatch advantage normalization is not that important for PPO, even the blog-post that the authors cite here says so.
 - What ranges for the parameters were chosen in section 4.1 ? It's easy to break an algorithm through a bad or uninformed choice for the learning rates... It's also not really clear from the text how the authors created the algorithm configurations to test, i.e., random-search? an exhaustive grid? (Found this in Appendix D, the ranges for the values are OK when speaking from experience, and the authors used exhaustive grid-search; this **must** be included in the main text)
 - Line 291, "It may ... be gained." This sentence is broken, maybe rephrase: "By finetuning a few important parameters for each specific environment, you can unlock most of an algorithm's best performance."
questions: questions: - Could the authors change the bound calculation (e.g., IQR or something similar), this should be an easy local change, and add the best-performing run metric (i.e., have the AUC and the best-run per algorithm side-by-side).
- Could the authors improve on the discussion and conclusions of the results, discussing the characteristics of the environment and how the choice of parameters hypothetically covaries with this.
limitations: limitations: Section 6 shortly discuss limitations, however, the actual limitations are discussed in appendix D. The shallow experimental setup that was performed resulted in 7 GPU years on modern NVIDIA GPUs. Since most code was implemented in Jax and the algorithm used was PPO, one of the most light-weight RL algorithms currently out there, the current framework does not offer much headroom for testing other methods. 

The severe computation costs are not included in the broader impact statement. The authors do state that reducing carbon footprint of DRL experiments are important, however, the presented methodology is not carbon friendly. If, the authors presented a better experimental setup, then this would alleviate the need for other practitioners from running these types of sensitivity analyses for PPO. However, this is currently not the case. Another way to improve the current setup is to use variance reduction methods in some way to reduce the immense compute requirements.

Comments: title: answer to rebuttal
A promising paper that needs more polish
comment: Here are two insights that can be drawn from Figure 2 of the pdf rebuttal. 
	
While it has been known in the literature that observation normalization matters a lot for performance, we can now see that the performance gain comes with, or is even partially enabled by, increased sensitivity.
It has also been previously reported that advantage normalization does not matter for performance. This appears to be true. However, it somewhat lowers sensitivity while retaining similar performance, making it possibly more interesting for practitioners than the literature has presented it.


Thank you for pointing out that section 4.3 could be deeper. For the camera-ready, we will drastically shorten section 4.2 (or move it to the appendix) and expand section 4.3 to include additional discussion and figures (different normalization schemes, environment subsets, etc.) as discussed in the rebuttal and responses.
We appreciate your comments and thank you again for your thoughtful review.
2D Plot
While color may not be the best choice for visualizing these segmentations, we don’t believe the segmentation is arbitrary. Each of the segmentations has a different interpretation of its relation to the reference point. The slope one line passing through the reference has unique importance as it marks the points where observed performance gains are directly attributable to per-environment hyperparameter tuning.  

Comparison with AutoRL

We agree that, like AutoRL, we are interested in promoting methods that require less human intervention and tuning to apply. The key difference we see between our work and AutoRL is that we are demonstrating the utility of our methodology with an experiment on PPO, not proposing a new way to tune hyperparameters in RL. While AutoRL methods tune an RL algorithm's hyperparameters internally, it should be noted that the AutoRL algorithms themselves often have hyperparameters (e.g., the scaling factor and crossover factor parameters in DEHB). For the camera-ready, we will include an additional discussion of the AutoRL literature and how the AutoRL community could use our method to measure their effectiveness at improving performance while reducing (hyper-)hyperparameter sensitivity.

Thank you for your additional comments. We appreciate your feedback and discussion.
Thank you for your response and for clarifying. We are sorry that we missed this point in our initial response. We used a confidence interval as we are interested in understanding the exact sensitivity and performance under different algorithmic variations and so plot the confidence in our sample estimates.  We will include a table in the appendix that reports the standard deviation across seeds of the AUC for each hyperparameter setting tested in each environment.
Thank you for your review and suggestions for improvement! We will incorporate them by modifying our broader impacts statement to include a discussion of the computational resources used as well as ideas for future work that promotes sustainable research.
Thank you for your answer.
1. Thanks for adding this figure, I think it is important to verify this stability to make sure the statement of the papers are applicable in generalized settings.
2. Great to hear, as mentioned in my review, this data can be valuable for other research.
4. The analysis you suggest makes sense and would provide value (I gave f-anova as an example but any analysis on hp importance would be valuable). I agree with you that studying hyperparameter importance is tangential to the main point of the paper but still seems related (it may be that some algorithms have the same hyperparameters to tune and other requires a larger set which is also interesting for practitioners, in addition to know which hyperparameters had the most effect)
5. I get your point about CDF, I dont mind if you use (p, 1-p) percentile transformation instead as long as you make sure that the results are not completely tied to one normalization. Regarding the downsides of the CDF you mention, I think those are standards points not necessarily tied to your use-case: the CDF gets rid of the scale (only the ordering matter), this has benefits (robustness to outliers, uniform distribution obtained) and downsides (sometimes the scale is important) which is why this normalization is good to have in addition to min-max as it has different trade-offs.

I have raised my score given that some of my points were addressed.
I thank the authors for the clarifications. 

Regarding the bootstrapped confidence intervals, what I mean is that the more samples available and the more points drawn at a time the smaller the interval will be.
This is different from a dispersion measure, like the variance, which would not shrink with more data.

I maintain my score.
Thank you for extensive response to my review.

A few thoughts are below. Where satisfied, I have not included response to each point for brevity's sake.

# 2D plot

I think the key for me about the colouring is that it enforces a fairly arbitrary segmentation that I think gets in the way of analysis. I think, personally, I would prefer to see results simply plotted on these axes, with analysis separated into text.

# Comparison with AutoRL

While I agree there is a difference, I think the takeaway is that both AutoRL and your work have similar motivations - to promote systems where an environment is put in and a policy comes out without human input. As such, I still think this is worthwhile comparison. In a sense, you can consider that AutoRL algorithms are 'hyperparameter free' in that they deal with hyperparameters internally, and thus have no hyperparameter sensitivity.

---
In addition to the above, I still feel there is a missing component considering how human-in-the-loop and prior work would focus a lot of these search efforts by offering intuition about the kind of hyperparameter ranges which are useful. That said, I am so far satisfied that some of my concerns have been addressed sufficiently, and thus have increased my score from 3 to 4. I remain open to discussion about the above.
I thank and commend the authors for the strong points laid out in the global response, the revision of their figures, and their direct comments. I find figure 2 in the additional pdf (global response) extremely interesting and am surprised by the extreme contrast in results when comparing figure 4 (main paper) with figure 1 (pdf rebuttal).

---
I agree with the point that mujoco/ brax was only a test-set to showcase your visualization method, and I did not find any overly strong claims or conclusions in these sections in particular. However, the main problem that I have is that despite the costs of the experiments, and the proposed visualization, I can still not draw any insights from these results. Which is unfortunate.

The issue that I had is that this section has too strong of a focus on the results, for example, section 4.2 is a lengthy discussion of most parameters (the details of which, I believe were a bit beside the point) and section 4.3 simply enumerates everything we can see in figure 4.

I want to see a critical discussion of: how can we use this visualization method to gather insight and what can we do with this information. This should be accompanied by a well executed test-case. This should include multiple ways to look at the same data (different normalization, environment subsets, conditional parameters, or more). Like I mentioned before, I think figure 2 of the rebuttal-pdf adds tremendously to this discussion. 

---
Considering all my previous points, I still think this paper lacks polish. It looks promising, but it is not quite there yet. I will raise my score from 3 to 4 and contribution from 1 to 3, since the authors did address many concerns.