Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes a new benchmark for RL that uses one set of hyperparameters across all environments. The authors argue that tuning hyperparameters per environment requires a large number of runs in order to get rid of the statistical errors, which also makes running RL algorithms quite compute-intensive. Therefore, the authors propose to tune a single hyperparameter across environments with a preliminary set of runs for hyperparameter selection and then evaluate the selected hyperparameter for hundreds of trials on each environment for the final performance. The authors perform the proposed evaluation protocol SHB on classical control tasks and RL algorithms and find that the algorithm rankings found by SHB are consistent across all environments. The paper also finds that the change in performance for each algorithm on every environment when using the SHB
versus conventional per-environment tuning is quite large on each environment, suggesting that RL algorithms are not quite able to be generally applied to all environments.
strengths: The proposed evaluation protocol makes the evaluation of RL algorithms more robust w.r.t. statistical errors and also focus more on generalization rather than overfitting to individual environments. SHB also makes RL evaluations less compute-intensive via only sweeping hyperparameters for 3 runs. SHB is very important for RL community since RL algorithms have been known for sensitivity to random seeds and susceptible to statistical errors. I think it would be of great significance for the whole RL community.
weaknesses: The authors claim SHB as a benchmark but it seems it is rather just an evaluation protocol. It would be best to clarify it in the paper. Moreover, evaluations of RL algorithms on continuous control tasks such as locomotion and robotic manipulation are missing, which makes the evaluation less thorough. I strongly suggest that the authors should include these evaluations and also add results of more recent RL algorithms such as TD3, SAC, etc.
correctness: Yes
clarity: Yes
relation to prior work: Yes
additional feedback: See above
review_title: Official Review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper proposes a new methodology for evaluating RL algorithms, which it terms the Single Hyperparameter Benchmark (SHB). Rather than separately tuning each algorithm on each benchmark environment, SHB advocates doing a small amount of tuning for each algorithm to find a single hyperparameter setting that gets high average return across all environments. Experiments show that this approach produces stable rankings across executions of the tuning process, even when using just a few seeds for tuning.
strengths: The strengths of this paper are straightforward and compelling:

[S1] The paper discusses a significant issue of practical importance to RL researchers and practitioners. Moreover, the proposed approach is simple, computationally inexpensive, and well argued for. The basic idea of using just one set of hyperparameters for an entire suite of environments is already common practice in the planning community, and RL could benefit greatly from similar community norms.

[S2] The experiments are quite thorough and addressed a number of objections that I had while reading the methods section (e.g. Does it matter if we aggregate scores across environments using something other than a summation during tuning? Does using a single hyperparameter setting actually change the rankings of algorithms? etc.).
weaknesses: However, I still objected to one of the major arguments in the paper and felt there were some lesser weaknesses in other areas:

[W1] The paper argues in various places that SHB does not require as much hyperparameter tuning as existing benchmarks, and in particular trumpets the fact that the experiments in Section 5 obtain good results using only three tuning runs. The conclusion takes this further, arguing that it allows "small university labs and tech giants alike to conduct rigorous and meaningful comparisons". My interpretation of these claims (which may not be the intended one) is that SHB-type approaches that average over many environments are not as sensitive to hyperparameter tuning as existing approaches, and so confer less benefit to well-resourced labs that currently do absurd amounts of tuning to make their algorithms shine.

Insofar as this is the intended argument, I am not yet convinced by it. In my experience, benchmarks are used throughout the development cycle to make design decisions and choose default values for hyperparameters (which may or may not be tuned over afterwards). There's a risk that an algorithms researcher using SHB might accidentally (or intentionally) do a huge amount of manual tuning on the SHB score during algorithm development by using it to make design decisions (e.g. should I use a replay buffer or not? What should be the base architecture for my network? etc.). The resulting ranking might be stable across tuning runs, but would give the new researcher's algorithm an unfair advantage over baselines that may have originally been developed to target a different population of environments. Figure 6 partially addresses this concern, but I'm not convinced that the limited number of hyperparameters considered (LR, replay buffer size, target network refresh rate) really capture the number of degrees of freedom that are typically available in deep RL. SHB definitely has value for an "honest" evaluator comparing existing algorithms on "held-out" combinations of environments, but I'm concerned that the paper overstates the degree to which it will address the current compute arms-race among benchmark-minded researchers.

[W2] This is not so much a weakness as an opportunity, but I feel that there are a few places where this paper could do more to encourage comparability of results across papers. One is in the choice of environments: as far as I can tell, the paper is trying to be somewhat agnostic to precisely what environments go into the SHB population, so that researchers could apply this methodology to any other benchmark suite. I think there is value in defining particular combinations of benchmarks and giving them "official" names (e.g. "classic control SHB" for the environment population in section 5, or "dm_control SHB" for the environment population in section 6). This would encourage researchers to use those specific combinations in future papers, rather than dropping environments that have inconvenient results.

I think there would also be some value in creating a predefined $\mathcal P_E$ for each environment in the paper so that future papers can copy those results instead of re-evaluating every algorithm anew each time. This might create some perverse incentives (e.g. to dedicate tuning budget to environments with weak baselines), but could be worth it if it encourages researchers to compare to more algorithms, since they don't have to re-run each one. It would also serve as another incentive for them to use the same combination of environments as previous papers instead of picking combinations that are favourable to them.

[W3] There are some lesser clarity issues that I discuss below.
correctness: I did not have any concerns about correctness.
clarity: [CL1] I was confused by the terminology in the title and introduction. Usually "hyperparameter" refers to a particular control knob for an algorithm such as, e.g., the step size of SGD. I initially assumed "single hyperparameter" meant that the algorithm was only meant to expose a single control knob, but that this knob could be tuned separately for each environment just like the (many) control knobs of existing algorithms. This misunderstanding was clarified fairly quickly, but using "hyperparameter setting" instead of "hyperparameter" might help readers who have the same confusion as me.

Similarly, I assumed that the "Single Hyperparameter Benchmark" referred to a particular set of environments that the paper was proposing (which is what I usually think of as a "benchmark"), but then was surprised in Section 6 when it was used to a different set of environments. I would personally find "single setting of hyperparameters" easier to understand than "single hyperparameter", although I admit "SSoHB" does not roll off the tongue.

[CL2] I didn't understand how the normalization was defined on lines 222 to 233 during my first read-through. After reading Jordan et al. & coming back to this I think all the necessary information is in fact present in the paragraph, but it took me a while to parse. Explicitly stating that $N(G) = CDF(G,E)$ might help.
relation to prior work: I'm not aware of missing related work, and the existing comparisons seem adequate.
additional feedback:  On the whole I enjoyed reading this paper and think that it would be a valuable addition to the track.

------

(review last checked/updated 2021-07-19)
review_title: A well-argued proposal for an alternative way to evaluate RL algorithms

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The work is aimed at standardizing the selection of hyperparameters for a distribution of environments. It does this in a simple manner that can be cheap and this appears to be a robust methodology for the subset of hyperparameters and environments evaluated. It shows how per-environment or even "subset of environments" tuning can lead to misleading conclusions on the whole set of environments one may be interested in. It also analyses the distribution of performance of RL agents on these environments.

strengths: The work is in the right spirit according to me. We definitely need such analysis of sensitivity to hyperparameter tuning for RL agents. It could also be used as a cheap proxy to check whether RL agents are progressing more generally to doing well over distributions of environments. So, it is broadly applicable. They also perform many experiments for the SHB vs per-environment vs subset of environments tuning comparison which is good.
weaknesses: The paper needs to make the bigger picture clearer I think. While it _is_ broadly applicable, the use-case is still not universal. The goals could be different for different people. For example, someone might be interested in dynamic tuning as in PBT (https://arxiv.org/abs/1711.09846) and only obtaining a good agent on a specific environment. While this is discussed indirectly by saying that so far research has been focused on what _can_ be done and not on generality, it's not explicitly stated. I think it needs to be made clearer that the SHB is for someone wanting to perform better on a (limited) distribution of environments.

Further, if the distribution is too large then the SHB is not cheap anymore. Someone might want good performance on a much larger set of environments than 6, e.g. 57 for Atari and then the SHB is not so cheap anymore imho. While they show experiments with DDPG on complex environments with 28 environments, they use fewer final runs (30) than the suggested number of 100 in L107. Also, I think the compute used for DMControlSuite is not mentioned in the paper. Basically, the point is that what generality is for an RL agent is not objectively defined - what number/distribution of environments at which to draw the line for SHB is a bit artificial. And if this number is too large (which it would need to be to measure "truer" general progress of RL agents), then the SHB will be slow. Even, the cited paper "Generalized domains for empirical evaluations in reinforcement learning" suggests that one may overfit to even the distribution of environments seen (page 3, column 2 in that paper) and the goal is not to be robust to an arbitrary distribution. So, the paper needs to be more explicit about its limitation that it is somewhere between true generality and per-environment specificity. 

Furthermore, it needs to provide guidance on how to choose the numbers of runs per environment. Were 30 and 100 above arbitrary? How was $n_{runs}$ chosen to be 3? Was 2 tried as a value for it? Was 3 chosen because it was the lowest possible value for which the SHB worked?
With respect to choosing these numbers, the danger with complex environments environments is that they are even more variant than simpler ones and, in my opinion, would need $n_{runs}$ to be even greater than for the small environments to get proper conclusions for the SHB.

The paper also needs to provide guidance on the environment sets it is expected to be run on. Do the authors want us to only use the 6 small environments in every RL paper as a proxy measure of general RL progress? Because, as mentioned, if SHB is used on large environment sets, it could be too expensive.

**Details on hyperhyperparameters are missing**:

How were the grids of hyperparameters (HPs) selected? Why were the values on the grid for stepsize of the tile-coded agents 2^-3 larger than for the deep RL agents? Couldn't comments about cherrypicking be made at the hyperhyperparameter level for this paper? How was the subset of HPs that was studied picked? There are so many possible RL HPs. Is it because the tested HPs are more important in the authors' opinion? Could the authors provide guidance on which HPs a researcher should use when using the SHB?

Because the conclusions on individual environments were very noisy in Figure 3, I find it surprising that the mean of such noisy results was perfectly consistent. This is why I'd be interested in seeing the SHB for a different subset of hyperparameters and/or on a different set of (possibly small) environments. This would hopefully also help with answering the questions in the previous paragraph.

As an aside, the name "stepsize" really confused me because I'm used to "learning rate" and I initially thought that "hyperparameter stepsize" was the stepsize used to perform grid search over all the HPs. So, I think this needs to be clarified in the paper as well.

The repo needs a README guide on how to run experiments.

The repo could allow users to easily use the SHB by defining, e.g., a *run_SHB.py* file that takes agent(s) and environment(s) as arguments. I know that agent APIs are not standardised, but one could create an "SHB API" for agents that one should conform to and then SHB could be run out-of-the-box as long as the user's agents conform to the SHB API. This is easy to do and would aid adoption much more.

I think there should be another experiment similar to the one for DDPG that provides insight on another combination of HP and environments.

Another key aspect which I think is not explicitly touched upon is the hyperparameter tuning budget used by researchers. Given the large variance of the models used by the community today, given enough time to tune hyperparameters, any method with enough variance can, in principle solve a task as far as I understand (this is related to Section 5). So, a guideline on limiting HP tuning budget should be added I feel (Or, a metric that quantifies/normalises the performance in terms of the $n_{runs}$ and "$n_{final\\_runs}$" chosen by the researcher).


correctness: They are generally correct as I see it. But some points would need to be addressed as mentioned later.
clarity: The paper does need to clarify many parts of the text which are further mentioned below.
relation to prior work: The paper discusses several important related works.
additional feedback: The "Ethical considerations" section was supposed to be in the main paper according to the paper checklist, so this allowed the authors more than half a page extra over the limit.
The checklist item asking for "the code, data, and instructions needed to reproduce the main experimental results" is a TODO.

Here are more comments pertaining to specific line numbers from the text (I mark them as L<n> where n is the line number):

L2: "allows comparison of RL algorithms across environments using only a single hyperparameter"

Did you mean a single hyperparameter "configuration/setting"? I found it confusing that "hyperparameter" was used to mean "hyperparameter configuration" multiple times.


L16: "AIGym"

I think it's called OpenAI Gym.


L144: "For example, the parameters determining the physics of classical control domains were randomized and sampled to avoid domain overfitting (Whiteson et al., 2009),"

Reference seems wrong.


L160: "y-axis the probability"

Is it the pdf and not the probability? (Can't be probabilities, since the sum would be >1.)


L162: The 2 paragraphs beginning here seem a bit vague.
"DQN is more likely to fail than to learn a policy which solves this relatively simple environment."

What does failure mean here? Performance worse than random? Performance <100, <200, ...?

L167: "First, using the expected value of this distribution as the summary statistic does not aptly demonstrate that the poor performance of DQN on Cartpole is due to occasional catastrophic failure."

How is catastrophic failure defined?

"Instead, mean performance might lead us to wrongly conclude that DQN on Cartpole usually finds a sub-optimal policy—better than random."

How is sub-optimal policy defined? Performance <300? Sentence seems to imply it's defined as being better than random.


"An alternative might be to consider percentile statistics, or if the goal is to evaluate mean performance, to avoid drawing strong conclusions about individual runs."

I did not understand this at all. Should the sentence be split in two?

L173: "If the goal is to report mean performance, ..."

What might be the alternative goals? I don't remember seeing alternative metrics being discussed before.

L179 and 181: Grammatically wrong.
Additionally, the definitions of sensitivity and stability are not completely clear to me.


L210: "aligns empirical practice with the goals of applied RL"

These goals were not explicit to me. As mentioned before, goals might be different for different researchers.


"2) is computationally feasible even in complex environments"

As mentioned before, to me, computational feasibility seems as it would not always hold.


"3) provides novel insights on old ideas (even with small environments)"

I think this claim is correct but would prefer a bulleted list of these insights.


L223: "We use a lightly modified version of the CDF normalization method from Jordan et al. (2020),"

Could you please justify why you used a modified version?

L238: Did you mean proportional to |E|? You run theta_SHB on every environment, right?
In any case, the claim only holds if n_runs * |E| * |Theta| << n_final_runs * |E|, right?

L247: What does "added variance" mean?

L248: "This evaluation methodology provides some insight into the difficulty of tuning, though requires a sensible distribution over hyperparameters to be chosen."

Even SHB would require sensible ranges for HPs. Isn't this why the HP grid values used are different for tile-coded and deep agents?

L253: "Both of these strategies help identify algorithms that are difficult to tune, but the SHB is easier to use and computationally cheaper."

Could you please substantiate this claim with concrete numbers and what you mean by "easier"?


L273: Why was LunarLander run for 250k timesteps as an exception? Is there an objective metric to decide total timesteps? Otherwise, it feels like a per-environment tuned HP.

L275: How were the stepsizes chosen?

L282: The dataset over which the bootstrap samples were taken was unclear to me. I guess it's defined in L64 but it took me quite a while to find again.

L287: How exactly is an experiment defined? I'm not clear what is plotted in Figure 3. Is it the means of 10k values (with each of these values itself a mean over three bootstrapped values)? Or is it the mean of the final 250 runs?
The caption for figure 3 is misleading because, by itself, it seems to suggest that perfect consistency is between per-environment and across-environment rankings. Only if you know the context from the text is it clear what is meant here.

L290-292 were very confusing to me. I think this stems from saying that selecting HPs was difficult. How can this be difficult when all that needs to be done is take averages?

L297: I don't know what "counting wins" means.

L306: Could you please explicitly show the calculations? I tried to get to the figures stated in the text but couldn't.

L377: Why was the number of bootstrap samples chosen to be 1k instead of the 10k used earlier? As I understand it, this step doesn't add much overhead to the computation, whether it's 1k or 10k.

L649: "Although there is a large difference in the WalkerRun environment, we point out this may be due to the SHB trading-off performance on other environments in order to pick a single hyperparameter setting; without explicit per-environment experimentation this remains unclear."

This can be seen as a limitation of the SHB - that we no longer have clear insights on a per-environment basis. Returning to the point about different use-cases for different researchers, there's a chance that if we ever get to an "optimal" RL agent, that it will have different HPs per environment. Such an agent may be hard to test using the SHB.

L396: The claim is a bit too strong for my liking. Because, the configuration space for HPs that was swept over (defined from L696 onwards) is far from exhaustive. Also, the sigma (L700) differs from previous work, so results may not be comparable.
Further, I am curious whether the claim in L701 is based on using the SHB to decide on choosing a smaller sigma. The sentence is unclear on this and it seems some kind of human judgement might have been involved in deciding what value of sigma was a good choice. Could you please clarify?


EDIT: Updated score based on authors' changes.
review_title: Work that is in a right direction but needs more documentation, code, explanation and clarity

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper introduces the Single Hyperparameter Benchmark (SHB), a new benchmark for comparison of RL algorithms across environments using a single hyperparameter. This benchmark shifts the focus from optimistic results based on environment overfitting to stability and generality of RL algorithms. The idea is to evaluate algorithms on a set of environments using the best hyperparameter setting across environments rather than per-environment. SHB is computationally inexpensive and easy to run.

The authors motivate and introduce SHB and evaluate it on an array of known RL algorithms. They discuss how to interpret the SHB results and how it behaves in certain settings. As an example, the SHB is used to demonstrate that Orstein-Uhlenbeck noise does not outperform Gaussian noise in the DMControl suite of tasks.  

strengths: Performance of SHB appears to heavily depend on the inclusion / exclusion of single environments. For example, the authors demonstrate how under SHB, the algorithm DeepQ sacrifices performance in Cliffworld and Lunar Lander to a point worse to a uniform random policy. Since this is clearly discussed in the paper, I would not deem this a terrible weakness of the approach, but this might be possible to improve on. 

weaknesses: Performance of SHB appears to heavily depend on the inclusion / exclusion of single environments. For example, the authors demonstrate how under SHB, the algorithm DeepQ sacrifices performance in Cliffworld and Lunar Lander to a point worse to a uniform random policy. Since this is clearly discussed in the paper, I would not deem this a terrible weakness of the approach, but this might be possible to improve on. 
correctness: The claims, evaluation methods and experiments in this paper seem technically sound to me.
clarity: The paper is very well written and the high-level is easily understandable even for researchers with little RL background.
relation to prior work: There is only little discussion of related work, however I can not speak to how much work has been done in this area. 
additional feedback: Some of the figures in the main text are missing captions.
review_title: RL benchmark focussing on stability and generality

Meta-review: This paper created a lot of discussions, involving the reviewers and multiple area chairs.
The paper received quite high scores by several reviewers, with 3 non-confindent (confidence:3) votes for acceptance (scores:6,8,8) and one very confident (confidence:5) vote for rejection (score:5).
I therefore read the paper myself in detail, discussed with the reviewers, and also with another area chair.

In these discussions, there was consensus that the paper points into the right direction, but that some points are problematic and could have undesired consequences rather than facilitate better experimentation in the RL community. There are no conditional accepts in this track, but I would expect that the paper will get accepted in round 2 of this track if it addresses the remarks below.

The paper's main contribution is a new empirical protocol. The authors clearly have a good point in suggesting to move beyond empirical protocols that perform unbounded hyperparameter tuning separately for every environment. However, the proposed protocol also has some issues (please see below for details), which is problematic since many reviewers would probably ask authors to follow it. (Note that this NeurIPS benchmarking track is likely going to be the most credible resource for benchmarking in machine learning, so the protocol would be accordingly prominent.)

In detail, the protocol / paper has the following issues:

1. The paper's suggested budgets are very high: 30 seeds for every hyperparameter setting and every environment in the preliminary phase. With a single run sometimes taking several days on complex environments this policy would (a) contribute substantially to the heavy carbon emissions bill of AI research and (b) disproportionally disadvantage researchers in typical academic institutions that do not have thousands of machines for this tuning. 

2. Likewise, the suggested budget for the final evaluation is extremely high. Not even DeepMind, despite its access to large compute, ran 100 seeds for all of their Atari environments.

Remark: The authors might feel that 1. and 2. are unfair since the numbers 30 and 100 are never explicitly mentioned as suggestions in the paper, but since they are used without a discussion that these numbers need to be adapted to the case at hand, future reviewers would likely ask for exactly these numbers of runs for the papers they are reviewing. (If they asked for very cheap experiments like the ones the authors are running throughout most of the paper that would of course be fine, but reviewers tend to ask for more expensive experiments, such as Mujoco, Atari, ProcGen, etc, and it should be clear from the paper that these are not the domains for which 100 repeats are suggested.)

3. The suggested protocol is wasteful as it does not take into account modern hyperparameter optimization techniques: it speaks about "parameter sweeps" throughout, i.e., grid searches. There are far more computationally efficient hyperparameter optimization methods, such as Bayesian optimization, multi-fidelity optimization, etc. Baking the least sample-efficient hyperparameter optimization technique (grid search) into the experimental protocol is not a good idea.

4. The paper misses the opportunity to make the connection to the rich literature on algorithm configuration that has studied the problem of effectively finding the best hyperparameter setting across tasks for over a decade. See, e.g., the ICML 2019 tutorial on algorithm configuration, for an overview. Algorithm configuration is basically an extension of hyperparameter optimization that tries to find a single hyperparameter setting which works well across tasks -- precisely the setting the authors propose. 
4a. Algorithm configuration methods automatically allocate more budget to better-performing hyperparameter settings, evaluating poor hyperparameter settings only on a few tasks and thereby substantially reducing the high cost of the authors' proposed method.
4b. Algorithm configuration methods also use the concept of "adaptive capping": terminating poorly performing runs early to save even more compute.

Remark: this relationship to algorithm configuration does not mean that the authors' experiments need to use the latest algorithm configuration methods, but it would be good to make the connection to make the point that the proposed experimental protocol can indeed be computationally inexpensive.

Relatedly, the argument the authors make about the computational cost of the standard protocol and the proposed protocol appears broken: in standard tuning, to avoid overfitting one should also do less runs for each of the hyperparameter settings than for the final evaluation, so the computational complexity of the two protocols is the same. (With algorithm configuration, particularly 4a, actually the authors' protocol would be faster again, since not all tasks need to be evaluated for poor hyperparameter settings.)

(continued below)
Decision: Reject