Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a new approach to HPO tailored towards RL. Specifically, a HPO approach that is sufficiently sample efficient such that the approach is effective even with limited resources (research lab as opposed to data center) is sought after. Bayesian optimization is used, with a Gaussian process (GP) surrogate. The GP has a novel deep kernel which consists of a parametrized Richard's curve, where the five coefficients of the curve are what the trained deep neural network is used to predict. The purpose of the kernel is to represent, and sample-efficiently infer, reward curves for specific hyperparameter configurations. In an extensive evaluation, on several problem domains comparing with several different kinds of state-of-the-art HBO methods, the proposed approach is demonstrated to be highly effective and clearly outperform competing methods.
strength_and_weaknesses: Strength
========

* The context of the work is well described and the approach is well motivated given related works.
* The approach is clever and attack the core problem of sample-efficient HPO for RL, namely to predict the continuation of reward curves given the training epochs so far. The connection to and use of multi-fidelity GP well made and highly suitable.
* The evaluation is strong, with a variety of methods and environments.
* Clear hypotheses, and interpretation of results in light of these.
* The ablation study enriches the contributions.

Weaknesses
========
Nothing obvious
clarity,_quality,_novelty_and_reproducibility: High clarity: Well written with excellent related work context. Most details are in the paper, but those that did not fit (e.g. BO, deep kernel learning, etc.) are referred to suitable related work.
High quality and seemingly high novelity. 
The method is well described and source code is available (I did not test it).
summary_of_the_review: The paper is well contained, well positioned and makes seemingly large and significant advances on an important problem for RL in general. It was a pleasure to read.

---

### Update
Thank you for engaging with the reviewers and clarifying many important things. However, after reading through the other reviews I reduce my score, mainly due to evaluation concerns as pointed out by h99m and eXNC. I still believe that the paper contain contributions of interest to the community, but the clarity and motivation for parts of the evaluation should probably be improved in accordance their comments. eXNC also raise important concerns in the latest comment & update, which affect my judgement. I am consequently still in favor of accept, but only marginally.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a Bayesian optimisation framework for tuning hyper-parameters reinforcement learning algorithms. The framework is based on a kernel formulation which uses a model reward learning curves to interpolate between multiple budget-dependent fidelity levels. An extensive experimental evaluation is presented comparing the proposed approach against existing hyper-parameter optimisation (HPO) baselines on problems with popular RL algorithms and multiple environments.
strength_and_weaknesses: ### Strengths
* The idea of using reward-curve models to inform multi-fidelity kernel functions for HPO is interesting.
* Experimental evaluation has a sufficient number of algorithmic baselines.
* The proposed algorithm seems relatively simple to implement compared to other baselines while achieving high performance in the low-budget regime.

### Weaknesses
Most of my concerns regard the experimental evaluation, which seems to be the main focus of the paper.

* **Number of trials** Experiments were repeated for only 3 independent trials (3 random seeds). This number seems to be a bit too small for state-of-the-art claims. DEHB, for example, had 50 trials for most experiments (Awad et al., 2021)

* **Baselines configuration** All of the HPO baselines have their own hyper-parameters and architectural choices (e.g., kernel, acquisition function, etc.). How were these configured to provide a fair comparison? Given the gap of around one std. deviation (out of only 3 random seeds), it might be possible to fine tune the baselines to obtain better results with them.

* Why use the maximum budget for the single-fidelity baselines? With 4 initial configurations provided, if I understood it correctly, it leaves them with only 6 attempts available to try to find anything. Meanwhile RCGP would have up to 10 times more iterations to run for with lower-fidelity observations.
clarity,_quality,_novelty_and_reproducibility: The paper is mostly well written. The quality of the experiments section could be improved (see comments above). A few minor issues are listed below.

* How is "rank" defined in the experimental comparisons?

* How long was full training routine for the tasks in Fig. 5 and 6? The time scales in these plots only go to $10^6$ steps, instead of $10^7$ as in the previous ones.

* Which of the three Mujoco environments is referred to as "MUJOCO" in the plots? Is it an average result over them?

* The acquisition function for the GP-BO baseline is not stated in the main text, nor the appendix. There are also no detail on how hyper-parameters are tuned for the other baselines.

Typos:
* Sec. 3: ``We define the history of $N$ evaluated configurations'', $N$ or $K$ (as in the rest of the paragraph)?
* Eq. 2: Definition of $p_H$ is missing. Also, I believe the $\log$ term is incomplete. Shouldn't it be the log of a probability (density)? If so, the $p$ is missing.

summary_of_the_review: The paper's contributions are novel, but the experiments section needs improvement and further clarifications. Otherwise, it imposes the question that the gains might be only marginal or fruit of the stochastic nature of the problems, given the low number of independent trials (3) and lack of clarity about the tuning of the baselines.


#### Post-rebuttal update
I've read the authors response and decided to raise my score (5 to 6).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The work proposes to incorporate the learning curve information in reinforcement learning as features in Bayesian optimization to more efficiently perform hyperparameter optimization.

Given an RL learning curve, one could fit a generalized logistic function on the curve to obtain the parameters of the curve as features that are used in a Gaussian process model. To generalize this procedure to hyperparameters where there might not already exist a learning curve, the authors propose to predict the parameters from a neural network that takes the hyperparameters as an input.

Experiments are performed on an AutoRL-Bench a benchmark for automatic reinforcement learning. This benchmark includes learning curves for a set of hyperparameter values. And it is possible to test the performance of BO algorithms by querying the learning curves for the different hyperparameters without requiring to rerun the whole RL procedure, allowing for efficient computationally inexpensive benchmarking of BO algorithm in AutoRL. Note that this benchmark appears to be still under review and does not seem to have passed peer review yet.

They experimentally compared with many different BO algorithms including random search, GP based methods, population based methods. And the new method outperformed the baselines.

They also performed ablation studies on different components of their algorithm.
strength_and_weaknesses: Strengths

- The performance appeared good.
- The ablation studies showed the necessity of the different components of the algorithm.

Weaknesses

- The AutoRL-Bench has not passed peer review yet and it is unclear whether it is suitable. Looking at the set of hyperparameters, it seems that the log10 spacing of the learning rate might be a bit large making it too easy to differentiate between the performance, so it was not clear to me whether this benchmark is really appropriate. At the same time, I understand their setup allows testing many different algorithms on a wide range of environments cheaply, and this is beneficial. I think it would be good to add your own smaller scale experiments including learning the hyperparameters from scratch on some proposed domain of hyperparameters (in an interval not between a finite selection of hyperparameters).
- The method included taking the max over past rewards in a window. This has been widely criticized in previous work in RL because it biases the evaluation. For example: Deep reinforcement learning that matters (https://arxiv.org/pdf/1709.06560.pdf)
“Due to the unstable nature of many of these algorithms, simply reporting the maximum returns is typically inadequate for fair comparison” or Deep reinforcement learning at the edge of the statistical precipice (https://arxiv.org/pdf/2108.13264.pdf)
While the ablation studies showed that this "max" was useful, comparing different algorithms when using such an evaluation protocol in RL is inappropriate. I would suggest trying some other technique, such as taking the average in the window.
- The presentation of the results could be more comprehensive and better executed. Currently they only present the ranks of the algorithms. This makes it difficult for future researchers to compare to the results in this paper. It would be useful to also present absolute performance based metrics such as the reward. This makes it also more clear how much the performance differs between the methods. Moreover, it seems that the errorbars are unrealistic in the article. They seem to wide given the smoothness of the curves. Moreover, sometimes the errorbar crosses below rank 1, which should be a hard minimum. I would suggest to change the statistical analysis performed to obtain these errorbars, e.g., using a 95% percentile bootstrap may provide better errorbars.
- There was no direct analysis of the accuracy of the newly proposed model based on the reward curves. The experiments only show the BO performance, which implies that the model should have accurately predicted the rewards; however, it would be better to directly check the prediction accuracy.
- It was unclear to me why the experiments were limited to RL. Learning curves are also present in other fields of machine learning. Is the method also useful in other domains? Why not expand the experiments to other domains? I would at least have liked to see some discussion or explanation of this.
clarity,_quality,_novelty_and_reproducibility: Clarity

Generally good, but I think a few points could be clarified:

What kernel was used in equation 7?

What do you mean by epoch? Typically 1 epoch is one pass through the data set in supervised learning. In RL the dataset keeps changing, so epoch is ill-defined.

> gray-boy Bayesian optimization

gray-box

Quality

The breadth of the experiments and amount of comparisons, as well as the inclusion of ablation studies was good. The data analysis and presentation could have been better. I was unsure of the used benchmark, and think that running at least some small additional experiments would be good. Also the max-smoothing seemed like a poor choice based on its criticism in the literature.

Novelty

I believe the idea of using the reward curve information to better predict the outcome of the experiment is novel. There are some previous works that take into account the iterative nature of these experiments, but I believe modeling the reward curve is new.

Reproducibility

The authors provide their code, so I believe it's reproducible.
summary_of_the_review: The work includes comprehensive experiments on BO in RL showing that the newly proposed method of modeling the learning curves improves the performance. However, I am unsure that the AutoRL-Bench that they performed their experiments on is a suitable way to evaluate BO algorithms as it does not appear to have passed peer review. Moreover, the analysis of the data and presentation of the results could be improved, e.g., improving the errorbars or also displaying reward information. Taking the max of the rewards as done in this paper has also been criticized in prior research. Considering these points, I am leaning towards suggesting to reject.

Update
------------------------------
_________________

After seeing additional details in the updated paper, and more discussion with the authors, I perceive more issues in the paper than initially, and I have reduced my score to 3. The main additional issues are that I believe the performance of the algorithms should also be compared based on the performance after fully training with the currently selected best hyperparameter (as opposed to looking at the reward at the current budget), and that the performance on several of the environments is quite poor (as can be seen from the reward curves). Regarding the second issue, it would have been good to include results for larger hyperparameter optimization budgets (the table based benchmark allows quickly testing large budgets, so I guess there is no computational issue with including larger budgets).

I list my main concerns below:
- Only indirect evidence is provided that the predictive model is performing well. I pointed this out in my initial review by saying that the accuracy of the predictive model was not compared. The authors responded that the accuracy is not that important, and the ranking is more important. While this may be true, it does not address the intention of my comment, and if the ranking is more important, the work could have instead investigated the ranking accuracy. Other ways to provide direct evidence that the predictive model is performing well would have been to visualize the predictive results. Providing this kind of evidence is important to prove that the algorithm is working as intended. There could be any number of reasons why the performance of the algorithm appeared to be good (e.g., bugs in the code or inappropriate evaluation), and providing multi-faceted evidence is a good way to mitigate such concerns. It may be particularly good if the authors compare with a simple ranking method, e.g., ranking the hyperparameters based on the reward at the same budget, and compare whether the predictive model outperforms such a simple method.
- The benchmark is not well established. In my original review, I said that the benchmark has not yet been published. The authors responded that after the reviewing started the paper was accepted for publication at a NeurIPS workshop. This does not mitigate the issue that the benchmark is not yet established in the community as a good way to evaluate algorithms as there are no prior works using this benchmark. Workshop papers are often work in progress that still have issues. The authors also asked whether I could provide any other benchmark for them to use. Actually, I don't necessarily think that the benchmark is bad, and it seems like a really good idea to have such a tabular benchmark for HPO; however, as it is very new, I am not convinced of its reliability. If the benchmark were well established, the requirement for additional evidence in the previous concern could be lessened; however, as the benchmark is not well established, I suggest that the authors should mitigate such a concern by providing more thorough experimental evidence, e.g., also include an evaluation of the predictive model's performance.
- Some missing details. I couldn't find reward curves for the BOHB, SMAC, DEHB methods. Moreover, the reward curves for the proposed method but without the max smoothing were also missing. The reward scale on the axes in Figures 18-22 was also missing (making it difficult to see whether the performance of algorithms is actually good, and also difficult to see whether the HPO found good configurations). Also, it was not written what kernel was used in equation 7 (the authors later clarified this for me, but not in the initial response, and the paper was also not updated). 
- The performance of all algorithms is poor on several environments (see figures 13 and 14). I believe the HPO optimization budget should be increased, so that it finds a good performance. The authors emphasized that their method works well for low budgets, but I don't see any good reason why they should withhold the results for high budgets. It would be important to know whether the new method is better for both low and high budgets, or whether it is only better in the early stages of HPO. Due to the use of the tabular benchmark that does not actually require running the RL algorithms, the computational speed of the experiments should be fast, and I don't see an issue with experimenting with higher budgets.
- **Evaluation** The exact method of evaluation is still a bit unclear to me due to miscommunication between me and the authors. But my current understanding is the following: There is a total maximum budget $B$, and each hyperparameter setting $i$ has their total allocated budget until the current step $b_i$. The database contains learning curves for each hyperparameter setting for $T > b_i$ steps. The algorithms work by selecting one configuration $i$, incrementing its budget to $b_i + \Delta b$ and querying the database for the performance at the budget $b_i + \Delta b$. This performance is stored in the history. I believe that these parts I was able to confirm from the code. How this history is finally used for evaluation, I am a bit unsure about, as the authors did not provide the code they used for postprocessing and plotting. I believe they take the max in this history, and take that as the evaluation of the algorithm. 

I perceive two issues with this evaluation method. First, the evaluation is only based on the performance during HPO with a policy that is not fully trained. In HPO we could consider two different ways of evaluating: evaluate the policy at the current budget, or evaluate the policy after fully training with the chosen hyperparameter. I believe that the current method of evaluating only the policy at the current budget is insufficient, as often we may be interested in the performance of the chosen hyperparameter rather than the currently best trained policy. For example, if a HPO method increments all hyperparameters by only a small fraction of $T$, and selects the best hyperparameter from there, the selection performance may be quite good, while because the policy is only trained for a little bit, the performance at the current budget will be poor. For this reason, I believe that the evaluation should at least include the final performance as well (perhaps providing both methods of evaluation would be good).

**Max** I had quite a bit of miscommunication with the authors about the max smoothing. My initial comment was about max smoothing during the budget increment in the interval $[b_i, b_i + \Delta b]$, and the authors responded that this does not affect the evaluation of the algorithm. This part I mostly agree with (although I believe that other comparisons with other smoothing methods would also be good, as I am skeptical of this method). However, my later comment is not about this smoothing. If my above explanation of the algorithm is correct, the history contains the final performance of each budget increment. This means that there will be multiple entries for each learning curve (if a hyperparameter is incremented multiple times). I believe for their evaluation, they pick the maximum value in the history, so essentially, it will be taking a max over multiple points from each learning curve. This would add a selection bias and overestimate the performance (as the method would select the point where the randomness made the reward be high). This would be particularly problematic when the increment size is different for the different HPO algorithms (e.g., in the comparison with the single fidelity methods). This explanation matches what is described in the algorithm 1, where the max is taken in line 9 (note that algorithm 1 is titled "Gray-Box HPO for RL" so I believe it corresponds to all their HPO algorithms). If the authors wish to include the performance comparison during training (i.e., not only after fully training as I suggested above), I believe they could mitigate this issue by comparing the performance at the highest budget for each hyperparameter (i.e., not the max in the history including all increments).

- The performance for the other algorithms is not provided for the max smoothing, so it remains unclear whether the new method performs well because of the max smoothing or because of the reward modeling. Here the authors did provide an ablation study showing that the max smoothing improved their performance. Moreover, in their rebuttal they argued that it is nonstandard to immediately use the new method for all previous algorithms as well. However, in this case it would at least be necessary to know how the reward modeling without the max smoothing compares to the previous algorithms. At the moment this is impossible to know, as the only results without max smoothing are rank comparisons in an ablation against the standard reward curve method. The work should at least have included reward curves for the method without max smoothing.

- Data processing and presentation could be better, e.g., more realistic errorbars.

Update 2
------------------------------
_________________

The authors performed substantial additional experiments, and provided a detailed new response, so I have increased my score to 6 and would be happy to see the work published. I refrained from setting the score higher because it is quite late in the review process, and it is no longer possible to update the paper, but I guess the changes for the camera ready will not be difficult if the paper is accepted.

Regarding the author response: they performed experiments looking at the predictive accuracy of the models, which also showed that the max smoothing improved the ranking accuracy while reducing the prediction accuracy (providing more evidence for the authors' intuition about why max smoothing may be useful). This could be further improved by showing a few visualizations of the predicted reward curves. 

They also provided the missing additional reward curves. Moreover, they explained that the performance is close to optimal on most environments (after looking at the plots on Fig 13 and 14 again, I guess the performance seems poor because the scale on the axis is inappropriate for some environments to properly see the performance).

The authors now also provided a clear explanation of their evaluation protocol as well as added the code for the post processing from where I was able to confirm the evaluation method, and it was fine.

They also performed new experiments trying out max-smoothing also with the other methods.

One remaining small issue in the paper are the errorbars. After looking at the code, it seems the errorbars are the standard deviation instead of the standard error (actually this was also written in 1 place in the paper), so I believe it should be simple to fix, and I'll give a suggestion in a comment below.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper considers the problem of hyper-parameter optimization for reinforcement learning algorithms. A Bayesian optimization based algorithm is proposed to tackle the problem where the key idea is to model the reward curve of a candidate hyper-parameter configuration with a generalized logistic function. Gaussian Processes are used as surrogate models with an augmented input feature space consisting of the hyper-parameters, budget allocated to the configuration and the reward curve estimate. Experiments are performed on multiple RL based algorithms.
strength_and_weaknesses: Strengths

- The paper considers an important problem as RL algorithms are well known to be finicky with the choice of their hyper-parameters.

- The release of source code is commendable and a good contribution towards reproducibility, which is commonly poor with hyper-parameter optimization algorithms.


Questions/Suggestions for improving the paper: 

- The discussion about some of the main points proposed in the paper is very limited. For example, the choice of reward model as generalized logistic function is introduced in few lines in the paper without any motivation. Please consider expanding discussion about why this is the right choice? What properties/assumption of underlying MDPs/reward model makes this an effective choice? What are the limitations of this choice? As another example, the smoothing of reward curves in Equation (1) is described in ad-hoc manner without any principle.



- Please provide more details about the training of the sigmoid coefficients and other hyper-parameters of the (multi-fidelity) Gaussian process model used in the proposed approach. 

- The experimental evaluation only shows the ranking of the proposed approach and the baselines. This gives limited information about the actual performance of the proposed approach. Please show runs of individual algorithms for a particular benchmark. 

- The search space consists of mainly two/three hyper-parameters which are chosen from a discrete set of values which seems too small. Can we employ the proposed approach on a continuous search space to robustly test it's effectiveness?

- The notion of gray-box has already been defined for a range of problem settings in the literature (please see [1] and references therein). Redefining the definition to `the concept of cheaper and approximate evaluations of the performance of hyperparameter configurations` makes it too specific and restricted. Please consider expanding this definition or atleast contextualizing the problem with respect to the existing (broad) literature.


[1] Astudillo, Raul, and Peter I. Frazier. "Thinking inside the box: A tutorial on grey-box Bayesian optimization." In 2021 Winter Simulation Conference (WSC), pp. 1-15. IEEE, 2021.

clarity,_quality,_novelty_and_reproducibility: The paper is mostly well-written. 
summary_of_the_review: Overall, the algorithm proposed in the paper is limited in novelty and mostly an application of existing Bayesian optimization techniques for hyper-parameter optimization of reinforcement learning algorithms. I want to stress that the limited novelty alone is not necessarily a bad thing and applying existing techniques to effectively solve an important problem is quite useful. However, most of the algorithmic components employed in the paper are not motivated and lack principled justification. Please consider working on this aspect to improve the paper's quality.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a new hyperparameter optimization (HPO) method for RL algorithms.

It takes advantage of the ability to run an RL algorithm using a smaller number of iterations than the algorithm requires to achieve its full performance. This produces a trace of the reward (the "learning curve") that can be extrapolated to quickly judge the quality of a set of hyperparameters. To take advantage of this, the paper proposes a novel kernel for GP regression that can extrapolate learning curves. It uses this with the expected improvement acquisition function within a Bayesian optimization algorithm. SOTA performance is demonstrated on problems from a collection of RL environments using 3 benchmark methods from the recent literature on HPO.

Specifically, the novel aspect of the kernel (equation 7) is that it uses an additional feature (equation 6), which is a Richard's curve whose parameters are given by a multi-layer perceptron taking the hyperparameters as input.


Strengths
- HPO for RL is important and is understudied
- Using learning curves is important and natural in HPO for RL
- Experimental evaluation shows a good improvement over recent methods on a wide collection of problems

Weaknesses
- Novelty --- BO with learning curves is well-studied for hyperparameter optimization in supervised learning (e.g., Falkner et al. 2018 as cited in the paper) and the approach taken here is not especially novel. This is the opinion of the AC --- some reviewers were not particularly concerned about novelty.
- Incomplete and unclear justification for the methodologically novel aspect of the approach. This is described in the review that I added.
- Lack of clarity in section 3 (beyond the specific part of section 3 discussed above). For example, the specific acquisition function used is not stated in the paper.

Minor Weaknesses 
- The reviewers remain concerned about the way in which error bars were produced, even after discussion with the authors
- Use of the term "gray-box" that is overly specific and restricted, as explained by reviewer 8pAg
justification_for_why_not_higher_score: - Weak methodological novelty
- Lack of clarity, particularly when describing the aspects of the methodology that is novel, but also throughout Section 3
- Some lingering questions about the experimental evaluation
justification_for_why_not_lower_score: - Important problem
- Method outperforms recently published benchmark methods on a wide variety of problems
Decision: Accept: poster