Gray-Box Gaussian Processes for Automated Reinforcement Learning
Authors: Gresa Shala
André Biedenkapp
Frank Hutter
Josif Grabocka
Venue: ICLR 2023 poster (2023)

Abstract: Despite having achieved spectacular milestones in an array of important real-world applications, most Reinforcement Learning (RL) methods are very brittle concerning their hyperparameters. Notwithstanding the crucial importance of setting the hyperparameters in training state-of-the-art agents, the task of hyperparameter optimization (HPO) in RL is understudied. In this paper, we propose a novel gray-box Bayesian Optimization technique for HPO in RL, that enriches Gaussian Processes with reward curve estimations based on generalized logistic functions. In a very large-scale experimental protocol, comprising 5 popular RL methods (DDPG, A2C, PPO, SAC, TD3), dozens of environments (Atari, Mujoco), and 7 HPO baselines, we demonstrate that our method significantly outperforms current HPO practices in RL.

Published as a conference paper at ICLR 2023
GRAY-BOX GAUSSIAN PROCESSES FOR AUTOMATED REINFORCEMENT LEARNING
Gresa Shala', André Biedenkapp', Frank Hutter' Josif Grabocka'
'Department of Computer Science. University of Freiburg
Bosch Center for Artificial Intelligence (shalag, biedenka, fh, grabocka (cs. uni frriiuure de
ABSTRACT
Despite having achieved spectacular milestones in an array of important real- world applications, most Reinforcement Learning (RL) methods are very brit- tle concerning their hyperparameters. Notwithstanding the crucial importance of setting the hyperparameters in training state-of-the-an agents, the task of hyper- parameter optimization (HPO) in RL. understudied. In this paper, we propose novel gray-box Bayesian Optimization technique for HPO in RL, that enriches Gaussian Processes with reward curve estimations based on generalized logis- tie functions. In very large-scale experimental protocol, comprising 5 popular RL methods (DDPG, A2C, PPO, SAC, TD3). 22 environments (OpenAl Gym: Mujoco, Atari, Classic Control), and 7 HPO baselines, we demonstrate that our method significantly outperforms current HPO practices in RL..
INTRODUCTION
While Reinforcement Learning (RL) has celebrated amazing successes in many applications (Miih et al. 2015] Silver et al.] [2166] OpenAl 2018
Andrychowicz et al.) 2020 Degrave et al.) 2022). remains very brittle (Henderson al., 2018 Engstrom et al., 2020). The successes of RL are achieved by leading experts in the field with many years of expertise in the "art" of RL, but the field does not yet provide a technology that broadly yields successes off the shelf. A crucial hindrance for both broader impact and faster progress in research is that an RL algorithm that has been well-tuned for one problem does not necessarily work for another one; especially, optimal hyperparameters are environment-speccfic and must be carefully tuned in order to yield strong performance.
Despite the crucial importance of strong hyperparameter settings in RL (Henderson et al., 2018 Chen et al.] 2018 Zhang al.) [202 Andrychowicz et al. (02T). the field of hyperparameter opti- mization (HPO) for RL is understudied. The field is largely dominated by manual tuning, compu- tationally expensive hyperparameter sweeps, or iauulation-based training which trains many agents in parallel that exchange hyperparameters and states (Jaderberg et al. (017). While these meth- ods are feasible for large industrial research labs, they are costly, substantially increase the CO2 footprint of artificial intelligence research (Dhar 2020). and make it very hard for smaller indus- trial and academic labs to partake RL research. In this paper, we address this gap, developing a computationally efficient yet robust HPO method for RL.
The method we propose exploits the fact that reward curves tend to have similar shapes. As a result, future rewards an agent collects with given hyperparameter setting can be predicted quite well based on initial rewards, providing computationally cheap mechanism to compare hyperparame- ter settings against each other. We combine this insight in novel gray-boy Bayesian optimization method that includes a parametric reward curve extrapolation layer in a neural network for comput- ing a Gaussian process kernel.
In a large-scale empirical evaluation using 5 popular RL methods (DDPG, A2C, PPO, SAC, TD3), 22 environments (OpenAI Gym: Mujoco, Atari, Classic Control). and 7 HPO baselines, we demon- strate that our resulting method, the Reward- Curve Gaussian Process (cCGP).yielde state-of-the-ar performance across the board. In summary, our contributions are as follows:

---

Published as a conference paper at ICLR 2023
We introduce novel method for extrapolating initial reward curves of RL agents with given hyperparameters based on partial learning curves with different hyperparameters.
We introduce RCGP, a novel Bayesian optimization method that exploits such predictions allocate more budget the most promising hyperparameter settings.
We carry out the most comprehensive experimental analysis of HPO for RL we are aware of to date (including 5 popular RL agents, 22 environments and 8 methods), concluding that RGCP sets new state of the art for optimizing RL hyperparameters in low compute budgets.
To ensure reproducibility (another issue in modern RL) and broad use of RGCP, all our code is open-sourced at http:///......to.o.//ee.eaaeea//
2
RELATED WORK
RL training pipelines are complex and often brittle (Henderson et al. 2018 Engstrom et al., 2020 Andrychowicz al. 202T). This makes RL difficult to for novel applications. To mitigate this, automated reinforcement learning (AutoRL; Parker-Holder et al. 2022) aims to alleviate a human practitioner from the tedious and error prone task of manually setting up the RL pipeline.
While there exists different approaches to automate the choice of algorithm, architecture (Miao al. 2022) or even environment components (Gleave et 202T). in this work we focus on hy- perparameter optimization (HPO; |Feurer & Hutter 2019] Bischl et al.) (021]) for RL. There exist various approaches in the literature HPO for RI. (see, e.g., Friksson et al., 2003 [Chen et al.)
2018 Hertel et al., 2020 Ashraf et al., 2021] for detailed survey we refer to Parker-Holder et al. (2022) ). Due to the non- sot-staiionarity of RL training, in recent years, most applications of hyper- parameter optimization for RL have focused on dynamically adapting hyperparameters throughout the run. For example, population based training (PBT: Daderberg et al. (017)) and variants thereof (see, e.g., Franke et al., 2021 Parker- community. This style of HPO uses a population of agents to optimize their hyperparameters while training. Parts of the population are used to explore different hyperparameter settings while the rest are kept to exploit the so far best performing configurations. While this has proven a successful HPO method, drawback of population based methods is that they come with an increased compute cost due to needing to maintain a population of parallel agents. Thus, most extensions of PBT, such as PB2 Marker-Holder al., 2020), aim at reducing the required population size. Still, to guarantee sufficient exploration, larger populations might be required which makes such methods hard to use with small compute budgets.
al. 2020) have found more wide-spread use in the
In the field of automated machine learning (AutoML; Hutter et al. 2019). multi-fidelity optimiza- tion has gained popularity to reduce the cost of the optimization procedure. Such methods (see, e.g., Kandasamy et al. 2017 Li et al., 2017 Klein et al.) 2017a] Falkner et al.] 2018 Li et al. 2020 Awad et al., 2021 leverage lower fidelities, such as dataset subsets, lower number of epochs or low numbers of repetitions, to quickly explore the configuration space. For the special case of num- ber of epochs as a fidelity, there also exists a rich literature on learning curve prediction (Swersky le al.) [014] Domhan et al. 2015 Baker et al., 2017]] Chandrashekaran & Lane 2017 Klein et al.) 2017 7b. Wistuba et al. (022). Multi-fidelity optimization typically evaluates the most promising configurations on higher fidelities, including the full budget. This style of optimization has proven a cost-efficient way of doing HPO for many applications. Still, multi-fidelity optimization has been explored only little in the context of RL. We are only aware of three such works: Runge et al. (2019) used a multi-fidelity optimizer to tune the hyperparameters of a PPO agent (Schulman et al. 2017)
that was tasked with learning to design RNA, allowing the so-tuned agent to substantially improve over the state of the art. Nguyen et al. (2020) also modelled the training curves, providing a signal to guide the search. In the realm of model-based RL, it was shown that dynamic tuning methods such PBT can produce well- rell-performing policies but often fail to generate robust results whereas static multi-fidelitg approaches produced much more stable configurations that might not result in as high final rewards (Zhang et al. 2021). Crucially, however, these previous studies did not evaluate how multi celli-fiielity and PBT style methods compare in the low budget regime, a setting that is more realistic for most research groups
2

---

Published as a conference paper at ICLR 2023
3 PROPOSED METHOD
Hyperparameter Optimization (HPO) focuses on discovering the best hyperparameter configura- tion E of a Machine Learning method. Gray-box HPO refers to the concept of cheaper and approximate evaluations of the performance of hyperparameter configurations. For example, we can approximately evaluate the final reward of the configurations of a deep RL method system after every epoch of stochastic gradient descent (gray-bor evaluations) (Swersky et al.) [014:] Wisttbbl let al. 2022), instead of waiting for the full convergence (black-box evaluations). The reward after a budget b (i.e. after b epochs) is defined as R (A, A x N - RR. To address the noisieness of re- ward curves of RI. algorithms we smooth the curves using a best-so-far transformation. We average rewards based on windows of h training steps, and select the highest reward at any past window, as:
h
R (max) (A,b) = max
Σ ((A 1=1
OSE<Pbh h
From now on, we refer to the smoothed (mxxx) as the reward R. In addition, let us define the cost (e.g., wall-clock time) of evaluating configuration for specific budget as (A, x N + R+. We define the history of N evaluated configurations and the respective bud- get as (K) = {(A1, by, R (A1, (1),b)))..... ...., (Ax, by: R (Ax,bx)) bx))). A gray-box algorithm A a evaluate and its budget as (AK+1,bk+1)
policy that recommends the next configuration
A (H) where A (A x N RR,) + x N. Gray-box HPO formally focuses on poli- cies A that are sequentially executed for as many steps (denoted K) as needed to reach a total budget Q. The best policy discovers the configuration with the largest reward at any budget, as arg max A max K R ((As11, b(+1) == A (H) s.t. K = maxjeN, Ω E' (A,, b.).
Bayesian optimization (BO) is very popular HPO policy that sequentially recommends hyper- parameters to evaluate. BO operates in sequences of two steps: by (i) fitting a probabilistic regres- sion model to approximate the observed performances R(X,b) of the evaluated configurations and budgets in H; and (ii) applying an acquisition to select the next configuration to evaluate.
In the first step, train Gaussian Processes (Snoek et al.) 2012) approximate the observed performances (i.e. R(A, b) ~ GP (X, (A,
0)) by finding the optimal GP parameters 6* via MLE:
0 (H) arg max (AAARA()))) log (R(),b) b,b; 0)
At the second step, we use an acquisition function, e.g. Expected Improvement (Snock et al. 2012) A x N4 + R4. that scores how "well" previously unevaluated configuration might perform at a future budget, based on the estimation the GP fitted above. A naive gray-box BO can be formalized a special HPO policy, based on fitted GP with parameters 0* from history of i evaluations (t), that recommends the (i + 1)-th configuration as:
(A--1,b411) AO (or (HO) arg max (x,b; or (1)))) DEA
3.1 MULTI-FIDELITY GAUSSIAN PROCESSES WITH REWARD CURVES
A multi-fidelity Gaussian Process can be modeled as standard GP with an augmented feature vector
: [X, b) E A x N (Nguyen et al. 2020 Song et al. 2019 Swersky et al. 2013). From the history of evaluations (K) we define the training features 2; = X. by] and their respective targets yi R (>,, b:) for i E {l,..., K)., A kernel function measures the similarity of features as A (2,, 2;) (A x N)2 + R4. The aim of the GP
estimate the posterior distribution the unknown target of new observed test instance 2, [A,, 6.]. The covariance matrix between training features' pairs defined as K (z, 2) [k(2; 2; Iv.. Similarly, the covariance between test-to-training features
K (z.,2) 2) = [k< 2;) 2))... and the test-to-test one as K (z., 2.)) = 2.). Ultimately, the posterior prediction of the unknown test target is:
3

---

Published as a conference paper at ICLR 2023
Algorithm/ 1: Gray-Box HPO for RL
Input Search space A, initial design H (0), budget increment As. max budget per agent bmux
Output: Best hyperparameter configuration X
Evaluate initial configurations and budgets H H7
while still budger do
Fit GP on using Equations for estimating pA b), of b);
Run acquisition a(js, d) to select A := arg max a (1(Λ, bmar), (2 bmax));
Define the next budget until which to train the selected agent Anext;
A
=r (Anext ,., E H
freu = min
Ab
b, otherwise
Resume training agent with Anext until boest and measure reward R (Anex, Bnext);
Append history H +- H {(Aeet fnext R (A next )next));;
end
return Best configuration X with highest reward
max (X(' BI H
R
((z.) = (z., (K (z, o y: 024))) = K (2+, K 2) (K (2,2) + 1 -1 K (zr,2)"
(4)
(5)
It was recently pointed out that a sigmoidal relationship exists between the reward curve of Rein- forcement Learning methods and the optimization budget (Nguyen et al. 2020). In this paper, we model the reward curve R (X, b) of configuration À at budget b as generalized logistic function (Richard's curve) with five coefficients (Richards) [959). Furthermore, we do not naively fit one sigmoid function each reward curve for each hyperparameter configuration. Instead, we pro- pose to condition the sigmoid coefficients on the hyperparameter configurations within a multi-layer perceptron g A R2 with weights (where g(,, b; w); represents the i-th output neuron)
g ( 9 (A; w) (1 g (A; a(A:o)) b) 1/0(Aww),
w)) = ();w) +
(6)
In this paper, we propose novel GP that exploits the pattern of the reward curve of the RL algorithm, by introducing the sigmoidal reward curve of (Equation ()) We augment the feature space with the estimation of the reward curve as [X, bj + b,, R (X, b; w)]. Therefore, the kernel becomes:
our bi,,, w) to (AN,, br, R (As, by; w) (Ay. R (1,, by: wi:)))
We train the parameters w using the established machinery of kernel learning for GPs Wilson et al. 2016) and then use this GP for gray-box HPO (Nguyen et al.) 2020 Song al. 2019). We use Ex- pected Improvement (Snoek et al. 2012) the highest budget juax as acquisition function. For brevity, we omit the basics of BO here, however, we refer the interested reader to Snoek al. (2012). In this context, the reward curve model of Equation full (bmax) performance of an unknown configuration (i.e., [X, pmax]] [X, gmux R (X, ymax, w))), and enables the acquisition function of the BO algorithm to discover performant hyper-parameter parameter con- figurations. We conducted analysis the predictive accuracy of our surrogate in Appendix [B.6]
6 offers crucial information in estimating the
Our novel gox HPO method is summarized by the pseudocode of Algorithm [
We stress that concurrently training one agent for each hyperparameter configuration, but advance
4

---

Published as a conference paper at ICLR 2023
only training procedure time, using an intelligent selection mechanism based on Bayesian optimization. In the first stage, we fit a GP (line 3) using the aforementioned novel kernel func- tion that combines hyperparameter configurations, budgets, and estimated rewards (Equations 7 Afterward, we select the next configuration with the highest estimated acquisition at the end of the convergence (line 4). Then train the RI agent corresponding to the selected configuration for one more budget increment (c.g. continue training for A = 10th more training steps) and measure the observed reward the end of the next budget (lines 5-6). Note that line 5 defines the next budget for both new configurations (žλneM : (A ...) H) as well as existing ones. We add the evaluation to the history (line 7) and continue the BO procedure until no budget is left (line 3).
4 EXPERIMENTAL PROTOCOL
4.1 EXPERIMENTAL. SETUP
We focus on evaluating the performance of our proposed method, RCGP (Reward-Curve GP), for optimizing the hyperparameters of five popular model-free RL algorithms: PPO (Schulman et al., 2017). A2C Mnih et al. (2016), DDPG (Lillicrap et al. 2016), SAC (Haarnoja et | and TD3 (Fujimoto et al. 2018). In total, consider 22 distinct Gym (Brockman et al. 2016) environ- ments, grouped into the Atari (Bellemare et al.) 2013), Classic Control, and Mujoco Todorov et al.) 2012) categories. We denote the full list of environments and their respective action space types in Appendix Al and we list the search spaces for the hyperparameters of each RI. algorithm in Table
Table I: Search spaces for HPO of PPO, A2C, DDPG, SAC, and TD3.
Algorithm
Hyperparameters
Hyperparameter Values
Learning rate (logıo)
1- -6,
PPO
7
0.9,0.95.95,000 0.99, 1.0}
Clip
(0.2,0.3,0.4) (66.-5.-4,-3,
Learning rate (logı0)
A2C
?
(..8,0.90..95,000 1.0)
Learning rate (logıo)
(-6. -3, 0.8.0.9,009900088000911111.
DDPG
7
T
Learning rate (log10)
SAC
7
(0.8, 00,5,0.,,0.55,00998,00 0.99, 1.0)
(0.0001,0.001.0.0000
Learning rate (log
(- 6, -,,
TD3
7
{0.8, 0.9, 0.95 0.98, 0.99. 1.0)
T
0.000,,,.001,..0000
We evaluated static hyperparameter optimization (HPO) methods by querying AutoRL-Benct which is tabular benchmark for AutoRL that contains reward curves for three different random seeds belonging to runs of RL algorithms with every possible combination of hyperparameter val- ues from the search spaces shown in Table T For the dynamic HPO methods (PBT (Jaderberg al., 2017) and PB2 (Parker-Holder et al.] 2020), details in Section 4.2). we ran our own evaluations of the RL pipelines. In every environment, we set the budget for all baselines to the run-time equiva- lent of 10 full training procedures, based on the expected run-time figures of AutoRL-Bench. Each training procedure consists of 10 steps on the training environment. All methods are evaluated for ten seeds in each environment and RL algorithm. The plots show the mean and standard devi- ations the relative ranks of all methods, with the training timesteps in the x-axis. We detail the evaluation protocol in Appendix C and the procedure we use to generate the plots in Appendix D Furthermore, we included the code for evaluating the performance of all the HPO methods in our GitHub httppp/pwps.ggggt..o...ooooaaaaa
httsss//ggithbbb.co//reeeaaa
5

---

Published as a conference paper at ICLR 2023
ATARI
CONTROL
MUJOCO
3.0
3.0
3.0
2.5
2.0
2.0
2.0
1.0
1.0
1.0
0.00
0.25
0.50
0.75
1.00
0.0
0.5
1.0
0.25
0.50 Timesteps
0.75
1.00
Timesteps
1e7
Timesteps
1e7
1e7
GP
RS
RCGP
Figure 1: Rank comparison of RS, GP, and RCGP for the PPO search space in the Atari, Classic Control, and MuloCo environments.
ATARI
CONTROL
MUJOCo
3.0
3.0
2.5
2.5
1.5
1.5
1.5
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Timesteps
1e7
Timesteps
1e7
Timesteps
1e7
Figure 2: Rank comparison of RS, GP, and RCGP for the A2C search space in the Atari, Classic Control, and MuJoCo environments.
4.2 BASELINES
We focus on comparing the performance of RCGP to existing HPO approaches within given time budget. We compare against three types of baselines. The first type includes standard baselines that do not utilize fidelity information during optimization, namely:
Random Search (RS) (Bergstra & Bengio 2012) a simple and common HPO baseline.
It optimizes hyperparameters by selecting configurations uniformly random.
Bayesian optimization with Gaussian Proccesses (GP) (Snoek et al.) 2012) another standard HPO baseline, using GPs the surrogate model standard blackbox Bayesian optimization. We used a GPytorch (Gardner et al. 2018) implementation with Matern 5/2 kernel.
The second type of baselines consists of multi-fidelity baselines which exploit intermediate learning (a.k.a. reward) curve information. Concretely, we compare against the following multi-fidelity HPO techniques:
BOHB FFlkknr et al.) 2018 is a multi-fidelity HPO baseline that combines Bayesian optimization and Hyperband (Li et al.) 2017). It uses tree-based Parzen estima- tors (TPE) (Bergstra et al. 2011) as surrogate model for Bayesian optimization. We used the source code provided by the authors.
2022 is recent variant of BOHB that uses Random Forests (RF) as surrogate model. Here again, we used the implementation provided by the authors.
SMAC (Lindauer et al.
DEHB (Awad et al. 2021 is state-of-the-art multi-fidelity HPO baseline that combines Differential Evolution and Hyperband. We used the source code released by the authors.
The third type of baselines includes online HPO techniques (which apply different hyperparameter configurations within a single RL agent training procedure). We compare against two state-of-ther the-art online HPO methods in RI. that are based evolutionary search:

---

Published as a conference paper at ICLR 2023
ATARI
CONTROL
MUJOCO
Rank
0.0
0.5
1.5
0.0
0.5
1.0 Timesteps
1.5
2.0
0.00
0.25
0.50 Timesteps
0.75
1.00
Timesteps
1e7
1e7
BOHB
DEHB
SMAC
RCGP
Figure 3: Ranks of BOHB, SMAC, DEHB, and RCGP in Atari, Classic control and MuJoCo envi- roments for the PPO search space..
ATARI
CONTROL
MUJoco
Rank
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Timesteps
1e7
Timesteps
1e7
Timesteps
1e7
BOHB
SMAC
RCGP
Figure 4: Ranks of BOHB, SMAC, DEHB, and RCGP in Atari, Classic control and MuloCo envi- roments for the A2C search space.
Population-Based Training (PBT) (Jaderberg et al. (017)) is evolutionary HPO method that dynamically optimizes the hyperparameters during the run of the algorithm (i.e., RI. agent training). discards the worst-performing members of the population after number steps and replaces them with new hyperparameter configurations that are generated by perturbing the best-performing configuration. We used the PBT implementation in the Ray Tune library (Liaw et al.) 2018). To facilitate a fair comparison on small compute budget we follow the protocol of Parker-Holder al. (2020) and use population of 4 individuals.
Population-based bandits (PB2) (Parker-Holder et al. 2020) is a PBT-like dynamic HPO method. It replaces the random perturbation with a time-varying GP, as mechanism to identify well-performing regions of the search space. Again, we used the implementation of PB2 in the Ray Tune library (Liaw et al., 2018) with a population of individuals.
RESEARCH HYPOTHESES AND EXPERIMENTAL RESULTS
Using the reward curve information helps in discovering efficient hyperparameters for model-free RL algorithms in the low budget regime.
Hypothesis 1:
We compare the performance of RCGP to RS and GP, as standard HPO baselines which do not utilize learning information. We evaluate RS and GP for 10 full RL algorithm runs. Initially, RS, GP, and RCGP start the search with the same hyperparameter configurations sampled uniformly random. RCGP queries the learning curve of evaluation returns of these initial configurations for the smallest budget of 10$ steps on the training environment. RS, and GP, being black-box optimization methods, query AutoRL-BBecch for final evaluation returns after 10 training steps, for both the initial and subsequently suggested configurations. Figure [ shows the comparison on the PPO search space for the Atari, Classic Control, and Mujoco classes environments. Similarly, Figure 2 presents the A2C results, while Figure 9 (Appendix [B) the results the DDPG, SAC and TD3 search spaces. all the cases, RCGP outperforms RS and GP within the wall-clock time budget of experimental protocol. In Appendix [.2) we also compare to enhanced variants of the black-box HPO methods, which evaluate configurations at smaller budgets. We conclude that gray-box HPO is more efficient than black-box HPO in RI..

---

Published as a conference paper at ICLR 2023
ATARI
CONTROL
MUjoCo
2.5
2.5
2.0
2.0
2.0
1.5
1.5
Timesteps
le6
Timesteps
1e6
Timesteps
1e6
PB2
PBT
RCGP
Figure 5: Ranks of PBT, PB2, and RCGP for the PPO search space in the Atari, Classic Control, and Mujoco environments.
CONTROL
MUJOCo
2.5
2.5
Rank
2.0
2.0
1.5
1.5
1.0
1.0
1.0
0
1
2
3
0
0
2
Timesteps
1e6
Timesteps
1e6
Timesteps
1e6
PB2
PBT
RCGP
Figure 6: Ranks of PBT, PB2, and RCGP for the A2C search space in the Atari, Classic Control, and Mujoco environments.
Hypothesis 2: RCGP outperforms state-of-the-ar multi-fidelity HPO methods in optimizing the hyperparameters of model-free RL algorithms.
We compare the performance of RCGP to BOHB, SMAC and DEHB, as state-of thee-- multi- fidelity HPO baselines. We evaluate each method for the equivalent time of 10 full RI. algorithm runs. Initially, all four methods start the search with the same hyperparameter configurations sampled uniformly random. Figure shows the performance comparison the PPO search space for the Atari, Classic Control, and Mujoco classes of environments. On the other hand Figure and B) include the experiments all experiments, our method RCGP on average outperforms BOHB, SMAC, and DEHB the low budget regime of up to 10 full function evaluations. We therefore conclude that RCGP sets the state- of-the-art in gray-box HPO for RL. We also compared to variants of the multi- multi-fidelity HPO basclines that use max-smoothed reward curves, as detailed Appendix B.4 In addition, we conducted an analysis of the gain of running the HPO procedure for longer budgets, as detailed in Appendix [B.]
10 (Appendix
the A2C, DDPG, SAC, and TD3 search spaces. In
Hypothesis 3: Our method oulperforms PBT and PB2, the state-of-the-art IIPO in RL.
In this experiment, we compare our method RCGP to PBT and PB2, to assess the efficiency of our gray-box HPO technique against -fate-of-the-arr dynamic HPO methods. For ensuring fair compar- ison, we evaluated PB2 and PBT using the recommended population size of 4 (Parker-Holder et al.) 2020), leading
budget equivalence of 4 full training routines. addition, all three optimiza- tion methods use an initial design consisting of the same hyperparameter configurations sampled uniformly at random. Figure 5 shows the performance comparison on the PPO search space for the Atari, Classic Control, and Mujoco environments. The associated experiments of Figure and and (Appendix B include experiments on A2C, DDPG, SAC, and TD3. The plots show RCGP clearly outperforms PBT and PB2 in the low budget regime of up 4 full function evaluations. Although PBT and PB2 are able to dynamically configure the hyperparameters of an RL algorithm, they require extensive parallel resources, and thus perform sub-optimally the low budget regime.
8

---

Published as a conference paper at ICLR 2023
ATARI
CONTROL
MUJOCO
3.0
3.0
3.0
2.5
2.5
2.0
2.0
2.0
1.5
1.0
1.0
1.0
Timesteps
1e7
Timestaps
1e7
Timesteps
1e7
RCGP (no reward curve)
RCGP (no max-smoothing)
RCGP
Figure 7: Rank comparison of RCGP (i) without reward curve model, (ii) with raw reward curve information, and (iii with max-smoothing of the reward curve, for the PPO search space in the Atari, Classic Control, and Mujoco environments.
ATARI
CONTROL
MUJOCO
1.5
1.00
Timesteps
1e7
Timesteps
1e7
Timesteps
1e7
RCGP (no reward curve)
RCGP (no max-smoothing)
RCGP
Figure 8: Rank comparison of RCGP (i) without reward curve model, (ii) with raw reward curve information, and (iii) with max-smoothing of the reward curve curve, for the A2C search space in the Atari, Classic Control, and Mujoco environments.
5.1 ABLATING DESIGN CHOICES
Throughout the paper we based our technical novelty on the hypothesis that reward curves have predictable shapes, and as a result, we can model them accurately with generalized logistic functions (Equation 6). In this section, we ablate the effect of enriching the feature-space of our surrogate with the reward curve estimations. i.c. [A, bj [A, (A, b; w)]. The ablations of Figure 7 8 demonstrate that using our novel reward curve modeling offers a major boost on the quality of the optimization. In addition, ablate the effect of the max-smoothing transformation of the reward curves (Equation []]. The empirical results further demonstrate that smoothing the noisy reward curves improves the performance of RCGP the low-budget regime, especially as the early segments of reward curves are very noisy.
6 CONCLUSION
Reinforcement Learning (RL) is one of the premier research sub-areas of Machine Learning, due to the impressive achievements of modern RL methods. Unfortunately, the performance trained RL agents depends heavily on the choice of the methods' hyperparameters. In this paper we introduced a novel gray-box HPO method that fits Gaussian Processes (GP) to partially-observed reward curves. Our GP variant fuses hyperparameter configurations, budget information and reward curve models based on generalized logistic functions. In a large-scale experimental protocol we demonstrated that our proposed method significantly advances the state-of-the-art for HPO in RL. Especially, we largely outperform evolutionary search HPO methods in RL (PBT and PB2), as well as existing gray-box HPO techniques.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
We would like to acknowledge the grant awarded by the Eva-Mayr-Stint Stiftung. In addition, this research was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Founda- tion) under grant number 417962828. In addition, Josif Grabocka acknowledges the support of the BrainLinks- Toll center of excellence. We acknowledge funding by European Research Coun- cil (ERC) Consolidator Grant "Deep Learning 2.0" (grant no. 101045765). Funded by the European Union. Views and opinions expressed are however those the author(s) only and do not necessarily reflect those of the European Union the ERC. Neither the European Union nor the ERC can be held responsible for them. The authors acknowledge funding by The Carl Zeiss Foundation through the research network "Responsive and Scalable Learning for Robots Assisting Humans" (ReScaLe) of the University of Freiburg.
Funded by European Union
Reproducibility Statement. To ensure reproducibility, give detailed descriptions of the exper- imental setup including the search spaces, environments, and seeds Section 4 and Appendix [A] We also make available the code used generate the results that present this paper. The benchmark we use is publicly available, and we reference it correspondingly.
REFERENCES
M. Andrychowicz, B. Baker, M. Chociej, R. Józefowicz. B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba. Learning dexterous in-hand manipulation. International Journal of Roboties Re-
search, 39(1), 2020.
M. Andrychowicz, A. Raichuk, P. Stanczyk, M. Orsini, S. Girgin, R.el Marinier, L. Hussenot, M. Geist, O. Pietquin, M. Michalski, S. Gelly, and O. Bachem. What matters for on-policy deep actor-critic methods? a large-scale study. In Proceedings of the International Conference on Learning Representations (ICLR'21) 2021. Published online: iclr. cc
N. M. Ashraf, R. R. Mostafa, R. H. Sakr, and M. Z. Rashad. Optimizing hyperparameters of deep reinforcement learning for autonomous driving based on whale optimization algorithm. PLOS ONE, 16(6):1-24, 2021.
N. Awad, N. Mallik, and F. Hutter. DEHB: Evolutionary hyberband for scalable, robust and efficient Hyperparameter Optimization. In Z. Zhou (ed.), Proceedings of the 30th International Joint Conference on Artificial Intelligence (ICAA'21), pp. 2147-2153, 2021.
B. Baker, O. Gupta, R. Raskar, and N. Naik. Practical neural network performance prediction for carly stopping. arXiv:1705.1082 [cs. 2017.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal Artificial Intelligence Research, 47:253-279, 2013.
Bergstra and Y. Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13:281-305, 2012.
J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl. Algorithms for hyper-parameter optimization. In Shawe-Tayee Taylor, Zemel, P. Bartlett, Pereira, and K. Weinberger (eds.), Proceedings of the 24th International Conference on Advances in Neural Information Processing Systems (NeurIPS' 11), pp.
Curran Associates, 2011.
B. Bischl, M. Binder. M. Lang. T. Pielok, J. Richter, S. Coors, J. Thomas, T. Ullmann, M. Becker, A. Boulesteix, D. Deng, and M. Lindauer. Hyperparameter Optimization: Foundations, algo- rithms, best practices and open challenges. arXiv:2107.05847 [stat.I.), 2021.
10

---

Published as a conference paper at ICLR 2023
G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Ope- nAI gym. arXiv:1606.01540 [cs. 2016.
A. Chandrashekaran and I. Lane. Speeding up Hyper-parameter Optimization by Extrapola- tion of Learning Curves using Previous Builds. In M. Ceci, J. Hollmen, L. Todorovski, C. Vens, and S. Džeroski (cds.). Machine Learning and Knowledge Discovery in Databases (ECMLPKDD' volume 10534 of Lecture Notes in Computer Science, pp. 477-492. Springer, 2017.
Y. Chen, A. Huang, Z. Wang, Antonoglou, Schrittwieser, D. Silver, and N. de Freitas. Bayesian optimization in alphago. vrXiv:/8/2.0660 [cs.LG), 2018.
J. Degrave, F. Felici, J. Buchli, M. Neunert, B. Tracey, F. Carpanese, T. Ewalds, R. Hafner, A. Abdol- maleki,
las Casas, C. Donner, L. Fritz, C. Galperti, A. Huber, J. Keeling, M. Tsimpoukelli, J. Kay, A. Merle, Jean-M. Moret, S. Noury, F. Pesamosca, D. Pfau, O. Sauter, C. Sommariva, S. Coda, B. Duval, A. Fasoli, P. Kohli, K. Kavukcuoglu, D. Hassabis, and M. Riedmiller. Mag- netic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):414- 419, 2022.
Dhar. The carbon impact of artificial intelligence. Nat. Mach. Intell., 2(8):423-425) 2020.
Domhan, Springenberg, and Hutter. Speeding up automatic Hyperparameter Optimization of deep neural networks by extrapolation of learning curves. In Q. Yang and M. Wooldridge (eds.), Proceedings of the 24th International Join! Conference on Artificial Intelligence (IJCAI'15, pp- 3460-3468, 2015.
L. Engstrom, A. Ilyas, S. Santurkar, D. Tsipras, F. Janoos, L. Rudolph, and A. Madry. Implemen- tation matters in deep RL: case study on PPO and TRPO. In Proceedings of the International Conference on Learning Representations (ICLR'20), 2020. Published online: iclr. cc
A. Eriksson, G. Capi, and K. Doya. Evolution of meta- pertapparamtters in reinforcement learning al- gorithm. In Proceedings of the International Conference on Intelligent Robots and Systems (IROS 003), pp. 412-411 417. IEEE, 2003.
S. Falkner, A. Klein, and F. Hutter. BOHB: Robust and efficient Hyperparameter Optimization at scale. In Dy and A. Krause (eds.), Proceedings of the 35th International Conference on Machine Learning (ICML'18) volume 80, pp. 1437-1446. Proceedings of Machine Learning Research, 2018.
M. Feurer and F. Hutter. Hyperparameter Optimization. In Hutter et al. 2019). chapter 1, pp. 3-38. Available for free at http://automl..rg//e
J. Franke, G. Köhler, A. Biedenkapp. and Hutter. Sample-efficient automated deep reinforcement learning. In Proceedings of the International Conference Learning Representations (ICLR'21) 2021. Published online: licrr.co
S. Fujimoto, H. van Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In J. Dy and A. Krause (eds.), Proceedings of the 35th International Conference Machine Learning (1CMI'18), volume 80, pp. 1587-1596. Proceedings of Machine Learning Research, 2018.
J. Gardner, G. Pleiss, K. Q Weinberger, D. Bindel, and A. G. Wilson. Gpytorch: Blackbox matrix- matrix gaussian process inference with gpu acceleration. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gamett (eds.), Proceedings of the s1 International Con- ference on Advances in Neural Information Processing Systems (NeurIPS' 18), Curran Associates, 2018.
A. Gleave, M. D Dennis, S. Legg, S. Russell, and J. Leike. Quantifying differences in reward func- tions. In Proceedings of the International Conference on Learning Representations (ICLR'2I), 2021. Published online: fclr.
11

---

Published as a conference paper at ICLR 2023
T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In J. Dy and A. Krause (eds.), Proceedings of the 35th International Conference on Machine Learning (ICML (8), volume 80, pp. 1861-1870. Proceedings of Machine Learning Research, 2018
Henderson, R. Islam, Bachman, J. Pineau, D. Precup, and D. Meger. Deep reinforcement learning that matters. In S. Mcllraith and K. Weinberger (eds.), Proceedings of the Thirty-Second Conference on Artificial Intelligence (AAAI'18). AAAI Press, 2018.
L. Hertel, P. Baldi, and D. L. Gillen. Quantity vs. quality: On hyperparameter optimization for deep reinforcement learning. 20X1:20077.1464 [cs.LG], 2020.
F. Hutter, L. Kotthoff, and J. Vanschoren (eds.). Automated Machine Learning: Methods, Systems, Challenges. Springer, 2019. Available for free at http://auutoml.org/bbon
M. Jaderberg, V. Dalibard, S. Osindero, W. Czarnecki, J. Donahue, A. Razavi, O. Vinyals, T. Green,
I. Dunning, K. Simonyan, C. Fernando, and K. Kavukcuoglu. Population based training of neural networks. 171iv: 1.09446 [cs.LG], 2017.
K. Kandasamy, G. Dasarathy, Schneider, and B. Póczos. Multi-fidelity Bayesian Optimisation with Continuous Approximations. In D. Precup and Y. Teh (eds.), Proceedings of the 34th Inter- national Conference on Machine Learning (ICML (T), volume 70, pp. 1799-1809 1808. Proceedings of Machine Learning Research, 2017.
A. Klein, S. Falkner, S. Bartels, P. Hennig, and F. Hutter. Fast bayesian Hyperparameter Optimiza- tion on large datasets. Electronic Journal of Statistics,
2017a.
A. Klein, Falkner, J. Springenberg, and F. Hutter. Learning curve prediction with Bayesian neural networks. In Proceedings of the International Conference on Learning Representations (I), 2017b. Published online: iclr.ce
L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar. Hyperband: Bandit-based configuration evaluation for Hyperparameter Optimization. In Proceedings of the International Conference on Learning Representations (ICLR' 17), 2017. Published online: iclr. cc
L. Li, K. G. Jamieson, A. Rostamizadeh, E. Gonina, J. Ben-tzur, M. Hardt, B. Recht, and A. Tal- walkar. A system for massively parallel hyperparameter tuning. In I. S. Dhillon, D. S. Papail- iopoulos, and V. Sze (eds.), Proceedings of Machine Learning and Systems 2020 (MLSys 20). mlsys.org, 2020.
R. Liaw, E. Liang, R. Nishihara, P. Moritz, J. E. Gonzalez, and I. Stoica. Tune: A research platform for distributed model selection and training. arXiv:1807.05116 [cs.LG), 2018.
T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Contin- uous control with deep reinforcement learning. In Proceedings of the International Conference on Learning Representations (ICLR' 16), 2016. Published online: ficcrieee .c
M. Lindauer, K. Eggensperger, M. Feurer, A. Biedenkapp. D. Deng. C. Benjamins, T. Ruhkopf, R. Sass, and F. Hutter. SMAC3: A versatile bayesian optimization package for Hyperparameter Optimization. Journal of Machine Learming Research (JMLR) MLOSS, 23(54):1-9, 2022.
Y. Miao, X. Song, J. D. Co-Reyes, D. Peng, S. Yue, E. Brevdo, and A. Faust. Differentiable archi- tecture search for reinforcement learning. In M. van der Schaar, M. Lindauer, I. Guyon, and H. H. Hoos (eds.), Proceedings of the First Conference on Automated Machine Learning (AutoML. 22)), 2022.
V. Mnih, K. Kavukcuuglu, D. Silver, A. Rusu, J. Veness, M. Bellemare, A. Graves, M. Riedmiller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529- 533, 02 2015.
12

---

Published as a conference paper at ICLR 2023
V. Mnih, A. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In M. Balcan and K. Weinberger (eds.), Proceedings of the 33rd International Conference Machine Learning (ICML' (EMLL17), volume 48, pp. 1928-1937. Proceedings Machine Learning Research, 2016.
V. Nguyen, S. Schulze, and M. A. Osborne. Bayesian optimization for iterative learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.-F. Balcan, and H. Lin (eds.), Proceedings of the 33rd International Conference Advances in Neural Information Processing Systems (NeurlPS'20). Curran Associates, 2020.
OpenAl. Openai five. fetps://bles httppp///hboog.oppaaa....oo/oopp five/ 2018.
ctm
Parker-Holder, Nguyen, and Roberts. Provably efficient online Hyperparameter Optimiza- tion with population- based bandits. In H. Larochelle, M. Ranzato, R. Hadsell, M.-F. Balcan, and H. Lin (eds.), Proceedings of the 33rd International Conference on Advances in Neural Informa- tion Processing Systems (NeurIPS' 20). Curran Associates, 2020.
Parker-Holder, Rajan, X. Song, A. Biedenkapp. Y. Miao, Eimer, Zhang, V. Nguyen, R. Calandra, A. Faust, F. Hutter, and M. Lindauer. Automated reinforcement learning (AutoRI)): A survey and open problems. Journal of Artificial Intelligence Research (JAIR), 74:517-568, 2022.
Richards. A Flexible Growth Function for Empirical Use. Journal of Experimental Botany, 10 (2):990-301, 06 1959.
Runge. D. Stoll, Falkner, and Hutter. Learning to Design RNA. In Proceedings of the Inter- national Conference on Learning Representations (ICLR" 19), 2019. Published online: iclr. cc.
Schulman, Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization [cs. 2017.
algorithms.
D. Silver, A. Huang, C. Maddison, A. Guez, I.. Sifre, G. Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484-48) 2016.
J. Snoek, H. Larochelle, and R. Adams. Practical Bayesian optimization of machine learning al- gorithms. In P. Bartlett, F. Pereira, C. Burges, L. Bottou, and K. Weinberger (eds.), Proceed- ings of the 25th International Conference on Advances Neural Information Processing Systems (NeurIPS'12) pp. 2960-2968, Curran Associates, 2012.
J. Song, Y. Chen, and Y Yue. A general framework for multi-fidelity bayesian optimization with gaussian processes. In K. Chaudhuri and M. Sugiyama (eds.), ). Proceedings of the 22nd Interna- tional Conference on Artificial Intelligence and Statistics (AISTATS' 19), volume 89, pp. 3158- 3167. PMLR, 2019.
K. Swersky, J. Snoek, and R. P. Adams. Multi-task bayesian optimization. In C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger (eds.), Proceedings of the 26th International Conference on Advances in Neural Information Processing Systems (NeurIPS'13), volume 26, pp. 2004-2222 20122. Curran Associates, 2013.
K. Swersky, J. Snoek, and R. Adams. Freeze-thaw Bayesian optimization. [stats. 2014.
arXiv:1406.3896
E. Todorov, T. Erez, and Y. Tassa. MuJoCo: A physics engine for model-based control. In Inter- national Conference on Intelligent Robots and Systems (IROS' (IRO pp. 5026-500000000 -0333, ieeecis, IEEE, 2012.
A. G. Wilson, Z. Hu, R. Salakhutdinov, and P. Xing. Deep kernel learning. In A. Gretton and C. Robert (eds.), Proceedings of the Seventeenth International Conference Artificial Intelli- gence and Statistics (AISTATS' 16), volume 51, pp. 370-378. Proceedings of Machine Learning Research, 2016.
13

---

Published as a conference paper at ICLR 2023
M. Wistuba, A. Kadra, and J. Grabocka. Supervising the multi-fidelityy race of hyperparameter configurations. In Proceedings of the 35th International Conference on Advances Neural In- formation Processing Systems (NeurIPS'22) 222). Curran Associates, 2022.
B. Zhang, R. Rajan, L. Pineda, N. Lambert, A. Biedenkapp. K. Chua, F. Hutter, and R. Calandra. On the importance of Hyperparameter Optimization for model-based reinforcement learning. In A. Banerjee and K. Fukumizu (eds.), Proceedings of the 24th International Conference on Artifi- cial Intelligence and Statistics (AISTATS' 21), pp. 4015-4023, Proceedings of Machine Learning Research, 2021.
14

---

Published as a conference paper at ICLR 2023
A RL ENVIRONMENTS
Table 2: List of environments for the experiments.
Environment Class
Environment Name
Action Space
Pong-vO
Alien-vO
BankHeist-v0
BeamRider-v0
Breakout-v0
Enduro-v0
Phoenix-v0
Atari
Seaquest-vO
Discrete
SpaceInvaders-v0
Riverraid-v0
Tennis-v0
Skiing-v
Boxing-v0
Bowling-v0
Asteroids-v0
CartPole-vl
MountainCar-vo
Discrete
Classic Control
Acrobot-v1
Pendulum-vO
Continuous
Ant-v2
MuJoCo
Hopper-v2
Continuous
Humanoid-v2
B ADDITIONAL EXPERIMENTAL. RESULTS
In this Section we present additional experimental results, including other search spaces (DDPG, SAC, and TD3) in B.I comparisons to the black-box HPO baselines modified to run suggested configurations for smaller budget than the budget of full RI. algorithm training procedure in B.2 comparison to PBT running continuous version of the PPO search space in B.3 as well as average episodic reward plots comparing PBT, PB2, SMAC, BOHB, DEHB, and RCGP in B.4 We additionally show average normalized regret plots comparing SMAC, BOHB, DEHB, and RCGP in B.5 In B.6 we evaluate the predictive performance of RCGP with the max-smoothing transforma- tion of the learning curves, and the version trained using -r-nrrasffrred learning curves.
B.1 RESULTS ON OTHER SEARCH SPACES
In Figures we show the average rank for the baselines and RCGP across 10 seeds and all MuloCo environments for the DDPG, SAC, and TD3 search spaces.
B.2 COMPARISON TO BLACK- BLXXK-BOX BASELINES
In Figures 12 protocol and RCGP. To allow the black-box baselines to search for more configurations within the maximum search budget, during the search the suggested configurations are evaluated for 1, 1 and of a full RL algorithm training procedure. During plotting, we still adhere the same plotting protocol as indicated in Algorithm
20
show the average rank for the black-box HPO baselines with modified search
B.3 SENSIBILITY OF SEARCH SPACE DISCRETIZATION
Figure 21 shows a comparison of RCGP, PBT optimizing on the discrete search space PPO (as specified in Table T). as well as PBT optimizing continuous version of this search space, with
15

---

Published as a conference paper at ICLR 2023
DDPG
SAC
TD3
Rank
0.25
0,50
0.75
1.00
0.25
0.50 Timesteps
0.00
0.25
0,50 Timesteps
0.75
Timesteps
1e7
le7
1e7
RS
RCGP
Figure 9: Rank comparison of RS, GP, and RCGP the MuJoCo enviroments for the DDPG, SAC, and TD3 search spaces.
DDPG
SAC
TD3
Bank
2
1
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.50 Timesteps
0.75
1.00 1e7
Timesteps
Timesteps
BOHB
DEHB
SMAC
RCGP
Figure 10: Ranks of BOHB, SMAC, DEHB, and RCGP in the MuJoCo enviroments for the DDPG, SAC, and TD3 search space.
the same bounds as the discrete version. We label the latter PBT-cont. The performance of PBT is visibly similar compared to PBT-cont on the MUJOCO environments, thus showing the sensibility of the discretization.
B.4 AVERAGE EPISODIC REWARD PLOTS
show the average episodic reward across the search budget for PBT, PB2, and RCGP in each environment for each search space. Figures (7 across the search budget for SMAC, BOHB, DEHB, and RCGP each environment for each search space. All these methods use non-transformed episodic reward curves. We show the comparison in terms of average episodic reward the grey-box baselines and RCGP using max-smoothed episodic
In Figures 22
26
277-3) 31 show the average episodic reward
reward curves in Figures 32 36
B.5 NORMALIZED REGRET PLOTS
A common metric in the HPO community for evaluating the distance of the discovered configuration (a.k.a. incumbent) to the optimal configuration (a.k.a. oracle) is the regret defined (oracle cumbent). If we let the HPO run longer we are going to discover the oracle in the best-case scenario. In our tabular benchmark, the oracle known (we can argmax the final returns of possible hyper- parameter configurations). therefore by computing the regret. Furthermore, a percentual difference variant is (oracle incumbent)/oracle. which shows the potential gain in percentage.
can directly assess the potential gain of running longer HPO
We show the performance of SMAC, BOHB, DEHB, and RCGP in terms average percentual regret for each environment in all search spaces in Figures 37 41 All four methods max- smoothed episodic reward curves. As can observed the percentual regret is between 10% and 1 %.
16

---

Published as a conference paper at ICLR 2023
DDPG
SAC
TD3
Rank
Mma M
2
Timesteps
1e6
Timesteps
1e6
Timestaps
1e6
PB2
PBT
RCGP
Figure 11: Ranks of PBT, PB2, and RCGP in the Mujoco enviroments for the TD3 search space.
ATARI
CONTROL
MUJOCO
3.0
3.0
2.5
2.5
2.0
2.0
2.0
1.0
1.0
1.0
Timesteps
1e6
Timesteps
1e6
Timesteps
1e6
GP
RS
RCGP
Figure 12: Average ranks of RS, GP, and RCGP in each environment for the PPO search space. RS and GP evaluate each suggested configuration during the search for g of the full budget (12 10a training timesteps).
Therefore, letting our HPO run till infinity will improve the results between 1% to 10% in the best- case scenario..
B.6 PREDICTIVE PERFORMANCE OF RCGP
In Figure 42 we show the predictive performance, and the rank correlation of RCGP. The plots show as y-axis (i) the predictive error (square error) and (ii) the Pearson correlation. The x-axis indicates the length of the reward curves which we train RCGP on, in order estimate the final return. The ground-truth final return is the final return of the max-smoothed reward curve.
In terms of predictive accuracy, the plots show that the error decreases as longer segment of the reward curve is observed. The squared error of RCGP using max-smoothed curves for training is higher than the version using the original reward curve train. However, in terms of the correlation the ground truth values of the final return, RCGP with max-smoothing
of ranks of the predicted has significantly larger correlation between estimated and final return values. As correlation more essential for HPO than the forecasting loss, these plots provide strong analysis explaining the superiority of our method in terms HPÖ performance.
C EVALUATION PROTOCOL
We detail the evaluation protocol for all baselines and RCGP in Algorithm|
2
D PLOTTING PROTOCOL
The detailed procedure that we use to generate the plots showing the rank of the methods is shown in Algorithm
17

---

Published as a conference paper at ICLR 2023
ATARI
CONTROL
MUJOCO
3.0
2.5
2.0
2.0
1.5-
1.5
1.0
1.0
1.0
Timesteps
1e6
Timesteps
1e6
Timestaps
1e6
RS
RCGP
Figure 13: Average ranks of RS, GP, and RCGP in each environment for the PPO search space. RS and GP evaluate each suggested configuration during the search for & of the full budget (25 10 training timesteps).
CONTROL
MUJOCo
3.0
2.0
1.5
0
2
4
2 Timesteps
4
Timesteps
1e6
Timesteps
GP
RS
RCGP
Figure 14: Average ranks of RS, GP, and RCGP in each environment for the PPO search space, RS and GP evaluate each suggested configuration during the search for 2 of the full budget (50 10a training timesteps).
Algorithm 2: Evaluation protocol
for seed in SEEDS do
for different budgets do
1. Search phase:
Run HPO for a budget and return the best hyperparameter configuration.
Evaluation phase:
Take the best configuration returned in step
Train it for the full budget.
Evaluate for 10 episodes and output the mean final return of these episodes.
end
10 end
E AUTORL-BENCH REWARD CURVES
To further motivate our design choice for modeling the reward curves using generalised logistic function, have plotted the reward curves in the AutoRL-Bench tabular benchmark. Figures 47 show the reward curves for the PPO, A2C, DDPG, SAC, and TD3 search spaces, respectively.
18

---

Published as a conference paper at ICLR 2023
CONTROL
MUJOCO
2,0
0
3
4 1e6
Timesteps
1e6
Timesteps
1e6
Timesteps
R5
RCGP
Figure 15: Average ranks of RS, GP, and RCGP in each environment for the A2C search space. RS and GP evaluate each suggested configurations during the search for of the full budget (12 10a training timesteps).
ATARI
CONTROL
MUJOCO
2.5
2.0
2.0
2.0
1.5
1.5
1.5
1.0
1.0
1.0
3
Timesteps
1e6
Timesteps
1e6
Timesteps
1e6
RS
RCGP
Figure 16: Average ranks RS, GP, and RCGP in each environment for the A2C search space. RS and GP evaluate each suggested configurations during the search for of the full budget (25 10a training timesteps).
Algorithm 3: Plotting protocol
for seed in SEEDS do
for different budgets B in the x-axis do Compute the y-axis by following steps (i) and (ii):
(i) for each environment in a benchmark do
for euch method do
Get final return (step 2 in Algorithm
end Compute the rank of each method for that environment based the returns.
end
10
(ii)
Compute the mean of the ranks across the environments step (i) for each method.
11
end
12 end
19

---

Published as a conference paper at ICLR 2023
ATARI
CONTROL
MUJOCO
2.5
2.5
2.5
2.0
2.0
2.0
1.5
1.5
1.5
1.0
1.0
1.0
Timesteps
1e6
Timesteps
1e6
Timesteps
1e6
GP
RS
RCGP
Figure 17: Average ranks RS, GP, and RCGP in each environment for the A2C search space. RS and GP evaluate each suggested configurations during the search for of the full budget (50 10a training timesteps).
DDPG
SAC
TD3
3.0
2.5
2.5
2.0
2.0
2.0
1.0
3
Timesteps
1e6
Timesteps
1e6
Timesteps
1e6
Figure 18: Average ranks of RS, GP, and RCGP in the MuJoCo enviroments for the DDPG, SAC,
and TD3 search spaces. RS and GP evaluate each suggested configurations during the search for of the full budget (12 10 training timesteps).
DDPG
SAC
TD3
3.0
3.0
3.0
2.5
2.5
1.5
1.0
1.0-
1.0
Timesteps
1e6
Timesteps
1e6
Timesteps
1e6
RS
RCGP
Figure 19: Average ranks of RS, GP, and RCGP in the MuJoCo enviroments for the DDPG, SAC, and TD3 search spaces. RS and GP evaluate each suggested configurations during the search for of the full budget (25 10 training timesteps).
20

---

Published as a conference paper at ICLR 2023
DDPG
SAC
TD3
3.0
1.5
1.0
1.0
Timesteps
1e6
Timesteps
1e6
Timesteps
1e6
GP
RS
RCGP
Figure 20: Average ranks of RS, GP, and RCGP in the MuJoCo enviroments for the DDPG, SAC, and TD3 search spaces. RS and GP evaluate each suggested configurations during the search for of the full budget (50 10 training timesteps).
MUJOCO
Rank
2
Timesteps
Figure 21: Ranks of RCGP, PBT (optimizing in the discrete search space of AutoRL-Bench), and PBT-cont (optimizing a continuous search space with the same boundaries as the AutoRL-Bench search space) in the MuJoCo enviroments for the PPO search space.
21

---

Published as a conference paper at ICLR 2023
Pong-v0
Alien-v0
BankHeist-vO
BeamRider-v0
Average Reward
1500
1000
1000
-10-
500
500
500
-20
0
0
5
Timesteps
Timesteps1e6
Timestepsle6
1e6
Breakout-v0
Enduro-vO
Phoenix-vO
Seaquest-v0
Average Reward
2000
2000
1000
1000
1000
1000
500
500
0
0
5
0
2
5
0
5
le6
1e6
1e6
1e6
Spacelnvaders-v0
Riverraid-v0
Tennis-v0
Skiing-v
Average Reward
2000
3000
1000
5000
2000
1000
500
1000
-5000
0-
0
5
0
2.5
5.0
0
5
1e6
1e6
le5
Boxing-v0
Bowling-v0
Asteroids-v0
CartPole-v1
Average Reward
5000
5000
5000
5000
0
-5000
-5000
5000
-5000
5
0.0
0.0
2.5
1e6
1e6
1e6
1e6
MountainCar.v0
Acrobot-v1
Pendulum-v0
Ant-v2
5000
5000
5000
Average Reward
0
-5000
-5000
5500
-5000
0.0
2.5
0.0
2.5
0.0
2.5
0.0
2.5
1e6
1e6
Timestepsle6
Hopper-v2
Humanoid-v2
5000
Average Reward
0
-5000
5000
0.0
2.5
0.0
2.5 Timestepsle6
e6
RCGP
PB2
PBT
22
Figure 22: Average reward of PBT, PB2, and RCGP in each enviroment for the PPO search space.

---

Published as a conference paper at ICLR 2023
Pong-v0
Alien-v0
BankHeist-v0
BeamRider-v0
1000
Average Reward
1000
1000
-15
500
500
-20
0
0.0
2.5
0.0
2.5
Timestepsle6
Timestepsle6
Timesteps1c6
1e6
Breakout-v0
Enduro-vO
Phoenix-v0
Seaquest-vo
1000
Average Reward
1000
1000
500
500
0
0.0
2.5
0.0
2.5
0.0
2.5
1e6
1e6
1e6
1e6
Spaceinvaders-v0
Riverraid-v0
Tennis-vO
Skiing-vO
Average Reward
2000
1000
1000
1000
500
0
-5000
0.0
2.5
0.0
2.5
5
2
1e6
1e6
1e6
Boxing-v0
Bowling-v0
Asteroids-v0
CartPole-vl
Average Reward
0
0
49
-5000
5000
-5000
-5000
2.5
0.0
le6
1e6
le6
1e6
MountainCar-vo
Acrobot-v1
Pendulum-vO
Ant-v2
Average Reward
2500
2500
2500
2500
0
-2500
2500
2500
-200
-5000
5000
5000
5000
0.0
2.5
0.0
2.5
0.0
2.5
2.5
1e6
1e6
Timestepsle6
Timestepsle6
Hopper-v2
Humanoid-v2
Average Reward
2500
2500
0
-2500
2500
0.0
2.5
Timestepsle6
Timestepsle6
RCGP
PB2
PBT
23
Figure 23: Average reward of PBT, PB2, and RCGP in each enviroment for the A2C search space.

---

Published as a conference paper at ICLR 2023
Ant-v2
Hopper-v2
Humanoid-v2
2000
Average Reward
2000
2000
1000
MA wwmm
-2000
-1000
Timesteps
1e6
Timesteps
1e6
Timesteps
1e6
RCGP
PB2
PBT
Figure 24: Average reward of PBT, PB2, and RCGP in each of the MuJoCo enviroments for the DDPG search space.
Ant-v2
Hopper-v2
Humanoid-v2
6000
6000
Average Reward
4000
4000
4000
2000
2000
2000
4
0
Timesteps
le6
Timesteps
1e6
Timesteps
1e6
RCGP
PB2
PBT
Figure 25: Average reward of PBT, PB2, and RCGP in each of the MuloCo enviroments for the SAC search space.
Ant-v2
Hopper-v2
Humanoid-v2
Average Reward
4000
4000
4000
2000
2000
2000
2
2
4
0
Timesteps
1e6
Timesteps
1e6
Timesteps
RCGP
PB2
PBT
Figure 26: Average reward of PBT, PB2, and RCGP in each of the the MuloCo enviroments for the TD3 search space.
24

---

Published as a conference paper at ICLR 2023
Pong-vO
Alien-v0
BankHeist-v0
BeamRider-v0
Average Reward
1000
1500
1000
1000
-10
500
500
-20
0
0
1
Timestepsle7
Timestepsle7
Timestepsle7
1c7
Breakout-v0
Enduro-vO
Phoenix-vO
Seaquest-v0
Average Reward
2000
2000
1000
1000
1000
1000
500
500
0
0
1
0
1
1e7
1e6
1e7
1e7
Spaceinvaders-v
Riverraid-vO
Tennis-v0
Skiing-vO
3000
Average Reward
5000
2000
2000
1000
1000
5000
0
0
1
0
5
1
1e7
1e6
1e7
Boxing-v0
Bowling-vO
Asteroids-v0
CartPole-v1
Average Reward
5000
5000
5000
5000
-5000
-5000
-5000
5000
1e7
1e7
1e7
1e7
MountainCar-vo
Acrobot-v1
Pendulum-vO
Ant-v2
5000
5000
5000
5000
Average Reward
-5000
-5000
-5000
-5000
2
1e7
1e7
Timestepsle7
Timestepsle7
Hopper-v2
Humanoid-v2
5000
Average Reward
55000
Timestepsle7
Timestepsle7
SMAC
DEHB
RCGP
BOHB
25
Figure 27: Average reward of SMAC, BOHE, DEHB, and RCGP in each environment for the PPO search space using con-transformed episodic reward curves.

---

Published as a conference paper at ICLR 2023
Pong-vO
Alien-vO
BankHeist-v0
BeamRider-v0
-12.5
Average Reward
1000
-15.0
-17.5
500
500
-20.0
0
0
Timestepsec6
Timestepa
Timestepsle7
1c6
Breakout-v0
Enduro-vO
Phoenix-v0
Seaquest-v0
Average Reward
1000
1000
2000
2000
500
500
1000
1000
0
5
1e7
1e6
1e6
1e7
Spacelnvaders-vO
Riverraid-v0
Tennis-v0
Skiing-vO
Average Reward
2000
2000
5000
1000
1000
1000
-5000
1e7
1e6
Boxing-v0
Bowling-vO
Asteroids-v0
CartPole-v1
Average Reward
5000
5000
5000
5000
-5000
-5000
-5000
-5000
le6
le6
1e7
1e7
MountainCar-vo
Acrobot-v1
Pendulum-v0
Ant-v2
5000
5000
5000
5000
Average Reward
-5000
-5000
5000
-5000
1e7
1e7
Timestepsle7
Timestepsle7
Hopper-v2
Humanoid-v2
5000
5000
Average Reward
-5000
-5000
Timestepsle7
Timestepale7
SMAC
DEHB
RCGP
BOHB
26
Figure 28: Average reward of SMAC, BOHB, DEHB, and RCGP in each environment for the A2C search space using con-transformed episodic reward curves.

---

Published as a conference paper at ICLR 2023
Ant-v2
Hopper-v2
Humanoid-v2
Average Reward
10a
10a
10a
102
10a
102
10
10
0.0
0.5
1.0
0.0
0.5
1.0
0.5 Timesteps
1.0
Timesteps
1e7
Timesteps
1e7
1e7
SMAC
BOHB
DEHB
RCGP
Figure 29: Average reward of SMAC, BOHB, DEHB, and RCGP in each of the MuJoCo enviro- ments for the DDPG search space using non-transformed episodic reward curves.
Ant-v2
Hopper-v2
Humanoid v2
Average Reward
10
10a
10a
102
102
0.0
0.5
1.0
0.0
2.5
5.0
7.5
0.0
0.5
1.0
Timesteps
1e7
Timesteps
1e6
Timesteps
1e7
SMAC
ВОНВ
DEHB
RCGP
Figure 30: Average reward of SMAC, BOHB, DEHB, and RCGP in each of the MuJoCo enviro- ments for the SAC search space using con-transformed episodic reward curves.
Ant-v2
Hopper-v2
Humanoid-v2
Average Reward
103
10a
10a
102
102
104
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5 Timesteps
1.0
Timesteps
1e7
Timesteps
1e7
1e7
SMAC
BOHB
DEHB
RCGP
Figure 31: Average reward of SMAC, BOHB, DEHB, and RCGP in each of the the MuloCo envi- roments for the TD3 search space using non-tree non-transformed episodic reward curves.
27

---

Published as a conference paper at ICLR 2023
Pong-v0
Alien-vO
BankHeist-v0
Ber-Vo
Average Reward
0
1000
1500
1000
500
500
500
-20
0
0
Timestepal.7
Timestepsle7
Timestepsle7
1c7
Breakout-v0
Enduro-v0
Phoenix-vO
Seaquest-v0
Average Reward
1500
2000
2000
1000
1000
1000
1000
500
500
0
0
0
0
1e7
1e6
1e7
1e7
Spacelnvaders-v0
Riverraid-v0
Tennis-v0
Skiing-vO
Average Reward
2000
3000
5000
2000
2000
0
1000
1000
1000
5000
0
0
10000
0
5
1e7
1e7
1e7
Boxing-vO
Bowling-v0
Asteroids-v0
CartPole-v1
Average Reward
5000
5000
5000
5000
0
0
-5000
-5000
5000
5000
-10000
10000
10000
1e7
1e7
1e7
1e7
MountainCar-vo
Acrobot-v1
Pendulum-vo
Ant-v2
Average Reward
5000
5000
5000
5000
0
0
0
-5000
5000
5000
5000
1e7
1e7
Timestepsle7
Timestepsle7
Hopper-v2
Humanoid-v2
Average Reward
5000
5000
0
0
-5000
5000
Timestepålc7
Timestepsle7
SMAC
DEHB
RCGP
BOHB
28
Figure 32: Average reward of SMAC, BOHE, DEHB, and RCGP in each environment for the PPO search space using max-smoothed episodic reward curves.

---

Published as a conference paper at ICLR 2023
Pong-vO
Alien-vO
BankHeist-v0
BeamRider-v0
1000
Average Reward
1000
1000
-15
500
500
500
-20
0
Timestepsec6
Timestepa
Timestepsle7
lc6
Breakout-vO
Enduro-vO
Phoenix-v0
Seaquest-v0
Average Reward
1000
1000
2000
2000
500
500
1000
1000
0
0
5
1e7
1e6
1e6
1e7
Spacelnvaders-vo
Riverraid-v0
Tennis-v0
Skiing-vO
Average Reward
2000
2000
1000
1000
1000
-5000
1e7
1e6
Boxing-v0
Bowling-vO
Asteroids-v0
CartPole-v1
Average Reward
5000
5000
5000
5000
-5000
-5000
-5000
-5000
1e6
1e6
1e7
1e7
MountainCar-vo
Acrobot-v1
Pendulum-v0
Ant-v2
5000
5000
Average Reward
-5000
5000
5000
-5000
1e7
1e7
Timestepsle7
Timestepsle7
Hopper-v2
Humanoid-v2
5000
5000
Average Reward
-5000
55000
Timestepsle7
Timestopāle7
SMAC
DEHB
RCGP
BOHB
29
Figure 33: Average reward of SMAC, BOHB, DEHB, and RCGP in each environment for the A2C search space using max-smoothed episodic reward curves.

---

Published as a conference paper at ICLR 2023
Ant-v2
Hopper-v2
Humanoid-v2
Average Reward
10a
10a
102
102
10
10
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5 Timesteps
1.0
Timesteps
1e7
Timesteps
1e7
1e7
SMAC
BOHB
DEHB
RCGP
Figure 34: Average reward of SMAC, BOHB, DEHB, and RCGP in each of the MuJoCo enviro- ments for the DDPG search space using max-smoothed episodic reward curves.
Ant-v2
Hopper-v2
Humanoid v2
Average Reward
10a
10a
10a
102
102
0.0
0.5
1.0
0.0
2.5
5.0
7.5
0.0
0.5
1.0
Timesteps
1e7
Timesteps
1e6
Timesteps
1e7
SMAC
ВОНВ
DEHB
RCGP
Figure 35: Average reward of SMAC, BOHB, DEHB, and RCGP in each of the MuJoCo enviro- ments for the SAC search space using max-smoothed episodic reward curves.
Ant-v2
Hopper-v2
Humanoid-v2
Average Reward
103
10a
10a
102
10a
102
104
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5 Timesteps
1.0
Timesteps
1e7
Timesteps
1e7
1e7
SMAC
BOHB
DEHB
RCGP
Figure 36: Average reward of SMAC, BOHB, DEHB, and RCGP in each of the the MuJoCo envi- roments for the TD3 search space using max-smoothed episodic reward curves.
30

---

Published as a conference paper at ICLR 2023
Pong-vO
Alien-vO
BankHeist-vO
BeamRider-v0
Normalized Regret
100
10
10
10
10 1
10-2
10
10-2
10
10-2
Timestepsle7
Timestepslc7
Timestepsle7
1c7
Breakout-vo
Enduro-v0
Phoenix-vO
Seaquest-vO
Normalized Regret
100
10a
10
10
10-1
10-1
10-1
10-1
10-2
10-2
10-2
10-2
1
0
5
0
0
1
1c7
1e6
1e7
1e7
Spacelnvaders-v0
Riverraid-vO
Tennis-vO
Skiing-vO
Normalized Regret
10"
109
10
1
10-1
10-2
10-2
10-2
10-2
1
0
1e7
1e7
1e7
Boxing-v0
Bowling-v0
Asteroids-v0
CartPole-v1
10a
Normalized Regret
109
100
10
10-1
10-1
10-1
10-2
10-3
10
10-2
10-4
1e7
1e7
1e7
1e7
Car-vainCar-v0
Acrobot-v1
Pendulum-v0
Ant-v2
Normalized Regret
10
10a
10
10a
10-1
10-1
10-1
10-1
10-2
10-2
10-2
10-2
1e7
1e7
Timestep
Timestepsle7
Hopper-v2
Humanoid-v2
Normalized Regret
10"
100
10-1
10-1
Timestepsle7
Timestepsle7
SMAC
BOHB
DEHB
RCGP
31
Figure 37: Performance terms of normalized regret for SMAC, BOHB, DEHB, and RCGP in each environment for the PPO search space using max-smoothed episodic reward curves.

---

Published as a conference paper at ICLR 2023
Pong-vO
Alien-vO
BankHeist-v0
BeamRider-v0
Normalized Regret
10"
109
100
10-1
10-1
10-1
10-2
10-2
10-²
5
Timestepsle6
Timestepsle7
Timestepsle7
1c6
Breakout-vo
Enduro-vO
Phoenix-vO
Seaquest-v0
Normalized Regret
100
10a
10a
10a
10-1
10-1
10
10-2
10-2
1
5
5
1e7
1e6
1e6
1e7
SpaceInvaders-v0
Riverraid-vO
Tennis-vO
Skiing-v0
Normalized Regret
100
101
10
10-1_
1
10-2.
10
10-2
5
0
1e7
1e7
1e6
1e6
Boxing-vO
Bowling-v0
Asteroids-vO
CartPole-v1
10a
Normalized Regret
10
10a
10-1
10-2
10-2
10-2
10-3
0.0
2.5
1e6
1e6
1e7
1e7
MountainCar-v0
Acrobot-v1
Pendulum-v0
Ant-v2
Normalized Regret
100
109
10a
10a
10-2
10-2
10-2
10-2 -2
1e7
1e7
Timestepsle7
Timestepsle7
Hopper-v2
Humanoid-v2
Normalized Regret
10"
109
10-1
10-2
10-2
Timestepsle7
Timestepsle7
SMAC
BOHB
DEHB
RCGP
32
Figure 38: Performance terms of normalized regret for SMAC, BOHB, DEHB, and RCGP in each environment for the A2C search space using max-smoothed episodic reward curves.

---

Published as a conference paper at ICLR 2023
Ant-v2
Hopper-v2
Humanoid-v2
Normalized Regret
100
109
10
10-1
10-1
10
10-2
10 2
0.0
Timesteps
1e7
Timesteps
1e7
Timesteps
1e7
SMAC
BOHB
DEHB
RCGP
Figure 39; Performance terms normalized regret for SMAC, BOHB, DEHB, and RCGP in each of the MuJoCo enviroments for the DDPG search space using max-smoothed episodic reward curves.
Ant-v2
Hopper-v2
Humanoid-v2
Normalized m Regret
100
109
10a
10-1
10
0.0
0.5
1.0
0.0
2.5
5.0
7.5
0.0
0.5 Timesteps
1.0
Timesteps
1e7
Timesteps
1e6
1e7
SMAC
ВОНВ
DEHB
RCGP
Figure 40: Performance terms of normalized regret for SMAC, BOHB, DEHB, and RCGP in each of the MuloCo environments for the SAC search space using max-smoothed episodic reward curves.
Ant-v2
Hopper-v2
Humanoid-v2
Normalized Regret
100
10a
10a
10-1
10-1
10-2
10-2
10-2
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5 Timesteps
1.0
Timesteps
1e7
Timesteps
1e7
1e7
SMAC
BOHB
DEHB
RCGP
Figure 41: Performance in terms of normalized regret for SMAC, BOHB, DEHB, and RCGP in each of the MuloCo enviroments for the TD3 search space using max-smoothed episodic reward curves.
33

---

Published as a conference paper at ICLR 2023
PPO
A2C
10a
Squared Erro
10a
5a Squared Error
10-1
10-1
10-1
10-3
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
1e6
le6
Pearson Correlation
1.0
Pearson Correlation
1.0
0.5
0.5
RCGP (no max-transform)
0.0
(no max-transform)
RCGP
0.6
0.8
Timesteps
1e6
Timesteps
1e6
(a)
(b)
DDPG
SAC
10a
Squared Error
10-1
10-1
Squared
10-3
10
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
1e6
Pearson Correlation
1.0
Correlation
1.0
RCGP (no max-transform)
RCGP
0.5
0.5
RCGP (na max-transform)
RCGP
0.2
0.6
0.8
0.2
0.4
0.6
1.0
Timesteps
1e6
Timesteps
1e6
(c)
TD3
Squared Error
10-1
1
0.2
0.4
0.6
0.8
1.0
1e6
Pearson Correlation
1.0
RCGP (no max-transform)
RCGP
0.4
0.6
0.8
Timesteps
1e6
Figure 42: Predictive performance of RCGP in terms of squared emror and Pearson correlation co- efficient for the PPO, A2C, DDPG. SAC and, TD3 search spaces using max-transformed episodic reward curves.
34

---

Published as a conference paper at ICLR 2023
Pong-v0
Alien-vO
BankHeist-vO
BeamRider-v0
Reward
Breakout-v0
Enduro-vO
Phoenix-v0
Seaquest-vo
Reward
Spacelnvaders-vO
Riverraid-vO
Tennis-vO
Skiing-v
Reward
Boxing-vO
Bowling- vv0
Asteroids-v0
CartPole-v1
Reward
MountainCar-v0
Acrobot-v1
Pendulum-vo
Ant-v2
Reward
Timesteps
Timesteps
Hopper-v2
Humanoid-V -v2
Reward
Timesteps
Timesteps
Figure 43: Reward curves of PPO on the environments included in AutoRL-Bench.
35

---

Published as a conference paper at ICLR 2023
Pong-v0
Alien-vO
BankHeist-vO
BeamRider-vO
Reward
Breakout-v0
Enduro-vO
Phoenix-vO
Seaquest-vo
Reward
Spacelnvaders-vO
Riverraid-v0
Tennis-vO
Skiing-0
Reward
Boxing-vO
Bowling-V -v0
Asteroids-vO
CartPole-v1
Reward
MountainCar-vo
Acrobot-v1
Pendulum-ve
Ant-v2
Reward
Timesteps
Timesteps
Hopper-v2
Humanoid- v2
Reward
Timesteps
Timesteps
Figure 44: Reward curves of A2C on the environments included in AutoRL-Bench.
36

---

Published as a conference paper at ICLR 2023
Ant-v2
Hopper-v2
Humanoid-v2
Reward
Timesteps
Timesteps
Timesteps
Figure 45: Reward curves of DDPG the environments included in AutoRL--Bechh.
Ant-v2
Hopper-v2
Humanoid-v2
Reward
Timesteps
Timesteps
Timesteps
Figure 46: Reward curves of SAC on the environments included in AutoRL-Bench.
Ant-v2
Hopper-v2
Humanoid-v2
Reward
Timesteps
Timesteps
Timesteps
Figure 47: Reward curves of TD3 on the environments included in AutoRL-BBench
37


Reviews:
[{"review_id": "jd0mzJvRIy", "reviewer": "Reviewer_Etq6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a new approach to HPO tailored towards RL. Specifically, a HPO approach that is sufficiently sample efficient such that the approach is effective even with limited resources (research lab as opposed to data center) is sought after. Bayesian optimization is used, with a Gaussian process (GP) surrogate. The GP has a novel deep kernel which consists of a parametrized Richard's curve, where the five coefficients of the curve are what the trained deep neural network is used to predict. The purpose of the kernel is to represent, and sample-efficiently infer, reward curves for specific hyperparameter configurations. In an extensive evaluation, on several problem domains comparing with several different kinds of state-of-the-art HBO methods, the proposed approach is demonstrated to be highly effective and clearly outperform competing methods.", "summary_of_the_review": "The paper is well contained, well positioned and makes seemingly large and significant advances on an important problem for RL in general. It was a pleasure to read.\n\n---\n\n### Update\nThank you for engaging with the reviewers and clarifying many important things. However, after reading through the other reviews I reduce my score, mainly due to evaluation concerns as pointed out by h99m and eXNC. I still believe that the paper contain contributions of interest to the community, but the clarity and motivation for parts of the evaluation should probably be improved in accordance their comments. eXNC also raise important concerns in the latest comment & update, which affect my judgement. I am consequently still in favor of accept, but only marginally.", "strength_and_weaknesses": "Strength\n========\n\n* The context of the work is well described and the approach is well motivated given related works.\n* The approach is clever and attack the core problem of sample-efficient HPO for RL, namely to predict the continuation of reward curves given the training epochs so far. The connection to and use of multi-fidelity GP well made and highly suitable.\n* The evaluation is strong, with a variety of methods and environments.\n* Clear hypotheses, and interpretation of results in light of these.\n* The ablation study enriches the contributions.\n\nWeaknesses\n========\nNothing obvious", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "ZsPMYSXU2Xl", "reviewer": "Reviewer_eXNC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work proposes to incorporate the learning curve information in reinforcement learning as features in Bayesian optimization to more efficiently perform hyperparameter optimization.\n\nGiven an RL learning curve, one could fit a generalized logistic function on the curve to obtain the parameters of the curve as features that are used in a Gaussian process model. To generalize this procedure to hyperparameters where there might not already exist a learning curve, the authors propose to predict the parameters from a neural network that takes the hyperparameters as an input.\n\nExperiments are performed on an AutoRL-Bench a benchmark for automatic reinforcement learning. This benchmark includes learning curves for a set of hyperparameter values. And it is possible to test the performance of BO algorithms by querying the learning curves for the different hyperparameters without requiring to rerun the whole RL procedure, allowing for efficient computationally inexpensive benchmarking of BO algorithm in AutoRL. Note that this benchmark appears to be still under review and does not seem to have passed peer review yet.\n\nThey experimentally compared with many different BO algorithms including random search, GP based methods, population based methods. And the new method outperformed the baselines.\n\nThey also performed ablation studies on different components of their algorithm.", "summary_of_the_review": "The work includes comprehensive experiments on BO in RL showing that the newly proposed method of modeling the learning curves improves the performance. However, I am unsure that the AutoRL-Bench that they performed their experiments on is a suitable way to evaluate BO algorithms as it does not appear to have passed peer review. Moreover, the analysis of the data and presentation of the results could be improved, e.g., improving the errorbars or also displaying reward information. Taking the max of the rewards as done in this paper has also been criticized in prior research. Considering these points, I am leaning towards suggesting to reject.\n\nUpdate\n------------------------------\n_________________\n\nAfter seeing additional details in the updated paper, and more discussion with the authors, I perceive more issues in the paper than initially, and I have reduced my score to 3. The main additional issues are that I believe the performance of the algorithms should also be compared based on the performance after fully training with the currently selected best hyperparameter (as opposed to looking at the reward at the current budget), and that the performance on several of the environments is quite poor (as can be seen from the reward curves). Regarding the second issue, it would have been good to include results for larger hyperparameter optimization budgets (the table based benchmark allows quickly testing large budgets, so I guess there is no computational issue with including larger budgets).\n\nI list my main concerns below:\n- Only indirect evidence is provided that the predictive model is performing well. I pointed this out in my initial review by saying that the accuracy of the predictive model was not compared. The authors responded that the accuracy is not that important, and the ranking is more important. While this may be true, it does not address the intention of my comment, and if the ranking is more important, the work could have instead investigated the ranking accuracy. Other ways to provide direct evidence that the predictive model is performing well would have been to visualize the predictive results. Providing this kind of evidence is important to prove that the algorithm is working as intended. There could be any number of reasons why the performance of the algorithm appeared to be good (e.g., bugs in the code or inappropriate evaluation), and providing multi-faceted evidence is a good way to mitigate such concerns. It may be particularly good if the authors compare with a simple ranking method, e.g., ranking the hyperparameters based on the reward at the same budget, and compare whether the predictive model outperforms such a simple method.\n- The benchmark is not well established. In my original review, I said that the benchmark has not yet been published. The authors responded that after the reviewing started the paper was accepted for publication at a NeurIPS workshop. This does not mitigate the issue that the benchmark is not yet established in the community as a good way to evaluate algorithms as there are no prior works using this benchmark. Workshop papers are often work in progress that still have issues. The authors also asked whether I could provide any other benchmark for them to use. Actually, I don't necessarily think that the benchmark is bad, and it seems like a really good idea to have such a tabular benchmark for HPO; however, as it is very new, I am not convinced of its reliability. If the benchmark were well established, the requirement for additional evidence in the previous concern could be lessened; however, as the benchmark is not well established, I suggest that the authors should mitigate such a concern by providing more thorough experimental evidence, e.g., also include an evaluation of the predictive model's performance.\n- Some missing details. I couldn't find reward curves for the BOHB, SMAC, DEHB methods. Moreover, the reward curves for the proposed method but without the max smoothing were also missing. The reward scale on the axes in Figures 18-22 was also missing (making it difficult to see whether the performance of algorithms is actually good, and also difficult to see whether the HPO found good configurations). Also, it was not written what kernel was used in equation 7 (the authors later clarified this for me, but not in the initial response, and the paper was also not updated). \n- The performance of all algorithms is poor on several environments (see figures 13 and 14). I believe the HPO optimization budget should be increased, so that it finds a good performance. The authors emphasized that their method works well for low budgets, but I don't see any good reason why they should withhold the results for high budgets. It would be important to know whether the new method is better for both low and high budgets, or whether it is only better in the early stages of HPO. Due to the use of the tabular benchmark that does not actually require running the RL algorithms, the computational speed of the experiments should be fast, and I don't see an issue with experimenting with higher budgets.\n- **Evaluation** The exact method of evaluation is still a bit unclear to me due to miscommunication between me and the authors. But my current understanding is the following: There is a total maximum budget $B$, and each hyperparameter setting $i$ has their total allocated budget until the current step $b_i$. The database contains learning curves for each hyperparameter setting for $T > b_i$ steps. The algorithms work by selecting one configuration $i$, incrementing its budget to $b_i + \\Delta b$ and querying the database for the performance at the budget $b_i + \\Delta b$. This performance is stored in the history. I believe that these parts I was able to confirm from the code. How this history is finally used for evaluation, I am a bit unsure about, as the authors did not provide the code they used for postprocessing and plotting. I believe they take the max in this history, and take that as the evaluation of the algorithm. \n\nI perceive two issues with this evaluation method. First, the evaluation is only based on the performance during HPO with a policy that is not fully trained. In HPO we could consider two different ways of evaluating: evaluate the policy at the current budget, or evaluate the policy after fully training with the chosen hyperparameter. I believe that the current method of evaluating only the policy at the current budget is insufficient, as often we may be interested in the performance of the chosen hyperparameter rather than the currently best trained policy. For example, if a HPO method increments all hyperparameters by only a small fraction of $T$, and selects the best hyperparameter from there, the selection performance may be quite good, while because the policy is only trained for a little bit, the performance at the current budget will be poor. For this reason, I believe that the evaluation should at least include the final performance as well (perhaps providing both methods of evaluation would be good).\n\n**Max** I had quite a bit of miscommunication with the authors about the max smoothing. My initial comment was about max smoothing during the budget increment in the interval $[b_i, b_i + \\Delta b]$, and the authors responded that this does not affect the evaluation of the algorithm. This part I mostly agree with (although I believe that other comparisons with other smoothing methods would also be good, as I am skeptical of this method). However, my later comment is not about this smoothing. If my above explanation of the algorithm is correct, the history contains the final performance of each budget increment. This means that there will be multiple entries for each learning curve (if a hyperparameter is incremented multiple times). I believe for their evaluation, they pick the maximum value in the history, so essentially, it will be taking a max over multiple points from each learning curve. This would add a selection bias and overestimate the performance (as the method would select the point where the randomness made the reward be high). This would be particularly problematic when the increment size is different for the different HPO algorithms (e.g., in the comparison with the single fidelity methods). This explanation matches what is described in the algorithm 1, where the max is taken in line 9 (note that algorithm 1 is titled \"Gray-Box HPO for RL\" so I believe it corresponds to all their HPO algorithms). If the authors wish to include the performance comparison during training (i.e., not only after fully training as I suggested above), I believe they could mitigate this issue by comparing the performance at the highest budget for each hyperparameter (i.e., not the max in the history including all increments).\n\n- The performance for the other algorithms is not provided for the max smoothing, so it remains unclear whether the new method performs well because of the max smoothing or because of the reward modeling. Here the authors did provide an ablation study showing that the max smoothing improved their performance. Moreover, in their rebuttal they argued that it is nonstandard to immediately use the new method for all previous algorithms as well. However, in this case it would at least be necessary to know how the reward modeling without the max smoothing compares to the previous algorithms. At the moment this is impossible to know, as the only results without max smoothing are rank comparisons in an ablation against the standard reward curve method. The work should at least have included reward curves for the method without max smoothing.\n\n- Data processing and presentation could be better, e.g., more realistic errorbars.\n\nUpdate 2\n------------------------------\n_________________\n\nThe authors performed substantial additional experiments, and provided a detailed new response, so I have increased my score to 6 and would be happy to see the work published. I refrained from setting the score higher because it is quite late in the review process, and it is no longer possible to update the paper, but I guess the changes for the camera ready will not be difficult if the paper is accepted.\n\nRegarding the author response: they performed experiments looking at the predictive accuracy of the models, which also showed that the max smoothing improved the ranking accuracy while reducing the prediction accuracy (providing more evidence for the authors' intuition about why max smoothing may be useful). This could be further improved by showing a few visualizations of the predicted reward curves. \n\nThey also provided the missing additional reward curves. Moreover, they explained that the performance is close to optimal on most environments (after looking at the plots on Fig 13 and 14 again, I guess the performance seems poor because the scale on the axis is inappropriate for some environments to properly see the performance).\n\nThe authors now also provided a clear explanation of their evaluation protocol as well as added the code for the post processing from where I was able to confirm the evaluation method, and it was fine.\n\nThey also performed new experiments trying out max-smoothing also with the other methods.\n\nOne remaining small issue in the paper are the errorbars. After looking at the code, it seems the errorbars are the standard deviation instead of the standard error (actually this was also written in 1 place in the paper), so I believe it should be simple to fix, and I'll give a suggestion in a comment below.", "strength_and_weaknesses": "Strengths\n\n- The performance appeared good.\n- The ablation studies showed the necessity of the different components of the algorithm.\n\nWeaknesses\n\n- The AutoRL-Bench has not passed peer review yet and it is unclear whether it is suitable. Looking at the set of hyperparameters, it seems that the log10 spacing of the learning rate might be a bit large making it too easy to differentiate between the performance, so it was not clear to me whether this benchmark is really appropriate. At the same time, I understand their setup allows testing many different algorithms on a wide range of environments cheaply, and this is beneficial. I think it would be good to add your own smaller scale experiments including learning the hyperparameters from scratch on some proposed domain of hyperparameters (in an interval not between a finite selection of hyperparameters).\n- The method included taking the max over past rewards in a window. This has been widely criticized in previous work in RL because it biases the evaluation. For example: Deep reinforcement learning that matters (https://arxiv.org/pdf/1709.06560.pdf)\n“Due to the unstable nature of many of these algorithms, simply reporting the maximum returns is typically inadequate for fair comparison” or Deep reinforcement learning at the edge of the statistical precipice (https://arxiv.org/pdf/2108.13264.pdf)\nWhile the ablation studies showed that this \"max\" was useful, comparing different algorithms when using such an evaluation protocol in RL is inappropriate. I would suggest trying some other technique, such as taking the average in the window.\n- The presentation of the results could be more comprehensive and better executed. Currently they only present the ranks of the algorithms. This makes it difficult for future researchers to compare to the results in this paper. It would be useful to also present absolute performance based metrics such as the reward. This makes it also more clear how much the performance differs between the methods. Moreover, it seems that the errorbars are unrealistic in the article. They seem to wide given the smoothness of the curves. Moreover, sometimes the errorbar crosses below rank 1, which should be a hard minimum. I would suggest to change the statistical analysis performed to obtain these errorbars, e.g., using a 95% percentile bootstrap may provide better errorbars.\n- There was no direct analysis of the accuracy of the newly proposed model based on the reward curves. The experiments only show the BO performance, which implies that the model should have accurately predicted the rewards; however, it would be better to directly check the prediction accuracy.\n- It was unclear to me why the experiments were limited to RL. Learning curves are also present in other fields of machine learning. Is the method also useful in other domains? Why not expand the experiments to other domains? I would at least have liked to see some discussion or explanation of this.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TqpZbAP00fj", "reviewer": "Reviewer_h99m", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a Bayesian optimisation framework for tuning hyper-parameters reinforcement learning algorithms. The framework is based on a kernel formulation which uses a model reward learning curves to interpolate between multiple budget-dependent fidelity levels. An extensive experimental evaluation is presented comparing the proposed approach against existing hyper-parameter optimisation (HPO) baselines on problems with popular RL algorithms and multiple environments.", "summary_of_the_review": "The paper's contributions are novel, but the experiments section needs improvement and further clarifications. Otherwise, it imposes the question that the gains might be only marginal or fruit of the stochastic nature of the problems, given the low number of independent trials (3) and lack of clarity about the tuning of the baselines.\n\n\n#### Post-rebuttal update\nI've read the authors response and decided to raise my score (5 to 6). ", "strength_and_weaknesses": "### Strengths\n* The idea of using reward-curve models to inform multi-fidelity kernel functions for HPO is interesting.\n* Experimental evaluation has a sufficient number of algorithmic baselines.\n* The proposed algorithm seems relatively simple to implement compared to other baselines while achieving high performance in the low-budget regime.\n\n### Weaknesses\nMost of my concerns regard the experimental evaluation, which seems to be the main focus of the paper.\n\n* **Number of trials** Experiments were repeated for only 3 independent trials (3 random seeds). This number seems to be a bit too small for state-of-the-art claims. DEHB, for example, had 50 trials for most experiments (Awad et al., 2021)\n\n* **Baselines configuration** All of the HPO baselines have their own hyper-parameters and architectural choices (e.g., kernel, acquisition function, etc.). How were these configured to provide a fair comparison? Given the gap of around one std. deviation (out of only 3 random seeds), it might be possible to fine tune the baselines to obtain better results with them.\n\n* Why use the maximum budget for the single-fidelity baselines? With 4 initial configurations provided, if I understood it correctly, it leaves them with only 6 attempts available to try to find anything. Meanwhile RCGP would have up to 10 times more iterations to run for with lower-fidelity observations.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MoTRjOXS0KH", "reviewer": "Reviewer_8pAg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers the problem of hyper-parameter optimization for reinforcement learning algorithms. A Bayesian optimization based algorithm is proposed to tackle the problem where the key idea is to model the reward curve of a candidate hyper-parameter configuration with a generalized logistic function. Gaussian Processes are used as surrogate models with an augmented input feature space consisting of the hyper-parameters, budget allocated to the configuration and the reward curve estimate. Experiments are performed on multiple RL based algorithms.", "summary_of_the_review": "Overall, the algorithm proposed in the paper is limited in novelty and mostly an application of existing Bayesian optimization techniques for hyper-parameter optimization of reinforcement learning algorithms. I want to stress that the limited novelty alone is not necessarily a bad thing and applying existing techniques to effectively solve an important problem is quite useful. However, most of the algorithmic components employed in the paper are not motivated and lack principled justification. Please consider working on this aspect to improve the paper's quality.", "strength_and_weaknesses": "Strengths\n\n- The paper considers an important problem as RL algorithms are well known to be finicky with the choice of their hyper-parameters.\n\n- The release of source code is commendable and a good contribution towards reproducibility, which is commonly poor with hyper-parameter optimization algorithms.\n\n\nQuestions/Suggestions for improving the paper: \n\n- The discussion about some of the main points proposed in the paper is very limited. For example, the choice of reward model as generalized logistic function is introduced in few lines in the paper without any motivation. Please consider expanding discussion about why this is the right choice? What properties/assumption of underlying MDPs/reward model makes this an effective choice? What are the limitations of this choice? As another example, the smoothing of reward curves in Equation (1) is described in ad-hoc manner without any principle.\n\n\n\n- Please provide more details about the training of the sigmoid coefficients and other hyper-parameters of the (multi-fidelity) Gaussian process model used in the proposed approach. \n\n- The experimental evaluation only shows the ranking of the proposed approach and the baselines. This gives limited information about the actual performance of the proposed approach. Please show runs of individual algorithms for a particular benchmark. \n\n- The search space consists of mainly two/three hyper-parameters which are chosen from a discrete set of values which seems too small. Can we employ the proposed approach on a continuous search space to robustly test it's effectiveness?\n\n- The notion of gray-box has already been defined for a range of problem settings in the literature (please see [1] and references therein). Redefining the definition to `the concept of cheaper and approximate evaluations of the performance of hyperparameter configurations` makes it too specific and restricted. Please consider expanding this definition or atleast contextualizing the problem with respect to the existing (broad) literature.\n\n\n[1] Astudillo, Raul, and Peter I. Frazier. \"Thinking inside the box: A tutorial on grey-box Bayesian optimization.\" In 2021 Winter Simulation Conference (WSC), pp. 1-15. IEEE, 2021.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster