Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRL
Authors: Jack Parker-Holder
Vu Nguyen
Shaan Desai
S Roberts
Venue: NeurIPS 2021 Poster (2021)
Keywords: AutoRL
Reinforcement Learning
Population Based Training
Bayesian Optimization
Generalization

Abstract: Despite a series of recent successes in reinforcement learning (RL), many RL algorithms remain sensitive to hyperparameters. As such, there has recently been interest in the field of AutoRL, which seeks to automate design decisions to create more general algorithms. Recent work suggests that population based approaches may be effective AutoRL algorithms, by learning hyperparameter schedules on the fly. In particular, the PB2 algorithm is able to achieve strong performance in RL tasks by formulating online hyperparameter optimization as time varying GP-bandit problem, while also providing theoretical guarantees. However, PB2 is only designed to work for \emph{continuous} hyperparameters, which severely limits its utility in practice. In this paper we introduce a new (provably) efficient hierarchical approach for optimizing \emph{both continuous and categorical} variables, using a new time-varying bandit algorithm specifically designed for the population based training regime. We evaluate our approach on the challenging Procgen benchmark, where we show that explicitly modelling dependence between data augmentation and other hyperparameters improves generalization.

Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRL
Jack Parker-Holder' University of Oxford
Vu Nguyen
Shaan Desai University of Oxford
Stephen Roberts University of Oxford
Amazon, Australia
Abstract
Despite a series of recent successes in reinforcement learning (RL), many RL algorithms remain sensitive to hyperparameters. As such, there has recently been interest the field of AutoRL, which seeks automate design decisions to create more general algorithms. Recent work suggests that population based approaches may be effective AutoRL algorithms,, by learning hyperparameter schedules on the fly. In particular, the PB2 algorithm is able to achieve strong performance in RI. tasks by formulating online hyperparameter optimization as time varying GP- bandit problem, while also providing theoretical guarantees. However, PB2 is only designed to work for continuous hyperparameters, which severely limits its utility practice. In this paper introduce new (provably) efficient hierarchical approach for optimizing both continuous and categorical variables, using a new time-varying bandit algorithm specifically designed for the population based training regime. We evaluate our approach on the challenging Procgen benchmark, where we show that explicitly modelling dependence between data augmentation and other hyperparameters improves generalization.
Introduction
Reinforcement Learning (RL [75]) is paradigm whereby agents learn to make sequential decisions through trial and error. In the past few years there have been series of breakthroughs using RL in games [70, 44, 6) and robotics [50, 33), which have led to a surge of interest the machine learning community. Beyond these examples, RL offers the potential to impact vast swathes of society, from autonomous vehicles to robotic applications in healtheare and industry.
Despite the promising results, RL is notoriously difficult to use in practice. In particular, RL algorithms are incredibly sensitive to hyperparameters [23, 2, 18, 27, 9, 47], with careful tuning often new and more complex algorithms are
the difference between success and failure. Furthermore, introduced, the search space continues grow [2]. As a result, it often challenging to reproduce RL results given the significant number of moving parts in modern algorithms [2). This makes it almost impossible to apply these methods novel settings, where optimal hyperparameters are unknown. On the other hand, the capability of current methods may be understated, as better configurations could boost performance [10, 84].
In this work we focus on the recent impressive results for Population Based Training (PBT, [29, 39]), which has demonstrated strong performance in variety of prominent RL settings [65, 84, 43, 19, 28]. PBT works by training agents in parallel, with an evolutionary outer loop optimizing hyperparameters, periodically replacing weaker agents with perturbations of stronger ones. However, since PBT relies on random search to explore the hyperparameter space, it requires large population size which can be prohibitively expensive for small and even medium-sized labs.
"ooreeponndecce to jackph@@rhots.ox..ccoo
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Table 1: The components of related approaches, and their relative trade-offs.
Algorithm
Population Based?
Efficient Continuous?
Efficient Categorical?
PBT/PBA [29, 25]
PB2 [52]
CoCaBO [63]
X
This work
In order to achieve similar success with a smaller computational budget, the recent Population Based Bandits (PB2. [52]) algorithm improved sample efficiency by introducing a probabilistic exploration step. backed by theoretical guarantees. Unfortunately. a key limitation of PB2 is that it only addresses the problem of efficiently selecting continuous hyperparameters, inheriting the random search method for categorical variables from the original PBT. This not only inefficient but also crucially ignores potential dependence between continuous and categorical variables. Since the Gaussian Process (GP, (58]) model is completely unaware of the changing categories, is unable to differentiate between trials that may have completely different categorical hyperparameters. In this paper we introduce a new hierarchical approach to PB2, which can efficiently model both continuous and categorical variables, with theoretical guarantees. Our main contributions are the following:
Technical:
We introduce a new PB2 explore step which can efficiently choose between both categori- cal and continuous hyperparameters in a population based training setup. In particular, we propose new time-varying batch multi-armed bandit algorithm, and introduce two hierarchical algorithms which condition on the selected categorical variables. We show our new approach achieves sublinear regret, extending the results for the continuous case with PB2.
Practical: We scale our approach to test generalization on the Procgen benchmark [13], an active area of research. We demonstrate improved performance when explicitly modelling dependence between data augmentation type and continuous hyperparameters (such as learning rate) vs. baselines using random search to select the data augmentation.
2 Related Work
This work contributes to the emerging field of AutoRL [11], which seeks to automate elements of the reinforcement learning (RL) training procedure. Automating RL hyperparameter tuning has been studied since the 1980s [4], with a surge in recent interest due to the increasing complexity of modern algorithms [84, 54]. The scope for AutoRL is broad, from using RL in novel real-world problems where hyperparameters are unknown [64], to improving the performance of existing methods [10]. Recent successes in AutoRL include learning differentiable hyperparameters with meta geta-graddeets [81, 83), while it has recently been shown it is even possible to learn algorithms [49, 12].
In this paper we focus on the class of Population Based Training (PBT [29]) methods. Inspired by how humans run experiments, PBT trains agents in parallel and periodically replaces the weakest agents with variations of the stronger ones, learning a hyperparameter schedule on the fly, in a single training run. While PBT is applicable in any AutoML [26] setting, it has been particularly impactful in deep RL [19, 84, 65]. PBT was recently shown work well with a shared replay buffer [20]. but this only applies for off policy methods. This paper builds on the recently introduced Population Based Bandits (PB2, [52]) algorithm, which employs a Bayesian Optimization (BO, [8. 24, 71, 17, 63, 46)) approach to select hyperparameters during the course of training. Concretely PB2 casts the hyperparameter selection step of PBT as a batch varying GP bandit optimization problem [7]. The strength of
PB2 lies in finding optimal configurations with a small population size, yet at present it has only been developed for continuous hyperparameters, relying on random search for the categorical variables.
Categorical variables are prominent in RI., for example the choice of exploration strategy [77], or even algorithm class. A prominent recent categorical variable is the choice of data augmentation. which has been shown to significantly improve efficiency and generalization in RL [37, 35, 38, 80), reducing observational overfitting [73]. Recent work introduced automatic data augmentation [57]. which shows significant improvement over static baseline by learning the data augmentation on the fly (with a single agent). However, this approach relies on a grid search over new hyperparameters, keeping all others fixed. We take inspiration from this result and learn both data augmentation and baseline RL algorithm hyperparameters jointly on the fly, with a population of agents. PBT has also been extended to data augmentation (PBA, (25]) with strong results in supervised learning tasks. However, in this setting the data augmentation was set up as a continuous variable. A high level summary of differences vs. prior work is shown in Table 1.
2

---

3 The Case for AutoRL
In this section we introduce the reinforcement learning (RL, [75]) paradigm, before making the case for automating hyperparameter selection for policy gradient algorithms.
3.1 Reinforcement Learning Background
A Markov Decision Process (MDP) is tuple P, where for each time step 0, 2.. the environment provides the agent with an observation 8; E S, the agent responds by selecting an action a, E S, and then the environment provides the next reward rt, discount YY+1, and state st+1- Reinforcement learning (RL) considers the problem of learning a policy π which selects actions that maximize the expected total discounted reward [5, 76, 75]. While method can be applied to any RL algorithm, for simplicity we focus our discussion on policy gradient algorithms, which directly seek maximize the policy with respect expected reward. For a policy л : parameterized by the objective is as follows:
J(Te) = Ey R(T)
where r = {s1, 01, ......... SH. 0H, TH } for some horizon H. Of course, we wish to maximize this, so optimize the policy by taking the following gradient steps:
0++1 = 02 + aVao
3.2 Case study: Proximal Policy Optimization
Proximal Policy Optimization (PPO [68]) is one of the most widely used RL algorithms, achieving strong performance in continuous control problems, and even scaling to large scale games [6]. Building on Trust Region Policy Optimization (TRPO [66]), the success PPO comes from using a clipped loss function follows:
Te(a|s)
TT(aas) (a(aas) e, c,1+t)
(P min
TH g(0.))A="" where g(0, 11) =clip [a](s
for a previous policy THe an advantage function and a clipping hyperparameter e. As can be seen, PPO's success hinges on several new hyperparameters. As well as the learning rate a. clip parameter decay 7, there is also often another hyperparameter in the advantage function À [67]. As if this wasn't 't enough, recent study showed we also need to consider code level hyperparameters [18] after recent work found nine implementation details (such as reward scaling) had a significant impact on performance. Thus, it is clear that while PPO can be effective, it requires careful tuning if it is to be used effectively for novel problems.
In addition, recent study investigating actor-critic algorithms [2] found that not only are there many more hyperparameters than previously thought (for example weight initialization strategy), but their optimal values are dependent
one another, producing combinatorially large search space. This problem is only increasing as novel methods extending upon PPO they almost always add additional heprpaarameters [57]. Thus, if it wasn't enough that PPO itself has tens of hyperparameters, new additions may render all previous values sub-optimal since they may be dependent on the new variables. The main hypothesis in this paper is that can improve performance of baseline algorithms by jointly learning several hyperparameters on the fly.
4 Population Based Bandits
We consider the problem of optimizing population of agents in parallel, dynamically adapting their weights and hyperparameters such that strong performance can be achieved in single training run. Following existing work [29, 52], we consider two sub-routines for a total of T steps, evaluating performance every bready < steps. For the exploit step, the weights of the bottom agents are replaced by those from randomly sampled agent from the set of best performing agents, in what is called truncation selection. The next step is to select new hyperparameters, with the explore step. The full procedure is shown in Algorithm I and Fig. 1.
explore and exploit. We train

---

Algorithm Population Based Training (Bandits)
Initialize: Population network weights, hyperparameters, empty dataset. for (in parallel) t= ,,..., do
1. Train Models
Evaluate Models and Record Data
3. Ift
mod fready = O:
Exploit: Replace the weights weaker agents with those from stronger agents.
Explore: If weights were replaced, select new hyperparameters.
Return the best trained model 0
The key difference between Population Based Training (PBT, [29]) and Population Based Bandits (PB2, [52]) is the explore step. PBT selects new hyperameters using random perturbations, hence it often performs poorly in a resource constrained setting. By contrast PB2 (blue in Alg. 1) models the data from previous trials to efficiently explore the hyperparameter space. We focus the PB2 framework and introduce new algorithms for mixed-input hyperparameters.
4.1 Online Hyperparameter Selection as GP-Bandit Optimization
We consider the problem of selecting optimal hyperparameters, xt. from a compact, convex subset E Rd where d the number of hyper- parameters. Here the index b refers to the bth agent population, and the subscript repre- sents the number of timesteps elapsed during the training of a neural network. In particular, we consider the schedule of optimal hyperparam- eters over time (=?) Let (x,)) be an objective function under a given set of hyperpa- rameters at timestep f. Here take (x.,) to be the reward for a deep RL agent. Our goal is maximize the final performance FTTx)).
R((h) #
x
Exptore
6/41
Explon
Figure 1: overview the PBT/PB2 framework. have population of B agents, each with weights of and hyperparameters 2/. t refers the stage training, with each increment corresponding to training for leady steps. The performance of the agent is represented by F,- At each the weights of the worst performing agents are replaced (exploit), where weights agents. The new hyperparameters are then selected using an explore procedure, the focus of this work.
Following [52] we formulate this problem as optimizing the time-varying black-box reward function ft. over D. Every tready steps, we observe and record noisy observations. y1 fe(xx)) + 6, where é ~ N(0, for some fixed a². Note the function fe represents the change in F, after training for fready steps, i.e. F. Fe-tyeady We define the best choice each timestep xt arg max fa()), and so the regret of each decision T. fe(æt) - ft (xx). Minimizing the regret of each decision is equivalent to maximizing the final reward [52], i.e. max Fr (xT) = min Σ re(z)). We now discuss how PB2 seeks to minimize regret for con- tinuous hyperparameters.
copied from better
4.2 Parallel Gaussian Process Bandits for Time- Vary-ng Function
PB2 models the data using a Gaussian Process (GP, [58]), thus assumes the data follows a Gaussian distribution with mean ne (2') and variance of (x') as:
pe(x') = (. (K, + oo2)
(1)
di(x') k( x') ky (2') (Ke + 1 -
(2)
where K, = [k(2;, I, J and K, = [K(I,, o 1 The GP predictive mean and variance trade-off in making decisions in
above will later be used to represent the exploration-explorie the presence of uncertainty. The key insight in PB2 is that it is possible to consider the time-varying nature of neural network hyperparameters. This is consistent with empirical results, for example, the use of learning rate schedules in almost all large scale neural network models [36]. PB2 follows [7] by modeling the reward function under the time- varying setting as follows:
f1(x) = 91(2), f(41()) = vI wf((x) vwge+1(x) Vt #²2 2,

---

where 91,92, .2.... are independent random functions with g ~ GP(O, k) and w E [0, 1] models how the function varies with time, such that if we return GP-UCB and if
1 then each evaluation is completely independent. This leads to the extensions of Eqs. (1) and (2) using the new covariance matrix K; = K, o K time where Kime [[] w) - R1=1 and kx(r) = k, k
time
with keme [(1 - Here refers to the Hadamard product.
Selecting hyperparameters for parallel agents. A key observation in [17] is that since a GP's variance (Eqn. 2) does not depend y/s the acquisition function can account for incomplete trials by updating the uncertainty at the pending evaluation points. Recall x6 is the b-th point selected in batch, after timesteps. This point may draw information from t (b 1) previously selected points. In the single agent, sequential case, we set B and recover t, = - 1. Thus, by sequentially maximizing the
at the iteration t, find a next batch of B samples [z], ".... [!] following acquisition function:
2 arg max Mr. (x) + (Booro()) Vb I....B ZED
(3)
for 3: > O. In Eqn. (3) we have the mean from the previous batch (x)) which is fixed, but can update the uncertainty using our knowledge of the agents currently training (r)))()). This significantly reduces redundancy, the model is able to explore distinct regions of the space.
To summarize, the PB2 explore step works by optimizing Eqn. (3) select a batch of new parameters. However, it is clear to see that this approach is not equipped with a means to select categorical variables. In fact, the existing PB2 algorithm inherits the random selection of categories from PBT. To reflect this, from this point onwards we refer to PB2 from [52] as PB2-Rand.
5 Efficient Selection of Continuous and Categorical Variables
We propose new class of hierarchical population based approaches for handling continuous and categorical variables. To the best of our knowledge, this is the first provably efficient approach for optimizing the categorical variables in the PBT family [29, 52]. First we introduce a new time-varying version of the parallel EXP3 algorithm, which we call TV. E.V.EX33... used in all of our methods to select categories. We then present three alternative approaches to subsequently select the continuous variables in a hierarchical fashion, varying in their degree of dependence on the categorical choices.
5.1 Time-varying Parallel EXP3
We introduce TV.EXP3.M, new algorithm for parallel multi-armed bandits (MAB) in time- varying setting with adversarial feedback. TV.EXP3. is an extension of the multiple play EXP3 algorithm (or EXP3.M) [78] for time-varying rewards. At each round, TV.XXP3.M M takes multiple actions from the set of all possible choices [78], making it possible to use this approach with population of agents. See Algorithm for high level summary and Algorithm 5 in Sec. D.I for further details.
Algorithm 2 TV. EXP3.M (brief)
Input: 7= V (c- B)BT a 3. C #categorical choice, T #max iteration, B #multiple play
CIn(C/B)
1: Init wi = 1, Ve C...C and denote 1) = = -
2: for t= to T do
3: Normalize to prevent from over exploitation
4: Compute the prob for each arm pf, Vc
5: Select batch of B choices S. o7.,,...... of) DepRound (B, [ptr?....p
6: Observe the reward after evaluating S₁
7: Update the weights using for each arm ws, Vc
8: end for
We treat each categorical choice as an arm in the bandit setting. Given a collection of C' categories, each includes reward probability, we select batch of B points
St Ict, o oi TV.EXP3.
(4)
5

---

To efficiently select set of B distinct arms from [C], we use the technique of dependent rounding (DepRound) while satisfying the condition that each arm c is selected with probability Pe. exactly [21]. TV.EXP3M is provably efficient, with a sublinear regret bound.
Theorem 1. Set a = + and = min { {
Cln(C/B) we assume the reward distributions changes (c-1)BT
at arbitrary instances, but the total number of change points is more than V < VT times. The expected regret of TV.EXP3MM satisfies the following sublinear bound
CT CT
E [RrB] S EI++e+Mi 1
5.2 New Exploration Strategies
Now we are ready to present two novel hierarchical exploration strategies. Both methods use TVEXXP3M to select categories before subsequently choosing continuous variables.
Algorithm 3 PB2-Mult: explore
Algorithm PB2--ixx explore
1. Select [b=1 using TV. E.E..3.
1. Select (h+ [ B using TV.EXP3M
Filter dataset.
2. Select x² by optimizing Eqn. (3), using the kernel from Eqn. (5)
3. Select æt by optimizing Eqn. (3).
Output: [h]]6- (a/))²
Output: (4) (=))
PB2-Mult (Algorithm 3): We consider a special version of dependency in which the presence of the continuous variables entirely depends on if the corresponding category is switched on. This fully incorporate dependence between hyperparameters, but also allows us
not only allows to have different hyperparameters depending on the category. For example, if our categorical variable is the choice of neural network optimizer, then the current optimal learning rate may vary significantly if using Adam [34] or SGD. Furthermore, Adam requires additional hyperparameters, 81 and B2. We handle this dependency by constructing multiple vimeinryyigg GP surrogate models, each corresponding to choice of category. We call this method PB2-Mult. PB2-Mult is the best choice the continuous variables are highly dependent on the category, or there are differing numbers of continuous variables for each category.
PB2-Mix (Algorithm 4): We also propose a mechanism incorporate dependence between contin- uous and categorical variables directly into the GP kernel, in what we call PB2-Mix. To do this, we extend the joint continuous-categortee kernel in CoCaBO [63] to the time-varying setting. Denote [x, c]. we have:
k=(2,z') (1- (K. + kat) + Ake(z)ket
(5)
where Ket = IIced) x k time and k2 (z), k kimee are defined in Sec. 4.2. In the above formulation, we first select a batch of B categorical choices ct, vb S B using TV.EXP3... Then, we utilize the kernel in Eqn. (5) to learn the time-varying GP model select the next batch of continuous points rt, vb < B following Eqn. (3), as in PB2-Rand. Each agent b will then train with [x cf). We do not need set the parameter À as it can be optimized alongside the other kernel parameters (see the Appendix, Sec: D.3). Thus, PB2-Mix in principle achieves the best of both worlds: it is able to learn the relative importance of the continuous and categorical kernels, while including all of the data.
5.3 Main theoretical results
We extend the sublinear regret bound from PB2-Rand to PB2-Mult, providing performance guarantees. Under the regret Rr is equivalent to maximizing the reward, presented Lem. in the appendix. Our main theoretical result is to derive the regret bound for PB2-Mult.
setting, we show the equivalence that minimizing the cumulative
Theorem 2. The PB2-Mult algorithm has cumulative regret bounded with high probability
T2
E[ So
V
CT
IT
(TaR + |B]3 w)
B
NB
for Ñ TT;, 0, and V < VT.
From Theorem 1, have To: xo when T - Asymptotically PB2-Mult achieves sublinear
convergence rate, i.c., limm- B[ra]] 0. We refer Sec. D.2 in the Appendix for proofs.

---

Experiments
The primary goal of our experiments is to test the efficacy of PB2-Mult and PB2-Mix against PB2- Rand, the primary baseline. We consider two settings: first, a synthetic task where we know that the continuous and categorical variables exhibit strong dependence. We then move to the AutoRL setting, where we apply them to select both continuous hyperparameters and categorical data augmentation type for challenging tixel-based procedurally generated environments [57].
6.1 Synthetic Function with Dependency
We begin with a simple synthetic function with one continuous and one categorical variable. The categorical variable has two options hy ~ {sin, cos} and the continuous variable has bounds x ~ [0, 21. The blackbox function [ [xr, A) + returns hxx), and the regret is equal to -+((). is clear to see the function is maximized (and regret minimized) at {sin, 3 } and {cos, 0}. We compare PB2-Mult and PB2-Mix against PB2-Rand, PBT and Random Search. At each iteration, the worst performing agent(s) is replaced by the best and the categories and continuous variables are re-selected. We run the example 20 times, with a population size of 4, 8 or 12 agents.
Fig 2 shows the mean cumulative regret, with the standard error (sem) shaded. As we see, random search has linear regret, while PBT and PB2-R both provide in- cremental improvements. Our new ap- proaches outperform, led by PB2-Mult. Thus, see that when there is strong de- pendence between the two hyperparame- ters, the methods which explicitly model this significantly outperform. For details, see the following notebook f func, including open source implementations of all the discussed algorithms.
Batchsize=4
Batchsize=8
Batchsize=12
30
50-
Random
Cumulative Megret
20
60-
PB2 Rand
30
PB2-Mix
10
20
10
10
20
10
20
Figure 2: Mcan + 1sem cumulative regret for 20 runs.
6.2 Learning Hyperparameters and Data Augmentation for Generalization in RL
There has recently been increased interest in testing RL algorithms procedurally generated environ- ments [48, 13]. These typically consist of a set of parameters which are used to generate levels, consisting of changes to the observation or layout of the environment, requiring agents to generalize [32, 61]. In this paper we focus on the Procgen environment [13]. Given the recent success of data augmentation in Procgen, DrAC [57], using a ResNet architecture [22] as originally introduced in IMPALA [19]. As well as the categorical data augmentation, we also tune the regularization coefficient a, introduced in DrAC, as well as more classically well-studied PPO hyperparameters such as the clip parameter e, the learning rate and entropy exploration coefficient. The bounds for each, well as the fixed hyperparameters, are given in the Appendix (Section A, Table A.1 and 3). We learn these hyperparameters, as well as the categorical data augmentation on the fly in single training run, starting with random values.
adopt the author's 's implementation of data regularized actor critic or
We train population of agents, each for 25M steps, with fready = 500k steps, thus the explore step is called approximately fifty times. All of the PBT-style methods use the same truncation selection approach, whereby the bottom 25% of agents are replaced with copies of the top 25%. Since we have population size of B = 4 this equates replacing the worst agent with the best. We use seven Procgen games: BigFish, CaveFlyer, CoinRun, FruitBot, Jumper, Leaper and StarPilot. At the end, each agent is evaluated on 100 train and test levels and we present the test performance from the agent with the best final training performance in each population. Our primary baseline is Rand, which we note
already shown to improve upon both Hyperband [40, 41] and vanilla BO [8] in [52]. As such, we choose to use our computational resources to run five trials for all seven games rather than by confirming this result. Instead, we do include the original PBT algorithm baseline, as well as single agent results from [57]. All of the PBT results can be replicated using our open source codebase, which was designed to make use of our single GPU available and thus is parallelized.2 For parallel version of PB2, see Ray Tune [45, 42].3
autorl
http://////boooooooo//////////// pttp:///eeeeeeeeeeeeeeee//////pp example html html
htttps:/ddoc..ray..oe
7

---

Table 2: Test performance for seven Procgen games. i indicates training conducted for single agent for 25M timesteps. Hyperparameters were initialized optimized values, meaning the effective number of timesteps much higher. Final performance the average of 100 trials and results were taken from [57]. indicates training was conducted by a population of four agents for 25M timesteps cach, with several hyperparameters initialized random. Each agent is evaluated for 100 trials on train and test levels and we present the mean test performance of the agent with the best training performance. The "Normalized Returns" with respect to the FB2-Rand baseline, * indicates p < 0.05 in l-test over PB2-Rand. Bold = within
the best mean.
Enviromment
PPO
UCB-DrACi
PBT:
PB2-Rand!
PB2-Mult
PB2-Mix
BigFish
4.0 1.2
9.7 1.0
6,6 16=2.4
8,2
9.6 1.7
10.6 = 1.9
CaveFlyer
5.1 0.9
5.3 0.9
4.4 1.0
5.8 1.8
5.6 0.8
6.0 0.7
CoinRun
8.5
0.5
6.8 1.0
8.1
8.5 0.4
8.1 0.2
FruitBot
26.7 0.8
28.3
2.2
29.5 1.9
29.6 0.8
29.0 0.6
Jumper
5.8 0.5
6.6
6.2 6.2 0.3
5.4
5.9
6.2 0.3
Leaper
4.9 0.7
0.3
0.6
5.0
5.0
6.6
StarPilot
24.7 3.4
30.2 2.8
26.9
33.6 3.1
35.3 1.5
36.3
Normalized Returns (%)
85.6 +: 22.4
100 + 23.8
105.3 14.6
112.1 20.4"
The final results are shown in Table 2, where each agent is evaluated for 100 trials, a common protocol in Procgen [57, 31]. For population based methods, we select the agent with the best training performance and show its' test performance. Given the challenge comparing results, we present an additional metric which is a score normalized by the PB2-Rand performance. For each seed in each game, the score is divided by the mean PB2-Rand score for that given game. We then show the mean of these normalized scores, comprising of 35 results. We show full train/test learning curves in Fig. 7 and 8 in the Appendix. In addition, we note the challenge in comparing aggregate metrics in RL, and such, also make use of the recently introduced rliable library, designed to provide robust metrics that take into account the uncertainty experimental results ]. In Fig.3 show aggregate performance statisties across all tasks, where clearly see that PB2-Mix and PB2-Mult outperform FB2-Rand Our other results confirm what was already known- wiit limited data (B 4) the original PBT performs poorly (worse than PPO without data augmentation), while PB2-Rand provides large improvements over this.
Algorithm
Algorithm
Median
IQM
Mean
Optimality Gap
PB2-MIX
PB2-Mix
PB2-Rand
P82-Mult
PB2-Rand
PB2-Rand
PBT
P82-Rand
PBT
UCB-DrAC
UCB-DrAC
PPO
0.45 0.60
0.5
0.48
0.64
Min-Max Normalized Score
P(X
(a) Aggregate Performance Statistics
(b) Probability of Improvement
Figure 3: (a) Aggregate performance statistics across all seven Procgen environments. The authors of [1]
recommend IQM. (b) The prohability of improvement on a new task from the same distribution.
Despite using small population size, state-of-the-art methods which used much larger grid search. We see that PB2-Rand significantly outperforms PBT, and furthermore PB2-Rand. Notably, we see the strongest performance from PB2-Mix. which is able to exploit the dependence between all variables in joint kernel. The average gains for PB2-Mix over PB2-Rand are statistically significant, with p = 0.044 in Welch's 's t-test.
able train policies that perform competitively with
new approaches modeling categorical variables improve upon
BigFish: clip
Jumper: clip,
FruitBet: clip
Efficacy
TV.EXP3.M Am
all-
<rop
jiter|
crp
cutout-color
rotate
autout
gepoode
0
-2
11
0
0:2
9.2
T-value
T-value
T-value
(a) Hyperparameter impact by augmentation type
(b) Augmentation selection efficacy
Figure (a) T-values for univariate regressions predicting reward change given a continuous hyperparameter, conditioncd on category. (b) Left: Mean reward change for each category in the BigFish environment, across all experiments, in the first and second half of training. Right: All arms selected by TV.EXP3M..

---

Do we need to model dependence? To explore the dependence of the parameters in the Procgen setting, we tested relationship between each continuous hyperparameter and the subsequent change in reward (.).... For each individual hyperparameter, we conditioned the data on the category currently being used, and fit a linear regression model. Fig. 4.a) we show the t-values for three separate examples, with the full grid in the Appendix (Fig. 9, Section B). As we see, the relationship between continuous variables and learning performance varies depending on the category selected, confirming dependence in Procgen [72]. In particular, we see for BigFish and Jumper the clip parameter relationship with training performance is heavily dependent on the data augmentation type used. Interestingly we also include example from FruitBot, where the dependence seems to be weaker as the clip param is positively related with improving policies for all but one category. This is likely the reason for relatively stronger performance for RB2-Rand in FruitBot.
We also investigate the effectiveness of TV.EXP3.M selecting the data augmentation type. Fig. 4.b) we show the mean change in reward for each category in the BigFish environment, as well as all the categories selected by the new PB2 variants using TV.EXP3.M M. We see that the three most selected augmentations (flip, crop and color-jitter) all lead to positive rewards. Interestingly, we also see that cutout-color is frequently selected at the beginning, but not used at all at the end where no longer improves training, thus demonstrating TVEXXP3M is able to adapt effectively.
Now we have seen there is clear evidence of dependence between the hyperparameters, we seek to answer this question on the downstream task by training RL agents. We introduce an ablation of our method called "PB2-Indep" which uses TV.EXP3. ignores this information when using the GP to select continuous hyperparameters. We once again evaluate this method with five runs on all seven Procgen games, and show the aggregate results in Fig. 5. Indeed, we see that PB2-Indep does improve upon PB2-Rand, since selecting the best data augmentation should provide performance gains, but we do not see the full benefit of PB2-Mix which is able to select continuous variables conditioned on this information. We see that the probability of improvement for PB2-Mix over Indelndep and PB2-Indep over PB2-Rand are both equal (just over 56%), which indicates the gain for PB2-Mix comes equally from efficiently selecting the data augmentation type and then subsequently conditioning on the output of that selection when selecting continuous variables. When combined, get over 69% probability of improvement the baseline.
select the data augmentation type but then
BigFish
IQM
Optimality Gap
Algorithm X
Algortthm
PB2-Mix PB2-Indep
PB2-Mix
P82-Rand
15-
Retum
PB2-Rand
FB2/MIX
P82-indep
P82-Rand
PBT
Ps2-Mult
PPO
PB2-Indea
P82-Rand
0.4 0.5
0.48 0.56 0.64
0
25M
Min-Max Normalized Score
P YI
Timestops
(a) Aggregate Statistics
(b) Probability of Improvement
(c) Test Curves (B=12)
Figure 5: Ablation Study:(a) and b) show aggregate performance statistics vs. an ablation "PB2-Indep" which independently (yet efficiently) selects continuous and categorical variables. Scaling up: c) shows the test curves with B= = 12 agents, plots show the mean of three runs with sem shaded.
Does it scale? The key benefit of PBT algorithms is the ability to make use of large population sizes with the same wall clock time of a single agent. Thus far, we have only considered the resource constrained setting which is important but not representative of setting that would be useful for labs with access to more resources. Thus, the question remains: how do these new methods scale? To test this we train with population size of 12 for PB2-Rand, PB2-Mult and PB2-Mix on the BigFish task. We repeat the experiment for three seeds and report the mean test return from the agent with the best training performance in 5c). As we see, PB2-Mix remains the strongest method, with final performance of 16 = 1.15. This is comparable to the state of the art performance [31, 56). Since our method is orthogonal to these, it is likely combining them could provide further gains.
6.3 Discussion
Our results show the benefit of AutoRL methods for adaptively selecting data augmentation type alongside other RL hyperparameters, with strong results in a set of procedurally generated environ- ments. We believe that with access to more compute, it may be possible to combine our AutoRI. framework with ideas from state-of-the-an methods [56, 15, 31] to achieve state-of th-thr-rrtttt results on

---

the Procgen benchmark. Furthermore, this may be an effective approach to find settings on new PCG environments, when the optimal base configuration is not known.
However, we note there are some limitations in our approach. Firstly, PB2-Mult, which has the strongest theoretical results and performance on the synthetic task, was less impressive in Procgen. This may be due to the continuous/categorica dependence not being strong enough to justify only using a small subset of the data when exploring. We thus believe PB2-Mix is likely the best bet for a novel problem, as it has the ability to trade off' between the continuous and categorical kernels. However, optimizing this parameter at each explore step does introduce additional computational complexity, which increases the runtime proportionally to the amount of data. The runtime can be improved by fixing the kernel hyperparameters, using a sliding window of data, or using faster GP implementation (such as sparse GP [69]), although in some cases this may not be desirable. Indeed, there inevitably a trade-off in terms of the desired efficiency of the explore step and the computational cost one is willing to incur. In the extreme setting of cheap evaluations, it may even be preferable to have a larger population of PBT agents, while in the case of Proegen (where each run takes up to 24h on GPU) it is certainly the opposite, as the explore step takes up trivial amount of time relative to RL training.
Finally we note another potential area for improvement in our approach: we make decisions in the explore/exploit step based on training performance (on changing levels). ignoring generalization ability. This may lead to accidentally removing the best agents on the test set, while it may also lead to inaccuracy in our GP models. One potentially promising approach may be consider a fixed set of "validation levels", learned using techniques such as Prioritized Level Replay [31].
7 Conclusion and Future Work
In this paper we expand the capabilities of population based bandits (PB2) by making it possible to efficiently select both continuous and categorical hyperparameters. We introduced a new time varying multi-armed bandit algorithm for selecting categorical hyperparameters, and presented a hierarchical approach to subsequently select continuous parameters. We believe this work is an important step in increasing the capability of ippulation-based approaches for AutoRL. This should provide benefits at both ends of the spectrum: it can make RL accessible to a wider audience by significantly lowering the cost of tuning hyperparameters, while also potentially leading stronger performance for those with larger resources. We also showed the effectiveness of learning both hyperparameters and data augmentation on the fly, for challenging procedurally generated environments. We are not the first to propose population-ben based methods in this setting [60], but we provided strong evidence that PBT-style hyperparameter tuning can provide significant benefits.
We are particularly excited by many future directions from here. The most natural is to learn more: can we learn policy architectures in the explore step, for example making of kernels between networks [79]? Can we learn algorithms altogether, extending recent work [49, 12]? We can even consider using our approach configure environments [31, 16, 30, 62], which may make it possible to jointly learn an environment and algorithm or architecture. Outside of increasing the search space, improvements can be made on the algorithm itself, for example exploring whether it is possible to share experience across agents [20]. We can also consider whether we can benefit from encouraging these agents be diverse with respect to one another [53, 55]. Finally, we are also curious to see whether recent innovations in other areas of AutoML can be combined in a PBT-style framework [59]. We think the combination of these ideas could lead to dramatic gains in performance for RL agents, ultimately making it possible to go beyond the capability of human -deigned configurationss
Acknowledgements
The experiments in this paper were conducted using AWS. SD would like to thank the Rhodes Trust for supporting this research. The authors would like to thank Roberta Raileanu for providing open source code and both Roberta and Minqi Jiang for discussion on reporting for Procgen results, as well as Xingyou Song and Yingjie Miao for discussion around AutoRI. Finally, this work was improved thanks to constructive feedback from anonymous reviewers.
10

---

References
[1] R. Agarwal, M. Schwarzer, S. Castro, A. Courville, and M. G. Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Advances in Neural Information Processing Systems. 2021.
[2] M. Andrychowicz, A. Raichuk, P. Stanczyk, M. Orsini, Girgin, R. Marinier, L. Hussenot, M. Geist, O. Pietquin, M. Michalski. S. Gelly, and O. Bachem. What matters for on-policy deep actor-critic methods? a large-scale study. International Conference on Learning Representations, 2021.
[3] P. Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire. The nonstochastic multiarmed bandit problem. SIAM journal on computing. 32(1):48-777 7706-777770 2002.
[4] A. G. Barto and R. S. Sutton. Goal seeking components for adaptive intelligence: An initial assessment. Technical report, Massachusetts Univ Amherst Dept of Computer and Information Science, 1981.
[5] R. Bellman. Dynamic Programming. Dover Publications, 1957.
[6] C. Berner, G. Brockman, B. Chan, V. Cheung, P. Debiak, C. Dennison, D. Farhi, Q. Fis- cher, S. Hashme, C. Hesse, R. Józefowicz, S. Gray, C. Olsson, J. Pachocki, M. Petrov, H. P. de Oliveira Pinto, J. Raiman, T. Salimans, J. Schlatter, J. Schneider, S. Sidor, I. Sutskever, Tang, F. Wolski, and S. Zhang. Dota 2 with large scale deep reinforcement learning. CoRR, abs/1912.066l 2019.
[7] I. Bogunovic, J. Scarlett, and V. Cevher. Time- Varying Gaussian Process Bandit Optimization. In Proceedings of the 19th International Conference on Artificial Intelligence and Statisties, 2016.
[8] E. Brochu, V. M. Cora, and N. de Freitas. A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. CoRR, abs/10012.2599 2010.
[9] J. S. O. Ceron and P. S. Castro. Revisiting rainbow: Promoting more insightful and inclusive deep reinforcement learning research. In Deep Reinforcement Learning Workshop, NeurIPS 2020, 2020.
[10] Y. Chen, A. Huang, Z. Wang, I. Antonoglou, J. Schritwieseer D. Silver, and N. de Freitas. Bayesian optimization in AlphaGo. CoRR, abs/1812.06855 2018.
[11] H. L. Chiang, A. Faust, M. Fiser, and A. G. Francis. Learning navigation behaviors end-to-end with autorl. IEEE Robotics Autom. Lett., 4(2):2007-2014, 2019.
[12] D. Co-Reyes, Y. Miao, D. Peng, Q. Le, S. Levine, H. Lee, and Faust. Evolving reinforcement learning algorithms. In International Conference on Learning Representations, 2021.
[13] K. Cobbe, C. Hesse, Hilton, and J. Schulman. Leveraging procedural generation benchmark reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, pages 2048-2056, 2020.
[14] K. Cobbe, O. Klimov, C. Hesse, T. Kim, and J. Schulman. Quantifying generalization in reinforcement learning. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, Proceedings of Machine Learning Research. PMLR, 2019.
[15] K. W. Cobbe, J. Hilton, O. Klimov, and J. Schulman. Phasic policy gradient. In Proceedings of the 38th International Conference Machine Learning, 2021.
[16] M. Dennis, N. Jaques, E. Vinitsky, A. M. Bayen, S. Russell, A. Critch, and S. Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 62, 2020, virtual, 2020.

---

[17] T. Desautels, A. Krause, and J. W. Burdick. Parallelizing Gaussian Process bandit optimization. Journal of Machine Learning Research, 15(149) 4103, 2014.
tradeoffs in
[18] I.. Engstrom, A. Ilyas, Santurkar, Tsipras, F. Janoos, L. Rudolph, and Madry. Imple- mentation matters in deep RI:: A case study on PPO and TRPO. In International Conference on Learning Representations, 2020.
[19] L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, Harley, Dunning. S. Legg, and K. Kavukcuoglu. IMPALA: Scalable distributed deep-RI. with importance weighted actor-learner architectures. In Proceedings of the 35th International Conference on Machine Learning, 2018.
[20] J. K. Franke, G. Koehler, A. Biedenkapp. and F. Hutter. Sample-efficient automated deep reinforcement learning. In International Conference on Learning Representations, 2021.
[21] R. Gandhi, S. Khuller, S. Parthasarathy, and A. Srinivasan. Dependent rounding and its applications to approximation algorithms. Journal of the ACM (JACM), 53(3):324-360, 2006.
[22] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. CoRR, abs/1512.033855 2015.
[23] P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger. Deep reinforcement learning that matters. AAAI, 2018.
[24] P. Hennig and C. J. Schuler. Entropy search for information-efficcint global optimization. J. Mach. Learn. Res., 13(mull):
June 2012.
[25] D. Ho, E. Liang, X. Chen, I. Stoica, and P. Abbeel. Population based augmentation: Efficient learning of augmentation policy schedules. In Proceedings of the 36th International Conference on Machine Learning, pages 2731 2019.
[26] F. Hutter, L. Kotthoff, and J. Vanschoren, editors. Automated Machine Learning: Methods, Systems, Challenges. Springer, 2018. In press, available at http://automl.orr/hoook
[27] R. Islam, P. Henderson, M. Gomrokchi, and D. Precup. Reproducibility of benchmarked deep reinforcement learning tasks for continuous control. CoRR, abbs/1708.0413. 04133, 2017.
[28] M. Jaderberg, W. M. Czarnecki, I. Dunning, L. Marris, G. Lever, A. G. Castañeda, C. Beattie, N. C. Rabinowitz, A. Morcos, A. Ruderman, N. Sonnerat, Green, Deason, J. Z. Leibo, D. Silver, D. Hassabis, K. Kavukcuoglu, and T. Graepel. Human-level performance in 3d multiplayer games with population-based reinforcement learning. Science, 361(6444):859-8855 2019.
[29] M. Jaderberg, V. Dalibard, S. Osindero, W. M. Czarnecki, J. Donahue, A. Razavi, O. Vinyals, T. Green, I. Dunning, K. Simonyan, C. Fernando, and K. Kavukcuoglu. Population based training of neural networks. CoRR. Abs/17111.0984. 2017.
[30] M. Jiang, M. Dennis, J. Parker-Holder, J. Foerster, E. Grefenstette, and T. Rocktäschel. Replay- guided adversarial environment design. Advances in Neural Information Processing Systems. 2021.
[31] M. Jiang, E. Grefenstette, and T. Rocktäschel. Prioritized level replay. In The International Conference Muchine Learning. 2021.
[32] N. Justesen, R. R. Torrado, P. Bontrager, A. Khalifa, J. Togelius, and S. Risi. Procedural level generation improves generality of deep reinforcement learning. NeurIPS Deep Reinforcement Learning Workshop, 2018
[33] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakr- ishnan, V. Vanhoucke, and Levine. Qt-opt: Scalable deep reinforcement learning for vision- based robotic manipulation. CoRR, abs/1806.10293, 2018.
12

---

[34] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[35] I. Kostrikov, D. Yarats, and R. Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations. 2021.
[36] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 25, 2012.
[37] M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas. Reinforcement learming with augmented data. Advances in Neural Information Processing Systems 33. 2020.
[38] M. Laskin, A. Srinivas, and P. Abbeel. CURL: Contrastive unsupervised representations for reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, 2020.
[39] A. Li, O. Spyra, S. Perel, V. Dalibard, M. Jaderberg, C. Gu, D. Budden, T. Harley, and Gupta. A generalized framework for population based training. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, 2019.
[40] L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar. Hyperband: A novel bandit- approach to hyperparameter optimization. J. Mach. Learn. Res., 18(1):6765-681 Jan. 2017.
[41] L. Li, K. G. Jamieson, A. Rostamizadeh, E. Gonina, M. Hardt, B. Recht, and A. Talwalkar. Massively parallel hyperparameter tuning. CoRR, 2018.
[42] R. Liaw, E. Liang, R. Nishihara, P. Moritz, J. E. Gonzalez, and I. Stoica. Tune: A research platform for distributed model selection and training. arXiv preprint, 2018.
[43] S. Liu, G. Lever, N. Heess, J. Merel, S. Tunyasuvunakool, and T. Graepel. Emergent coordination through competition. In International Conference on Learning Representations, 2019.
[44] V. Mnih, K. Kavukcuoglu, Silver, A. Graves, I. Antonoglou, Wierstra, and M. A. Ried- miller. Playing Atari with deep reinforcement learning. CoRR, abs/1312.5602. 2013.
[45] Moritz, R. Nishihara, Wang, A. Tumanov, R. Liaw, E. Liang, W. Paul, M. Jordan, and I. Stoica. Ray: A distributed framework for emerging AI applications. CoRR, abs/ 171.00589, 2017.
[46] V. Nguyen and M. A. Oshorne. Knowing the what but not the where in bayesian optimization. In International Conference on Machine Learning, pages 7317-7326. PMLR, 2020.
[47] V. Nguyen, S. Schulze, and M. A. Osborne. Bayesian optimization for iterative Icarning. In Advances in Neural Information Processing Systems. 2020.
[48] A. Nichol, V. Pfau, C. Hesse, O. Klimov, and J. Schulman. Gotta learn fast: A new benchmark for generalization in RI. CoRR, abs/1 Abs//884.0855 804.03720 2018.
[49] J. Oh, M. Hessel, W. M. Czarnecki, Z. Xu, H. van Hasselt, S. Singh, and D. Silver. Discovering reinforcement learning algorithms. In Advances in Neural Information Processing Systems 33. 2020.
[50] OpenAL, M. Andrychowicz, B. Baker, M. Chociej, R. Józefowicz, B. McGrew. J. W. Pachocki, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba. Learning dexterous in-hand manipulation. CoRR, his//808.00017) 2018.
[51] C. Packer, K. Gao, J. Kos, P. Krahenbuhi, V. Koltun, and Song. Assessing generalization in deep reinforcement learning. CoRR, 2019.
13

---

[52] J. Holder, V. Nguyen and S. Roberts. Provably efficient online hyperparameter optimiza- tion with population-bas based bandits. In Advances in Neural Information Processing Systems. 2020.
[53] Parker-Holder, A. Pacchiano, K. Choromanski, and Roberts. Effective diversity in population-based reinforcement learning. In Advances in Neural Information Processing Systems 33. 2020.
[54] S. Paul, V. Kurin, and S. Whiteson. Fast efficient hyperparameter tuning for policy gradients. Advances in Neural Information Processing Systems (NeurIPS), 2019.
[55] J. K. Pugh, L. B. Soros, and K. O. Stanley. Quality diversity: A new frontier for evolutionary computation. Frontiers in Robotics and AI, 3:40, 2016.
[56] R. Raileanu and R. Fergus. Decoupling value and policy for generalization in reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning. 2021.
[57] R. Raileanu, M. Goldstein, D. Yarats, I. Kostrikov, and R. Fergus. Automatic data augmentation for generalization in deep reinforcement learning. In Advances in Neural Information Processing Systems. 2021.
[58] C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press, 2005.
[59] E. Real, C. Liang, D. R. So, and Q. V. Le. Automl-zero: Evolving machine learning algorithms from scratch. In Proceedings of the 37th International Conference on Machine Learning. 2020.
[60] S. Risi and J. Togelius. Neuroevolution in games: State of the art and open challenges. IEEE Transactions on Computational Intelligence and Al Games, 2017.
[61] S. Risi and J. Togelius. Procedural content generation: From automatically generating game levels to increasing generality in machine learning. Nature, 2020.
[62] C. Romac, R. Portelas, K. Hofmann, and P. Oudeyer. Teachmyagent: benchmark for automatic curriculum learning in deep RL. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 9052-9063. PMLR, 2021.
[63] B. X. Ru, A. S. Alvi, V. Nguyen, M. A. Osborne, and S. J. Roberts. Bayesian optimisation over multiple continuous and categorical inputs. In Proceedings of the 37th International Conference on Machine Learning. 2020.
[64] F. Runge, D. Stoll, S. Falkner, and F. Hutter. Learning to design RNA. In International Conference Learning Representations, 2019.
[65] S. Schmitt, J. J. Hudson, A. Zidek, S. Osindero, C. Doersch, W. M. Czarnecki, J. Z. Leibo, H. Küttler, A. Zisserman, K. Simonyan, and S. M. A. Eslami. Kickstarting deep reinforcement learning. CoRR, abs/1 803.03335 2018.
[66] Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning, 2015.
[67] Schulman, Moritz, Levine, M. Jordan, and Abbeel. High-dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference on Learning Representations (ICLR), 2016.
[68] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. CORR, 2017.
[69] M. W. Seeger, C. K. I. Williams, and N. D. Lawrence. Fast forward selection to speed up sparse gaussian process regression. In Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics, pages 254-261, 2003.
14

---

[70] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonogiou. V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham. N. Kalch- brenner, I. Sutskever, T. P. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, 529:488-489, 2016.
[71] J. Snock, H. Larochelle, and R. P. Adams. Practical Bayesian optimization of machine learning algorithms. F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 25, 2012.
[72] X. Song. Y. Du, and J. Jackson. An empirical study hyperparameters and their inter- dependence for RL generalization. ICML 2019 Workshop "Understanding and Improving Generalization in Deep Learning", 2019.
[73] X. Song. Y. Jiang, S. Tu, Y. Du, and B. Neyshabur. Observational overfitting in reinforcement learning. In International Conference on Learning Representations, 2020.
[74] N. Srinivas, A. Krause, S. Kakade, and M. Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In Proceedings of the 27th International Conference on International Conference Machine Learning, ICML' 10, 2010.
[75] R. S. Sutton and A. G. Barto. Introduction to Reinforcement Learning. MIT Press, Cambridge, MA, USA, 1st edition, 1998.
[76] C. Szepesvári. Algorithms for reinforcement learning. Morgan & Claypool Publishers, 2010.
[77] A. A. Taiga, W. Fedus, M. C. Machado, A. C. Courville, and M. G. Bellemare. Benchmarking bonus-based exploration methods on the arcade learning environment. CoRR, abs//908.00 023388, 2019.
[78] T. Uchiya, A. Nakamura, and M. Kudo. Algorithms for adversarial bandit problems with multiple plays. In International Conference on Algorithmic Learning Theory, pages 375-389. Springer, 2010.
[79] X. Wan, B. Ru, X. Dong, and M. Osbome. Interpretable neural architecture search via Bayesian optimisation with Weisfeiler-Lehme kernels. In International Conference on Learning Repre- sentations, 2021.
[80] K. Wang, B. Kang. J. Shao, and J. Feng, Improving generalization in reinforcement learning with mixture regularization. In Advances in Neural Information Processing Systems 33. 2020.
[81] 7. Xu, H. P. van Hasselt, and D. Silver. Meta-gradient reinforcement learning. In Advances in Neural Information Processing Systems, volume 31, pages 1396-2407, 2018.
[82] C. Ye, A. Khalifa, Bontrager, and J. Togelius. Rotation, translation, and cropping for zero-shot generalization. In 2020 IEEE Conference on Games (CoG), pages 57-64, 2020.
[83] T. Zahavy, Z. Xu, V. Veeriah, M. Hessel, J. Oh, H. van Hasselt, D. Silver, and S. Singh. A self-tuning actor-critic algorithm. In Advances in Neural Information Processing Systems, 2020.
[84] B. Zhang, R. Rajan, L. Pineda, N. Lambert, A. Biedenkapp, K. Chua, F. Hutter, and R. Calandra. On the importance of hyperparameter optimization for model-based reinforcement learning. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, 2021.
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 6.3.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] The sole aim of this work make possible use existing methods more effectively.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Theory is presented in Section 5.
(b) Did you include complete proofs of all theoretical results? [Yes] Please see Section
D.2 and D.I in the Appendix
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [Yes]] Code for the Proegen experiments is attached in the supplementary materials. Code for the synthetic problem is released as notebook. The full algorithm will be released part of the ray library once published.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix A.
(c) Did you report error bars (c.g., with respect to the random seed after running exper- iments multiple times)? [Yes] All Procgen results include five seeds, with standard deviations the tables and plots.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix A.
4. If you using existing assets (e.g., code, data, models) curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Cited the Procgen
benchmark.
(b) Did you mention the license of the assets? [Yes] Cited the Procgen benchmark.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A] Not applicable.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] Not applicable.
(c) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A] Not applicable.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A] Did not use human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A] Did not use human subjects.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A] Did not use human subjects.
16


Reviews:
[{"review_id": "q6MLoHv9aWW", "reviewer": "Reviewer_3P6N", "summary": "This paper extends the existing AutoRL algorithm, PB2, that was specifically designed to work with continuous hyperparameters by enabling it to efficiently search over both continuous and categorical hyperparameters of reinforcement learning solution methods. To do so, it first proposes a new multi-armed bandit algorithm called TV.EXP3.M (an extension of EXP3.M that varies with time) to select a subset of a given set of categories. Then, the paper incorporates this technique into the exploration stage of the PB2 algorithm to explore and select the categorical variables by taking into account the dependence of continuous variables on them. It takes two different approaches for modeling this dependency, one by incorporating it into the GP kernel and the other by filtering continuous variables based on the chosen categories that result in PB2-Mix and PB2-Mult algorithms. To evaluate the proposed algorithms, they first theoretically analyze PB2-Mult and TV.EXP3.M algorithm by showing they have cumulative regret bound and sublinear regret bound respectively. Then, they empirically evaluate their techniques along with PBT and original PB2 as the baselines on a toy environment and seven Procgen games and show that PB2-Mix achieves a similar or better performance compared to other baselines on these environments. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is an interesting paper and the proposed contributions seem useful for the RL community. However, there are some issues with the work that preclude acceptance in the current form. To elaborate, (1) there are many missing details in the paper and weakly supported claims that make it hard to understand the paper and proposed techniques, and (2) the empirical study done in the paper fails to justify the stated contributions.\n\nDetailed comments below: \n\nThere are some claims in the paper that needs further clarification. “As a result, many lauded approaches are impossible to reproduce without the vast resources required to sweep through hundreds or thousands of possible configurations [6].” The cited paper, [6], proposes an AutoML algorithm specifically designed to solve computer vision problems and not a reinforcement learning (RL) algorithm. This is happening while the main topic of the paragraph is about the practicality of RL techniques. This claim needs further justification. \n\nThe main claim of the paper is that the proposed algorithms enable the search over the space of categorical variables. Nevertheless, TV.EXP3.M seems to only consider selecting categorical variables that are binary. In another word, it can select whether a certain operation will be part of the algorithm or not. This observation is further supported by the hyperparameter search space that has been chosen for the categorical hyperparameters that are selecting a subset of different data augmentation operations. It is possible that I couldn’t understand this part really well, so I ask the following question. Is it possible to search over different data augmentations and optimizers using the proposed technique? If so, please describe how you would do it using the TV.EXP3.M algorithm. In this setting, you only get to select one optimizer out of n optimizers in your search space. But, it seems that TV.EXP3.M would select a subset of optimizers by taking multiple arms and this problem is not addressed in the paper.\n\nIn algorithms 2 and 5, it is not clear how the output of this algorithm, D_t, is chosen. It is not mentioned in the algorithms’ procedure, but it suddenly appears as output. Also, whereas this output is denoted by D_t in these algorithms, it is denoted by A_t in equation 4. Is there a reason for this notational difference here? Also, the style of “D” (D_{t-1}) used in equation 4 and the one in the output of Algorithm 4 (D_t) is different. Are they refer to different collections or datasets? Furthermore, there are not enough details about Algorithm 3 in the paper to understand how the filtering dataset procedure takes place. This part needs further explanation. “For the exploit step, the weights of the bottom agents are replaced by those from a randomly sampled agent from the set of best-performing agents, in what is called truncation selection.” The truncation selection procedure is not mentioned in the paper and there is not talk about its hyperparameters in the paper. Does this technique use any hyperparameter, such as the number of bottom agents that their weights should be replaced?\n\n\nThe experiments done in the paper do not fully support the stated contributions. In the abstract, it is mentioned: “where we show that explicitly modeling dependence between data augmentation and other hyperparameters improves generalization.” However, there is no baseline in the paper that uses TV.EXP3.M algorithm for selecting categories without modeling the dependence between continuous and categorical variables. In another part, it is stated “Despite using a small population size, we are able to train policies that match or outperform state-of-the-art methods which used a much larger grid search.” However, the evidence in the paper does not support this claim because the comparison with the UCB-DrAC algorithm in Table 2 is not justified. As is shown in Table 2, the proposed techniques in the paper and the UCB-DrAC algorithm have a highly similar performance to PB2-Mix and PB2-Mult, by their standard deviations overlapping each other in most of the environments. Also, based on the citation [21] in the paper, only 5 seeds for deep RL algorithms do not necessarily result in statistically significant results. So, to make a better comparison, the number of seeds needs to be increased. \n\n\nFinally, this is a list of questions that need to be answered about the empirical study:\n1. What is the value of \\omega (in the equation on page 4) in your experiments? Also, how much your algorithm is sensitive to this hyperparameter?\n2. A lot of new modules have been added to PB2 and eventually your algorithm to do AutoRL makes the original algorithm more computationally expensive. For instance, the computational cost of the gradient-based optimization approach that you take to optimize the GP hyperparameters can increase by the number of points in the training set. So, I wonder whether have you done any study to show how much computational cost these new modules add to the original algorithm that in this case is PPO?\n3. Why this specific hyperparameter search space has been chosen for the empirical study? There are only four continuous hyperparameters in the paper, while there the used algorithm has much more hyperparameters to tune such as \\beta_1 and \\beta_2 for Adam optimizer. So, a better question would be: how does this algorithm scale as the number of hyperparameters increases?\n4. What model is used in the paper to run the experiments and what specific hyperparameters have been used for training this model? The details of the neural architecture and Adam optimizer’s parameters are missing, such as the value of \\beta_1 and the number of layers.\n\n\nMinor comments:\n1. Thus, if it wasn’t enough that PPO -> Thus, if it was not enough that PPO (Line 115)\n2. Notation inconsistency in line 155: r_t is getting x_t as input while in line 153 it is defined without any input. So, replacing r_t to r_t(x_t) can clarify it here.\n3. Line 90: “provides the agent with an observation s_t” -> “provides the agent with a state s_t”\n4. Line 104: for a previous policy -> for the behavior policy. It is good to also add a brief definition of behavior policy such as the policy used to gather the data.\n5. Line 259: PB2-R -> PB2-Rand\n5. The name of the paper does not reflect the content of the paper and requires modification: “Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRL”. Based on the current name, it seems that the hyperparameter selection is designed for the AutoRL algorithm while it is an AutoRL algorithm that is designed for tuning the parameters of the RL algorithm.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper stated its limitations in section 6.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "15"}, "extra_text": {}}, {"review_id": "nD17GL7bEf", "reviewer": "Reviewer_GNeX", "summary": "The paper extends PB2 to allow for categorical hyperparameters in two different ways (direct integration into the kernel or two-step exploration depending on categorical choices). The authors show regret and convergence properties similar to PB2. In their evaluations, they show the benefits of integrating categorical hyperparameters by tuning the data augmentation alongside agent hyperparameters.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a well-written and technically sound paper extending an existing promising approach in hyperparameter optimization in RL. Incorporating categorical hyperparameters into PB2 in a way that allows for optimization with regards to dependencies between categorical and continuous variables will increase the utility of the algorithm and be of interest to its users. The paper gives the core insights into the extensions, mostly through its experiments which focus on the ProcGen benchmark. On one hand, this shows the improvements in a very challenging setting within RL. On the other, its computational expense leads to a less thorough exploration of the research questions. Overall an interesting addition.\n\n## Novelty\nThe paper extends an existing approach in an important and useful way.\n\n## Significance / Impact\nPB methods are an important approach for hyperparameter optimization in AutoRL, so this will be very useful and probably have a large impact for existing users of automated HPO methods in RL (especially since it will also be integrated into Ray). The impact on the research community overall depends on how well AutoRL methods in general will be adopted.\n\nI wonder a bit about the limited scope to AutoRL. PB2 and this extension could also be very well applied to other continuous adaptation problems, right? I think it would make the paper stronger and increase its impact outside of RL, if the authors could show the generic applicability to many problems and not only AutoRL.\n\n## Soundness (method and experimental setup)\nThe authors use a very thorough setup for the toy example, but time constraints of course limit this on ProcGen. Still, 5 repetitions are an acceptable compromise and including the PB2 results without reruns is a fine trade-off in this case. \n\nThe research questions are interesting, relevant to the method and answered well for the most part. I’m only a bit disappointed in the scaling study since it does not show how well it would perform with different population sizes. This could have been done on a different benchmark than ProcGen if time constraints were the deciding factor, maybe by reducing the number of ProcGen games used by one or two. That would have given the research questions beside improved performance more space.\n\nAs the paper extends PB2, it addresses points like the regret bounds of that paper as it should. Overall, the experiments are done soundly and answer the most important questions of this extension. A comparison to similar non-population-based methods (especially CoCaBO) would have been nice, but given the computational budget it is understandable the authors chose not to include any.\n\n## Clarity\nThe authors used a clear language; critical points are addressed straightforwardly (e.g. details of the experiments) and the visuals help with understanding. Background is sufficiently explained to researchers familiar with RL methods. Overall very good.\n\n## Minor Comments\n\n* For Table 1, I wondered whether it is important in practice whether it is population-based or not. I think this is not as important as whether the methods are efficient on categorical and continuous spaces. If the authors think there is a relevant difference, the text should reflect that and explicitly mention why e.g. CaCoBO suffers from not being population-based.\n* Similarly, in the discussion it is stated that the paper makes a strong case for population-based methods specifically. I don’t really see how this is true as there is no comparison to other HPO methods or a strong demonstration that dynamic hyperparameter settings are important on ProcGen. In fact, UCB-DrAC is very close to the PB2 extensions in performance. So I would argue the results are more indicative of the extensions improving over vanilla PB2 and PBT-style training being a viable option on ProcGen rather than making a case for it over any others.\n* Sometimes the authors call s_t a state and sometimes an observation. Should be consistent.\n* I wouldn’t call reward scaling an implementation detail as it can decide whether an agent can learn efficiently or not (Section 3.2)\n* What is meant by “combinatorially large search space”? If the space includes continuous hyperparameters, it is infinitely large anyway.\n* I think that it is very unfortunate to call the compact, convex subset $\\mathcal{D}$ which is typically used for datasets. This confused me at first.\n* Since $r$ is normally used for the reward in RL, I would propose to use $\\rho$ for the regret to not confuse readers.\n\n## Questions for Rebuttal\n\n* Eq 3: Is it possible to also use other acquisition functions (such as EI) instead of LCB?\n* Section 6.1: The synthetic benchmark was not quite clear to me since it seems that h_t^b does not really depend on t and therefore, there is no real advantage to use PBT/PB2 over other HPO approaches\n* Section 6.2. Why have you chosen a linear regression model? Is it a reasonable assumption that a linear model can fit the data well?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, adequately addressed in view of that the paper deals with a meta-problem and not with a specific application.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "RXTx6bOoJRd", "reviewer": "Reviewer_QwWX", "summary": "The paper presents a population-based approach for learning hyperparameter schedules on the fly. Its contribution to the field is to allow both continuous and categorical variables to be optimized in a non-random way.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Overview\n\nI had trouble understanding the paper, in part due to my background (no prior experience with PPO and PB2), but I’ve provided some suggestions that I believe would have helped with my understanding. \n\nThe justification for where this method fits among other population-based methods was clear and at a very high level the theory seemed like a great advancement, but most of the theory was in the appendix, which I didn’t review. \n\nThe empirical evidence was not great. I’m not convinced the performance was meaningfully different between the baseline and the two new methods. The theory might justify the poor performance, but I didn’t assess the theory in detail.\n\n### Major\n\n* The new approaches (PB2-Mult and PB2-Mix) were both within 1 standard deviation of the baseline (PB2-Rand) according to Table 2. The PB2-Mix improvement was statistically significant (line 298), but the p-value was a barely acceptable 0.044 (i.e. if PB2-Mix was equal to the baseline, we’d see a difference as large as this once every 23 times we ran the experiment). Furthermore, if a p-value is being used to consider significance, it is often scaled by the number of comparisons done (i.e. two: PB2-Mult and PB2-Mix), which would put this above the significance threshold.\n\n### Clarity\n\n* Lines 134–136 consider a *convex* hyperparameter space. Why is this assumption being made? This seems like a very strong assumption I’d like to see defended (even if just to say this is the assumption made by prior work, e.g. PBT).\n* Figure 1 (near line 140) was not helpful to me. I'm not sure exactly how to improve it, but I think I would have appreciated seeing the entire pipeline of hyperparameter selection and evaluation. (Perhaps familiarity with the background, such as the Procgen protocol mentioned in line 286–287, would obviate this.)\n* I didn’t find algorithm 2 (near line 198) to provide the right amount of high-level detail. I had to go to the appendix because on its own, Alg2 didn’t give me enough understanding. For example, I thought “normalize $\\omega$” might be a simple [0,1] normalization, but it’s a more complex scaling that ensures no $\\omega_c$ is larger than $\\eta$ percent of the sum of weights. Perhaps indicate that $\\eta$ is used as a scaling factor? Alg2 doesn’t indicate what $\\eta$ is used for.\n* I suggest wording line 198 as “Algorithm 5 *in* Sec. D.1” to indicate that Alg5 is in the appendix. I assumed Algorithm 5 was coming later in the paper after this high-level summary.\n\n### Minor (typos and improved phrasing)\n\nSome of these points are trivial, but you might find them helpful...\n* Some acronyms were undefined. E.g. GP was used on lines 7, 42, 71, and 133 before finally being defined on line 158. DrAC on line 272 was never defined.\n* Some hyphens were missing. Consider reading about how to [hyphenate compound adjectives](https://www.grammarly.com/blog/hyphen/). E.g. I suggest “population-based AutoRL”, “time-varying problem”, “high-level summary”, and “code-level hyperparameters”.\n* Line 75 discusses data augmentation as a categorical variable. I’d have appreciated a couple of examples or a bit more background here.\n* Line 100 says PPO is “one of the most widely used RL algorithms”. Compared to what? Sarsa? If “one of the most widely used” changes to “a widely used”, I can’t argue anymore and I won’t ask for a citation. :D \n* Line 104 says “As can be seen,” but so far the paper has only pointed out one hyperparameter ($\\epsilon$), so maybe delete that phrase.\n* Line 106 would be more clear as “there is also often another hyperparameter, $\\lambda$, in the advantage function [60].” (As written, it implied that $\\lambda$ is the advantage function.)\n* Lines 107–108: it’s unclear to me whether the “recent study” and the “recent work” referenced here are the same work or a different one. Since there’s only one citation [17], I think they both refer to the same study? Either the phrasing should be improved or this sentence needs another citation.\n* On line 114, “they” should be deleted.\n* Lines 114–115 refers to methods (plural), but only one citation [51] is provided.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There's an acknowledgement of limitations (i.e. that the performance wasn't that great).\n\nSocietal Impact: N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "-NyqDUKHMYa", "reviewer": "Reviewer_GDLx", "summary": "This paper presents an extension of population based bandits (PB2) and the proposed method can handle both continuous variables and categorical variables at the same time. A new algorithm for parallel multi-armed bandits is also presented. The proposed hierarchical approach shows better empirical performance than other approaches. Authors also give theoretical results w.r.t. the cumulative regret bound. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Jointly optimizing categorical and continuous parameters  is a very important problem.  I don't have much experience on RL, so the RL parts in this work will not be commented. \n\nadvantages of this paper:\n1. This paper is well-written. \n2. Empirical experiments are detailed and extensive.\n3. Sound theoretical results are given w.r.t. the cumulative regret. \n\nlimitations of this work:\n1. The proposed PB2-Mult treats continuous parameters conditioned on different categorical variables are independent. It is equivalent to model Gaussian processes independently. However, the assumed independence usually doesn't hold in practice.  This explains why PB2-Mult performs best among all competing algorithms in Fig2 and it doesn't perform better than PB2-Mix in Fig3. For the synthetic function, the independence assumption is satisfied while in the RL experiments, possibility such an independence doesn't hold. See [1] for a possible solution, which is also a hierarchical approach and jointly models continuous and categorical variables. \n2. Maybe it is common in RL literature, I feel it is strange to use mean to measure performance instead of median in Fig 3 and Table2.  Median is a more common in BO  literature and is more robust. \n\n\n1.Ma, X., Blaschko, M.: Additive Tree-Structured Covariance Function for Conditional Parameter Spaces in Bayesian Optimization. In: International Conference on Artificial Intelligence and Statistics. pp. 1015–1025. PMLR (2020).\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See the above review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: There was extensive discussion between the authors and reviewers, and amongst the reviewers privately. Many concerns were raised, and largely addressed. This is particularly the case with the number of runs and statistical significance, which is a concern that has largely been allayed.

The primary remaining concern is simply in the simplicity of the BO (Bayesian optimization) results. An expert in BO assessed the BO theory to be relatively solid, but not particularly exciting in the BO literature. The claims about novelty for addressing both categorial and continuous variables is a bit overstated, considering there are several works that already address this problem in BO (that can be used here). This work extends an existing algorithm, developed for the RL setting, that only works for continuous hyperparameters, so the contribution is placed relative to that. But, the limitation of this previous approach does not imply that these existing BO algorithms could not have already been applied to RL to solve this issue of mixed hyperparameters. This claim on the technical novelty should be clarified, especially given that there appears to be more effective BO algorithms for mixed hyperparameters. 

Additionally, the work begins by highlighting the importance of making RL more usable in practice. But, the work focuses on the setting where you can have populations of agents learning in parallel. The primary setting where it is feasible to train many agents in parallel is in simulation. Many real-world RL settings are restricted to a single stream of experience. This contrasts AutoML, which is typically done on a dataset, and so many hyperparameters can be tested. In RL, it is often not feasible to test many hyperparameters, except (a) in experiments to better understand our algorithms (not a deployment setting) and (b) learning in simulators (a useful but relatively restricted setting). The paper should better place the problem setting for which you are developing AutoRL algorithms. This statement may seem to be more generally about AutoRL (where many papers do not do this); but we are here to discuss and improve this paper, and clarity of problem setting is critical. 

Decision: Accept (Poster)