Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRL

1 Introduction . Reinforcement Learning ( RL [ 75 ] ) is a paradigm whereby agents learn to make sequential decisions through trial and error . In the past few years there have been a series of breakthroughs using RL in games [ 70 , 44 , 6 ] and robotics [ 50 , 33 ] , which have led to a surge of interest in the machine learning community . Beyond these examples , RL offers the potential to impact vast swathes of society , from autonomous vehicles to robotic applications in healthcare and industry . Despite the promising results , RL is notoriously difficult to use in practice . In particular , RL algorithms are incredibly sensitive to hyperparameters [ 23 , 2 , 18 , 27 , 9 , 47 ] , with careful tuning often the difference between success and failure . Furthermore , as new and more complex algorithms are introduced , the search space continues to grow [ 2 ] . As a result , it is often challenging to reproduce RL results given the significant number of moving parts in modern algorithms [ 2 ] . This makes it almost impossible to apply these methods in novel settings , where optimal hyperparameters are unknown . On the other hand , the capability of current methods may be understated , as better configurations could boost performance [ 10 , 84 ] . In this work we focus on the recent impressive results for Population Based Training ( PBT , [ 29 , 39 ] ) , which has demonstrated strong performance in a variety of prominent RL settings [ 65 , 84 , 43 , 19 , 28 ] . PBT works by training agents in parallel , with an evolutionary outer loop optimizing hyperparameters , periodically replacing weaker agents with perturbations of stronger ones . However , since PBT relies on random search to explore the hyperparameter space , it requires a large population size which can be prohibitively expensive for small and even medium-sized labs . úCorrespondence to jackph @ robots.ox.ac.uk . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In order to achieve similar success with a smaller computational budget , the recent Population Based Bandits ( PB2 , [ 52 ] ) algorithm improved sample efficiency by introducing a probabilistic exploration step , backed by theoretical guarantees . Unfortunately , a key limitation of PB2 is that it only addresses the problem of efficiently selecting continuous hyperparameters , inheriting the random search method for categorical variables from the original PBT . This is not only inefficient but also crucially ignores potential dependence between continuous and categorical variables . Since the Gaussian Process ( GP , [ 58 ] ) model is completely unaware of the changing categories , it is unable to differentiate between trials that may have completely different categorical hyperparameters . In this paper we introduce a new hierarchical approach to PB2 , which can efficiently model both continuous and categorical variables , with theoretical guarantees . Our main contributions are the following : Technical : We introduce a new PB2 explore step which can efficiently choose between both categorical and continuous hyperparameters in a population based training setup . In particular , we propose a new time-varying batch multi-armed bandit algorithm , and introduce two hierarchical algorithms which condition on the selected categorical variables . We show our new approach achieves sublinear regret , extending the results for the continuous case with PB2 . Practical : We scale our approach to test generalization on the Procgen benchmark [ 13 ] , an active area of research . We demonstrate improved performance when explicitly modelling dependence between data augmentation type and continuous hyperparameters ( such as learning rate ) vs. baselines using random search to select the data augmentation . 2 Related Work . This work contributes to the emerging field of AutoRL [ 11 ] , which seeks to automate elements of the reinforcement learning ( RL ) training procedure . Automating RL hyperparameter tuning has been studied since the 1980s [ 4 ] , with a surge in recent interest due to the increasing complexity of modern algorithms [ 84 , 54 ] . The scope for AutoRL is broad , from using RL in novel real-world problems where hyperparameters are unknown [ 64 ] , to improving the performance of existing methods [ 10 ] . Recent successes in AutoRL include learning differentiable hyperparameters with meta-gradients [ 81 , 83 ] , while it has recently been shown it is even possible to learn algorithms [ 49 , 12 ] . In this paper we focus on the class of Population Based Training ( PBT [ 29 ] ) methods . Inspired by how humans run experiments , PBT trains agents in parallel and periodically replaces the weakest agents with variations of the stronger ones , learning a hyperparameter schedule on the fly , in a single training run . While PBT is applicable in any AutoML [ 26 ] setting , it has been particularly impactful in deep RL [ 19 , 84 , 65 ] . PBT was recently shown to work well with a shared replay buffer [ 20 ] , but this only applies for off policy methods . This paper builds on the recently introduced Population Based Bandits ( PB2 , [ 52 ] ) algorithm , which employs a Bayesian Optimization ( BO , [ 8 , 24 , 71 , 17 , 63 , 46 ] ) approach to select hyperparameters during the course of training . Concretely PB2 casts the hyperparameter selection step of PBT as a batch time-varying GP bandit optimization problem [ 7 ] . The strength of PB2 lies in finding optimal configurations with a small population size , yet at present it has only been developed for continuous hyperparameters , relying on random search for the categorical variables . Categorical variables are prominent in RL , for example the choice of exploration strategy [ 77 ] , or even algorithm class . A prominent recent categorical variable is the choice of data augmentation , which has been shown to significantly improve efficiency and generalization in RL [ 37 , 35 , 38 , 80 ] , reducing observational overfitting [ 73 ] . Recent work introduced automatic data augmentation [ 57 ] , which shows significant improvement over a static baseline by learning the data augmentation on the fly ( with a single agent ) . However , this approach relies on a grid search over new hyperparameters , keeping all others fixed . We take inspiration from this result and learn both data augmentation and baseline RL algorithm hyperparameters jointly on the fly , with a population of agents . PBT has also been extended to data augmentation ( PBA , [ 25 ] ) with strong results in supervised learning tasks . However , in this setting the data augmentation was set up as a continuous variable . A high level summary of differences vs. prior work is shown in Table 1 . 3 The Case for AutoRL . In this section we introduce the reinforcement learning ( RL , [ 75 ] ) paradigm , before making the case for automating hyperparameter selection for policy gradient algorithms . 3.1 Reinforcement Learning Background . A Markov Decision Process ( MDP ) is a tuple ( S , A , P , R , “ ) , where for each time step t = 0 , 1 , 2 . . . the environment provides the agent with an observation st œ S , the agent responds by selecting an action at œ S , and then the environment provides the next reward rt , discount “ t+1 , and state st+1 . Reinforcement learning ( RL ) considers the problem of learning a policy fi which selects actions that maximize the expected total discounted reward [ 5 , 76 , 75 ] . While our method can be applied to any RL algorithm , for simplicity we focus our discussion on policy gradient algorithms , which directly seek to maximize the policy with respect to expected reward . For a policy fi : S æ A parameterized by ◊ the objective is as follows : J ( fi◊ ) = E·≥fi◊ R ( · ) where · = { s1 , a1 , r1 , . . . , sH , aH , rH } for some horizon H . Of course , we wish to maximize this , so optimize the policy by taking the following gradient steps : ◊t+1 = ◊t + –Ò◊J ( fi◊t ) . 3.2 Case study : Proximal Policy Optimization . Proximal Policy Optimization ( PPO [ 68 ] ) is one of the most widely used RL algorithms , achieving strong performance in continuous control problems , and even scaling to large scale games [ 6 ] . Building on Trust Region Policy Optimization ( TRPO [ 66 ] ) , the success of PPO comes from using a clipped loss function as follows : LPPO ( ◊ ) = min A fi◊ ( a|s ) fiµ ( a|s ) A fiµ , g ( ◊ , µ ) Afiµ B , where g ( ◊ , µ ) = clip 3 fi◊ ( a|s ) fiµ ( a|s ) , 1 ≠ ‘ , 1 + ‘ 4 for a previous policy fiµ , an advantage function A and a clipping hyperparameter ‘ . As can be seen , PPO ’ s success hinges on several new hyperparameters . As well as the learning rate – , clip parameter ‘ , decay “ , there is also often another hyperparameter in the advantage function ⁄ [ 67 ] . As if this wasn ’ t enough , a recent study showed we also need to consider code level hyperparameters [ 18 ] after recent work found nine implementation details ( such as reward scaling ) had a significant impact on performance . Thus , it is clear that while PPO can be effective , it requires careful tuning if it is to be used effectively for novel problems . In addition , a recent study investigating actor-critic algorithms [ 2 ] found that not only are there many more hyperparameters than previously thought ( for example weight initialization strategy ) , but their optimal values are dependent on one another , producing a combinatorially large search space . This problem is only increasing as novel methods extending upon PPO they almost always add additional hyperparameters [ 57 ] . Thus , if it wasn ’ t enough that PPO itself has tens of hyperparameters , new additions may render all previous values sub-optimal since they may be dependent on the new variables . The main hypothesis in this paper is that we can improve performance of baseline algorithms by jointly learning several hyperparameters on the fly . 4 Population Based Bandits . We consider the problem of optimizing a population of agents in parallel , dynamically adapting their weights and hyperparameters such that strong performance can be achieved in a single training run . Following existing work [ 29 , 52 ] , we consider two sub-routines , explore and exploit . We train for a total of T steps , evaluating performance every tready < T steps . For the exploit step , the weights of the bottom agents are replaced by those from a randomly sampled agent from the set of best performing agents , in what is called truncation selection . The next step is to select new hyperparameters , with the explore step . The full procedure is shown in Algorithm 1 and Fig . 1 . Algorithm 1 Population Based Training ( Bandits ) Initialize : Population network weights , hyperparameters , empty dataset . for ( in parallel ) t = 1 , . . . , T ≠ 1 do 1 . Train Models 2 . Evaluate Models and Record Data 3 . If t mod tready = 0 : • Exploit : Replace the weights of weaker agents with those from stronger agents . • Explore : If weights were replaced , select new hyperparameters . Return the best trained model ◊ The key difference between Population Based Training ( PBT , [ 29 ] ) and Population Based Bandits ( PB2 , [ 52 ] ) is the explore step . PBT selects new hyperameters using random perturbations , hence it often performs poorly in a resource constrained setting . By contrast PB2 ( blue in Alg . 1 ) models the data from previous trials to efficiently explore the hyperparameter space . We focus on the PB2 framework and introduce new algorithms for mixed-input hyperparameters . 4.1 Online Hyperparameter Selection as GP-Bandit Optimization We consider the problem of selecting optimal hyperparameters , xb t , from a compact , convex subset D œ Rd where d is the number of hyperparameters . Here the index b refers to the bth agent in a population , and the subscript t represents the number of timesteps elapsed during the training of a neural network . In particular , we consider the schedule of optimal hyperparameters over time ! x b t `` t=1 , ... T . Let Ft ( xt ) be an objective function under a given set of hyperparameters at timestep t. Here we take Ft ( xt ) to be the reward for a deep RL agent . Our goal is to maximize the final performance FT ( xT ) . Following [ 52 ] we formulate this problem as optimizing the time-varying black-box reward function ft , over D. Every tready steps , we observe and record noisy observations , yt = ft ( xt ) + ‘ t , where ‘ t ≥ N ( 0 , ‡2I ) for some fixed ‡2 . Note the function ft represents the change in Ft after training for tready steps , i.e . Ft ≠ Ft≠tready . We define the best choice at each timestep as xú t = arg maxxtœD ft ( xt ) , and so the regret of each decision as rt = ft ( xút ) ≠ ft ( xt ) . Minimizing the regret of each decision is equivalent to maximizing the final reward [ 52 ] , i.e . max FT ( xT ) = min q T t=1 rt ( xt ) . We now discuss how PB2 seeks to minimize regret for continuous hyperparameters .

Summary: This paper presents an extension of population based bandits (PB2) and the proposed method can handle both continuous variables and categorical variables at the same time. A new algorithm for parallel multi-armed bandits is also presented. The proposed hierarchical approach shows better empirical performance than other approaches. Authors also give theoretical results w.r.t. the cumulative regret bound.