Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRL

1 Introduction . Reinforcement Learning ( RL [ 75 ] ) is a paradigm whereby agents learn to make sequential decisions through trial and error . In the past few years there have been a series of breakthroughs using RL in games [ 70 , 44 , 6 ] and robotics [ 50 , 33 ] , which have led to a surge of interest in the machine learning community . Beyond these examples , RL offers the potential to impact vast swathes of society , from autonomous vehicles to robotic applications in healthcare and industry . Despite the promising results , RL is notoriously difficult to use in practice . In particular , RL algorithms are incredibly sensitive to hyperparameters [ 23 , 2 , 18 , 27 , 9 , 47 ] , with careful tuning often the difference between success and failure . Furthermore , as new and more complex algorithms are introduced , the search space continues to grow [ 2 ] . As a result , it is often challenging to reproduce RL results given the significant number of moving parts in modern algorithms [ 2 ] . This makes it almost impossible to apply these methods in novel settings , where optimal hyperparameters are unknown . On the other hand , the capability of current methods may be understated , as better configurations could boost performance [ 10 , 84 ] . In this work we focus on the recent impressive results for Population Based Training ( PBT , [ 29 , 39 ] ) , which has demonstrated strong performance in a variety of prominent RL settings [ 65 , 84 , 43 , 19 , 28 ] . PBT works by training agents in parallel , with an evolutionary outer loop optimizing hyperparameters , periodically replacing weaker agents with perturbations of stronger ones . However , since PBT relies on random search to explore the hyperparameter space , it requires a large population size which can be prohibitively expensive for small and even medium-sized labs . úCorrespondence to jackph @ robots.ox.ac.uk . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In order to achieve similar success with a smaller computational budget , the recent Population Based Bandits ( PB2 , [ 52 ] ) algorithm improved sample efficiency by introducing a probabilistic exploration step , backed by theoretical guarantees . Unfortunately , a key limitation of PB2 is that it only addresses the problem of efficiently selecting continuous hyperparameters , inheriting the random search method for categorical variables from the original PBT . This is not only inefficient but also crucially ignores potential dependence between continuous and categorical variables . Since the Gaussian Process ( GP , [ 58 ] ) model is completely unaware of the changing categories , it is unable to differentiate between trials that may have completely different categorical hyperparameters . In this paper we introduce a new hierarchical approach to PB2 , which can efficiently model both continuous and categorical variables , with theoretical guarantees . Our main contributions are the following : Technical : We introduce a new PB2 explore step which can efficiently choose between both categorical and continuous hyperparameters in a population based training setup . In particular , we propose a new time-varying batch multi-armed bandit algorithm , and introduce two hierarchical algorithms which condition on the selected categorical variables . We show our new approach achieves sublinear regret , extending the results for the continuous case with PB2 . Practical : We scale our approach to test generalization on the Procgen benchmark [ 13 ] , an active area of research . We demonstrate improved performance when explicitly modelling dependence between data augmentation type and continuous hyperparameters ( such as learning rate ) vs. baselines using random search to select the data augmentation . 2 Related Work . This work contributes to the emerging field of AutoRL [ 11 ] , which seeks to automate elements of the reinforcement learning ( RL ) training procedure . Automating RL hyperparameter tuning has been studied since the 1980s [ 4 ] , with a surge in recent interest due to the increasing complexity of modern algorithms [ 84 , 54 ] . The scope for AutoRL is broad , from using RL in novel real-world problems where hyperparameters are unknown [ 64 ] , to improving the performance of existing methods [ 10 ] . Recent successes in AutoRL include learning differentiable hyperparameters with meta-gradients [ 81 , 83 ] , while it has recently been shown it is even possible to learn algorithms [ 49 , 12 ] . In this paper we focus on the class of Population Based Training ( PBT [ 29 ] ) methods . Inspired by how humans run experiments , PBT trains agents in parallel and periodically replaces the weakest agents with variations of the stronger ones , learning a hyperparameter schedule on the fly , in a single training run . While PBT is applicable in any AutoML [ 26 ] setting , it has been particularly impactful in deep RL [ 19 , 84 , 65 ] . PBT was recently shown to work well with a shared replay buffer [ 20 ] , but this only applies for off policy methods . This paper builds on the recently introduced Population Based Bandits ( PB2 , [ 52 ] ) algorithm , which employs a Bayesian Optimization ( BO , [ 8 , 24 , 71 , 17 , 63 , 46 ] ) approach to select hyperparameters during the course of training . Concretely PB2 casts the hyperparameter selection step of PBT as a batch time-varying GP bandit optimization problem [ 7 ] . The strength of PB2 lies in finding optimal configurations with a small population size , yet at present it has only been developed for continuous hyperparameters , relying on random search for the categorical variables . Categorical variables are prominent in RL , for example the choice of exploration strategy [ 77 ] , or even algorithm class . A prominent recent categorical variable is the choice of data augmentation , which has been shown to significantly improve efficiency and generalization in RL [ 37 , 35 , 38 , 80 ] , reducing observational overfitting [ 73 ] . Recent work introduced automatic data augmentation [ 57 ] , which shows significant improvement over a static baseline by learning the data augmentation on the fly ( with a single agent ) . However , this approach relies on a grid search over new hyperparameters , keeping all others fixed . We take inspiration from this result and learn both data augmentation and baseline RL algorithm hyperparameters jointly on the fly , with a population of agents . PBT has also been extended to data augmentation ( PBA , [ 25 ] ) with strong results in supervised learning tasks . However , in this setting the data augmentation was set up as a continuous variable . A high level summary of differences vs. prior work is shown in Table 1 . 3 The Case for AutoRL . In this section we introduce the reinforcement learning ( RL , [ 75 ] ) paradigm , before making the case for automating hyperparameter selection for policy gradient algorithms . 3.1 Reinforcement Learning Background . A Markov Decision Process ( MDP ) is a tuple ( S , A , P , R , “ ) , where for each time step t = 0 , 1 , 2 . . . the environment provides the agent with an observation st œ S , the agent responds by selecting an action at œ S , and then the environment provides the next reward rt , discount “ t+1 , and state st+1 . Reinforcement learning ( RL ) considers the problem of learning a policy fi which selects actions that maximize the expected total discounted reward [ 5 , 76 , 75 ] . While our method can be applied to any RL algorithm , for simplicity we focus our discussion on policy gradient algorithms , which directly seek to maximize the policy with respect to expected reward . For a policy fi : S æ A parameterized by ◊ the objective is as follows : J ( fi◊ ) = E·≥fi◊ R ( · ) where · = { s1 , a1 , r1 , . . . , sH , aH , rH } for some horizon H . Of course , we wish to maximize this , so optimize the policy by taking the following gradient steps : ◊t+1 = ◊t + –Ò◊J ( fi◊t ) . 3.2 Case study : Proximal Policy Optimization . Proximal Policy Optimization ( PPO [ 68 ] ) is one of the most widely used RL algorithms , achieving strong performance in continuous control problems , and even scaling to large scale games [ 6 ] . Building on Trust Region Policy Optimization ( TRPO [ 66 ] ) , the success of PPO comes from using a clipped loss function as follows : LPPO ( ◊ ) = min A fi◊ ( a|s ) fiµ ( a|s ) A fiµ , g ( ◊ , µ ) Afiµ B , where g ( ◊ , µ ) = clip 3 fi◊ ( a|s ) fiµ ( a|s ) , 1 ≠ ‘ , 1 + ‘ 4 for a previous policy fiµ , an advantage function A and a clipping hyperparameter ‘ . As can be seen , PPO ’ s success hinges on several new hyperparameters . As well as the learning rate – , clip parameter ‘ , decay “ , there is also often another hyperparameter in the advantage function ⁄ [ 67 ] . As if this wasn ’ t enough , a recent study showed we also need to consider code level hyperparameters [ 18 ] after recent work found nine implementation details ( such as reward scaling ) had a significant impact on performance . Thus , it is clear that while PPO can be effective , it requires careful tuning if it is to be used effectively for novel problems . In addition , a recent study investigating actor-critic algorithms [ 2 ] found that not only are there many more hyperparameters than previously thought ( for example weight initialization strategy ) , but their optimal values are dependent on one another , producing a combinatorially large search space . This problem is only increasing as novel methods extending upon PPO they almost always add additional hyperparameters [ 57 ] . Thus , if it wasn ’ t enough that PPO itself has tens of hyperparameters , new additions may render all previous values sub-optimal since they may be dependent on the new variables . The main hypothesis in this paper is that we can improve performance of baseline algorithms by jointly learning several hyperparameters on the fly . 4 Population Based Bandits . We consider the problem of optimizing a population of agents in parallel , dynamically adapting their weights and hyperparameters such that strong performance can be achieved in a single training run . Following existing work [ 29 , 52 ] , we consider two sub-routines , explore and exploit . We train for a total of T steps , evaluating performance every tready < T steps . For the exploit step , the weights of the bottom agents are replaced by those from a randomly sampled agent from the set of best performing agents , in what is called truncation selection . The next step is to select new hyperparameters , with the explore step . The full procedure is shown in Algorithm 1 and Fig . 1 . Algorithm 1 Population Based Training ( Bandits ) Initialize : Population network weights , hyperparameters , empty dataset . for ( in parallel ) t = 1 , . . . , T ≠ 1 do 1 . Train Models 2 . Evaluate Models and Record Data 3 . If t mod tready = 0 : • Exploit : Replace the weights of weaker agents with those from stronger agents . • Explore : If weights were replaced , select new hyperparameters . Return the best trained model ◊ The key difference between Population Based Training ( PBT , [ 29 ] ) and Population Based Bandits ( PB2 , [ 52 ] ) is the explore step . PBT selects new hyperameters using random perturbations , hence it often performs poorly in a resource constrained setting . By contrast PB2 ( blue in Alg . 1 ) models the data from previous trials to efficiently explore the hyperparameter space . We focus on the PB2 framework and introduce new algorithms for mixed-input hyperparameters . 4.1 Online Hyperparameter Selection as GP-Bandit Optimization We consider the problem of selecting optimal hyperparameters , xb t , from a compact , convex subset D œ Rd where d is the number of hyperparameters . Here the index b refers to the bth agent in a population , and the subscript t represents the number of timesteps elapsed during the training of a neural network . In particular , we consider the schedule of optimal hyperparameters over time ! x b t `` t=1 , ... T . Let Ft ( xt ) be an objective function under a given set of hyperparameters at timestep t. Here we take Ft ( xt ) to be the reward for a deep RL agent . Our goal is to maximize the final performance FT ( xT ) . Following [ 52 ] we formulate this problem as optimizing the time-varying black-box reward function ft , over D. Every tready steps , we observe and record noisy observations , yt = ft ( xt ) + ‘ t , where ‘ t ≥ N ( 0 , ‡2I ) for some fixed ‡2 . Note the function ft represents the change in Ft after training for tready steps , i.e . Ft ≠ Ft≠tready . We define the best choice at each timestep as xú t = arg maxxtœD ft ( xt ) , and so the regret of each decision as rt = ft ( xút ) ≠ ft ( xt ) . Minimizing the regret of each decision is equivalent to maximizing the final reward [ 52 ] , i.e . max FT ( xT ) = min q T t=1 rt ( xt ) . We now discuss how PB2 seeks to minimize regret for continuous hyperparameters .

Summary: This paper extends the existing AutoRL algorithm, PB2, that was specifically designed to work with continuous hyperparameters by enabling it to efficiently search over both continuous and categorical hyperparameters of reinforcement learning solution methods. To do so, it first proposes a new multi-armed bandit algorithm called TV.EXP3.M (an extension of EXP3.M that varies with time) to select a subset of a given set of categories. Then, the paper incorporates this technique into the exploration stage of the PB2 algorithm to explore and select the categorical variables by taking into account the dependence of continuous variables on them. It takes two different approaches for modeling this dependency, one by incorporating it into the GP kernel and the other by filtering continuous variables based on the chosen categories that result in PB2-Mix and PB2-Mult algorithms. To evaluate the proposed algorithms, they first theoretically analyze PB2-Mult and TV.EXP3.M algorithm by showing they have cumulative regret bound and sublinear regret bound respectively. Then, they empirically evaluate their techniques along with PBT and original PB2 as the baselines on a toy environment and seven Procgen games and show that PB2-Mix achieves a similar or better performance compared to other baselines on these environments.