ROMUL: Scale Adaptative Population Based Training

In most pragmatic settings , data augmentation and regularization are essential , and require hyperparameter search . Population based training ( PBT ) is an effective tool for efficiently finding them as well as schedules over hyperparameters . In this paper , we compare existing PBT algorithms and contribute a new one : ROMUL , for RObust MULtistep search , which adapts its stepsize over the course of training . We report competitive results with standard models on CIFAR ( image classification ) as well as Penn Tree Bank ( language modeling ) , which both depend on heavy regularization . We also open-source hoptim , a PBT library agnostic to the training framework , which is simple to use , reentrant , and provides good defaults with ROMUL . 1 INTRODUCTION . Hyperparameter tuning is essential for good performance in most machine learning tasks , and poses numerous challenges . First , optimal hyperparameter values can change over the course of training ( schedules ) , e.g . for learning rate , fine tuning phases , data augmentation . Hyperparameters values are also rarely independent from each other ( e.g . the magnitude of individual data augmentations depends on the number of data augmentations applied ) , and the search space grows exponentially with the number of hyperparameters . All of that search has to be performed within a computational budget , and sometimes even within a wall-clock time budget ( e.g . models that are frequently retrained on new data ) , requiring efficient parallelization . In practice , competitive existing methods range from random search ( Bergstra & Bengio , 2012 ) to more advanced methods ( that aim at being more compute-efficient ) like sequential search ( Bergstra et al. , 2011 ; 2013 ; Li et al. , 2018 ) , population based training ( PBT , e.g . Jaderberg et al . ( 2017 ) ; Ho et al . ( 2019 ) ) and search structured by the space of the hyperparameters ( Liu et al. , 2018 ; Cubuk et al. , 2019b ) . A major drawback of advanced hyperparameter optimization methods is that they themselves require attention from the user to reliably outperform random search . In this work , we empirically study the different training dynamics of data augmentation and regularization hyperparameters across vision and language modeling tasks , in particular for multistep ( sequential ) hyperparameter search . A common failure mode ( i ) is due to hyperparameters that have a different effect on the validation loss in the short and long terms , for instance using a smaller dropout often leads to faster but worse convergence . Another common problem ( ii ) is that successful searches are constrained on adequate “ hyper-hyperparameters ” ( such as value ranges or the search policy used , which in current methods are non-adaptative mutation steps ) . Our contributions can be summarized as follows : • We present a robust algorithm for leveraging population based training for hyperparameter search : ROMUL ( RObust MULtistep ) search , which addresses ( i ) and ( ii ) . We empirically study its benefits and limitations , and show that it provides good defaults that compare favorably to existing methods . • We open-source hoptim , a simple library for sequential hyperparameter search , that provides multiple optimizers ( including ROMUL ) , as well as toy benchmarks showcasing hyperparameter optimization problems we identified empirically and standard datasets . 2 HYPERPARAMETER OPTIMIZATION WITH POPULATION-BASED TRAINING . In this article , we refer to the family of algorithms that continuously tunes hyperparameters of a set of models over the course of their training as “ PBT algorithms ” or “ PBT optimizers ” . Hyperparameter optimization is thus a zero order optimization performed at a slower frequency than the ( often first order , e.g . SGD ) optimization of the model . A PBT step happens typically after a fixed number of epochs or updates of the model , often optimizing the loss from the validation set , continuing from an already produced “ parent ” checkpoint , and producing and evaluating a new checkpoint . At every PBT step , hyperparameters can be updated ( mutated ) , incremented or decremented by some number ( step size ) , or sampled . There are multiple aspects to consider when designing a PBT algorithm . Technical constraints : how the optimization is distributed , with a centralized or decentralized algorithm , workers to run the trainings , how failed workers are handled . They are solved in a unified manner in the experiments we performed , by the hoptim library to implement and compare multiple algorithms . It is decoupled from the scheduling of the jobs and designed to accommodate adding more workers to scale up the training , or fewer when some are killed , for example through preemption or time-out on a shared cluster . Optimization method : how the hyper-parameters are modified throughout the training , for instance through mutations . Selection process : which individual of the population are kept , both in term of hyper-parameters and state of the neural network ( checkpoint ) . For those last two points , some solutions are described below . 2.1 CHALLENGES . In order to have a clearer understanding of our proposed methods , we show below the main concerns we have observed in PBT : Anisotropy : by definition , the optimal value of the hyperparameters considered is unknown , and oftentimes the range ( or mutation scheme ) provided to the algorithm is a loose estimate only . As modifying two hyperparameters with the same step size can produce effects with very different magnitudes , the user is required to to normalize the search space . But pre-tuning the hyperparameter tuner itself can be cumbersome as dynamics evolve during training . Section 3.1 provides an example based on the Rosenbrock function which illustrates this issue and highlights the interest of adaptative mutations . Checkpoint vs. hyperparameters : comparing individuals in the population is extremely hard as improvements can be due to better hyperparameters , or better checkpoints ( including potentially better batches ) . Better performance through better checkpoints is an optimization phenomenon ( e.g . random restarts ) , that can bias the hyperparameter selection . We will detail this aspect in Section 4.2 . Short-term-long-term discordance : we observed empirically that hyperparameters which induce better performance in the short term are not always optimal in the longer term . This is a challenge that does not exist in classical static optimization , but is crucial for PBT since local minima are easy to reach and pose a danger for greedy algorithms . An example of such a parameter is the learning rate . Dropping the learning rate often induces a drop in the validation loss , even early in the training , and increasing it has the opposite effect , causing greedy PBT algorithms to reduce it to the minimum value too early , without being able to recover . We will detail this aspect in Section 4.1 . 2.2 DIFFERENTIAL EVOLUTION AND ROMUL . Differential Evolution Storn & Price ( 1997 ) ( DE ) is a standard black-box optimization method , for minimizing f : Rn → R. It operates on a population xi ∈ Rn for all i ∈ { 1 , ... , M } , M ≥ 4 , and indefinitely repeats the following steps for each individual xbase in the population to generate another individual called mutated vector that could replace xbase if better : 1. given the best individual xbest which minimizes f in the population , as well as two randomly selected ones xa and xb , compute the donor d , which will give part of its coefficients to the mutated vector . In the current-to-best/1 scheme we use , these are the base coefficients plus a term attracting to the best set of coefficients from the current population , and an additional random variation ( a standard value for Fi is F1 = F2 = 0.8 ) : d = xbase + F1 ( x best − xbase ) + F2 ( xb − xa ) ( 1 ) 2. create the new mutated vector x̃base by randomly selecting each component j ∈ { 1 , ... , n } of the base xbase or the donor d through the binary crossover operator : x̃basej = CHOICE ( xbasej , dj ) . This non-linear operation lets the optimization leave the vector span of the population . 3. compute f ( x̃base ) and replace xbase by x̃base within the population if and only if f ( x̃base ) ≤ f ( xbase ) . This method is interesting as it is already based on a population and adapts well to parameters with different dynamics while being simple and fully parallelizable . In particular , it does not rely on mutation ranges or step sizes - Equation 1 samples new parameters close to the current population , and as the population individuals go through selection this sampling is refined and becomes sharper around optimal values . In practice , if a parameter ’ s bounds are too loose or wrong , DE will eventually adapt after iterations of selection by removing individuals too far from the optimal value , and concentrate its computation budget on relevant values for this parameter . In order to use it for PBT , the set of hyper-parameters is converted to a vector in Rn using nevergrad parametrization system ( Rapin & Teytaud , 2018 ) . However , this basic version of differential evolution ( also implemented in nevergrad ) is not adapted to PBT . Indeed the training function f changes with the checkpoint as we are updating the parameters ( not the hyperparameters ) with a stochastic gradient from the task loss . The trend of f is therefore typically downwards during the training , younger generations/later epochs tending to have a lower loss than their parents ’ , biasing the hyperperameter selection process in favor of those of the children ( later steps of SGD updates ) instead of in favor of better hyperparameters . ROMUL We therefore propose an adaptation : a population of n individuals is trained , after finishing their step , individuals are compared to the rest of the population . If they have one of the n/k best loss ( we use k = 2 throughout ) , the training continues without changing the hyperparameters , otherwise , the hyperparameters are mutated . If the hyperparameters of an individual are mutated m times in a row ( we use m = 3 throughout ) , its checkpoint is killed and replaced by one of the n/k best individuals . The values of k and m are hyperparameters , although we did not vary them in any experiments : k = 2 allows to have , on average , one alternative ( mutated ) version to each of the ones we keep training without hyperparameter change , and m = 3 proved to be robust across our experiments , to select when to discard a checkpoint . If using lower m values , one should consider increasing the number of epochs per PBT step to prevent culling checkpoints too early ( see Section 4.1 and 4.2 ) . The mutation scheme is adapted to fit this use case . In Eq . 1 , xbase and xbest are both replaced by a randomly selected set of hyperparameters xc and xd from the best n/2 individuals ( “ rand-to-rand/1 ” scheme following ( Storn & Price , 1997 ; Das & Suganthan , 2011 ) notations ) . Replacing xbase aims at keeping the path through checkpoints unimodal , since keeping several modes with corresponding checkpoints is unnecessary . Replacing xbest by any other ” good ” ( top 50 % ) set of hyperparameters aims at avoiding early convergence , which we observed as one of the main problems during trainings . This also avoids a strong bias by a good checkpoint ( more on this in 4.2 ) . To avoid duplication of hyperparameters , we opt for making F1 and F2 random vectors instead of using the binary crossover non-linearity . In order to keep the initial scaling of DE , we chose F1 [ i ] uniformly distributed between 0 and 2F ( we use the common value for F from vanilla DE : F = 0.8 ) , and F2 [ i ] = 2F − F1 [ i ] , ∀i . This ensures that the sum F1 [ i ] + F2 [ i ] = 2F , ∀i , as in vanilla DE . With the elementwise multiplication , this yields d = xc + F1 ( xd − xc ) + F2 ( xb − xa ) .

Summary: The paper provides a new variant of PBT which utilizes ideas from differential evolution and cross-over. The original PBT and even initiator PBT do not perform crossover on the hyper-parameters, and insufficient cross-over may cause PBT to perform greedy in the initial phases which ends up with a suboptimal convergence. The investigation of better cross-over in PBT is itself an interesting research direction and the authors demonstrated its effectiveness in standard benchmarks and data augmentation tasks. The improvements of ROMUL-PBT are also helpful to the community since PBT has been applied in a variety of real world applications.