ROMUL: Scale Adaptative Population Based Training

In most pragmatic settings , data augmentation and regularization are essential , and require hyperparameter search . Population based training ( PBT ) is an effective tool for efficiently finding them as well as schedules over hyperparameters . In this paper , we compare existing PBT algorithms and contribute a new one : ROMUL , for RObust MULtistep search , which adapts its stepsize over the course of training . We report competitive results with standard models on CIFAR ( image classification ) as well as Penn Tree Bank ( language modeling ) , which both depend on heavy regularization . We also open-source hoptim , a PBT library agnostic to the training framework , which is simple to use , reentrant , and provides good defaults with ROMUL . 1 INTRODUCTION . Hyperparameter tuning is essential for good performance in most machine learning tasks , and poses numerous challenges . First , optimal hyperparameter values can change over the course of training ( schedules ) , e.g . for learning rate , fine tuning phases , data augmentation . Hyperparameters values are also rarely independent from each other ( e.g . the magnitude of individual data augmentations depends on the number of data augmentations applied ) , and the search space grows exponentially with the number of hyperparameters . All of that search has to be performed within a computational budget , and sometimes even within a wall-clock time budget ( e.g . models that are frequently retrained on new data ) , requiring efficient parallelization . In practice , competitive existing methods range from random search ( Bergstra & Bengio , 2012 ) to more advanced methods ( that aim at being more compute-efficient ) like sequential search ( Bergstra et al. , 2011 ; 2013 ; Li et al. , 2018 ) , population based training ( PBT , e.g . Jaderberg et al . ( 2017 ) ; Ho et al . ( 2019 ) ) and search structured by the space of the hyperparameters ( Liu et al. , 2018 ; Cubuk et al. , 2019b ) . A major drawback of advanced hyperparameter optimization methods is that they themselves require attention from the user to reliably outperform random search . In this work , we empirically study the different training dynamics of data augmentation and regularization hyperparameters across vision and language modeling tasks , in particular for multistep ( sequential ) hyperparameter search . A common failure mode ( i ) is due to hyperparameters that have a different effect on the validation loss in the short and long terms , for instance using a smaller dropout often leads to faster but worse convergence . Another common problem ( ii ) is that successful searches are constrained on adequate “ hyper-hyperparameters ” ( such as value ranges or the search policy used , which in current methods are non-adaptative mutation steps ) . Our contributions can be summarized as follows : • We present a robust algorithm for leveraging population based training for hyperparameter search : ROMUL ( RObust MULtistep ) search , which addresses ( i ) and ( ii ) . We empirically study its benefits and limitations , and show that it provides good defaults that compare favorably to existing methods . • We open-source hoptim , a simple library for sequential hyperparameter search , that provides multiple optimizers ( including ROMUL ) , as well as toy benchmarks showcasing hyperparameter optimization problems we identified empirically and standard datasets . 2 HYPERPARAMETER OPTIMIZATION WITH POPULATION-BASED TRAINING . In this article , we refer to the family of algorithms that continuously tunes hyperparameters of a set of models over the course of their training as “ PBT algorithms ” or “ PBT optimizers ” . Hyperparameter optimization is thus a zero order optimization performed at a slower frequency than the ( often first order , e.g . SGD ) optimization of the model . A PBT step happens typically after a fixed number of epochs or updates of the model , often optimizing the loss from the validation set , continuing from an already produced “ parent ” checkpoint , and producing and evaluating a new checkpoint . At every PBT step , hyperparameters can be updated ( mutated ) , incremented or decremented by some number ( step size ) , or sampled . There are multiple aspects to consider when designing a PBT algorithm . Technical constraints : how the optimization is distributed , with a centralized or decentralized algorithm , workers to run the trainings , how failed workers are handled . They are solved in a unified manner in the experiments we performed , by the hoptim library to implement and compare multiple algorithms . It is decoupled from the scheduling of the jobs and designed to accommodate adding more workers to scale up the training , or fewer when some are killed , for example through preemption or time-out on a shared cluster . Optimization method : how the hyper-parameters are modified throughout the training , for instance through mutations . Selection process : which individual of the population are kept , both in term of hyper-parameters and state of the neural network ( checkpoint ) . For those last two points , some solutions are described below . 2.1 CHALLENGES . In order to have a clearer understanding of our proposed methods , we show below the main concerns we have observed in PBT : Anisotropy : by definition , the optimal value of the hyperparameters considered is unknown , and oftentimes the range ( or mutation scheme ) provided to the algorithm is a loose estimate only . As modifying two hyperparameters with the same step size can produce effects with very different magnitudes , the user is required to to normalize the search space . But pre-tuning the hyperparameter tuner itself can be cumbersome as dynamics evolve during training . Section 3.1 provides an example based on the Rosenbrock function which illustrates this issue and highlights the interest of adaptative mutations . Checkpoint vs. hyperparameters : comparing individuals in the population is extremely hard as improvements can be due to better hyperparameters , or better checkpoints ( including potentially better batches ) . Better performance through better checkpoints is an optimization phenomenon ( e.g . random restarts ) , that can bias the hyperparameter selection . We will detail this aspect in Section 4.2 . Short-term-long-term discordance : we observed empirically that hyperparameters which induce better performance in the short term are not always optimal in the longer term . This is a challenge that does not exist in classical static optimization , but is crucial for PBT since local minima are easy to reach and pose a danger for greedy algorithms . An example of such a parameter is the learning rate . Dropping the learning rate often induces a drop in the validation loss , even early in the training , and increasing it has the opposite effect , causing greedy PBT algorithms to reduce it to the minimum value too early , without being able to recover . We will detail this aspect in Section 4.1 . 2.2 DIFFERENTIAL EVOLUTION AND ROMUL . Differential Evolution Storn & Price ( 1997 ) ( DE ) is a standard black-box optimization method , for minimizing f : Rn → R. It operates on a population xi ∈ Rn for all i ∈ { 1 , ... , M } , M ≥ 4 , and indefinitely repeats the following steps for each individual xbase in the population to generate another individual called mutated vector that could replace xbase if better : 1. given the best individual xbest which minimizes f in the population , as well as two randomly selected ones xa and xb , compute the donor d , which will give part of its coefficients to the mutated vector . In the current-to-best/1 scheme we use , these are the base coefficients plus a term attracting to the best set of coefficients from the current population , and an additional random variation ( a standard value for Fi is F1 = F2 = 0.8 ) : d = xbase + F1 ( x best − xbase ) + F2 ( xb − xa ) ( 1 ) 2. create the new mutated vector x̃base by randomly selecting each component j ∈ { 1 , ... , n } of the base xbase or the donor d through the binary crossover operator : x̃basej = CHOICE ( xbasej , dj ) . This non-linear operation lets the optimization leave the vector span of the population . 3. compute f ( x̃base ) and replace xbase by x̃base within the population if and only if f ( x̃base ) ≤ f ( xbase ) . This method is interesting as it is already based on a population and adapts well to parameters with different dynamics while being simple and fully parallelizable . In particular , it does not rely on mutation ranges or step sizes - Equation 1 samples new parameters close to the current population , and as the population individuals go through selection this sampling is refined and becomes sharper around optimal values . In practice , if a parameter ’ s bounds are too loose or wrong , DE will eventually adapt after iterations of selection by removing individuals too far from the optimal value , and concentrate its computation budget on relevant values for this parameter . In order to use it for PBT , the set of hyper-parameters is converted to a vector in Rn using nevergrad parametrization system ( Rapin & Teytaud , 2018 ) . However , this basic version of differential evolution ( also implemented in nevergrad ) is not adapted to PBT . Indeed the training function f changes with the checkpoint as we are updating the parameters ( not the hyperparameters ) with a stochastic gradient from the task loss . The trend of f is therefore typically downwards during the training , younger generations/later epochs tending to have a lower loss than their parents ’ , biasing the hyperperameter selection process in favor of those of the children ( later steps of SGD updates ) instead of in favor of better hyperparameters . ROMUL We therefore propose an adaptation : a population of n individuals is trained , after finishing their step , individuals are compared to the rest of the population . If they have one of the n/k best loss ( we use k = 2 throughout ) , the training continues without changing the hyperparameters , otherwise , the hyperparameters are mutated . If the hyperparameters of an individual are mutated m times in a row ( we use m = 3 throughout ) , its checkpoint is killed and replaced by one of the n/k best individuals . The values of k and m are hyperparameters , although we did not vary them in any experiments : k = 2 allows to have , on average , one alternative ( mutated ) version to each of the ones we keep training without hyperparameter change , and m = 3 proved to be robust across our experiments , to select when to discard a checkpoint . If using lower m values , one should consider increasing the number of epochs per PBT step to prevent culling checkpoints too early ( see Section 4.1 and 4.2 ) . The mutation scheme is adapted to fit this use case . In Eq . 1 , xbase and xbest are both replaced by a randomly selected set of hyperparameters xc and xd from the best n/2 individuals ( “ rand-to-rand/1 ” scheme following ( Storn & Price , 1997 ; Das & Suganthan , 2011 ) notations ) . Replacing xbase aims at keeping the path through checkpoints unimodal , since keeping several modes with corresponding checkpoints is unnecessary . Replacing xbest by any other ” good ” ( top 50 % ) set of hyperparameters aims at avoiding early convergence , which we observed as one of the main problems during trainings . This also avoids a strong bias by a good checkpoint ( more on this in 4.2 ) . To avoid duplication of hyperparameters , we opt for making F1 and F2 random vectors instead of using the binary crossover non-linearity . In order to keep the initial scaling of DE , we chose F1 [ i ] uniformly distributed between 0 and 2F ( we use the common value for F from vanilla DE : F = 0.8 ) , and F2 [ i ] = 2F − F1 [ i ] , ∀i . This ensures that the sum F1 [ i ] + F2 [ i ] = 2F , ∀i , as in vanilla DE . With the elementwise multiplication , this yields d = xc + F1 ( xd − xc ) + F2 ( xb − xa ) .

Summary: This paper focuses on issues in the popular PBT algorithm for hyperparameter optimization. It investigates the 1) step size (which is typically a constant multiplier) 2) the variance induced by better weights and 3) the greediness of the algorithm, which they refer to as short-term vs. long term effects. These issues are well motivated, and it is intuitive that they are flaws in the original algorithm. The proposed approach is to use Differential Evolution which the authors claim makes the hyperparameter selection more robust. The paper also introduces a new library for online hyperparameter tuning.