ROMUL: Scale Adaptative Population Based Training
Authors: Daniel HAZIZA
Jérémy Rapin
Gabriel Synnaeve
Venue:  (2021)
Keywords: hyperparameter search
population based training
differential evolution
hyperparameter optimization
online optimization
deep learning

Abstract: In most pragmatic settings, data augmentation and regularization are essential, and require hyperparameter search.
Population based training (PBT) is an effective tool for efficiently finding them as well as schedules over hyperparameters.
In this paper, we compare existing PBT algorithms and contribute a new one: ROMUL, for RObust MULtistep search, which adapts its stepsize over the course of training.
We report competitive results with standard models on CIFAR (image classification) as well as Penn Tree Bank (language modeling), which both depend on heavy regularization.
We also open-source hoptim, a PBT library agnostic to the training framework, which is simple to use, reentrant, and provides good defaults with ROMUL.

Under review as a conference paper at ICLR 2021
ROMUL: SCALE ADAPTATIVE POPULATION BASED TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
In most pragmatic settings, data augmentation and regularization are essential, and require hyperparameter search. Population based training (PBT) is an effec- tive tool for efficiently finding them as well as schedules over hyperparameters. this paper, we compare existing PBT algorithms and contribute new one: RO- MUL, for RObust MULtistep search, which adapts its stepsize over the course of training. We report competitive results with standard models CIFAR (image classification) as well as Penn Tree Bank (language modeling), which both depend on heavy regularization. We also open-source hoptim, PBT library agnostic to the training framework, which is simple to use, reentrant, and provides good defaults with ROMUL.
INTRODUCTION
Hyperparameter tuning is essential for good performance in most machine learning tasks, and poses numerous challenges. First, optimal hyperparameter values can change over the course of training (schedules), e.g. for learning rate, fine tuning phases, data augmentation. Hyperparameters values also rarely independent from each other (e.g. the magnitude of individual data augmentations depends on the number of data augmentations applied), and the search space grows exponentially with the number of hyperparameters. All of that search has be performed within computational budget, and sometimes even within a wall-clock time budget (e.g. models that are frequently re- trained on new data), requiring efficient parallelization. In practice, competitive existing methods range from random search (Bergstra & Bengio 2012) to more advanced methods (that aim at being more compute-effie (fficceetfi like sequential search Bergstra et al., 2011 2013 Li al. 2018), popula- tion based training (PBT. e.g. Jaderberg et al. (2017): Ho et al. (2019)) and search structured by the
space of the hyperparameters (Liu al. 2018 Cubuk | (011bb).
A major drawback of advanced hyperparameter optimization methods is that they themselves require attention from the user reliably outperform random search. In this work, we empirically study the different training dynamics of data augmentation and regularization hyperparameters across vision and language modeling tasks, particular for multistep (sequential) hyperparameter search. A common failure mode (i) is due to hyperparameters that have different effect on the validation loss the short and long terms, for instance using a smaller dropout often leads to faster but worse convergence. Another common problem (ii) is that successful searches constrained adequate tiyper-lyperppramettrr"" (such as value ranges or the search policy used, which in current methods non-adaptative mutation steps). Our contributions
be summarized as follows:
We present a robust algorithm for leveraging population based training for hyperparameter search: ROMUL (RObust MUL.tistep) search, which addresses (i) and (ii). We empirically study its benefits and limitations, and show that it provides good defaults that compare favorably to existing methods.
We open-source hopt im, simple library for sequential hyperparameter search, that pro- vides multiple optimizers (including ROMUL), as well as toy benchmarks showcasing hy- perparameter optimization problems identified empirically and standard datasets.

---

Under review as a conference paper at ICLR 2021
2 HYPERPARAMETE. OPTIMIZATION WITH POPULATION-BASED TRAINING
In this article, we refer to the family of algorithms that continuously tunes hyperparameters of set of models over the course of their training as "PBT algorithms" "PBT optimizers". Hyperparameter optimization is thus a zero order optimization performed at a slower frequency than the (often first order, e.g. SGD) optimization of the model. A PBT step happens typically after a fixed number of epochs or updates of the model, often optimizing the loss from the validation set, continuing from an already produced "parent" checkpoint, and producing and evaluating a new checkpoint. At every PBT step. hyperparameters can be updated (mutated), incremented or decremented by some number (step size), or sampled.
There are multiple aspects to consider when designing a PBT algorithm. Technical constraints: how the optimization distributed, with centralized or decentralized algorithm, workers to run the trainings, how failed workers are handled. They are solved in unified manner in the experiments we performed, by the hopt im library to implement and compare multiple algorithms. It is decoupled from the scheduling of the jobs and designed to accommodate adding more workers to scale up the training, or fewer when some are killed, for example through preemption or time-out on a shared cluster. Optimization method: how the hyper-paramett instance through mutations. Selection process: which individual of the population are kept, both in term of hyper-parameters and state of the neural network (checkpoint). For those last two points, some solutions are described below.
modified throughout the training, for
2.1 CHALLENGES
In order to have a clearer understanding of our proposed methods, we show below the main concerns have observed in PBT:
Anisotropy: by definition, the optimal value of the hyperparameters considered is unknown, and oftentimes the range (or mutation scheme) provided to the algorithm is loose estimate only. As modifying two hyperparameters with the same step size can produce effects with very different magnitudes, the user required to normalize the search space. But pre-tuning the hyperparameter tuner itself can be cumbersome as dynamics evolve during training. Section[..] provides an example based on the Rosenbrock function which illustrates this issue and highlights the interest of adaptative mutations.
Checkpoint vs. hyperparameters: comparing individuals in the population is extremely hard as improvements can be due to better hyperparameters, or better checkpoints (including potentially better batches). Better performance through better checkpoints an optimization phenomenon (e.g. random restarts), that can bias the hyperparameter selection. We will detail this aspect in Section 4.2
thort-termm-long-tern discordance: we observed empirically that hyperparameters which induce better performance in the short term are not always optimal in the longer term. This is challenge that does not exist in classical static optimization, but crucial for PBT since local minima are easy to reach and pose a danger for greedy algorithms. An example of such parameter is the learning rate. Dropping the learning rate often induces a drop in the validation loss, even early in the training, and increasing it has the opposite effect, causing greedy PBT algorithms to reduce to the minimum value too early, without being able to recover. We will detail this aspect in Section
2.2 DIFTERENTIAL EVOLUTION AND ROMUL
Differential Evolution Storn & Price 1997) (DE) a standard black-box optimization method, for minimizing f R" + R. It operates population z' E Rn for all i E M),, M 2 4, and indefinitely repeats the following steps for each individual hose in the population to generate another individual called mutated vector that could replace hase if better:
1. given the best individual b hest which minimizes f in the population, as well as two randomly selected ones z" and z², compute the donor d, which will give part of its coefficients to the mutated vector. In the current-to-best/ scheme we use, these are the base coefficients plus term attracting to the best set of coefficients from the current population, and an
2

---

Under review as a conference paper at ICLR 2021
additional random variation (a standard value for F; is Fi F2 0.8):
d = x buse + Fi (xrhes - -xhwee Fs F2 x²")
2. create the new mutated vector Tbase by randomly selecting each component j E {1, .I..... nn
the base gh or the donor d through the binary crossover operator: whhse CHOICE ,d)). This non-linear operation lets the optimization leave the vector span of the population.
3. compute ((thue and replace buse by huse within the population if and only if f S farhnse)
This method is interesting as it is already based on population and adapts well to parameters with different dynamics while being simple and fully parallelizable. In particular, it does not rely on samples new parameters close to the current population,
mutation ranges step sizes Equation | and as the population individuals go through selection this sampling is refined and becomes sharper around optimal values. In practice, if parameter's bounds are too loose or wrong, DE will eventu- ally adapt after iterations of selection by removing individuals too far from the optimal value, and concentrate its computation budget on relevant values for this parameter.
In order to use it for PBT, the set of hapareparameters is converted to a vector in IR" using never- Teytaud, 2018). However, this basic version of differential grad parametrization system (Rapin & evolution (also implemented in nevergrad) is not adapted to PBT. Indeed the training function changes with the checkpoint as we are updating the parameters (not the hyperparameters) with stochastic gradient from the task loss. The trend of f is therefore typically downwards during the training. younger generations/later epochs tending to have a lower loss than their parents', biasing the hyperperameter selection process in favor of those of the children (later steps of SGD updates) instead of in favor of better hyperparameters.
We therefore propose adaptation: a population of individuals is trained, after finishing their step, individuals are compared to the rest of the population. If they have one of the n/k best loss (we use k = 2 throughout), the training continues without changing the hyperparameters, otherwise, the hyperparameters are mutated. If the hyperparameters of an individual are mutated (we
ROMUL
m times in the n/k k best individuals. The values of & and m are hyperparameters, although we did not vary them in any experiments: k 2 allows have, on average, one alternative (mutated) version to each of the ones we keep training without hyperparameter change, and m 3 proved to be robust across our experiments, to select when to discard a checkpoint. using lower m values, one should consider increasing the number of epochs per PBT step to prevent culling checkpoints too early (see Section and 4.2
throughout), its checkpoint is killed and replaced by one of
The mutation scheme is adapted to fit this use case. In Eq. 1 J. bose and best are both replaced by randomly selected set of hyperparameters xº and 2d from the best n/2 individuals ("rand-to-rand/1" scheme following (Storn & Price, 1997, Das & Suganthan| 2011) notations). Replacing huse aims at keeping the path through checkpoints unimodal, since keeping several modes with corresponding checkpoints is unnecessary. Replacing best by any other "good" (top 50%) set of hyperparame- ters aims at avoiding early convergence, which we observed as one of the main problems during trainings. This also avoids strong bias by good checkpoint (more this in 4.2. To avoid du- plication of hyperparameters, we opt for making Fi and F2 random vectors instead of using the binary crossover non-linearity. In order keep the initial scaling of DE, we chose F, [i] uniformly distributed between 0 and 2/'' (we use the common value for I from vanilla DE: F 0.8), and
F2 2f Fi], Vi. This ensures that the sum Fil1] + F2[i] 2F, Vi, as vanilla DE. With @ the elementwise multiplication, this yields d = r + F1 @ (x x²) + F2 @ (I x²))
2.3 OTHER ALGORITIIMS
In our experiments, we compare several algorithms briefly presented below, We aim to compare how effective they can be for practical use-cases of hyperparameter tuning, where the user does not want to tune the hyperparameters of its hyperparameter tuner, and desires meaningful defaults. The only input they take are the number parallel trainings, the range of hyperparameters, and a hint for initial value (e.g. the same value ( for dropout values and data augmentation magnitudes).
3

---

Under review as a conference paper at ICLR 2021
Initiator PBT: We reimplemented Initiator Based Evolution, presented in Li al. (0119). New nyper-parameters are sampled from parent stant (for instance dropoutChald pared to a randomly sampled checkpoint in the population: if the latter is better, the new checkpoint is discarded and the latter is forked with its hyperparameters this ensures that only the best per- forming models remain eventually, and allows it to run asynchronously. For each parameter, we specify range, and use (hi - lo)/30 as a mutation constant unless specified otherwise.
by adding/removing a mutation con- StropoutParente + 0.1). A newly created checkpoint is com-
Truncation Selection: N models are trained in parallel. Regularly, the M worst performing models are stopped and replaced with clones of the M bests, and hyperparameters are randomly pertur- bated. This scheme was first introduced in Daderberg et al. (2017). In our experiments, we use M N/4. For hyperparameter perturbation, we generalize the mutation scheme introduced in Ho et al. 2019): : each parameter is sampled uniformly in its range (lo, hi] with 20% probability, or incre-
mented by random. choi ce ( [-3, 3, -2, -1, 0, 0, 1, 2, 31) (hi 1o) /
and then clipped to stay within [lo, hi].
ASHA: This is not a PBT algorithm, but a strong hyperparameter search algorithm that we compare to. In the Asynchronous Successive Halving Algorithm (ASHA, Li et al. (2018)) hyperparameters are sampled uniformly like in Random Search, but models are evaluated early and stopped if not in the top 1/1) percentile. For given model, the first evaluation can happen after 1, 17. 172 steps, making this algorithm robust to hyperparameters whose optimal value does not perform well until late in the training (Section(4.1). Unlike Initiator PBT or Truncation Selection, ASHA finds constant values for hyperparameters rather than schedules. We set the reduction factor to 3.
3 EXPERIMENTS
We ran experiments on a toy optimization problem (the Rosenbrock function), CIFAR, and Penn Tree Bank, all with the same ROMUL hyperparameters to test its robustness. Each of these experi- ments train in around 100 to 300 epochs, and we used step per epoch, so that they all have similar time scales.
3.1 EXAMPLE ON TOY OPTIMIZATION PROBLEM
Current PBT mutation schemes have fixed steps and therefore do not automatically adapt to the land- scape of the optimized function. This means that they are not well-suited for anisotropic problems, which often arise in real life applications since some hyperparameters may be very important to tune finely, while other do not require the same precision. To highlight this issue, we experiment below on the Rosenbrock function: Ra, y) = (a x² + bly x²)²
We will aim at minimizing 11,100 through the surrogate R2 with à and two hyperparameters handled with PBT. We initialize both parameters at 20 and bound them by -12.12 and 212.12 (using integers would be a special case since actual a and values are integers). This experiment can be reproduced using the hopt im toolbox with the command: hop bench rosenbrock.
While standard PBT with random steps wastes mutations on à, DE is able to adapt its step-size to large steps on à until getting close, then smaller steps on to tune and i more finely. This is visible in Fig. Ta with ROMUL values of å converging quickly around 1. The mutations then become sharper, while the ones for Initiator- PBT (small steps) are still too large and oscillate around the optimal value. Arguably, the mutation step could have been even smaller, but that would have slowed down the convergence, and these steps would be painful meta- peeta-prameeeess to tune at scale. Initiator-Pa PBT with larger steps and Truncation selection are not displayed in this figure because their variations are too large.
The impact on the loss R. is then visible in Fig. 1b Initiator PBT can't decrease past 0.2 with large steps, and 0.048 with smaller steps, since it is trapped trying to optimize while DE is able to reach better values. Fig. [Icladd Idwwow the trajectory of (I. v) for Truncation selection and ROMUL, with the same number of training steps. Truncation selection is hampered by more random mutations. On the other hand, ROMUL is able to reach much lower value after exhibiting more chaotic behavior when it initially adapts to the scale of the problem. The trajectory for both versions of mnitiator-PBT can be found in Fig. of the appendix. Initiator-PBT with large steps (Fig. 2a moves
4

---

Under review as a conference paper at ICLR 2021
10
2-
10
20
(a) Rosenbrock parameter with respect the num- of steps (optimal 1). After =50 steps, ROMUL's distribution for à gets sharper around Initiator ways uses an hardooded mutation step size.
(b) Loss R1.100 with respect the number of steps (lower better, minimum value 0). ROMUL keeps adapting and decreasing while other optimizers are locked to higher levels depending on their step sizes.
200,005
200.0000
1.25
20.000
1.25
0.75
200 dio
0.50
0.50
0.25
0.50
0.00
-0.50
-0.50
Truncation Selection PBT (minimum loss=0.14)
ROMUL (minimum loss=0.011)
Figure 1: Training on the Rosenbrock benchmark. ROMUL outperforms initiator and truncation selection because can adapt its step size. Bottom plots: Trajectories of 100 PBT training steps (16 jobs per step) on the Rosenbrock function with a = 1 and 100 (minimum at the red cross (1, 1), trajectories go from blue to green)
very slowly the minimum because of big extra oscillations. With smaller steps (Fig. 2b it reaches better values through a slow and non-direct path. Tab.Bin the appendix provides quantitative results by averaging over 20 runs, including version of Initiator PBT in which updates are performed through multiplications by 0.8 or 1.2. In particular, ROMUL performs statically better than all over optimizers on this testbed (p 1.1e with a two sample Welch's t-test). In Fig. in Appendix, we also show the behavior with more variables by performing optimization an average of Rosenbrocks functions, each with independent a and b variable to estimated by PBT. Overall ROMUL performs consistently well across the board for wide range of number of variables.
3.2 APPLICATION CIFAR (IMAGE CLASSIFICATION)
In this section, we compare various algorithms for tuning hyperparameters for image classification on CIFAR (Krizhessky et al. 2009). We reproduce the population based augmentation (PBA) setup from (Ho et al. 2019) with their original implementation. Our algorithms train a Wide-ResNet-22 10 model on Reduced CIFAR- 10 (using 10%, i.e. 4000 images, of the training set for actual training, and the remainder
validation set), and optimize the same 60 hyperparameters as in Ho et al. (011): 2 magnitudes and probabilities for each of the 15 possible data augmentations. For each algorithm, take the best model in the validation set epoch 200, and use its hyperparameters schedules to train another Wide- ResNet-28- model on CIFAR-10 10 and CIFAR- -00 and finally report test accuracies at epoch 200 in Table Trainings can be reproduced with the hoptim
5

---

Under review as a conference paper at ICLR 2021
package and its benchmarking counterpart hopt im.benhhaakss in the cifar folder. ROMUL recovers most of the gains on CIFAR-10 (2.8% error vs. 2.6% for the SOTA and 3.9% for the baseline), and is bit further away CIFAR- 100 (17.1% vs. 16.7% for the SOTA and 18.8% for the baseline). PBA, which yields state-of-the-ar results on CIFAR, used Truncation selection PBT introduced in (Jaderberg et al. 2017), which implemented and compared PBA hyperparameters and observe 2.7% on CIFAR-10 10 (ROMUL: 2.8%) and 17.7% on CIFAR-100 (ROMUL: 17.1%). The differences in the job and population management in hopt im may explain the difference between our implementation and theirs, which is particularly marked on the training set reduced CIFAR-10: 12.8% for their vs. 13.9% for our implementation.
We adopted all the
Table I: Classification error (lower is better) on CIFAR-IO and CIFAR- 100 test sets for a Wide- ResNet-28- (36M params). The algorithms run with 16 workers in parallel with the same compute budget (except when stated otherwise) on reduced CIFAR-10 10. After that, the schedule found is used for training the same model from scratch CIFAR- 10 and CIFAR-100
Algorithm
Reduced CIFAR- 10 (10%)
CIFAR-10
CIFAR-
Baseline: Wide-ResNet-28-10
n/a
3.9
18.8
RandAugment (Cubuk et al. 2019b
n/a
2.7
16.7
PBA (3 epochs/step) (Ho et al. 2019)
12.8
2.6
16.7
ASHA
14.7
2.8
17.6
ASHA (running for double the time)
14.1
2.7
17.2
Truncation Selection (PBA, ours)
13.9
2.7
17.7
Initiator PBT (Li et al. (2019) ours)
14.7
2.9
17.9
ROMUL
14.0
2.8
17.1
As PBA waits for more than epoch/step to evaluate a set of hyperparameters, we compared epoch/step and 3 epochs/step (their setting), as it could help thwarting short-term/long-tems discrep- ancy effects II) and noise as explained above, but we could not identify a sufficiently generic scheme for all applications. In general, this is part of the PBT hyperparameters that are tuned in PBA, that we try to completely remove as hyperparameters (in this case it does not seem affect Truncation Selection either). For this hyperparameter, the constraint is to do PBT steps slow enough so that the number of updates sufficiently large for the model to adapt to new mutated hyperparameters, and high enough that PBT has enough steps to optimize the hyperparameters. In practice, trainings are long enough (regarding the number of SGI) updates of the model) for wide range of PBT steps frequencies work.
ROMUL, by being insensitive to
3.3 APPLICATION TO THE PENN TREEBANK DATASET (LANGUAGE MODELING)
We experiment with the TransformerXL model (Dai ct al. 2019 on the PTB dataset (Mareus) 1993). TranformerXI.Ss code is open-source and is the state-of-the the-art for tranformer models this dataset when using proper regularization, making it an interesting challenge for PBT. It comes with several dropout hyperparameters: we search for optimal values for five different dropout hyperparameters, that we describe in Table in appendix. They are all initialized to 0 with standard deviation of 0.1 for ROMUL (negative values are reflected to positive values), hence not at the baseline values.
Results are reported in Table 2 Trainings can be reproduced (up to random variance) with the hoptim package and its benchmarking counterpart The baseline TransformerXL was obtained with the author's code and is close to the one reported in the initial paper. Noticeably, ASHA and Random Search (with a uniform prior) are not able come close to the baseline, with more than 4 points difference in both validation and test perplexity (PPL). Truncation selection and Initiator PBT on the other hand are able to reach the baseline although they were not able to excel it in test perplexity. Only ROMUL is able to reach (marginally) better results than the reproduced baseline in test PPL with both 16 and 32 workers. A found dropout schedule is displayed in the appendix (Fig. and show dropouts rapidly increasing in the beginning and stabilizing to different levels. Using 16 and 32 workers provided similar results up to noise for ROMUL (in this very case, 32 workers does not actually perform better than with 16 workers).
in the ptb folder.

---

Under review as a conference paper at ICLR 2021
However, using 8 workers results in notable drop in performance for all optimizers (Test PPL ROMUL 56.39, TruncSel 57.98, Initiator 56.33).
Table 2: Perplexity (lower is better) on PTB for a Transformer-XI with 16 layers and 24M pa- rameters, best validation PPL before iteration 175 and corresponding test PPL, given the resources needed these values are not averaged, numbers excelling our training baseline are in bold.
Training
workers
Validation PPI.
Test PPL.
TransformerXI. SOTA DDa et al. (2019)
54.52
TransformerXI. SOTA (our training, their code)
59.65
55.43
ASHA
16
63.20
58.35
Truncation Selection PBT
16
60.24
57.29
Initiator PBT
16
59.42
55.80
ROMUL PBT
16
57.83
55.16
Random Search
32
63.84
60.90
ASHA
32
64.31
61.63
Truncation Selection PBT
32
58.45
55.93
Initiator PBT
32
59.36
55.73
ROMUL PBT
32
58.63
55.28
4 DISCUSSION
4.1 SHORT TERM LONG TERM DISCORDANCE
We have observed on PTB and other applications that some hyperparameters were never contributing positively the model's performance in the short term (eg: 1 step) but could become better longer term (eg: 5 steps or more). hence checkpoints need time to adapt to new parameter set (e.g.: building more redundancy). In experiment on PTB, we used one epoch per step for half the training and then modified it to 10 epochs per step. We observed that increasing one of the dropout contributed negatively for small steps (1 epoch), but positively for long steps (10th epochs). This is major roadblock for PBT-based approaches since two models with different hyperparameters can't be straightforwardly compared at every step, but only after an unknown delay. This is partially handled by being conservative models to keep: keeping the best 50% unchanged in ROMUL, or the random tournament scheme that allows bad models to continue in Initiator PBT when assigned an worse opponent. Fig. iiin appendix shows such example in another domain: large dropout seems very detrimental early on, but very beneficial in the longer term. While this is expected for regularizations, we observe that straightforward schedule increasing the dropout in steps (in red) is not able to compensate this we observed the same effect with continuous schedule. This behavior adds complexity the task PBT algorithms, because bad early choices Arguably, it can be due to interactions with the learning rate scheduler used and more appropriate schedule could help solve this issue (although it is not clear what such schedule should be).
compensated later.
4.2 CHECKPOINTS VS HYPERPARAMETERS SELECTION BIASES
For PBT optimizers, one critical question is how much the loss difference between two indi- viduals is caused by different hyperparameters, and how much about different checkpoints. Both contributions are tightly entwined making it harder to identify which hyperparameters are best.
A naive initial option counter this is always start PBT step from the best checkpoint in the population. our experiments this performed worse, at least because the noise in the evaluation metric, but also because of the short-term long-term discrepancy detailed above (Section 4.1 We also expect that doing so could make optimizers less robust by getting trapped in local minima too easily, aggressively discarding more promising models in the longer term.
On the opposite side the spectrum, we experimented with never culling checkpoints, effectively performing full trainings in parallel. Conceptually, keeping checkpoints is attractive since it should

---

Under review as a conference paper at ICLR 2021
add robustness the optimizer: selected hyperparameters have to work well for more than one par- ticular checkpoint. Indeed, this way the performance can be attributed to actual parameter schedules fitting different trainings, instead of being biased by checkpoints culling/random restarts. It also adds more variability which could be beneficial especially with respect to the short-term long-term discrepancy. That being said, we have neither observed significant improvement nor deterioration when keeping checkpoints, as long as the mutation schemes were not biased towards the best set of hyperparameters (e.g.: removing the best term in Eq. I. because doing so can make all hyperpa- rameters converge towards the best checkpoint, making the optimization process early converge to values which are not necessarily adapted to other checkpoints.
Still, even with ROMUL's loss-agnostic mutation scheme, some checkpoints were observed to fall behind and waste resources if not culled, so we expect that trade-off like the one we implemented (killing checkpoints after 3 failed mutations in row) is necessary.
Another source of selection bias noise. While the trainings are well behaved in PTB because the shuffling of the training set is synchronized by epoch, the trainings in CIFAR are much noisier because of the randomness introduced by data augmentation and the very small training set size. PBT optimizers based on more noise-robust blackbox optimization methods could be beneficial, but it is not clear how to adapt them.
5 RELATED WORK
Several families of methods exist for tuning hyperparameters of neural networks. Methods closest to grid search like random search (Bergstra Bengio 2012) and ASHA (Li et al.) 2018) are based on minimal constraints and can be parallelized extensively, Methods striving for more data-efficient search (Bergstra et al., 2011 2013 Feurer & Hutter 2019) are more sequential in nature, requiring convergence of some trainings before launching
ones. Population-base training approaches (Jaderberg et al., 2017 Ho et al. 2019: Li et al. 2019) loosen the requirements of training different models, as hyperparameters are changed on-the-fi fly during training, which also makes the search for schedules easier and less structured, i.e. not based on a predefined function.
Recent advances in automatic discovery of data augmentation policies include Population Based Augmentation (Ho et al.) 2019) which we compared to this paper (denoted Truncation Selection). Another line of work on structuring the hyperparameter space for data augmentation policy search is AutoAugment (Cubuk et al. (2119)), FastAuto [im et al. (2019) and RandAugment Cubuk ct al. 2019b), the later being faster and reaching top performance on CIFAR.
PBT is used successfully in reinforcement learning (Jaderberg et al. 2017), providing diversity in self-play and progressive difficulty, so other experimental comparisons that we did include Initiator PBT from (Li et al.] (209)). which presented generic PBT setup that inspired hopt i.. For non- PBT baselines we used random search (Bergstra & Bengio [012]. and ASHA (Li et al. 2018).
which is an update on HyperBand (Li et 2017).
6 CONCLUSION
We introduced ROMUL, robust PBT algorithm that we benchmarked on standard datasets with multiple regularization and data augmentation hyperparameters. Its main strength comes from its robustness to hyperparameters definitions by automatically adapting to the scale of each parame- ter. Although it did not show better performance on CIFAR than PBA that was tuned for this benchmark we demonstrated that it is more robust to domain changes. More importantly for the practical use-cases, it constitutes a good default that does not require extensive tuning to work well. We open-sourced its implementation as well as a simple and broadly compatible PBT library.
The main difficulties we observed for PBT-based optimizers came from short-term vs. long-term effects: parameters can have a positive impact in the short term but a negative one in the longer term which may not be rectifiable. Learning rate falls in this category, since decreasing it often provides quick gains at the risk of being trapped in a local minimum. Studying how to deal with such behaviors in our opinion the main challenge of future work.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
James Bergstra and Yoshua Bengio. Random search for hyper-parametie optimization. The Journal of Machine Learning Research, 13(1):281-305, 2012.
James Bergstra, Dan Yamins, and David D Cox. Hyperopt: A python library for optimizing the hyperparameters machine learning algorithms. In Proceedings of the 12th Python in science conference, volume 13, pp. 20. Citeseer, 2013.
James S Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper- parametr optimization. In Advances in neural information processing systems, pp. 2546-2554, 2011.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 123, 2019a.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with reduced search space. arXiv preprint 190.vv//99917///// 2019b.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoe V. Le, and Ruslan Salakhutdinov. Transformer-XL. XX:: Attentive Language Models Beyond Fixed-Length Context. arXiv:1901.00800 [cs, stat], June 2019.
Swagatam Das and Ponnuthurai Nagaratnam Suganthan. Differential Evolution: A Survey of the Srate-of-the-Art IEEE Transactions on Evolutionary Computation, 15(1):4-31, February 2011. ISSN 1089-778X, 1941-0026. doi: 1011100TEVC222000000033.
Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. arXiv preprint arXiv: /909. 1556, 2019.
Matthias Feurer and Frank Hutter. Hyperparameter optimization. In Automated Machine Learning, pp. 3-33. Springer, Cham, 2019.
Daniel Ho. Eric Liang, Ion Stoica, Pieter Abbeel, and Xi Chen. Population based augmentation: Efficient learning of augmentation policy schedules. arXiv preprint (00i:::90000539333 2019.
Max Jaderberg, Valentin Dalibard, Simon Osindero. Wojeiech M. Czamecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population Based Training of Neural Networks. 1:Xiv:/711.09800 [es], November 2017. arXiv: 1711.09846.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ang Li, Ola Spyra, Sagi Perel, Valentin Dalibard. Max Jaderberg, Chenjie Gu, David Budden, Tim Harley, and Pramod Gupta. A generalized framework for population based training, 2019.
Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Moritz Hardr, Benjamin Recht, and Ameet Talwalkar. Massively parallel hyperparameter tuning. arXiv preprint 11X1v:1810.05944, 2018.
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar, Hyperband: A novel bandit- based approach to hyperparameter optimization. The Journal of Machine Learning Research, 18(1):6765-688 1616, 2017.
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. In Advances in Neural Information Processing Systems, pp. 6662-6672, 2019.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.090 09055, 2018.
Mitch Marcus. Building a Large Annotated Corpus English: The Penn Treebank: Technical report, Defense Technical Information Center, Fort Belvoir, VA, April 1993.

---

Under review as a conference paper at ICLR 2021
J. Rapin and O. Teytaud. Nevergrad- A gradient-free optimization platform. htttps://GitHbb
htw.aa/bbooeeeerrrreeeeeegaaaaaa 2018.
Rainer Storn and Kenneth Price. Differential Evolution A Simple and Efficient Heuristic for global Optimization over Continuous Spaces. Journal Global Optimization, 11(4):311339,
December 1997. ISSN 1573-2916.
doi: 10. 00:0023/A1100522222133388.
10

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1
ROSENBROCK EXPERIMENTS
250,000
200.000
27.0000
20.000
1.00
1.22
000
000000
00.000
200000
0.30
0.50
0.00
0.00
-0225
-0.2
-0.50
-0.50
-0.25
0.000
0.25
0.50
0.75
1.00
1.75
-0.50
-0.2
0.00
0.25
0.50
0.75
1.00
1.25
(a) Initiator PBT (minimum loss=0.20)
(b) Initiator PBT, small steps (minimum loss=0.048)
Figure 2: Trajectories of 100 Initiator PBT training steps (16 jobs per step) on the Rosenbrock
function with a 1 and b 100 (minimum at the red cross (1, 1), trajectories go from blue to green)
Training
mean (log10) 10)
std (log 10)
Initiator (0.8/1.2 1.2 mult. steps)
-1.18
0,045
Initiator (big steps)
-0.707
0.069
Initiator (small steps)
-0.992
0.143
ROMUL Truncation Selection
-2.101
0.678
-0.834
0.327
Table 3: Final loss (in log10) mean and standard deviation for independent runs on the Rosenbrock testhed, computed over 20 runs (two-sample Welsh's 'ss test provides p < 1.1e 5 when comparing each algorithm with ROMUL).
Score on Rosenbrock(n [1
-0.5
-1.5
(big steps)
-2.0
Initiator (smail steps)
2.5
Truncation Selection
Initiator (0.8/1. steps)
10
15
25
30
Number variables
Figure 3: Score on the multi-variate Rosenbrock benchmark (explained in 3.1 over 20 experiments for each point. Lower is better, standard deviations indicated, ROMUL performs well across the board for a wide range of number of variables, being surpassed only by Truncated selection in some regime (n E [8... 14]].
11

---

Under review as a conference paper at ICLR 2021
Table 4: The dropouts from Transformer-Xe XL that we tune through PBT.
dropouta dropoute dropoutf dropouti dropouto
applied to multi-head attention layers
to remove words from embedding layer applied to positionwise ff layers for input embedding vectors applied to the output (before the logit)
Drepouts
dropouta dropoute
2.0
drapoutf
dropouti
dropouto
1.5
Sumpersppuu)
1.0
0.5
0.0
0
20
40
80
80
100
120
140
160
Epochs
Figure 4: Dropout schedule of the best run of ROMUL 32 workers PTB
A.2 LANGUAGE MODELING ON PENN TREE BANK
A.3
REGULARIZATION SCHEDULES ON WIKITEXT-103 103
21.5
dropout 0.1
dropout 0.2
dropout 0.3 (baseline)
validation set perplexity
dropout schedule 0.1, 0.2, 0.3
20.5
dropout 0.3 ~> 0.0
20.0
19.5
19.0
20
80
100
120
140
160
180
200
epoch
Figure 5: Lower dropout values are better early, but are outperformed by more strongly regularized models later (red, orange and blue lines) here on wikitext103 03 with 247M parameters language model from Fan et al. (2019) (Adaptive Inputs + LayerDrop). PBT algorithms would tend to reduce dropout aggressively early on: after that, even if the dropout is increased later, the performance remains worse than training with a high dropout from the beginning (red line). Perhaps counterin- tuitively, this hints against increasing regularization over the course of the training in the opposite, we observe that fine-tuning the model without dropout significatively improves test performance (purple line reaches 17.98 test perplexity) compared to the baseline (green: 18.42 test perplexity)
12

---

Under review as a conference paper at ICLR 2021
A.4
SLIDES FOR INTERNAL PRESENTATION
Training
Parallelism
Validation PPL.
Test PPL.
XransformerXI
59.65
55.43
ASHA
16
63.20
58.35
Truncation Selection PBT
16
60.24
57.29
Initiator PBT
16
59.42
55.80
ROMUL PBT
16
57.83
55.16
Table 5: Perplexity (lower is better) PTB for Transformer-XI. with 16 layers and 24M parame- ters
Algorithm
CIFAR-10
CIFAR-100
Baseline: Wide-ResNet-28-11 10
3.9
18.8
RandAugment (Cubuk et al. 2019b)
2.7
16.7
PBA (3 epochs/step) (Ho et 2019
2.6
16.7
ASHA
2.8
17.6
Truncation Selection (PBA, ours)
2.7
17.7
Initiator PBT (Li et al. (0119) ours)
2.9
17.9
ROMUL
2.8
17.1
Table 6: Classification error (lower is better) on CIFAR-IO and CIFAR-100 test sets for Wide- ResNet-28-10 10 (36M params)
13


Reviews:
[{"review_id": "ms9ZpSTvUz", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary\nThe paper provides a new variant of PBT which utilizes ideas from differential evolution and cross-over. The original PBT and even initiator PBT do not perform crossover on the hyper-parameters, and insufficient cross-over may cause PBT to perform greedy in the initial phases which ends up with a suboptimal convergence. The investigation of better cross-over in PBT is itself an interesting research direction and the authors demonstrated its effectiveness in standard benchmarks and data augmentation tasks. The improvements of ROMUL-PBT are also helpful to the community since PBT has been applied in a variety of real world applications.\n\n#### Pros\n1. Quality: The paper quality is in general good. The experiments are well designed and the results are good. So the experiments clearly supports the argument that differential evolution helps PBT.\n2. Clarity: The paper is well written and easy to follow. The organization is also clear.\n3. Originality: I think that adapting ideas from differential evolution to PBT is new, even though differential evolution itself is not something new.\n4. The paper provides some benchmarking of PBT related algorithms in image classification, language modeling and data augmentation which is good for the community to understand these approaches.\n\n\n#### Cons\n1. Significance: the improvements over existing methods seem slight. The experiments do not provide sensitivity analysis so it is a bit hard to conclude whether the results are statistically significant. But at the same time, the proposed method does show promise.\n2. As a thorough evaluation purpose, it would be interesting to see how the proposed methods work in large set of hyperparameters (magnitude of 10-100). \n\n#### Questions\n1. PBT needs to use validation loss to obtain fitness. Is your result evaluated on the validation data or the test data? If only evaluating on the validation data, the result may not reveal potential overfitting to the validation set. So it would be nice to have results on a held-out test set.", "extra_scores": {}, "extra_text": {}}, {"review_id": "dcE-J8OQYiq", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission studies Population Based Training (PBT) methods for tuning and adapting hyperparameters over the course of training. It makes two contributions: (1) a novel PBT algorithm, ROMUL, and (2) a library for PBT-based training, hoptim.\n\nAfter reading the paper and the supplementary material, I can only comment on the first contribution. While the authors highlight the hoptim library as one of the main contributions of the paper, they do not describe what are their advantages over existing hyperparameter optimization frameworks or existing PBT implementations (e.g. the one in Ray). As far as I can tell, there is no source code or link to an anonymous repository so that we can evaluate this contribution. Commands for replicating experiments using hoptim are scattered throughout the manuscript, but please note that this is not enough to evaluate the quality or impact of the software. This is an important issue because the paper justifies weaker empirical results based on implementation differences that are never discussed (e.g. “The differences in the job and population management in hoptim may explain the difference between our implementation and theirs, which is particularly marked on the training set reduced CIFAR-10: 12.8% for their vs. 13.9% for our implementation.”, or how the number of workers has a strong impact in the final result for PTB experiments.).\n\nBy leveraging ideas from Differential Evolution, ROMUL eases the task of defining the search space when considering hyperparameters with different magnitudes. In other words, this simplifies the task of “tuning the hyperparameter tuner”. The benefits of the proposed strategy are showcased by optimizing a 2D Rosenbrock function where the optimal values for the two parameters differ in magnitude (a=1, b=100). ROMUL is then applied to optimize Population Based Augmentation (PBA) on a reduced training set for CIFAR-10 and to tune the dropout rates in TransformerXL for Penn Treebank (PTB). Results for baseline methods are slightly worse than those in the literature even when using the code provided by the authors, and authors are encouraged to explain the reason for this. ROMUL seems to outperform other PBT methods when tuning TransformerXL, but the benefits on PBA when applied to CIFAR-10 are not so clear. It is difficult to evaluate the significance of these figures, as no standard deviation across seeds is reported. \n\nMy main concern regarding the experimental setup has to do with the mutation constant used for Initiator PBT and Truncation PBT. The works by Li et al. (2019) and Jaderberg et al. (2017) perturb hyperparameters by a multiplicative factor of 1.2 or 0.8 instead. This enables a much finer-grained search space than the one implemented in this submission, where the additive mutation constant might be too large for some parameters given the range in which these parameters are defined. For instance, the optimal value of $a$ in the Rosenbrock function is 1 but the step size for Initiator PBT is $(hi-lo)/30=224.24/30=7.47$. Since $\\hat{a}$ is initialized to 20, it is impossible for this method to even get close to the optimal value.\n\nThe discussion section provides some interesting experiments showcasing the effect of some design choices on PBT methods. It discusses the importance of the patience of the algorithm in order to account for the long-term impact of some hyperparameters (e.g. learning rate, dropout rate) as well as the impact of reusing existing checkpoints after mutation.\n\nWhile this submission discusses important research topics, I do not believe it is ready for publication yet. The authors highlighted two main contributions, but I believe there are three potential ones: (1) a PBT method that does not need extensive tuning, (2) a software library for PBT training, and (3) an empirical evaluation of different design choices for PBT methods. However, these need to be developed further (and potentially in separate papers) before they can be published at ICLR. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "K4g1caSfj_r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this submission, the authors propose a modification to the PBT (population-based training) method for HPO. It is interesting, however, there are several important issues to consider:\n\n1) The major part of the proposed method ROMUL is to replace some update rules based on Differential Evolution, which is a well-studied method in Evolutionary Algorithms. The novelty of the proposed method ROMUL is not high. But more important, it is unclear why such modifications are necessary. In other words, what new challenges in HPO can be addressed by conducting these modifications to PBT. Without clear and strong reasons to motivate these modifications, it is hard to evaluate the proposed method.\n\n2) The writing of this submission can be further improved. Many paragraphs and sentences are not logically organized, and it is difficult to understand the main points of the submission. For example, based on the Introduction section, it seems that the main part of this submission is to \"empirically study the different training dynamics of ...\" (second paragraph). And in introduction, the authors didn't well motivate the proposal of their method. Although several challenges are mentioned in the first paragraph, it is not clear which ones are solved by the proposed method, and how they are tackled.\n\nPersonally I feel that \"fixed step\" issue in PBT is important, which should be mentioned early.\n\nSeveral interesting findings are provided in the experiments. The authors can make them clear and highlight them by improving the writing of current version.", "extra_scores": {}, "extra_text": {}}, {"review_id": "2QSZC-kXVHt", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "** Summary **\n\nThis paper focuses on issues in the popular PBT algorithm for hyperparameter optimization. It investigates the 1) step size (which is typically a constant multiplier) 2) the variance induced by better weights and 3) the greediness of the algorithm, which they refer to as short-term vs. long term effects. These issues are well motivated, and it is intuitive that they are flaws in the original algorithm. The proposed approach is to use Differential Evolution which the authors claim makes the hyperparameter selection more robust. The paper also introduces a new library for online hyperparameter tuning.\n\n** Primary Reason for Score **\n\nThe strengths of this work are that it identifies and discusses some interesting issues with PBT, a commonly used algorithm. However, as someone who frequently uses variants of the PBT algorithm, the evidence provided in this work is not sufficient for me to adopt their recommendations. The method is based on heuristics and the experiments are unfortunately not rigorous: the gains are small and it is a single seed. To increase my score, I would need to see more robust results that make these heuristics convincing, for example multiple seeds with clear outperformance (ideally statistically significant). It would also be important to see ablation studies for the newly introduced parameters (e.g. m). In addition, some demonstration of the phenomena described having an influence on the performance would be helpful.\n\n** Strengths **\n\n1) The issues the paper addresses are well motivated, and well described. \n2) The topic of the paper (PBT) is one that I think has not been sufficiently addressed by the community. In particular, the present PBT algorithm is commonly used but none of the improvements since 2017 have been widely adopted. It seems like a fruitful direction for research.\n3) I appreciate the discussion of the results, which do not claim SoTA but instead go into detail on possible drivers of performance. \n\n** Weaknesses **\n\n1) The main contribution of the work is not convincing. It simply replaces one heuristic for another. While the results show an improvement, it is not clear. \n2) Experiments are only run a single time, and this is surely a noisy process. Given that, the gains vs. PBT seem small. It is entirely possible that this small gain is reversed in a second run. If the TransformerXL is too expensive, then a smaller experiment which can be repeated multiple times would be a stronger piece of evidence for the method’s efficacy. \n3) The authors claim to reduce the meta-parameters, yet introduce new parameters (F_1, F_2 and m). Also how was the size of the PBT step chosen? For the transformer experiment it goes from 1 epoch -> 10 epochs. Some ablation studies for these parameters would be needed for a reader to fully understand how to use this method on a new task. \n4) The library is presented as a second major contribution, but it is not clear why the reader would choose to use it over existing libraries such as ray tune, which are popular and widely used. There is no comparison or discussion here, other than just saying that the new library is better. I also couldn’t find the library anywhere, the supplementary material is just a two page pdf, and there is no anonymized link. Please correct me if I missed this. \n\n** Minor issues **\n\ni) The ICLR 2020 template was used (rather than 2021). \n\nii) Bottom of page 7, “raw” -> “row”", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission proposes a variant of population based training (PBT) for hyperparameter selection/evolution, aimed at addressing drawbacks of existing variants (e.g. the coupling of the choice of checkpoint with the choice of hyperparameters). Reviewers generally agreed that the paper is interesting and covers an important topic, and the evaluation does show improvements over existing PBT variants. On the other hand they also raised a few important issues:

1. The `hoptim` library is claimed as a primary contribution of the work, but it is not clear from the manuscript what benefits this library offers over existing software. When claiming a library as a main contribution, it is helpful to provide a more thorough description of the software and its benefits, and/or ideally a link (anonymized for review) to the software. The authors did respond by providing a brief description of the benefits of the library, mitigating this issue somewhat. However it's still difficult to discern how/whether to weigh the open source library as a main contribution of the paper.

2. The evaluation is not very convincing: the differences are small and error margins are not provided for the neural network-based experiments, meaning that any differences could be due to noise. The authors fairly point out that it is difficult to perform multiple runs of these experiments as the resource requirements are large, and they have done 20 runs of the Rosenbrock experiment with smaller compute requirements. But the reviewers were not convinced that the Rosenbrock experiment reflects the method's application to neural network hyperparameter selection; the problems are too different. The submission would be significantly stronger if it included results over multiple runs of an "intermediate" sized experiment on a problem involving a neural network demonstrating that ROMUL outperforms competing approaches by a statistically significant margin.

3. The proposed approach is ultimately heuristic. This is not necessarily a problem if there are strong empirical results demonstrating the efficacy of the proposed heuristic, but in this case the empirical results didn't convince (see point 2).

Given these concerns raised by reviewers, the submission is not quite ready for ICLR. I hope the authors will consider resubmitting the paper after improving it based on the reviewers' feedback.
Decision: Reject