TaskSet: A Dataset of Optimization Tasks
Authors: Luke Metz
Niru Maheswaranathan
Ruoxi Sun
C. Daniel Freeman
Ben Poole
Jascha Sohl-Dickstein
Venue:  (2021)
Keywords: optimizers
meta-learning

Abstract: We present TaskSet, a dataset of tasks for use in training and evaluating optimizers. TaskSet is unique in its size and diversity, containing over a thousand tasks ranging from image classification with fully connected or convolutional neural networks, to variational autoencoders, to non-volume preserving flows on a variety of datasets. As an example application of such a dataset we explore meta-learning an ordered list of hyperparameters to try sequentially. By learning this hyperparameter list from data generated using TaskSet we achieve large speedups in sample efficiency over random search. Next we use the diversity of the TaskSet and our method for learning hyperparameter lists to empirically explore the generalization of these lists to new optimization tasks in a variety of settings including ImageNet classification with Resnet50 and LM1B language modeling with transformers. As part of this work we have opensourced code for all tasks, as well as ~29 million training curves for these problems and the corresponding hyperparameters.

Under review as a conference paper at ICLR 2021
TASKSET: A DATASET OF OPTIMIZATION TASKS
Anonymous authors Paper under double-blind review
ABSTRACT
We present TaskSet, a dataset tasks for use training and evaluating optimizers. TaskSet is unique in its size and diversity, containing over thousand tasks ranging from image classification with fully connected convolutional neural networks, variational autoencoders, to non-volume preserving flows on variety of datasets. As an example application such a dataset we explore meta-learning an ordered list of hyperparameters try sequentially. By learning this hyperparameter list from data generated using TaskSet we achieve large speedups in sample efficiency over random search. Next we use the diversity of the TaskSet and our method for learning hyperparameter lists to empirically explore the generalization of these lists new optimization tasks in variety of settings including ImageNet classification with Resnet50 and LMIB language modeling with transformers. As part of this work we have opensourced code for all tasks, as well as 29 million training curves for these problems and the corresponding hyperparameters
INTRODUCTION
As machine learning moves to new domains, collecting diverse, rich, and popliccation-rlevaan datasets critical for its continued success. Historically, research on learning optimization algorithms have only leveraged single tasks (Andrychowicz et al. 2016;) Metz al.) (019a), or parametric synthetic tasks /Wichrowska et al. 2017), due to the difficulty of obtaining large sets of tasks.
1.1 TASKSET: A SET OF TASKS
We present a set of tasks significantly larger than any optimizer dataset previously studied. We aim to better enable standardized research on optimizers, be that analysis of existing optimizers, or development of new learned learning algorithms. We call this suite of tasks TaskSet.
Much in the same way that learned features in computer vision outpaced hand designed features (Krizhevsky et al. 2012 LeCun et al. 2015). we believe that data driven approaches to discover opti- mization algorithms will replace their hand designed counterparts resulting increased performance and usability. To this end, standardizing large suite of optimization tasks is important first step towards more rigorous leamned optimizer research.
In this setting, a single "example" is an entire training procedure for a task defined by data, loss function, and architecture. Thus, TaskSet consists of over thousand optimization tasks, largely focused on deep learning (neural networks). They include image classification using fully connected and convolutional models, generative models with variational autoencoders (Kingma & Welling (2013) or flows (Dinh et al.) 2016; Papamakarios et al. 2017), natural language processing tasks including both language modeling and classification, as well as synthetic tasks such quadratics, and optimization test functions. The problems themselves are diverse in size, spanning 7 orders of magnitude in parameter count, but remain reasonably fast to compute as almost all tasks can be trained 10k iterations on CPU in under hour. To demonstrate the breadth of this dataset show an embedding of all the tasks in Appendix A.1 in Figure ST
redacted url

---

Under review as a conference paper at ICLR 2021
1.2 AMORTIZING HYPERPARAMETER SEARCH
Machine learning methods are growing ever more complex, and their computational demands are increasing at a frightening pace (Amodei & Hermandez 2018). Unfortunately, most modern machine learning models also require extensive hyperparameter tuning. Often, hyperparameter search many times more costly than the final algorithm, which ultimately has large economic and environmental costs (ttubelll et al., (2119).
The most common approach to hyperparameter tuning involves some form of quasi-random search over pre-specified grid of hyperparameterss. Building on past work (Wistuba et al. 2015b Pfisterer ct al. 2018), and serving as typical example problem illustrative of the sort of research enabled by TaskSet, we explore a hyperparameter search strategy consisting hyperparameters try. The idea is that the first few elements in this list will cover most of the variation in good hyperparameters found in typical machine learning workloads.
simple ordered list of
We choose the elements in this list by leveraging the diversity of tasks in TaskSet, by meta-learning a hyperparameter list that performs the best on the set of tasks in TaskSet. We then test this list of hyperparameters new, larger machine learning tasks.
Although learning the list of hyperparameters is costly (in total we train ~29 million models consisting of over 4,000 distinct hyper parameter configurations), our final published list is now available as good starting guess for new tasks.
Furthermore, we believe the raw training curves generated by this search will be useful for future hy hyprraarreeter analysis and meta-learning research, and we release as part of this work. We additionally release code in Tensorflow (Abadi al. (0166), Jax (Bradbury et al. 2018), and Py- Torch (Paszke et al.) (219)) for reference optimizer which uses our learned hyperparameter list, and can be easily applied to any model.
2 TASKSET: A SET OF TASKS
How should one choose what problems to include in set of optimization tasks? In our case, strive to include optimization tasks that have been influential in deep learning research over the last several decades, and will representative of many common machine learning problems. Designing this dataset requires striking balance between including realistic large-scale workloads and ensuring that tasks are fast to train so that using it for meta-learning is tractable. We construct our dataset largely out of neural network based tasks. Our chosen tasks have between ten thousand and one million parameters (much smaller than the billions commonly used today), as a result most problems can train in under an hour cloud CPU with 5 cores. We additionally focus on increased "task diversity" by including many different kinds of training algorithms, architectures, and datasets - inspired by past work in reinforcement learning which has demonstrated large numbers of problems and increased diversity around some domain of interest is useful for both training and generalization Heess et al. (2017): Tobin et al. (2017); Cobbe et al. (0188) 8): OpenAl et al. (201)). Again though, balance must be struck, as in the limit of too much diversity no learning can occur due to the no free lunch theorem (Wolpert& & Macready 1997). Our dataset, TaskSet, is made up of 1 1162 tasks in total. We define task the combination of loss function, dataset, and initialization.
Specifically we define task as a set of 4 functions:
Initialization: ( + parameter initial values
Data generator: data split (e.g. train / valid / test) - batch of data
Forward pass: (batch of data, params) -> loss
Gradient function: (input data, params) - gradients Jpuranns
dloss
A task has tunable hyperparamcters and, coupled with an optimizer, provides all the necessary
information to train using first order optimization. This makes experimentation easier, as each task
definition specifies hyperparameters such as batch size (Shallue et al. 2018 McCandlish al. 2018)
initialization (Schoenholz al. 2016 Yang & Schoenholz 2017
Xiao al. 2018 Li & Nguyen
2

---

Under review as a conference paper at ICLR 2021
parcant of optmmers
number tooke
120
100
0.8
80
58
60
0.1
40
44
0.01
62-
10 500
5mhn
logT0 paramoter count
tndk
training time
Figure 1: (a) A histogram of parameter counts for each problems in the task suite. Our task suite spans more than 7 orders of magnitude model size. (b) Percentage of optimizers (y-axis) capable of reaching given Ioss value (color) for tasks (X-axis). We find there exists around 100 "easy" tasks on which more than half of the optimizers perform well, and large number of "difficult" tasks for which almost no optimizers perform well. (c) histogram of training times. Almost tasks can be trained in under hour.
2019
Pretorius et al. 2018 Hayou et al. 2018 Karakida et al. 2018 Blumenfeld al. 2019 Hayou
et al. 2019) that no longer need to be tuned.
We augment a set of "fixed" tasks which have been designed by hand, with "sampled" tasks that are randomly generated task instances.
2.1 SAMPLED FAMILIES OF TASKS
Sampled tasks are created by sampling neural network architectures (c.g., MLPs, convnets), activation functions, datasets (e.g., images, text, quadratic functions, and synthetic tasks), and other properties. organize these sampled tasks into similar families of tasks. See Appendix H for complete description of these sampled tasks. Broadly, these separated into tasks sampling image models (mlp, mip_ae (Hinton & Salakhutdinov (0066), mlp_vae (Kingma & Welling, 2013), conv_pooling, conv_fc, mvp (Dinh et al. 2016), maf (Papamakarios et al. 2017)), tasks sampling language models (char_rmi rnn laer_nnnnannaan lan_languageee_e sunguage_model (Graves) 2013). word_rna run language _model, rnn text classification), quadratics (quadratic) and other synthetic tasks (losg tasks (Wichrowska al. (2017)). Defining sampling distribution that generates tasks that always valid, and that run within time constraint, is difficult. Instead, we define broad distribution and make use of rejection sampling to remove tasks that are either too slow or that we are unable to optimize at all. By starting with a distribution that is too broad, and pruning
hope achieve better coverage of tasks.
2.2 HAND DESIGNED TASKS
In addition to the sampled tasks, we also include 107 hand designed tasks. These consist more common tasks that both improve the coverage beyond the sampled tasks, and provide for better interpretability through a closer match to existing tasks in the literature. These tasks span image classification, text classification, language modeling, and generative modeling, as well as some synthetic tasks such associative retrieval (Ba et al. 2016). We leave the description of each of these tasks to Appendix H.3
2.3 AGGREGATE STATISTICS OF TASKSET
In Figure we show histograms of compute times for all problems and find almost all problems train under an hour (see Appendix C for per task family histograms). In Figure I we plot histogram of the number of parameters per tasks. Finally, in Figure Dp we show distribution of task difficulty by plotting the fraction of optimizer configurations that achieve certain loss value. We find that for some tasks as many as 50% of optimizers perform well while for others < 1% achieve loss close to the smallest observed loss. For qualitative visualization of TaskSet, see Appendix A
AMORTIZED HYPERPARAMETER SEARCH
As simple demonstration of using TaskSet meta-learning research, we consider learning hyper- parameter lists. This idea of learning lists of hyper parameters has been explored in (Wistuba et al..
3

---

Under review as a conference paper at ICLR 2021
2015b;
Pfisterer al. 2018). We define optimizer as the pairing of optimization algorithm and all its corresponding hyperparameters (c.g, learning rate). While sometimes practitioners use single optimizer e.g. Adam (Kingma Ba 2014) with default hyperparameters most practitioners will often multiple optimizers and validation set to select the best performer.
3.1
OPTIMIZER FAMILIES
We define different parameterizations of hand designed optimizers as an optimizer family. The optimizer families consider consist of:
Adam/p: One hyperparameter, the fixed learning rate
Adam4p: Four Adam hyperparameters, a, 91, 32, and c
Adam6p: Adam4p hyperparameters, and two additional hyperparameters controlling linear and exponential learning rate decays
Adam8p: The hyperparameters in Adam6p plus two additional hyperparameters for é1 and 82 regularization terms
NAdamW: 10 hyperparameter search space based on NAdam Dozal 2016) with cosine learning rate decay, and weight decay.
For the full update equations see Appendix D. for Adam and D.2 for Nadam We chose Adam based on its use in existing work, and NAdam based on performance shown in (Choi et al. 2019).
3.2 LEARNED HYPERPARAMETTR LISTS
Traditionally researchers tune hyperparameters performance gains; it comes at the cost of immense compute, and researchers are almost never able to expend enough compute to saturate model performance (Shallue et al. 2018). As an alternative to per-problem tuning, we proposes instead tuning the search strategy itself on dataset of tasks and transferring the knowledge gained to new tasks of interest. This idea is already implicitly done by humans - e.g. we don't start a hyperparameter search with a learning rate 10 we use values that the community has found useful.
per model basis. While this often results in
This dataset-based tuning has number of desirable properties. First, the resulting search strategies are much more efficient, resulting in large speedups in sample efficiency on unseen tasks over a random search baseline. Second, we are less restricted by the number of optimizer parameters we search by needing define reasonable search spaces. For example, if there redundant regions of search space, our learned optimizer will be less likely to sample them repeatedly, unlike random search. If there is a region of hyperparameter space that performs poorly on all problems, the learned search strategy will avoid it.
In this work we parameterize the learned search strategy an ordered list of optimizers to try (i.e. list of hyperparameter configurations). Given fixed number of task evaluations we would like to achieve the best possible performance on all tasks the training set of tasks. For length le list of optimizers define our loss
Σ
min [(1, (r) HEL..
where 8, are the optimizer hyperparameters for element the list, and f an appropriately normalized loss computed after training task
We seek to find an optimal list of optimizers as (similar to Wistuba et al. 2015b));
0
arg min J(B (....)).
O1.....k
This is meant serve as an example task, illustrative of the sort of research enabled by TaskSet. More advanced hyperparameter search strategies would doubt yield even more performant results.
4

---

Under review as a conference paper at ICLR 2021
3.3 SCORING AN OPTIMIZER BY AVERAGING OVER TASKS
To score task, we initialize the parameters of the task and run 10,000 iterations of an optimizer. We monitor loss on each data split (train, validation, test) every 200 steps using an average over 50 mini-batches per evaluation. For all data presented in this paper we also compute averages over random task parameter initializations.
A side effect of the diverse task dataset is that losses span multiple orders of magnitude, making direct aggregation of performance problematic. To remedy this we normalize the loss values for all tasks linearly between 0 and where 1 is validation loss initialization and zero is the lowest validation loss achieved by any tested optimizer. Loss values greater than the loss at initialization are clipped to 1. To collapse an entire normalized training curve into a scalar cost, we compute the mean normalized loss over the 10,000 iterations. We find empirically that this choice is similar to taking the minimum (Appendix B.5). We leave exploring alternative methods such performance profiles (Dolan & Moré. 2002) and Nash averaging (Balduzzi et al. 20018) for future work.
3.4 GREEDY LEARNING FROM RANDOM SEARCH
Optimizing Eq. is combinatorially expensive. To tractably solve this optimization problem, introduce two approximations (Wistuba et al., 20155). First, we shift the unconstrained search over the full space of optimizers to search over finite set of optimizers, O. This finite set can be computed ahead of time and decouples the expensive procedure of training each task with an optimizer from training the learned search space. Separating data and training in this way has been done for both hyperparameter search (Eggensperger et al. 2015), and neural architecture search (Klein Hutter 2019; Ying al. 2019). In total we trained 1,000 optimizer configurations for each of Adamlp. Adam4p, Adamóp, Adam8p, and NAdamW on all 1,162 tasks with 5 random seeds per pair. Second, we use a greedy heuristic to approximate the combinatorial search over sets of k optimizers. For a single optimizer trial, 1, we select the best performing optimizer on average across all training tasks. We then continue to select optimizer parameters such that the minimum of all optimizer-praameters parameters per task, aggregated over all tasks is minimized. This shifts the complexity from exponential in I to linear. Finding a length set of optimizers can thus be efficiently computed as follows:
0 = arg min 8
Σ S(T,0)
(3)
T Elasks
0 arg min
min (b, f(T, 0])]
where 6
min iEI..(k-1)
01),
(4)
Ag
Lllskk
We note that the first argument of the outer min, can be computed once per set of hyperparameters it does not depend on 0. Finally, as our tasks are stochastic, we order optimizers based validation loss and report test loss (Van Hasselt et al. 2016
This training strategy requires an original search space from which collect data and build @. The search space use is described in Appendix E.2] While large, we find that the optimal parameters for each task end up covering almost the entire space. At some point, no improvement can be obtained on any of the tasks in the dataset. At this point, we simply randomly order the remaining optimizers though expect more sophisticated methods could be employed.
4 EXPERIMENTS: TRAINING AND GENERALIZATION OF LEARNED HYPERPARAMETER LISTS
With our dataset of tasks and data collected, we turn our attention to exploring training of the hyperparameter lists, and generalization beyond the suite of tasks in TaskSet. In this exploration,
Thii technically means that increasing the number of optimizes could potentially decrease performance, but find this rarely happens in practice.
5

---

Under review as a conference paper at ICLR 2021
0.5
Learned
on holdaut tacks
0.4
Rand: AdamAp
Rand: Acamáp
Rand. Adamhp
0.3
Relined, mimmax
Relined: pero
0.1
10
10
10P
10P
number of optimizere tried
By learning a search space we achieve large speedups over random search. On the y-axis we show or the best aggregated and normalized performance achieved given some number of optimizer trials (x-axis). This is computed we show median performance with 25-75 percentile shown with error bars over 50 resamplings of the train-test split of tasks, as well as random samplings. In black we show a learned search space computed from the Adam8p family of optimizes. In color we show various random search baselines. See $4.1 I for description of these.
Figure 2:
heldout tasks not used to train the hyperparameter list. In solid
we hope to give a flavor of the types of research possible with TaskSet. Our main tool show performance figures that sweep the number of optimizers configurations the x-axis, and show the best performance achieved for each number of optimizers tried, averaged over some set of tasks (Eq.
4.1 LEARNED HYPERPARAMETER LISTS ARE MORE EFFICIENT THAN RANDOM SEARCH
To demonstrate the impact of learning search space, take the 1,162 tasks split them into even train and test tasks. We then learn a search strategy using optimizers from the Adam&p family following on n the train tasks. Results in Figure 3 As baselines, we use random search with different search spaces, including just learning rate (Rand: Adamlp), the default Adam hyper parameters (Rand: Adam4p), well the Adam 8 dimensional search space (Rand: Adam8p). To better get sense of performance, we show two additional "Refined" baselines which involve random sampling from better search space. For min/max, we sample from the minimum bounding box containing the best hyperparameters for each task. To improve the search space quality, shrink this bounding box so 90% of the best hyperparameters are enclosed. Further considerations regarding search space volume are treated in I and the precise search spaces are specified in Appendix E22] Finally, one difficulty of working with offline data is the difficulty of running online hyperparameter optimization methods such Bayesian Optimization without running additional compute. Future work will explore offline Bayesian methods.
4.2
MORE TASKS LEAD TO BETTER GENERALIZATION
We next look at the effects of the number of training tasks on generalization. We take subsets of tasks of different size, and train hyperparameter lists using Eq We compute test performance the remainder of the tasks and plot loss averaged over different splits in Fig. 3 We find that large number of tasks (more than 100) are required to achieve near-optimal test performance. This is surprising us given how simple our learned search strategy is (simply list of hyperparameters), but not wholly so given past work studying generalization in RL (Cobbe al. 2018).
4.3 GENERALIZATION TO DIFFERENT TYPES OF PROBLEM
For learned algorithms to be generally useful, some amount of generalization to unseen task families required. To test this, we split our data into disjoint task types. We perform two splits: testing on RNN tasks and training on all others, and testing on autoencoder tasks and training all others. As a best case baseline we additionally train search spaces on the test task families directly. We find an order magnitude better sample efficiency than random search for both cases and find our learned search space is close in performance search spaces trained on just the testing tasks (Fig. 3

---

Under review as a conference paper at ICLR 2021
0,160
RNN
0.40
AE Tooks
0,150
0.155
0.145
0.
0.155
0.28
0.84
0.28
0.35
0.90
Leart Sorvo tasks
Rend Adbinit
Fannd Adknts
Other tance
0.22
0.136
10 Txme
0.
0.20
0.20
0.130
0.125
30 Tasto
120 Tade
0.
0.130
35 Opprsiooss
Opeimoons
0.18
0.18
0.15
0.10
0.120
0.120
0.14
10
(a)
Optimzuus tried
(b)
Namber laaike used learning
number of optimizers
(d)
number of ofoptiizurs tried
Figure 3: Using more tasks to train the search space results in improved performance heldout tasks. (a) The number of optimizers tried vs performance on heldout tasks. In color, we show different numbers of tasks used to learn the search spaces. We show median performance with error bars denoting 25 and 75 percentile. (b) Performance at a fixed number of optimizers tried vs number of tasks used for meta-training We find performance continues to improve as we meta-train on more tasks. These plots are slices out of (a), with colors matching the vertical dashed lines. (c,d) We show aggregate performance (J) as function of number of optimizers tried when training the hyperparameter list different distributions of tasks than those we test on. We show testing on RNNs (c) and auto encoders (d), and train on 700 tasks sampled from the remainder set of tasks. We find that these learned search spaces perform much better than random search both the learning rate search space and in the original Adam8p search space. We additionally plot the best case performance the case where we train and test on the same problem type. We show median and 25-75 percentile averaged over 50 different samplings.
EXPERIMENTS: REALISTIC PROBLEMS
In and we explored generalization of learned hyperparameter lists to held out tasks within the TaskSet dataset. While useful for analysis, these tasks are still far from the workloads commonly employed to solve real problems. In this section, we explore the performance of our learned search space on number of state of the art models. These models drastically differ from the training set of tasks in parameter count and compute cost. We see these experiments as evidence that the tasks presented in TaskSet capture enough of the structure of "realistic" problems that Task can be used to improve larger scale workloads. For all experiments in this section take the optimizer ordering using the NAdamW optimizer family on all TaskSet tasks then apply the resulting search space to the target problem. The final ordered list of hyperparameters used is in Appendix G We show results for ResNet50 on ImageNet, and Transformers on LMIB. Additional results with reinforcement learning using PPO are in Appendix B.2
First we explore ImageNet classification using a ResNet50. on We take the TPU implementation with default settings from the official Tensorflow models repository (Tensorllow 2019). and swap out different optimizers. We show accuracy computed over the course of training well as best performance for a given hyperparameter budget in Figure 4 vastly outperforms learning rate tuned Adam.
We find that the learned search space
Next we explore language modeling LMIB with a Transformer. We take the transformer (Vawwani et al. 2017) example implemented in Jax (Bradbury et al.) 2018) with Flax (Flax Developers 2020 We train using a 2x2 TPU V2 configuration for 100k iterations. Once again we take all other hyperparameters as is and simply swap optimizer implementation. We find the learned hyperparameter list dramatically outperforms the default optimizer setting and the fixed learning rate baseline. Nevertheless, we emphasize that our method does not require any knowledge of the underlying problem to achieve faster results. See Appendix B.3 for this same transformer with a budget of 20k iterations.
RELATED WORK
The idea of sets of tasks has been explored throughout machine learning. The majority of these suites are for use in evaluation where as our suite is targeted for meta-learning. The closest family of optimization tasks for evaluation those presented here is DeepObs (Schneider al. 2019) which
7

---

Under review as a conference paper at ICLR 2021
0)
4)
Insueell Rartaro
0.7
Imagenet Resnento
0.78
3.9
3.8
Plyntt Acamer
lest acturey
0.8
0.5
feet feet arturary
0.72
0.76
Teoo cross antrog
3.8
Teon cross ent
3.8
Flunet Nilaa
Foontt Schedsll
0.4
0.68
0.70
Detuul SCOM
Learno)
Rand Adami.
3.8
5.8
0.88
Rand HAvden
0.2
50000
100000
traininn spach
(umber of opoliiizerrs tried
Training haraton
number optimbzer tried
We find our learned optimizer list outperforms both learning rate tuned Adam and default training hyperparameters for ResNet50 and a 53M parameter Transformer trained on LMIB. a) Learning curves for the best hyperparameter from each optimizer class on the ResNet50 task. b) Number of optimizers tried best top ImageNet accuracy achieved on the ResNer50 task. For NAdamW, we first train models with a validation set created by splitting the training set. We compute maxes over this set, and show the performance when retraining on the full training set on the official ImageNet validation set. For AdamLR, we simply compute max over the official ImageNet validation set. c) Training curves for the best hyperparameter for each optimizer class on the Transformer task. d) Number of optimizers tried vs the test loss obtained given the best validation performance on the Transformer task.
Figure 4:
includes 20 neural network tasks. Our task suite focuses on smaller problems and contains 50x more tasks. Outside of evaluation, task suites in reinforcement learning such as Obstacle Tower (Juliani
al.
2019). ProeGen (Cobbe al. 2019), CoinRun (Cobbe al. 2018), and Sonic (Nichol et al. focus on training algorithms that work across variety of settings.
2018 8
The creation of TaskSet was motivated by the goal of learning learning algorithms, or meta- learning (Schmidhuber 1987| [995] Hochreiter et al. (001), and in particular learned optimizers
1990 Andrychowicz et al. 22016] Bello et al. 2017: Wichrowska et al. 2017 Li & Malik 2017 Lv et al. 2017] Metz et al. 2019a b). This use case is explored with this dataset in (Metz et al. 2020). In this work we do not use this task suite to train learned optimizers, but instead
(Bengio et al.
focus on learning hyperparameter search strategy. Tuning hyperparameters by leveraging multiple
tasks has been explored within the contexts of Bayesian optimization (Swersky et al.) (2013); Perrone
& Shen (2019); Perrone et al. (2018) as well as meta-learning (Reif et al., [012] Gomes et al. 2012)
Feurer al. 2014] Wistuba et al. 20155b a. Chen et al. 2017 Pfisterer et al. 2018). See Appendix
for full discussion of sets of tasks in machine learning, Appendix F.2 for more info on optimization
in machine learning, and Appendix F3 for a discussion on existing hyper parameter search methods.
7 DISCUSSION
Learning optimization algorithms represents promising direction for accelerating machine learning research. For the resulting algorithms to become useful tools, however, we must further understand the relationships between training tasks, meta-optimization, and both iid and out of distribution generalization.
This work takes steps towards this goal by introducing significantly larger set of optimization tasks than ever previously considered. As an example use-case, we provide a thorough analysis of how TaskSet enables meta-oprimization of simple, but performant hyperparameter lists. Despite this approach's 's simplicity, the training of learned learning algorithms is computationally expensive. We hope to explore alternative parameterizations which will increase efficiency by, e.g., leveraging
previous evaluations or partial model training (Swersky et al. 2014 Li al. 2016).
We are releasing the optimal hyperparameter list have found as drop-in replacement optimizer in variety of deep learning frameworks (Tensorflow (Abadi et al.] 2016)) PyTorct (Paszke et al.) 2019), and JAX (Bradbury et al.) 2018) in the hopes that the research community finds them useful. We believe this represents new set of reasonable optimizer defaults for new problems. Finally, we hope Tas encourages more standardized research general purpose optimizers.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
URL. sittps://s3. awazanaasass.............. http://wwwwaa..cc.aaaaaaaaaaaaaa
.ttm
Martín Abadi, Paul Barham, Jianmin Chen. Zhifeng Chen, Andy Davis, Jeftrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In OSDI, volume 16, pp. 263-283, 2016.
Dario Amodei and Danny Hernandez. Ai and compute. Heruntergeladen https://bllg. openai. com/aiand-compute, 2018.
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Plau, Tom Schaul, and Nando de Freitas. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems, pp. 3981-3989, 2016.
Jimmy Ba, Geoffrey E Hinton, Volodymyr Mnih, Joel Z Leibo, and Catalin Ionescu. Using fast weights to attend to the recent past. In Advances in Neural Information Processing Systems, pp. 2016.
David Balduzzi, Karl Tuyls, Julien Perolat, and Thore Graepel. Re-evaluating evaluation. In Advances in Neural Information Processing Systems, pp. 3268-3279, 2018.
David Balduzzi, Marta Gareel,, Yoram Bachrach, Wojciech M Czameekki, Julien Perolat, Max Jaderberg, and Thore Graepel. Open-ended learning symmetric zero-sum games. arXiv preprint arXiv:1901.01100 2019.
Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Mareus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, al. Deepmind lab. arXiv preprint arXiv:1612.08801, 2016.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Irwan Bello, Barret Zoph, Vijay Vasudevan, and Quoe Le. Neural optimizer search with reinforcement learning. 2017. URL hittps://arxiv.or//pptat wttp:ooog/pppoo/oooo/ 07417. pdf
Yoshua Bengio, Samy Bengio, and Jocelyn Cloutier. Learning a synaptic learning rule. Université de Montréal, Département d' d'informatiuue et de recherche opérationnelle, 1990.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(Feb):2813303 2012.
James S. Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-parameter optimization. In J. Shawe- Shawe-Taylor, R. S. Zemel, L. Bartlett, F. Pereira, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 24, pp. 2546-2554. Curran Associates, Inc., 2011.
Yaniv Blumenfeld, Dar Gilboa, and Daniel Soudry. A mean field theory of quantized deep networks: The quamtization-depth trade-off. arXiv preprint arXiv:1906.0077) 2019.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative compo- nents with random forests. In European Conference Computer Vision, 2014.
Olivier Bousquet, Sylvain Gelly, Karol Kurach, Olivier Teytaud, and Damien Vincent. Critical hyper-parameters No random, cry. arXiv preprint arXiv:1706.03200, 2017.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclau- rin, and Skye Wanderman-Milne. JAX: composable transformations of Python+NumPy programs, 2018. URL http::/tt///t cww/ comm/oogge//jax
G. Bradski. The OpenCV Library. Dr. Dobb's Journal of Software Tools, 2000.

---

Under review as a conference paper at ICLR 2021
Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Smash: one-shot model architecture search through hypernetworks. arXiv preprint
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, and Phillipp Koehn. One billion word benchmark for measuring progress in statistical language modeling. CoRR, abs/1312.3005, 2013. /tL//ttp://axvvvoor//////
3005
Yutian Chen, Matthew W Hoffman, Sergio Gómez Colmenarejo, Misha Denil, Timothy P Lillicrap, Matt Botvinick, and Nando de Freitas. Learning to learn without gradient descent by gradient descent. In Proceedings of the 34th International Conference on Machine Learning-Volum 70, pp- 748-756, JMLR. org. 2017.
Yutian Chen, Aja Huang, Ziyu Wang, Ioannis Antonoglou, Julian Schrittwieser, David Silver, and Nando de Freitas. Bayesian optimization in alphago. arXiv preprint 1/X.n:18/2..06855, 2018.
Dami Choi, Christopher Shallue, Zachary Nado, Jaehoon Lee, Chris J Maddison, and George Dahl. On empirical comparisons of optimizers for deep learning. arXiv preprint arXiv: 1XXx::9900005400 2019.
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint arXiv: 170.008777.088// 2017.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv: 2014.
Karl Cobbe, Oleg Klimov, Chris Hesse, Tachoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. arXiv preprint arXiv:1812.0234/ 2018.
Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation benchmark reinforcement learning, 2019.
Ian Dewancker, Michael McCourt, Scott Clark, Patrick Hayes, Alexandra Johnson, and George Ke.
A stratified analysis of bayesian optimization methods. arXiv preprint 1rXiv:1603.000 09441, 2016.
Laurent Dinh, Jascha Sohk-ticks,,,,, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv: 08803, 2016.
Elizabeth D Dolan and Jorge J Moré. Benchmarking optimization software with performance profiles. Mathematical programming, 91(2):201-213, 2002.
Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.
Timothy Dozat. Incorporating nesterov momentum into adam. 2016.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Ju)):1211-115, 2011.
Katharina Eggensperger, Frank Hutter, Holger Hoos, and Kevin Leyton-Brown Efficient benchmark- ing of hyperparameter optimizers via surrogates. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Jeffrey L Elman. Finding structure in time. Cognitive science, 14(2):19--111, 1990.
Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter optimiza- optiiin tion scale. arXiv preprint arXiv: 807.0/77777 2018
10

---

Under review as a conference paper at ICLR 2021
Matthias Feurer, Jost Tobias Springenberg, and Frank Hutter. Using meta- meta-learning initialize bayesian optimization of hyperparameters. In Proceedings of the 2014 International Conference on Mheta-learning and Algorithm Inlection-Volumeae 1201, pp. 3-10. Citeseer, 2014.
Flax Developers. Flax: A neural network library for jax designed for flexibility, 2020. URL https: /ttt://///thubbb com/ httpa//eeeeeeaaa/aa///e/////e//e
Wikimedia Foundation. Wikimedia downloads. URL
Boris Ginsburg. Patrice Castonguay, Olcksii Hrinchuk, Oleksii Kuchaiev, Vitaly Lavrukhin, Ryan Leary, Jason Li, Huyen Nguyen, and Jonathan M Cohen. Stochastic gradient methods with layer-wise adaptive moments for training of deep networks. arXiv preprint arXiv: 2019.
arXiv:1905. 1905. 1226,
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, 2010.
Taciana AF Gomes, Ricardo BC Prudêncio, Carlos Soares, André LD Rossi, and André Carvalho. Combining meta-learning and search techniques to select parameters for support vector machines. Nearocomputing, 75(1):3-13, 2012.
Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv: 1308.003 0850, 2013.
Sergio Guadarrama, Anoop Korattikara, Oscar Ramirez, Pablo Castro, Ethan Holly, Sam Fishman, Ke Wang, Ekaterina Gonina, Neal Wu, Efi Kokiopoulou, I Luciano Shaiz, Jamie Smith, Gábor Bartók, Jesse Berent, Chris Harris, Vincent Vanhoucke, and Eugene Brevdo. TF-Agents: A library for reinforcement learning in tensorflow, /tttps:/
http:/eensorfffoo agents 2018. [Online; accessed 25-June-2019
Nikolaus Hansen, Anne Auger. Olaf Mersmann, Tea Tusar, and Dimo Brockhoff Coco: A platform for comparing continuous optimizers black-box setting. arXiv preprint vrXiv:1603.003 2016.
Soufiane Hayou, Arnaud Doucet, and Judith Rousseau. On the selection of initialization and activation function for deep neural networks. arXiv preprint arXiv: 08266,, 2018
Soufiane Hayou, Arnad Doucet, and Judith Rousseau. Mean-field behaviour of neural tangent kernel for deep neural networks, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034 1034, 2015.
Nicolas Heess, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, Martin Riedmiller, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint a 2017.
Geoffrey E Hinton and Ruslan Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 3 31365786) 2006.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to learn using gradient descent. In International Conference on Artificial Neural Networks, pp. 87-94. Springer, 2001.
Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren (eds.). Automatic Machine Learning: Methods, Systems, Challenges. Springer, 2018. In press, available http://auuooll.or/bbboooo
11

---

Under review as a conference paper at ICLR 2021
Arthur Juliani, Ahmed Khalifa, Vincent- Pierre Berges, Jonathan Harper, Hunter Henry, Adam Crespi, Julian Togelius, and Danny Lange. Obstacle tower: A generalization challenge in vision, control, and planning. arXiv preprint arXiv:1902.01377, 2019.
Ryo Karakida, Shotaro Akaho, and Shun-ichi Amari. Universal statistics of fisher information in deep neural networks: mean field approach. 2018.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv:1312.6114 14, 2013.
arXiv preprint
Aaron Klein and Frank Hutter. Tabular benchmarks for joint architecture and hyperparameter optimization. arXiv preprint arXiv: 1905.04970. 2019.
Aaron Klein, Stefan Falkner, Jost Tobias Springenberg, and Frank Hutter. Learning curve prediction with bayesian neural networks. 2016.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- 10 and cifar- 100 datasets. URI: https://www. cs. toronto. edukrii/cifar. html, 6, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton, Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Manoj Kumar, George E Dahl, Vijay Vasudevan, and Mohammad Norouzi. Parallel architecture and hyperparameter search via successive halving and classification. arXiv preprint arXiv: /805. 10255, 2018.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mmist// 1998.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning, nature, 521(7553))436-444 2015.
Ke Li and Jitendra Malik. Learning to optimize neural nets. arXiv preprint arXiv: 2017.
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization, arXiv preprint 2016.
Ping Li and Phan-Minh Nguyen. On random deep weight-tied autoencoders: Exact asymptotic analysis, phase transitions, and implications to training. In International Conference Learning Representations, 2019. URL https:://penrrviies net outp://wwwweeee/ooooer///doodddd
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arXiv preprint 1ttf:::990000.255. 2019.
Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. arXiv preprint 17Xiv:1711.05101, 2017.
Kaifeng Lv, Shunhua Jiang, and Jian Li. Learning gradient descent: Better generalization and longer horizons. arXiv preprint arXiv: 1703.03333 2017.
Andrew L. Maas, Raymond E. Daly, Peter Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http 7/ 7/www. web. www.ee/wrg//aaaoooooo///111/
12

---

Under review as a conference paper at ICLR 2021
Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9Noov) 2008.
Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model of large-batch training. arXiv preprint arXiv:1812.06162, 2018.
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning question answering. arXiv preprint arXiv: 000600 2018.
Gábor Melis, Chris Dyer, and Phil Blunsom. On the state of the art of evaluation in neural language models. arXiv preprint arXiv: /77vv0/77.00588, 2017.
Luke Metz, Niru Maheswaranathan, Jeremy Nixon, Daniel Freeman, and Jascha Sohl-Dickstein. Understanding and correcting pathologies in the training of learned optimizers. In International Conference on Machine Learning, pp. 4556 4565, 2019a.
Luke Metz, Niru Maheswaranathan, Jonathon Shlens, Jascha Sohl-Dickstein, and Ekin Cubuk. Using learned optimizers make models robust input noise. arXiv preprint arXiv: (eee//90600030 2019b.
Luke Metz, Niru Maheswaranathan, Daniel Freeman, Ben Poole, and Jascha Sohl-Dickstein. Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves. arXiv preprint WxXiv22009.11244 2020.
Sameer A Nene, Shree K Nayar, Hiroshi Murase, et al. Columbia object image library (coil-20). 1996.
Yurii Nesterov. method for unconstrained convex minimization problem with the rate of conver- gence (1/k" 2). In Doklady AN USSR, volume 269, pp. 543-547, 1983.
Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman. Gotta learn fast: A new benchmark for generalization in rl. arXiv preprint arXiv: /804033700 2018.
Alex Olsen, Dmitry A. Konovalov, Bronson Philippa, Peter Ridd, Jake C. Wood, Jamie Johns, Wesley Banks, Benjamin Girgenti, Owen Kenny, James Whinney, Brendan Calvert, Mostafa Rahimi Azghadi, and Ronald White. DeepWeeds: Multiclass Weed Species Image Dataset for Deep Learning. Scientific Reports, 9(058)), 2 2019. doi: 111100//4111--01--888888888
URL
OpenAI, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng. Qiming Yuan, Wojeiech Zaremba, and Lei Zhang. Solving rubik's cube with robot hand. arXiv preprint, 2019.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems, pp.
2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjic Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, c E. Fox, and R. Garnett (eds.), Advances Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Valerio Perrone and Huibin Shen. Learning search spaces for bayesian optimization: Another view of hyperparameter transfer learning. In Advances in Neural Information Processing Systems, pp. 12751-12761, 2019.
Valerio Perrone, Rodolphe Jenatton, Mathias W Seeger, and Cédric Archambeau. Scalable hyperpa- rameter transfer learning. In Advances Neural Information Processing Systems, pp. 6845-6855, 2018.
13

---

Under review as a conference paper at ICLR 2021
Johann Petrak. Fast subsampling performance estimates for classification algorithm selection. In Proceedings of the ECML-00 Workshop on Meta-nearnnn:: Building Automatic Advice Strategies for Model Selection and Method Combination, pp. 3-14. Citeseer, 2000.
Florian Pfisterer, Jan N van Rijn, Philipp Probst, Andreas Müller, and Bernd Bischl. Learning multiple defaults for machine learning algorithms. arXiv preprint arXiv: /8/ I. 1.09409, 2018.
Arnu Pretorius, Elan Biljon, Steve Kroon, and Herman Kamper. Critical initialisation for deep signal propagation in noisy rectifier neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Baaa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31. pp- 5717-5726. Curran Associates, Inc., 2018.
Prajit Ramachandran, Barret Zoph, and Quoe Le. Searching for activation functions. 2017.
Rapin and O. Teytaud. Nevergrad A gradient-free free optimization platform. https://GitHu:
com/ htt.://ccbooceeeeerrre/eeeeerrrr
2018
Matthias Reif, Faisal Shafait, and Andreas Dengel. Meta-learning for evolutionary parameter optimization of classifiers. Machine learning. 87())3577-380, 2012.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (3):211-252, 2015. doi: 10.1007/s1 126001150111
Tom Schaul, Ioannis Antonoglou, and David Silver. Unit tests for stochastic optimization. arXiv preprint
2013.
Juergen Schmidhuber. On learning how to learn learning strategies. 1995.
Jürgen Schmidhuber. Evolutionary principles in self-referential learning, learning how learn: the meta-meta-.. hook. PhD thesis, Technische Universität München, 1987.
Frank Schneider, Lukas Balles, and Philipp Hennig. Deepobs: deep learning optimizer benchmark suite. International Conference Learning Representations, 2019.
Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. arXiv preprini arXiv: 1. 1.01232, 2016.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv: 1508. 0598007909 2015.
Christopher J Shallue, Jaehoon Lee, Joe Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E Dahl. Measuring the effects of data parallelism on neural network training. arXiv preprint 181 2018
Prabhu Teja Sivaprasad, Florian Mai, Thijs Vogels, Martin Jaggi, and François Fleuret. On the tunability of optimizers deep learning. arXiv preprint A910 1758, 2019.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In Advances in neural information processing systems, pp.
2012.
Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. Scalable bayesian optimization using deep neural networks. In International conference on machine learning, pp. 2171-2180 2015.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nip. arXiv preprint 1rXiv:1906.0220 2019.
Kevin Swersky, Jasper Snoek, and Ryan P Adams. Multi-task bayesian optimization. In Advances in neural information processing systems, pp. 2004-2012, 2013.
14

---

Under review as a conference paper at ICLR 2021
Kevin Swersky, Jasper Snoek, and Ryan Prescott Adams. Freeze-thaw bayesian optimization. arXiv preprint arXi:: 2014.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. DeepMind control suite. Technical report, DeepMind, January 2018. URL https
Tensorflow. tensorflow tpu resnet50, Oct 2019. URL Eppp/treee/mastrr/mmeee s/officiifiiiioooooooooooo resnet
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-mmsprop: Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26-31, 2012.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEFE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 23-30. IEEE, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSI International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta-dataset: dataset of datasets for learning to learn from few examples. arXiv preprint arXiv: 1903v:9003.30000 2019.
Laurens Van Der Maaten. Accelerating t-sne using tree-based algorithms. The Journal of Machine Learning Research, 15(1):3221-322 -3245, 2014.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Thirlieth AAAI conference artificial intelligence, 2016.
Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo Openml: Networked science in machine learning. SIGKDD Explovations, 15(2):49 60, 2013. doi: 10.1145/2641 190.2641 198.
acm. org 0ttpg/.1..15/224411111
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Daan Wierstra, al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. 3630-3638, 2016.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Oner Levy, and Samuel R Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv preprint 190500000 2019.
Paul Werbos. Backpropagation through time: what it does and how do it. Proceedings of the IEEE, 18510):5500-1560) 1990.
Olga Wichrowska, Niru Maheswaranathan. Matthew W Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando de Freitas, and Jascha Sohl-Dickstein. Learned optimizers that scale and generalize. International Conference Machine Learning, 2017.
15

---

Under review as a conference paper at ICLR 2021
Ashia C Willsn, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In Advances in Neural Information Processing Systems, pp. 4148 4 4188, 2017.
Martin Wistuba, Nicolas Schilling. and Lars Schmidt-Thieme. Learning hyperparameter optimization initializations. In 2015 IEEE international conference on data science and advanced analytics (DSAA), pp. 1-10. IEEE, 2015a.
Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thieme Sequential model-free hyperparameter tuning. In 2015 IEEE international conference on data mining, pp. 1033-1038. IEEE, 2015b.
David H Wolpert and William G Macready. No free lunch theorems for optimization. IEEE transactions on evolutionary computation, 1(1):67-82, 1997.
Yuhuai Wu, Mengye Ren, Renjie Liao, and Roger B Grosse. Understanding short-horizon bias in
stochastic meta-optimization. pp. 478-487, 2016.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion- a novel image dataset for benchmarking machine learning algorithms, 2017.
Xiao, J. Hays, K. A. Ehinger. A. Oliva, and A. Torralba. Sun database: Large-scale scene recognition from abbey zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 3485-3492, June 2010. doi: 10.1 00.109/CVPR.220055599977
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel Schoenholz, and Jeffrey Penning- ton. Dynamical isometry and mean field theory of CNNs: How to train 10,000-layer vanilla convolutional neural networks. In International Conference on Machine Learning, 2018.
Ge Yang and Samuel Schoenholz. Mean field residual networks: On the edge of chaos. In Advances In Neural Information Processing Systems, 2017.
Chris Ying, Aaron Klein, Esteban Real, Eric Christiansen, Kevin Murphy, and Frank Hut- ter. NAS-Bench-101: Towards Reproducible Neural Architecture Search. arXiv e-prints, art. arXiv:1902.09635. Feb 2019.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, al. The visual task adaptation benchmark. arXiv preprint 190.00011 2019.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint 16.05578 2016.
Barret Zoph and Quoe Le. Neural architecture search with reinforcement learning. International Conference on Learning Representations, 2017. URL http:/::///xxxvw.///eeeee 01578
16

---

Under review as a conference paper at ICLR 2021
model Surily
Opth
language
poolig lamily
ling boaks lamily
mat Jamiy
mip tinis
mip av Jumily
cony Jurbly
Figure S1: A 2D TSNE embedding of all 1162 tasks. This embedding is produced from 1,000 dimensional feature vector consisting of task loss evaluated with many different hyperparameter configurations. We find similar tasks - c.g. masked auto regressive flow models, and character word RNN models cluster, suggesting similarity in the optimizers that perform well. See $?? for more details.
A TASKSET VISUALIZATION
For qualitative view, we constructed a feature space consisting of performance measurements for each task+optimizer pair (See (3.3). This forms dense matrix of size number of tasks by number of optimizers. We then perform T-SNE Maaten & Hinton 2008 Van Der Maaten 2014) to reduce the dimensionality to two and plot the results coloring by task family (Figure (SI). Clusters in this space correspond to tasks that work well with similar optimizers. We find diversity of tasks with clusters occurring around similar families of tasks.
A.1 TSNE OF TASKSET
B
ADDITIONAL EXPERIMENTS
B.1 GENERALIZATION TO DIFFERENT SIZED PROBLEMS
Training learned algorithms large models is often infeasible for computational reasons. As such, one form of generalization needed when building learned algorithms is the ability to transfer to different sized models. As shown in Figure the tasks in this suite contain a wide range of parameter counts, and can thus be used test this kind of generalization. We split the tasks into groups group per order of magnitude in parameter count, and train hyperparameter lists on one range and test on the rest. In Figure S2 we plot the fraction of the training loss achieved by the test loss on the target parameter range.
find peak performance around the model sizes used for training, and smooth falloff as the testing tasks become more dissimilar measured by parameter count. We note that our problems are not evenly distributed across these groups thus each group will contain a different percentage of the underlying tasks. While this potentially confounds these results, we believe a similar bias occurs in realistic workloads as well.
0.9
0.7
0.8
3-4
0.4
0.2
0
3
number of parameters (log10)
Figure S2: We show learned search space generalization, measured as ratio of the loss achieved
training and testing, versus the number of task parameters used during search space training. Generalization falls off as one uniform mixture of the 7 parameter buckets does not fall off.
further away from the training regime. In black show that
17

---

Under review as a conference paper at ICLR 2021
B.2
REINFORCEMENT LEARNING WITH PPO
Half Chestah
Humanold
500
1500
400
Reward
300
1000
500
Learned
100
Rend: AdamLr
Rand: NAdam
0
100
101
102
109
109
10
102
109
number of optimizers tried
number of optimizers tried
Figure S3: find our learned hyperparameter lists performs about as well as random search on the NAdam search space, and worse than the random search on the learning rate tuned Adam search space.
We test the learned hyperparameter lists continuous control reinforcement learning environ- ments, half cheetah and humanoid, from Gym's 's Mujoco environments (Todoro et al. 2012 Brockman et al. 201 We use TF-Agents (Guadarrama et al. 2018) with all non-optimizer hyperparameters set via searching mixture of environments. In figure B.2 we find our learned hyperparameter lists achieves comparable to slightly worse performance does not out perform learning rate tuning of Adam in both efficiency nor final performance. To diagnose this behavior we ran all 1k optimizers for both problems and found the learned hyperparameter list performs comparable to random search in the underlying space. To probe further, we computed spearman correlation on the performance of each optimizer as compared to the rest of the tasks in the task suite. We found considerably worse correlations than where present for tasks in the TaskSet. This not surprising as TaskSet contains no reinforcement learning problems.
B.3 LMIB TARGETING 20K ITERATIONS
We show a transformer on LMIB similar to that shown in $ except run for only 20k iterations, fith of the steps. Results in Figure S4 than either of the baselines.
We find the learned hyperparameter lists much more efficient
5.0
4.8
Learned
entoppy
entroyy
Fannd Lr Schodulo
4.4
3.9
Raand Nadeum
4.2
4.0
3.6
10000
20000
10
10P
10a
Training iteration
optimizer
Figure S4: We find our learned hyperparameter lists out performs learning rate tuned Adam with both constant, and a fixed learning rate schedule 53M parameter Transformer trained LMIB. Left: Learning curves for the best of the optimizers. Right: Number of optimizers tried vs best test loss.
B.4 PROBING SHORT HORIZON
Often the goal when training learned optimizers to minimize performance after training some number of iterations. This is extremely computationally expensive and in practice approximations must be used, One common family of approximations is short horizon based methods. These methods rely upon somehow truncating training so that updates can made to the learned optimizer more frequently. This is commonly done via truncated backprop (Werbos 1990) Wichrowska al. 2017
18

---

Under review as a conference paper at ICLR 2021
min aggregate task perfomance
144
meta train (ft)
meta-train shorthorizon (H-11)
meta- train shorthorizon (H- random
1.0
0.6
02
-02
optimizers tried
Figure S5: Hyperparameter lists trained on short horizon data generalize remarkably well. On the y-axis we show performance evaluated on the the full 10k training iterations for given number of optimizers tried (x-axis). In color show different number of steps used when evaluating task optimizer performance when training the hyperparameter list.
Metz et al. 2019a Wu al., 2016), proxy objectives such only training for handful of epoch Zoph & Le 2017 While this short horizon proxy certainly not optimal(Wu et al. 2016). the performance gains are immense and in practice what makes meta-training optimizers feasible. In task suite, we test this short horizon learning by training hyperparameter lists only using some finite amount of training iterations per task and testing in the full training regieme (10k steps). Results in figure S5 We find that even when learning the hyperparameter list on a mere 200 steps, our hyperparameter list continues generalize outperform random search on Adam8p. This promising as this suggests that training the learned hyperparameter list can be done with 1/50th of the total compute. This result is surprising to us as prior work indicates the effect of this bias can be severe (Wu et al., [2016] |etz al.) (2019)). We suspect is due to the simplicity of the learned parameter space but leave thorough amalysis of this for future work.
detbult norm
quantie nor
quantie nom
0.8
0.8
0.6
0.6
0.5
random
0.4
0.1
0.0
0.0
10a
10a
0.0
0.2
0.4
0.6
0.2
0.4
0.8
1.0
default norm
default norm
Figure S6: Left: Aggregate performance (y-axis) number of optimizer tried (x-axis) for different normalization and aggregation techniques. In each curve we train the hyperparameter list with dif- ferent normalization and aggregation strategy and test with the default normalization and aggregation technique described in 3.3 We find some some strategies are identical in performance (e.g. min norm). while others perform significantly worse e.g. last quantile norm. In both cases, however, we still perform better than the underlying random search. Center: Correlation between default nor- malization and the quantile based normalization strategy. Correlation is quite low 0.193 Pearson's correlation. Right: Correlation between the default normalization using validation over the course of training vs using a min over validation over the course training. We find a much higher correlation of 0.911.
to aggregate
B.5 CHOICE OF NORMALIZATION FUNCTION
There is no easy way define single metric for optimizer performance over mixture of tasks. This paper picks single normalization strategy based on minimum validation loss and the validation loss at initialization presented in $3.3 In this section we show the impact of choosing different normalization and or aggregation technique. First, instead of computing the mean over learning curves as described in $3.3 we compute min. Second, instead of rescaling based init and min,
19

---

Under review as a conference paper at ICLR 2021
linearly rescale based the 95 percentile of validation loss and the min validation loss achieved at the end of training each task.In Figure S6 we show learned hyperparameter list training and testing performance as a function of number of optimizers tried when training with different normalization techniques. We find using the min instead of mean results in negligible change, while using the percentile loss more significantly hurts performance. This difference can be explained by Figure SSop and [S6p where we show correlations between the two losses. We find the percentile loss has a much weaker correlation the default normalizer. We suspect this difference due the fact that many optimizers diverage on tasks. By using the 95 percentile upweight optimizers that do not diverge.
B.6 TASK FAMILIES ARE DIVERSE
To show the effects of diversity train and test hyperparameter lists each pair of task family. We additionally normalize each column from 0-1 to account for different mean losses across tasks. Results in Figure(ST S7] While we do find some similarity in tasks e.g. between MAF and NVP models, but no tasks behave the same performance characteristies (no duplicate columns) suggesting that each task family is providing different contribution to the space of all tasks. We also find when training on certain "far away" tasks, e.g. the quadratic family, we find poor performance on most other task families.
0.9
mip_vae
0.8
Taat family
mlp ae-
0.7
0.6
0.5
mlp
0.4
mn class
char Im__mm
losg lasks
0.1
quadratic
0.0
mvp
mlp ne conv. pooling conv fo mip mm Jm
mn toxt class- char ron Iin long tasks
quadratic
mip
word,
Train task family
Figure S7: Learning hyperparameter lists using one task family and testing on the remainder of task families. We normalize each column from 0-1 account for different mean losses across tasks. Lower loss means better performance. We find some groups of similar tasks, but in general no two task families behave identically.
B.7 EFFECTS OF THE
SEARCH SPACE SIZE
Our offline learning technique described in $3.4 hinges on finite of optimizers collected via random search. This set is denote by θ in Eq 4 In this section we probe the impact of this size. We take different sized subsets the the thousand Adam8p optimizer configurations and train and test search spaces on different iid splits of tasks. We then plot performance as function of this number of optimizers in Figure $9 Moving left in this figure corresponds increasing the compute needed to train the learned hyperparameter list. We find performance continues to improve as the size of θ grows. Given the high dimension of our meta peeta-paramtter, 8, this is not a surprise as the number of evaluations needed explore the space will grow exponentially, We find that the full thousand trials needed to out perform learning rate tuned Adam when only given single optimizer evaluation. We find around 100 optimizers (size of +) are needed the case of 10 optimizer trials (A: = 10).
Overall this sugjests that randomsearch might not be the most efficient learning method for creating perparramete lists. This is especially true hyperparameters. Other approximate learning methods should likely be explored such as truncated backprop through time as used by the learned optimizer community et al., 2019a), and/or population based methods (Balduzzi al. 2019
work with optimizer families that have more
20

---

Under review as a conference paper at ICLR 2021
2 hr
1 hr
10k
1 min
10 sec
sec
losg tasks
quadratic
mlp
conv ft
mip vae
nvv
rn text, classification
maf
fixed
chat rnn ionpuae ape mode
pooo yan, Jangusge model
mip
con
Timings computed for each task family. We find most task families have narrow distribution compute times.
Figure $8:
0.40
hparams (Adam8p)
0.35
hparams (Adam4p)
0.30
(Adam8p)
10 hparams (Adam44p)
100 hparams (Adamäp)
0.20
100 hparms (Adam4p)
0.15
Adam
0.10
10
102
10a
number optimizers
Figure S9: Performance continues to improve search spaces. On the x-axis we show number of optimzers (size of e, the number of hyperparameter evaluations used in training the learned hyperparameter list) and y-axis we show test loss achieved when applying the learned search space for given fixed length, e.g. different values of A: shown in color). We plot median with 25-75 percentile shaded over different random optimizer samples and iid task splits. Stars (with horizontal guide lines) denote best search for the corresponding number of hyperparameters for learning rate tuned Adam in half orders of magnitude.
and more optimizers used when training the
C TASK TIMINGS
In Figure S8 we show box plots of training times for each problem. For each task we use the median step time recorded over mixture of different physical devices and multipled by 10k to estimate full training time. Future versions of this dataset of tasks will contain more variation within each task family.
21

---

Under review as a conference paper at ICLR 2021
D OPTIMIZER FAMILY UPDATE EQUATIONS
D.1 ADAM8P UPDATE EQUATIONS
The 8 meta-parameters are: the learning rate, a, first and second moment momentum, 31, 82, the numerical stability term, e, &a and & regularization strength, and learning rate schedule constants decay and Alinear decay. For Adam6p, set & and & zero.
(00) problem specified random initialization
(S1)
m0 =0
(S2)
(0) =0
(S3)
(0) d
( (60) f2 11 + 111))(0011)
(S4)
døl
m0 -mm + (1)
(S5)
(1) = 120 + (1(0))(1 (1) 32)
(S6)
(0 ml)
(S7)
8541
u(1)
6(0)
(S8)
Bt
mt)
(1)
(S9)
V Vot +
lincar max (1 -Alincar_deeee 0)
(4)
(S10)
(e) exx
expp +.eexp_dccas docay)
(S11)
(((+)) 1)
-LS Iibcar 'exp (9))
s(1)
(S12)
D.2 NADAMW UPDATE EQUATIONS
This optimizer family has 10 hyper parameters. The base learning rate, Oiases first and second moment momentum, 31. 82, the numerical stability term, f2WD & regularization strength, A AdamW style weight decay, and boolean to switch between NAdam and Adam, base nesterw- The learning rate schedule is based off of single cycle cosine decay with warmup. It is controlled by additional parameters Cwarmups Gromstant and Cmin learning rale mult.
The learning rate defined by:
Cwarmup T >> L
(S13)
Oldecay Rcconstant
= =(abase Cmin learning rate mult) (0.5
(S14)
cos tT (T Coonstant) )) + 0.5)+ +
(S15)
Cmin learning rate mult
(S16)
Qwwarmup
(
('Carmupp)
(S17)
a = (1 u decayyeamommmm + 440laarm
(S18)
The update equations of NAdamW are quite similar to that of Adam8p. For clarity we list the full update here.
22

---

Under review as a conference paper at ICLR 2021
problem specified random initialization
(S19)
m (0) =0 [0 =0
(S20)
(S21)
((4) d
d (f)) + 2
(S22)
m (4) m (1-1) + g (0 B1)
(S23)
(1) = 11 + (g(1)) 32)
(S24)
in (t)
(S25)
-5
6(0)
(S26)
(9(
(e)
2 heavy ball Vi(t) +E
(S27)
1, + - (1-13)g (0)
nesterow
(t)
Vi(()= +
(S28)
(((+1)
=010] (1 buse besterow )a ) heavy ball
(S29)
buse nesteroy Orll nesterov 0l22 Ademm (1))
(S30)
E OPTIMIZER FAMILY SEARCH SPACES
E. SEARCH SPACE CONSIDERATIONS
The performance of random search critically depends on the boundaries the original search space. Without prior knowledge about the problems, however, picking a good search space is difficult. To explore this we additionally choose search spaces after collecting and looking at the data. We then use this search space to simulate random search within the constraints via rejection sampling. To find these search spaces we find the best hyper parameters for each task and construct new hyperparameter ranges with min and max values determined by the smallest and largest values of each hyperparameter which were the best hyperparameter for some task. This removes regions of the search space not used by any task. We also tested bounds based on the 5th and 95th percentile of best performing hyperparameters computed over all tasks. In the case of min and max, we find the optimal hyperparameters cover nearly all of the existing space, whereas the percentile based search spaces reduces the volume of the search hypercube by more than 90% leaving us with only ~ 100 hyperparameter configurations. In Figure 3] we find, in all cases, learning the hyperparameter list is much more efficient.
E.2 ADAM8P, ADAM6P, ADAM4P, ADAMLR SEARCH SPACES
For Adamlp, Adam4p, Adamóp, and Adam8p sample learning rate logritmically between 1e-8 and 10, betal and beta2 we parametrize as 1 - v and sample logrithmically between le-4 and and 1c-6 and respectively. For learning rate schedules we sample linear decay between 1e-7, 1c-4 logrithmically and exponential decay logrithmically between 1e-3, 1e-6. We sample both e and lz logrithmcally between 1e-8, lel.
E.3 NADAMW SEARCH SPACE
This search space was chosen heuristically an effort to generalize to new problems. We would like to emphasize that it not tuned. We used our insight from Adam based optimizer families and
23

---

Under review as a conference paper at ICLR 2021
chose this. No iterations where done. We expect more iterations will improve not only in distribution performance, but alsos generalization performance.
The initial learning rate, Obass is sampled from log space between le - and 1.0. 1 - 81 is sampled
logrithmically between le 3. and 1.0. 1 - B2 is sampled between 1e and 1.0. is sampled
logarithmically between 1e 8 and 1e4. We sample using nesterov (buse nesterov) 50% of the time. We sample f2WD D and (2Adam logrithmically between 1e- - 5 and 1e - 1. Equal probabilities of a third we either use both terms, zero out l2W D. or zero out L2AdomW. With 50% probability we use nonzero min learning rate multiplier sampled logrithmically between 1e 5 and 1.0. With 50% probability we sample the warm up fraction, Cwarmup between le-5 and le-1, otherwise it is set to zero Finally, we uniformly sample the amount of time the learning rate is held constant(Ccomstann between 0 and 1.
F EXTENDED RELATED WORK
F.1 SETS OF TASKS
Benchmarks consisting of multiple tasks are becoming increasingly common technique for
measuring improvement in algorithm design. Reinforcement learning has Atari Bellemare et al. (2013), DMLab DMLabbBeattie et al. (2016), gym [Brockman et al. (2016), and dm_control Tassa et al. (2018) 8). Natural language processing has evaluation sets such GLUE (Wang et al., 2018). Super GLUE (Wang et al.
2019), and the NLPDecathalon (McCann et al. 2018). In computer vision there is Zhai ct al. 2019 which studies transfer learning of image features. In black box optimization there is
Nevergrad Rapin & Teytaud 2018), COmparing Continuous Optimizers (COCO) (Hansen et al. 2016) and a number of tasks to test Bayesian hyperparameter optimization presented in (Dewancker 2016). For first order gradient methods there are unit tests for stochastic optimization (Schaul ct al.
et al.
2013) which studies toy optimization functions, and DeepObs (Schneider al., 2019) which includes 20 neural network tasks. Hyperparameter tuning practices these benchmarks vary between tuning on each task separately, to tuning one set of hyperparameters for all problems. In Atari (Bellemare et al. 2013), for example, it is common practice to tune hyperparameters on a subset of tasks and evaluate on the full set. This protocol can further be extended by leveraging unseen levels or games at test time as done in Obstacle Tower (Juliani et al. 2019), ProcGen (Cobbe et al., 2019). CoinRun (Cobbe et al. 2018). and Sonic (Nichol et al. 2018). We believe generalization to unseen tasks is key for learned algorithms to be useful thus our learned search space experiments mirror this setting by making use of hold out tasks.
Existing meta-learning data sets share similar goals to our work but focus on different domains. In few shot learning there is MinilmageNet (Vinyals et al. 2016) which built procedurally from the ImageNet dataset (Russakovsky et al. 2015). Meta-Dataset (Triantafillou et al., 2019) takes this further and also focuses on generalization by constructing few shot learning tasks using images from a number of different domains for evaluation purposes. The automated machine learning community has OpenML ( (Vanschoren et al. (013) with a focus on selecting and tuning non-neural algorithms. For learning optimizers, the use of task suites has been limited and ad-hoc. Many works use single or small number of standard machine learning tasks (Andrychowicz et al. 2016 Li & Malik 2017] Lv et al., 2017; Metz et al. 2019a). Wichrowska et al. (2017) uses a set of synthetic problems meant emulate many different kinds of loss surfaces. While existing collections of tasks exist for optimizer evaluation, e.g. (Schneider et al. 2019), they contain too small number of tasks to act as a comprehensive training set for learning algorithms, and many of their tasks are additionally too computationally expensive to useful during learning.
F.2 HAND DESIGNED AND LEARNED OPTIMIZERS
Optimization is core to machine learning and thus the focus of extensive work. Methods such as Nesterov momentum Nesterov| 1983,, AdaGrad (Duchi al. 2011), RMSProp (Tieleman & Hinton 2012), and Adam (Kingma & Ba 2014) have all shown considerable improvements in both the speed of optimization and ease of use by exposing robust, and easier to tune hyperparameters than SGD (Sivaprasad et al., 2019). Adaptive step size methods in particular have emerged at the forefront with
24

---

Under review as a conference paper at ICLR 2021
many works building from it including AdamW (Loshchiloy & Hutter 2017). RAdam (Lu et al. 2019), Novograd (Ginsburg et al. 2019 and NAdam Dozat (016). Recently, there has been a focus on comparing optimizers either for best performance, or ease of use (Wilson et al. 2017] Choi et al.)
2019 Schneider et al. 2019; Sivaprasad al., 2019). This has proven difficult performance is heavily dependent on the choice of search space for optimization hyperparameters (Choi et al. 2019).
Learned optimizers represent a parallel thread in the development optimizers. By learning as opposed to dand-nesggn optimizers, researchers hope not only increase performance but also ease of use (c.g. minimize the number of hyperparameters required or lower hyperparameter sensitivity) (Bengio et al. 1990 Schmidhuber 1995 Hochreiter et al. 2001). Recently, there has been renewed
interest in parameterizating learning algorithms with neural networks and learning these optimizers
neural network based losses (Andrychowicz al. 2016 Wichrowska et al. 2017 Li & Malik
2017 Lv et al. 2017, Metz et al.) 2019a b). Other approaches make learn symbolic parameterizations for new optimizers [Bello et al., 2017). These various methods are all trained and evaluated on different distributions of tasks making comparison across papers challenging, The dataset of tasks presented here will hopefully aid in the ability to compare and evaluate progress in learned optimizer
research.
In this work, we develop a much more minimal type of "learned optimizer" than previous work which developed new functional forms for the optimizer. Optimization involves not only the functional form of the optimizer, but also the rules for choosing hyperparameters and applying the optimizer. We focus on this second aspect of optimization and learn a hyperparameter search space to improve the performance of existing hand designed methods.
F.3 HYPERPARAMETER SEARCH
Hyperparameter search a key component in machine learning. Considerable improvements have been made in language Melis al. (2017). computer vision (Snock al. 2012), and RL (Chen et al. 2018) simply by tuning better. Often no single hyperparameter configuration works well across all tasks for existing optimization methods. Most current hyperparameter search methods involve trying very large number of hyperparameters for every new task, which is computationally infeasible for large tasks, and additionally can severely limit the number of hyperparameters that can be tuned. Many common techniques such as random search (Bergstra & Bengio [002 Bousquet et al. (017). Bayesian optimization (Snoek et al. 2012) 2015). tree parzen estimators (Bergstra et al.) (001). or sequential halving (Kumar et al. 2018) require setting hyperparameter search space by hand which is not only difficult but often wildly inefficient.
Learning hyperparameters or search strategies by leveraging multiple tasks has been explored within the context of Bayesian optimization (Swersky et al. (2013) Perrone & Shen (2019) y Perrone al. (2018) as well as under the term meta-learning in Chen et al. (2017) in which an LSTM is meta-trained to produce function locations to query.
The cost of hyperparameter search is often large each evaluation requires training model to completion. Often multi-fidelity based approaches are used which leverage "simpler" tasks and transfer the resulting hyperparameters (Hutter et al. (001)). Common approaches include training on partial function evaluations Swersky et al. (2014); Domhan et al.) (2015): Li et al. (2016); Klein et al.) (2016); Falkner et al. (2018), or leveraging simplified data and models (Petrak 2000 Zoph & Le 2016 Brock et al.) 2017). Our dataset of tasks serves as a: "simpler" set of tasks to train on; large and diverse enough set of problems that optimization algorithms trained on it may be expected to generalize; and framework to transfer across different types of problems.
25

---

Under review as a conference paper at ICLR 2021
G
LIST OF NADAM HPARAMS
ldx
Lr
warmup
constant
Min LR mult
betal
beta2
epsilon
nesterov
reg
12 weight decay
0
1.24e-3
0.000
0.477
1.0e-3
0.94666
0.94067
8.114e-8
False
0.000e+00
7.258e-5
1
5.33e-3
0,000
0.172
0.0
0.96047
0,99922
8.665e-8
True
0.000c+00
5.563e-3
2
2.12c-4
0.000
0.210
1.39c-3
0.62297
0.97278
1.540e-7
False
0.000c+00
5.361e-2
3
4.06e-1
0.000
0.324
0.0
0.99724
0.98680
1.099+002
True
0.000c+00
1,562c-2
4
2.05e-2
0.000
0.885
1.57e-5
0.35731
0.86043
8.874e-5
True
0.000e+00
7.217e-2
5
5.95e-4
0.008
0.378
0.0
0.89130
0.99983
1.483e-7
True
0.000c+00
4.087e-2
6
7.53e-3
0.000
0.422
9.55e-4
0.69192
0.98434
3.593e-8
False
0.000e+00
3.060e-4
7
4.69c-3
0,000
0,509
0.0
0,99639
0,98820
2.056e-5
False
0,000c+00
3,552e-2
8
2.95e-1
0.000
0.201
0.0
0.99678
0.99981
7.498c+00
False
3.792c-4
3.463c-4
9
2.04c-3
0,000
0.527
0.0
0.49995
0,99755
5.630c-8
True
0,000c+00
2.796c-2
10
7.39c-1
0.001
0.556
3.31c-3
0.99691
0.80639
2.900c+03
False
0.000c+00
7.851c-2
8.12c-3 11
0.000
0.207
0.0
0.17785
0.96033
7.971c-2
False
0.000c+00
1,489-22
12
3.33e-2
0.000
0.369
0.0
0.69592
0.99997
5.510c-6
True
0.000c+00
1.362e-5
13
6.95e-3
0.000
0.014
0.0
0.99412
0,99305
4.352e-7
False
0,000e+00
3,142e-5
14
1.88e-1
0.000
0.205
1.08e11
0.98597
0.56531
3.335e+00
True
1.265e-5
3.868e-3
15
9,47e-4
0,007
0.452
0.0
0.43977
0,09422
2.120e-7
False
0,000e+00
6.902e-3
16 3.75e-3
0.000
0.184
0.0
0.87756
0.96128
3.163e-3
True
7.468e-5
2.627e-3
17
7.25e-1
0,000
0.495
0.0
0.99800
0,99781
3,608c+000
True
1.656e-5
3.911e-2
18
4.58c-3
0.000
0.107
3.66e-1
0.42294
0.99963
4.174c-6
True
0.000c+00
4.446c-3
19
3.07e-4
0.007
0.518
0.0
0.57863
0.99625
9.881e-6
False
0,000e+00
5.521e-2
20
2.94e-5
0.000
0.830
8.27e-5
0.96916
0.99896
7.782e-7
True
3.364e-4
3.466-33
21
1.65e-4
0.002
0.457
2.70e-1
0.95280
0.04565
2.832e-6
True
0,000e+00
1.14le-2
22
9.17e-1
0.010
0.897
2.67e-2
0.45061
0.99244
4.945e-1
False
1.253e-3
0.000c+00
23
2.36e-3
0,000
0.986
0.0
0.98560
0,99997
1.080-88
True
0,000e+00
3,023e-3
24
2.14e-2
0.000
0.128
0.0
0.98741
0.99336
1.266e-4
False
0,000e+00
5.194e-4
25
5.91e-2
0,000
0.062
0.0
0.99794
0.99383
3.447c+02
True
0.000c+00
3.935e-2
26
1.57e-3
0.000
0.251
0.0
0.91820
0.99991
4.675e-5
False
0.000e+00
4.112e-5
27
4.43e-1
0,000
0.702
0.0
0.94375
0.93551
2.335e-8
True
0,000e+00
8.325e-5
28
2.98e-3
0.008
0.046
0.0
0.68612
0.94232
6.614e-2
False
6.489e-5
0.000e+00
29
1.65e-2
0.004
0.082
4.92e-4
0.95717
0.99789
3.068e+01
True
0,000e+00
8.920e-2
30
5.58e-3
0.000
0.538
0.0
0.97559
0.99990
3.238e-8
True
0.000e+00
4.896e-4
31
8.54c-1
0,000
0.229
0.0
0.93129
0,50200
2.051e-2
False
2.068c-4
2.801c-2
32
7.38e-3
0.000
0.722
8.78e-2
0.21456
0.99752
2.862e-2
False
0.000c+00
8.439c-2
33
4.26e-4
0.001
0.923
2.06c-1
0.47239
0.99974
8.221e-5
False
1.248e-5
0.000c+00
34
6.04e-3
0.000
0.698
0.0
0.97849
0.91449
1.806c400
False
3.183e-3
1.762e-2
35
8.86e-3
0,000
0.104
1.66e-1
0.98967
0.99720
1.493e-2
True
0.000e+00
2.253e-2
36
1.51e-2
0.000
0.431
1.99e-3
0.80488
0.97878
2.538c-8
True
0.000e+00
2.269e-5
37
2.50e-3
0.000
0.009
0.0
0,98127
0.99988
1.799c-7
False
0.000c+00
1.303e-2
38
3,42c-4
0.000
0.827
6.38c-1
0.25217
0.96572
2.928c-7
True
0.000c+00
1.318c-3
39
6.94c-5
0.000
0.085
0.0
0.98674
0.42709
2.387c-7
False
0.000c+00
2.071c-4
40
3.03e-2
0.001
0.313
0.0
0.90610
0.99997
4.449e-3
True
0.000c+00
2.813c-5
41
4.64c-3
0.000
0.495
2.26c-5
0.64658
0.54108
3.528c-8
False
0.000c+00
2.996c-5
42
2.25e-3
0.000
0.722
0.0
0.97967
0.97518
1.488-77
True
1.812e-5
2.180c-2
43
6.66e-4
0.000
0.632
2.79e-5
0.65968
0,99997
6.848e-6
True
0.000e+00
3.130e-3
44
3.31e-3
0.000
0.146
0.0
0,90447
0,99970
6.618e-6
True
0.000e+00
2.184e-2
45
7.84e-4
0.016
0.124
0.0
0.95065
0,99685
2.141e-2
False
0.000c+00
4.024e-5
46
6.16e-3
0.016
0.623
0.0
0,98823
0,98744
1.666--6
False
0.000e+00
1.544e-2
47
3.26e-4
0.000
0.738
1.6le-4
0.78425
0,99998
3,468e-3
False
0.000c+00
4.709c-2
48
4.12c-3
0.001
0.205
0.0
0.99561
0.75382
2,390c-6
True
0.000c+00
3.631e-2
49
6.26c-1
0.000
0.932
2.52c=3
0.99401
0.83521
2.431c+00
True
0.000c+00
1.048c-2
Top 50 hyper parameters found using the NAdamW search space. We find diverse learning rates, with very little warmup used. We additionally find most good performing optimizers make use of AdamW
style weight decay. Finally, matching insight from (Choi et al., 2019), we find large values of c.
26

---

Under review as a conference paper at ICLR 2021
H DESCRIPTION OF TASKS IN TASK SUITE
In this section detail the task distribution used throughout this work. In addition to this text, Tensorflow (Abadi et al. 2016) implementation is also released at fgjihub.com/gogggrrsssrrrgggggg
researchfree//maseellasskk set
H.1 SAMPLED TASKS
H.1.1 DEFAULT SAMPLED COMPONENTS
As many of the sampled tasks are neural networks. We define common sampling routines used by all the sampled tasks.
Activation functions: We define distribution of activation functions which is sampled correspond- ing the following listing both name and weight. These are a mix standard functions (relu, tanh) to less standard (cos).
relu:
tanh: 3
cos: 1
elu:
sigmoid:
swish (Ramachandran et al.
2017) :
leaky relu (with a 0.4):
leaky relu (with a 0.2):
leaky relu (with 0.1):
Initializations:
We sample initializers according weighted distribution. Each initialization sample also optionally samples hyperparameters (e.g. for random normal initializers we sample standard deviation of the underlying distribution).
he normal (He et al.
2015 :
uniform (He et al.
2015):
glorot normal Glorot & Bengio 2010 2
glorot uniform GGloot & Bengio 2010):
orthogonal: 1. We sample the "gain", or multiplication of the orthogonal matrix logarithmi- cally between 10.1, 10).
random uniform 1.0: This is defined between [-s, s] where is sampled logarithmically
between [0.1, 10].
random normal: 1.0: The std is sampled logarithmically between (0.1, 10).
truncated normal: 1.0 1.:: The std is sampled logarithmically between 10 10).
variance scaling: 1.0: The scale is sampled logarithmically between (0.1, 10).
We define distribution over different types of RNN cores used by the sequential tasks, With equal probability we sample either vanilla RNN (Elman 1990), GRU(Chung et al.) (014). or LSTMMHochreiter & Schmidhuber 1997). For each cell we either sample shared initialization method or sample a different initialization method per parameter vector with 4:1 ratio. We sample the core hidden dimension logarithmically between [32, 128).
RNN Cores:
1.1.2 SAMPLED DATASETS
Image Datasets: We sample uniformly from the following image datasets. Each dataset additionally has sampled parameters. For datasets we make use of four data splits: train, valid-inner, valid-outer, test. Train is used to train models, valid-inner is used while training models to allow for modification
27

---

Under review as a conference paper at ICLR 2021
of the training procedure (e.g. if validation loss doesn't 'tss'ttiinn increase, drop learning rate). Valid-outer is used to select meta- parameters. Test should not be used during meta-training
For all datasets, we sample switch with low probability (10% of the time) to only use training data and thus not test generalization. This ensures that learned optimizers capable of optimizing loss as opposed to mix of optimizing and generalizing.
Mnist: Batch size is sampled logarithmically between [8, 512]. We sample the number of training images logarithmically between [1000, 55000] (LeCun 1998).
Fashion Mnist: Batch size is sampled logarithmically between [8 512].
We sample the number of (Xiao et al. 2017).
training images logarithmically between [1000, 55000]
Cifar10: Batch size is sampled logarithmically between [8. 256]. The number of training examples is sampled logarithmically [1000, 50000] (Krizhevsky et al.
2009).
Cifar100: Batch size is sampled logarithmically between [8, 256). The number of training examples 2009).
sampled logarithmically [1000, 50000] (Krizhevsky et al.
[0ood101 10x2_33 mail100332x333 deep_weeds. 32x32, sun397 sun.99_33xx32) 32x332]: These dataset take the orig-
inal set of images and resize them to 32x32 using OpenCV's (Bradski 2000) cubic interpolation. We
ignore aspect ratio for this resize. Batch size sampled logarithmically between [8, 256) (Bossard
al. 2014 Nene et al. 1996 Olsen al. 2019 Xiao al. 2010).
Imagenet32x32// / Imagenet16x16: The ImageNet 32x32 and 16x 16 dataset as created by Chrabaszcz et al. 2017). Batch size is logrithmically sampled between [8, 256).
H.1.3 TEXT CLASSIFICATION:
IMDB sentiment classification: We use text from the IMDB movie reviews dataset Maas et al.
201T) and tokenize using subwords using vocab size of 8k(Sennrich al.) 2015). We then take length s random slice from each example where is sampled logarithmically between [8, 64). These examples are then batched into a batch size logarithmically sampled between [8, 512]. We sample the
number of training examples logarithmically between [1000, 55000) and with 10% probability just training data instead of valid/ test to test pure optimization opposed generalization.
1.4 CHARACTER AND WORD LANGUAGE MODELING
For the character and word language modeling datasets we make use of the following data sources: imdb movie reviews(Maas et al. 2011). amazon product reviews (ama) using the Books, Camera, Home, and Video subset each as separate datasets, LMIB(Chelba et al. 2013), and taken from the 20190301 dump using the zh, ru, ja, hab, and en language codes. We split each article by new lines and only keep resulting examples that contain more than 5 characters. For infrastructure reasons, we only use a million articles from each language and only 200k examples to build the tokenizer.
Byte encoding: We take length random slices of each example where is sampled logarithmically between [10, 160). These examples are then batched into a batch size logarithmically sampled between [8, 512]. With probability 0.2 we restrict the number of training examples to a number logarithmically sampled between [1000, 50000). Finally, with 10% probability just use training data instead of valid/ test to test pure optimization opposed to generalization.
subword encoding: We encode the text as subwords with a vocabsize of 8k (Sennrich et al. 2015).
We then take length s random slices of each example where is sampled logarithmically between [10, 256). These examples are then batched into batch size logarithmically sampled between [8, 512]. With probability 0.2 we restrict the number of training examples to a number logarithmically sampled between [1000, 50000). Finally, with a 10% probability just use training data instead of valid test to test pure optimization opposed generalization.
28

---

Under review as a conference paper at ICLR 2021
H.2 SAMPLED TASKS
H.2.1 MLP
This task family consists of multi layer perceptron trained flattened image data. The amount of layers is sampled uniformly from [1. 6]. Layer hidden unit sizes are sampled logarithmically between [16, 128] with different number of hidden units per layer. One activation function is chosen for the whole network and is chosen as described in H.I.1 One shared initializer strategy is also sampled. The image dataset used is also sampled.
Two sampled configurations are shown below.
2
71
1,
5
"act Factivation"; "leaky_relu2",
"w_init":
7
"he_normal",
null
1,
10
"dataset": [ "sun397_32x32
12
13
"bs": 32,
"just_train": false, "num_train"; null
15
16
18
"crop_amount": 0, "flip_left_rrghht; false, "flip_up pro_up__dww"" true,
21
"do_color_aug": false,
"brightness": 9.00296489112855221,,
23
"saturation"; 0.4308521744067500,, "hue": 0.19648945665587863, "contrast"; 0. 0.0360963201339111444
26
28
"center_data". false
2
"layer_sizee sizes":
3
68,
37,
5
78
"activation": "relu",
8
"w iniiiit: [ "glorot_normal",,
null
11
},
"dataset":
13
"food 10111111222000 13odl00__2x32,, 32x32",
14
"bs": 117,
16
"just_train"; true, "num train"; mull
null 18
}
29

---

Under review as a conference paper at ICLR 2021
"center_date data":
22
H.2.2 MLP_AE
This task family consists of multi layer perceptron trained with auto encoding loss. The amount of layers is sampled uniformly from [2, 7] Layer hidden unit sizes are sampled logarithmically between [16, 128] with different number of hidden units per layer. The last layer always maps back the input dimension. The output activation function is sampled with the following weights: tanh:2, sigmoid:1, linear 1ettrerr linear: where linear linearecenter is an identity mapping. When using the linear center and tanh activation we shift the ground truth image to [-1, 1] before performing a comparison to the model's predictions. We sample the per dimension distance function used to compute loss with weights 12:2, 11:1, and the reduction function across dimensions to be either mean or sum with equal probability. A single activation function, and initializer is sampled. We train on image datasets which are also sampled.
sample configurations is shown below.
thidden_units""
73,
103,
105,
104,
76
9
"activation"; "relu",
10
"w "w_init":
"glorot__niform",
12
null
14
"dataset"; [
15
"mnist",
17
"bs": 39, "num_train"; 43753,
18
19
"num classes"; 10, "just_train": false
20
22
null
"oupput_typ"": "tanh",
25
"loss_ :ess"t "12",
26
'reduction_type:; "reduce_sun"
H.2.3
MLP VAB
This task has an encoder with sampled number of layers between [1,3]. For each layer we sample the number of hidden units logarithmically between [32, 128). For the decoder we sample the number of layers uniformly between 3] For each layer we sample the number of hidden units logarithmically between [32, 128]. We use gaussian prior of dimensionality logarithmically sampled between [32, 128]. A single activation function and initialization is chosen for the whole network. The output of the encoder is projected to both mean, and log standard deviation which parameterizes the variational distribution, (2][z). The decoder maps samples from the latent space to a quantized gaussian distribution in which we compute data log likelihoods log p(x[z). The loss optimize is the evidence lower bound (ELBO) which is computed by adding this likelihood to the kl divergence between our normal distribution prior and (( lr). We use the reparameterization trick to compute gradients. This model trained sampled image datasets.
30

---

Under review as a conference paper at ICLR 2021
A sample configuration is listsed below.
"enc_hidden_units""
3
73
}, wwwphhiddeeuuiieee
74
8
"activation"; "relu",
"w "w_init": "hc_normal",
11
null
13
"dataset": [ "food101_32x32"
14
16
"bs": 22,
17
true, "num_train"; null
19
}
null
22
H.2.4 CONV POOLING
This task consists of small convolutional neural networks with pooling. We sample the number of layers uniformly between [1, 5J. We sample a stride pattern to be either all stride 2, repeating the stride pattern of 1,2,1,2... for the total number layers, or 2,1,2, for the total number of layers. The hidden units logarithmically sampled for each layer between [8, 64). We sample one activation function and weight init for the entire network. Padding for the convolutions are sampled per layer to either be same or valid with equal probability. For the convnet we also sample whether or use bias with equal probability. At the last layer of the convnet we do reduction spatially using either the mean, max, squared mean sampled uniformly. This reduced output is fed into linear layer and a softmax cross entropy loss. These models are trained on sampled image dataset.
sample configuration shown below.
2
"strides":
[1, 11,
[2, 21.
5
[1, 11.
[2, 21,
7
[1, 1]
"hidden_units"; [
10
46,
48,
12
47.
13
29,
18
15
},
16
"activation"; "leaky_relu4",
"w "w_init": "glorot_normal",
18
mull
1,
"paccing": "SAMK""
31

---

Under review as a conference paper at ICLR 2021
"SAME",
24
"VALIC",
25
"SAME",
"VALID"
27
28
"pool_type": "squared_mean",
"use bias": true, "dataset"; [ *cifar100",
30
32
"bs": 10,
"num_train"; 5269, "just_ train"train; true
35
36
)
null
38
},
"center_dates data"; false
40
H.2.5 CONV FC
This task consists of small convolutional neural networks, flattened, then run through a MLP. We sample the number of conv layers uniformly between [1, 5). We sample stride pattern to be either all stride repeating the stride pattern of 1,2,1,2... for the total number of layers, or 2,1,2,1... for the total number of layers. The hidden units are logarithmically sampled for each layer between [8, 64]. Padding for the convolutions are sampled per layer to either same or valid with equal probability.
The output is then flattened, and through MLP with hidden layers sampled uniformly from [0, 4] and with sizes sampled logrithmically from [32, 128). The loss is then computed via softmax cross entropy.
We sample one activation function and weight init for the entire network. For the convnet also sample whether not to use bias with equal probability. These models trained on sampled image dataset.
An example configuration is shown below.
"strides":
[2, 21,
[2, 2],
[2, 21,
[2, 21
7
},
Saidden_units""
9
17,
10
30,
13,
12
16
13
},
"activation"; "relu",
15
"w_init": "glorot_uniform!
17
null
18
19
"pacaing":
20
"VALID""
"VALID"
"VALII""
23
"SAME"
32

---

Under review as a conference paper at ICLR 2021
"fc hicden units": "use_bias", true, "dataset": [ "coi1100_32x32""
26
29
30
"bs": 49, "just_ train:: false, "num_train"; null
32
null
35
" "center_data";
37
H.2.6 CHARACTER RNN LANGUAGE MODEL
This task takes character embedded data, and embeds in a size embedding vector where is sampled logarithmically between [8, 128] with random normal initializer with std 1.0. With 80% we use all 256 tokens, and with 20% chance we only consider subset of tokens sampled logarithmically [100, 256]. We then pass this embedded vector to RNN with teacher forcing with equal probability we use a trainable initializer or zeros. A linear projection is then applied to the number of vocab tokens. Losses are computed using a softmax cross entropy vector and mean across the sequence.
A sample configuration is shown below.
"embed__inn dim": 30,
3
"w_init": [ "he_normal", null
procab_sizer 256,
8
"corc": [
"gru",
11
"core_dim", 84,
"wh": "glorot_uniform", null
13
14
1,
16
"wz":
"random_norma"
0. 0.402264178800722
19
1,
"wr":
21
"he_ mee_nniforee
null
1,
24
"uh":
"he_normal",
null
27
},
"uz":
29
"glorot_normal",
30
null
1,
32
"ur": [
null
35
33

---

Under review as a conference paper at ICLR 2021
38
"trainable_init" true,
"dataset": [ "Imlb/bytes",
41
"patch_length"; 147,
bbatch_size" 63, "just_train"; false, "nun_train"; null
44
46
H.2.7 WORD RNN LANGUAGE MODEL
This task takes word embedded data, and embeds in a size s embedding vector where s sampled logarithmically between [8, 128] with random normal initializer with std 1.0. vocab size for this embedding table is sampled logarithmically between [1000, 30000). We then pass this embedded vector to a RNN with teacher forcing with equal probability we use trainable initializer or zeros. linear projection is then applied the number of vocab tokens. Losses are computed using a softmax cross entropy vector and mean across the sequence.
A sample configuration shown below.
"embed_dime 91,
"w_init": "glorot_uniform", null
), "vocab_size"" 13494,
"core": [
9
"gru",
10
"core_din"; 96, "wh":
12
"he__ow "e__normal", null
14
15
},
"wz":
17
"he_ ne__normal, null
18
1,
20
"wr":
"he_ normall null
22
23
"uh":
25
"he ne_normaa",, null
26
1,
28
"uz":
"he normal", null
30
1,
32
"ur":
33
"he_normal",
mull
36
34

---

Under review as a conference paper at ICLR 2021
"trainable init": true,
39
"dataset"; [ "tokenized amazon anazzon reviews/vie v1 Lttp:/de_vv__00uuuuorrreee _0__sbbordds'"",
42
14,
"batch_size"; 59,
"just_train" false,
45
"num_train"; null
47
48
H.2.8 LOSG PROBLEMS
These tasks consist of mixture of many other tasks. We sample uniformly over the following types of problems. We brielfy describe them here but refer reader to the provided source for more information. this work we took all the base problems from (Wichrowska al., (017)) but modified the sampling distributions better cover the space opposed to narrowly sampling particular problem families. Future work will consist of evaluating which sets of problems or which sampling decisions are required.
quadratic: n dimensional quadratic problems where is sampled logarithmically between [10, 1000]. Noise is optionally added with probability 0.5 and of the scale s where is sampled logarithmically between [0.01, 10].
bowl: A 2d qaudratic bowl problem with a sampled condition number (logrithmically between [0.01, 100)). Noise is optionally added with probability 0.5 and of the scale where is sampled logarithmically between [0.01, 10].
gparse_softmax_eegressinnn
synthetic random sparse logistic regression task.
optimization Battooo_tttt____eeeeee prsslpprobems:: A uniform sample over the following functions: Ackley, Beale, Branin, logsumexp, Matyas, Michalewicz, Rosenbrock, StyblinskiTang.
fully_ colle_coneeted: A sampled random fully connected classification neural network predicting classes synthetic data. Number of input features is sampled logrithmically between 1 and 16, with a random activation function, and sampled number of layers uniformly sampled from 2-5.
norm: A problem that finds a minimum error an arbitrary norm. Specifically: (E ()) where W NNN, y R Na1, The dimentionality, N, is sampled logrithmically between and 1000. The power, is sampled uniformly between 0.1 and 5.0. W. and are drawn from a standard normal distribution.
dependency chain: A synthetic problem where each parameter must be brought to sequentially. We sample dimensionality logrithmically between 3, 100.
outward snake: This loss creates a winding path to infinity. Step size should remain constant across this path. We sample dimensionality logrithmically between and 100.
min max max_well: A loss based on the sum of min and max over parameters: max x + 1/(min .: 2. Note that the gradient is zero for all but parameters. We sample dimentaionlity logrithmically between 10 and 1000. Noise is optionally added with probability 0.5 and of the scale where is sampled logarithmically between [0.01. 10).
sum_of of ___uudratiic: A least squares loss of a dimentionality sampled logrithmically between 3 and 100 to a synthetic dataset.
projection quadratic: A quadratic minimized by probing different directions. Dimentionality is sampled from 3 to 100 logrithmically.
In addition to these base tasks, we also provide a variety of transformations described bellow. The of these transformations also sampled.
35

---

Under review as a conference paper at ICLR 2021
sparse_ problems: With probability 0.9 to 0.99 the gradient per parameter is Additional noise is added with probability 0.5 sampled from normal with std sampled logrithmically between [0.01, 10.0].
rescale paoblees-obm:: Rescales the loss value by 0.001 to 1000.0 sampled logrithmically.
log Log_objective: Takes the log of the objective value.
Sample configurations shown below.
3
features": 16,
5
"n_classes"; 2,
"activation"; feaakyreelu"",
"bs": 7, "n ur__smplees; 12, "hidden_siee
8
32,
11
8,
5,
13
9,
14
8
]
16
},
17
36641
"outward_snake",
"cim": 9, "bs": 30, "n_samples": 249
} 79416
"rescale_problens:
3
{
"base": "sum_od "sum_of_quadiii _quddrttasadd
"dim": 36, "bs" ; 5, "n_samples"; 1498
8
11
},
"scale": 227. 000886615222 87778222229000
13
}
14
89629
H.2.9 MASKED AUTOREGRESSIVE FLOWS
Masked autoregressive flows family of tractable density generative models. See XX for more information. The MAF is defined by sequence of bijectors. For one bijector samples number of layers to either be or with equal probability, and number of hidden layers sampled logarithmically between 116, 128). We sample the number of bijector uniformly from [1, and use the same hidden
36

---

Under review as a conference paper at ICLR 2021
layers across all bijector. We sample activation function, and initializer once for the whole model. In this task we model image datasets which are also sampled.
sample configuration shown below.
"activation "relu",
"w_init";
"he_uniform",
5
null
"dataset";
8
Lmmggnet_resized/11a..
"bs": 19, "just_train"; true, "nun_train": null
11
13
14
mul1
1,
16
Wriden_unitts"
44,
18
24
"nunobi.jettor";
21
H.2.10
NON VOLUME PRESERVING FLOWS
NVP are family of tractable density generative models. See Dinh et al, (2016) for more information. The NVP is defined by a sequence of bijectors. For one bijector samples a number layers to either be with equal probability. and a number of hidden layers sampled logarithmically between [16, 128]. We sample the number of bijector uniformly from [1, 4] and use the same hidden layers across all bijector. We sample activation function, and initializer once for the whole model. In this task we model image datasets which are also sampled.
A sample configuration shown below.
"activation": "cos",
"w_init": "glorot_normal null
"dataset"; [ ssun397_32x322
"bs": 228,
11
"just_train" false, "num Traint mull }
12
14 null
16
"hidden_units/" [
17
21,
121
19
"num_ bi jectors": 4
37

---

Under review as a conference paper at ICLR 2021
H.2.11 QUADRATIC LIKE PROBLEMS
This task distribution defines a synthetic problem based on a non-linear modification to quadratic. The dimensionality of the problem is sampled logarithmically between [2, 3000].
The loss for this task is described by:
Cutput_fn(AAA)) - B)² + C)
(S31)
where X = param * weight_rescale rescale and where param is initialized by initial sampsll / weight_rescale
The output fn is sampled uniformly between identity, and ((x) log(mmx(OO, x)). The loss scale is sampled logarithmically between [10-5, 10%].
We define distribution over matrices A as a sample from one of the following: normal: we sample a mean from a normal draw with a standard deviation of 0.05 and a std from a uniform [0, 0.05]. The elements of A are drawn from the resulting distribution. uniform: linspace_ eigen: togspace_eigee
We define a distribution over B to be either normal with mean and std sampled from N(0, 1), U(0, 2) respectively or uniform with min and range equal to U(-5, 2.5), U(0, 5) respectively.
With probability 50% we add noise from distribution whose parameters also sampled.
sample configuration shown below.
"A :isiitt::
"linspace_eigen"
"min": 32.
"max": 122. 789486689666
9
"initial dist": "uni form",
10
12
"min": 2. 291 091119987833995
"max": 6./2394005/7/1417
14
15
"output_fn"" fn": "log",
17
"dims": 212,
18
"seed": 68914,
19
loss_scales 0. 0.6030061302855556
20
"noise": null
H.2.12 RNN TEXT CLASSIFICATION
This task consists of using an RNN classify tokenized text. We first trim the vocab length to be of size logarithmically sampled between [100, 10000]. The text is then embedded into vocab size logarithmically sampled between [8, 128]. These embeddings get fed into sampled config RNN. With equal probability the initial state of the rnn is either sampled, or zeros. With equal probability we either take the last RNN prediction, the mean over features, or the per feature max over the sequence. This batch of activations is then passed through a linear layer and a softmax cross entropy loss. The initialization for the linear projection is sampled.
An example configuration shown below. In this version of TaskSet the dataset sampling contains bug. All data used is from the indd_reviewwweeee wewbbrrriie/subworrss dataset.
"embed dim": 111,
38

---

Under review as a conference paper at ICLR 2021
init": "random_normal",, 0.1193048629073732
1,
7
"dataset": | httpo//wwwwreeeoooooooodddeeeeee
10
"bs": 43,
Fnum_train", null, "max_token"; 8185,
12
13
"just_train! true, "patch_length": 20
15
16
},
"vocab size": " 3570,
18
"core": [ "vrnn",
20
"hidden_to_hidden"" "he_uniform", null
23
"in_to_hidden"; [ the_uniform", null
26
28
|,
"act_fn" "leaky_relu2", "core_dim;;
31
"trainable_init" _niit false, "loss_compute" "max"
34
39

---

Under review as a conference paper at ICLR 2021
H.3 FIXED TASKS
In addition sampled tasks, we also define set of hand designed and hand specified tasks. These tasks are either more typical of what researcher would do (c.g. using default initializations) or specific architecture features such as bottlenecks in autoencoders, normalization, or dropout.
In total there are 107 fixed tasks. Each task is labeled by name with some information about the underlying task. We list all tasks, discuss groups of tasks, but will not describe each task in detail. Please see the source for exact details.
SonciattiveGRRu122 BS128 2 Associative_ GRUU26_BB1188 Pairs20 Tokens50
Associative__L.TTE L.STM128 BS128 Chirs00TToknnee
BS128 P Tokens50
Associative LSTM128 BS128 Pairs5 Tokens20
Associative LSTM256 BS128 Pairs20 Tokens50
Associative__LSTTMEEE BS128 Tooenn10Tool1000
Associative_ Associativ__VRNNI228 VRRNN288 BS128 Pairs10
Tookes50
Pairs20_Tokkee
These tasks use RNN's to perform an associative memory task. Given a vocab of tokens, and some number of pairs store and a query the RNN's goal is produce the desired value. For example given the input sequence AIB2C3 the RNN should produce
B.
This model embeds tokens, applies an RNN, and applies a linear layer to map back to the output space. Softmax cross entropy loss is used to compare outputs. A weight is also placed on the losses so that loss is incurred only when the RNN is supposed to predict. For RNN cells use LSTM (Hochreieer & Schmidhuber 1997), GRU (Chung et al. 2014), and VRNN a vanilla RNN. The previous tasks are defined with the corresponding RNN cell, number of units, batch size, sequence lengths, and number of possible tokens for the retrieval task.
Copy moy__GR128_P BS128 Length20 Tokens10
BS1/2256_S11188111 Length-0_Tokens5i Tokens50
Copy LSTM128 BS128 Length20 Tokens10
Copy LSTM128 BS128 Length20 Tokens20
Copy LOSMILS BS128 Length50_Tokeee
Copy_LSTM128 BS128
Tokens10
CSpy_LSTM226 BS128
Length40_ooooooeee Tokens50
Copy_VRNNI28 BS128 _.eggthh__ Tooeens00
Copy_VRNN256 BS128 Leee_nnennntto_eeeeeeeee Tokens50
These tasks use RNN's to perform a copy task. Given vocab of tokens and some number of tokens the RNN's job is to read the tokens and to produce the corresponding outputs. For example an input might be: ABBCI description of the task. Each task in this set varies the RNN core, well the dataset structure.
and the RNN should output JABBC. See the source for a complete
This model embeds tokens, applies an RNN, and applies a linear layer to map back to the output space. Softmax crossentropy loss used to compare outputs. A weight also placed on the losses so that loss is incurred only when the RNN supposed to predict. For RNN cells we use LSTM (Hochreiter & Schmidhuber] 1997), GRU (Chung et al.) 2014), and VRNN vanilla RNN. The previous tasks are defined with the corresponding RNN cell, number of units, batch size, sequence lengths, and number of possible tokens.
FieedImageConvAEE cifar10 32x32x33x3322222 bs128
FixedImageConvAAB AAT.://AAA...........2222626.... bs128
32x64488x64x32
FixellmageConn AnF__mmmmmimnm22x222222233 128
FixedInageConn mnist 32x64x32xx4x322
hatpa//ooooooommmmmmmmmmdnddaeed bs512
40

---

Under review as a conference paper at ICLR 2021
FixedImageConnAE mnist 32x64x8x64x32 bs128
Convolutional autoencoders trained on different datasets and with different architectures (sizes of hidden units).
FixedImageConv VAE cifar10 32204x122Bxx441128xxxx3333 bs128
FixedImageCon VAE cifar10 52x661128xx44x13xx44322 bs5512222
bs512
FixedImageConv VAE__ VAE_cifar10 wwfferl0_3xxx411111
32x64x122xx64322_bbbl
FixedlmageCony VAE_ _se__fffll11 644x128x255x112885566 b8128
FixedImageCon' VAE mnist 32x32x32xx2xx32 bs128
FixedImageCom VAE mnist 32x64x32x64x32 bs128
FixedImageCony VAE, mnist_66x1225555 1as____xxx2xxx221228222220 bs128
Convolutional variational autoencoders trained on different datasets, batch sizes, and with different architectures.
FixedlmageCony
TixellmasCCov__cirr11111333
FC64x32 tanh__ta______ variance_scalingg bs64
_3060___22xxx4____1 flatten bs128
FiedImaggConm,_fffr10000..
21111100____3xxx112333 28x128 bs128
FDe___C64332___ _s_
TxcdlmaggConn,ccirr10222222 32110___xxxxxx2_88_ faatttllaa_______ flatten_FC64x32
FC64x32_taam
Lart__taaaaavaieeana__e variance_scaling bs64 _s64
FixedlmageConv_cffrtll
_...ffrr10033xx4x1222 32x64x128 he bs64
2Gffar10_22x6xx223 largenormal bs64
FixedImageConv.coo
normal bs64
1964
SonelmmaaCooycfffrllll
32x64x128 322a4xi22_smallnormmaal bs64
__v____a_____ bs64
FiceliaggComm..itaa1000222444 bs128
fc____________888____
TxcdlmageComm,ccfaa100.2222.2 flatten bs128
FieellnageCom..cfffr10122222222 tanh bs64
FxedImageCCnn_ccar1100... batchnorm 32x32x32x64x64
1s1288
CatTarr0,batchnorm batchnorm 32x64x64 bs128
AxcdllaggConnyccoil103332222 bn 32x64x128x125 bs128
32x64x64 flatten bs128
TicellngeConnnfooollldd444L_LL_ BttpllxxxoCoov_32222 321___32xxxx64____ 11164x66_flaatte____ bs64
FixedimageConnffood100.. Baaptto0__bathhhrmm..om
bs128
FixedImageCon
2ttp://gggeCCoonnuunn3333222222 Ss__unn397322322___ 2n 220p2/222__n333xx11222828888 32x64x128x122_bbss
Mnist Cony 32x16x64 flatten
FC32_tanh ba32
Convolutional neural networks doing supervised classification. These models vary in dataset, archi- tecture, and initializations.
FixedLM MaadLLM_mlhh__ 2asel__ptchh28._111 GRU128 hatpi/2__mmeee___a___aaaa bs128
FixedLM Imlb patch128 GRU256 GRU256 s_eeemee6___aag bs128
FixedLM Imlb patch128 GRU64 embed64 avg bs128
FixedLM Imlb Bacllbppatii28__11 LSTM128 embed64 avg bs128
FixedLM Imlb ballbbptthh122111111 LSTM256 40we2555eebee6444__11 avg bs128
Language modeling tasks on different RNN cell types and sizes.
FiXedMAF aalfaaa 10_aa1 bs64
FiXedMAF _acisst____t__ 2aae__llae__r______ bs64
FiXedMAF faasttt3laye_thhaa saar___tt___e___ee
Masked auto regressive flows models with different architectures (number of layers and sizes).
41

---

Under review as a conference paper at ICLR 2021
FixedMLPAE ciffar10 128x32x128 bs128
FixedMLPAE mnist 128x32x128 bs128 FixedMIPAE, 11____mi__33__ 32x32x32 bs128
Autoencoder models based on multi layer perceptron with different number of hidden layers and dataset.
FixedMLPVAE, 10 128x1288322128x1225 21 bs128
FixedMLPVAE 1tt.-/a1___..111111111111 _______xxx2xx___2hh_
FixedMLPVAE food10132x32 128x64x32x64x112 bs64
FixedMLPVAE mnist 128x128x8x128 bs128
FixedMLPVAE
manir. 2ttIs__128xxxxx22xxx222222 bs64
FixedMLPVAE_ (is_mnntt__222_ 128x8x128x128 bs128
SF__e__33XXX0___________ 128x64x32x64x1222 1112__relu__s__ bs256
Variational autoencoder models built from multi layer perceptron with different datasets, batchsizes, and architectures.
TixedMLP_cifart BatchNoorm_112x12282222
rel__bsu11 bs128
TixedMIP_cifar/// BatchNorm 64x64x64x64x64
relu_eeluueeeeeee bs128
TixeMMIP_cifarll 2reti_DrroooO0__1111 12tx___22xxx2282 Beaiix12__reuu______1_1 relu_bs12g
FixedMI.P_cifaaret Sr00__rrrp0055111111 1ropout055128x1253 1022__eel_____ relu_bs128
FiXedMLP_ciiie ci Dropout08 128x128 relu bs128
1ropout08_128x1288
1 FixedMLP eifar10 LayerNorm 128x128x122 relu bs128
FiXedMLP_cirn _iif__iill LayerNorm 128x128x128 tanh bs128
TixedMLP_cifarlll ce ce_128x12xx128 relu relu_bs128
FixedMLP_cifar100 _a___mse_______ 12x__xxxxxxxx88 Leruul2____uu_______
FixedMMLP_foo1111332333
128x128xxxxxxxx 28 @elu_bs128
FixcdMI.__.ood1013333222222 mannse_2xx12___0 28 1is_rrlu_bss220
FixedMI._mmnist 128x128x128 1
relu_bs122
FiXedMLP Saaie__mis__mi _as____s________________________ 128x128x122 relu bs128
LxxdNVP_mmile 2layer
bs64 4564
Image classification based multi layer perceptron. We vary architecture, data, batchsize, normal- ization techniques, dropout, and loss type across problems.
FiXedNVP _______ns_____ 3layer thin bs64
FiXedNVP mnist 5layer bs64
FiXedNVP mnist _laa__lly___ta_ t_ti_thin______ bs64
BoaaNVP_mnisse Slayer_thin_bee bs16
Non volume preserving flow models with different batchsizesm and architectures.
LSTM128 avg bs64
ShceTisiKKhNCCaasffffcoooooiinnd LSTM128 bs64
2ttnb__patch12___SSM1122 LSTM1188.... e0eeemmee222 bs64
S GRU128 bs128
GRU64 ________________________________ _s18_s1288
FieeTTexIRNNNCLaasfffcttioon imdb indb_patch322 IRNN64 relu avg bs128
FixeITexIRNNNCLaasiftttttoo imdb 2acbb__atchh2 IRNN64 relu last bs128
FixeeTlesRRNNCCaassiiccttoo imdb Baabb_patch32 LSTM128 E128 bs128
Fixed/ExxIANNCCasstiitttiinnnnn Lmcpsm__mmmm_____ Sundb_patch32_LSS) LSTM128 bs128
TixeITextIRNNCCaassticacaaoo Barraa__iddd_dd____ VRNNI28 tanh_ bs128
inddb_paich322_RRNNN2
MANN64_relu__0 relu_avg_bsI122 bs128
CxeedTeeRRNNClasfffccttioo
htttffcatttoiimd___athch222 han TRNN64_tanh__0 bs128
42

---

Under review as a conference paper at ICLR 2021
RNN text classification problems with different RNN cell, sizes, embedding sizes, and batchsize.
TwoD_ Bowll
TwoD Bowl10
TwoD Bowl100
TwoD Bowl1000
2D quadratic bowls with different condition numbers.
TwoD Rosenbrock
TwoD Sann
TwoD Ackley
TwoD Beale
Toy 2D test functions.
43


Reviews:
[{"review_id": "nt4G4HrUQ2Q", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work presents TaskSet, a collection of optimization tasks consisting of different combinations of data, loss function, and network architecture. The tasks are useful when choosing and evaluating different optimizers (e.g. ADAM) for learning tasks. The usefulness of this collection is demonstrated for a hyperparameter search problem. \n\nThe main question I had about this work is why is the chosen collection the right set of tasks to be considering? Do I have any assurance that an optimizer chosen using TaskSet will be any good on future tasks? How can we know that we don't overfit to these particular tasks when choosing an optimizer? Is there any notion of two tasks being drawn from the same distribution? \n\nHowever, my main concern with this paper is that, while TaskSet may be a useful tool for facilitating future research, it is not clear to me that it itself represents an advancement of novel research, which I think should be the bar for acceptance to a major conference. The work does not make any claims, or present any results beyond a use-case for the set of tasks. That's not to say that TaskSet isn't a useful tool, helpful for future research. But it itself does not represent such research. Because of this I recommend the work be rejected. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "fvXjgBAo_L6", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \n\nThis paper proposes a dataset of tasks to help evaluate learned optimizers. The learned optimizers are evaluated by the loss that they achieve on held-out tasks after 10k steps. Using this dataset, the main strategy considered is to use search spaces that parametrize optimizers and learn a list of hyperparameter configurations for the optimizer that are tried sequentially. The authors show that the learned hyperparameter configuration list learned achieves better performance than (constrained) random search on multiple optimizer search spaces. Finally, they show that the learned hyperparameter list transfer well to realistic problems such as training a ResNet-50 model on ImageNet and training a transformer architecture on LM1B, outperforming reasonable baselines.\n\nPros:\n\n+ Creating a dataset of tasks for learning optimizers is a interesting and useful goal. While there have been some sets of tasks used in the learned optimizers literature, there isn't a standard dataset for this task. A large number of tasks is proposed.\n+ The hyperparameter list trained compares favorably with random search across the other tasks. The experiments are interesting overall and show some insights about the performance of the learned list with increasing number of tasks. \n\nCons:\n\n- While the goal of finding a good dataset of tasks for learned optimizers is a worthy, I find that the paper does not adequately discuss and explore the choices that went into creating this dataset. Namely, how were tasks picked? What code implementations were used? What are some limitations of the current dataset that could be addressed in future research? How are the tasks represented? How can a researcher use this dataset of tasks to explore new algorithms? Most of the value of proposing a new benchmark or dataset is explaining the choices that went into creating it and packaging well so that other researchers can use it easily. I think that this could be better realized in the paper. For example, is future research based on this dataset meant to be done offline (on the optimization curves collected) or online (by running additional configurations for these search spaces)? How are new methods to be benchmarked through this dataset? How are existing datasets for this missing important aspects? This is not adequately defined. Given this, I think that the paper could do a better job setting the stage for future research building on this dataset. \n\n- The focus on the dataset of tasks is poorly realized in the paper, which is devoted in great part to how an ordered list of 1000 hyperparameter configurations for a learned optimizer performs well in comparison with other random search baselines. While this was well executed overall, it is not initially the focus of the paper. I believe that more value for the community could be derived by focusing on the creation of the dataset rather than on the introduction of a new heuristic.\n\nTypo: . on We take the TPU... ==> . We take the TPU...", "extra_scores": {}, "extra_text": {}}, {"review_id": "f_lYl-vwbB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new dataset which contains experiment / model details coupled with optimizer information so as to model the behavior of optimizer, and their effect on performance on test set. The paper is not very difficult to follow, but I am not super convinced of an actual practical use cases. \n\nI think that the authors should provide a concrete examples for real life test time applications. I suppose the meta-learning algorithm for the optimizer would take the experiment definition, and map this information to an optimal optimizer, but I think that it would be easier for the reader if this information could be made mode explicit in the paper, perhaps with a concrete example. \n\nI also think that the comparison between the proposed meta learning approach, vs. the regular hyperparameter search on a given dataset should be made clearer. Right now it is limited to figure 3, and in my opinion the details on how the random search is carried out is not clear enough. What is the range of hyper parameters that are sampled? What are the distributions from which the hyperparameters come from? \n\nAlso, it is hard to make the conclusion only from the experiments provided in Figure 3 that, the proposed meta-learning approach would be preferable over the standard hyperparameter search just by the two tasks explored in this particular figure. Ideally a third dimension of tasks should also be added to the figures so that we know that this meta learning approach generalizes over a variety of tasks. (The same comments apply to figure 4, if I understand correctly, which does similar experiments on more realistic models/tasks)\n\nIf I am missing something that is already in the paper, I apologize, but without further experimental evidence which suggests that the proposed meta learning scheme would be clearly preferable over standard hyperparameter search, it is hard to see a clear-cut application for this paper. \n\nI appreciate the ambitious task that this paper is trying to tackle, but I feel more convincing experimental evidence, and better presentation of the experiments is required to consolidate the case that this paper is trying to make. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "WxJE4jpsP3n", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a suite of deep learning focused optimization problems that would facilitate the development of learned optimizers. This is very useful and can streamline research in learned optimizers while providing a benchmarking suite that can be used for training as well as evaluation. In my opinion, this is very valuable.\n\nHowever, the presentation of the task suite and then the subsequent application could use significant clarification. For example, it was not clear to me until I made multiple passes that the application presented in the paper is about meta-learning hyper-parameter initializations for deep learning optimizers by levaraging the suite to generate the meta-learning data set with various task executions. Moreover, given that the main advantage, to the best of my understanding, of the proposed TaskSet is in learning optimizers, the choice of tasks in the paper lack proper qualitative or quantitative justification. As mentioned in the paper, it is useful to have a wide set of tasks for better generalization of learned optimizers, too broad a set could hinder the meta-learning. In that case, it is not clearly discussed (at least in the main paper) why these set of tasks are selected and why/how they lead to better learned optimizers. If the suite consists of all combinations of deep learning architectures, applications and data sets without any meaningful discussion of why it is better for learning optimizers (or even for meta-learning hyper-parameter initializations) than just learning optimizers separately on each task (or even task class), it reduces the possible usefulness of the suite. \n\n\nGiven the motivation and the utility of the suite, I am leaning towards an accept. However, the lack of proper justification for the choices (beyond just covering a laundry list of deep learning architectures, data sets and applications) to the best of my understanding leaves me partially unsatisfied.\n\n\nBeyond the above, I have a few minor comments:\n\n\n- Subsection 1.2 is very confusing. Why are we talking about hyperparameters if the taskset consists of first order optimization problems? It should better (and more explicitly) presented as a potential application of the TaskSet suite for meta-learning hyper-parameter initializations for some optimizers.\n- Figure 1b is confusing to me. I'm unable to wrap my head around what each line means and how the x axis ordered and why the fraction is always increasing? Moreover, given the nonconvex and randomized nature of many of the discussed tasks, how is it decided that an optimizer achieves a particular loss -- the optimizer might reach different level with different restarts.\n- This paper shows an application of TaskSet for meta-learning hyper-parameters of specific optimization algorithms. I am not sure, in the context of HPO, why one would use the TaskSet given something like the Bayesmark framework [A]. It has HPO tasks with associated data with ability to add new tasks/data, and the task set covers a wider class of methods. \n\n\n[A] bayesmark.readthedocs.io\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The contributions of this paper are twofold: 1) datasets of tasks are provided, and 2) based on the datasets and hyperparameter lists on the datasets, a transfer learning approach for hyperparameter optimization (HPO) is proposed. Many reviewers positively evaluated the idea and approach discussed in this paper. However, the common concern of multiple reviewers and area chair is that it is not clear whether the provided datasets and their hyperparameter lists are generally applicable to other practical problems. Since there is no discussion on how the datasets are constructed, it is not clear whether they have generally or not. In addition, the comparison with existing HPO approaches is not sufficiently made, and it is not clear whether the performance of the proposed method is advantageous over existing methods. Overall, although the idea is very interesting and potentially useful, I cannot recommend the acceptance in its current form due to the lack of evidence on its generality. 
Decision: Reject