Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: 

This paper proposes a dataset of tasks to help evaluate learned optimizers. The learned optimizers are evaluated by the loss that they achieve on held-out tasks after 10k steps. Using this dataset, the main strategy considered is to use search spaces that parametrize optimizers and learn a list of hyperparameter configurations for the optimizer that are tried sequentially. The authors show that the learned hyperparameter configuration list learned achieves better performance than (constrained) random search on multiple optimizer search spaces. Finally, they show that the learned hyperparameter list transfer well to realistic problems such as training a ResNet-50 model on ImageNet and training a transformer architecture on LM1B, outperforming reasonable baselines.

Pros:

+ Creating a dataset of tasks for learning optimizers is a interesting and useful goal. While there have been some sets of tasks used in the learned optimizers literature, there isn't a standard dataset for this task. A large number of tasks is proposed.
+ The hyperparameter list trained compares favorably with random search across the other tasks. The experiments are interesting overall and show some insights about the performance of the learned list with increasing number of tasks. 

Cons:

- While the goal of finding a good dataset of tasks for learned optimizers is a worthy, I find that the paper does not adequately discuss and explore the choices that went into creating this dataset. Namely, how were tasks picked? What code implementations were used? What are some limitations of the current dataset that could be addressed in future research? How are the tasks represented? How can a researcher use this dataset of tasks to explore new algorithms? Most of the value of proposing a new benchmark or dataset is explaining the choices that went into creating it and packaging well so that other researchers can use it easily. I think that this could be better realized in the paper. For example, is future research based on this dataset meant to be done offline (on the optimization curves collected) or online (by running additional configurations for these search spaces)? How are new methods to be benchmarked through this dataset? How are existing datasets for this missing important aspects? This is not adequately defined. Given this, I think that the paper could do a better job setting the stage for future research building on this dataset. 

- The focus on the dataset of tasks is poorly realized in the paper, which is devoted in great part to how an ordered list of 1000 hyperparameter configurations for a learned optimizer performs well in comparison with other random search baselines. While this was well executed overall, it is not initially the focus of the paper. I believe that more value for the community could be derived by focusing on the creation of the dataset rather than on the introduction of a new heuristic.

Typo: . on We take the TPU... ==> . We take the TPU...
review_title: A dataset of optimization tasks is an interesting contribution, but lacking depth in the discussion of its construction and intended use 

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a new dataset which contains experiment / model details coupled with optimizer information so as to model the behavior of optimizer, and their effect on performance on test set. The paper is not very difficult to follow, but I am not super convinced of an actual practical use cases. 

I think that the authors should provide a concrete examples for real life test time applications. I suppose the meta-learning algorithm for the optimizer would take the experiment definition, and map this information to an optimal optimizer, but I think that it would be easier for the reader if this information could be made mode explicit in the paper, perhaps with a concrete example. 

I also think that the comparison between the proposed meta learning approach, vs. the regular hyperparameter search on a given dataset should be made clearer. Right now it is limited to figure 3, and in my opinion the details on how the random search is carried out is not clear enough. What is the range of hyper parameters that are sampled? What are the distributions from which the hyperparameters come from? 

Also, it is hard to make the conclusion only from the experiments provided in Figure 3 that, the proposed meta-learning approach would be preferable over the standard hyperparameter search just by the two tasks explored in this particular figure. Ideally a third dimension of tasks should also be added to the figures so that we know that this meta learning approach generalizes over a variety of tasks. (The same comments apply to figure 4, if I understand correctly, which does similar experiments on more realistic models/tasks)

If I am missing something that is already in the paper, I apologize, but without further experimental evidence which suggests that the proposed meta learning scheme would be clearly preferable over standard hyperparameter search, it is hard to see a clear-cut application for this paper. 

I appreciate the ambitious task that this paper is trying to tackle, but I feel more convincing experimental evidence, and better presentation of the experiments is required to consolidate the case that this paper is trying to make. 


review_title: Better comparison with regular hyperparameter search is required 

final_score: rating: 7: Good paper, accept
review_content: The paper presents a suite of deep learning focused optimization problems that would facilitate the development of learned optimizers. This is very useful and can streamline research in learned optimizers while providing a benchmarking suite that can be used for training as well as evaluation. In my opinion, this is very valuable.

However, the presentation of the task suite and then the subsequent application could use significant clarification. For example, it was not clear to me until I made multiple passes that the application presented in the paper is about meta-learning hyper-parameter initializations for deep learning optimizers by levaraging the suite to generate the meta-learning data set with various task executions. Moreover, given that the main advantage, to the best of my understanding, of the proposed TaskSet is in learning optimizers, the choice of tasks in the paper lack proper qualitative or quantitative justification. As mentioned in the paper, it is useful to have a wide set of tasks for better generalization of learned optimizers, too broad a set could hinder the meta-learning. In that case, it is not clearly discussed (at least in the main paper) why these set of tasks are selected and why/how they lead to better learned optimizers. If the suite consists of all combinations of deep learning architectures, applications and data sets without any meaningful discussion of why it is better for learning optimizers (or even for meta-learning hyper-parameter initializations) than just learning optimizers separately on each task (or even task class), it reduces the possible usefulness of the suite. 


Given the motivation and the utility of the suite, I am leaning towards an accept. However, the lack of proper justification for the choices (beyond just covering a laundry list of deep learning architectures, data sets and applications) to the best of my understanding leaves me partially unsatisfied.


Beyond the above, I have a few minor comments:


- Subsection 1.2 is very confusing. Why are we talking about hyperparameters if the taskset consists of first order optimization problems? It should better (and more explicitly) presented as a potential application of the TaskSet suite for meta-learning hyper-parameter initializations for some optimizers.
- Figure 1b is confusing to me. I'm unable to wrap my head around what each line means and how the x axis ordered and why the fraction is always increasing? Moreover, given the nonconvex and randomized nature of many of the discussed tasks, how is it decided that an optimizer achieves a particular loss -- the optimizer might reach different level with different restarts.
- This paper shows an application of TaskSet for meta-learning hyper-parameters of specific optimization algorithms. I am not sure, in the context of HPO, why one would use the TaskSet given something like the Bayesmark framework [A]. It has HPO tasks with associated data with ability to add new tasks/data, and the task set covers a wider class of methods. 


[A] bayesmark.readthedocs.io

review_title: Significantly important contribution for developing learned optimizers; justification for task choices and presented application somewhat limited

final_score: rating: 3: Clear rejection
review_content: This work presents TaskSet, a collection of optimization tasks consisting of different combinations of data, loss function, and network architecture. The tasks are useful when choosing and evaluating different optimizers (e.g. ADAM) for learning tasks. The usefulness of this collection is demonstrated for a hyperparameter search problem. 

The main question I had about this work is why is the chosen collection the right set of tasks to be considering? Do I have any assurance that an optimizer chosen using TaskSet will be any good on future tasks? How can we know that we don't overfit to these particular tasks when choosing an optimizer? Is there any notion of two tasks being drawn from the same distribution? 

However, my main concern with this paper is that, while TaskSet may be a useful tool for facilitating future research, it is not clear to me that it itself represents an advancement of novel research, which I think should be the bar for acceptance to a major conference. The work does not make any claims, or present any results beyond a use-case for the set of tasks. That's not to say that TaskSet isn't a useful tool, helpful for future research. But it itself does not represent such research. Because of this I recommend the work be rejected. 

review_title: Collection of tasks

Meta-review: The contributions of this paper are twofold: 1) datasets of tasks are provided, and 2) based on the datasets and hyperparameter lists on the datasets, a transfer learning approach for hyperparameter optimization (HPO) is proposed. Many reviewers positively evaluated the idea and approach discussed in this paper. However, the common concern of multiple reviewers and area chair is that it is not clear whether the provided datasets and their hyperparameter lists are generally applicable to other practical problems. Since there is no discussion on how the datasets are constructed, it is not clear whether they have generally or not. In addition, the comparison with existing HPO approaches is not sufficiently made, and it is not clear whether the performance of the proposed method is advantageous over existing methods. Overall, although the idea is very interesting and potentially useful, I cannot recommend the acceptance in its current form due to the lack of evidence on its generality. 
Decision: Reject