Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper considers the problem of automatic early termination in hyper-parameter search and neural architecture search.  The authors propose to form this problem as learning curve ranking and transfer learning. Unlike most previous approaches for learning curve prediction, which estimates the probability of whether the current model is better than the current best model or not by first extrapolating the learning curve and then invoke a heuristic measure, this paper proposes to predict the probability directly. The pairwise comparison probability is modeled as the logistic function of the difference of a scoring function f, where f is modeled as a neural network with the learning curve data as the input, and constructed with three components, including a learning curve component, an architecture component and a data set component. The neural network f is trained with some meta dataset, and then the early termination is then decided based on this pairwise comparison probability. The paper applied the proposed early termination approach to the neural architecture search of five image classification data sets, and evaluated the performance in terms of the Spearman correlation of the learning curve ranking and the regret and time for the architecture search. It also analyzes the learning curve prediction characteristics through a few concrete examples, followed by some ablation analysis.

The proposed approach is novel to my knowledge, and the numerical performance is also satisfying and convincing. However, there are a few issues that this paper should better improve upon. 

Firstly, the methodology description at the beginning of section 3 is not clear enough. In particular, the motivation of choosing logistic function in (1) is not explained, and how the calculation of p_{i,j} from the final learning curves is done is also not elaborated. Some of the terminologies are also not clearly defined or specified. For example, what does "posteriors" mean (cf. the line before (2))? What is the meta-knowledge \mathcal{D}? Is it the one used to train the neural network of f? There is also a small typo in line 2 of Algorithm 1, where d should be D probably. 

Secondly, although the numerical experiments are convincing within the framework of learning curves prediction, they are not sufficiently convincing when it comes to the scope of the neural architecture search (NAS) or hyper-parameter optimization (HPO). Comparisons with state-of-the-art NAS and HPO algorithms that do not use learning curves prediction (e.g., HyperBand, learning-by-learning algorithms using LBFGS, etc.) are not mentioned or compared with. The authors may either want to add comparisons with those algorithms, or provide some more applications of learning curve prediction to showcase the flexibility of the proposed approach.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: Learning curve prediction methods try to predict the final performance of a candidate model before it gets fully trained. In this way, learning curve prediction can act as a fast method for performance measurements in AutoML. Compared with previous approaches, the proposed method allows for transferring useful knowledge among different datasets.

While the proposed method is simple, the authors have shown promising results. However, I have some concerns about the motivations and usage of the proposed method. Please see the questions below:

Q1. Learning curve prediction is a general approach that can be combined with many zero-order optimization methods. Does this paper focus on learning curve prediction on general AutoML problems or just NAS?
- If the authors want to target general AutoML problems, they should perform experiments with (1) more datasets (come from various domains) and consider more (2) search algorithms. 
- For (1), the authors can follow Auto-sklearn and Auto-weka, or some experiments on graph (e.g., GCN) or some experiments on language modeling. For (2), they can combine the proposed approach with Bayesian optimization or genetic algorithms. CNN has nice transfer learning ability, to show the wide applicability of the proposed approach is important.
- If authors want to target at NAS, then the proposed method is not useful. Parameter sharing is a better method in NAS for fast performance evaluation. The authors need to compare with some recent NAS papers in CV (e.g., DARTS).

Q2. "Learning curve component". Learning curves are naturally a sequence of data points. The proposed method simply does a maximum pooling over all positions, which ignores the sequence of data. Could the authors give some explanations? Is it better to carry an ablation study on this point?

Q3. "Architecture component". How will the changes in the search space affect the proposed method? 

Q4. Could the authors random draw many architectures from "NASNet search space" and then plot their learning curve? It is better to show what kinds of curves will the proposed method likely to give early stops.

Q5. Another main pitfall of the proposed method cannot offer a probabilistic estimation, which can be done by Baker et al. (2018). Since the model is early stopped, it is naturally that there are some uncertainty in the estimated ranks.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: 
The paper proposes a new method to rank learning curves of neural networks which can be used to speed up neural architecture search.
Compared to previous work, the learning curve model not only takes hyperparameter configurations into account, but by training it on offline generated data, it is able to model learning curves across different datasets.
Applied to a simple neural architecture search strategy, the proposed method achieves higher speed-ups on image classification tasks than other methods from the literature.

While the method seems interesting, I don't think the paper is ready for acceptance yet, since it a) misses some important details and b) the empirical evaluation is not sufficient.
More precisely the following points need to be addressed:

 - In section 3, the paper says that all automated methods follow broadly the same principal that the first model is trained to completion. This is not correct, commonly used methods, such as Hyperband (Li et al.) or BOHB (Falkner et al.), use successive halving (Jamieson et al.) which trains a batch of configurations for just a minimum budget and then already discards poorly performing configurations. I also miss a discussion of these methods in the related work section.

Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization
Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., and Talwalkar, A.
Journal of Machine Learning Research 2018

BOHB: Robust and efficient hyperparameter optimization at scale
S Falkner, A Klein, F Hutter
Proceedings of the 35th International Conference on Machine Learning

Non-stochastic best arm identification and hyperparameter optimization
K Jamieson, A Talwalkar
Artificial Intelligence and Statistics, 240-248


 - Is the learning curve model updated during the optimization process with the new observed data? If not, is there any way the model can adapt to new data? Also, what happens if learning curves are fundamentally different to the training data, e.g if different learning rate schedules are used than for generating the offline data?

 - A simple baseline that is missing, is to use the last observed value as approximation for the final performance which often works competitively to more powerful learning curve extrapolation methods (e.g see Klein et al.).

 - The method is only applied for a simple random search strategy, however, in practice, one would use more sophisticated methods, such as for example Bayesian optimization. The question is, how effective is the proposed method in this setting, since the distribution of learning curves might be dramatically different and biased towards more well-performing configurations with almost identical learning curves.

- I think the experiments would be more convincing, if the method shows strong performance when deploy in commonly used NAS/HPO methods, such as Hyperband or successive halving. This should be straight forward, since decisions which configuration is promoted to a higher budget can be made based on the model instead of just the last observed value.

- What is delta in the experiments? How sensitive is this hyperparameter in practice?


Minor comments:

- Related to this paper is the work by Gargiani et al. which also models learning curves across datasets on offline generated data
 Probabilistic Rollouts for Learning Curve Extrapolation Across Hyperparameter Settings
 M Gargiani, A Klein, S Falkner, F Hutter
 arXiv preprint arXiv:1910.04522

- Figure 5: visually it seems that all learning curves are almost identical, maybe it would be better if the plot could zoom in at least for the final stage of training.



after the rebuttal
-----------------------

I thank the authors for answering my  questions. In the rebuttal the authors addressed my concerned about the insufficient empirical evaluation and included other relevant baselines. I will increase my score.
review_title: Official Blind Review #1

Meta-review: Authors propose a new way of early stopping for neural architecture search. In contrast to making keep or kill decisions based on extrapolating the learning curves then making decisions between alternatives, this work learns a model on pairwise comparisons between learning curves directly. Reviewers were concerned with over-claiming of novelty since the original version of this paper overlooked significant hyperparameter tuning works. In a revision, additional experiments were performed using some of the suggested methods but reviewers remained skeptical that the empirical experiments provided enough justification that this work was ready for prime time.   
Decision: Reject