Learning to Rank Learning Curves
Authors: Martin Wistuba
Tejaswini Pedapati
Venue:  (2020)

Abstract: Many automated machine learning methods, such as those for hyperparameter and neural architecture optimization, are computationally expensive because they involve training many different model configurations. In this work, we present a new method that saves computational budget by terminating poor configurations early on in the training. In contrast to existing methods, we consider this task as a ranking and transfer learning problem. We qualitatively show that by optimizing a pairwise ranking loss and leveraging learning curves from other data sets, our model is able to effectively rank learning curves without having to observe many or very long learning curves. We further demonstrate that our method can be used to accelerate a neural architecture search by a factor of up to 100 without a significant performance degradation of the discovered architecture. In further experiments we analyze the quality of ranking, the influence of different model components as well as the predictive behavior of the model.

Under review as a conference paper at ICLR 2020
LEARNING TO RANK LEARNING CURVES
Anonymous authors Paper under double-blind review
ABSTRACT
Many automated machine learning methods, such as those for hyperparameter and neural architecture optimization, computationally expensive because they involve training many different model configurations, In this work, we present a new method that saves computational budget by terminating poor configurations early on in the training. In contrast to existing methods, we consider this task as a ranking and transfer learning problem. We qualitatively show that by optimizing a pairwise ranking loss and leveraging learning curves from other data sets, our model is able to effectively rank learning curves without having to observe many very long learning curves. We further demonstrate that our method can be used to accelerate a neural architecture search by a factor of up to 100 without significant performance degradation of the discovered architecture. In further experiments we analyze the quality of ranking, the influence of different model components well the predictive behavior of the model.
INTRODUCTION
A method commonly used by human experts to speed up the optimization of neural architectures or hyperparameters is the early termination of iterative training processes that are unlikely to im- prove the current solution. A common technique to determine the likelihood of no improvement is to compare the learning curve of a new configuration to the one of the currently best configuration. This idea can also be used to speed up automated machine learning processes. For this purpose, it is common practice to extrapolate the partial learning curve in order to predict the final performance of the currently investigated model. Current extrapolation techniques have several weaknesses that make them unable to realize their full potential in practice. Many of the methods require sufficient sample learning curves make reliable predictions (Chandrashekaran & Lane| 2017 Klein al., 2017 Baker et al. 2018). Thus, the extrapolation method for the first candidates can not be used yet, which means more computational effort. Other methods do not have this disadvantage, but require sufficiently long learning curves to make reliable predictions which again means unnecessary over- head (Domhan al. 2015). Many of these methods also do not take into account other information such as the hyperparameters of the model being examined or its network architecture.
We address the need for sample learning curves by devising a transfer learning technique that uses learning curves from other problems. Since the range of accuracy varies from data set to data set, we are forced to consider this in our modeling. But since we are not interested in predicting the performance of model anyway, use ranking model that models the probability that the model currently being investigated surpasses the best solution so far. This does not only solve the problem but also provides better modeling of the actual task. In order to be able to make reliable predictions for short learning curves, we consider further characteristics of the model such as its network architecture. We compare our ranking method with respect to a ranking measure against different methods on five different image classification data sets. We also show that our method is capable of significantly accelerating a neural architecture search. Furthermore, we conduct several ablation studies to provide a better motivation of our model and its behavior.
RELATED WORK
Most of the prior work for learning curve prediction is based on the idea of extrapolating the partial learning curve by using combination of continuously increasing basic functions.

---

Under review as a conference paper at ICLR 2020
Domhan et al. (2015) define set of 11 parametric basic functions, estimate their parameters and combine them in an ensemble. Klein et al. (2017) propose a heteroscedastic Bayesian model which learns weighted average of the basic functions. Chandrashekaran & Lane (2017) do not use basic functions but use previously observed learning curves of the current data set. An affine transforma- tion for each previously seen learning curve is estimated by minimizing the mean squared error with respect to the partial learning curve. The best fitting extrapolations are averaged as the final predic- tion. Baker et al. (2018) use different procedure. They use support vector machines as sequential regressive models to predict the final accuracy based on features extracted from the learning curves, its gradients, and the neural architecture itself.
The predictor by Domhan et al. (2015) is able to forecast without seeing any learning curve before but requires observing more epochs for accurate predictions. The model by [handdraheearan & Lane (2017) requires seeing few learning curves to extrapolate future learning curves. However, accurate forecasts are already possible after few epochs. Algorithms proposed by byleii et al. al. (2017);
Baker et al. (2018) need to observe many full-length learning curves before providing any useful forecasts. However, this is prohibiting in the scenarios where learning is time-consuming such as in
large convolutional neural networks.
All previous methods for automatically terminating iterative learning processes are based on meth- ods that predict the learning curve. Ultimately, however, we less interested in the exact learning curve but rather whether the current learning curve leads to a better result. This way of obtaining ranking is referred to as pointwise ranking methods (Liu 201T). They have proven to be less efficient than pairwise ranking methods which directly optimize for the objective function (Burges) al. 2005). Yet, we are the first to consider a pairwise ranking loss for this application.
There are some bandit-based methods which leverage early termination as well. Successive Halv- ing (Jamieson & Talwalkar 2016) is a method that trains multiple models with settings chosen at random simultancously and terminates the worst performing half in predefined intervals. Hyper- band (Li et al.] 2017) identifies that the choice of the intervals is vital and therefore proposes to run Successive Halving with different intervals. BOHB (Falkner et al. 2018) is an extension of Hyperband which proposes to replace the random selection of settings with Bayesian optimization.
The use of methods that terminate less promising iterative training processes early are of particular interest in the field of automated machine learning, as they can, for example, significantly accel- erate hyperparameter optimization. The most computationally intensive subproblem of automated machine learning is Neural Architecture Search (Zoph & Le, 2017). the optimization of the neu- ral network topology. Our method is not limited to this problem, but as it is currently of the major challenges in automated machine learning, we use this problem as a sample application in evaluation. For this particular application, optimization methods that leverage parameter shar- ing (Pham et al., 2018) have become established as a standard method. Here, a search space span- ning, overparametrised neural network is learned and finally used to determine the best architecture DARTS (Liu et al. 2019)) is one of the last extensions of this idea which uses a continuous relaxation of the search space. which allows learning of the shared parameters as well as of the architecture by means of gradient descent. This method can be a strong alternative to early termination methods but does not transfer to general machine learning or arbitrary architecture search spaces.
3 LEARNING CURVE RANKING
With learning curve we refer to the function of qualitative performance with growing number of iterations of an iterative learning algorithm. We use the fena learning curve to explicitly denote the entire learning curve, .i...., Vt., reflecting the training process from beginning to end. Here, yi is measure of the performance of the model (e.g.. classification accuracy), which is determined at regular intervals. Contrary, partial learning curve, ......, Ut, refers to learning curves that are observed only up to a time I. We visualize the concepts of the terms in Figure |
There is a set of automated early termination methods which follow broadly the same principle. The first model is trained to completion and is considered as the current best model m max with its performance being max For each further model m, the probability that it is better than the current best model, p (m, > max), is monitored at periodic intervals during training. If it is below a given threshold, the model's training is terminated (see Algorithm I All existing methods rely on a two-
2

---

Under review as a conference paper at ICLR 2020
Final Performance
Validation Accuracy
Intermediate Performance
Iterations
Figure 1: Learning curve prediction tries to predict from the partial learning curve (solid line) the final performance.
step approach to determine this probability which involves extrapolating the partial learning curve and a heuristic measure. Instead the two-step process determine the probability, we propose LCRankNet to predict the probability that model m, is better than my directly. LCRankNet is based on neural network f which considers model characteristics and the partial learning curve X; as input. We define the probability that m, is better than my as
(xx))_((x))
pmm, m,) P..j 1 + (x))))((3)))
Using the logistic function is established modelling approach in the learning-to- -torinng-t-rankk commu- nity (Burges et al. 2005). Given a set of final learning curves for some models, the estimation of the posterior values pr, between these models is trivial since know whether m, is better than my or not. Therefore, we set
m; > mj
Pi.j =
0.5
if m, m
0
if my m.
We minimize the cross-entropy loss
Σ
-Ps Pi. log (1 Pt.j) log(1 - -pis)
(3)
to determine the parameters of f. Now probability p(m; m2) can be predicted for arbitrary pairs m, and my using Equation IT.
Algorithm Early Termination Method
Input: Data set d, model m. performance of best model max far max
Output: Learning curve.
for / +- 11..1..... do
2: Train m on d for a step and observe further part of the learning curve
3: if maxigigi y/ max then
4: continue
5: else if (m > m max S then
6: return
7: return y
3.1 RANKING MODEL TO LEARN ACROSS DATA SETS
We have defined the prediction of our model in Equation T It depends the outcome of the function if which takes a model representation x as input. The information contained in this rep- resentation depends on the task at hand. Since will consider Neural Architecture Search, the representation consists of three different parts. First, the partial learning curve y1..... V/- Second, the description of the model's architecture which is sequence of strings representing the layers it is
3

---

Under review as a conference paper at ICLR 2020
f(x)
Fully Connected Layer
max pool
max pool
max pool
LSTM
Embedding
conv k=2
conv k=3
cony k=4
Embedding
Architecture
y //,............ 1/
Data Set ID
Figure 2: LCRankNet has three different components, each dealing with one type of input: partial learning curve, architecture encoding and data ID.
comprised Third, a data set ID to indicate on which data set the corresponding architecture and learning curve was trained and observed.
We model f using neural network and use special layers to process the different parts of the representation. The learned representation for the different parts are finally concatenated and fed to a fully connected layer. The architecture visualized in Figure 2 We will now describe the different components in detail.
Learning curve component convolutional neural network used to process the partial learning curve. We consider up four different convolutional layers with different kernel sizes. The exact number depends on the length of the learning curve since we consider no kernel sizes larger than the learning curve length. Each convolution is followed by global max pooling layer and their outputs are concatenated. This results in the learned representation for the learning curve.
Architecture component For our learning curve prediction we use architectures of the popular NASNet search space (Zoph et al. 2018). We use the most common configuration which consists of two cells and every cell of five blocks. This results into 40 choices to be made which is encoded as sequence of integers. Please refer to Zoph et al. (2018) for further details regarding the search space. We learn an embedding for every choice. An LSTM takes this embedding and generates the architecture embedding.
Data set component As we intend to learn from learning curves of other data sets, we include data embeddings used whenever a learning curve observed on this data set is selected. If the data set is new, then the corresponding embedding initialized at random. As the model observes more learning curves from this data set, it improves the data set embedding. The embedding helps us to model data-set-speciie peculiarities in order to adjust the ranking if necessary.
of the components. Every data has its embedding and it will be
Technical details During the development process, we found that the architecture component leads to instabilities during training. To avoid this, we regularize the output of the LSTM layer by using it as an input to autoregressive model, similar to sequence-to-sequence model (Sutskever) et al. 2014) that recreates the original description of the architecture. In addition, we use the atten- tion mechanism (Bahdanau et al. 2015) to facilitate this process. All parameters of the layers in are trained jointly by means of Adam (Kingma & Ba 2015) by minimizing a weighted linear com- bination of the ranking loss (Equation (3)) and the reconstruction loss with respect to its parameters.
4

---

Under review as a conference paper at ICLR 2020
Spaarmn Correlation
Speerman Correlaion
CIFAR-100
Spearman Correlation
Fashion-Mi
0.50
0.75
0.25
0.50
0.00
0.25
8.0
0.00
20
30
10
20
30
10
20
Learning Curve
Learning
Seen Leariing Curve
Spearment Correlation
Quick, Draw!
Spearmant Correlation
SVHN
0.75
0.50
Baker al.
Chandrashekaran
0.50
0.25
Domhan al.
0.25
0.00
Last Seen Value LCRankNet
0.00
-0.25
10
20
30
10
30
Seen Learning Curve 1%)
Seen Learning Durva 196)
Figure 3: The x-axis indicates length observed of the learning curve. We report the mean of ten repetitions. The shaded area is the standard deviation. Our method LCRankNet outperforms its competitors on all data sets.
The hyperaarameter å allows for trading precision vs. recall search time vs. regret, respectively. There are two extreme case: if we set 5 > every run is terminated immediately. If we set 5 s O we never terminate any run. For our experiment we set 0.45 which means that if the predicted probability that the new model is better than the best one is below 45%, the run is terminated early.
4 EXPERIMENTS
In this section, first discuss how to create the meta-knowledge and then analyze model in terms of learning curve ranking and the ability to use as way to accelerate Neural Architecture Search. Finally, examine its individual components and behavior in certain scenarios.
4.1 META-KNOWLEDGE
We compare our method similar methods on five different data sets: CIFAR-IO, CIFAR-100, Fashion-MNIST, Quickdraw, and SVHN, We use the original train/test splits if available. Quickdraw has total of 50 million data points and 345 classes. To reduce the training time, select subset of this data set. We use 100 different randomly selected classes and choose 300 examples per class for the training split and 100 per class for the test split. 5,000 random data points of the training data set serve as validation split for all data sets.
To create the knowledeowe, we choose 200 architectures per data set random from the NASNet search space (Zoph et al. 2018) such that we train total of 1,000 architectures. We would like to point out that these are 1,000 unique architectures. there is no architecture that has been trained on several different data sets. Each architecture is trained for 100 epochs with stochastic gradient descent and cosine learning rate schedule without restart (Loshchilov Hutter 2017).
We use standard image preprocessing and augmentation: for every image, first subtract the channel mean and then divide by the channel standard deviation. Images are padded by margin of four pixels and randomly cropped back to the original dimension. For all data sets but SVHN apply random horizontal flipping. Additionally, we use Cutout (DeVsies Taylor 2017).
The following experiments conducted in wnweee--o--dtt--ettee cross-validation. That means when considering one data set, all meta- knowleee but the one for this particular data set is used as meta-knowledge.
5

---

Under review as a conference paper at ICLR 2020
Table 1: Results obtained by the different methods five different data sets. For both metrics the smaller, the better. Regret reported in percent, time in GPU hours.
Method
CIFAR-10
CIFAR-100
Fashion Regret
Regret
Time
Regret
Time
Time
No Early Termination
0.00
1023
0.00
1021
0.00
1218
Domhan et al. (2015)
0.56
346
0.82
326
0.00
460
Li et al. 20177)
0.22
106
0.78
102
0.32
132
Baker al. (2018
0.00
89
0.00
77
0.00
129
Jamicson & Talwalkar 2016
0,62
62
0.00
54
0.18
70
Chandrashekaran Lane (0077)
0.62
30
0.00
35
0.28
41
LCRankNet
0.22
20
0.00
11
0.10
19
Method
Quickdraw
SVHN
Regret
Time
Regret
Time
No Early Termination
0.00
1045
0.00
1485
Domhan et al. (0155)
0.44
331
0.28
471
Li cl al. (0017)
0.54
109
0.00
156
Baker al. (2018
0.00
107
0.00
241
Jamieson & Talwalkar 2016
0.40
60
0.28
88
Chandrashekaran Lane 2017)
0.30
0.06
164
LCRankNet
0.00
28
0.10
74
4.2 RANKING PERFORMANCE
First, we analyze the quality of the learning curve rankings by different learning curve prediction methods. In this experiment choose 50 different learning curves at random random learning curves are used as a training set for every repetition, Each learning curve predic- tion method ranks the 50 architectures by observing the partial learning curve whose length varies from 0 to 30 percent of the final learning curve. We repeat the following experiment ten times and report the mean and standard deviation of the correlation between the true and the predicted rank- ing in Figure 3 As correlation metric, the correlation is for a perfect ranking and 0 for an uncorrelated, random ranking. Our method LCRankNet shows for all data sets better performance. If there are learning curves available, our method shows the biggest difference to the existing methods. The rea- son for this is combination of the consideration of the network architecture together with additional meta-knowledge. We analyze the impact of each component in detail in Section 4.5
set. Five
Spearman's 's rank correlation coefficient. Thus,
only very short partial
The method of Chandrashekarann & Lane (017) consistently shows the second best results and in some cases can catch up to the results of our method. The method of Baker et al. (2018) stands due the high standard deviation. It is by far the method with the smallest squared error on test. However, the smallest changes in the prediction lead to a significantly different ranking, which explains the high variance in their results. The method of Domhan al. (2015) requires minimum length of the learning curve to make predictions. Accordingly, we observe rank correlation values starting from a learning curve length 4%. Using the last seen value to determine the ranking of learning curves is simple yet efficient method (Klein et al. 2017). In fact, it is able to outperform some the more elaborate methods.
4.3 ACCELERATING RANDOM NEURAL ARCHITECTURE SEARCH
In this experiment, demonstrate the utility of learning predictors in the search for net- work architectures. For the sake of simplicity we accelerate a random search the NASNet search space (Zoph et al. 2018).
The random search samples 200 models and trains each them for 100 epochs to obtain the final accuracy. In the end, the best of all these models is returned. Now each learning curve predictor

---

Under review as a conference paper at ICLR 2020
0.35
Validation Error
0.30 8
150
0.25
0.2
0.20
100
0.15
0.1
0.10
50
0.05
20
100
0
Learning Curve Length (%6
20 40 60 100
Learning Curve Length (%)
Figure 4: LCRankNet speeding up architecture search for SVHN. Difference between learning curves is small, making this the hardest task.
iterates over these sampled architectures the same order and determines at every third epoch the training should be aborted. For Successive Halving (Jamieson & Talwalkar (016) and Hyper- band (Li et al.] (017)) we follow the algorithm defined by the authors and use the recommended settings. The current best model discovered after iterating over all 200 architectures is returned as the best model for this learning curve predictor. The goal the methods to minimize the regret compared to random search without early stopping and at the same time reduce the computational effort.
One of our observations is that Domhan et al. (2015)'s method underestimates performance when the learning curve is short. As a result, the method ends each training process early after only a follow the recommendation of Domhan et al. (2015) and do not end
few epochs. Therefore, processes before we have seen the first 30 epochs. We summarize the results in Table [ Our first observation is that all methods accelerate the search with little regret. Here we define regret as the difference between the accuracy of the model found by the random search and the accuracy of the model found by one of the methods. Not surprisingly. Domhan et (2015) 's method takes up most of the time, as it requires significantly longer learning curves to make its decision. In addition, we can confirm the results of Baker et al. (201)); [Chandrashekaran (0 better results than Domhan et al. (2015). Our method requires the least amount of time for each data set. For CIFAR-100 we do not observe any regret, but a reduction of the time by a factor 100. In some cases observe an insignificantly higher regret than some of the other methods. In our opinion, the time saved makes up for it.
& Lane (2017). both of which report
In Figure A visualize the random search for SVHN. As you see, many curves not only have similar behavior but also similar accuracy. For this reason, it is difficult to decide whether to discard model safely, which explains the increased runtime. The only methods that do not show this behavior are Successive Halving and Hyperband. The simple reason is that the number of runs terminated and the time they are discarded are fixed by the algorithm and do not rely on the properties of the learning curve. The disadvantages are obvious: promising runs may be terminated and unpromising runs may run longer than needed,
Finally, we compare our method with DARTS (Liu et al. 2019). method that accelerates the search by using the parameter sharing. We train the architecture discovered in the previous experiment with an equivalent training setup like DARTS for fair comparison. This architecture reached 2.99% classification error on CIFAR-10 10 after only 20 GPU hours were searched. By comparison, DARTS (1st order) requires 36 GPU hours for similarly good architecture (3.00% error). DARTS (2nd order) can improve these results 2.76%, but requires 96 GPU hours.
4.4 LCRANKNET PREDICTION ANALYSIS
We saw in the previous selection that LCRankNet does not perform perfectly. There are cases where its regret is greater than where the search time is higher which indicates that early stopping

---

Under review as a conference paper at ICLR 2020
Validation Accuracy
0.8
0.6
0.4
m
Validation
pím m))))
0.2-
0.0
75
100
50
75
100
Learning Curve Length (%)
Learning Curve Length (%)
Validation Accuracy
Validation Accuracy
0.8
0.6
0.4
0.4
0.2
0.2
0.0
25
50
75
100
25
50
75
100
Learning
Length (%)
Learning
Length
Figure 5: Analysis of the predicted probability of LCRankNet with growing learning curve length. The plots in the top row show examples for correct decisions by LCRankNet. The bottom row shows two examples where it's decision was wrong.
was applied later than maybe possible. In this section we try to shed some light on the decisions made by the model and try to give the reader some insight of the model's behavior.
We provide four example decisions of LCRankNet in Figure 5 The plots in the top row show cases where LCRankNet made correct decision, the plots the bottom row are examples for incorrect decisions.
In both of the correct cases (top row), LCRankNet assigns a higher probability to begin with, using meta-knowledge. However, in the top left case it becomes evident after only very few epochs that m max is consistently better than m such that the probability p (m > m max) reduces sharply to values close to zero which would correctly stop training early on. The case in the top right plot is more tricky. The probability is increasing until learning curve length 12% as the learning curve seem to indicate that is better than mux However, the learning curve of muu approaches the values of m and then the probability decreases and training is stopped early.
We continue now with the discussion of the two examples in the bottom false terminated early which cause a regret higher than zero in Table [ In the bottom left we show example where sometimes is better for longer period and sometimes max This is arguable a very difficult problem and it is hard to predict which curve will eventually be better. However, LCRankNet shows a reasonable behavior. As the learning curve of m consistently worse than m in the segment from 15% to 42%, the probability is decreasing. Starting from a length of 57%, where shows superior performance than max for several epochs, the probability starts to raise. From learning curve length of 70% onwards, both the learning curves are very close and the difference in the final accuracy is only 0.0022. The example visualized in the bottom right plot is very interesting one. The learning curve of m max is consistently better than or almost equal to m up to the very end. Towards the end, learning curves are getting very close. In fact, from learning curve length 63% onwards, the maximum difference between and mux per epoch is only 0.008. Hence, the beginning it seems like a trivial decision to reject m. However, eventually m turns out to be better than m mux
which LCRankNet
In conclusion, deciding whether one model will be better than another one based on partial learning curve is challenging task. A model that turns out to be (slightly) better than another can be dominated consistently for most of the learning curve. This makes not only a very challenging problem for automated methods but for human experts well.
8

---

Under review as a conference paper at ICLR 2020
Spearmnn Corelaaion
1.0
CIFAR-10
Spearman Correlalion
CIFAR-100
Spearman Correlation
Fashion-
1.0
0.4
0.5
0.0
0.0
-0.2
20
30
10
20
10
20
30
Learning Curve
Leariin Curve
Seen Learning Curve
Quick, Draw!
SVHN
Correaatoon
Spearman Corretation
Proposed Configuration
0.75
0.50
Architecture only
0.50
0.25
Proposed Configuration
Proposed Configuration w/o Metadala
0.25
0.00
Learning Curve only
0.00
-0.25
Learning only
-0.25
0
30
20
30
10
20
30
Seen Learning Curve (%)
Seen Learning Curve (%)
Figure 6: Analysis of the different components of LCRankNet. Every single component, metadata, consideration of the learning curve and architecture description, vital.
4.5 ANALYSIS OF LCRANKNET'S COMPONENTS
We briefly mentioned before which components of our learning curve ranker have an essential in- fluence on its quality. We would like deepen the analysis at this point and compare the configu- ration we have proposed with different variants in Figure We consider variants with and without metadata, architecture description or learning curve consideration.
addition, we compare our configuration trained with pairwise ranking loss to one trained with pointwise ranking loss.
One of the most striking observations that metadata is essential for model. This is not surprising since in particular the learning of architecture embedding needs sufficient data. Sufficient data is not available in this setup, so we observe a much higher variance for these variants. Even the variant that only considers the learning curve benefits from additional meta-knowledge. But even this is not surprising, since stagnating learning processes show similar behavior regardless of the data set. Using the meta--nnoledde, both components achieve good results their own. It can be clearly seen that these components are orthogonal to one another. The variant, which only considers the architecture, shows very good results for short learning curves. If only the learning curve is considered, the results are not very good at first, but improve significantly with the length of the learning curve. combination of both methods ultimately leads to our method and further improves the results. Finally, we compare the use of pointwise ranking loss (L2 loss) versus pairwise ranking loss. Although our assumption was that the latter would have to fundamentally better, since it optimizes the model parameters directly for the task at hand, in practice this does not necessarily seem be the case. Especially for short learning curves, the simpler method achieves better results. However, once the learning curve is long enough, the pairwise ranking loss pays off.
5 CONCLUSION
In this paper we present LCRankNet, a method to automatically terminate unpromising model con- figurations early. The two main novelties of the underlying model are that it is able to consider learning curves from other data sets and that it uses pairwise ranking loss. The former allows to predict for relatively short, and in extreme cases even without, learning curves. The latter directly allows to model the probability that one configuration is better than the another. We analyze our method on five different data sets against three alternatives. In an experiment to optimize network architectures, obtain the fastest results. In the best case, LCRankNet is 100 times faster without sacrificing accuracy. We also examine the components and predictions of our method to give the reader a better understanding of the design choices and functionalities.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv/ org.
herg/bbs.. abs/ 0t5//149.0000
Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Accelerating neural architecture search using performance prediction. In 6th International Conference on Learning Representa- tions, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Workshop Track Proceedings, 2018. URL httthhtts///oom http:::://pperrvieee. www. forum?/fmmm H
Christopher J. C. Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamil- ton, and Gregory N. Hullender. Learning to rank using gradient descent. In Machine Learn- ing. Proceedings of the Twenty-Second International Conference (ICML 2005), Bonn, Ger- many, August 7-11, 2005, pp. 89-96, 2005. doi: 10.1145/1103511.11222 00351 1.....11220 1102363
URL https:
Akshay Chandrashekaran and Ian R. Lane. Speeding up hyper-pare pprrmparreettr optimization by extrapo- lation of learning curves using previous builds. In Machine Learning and Knowledge Discovery in Databases- European Conference, ECML PKDD 2017, Skopje, Macedonia, September 18-22, 2017, Proceedings, Part I, volume 10534 of Lecture Notes in Computer Science, pp. 477-492. Springer, 2017.
Terrance De Vries and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout. CORR, abs/1708.00455 2017.
abrrgg/bb/17000 04552]
Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. Speeding up automatic hyperparam- eter optimization of deep neural networks by extrapolation of learning curves. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos AAAI Press, 2015.
Aires, Argentina, July 25-31, 2015. pp. 3460-3468
Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: robust and efficient hyperparameter opti- mization at scale. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüssson, Stockholm, Sweden, July 10-15, 2018, pp. 1436-1445, 2018. Ba .tmll
URL httptttp///prrceednggss .mlr. http:://oooeeeennn.....eeeeeeee.
Kevin G. Jamieson and Ameet Talwalkar. Non-stochastic best arm identification and hyperparameter optimization. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016, Cadiz, Spain, May 9-11, 2016, pp. 240,248, 2016. URL http:// html
proceedings . mwr. prress/v5//jamieee
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http:
abs////b//aaa 1111..11... 6980
Aaron Klein, Stefan Falkner, Jost Tobias Springenberg, and Frank Hutter. Learning curve predic- tion with bayesian neural networks. In 5th International Conference on Learning Representa- tions, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017. URL /tttps://operreee
net _et.:/oorummdd--11153....
Lisha Li, Kevin G. Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyper- band: A novel bandit-based approach to hyperparameter optimization. J. Mach. Learn. Res., 18:
188:1-1 2017. URL
httpp//ww.wggppppps///////--a--. 55mmmm11
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. In
7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019. URI. https:: http:://opeeeeeeeeeeeeee wwt.:/iwwwwweeeeeeeeeeeeeeeeee
net formmt d== eY
Tie- Yan Liu. Learning to Rank for Information Retrieval. Springer, 2011. ISBN 978-3- 642616.. doi: 10.1007/978-3-642. 14267-3. URL /attp::/all.org//100...
578-3-642-14277-
10

---

Under review as a conference paper at ICLR 2020
Iya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In 5th International Conference on Learning Representations. ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017. URL Ttttps:/oppezeeeee id= 9Scxx
net forum?
Hieu Pham, Melody Y. Guan, Barret Zoph, Quoe V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In Proceedings of the 35th International Conference Machine Learning. ICML 2018, Stockholmsmassan, Stockholm. Sweden, July 10-15, 2018, pp. 4092-4101, 2018. URL
mlr www..ppeessseeppeeeeeee. 8a html
Ilya Sutskever, Oriol Vinyals, and Quoe Le. Sequence sequence learning with neu- ral networks. In Advances in Neural Information Processing Systems 27: Annual Con- ference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pp. 3104-3112, 2014. URL http://papers. nips cc/paper/
Barret Zoph and Quae V. Le. Neural architecture search with reinforcement learning. In 5th In- ternational Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017. URL. https: /ttps///ppeeeeeeeeeee net. forum? Hd=r1Ue@Hcx
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. Learning transfer- able architectures for scalable image recognition. In 2018 IEEE Conference Com- puter Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June
18-22, 2018, pp. 8697-8710, 2018. doi: 10.1 10.1009/VVR.22018.000000 aaanspaaaaaaaaacaacccueueerrrrrr
URL http:
//openaccess. thecvf. wtt.


Reviews:
[{"review_id": "rJlhVTqTKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Learning curve prediction methods try to predict the final performance of a candidate model before it gets fully trained. In this way, learning curve prediction can act as a fast method for performance measurements in AutoML. Compared with previous approaches, the proposed method allows for transferring useful knowledge among different datasets.\n\nWhile the proposed method is simple, the authors have shown promising results. However, I have some concerns about the motivations and usage of the proposed method. Please see the questions below:\n\nQ1. Learning curve prediction is a general approach that can be combined with many zero-order optimization methods. Does this paper focus on learning curve prediction on general AutoML problems or just NAS?\n- If the authors want to target general AutoML problems, they should perform experiments with (1) more datasets (come from various domains) and consider more (2) search algorithms. \n- For (1), the authors can follow Auto-sklearn and Auto-weka, or some experiments on graph (e.g., GCN) or some experiments on language modeling. For (2), they can combine the proposed approach with Bayesian optimization or genetic algorithms. CNN has nice transfer learning ability, to show the wide applicability of the proposed approach is important.\n- If authors want to target at NAS, then the proposed method is not useful. Parameter sharing is a better method in NAS for fast performance evaluation. The authors need to compare with some recent NAS papers in CV (e.g., DARTS).\n\nQ2. \"Learning curve component\". Learning curves are naturally a sequence of data points. The proposed method simply does a maximum pooling over all positions, which ignores the sequence of data. Could the authors give some explanations? Is it better to carry an ablation study on this point?\n\nQ3. \"Architecture component\". How will the changes in the search space affect the proposed method? \n\nQ4. Could the authors random draw many architectures from \"NASNet search space\" and then plot their learning curve? It is better to show what kinds of curves will the proposed method likely to give early stops.\n\nQ5. Another main pitfall of the proposed method cannot offer a probabilistic estimation, which can be done by Baker et al. (2018). Since the model is early stopped, it is naturally that there are some uncertainty in the estimated ranks.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1xmXOVhYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper proposes a new method to rank learning curves of neural networks which can be used to speed up neural architecture search.\nCompared to previous work, the learning curve model not only takes hyperparameter configurations into account, but by training it on offline generated data, it is able to model learning curves across different datasets.\nApplied to a simple neural architecture search strategy, the proposed method achieves higher speed-ups on image classification tasks than other methods from the literature.\n\nWhile the method seems interesting, I don't think the paper is ready for acceptance yet, since it a) misses some important details and b) the empirical evaluation is not sufficient.\nMore precisely the following points need to be addressed:\n\n - In section 3, the paper says that all automated methods follow broadly the same principal that the first model is trained to completion. This is not correct, commonly used methods, such as Hyperband (Li et al.) or BOHB (Falkner et al.), use successive halving (Jamieson et al.) which trains a batch of configurations for just a minimum budget and then already discards poorly performing configurations. I also miss a discussion of these methods in the related work section.\n\nHyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization\nLi, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., and Talwalkar, A.\nJournal of Machine Learning Research 2018\n\nBOHB: Robust and efficient hyperparameter optimization at scale\nS Falkner, A Klein, F Hutter\nProceedings of the 35th International Conference on Machine Learning\n\nNon-stochastic best arm identification and hyperparameter optimization\nK Jamieson, A Talwalkar\nArtificial Intelligence and Statistics, 240-248\n\n\n - Is the learning curve model updated during the optimization process with the new observed data? If not, is there any way the model can adapt to new data? Also, what happens if learning curves are fundamentally different to the training data, e.g if different learning rate schedules are used than for generating the offline data?\n\n - A simple baseline that is missing, is to use the last observed value as approximation for the final performance which often works competitively to more powerful learning curve extrapolation methods (e.g see Klein et al.).\n\n - The method is only applied for a simple random search strategy, however, in practice, one would use more sophisticated methods, such as for example Bayesian optimization. The question is, how effective is the proposed method in this setting, since the distribution of learning curves might be dramatically different and biased towards more well-performing configurations with almost identical learning curves.\n\n- I think the experiments would be more convincing, if the method shows strong performance when deploy in commonly used NAS/HPO methods, such as Hyperband or successive halving. This should be straight forward, since decisions which configuration is promoted to a higher budget can be made based on the model instead of just the last observed value.\n\n- What is delta in the experiments? How sensitive is this hyperparameter in practice?\n\n\nMinor comments:\n\n- Related to this paper is the work by Gargiani et al. which also models learning curves across datasets on offline generated data\n Probabilistic Rollouts for Learning Curve Extrapolation Across Hyperparameter Settings\n M Gargiani, A Klein, S Falkner, F Hutter\n arXiv preprint arXiv:1910.04522\n\n- Figure 5: visually it seems that all learning curves are almost identical, maybe it would be better if the plot could zoom in at least for the final stage of training.\n\n\n\nafter the rebuttal\n-----------------------\n\nI thank the authors for answering my  questions. In the rebuttal the authors addressed my concerned about the insufficient empirical evaluation and included other relevant baselines. I will increase my score.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HylXcz8GcB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the problem of automatic early termination in hyper-parameter search and neural architecture search.  The authors propose to form this problem as learning curve ranking and transfer learning. Unlike most previous approaches for learning curve prediction, which estimates the probability of whether the current model is better than the current best model or not by first extrapolating the learning curve and then invoke a heuristic measure, this paper proposes to predict the probability directly. The pairwise comparison probability is modeled as the logistic function of the difference of a scoring function f, where f is modeled as a neural network with the learning curve data as the input, and constructed with three components, including a learning curve component, an architecture component and a data set component. The neural network f is trained with some meta dataset, and then the early termination is then decided based on this pairwise comparison probability. The paper applied the proposed early termination approach to the neural architecture search of five image classification data sets, and evaluated the performance in terms of the Spearman correlation of the learning curve ranking and the regret and time for the architecture search. It also analyzes the learning curve prediction characteristics through a few concrete examples, followed by some ablation analysis.\n\nThe proposed approach is novel to my knowledge, and the numerical performance is also satisfying and convincing. However, there are a few issues that this paper should better improve upon. \n\nFirstly, the methodology description at the beginning of section 3 is not clear enough. In particular, the motivation of choosing logistic function in (1) is not explained, and how the calculation of p_{i,j} from the final learning curves is done is also not elaborated. Some of the terminologies are also not clearly defined or specified. For example, what does \"posteriors\" mean (cf. the line before (2))? What is the meta-knowledge \\mathcal{D}? Is it the one used to train the neural network of f? There is also a small typo in line 2 of Algorithm 1, where d should be D probably. \n\nSecondly, although the numerical experiments are convincing within the framework of learning curves prediction, they are not sufficiently convincing when it comes to the scope of the neural architecture search (NAS) or hyper-parameter optimization (HPO). Comparisons with state-of-the-art NAS and HPO algorithms that do not use learning curves prediction (e.g., HyperBand, learning-by-learning algorithms using LBFGS, etc.) are not mentioned or compared with. The authors may either want to add comparisons with those algorithms, or provide some more applications of learning curve prediction to showcase the flexibility of the proposed approach.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Authors propose a new way of early stopping for neural architecture search. In contrast to making keep or kill decisions based on extrapolating the learning curves then making decisions between alternatives, this work learns a model on pairwise comparisons between learning curves directly. Reviewers were concerned with over-claiming of novelty since the original version of this paper overlooked significant hyperparameter tuning works. In a revision, additional experiments were performed using some of the suggested methods but reviewers remained skeptical that the empirical experiments provided enough justification that this work was ready for prime time.   
Decision: Reject