Learning to Rank Learning Curves

1 INTRODUCTION . A method commonly used by human experts to speed up the optimization of neural architectures or hyperparameters is the early termination of iterative training processes that are unlikely to improve the current solution . A common technique to determine the likelihood of no improvement is to compare the learning curve of a new configuration to the one of the currently best configuration . This idea can also be used to speed up automated machine learning processes . For this purpose , it is common practice to extrapolate the partial learning curve in order to predict the final performance of the currently investigated model . Current extrapolation techniques have several weaknesses that make them unable to realize their full potential in practice . Many of the methods require sufficient sample learning curves to make reliable predictions ( Chandrashekaran & Lane , 2017 ; Klein et al. , 2017 ; Baker et al. , 2018 ) . Thus , the extrapolation method for the first candidates can not be used yet , which means more computational effort . Other methods do not have this disadvantage , but require sufficiently long learning curves to make reliable predictions which again means unnecessary overhead ( Domhan et al. , 2015 ) . Many of these methods also do not take into account other information such as the hyperparameters of the model being examined or its network architecture . We address the need for sample learning curves by devising a transfer learning technique that uses learning curves from other problems . Since the range of accuracy varies from data set to data set , we are forced to consider this in our modeling . But since we are not interested in predicting the performance of a model anyway , we use a ranking model that models the probability that the model currently being investigated surpasses the best solution so far . This does not only solve the problem but also provides a better modeling of the actual task . In order to be able to make reliable predictions for short learning curves , we consider further characteristics of the model such as its network architecture . We compare our ranking method with respect to a ranking measure against different methods on five different image classification data sets . We also show that our method is capable of significantly accelerating a neural architecture search . Furthermore , we conduct several ablation studies to provide a better motivation of our model and its behavior . 2 RELATED WORK . Most of the prior work for learning curve prediction is based on the idea of extrapolating the partial learning curve by using a combination of continuously increasing basic functions . Domhan et al . ( 2015 ) define a set of 11 parametric basic functions , estimate their parameters and combine them in an ensemble . Klein et al . ( 2017 ) propose a heteroscedastic Bayesian model which learns a weighted average of the basic functions . Chandrashekaran & Lane ( 2017 ) do not use basic functions but use previously observed learning curves of the current data set . An affine transformation for each previously seen learning curve is estimated by minimizing the mean squared error with respect to the partial learning curve . The best fitting extrapolations are averaged as the final prediction . Baker et al . ( 2018 ) use a different procedure . They use support vector machines as sequential regressive models to predict the final accuracy based on features extracted from the learning curves , its gradients , and the neural architecture itself . The predictor by Domhan et al . ( 2015 ) is able to forecast without seeing any learning curve before but requires observing more epochs for accurate predictions . The model by Chandrashekaran & Lane ( 2017 ) requires seeing few learning curves to extrapolate future learning curves . However , accurate forecasts are already possible after few epochs . Algorithms proposed by Klein et al . ( 2017 ) ; Baker et al . ( 2018 ) need to observe many full-length learning curves before providing any useful forecasts . However , this is prohibiting in the scenarios where learning is time-consuming such as in large convolutional neural networks . All previous methods for automatically terminating iterative learning processes are based on methods that predict the learning curve . Ultimately , however , we are less interested in the exact learning curve but rather whether the current learning curve leads to a better result . This way of obtaining a ranking is referred to as pointwise ranking methods ( Liu , 2011 ) . They have proven to be less efficient than pairwise ranking methods which directly optimize for the objective function ( Burges et al. , 2005 ) . Yet , we are the first to consider a pairwise ranking loss for this application . There are some bandit-based methods which leverage early termination as well . Successive Halving ( Jamieson & Talwalkar , 2016 ) is a method that trains multiple models with settings chosen at random simultaneously and terminates the worst performing half in predefined intervals . Hyperband ( Li et al. , 2017 ) identifies that the choice of the intervals is vital and therefore proposes to run Successive Halving with different intervals . BOHB ( Falkner et al. , 2018 ) is an extension of Hyperband which proposes to replace the random selection of settings with Bayesian optimization . The use of methods that terminate less promising iterative training processes early are of particular interest in the field of automated machine learning , as they can , for example , significantly accelerate hyperparameter optimization . The most computationally intensive subproblem of automated machine learning is Neural Architecture Search ( Zoph & Le , 2017 ) , the optimization of the neural network topology . Our method is not limited to this problem , but as it is currently one of the major challenges in automated machine learning , we use this problem as a sample application in our evaluation . For this particular application , optimization methods that leverage parameter sharing ( Pham et al. , 2018 ) have become established as a standard method . Here , a search space spanning , overparametrised neural network is learned and finally used to determine the best architecture . DARTS ( Liu et al. , 2019 ) is one of the last extensions of this idea which uses a continuous relaxation of the search space , which allows learning of the shared parameters as well as of the architecture by means of gradient descent . This method can be a strong alternative to early termination methods but does not transfer to general machine learning or arbitrary architecture search spaces . 3 LEARNING CURVE RANKING . With learning curve we refer to the function of qualitative performance with growing number of iterations of an iterative learning algorithm . We use the term final learning curve to explicitly denote the entire learning curve , y1 , . . . , yL , reflecting the training process from beginning to end . Here , yi is a measure of the performance of the model ( e.g. , classification accuracy ) , which is determined at regular intervals . Contrary , a partial learning curve , y1 , . . . , yl , refers to learning curves that are observed only up to a time l. We visualize the concepts of the terms in Figure 1 . There is a set of automated early termination methods which follow broadly the same principle . The first model is trained to completion and is considered as the current best model mmax with its performance being ymax . For each further model mi , the probability that it is better than the current best model , p ( mi > mmax ) , is monitored at periodic intervals during training . If it is below a given threshold , the model ’ s training is terminated ( see Algorithm 1 ) . All existing methods rely on a two- step approach to determine this probability which involves extrapolating the partial learning curve and a heuristic measure . Instead of the two-step process to determine the probability , we propose LCRankNet to predict the probability that a model mi is better than mj directly . LCRankNet is based on a neural network f which considers model characteristics and the partial learning curve xi as input . We define the probability that mi is better than mj as p ( mi > mj ) = p̂i , j = ef ( xi ) −f ( xj ) 1 + ef ( xi ) −f ( xj ) . ( 1 ) Using the logistic function is an established modelling approach in the learning-to-rank community ( Burges et al. , 2005 ) . Given a set of final learning curves for some models , the estimation of the posterior values pi , j between these models is trivial since we know whether mi is better than mj or not . Therefore , we set pi , j = 1 if mi > mj 0.5 if mi = mj 0 if mi < mj . ( 2 ) We minimize the cross-entropy loss∑ i , j −pi , j log p̂i , j − ( 1− pi , j ) log ( 1− p̂i , j ) ( 3 ) to determine the parameters of f . Now the probability p ( mi > mj ) can be predicted for arbitrary pairs mi and mj using Equation ( 1 ) . Algorithm 1 Early Termination Method Input : Data set d , model m , performance of best model mmax so far ymax . Output : Learning curve . 1 : for l← 1 . . . L do 2 : Train m on d for a step and observe a further part of the learning curve yt . 3 : if max1≤i≤l yi > ymax then 4 : continue 5 : else if p ( m > mmax ) ≤ δ then 6 : return y 7 : return y 3.1 RANKING MODEL TO LEARN ACROSS DATA SETS . We have defined the prediction of our model in Equation ( 1 ) . It depends on the outcome of the function f which takes a model representation x as input . The information contained in this representation depends on the task at hand . Since we will consider Neural Architecture Search , the representation consists of three different parts . First , the partial learning curve y1 , . . . , yl . Second , the description of the model ’ s architecture which is a sequence of strings representing the layers it is comprised of . Third , a data set ID to indicate on which data set the corresponding architecture and learning curve was trained and observed . We model f using a neural network and use special layers to process the different parts of the representation . The learned representation for the different parts are finally concatenated and fed to a fully connected layer . The architecture is visualized in Figure 2 . We will now describe the different components in detail . Learning curve component A convolutional neural network is used to process the partial learning curve . We consider up to four different convolutional layers with different kernel sizes . The exact number depends on the length of the learning curve since we consider no kernel sizes larger than the learning curve length . Each convolution is followed by a global max pooling layer and their outputs are concatenated . This results in the learned representation for the learning curve . Architecture component For our learning curve prediction we use architectures of the popular NASNet search space ( Zoph et al. , 2018 ) . We use the most common configuration which consists of two cells and every cell of five blocks . This results into 40 choices to be made which is encoded as a sequence of integers . Please refer to Zoph et al . ( 2018 ) for further details regarding the search space . We learn an embedding for every choice . An LSTM takes this embedding and generates the architecture embedding . Data set component As we intend to learn from learning curves of other data sets , we include data set embeddings as one of the components . Every data set has its own embedding and it will be used whenever a learning curve observed on this data set is selected . If the data set is new , then the corresponding embedding is initialized at random . As the model observes more learning curves from this data set , it improves the data set embedding . The embedding helps us to model data-set-specific peculiarities in order to adjust the ranking if necessary . Technical details During the development process , we found that the architecture component leads to instabilities during training . To avoid this , we regularize the output of the LSTM layer by using it as an input to an autoregressive model , similar to a sequence-to-sequence model ( Sutskever et al. , 2014 ) that recreates the original description of the architecture . In addition , we use the attention mechanism ( Bahdanau et al. , 2015 ) to facilitate this process . All parameters of the layers in f are trained jointly by means of Adam ( Kingma & Ba , 2015 ) by minimizing a weighted linear combination of the ranking loss ( Equation ( 3 ) ) and the reconstruction loss with respect to its parameters . 0 10 20 30 Seen Learning Curve ( % ) 0.00 0.25 0.50 0.75 Sp ea rm an C or re la tio n CIFAR-10 0 10 20 30 Seen Learning Curve ( % ) 0.0 0.5 1.0 Sp ea rm an C or re la tio n CIFAR-100 0 10 20 30 Seen Learning Curve ( % ) 0.25 0.00 0.25 0.50 Sp ea rm an C or re la tio n Fashion-MNIST 0 10 20 30 Seen Learning Curve ( % ) 0.00 0.25 0.50 0.75 Sp ea rm an C or re la tio n Quick , Draw ! 0 10 20 30 Seen Learning Curve ( % ) 0.25 0.00 0.25 0.50 Sp ea rm an C or re la tio n SVHN 0 10 20 30 Seen Learning Curve ( % ) 0.4 0.2 0.0 0.2 0.4 0.6 Sp ea rm an C or re la tio n SVHN Baker et al . Chandrashekaran Domhan et al . Last Seen Value LCRankNet Figure 3 : The x-axis indicates the length observed of the learning curve . We report the mean of ten repetitions . The shaded area is the standard deviation . Our method LCRankNet outperforms its competitors on all data sets . The hyperparameter δ allows for trading precision vs. recall or search time vs. regret , respectively . There are two extreme case : if we set δ ≥ 1 every run is terminated immediately . If we set δ ≤ 0 we never terminate any run . For our experiment we set δ = 0.45 which means that if the predicted probability that the new model is better than the best one is below 45 % , the run is terminated early .

Summary: This paper considers the problem of automatic early termination in hyper-parameter search and neural architecture search.  The authors propose to form this problem as learning curve ranking and transfer learning. Unlike most previous approaches for learning curve prediction, which estimates the probability of whether the current model is better than the current best model or not by first extrapolating the learning curve and then invoke a heuristic measure, this paper proposes to predict the probability directly. The pairwise comparison probability is modeled as the logistic function of the difference of a scoring function f, where f is modeled as a neural network with the learning curve data as the input, and constructed with three components, including a learning curve component, an architecture component and a data set component. The neural network f is trained with some meta dataset, and then the early termination is then decided based on this pairwise comparison probability. The paper applied the proposed early termination approach to the neural architecture search of five image classification data sets, and evaluated the performance in terms of the Spearman correlation of the learning curve ranking and the regret and time for the architecture search. It also analyzes the learning curve prediction characteristics through a few concrete examples, followed by some ablation analysis.