Revisiting the Train Loss: an Efficient Performance Estimator for Neural Architecture Search

1 INTRODUCTION . Reliably estimating the generalisation performance of a proposed architecture is crucial to the success of Neural Architecture Search ( NAS ) but has always been a major bottleneck in NAS algorithms ( Elsken et al. , 2018 ) . The traditional approach of training each architecture for a large number of epochs and evaluating it on validation data ( full evaluation ) provides a reliable performance measure , but requires prohibitively high computational resources on the order of thousands of GPU days ( Zoph & Le , 2017 ; Real et al. , 2017 ; Zoph et al. , 2018 ; Real et al. , 2019 ; Elsken et al. , 2018 ) . This motivates the development of methods for speeding up performance estimation to make NAS practical for limited computing budgets . A popular simple approach is early-stopping which offers a low-fidelity approximation of generalisation performance by training for fewer epochs ( Li et al. , 2016 ; Falkner et al. , 2018 ; Li & Talwalkar , 2019 ) . However , if we stop the training early at a small number of epochs and evaluate the model on validation data , the relative performance ranking may not correlate well with the performance ranking of the full evaluation ( Zela et al. , 2018 ) . Another line of work focuses on learning curve extrapolation ( Domhan et al. , 2015 ; Klein et al. , 2016b ; Baker et al. , 2017 ) , which trains a surrogate model to predict the final generalisation performance based on the initial learning curve and/or meta-features of the architecture . However , the training of the surrogate often requires hundreds of fully evaluated architectures to achieve satisfactory extrapolation performance and the hyper-parameters of the surrogate also need to be optimised . Alternatively , the idea of weight sharing is adopted in one-shot NAS methods to speed up evaluation ( Pham et al. , 2018 ; Liu et al. , 2019 ; Xie et al. , 2019b ) . Despite leading to significant cost-saving , weight sharing heavily underestimates the true performance of good architectures and is unreliable in predicting the relative ranking among architectures ( Yang et al. , 2020 ; Yu et al. , 2020 ) . In view of the above limitations , we propose a simple model-free method which provides a reliable yet computationally cheap estimation of the generalisation performance ranking of architectures : the Sum over Training Losses ( SoTL ) . Our method harnesses the training losses of the commonly-used SGD optimiser during training , and is motivated by recent empirical and theoretical results linking training speed and generalisation ( Hardt et al. , 2016 ; Lyle et al. , 2020 ) . We ground our method in the Bayesian update setting , where we show that the SoTL estimator computes a lower bound to the model evidence , a quantity with sound theoretical justification for model selection ( MacKay , 1992 ) . We show empirically that our estimator can outperform a number of strong existing approaches to predict the relative performance ranking among architectures , while speeding up different NAS approaches significantly . 2 METHOD . We propose a simple metric that estimates the generalisation performance of a deep neural network model via the Sum of its Training Losses ( SoTL ) . After training a deep neural network whose prediction is fθ ( · ) for T epochs1 , we sum the training losses collected so far : SoTL = T∑ t=1 [ 1 B B∑ i=1 l ( fθt , i ( Xi ) , yi ) ] ( 1 ) where l is the training loss of a mini-batch ( Xi , yi ) at epoch t and B is the number of training steps within an epoch . If we use the first few epochs as the burn-in phase for θt , i to converge to certain distribution P ( θ ) and start the sum from epoch t = T −E + 1 instead of t = 1 , we obtain a variant SoTL-E . In the case where E = 1 , we start the sum at t = T and our estimator corresponds to the sum over training losses within epoch t = T . We discuss SoTL ’ s theoretical interpretation based on Bayesian marginal likelihood and training speed in Section 3 , and empirically show that SoTL , despite its simple form , can reliably estimate the generalisation performance of neural architectures in Section 5 . If the sum over training losses is a useful indicator for the generalisation performance , one might expect the sum over validation losses to be a similarly effective performance estimator . The sum over validation losses ( SoVL ) lacks the link to the Bayesian model evidence , and so its theoretical motivation is different from our SoTL . Instead , the validation loss sum can be viewed as performing a bias-variance trade-off ; the parameters at epoch t can be viewed as a potentially high-variance sample from a noisy SGD trajectory , and so summation reduces the resulting variance in the validation loss estimate at the expense of incorporating some bias due to the relative ranking of models ’ test performance changing during training . We show in Section 5 that SoTL clearly outperforms SoVL in estimating the true test performance . 3 THEORETICAL MOTIVATION . The SoTL metric is a direct measure of training speed and draws inspiration from two lines of work : the first is a Bayesian perspective that connects training speed with the marginal likelihood in the model selection setting , and the second is the link between training speed and generalisation ( Hardt et al. , 2016 ) . In this section , we will summarize recent results that demonstrate the connection between SoTL and generalisation , and further show that in Bayesian updating regimes , the SoTL metric corresponds to an estimate of a lower bound on the model ’ s marginal likelihood , under certain assumptions . 3.1 TRAINING SPEED AND THE MARGINAL LIKELIHOOD . We motivate the SoTL estimator by a connection to the model evidence , also called the marginal likelihood , which is the basis for Bayesian model selection . The model evidence quantifies how likely a dataset D is to have been generated by a model , and so can be used to update a prior belief distribution over which model from a given set is most likely to have generated D. Given a model with parameters θ , prior π ( θ ) , and likelihood P ( D|θ ) for a training data set D = { D1 , . . . , Dn } with data points Di = ( xi , yi ) , the ( log ) marginal likelihood is expressed as follows . logP ( D ) = logEπ ( θ ) [ P ( D|θ ) ] ⇔ logP ( D ) = n∑ i=1 logP ( Di|D < i ) = n∑ i=1 log [ EP ( θ|D < i ) [ P ( Di|θ ) ] ] Interpreting the negative log posterior predictive probability− logP ( Di|D < i ) of each data point as a ‘ loss ’ function , the log evidence then corresponds to the area under a training loss curve , where each 1T can be far from the total training epochs Tend used in complete training training step would be computed by sampling a data point Di , taking the log expected likelihood under the current posterior P ( θ|D < i ) as the current loss , and then updating the posterior by incorporating the new sampled data point : D < i+1 : = D < i ∪ { Di } . One can therefore interpret the marginal likelihood as a measure of training speed in a Bayesian updating procedure . In the setting where we can not compute the posterior analytically and only samples θ̂ from the posterior over parameters are available , we obtain an unbiased estimator of a lower bound L ( D ) = ∑ EP ( θ|D < i ) [ logP ( Di|θ ) ] on the marginal likelihood by Jensen ’ s inequality , which again corresponds to minimizing a sum over training losses∑ logP ( Di|θ̂ ) ≈ ∑ EP ( θ|D < i ) [ logP ( Di|θ ) ] ≤ ∑ log [ EP ( θ|D < i ) [ P ( Di|θ ) ] ] = logP ( D ) with ≈ denoting equality in expectation . A full analysis of the Bayesian setting is outside of the scope of this work . We refer the reader to ( Lyle et al. , 2020 ) for more details of the properties of this estimator in Bayesian models . Although the NAS setting does not yield the same interpretation of SoTL as model evidence estimation , we argue that the SoTL metric is still plausibly useful for model selection . Just as the marginal likelihood measures the utility of updates based on early data points in predicting later data points , the SoTL of a model trained with SGD will be lower for models whose mini-batch gradient descent updates improve the loss of later mini-batches seen during optimisation . We refer the reader to Apppendix B to see a demonstration of the SoTL metric in the Bayesian linear regression setting . We emphasize that the Bayesian connection thus justifies the sum over training losses as a tool for model selection , but not the training loss from a single parameter update . 3.2 TRAINING SPEED AND GENERALISATION . Independent of the accuracy of SoTL in estimating the Bayesian model evidence , it is also possible to motivate our method by its relationship with training speed : models which achieve low training loss quickly will have low SoTL . There are both empirical and theoretical lines of work that illustrate a deep connection between training speed and generalisation . On the theoretical front , we find that models which train quickly can attain lower generalisation bounds . Training speed and generalisation can be related via stability-based generalisation bounds ( Hardt et al. , 2016 ; Liu et al. , 2017 ) , which characterize the dependence of the solution found by a learning algorithm on its training data . In networks of sufficient width , ( Arora et al. , 2019 ) propose a neural-tangent-kernel-based data complexity measure which bounds both the convergence rate of SGD and the generalisation error of the model obtained by optimisation . A similar generalisation bound and complexity measure is obtained by ( Cao & Gu , 2019 ) . While theoretical work has largely focused on ranking bounds on the test error , current results do not provide guarantees on consistency between the ranking of different models ’ test set performance and their generalisation bounds . The empirical work of ( Jiang * et al. , 2020 ) demonstrates that many complexity measures are uncorrelated or negatively correlated with the relative performance of models on their test data but notably , a particular measure of training speed – the number of steps required to reach cross-entropy loss of 0.1 , was highly correlated with the test set performance ranking of different models . The connection between training speed and generalisation is also observed by ( Zhang et al. , 2016 ) , who find that models trained on true labels converge faster than models trained on random labels , and attain better generalisation performance . 4 RELATED WORK . Various approaches have been developed to speed up architecture performance estimation , thus improving the efficiency of NAS . Low-fidelity estimation methods accelerate NAS by using the validation accuracy obtained after training architectures for fewer epochs ( namely early-stopping ) ( Li et al. , 2016 ; Falkner et al. , 2018 ; Zoph et al. , 2018 ; Zela et al. , 2018 ) , training a down-scaled model with fewer cells during the search phase ( Zoph et al. , 2018 ; Real et al. , 2019 ) or training on a subset of the data ( Klein et al. , 2016a ) . However , low-fidelity estimates underestimate the true performance of the architecture and can change the relative ranking among architectures ( Elsken et al. , 2018 ) . This undesirable effect on relative ranking is more prominent when the cheap approximation set-up is too dissimilar to the full evaluation ( Zela et al. , 2018 ) . As shown in our Fig . 2 below , the validation accuracy at early epochs of training suffers low rank correlation with the final test performance . Another way to cheaply estimate architecture performance is to train a regression model to extrapolate the learning curve from what is observed in the initial phase of training . Regression model choices that have been explored include Gaussian processes with a tailored kernel function ( Domhan et al. , 2015 ) , an ensemble of parametric functions ( Domhan et al. , 2015 ) , a Bayesian neural network ( Klein et al. , 2016b ) and more recently a ν-support vector machine regressor ( ν-SVR ) ( Baker et al. , 2017 ) which achieves state-of-the-art prediction performance . Although these model-based methods can often predict the performance ranking better than their model-free early-stopping counterparts , they require a relatively large amount of fully evaluated architecture data ( e.g . 100 fully evaluated architectures in ( Baker et al. , 2017 ) ) to train the regression surrogate properly and optimise the model hyperparameters in order to achieve good prediction performance . The high computational cost of collecting the training set makes such model-based methods less favourable for NAS unless the practitioner has already evaluated hundreds of architectures on the target task . Moreover , both low-fidelity estimates and learning curve extrapolation estimators are empirically developed and lack theoretical motivation . Finally , one-shot NAS methods employ weight sharing to reduce computational costs ( Pham et al. , 2018 ; Liu et al. , 2019 ; Xie et al. , 2019b ) . Under the one-shot setting , all architectures are considered as subgraphs of a supergraph . Only the weights of the supergraph are trained while the architectures ( subgraphs ) inherit the corresponding weights from the supergraph . Weight sharing removes the need for retraining each architecture during the search and thus achieves a significant speed-up . However , the weight sharing ranking among architectures often correlates very poorly with the true performance ranking ( Yang et al. , 2020 ; Yu et al. , 2020 ; Zela et al. , 2020 ) , meaning architectures chosen by one-shot NAS are likely to be sub-optimal when evaluated independently ( Zela et al. , 2020 ) . Moreover , one-shot methods are often outperformed by sample-based NAS methods ( Dong & Yang , 2020 ; Zela et al. , 2020 ) . Apart from the above mentioned performance estimators used in NAS , many complexity measures have been proposed to analyse the generalisation performance of deep neural networks . ( Jiang * et al. , 2020 ) provides a rigorous empirical analysis of over 40 such measures . This investigation finds that sharpness-based measures ( McAllester , 1999 ; Keskar et al. , 2016 ; Neyshabur et al. , 2017 ; Dziugaite & Roy , 2017 ) ( including PAC-Bayesian bounds ) provide good correlation with test set performance , but their estimation requires adding randomly generated perturbations to the network parameters and the magnitude of the perturbations needs to be carefully optimised with additional training , making them unsuitable performance estimators for NAS . Optimisation-based complexity measures also perform well in predicting generalisation . Specifically , the number of steps required to reach loss of 0.1 , as mentioned in Section 3.2 , is closely related to our approach as both quantities measure the training speed of architectures . To our knowledge though , this measure has never been used in the NAS context before .

Summary: This paper proposes a simple model-free method to estimate the generalization performance of deep neural architectures based on their early training losses. The proposed method uses the sum of training losses during training to estimate the performance and is motivated by recent empirical and theoretical results. The experimental results show that the proposed estimator outperforms the existing methods that predict the performance ranking among architectures.