Teacher Guided Architecture Search

Strong improvements in neural network performance in vision tasks have resulted from the search of alternative network architectures . Prior work has shown that this search process can be automated and guided by evaluating candidate network performance following limited training ( Performance Guided Architecture Search or PGAS ) . However , because of the large architecture search spaces and the high computational cost associated with evaluating each candidate network , further gains in computational efficiency are needed . Here we present a method termed Teacher Guided Search for Architectures by Generation and Evaluation ( TG-SAGE ) that produces up to an order of magnitude in search efficiency over PGAS methods . Specifically , TG-SAGE guides each step of the architecture search by evaluating the similarity of internal representations of the candidate networks with those of the ( fixed ) teacher network . We show that this procedure leads to significant reduction in required per-sample training and that , this advantage holds for two different search spaces of architectures , and two different search algorithms . We further show that in the space of convolutional cells for visual categorization , TG-SAGE finds a cell structure with similar performance as was previously found using other methods but at a total computational cost that is two orders of magnitude lower than Neural Architecture Search ( NAS ) and more than four times lower than progressive neural architecture search ( PNAS ) . These results suggest that TG-SAGE can be used to accelerate network architecture search in cases where one has access to some or all of the internal representations of a teacher network of interest , such as the brain . 1 INTRODUCTION . The accuracy of deep convolutional neural networks ( CNNs ) for visual categorization has advanced substantially from 2012 levels ( AlexNet ( Krizhevsky et al. , 2012 ) ) to current state-of-the-art CNNs like ResNet ( He et al. , 2015 ) , Inception ( Szegedy et al. , 2014 ) , DenseNet ( Huang et al. , 2016 ) . This progress is mostly due to discovery of new network architectures . Yet , even the space of feedforward neural network architectures is essentially infinite and given this complexity , the design of better architectures remains a challenging and time consuming task . Several approaches have been proposed to automate the discovery of neural network architectures , including random search ( Pinto et al. , 2009 ) , reinforcement learning ( Zoph & Le , 2017 ) , evolution ( Real et al. , 2016 ) , and sequential model based optimization ( SMBO ) ( Liu et al. , 2017 ; Bergstra et al. , 2012a ) . These methods operate by iteratively sampling from the hyperparameter space , training the corresponding architecture , evaluating it on a validation set , and using the search history of those scores to guide further architecture sampling . But even with recent improvements in search efficiency , the total cost of architecture search is still outside the reach of many groups and thus impedes the research in this area ( e.g . some of the recent work in this area has spent 40-557k GPU-hours for each search experiment ( Real et al. , 2018 ; Zoph & Le , 2017 ) ) . What drives the total computational cost of running a search ? For current architectural search procedures ( above ) , the parameters of each sampled architecture must be trained before its performance can be evaluated and the amount of such training turns out to be a key driver in the total computational cost . Thus , to reduce that total cost , each architecture is typically only partially trained to a premature state and its premature performance is used as a proxy of its mature performance ( i.e . the performance it would have achieved if was actually fully trained ) . Because the search goal is high mature performance in a task of interest , the most natural choice of an architecture evaluation score is its premature performance . However , this may not be the best choice of evaluation score . For example , it has been observed that , as a network is trained , multiple sets of internal features begin to emerge over network layers , and the quality of these internal features determines the ultimate behavioral performance of the neural network as a whole . Based on these observations , we reasoned that , if we could evaluate the quality of a network ’ s internal features even in a very premature state , we might be able to more quickly determine if a given architecture is likely to obtain high levels of mature performance . But without a reference set of high quality internal features , how can we determine the quality of a network ’ s internal features ? The main idea proposed here is to use features of a high performing “ teacher ” network as a reference to identify promising sample architectures at a much earlier premature state . Our proposed method is inspired by prior work showing that the internal representations of a high-performing teacher network can be used to optimize the parameters of smaller , shallower , or thinner student networks ( Ba & Caruana , 2014 ; Hinton et al. , 2015 ; Romero et al. , 2014 ) . It is also inspired by the fact that such internal representation measures can potentially be obtained from primate brain and thus could be used as an ultimate teacher . While our ability to simultaneously record from large populations of neurons is fast growing ( Stevenson & Kording , 2011 ) , these measurements have already been shown to have remarkable similarities to internal features of CNNs ( Yamins et al. , 2014 ; Schrimpf et al. , 2018 ) . We refer to this method as Teacher Guided Search for Architectures by Generation and Evaluation ( TG-SAGE ) . Specifically , TG-SAGE guides each step of an architecture search by evaluating the similarity of several internal feature representations of each sampled architecture with those of a fixed , high-performing teacher network with unknown architectural parameters but observable internal states . We found that when this evaluation is combined with the usual performance evaluation ( above ) , we can predict the mature performance of sampled architectures with an order of magnitude less premature training and thus an order of magnitude less total computational cost . We then use this observation to execute multiple runs of TG-SAGE for different architecture search spaces to confirm that TG-SAGE can indeed discover network architectures of comparable mature performance to those discovered with performance-only search methods , but with far less total computational cost . 2 PREVIOUS WORK . There have been several studies on automatic design of neural network architectures in the past few years . Real et al . ( Real et al. , 2016 ; 2018 ) used an evolutionary approach in which samples taken from a pool of networks were engaged in a pairwise competition game . This method searched for optimal architectures and weights jointly by reusing all or part of weights from the parent network in an effort to reduce the computation cost associated with training the candidate networks as well as the final retraining of the best found networks . However , it is not clear to what degree this procedure has cut down on the computational cost compared to alternative search method that depend on ( some ) training for each candidate network starting from an initial point . There have also been several studies on using reinforcement learning in agents that learn to design high performing neural network architectures ( Baker et al. , 2016 ; Zoph & Le , 2017 ) . Of special relevance to this work is Neural Architecture Search ( NAS ) ( Zoph & Le , 2017 ) in which a long short-term memory network ( LSTM ) trained using REINFORCE was used to learn to design neural network architectures for object recognition and natural language processing tasks . A variation of this approach was later used to design convolutional cell structures similar to those used in Inception network that could be transferred to larger datasets like Imagenet ( Zoph et al. , 2017 ) . While most of these works have focused on discovering higher performing architectures , there has been a number of efforts emphasizing the computational efficiency in hyperparameter search . In order to reduce the computational cost of architecture search , Brock et al . ( Brock et al. , 2017 ) proposed to use a hypernetwork ( Ha et al. , 2016 ) to predict the layer weights for any arbitrary candidate architecture instead of retraining from random initial values . Hyperband Li et al . ( 2017 ) formulated hyperparameter search as a resource allocation problem and improved the efficiency by controlling the amount of resources ( e.g . training ) allocated to each sample . Similarly , several other methods proposed to increase the search efficiency by introducing early-stopping criteria during training Baker et al . ( 2017 ) or by extrapolating the learning curve Domhan et al . ( 2015 ) . These approaches are closely related to our proposed method in that , their main focus is to reduce the per-sample training cost . Some recent work attempted to jointly optimize for the network hyperparameters as well as the trainable weights themselves . While this is a very interesting idea that significantly reduces the computational cost of architecture search , in its current form it can only be applied to the spaces of network architectures in which the number of trainable weights do not change as a result of hyperparameter choices ( e.g . when the number of filters in a CNN is fixed ) . Efficient NAS ( Pham et al. , 2018 ) and DARTS ( Liu et al. , 2018 ) methods proposed to share the trainable parameters across all candidate networks and to jointly optimize for the hyperparameters and the network weights during the search . While these approaches led to significant reduction in total search cost , they did so by constraining the search space due to considerations regarding the shared trainable weights . More recently progressive neural architecture search ( PNAS ) ( Liu et al. , 2017 ) proposed a sequential model based optimization ( SMBO ) approach that learned a predictive model of performance given the hyperparameters through a procedure which gradually increased the complexity of the space . This approach led to an impressive 20× improvement in the computational cost of search compared to NAS . 3 METHODS . 3.1 REPRESENTATIONAL DISSIMILARITY MATRIX : . Representational Dissimilarity Matrix ( RDM ) Kriegeskorte et al . ( 2008 ) is a statistic computed for a representation space that quantifies the dissimilarity between activity patterns in that space in response to pairs of inputs or input categories . For a given feature matrix F ∈ Rni×nf which contains nf features measured in response to ni images , we derive RDM ( MF ) by computing the pairwise distances between each pair of feature vectors ( rows in F ) using a distance measure like correlation residual . MF ∈ Rnc×nc , Mi , j = 1− corr ( F ci , F cj ) ( 1 ) When calculating RDM for object categories ( instead of individual images ) we substitute the matrix F with F c in which each row c contains the average activity pattern across all images of category c. Once RDM is calculated for two representation spaces , we can evaluate the similarity of those spaces by calculating the correlation coefficient ( e.g . Pearson ’ s r ) between the two RDM matrices . 3.2 REPRESENTATIONAL SIMILARITY WITH A TEACHER NETWORK AS SURROGATE . The largest portion of cost associated with neural network architecture search comes from training the sampled networks , which is proportional to the number of training steps ( SGD updates ) performed on the network . Due to the high cost of fully training each sampled network , in most cases a surrogate score is used as a proxy for the mature performance . Correlation between the surrogate and mature score may affect the architecture search performance as poor proxy values could guide the search algorithm towards suboptimal regions of the space . Previous work on architecture search in the space of Convolutional Neural Networks ( CNN ) have concurred with the empirical surrogate measure of premature performance after about 20 epochs of training . While 20 epochs is much lower than the usual number of epochs used to fully train a CNN network ( 300-900 epochs ) , it still forces a large cost on conducing architecture searches . We propose that evaluating the internal representations of a network would be a more reliable measure of architecture quality during the early phase of training ( e.g . after several hundreds of SGD iterations ) , when the features are starting to be formed but the network is not yet performing reliably on the task . An overview of the procedure is illustrated in Figure-1 . We evaluate each sampled model by measuring the similarity between its RDMs at different layers ( e.g . RDMC1−C3 ) to those extracted from the teacher network ( e.g . RDMT1−T3 ) . To this end , we compute RDM for all or a subset of layers in the network and then compute the correlation between all pairs of student and teacher RDMs . To score a candidate network against a given layer of the teacher network , we consider the highest RDM similarity to teacher layer calculated over all layers of the student network ( e.g . RDMS1−S3 ) . Finally , we construct an overall score by taking the mean of the RDM scores which we call TG ( Teacher Guidance ) . We also define a combined Performance + TG ( P+TG ) score which is formulated as weighted sum of premature performance and TG score in the form of P + αTG . In this fashion , the combined validation score guides the architecture search to maximize performance as well as representational similarity with the teacher architecture . We consider the teacher architecture as any high-performing network with unknown architecture but observable activations . We can have one or several measured endpoints from the teacher network that each could potentially be used to generate a similarity score .

Summary: This work tries to accelerate neural architecture search by reducing the computational workload spend to evaluate a single architecture. Given a network with premature performance (i.e. one trained only for few epochs), it tries to compute the mature performance (accuracy achieved when training the network to completion). A score for each architecture is given by "P+TG" which considers the validation accuracy and the similarity of instance representations at different layers to a given teacher network. In an experiment against using the validation accuracy as a score only, they achieve higher validation accuracies on CIFAR-10/100. In a comparison against NASNet and PNASNet, their experiments indicate higher validation accuracy in orders of magnitudes faster run time.