Neural Architecture Search without Training

1 INTRODUCTION . The success of deep learning in computer vision is in no small part due to the insight and engineering efforts of human experts , allowing for the creation of powerful architectures for widespread adoption ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2015 ; He et al. , 2016 ; Szegedy et al. , 2016 ; Huang et al. , 2017 ) . However , this manual design is costly , and becomes increasingly more difficult as networks get larger and more complicated . Because of these challenges , the neural network community has seen a shift from designing architectures to designing algorithms that search for candidate architectures ( Elsken et al. , 2019 ; Wistuba et al. , 2019 ) . These Neural Architecture Search ( NAS ) algorithms are capable of automating the discovery of effective architectures ( Zoph & Le , 2017 ; Zoph et al. , 2018 ; Pham et al. , 2018 ; Tan et al. , 2019 ; Liu et al. , 2019 ; Real et al. , 2019 ) . NAS algorithms are broadly based on the seminal work of Zoph & Le ( 2017 ) . A controller network generates an architecture proposal , which is then trained to provide a signal to the controller through REINFORCE ( Williams , 1992 ) , which then produces a new proposal , and so on . Training a network for every controller update is extremely expensive ; utilising 800 GPUs for 28 days in Zoph & Le ( 2017 ) . Subsequent work has sought to ameliorate this by ( i ) learning stackable cells instead of whole networks ( Zoph et al. , 2018 ) and ( ii ) incorporating weight sharing ; allowing candidate networks to share weights to allow for joint training ( Pham et al. , 2018 ) . These contributions have accelerated the speed of NAS algorithms e.g . to half a day on a single GPU in Pham et al . ( 2018 ) . For some practitioners , NAS is still too slow ; being able to perform NAS quickly ( i.e . in seconds ) would be immensely useful in the hardware-aware setting where a separate search is typically required for each device and task ( Wu et al. , 2019 ; Tan et al. , 2019 ) . Moreover , recent works have scrutinised NAS with weight sharing ( Li & Talwalkar , 2019 ; Yu et al. , 2020 ) ; there is continued debate as to whether it is clearly better than simple random search . The issues of cost and time , and the risks of weight sharing could be avoided entirely if a NAS algorithm did not require any network training . In this paper , we show that this can be achieved . We explore two recently released NAS benchmarks , NAS-Bench-101 ( Ying et al. , 2019 ) , and NASBench-201 ( Dong & Yang , 2020 ) and examine the relationship between the linear maps induced by an untrained network for a minibatch of augmented versions of a single image ( Section 3 ) . These maps are easily computed using the Jacobian . The correlations between these maps ( which we denote by ΣJ ) are distinctive for networks that perform well when trained on both NAS-Benches ; this is immediately apparent from visualisation alone ( Figure 1 ) . We devise a score based on ΣJ and perform an ablation study to demonstrate its robustness to inputs and network initialisation . We incorporate our score into a simple search algorithm that doesn ’ t require training ( Section 4 ) . This allows us to perform architecture search quickly , for example , on CIFAR-10 ( Krizhevsky , 2009 ) we are able to search for a network that achieve 93.36 % accuracy in 29 seconds within the NAS-Bench-201 search space ; several orders of magnitude faster than traditional NAS methods for a modest change in final accuracy ( e.g . REINFORCE finds a 93.85 % net in 12000 seconds ) . Finally , we show that we can combine our approach with regularised evolutionary search ( REA , Pham et al. , 2018 ) to produce a new NAS algorithm , Assisted-REA ( AREA ) that outperforms its precedessor , attaining 94.16 % accuracy on NAS-Bench-101 in 12,000 seconds . Code for reproducing our experiments is available in the supplementary material . We believe this work is an important proof-of-concept for NAS without training , and shows that the large resource costs associated with NAS can be avoided . The benefit is two-fold , as we also show that we can integrate our approach into existing NAS techniques for scenarios where obtaining as high an accuracy as possible is of the essence . 2 BACKGROUND . Designing a neural architecture by hand is a challenging and time-consuming task . It is extremely difficult to intuit where to place connections , or which operations to use . This has prompted an abundance of research into neural architecture search ( NAS ) ; the automation of the network design process . In the pioneering work of Zoph & Le ( 2017 ) , the authors use an RNN controller to generate descriptions of candidate networks . Candidate networks are trained , and used to update the controller using reinforcement learning to improve the quality of the candidates it generates . This algorithm is very expensive : searching for an architecture to classify CIFAR-10 required 800 GPUs for 28 days . It is also inflexible ; the final network obtained is fixed and can not be scaled e.g . for use on mobile devices or for other datasets . The subsequent work of Zoph et al . ( 2018 ) deals with these limitations . Inspired by the modular nature of successful hand-designed networks ( Simonyan & Zisserman , 2015 ; He et al. , 2016 ; Huang et al. , 2017 ) , they propose searching over neural building blocks , instead of over whole architectures . These building blocks , or cells , form part of a fixed overall network structure . Specifically , the authors learn a standard cell , and a reduced cell ( incorporating pooling ) for CIFAR-10 classification . These are then used as the building blocks of a larger network for ImageNet ( Russakovsky et al. , 2015 ) classification . While more flexible—the number of cells can be adjusted according to budget— and cheaper , owing to a smaller search space , this technique still utilises 500 GPUs across 4 days . ENAS ( Pham et al. , 2018 ) reduces the computational cost of searching by allowing multiple candidate architectures to share weights . This facilitates the simultaneous training of candidates , reducing the search time on CIFAR-10 to half a day on a single GPU . Weight sharing has seen widespread adoption in a host of NAS algorithms ( Liu et al. , 2019 ; Luo et al. , 2018 ; Cai et al. , 2019 ; Xie et al. , 2019 ; Brock et al. , 2018 ) . However , there is evidence that it inhibits the search for optimal architectures ( Yu et al. , 2020 ) . Moreover , random search proves to be an extremely effective NAS baseline ( Yu et al. , 2020 ; Li & Talwalkar , 2019 ) . This exposes another problem : the search space is still vast—there are 1.6× 1029 possible architectures in Pham et al . ( 2018 ) for example—that it is impossible to isolate the best networks and demonstrate that NAS algorithms find them . An orthogonal direction for identifying good architectures is the estimation of accuracy prior to training ( Deng et al. , 2017 ; Istrate et al. , 2019 ) , although these differ from this work in that they rely on training a predictive model , rather than investigating more fundamental architectural properties . 2.1 NAS BENCHMARKS . A major barrier to evaluating the effectiveness of a NAS algorithm is that the search space ( the set of all possible networks ) is too large for exhaustive evaluation . Moreover , popular search spaces have been shown to be over-engineered , exhibiting little variety in their trained networks ( Yang et al. , 2020 ) . This has led to the creation of several benchmarks ( Ying et al. , 2019 ; Zela et al. , 2020 ; Dong & Yang , 2020 ) that consist of tractable NAS search spaces , and metadata for the training of networks within that search space . Concretely , this means that it is now possible to determine whether an algorithm is able to search for a good network . In this work we utilise NAS-Bench-101 ( Ying et al. , 2019 ) and NAS-Bench-201 ( Dong & Yang , 2020 ) to evaluate the effectiveness of our approach . NAS-Bench-101 consists of 423,624 neural networks that have been trained exhaustively , with three different initialisations , on the CIFAR-10 dataset for 108 epochs . NAS-Bench-201 consists of 15,625 networks trained multiple times on CIFAR-10 , CIFAR-100 , and ImageNet-16-120 ( Chrabaszcz et al. , 2017 ) . Both benchmarks are described in detail in Appendix B . 3 SCORING NETWORKS AT INITIALISATION . Our goal is to devise a means to score a network architecture at initialisation in a way that is indicative of its final trained accuracy . This can either replace the expensive inner-loop training step in NAS , or better direct exploration in existing NAS algorithms . Given a neural network with rectified linear units , we can , at each unit in each layer , identify a binary indicator as to whether the unit is inactive ( the value is negative and hence is multiplied by zero ) or active ( in which case its value is multiplied by one ) . Fixing these indicator variables , it is well known that the network is now locally defined by a linear operator ( Hanin & Rolnick , 2019 ) ; this operator is obtained by multiplying the linear maps at each layer interspersed with the binary rectification units . Consider a minibatch of data X = { xi } Ni=1 . Let us denote the linear map for input xi ∈ RD by column vector wi , which maps the input through the network f ( xi ) to a final choice of scalar representation zi ∈ R1 . This linear map can be easily computed using the Jacobian wi = ∂f ( xi ) ∂x . How differently a network acts at each data point can be summarised by comparing the corresponding local linear operators . Correlated operators for nearby points ( such as small perturbations from a training point ) relate to a potential difficulty in handling the two points differently during learning . The Frobenius inner product Tr [ ( wi − µi ) T ( wj − µj ) ] provides a natural basis for defining how two linear operators corresponding to data points xi and xj covary ( µ are mean Jacobian elements , and usually close to zero ) . We can examine the correspondences for the whole minibatch by computing J = ( ∂f ( x1 ) ∂x ∂f ( x2 ) ∂x · · · ∂f ( xN ) ∂x ) > ( 1 ) and observing the covariance matrix CJ = ( J −MJ ) ( J −MJ ) T where MJ is the matrix with entries ( MJ ) i , t = 1 D ∑D d=1 Ji , d ∀t , where d , t index over the D elements of each input ( i.e . channels × pixels ) . It is more salient to focus on the the correlation matrix ΣJ as the appropriate scaling in input space around each point is arbitrary . The ( i , j ) th element of ΣJ is given by ( ΣJ ) i , j = ( CJ ) i , j√ ( CJ ) i , i ( CJ ) j , j . We want an untrained neural network to be sufficiently flexible to model a complex target function . However , we also want a network to be invariant to small perturbations . These two requirements are antagonistic . For an untrained neural network to be sufficiently flexible it would need to be able to distinguish the local linear operators associated with each data point : if two are the same then the two points are coupled . To be invariant to small perturbations the same local linear operators would need to be weakly coupled . Ideally a network would have low correlated local maps associated with each data point to be able to model each local region . We empirically demonstrate this by computing ΣJ for a random subset of NAS-Bench101 ( Ying et al. , 2019 ) and NAS-Bench-201 ( Dong & Yang , 2020 ) networks at initialisation for a minibatch of a single CIFAR-10 image replicated 256 times with a different cutout ( DeVries & Taylor , 2017 ) perturbation applied to each replicant . We use torchvision.transforms.RandomErasing ( p=0.9 , scale= ( 0.02 , 0.04 ) ) in our experiments . To form J we flatten the Jacobian for each input ( so D = 3× 32× 32 = 3072 ) , and adjust the final classifier layer to output a scalar . The plots of the histograms of ΣJ for different networks , categorised according to the validation accuracy when trained is given in Figure 1 for a sample of networks in both benchmarks . Further plots are given in Appendix A . The histograms are very distinct : high performing networks in both benchmarks have their mass tightly around zero with a small positive skew . We can therefore use these histograms to predict the final performance of untrained networks , in place of the expensive training step in NAS . Specifically , we score networks by counting the entries in ΣJ that lie between 0 and an small upper bound β . A ΣJ where inputs are marginally positively-correlated will have a higher score . Our score is given by S = ∑ i , j 1 ( 0 < ( ΣJ ) i , j < β ) ( 2 ) where 1 is the indicator function . In this work we set β = 14 . An overview is provided in Figure 2 . To be clear , we are not claiming this score is particularly optimal ; rather we use it to demonstrate that there are ways of scoring untrained networks that provide significant value for architecture search . We sample 1000 different architectures at random from NAS-Bench-101 and NAS-Bench-201 and plot our score on the untrained network versus their validation accuracies when trained for the datasets in these benchmarks in Figure 3 . In all cases there is a strong correlation between our score and the final accuracy , although it is noisier for ImageNet-16-120 ; this dataset has smaller images compared to the other datasets so it may be that different cutout parameters would improve this . In Section 4 we demonstrate how our score can be used in a NAS algorithm for extremely fast search .

Summary: This paper attempts to infer a network's accuracy at initialization without training it, which can speed up neural architecture search and greatly reduce the search cost. Specifically, they propose a metric based on the Jacobian of the loss with respect to a minibatch of input data. The authors show that with this metric, they can find architectures with reasonable accuracy on CIFAR-10/CIFAR-100 in the NAS-Bench-201, while using much less search cost compared to previous NAS methods.