Neural Architecture Search without Training

1 INTRODUCTION . The success of deep learning in computer vision is in no small part due to the insight and engineering efforts of human experts , allowing for the creation of powerful architectures for widespread adoption ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2015 ; He et al. , 2016 ; Szegedy et al. , 2016 ; Huang et al. , 2017 ) . However , this manual design is costly , and becomes increasingly more difficult as networks get larger and more complicated . Because of these challenges , the neural network community has seen a shift from designing architectures to designing algorithms that search for candidate architectures ( Elsken et al. , 2019 ; Wistuba et al. , 2019 ) . These Neural Architecture Search ( NAS ) algorithms are capable of automating the discovery of effective architectures ( Zoph & Le , 2017 ; Zoph et al. , 2018 ; Pham et al. , 2018 ; Tan et al. , 2019 ; Liu et al. , 2019 ; Real et al. , 2019 ) . NAS algorithms are broadly based on the seminal work of Zoph & Le ( 2017 ) . A controller network generates an architecture proposal , which is then trained to provide a signal to the controller through REINFORCE ( Williams , 1992 ) , which then produces a new proposal , and so on . Training a network for every controller update is extremely expensive ; utilising 800 GPUs for 28 days in Zoph & Le ( 2017 ) . Subsequent work has sought to ameliorate this by ( i ) learning stackable cells instead of whole networks ( Zoph et al. , 2018 ) and ( ii ) incorporating weight sharing ; allowing candidate networks to share weights to allow for joint training ( Pham et al. , 2018 ) . These contributions have accelerated the speed of NAS algorithms e.g . to half a day on a single GPU in Pham et al . ( 2018 ) . For some practitioners , NAS is still too slow ; being able to perform NAS quickly ( i.e . in seconds ) would be immensely useful in the hardware-aware setting where a separate search is typically required for each device and task ( Wu et al. , 2019 ; Tan et al. , 2019 ) . Moreover , recent works have scrutinised NAS with weight sharing ( Li & Talwalkar , 2019 ; Yu et al. , 2020 ) ; there is continued debate as to whether it is clearly better than simple random search . The issues of cost and time , and the risks of weight sharing could be avoided entirely if a NAS algorithm did not require any network training . In this paper , we show that this can be achieved . We explore two recently released NAS benchmarks , NAS-Bench-101 ( Ying et al. , 2019 ) , and NASBench-201 ( Dong & Yang , 2020 ) and examine the relationship between the linear maps induced by an untrained network for a minibatch of augmented versions of a single image ( Section 3 ) . These maps are easily computed using the Jacobian . The correlations between these maps ( which we denote by ΣJ ) are distinctive for networks that perform well when trained on both NAS-Benches ; this is immediately apparent from visualisation alone ( Figure 1 ) . We devise a score based on ΣJ and perform an ablation study to demonstrate its robustness to inputs and network initialisation . We incorporate our score into a simple search algorithm that doesn ’ t require training ( Section 4 ) . This allows us to perform architecture search quickly , for example , on CIFAR-10 ( Krizhevsky , 2009 ) we are able to search for a network that achieve 93.36 % accuracy in 29 seconds within the NAS-Bench-201 search space ; several orders of magnitude faster than traditional NAS methods for a modest change in final accuracy ( e.g . REINFORCE finds a 93.85 % net in 12000 seconds ) . Finally , we show that we can combine our approach with regularised evolutionary search ( REA , Pham et al. , 2018 ) to produce a new NAS algorithm , Assisted-REA ( AREA ) that outperforms its precedessor , attaining 94.16 % accuracy on NAS-Bench-101 in 12,000 seconds . Code for reproducing our experiments is available in the supplementary material . We believe this work is an important proof-of-concept for NAS without training , and shows that the large resource costs associated with NAS can be avoided . The benefit is two-fold , as we also show that we can integrate our approach into existing NAS techniques for scenarios where obtaining as high an accuracy as possible is of the essence . 2 BACKGROUND . Designing a neural architecture by hand is a challenging and time-consuming task . It is extremely difficult to intuit where to place connections , or which operations to use . This has prompted an abundance of research into neural architecture search ( NAS ) ; the automation of the network design process . In the pioneering work of Zoph & Le ( 2017 ) , the authors use an RNN controller to generate descriptions of candidate networks . Candidate networks are trained , and used to update the controller using reinforcement learning to improve the quality of the candidates it generates . This algorithm is very expensive : searching for an architecture to classify CIFAR-10 required 800 GPUs for 28 days . It is also inflexible ; the final network obtained is fixed and can not be scaled e.g . for use on mobile devices or for other datasets . The subsequent work of Zoph et al . ( 2018 ) deals with these limitations . Inspired by the modular nature of successful hand-designed networks ( Simonyan & Zisserman , 2015 ; He et al. , 2016 ; Huang et al. , 2017 ) , they propose searching over neural building blocks , instead of over whole architectures . These building blocks , or cells , form part of a fixed overall network structure . Specifically , the authors learn a standard cell , and a reduced cell ( incorporating pooling ) for CIFAR-10 classification . These are then used as the building blocks of a larger network for ImageNet ( Russakovsky et al. , 2015 ) classification . While more flexible—the number of cells can be adjusted according to budget— and cheaper , owing to a smaller search space , this technique still utilises 500 GPUs across 4 days . ENAS ( Pham et al. , 2018 ) reduces the computational cost of searching by allowing multiple candidate architectures to share weights . This facilitates the simultaneous training of candidates , reducing the search time on CIFAR-10 to half a day on a single GPU . Weight sharing has seen widespread adoption in a host of NAS algorithms ( Liu et al. , 2019 ; Luo et al. , 2018 ; Cai et al. , 2019 ; Xie et al. , 2019 ; Brock et al. , 2018 ) . However , there is evidence that it inhibits the search for optimal architectures ( Yu et al. , 2020 ) . Moreover , random search proves to be an extremely effective NAS baseline ( Yu et al. , 2020 ; Li & Talwalkar , 2019 ) . This exposes another problem : the search space is still vast—there are 1.6× 1029 possible architectures in Pham et al . ( 2018 ) for example—that it is impossible to isolate the best networks and demonstrate that NAS algorithms find them . An orthogonal direction for identifying good architectures is the estimation of accuracy prior to training ( Deng et al. , 2017 ; Istrate et al. , 2019 ) , although these differ from this work in that they rely on training a predictive model , rather than investigating more fundamental architectural properties . 2.1 NAS BENCHMARKS . A major barrier to evaluating the effectiveness of a NAS algorithm is that the search space ( the set of all possible networks ) is too large for exhaustive evaluation . Moreover , popular search spaces have been shown to be over-engineered , exhibiting little variety in their trained networks ( Yang et al. , 2020 ) . This has led to the creation of several benchmarks ( Ying et al. , 2019 ; Zela et al. , 2020 ; Dong & Yang , 2020 ) that consist of tractable NAS search spaces , and metadata for the training of networks within that search space . Concretely , this means that it is now possible to determine whether an algorithm is able to search for a good network . In this work we utilise NAS-Bench-101 ( Ying et al. , 2019 ) and NAS-Bench-201 ( Dong & Yang , 2020 ) to evaluate the effectiveness of our approach . NAS-Bench-101 consists of 423,624 neural networks that have been trained exhaustively , with three different initialisations , on the CIFAR-10 dataset for 108 epochs . NAS-Bench-201 consists of 15,625 networks trained multiple times on CIFAR-10 , CIFAR-100 , and ImageNet-16-120 ( Chrabaszcz et al. , 2017 ) . Both benchmarks are described in detail in Appendix B . 3 SCORING NETWORKS AT INITIALISATION . Our goal is to devise a means to score a network architecture at initialisation in a way that is indicative of its final trained accuracy . This can either replace the expensive inner-loop training step in NAS , or better direct exploration in existing NAS algorithms . Given a neural network with rectified linear units , we can , at each unit in each layer , identify a binary indicator as to whether the unit is inactive ( the value is negative and hence is multiplied by zero ) or active ( in which case its value is multiplied by one ) . Fixing these indicator variables , it is well known that the network is now locally defined by a linear operator ( Hanin & Rolnick , 2019 ) ; this operator is obtained by multiplying the linear maps at each layer interspersed with the binary rectification units . Consider a minibatch of data X = { xi } Ni=1 . Let us denote the linear map for input xi ∈ RD by column vector wi , which maps the input through the network f ( xi ) to a final choice of scalar representation zi ∈ R1 . This linear map can be easily computed using the Jacobian wi = ∂f ( xi ) ∂x . How differently a network acts at each data point can be summarised by comparing the corresponding local linear operators . Correlated operators for nearby points ( such as small perturbations from a training point ) relate to a potential difficulty in handling the two points differently during learning . The Frobenius inner product Tr [ ( wi − µi ) T ( wj − µj ) ] provides a natural basis for defining how two linear operators corresponding to data points xi and xj covary ( µ are mean Jacobian elements , and usually close to zero ) . We can examine the correspondences for the whole minibatch by computing J = ( ∂f ( x1 ) ∂x ∂f ( x2 ) ∂x · · · ∂f ( xN ) ∂x ) > ( 1 ) and observing the covariance matrix CJ = ( J −MJ ) ( J −MJ ) T where MJ is the matrix with entries ( MJ ) i , t = 1 D ∑D d=1 Ji , d ∀t , where d , t index over the D elements of each input ( i.e . channels × pixels ) . It is more salient to focus on the the correlation matrix ΣJ as the appropriate scaling in input space around each point is arbitrary . The ( i , j ) th element of ΣJ is given by ( ΣJ ) i , j = ( CJ ) i , j√ ( CJ ) i , i ( CJ ) j , j . We want an untrained neural network to be sufficiently flexible to model a complex target function . However , we also want a network to be invariant to small perturbations . These two requirements are antagonistic . For an untrained neural network to be sufficiently flexible it would need to be able to distinguish the local linear operators associated with each data point : if two are the same then the two points are coupled . To be invariant to small perturbations the same local linear operators would need to be weakly coupled . Ideally a network would have low correlated local maps associated with each data point to be able to model each local region . We empirically demonstrate this by computing ΣJ for a random subset of NAS-Bench101 ( Ying et al. , 2019 ) and NAS-Bench-201 ( Dong & Yang , 2020 ) networks at initialisation for a minibatch of a single CIFAR-10 image replicated 256 times with a different cutout ( DeVries & Taylor , 2017 ) perturbation applied to each replicant . We use torchvision.transforms.RandomErasing ( p=0.9 , scale= ( 0.02 , 0.04 ) ) in our experiments . To form J we flatten the Jacobian for each input ( so D = 3× 32× 32 = 3072 ) , and adjust the final classifier layer to output a scalar . The plots of the histograms of ΣJ for different networks , categorised according to the validation accuracy when trained is given in Figure 1 for a sample of networks in both benchmarks . Further plots are given in Appendix A . The histograms are very distinct : high performing networks in both benchmarks have their mass tightly around zero with a small positive skew . We can therefore use these histograms to predict the final performance of untrained networks , in place of the expensive training step in NAS . Specifically , we score networks by counting the entries in ΣJ that lie between 0 and an small upper bound β . A ΣJ where inputs are marginally positively-correlated will have a higher score . Our score is given by S = ∑ i , j 1 ( 0 < ( ΣJ ) i , j < β ) ( 2 ) where 1 is the indicator function . In this work we set β = 14 . An overview is provided in Figure 2 . To be clear , we are not claiming this score is particularly optimal ; rather we use it to demonstrate that there are ways of scoring untrained networks that provide significant value for architecture search . We sample 1000 different architectures at random from NAS-Bench-101 and NAS-Bench-201 and plot our score on the untrained network versus their validation accuracies when trained for the datasets in these benchmarks in Figure 3 . In all cases there is a strong correlation between our score and the final accuracy , although it is noisier for ImageNet-16-120 ; this dataset has smaller images compared to the other datasets so it may be that different cutout parameters would improve this . In Section 4 we demonstrate how our score can be used in a NAS algorithm for extremely fast search .

Summary: The paper mainly introduces a metric to benchmark the performance of neural networks without training – the correlation of Jacobian subject to different augmented versions of a single image. The key motivation is, high-performance networks tend to represent data of small perturbations with different hyperplanes at initialization, so that the distinguishing capability may also be stronger. The divergence of the hyperplanes can be efficiently estimated via the correlation of the Jacobian, thus quantified by the score in Eq. 2. The effectiveness of the proposed metric is mainly verified in two NAS benchmarks (NAS-Bench-101 and NAS-Bench-201), whose correlation to the actual accuracy is relatively significant (Fig 3). Compared with existing NAS frameworks, the proposed method is very efficient, moreover, able to obtain competitive performance.