One-Shot Neural Architecture Search via Compressive Sensing

1 INTRODUCTION . Motivation . Choosing a suitable neural network architecture for complex prediction tasks such as image classification and language modeling often requires a substantial effort of trial-and-error . Therefore , there has been a growing interest to automatically learn ( or meta-learn ) the architecture of neural networks that can achieve competitive ( or better ) results over hand-designed architectures . The sub-field of neural architecture search ( NAS ) addresses the problem of designing competitive architectures with as small computational budget as possible . Numerous approaches for neural architecture search already exist in the literature , each with their own pros and cons : these include black-box optimization based on reinforcement learning ( RL ) ( Zoph & Le , 2017 ) , evolutionary search ( Real et al. , 2019 ) , and Bayesian optimization ( Cao et al. , 2019 ; Kandasamy et al. , 2018 ) . Though the algorithmic details vary , most of these NAS methods face the common challenge of evaluating the test/validation performance of a ( combinatorially ) large number of candidate architecture evaluations . Our Contributions . In a departure from traditional methods , we approach the NAS problem via the lens of compressive sensing . The field of compressive sensing ( or sparse recovery ) , introduced by the seminal works of ( Candes et al. , 2006 ; Donoho et al. , 2006 ) , has received significant attention in both ML theory and applications over the last decade , and has influenced the development of numerous advances in nonlinear and combinatorial optimization . We leverage these advances for the NAS problem . In particular , we develop a new NAS method called CoNAS ( Compressive sensing-based Neural Architecture Search ) , which merges ideas from sparse recovery with so-called “ one-shot ” architecture search methods ( Bender et al. , 2018 ) , described in greater detail below . CoNAS consists of two new innovations : ( i ) a new search space that permits exploration of a large ( r ) number of diverse candidate architectures , and ( ii ) a new search strategy that borrows ideas from recovery of Boolean functions from their ( sparse ) Fourier expansions . Our experiments show that CoNAS is able to discover a deep convolutional neural network with test error 2.74± 0.12 % on CIFAR-10 classification , outperforming existing state-of-the-art methods , including DARTs ( Liu et al. , 2018b ) , ENAS ( Pham et al. , 2018 ) , random search with weight-sharing ( RSWS ) ( Li & Talwalkar , 2019 ) , and the baseline vanilla random search method ( Liu et al. , 2018b ) in terms of test error , search time , model size , and number of multiply-add operations . Moreover , CoNAS can achieve the comparable performance as NASNet ( Zoph et al. , 2018 ) and AmoebaNet ( Real et al. , 2019 ) with less than one GPU-day of computation . Our experiments on designing recurrent neural networks for language modeling are somewhat short of the state-of-the-art ( Zilly et al. , 2017 ) , but we find that CoNAS still finds competitive results with less search time than previous NAS approaches . Our results are exactly reproducible ( having been trained with fixed pseudorandom seeds ) , and an implementation of CoNAS will be made publicly available post-peer review . Finally , while our original motivation was to devise an empirically useful NAS method , a nice benefit is that CoNAS can also be theoretically analyzed , since existing theoretical results for Fourier-sparse Boolean functions can be ported over in order to provide upper bounds for the required number of performance evaluations of subarchitectures of the one-shot model . This , to our knowledge , is one of the first results of their kind in the NAS literature and may be of independent interest . We defer discussion of our approach to Section 3 . Our Techniques . The intuition behind compressive sensing is that if a signal ( or function ) can be represented via a sparse basis expansion , then it can be recovered ( either exactly or approximately ) from a small number of randomized measurements . CoNAS leverages this intuition in the context of one-shot architecture search ( Bender et al. , 2018 ) . In one-shot NAS , instead of evaluating several candidate architectures , a single “ base ” neural network model is pre-trained ; a class of sub-networks is identified ( called the search space ) and the performance of each sub-network is evaluated on a validation set ; and the best-performing sub-network is finally selected and fine-tuned . Following Hazan et al . ( 2017 ) ; Stobbe & Krause ( 2012 ) , we model the sub-network selection as a sparse recovery problem . Concretely , consider a function f that maps sub-architectures to a measure of performance ( validation loss ) . We assume that f can be written as a sparse , low-degree polynomial in the ( discrete ) Fourier basis . If the sparsity assumption is satisfied , then we claim the function f can be reconstructed using a very small number of sub-network evaluations , thus reducing overall compute time . A key challenge lies in defining a suitable search space ; we propose one that is considerably larger than the one used in DARTS or ENAS , allowing us to ( putatively ) search over a more diverse set of candidate architectures . 2 BACKGROUND . We briefly describe one-shot neural architecture search techniques ( Bender et al. , 2018 ; Li & Talwalkar , 2019 ) ; a supplementary description is available in Appendix A.1 . Following the treatment given in the recent survey paper ( Elsken et al. , 2018 ) , one-shot NAS approaches have three main components : a search space , a search strategy , and a performance estimation strategy . Search Space . The goal of one-shot NAS is to find the best performing cell , a fundamental component from which more complex architectures are constructed via stacking . Following Liu et al . ( 2018b ) , a cell is a directed acyclic graph ( DAG ) where a node corresponds to the latent representation , and a directed edge transforms predecessor nodes using a given operation ; common operations used in CNNs include 3× 3 and 5× 5 separable convolutions , 3× 3 max pooling , and 3× 3 average pooling . Each cell has two input nodes and one output node , and intermediate nodes can only be connected by predecessor nodes including input nodes . Intermediate nodes are wired to two predecessor nodes in CNNs and one predecessor node in RNNs . Search Strategy and Performance Estimation Strategy . Having defined a search space , oneshot NAS approaches employ four steps : ( i ) train a single “ one-shot ” base model that is capable of predicting the performance of sub-architectures1 ; ( ii ) randomly sample sub-architectures of a trained one-shot model and measure performance over a hold-out validation set of samples ; ( iii ) select the candidate ( cell ) with best validation performance ; ( iv ) retrain a deeper final architecture using the best cell . Using the one-shot model as the proxy measurements of the candidate architecture corresponds to the performance estimation strategy ( first and second step ) . Fourier analysis of Boolean functions . We follow the treatment given in O ’ Donnell ( 2014 ) . A real-valued Boolean function is one that maps n-bit binary vectors ( i.e. , nodes of the hypercube ) to real values : f : { −1 , 1 } n → R. Such functions can be represented in a basis comprising real multilinear polynomials called the Fourier basis , defined as follows . ( We denote the vectors with bold letters . Also , [ n ] denotes the set { 1 , 2 . . . , n } . ) Definition 2.1 . For S ⊆ [ n ] , define the parity function χS : { −1 , 1 } n → { −1 , 1 } such that χS ( α ) = ∏ i∈S αi . Then , the Fourier basis is the set of all 2 n parity functions { χS } . The key fact is that the basis of parity functions forms an K-bounded orthonormal system ( BOS ) with K = 1 , therefore satisfying two properties : 〈χS , χT 〉 = { 1 , if S = T 0 , if S 6= T and supα∈ { −1,1 } n |χS ( α ) | ≤ 1 for all S ⊆ [ n ] , ( 2.1 ) Due to orthonormality , any Boolean function f has a unique Fourier representation , given by f ( α ) = ∑ S⊆ [ n ] f̂ ( S ) χS ( α ) , with Fourier coefficients f̂ ( S ) = Eα∈ { −1,1 } n [ f ( α ) χS ( α ) ] where expectation is taken with respect to the uniform distribution over the nodes of the hypercube . A modeling assumption is that the Fourier spectrum of the function is concentrated on monomials of small degree ( ≤ d ) . This corresponds to the case where f is a decision tree ( Hazan et al. , 2017 ) , and allows us to simplify the Fourier expansion by limiting its support . Let Pd ⊆ 2 [ n ] be a fixed collection of Fourier basis such that Pd : = { χS ⊆ 2 [ n ] : |S| ≤ d } . Then Pd ⊆ 2 [ n ] induces a function space consisting of all functions of order d or less , denoted byHPd : = { f : Supp [ f̂ ] ⊆ Pd } . For example , P2 allows us to express the function f with at most ∑d l=0 ( n l ) ≡ O ( n2 ) Fourier coefficients . Lastly , if we have prior knowledge of some set of bits J , we use an operation called restriction . Definition 2.2 . Let f : { −1 , 1 } n → R , ( J , J ) be a partition of [ n ] , and z ∈ { −1 , 1 } J . The restriction of f to J using z denoted by fJ|z : { −1 , 1 } J → R is the subfunction of f given by fixing the coordinates in J to the bit values z . 3 PROPOSED ALGORITHM : CONAS . Overview . Our proposed algorithm , Compressive sensing-based Neural Architecture Search ( CoNAS ) , infuses ideas from learning a sparse graph ( Boolean Fourier analysis ) into one-shot NAS . CoNAS consists of two novel components : an expanded search space , and a more effective search strategy . Search Space . Our first ingredient is an expanded search space . Following the approach of DARTS ( Liu et al. , 2018b ) , we define a directed acyclic graph ( DAG ) where all predecessor nodes are connected to every intermediate node with all possible operations . We represent any sub-graph of the DAG using a binary string α called the architecture encoder . Its length is the total number of edges in the DAG , and a 1 ( resp . −1 ) in α indicates an active ( resp . inactive ) edge . 1We note that the quality of sub-network performance predictions is heavily dependent on the base model that is trained . Choosing the correct base model is itself a separate challenge , which earlier papers such as Bender et al . ( 2018 ) have addressed in detail . We do not pursue that direction here since our focus is on the sub-network selection problem , and assume that the base model is well-trained . Figure 2 gives an example of how the architecture encoder α samples the sub-architecture of the fully-connected model in case of a convolutional neural network . The goal of CoNAS is to find the “ best ” encoder α∗ , which is ” close enough ” to the global optimum returning the best validation accuracy by constructing the final model with α∗ encoded sub-graph . Since each edge can be switched on and off independently , the proposed search space allows exploring a cell with more diverse connectivity patterns than DARTS Liu et al . ( 2018b ) . Moreover , the number of possible configurations exceeds similar previously proposed search spaces with constrained wiring rules Li & Talwalkar ( 2019 ) ; Pham et al . ( 2018 ) ; Real et al . ( 2019 ) ; Zoph et al . ( 2018 ) . Search Strategy . We propose a compressive measuring strategy to approximate the one-shot model with a Fourier-sparse Boolean function . Let f : { −1 , 1 } n → R map the sub-graph of the one-shot pre-trained model encoded by α to its validation performance . Similar to Hazan et al . ( 2017 ) , we collect a small number of function evaluations of f , and reconstruct the Fourier-sparse function g ≈ f via sparse recovery algorithms with randomly sampled measurements . Then , we solve argminα g ( α ) by exhaustive enumeration over all coordinates in its sub-cube { −1 , 1 } J where ( J , J ) partitions [ n ] ( Definition 2.2 ) 2 . If the solution of the argminα g ( α ) does not return enough edges to construct the cell ( some intermediate nodes are disconnected ) , we simply connect the intermediate nodes to the previous cell output , Cellk−2 , using the Identity operation ( this does not increase neither the model size nor number of multiply-add operations ) . Larger cells can be found from multiple iterations by restricting the approximate function g and with fixing the bit values found in the previous solution , and randomly sampling sub-graphs in the remaining edges . Full Algorithm . We now describe CoNAS in detail , with pseudocode provided in Appendix A.3 . We first train a one-shot model with standard backpropagation but only updates the weight corresponding to the randomly sampled sub-graph edges for each minibatch . Then , we randomly sample sub-graphs by generating architecture encoder strings α ∈ { −1 , 1 } n using a Bernoulli ( p ) distribution for each bit of α independently ( We set p = 0.5 ) . In the second stage , we collect m measurements of randomly sampled sub-architecture performance denoted by y = ( f ( α1 ) , f ( α2 ) , . . . , f ( αm ) ) T . Next , we construct the graph-sampling matrix A ∈ { −1 , 1 } m×|Pd| with entries Al , k = χSk ( αl ) , l ∈ [ m ] , k ∈ [ |Pd| ] , S ⊆ [ n ] , |S| ≤ d , ( 3.1 ) where d is the maximum degree of monomials in the Fourier expansion , and Sk is the index set corresponding to kth Fourier basis . 2This is similar to the idea of de-biasing in the Hard-Thresholding ( HT ) algorithm ( Foucart & Rauhut , 2017 ) where the support is first estimated , and then within the estimated support , the coefficients are calculated through least-squares estimation . We solve the familiar Lasso problem ( Tibshirani , 1996 ) : x∗ = argmin x∈R|Pd| ‖y −Ax‖22 + λ‖x‖1 , ( 3.2 ) to ( approximately ) recover the global optimizer x∗ , the vector contains the Fourier coefficients corresponding to Pd . We define an approximate function g ≈ f with Fourier coefficients with the top-s ( absolutely ) largest coefficients from x∗ , and compute α∗ = argminα g ( α ) , resulting all the possible points in the subcube defined by the support of g ( this computation is feasible if s is small ) . Multiple stages of sparse recovery ( with successive restrictions to previously obtained optimal α∗ ) enable us to approximate additional monomial terms . Finally , we obtain a cell to construct the final architecture by activating the edges corresponding to all i ∈ [ n ] such that α∗i = 1 . Theoretical support for CoNAS . The system of linear equations y = Ax with the graph-sampling matrix A ∈ { −1 , 1 } m×O ( nd ) , measurements y ∈ Rm , and Fourier coefficient vector x ∈ RO ( nd ) is an ill-posed problem when m O ( nd ) for large n. However , if the graph-sampling matrix satisfies Restricted Isometry Property ( RIP ) , the sparse coefficients , u can be recovered : Definition 3.1 . A matrix A ∈ Rm×O ( nd ) satisfies the restricted isometry property of order s with some constant δ if for every s-sparse vector u ∈ RO ( nd ) ( i.e. , only s entries are non-zero ) the following holds : ( 1− δ ) ‖u‖22 ≤ ‖Au‖22 ≤ ( 1 + δ ) ‖u‖22 . We defer the history of improvements on the upper bounds of the number of rows from bounded orthonormal dictionaries ( matrix A ) for which A is guaranteed to satisfy the restricted isometry property with high probability in Appendix A.2 . To the best of our knowledge , the best known result with mild dependency on δ ( i.e. , δ−2 ) is due to Haviv & Regev ( 2017 ) , which we can apply for our setup . It is easy to check that the graph-sampling matrix A in our proposed CoNAS algorithm satisfies BOS for K = 1 ( Eq 3.1 ) . Theorem 3.2 . Let the graph-sampling matrix A ∈ { −1 , 1 } m×O ( nd ) be constructed by taking m rows ( random sampling points ) uniformly and independently from the rows of a square matrix M ∈ { −1 , 1 } O ( nd ) ×O ( nd ) . Then the normalized matrix A with m = O ( log2 ( 1δ ) δ −2s log2 ( sδ ) d log ( n ) ) with probability at least 1 − 2−Ω ( d logn log ( sδ ) ) satisfies the restricted isometry property of order s with constant δ ; as a result , every s-sparse vector u ∈ RO ( nd ) can be recovered from the sample yi ’ s : y = Au = ( |O ( nd ) |∑ j=1 ujAi , j ) m i=1 , by LASSO ( equation 3.2 ) . Proof . First , we note that the graph-sampling matrix A is a BOS matrix with K = 1 ; hence , directly invoking Theorem 4.5 of Haviv & Regev ( 2016 ) to our setting , we can see that matrix A satisfies RIP . Now according to Theorem 1.1 of Candes ( 2008 ) , letting δ < √ 2− 1 , the l1 minimization or LASSO will recover exactly the s sparse vector u . For instance , in our experiments , we have selected m = 1000 which is consistent with our parameters , d = 2 , s = 10 , n = 140 . Here , it is worthwhile to mention two points : first , the above upper bound on the number of rows of the graph-sampling matrix A is the tightest bound ( according to our knowledge ) for the BOS matrices to satisfy RIP . There exist series of results establishing the RIP for BOS matrices during the last 15 year . We have reviewed these results in the appendix A.2 . Second , instead of LASSO , one can use any sparse recovery method ( such as IHT ( Blumensath & Davies , 2009 ) ) in our algorithm . In essence , Theorem 3.2 provides a successful guarantee for recovering the optimal sub-network of a given size given a sufficient number of performance measurements .

Summary: This paper proposes a new algorithm for one-shot neural architecture search (NAS) via compressive sensing. The authors propose a new search strategy, as well as a slightly different search space compared to DARTS [1], ProxylessNAS [2], etc. They use architecture samples from the one-shot model evaluated with the search parameters as a surrogate of the true objective in order to speed-up the search. Afterwards, these surrogate function evaluations are used to compute Fourier coefficients which are eventually used to optimize the vector of binary parameters encoding the architecture.