On Redundancy and Diversity in Cell-based Neural Architecture Search

1 INTRODUCTION . Neural Architecture Search ( NAS ) , which automates designs of neural networks by task , has seen enormous advancements since its invention . In particular , cell-based NAS has become an important technique in NAS research : in contrast to attempts that directly aim to design architectures at once and inspired by the classical manually-designed architectures like VGGNet and ResNet that feature repeated blocks , cell-based NAS similarly searches for repeated cells only and later stack them into full architectures . This simplification reduces the search space ( although it remains highly complex ) , and allows easy transfer of architectures across different tasks/datasets , application scenarios and resources constraints ( Elsken et al. , 2019 ) . Indeed , although alternative search spaces exist , cell-based NAS has received an extraordinary amount of research attention : based on our preliminary survey , almost 80 % of the papers ( detailed in App C ) proposing new NAS methods published in the top machine learning conferences ( ICLR , ICML , NEURIPS ) in the past year show at least one part of their major results in standard Differentiable Architecture Search ( DARTS ) cell-based spaces and/or highly related ones , and approx . 60 % demonstrate results in such spaces only ; it is fair to state that such cell-based spaces currently dominate . However , the lagging understanding of these architectures and the search space itself stands in stark contrast with the volume and pace of new search algorithms . The literature on understanding why this dominant search space works and comparing what different NAS methods have found is much more limited in depth and scope , with most existing works typically focusing exclusively on a few search methods or highlighting high-level patterns only ( Shu et al. , 2020 ; Zela et al. , 2020 ) . We argue that strengthening such an understanding is crucial on multiple fronts , and the lack thereof is concerning : first , studying a diverse set of architectures enables us to discover patterns shared across the search space , the foundation of comparison amongst search methods that heavily influences the resultant performances ( Yang et al. , 2020a ) : if the search space itself is flawed , designing and iterating new search methods on it , which is typically computationally intensive , can be misleading and unproductive . Conversely , understanding any pitfalls of the existing search spaces informs us on how to design better search spaces in the future , which is critical in advancing the goal of NAS in finding novel and high-performing architectures , not only in conventional CNNs but also in emerging NAS paradigms such as Transformers ( which may also take the form of a cell-based design space ) . Second , opening the NAS black box enables us to distill the essence of the strong-performing NAS architectures beneath their surface of complexity . Unlike manually designed architectures where usually designers attribute performance to specific designs , currently owing to the apparent complexity of the design space , the NAS architectures , while all discovered in a similar or identical search space , are often compared to in terms of final performance on a standard dataset only ( e.g . CIFAR-10 test error ) . This could be problematic , as we do not necessarily understand what NAS has discovered that led to the purported improvements , and the metric itself is a poor one on which external factors such as hyperparameter settings , variations in training/data augmentation protocols and even just noise could exert a greater influence than the architectural design itself ( Yang et al. , 2020a ) . However , by linking performance to specific designs , we could more ascertain whether any performance differences stem from the architectures rather than the interfering factors . We aim to address this problem by presenting a post-hoc analysis of the well-performing architectures produced by technically diverse search methods . Specifically , we utilise explainable machine learning tools to open the NAS black box by inspecting the good- and bad-performing architectures produced by a wide range of NAS search methods in the dominant DARTS search space . We find : • Performances of architectures can often be disproportionately attributed to a small number of simple yet critical features that resemble known patterns in classical network designs ; • Many designs almost universally adopted simply contribute to complexity but not performance ; • The nominal complexity of the search spaces poorly reflects the actual diversity of the ( high- performing ) architectures discovered , the functional parts of which are often very similar despite the technical diversity in search methods and the seeming disparity in topology . In fact , with few simple and human-interpretable constraints , almost any randomly sampled architecture can perform on par or exceed those produced by the state-of-the-art NAS methods over varying network sizes and datasets ( CIFAR-10/IMAGENET ) . Ultimately , these findings prompt us to rethink the suitability of the current standard protocol in evaluating NAS and the capability to find truly novel architectures within the search space . We finally provide suggestions for prospective new search spaces inspired by these findings . input0 input1 0 1 2 3 out avg_pool_3x3 ( ap3 ) max_pool_3x3 ( mp3 ) skip_connect ( skip ) dil_conv_5x5 ( d5 ) dil_conv_3x3 ( d3 ) sep_conv_5x5 ( s5 ) sep_conv_3x3 ( s3 ) x ( j ) = i < j o ( i , j ) ( x ( i ) ) and finally the out node concatenates all outputs from the intermediate nodes . As shown in Fig 1 , the DARTS cell allows for up to 14 possible spots to place operations . However , to ensure that all intermediate nodes are connected in the computational graph , each intermediate node is constrained to have exactly 2 in-edges connected to it from 2 different preceding nodes . Lastly , it is conventional to search for two cells , namely normal αn and reduce αr cells simultaneously , with αr placed at 1/3 and 2/3 of the total depth of the networks and αn everywhere else . In total , there are ∏4 k=1 72k ( k+1 ) 2 ≈ 10 9 distinct cells without considering graph isomorphism , and since both αn and αr are required to fully specify an architecture , there exist approx . ( 109 ) 2 = 1018 distinct architectures ( Liu et al. , 2019 ) . Other cells commonly used in are almost invariably closely related to , and can be viewed as simplified or more complicated variants of the DARTS space . For example , NASBench-201 ( NB201 ) search space used in the NAS benchmark is similar to but simplified from the DARTS cell ( detailed and analysed in App D ) . Other works have increased the number of intermediate nodes ( Wang et al. , 2021b ) , expanded the primitive pool A ( Hundt et al. , 2019 ) and/or relaxed other constraints ( e.g . Shi et al . ( 2020 ) allow both incoming edges of the intermediate nodes to be from the same preceding nodes ) , but do not fundamentally differ from the DARTS space . Additionally , search spaces like the NAS-Bench-101 ( NB101 ) ( Ying et al. , 2019a ) feature cells where the operations are represented as node features , although we believe that the way the cells are represented should not affect any findings we will demonstrate , as the DARTS/NB201 spaces can similarly be equivalently represented in a feature-on-node style ( Ru et al. , 2021 ; Pham et al. , 2018 ) . We do not experiment on NB101 in the present paper , as it features CIFAR-10 only similar to NAS-Bench-301 ( NB301 ) , but the latter is much closer in terms of size to a “ realistic ” search space the community expects and is more amenable to one-shot methods which are currently mainstream in NAS . 3 OPERATION-LEVEL ANALYSIS : REDUNDANCIES IN SEARCH SPACES . Cell-based NAS search space contains multiple sources of complexity ( deciding both the specific wiring and operations on the edges ; searching for 2 cells independently , etc . ) , each expanding the search space combinatorially . While this complexity is argued to be necessary for the search space to be expressive for good-performing novel architectures to be found , it is unknown whether all these sources of complexities are equally important , or whether the performance critically depend on some sub-features only . We argue that answering this question , and identifying such features , if they are indeed present , are highly significant : it enables us to separate the complexities that positively contribute to performances from those that do not , and subsequently would fundamentally affect the design of search methods . At individual architectures , it helps us understand what NAS has truly discovered , by removing confounding factors and focusing on the key aspects of the architectures . For findings to be generally applicable , we aim to be not specific to any search method ; this requires us to study a large set of architectures that are high-performing but technically diverse in terms of the search methods that produce them . Fortunately , the training set of NB301 ( Siems et al. , 2020 ) , which include > 50,000 architectureperformance pairs in the DARTS space using a combination of random sampling and more than 10 state-of-the-art yet technically diverse methods ( detailed in App . B.1 ) , could be used for such an analysis : to build a surrogate model that accurately predicts architecture performance across the entire space . We primarily focus on the top-5 % ( or top-2,589 ) architectures of the training set since we overwhelmingly care about the good-performing architectures by definition in NAS , although as we will show , the main findings hold true also for architectures found by methods not covered by NB301 and for other search spaces like the NB201 space . As shown in Fig 2 , the top architectures are well-spread out in the search space and are well-separated by search methods , seemingly suggesting that the architectures discovered by different methods are diverse in characteristics . Lastly , the worse-performing cells could also be of interest , as any features observed could be the ones we would actively like to avoid , and we analyse them in App . A . Operation Importance To untangle the influence of each part of the architecture , we introduce Operation Importance ( OI ) , which measures the incremental effect of the individual operations , which are the smallest possible features , to the overall performance . We quantify this via measuring the expected change in the performance by perturbing the type or wiring of the operation in question : considering an edge-attributed cell α with edge ei , j currently assigned with primitive ok , then the operation importance of ok on that edge of cell α is given by : OI ( α , ei , j : = ok ) = 1 |N ( α , ei , j : = ok ) | |N ( α , ei , j : =ok ) |∑ m=1 [ y ( αm ) ] − y ( α ) , ( 1 ) where y ( α ) denotes the validation accuracy or another appropriate performance metric of the fullytrained architecture induced by cell α . We are interested in measuring the importance of both the primitive choice and where the operation is located relative to the entire cell , and we use N ( α , ei , j = ok ) to denote a set of neighbour cells that differ to α only with the edge in question assigned with another primitive : ei , j ∈ A \ { ok } , or with the same primitive ok but with one end node of ei , j being rewired to another node , subjected to any constraints in the search space . It is worth noting that OI is an instance of the Permutation Feature Importance ( PFI ) ( Breiman , 2001 ; Fisher et al. , 2019 ; Molnar , 2019 ) . Given the categorical nature of the “ features ” in this case , we may enumerate all permutations on the edge in question instead of having to rely on random sampling as conventional PFI does . An important operation by Eq ( 1 ) would therefore be accorded with an OI with a large magnitude in either direction , whereas an irrelevant one would have a value of zero since altering it on expectation leads to no change in architecture performance . We compute the OI of each operation of 2,589 architectures , and to circumvent the computational challenge of having to train all neighbour architectures of α ( which are outside the NB301 training set ) from scratch to compute y ( · ) , we use the performance prediction ỹ ( · ) from NB301 . However , as we will show , we validate all key findings by actually training some architectures to ensure that they are not artefacts of the statistical surrogate ( training protocols detailed in App . B.2 ) . Findings The most natural way to group the operations is by their primitive and cell ( i.e . normal or reduce ) types , and we show the main results in Figs . 3 and 4 . In Fig . 3 ( b ) , we discretise the OI scores using the threshold 0.001 ( 0.1 % ) , which is similar to the observed noise standard deviation of the better-performing architectures from NB301 which quantifies the expected variation in performance if an identical architecture is re-trained from scratch , to highlight the important operations with |OI| ≥ 0.001 : these the ones we could more confidently assert to affect the architecture performance beyond noise effects . We summarise the key findings below : ( # 1 ) Only a fraction of operations is critical for good performance within cells : If all operations need to be fully specified ( i.e. , with both the primitive choice and the specific wiring determined ) for good performance , perturbing any of them should have led to significant performance deterioration . However , this is clearly not the case in practice : comparing Fig . 3 ( a ) and ( b ) , we observe that only a fraction of the operations are important based on our definition . To verify this directly beyond predicted performances , we randomly select 30 architectures from the top 5 % training set . Within each cell , we sort their 16 operations by their OI in both ascending and descending orders . We then successively disable the operations by zeroising them , and train the resulting architectures with increasing number of operations disabled from scratch1 until there are only half of the active operations remain ( Fig . 5 ) . The results largely confirms our findings and shows that the OI , although computed via predicted performance , is accurate in representing the ground-truth importance of the operations : on average , we need to disable 6 low-OI operations to see a statistically significant drop in performance , and almost half of the operations to match the effect of disabling just 2 high-OI ones . On the other hand , disabling the high-OI operations quickly reduce the performance and in some cases stall the training altogether : noting that the overall standard deviation of the entire NB301 training set is just approx . 0.8 % , the drop in performance is quite dramatic . ( # 2 ) Reduce cells are relatively unimportant for good performance : Searching independently for reduce cells scale the search space quadratically , but Fig . 3 ( b ) shows that they contain much fewer important operations , and Fig . 4 shows that the OI distribution across all primitives are centered close to zero in reduce cells : both suggest that reduce cell is less important to the architecture performance . To verify this , we draw another 30 random architectures . For each of the them , we construct and train from scratch the original architecture and 4 derived ones , with ( a ) reduce cell set identical to normal cell , ( b ) reduce cell with all operations set to parameterless skip connections , ( c ) normal cell set identical to reduce cell and ( d ) normal cell with operations set to skip connections . As shown in Fig . 1Note that we may not obtain NB301 performance prediction on these architectures , as NB301 requires all 16 operations to be enabled with valid primitives . 6 : setting reduce cell to be identical to the normal cell leads to no significant change in performance , while the reverse is not true . A more extreme example is that while setting cells to be consisted of skip connections only is unsurprisingly sub-optimal in both cases , doing so on the reduce cells harms the performance much less – this suggests that while searching separately for reduce cells are well-motivated , the current design , which places much fewer reduce cells than normal cells in the overall architecture yet treats them equally in search might be a sub-optimal trade-off , and searching two separate cells may yield little benefits over the simple strategy of searching for one graph only and applying it on both normal and reduce cells . ( # 3 ) Different primitives have vastly different importance profiles with many of them redundant : The set of candidate primitives A consists of operators that are known to be useful in manuallydesigned architectures , with the expectation that they should also be , to varying degrees , useful to NAS . However , this is clearly not the case : while it is already known that some primitives are favoured more by certain search algorithms ( Zela et al. , 2020 ) , the observed discrepancy in the relative importance of the primitives is , in fact , more extreme : the normal cells ( which are also the important cells by Finding 2 ) across the entire spectrum of good performing architectures overwhelmingly favour only 3 ( separable convolutions & skip connection ) out of 7 possible primitives ( Fig . 3 ( a ) ) . Even when the remaining 4 primitives are occasionally selected , they are almost never important ( Fig . 3 ( b ) ) – this is also observed in Fig . 4 ( a ) which shows that only they all have distributions of OI close to 0 . As we will show later in Sec 4 , we could essentially remove these primitives from A without impacting the performances . Even within the 3 favoured primitives , there is a significant amount of variation . First , comparing Figs 1 ( a ) and ( b ) , skips , when present in good architectures , are very likely to be important . We also note that the distribution of OI of skip has a higher variance – these suggest that the performance of an architecture is highly sensitive towards the specific locations and patterns of skip connections , a detailed study of which we defer to Sec . 4 . On the other hand , while both separable convolutions ( s3 and s5 ) are highly over-represented in the good-performing architectures , it seems that they are less important on an operation level than skip . A possible explanation is that while their presence is required for good performance , their exact locations in the cell matter less which we again verify in Sec . 4 . Increasingly in NAS we are also interested in optimising for multiple objectives ( e.g . maximising performance while minimising costs ) ; we show that even accounting for the conflicting objectives , redundant primitives , especially pooling , remain largely redundant . ( see App . G ) . Discussions The findings confirm that both the search space and the cells are rampant with various redundancies that increase the search complexity but do not actually contribute as much to the performance , and good performance most often does not depend on an entire cell but a few key features and primitives in both individual cells and in the search space . This clearly shows that the search space design can be further optimised , but consequently , many beliefs often ingrained in existing search methods can also be sub-optimal or unnecessary . For example , barring a few exceptions ( and none to our knowledge in the standard cell-based design space ) ( Xie et al. , 2019a ; You et al. , 2020 ; Ru et al. , 2020 ) , the overwhelming majority of the current approaches aims to search for a single , fully deterministic architecture , and this often results in high-dimensional vector encoding of the cells ( e.g . the path encoding of DARTS cell in White et al . ( 2021 ) is > 104 dimensions without truncation ) ; this affects the performance in general ( White et al. , 2020a ) and impedes methods that suffer from curse of dimensionality , such as Gaussian Processes , in particular . However , exact encoding could be in fact unnecessary if good performance simply hinges upon only a few key designs while the rest does not matter as much , and finding relevant low-dimensional , approximate representations could be beneficial instead .

Summary: This paper examines the NASNET cell spaced search space in NASBENCH-301 (darts search space) and NASBENCH-201. They conclude that Separable convolution and skip connections are the most important operations. By replacing the current operations with just this subset and also enforcing the skip connection to be used as a residual connection, they are able to achieve accuracy very similar to the networks discovered by other SOTA algorithms. They also demonstrate that one does not have to search for reduce cell and that it can just use the same architecture as that of the normal cell. Finally, they replace the SOTA architectures found by the most popular NAS algorithms and replace the cells to use only Separable conv and skip connections and show that the accuracy of the network is very close to the original network.