Understanding Architectures Learnt by Cell-based Neural Architecture Search

1 INTRODUCTION . Various neural network architectures ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2015 ; He et al. , 2016 ; Huang et al. , 2017 ) have been devised over the past decades , achieving superhuman performance for a wide range of tasks . Designing these neural networks typically takes substantial efforts from domain experts by trial and error . Recently , there is a growing interest in neural architecture search ( NAS ) , which automatically searches for high-performance architectures for the given task . The searched NAS architectures ( Zoph et al. , 2018 ; Real et al. , 2019 ; Pham et al. , 2018 ; Liu et al. , 2019 ; Xie et al. , 2019b ; Luo et al. , 2018 ; Cai et al. , 2019 ; Akimoto et al. , 2019 ; Nayman et al. , 2019 ) have outperformed best expert-designed architectures on many computer vision and natural language processing tasks . Mainstream NAS algorithms typically search for the connection topology and transforming operation accompanying each connection from a predefined search space . Tremendous efforts have been exerted to develop efficient and effective NAS algorithms ( Liu et al. , 2019 ; Xie et al. , 2019b ; Luo et al. , 2018 ; Akimoto et al. , 2019 ; Nayman et al. , 2019 ) . However , less attention has been paid to these searched architectures for further insight . To our best knowledge , there is no related work in the literature examining whether these NAS architectures share any pattern , and how the pattern may impact the architecture search if there exists the pattern . These questions are fundamental to understand and improve existing NAS algorithms . In this paper , we endeavour to address these questions by examining the popular NAS architectures1 . The recent work ( Xie et al. , 2019a ) shows that the architectures with random connection topologies can achieve competitive performance on various tasks compared with expert-designed architectures . Inspired by this result , we examine the connection topologies of the architectures generated by popular NAS algorithms . In particular , we find a connection pattern of the popular NAS architectures . These architectures tend to favor wide and shallow cells , where the majority of intermediate nodes are directly connected to the input nodes . 1The popular NAS architectures refer to the best architectures generated by state-of-the-art ( SOTA ) NAS algorithms throughout the paper . Notably , we research on the cell-based NAS algorithms and architectures . To appreciate this particular connection pattern , we first visualize the training process of the popular NAS architectures and their randomly connected variants . Fast and stable convergence is observed for the architectures with wide and shallow cells . We further empirically and theoretically show that the architectures with wider and shallower cells consistently enjoy a smoother loss landscape and smaller gradient variance than their random variants , which helps explain their better convergence and consequently the selection of these NAS architectures during the architecture search . We finally evaluate the generalization performance of the popular NAS architectures and their randomly connected variants . We find that the architectures with wide and shallow cells may not generalize better than other candidate architectures despite their faster convergence . We therefore believe that rethinking NAS from the perspective of the true generalization performance rather than the convergence of candidate architectures should potentially help generate better architectures . 2 RELATED WORKS . Neural Architecture Search Neural architecture search ( NAS ) searches for best-performing architectures automatically for a given task . It has received increasing attention in recent years due to its outstanding performance and the demand for automated machine learning ( AutoML ) . There are three major components in NAS as summarized by Elsken et al . ( 2019 ) , namely search space , search policy ( or strategy , algorithm ) , and performance evaluation ( or estimation ) . To define the search space , the prior knowledge extracted from expert-designed architectures is typically exploited . As for the search policy , different algorithms are proposed to improve the effectiveness ( Zoph et al. , 2018 ; Real et al. , 2019 ; Tan et al. , 2019 ; Cai et al. , 2019 ) and the efficiency ( Pham et al. , 2018 ; Liu et al. , 2019 ; Xie et al. , 2019b ; Luo et al. , 2018 ; Nayman et al. , 2019 ; Akimoto et al. , 2019 ) of the architecture search . However , no effort has been devoted to understanding the best architectures generated by various NAS approaches . Detailed analysis of these architectures may give insights about the further improvement of existing NAS algorithms . Evaluation of NAS algorithms Recent works evaluate NAS algorithms by comparing them with random search . Li & Talwalkar ( 2019 ) and Sciuto et al . ( 2019 ) compare the generalization performance of architectures generated from random search and existing NAS algorithms . Interestingly , the random search can find architectures with comparable or even better generalization performance . Particularly , Sciuto et al . ( 2019 ) show empirically that the ineffectiveness of some NAS algorithms ( Pham et al. , 2018 ) could be the consequence of the weight sharing mechanism during the architecture search . While these evaluations help understand the general disadvantages of NAS algorithms , what kind of architectures the NAS algorithms are learning and why they learn these specific architectures are still not well understood . 3 THE CONNECTION PATTERN OF POPULAR NAS CELLS . Mainstream NAS algorithms ( Zoph et al. , 2018 ; Real et al. , 2019 ; Pham et al. , 2018 ; Liu et al. , 2019 ; Xie et al. , 2019b ; Luo et al. , 2018 ) typically search for the cell structure , including the connection topology and the corresponding operation ( transformation ) coupling each connection . The generated cell is then replicated to construct the entire neural network . We therefore mainly investigate these cell-based NAS architectures . In this section , we first introduce the commonly adopted cell representation , which is useful to understand the connection and computation in a cell space . We then sketch the connection topologies of popular cell-based NAS architectures to investigate their connection patterns . By comparison , we show that there is a common connection pattern among the cells learned by different NAS algorithms ; particularly , these cells tend to be wide and shallow . 3.1 CELL REPRESENTATION . Following DARTS ( Liu et al. , 2019 ) , we represent the cell topology as a directed acyclic graph ( DAG ) consisting of N nodes , including M input nodes , one output node and ( N −M − 1 ) intermediate nodes . Each node forms a latent representation of the input instance . The input nodes consist of the outputs from M preceding cells . And the output node aggregates ( e.g. , concatenate ) the representations from all intermediate nodes . Each intermediate node is connected to M proceeding nodes in the same cell . Each connection transforms the representation from one node via an operation from a predefined operation set , e.g. , 3×3 convolution , 3×3 max pooling , etc . The target of NAS algorithm is to search for the best M source nodes for each intermediate node and the best operation for each of the connections between nodes . In the literature , the searched cell is then replicated by L times to build the entire neural network architecture2 . We abuse the notation C to denote a cell and also the architecture built with the specific cell in the following sections . Besides , we shall use CA to denote the best architecture ( or cell ) searched with the NAS algorithm A ( e.g. , DARTS ( Liu et al. , 2019 ) , ENAS ( Pham et al. , 2018 ) ) . Details on how to build the architecture with given cells are provided in Appendix A.3 . 3.2 THE COMMON CONNECTION PATTERN . Recently , Xie et al . ( 2019a ) shows that neural networks constructed by cells with random connection patterns can achieve compelling performance on multiple tasks . Taking this a step further , we wonder whether cells generated from popular NAS algorithms share any connection patterns , which may explain why these cells are chosen during the architecture search . To investigate the connection patterns , we sketch the topologies of the popular NAS cells with detailed operations omitted . Figure 1 illustrates topologies of 5 popular NAS cells3 . To examine the connection pattern formally , we introduce the concept of ‘ depth ’ and ‘ width ’ for a cell . The depth of a cell is defined as the number of connections along the longest path from input nodes to the output node . The width of a cell is defined as the total width of the intermediate nodes that are connected to the input nodes . In particular , if some intermediate nodes are only partially connected to input nodes ( i.e. , have connections to other intermediate nodes ) , their width is reduced by the percentage of the number of connections to intermediate nodes over all connections . The width of a node is the number 2We omit reduction cell here for brevity , which is used for dimension reduction in NAS . 3We only visualize normal cells since the number of normal cells is significantly larger than the reduction cells in popular NAS architectures . of channels for convolution operations ; and the width is the dimension of the features for linear operations . Supposing that the width of each intermediate node is c , as shown in Figure 1 , the width and depth of the DARTS ( Liu et al. , 2019 ) cell are 3.5c and 3 respectively , and the width and depth of the AmoebaNet ( Real et al. , 2019 ) cell are 4c and 4 correspondingly . Following the above definitions , the smallest depth and largest width for a cell with N = 7 and M = 2 are 2 and 4c respectively . Similarly , for a cell with N = 8 and M = 2 , the smallest depth and largest width are 2 and 5c respectively . In Figure 1 , we can observe that cells from popular NAS architectures tend to be the widest and shallowest ones ( with width close to 4c/5c and depth close to 2 ) among all candidate cells in the same search space . Regarding this as the common connection pattern , we have the following observation : Observation 3.1 ( The Common Connection Pattern ) NAS architectures generated by popular NAS algorithms tend to have the widest and shallowest cells among all candidate cells in the same search space . 4 THE IMPACTS OF CELL WIDTH AND DEPTH ON OPTIMIZATION . Given that popular NAS cells share the common connection pattern , we then explore the impact of this common connection pattern from the optimization perspective to answer the question : why the wide and shallow cells are selected during the architecture search ? We sample and train variants of popular NAS architectures with random connections . Comparing randomly connected variants with the popular NAS architectures , we find that architectures with wider and shallower cells indeed converge faster so that they are selected by NAS algorithms ( Section 4.1 ) . To understand why the wider and shallower cell contributes to faster convergence , we further investigate the loss landscape and gradient variance of popular NAS architectures and their variants via both empirical experiments ( Section 4.2 ) and theoretical analysis ( Section 4.3 ) .

Summary: This paper tries to understand the characteristics of the architectures found by common NAS methods in the cell-search space. Specifically it characterizes the cell-search space used by DARTS, SNAS, AmeobaNet and finds that a most of these search methods find cells which are wide and shallow in depth (they give a specific definition of width and depth for characterizing cells). In fact these cells are usually the widest and shallowest architectures in their search space. The author empirically find that because these kinds of topologies converge faster during training and inevitably every NAS algorithm during search don't train upto convergence but only up to a bit and make decisions based on partially converged statistics there is a bias in selection towards these topologies. They also provide theoretical intuition to back-up these empirical findings.