Title: NEURON HIERARCHICAL NETWORKS

Abstract: In this paper, we propose a neural network framework called neuron hierarchical network (NHN), that evolves beyond the hierarchy in layers, and concentrates on the hierarchy of neurons. We observe mass redundancy in the weights of both handcrafted and randomly searched architectures. Inspired by the development of human brains, we prune low-sensitivity neurons in the model and add new neurons to the graph, and the relation between individual neurons are emphasized and the existence of layers weakened. We propose a process to discover the best base model by random architecture search, and discover the best locations and connections of the added neurons by evolutionary search. Experiment results show that the NHN achieves higher test accuracy on Cifar-10 than state-of-theart handcrafted and randomly searched architectures, while requiring much fewer parameters and less searching time.

1. INTRODUCTION:

Neural networks can be designed either by human experts or search algorithms, both of which have gained great success in image classification and language modeling (Huang et al., 2017; Yang et al., 2018; Pham et al., 2018) . Network architectures designed by both means are mostly layer-based or block-based, which means that the fundamental components are either operation layers or blocks that consist of several layers. A clear tendency can be observed that models with more parameters generally have better performances. It is a well-established fact that redundancy of parameters exists widely in handcrafted neural networks (Han et al., 2015b; Tai et al., 2015; Lin et al., 2017) . We find that such claim holds for architectures discovered by random search or evolutionary search as well. The pruning of unimportant neurons emphasizes the hierarchical relation between individual neurons. Additionally, the decrease in accuracy after parameter reduction is generally inevitable. Therefore, we propose a heuristic procedure to construct neuron-based network architectures by pruning redundant connections and neurons in layer-based models and adding new neurons to strengthen the neuron hierarchy while achieving competitive performances as layer-hierarchy models. Experiments show that NHN achieves higher test accuracy than DenseNet (Huang et al., 2017) , SMASH (Brock et al., 2018) and hierarchical representation (Liu et al., 2018) with much fewer parameters.

Handcrafted architectures. Successful convolutional neural networks (CNNs) designed by human experts can be sketchily categorized by the way data flow through the networks, i.e., plain networks and branching networks. A notable example of plain networks would be VGG nets (Simonyan and Zisserman, 2014) , where there are only one input and output path in each hidden layer. However, in a branching network, the computation flow splits somewhere in the network and merges in a latter layer (Ahmed and Torresani, 2017) . The splitting and aggregation may occur multiple times in a single network. Many have discovered numerous branching network architectures whose performances surpass plain ones while requiring fewer parameters. Skip connections (He et al., 2015; 2016a; Huang et al., 2017) are increasingly popular in improving the performance of deep neural networks, and it becomes common to observe additional convolutions (or other forms of operations) stacked between large layers (Lin et al., 2013; Szegedy et al., 2017; Chen et al., 2017) . In fact, the "stacked-between" operations can be considered as part of a generalized residual block. Multi-branching computation graphs benefit addressing the gradient vanishing problem during the gradient descent training (He et al., 2016a; Huang et al., 2017) . The distinguished techniques mentioned above (plus more listed in Table 1 ) share the same idea of weakening the hierarchy between layers by introducing complex paths to the data flow. The idea is further highlighted by architecture search algorithms. Random and evolutionary architectures. Machine learning algorithms evolve fast. Designing neural networks that perform remarkably on a given task requires ample experience. It has been found that neural networks are not only good at autonomically extracting useful features from raw data, but also capable of finding optimal network architectures to that end. Neural architecture search (NAS) (Zoph and Le, 2017; Zoph et al., 2017) has been attested to its ability to design network architectures for language modeling and image classification. However, candidate models have to be entirely randomly generated and fully trained, therefore NAS is extremely computation intensive, which dims its competitive performances to handcrafted architectures. Many efforts have been devoted to reducing the computational costs of NAS while ensuring sufficient capacity of search space (Liu et al., 2018; Zhong et al., 2018; Liu et al., 2017; Cai et al., 2017) . Two major ideas to achieve the purpose are to design individual reusable cells rather than the entire networks or to share trained weights between models. Recently, Pham et al. (2018) proposed to describe each candidate model as a subgraph of a single directed acyclic graph (DAG) . By sharing weights between submodels, the searching time of NAS is reduced by 1000X in term of GPU-hours. Genetic algorithms are also applied in searching the optimal architectures of CNNs (Miikkulainen et al., 2017; Xie and Yuille, 2017; Real et al., 2017) . Real et al. (2018) proposed regularized evolution (RE) to remove the oldest models from the population, instead of removing the worst ones as in traditional tournament selection (Goldberg and Deb, 1991) . The best CNN model discovered by RE achieves the state-of-the-art performance on Cifar-10, i.e., 2.13% test accuracy on average. However, the number of parameters it requires is as large as nearly 35 million. Convolutional neural fabrics (CNF) (Saxena and Verbeek, 2016; Veniat and Denoyer, 2017) and other forms of random search methods (Baker et al., 2017; Brock et al., 2018; Dong et al., 2018) have been investigated as well.

Layer-wise to neuron-wise hierarchy. Take the best model discovered by macro search in ENAS (Pham et al., 2018) for example. The 12-layer CNN model contains over 21 million parameters and achieves 4.23% test accuracy on Cifar-10. If we remove 75% -90% parameters in all 3 × 3 and 5 × 5 convolutions, the test accuracy is hardly compromised after the same duration of retraining. Even though the architectures in all the search methods are described as directed graphs, each node in these graphs represents either an operation layer (e.g. convolution or pooling layer) or an operation block (e.g. residual block). None of the nodes stands for an actual individual neuron in the network. On one hand, random search and evolutionary search tend to discover architectures that contain complex branching paths. On the other hand, the pruned versions of such architectures work nearly as well as intact ones. These facts bring about the hypothesis that the hierarchy of neurons should work as well as the hierarchy of layers. Please note that we do not simply replace layers with individual neurons, considering layers are composed of abundant neurons. We need the sufficient number of neurons to meet the feature representation requirements. A good hierarchical architecture of neurons may be discovered by either random search or evolutionary search, or combined. The search process must be carefully designed.

In this paper, we propose a three-step course to discover the optimal neuron hierarchy for image classification (see Figure 1 ), i.e., (1) discover the optimal layer hierarchy with ENAS, (2) prune unimportant neurons in the discovered layer-wise model, and (3) randomly add new neurons to the pruned model to enrich the expressive capacity of neuron hierarchy networks. It is worth pointing out that the three-step procedure is also an imitation of the natural development of human brains ( Craik and Bialystok, 2006) . For example, the creation and searching by ENAS correspond to the mass neurogenesis before birth. The pruning of unimportant neurons corresponds to the apoptosis before puberty (Elmore, 2007) . The addition of new neurons to the pruned model corresponds to the persisting neurogenesis during adulthood (Ming and Song, 2011) . Although the existence of neurogenesis in mature brains is being debated in the field of neuroscience (Sorrells et al., 2018; Boldrini et al., 2018) , the software simulation of such process by our work indicates that it is helpful in improving the learning capability of neural networks.

2. THE PROPOSED APPROACH:

In this section, we propose a heuristic algorithm to derive network architectures that emphasize the relation between individual neurons based on traditional layer-based models by network pruning and adding new neurons.

2.1. DISCOVER THE LAYER HIERARCHY:

We directly employ the macro search method proposed by Pham et al. (2018) to discover a 12-layer CNN architecture for image classification. Note that we need to distinguish the important neurons from the less important ones, thus we are not performing Dropout (Srivastava et al., 2014) during the searching and re-training in ENAS. Even though Dropout has proved to be helpful in increasing the generalization power of networks, it distributes the feature extracting capabilities evenly in all neurons, which makes it hard for us to tell which neurons are more important than others. We will, however, employ Dropout in the final fine-tuning stage. We use the cosine schedule (Loshchilov and Hutter, 2017) that involves 4 warm restarts (for a total of 310 epochs) to adjust the learning rate during training, as in Pham et al. (2018) , with the maximum value of 0.05 and minimum of 0.001. Cutout (DeVries and Taylor, 2017) is used to enrich the sample capacity and improve the generalization of the model. The architecture discovered by ENAS is illustrated in Figure 2 (the large blocks connected with solid lines)

2.2. NETWORK PRUNING:

Network pruning (Han et al., 2015b; a; Lin et al., 2017) has been used to get rid of unimportant parameters in CNNs. When followed by quantization and sparsification techniques, the network pruning methods can considerably reduce the memory and bandwidth consumption of deep networks in distributed systems. But it requires pre-learning of the connectivity and a very long time of re-training to restore the performance. We employ the dynamic network pruning (DNP) method proposed by Anonymous (2018) to remove redundant weights. Specifically, during the search of optimal layer-hierarchy model, we replace the global average pooling (Lin et al., 2013) with grouped convolution (Xie et al., 2017; Kaiser et al., 2016) , in which the group number equals the number of both input and output channels and the kernel sizes equal the sizes of input feature maps, so that the grouped convolution exactly maps each feature map to a single value. We also smooth out all 3 × 3 and 5 × 5 convolution kernels with 3 × 3 Gaussian filters after initialization. After training the optimal model discovered by ENAS from scratch, we find that only a small portion of trained kernels have apparent texture appearance, while a majority of them are practically flat. We remove nearly 85% of the kernels based on standard deviation thresholds. Accuracy drop is observed on the pruned model, it expresses the need for finding a way to improve the representation capability of our model, which leads to the idea of adding additional neurons to the network.

2.3. EVOLUTIONARY ADD-ON NEURONS:

In order to distinguish the newly added neurons from the existing neurons, we refer to the former ones as add-on neurons. The add-on neurons are free to be inserted between two arbitrary neurons that are originally located in different layers, since a connection within a layer could cause confusion about computation order. The input and output neurons of an add-on neuron must share the same data dimension, so that we can define a standard rule to create add-on neurons. Take the best architecture discovered by ENAS for example. The 12-layer architecture is divided by two average pooling layers into three segments, with each containing 4 convolution cells. The data dimensions are uniformed in each segment, therefore add-on neurons are not allowed to be inserted across separate segments. Two types of operations are allowed in an add-on neuron, i.e., 3 × 3 convolution and 5 × 5 convolution. 

} ← Recombine(p 1 l , p 2 l , p 3 l , p 4 l ) for i ← 1 to 5 do p i+15 l ← Mutate(p i l ) end end G c ← best model from P l Direction search Population P d = {p d }, p d ← GenerateDir(G c ) return G ← p d

Location and direction search. In order to increase the search speed for add-on neurons, we propose a location-direction search (LDS) method to search the specific positions and connections independently, which means that we first search for the most possible locations where the add-on neurons may occur, then decide on their specific input and output connections. The location search is carried out in an evolutionary fashion, as described in Algorithm 1. First, we define an initial population (GenerateLoc). Each individual in the population stands for a possible combination of add-on locations. During each evolution cycle, we train every model in the population for several iterations (M ) and we (1) crossover the best models (Recombine) to reproduce new models, (2) mutate the best models (M utate) to produce new models, and (3) remove the worst models from the population. The crossover (Recombine) is performed as follows. Let p 1 l , p 2 l , p 3 l , p 4 l denote the best 4 models, in which lower indices are assigned to better models. The crossover pairs to generate 5 new models are selected as {(p 1 l , p 2 l ), (p 1 l , p 3 l ), (p 1 l , p 4 l ), (p 2 l , p 3 l ), (p 2 l , p 4 l )}. After the best combination of add-on locations is discovered, an evolutionary search on add-on directions is performed. The direction of an add-on neuron (i.e. to choose which neurons to connect with) at given location is selected sparsely (GenerateDir), since we do not want add-on neurons at the same location to marginally form a new layer. Because the potential number of different combinations of add-on directions can be extremely large, we simply re-generate all combinations at each evolution cycle to find the best one. No mutation or crossover will be performed during direction search.

For example, let us employ LDS to the first segment in Figure 2 , i.e., layers {0, 1, 2, 3}, as separately shown in Figure 3a . We define the location where one or several add-on neurons may occur as a new sparse convolution layer (e.g., a 0 in Figure 2 ). All possible locations for add-on neurons are {(0, 1), (0, 2), (1, 2), (0, 3), (1, 3), (2, 3)}, in which tuple (a, b) means that add-on neurons are inserted between layer a and b, as illustrated as gray blocks in Figure 3a . If we enforce a restraint that only one input add-on location with the same kernel size is allowed for each existing layer and include the pooling layer into search space, the overall location search complexity will be 1.9 × 10 8 (which is (4!) 3 • (4!) 3 , since there are three segments in the network and we are using two different kernel sizes of add-on neurons). When the optimal add-on locations are determined, we perform the direction search with the sparsity of 0.1, which means that we will leave out 90% of all possible connections and leave the remaining effective neurons as the search results. Keeping neurons with high sensitivities is recommended, because we need the add-on neurons to improve the performance of the model. Therefore, the standard deviation thresholds are used to determine the optimal add-on directions.

Weight sharing in add-on search. We adopt the idea of sharing weights between submodels by Pham et al. (2018) to increase the search speed in LDS. Specifically, we define all possible connections at all possible locations beforehand. Let W = {w l,d |l ∈ L, d ∈ D} denote all shared weights for LDS, in which w l,d denotes the weight at location l and direction d, and L and D stand for all possible locations and directions, respectively. When generating new subgraphs for location or direction search, we simply select the corresponding weights in W k,t = {w l,d |l ∈ L k,t , d ∈ D k,t }, in which (k, t) stands for the kth model in the search population at evolution cycle t. In general, all w l,d in W are trained in an asynchronous fashion.

The proposed approach for neuron hierarchical networks can be described as Algorithm 2. Liu et al. (2018) .

Method:

Parameters Error Score ResNet-110 (He et al., 2016a) 1.7M 6.43% 3.22 pre-act-ResNet-1001 (He et al., 2016b) 10.2M 4.62% 2.35 WRN-40-10 + d/o (Zagoruyko and Komodakis, 2016) 56M 3.80% 2.94 WRN-28-20 + SGDR (Loshchilov and Hutter, 2017) 145.8M 3.74% 6.12 DenseNet (Huang et al., 2017) 27.2M 3.74% 2.16 ResNeXt (Xie et al., 2017) 68.1M 3.58% 3.26 DenseNet + s/s (Gastaldi, 2017) 26.2M 2.86% 1.77

CNF (Saxena and Verbeek, 2016) 21.2M 7.43% 3.81 MetaQNN (Baker et al., 2017) 11.2M 6.92% 3.49 Budgeted CNF (Veniat and Denoyer, 2017) 7.6M 5.12% 2.58 FractalNet + d/p + d/o (Larsson et al., 2017) 38.6M 4.60% 2.77 PPP-Net (Dong et al., 2018) 11.3M 4.36% 2.23 ENAS macro (Pham et al., 2018) 21.3M 4.23% 2.28 SMASH (Brock et al., 2018) 16.0M 4.03% 2.11 NAS + more filters (Zoph and Le, 2017) 37.4M 3.65% 2.36 Block-QNN-S + more filters (Zhong et al., 2018) 39.8M 3.54% 2.38 ENAS micro (Pham et al., 2018) 4.6M 3.54% 1.78 EAS by NT (DenseNet) (Cai et al., 2017) 10.7M 3.44% 1.77 Progressive NAS (Liu et al., 2017) 3.2M 3.41% 1.71 CL on ResNeXt (Ahmed and Torresani, 2017) 34.4M 3.17% 2.10

Genetic CNN (Xie and Yuille, 2017) -7.10% -Evolving DNN (Miikkulainen et al., 2017)* -7.30% -Large-scale evolution (Real et al., 2017) -5.40% -Evo. search on hier. repr. (Liu et al., 2018) 61.3M 3.63% 3.05 AmoebaNet + c/o (Real et al., 2018) 34.9M Network pruning. We perform the DNP method to prune low sensitivity connections in the best model discovered by ENAS, as described in Section 2.2. We also clean up the network by removing all the neurons with zero input or output connection after DNP. We prune nearly 85% of input connections of all 3 × 3 and 5 × 5 convolution layers on average. In total, more than 70% of the parameters are pruned. The overall pruning ratio decreases because of the immense applications of 1 × 1 convolutions in ENAS, which are not directly removed by DNP. Evident impact on the performance by pruning is observed. We re-train the pruned network for another 160 epochs, which takes about 5 hours. The test accuracy is slightly dropped to 96.20% (with Cutout).

Neuron hierarchy search. The population capacity (K) for location search is set to 20. The individual model p k l is described as the gene code of length of 24, in which the first 12 genes indicate whether an add-on location exists with kernel size of 3 × 3 at each existing layer (since there are 3 segments and we need 4 genes for each segment), and the last 12 genes are for 5 × 5. A positive gene indicates there are add-on neurons connecting into this layer and the value denotes the input layer index. A negative gene value (-1) means that no add-on neuron is connected to this layer. Out of avoiding too many add-on neurons being added to the model, the gene value of -1 is generated by a probability of 0.8, and the rest of the values are sampled uniformly. Each model is trained for 25 iterations (M ) with the batch size of 100, so that a complete training cycle of the population exactly traverses the entire training set of Cifar-10. Submodels are evaluated by single iteration (N = 1) with the batch size of 500. Considering the sparsity of add-on locations and that a slight change in the gene value could lead to dramatic changes in performance, we simply re-generate 5 new models instead of performing mutation on the best models. A cosine schedule for learning rate that gradually decreases from 0.05 to 0.001 is used. The location search takes around 10 hours (300 evolution cycles) to complete. The best 3 sets of discovered locations are illustrated in Figure 2 (the small blocks). It is apparent that having more add-on neurons at low-level layers works better for image classification. The direction search is performed as described in Section 2.3. The direction search takes about 9 hours (300 cycles) to finish. Only weights in add-on neurons are trained and weights in existing neurons are fixed during the LDS, because we are trying to find the add-on neurons that are most compatible with existing neurons. It also saves a great number of computation resources when the majority of parameters are not trained.

After the best locations and directions of add-on neurons have been discovered, we re-train the entire network from scratch. Cutout (DeVries and Taylor, 2017) and Dropout (Srivastava et al., 2014) are used for better generalization. The training schedule resembles the one used by Pham et al. (2018) . Nesterov (Nesterov, 1983) is used as the gradient descent method, in which the momentum term is set to 0.9 and l 2 regularization of 2 × 10 -4 is used. Gradient clipping (Pascanu et al., 2013) with the threshold of 5.0 is used to overcome the potential gradient explosion. In fact, most of the weights (i.e. the weights from the original layer-wise network) have undergone three passes of training, i.e., during ENAS, after DNP, and after LDS. It is reasonable to believe that the overall training has seen several warm restarts, hence the reason why no warm restart is introduced in the training stages after DNP and LDS. The final training takes 6 hours (260 epochs) to complete.

The final test accuracy of NHN is 96.52%. The performance of NHN in comparison with other methods is listed in Table 1 . The total number of parameters in the NHN is approximately 7M. We can see that the test accuracy of NHN is better than most of the methods listed in Table 1 . However, several methods achieve lower test errors than NHN, e.g. DenseNet with Shake-Shake, connectivity learning on ResNeXt and AmoebaNet with Cutout. However, the amounts of parameters of these methods are huge. In order to examine the overall performances of methods in terms of test errors and parameter numbers, we define the performance score as follows. score = (e/2) 2 + (n/25) 2 , where e denotes test error (%) and n denotes number of parameters (M). A lower score value indicates better performance of the model. We can see that except for Progressive NAS which has marginally better performance score, the NHN is highly competitive against all other methods in terms of low test error and small parameter number.

On the whole, the searching and training of the neuron hierarchy network take approximately 46 hours in total, in which the search time of neuron hierarchy is 10 hours. The search time of NHN, however, is indeed more than that of ENAS, since we actually perform a complete ENAS process among other operations during the search. When comparing to NAS (Zoph and Le, 2017) (which takes 450 GPUs to search for 3-4 days), the speedup is as high as 700X-940X and parameter number is reduced by 81%. NHN achieves higher accuracy than SMASH (Brock et al., 2018) with 56% fewer parameters. Moreover, it also has higher accuracy than hierarchical representation (Liu et al., 2018) with 89% fewer parameters and a speedup in search time of 152X, when the latter costs 1.5 days on 200 GPUs to evaluate 7000 samples. Our approach achieves more than 5X speedup with 35% fewer parameters compared to the EAS by NT (Cai et al., 2017) , which costs 2 days on 5 GPUs to achieve similar accuracy. Compared to another method that achieves similar accuracy, Progressive NAS (Liu et al., 2017) , NHN achieves speedup in GPU time of 50X. Cai et al. (2017) proposed a network architecture search method which allows layer transformations during architecture search by enforcing Actor Networks for widening or deepening. Although it allows more neurons and parameters being added to the network during architecture search, the models it discovers are still layer-based, thus redundancy in weights cannot be overlooked. Ahmed and Torresani (2017) proposed to learn the connectivity in ResNeXt (Xie et al., 2017) . They create a uniform grid on which each node represents a residual block, and connections are only allowed between neighboring rows of nodes, whereas in our work, each node represents an actual neuron, the network is not necessarily described as a grid, and connections are not under strict restrains. The performance of the best architecture discovered by connectivity learning is impressive, but the method only applies to ResNeXt.

4. RELATED WORK AND DISCUSSION:

We did not choose fixed or handcrafted architectures as the base model because we believe that experiments conducted on a randomly searched model would be more compelling. There are also pruning-related issues with fixed models, for example, performances of ResNets and DenseNet are extremely sensitive to pruning. The training from scratch on the pruned architecture is crucial, because without it, the model only has a test accuracy of 15.7%. The NHN is not built upon the results by ENAS micro search even though it presents higher test accuracy while requiring fewer parameters than macro search. It is mainly due to the mass employment of depthwise separable convolution (Chollet, 2017) in which kernels are pairs of vectors and cannot be directly pruned. If we replace all the depthwise separable convolutions with normal convolutions, the micro model merely gains accuracy advantage of 0.3% over the macro model. However, it instead contains 67.8M parameters, which is more than 4 times of macro (16.3M). Also, it will consume more than 28GB of memory space to perform the layer-hierarchy search. LDS results (Figure 2 ) show that add-on neurons at lower layers work better, which indicates that rich representation of low-level features is crucial to the performance of NHN. When comparing the final test accuracy (96.52%) to the network without any add-on neuron (96.20%), we know that add-on neurons are helpful in increasing the performance of NHN. In fact, perturbation on the add-on genes discovered by LDS almost always leads to degradation of performance, and the total ablation of added neurons in the final model causes accuracy drop of 1.08%, which proves that the search results are optimal.

The main goal of this paper is neither to comprehensively discuss the properties of neuron fields (Fischer, 1973) , nor to investigate a training method on an entirely randomly generated neuron graph. We'd like to point out that it is quite possible to directly generate a large number of free neurons with somewhat arbitrary connections and train this "random neuron field" to address the same task presented in this work. However, because modern GPUs, or to be more precise, the computation softwares that run on these GPUs are mainly designed for dense 4-d tensor calculation. It is hard to efficiently train such random neuron field at present. Therefore, as sophisticated as our approach may seem, it's an efficient method to construct network architectures that highlight the significance of individual neurons and perform competitively against other state-of-the-art methods.

5. CONCLUSION:

Neural networks that are designed by human experts and search algorithms perform outstandingly in image classification. However, redundancy in parameters exists widely in layer-based architectures. We propose a heuristic method to construct neuron-based architectures, namely, neuron hierarchical networks (NHN), that obviate the redundancy in weights and emphasize the relation between individual neurons. Experiments show that the NHN discovered based on ENAS and by locationdirection search (LDS) outperforms the original ENAS architecture and many other handcrafted and randomly searched models in Cifar-10 classification, while requiring much fewer parameters. Also, the search time of NHN is very efficient compared to several state-of-the-art network architecture search methods, while achieving competitive performance.

:



Figures and Tables:

Figure 1: Neuron hierarchical networks as a generalization of various network structures. Each node in the four graphs indicates an actual neuron in the network. NHN may be derived by performing network pruning (c) on a plain/branching network (a,b), then adding new neurons to break the layer hierarchy (d).

Figure 2: Illustration of discovered layer hierarchy and potential locations for add-on neurons. Block p denotes the average pooling cell. Block a denotes the location where add-on neurons may occur. The yellow and orange colored blocks represent 3 × 3 and 5 × 5 convolutions, respectively. Better viewed in color.

Figure3: Illustration of location-direction search (LDS). First, search for optimal locations where add-on neurons may occur (a). Second, search for optimal direction at each given location (b) that add-on neurons point to the existing neurons. White blocks stand for existing layers and neurons. Gray blocks stand for add-on neurons.

Output: G = G + A, in which A represents the discovered add-on neurons and their connections to the input network G. Require: Population capacity K, cycle number T , train samples B train , test samples B test Location search Initialize: Population P l = {p k l , k = 1, 2, . . . , K}, in which p k

Table content: <table><tr><td>for t ← 1 to T do</td><td/><td/><td>l ← GenerateLoc(G)</td></tr><tr><td>foreach p k l do Train p k l with train sample batches {B</td><td>(k-1)•M +1 train</td><td colspan="2">, . . . , B k•M train }</td></tr><tr><td colspan="3">Evaluate p k l with test sample batches {B test (k-1)•N +1</td><td>, . . . , B k•N test }</td></tr><tr><td>end</td><td/><td/></tr><tr><td colspan="2">P l ← Sort(P l ) by test accuracy decreasingly {p 11 l , . . . , p 15 l</td><td/></tr></table>

Cifar-10 test errors of NHN and other methods. The first block represents handcrafted architectures. The second block represents architectures discovered by random search methods. The third block represents evolutionary architectures. Note that "d/o" stands for Dropout, "s/s" stands for Shake-Shake regulation, "c/o" stands for Cutout, and "d/p" stands for Drop-path. Score = (e/2) 2 + (n/25) 2 , where e denotes the test error in percentage and n denotes the number of parameters in million. Smaller score indicates better overall performance. *The performance of Evolving DNN on Cifar-10 is reported by

Table content: <table><tr><td>Algorithm 2: Construct a neuron hierarchical network (NHN).</td></tr><tr><td>Require: Number of layers L</td></tr><tr><td>Initialize: Layer-hierarchy network G</td></tr><tr><td>Layer hierarchy search</td></tr><tr><td>G ← SmoothInit(G)</td></tr><tr><td>G ← ENAS(G)</td></tr><tr><td>Network pruning</td></tr><tr><td>G ← DNP(G)</td></tr><tr><td>Train G from scratch</td></tr><tr><td>Neuron hierarchy search</td></tr><tr><td>G ← LDS(G) Train G from scratch return G</td></tr></table>

Krizhevsky and Hinton, 2009) dataset to train and evaluate our neuron hierarchy network. The Cifar-10 dataset consists of 50k 32 × 32 3-channel training images in 10 classes, and 10k test images. The standard pre-processing is applied, i.e., subtracting the mean and dividing the standard deviation on each channel. The standard data augmentation is used for the training set, i.e., padding the images to 40 × 40 and randomly cropping down to 32 × 32, then flipping horizontally by a probability of 0.5. All experiments are run on a single NVIDIA GTX 1080Ti video card, which has sufficient GPU memory for the searching and training of the NHN.Layer hierarchy search. We use the same hyperparameters as inPham et al. (2018) to perform the macro search on the entire architecture, as explained in Section 2.1. It takes ENAS no more than 16 hours to find a layer-hierarchy model with test accuracy of 96.55% on Cifar-10 with Cutout. If trained without Cutout, the model's test accuracy is 95.08%. The re-training of the best model from scratch takes almost 9 hours.

Table content: <table><tr><td/><td/><td>2.13% 1.76</td></tr><tr><td>NHN</td><td>7.0M</td><td>3.72% 1.88</td></tr><tr><td>NHN + c/o</td><td>7.0M</td><td>3.48% 1.76</td></tr><tr><td>3 EXPERIMENTS</td><td/><td/></tr><tr><td>We are using the Cifar-10 (</td><td/><td/></tr></table>

Peer review (iclr2019):
 Summary: This paper proposes a method to design architectures for neural networks. Their method has 3 steps. 1. Search for an architecture using an existing automatic model designing algorithm. ENAS (Pham et al 2018) is used in this work. 2. Prune some existing neurons in the model found in step (1), using a pruning algorithm. The algorithm (dynamic network pruning, DNP) comes from an Anonymous submission, so I don’t know what it is. However, I suspect that other algorithms can also be used here. 3. Add extra neurons and computations to the pruned network using evolutionary search. The ultimate architecture resulting from the search indeed improves compared to the architecture found in step (1): having fewer parameters and achieving a smaller error rate on CIFAR-10. Strengths: The NHN method is well-motivated. Pruning of networks is an important technique, but there does not seem to be methods to undo over-prunning. NHN can be applied there. The empirical results of the paper also demonstrated that (ENAS + pruning + NHN) is better than both ENAS and (ENAS + pruning). I think this is a strong point. Weaknesses: 1. The paper is not very well-written. Followings are some points that confused me: - Section 2.3: “the standard deviation thresholds are used to determine the optimal add-on directions”. Which stddev is it? Are you running the existing structure through a batch of data and measure the stddev of each neuron? - Figure 3 is not very illustrative. My understanding of Figure 3 is as follows. Each foll. Please correct me if I’m wrong, but if I am correct, then I strongly suggest that you explain each circle in Figure 3 corresponds to a channel in the input and output layer. - When you perform add-on search, how are the add-on neuron used? Are there outputs added to the existing output layers, or are they concatenated, followed by 1x1 convs etc.? - In Section 2.2, the dynamic network pruning (DNP) algorithm is from a paper in submission. In this case, I suspect that the authors should take the responsibility to briefly explain how the algorithm works. At least, some intuitions should be given. 2. Experimental results of this paper are relatively weak. First, the accuracy achieved by NHN is almost always behind that of block-based search methods, as reported in Table 3 (Zoph et al 2018; Pham et al 2018; Liu et al 2018). It has already been established that block-based search methods are stronger, since they have a smaller search space. Perhaps NHN should be applied on a cell found by a search-based method, rather than on a whole convnet found by ENAS. Second, I am not a fan of the Score metric in Table 1. .sqrt{(e/2)^2 + (n/25)^2}, where e is the error rate and n is the number of parameters, is a strange metric. I wonder where do the numbers 2 and 25 come from, and had they been chosen differently (really, one can SGD-search for these numbers!), some other entries in Table 1 would have been bolded. 3. It’s unclear how much SmoothInit affects (i.e. improves) the performance of NHN. Can you comment on this? 4. Missing citations: Net2Net [1] also expands existing network architectures, albeit with methods to ensure the stability of the “add-on layers”, which can perfectly be applied to the search for add-on neurons in your method. 5. Nit-picking: - Section 2.2: grouped convolutions where the number of groups is equal to the number of input (and output) channels is called “depthwise convolutions”. I think this is a less confusing name. References. [1] Net2Net: Accelerating Learning via Knowledge Transfer. https://arxiv.org/pdf/1511.05641.pdf