Title: DIVING INTO OPTIMIZATION OF TOPOLOGY IN NEURAL NETWORKS

Abstract: Seeking effective networks has become one of the most crucial and practical areas in deep learning. The architecture of a neural network can be represented as a directed acyclic graph, whose nodes denote the transformation of layers and edges represent information flow. Despite the selection of micro node operations, macro connections among the whole network, noted as topology, largely affect the optimization process. We first rethink the residual connections via a new topological view and observe the benefits provided by dense connections to the optimization. Motivated by this, we propose an novel method to optimize the topology of a neural network. The optimization space is defined as a complete graph, and through assigning learnable weights which reflect the importance of connections, the optimization of topology is transformed into learning a set of continuous variables of edges. To extend the optimization to larger search spaces, a new series of networks, called TopoNet, are designed. To further focus on critical edges and promote generalization ability in dense topologies, an auxiliary sparsity constraint is adopted to constrain the distribution of edges. Experiments on classical networks prove the effectiveness of the optimization of topology. Experiments with To-poNets further verify both availability and transferability of the proposed method in different tasks e.g. image classification, object detection, and face recognition.

1. INTRODUCTION:

Deep learning successfully transits the feature engineering from manual to automatic design. As a tendency, seeking effective neural networks gradually becomes an important and practical direction. Topologically, the architecture of a network can be expressed as a direct acyclic graph, whose nodes denote the transformation of layers and edges represent information flow. Corresponding to micro operations within a layer/block, the macro connections (Atwood & Towsley, 2016; Pérez-Rúa et al., 2018) between layers also experienced a series of evolutions. And we denote the macro connections as topology in this work.

In initial literature, AlexNet (Krizhevsky et al., 2012) , VGGNet (Simonyan & Zisserman, 2014) with plain topology were proposed. Due to the problems of gradient vanishing and exploding, extending the network to a deeper level for better representation is nearly difficult. To better adapt the optimization process of gradient descent, GoogleNet (Szegedy et al., 2015) adopted parallel modules, and Highway networks (Srivastava et al., 2015) utilized gating units to regular the flow of information, resulting in multipath and elastic topologies. Driven by the significance of depth, the residual block consisted of residual mapping and shortcut was raised in ResNet (He et al., 2016) . Topological changes in neural networks successfully scaled up neural networks to hundreds or even thousands of layers. The residual topology was widely approved and applied in the following works, e.g. MobileNet (Sandler et al., 2018; Howard et al., 2019) and ShuffleNet (Zhang et al., 2018) . Divergent from relative sparse topologies, DenseNet (Huang et al., 2017) wired densely within a block to reuse features fully. Recent advances in computer vision also explore neural architecture search (NAS) methods (Zoph et al., 2018; Liu et al., 2019b; Tan et al., 2019) to jointly search micro operations and topology. To trade-off efficiency and performance, their topologies are handdesigned stacked patterns and constrained in limited searching spaces. Orthogonally, randomly wired networks (Xie et al., 2019) explored random topology as architecture by different generators, and obtained competitive performance. These trends reflect the great impact of topology on the Figure 1 : From natural view to topological view of networks with residual connections. We map both addition and unary layer to nodes, and data flow to edges. Red node denotes input x and green one means output y. Red arrows give an example of this mapping for a node with in-degree of 3.

optimization of neural networks. Echoing this perspective, we wonder: Can topology itself in neural networks be optimized? What is the suitable route to do this?

To answer these questions, we start by understanding the most representative residual connection in terms of topology. It formulates x + F(x) as a basic component, in which x represents the identity shortcut, and F(x) denotes the residual mapping. Figure 1 presents two residual architectures with a interval of 1 and 2 respectively. And the number of intervals represents the number of layers that make up a block. If we map both combining (e.g., addition) and transformation (e.g., 3 × 3 conv) to a node, and flow of features to an edge, then the architecture can be expressed as a directed acyclic graph (DAG). From the topological view, the residual connections are rather denser than the natural perspective. This novel representation illustrates residual networks perform multiple feed-forward paths instead of a single deep network. When the interval degrades to 1, its topology evolves into a complete DAG. For a complete DAG with n nodes, the number of available paths from input to output is (n -1)!. As a result, most layers are directly connected to the input and output, resulting in direct access to the gradients and the original input. So we make a hypothesis that dense wirings in topology benefit the optimization process.

To verify the hypothesis and explore the effects of differently wired topologies, toy experiments on CIFAR-100 (Krizhevsky et al., 2009) are conducted. To ensure universality, three types of layer are selected respectively, including 3 × 3 conv, 3 × 3 inverted bottleneck (Sandler et al., 2018) and 3 × 3 separable depthwise conv (Howard et al., 2017) . Defining the total number of nodes, we establish three series of networks under different intervals. Loss curves during training and test performance are given in Figure 2 . It demonstrates an obvious topology-induced impact on optimization. First, it shows denser networks (intervals of 1 and 2) can achieve lower training losses in the majority of cases. Second, the optimal topology is different for different node types (2 for conv, 1 for depthwise and inverted bottleneck). It is possible due to that not all connections are beneficial, and the aggregation of features should be discretionary. So we modify the hypothesis as the topology with dense and effective connections are necessary for networks that are compatible with the optimization process. Regardless of the node types, the optimal structure is always a subset of the complete graph. Naturally, finding the optimal topology in a full search space is equivalent to finding the optimal sub-graph in the complete graph.

To focus on the optimization of topology itself, in this work, we exclude the influence of the mixture of different layers/nodes and select the complete DAG as the search space. Through assigning learnable weights which reflect the importance of connections of edges, the topology can be optimized through gradient descent. Then task-related architectures can be achieved conveniently. Besides, this optimization method is also compatible with existing networks. Given a basic network, optimization of topology can lead to improvement without additional computing burdens. To verify the effectiveness of the optimization of topology in larger search spaces, we also propose a welldesigned network named TopoNet. To solve the optimization problem in a large space and promote generalization, we attach auxiliary sparsity regularization to the distribution of edges, resulting in critical connections and better generalization. Our contributions are as follows:

• We first rethink the residual connections via a new topological view and observe the benefits provided by dense connections to the optimization. • We propose an innovative method to optimize the topology of a neural network. The optimization space is defined as a complete graph, through assigning learnable weights which reflect the importance of connections, the optimization of topology is transformed into learning a set of continuous variables of edges. • The proposed optimization method of topology is effective for existing networks.

• When extended to larger search space, TopoNets and optimization using sparsity constraint verify the availability and transferability in different tasks, including image classification on ImageNet, object detection on COCO and face recognition on MegaFace.

2. METHODOLOGY:



2.1. TOPOLOGICAL REPRESENTATION OF ARCHITECTURE:

We represent the neural network using a directed acyclic graph (DAG) in topology: G = (N , E). Each node n i ∈ N owns an operation o i , such as convolution or identity mapping, parametrized by w i , where i stands for the topological ordering. Edge e ji = (j, i, α ji ) ∈ E means the flow of features from node j to node i, and the importance of the connection is determined by α ji . Each node aggregates inputs from connected pre-order nodes, and performs a transformation to get an output tensor x i . Then x i is sent out to the post-order nodes by the output edges. Among these nodes, the first in topological ordering is appointed to be input, and the last to be output. These two nodes do not transform features, just send/gather features. Except for the input and output nodes, other nodes perform feature transformation, such as convolution, activation and normalization. To focus on the impact of topological changes, the same type of operation is applied for all nodes. It can be formulated as follows:

EQUATION

For a network with k stages, k DAGs are initialized and connected in series. Each graph is linked to its preceding or succeeding stage by output or input node. The topology of network can be denoted as a set α = {α k ji }, and weights of transformation as a set w = {w k i }. Given an input x, the mapping function from sample to feature can be written as:

EQUATION

where T (•) denotes the mapping function computing feature maps of G. 

× 7 conv, C - - - Stage 1 56 × 56 N1 nodes, 2 × C p • N 1 2 N 1 -2 l + N 1 -2 l +2 2 N 1 2 Stage 2 28 × 28 N2 nodes, 4 × C p • N 2 2 N 2 -2 l + N 2 -2 l +2 2 N 2 2 Stage 3 14 × 14 N3 nodes, 8 × C p • N 3 2 N 3 -2 l + N 3 -2 l +2 2 N 3 2 Stage 4 7 × 7 N4 nodes, 16 × C p • N 4 2 N 4 -2 l + N 4 -2 l +2 2 N 4 2 Classifier 1 × 1 GAP, 1k-d FC, softmax - - -

2.2. SEARCH SPACE AND OPTIMIZATION METHOD:

In most previous literature (Tan et al., 2019; Guo et al., 2019) , searching for a cell that is later stacked as blocks for a deep network is an expedient solution to trade-off search efficiency and result optimality. But it constrains the possible topologies in a small subset of all possible graphs. Different from them, we relax connections from block-wise to layer-wise, and define the search space as a complete DAG. It provides all possible connections and is much wider than cell-based or block-based approaches. For a complete DAG with N nodes, the search space contains 2 N (N -1)/2 possible topological structures. Assigning learnable weights to edges transforms the optimization of topology into learning a set of continuous variables weights for edges α. It provides a differentiable type to update topology through gradients according to the loss of task noted as L task . Compared with existing differentiable methods (e.g. DARTS), there is no need to conduct arg max operation for α since it determines the fusion of features instead of the selection of operations. Continuous α also guarantees the consistency between training and testing. During the process of optimization, both w and α are optimized jointly. This optimization process can be viewed as:

EQUATION

2.3. EXPANDING TO LARGER SEARCH SPACES BY TOPONET:

The verification of the optimization method requires a network as a base, which determines the upper bound of the search space. Existing networks are limited by the number of nodes, and can only provide narrow possible structures. These may cover the influence caused by topological changes and affect the search for optimal topology.

Therefore, we design a series of architectures named as TopoNets that can flexibly adjust search space, types of topology and node. As shown in Table 1 , it consists of four stages with number of nodes of {N 1 , N 2 , N 3 , N 4 }. The topology in each stage is defined by a graph, whose type can be chosen from {complete, random, residual}. The complete graph is used for the optimization of topology. For a more strict comparison, we also take the other two types as baselines. The residual one has been described in detail before, and the interval is referred to l. In the random one, an edge between two nodes is linked with probability p, independent of all other nodes and edges. The higher the probability, the denser it is. We follow two simple design rules used in (He et al., 2016) , (i) in each stage, the nodes have the same number of filters C; (ii) and if the feature map size is halved, the number of filters is doubled. The change of filters is implied by the first calculation node in each graph. For the head of the network, we use a single convolutional layer for simplicity. The network ends with a classifier composed of a global average pooling (GAP), a 1000-dimensional fully-connected (FC) layer and softmax function.

2.4. SPARSITY CONSTRAINT:

Much as the mammalian brain (Rauschecker, 1984) in biology, where synapses are created in the first few months of a child's development, followed by gradual re-weighting through postnatal knowledge, growing into a typical adult with relative sparse connections. Same observation also be found in Figure 2 (left) that moderate sparsity benefits the generalization. Inspired from these, we choose L1 regularization, denoted as L sparse , to penalize non-zero parameters of edges resulting in more parameters near zero. This sparsity constraint promotes attention to more critical connections. Then the objective function of our proposed method can be formulated as:

EQUATION

where y stands for the label w.r.t the input x, and λ is a hyper-parameter to balance the sparsity level.

During the process of back-propagation, w can be updated through ∇ w L task . Due to the properties of a complete graph, we propose two types to update α. The first one attaches sparsity constraint on all weights of edges uniformly by computing ∇ α L task + λ • ∇ α L sparse , and is called uniform sparse. The other one is logarithmically related to the in-degree δ i of a node n i with input edges {e ji |j < i}. It performs larger constraint on dense input and smaller on sparse input, and is noted as adaptive sparse. These two types will be further discussed in the experiments section. Algorithm 1 summarizes the procedure for the optimization of topology in neural networks.

3. EXPERIMENTS AND ANALYSIS:



3.1. OPTIMIZATION FOR CLASSICAL NETWORKS:

To investigate the applicability of the optimization method on representative networks, we select ResNet-CIFAR (He et al., 2016 ) consisted of 3 × 3 conv and MobileNetV2-1.0 (Sandler et al., 2018) consisted of inverted bottleneck as node operation. For the optimization of ResNets, we change the interval of 2 in the BasicBlock to 1 to form the complete graph. For MobileNetV2-1.0, each node involves a residual connection and can be viewed as complete graph naturally. As a complement, we increase the depth by increasing the node in each stage, resulting in larger search spaces. It is also a common skill to expand networks (Tan & Le, 2019) . Through assigning learnable α to their edges, the topologies can be optimized using our method. We evaluate our method on CIFAR100 and ImageNet 2012 classification dataset (Russakovsky et al., 2015) that consists of 100 and 1000 classes respectively. Comparative experiments are shown in Table 3 . Under similar Params and FLOPs, the optimization of topology brings 2.22% improvement on Top-1 for ResNet-110 on CIFAR100. For MobileNetV2-1.0 and its extension, the improvements range from 0.24% to 0.75% separately. These confirm the optimization of topology is effective to existing networks and depth-friendly, and larger search space can bring more improvements.

A detailed experimental setup for this section can be found in Sect. A.1. Due to restricted optional topologies of classical networks, the topology can be only optimized in small search spaces, which limits the representation ability of topology. In this section, we design a larger search space using TopoNets, and fully illustrate the improvement brought by topology optimization. The properties of edges and nodes in the optimized topology are also analyzed.

3.2.1. EXPERIMENTAL SETUP FOR TOPONET:

To initialize as many nodes as possible with defined parameters and calculations, we select the separable depthwise convolution that includes a 3 × 3 depthwise convolution followed by a 1 × 1 pointwise convolution. Similar with (Xie et al., 2019) , a triplet unit ReLU-conv-BN forms the transformation in a node. The number of nodes in each stage is {14, 20, 26, 14}. In this setting, the number of possible discrete topologies is 2

14 2 • 2 20 2 • 2 26 2 • 2 14 2 ≈ 6 × 10 209 .

The weights of α are initialized to be 1. And C is set to be 64 to match ResNet-50 (He et al., 2016) for equivalent comparison in the amount of parameters and calculations. 

3.2.2. IMAGE CLASSIFICATION:

We conduct experiments in image classification as in Sect. 3.1. The validation results are shown in Table 4 . Some conclusions can be drawn from the results. (i) Compared with ResNet-50, TopoNets achieve higher performances overall. These reflect the efficiency and elegance of the designed network. (ii) TopoNets of random (0.4 ∼ 0.8) and residual (2 ∼ 4) reflect the benefits of moderate sparsity using prior design. (iii) Under complete graphs, direct optimization of topology can yield 0.98% improvement on Top-1. A similar result is obtained by residual (2), which further reflects residual connection is a pretty manual-designed topology. (iv) Through assigning sparsity constraints, performances have been further improved. TopoNet of adaptive sparse gets the best Top-1 by 78.61%. This proves the benefits of sparseness for dense topology.

In order to intuitively understand the optimization effect of sparsity constraints on dense topological connections, distributions of α in optimized topologies are given in Figure 3 . Compared with w/o sparse, sparsity constraints push more parameters near zero, resulting in focusing on critical con- nections. With the enhancement of constraints, more connections disappear. Excessive constraints will damage the representation of features, so λ is set to be e-4 in all experiments. In the right, two types of constraints are shown in different stages. The adaptive one penalizes denser connections a lot and further strengthens relatively sparser but critical connections.

A detailed experimental setup for this section can be found in Sect. A.2.1. To analyze the optimized topology, we select the trained complete model in Table 4 .

3.2.3. GRAPH DAMAGE FOR TOPOLOGICAL PROPERTIES:

First, we explore the influence by removing individual nodes according to its topological ordering in the graph and evaluating without extra training. We expect them to break because dropping any layer drastically changes the input distribution of all subsequent layers. Surprisingly, most removals do not lead to a noticeable change as shown in Figure 4 (left). It can be explained that the available paths is reduced from (n -1)! to (n -2)!, leaving sufficient paths. This suggests that each node in the complete graph do not strongly rely on others, although it is trained jointly. Direct links with input/output nodes makes each node contribute to the final feature representation, and benefits the optimization process. Another observation is that nodes in the front of topological orderings contribute more to a stage. This can be explained that for a node with ordering of i, the generated x i can be only received by node j (where j > i). This causes the feature generated by the front nodes can participate in aggregation as a downstream input. It makes the front nodes contribute more. This can be used to reallocate calculation resources in future work.

Second, we consider the impact of edge removal. All edges with α below a threshold are pruned from the graph, only remaining the important connections. Accuracies before and after retraining are given in Figure 4 (right). Without retraining, accuracy decreases as the degree of pruning deepens.

It is interesting to see that we have the "free lunch" of reducing less than 40% without losing much accuracy. If we fix α of remaining edges and retrain the weights, it can maintain accuracy with 80% of the nodes removed. This proves that the optimization process has found indeed important connections. After pruning edges, nodes with zero in-degree or zero out-degree may be safely removed. It can be used to reduce the parameters and accelerate inference in practical applications.

3.2.4. TRANSFERABILITY ON DIFFERENT TASKS:

To evaluate the generalization and transferability for both optimization method and TopoNets, we also conduct experiments on COCO object detection task (Lin et al., 2014) . We adopt FPN (Lin et al., 2017) as the object detection method. The backbone is replaced with corresponding pretrained one in Table 4 , and is fine-tuned on COCO train2017 dataset. The training configurations of different models are consistent. Test performances are given in Table 5 . To comparable ResNet-50, TopoNets obtain significant promotions in AP with lower computation costs. Contrast with elegant residual topology, our optimization method can also achieve increase by 0.95%. These results indicate the effectiveness of the proposed network and the optimization method.

A detailed experimental setup for this object detection can be found in Sect. A.2.2. Our method also succeeds in face recognition, whose details are given in Sect. A.2.3. 

4. RELATED WORKS:

We briefly review related works in the aspects of neural network structure design and relevant optimization methods.

Neural network design is widely studied in previous literature. From shallow to deep, the shortcut connection plays an important role. Before ResNet, an early practice (Venables & Ripley, 2013) also added linear layer connected from input to the output to train multi-layer perceptrons. Besides, "inception" layer was proposed in (Szegedy et al., 2015) that is composed of a shortcut branch and a few deeper branches. Except on large networks, shortcut also proved effective in small networks, e.g. MobileNet (Sandler et al., 2018) , ShuffleNet (Zhang et al., 2018) and MnasNet (Tan et al., 2019) . The existence of shortcut eases vanishing/exploding gradients (He et al., 2016; Srivastava et al., 2015) . In this paper, we explain from a topological view that shortcuts offer dense connections and benefit optimization. On the macro structure, there also exist many networks with dense connections. DenseNet (Huang et al., 2017) contacted all preceding layers and passed on the feature maps to all subsequent layers in a block. HRNet (Sun et al., 2019) benefits from dense high-to-low connections for fine representations. Densely connected networks promote the specific task of localization (Tang et al., 2018) . Moreover, we optimize the desired network from the complete graph. This is different from MaskConnect (Ahmed & Torresani, 2018) which constrained by K discrete in-degree. This also provides a supplement to (Xie et al., 2019) where random graphs generated by different generators are employed to form a network.

Above the optimization process, our method is consistent with DARTS (Liu et al., 2019b) which is differentiable. Different from it, we do not adopt alternative optimization strategies for weights and architecture. Joint training can replace the transferring step from one task to another, and obtain taskrelated topology. Different from sample-based optimization methods (Real et al., 2019) , our method obtains optimized architecture in the manner of one-shot. (Bender et al., 2018; Guo et al., 2019) also explored this type and utilized weight-sharing across models to amortize the cost of training. Searching from the full space is evaluated in object detection by NAS-FPN (Ghiasi et al., 2019) , in which the feature pyramid is sought in all cross-scale connections. In the aspect of semantic segmentation, Auto-DeepLab (Liu et al., 2019a ) also formed a hierarchical architecture to enlarge possible spaces. The sparsity optimization can also be observed in other applications, e.g. path selection for a multi-branch network (Huang & Wang, 2018) , and pruning unimportant channels for fast inference (Han et al., 2015) .

5. CONCLUSION:

In this work, we have found a feasible way for the optimization of topology in neural networks. Motivated by the topological view, the optimization space is defined as a complete graph, through assigning learnable weights which reflect the importance of connections, the optimization of topology is transformed into learning a set of continuous variables of edges. This method is compatible with existing networks, such as ResNet and MobileNet. TopoNet and corresponding sparsity constraint also proved the effectiveness and transferability in the larger optimization space. Moreover, the observed properties of topology can be used for future works.

A.2.2 TRANSFERABILITY ON OBJECT DETECTION:

We perform experiments on the 80 category COCO detection dataset. We train using the COCO train2017 dataset, and test using the COCO val2017 dataset. Following (Lin et al., 2017) , the input image is resized such that its shorter side has 800 pixels. We adopt synchronized SGD training on 16 GPUs. A mini-batch involves 2 images per GPU and 256 anchors per image. We use a weight decay of e-4 and a momentum of 0.9 without dampening. Our fine-tuning is based on 1× setting of the publicly available Detectron (Girshick et al., 2018) .

A.2.3 TRANSFERABILITY ON FACE RECOGNITION:

We evaluate and compare the performance of proposed methods as feature extractors for face recognition. The most challenging face identification task on MegaFace dataset (Kemelmacher-Shlizerman et al., 2016) is selected. The MegaFace dataset includes 1M images of 690K different individuals as the gallery set and 100K photos of 530 unique individuals from FaceScrub (Ng & Winkler, 2014) as the probe set. Following (Deng et al., 2019) , we select ArcFace as loss, and MS1MV2 (Guo et al., 2016) for training set. For feature extraction, a hidden FC layer is added before the classifier. And the output dimension of FC in the classifier is set equal to the number of individuals of the training set. And the dimension of feature representation is set to 256 for all experiments. We train for 80k iterations with batch size of 512 using 8 GPUs. The initial learning rate is 0.0375. Following (Deng et al., 2019) , the learning rate is lowered by 10 times at 30k, 50k and 70k iterations. We use a weight decay of e-4 and a momentum of 0.9 without dampening.

Identification results are given in Table 6 , and it should be noted that the Rank-1 accuracy of one million distractors is extremely strict. TopoNets achieve outperforming performances against ResNet-50 with less calculation costs. Comparisons among TopoNets also illustrate the applicability of the optimization for topology in face recognition. Since Xie et al. (2019) has not released their codes, we conduct comparisons using TopoNet through replacing graphs to form networks. The three graph generators of ER, BA, WS are performed using NetworkX2 . with the best configs in their paper. For fair comparison, we do not adopt droppath and dropout for all experiments. The results are given as follows using 5 repeat runs. These results further prove the effectiveness of our optimization method over random baselines. 

:

https://networkx.github.io/documentation/stable/tutorial.html

Figures and Tables:

Figure 2: The impact of topology changes on the training process in CIFAR-100. Each subgraph represents the loss curves of networks composed of a single node type when the topology changes. The tags at the top give the type of node used, interval of residual blocks and test Top-1 accuracy.

Figure 3: The effect of sparsity constraint on distributions of α. Histogram on the left indicates that sparsity drives most of the weights near zero. Adjacency matrices on the right shows the difference between uniform and adaptive one, whose rows correspond to the input edges for a particular node and columns represent the output ones. Colors indicate the weights of edges.

Figure 4: Impact of node (left) and edge (right) removal for complete type of TopoNet.

Architectures of TopoNets for ImageNet.

Table content: <table><tr><td>Layers</td><td>Output Size</td><td>Component and Channels</td><td>Edges of Different Topologies Random, p Residual, l Complete</td></tr><tr><td>Head</td><td>112 × 112</td><td>7</td><td/></tr></table>

Algorithm 1: Optimization of Topology in Neural Networks. Network wired with k DAGs G(N , E; w, α); training set {(x, y) (s) } S s=1 ; SparseType; λ. Output: Optimized weights w and topology α. for s = 1 . . . S do Conduct forward propagation F(x), obtain L task and Lsparse ; Calculate backward gradient and update w with ∇wL task ; if SparseType is uniform then Update α with ∇αL task + λ • ∇αLsparse ; else if SparseType is adaptive then Count input edges {eji|j < i} and its in-degree δi for each node ni; Update αji according to δi with ∇α ji L task + λ • log(δi) • ∇α ji Lsparse ; end

Table content: <table/>

Top-1 accuracy (%) of networks on CIFAR100.

Table content: <table><tr><td colspan="4">Basic Network Params(M) FLOPs(G) Original</td><td>Optimized</td><td>Gain</td></tr><tr><td>ResNet-20</td><td>0.28</td><td>0.04</td><td>69.01</td><td colspan="2">69.91 ± 0.12 0.90</td></tr><tr><td>ResNet-32</td><td>0.47</td><td>0.07</td><td>72.07</td><td colspan="2">73.34 ± 0.09 1.37</td></tr><tr><td>ResNet-44</td><td>0.67</td><td>0.10</td><td>73.73</td><td colspan="2">75.60 ± 0.14 1.87</td></tr><tr><td>ResNet-56</td><td>0.86</td><td>0.13</td><td>75.22</td><td colspan="2">76.90 ± 0.03 1.68</td></tr><tr><td>ResNet-110</td><td>1.74</td><td>0.25</td><td>76.31</td><td colspan="2">78.54 ± 0.15 2.23</td></tr></table>

Top-1 accuracy (%) of scaled networks on ImageNet.

Table content: <table><tr><td>Basic Network</td><td colspan="3">Params(M) FLOPs(G) Original</td><td>Optimized</td><td>Gain</td></tr><tr><td>MobileNetV2-1.0</td><td>3.51</td><td>0.31</td><td>72.60</td><td colspan="2">72.86 ± 0.13 0.24</td></tr><tr><td>MobileNetV2-1.0-2N</td><td>6.43</td><td>0.62</td><td>75.93</td><td colspan="2">76.40 ± 0.05 0.47</td></tr><tr><td>MobileNetV2-1.0-4N</td><td>9.62</td><td>1.10</td><td>77.33</td><td colspan="2">77.87 ± 0.09 0.54</td></tr><tr><td>MobileNetV2-1.0-6N</td><td>12.00</td><td>2.06</td><td>77.61</td><td colspan="2">78.36 ± 0.14 0.75</td></tr><tr><td colspan="3">3.2 OPTIMIZATION FOR DESIGNED TOPONETS</td><td/><td/></tr></table>

Optimization results for designed TopoNets on ImageNet. ± 0.06 †The changes of Params and FLOPs caused by edges for different topologies are negligible.

Table content: <table><tr><td>Network</td><td>Initial Edges</td><td colspan="2">Params(M) FLOPs(G)</td><td>Top-1(%)</td></tr><tr><td>ResNet-50</td><td>-</td><td>25.57</td><td>4.08</td><td>76.50</td></tr><tr><td>Random, p = 0.8</td><td>{56,130,221,56}</td><td>23.23</td><td>3.95</td><td>77.56 ± 0.22</td></tr><tr><td>Random, p = 0.6</td><td>{44,104,172,44}</td><td>23.23</td><td>3.95</td><td>77.84 ± 0.19</td></tr><tr><td>Random, p = 0.4</td><td>{31,64,117,31}</td><td>23.23</td><td>3.95</td><td>77.90 ± 0.27</td></tr><tr><td>Residual, l = 4</td><td>{19,36,46,19}</td><td>23.23</td><td>3.95</td><td>77.72 ± 0.13</td></tr><tr><td>Residual, l = 3</td><td>{23,40,61,23}</td><td>23.23</td><td>3.95</td><td>78.10 ± 0.07</td></tr><tr><td>Residual, l = 2</td><td>{34,64,103,34}</td><td>23.23</td><td>3.95</td><td>78.26 ± 0.04</td></tr><tr><td>Complete</td><td>{91,190,325,91}</td><td>23.23</td><td>3.95</td><td>77.24 ± 0.12</td></tr><tr><td>Complete, α</td><td>{91,190,325,91}</td><td>23.23</td><td>3.95</td><td>78.22 ± 0.13</td></tr><tr><td>Complete, α, uniform sparse</td><td>{91,190,325,91}</td><td>23.23</td><td>3.95</td><td>78.46 ± 0.04</td></tr><tr><td colspan="2">Complete, α, adaptive sparse {91,190,325,91}</td><td>23.23</td><td>3.95</td><td>78.60</td></tr></table>

Transferability results (%) on COCO object detection.

Table content: <table><tr><td>Backbone</td><td>AP</td><td>AP50 AP75</td><td>APS</td><td>APM</td><td>APL</td></tr><tr><td>ResNet-50</td><td>36.42</td><td colspan="4">58.66 38.90 21.93 39.84 46.74</td></tr><tr><td>Residual, l = 2</td><td colspan="5">40.74(+4.32) 63.22 44.62 25.01 44.18 52.74</td></tr><tr><td>Complete, α</td><td colspan="5">41.35(+4.93) 63.32 45.08 25.63 44.99 53.47</td></tr><tr><td>Complete, α, uniform sparse</td><td colspan="5">41.46(+5.04) 63.83 44.91 25.07 45.31 53.52</td></tr><tr><td colspan="6">Complete, α, adaptive sparse 41.69(+5.27) 63.86 45.45 25.58 45.52 53.69</td></tr></table>

Optimization results on MegaFace face identification.

Table content: <table><tr><td>Network</td><td colspan="2">Params (M) Conv/BN FC</td><td>Rank-1 (%) (1M distractors)</td></tr><tr><td>ResNet-50</td><td>23.51</td><td>25.69</td><td>94.69</td></tr><tr><td>Residual, l = 2</td><td>20.64</td><td>12.85</td><td>96.10(+1.41)</td></tr><tr><td>Complete, α</td><td>20.64</td><td>12.85</td><td>96.44(+1.75)</td></tr><tr><td>Complete, α, uniform sparse</td><td>20.64</td><td>12.85</td><td>96.51(+1.82)</td></tr><tr><td>Complete, α, adaptive sparse</td><td>20.64</td><td>12.85</td><td>96.62(+1.93)</td></tr><tr><td colspan="4">A.3 REIMPLEMENTATION AND COMPARISION WITH RANDOMLY-WIRED NETWORKS</td></tr></table>

Comparasion with randomly-wired neural networks.

Table content: <table><tr><td>Topology</td><td>Top-1 Acc (%)</td></tr><tr><td>ER (P=0.2)</td><td>77.76 ± 0.23</td></tr><tr><td>BA (M=5)</td><td>78.08 ± 0.17</td></tr><tr><td>WS (K=4, P=0.75)</td><td>78.19 ± 0.25</td></tr><tr><td>our method</td><td>78.60 ± 0.06</td></tr></table>

Peer review (iclr2020):
I. Summary of the paper This paper describes a principled strategy for searching for the most suitable neural network architecture out of a particular class of architectures. Specifically, the problem is framed as an optimisation problem over a set of directed acyclic graphs (DAGs) that correspond to potential network architectures. By optimising the edge weights of this representation, a suitable architecture can be generated. In addition to the aforementioned optimisation scheme, the paper also presents a regularisation that results in *sparse* networks, i.e. networks with a smaller number of edges. Multiple experiments on *tuning* existing architectures on several data sets conclude the paper. II. Summary of the review This paper discusses a highly relevant subject, namely how to select neural network architectures in a principled manner. While the presented work already goes into a good direction, I cannot give it my endorsement for acceptance because of the following reasons: - The paper is lacking clarity: concepts could be explained somewhat better, and the paper is suffering from language/grammar issues that make it harder to understand the contents. - Lack of experimental or theoretical depth: the proposed method is presented as-is; no theoretical analysis of its behaviour is performed; while this is not necessarily a problem, as there are several empirical experiments, the experimental section is not sufficiently detailed: for example, no limitations of the method are being discussed and the presented results are not state-of-the-art accuracies. Nevertheless, I want to point out that this paper has the potential to become an important contribution to the community; it is absolutely clear that more principled approaches are required to select network architectures. In the following, I will comment on the individual aspects in more detail. III. Detailed comments (clarity) - The abstract could be improved in terms of its logical flow. Instead of trying to introduce new terminology (macro/micro etc.) here, the abstract should rather state directly that this paper frames network architecture selection as an optimisation problem over DAGs. - The use of topology is slightly non-standard here. What is the meaning behind the *macro* and *micro* operations? This should be explained somewhat better. - Figure 1 should be extended to show an example of how the depicted graphs are described through the terminology mentioned in the paper. For example, individual edges or nodes could be highlighted and referred to in the text to make the *mapping* clearer. - I do not understand how operations such as *addition* are represented in the DAG. Ideally, this should also be elucidated by a figure. - When discussing *intervals* of residual connections, I am assuming that the paper refers to how many layers are skipped? If so, this should be mentioned and defined explicitly. - The term *searching space* should be replaced by *search space*, as the latter is more standard usage. - I do not understand why the optimisation of the topology can decrease the computational burden, as claimed on p. 2. The optimisation process still has to be performed, just like the training of the network, correct? Am I misunderstanding this? - The caption of Figure 2 could be extended; does a single node type mean that the complete network only consists of nodes of that type? Moreover, accuracies/errors should be shown in addition to the loss curves. - The term *dense connection* is vague; I think the paper should use *densely-connected graph* here. - The sentence *Among these nodes, [...]* refers to the *whole* network, and not to the way the output tensor is processed. Am I understanding this correctly? - I do not understand the initial sentences in Section 2.2; what is the meaning of *cell* in this case? - The term *topological structures* should be renamed to *(sub)graphs* in order to improve clarity. - I do not understand the comment on sparsity in Section 2.4. How are *moderate sparsity* and Figure 2 connected? - In the algorithm, I would use *Sparsity Type* instead of *Sparse Type* to refer to the parameter. - What does *Complete* (without ) mean in Table 3? - The footnote below Table 3 is not referenced anywhere in the text or in the table. - In Figure 3, are the adjacency matrices consistent? What happens if the training process is repeated? It would be highly interesting to show *averaged* matrices over multiple runs. IV. Detailed comments (experiments & theory) - A theoretical analysis of the proposed method would be interesting. Does the optimisation always converge? Are minima unique? What is the computational complexity? At least some of these aspects should be discussed. - The limitations of the proposed method are not explained. For example, what is the meaning of the sentence on p. 2 about *excluding the influence of the mixture of different layers/nodes*? It is my understanding that the proposed method can only change the *connections* between blocks of a network, but not the type of layers. Is this correct? If so, it would be a major limitation that should be mentioned explicitly. - Another limitation that is not discussed is the scaling to very deep networks. How problematic is it to model all potential connections in such a network? Are there limits to the current optimisation scheme? This needs to be assessed in the experimental section. - For all experimental tables, standard deviations and means should be provided. This is necessary in order to assess the stability of the proposed method, because there are multiple sources of stochasticity: one arising from the optimisation procedure, the other one arising from the training of the network itself. - The results reported for the experiments are somewhat behind the state-of-the-art in terms of accuracy values. This should be stated more clearly; I assume that it is caused by limitations of the proposed method, which prohibit an application to very recent architectures. Is this correct? If so, it should at least be mentioned. - The claim that nodes at the start of a topological ordering contribute more to specific stages needs to be (empirically) proven. V. Style issues The paper is not easy to read because of several non-standard phrases or expressions. - The phrase *in topology* is often added to a sentence where it does not entirely make sense. For example, *[The] architecture can be expressed as a directed acyclic graph (DAG) in topology*. I do not see the necessity of adding *in topology* here. There are other places at well from which I would remove this phrase. - *effective networks* --> *effective network architectures* - *largely affects* --> *largely affect* - *Motivated by which* --> *Motivated by this* - *innovative method* --> *method* (or *novel method*) - *as a complete graph, through* --> *as a complete graph, and through* - *auxiliary sparsity constraint* --> *an auxiliary sparsity constraint* - *named as TopoNet* --> *called TopoNet* - *At initial periods* --> *Previously* (I am not sure I understand this correctly) - *red signs* --> *red arrows* - *for its topology* --> *in terms of its topology* (?) - *both combining* --> *both a combination* (?) - *number of interval* --> *number of intervals* - *straight connected* --> *directly connected* - *conduct transformation* --> *performs a transformation* - What is the meaning of the phrase *These may cover the influence [...]*? Is this a reference to limitations of existing networks? - *opted from* --> *chosen from* - *Following two simple design rules* --> *We follow two simple design rules* - *1000-dimension* --> * -dimensional* - *We raise two ways* --> *We describe two ways* - *consited* --> *consisted* / *consists* - *deepen the depth* --> *increase the depth* - *origin* --> *original* - *promotions* --> *improvements* - *can make more profit* --> *can be useful to improve performance* (I am guessing this from the context) - *sparseness on representation* --> *sparsity* - *Adaptive one* --> *The adaptive one* - *At the fore* --> *At the beginning/start* - I do not understand the sentence about the *free lunch*. Does it refer to the fact that some connections can still be removed from the network without decreasing accuracy? - *less computation costs* --> *lower computation costs* - *shortcut offers* --> *shortcuts offer* - *benefits optimization* --> *benefit optimization* - *feasible way to the optimization* --> *feasible way for the optimization* - Some references in the bibliography are not capitalised consistently