Title: DIVING INTO OPTIMIZATION OF TOPOLOGY IN NEURAL NETWORKS

Abstract: Seeking effective networks has become one of the most crucial and practical areas in deep learning. The architecture of a neural network can be represented as a directed acyclic graph, whose nodes denote the transformation of layers and edges represent information flow. Despite the selection of micro node operations, macro connections among the whole network, noted as topology, largely affect the optimization process. We first rethink the residual connections via a new topological view and observe the benefits provided by dense connections to the optimization. Motivated by this, we propose an novel method to optimize the topology of a neural network. The optimization space is defined as a complete graph, and through assigning learnable weights which reflect the importance of connections, the optimization of topology is transformed into learning a set of continuous variables of edges. To extend the optimization to larger search spaces, a new series of networks, called TopoNet, are designed. To further focus on critical edges and promote generalization ability in dense topologies, an auxiliary sparsity constraint is adopted to constrain the distribution of edges. Experiments on classical networks prove the effectiveness of the optimization of topology. Experiments with To-poNets further verify both availability and transferability of the proposed method in different tasks e.g. image classification, object detection, and face recognition.

1. INTRODUCTION:

Deep learning successfully transits the feature engineering from manual to automatic design. As a tendency, seeking effective neural networks gradually becomes an important and practical direction. Topologically, the architecture of a network can be expressed as a direct acyclic graph, whose nodes denote the transformation of layers and edges represent information flow. Corresponding to micro operations within a layer/block, the macro connections (Atwood & Towsley, 2016; Pérez-Rúa et al., 2018) between layers also experienced a series of evolutions. And we denote the macro connections as topology in this work.

In initial literature, AlexNet (Krizhevsky et al., 2012) , VGGNet (Simonyan & Zisserman, 2014) with plain topology were proposed. Due to the problems of gradient vanishing and exploding, extending the network to a deeper level for better representation is nearly difficult. To better adapt the optimization process of gradient descent, GoogleNet (Szegedy et al., 2015) adopted parallel modules, and Highway networks (Srivastava et al., 2015) utilized gating units to regular the flow of information, resulting in multipath and elastic topologies. Driven by the significance of depth, the residual block consisted of residual mapping and shortcut was raised in ResNet (He et al., 2016) . Topological changes in neural networks successfully scaled up neural networks to hundreds or even thousands of layers. The residual topology was widely approved and applied in the following works, e.g. MobileNet (Sandler et al., 2018; Howard et al., 2019) and ShuffleNet (Zhang et al., 2018) . Divergent from relative sparse topologies, DenseNet (Huang et al., 2017) wired densely within a block to reuse features fully. Recent advances in computer vision also explore neural architecture search (NAS) methods (Zoph et al., 2018; Liu et al., 2019b; Tan et al., 2019) to jointly search micro operations and topology. To trade-off efficiency and performance, their topologies are handdesigned stacked patterns and constrained in limited searching spaces. Orthogonally, randomly wired networks (Xie et al., 2019) explored random topology as architecture by different generators, and obtained competitive performance. These trends reflect the great impact of topology on the Figure 1 : From natural view to topological view of networks with residual connections. We map both addition and unary layer to nodes, and data flow to edges. Red node denotes input x and green one means output y. Red arrows give an example of this mapping for a node with in-degree of 3.

optimization of neural networks. Echoing this perspective, we wonder: Can topology itself in neural networks be optimized? What is the suitable route to do this?

To answer these questions, we start by understanding the most representative residual connection in terms of topology. It formulates x + F(x) as a basic component, in which x represents the identity shortcut, and F(x) denotes the residual mapping. Figure 1 presents two residual architectures with a interval of 1 and 2 respectively. And the number of intervals represents the number of layers that make up a block. If we map both combining (e.g., addition) and transformation (e.g., 3 × 3 conv) to a node, and flow of features to an edge, then the architecture can be expressed as a directed acyclic graph (DAG). From the topological view, the residual connections are rather denser than the natural perspective. This novel representation illustrates residual networks perform multiple feed-forward paths instead of a single deep network. When the interval degrades to 1, its topology evolves into a complete DAG. For a complete DAG with n nodes, the number of available paths from input to output is (n -1)!. As a result, most layers are directly connected to the input and output, resulting in direct access to the gradients and the original input. So we make a hypothesis that dense wirings in topology benefit the optimization process.

To verify the hypothesis and explore the effects of differently wired topologies, toy experiments on CIFAR-100 (Krizhevsky et al., 2009) are conducted. To ensure universality, three types of layer are selected respectively, including 3 × 3 conv, 3 × 3 inverted bottleneck (Sandler et al., 2018) and 3 × 3 separable depthwise conv (Howard et al., 2017) . Defining the total number of nodes, we establish three series of networks under different intervals. Loss curves during training and test performance are given in Figure 2 . It demonstrates an obvious topology-induced impact on optimization. First, it shows denser networks (intervals of 1 and 2) can achieve lower training losses in the majority of cases. Second, the optimal topology is different for different node types (2 for conv, 1 for depthwise and inverted bottleneck). It is possible due to that not all connections are beneficial, and the aggregation of features should be discretionary. So we modify the hypothesis as the topology with dense and effective connections are necessary for networks that are compatible with the optimization process. Regardless of the node types, the optimal structure is always a subset of the complete graph. Naturally, finding the optimal topology in a full search space is equivalent to finding the optimal sub-graph in the complete graph.

To focus on the optimization of topology itself, in this work, we exclude the influence of the mixture of different layers/nodes and select the complete DAG as the search space. Through assigning learnable weights which reflect the importance of connections of edges, the topology can be optimized through gradient descent. Then task-related architectures can be achieved conveniently. Besides, this optimization method is also compatible with existing networks. Given a basic network, optimization of topology can lead to improvement without additional computing burdens. To verify the effectiveness of the optimization of topology in larger search spaces, we also propose a welldesigned network named TopoNet. To solve the optimization problem in a large space and promote generalization, we attach auxiliary sparsity regularization to the distribution of edges, resulting in critical connections and better generalization. Our contributions are as follows:

• We first rethink the residual connections via a new topological view and observe the benefits provided by dense connections to the optimization. • We propose an innovative method to optimize the topology of a neural network. The optimization space is defined as a complete graph, through assigning learnable weights which reflect the importance of connections, the optimization of topology is transformed into learning a set of continuous variables of edges. • The proposed optimization method of topology is effective for existing networks.

• When extended to larger search space, TopoNets and optimization using sparsity constraint verify the availability and transferability in different tasks, including image classification on ImageNet, object detection on COCO and face recognition on MegaFace.

2. METHODOLOGY:



2.1. TOPOLOGICAL REPRESENTATION OF ARCHITECTURE:

We represent the neural network using a directed acyclic graph (DAG) in topology: G = (N , E). Each node n i ∈ N owns an operation o i , such as convolution or identity mapping, parametrized by w i , where i stands for the topological ordering. Edge e ji = (j, i, α ji ) ∈ E means the flow of features from node j to node i, and the importance of the connection is determined by α ji . Each node aggregates inputs from connected pre-order nodes, and performs a transformation to get an output tensor x i . Then x i is sent out to the post-order nodes by the output edges. Among these nodes, the first in topological ordering is appointed to be input, and the last to be output. These two nodes do not transform features, just send/gather features. Except for the input and output nodes, other nodes perform feature transformation, such as convolution, activation and normalization. To focus on the impact of topological changes, the same type of operation is applied for all nodes. It can be formulated as follows:

EQUATION

For a network with k stages, k DAGs are initialized and connected in series. Each graph is linked to its preceding or succeeding stage by output or input node. The topology of network can be denoted as a set α = {α k ji }, and weights of transformation as a set w = {w k i }. Given an input x, the mapping function from sample to feature can be written as:

EQUATION

where T (•) denotes the mapping function computing feature maps of G. 

× 7 conv, C - - - Stage 1 56 × 56 N1 nodes, 2 × C p • N 1 2 N 1 -2 l + N 1 -2 l +2 2 N 1 2 Stage 2 28 × 28 N2 nodes, 4 × C p • N 2 2 N 2 -2 l + N 2 -2 l +2 2 N 2 2 Stage 3 14 × 14 N3 nodes, 8 × C p • N 3 2 N 3 -2 l + N 3 -2 l +2 2 N 3 2 Stage 4 7 × 7 N4 nodes, 16 × C p • N 4 2 N 4 -2 l + N 4 -2 l +2 2 N 4 2 Classifier 1 × 1 GAP, 1k-d FC, softmax - - -

2.2. SEARCH SPACE AND OPTIMIZATION METHOD:

In most previous literature (Tan et al., 2019; Guo et al., 2019) , searching for a cell that is later stacked as blocks for a deep network is an expedient solution to trade-off search efficiency and result optimality. But it constrains the possible topologies in a small subset of all possible graphs. Different from them, we relax connections from block-wise to layer-wise, and define the search space as a complete DAG. It provides all possible connections and is much wider than cell-based or block-based approaches. For a complete DAG with N nodes, the search space contains 2 N (N -1)/2 possible topological structures. Assigning learnable weights to edges transforms the optimization of topology into learning a set of continuous variables weights for edges α. It provides a differentiable type to update topology through gradients according to the loss of task noted as L task . Compared with existing differentiable methods (e.g. DARTS), there is no need to conduct arg max operation for α since it determines the fusion of features instead of the selection of operations. Continuous α also guarantees the consistency between training and testing. During the process of optimization, both w and α are optimized jointly. This optimization process can be viewed as:

EQUATION

2.3. EXPANDING TO LARGER SEARCH SPACES BY TOPONET:

The verification of the optimization method requires a network as a base, which determines the upper bound of the search space. Existing networks are limited by the number of nodes, and can only provide narrow possible structures. These may cover the influence caused by topological changes and affect the search for optimal topology.

Therefore, we design a series of architectures named as TopoNets that can flexibly adjust search space, types of topology and node. As shown in Table 1 , it consists of four stages with number of nodes of {N 1 , N 2 , N 3 , N 4 }. The topology in each stage is defined by a graph, whose type can be chosen from {complete, random, residual}. The complete graph is used for the optimization of topology. For a more strict comparison, we also take the other two types as baselines. The residual one has been described in detail before, and the interval is referred to l. In the random one, an edge between two nodes is linked with probability p, independent of all other nodes and edges. The higher the probability, the denser it is. We follow two simple design rules used in (He et al., 2016) , (i) in each stage, the nodes have the same number of filters C; (ii) and if the feature map size is halved, the number of filters is doubled. The change of filters is implied by the first calculation node in each graph. For the head of the network, we use a single convolutional layer for simplicity. The network ends with a classifier composed of a global average pooling (GAP), a 1000-dimensional fully-connected (FC) layer and softmax function.

2.4. SPARSITY CONSTRAINT:

Much as the mammalian brain (Rauschecker, 1984) in biology, where synapses are created in the first few months of a child's development, followed by gradual re-weighting through postnatal knowledge, growing into a typical adult with relative sparse connections. Same observation also be found in Figure 2 (left) that moderate sparsity benefits the generalization. Inspired from these, we choose L1 regularization, denoted as L sparse , to penalize non-zero parameters of edges resulting in more parameters near zero. This sparsity constraint promotes attention to more critical connections. Then the objective function of our proposed method can be formulated as:

EQUATION

where y stands for the label w.r.t the input x, and λ is a hyper-parameter to balance the sparsity level.

During the process of back-propagation, w can be updated through ∇ w L task . Due to the properties of a complete graph, we propose two types to update α. The first one attaches sparsity constraint on all weights of edges uniformly by computing ∇ α L task + λ • ∇ α L sparse , and is called uniform sparse. The other one is logarithmically related to the in-degree δ i of a node n i with input edges {e ji |j < i}. It performs larger constraint on dense input and smaller on sparse input, and is noted as adaptive sparse. These two types will be further discussed in the experiments section. Algorithm 1 summarizes the procedure for the optimization of topology in neural networks.

3. EXPERIMENTS AND ANALYSIS:



3.1. OPTIMIZATION FOR CLASSICAL NETWORKS:

To investigate the applicability of the optimization method on representative networks, we select ResNet-CIFAR (He et al., 2016 ) consisted of 3 × 3 conv and MobileNetV2-1.0 (Sandler et al., 2018) consisted of inverted bottleneck as node operation. For the optimization of ResNets, we change the interval of 2 in the BasicBlock to 1 to form the complete graph. For MobileNetV2-1.0, each node involves a residual connection and can be viewed as complete graph naturally. As a complement, we increase the depth by increasing the node in each stage, resulting in larger search spaces. It is also a common skill to expand networks (Tan & Le, 2019) . Through assigning learnable α to their edges, the topologies can be optimized using our method. We evaluate our method on CIFAR100 and ImageNet 2012 classification dataset (Russakovsky et al., 2015) that consists of 100 and 1000 classes respectively. Comparative experiments are shown in Table 3 . Under similar Params and FLOPs, the optimization of topology brings 2.22% improvement on Top-1 for ResNet-110 on CIFAR100. For MobileNetV2-1.0 and its extension, the improvements range from 0.24% to 0.75% separately. These confirm the optimization of topology is effective to existing networks and depth-friendly, and larger search space can bring more improvements.

A detailed experimental setup for this section can be found in Sect. A.1. Due to restricted optional topologies of classical networks, the topology can be only optimized in small search spaces, which limits the representation ability of topology. In this section, we design a larger search space using TopoNets, and fully illustrate the improvement brought by topology optimization. The properties of edges and nodes in the optimized topology are also analyzed.

3.2.1. EXPERIMENTAL SETUP FOR TOPONET:

To initialize as many nodes as possible with defined parameters and calculations, we select the separable depthwise convolution that includes a 3 × 3 depthwise convolution followed by a 1 × 1 pointwise convolution. Similar with (Xie et al., 2019) , a triplet unit ReLU-conv-BN forms the transformation in a node. The number of nodes in each stage is {14, 20, 26, 14}. In this setting, the number of possible discrete topologies is 2

14 2 • 2 20 2 • 2 26 2 • 2 14 2 ≈ 6 × 10 209 .

The weights of α are initialized to be 1. And C is set to be 64 to match ResNet-50 (He et al., 2016) for equivalent comparison in the amount of parameters and calculations. 

3.2.2. IMAGE CLASSIFICATION:

We conduct experiments in image classification as in Sect. 3.1. The validation results are shown in Table 4 . Some conclusions can be drawn from the results. (i) Compared with ResNet-50, TopoNets achieve higher performances overall. These reflect the efficiency and elegance of the designed network. (ii) TopoNets of random (0.4 ∼ 0.8) and residual (2 ∼ 4) reflect the benefits of moderate sparsity using prior design. (iii) Under complete graphs, direct optimization of topology can yield 0.98% improvement on Top-1. A similar result is obtained by residual (2), which further reflects residual connection is a pretty manual-designed topology. (iv) Through assigning sparsity constraints, performances have been further improved. TopoNet of adaptive sparse gets the best Top-1 by 78.61%. This proves the benefits of sparseness for dense topology.

In order to intuitively understand the optimization effect of sparsity constraints on dense topological connections, distributions of α in optimized topologies are given in Figure 3 . Compared with w/o sparse, sparsity constraints push more parameters near zero, resulting in focusing on critical con- nections. With the enhancement of constraints, more connections disappear. Excessive constraints will damage the representation of features, so λ is set to be e-4 in all experiments. In the right, two types of constraints are shown in different stages. The adaptive one penalizes denser connections a lot and further strengthens relatively sparser but critical connections.

A detailed experimental setup for this section can be found in Sect. A.2.1. To analyze the optimized topology, we select the trained complete model in Table 4 .

3.2.3. GRAPH DAMAGE FOR TOPOLOGICAL PROPERTIES:

First, we explore the influence by removing individual nodes according to its topological ordering in the graph and evaluating without extra training. We expect them to break because dropping any layer drastically changes the input distribution of all subsequent layers. Surprisingly, most removals do not lead to a noticeable change as shown in Figure 4 (left). It can be explained that the available paths is reduced from (n -1)! to (n -2)!, leaving sufficient paths. This suggests that each node in the complete graph do not strongly rely on others, although it is trained jointly. Direct links with input/output nodes makes each node contribute to the final feature representation, and benefits the optimization process. Another observation is that nodes in the front of topological orderings contribute more to a stage. This can be explained that for a node with ordering of i, the generated x i can be only received by node j (where j > i). This causes the feature generated by the front nodes can participate in aggregation as a downstream input. It makes the front nodes contribute more. This can be used to reallocate calculation resources in future work.

Second, we consider the impact of edge removal. All edges with α below a threshold are pruned from the graph, only remaining the important connections. Accuracies before and after retraining are given in Figure 4 (right). Without retraining, accuracy decreases as the degree of pruning deepens.

It is interesting to see that we have the "free lunch" of reducing less than 40% without losing much accuracy. If we fix α of remaining edges and retrain the weights, it can maintain accuracy with 80% of the nodes removed. This proves that the optimization process has found indeed important connections. After pruning edges, nodes with zero in-degree or zero out-degree may be safely removed. It can be used to reduce the parameters and accelerate inference in practical applications.

3.2.4. TRANSFERABILITY ON DIFFERENT TASKS:

To evaluate the generalization and transferability for both optimization method and TopoNets, we also conduct experiments on COCO object detection task (Lin et al., 2014) . We adopt FPN (Lin et al., 2017) as the object detection method. The backbone is replaced with corresponding pretrained one in Table 4 , and is fine-tuned on COCO train2017 dataset. The training configurations of different models are consistent. Test performances are given in Table 5 . To comparable ResNet-50, TopoNets obtain significant promotions in AP with lower computation costs. Contrast with elegant residual topology, our optimization method can also achieve increase by 0.95%. These results indicate the effectiveness of the proposed network and the optimization method.

A detailed experimental setup for this object detection can be found in Sect. A.2.2. Our method also succeeds in face recognition, whose details are given in Sect. A.2.3. 

4. RELATED WORKS:

We briefly review related works in the aspects of neural network structure design and relevant optimization methods.

Neural network design is widely studied in previous literature. From shallow to deep, the shortcut connection plays an important role. Before ResNet, an early practice (Venables & Ripley, 2013) also added linear layer connected from input to the output to train multi-layer perceptrons. Besides, "inception" layer was proposed in (Szegedy et al., 2015) that is composed of a shortcut branch and a few deeper branches. Except on large networks, shortcut also proved effective in small networks, e.g. MobileNet (Sandler et al., 2018) , ShuffleNet (Zhang et al., 2018) and MnasNet (Tan et al., 2019) . The existence of shortcut eases vanishing/exploding gradients (He et al., 2016; Srivastava et al., 2015) . In this paper, we explain from a topological view that shortcuts offer dense connections and benefit optimization. On the macro structure, there also exist many networks with dense connections. DenseNet (Huang et al., 2017) contacted all preceding layers and passed on the feature maps to all subsequent layers in a block. HRNet (Sun et al., 2019) benefits from dense high-to-low connections for fine representations. Densely connected networks promote the specific task of localization (Tang et al., 2018) . Moreover, we optimize the desired network from the complete graph. This is different from MaskConnect (Ahmed & Torresani, 2018) which constrained by K discrete in-degree. This also provides a supplement to (Xie et al., 2019) where random graphs generated by different generators are employed to form a network.

Above the optimization process, our method is consistent with DARTS (Liu et al., 2019b) which is differentiable. Different from it, we do not adopt alternative optimization strategies for weights and architecture. Joint training can replace the transferring step from one task to another, and obtain taskrelated topology. Different from sample-based optimization methods (Real et al., 2019) , our method obtains optimized architecture in the manner of one-shot. (Bender et al., 2018; Guo et al., 2019) also explored this type and utilized weight-sharing across models to amortize the cost of training. Searching from the full space is evaluated in object detection by NAS-FPN (Ghiasi et al., 2019) , in which the feature pyramid is sought in all cross-scale connections. In the aspect of semantic segmentation, Auto-DeepLab (Liu et al., 2019a ) also formed a hierarchical architecture to enlarge possible spaces. The sparsity optimization can also be observed in other applications, e.g. path selection for a multi-branch network (Huang & Wang, 2018) , and pruning unimportant channels for fast inference (Han et al., 2015) .

5. CONCLUSION:

In this work, we have found a feasible way for the optimization of topology in neural networks. Motivated by the topological view, the optimization space is defined as a complete graph, through assigning learnable weights which reflect the importance of connections, the optimization of topology is transformed into learning a set of continuous variables of edges. This method is compatible with existing networks, such as ResNet and MobileNet. TopoNet and corresponding sparsity constraint also proved the effectiveness and transferability in the larger optimization space. Moreover, the observed properties of topology can be used for future works.

A.2.2 TRANSFERABILITY ON OBJECT DETECTION:

We perform experiments on the 80 category COCO detection dataset. We train using the COCO train2017 dataset, and test using the COCO val2017 dataset. Following (Lin et al., 2017) , the input image is resized such that its shorter side has 800 pixels. We adopt synchronized SGD training on 16 GPUs. A mini-batch involves 2 images per GPU and 256 anchors per image. We use a weight decay of e-4 and a momentum of 0.9 without dampening. Our fine-tuning is based on 1× setting of the publicly available Detectron (Girshick et al., 2018) .

A.2.3 TRANSFERABILITY ON FACE RECOGNITION:

We evaluate and compare the performance of proposed methods as feature extractors for face recognition. The most challenging face identification task on MegaFace dataset (Kemelmacher-Shlizerman et al., 2016) is selected. The MegaFace dataset includes 1M images of 690K different individuals as the gallery set and 100K photos of 530 unique individuals from FaceScrub (Ng & Winkler, 2014) as the probe set. Following (Deng et al., 2019) , we select ArcFace as loss, and MS1MV2 (Guo et al., 2016) for training set. For feature extraction, a hidden FC layer is added before the classifier. And the output dimension of FC in the classifier is set equal to the number of individuals of the training set. And the dimension of feature representation is set to 256 for all experiments. We train for 80k iterations with batch size of 512 using 8 GPUs. The initial learning rate is 0.0375. Following (Deng et al., 2019) , the learning rate is lowered by 10 times at 30k, 50k and 70k iterations. We use a weight decay of e-4 and a momentum of 0.9 without dampening.

Identification results are given in Table 6 , and it should be noted that the Rank-1 accuracy of one million distractors is extremely strict. TopoNets achieve outperforming performances against ResNet-50 with less calculation costs. Comparisons among TopoNets also illustrate the applicability of the optimization for topology in face recognition. Since Xie et al. (2019) has not released their codes, we conduct comparisons using TopoNet through replacing graphs to form networks. The three graph generators of ER, BA, WS are performed using NetworkX2 . with the best configs in their paper. For fair comparison, we do not adopt droppath and dropout for all experiments. The results are given as follows using 5 repeat runs. These results further prove the effectiveness of our optimization method over random baselines. 

:

https://networkx.github.io/documentation/stable/tutorial.html

Figures and Tables:

Figure 2: The impact of topology changes on the training process in CIFAR-100. Each subgraph represents the loss curves of networks composed of a single node type when the topology changes. The tags at the top give the type of node used, interval of residual blocks and test Top-1 accuracy.

Figure 3: The effect of sparsity constraint on distributions of α. Histogram on the left indicates that sparsity drives most of the weights near zero. Adjacency matrices on the right shows the difference between uniform and adaptive one, whose rows correspond to the input edges for a particular node and columns represent the output ones. Colors indicate the weights of edges.

Figure 4: Impact of node (left) and edge (right) removal for complete type of TopoNet.

Architectures of TopoNets for ImageNet.

Table content: <table><tr><td>Layers</td><td>Output Size</td><td>Component and Channels</td><td>Edges of Different Topologies Random, p Residual, l Complete</td></tr><tr><td>Head</td><td>112 × 112</td><td>7</td><td/></tr></table>

Algorithm 1: Optimization of Topology in Neural Networks. Network wired with k DAGs G(N , E; w, α); training set {(x, y) (s) } S s=1 ; SparseType; λ. Output: Optimized weights w and topology α. for s = 1 . . . S do Conduct forward propagation F(x), obtain L task and Lsparse ; Calculate backward gradient and update w with ∇wL task ; if SparseType is uniform then Update α with ∇αL task + λ • ∇αLsparse ; else if SparseType is adaptive then Count input edges {eji|j < i} and its in-degree δi for each node ni; Update αji according to δi with ∇α ji L task + λ • log(δi) • ∇α ji Lsparse ; end

Table content: <table/>

Top-1 accuracy (%) of networks on CIFAR100.

Table content: <table><tr><td colspan="4">Basic Network Params(M) FLOPs(G) Original</td><td>Optimized</td><td>Gain</td></tr><tr><td>ResNet-20</td><td>0.28</td><td>0.04</td><td>69.01</td><td colspan="2">69.91 ± 0.12 0.90</td></tr><tr><td>ResNet-32</td><td>0.47</td><td>0.07</td><td>72.07</td><td colspan="2">73.34 ± 0.09 1.37</td></tr><tr><td>ResNet-44</td><td>0.67</td><td>0.10</td><td>73.73</td><td colspan="2">75.60 ± 0.14 1.87</td></tr><tr><td>ResNet-56</td><td>0.86</td><td>0.13</td><td>75.22</td><td colspan="2">76.90 ± 0.03 1.68</td></tr><tr><td>ResNet-110</td><td>1.74</td><td>0.25</td><td>76.31</td><td colspan="2">78.54 ± 0.15 2.23</td></tr></table>

Top-1 accuracy (%) of scaled networks on ImageNet.

Table content: <table><tr><td>Basic Network</td><td colspan="3">Params(M) FLOPs(G) Original</td><td>Optimized</td><td>Gain</td></tr><tr><td>MobileNetV2-1.0</td><td>3.51</td><td>0.31</td><td>72.60</td><td colspan="2">72.86 ± 0.13 0.24</td></tr><tr><td>MobileNetV2-1.0-2N</td><td>6.43</td><td>0.62</td><td>75.93</td><td colspan="2">76.40 ± 0.05 0.47</td></tr><tr><td>MobileNetV2-1.0-4N</td><td>9.62</td><td>1.10</td><td>77.33</td><td colspan="2">77.87 ± 0.09 0.54</td></tr><tr><td>MobileNetV2-1.0-6N</td><td>12.00</td><td>2.06</td><td>77.61</td><td colspan="2">78.36 ± 0.14 0.75</td></tr><tr><td colspan="3">3.2 OPTIMIZATION FOR DESIGNED TOPONETS</td><td/><td/></tr></table>

Optimization results for designed TopoNets on ImageNet. ± 0.06 †The changes of Params and FLOPs caused by edges for different topologies are negligible.

Table content: <table><tr><td>Network</td><td>Initial Edges</td><td colspan="2">Params(M) FLOPs(G)</td><td>Top-1(%)</td></tr><tr><td>ResNet-50</td><td>-</td><td>25.57</td><td>4.08</td><td>76.50</td></tr><tr><td>Random, p = 0.8</td><td>{56,130,221,56}</td><td>23.23</td><td>3.95</td><td>77.56 ± 0.22</td></tr><tr><td>Random, p = 0.6</td><td>{44,104,172,44}</td><td>23.23</td><td>3.95</td><td>77.84 ± 0.19</td></tr><tr><td>Random, p = 0.4</td><td>{31,64,117,31}</td><td>23.23</td><td>3.95</td><td>77.90 ± 0.27</td></tr><tr><td>Residual, l = 4</td><td>{19,36,46,19}</td><td>23.23</td><td>3.95</td><td>77.72 ± 0.13</td></tr><tr><td>Residual, l = 3</td><td>{23,40,61,23}</td><td>23.23</td><td>3.95</td><td>78.10 ± 0.07</td></tr><tr><td>Residual, l = 2</td><td>{34,64,103,34}</td><td>23.23</td><td>3.95</td><td>78.26 ± 0.04</td></tr><tr><td>Complete</td><td>{91,190,325,91}</td><td>23.23</td><td>3.95</td><td>77.24 ± 0.12</td></tr><tr><td>Complete, α</td><td>{91,190,325,91}</td><td>23.23</td><td>3.95</td><td>78.22 ± 0.13</td></tr><tr><td>Complete, α, uniform sparse</td><td>{91,190,325,91}</td><td>23.23</td><td>3.95</td><td>78.46 ± 0.04</td></tr><tr><td colspan="2">Complete, α, adaptive sparse {91,190,325,91}</td><td>23.23</td><td>3.95</td><td>78.60</td></tr></table>

Transferability results (%) on COCO object detection.

Table content: <table><tr><td>Backbone</td><td>AP</td><td>AP50 AP75</td><td>APS</td><td>APM</td><td>APL</td></tr><tr><td>ResNet-50</td><td>36.42</td><td colspan="4">58.66 38.90 21.93 39.84 46.74</td></tr><tr><td>Residual, l = 2</td><td colspan="5">40.74(+4.32) 63.22 44.62 25.01 44.18 52.74</td></tr><tr><td>Complete, α</td><td colspan="5">41.35(+4.93) 63.32 45.08 25.63 44.99 53.47</td></tr><tr><td>Complete, α, uniform sparse</td><td colspan="5">41.46(+5.04) 63.83 44.91 25.07 45.31 53.52</td></tr><tr><td colspan="6">Complete, α, adaptive sparse 41.69(+5.27) 63.86 45.45 25.58 45.52 53.69</td></tr></table>

Optimization results on MegaFace face identification.

Table content: <table><tr><td>Network</td><td colspan="2">Params (M) Conv/BN FC</td><td>Rank-1 (%) (1M distractors)</td></tr><tr><td>ResNet-50</td><td>23.51</td><td>25.69</td><td>94.69</td></tr><tr><td>Residual, l = 2</td><td>20.64</td><td>12.85</td><td>96.10(+1.41)</td></tr><tr><td>Complete, α</td><td>20.64</td><td>12.85</td><td>96.44(+1.75)</td></tr><tr><td>Complete, α, uniform sparse</td><td>20.64</td><td>12.85</td><td>96.51(+1.82)</td></tr><tr><td>Complete, α, adaptive sparse</td><td>20.64</td><td>12.85</td><td>96.62(+1.93)</td></tr><tr><td colspan="4">A.3 REIMPLEMENTATION AND COMPARISION WITH RANDOMLY-WIRED NETWORKS</td></tr></table>

Comparasion with randomly-wired neural networks.

Table content: <table><tr><td>Topology</td><td>Top-1 Acc (%)</td></tr><tr><td>ER (P=0.2)</td><td>77.76 ± 0.23</td></tr><tr><td>BA (M=5)</td><td>78.08 ± 0.17</td></tr><tr><td>WS (K=4, P=0.75)</td><td>78.19 ± 0.25</td></tr><tr><td>our method</td><td>78.60 ± 0.06</td></tr></table>

Peer review (iclr2020):
------------------------- Update after rebuttal --------------------------- Thank you for addressing my concerns. I feel the rebuttal did improve the paper, e.g., the significance of results can be evaluated better now. I still like the overall idea of the paper as optimizing connectivity patterns in architectures has so far mostly been ignored while it is actually straight-forward to do (as shown in this work). I increased my score accordingly. However, the novelty and significance of this work is still limited in my option and therefore I do not argue heavily in favor of accepting this submission. -------------------------------------------------------------------------------------------------------------------------------------------------- The authors propose a method for learning the connections/ connectivity pattern (dubbed: the topology; meaning which layer is directly connected to which other layer) in neural networks. They do so by weighting connections between layers (e.g., by weighting skip connections) with a real valued parameter. This real-valued parameterization of the connections is then optimized by gradient descent along with the weights of neural networks. The authors also propose L1 regularization on the connectivity parameters to induce sparsity. The proposed method is evaluated by optimizing the topology for ResNets, MobileNetsV2 and their proposed “TopoNets”. Originality and significance. The manuscript addresses an interesting problem: while there has been lots of work on manually designing better architectures as well as automated design (a.k.a. neural architecture search, NAS), there is little work on optimizing the overall topology (meaning the connectivity patterns between layers). Most prior work solely focuses on search for blocks or cells and then stacking these cells in a pre-defined, not-optimized manner. However, there has been some work also including this in architecture search (e.g., [1,2,3]), and especially the work [4] seems very related but is not discussed. The authors of [4] propose, very similar to this submission, a gradient-based optimization of the connections (in a different way though). The proposed method for optimizing the topology is also very similar to DARTS, simply applied to the connectivity pattern rather than on the operations-level. However, here the topology is optimized along with the network’s weights on the training data rather than the bi-level optimization from DARTS, where the architectural parameters are optimized on the validation data instead (which is very reasonable as one usually considers the architecture as a hyperparameter). I wonder if this has also been considered/tested by the authors of this paper as I would consider the topology to be a hyperparameter which should not be optimized on training but rather validation data. Knowing [4] and DARTS, the proposed method seems to be rather incremental and straightforward rather than ground breaking. While the proposed method allows for more flexible topologies, it introduces different “stages” for their TopoNets, which are actually a similar concept as blocks or cells from the NAS literature. This again does not allow connections between arbitrary layers (but rather only between layers in the same stage; to the best of my understanding). Empirical results show rather small improvements and their significance is unclear (see next paragraph). Clarity and quality. The paper is mostly well written, well motivated and easy to follow. The mathematical formalism is a little vague at some points (e.g., in Section 2.1., the notation G is used for defining a graph and later in Equation (2) as a function computing feature maps in networks). While the literature on manual design of architectures is thoroughly reviewed, there is missing related work in the context of neural architecture search, as already discussed above. The quality of the results is questionable as differences in accuracies are in almost all experiment rather small (e.g., tuning MobileNetsV2 on Imagenet: 72.62% (original) Top-1 accuracy vs. 72.84% (optimized) and it seems that the authors do only report results for a single run of experiments. In order to assess if the differences are actually statistically significant, the authors would need to report several runs and would need to state, e.g., means and standard deviations. Overall, the authors address an interesting problem, which seems to have fallen into oblivion in current NAS literature: while researcher optimize cells, which are then stacked to build the final model, not many researcher look into connectivity patterns / topology on the macro level, meaning connections across cells and how cells should be stacked. This paper addresses this problem to some extent. However, the proposed optimization method is, in my opinion, rather incremental (with respect to DARTS and [4]) and therefore of limited novelty and significance. It is currently hard to assess the empirical results due to rather small improvements and the lack of repeated runs of experiments. Mainly for these two reasons, I do not recommend the paper for acceptance. [1] Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Quoc V. Le, and Alex Kurakin. Large-scale evolution of image classifiers. ICML, 2017. [2] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Efficient multi-objective neural architecture search via lamarckian evolution. ICLR, 2019. [3] Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. ICML, 2018 [4] Karim Ahmed and Lorenzo Torresani. Maskconnect: Connectivity learning by gradient descent. ECCV, 2018