Diving into Optimization of Topology in Neural Networks
Authors: Kun Yuan
Quanquan Li
Yucong Zhou
Jing Shao
Junjie Yan
Venue:  (2020)

Abstract: Seeking effective networks has become one of the most crucial and practical areas in deep learning. The architecture of a neural network can be represented as a directed acyclic graph, whose nodes denote transformation of layers and edges represent information flow. Despite the selection of \textit{micro} node operations, \textit{macro} connections among the whole network, noted as \textit{topology}, largely affects the optimization process. We first rethink the residual connections via a new \textit{topological view} and observe the benefits provided by dense connections to the optimization. Motivated by which, we propose an innovation method to optimize the topology of a neural network. The optimization space is defined as a complete graph, through assigning learnable weights which reflect the importance of connections, the optimization of topology is transformed into learning a set of  continuous variables of edges. To extend the optimization to larger search spaces, a new series of networks,
named as TopoNet, are designed. To further focus on critical edges and promote generalization ablity in dense topologies, auxiliary sparsity constraint is adopted to constrain the distribution of edges. Experiments on classical networks prove the effectiveness of the optimization of topology. Experiments with TopoNets further verify both availability and transferability of the proposed method in
different tasks e.g. image classification, object detection and face recognition.

Under review as a conference paper at ICLR 2020
DIVING INTO OPTIMIZATION OF TOPOLOGY IN NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Seeking effective networks has become one of the most crucial and practical areas in deep learning. The architecture of neural network can be represented di- rected acyclic graph, whose nodes denote the transformation of layers and edges represent information flow. Despite the selection of micro node operations, macro connections among the whole network, noted as topology, largely affect the opti- mization process. We first rethink the residual connections via new topological view and observe the benefits provided by dense connections to the optimization. Motivated by this, we propose an novel method to optimize the topology of neu- ral network. The optimization space is defined as complete graph, and through assigning learnable weights which reflect the importance of connections, the op- timization of topology is transformed into learning set of continuous variables of edges. To extend the optimization to larger search spaces, a new series net- works, called TopoNet, are designed. To further focus critical edges and pro- mote generalization ability in dense topologies, an auxiliary sparsity constraint is adopted constrain the distribution of edges. Experiments on classical networks prove the effectiveness of the optimization of topology. Experiments with To- poNets further verify both availability and transferability of the proposed method in different tasks e.g. image classification, object detection, and face recognition.
INTRODUCTION
Deep learning successfully transits the feature engineering from manual to automatic design. As tendency, seeking effective neural networks gradually becomes an important and practical direction. Topologically, the architecture of network can be expressed as direct acyclic graph, whose nodes denote the transformation of layers and edges represent information flow. Corresponding to micro operations within layer/block, the macro connections (Atwood & Towsley, 2016; Pérez-Rúa et al., 2018) between layers also experienced series of evolutions. And we denote the macro connections topology in this work.
In initial literature, AlexNet (Krizhevsky et al., 2012), VGGNet (Simonyan & Zisserman, 2014) with plain topology were proposed. Due to the problems of gradient vanishing and exploding, extending the network to deeper level for better representation is nearly difficult. To better adapt the optimization process of gradient descent, GoogleNet (Szegedy et al., 2015) adopted parallel modules, and Highway networks (Srivastava et al., 2015) utilized gating units to regular the flow of information, resulting in multipath and elastic topologies. Driven by the significance of depth, the residual block consisted of residual mapping and shortcut was raised in ResNet (He et al., 2016). Topological changes in neural networks successfully scaled up neural networks to hundreds or even thousands of layers. The residual topology was widely approved and applied in the following works, e.g. MobileNet (Sandler al., 2018: Howard et al., 2019) and ShuffleNet (Zhang et al., 2018). Divergent from relative sparse topologies, DenseNet (Huang et al., 2017) wired densely within block to reuse features fully. Recent advances in computer vision also explore neural architecture search (NAS) methods (Zoph et al., 2018; Liu et al., 2019b; Tan et al., 2019) to jointly search micro operations and topology. To trade-off efficiency and performance, their topologies are hand- designed stacked patterns and constrained in limited searching spaces. Orthogonally, randomly wired networks (Xie et al., 2019) explored random topology as architecture by different generators, and obtained competitive performance. These trends reflect the great impact of topology the

---

Under review as a conference paper at ICLR 2020
segght Syyt
night lgaer
WRI light
wegh
Kenrity
Weggi
Iduntity
wiigHt laper
Ways/twy
Meenity
layer
welghs
Marrtyy
Werky
Architecture Interval 2
Topological
Architecture
Topological
View
Interval
View
Figure 1: From natural view to topological view of networks with residual connections. We map both addition and unary layer to nodes, and data flow to edges. Red node denotes input x and green one means output y. Red arrows give an example of this mapping for node with in-degree of 3.
optimization of neural networks. Echoing this perspective, we wonder: Can topology itself in neural networks be optimized? What the suitable route to do this?
To answer these questions, we start by understanding the most representative residual connection in terms of topology. It formulates F(x) basic component, in which represents the identity shortcut, and F Fxx) denotes the residual mapping. Figure presents two residual architectures with interval of and 2 respectively. And the number of intervals represents the number of layers that make up a block. If we map both combining (e.g., addition) and transformation (e.g., 3 x 3 conv) to node, and flow of features an edge, then the architecture can be expressed directed acyelic graph (DAG). From the topological view, the residual connections are rather denser than the natural perspective. This novel representation illustrates residual networks perform multiple feed-forward paths instead of single deep network. When the interval degrades to 1, its topology evolves into a complete DAG, For a complete DAG with n nodes, the number available paths from input to output is (n - 1)!. As result, most layers are directly connected to the input and output, resulting in direct access to the gradients and the original input. So we make a hypothesis that dense wirings in topology benefit the optimization process.
To verify the hypothesis and explore the effects of differently wired topologies, toy experiments on CIFAR-100 (Krizhevsky al., 2009) are conducted. To ensure universality, three types of layer are selected respectively, including 3 x 3 conv, 3 x 3 inverted bottleneck (Sandler et al., 2018) and 3 x 3 separable depthwise conv (Howard we establish three series of networks under different intervals. Loss curves during training and test performance are given in Figure 2. demonstrates an obvious topology-induced impact on optimization. First, it shows denser networks (intervals of 1 and 2) can achieve lower training losses in the majority of cases. Second, the optimal topology is different for different node types (2 for conv, for depthwise and inverted bottleneck). It is possible due to that not all connections are beneficial, and the aggregation of features should discretionary. So we modify the hypothesis as the topology with dense and effective connections with the optimization process. Regardless of the node types, the optimal structure is always a subset of the complete graph. Naturally, finding the optimal topology in full search space is equivalent to finding the optimal sub-graph in the complete graph.
al., 2017). Defining the total number of nodes,
necessary for networks that are compatible
To focus the optimization of topology itself, in this work, exclude the influence of the mixture of different layers/nodes and select the complete DAG as the search space. Through assigning learn- able weights which reflect the importance of connections of edges, the topology be optimized
2

---

Under review as a conference paper at ICLR 2020
Cam-1165 10w..
Comy 23933 79
2.00
1.75
Seppene- 1gg00.12
Sepcont 209 07
2.00
MOON- 10
HEOH 2
Conv 3503-75 75
Septonn 3,78 86
3.75
HBOP 3
Comv 197955
1.99
3.2h
3.50
1.00
1:00
1.25
0.5
146
0.3
6.15
0.25
023
2590 3530 1990 Lokee tiber ticce Lreer
033
0:5
Eatr
Tars
Figure 2: The impact of topology changes on the training process in CIFAR-I00. Each subgraph represents the loss curves of networks composed of single node type when the topology changes. The tags the top give the rype of node used, interval of residual blocks and test Top-1 accuracy.
through gradient descent. Then task-related architectures can be achieved conveniently. Besides, this optimization method is also compatible with existing networks. Given a basic network, op- timization of topology can lead to improvement without additional computing burdens. To verify the effectiveness of the optimization of topology in larger search spaces, we also propose a well- designed network named TopoNet. To solve the optimization problem in large space and promote generalization, we attach auxiliary sparsity regularization to the distribution of edges, resulting in critical connections and better generalization. Our contributions are as follows:
We first rethink the residual connections via a new topological view and observe the benefits provided by dense connections to the optimization.
We propose innovative method to optimize the topology of neural network. The opti- mization space is defined as a complete graph, through assigning learnable weights which reflect the importance of connections, the optimization of topology is transformed into learning a set of continuous variables of edges.
The proposed optimization method of topology is effective for existing networks.
When extended to larger search space, TopoNets and optimization using sparsity constraint verify the availability and transferability in different tasks, including image classification ImageNet, object detection COCO and face recognition on MegaFace.
2 METHODOLOGY
2.1 TOPOLOGICAL REPRESENTATION OF ARCHITECTURE
We represent the neural network using a directed acyclic graph (DAG) in topology: 9 = EN.
Each node n, N owns operation o,, such as convolution or identity mapping, parametrized by W,, where i stands for the topological ordering. Edge e; = (j, i, i, E & means the flow of features from node j to node and the importance of the connection is determined by Ozi- Each node aggregates inputs from connected pre-order nodes, and performs a transformation to get an output tensor X,. Then x; is sent out to the post-order nodes by the output edges. Among these nodes, the first in topological ordering is appointed to be input, and the last to be output. These two nodes do not transform features, just send/gather features. Except for the input and output nodes, other nodes perform feature transformation, such as convolution, activation and normalization. To focus on the impact topological changes, the same type of operation applied for all nodes. can be formulated as follows:
o, (x/: w.), where X
Σ ayt- Xj. E3
(1)
For network with A stages, A DAGs are initialized and connected in series. Each graph is linked to its preceding or succeeding stage by output or input node. The topology network can be denoted
set (a ). and weights of transformation mapping function from sample to feature can be written as:
set [w/).. Given an input x, the
F(x)
(x; a', a²) ... : w*),
(2)
where T(-) denotes the mapping function computing feature maps of g.
3

---

Under review as a conference paper at ICLR 2020
Table I: Architectures of TopoNets for ImageNet.
Output Size
Component and Channels
Edges of Different Topologies
Layers
Random,
Residual, I
Complete
Head
112 x 112
7 x conv, C
Stage
56 x 56
N1 nodes, 2 x C
p (N)
N
Ni
N
3
Stage 2
28 x 28
N2 nodes, 4 X C
p (2)
(Na)
Nx-2
(Na)
No
Stage 3
14
Ns nodes, 8 C
p (.())
Na=2
3
Stage 4
7 X 7
Na nodes, 16 x C
p: (N1)
Na-2
(1)
Classifier
X 1
GAP, Ik-d FC, softmax
2.2
SEARCH SPACE AND OPTIMIZATION METHOD
In most previous literature (Tan et al., 2019; Guo et al., 2019), searching for a cell that is later stacked blocks for deep network is expedient solution to trade-off search efficiency and result optimality. But it constrains the possible topologies in a small subset of all possible graphs. Different from them, we relax connections from block-wise layer-wise, and define the search space as a complete DAG. It provides all possible connections and is much wider than cell-based or block-based approaches. For a complete DAG with N nodes, the search space contains 22 possible topological structures. Assigning learnable weights edges transforms the optimization of topology into learning set of continuous variables weights for edges a. It provides a differentiable type to update topology through gradients according to the loss of task noted as Ltesk. Compared with existing differentiable methods (e.g. DARTS), there is no need to conduct arg max operation for a since it determines the fusion of features instead of the selection of operations. Continuous a also guarantees the consistency between training and testing. During the process of optimization, both w and a are optimized jointly. This optimization process can viewed as:
min Lrosk min LXF((x; w, a)).)
2.3 EXPANDING TO LARGER SEARCH SPACES BY TOPONET
The verification of the optimization method requires a network as base, which determines the upper bound of the search space. Existing networks are limited by the number of nodes, and can only provide narrow possible structures. These may cover the influence caused by topological changes and affect the search for optimal topology.
Therefore, we design a series of architectures named as TopoNets that can flexibly adjust search space, types of topology and node. As shown in Table 1, it consists of four stages with number of nodes of {N1, N2, Na, N4). The topology in each stage is defined by a graph, whose type can be chosen from {complete, random, residual). The complete graph is used for the optimization of topology. For a more strict comparison, we also take the other two types as baselines. The residual one has been described in detail before, and the interval is referred to I. In the random one, an edge between two nodes is linked with probability p. independent of all other nodes and edges. The higher the probability, the denser it is. We follow two simple design rules used in (He al., 2016), (i) in each stage, the nodes have the same number of filters C; (ii) and if the feature map size is halved, the number of filters is doubled. The change of filters implied by the first calculation node in each graph. For the head of the network, we use a single convolutional layer for simplicity. The network ends with classifier composed of global average pooling (GAP), 000-dimensi fully-connected (FC) layer and softmax function.
2.4 SPARSITY CONSTRAINT
Much as the mammalian brain (Rauschecker, 1984) in biology, where synapses are created in the first few months of child's development, followed by gradual re-weighting through postnatal knowl- edge, growing into a typical adult with relative sparse connections. Same observation also be found in Figure 2 (left) that moderate sparsity benefits the generalization. Inspired from these, we choose
4

---

Under review as a conference paper at ICLR 2020
Algorithm 1: Optimization of Topology in Neural Networks.
Input: Network wired with & DAGs G(N, E; w, a)); training set {(x, y)00) SparseType; X. Output: Optimized weights and topology
for S...S
Conduct forward propagation F(x), obtain Ltask and Lipprrs
Calculate backward gradient and update w with V Leask if SparseType is uniform then
Update with Va A VaLeparse:
else SpurseType is adaptive then
Count input edges [egili < i] and its in-degree ó, for each node nn: end
Update Ozy according to ó, with Vagi Ltask + X log(ó.) Vay Exparse
LI regularization, denoted as Lsparses to penalize non-zero parameters edges resulting more parameters near zero. This sparsity constraint promotes attention more critical connections. Then the objective function our proposed method can be formulated as:
min Leask + A Lsparse = min [(F(x; y) + X A|aall1,
(4)
where y stands for the label w.r.t the input x, and À is a hyper parameter to balance the sparsity level. During the process of back propppropgg,tin, w can be updated through Wtakkk Due to the properties of a complete graph, we propose two types to update a. The first one attaches sparsity constraint on all weights of edges uniformly by computing VaLtask + VaLsparses and called uniform sparse. The other one is logarithmically related to the in-degree ó, of node n, with input edges {eyli < i}. It performs larger constraint on dense input and smaller on sparse input, and is noted as adaptive sparse. These two types will be further discussed in the experiments section. Algorithm summarizes the procedure for the optimization of topology in neural networks.
EXPERIMENTS AND ANALYSIS
3.1 OPTIMIZATION FOR CLASSICAL NETWORKS
To investigate the applicability of the optimization method representative networks, select CIFNR (He et al., 2016) consisted of x 3 conv and MobileNerV2-1100 (Sandler al., 2018) consisted of inverted bottleneck node operation. For the optimization of ResNets, change the interval of the BasicBlock to to form the complete graph. For MobileNetV22-100, each node involves residual connection and be viewed complete graph naturally. As complement, we increase the depth by increasing the node in each stage, resulting in larger search spaces. is also common skill expand networks (Tan Le. 2019). Through assigning learnable a their edges, the topologies can be optimized using our method.
Table 2: Top-1 accuracy (%) of networks on CIFAR100.
Basic Network
Params(M)
FLOPs(G)
Original
Optimized
Gain
ResNet-20
0.28
0.04
69.01
69.91 +: 0.12
0.90
ResNet-32
0.47
0.07
72.07
73.34 0.09
1.37
ResNet-44
0.67
0.10
73.73
75.60 0.14
1.87
ResNet-56
0.86
0.13
75.22
76,90
+ 0.03
1.68
ResNet-1
1.74
0.25
76.31
78.54 + 0.15
2.23
We evaluate method on CIFARI00 and ImageNet 2012 classification dataset (Russakovsky et al., 2015) that consists 100 and 1000 classes respectively. Comparative experiments are shown in Table 3. Under similar Params and FLOPs, the optimization of topology brings 2.22% improvement on Top-1 for ResNet- 1 10 on CIFAR100. For MobileNetV2-1.00 and its extension, the improvements range from 0.24% 0.75% separately. These confirm the optimization of topology is effective to existing networks and depth--riendly, and larger search space can bring more improvements.
A detailed experimental setup for this section be found in Sect. A.1.

---

Under review as a conference paper at ICLR 2020
Table 3: Top-1 accuracy (%) of scaled networks ImageNet.
Basic Network
Params(M)
FLOPs(G)
Original
Optimized
Gain
MobilcNerV2-110
3.51
0.31
72.60
72.86 = 0.13
0.24
MobilcNeeV2-1.0022
6,43
0.62
75.93
= 0.05
0.47
MobileNeeVV2-10044
9.62
1.10
77.33
77.87 0.09
0.54
MobileNerV2-1.066
12.00
2.06
77.61
78.36 0.14
0.75
3.2 OPTIMIZATION FOR DESIGNED TOPONETS
Due to restricted optional topologies of classical networks, the topology can be only optimized in small search spaces, which limits the representation ability of topology. In this section, we design a larger search space using TopoNets, and fully illustrate the improvement brought by topology optimization. The properties of edges and nodes in the optimized topology also analyzed.
3.2.1
EXPERIMENTAL SETUP FOR TOPONET
To initialize as many nodes as possible with defined parameters and calculations, we select the separable depthwise convolution that includes a 3 x 3 depthwise convolution followed by x pointwise convolution. Similar with (Xie al., 2019), triplet unit ReLU--cov-BN forms the transformation in node. The number of nodes in each stage is (14, 20, 26, 14). In this setting, the
number possible discrete topologies is 2 2(2) 2 (2) 2 (2) 2 ((2))
x 10209, The weights of are initialized to be 1. And C' is set to be 64 to match ResNet-50 (He al., 2016) for equivalent comparison in the amount of parameters and calculations.
Table 4: Optimization results for designed TopoNets on ImageNet.
Network
Initial Edges
Params(M)
FLOPs(()
Top-1(5)
ResNet-50
25.57
4.08
76.50
Random, 0.8
(56.130,221,56]
23.23
3.95
77.56 1: 0.22
Random, 0.6
(44.104.172,44)
23.23
3,95
77.84 + 0.19
Random, p 0.4
(31,64.117,31}
23.23
3.95
77.90 + 0.27
Residual, 1 4
(19.36,46.99)
23.23
3.95
77.72 + 0.13
Residual, 3
{23,40.61,23)
23,23
3.95
78.10
0.07
Residual, 1 2
(34.64.103.44)
23,23
3,95
78.26 + 0.04
Complete Complete, a
(91,190,325,91)
23.23
3.95
77.24 + 0.12
(91.190,325,91
23,23
3.95
78.22 + 0.13
Complete, uniform sparse Complete, ox, adaptive sparse
[91.190.325,91] [91.190.325,9]
23.23
3.95
78.46 0.04
23.23
3.95
78.60 0.06
The changes of Params and FLOPs caused edges for different topologies negligible.
3.2.2 IMAGE CLASSIFICATION
We conduct experiments in image classification as Sect. 3.1. The validation results are shown in Table 4. Some conclusions can be drawn from the results. (i) Compared with ResNet-50, TopoNets achieve higher performances overall. These reflect the efficiency and elegance of the designed net- work. (ii) TopoNets of random (0.4 ~ 0.8) and residual (2 4) reflect the benefits of moderate sparsity using prior design. (iii) Under complete graphs, direct optimization of topology can yield 0.98% improvement
Top-1. A similar result is obtained by residual (2), which further reflects residual connection is a pretty manual-designed topology. (iv) Through assigning sparsity con- straints, performances have been further improved. TopoNet of adaptive sparse gets the best Top-1 by 78.61%. This proves the benefits of sparseness for dense topology.
In order to intuitively understand the optimization effect of sparsity constraints dense topological connections, distributions of a in optimized topologies are given in Figure 3. Compared with w/o sparse, sparsity constraints push more parameters zero, resulting in focusing on critical

---

Under review as a conference paper at ICLR 2020
Uniform
weo sparse
mild sparse
100
Nunber
40-
Weights Edges
Stage1
Stagee
Stage3
Staged
Figure 3: The effect of sparsity constraint on distributions of a. Histogram the left indicates that sparsity drives most of the weights near zero. Adjacency matrices on the right shows the difference between uniform and adaptive one, whose rows correspond to the input edges for particular node and columns represent the output ones. Colors indicate the weights of edges.
nections. With the enhancement of constraints, more connections disappear. Excessive constraints will damage the representation features, so À is set be e-4 in all experiments. In the right, two types of constraints are shown in different stages. The adaptive one penalizes denser connections lot and further strengthens relatively sparser but critical connections.
A detailed experimental setup for this section can be found in Sect. A.2.1.
3.2.3 GRAPH DAMAGE FOR TOPOLOGICAL PROPERTIES
200044 beaaire
Rurting Test
Ratraining Text
20
0.0
0.1
0.5
0.5
Topa ogiical Orcering
Pruniing Rabio
Figure 4: Impact node (left) and edge (right) removal for complete type of TopoNet.
To analyze the optimized topology, we select the trained complete model in Table 4.
First, explore the influence by removing individual nodes according to its topological ordering in the graph and evaluating without extra training. We expect them to break because dropping any layer drastically changes the input distribution of all subsequent layers. Surprisingly, most removals do not lead to a noticeable change as shown in Figure 4 (left). It can be explained that the available paths is reduced from (n 1)! to (n !)!, leaving sufficient paths. This suggests that each node in the complete graph do not strongly rely on others, although it is trained jointly. Direct links with input/output nodes makes each node contribute the final feature representation, and benefits the optimization process. Another observation is that nodes in the front of topological orderings contribute more to stage. This can be explained that for a node with ordering of the generated a; can be only received by node j (where j i). This causes the feature generated by the front nodes can participate in aggregation as a downstream input. makes the front nodes contribute more. This can be used reallocate calculation resources in future work.
Second, we consider the impact of edge removal. All edges with a below a threshold are pruned from the graph, only remaining the important connections. Accuracies before and after retraining given Figure 4 (right). Without retraining, accuracy decreases as the degree of pruning deepens. It is interesting to see that have the "free lunch" of reducing less than 40% without losing much accuracy. we fix a of remaining edges and retrain the weights, it can maintain accuracy with 80% of the nodes removed. This proves that the optimization process has found indeed important
7

---

Under review as a conference paper at ICLR 2020
connections. After pruning edges, nodes with zero in-degree out-degree may be safely removed. can be used to reduce the parameters and accelerate inference practical applications.
3.2.4 TRANSFERABILIT ON DIFFERENT TASKS
To evaluate the generalization and transferability for both optimization method and TopoNets, we also conduct experiments on COCO object detection task (Lin et al., 2014). We adopt FPN (Lin et al., 2017) the object detection method. The backbone is replaced with corresponding pretrained one in Table 4, and is fine-tuned on COCO train2017 dataset. The training configurations different models consistent. Test performances are given in Table 5. To comparable ResNet-50, TopoNets obtain significant promotions in AP with lower computation costs. Contrast with elegant residual topology, optimization method can also achieve increase by 0.95%. These results indicate the effectiveness of the proposed network and the optimization method.
A detailed experimental setup for this object detection can be found in Sect. A.2.2. Our method also succeeds in face recognition, whose details are given in Sect. A.2.3.
Table 5: Transferability results (%) COCO object detection.
Backbone
AP
APso
AP7s
AP3
A
AP
ResNet-50
36.42
58.66
38.90
21.93
39.84
46.74
Residual,
40.74(+4.32)
63.22
44.62
25.01
44.18
52.74
Complete, a
41.35(+4.93)
63.32
45.08
25.63
44,99
53.47
Complete, a,, uniform sparse Complete, adaptive sparse
41.46(45.04)
63.83
44.91
25.07
45.31
53.52
41.69(45.27)
63.86
45.45
25.58
45.52
53.69
RELATED WORKS
We briefly review related works in the aspects of neural network structure design and relevant opti- mization methods.
Neural network design is widely studied in previous literature. From shallow to deep. the shortcut connection plays an important role. Before ResNet, an early practice (Venables & Ripley. 2013) also added linear layer connected from input to the output to train multi-layer perceptrons. Besides, "inception" layer was proposed in (Szegedy et al., 2015) that is composed of shorteut branch and few deeper branches. Except on large networks, shortcut also proved effective in small networks, e.g. MobileNet (Sandler al., 2018), ShuffleNet (Zhang et al., 2018) and MnasNet (Tan et al., 2019). The existence of shortcut eases vanishing/explodingg gradients (He et al., 2016; Srivastava et al., 2015). In this paper, we explain from a topological view that shortcuts offer dense connec- tions and benefit optimization. On the macro structure, there also exist many networks with dense connections. DenseNet (Huang et al., 2017) contacted all preceding layers and passed on the feature maps all subsequent layers in block. HRNet (Sun et al., 2019) benefits from dense high-to-low connections for fine representations. Densely connected networks promote the specific task lo- calization (Tang et al., 2018). Moreover, we optimize the desired network from the complete graph. This is different from MaskConnect (Ahmed & Torresani, 2018) which constrained by K discrete in-degree. This also provides supplement to (Xie et al., 2019) where random graphs generated by different generators are employed to form network.
Above the optimization process, our method is consistent with DARTS (Liu et al., 2019b) which is differentiable. Different from it, we do not adopt alternative optimization strategies for weights and architecture. Joint training can replace the transferring step from one task to another, and obtain task- related topology. Different from sample-based optimization methods (Real et al., 2019), our method obtains optimized architecture in the manner of one-shot. (Bender et al., 2018; Guo et al., 2019) also explored this type and utilized weight-sharing across models to amortize the cost of training. Searching from the full space is evaluated in object detection by NAS-FPN (Ghiasi et al., 2019), in which the feature pyramid is sought in all cross- -ross- connections. In the aspect of semantic segmentation, DeepLab (Liu et al., 2019a) also formed hierarchical architecture to enlarge possible spaces. The sparsity optimization also be observed in other applications, e.g. path
8

---

Under review as a conference paper at ICLR 2020
selection for multi-branch network (Huang & Wang, 2018), and pruning unimportant channels for fast inference (Han et al., 2015).
5 CONCLUSION
In this work, we have found a feasible way for the optimization of topology in neural networks. Motivated by the topological view, the optimization space is defined as a complete graph, through assigning learnable weights which reflect the importance of connections, the optimization of topol- ogy is transformed into learning set of continuous variables of edges. This method is compatible with existing networks, such as ResNet and MobileNet. TopoNet and corresponding sparsity con- straint also proved the effectiveness and transferability in the larger optimization space. Moreover, the observed properties of topology can be used for future works.
REFERENCES
Karim Ahmed and Lorenzo Torresani. Maskconnect: Connectivity learning by gradient descent. In ECCV, 2018.
James Atwood and Don Towsley. Diffussio-coovvllttionaal neural networks. In NeurlPS, 2016.
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc Le. Understand- ing and simplifying one-shot architecture search. In ICML, 2018.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In CVPR, 2019.
Terrance DeVries and Graham W Taylor. Improved regularization convolutional neural networks with cutout. arXiv preprint arXiv: 04552, 2017.
Golnaz Ghiasi, Tsung-Y. Yi Lin, and Quoc V Le. Nas-fpn: Learning scalable feature pyramid archi- tecture for object detection. CVPR, 2019.
Ross Girshick, Ilija Radosavovic, Georgia Gkioxari, Piotr Dollár, and Kaiming He. Detectron, 2018.
Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. Ms-celeb-Imi A dataset and benchmark for large-scale face recognition. In ECCV, 2016
Zichao Guo, Xiangyu Zhang, Haoyuan MU, Wen Heng. Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. arXiv preprint arXiv:1904.00420 2019.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In NeurlPS, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi- nov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv: 1207.0007 2012.
Andrew Howard. Mark Sandler, Grace Chu, Chinh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. arXiv preprint arXiv:
2019.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04888 2017.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks, In CVPR, 2017.

---

Under review as a conference paper at ICLR 2020
Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks. In ECCV, 2018.
Ira
Steven M Seitz, Daniel Miller, and Evan Brossard. The megaface benchmark: million faces for recognition at scale. In CVPR, 2016.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In NeurlPS, 2012.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Tsung Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
Chenxi Liu, I Liang-Chien Chieh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan Yuille, and Li Fei- Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In CVPR, 2019a.
Hanxiao Liu. Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search, In ICLR, 2019b.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. ICIR, 2017.
Hong-Wo Wei Ng and Stefan Winkler. A data-driven approach to cleaning large face datasets. In ICIP, 2014.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Juan-Manuel Pérez-Rúa, Moez Baccouche, and Stephane Pateux. Efficient progressive neural archi- tecture search. arXiv preprint arXiv: 0039), 2018.
JP Rauschecker. Neuronal mechanisms of developmental plasticity in the cat's visual system. Hu- man neurobiology. 1984.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. AAAI, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.155 2014.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint NX05::550000388 2015.
Ke Sun, Yang Zhao, Borui Jiang, Tianheng Cheng, Bin Xiao, Dong Liu, Yadong Mu, Xinggang Wang, Wenyu Liu, and Jingdong Wang. High-resolution representations for labeling pixels and regions. arXiv preprint 1rXiv:1904.445//1 2019.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initializa- tion and momentum in deep learning. In ICMI., 2013.
10

---

Under review as a conference paper at ICLR 2020
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. AAAI, 2017.
Mingxing Tan and Quoc V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. JCML, 2019.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe Le. Mnasnet: Platform-aware neural architecture search for mobile. In CVPR, 2019.
Zhiqiang Tang, Xi Peng, Shijie Geng, Lingfei Wu, Shaoting Zhang, and Dimitris Metaxas. Quan- tized densely connected u-nets for efficient landmark localization. In ECCV, 2018.
William N Venables and Brian D Ripley. Modern applied statistics with S-PLUS. Springer Science & Business Media, 2013.
Saining Xie, Alexander Kirillov, Ross Girshick, and Kaiming He. Exploring randomly wired neural networks for image recognition. arXiv preprint
2019.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In CVPR, 2018.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In CVPR, 2018
A APPENDIX
A.1 EXPERIMENTAL DETAILS FOR CLASSICAL NETWORKS
A.1.1 RESNET-CIFAR
On CIFARI00, we train ResNets using 2 GPUs. We follow the hyperparameter settings in paper (DeVries & Taylor, 2017), which is init Ir = 0.1 divide by 5 times at 60th, 120th, 160th epochs. We train for 200 epochs with batchsize 128 and weight decay 5e-4. Nesterov (Sutskever et al., 2013) momentum of 0.9 without dampening is also used. The training and test size 32 x 32, We report classification accuracy the validation set.
A.1.2 MOBILENETV2
On ImageNet, The models are trained on the 1.28 million training images and evaluated on the 50k validation images. We train MobileNetV2 using 16 GPUs for 200 epochs with batch size of 1024. The initial learning rate is 0.4 and cosine shaped learning rate decay (Loshchilov & Hutter, 2017) is adopted. Following (Sandler et al., 2018), we use a weight decay of 4e-5 and dropout (Hinton al., 2012) of 0.2. Nesterov momentum of 0.9 without dampening is also used. The training and test size 224 x 224. We report classification accuracy on the validation set. And the data augmentation used the same Sect. A.1.1.
A.2 EXPERIMENTAL DETAILS FOR TOPONETS
A.2.1 IMAGE CLASSIFICATION
On ImageNet, TotoNets of random, residual and complete are trained using 16 GPUs for 100 epochs with batch size of 1024. The initial learning rate is 0.4, and cosine shaped learning rate decay is adopted. We use weight decay of e-4 and a Nesterov momentum of 0.9 without dampening. Dropout is not used. Label smoothing regularization (Szegedy al., 2017) with a coefficient of 0.1 is also used. The training and test size is 224 224. For fair comparison, ResNet-50 are trained under the same setting. We report classification accuracy on the validation set using repeat runs.
All of our experiments were performed using NVIDIA Tesla V100 GPUs with our implementation in PyTorch (Paszke al., 2017).
11

---

Under review as a conference paper at ICLR 2020
A.2.2 TRANSFERABILITY ON OBJECT DETECTION
We perform experiments on the 80 category COCO detection dataset. We train using the COCO train2017 dataset, and test using the COCO val2017 dataset. Following (Lin et al., 2017), the input image is resized such that its shorter side has 800 pixels. We adopt synchronized SGD training on 16 GPUs. A mini-batch involves 2 images per GPU and 256 anchors per image. We use a weight decay of e-4 and a momentum of 0.9 without dampening, Our fine-tuning is based on 1x setting of the publicly available Detectron (Girshick et al., 2018).
A.2.3 TRANSFERABILITY ON FACE RECOGNITION
We evaluate and compare the performance of proposed methods as feature extractors for face recognition. The most challenging face identification task MegaFace dataset (Kemelmacher- Shlizerman et al., 2016) is selected. The MegaFace dataset includes 1M images of 690K different individuals as the gallery Winkler, 2014) as the probe set. Following (Deng et al., 2019), we select ArcFace as loss, and MSIMV2 (Guo al., 2016) for training set. For feature extraction, hidden FC layer is added before the classifier. And the output dimension of FC in the classifier is set equal to the number of individuals of the training set. And the dimension of feature representation set 256 for all experiments. We train for 80k iterations with batch size of 512 using 8 GPUs. The initial learning rate is 0.0375. Following (Deng et al., 2019), the learning rate is lowered by 10 times at 30k, 50k and 70k iterations. We use weight decay of e-4 and a momentum of 0.9 without dampening.
and 100K photos of 530 unique individuals from FaceScrub (Ng &
Identification results are given in Table 6, and it should be noted that the Rank-1 accuracy of one mil- lion distractors is extremely strict. TopoNets achieve outperforming performances against ResNet- 50 with less calculation costs. Comparisons among TopoNets also illustrate the applicability of the optimization for topology in face recognition.
Table 6: Optimization results on MegaFace face identification.
Params (M) Conv/BN
Rank (%) (1M distractors)
Network
FC
ResNet-50
23.51
25.69
94.69
Residual, 1
20.64
12.85
16(10())44
Complete, a
20.64
12.85
96.44(+1.75)
Complete, a, uniform sparse
20.64
12.85
96.51(+1.82)
Complete, a. adaptive sparse
20.64
12.85
96.62(+1.933
A.3 REIMPLEMENTATION AND COMPARISION WITH RANDOMLY- WIRED NETWORKS
Since Xie et al. (2019) has not released their codes, we conduct comparisons using TopoNet through replacing graphs to form networks. The three graph generators of ER, BA, WS are performed using NetworkX with the best configs in their paper. For fair comparison, we do not adopt droppath and dropout for all experiments. The results are given as follows using repeat runs. These results further prove the effectiveness of our optimization method over random baselines.
Table 7: Comparasion with handomly-wired neural networks.
Topology
Top-1 Ace (%)
ER (P=0.2)
77.76 + 0.23
(M=5)
78.08 + 0.17
WS (K=4, P=0.75)
78.19 + 0.25
our method
78.60 + 0.06
hhtml
12

---

Under review as a conference paper at ICLR 2020
A.4 OPTIMIZED TOPOLOGIES OVER MULTIPLE RUNS
We give results of multiple runs in Fig 5.
run1 @78.51
run2@78.71
run3@78.61
run4@78.60
run5@78.58
Figure 5: We run several times and analyze the optimized topologies, which achieve top-1 accuracy of 78.60 + 0.06%. The matrices optimized from multiple runs have high similarity, where weights on the diagonal are larger, and connections on the off-diagonal are relatively sparse. This indicates nodes with adjacent topological orderings have more information interactions. Besides, some essen- tial long-range connections are kept as inputs to intermediate layers. Many nodes contribute to the final output, leading better representations for the subsequent layers.
A.5 STAND-ALONE ACCURACIES DURING THE SEARCH PROCESS
We give the Htand-alone accuracies in Fig. 6.
13

---

Under review as a conference paper at ICLR 2020
ckpt1 @77.47
ckpt2@77.89
Training Process
ckpt3@77.98
ckpt4@78.18
ckpt5@78.31
ckpt6@78.60
Figure 6: Stand-alone top-1 accuracies the validation set during optimization. We select topolo- gies under different iterations and freeze their a. This reveals the process topology changes. Then stand-alone accuracies are obtained by training the parameters of network from scratch. It is clear that the expression ability of topology itself increases with the training process.
14


Reviews:
[{"review_id": "SJlSSBqYcr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*UPDATE* I have read the other reviews, authors' comments and the revised version of the manuscript. I have modified my rating to accept. The updated version, with variance across runs reported, comparison to randomly wired networks, and clearer writing, is substantially better. The core idea is simple to understand in retrospect, and could lead to more follow-up work in the vein of DenseNets and DART (with a more constrained search space).\n\nThe paper proposes a refinement of the idea behind DenseNets -- rather than summing over all previous layers' outputs, sum a weighted combination instead where the weights are learned. This idea can be extended to search through the space of all possible residual connections, which they call TopoNet. This is practically achieved by enforcing a sparsity constraint on the learned weights. There is an additional nuance when enforcing the sparsity constraint: downstream layers have many more incoming residual connections, and may need an appropriately scaled sparsity penalty.\n\n It may help to clarify in the text that weights can be positive or negative (the current motivation from the point of view of residuals at different intervals suggests all the weights should be non-negative).\nTable 3 is baffling. Were the initial edge values (column 2) chosen so as to make the number of params and FLOPS somewhat comparable across different rows? How were these initial edge values set (seems very specific for N_3 to go from 46 to 103 when residual interval goes from 4 to 2, etc.)? The comment that number of params and FLOPS changes are negligible is puzzling; clearly Random, p=0.01 should use much fewer FLOPS than Random, p=0.99.\nWithout additional information behind the numbers for the baselines in Table 3, it is unclear if TopoNets indeed give an improvement over the baselines (Random and Residual).\nThe text will also benefit from a careful elaboration of the differences in the Random baselines in the paper vs. the Xie et al approach of trying random architectures.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1gIRLVo5r", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I. Summary of the paper\n\nThis paper describes a principled strategy for searching for the most\nsuitable neural network architecture out of a particular class of\narchitectures. Specifically, the problem is framed as an optimisation\nproblem over a set of directed acyclic graphs (DAGs) that correspond to\npotential network architectures. By optimising the edge weights of this\nrepresentation, a suitable architecture can be generated.\nIn addition to the aforementioned optimisation scheme, the paper also\npresents a regularisation that results in *sparse* networks, i.e.\nnetworks with a smaller number of edges. Multiple experiments on\n'tuning' existing architectures on several data sets conclude the paper.\n\nII. Summary of the review\n\nThis paper discusses a highly relevant subject, namely how to select\nneural network architectures in a principled manner. While the presented\nwork already goes into a good direction, I cannot give it my endorsement\nfor acceptance because of the following reasons:\n\n  - The paper is lacking clarity: concepts could be explained somewhat\n    better, and the paper is suffering from language/grammar issues that\n    make it harder to understand the contents.\n\n  - Lack of experimental or theoretical depth: the proposed method is\n    presented as-is; no theoretical analysis of its behaviour is\n    performed; while this is not necessarily a problem, as there are\n    several empirical experiments, the experimental section is not\n    sufficiently detailed: for example, no limitations of the method are\n    being discussed and the presented results are not state-of-the-art\n    accuracies.\n\nNevertheless, I want to point out that this paper has the potential to\nbecome an important contribution to the community; it is absolutely\nclear that more principled approaches are required to select network\narchitectures.\n\nIn the following, I will comment on the individual aspects in more\ndetail.\n\nIII. Detailed comments (clarity)\n\n- The abstract could be improved in terms of its logical flow. Instead\n  of trying to introduce new terminology (macro/micro etc.) here, the\n  abstract should rather state directly that this paper frames network\n  architecture selection as an optimisation problem over DAGs.\n\n- The use of topology is slightly non-standard here. What is the meaning\n  behind the 'macro' and 'micro' operations? This should be explained\n  somewhat better.\n\n- Figure 1 should be extended to show an example of how the depicted\n  graphs are described through the terminology mentioned in the paper.\n  For example, individual edges or nodes could be highlighted and\n  referred to in the text to make the 'mapping' clearer.\n\n- I do not understand how operations such as *addition* are represented\n  in the DAG. Ideally, this should also be elucidated by a figure.\n\n- When discussing 'intervals' of residual connections, I am assuming\n  that the paper refers to how many layers are skipped? If so, this\n  should be mentioned and defined explicitly.\n\n- The term 'searching space' should be replaced by 'search space', as\n  the latter is more standard usage.\n\n- I do not understand why the optimisation of the topology can decrease\n  the computational burden, as claimed on p. 2. The optimisation process\n  still has to be performed, just like the training of the network,\n  correct? Am I misunderstanding this?\n\n- The caption of Figure 2 could be extended; does a single node type\n  mean that the complete network only consists of nodes of that type?\n  Moreover, accuracies/errors should be shown in addition to the loss\n  curves.\n\n- The term 'dense connection' is vague; I think the paper should use\n  'densely-connected graph' here.\n\n- The sentence 'Among these nodes, [...]' refers to the *whole* network,\n  and not to the way the output tensor $\\mathbf{x}_i$ is processed. Am I\n  understanding this correctly?\n\n- I do not understand the initial sentences in Section 2.2; what is the\n  meaning of 'cell' in this case?\n\n- The term 'topological structures' should be renamed to '(sub)graphs'\n  in order to improve clarity.\n\n- I do not understand the comment on sparsity in Section 2.4. How are\n  'moderate sparsity' and Figure 2 connected?\n\n- In the algorithm, I would use 'Sparsity Type' instead of 'Sparse Type'\n  to refer to the parameter.\n\n- What does 'Complete' (without $\\mathbf{\\alpha}$) mean in Table 3?\n\n- The footnote below Table 3 is not referenced anywhere in the text or\n  in the table.\n\n- In Figure 3, are the adjacency matrices consistent? What happens if\n  the training process is repeated? It would be highly interesting to\n  show 'averaged' matrices over multiple runs.\n\nIV. Detailed comments (experiments & theory)\n\n- A theoretical analysis of the proposed method would be interesting.\n  Does the optimisation always converge? Are minima unique? What is the\n  computational complexity?\n\n  At least some of these aspects should be discussed.\n\n- The limitations of the proposed method are not explained. For example,\n  what is the meaning of the sentence on p. 2 about 'excluding the\n  influence of the mixture of different layers/nodes'?\n\n  It is my understanding that the proposed method can only change the\n  *connections* between blocks of a network, but not the type of layers.\n  Is this correct? If so, it would be a major limitation that should be\n  mentioned explicitly.\n\n- Another limitation that is not discussed is the scaling to very deep\n  networks. How problematic is it to model all potential connections in\n  such a network? Are there limits to the current optimisation scheme?\n  This needs to be assessed in the experimental section.\n\n- For all experimental tables, standard deviations and means should be\n  provided. This is necessary in order to assess the stability of the\n  proposed method, because there are multiple sources of stochasticity:\n  one arising from the optimisation procedure, the other one arising\n  from the training of the network itself.\n\n- The results reported for the experiments are somewhat behind the\n  state-of-the-art in terms of accuracy values. This should be stated\n  more clearly; I assume that it is caused by limitations of the\n  proposed method, which prohibit an application to very recent\n  architectures. Is this correct? If so, it should at least be\n  mentioned.\n\n- The claim that nodes at the start of a topological ordering contribute\n  more to specific stages needs to be (empirically) proven.\n\nV. Style issues\n\nThe paper is not easy to read because of several non-standard phrases or\nexpressions.\n\n- The phrase 'in topology' is often added to a sentence where it does\n  not entirely make sense. For example, '[The] architecture can be\n  expressed as a directed acyclic graph (DAG) in topology'. I do not\n  see the necessity of adding 'in topology' here. There are other places\n  at well from which I would remove this phrase.\n\n- 'effective networks' --> 'effective network architectures'\n\n- 'largely affects' --> 'largely affect'\n\n- 'Motivated by which' --> 'Motivated by this'\n\n- 'innovative method' --> 'method' (or 'novel method')\n\n- 'as a complete graph, through' --> 'as a complete graph, and through'\n\n- 'auxiliary sparsity constraint' --> 'an auxiliary sparsity constraint'\n\n- 'named as TopoNet' --> 'called TopoNet'\n\n- 'At initial periods' --> 'Previously' (I am not sure I understand\n  this correctly)\n\n- 'red signs' --> 'red arrows'\n\n- 'for its topology' --> 'in terms of its topology' (?)\n\n- 'both combining' --> 'both a combination' (?)\n\n- 'number of interval' --> 'number of intervals'\n\n- 'straight connected' --> 'directly connected'\n\n- 'conduct transformation' --> 'performs a transformation'\n\n- What is the meaning of the phrase 'These may cover the influence\n  [...]'? Is this a reference to limitations of existing networks?\n\n- 'opted from' --> 'chosen from'\n\n- 'Following two simple design rules' --> 'We follow two simple design\n  rules'\n\n- '1000-dimension' --> '$1000$-dimensional'\n\n- 'We raise two ways' --> 'We describe two ways'\n\n- 'consited' --> 'consisted' / 'consists'\n\n- 'deepen the depth' --> 'increase the depth'\n\n- 'origin' --> 'original'\n\n- 'promotions' --> 'improvements'\n\n- 'can make more profit' --> 'can be useful to improve performance' (I\n  am guessing this from the context)\n\n- 'sparseness on representation' --> 'sparsity'\n\n- 'Adaptive one' --> 'The adaptive one'\n\n- 'At the fore' --> 'At the beginning/start'\n\n- I do not understand the sentence about the 'free lunch'. Does it refer\n  to the fact that some connections can still be removed from the\n  network without decreasing accuracy?\n\n- 'less computation costs' --> 'lower computation costs'\n\n- 'shortcut offers' --> 'shortcuts offer'\n\n- 'benefits optimization' --> 'benefit optimization'\n\n- 'feasible way to the optimization' --> 'feasible way for the optimization'\n\n- Some references in the bibliography are not capitalised consistently", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HklYNPCY5r", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "-------------------------      Update after rebuttal    ---------------------------\n\n\nThank you for addressing my concerns. I feel the rebuttal did improve the paper, e.g., the significance of results can be evaluated better now. I still like the overall idea of the paper as optimizing connectivity patterns in architectures has so far mostly been ignored while it is actually straight-forward to do (as shown in this work).  I increased my score accordingly. However, the novelty and significance of this work is still limited in my option and therefore I do not argue heavily in favor of accepting this submission.\n\n\n--------------------------------------------------------------------------------------------------------------------------------------------------\n\n\n\n\nThe authors propose a method for learning the connections/ connectivity pattern (dubbed: the topology; meaning which layer is directly connected to which other layer) in neural networks. They do so by weighting connections between layers (e.g., by weighting skip connections) with a real valued parameter. This real-valued parameterization of the connections is then optimized by gradient descent along with the weights of neural networks. The authors also propose L1 regularization on the connectivity parameters to induce sparsity. The proposed method is evaluated by optimizing the topology for ResNets, MobileNetsV2 and their proposed “TopoNets”.\n \nOriginality and significance. The manuscript addresses an interesting problem: while there has been lots of work on manually designing better architectures as well as automated design (a.k.a. neural architecture search, NAS), there is little work on optimizing the overall topology (meaning the connectivity patterns between layers). Most prior work solely focuses on search for blocks or cells and then stacking these cells in a pre-defined, not-optimized manner. However, there has been some work also including this in architecture search (e.g., [1,2,3]), and especially the work [4] seems very related but is not discussed. The authors of [4] propose, very similar to this submission, a gradient-based optimization of the connections (in a different way though). The proposed method for optimizing the topology is also very similar to DARTS, simply applied to the connectivity pattern rather than on the operations-level. However, here the topology is optimized along with the network’s weights on the training data rather than the bi-level optimization from DARTS, where the architectural parameters are optimized on the validation data instead (which is very reasonable as one usually considers the architecture as a hyperparameter). I wonder if this has also been considered/tested by the authors of this paper as I would consider the topology to be a hyperparameter which should not be optimized on training but rather validation data. Knowing [4] and DARTS, the proposed method seems to be rather incremental and straightforward rather than ground breaking. While the proposed method allows for more flexible topologies, it introduces different “stages” for their TopoNets, which are actually a similar concept as blocks or cells from the NAS literature. This again does not allow connections between arbitrary layers (but rather only between layers in the same stage; to the best of my understanding). Empirical results show rather small improvements and their significance is unclear (see next paragraph).\nClarity and quality. The paper is mostly well written, well motivated and easy to follow. The mathematical formalism is a little vague at some points (e.g., in Section 2.1., the notation G is used for defining a graph and later in Equation (2) as a function computing feature maps in networks). While the literature on manual design of architectures is thoroughly reviewed, there is missing related work in the context of neural architecture search, as already discussed above. The quality of the results is questionable as differences in accuracies are in almost all experiment rather small (e.g., tuning MobileNetsV2 on Imagenet: 72.62% (original) Top-1 accuracy vs. 72.84% (optimized) and it seems that the authors do only report results for a single run of experiments. In order to assess if the differences are actually statistically significant, the authors would need to report several runs and would need to state, e.g., means and standard deviations. \n\nOverall, the authors address an interesting problem, which seems to have fallen into oblivion in current NAS literature: while researcher optimize cells, which are then stacked to build the final model, not many researcher look into connectivity patterns / topology on the macro level, meaning connections across cells and how cells should be stacked. This paper addresses this problem to some extent. However, the proposed optimization method is, in my opinion, rather incremental (with respect to DARTS and [4]) and therefore of limited novelty and significance. It is currently hard to assess the empirical results due to rather small improvements and the lack of repeated runs of experiments. Mainly for these two reasons, I do not recommend the paper for acceptance.\n\n\n\n[1] Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Quoc V. Le, and Alex Kurakin. Large-scale evolution of image classifiers. ICML, 2017.\n[2] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Efficient multi-objective neural architecture search via lamarckian evolution. ICLR, 2019.\n[3] Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. ICML, 2018\n[4] Karim Ahmed and Lorenzo Torresani. Maskconnect: Connectivity learning by gradient descent. ECCV, 2018\n\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Byxx0fHTtr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors proposed a method to optimize the topology of neural networks in a soft fashion. The main idea is to formulate the network as a complete graph (or a sequence of complete subgraphs), and to optimize the relative importance of each edge using gradient descent. The overall approach is similar to differentiable architecture search, except that (1) the continuous architecture is optimized wrt the training set (instead of the validation set), and (2) the learned architecture is never discretized at the end of training.\n\nThe paper is well-organized and easy to follow. The authors have also conducted controlled experiments to convincingly show that the method is leading to improvement. \n\nI'm a bit concerned about the technical novelty, however, as the approach can be viewed as an application of (a simplified version of) differentiable NAS to a search space analogous to the one used in [1]. In fact, the notion of soft topology has already been introduced in this prior work (Figure 2 in [1]: \"The aggregation is done by weighted sum with learnable positive weights w0, w1, w2\"), which was also optimized using gradient descent. A difference between this work and [1] is that whether the underlying graph is complete or randomly generated, but such a distinction is minor (we can always get densely connected random graphs by adjusting the hyperparameters of the graph generator).\n\nIn addition, I'm not sure whether the learned continuous \\alpha can be conveniently referred to as \"topology\". Note the mathematical definition of topology is discrete by nature. I believe the authors would need to either revise this terminology (e.g., by referring to it as “soft topology”, as a generalized definition of hard topology), or provide a way to induce discrete subgraphs from the continuous architecture. Sparsity regularization alone may not be sufficient as the non-zero \\alpha's are still real-valued.\n\nMinor issue:\nI like Figure 1 a lot. However, it seems the equivalence between the 3rd and the 4th sub-figures in Figure 1 can only be established for ResNet-V2 blocks, where there is no ReLU after each addition. It is not immediately obvious how this analysis can generalize to ResNet-V1 blocks (which still offers reasonably good empirical performance).\n\n[1] Xie, Saining, et al. \"Exploring randomly wired neural networks for image recognition.\" arXiv preprint arXiv:1904.01569 (2019).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an approach for architecture search by framing it as a differentiable optimization over directed acyclic graphs. While the reviewers appreciated the significance of architecture search as a problem and acknowledged that the paper proposes a principled approach for this problem, there were concerns about lack of experimental rigor, and limited technical novelty over some existing works. 
Decision: Reject