Diving into Optimization of Topology in Neural Networks

Seeking effective networks has become one of the most crucial and practical areas in deep learning . The architecture of a neural network can be represented as a directed acyclic graph , whose nodes denote the transformation of layers and edges represent information flow . Despite the selection of micro node operations , macro connections among the whole network , noted as topology , largely affect the optimization process . We first rethink the residual connections via a new topological view and observe the benefits provided by dense connections to the optimization . Motivated by this , we propose an novel method to optimize the topology of a neural network . The optimization space is defined as a complete graph , and through assigning learnable weights which reflect the importance of connections , the optimization of topology is transformed into learning a set of continuous variables of edges . To extend the optimization to larger search spaces , a new series of networks , called TopoNet , are designed . To further focus on critical edges and promote generalization ability in dense topologies , an auxiliary sparsity constraint is adopted to constrain the distribution of edges . Experiments on classical networks prove the effectiveness of the optimization of topology . Experiments with TopoNets further verify both availability and transferability of the proposed method in different tasks e.g . image classification , object detection , and face recognition . 1 INTRODUCTION . Deep learning successfully transits the feature engineering from manual to automatic design . As a tendency , seeking effective neural networks gradually becomes an important and practical direction . Topologically , the architecture of a network can be expressed as a direct acyclic graph , whose nodes denote the transformation of layers and edges represent information flow . Corresponding to micro operations within a layer/block , the macro connections ( Atwood & Towsley , 2016 ; Pérez-Rúa et al. , 2018 ) between layers also experienced a series of evolutions . And we denote the macro connections as topology in this work . In initial literature , AlexNet ( Krizhevsky et al. , 2012 ) , VGGNet ( Simonyan & Zisserman , 2014 ) with plain topology were proposed . Due to the problems of gradient vanishing and exploding , extending the network to a deeper level for better representation is nearly difficult . To better adapt the optimization process of gradient descent , GoogleNet ( Szegedy et al. , 2015 ) adopted parallel modules , and Highway networks ( Srivastava et al. , 2015 ) utilized gating units to regular the flow of information , resulting in multipath and elastic topologies . Driven by the significance of depth , the residual block consisted of residual mapping and shortcut was raised in ResNet ( He et al. , 2016 ) . Topological changes in neural networks successfully scaled up neural networks to hundreds or even thousands of layers . The residual topology was widely approved and applied in the following works , e.g . MobileNet ( Sandler et al. , 2018 ; Howard et al. , 2019 ) and ShuffleNet ( Zhang et al. , 2018 ) . Divergent from relative sparse topologies , DenseNet ( Huang et al. , 2017 ) wired densely within a block to reuse features fully . Recent advances in computer vision also explore neural architecture search ( NAS ) methods ( Zoph et al. , 2018 ; Liu et al. , 2019b ; Tan et al. , 2019 ) to jointly search micro operations and topology . To trade-off efficiency and performance , their topologies are handdesigned stacked patterns and constrained in limited searching spaces . Orthogonally , randomly wired networks ( Xie et al. , 2019 ) explored random topology as architecture by different generators , and obtained competitive performance . These trends reflect the great impact of topology on the optimization of neural networks . Echoing this perspective , we wonder : Can topology itself in neural networks be optimized ? What is the suitable route to do this ? To answer these questions , we start by understanding the most representative residual connection in terms of topology . It formulates x+ F ( x ) as a basic component , in which x represents the identity shortcut , and F ( x ) denotes the residual mapping . Figure 1 presents two residual architectures with a interval of 1 and 2 respectively . And the number of intervals represents the number of layers that make up a block . If we map both combining ( e.g. , addition ) and transformation ( e.g. , 3× 3 conv ) to a node , and flow of features to an edge , then the architecture can be expressed as a directed acyclic graph ( DAG ) . From the topological view , the residual connections are rather denser than the natural perspective . This novel representation illustrates residual networks perform multiple feed-forward paths instead of a single deep network . When the interval degrades to 1 , its topology evolves into a complete DAG . For a complete DAG with n nodes , the number of available paths from input to output is ( n − 1 ) ! . As a result , most layers are directly connected to the input and output , resulting in direct access to the gradients and the original input . So we make a hypothesis that dense wirings in topology benefit the optimization process . To verify the hypothesis and explore the effects of differently wired topologies , toy experiments on CIFAR-100 ( Krizhevsky et al. , 2009 ) are conducted . To ensure universality , three types of layer are selected respectively , including 3 × 3 conv , 3 × 3 inverted bottleneck ( Sandler et al. , 2018 ) and 3 × 3 separable depthwise conv ( Howard et al. , 2017 ) . Defining the total number of nodes , we establish three series of networks under different intervals . Loss curves during training and test performance are given in Figure 2 . It demonstrates an obvious topology-induced impact on optimization . First , it shows denser networks ( intervals of 1 and 2 ) can achieve lower training losses in the majority of cases . Second , the optimal topology is different for different node types ( 2 for conv , 1 for depthwise and inverted bottleneck ) . It is possible due to that not all connections are beneficial , and the aggregation of features should be discretionary . So we modify the hypothesis as the topology with dense and effective connections are necessary for networks that are compatible with the optimization process . Regardless of the node types , the optimal structure is always a subset of the complete graph . Naturally , finding the optimal topology in a full search space is equivalent to finding the optimal sub-graph in the complete graph . To focus on the optimization of topology itself , in this work , we exclude the influence of the mixture of different layers/nodes and select the complete DAG as the search space . Through assigning learnable weights which reflect the importance of connections of edges , the topology can be optimized through gradient descent . Then task-related architectures can be achieved conveniently . Besides , this optimization method is also compatible with existing networks . Given a basic network , optimization of topology can lead to improvement without additional computing burdens . To verify the effectiveness of the optimization of topology in larger search spaces , we also propose a welldesigned network named TopoNet . To solve the optimization problem in a large space and promote generalization , we attach auxiliary sparsity regularization to the distribution of edges , resulting in critical connections and better generalization . Our contributions are as follows : • We first rethink the residual connections via a new topological view and observe the benefits provided by dense connections to the optimization . • We propose an innovative method to optimize the topology of a neural network . The optimization space is defined as a complete graph , through assigning learnable weights which reflect the importance of connections , the optimization of topology is transformed into learning a set of continuous variables of edges . • The proposed optimization method of topology is effective for existing networks . • When extended to larger search space , TopoNets and optimization using sparsity constraint verify the availability and transferability in different tasks , including image classification on ImageNet , object detection on COCO and face recognition on MegaFace . 2 METHODOLOGY . 2.1 TOPOLOGICAL REPRESENTATION OF ARCHITECTURE . We represent the neural network using a directed acyclic graph ( DAG ) in topology : G = ( N , E ) . Each node ni ∈ N owns an operation oi , such as convolution or identity mapping , parametrized by wi , where i stands for the topological ordering . Edge eji = ( j , i , αji ) ∈ E means the flow of features from node j to node i , and the importance of the connection is determined by αji . Each node aggregates inputs from connected pre-order nodes , and performs a transformation to get an output tensor xi . Then xi is sent out to the post-order nodes by the output edges . Among these nodes , the first in topological ordering is appointed to be input , and the last to be output . These two nodes do not transform features , just send/gather features . Except for the input and output nodes , other nodes perform feature transformation , such as convolution , activation and normalization . To focus on the impact of topological changes , the same type of operation is applied for all nodes . It can be formulated as follows : xi = oi ( x ′ i ; wi ) , where x ′ i = ∑ eji∈E αji · xj . ( 1 ) For a network with k stages , k DAGs are initialized and connected in series . Each graph is linked to its preceding or succeeding stage by output or input node . The topology of network can be denoted as a set α = { αkji } , and weights of transformation as a set w = { wki } . Given an input x , the mapping function from sample to feature can be written as : F ( x ) = T k ( · · · T 2 ( T 1 ( x ; α1 , w1 ) ; α2 , w2 ) · · · ; αk , wk ) , ( 2 ) where T ( · ) denotes the mapping function computing feature maps of G . 2.2 SEARCH SPACE AND OPTIMIZATION METHOD . In most previous literature ( Tan et al. , 2019 ; Guo et al. , 2019 ) , searching for a cell that is later stacked as blocks for a deep network is an expedient solution to trade-off search efficiency and result optimality . But it constrains the possible topologies in a small subset of all possible graphs . Different from them , we relax connections from block-wise to layer-wise , and define the search space as a complete DAG . It provides all possible connections and is much wider than cell-based or block-based approaches . For a complete DAG with N nodes , the search space contains 2N ( N−1 ) /2 possible topological structures . Assigning learnable weights to edges transforms the optimization of topology into learning a set of continuous variables weights for edges α . It provides a differentiable type to update topology through gradients according to the loss of task noted as Ltask . Compared with existing differentiable methods ( e.g . DARTS ) , there is no need to conduct argmax operation for α since it determines the fusion of features instead of the selection of operations . Continuous α also guarantees the consistency between training and testing . During the process of optimization , both w and α are optimized jointly . This optimization process can be viewed as : min w , α Ltask = min w , α L ( F ( x ; w , α ) , y ) ( 3 )

Summary: The authors proposed a method to optimize the topology of neural networks in a soft fashion. The main idea is to formulate the network as a complete graph (or a sequence of complete subgraphs), and to optimize the relative importance of each edge using gradient descent. The overall approach is similar to differentiable architecture search, except that (1) the continuous architecture is optimized wrt the training set (instead of the validation set), and (2) the learned architecture is never discretized at the end of training.