Dynamic Graph: Learning Instance-aware Connectivity for Neural Networks

1 INTRODUCTION . Deep neural networks have driven a shift from feature engineering to feature learning . The great progress largely comes from well-designed networks with increasing capacity of models ( He et al. , 2016a ; Xie et al. , 2017 ; Huang et al. , 2017 ; Tan & Le , 2019 ) . To achieve the superior performance , a useful practice is to add more layers ( Szegedy et al. , 2015 ) or expand the size of existing convolutions ( kernel width , number of channels ) ( Huang et al. , 2019 ; Tan & Le , 2019 ; Mahajan et al. , 2018 ) . Meantime , the computational cost significantly increases , hindering the deployment of these models in realistic scenarios . Instead of adding much more computational burden , we prefer adding sampledependent modules to networks , increasing the model capacity by accommodating the data variance . Several existing work attempt to augment the sample-dependent modules into network . For example , Squeeze-and-Excitation network ( SENet ) ( Hu et al. , 2018 ) learns to scale the activations in the channel dimension conditionally on the input . Conditionally Parameterized Convolution ( CondConv ) ( Yang et al. , 2019 ) uses over-parameterization weights and generates individual convolutional kernels for each sample . GaterNet ( Chen et al. , 2018 ) adopts a gate network to extract features and generate sparse binary masks for selecting filters in the backbone network based upon inputs . All these methods focus on the adjustment of the micro structure of neural networks , using a data-dependent module to influence the feature representation at the same level . Recall the deep neural network to mammalian brain mechanism in biology ( Rauschecker , 1984 ) , the neurons are linked by synapses and responsible for sensing different information , the synapses are activated to varying degrees when the neurons perceive external information . Such a phenomenon inspires us to design a data-dependent network structure so that different samples will activate different network paths . In this paper , we learn to optimize the connectivity of neural networks based upon inputs . Instead of using stacked-style or hand-designed manners , we allow more flexible selection for forwarding paths . Specifically , we reformulate the network into a directed acyclic graph , where nodes represent the convolution block while edges indicate connections . Different from randomly wired neural networks ( Xie et al. , 2019 ) that generate random graphs as connectivity using predefined generators , we rewire the graph as a complete graph so that all nodes establish connections with each other . Such a setting allows more possible connections and makes the task of finding the most suitable connectivity for each sample equivalent to finding the optimal sub-graph in the complete graph . In the graph , each node aggregates features from the preceding nodes , performs feature transformation ( e.g . convolution , normalization , and non-linear operations ) , and distributes the transformed features to the succeeding nodes . The output of the last node in the topological order is employed as the representation through the graph . To adjust the contribution of different nodes to the feature representation , we further assign weights to the edges in the graph . The weights are generated dynamically for each input via an extra module ( denoted as router ) along with each node . During the inference , only crucial connections are maintained , which creates different paths for different instances . As the connectivity for each sample is generated through non-linear functions determined by routers , our method can enable the networks to have more representation power than the static network . We call our method as the Dynamic Graph Network ( DG-Net ) . It doesn ’ t increase the depth or width of the network , while only introduces an extra negligible cost to compute the edge weights and aggregate the features . To facilitate the training , we represent the network connection of each sample as a adjacent matrix and design a buffer mechanism to cache the matrices of a sample batch during training . With the buffer mechanism , we can conveniently aggregate the feature maps in the forward pass and compute the gradient in the backward pass by looking up the adjacent matrices . The main contributions of our work are as follows : • We first introduce the dynamic connectivity based upon inputs to exploit the model capacity of neural networks . Without bells and whistles , simply replacing static connectivity with dynamic one in many networks achieves solid improvement with only a slight increase of ( ∼ 1 % ) parameters and ( ∼ 2 % ) computational cost ( see table 1 ) . • DG-Net is easy and memory-conserving to train . The parameters of networks and routers can be optimized in a differentiable manner . We also design a buffer mechanism to conveniently access the network connectivity , in order to aggregate the feature maps in the forward pass and compute the gradient in the backward pass . • We show that DG-Net not only improves the performance for human-designed networks ( e.g . MobielNetV2 , ResNet , ResNeXt ) but also boosts the performance for automatically searched architectures ( e.g . RegNet ) . It demonstrates good generalization ability on ImageNet classification ( see table 1 ) and COCO object detection ( see table 2 ) tasks . 2 RELATED WORK . Non-modular Network Wiring . Different from the modularized designed network which consists of topologically identical blocks , there exists some work that explores more flexible wiring patterns . MaskConnect ( Ahmed & Torresani , 2018 ) removes predefined architectures and learns the connections between modules in the network with k conenctions . Randomly wired neural networks ( Xie et al. , 2019 ) use classical graph generators to yield random wiring instances and achieve competitive performance with manually designed networks . DNW ( Wortsman et al. , 2019 ) treats each channel as a node and searches a fine-grained sparse connectivity among layers . TopoNet ( Yuan et al. , 2020 ) learns to optimize the connectivity of neural networks in a complete graph that adapt to the specific task . Prior work demonstrates the potential of more flexible wirings , our work on DG-Net pushes the boundaries of this paradigm , by enabling each example to be processed with different connectivity . Dynamic Networks . Dynamic networks , adjusting the network architecture to the corresponding input , have been recently studied in the computer vision domain . SkipNet ( Wang et al. , 2018b ) , BlockDrop ( Wu et al. , 2018 ) and HydraNet ( Mullapudi et al. , 2018 ) use reinforcement learning to learn the subset of blocks needed to process a given input . Some approaches prune channels ( Lin et al. , 2017a ; You et al. , 2019 ) for efficient inference . However , most prior methods are challenging to train , because they need to obtain discrete routing decisions from individual examples . Different from these approaches , DG-Net learns continuous weights for connectivity to enable ramous propagation of features , so can be easily optimized in a differentiable way . Conditional Attention . Some recent work proposes to adapt the distribution of features or weights through attention conditionally on the input . SENet ( Hu et al. , 2018 ) boosts the representational power of a network by adaptively recalibrating channel-wise feature responses by assigning attention over channels . CondConv ( Yang et al. , 2019 ) and dynamic convolution ( Chen et al. , 2020 ) are restricted to modulating different experts/kernels , resulting in attention over convolutional weights . Attention-based models are also widely used in language modeling ( Luong et al. , 2015 ; Bahdanau et al. , 2015 ; Vaswani et al. , 2017 ) , which scale previous sequential inputs based on learned attention weights . In the vision domain , previous methods most compute attention over micro structure , ignoring the influence of the features produced by different layers on the final representation . Unlike these approaches , DG-Net focuses on learning the connectivity based upon inputs , which can be seen as attention over features with different semantic hierarchy . Neural Architecture Search . Recently , Neural Architecture Search ( NAS ) has been widely used for automatic network architecture design . With evolutionary algorithm ( Real et al. , 2019 ) , reinforcement learning ( Pham et al. , 2018 ) or gradient descent ( Liu et al. , 2019 ) , one can obtain task-dependent architectures . Different from these NAS-based approaches , which search for a single architecture , the proposed DG-Net generates forward paths on the fly according to the input without searching . We also notice a recent method InstaNAS ( Cheng et al. , 2020 ) that generates domain-specific architectures for different samples . It trained a controller to select child architecture from the defined meta-graph , achieving latency reduction during inference . Different from them , DG-Net focuses on learning connectivity in a complete graph using a differentiable way and achieves higher performance . 3 METHODOLOGY . 3.1 NETWORK REPRESENTATION WITH DAGS . The architecture of a neural network can be naturally represented by a directed acyclic graphs ( DAG ) , consisting of an ordered sequence of nodes . Specifically , we map both combinations ( e.g. , addition ) and transformation ( e.g. , convolution , normalization , and activation ) into a node . At the same time , connections between layers are represented as edges , which determine the path of the features in the network . For simplicity , we denote a DAG with N ordered nodes as G = ( N , E ) , where N is the set of nodes and E is the set of edges . We show E = { e ( i , j ) |1 ≤ i < j ≤ N } , where e ( i , j ) indicates a directed edge from the i-th node to the j-th node . Most traditional convolutional neural networks can be represented with DAGs . For example , VGGNet ( Simonyan & Zisserman , 2015 ) is stacked directly by a series of convolutional layers , where a current layer is only connected to the previous layer . The connectivity in each stage can be represented as Evgg = { e ( i , j ) |j = i+ 1|1≤i < N } . To ease problems of gradient vanishing and exploding , ResNets ( He et al. , 2016a ) build additional shortcut and enable cross-layer connections whose nature view 1 can be denoted by Eres = { e ( i , j ) |j ∈ { i + 1 , i + 2 } |1≤i < N } . It is worth noting that some NAS methods ( Real et al. , 2019 ; Liu et al. , 2019 ) also follow this wiring pattern that blocks connect two immediate preceding blocks . Differently , DenseNets ( Huang et al. , 2017 ) aggregate features from all previous layers in the manner of Edense = { e ( i , j ) |i ∈ [ 1 , j − 1 ] |1 < j≤N } . Given these patterns of connectivity , the forward procedure of network can be performed according to the topological order . For the j-th node , the output feature x ( j ) is computed by : x ( j ) = f ( j ) ( ∑ i < j 1E ( e ( i , j ) ) · x ( i ) ) , s.t . 1E ( e ( i , j ) ) ∈ { 0 , 1 } ( 1 ) where f ( j ) ( · ) is the corresponding mapping function for transformations , and 1E ( e ( i , j ) ) stands for the indicator function and equals to one when e ( i , j ) exists in E . In each graph , the first node in topological order is the input one that only performs the distribution of features . The last node is the output one that only generates final output by gathering preceding inputs . For a network with K stages , K DAGs are initialized and connected sequentially . Each graph is linked to its preceding or succeeding stage by output or input node . Let F ( k ) ( · ) be the mapping 1In ( Veit et al. , 2016 ) , its unrolled type can be viewed as Edense = { e ( i , j ) |i ∈ [ 1 , j − 1 ] |1 < j≤N } . function of the k-th stage , which is established by G ( k ) with nodesN ( k ) and connectivity E ( k ) . Given an input x , the mapping function from the sample to the feature representation can be written as : T ( x ) = F ( K ) ( · · · F ( 2 ) ( F ( 1 ) ( x ) ) ) ( 2 )

Summary: This paper presents a novel approach (DG-Net) to “generate” a dynamic structure for the neural network, by learning to predict and select the edges between computational nodes in an end-to-end manner. The method is based on a gating mechanism, applied on top of a fully connected graph (similar to the connectivity in a DenseNet), designed to control the quantity of information received from each previous layer. The experiments show consistent improvement in image classification (ImageNet) and object detection (COCO).