Dynamic Graph: Learning Instance-aware Connectivity for Neural Networks

1 INTRODUCTION . Deep neural networks have driven a shift from feature engineering to feature learning . The great progress largely comes from well-designed networks with increasing capacity of models ( He et al. , 2016a ; Xie et al. , 2017 ; Huang et al. , 2017 ; Tan & Le , 2019 ) . To achieve the superior performance , a useful practice is to add more layers ( Szegedy et al. , 2015 ) or expand the size of existing convolutions ( kernel width , number of channels ) ( Huang et al. , 2019 ; Tan & Le , 2019 ; Mahajan et al. , 2018 ) . Meantime , the computational cost significantly increases , hindering the deployment of these models in realistic scenarios . Instead of adding much more computational burden , we prefer adding sampledependent modules to networks , increasing the model capacity by accommodating the data variance . Several existing work attempt to augment the sample-dependent modules into network . For example , Squeeze-and-Excitation network ( SENet ) ( Hu et al. , 2018 ) learns to scale the activations in the channel dimension conditionally on the input . Conditionally Parameterized Convolution ( CondConv ) ( Yang et al. , 2019 ) uses over-parameterization weights and generates individual convolutional kernels for each sample . GaterNet ( Chen et al. , 2018 ) adopts a gate network to extract features and generate sparse binary masks for selecting filters in the backbone network based upon inputs . All these methods focus on the adjustment of the micro structure of neural networks , using a data-dependent module to influence the feature representation at the same level . Recall the deep neural network to mammalian brain mechanism in biology ( Rauschecker , 1984 ) , the neurons are linked by synapses and responsible for sensing different information , the synapses are activated to varying degrees when the neurons perceive external information . Such a phenomenon inspires us to design a data-dependent network structure so that different samples will activate different network paths . In this paper , we learn to optimize the connectivity of neural networks based upon inputs . Instead of using stacked-style or hand-designed manners , we allow more flexible selection for forwarding paths . Specifically , we reformulate the network into a directed acyclic graph , where nodes represent the convolution block while edges indicate connections . Different from randomly wired neural networks ( Xie et al. , 2019 ) that generate random graphs as connectivity using predefined generators , we rewire the graph as a complete graph so that all nodes establish connections with each other . Such a setting allows more possible connections and makes the task of finding the most suitable connectivity for each sample equivalent to finding the optimal sub-graph in the complete graph . In the graph , each node aggregates features from the preceding nodes , performs feature transformation ( e.g . convolution , normalization , and non-linear operations ) , and distributes the transformed features to the succeeding nodes . The output of the last node in the topological order is employed as the representation through the graph . To adjust the contribution of different nodes to the feature representation , we further assign weights to the edges in the graph . The weights are generated dynamically for each input via an extra module ( denoted as router ) along with each node . During the inference , only crucial connections are maintained , which creates different paths for different instances . As the connectivity for each sample is generated through non-linear functions determined by routers , our method can enable the networks to have more representation power than the static network . We call our method as the Dynamic Graph Network ( DG-Net ) . It doesn ’ t increase the depth or width of the network , while only introduces an extra negligible cost to compute the edge weights and aggregate the features . To facilitate the training , we represent the network connection of each sample as a adjacent matrix and design a buffer mechanism to cache the matrices of a sample batch during training . With the buffer mechanism , we can conveniently aggregate the feature maps in the forward pass and compute the gradient in the backward pass by looking up the adjacent matrices . The main contributions of our work are as follows : • We first introduce the dynamic connectivity based upon inputs to exploit the model capacity of neural networks . Without bells and whistles , simply replacing static connectivity with dynamic one in many networks achieves solid improvement with only a slight increase of ( ∼ 1 % ) parameters and ( ∼ 2 % ) computational cost ( see table 1 ) . • DG-Net is easy and memory-conserving to train . The parameters of networks and routers can be optimized in a differentiable manner . We also design a buffer mechanism to conveniently access the network connectivity , in order to aggregate the feature maps in the forward pass and compute the gradient in the backward pass . • We show that DG-Net not only improves the performance for human-designed networks ( e.g . MobielNetV2 , ResNet , ResNeXt ) but also boosts the performance for automatically searched architectures ( e.g . RegNet ) . It demonstrates good generalization ability on ImageNet classification ( see table 1 ) and COCO object detection ( see table 2 ) tasks . 2 RELATED WORK . Non-modular Network Wiring . Different from the modularized designed network which consists of topologically identical blocks , there exists some work that explores more flexible wiring patterns . MaskConnect ( Ahmed & Torresani , 2018 ) removes predefined architectures and learns the connections between modules in the network with k conenctions . Randomly wired neural networks ( Xie et al. , 2019 ) use classical graph generators to yield random wiring instances and achieve competitive performance with manually designed networks . DNW ( Wortsman et al. , 2019 ) treats each channel as a node and searches a fine-grained sparse connectivity among layers . TopoNet ( Yuan et al. , 2020 ) learns to optimize the connectivity of neural networks in a complete graph that adapt to the specific task . Prior work demonstrates the potential of more flexible wirings , our work on DG-Net pushes the boundaries of this paradigm , by enabling each example to be processed with different connectivity . Dynamic Networks . Dynamic networks , adjusting the network architecture to the corresponding input , have been recently studied in the computer vision domain . SkipNet ( Wang et al. , 2018b ) , BlockDrop ( Wu et al. , 2018 ) and HydraNet ( Mullapudi et al. , 2018 ) use reinforcement learning to learn the subset of blocks needed to process a given input . Some approaches prune channels ( Lin et al. , 2017a ; You et al. , 2019 ) for efficient inference . However , most prior methods are challenging to train , because they need to obtain discrete routing decisions from individual examples . Different from these approaches , DG-Net learns continuous weights for connectivity to enable ramous propagation of features , so can be easily optimized in a differentiable way . Conditional Attention . Some recent work proposes to adapt the distribution of features or weights through attention conditionally on the input . SENet ( Hu et al. , 2018 ) boosts the representational power of a network by adaptively recalibrating channel-wise feature responses by assigning attention over channels . CondConv ( Yang et al. , 2019 ) and dynamic convolution ( Chen et al. , 2020 ) are restricted to modulating different experts/kernels , resulting in attention over convolutional weights . Attention-based models are also widely used in language modeling ( Luong et al. , 2015 ; Bahdanau et al. , 2015 ; Vaswani et al. , 2017 ) , which scale previous sequential inputs based on learned attention weights . In the vision domain , previous methods most compute attention over micro structure , ignoring the influence of the features produced by different layers on the final representation . Unlike these approaches , DG-Net focuses on learning the connectivity based upon inputs , which can be seen as attention over features with different semantic hierarchy . Neural Architecture Search . Recently , Neural Architecture Search ( NAS ) has been widely used for automatic network architecture design . With evolutionary algorithm ( Real et al. , 2019 ) , reinforcement learning ( Pham et al. , 2018 ) or gradient descent ( Liu et al. , 2019 ) , one can obtain task-dependent architectures . Different from these NAS-based approaches , which search for a single architecture , the proposed DG-Net generates forward paths on the fly according to the input without searching . We also notice a recent method InstaNAS ( Cheng et al. , 2020 ) that generates domain-specific architectures for different samples . It trained a controller to select child architecture from the defined meta-graph , achieving latency reduction during inference . Different from them , DG-Net focuses on learning connectivity in a complete graph using a differentiable way and achieves higher performance . 3 METHODOLOGY . 3.1 NETWORK REPRESENTATION WITH DAGS . The architecture of a neural network can be naturally represented by a directed acyclic graphs ( DAG ) , consisting of an ordered sequence of nodes . Specifically , we map both combinations ( e.g. , addition ) and transformation ( e.g. , convolution , normalization , and activation ) into a node . At the same time , connections between layers are represented as edges , which determine the path of the features in the network . For simplicity , we denote a DAG with N ordered nodes as G = ( N , E ) , where N is the set of nodes and E is the set of edges . We show E = { e ( i , j ) |1 ≤ i < j ≤ N } , where e ( i , j ) indicates a directed edge from the i-th node to the j-th node . Most traditional convolutional neural networks can be represented with DAGs . For example , VGGNet ( Simonyan & Zisserman , 2015 ) is stacked directly by a series of convolutional layers , where a current layer is only connected to the previous layer . The connectivity in each stage can be represented as Evgg = { e ( i , j ) |j = i+ 1|1≤i < N } . To ease problems of gradient vanishing and exploding , ResNets ( He et al. , 2016a ) build additional shortcut and enable cross-layer connections whose nature view 1 can be denoted by Eres = { e ( i , j ) |j ∈ { i + 1 , i + 2 } |1≤i < N } . It is worth noting that some NAS methods ( Real et al. , 2019 ; Liu et al. , 2019 ) also follow this wiring pattern that blocks connect two immediate preceding blocks . Differently , DenseNets ( Huang et al. , 2017 ) aggregate features from all previous layers in the manner of Edense = { e ( i , j ) |i ∈ [ 1 , j − 1 ] |1 < j≤N } . Given these patterns of connectivity , the forward procedure of network can be performed according to the topological order . For the j-th node , the output feature x ( j ) is computed by : x ( j ) = f ( j ) ( ∑ i < j 1E ( e ( i , j ) ) · x ( i ) ) , s.t . 1E ( e ( i , j ) ) ∈ { 0 , 1 } ( 1 ) where f ( j ) ( · ) is the corresponding mapping function for transformations , and 1E ( e ( i , j ) ) stands for the indicator function and equals to one when e ( i , j ) exists in E . In each graph , the first node in topological order is the input one that only performs the distribution of features . The last node is the output one that only generates final output by gathering preceding inputs . For a network with K stages , K DAGs are initialized and connected sequentially . Each graph is linked to its preceding or succeeding stage by output or input node . Let F ( k ) ( · ) be the mapping 1In ( Veit et al. , 2016 ) , its unrolled type can be viewed as Edense = { e ( i , j ) |i ∈ [ 1 , j − 1 ] |1 < j≤N } . function of the k-th stage , which is established by G ( k ) with nodesN ( k ) and connectivity E ( k ) . Given an input x , the mapping function from the sample to the feature representation can be written as : T ( x ) = F ( K ) ( · · · F ( 2 ) ( F ( 1 ) ( x ) ) ) ( 2 )

Summary: This work proposes a novel method, called Dynamic Graph Network (DG-Net), for optimizing the architecture of a neural network. Building on the previous work introduced by (Xie et al., 2019), the authors propose to consider the network as a complete directed acyclic graph (DAG). Then, the edge weights of the DAG are generated dynamically for each input of the network. At each node of the network, the authors introduce an extra-module, called router, to estimate the edge weights as function of the input features.