Dynamic Graph: Learning Instance-aware Connectivity for Neural Networks
Authors: Kun Yuan
Quanquan Li
Dapeng Chen
Aojun Zhou
Junjie Yan
Venue:  (2021)
Keywords: dynamic network
data-dependent
complete graph

Abstract: One practice of employing deep neural networks is to apply the same architecture to all the input instances. However, a fixed architecture may not be representative enough for data with high diversity. To promote the model capacity, existing approaches usually employ larger convolutional kernels or deeper network structure, which may increase the computational cost. In this paper, we address this issue by raising the Dynamic Graph Network (DG-Net). The network learns the instance-aware connectivity, which creates different forward paths for different instances. Specifically, the network is initialized as a complete directed acyclic graph, where the nodes represent convolutional blocks and the edges represent the connection paths. We generate edge weights by a learnable module \textit{router} and select the edges whose weights are larger than a threshold, to adjust the connectivity of the neural network structure. Instead of using the same path of the network, DG-Net aggregates features dynamically in each node, which allows the network to have more representation ability. To facilitate the training, we represent the network connectivity of each sample in an adjacency matrix. The matrix is updated to aggregate features in the forward pass, cached in the memory, and used for gradient computing in the backward pass.  We verify the effectiveness of our method with several static architectures, including MobileNetV2, ResNet, ResNeXt, and RegNet. Extensive experiments are performed on ImageNet classification and COCO object detection, which shows the effectiveness and generalization ability of our approach.

Under review as a conference paper at ICLR 2021
DYNAMIC GRAPH: LEARNING LNSTANCE-AWARE CONNECTIVITY FOR NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
One practice of employing deep neural networks is to apply the same architecture all the input instances. However, a fixed architecture may not representative enough for data with high diversity. To promote the model capacity, existing approaches usually employ larger convolutional kernels or deeper network structure, which may increase the computational cost. In this paper, we address this issue by raising the Dynamic Graph Network (DG-Net). The network learns the instance- aware connectivity, which creates different forward paths for different instances. Specifically, the network initialized as a complete directed acyelic graph, where the nodes represent convolutional blocks and the edges represent the connection paths. We generate edge weights by a learnable module router and select the edges whose weights are larger than threshold, to adjust the connectivity of the neural network structure. Instead of using the same path of the network, DG-Net aggregates features dynamically in each node, which allows the network have more representation ability. To facilitate the training, we represent the network connectivity of each sample in an adjacency matrix. The matrix updated aggregate features in the forward pass, cached in the memory, and used for gradient computing in the backward pass. We verify the effectiveness of our method with several static architectures, including MobileNetV2, ResNet, RtsNeXt, and RegNet. Extensive experiments are performed on ImageNet classification and COCO object detection, which shows the effectiveness and generalization ability of our approach.
INTRODUCTION
Deep neural networks have driven shift from feature engineering to feature learning. The great progress largely comes from well-designed networks with increasing capacity of models (He et al., 2016a; Xie et al., 2017; Huang al., 2017; Tan & Le, 2019). To achieve the superior performance, useful practice is to add more layers (Szegedy et al., 2015) or expand the size of existing convolutions (kernel width, number of channels) (Huang al., 2019; Tan & Le, 2019; Mahajan et al., 2018). Meantime, the computational cost significantly increases, hindering the deployment of these models in realistic scenarios. Instead of adding much more computational burden, we prefer adding sample- dependent modules to networks, increasing the model capacity by accommodating the data variance.
Several existing work attempt augment the sample-dependent modules into network. For example, Squeeze-and-Excitaal
network (SENet) (Hu et al., 2018) learns to scale the activations in the channel dimension conditionally on the input. Conditionally Parameterized Convolution (CondConv) ( (Yang et al., 2019) uses over parmmterrizationn weights and generates individual convolutional kernels for each sample. GaterNet (Chen et al., 2018) adopts gate network extract features and generate sparse binary masks for selecting filters in the backbone network based upon inputs. All these methods focus on the adjustment of the micro structure of neural networks, using a data-dependent module to influence the feature representation at the same level. Recall the deep neural network to mammalian brain mechanism biology (Rauschecker, 1984), the neurons are linked by synapses and responsible for sensing different information, the synapses neurons perceive external information. Such a phenomenon inspires us to design a data-dependent network structure that different samples will activate different network paths.
activated to varying degrees when the
In this paper, we learn to optimize the connectivity neural networks based upon inputs. Instead of using stayked-style or handndesigned manners, allow more flexible selection for forwarding paths.

---

Under review as a conference paper at ICLR 2021
Specifically, reformulate the network into directed acyclicc graph, where nodes represent the convolution block while edges indicate connections. Different from randomly wired neural networks (Xie et al., 2019) that generate random graphs as connectivity using predefined generators, we rewire the graph as complete graph so that all nodes establish connections with each other. Such a setting allows more possible connections and makes the task of finding the most suitable connectivity for each sample equivalent to finding the optimal sub-graph in the complete graph. In the graph, each node aggregates features from the preceding nodes, performs feature transformation (e.g. convolution, normalization, and non-linear operations), and distributes the transformed features to the succeeding nodes. The output of the last node in the topological order is employed as the representation through the graph. To adjust the contribution of different nodes to the feature representation, we further assign weights to the edges in the graph. The weights are generated dynamically for each input via an extra module (denoted as router) along with each node. During the inference, only crucial connections are maintained, which creates different paths for different instances. As the connectivity for each sample is generated through non-linear functions determined by routers, our method can enable the networks to have more representation power than the static network.
We call our method as the Dynamic Graph Network (DG-Net). It doesn't 't increase the depth or width of the network, while only introduces an extra negligible cost to compute the edge weights and aggregatte the features. To facilitate the training, we represent the network connection of each sample as a adjacent matrix and design a buffer mechanism to cache the matrices of a sample batch during training. With the buffer mechanism, we can conveniently aggregate the feature maps in the forward pass and compute the gradient in the backward pass by looking up the adjacent matrices. The main contributions of our work are as follows:
We first introduce the dynamic connectivity based upon inputs to exploit the model capacity of neural networks. Without bells and whistles, simply replacing static connectivity with dynamic one in many networks achieves solid improvement with only slight increase of (~ 1%) parameters and (~ 2%) computational cost (see table l).
DG-Net is easy and memory- memorr-consereee conserving to train. The parameters of networks and routers can be optimized in differentiable manner. We also design a buffer mechanism conveniently access the network connectivity, in order to aggregate the feature maps in the forward pass and compute the gradient in the backward pass.
We show that DG-Net not only improves the performance for human--esignee networks (e.g. MobielNetV2, ResNet, ResNeXt) but also boosts the performance for automatically searched architectures (e.g. RegNet). It demonstrates good generalization ability ImageNet classification (see table 1) and COCO object detection (see table 2) tasks.
2 RELATED WORK
Non-modular Network Wiring, Different from the modularized designed network which consists of topologically identical blocks, there exists some work that explores more flexible wiring patterns. MaskConnect (Ahmed & Torresani, 2018) removes predefined architectures and learns the connec- tions between modules in the network with & conenctions. Randomly wired neural networks (Xie et al., 2019) use classical graph generators to yield random wiring instances and achieve competitive performance with manually designed networks. DNW (Wortsman et al., 2019) treats each channel as node and searches fine- fine-grained sparse connectivity among layers. TopoNet (Yuan et al., 2020) learns to optimize the connectivity of neural networks in a complete graph that adapt to the specific task. Prior work demonstrates the potential of more flexible wirings, our work on DG-Net pushes the boundaries of this paradigm, by enabling each example to be processed with different connectivity.
Dynamic Networks. Dynamic networks, adjusting the network architecture to the corresponding input, have been recently studied in the computer vision domain. SkipNet (Wang et al., 8b)),, BlockDrop (Wu et al., 2018) and HydraNet (Mullapudi et al., 2018) use reinforcement learning to learn the subset of blocks needed to process given input. Some approaches prune channels (Lin et al., 2017a; You et al., 2019) for efficient inference. However, most prior methods are challenging to train, because they need to obtain discrete routing decisions from individual examples. Different from these approaches, DG-Net learns continuous weights for connectivity to enable ramous propagation of features, so can be easily optimized in differentiable way.
2

---

Under review as a conference paper at ICLR 2021
Conditional Attention. Some recent work proposes adapt the distribution of features weights through attention conditionally on the input. SENet (Hu et al., 2018) boosts the representational power of network by adaptively recalibrating channel-wise feature responses by assigning attention over channels. CondConv (Yang et al., 2019) and dynamic convolution (Chen et al., 2020) are restricted modulating different experts/kemells, resulting in attention over convolutional weights. Attention-basee models are also widely used in language modeling (Luong et al., 2015; Bahdanau al., 2015; Vaswani et al., 2017), which scale previous sequential inputs based on learned attention weights. In the vision domain, previous methods most compute attention over micro structure, ignoring the influence of the features produced by different layers on the final representation. Unlike these approaches, DG-Net focuses on learning the connectivity based upon inputs, which can be seen as attention over features with different semantic hierarchy.
Neural Architecture Search. Recently, Neural Architecture Search (NAS) has been widely used for automatic network architecture design. With evolutionary algorithm (Real et al., 2019), reinforce- ment learning (Pham et al., 2018) or gradient descent (Liu et al., 2019), one can obtain task-dependent architectures. Different from these NAS-based approaches. which search for single architecture, the proposed DG-Net generates forward paths on the fly according to the input without searching. We also notice a recent method InstaNAS (Cheng et al., 2020) that generates Nomain--peciiff architectures for different samples. It trained a controller to select child architecture from the defined meta- veta-graph, achieving latency reduction during inference. Different from them, DG-Net focuses on learning connectivity in a complete graph using a differentiable way and achieves higher performance.
3 METHODOLOGY
3.1 NETWORK REPRESENTATION WITH DAGS
The architecture of a neural network can be naturally represented by a directed acyclic graphs (DAG), consisting of an ordered sequence of nodes. Specifically, we map both combinations (e.g., addition) and transformation (e.g., convolution. normalization, and activation) into node. At the same time, connections between layers are represented as edges, which determine the path of the features in the network. For simplicity, we denote DAG with N ordered nodes as G = (N, where N is the set of nodes and & is the set of edges. We show E = (c(t-) < < j I N). where c(t.)) indicates a directed edge from the i-th node to the j-th node.
Most traditional convolutional neural networks can be represented with DAGs. For example, VGGNet (Simonyan & Zisserman, 2015) is stacked directly by a series of convolutional layers, where a current layer is only connected to the previous layer. The connectivity in each stage can be represented as Engy (el (He et al., 201 build additional shorteut and enable cross-layer connections whose nature view can be denoted by Eres { {i + 1, + 2)/l<i<n) }. It is worth noting that some NAS methods (Real et al., 2019; Liu et al., 2019) also follow this wiring pattern that blocks connect two immediate preceding blocks. Differently, DenseNets (Huang et al., 2017) aggregate features from all previous layers in the manner of Edense (e(--1)j E [1.j - I]li<iss]; Given these patterns of connectivity, the forward procedure of network can be performed according to the topological order. For the j-th node, the output feature x(i is computed by:
i + ll< }. To ease problems of gradient vanishing and exploding, ResNets
x(1 = f(O (e(e))))) -(())))))()))).)) xx),, s.t. (e(e)))) E (0,1}
(I)
where (()) is the corresponding mapping function for transformations, and (e(()))) stands for the indicator function and equals to one when e(f.j) exists in E.
In each graph, the first node in topological order is the input one that only performs the distribution of features. The last node is the output one that only generates final output by gathering preceding inputs. For a network with K stages, K DAGs are initialized and connected sequentially. Each graph is linked to its preceding or succeeding stage by output or input node, Let (.) be the mapping
In (Veit al., 2016), its unrolled type can he viewed Edense
(e [1,} 1]le<gnn)
3

---

Under review as a conference paper at ICLR 2021
Aggrisaation
Graph
Graph 2
Consolition
Transformation
Distribsiion
Output
The framework of DG-Net. Left: For a training batch, each sample performs different forward paths that determined by the sample-dependent macro connectivity. Right: Node operations at the micro level. Here we illustrate a node with 2 active input edges and output edges. First, it aggregates input features from preceding nodes by weighted sum. Second, convolutional blocks transform the aggregated features. Third, a router predicts routing weights for each sample on the output edges according to the transformed features. Last, the transformed data sent out by the output edges to the following nodes. Arrows indicate the data flow.
Figure 1:
function of the k-th stage, which is established by ((k) with nodes (K(k) and connectivity g()) Given an input x, the mapping function from the sample the feature representation be written as:
(.. (F(
3.2 EXPANDING SEARCH SPACE FOR THE CONNECTIVITY
As shown in Eq.(1), most traditional networks adopt binary codes to formulate the connectivity, resulting in relatively sparse and static connection pattern. But these prior- based methods limit the connection possibilities, the type of feature fusion required by different samples may different. In this paper, we raise two modifications in DG-Net to expand the search space with more possible connectivity. First, we remove the constraint on in/out-degree tivity be complete graph where edges exist between any nodes. The search space is different from DenseNet that we replace the aggregation method from concatenation to addition. This avoids misalignment of channels caused by the removal addition of edges. In this way, finding good connectivity is akin to finding optimal sub-graphs. Second, instead of selecting discrete edges in the binary type, we assign a soft weight a (+J) to the edge which reflects the magnitude connections. This also benefits the connectivity can be optimized in differentiable manner.
nodes and initialize the connec-
In neural networks, features generated by different layers exhibit various semantic representations (Zhou al., 2016; Zeiler & Fergus, 2014). Recall the mammalian brain mechanism in biology (Rauschecker, 1984) that the synapses are activated to varying degrees when the neurons perceive external information, the weights of edges in the graph be parameterized upon inputs. As shown in the left of Fig. 1, DG-Net can generate appropriate connectivity for each sample. Different from (q.(1), the output feature can be computed by:
x
=(() Σ (t.) x())
(3)
where (4-5) is vector that contains the weights related to samples in batch.
4

---

Under review as a conference paper at ICLR 2021
3.3 INSTANCE-AWARD CONNECTIVITY THROUGH ROUTING MECHANISM
To obtain ((+-)) and allow instance- aware connectivity for the network, we add an extra conditional router module along with each node, as presented in the right of Fig. 1. The calculation procedure in node can be divided into four steps. First, the node aggregates features from preceding connected nodes by weighted addition. Second, the node performs feature transformation with convolution, normalization, and activation layers (determined by the network). Third, the router receives the transformed feature and applies squeeze-and setezee-aaddxxiittiioonnn to compute instance-aware weights over edges with succeeding nodes. Last, the node distributes the transformed features to succeeding nodes according to the weights. We also discuss the different implementation of routers in appendix 6.2. This method is equivalent to building independent router for each edge.
Structurally, the router applies a lightweight module consisting of a global average pooling o(.), a fully-connected layer and a sigmoid activation o.)) The global spatial information is firstly squeezed by global average pooling. Then we use a fully-connected layer and sigmoid to generate normalized routing weights (1.) for output edges. The mapping function of the router can be written as:
(xx) (w)")(x) + b), s.t. (p()) [0, 1),
where and b are weights and bias of the fully-connected layer. Particularly, DG-Net is compu- tationally efficient because of the 1-D dimension reduction of o(.). For an input feature map with dimension H x W Cin, the convolutional operation requires Cin Cout HW DJ Multi-Adds (for simplicity, only one layer of convolution is calculated), where D3 is the kernel size. As for the routing mechanism, it only introduces extra ((p(x)) Cin Sout Multi-Adds, where Sour is the number of output edges for the node. This is much less than the computational cost of convolution.
Besides, set a learnable weight of r that acts threshold for each node to control the connectivity. When the weight is less than the threshold, the connection will be closed during inference. When (.(..) = 0. the edge from i-th node to j-th node will be marked as closed. If the input or output edges for node are all closed, the node will be removed to accelerate inference time. Meanwhile, all the edges with (i-))) > 0 will be reserved, continuously enabling feature fusion. This can be noted by:
{ (+6)))
(Ga) a (13) 27
a
(5)
During training, this can be implemented by a differentiable manner of (a)a) a aaaaaa--aa-- r).
3.4 BUFFER MECHANISM FOR MACRO FEATURE AGGREGATION
DG-Net allows flexible wiring patterns for the connectivity, which requires the aggregation of the features among nodes that need to be recorded and shared within graph. For this purpose, we represent the connectivity in an adjacency matrix (denoted as M E R R-NxN). The order of rows and columns indicate the topological order of the nodes in the graph. Elements in the matrix represent the weights of edges, as shown in the left of Fig. 2, where rows reflect the weights of input edges and columns are of output edges for a node. During the forward procedure, the i-th node performs aggregation through weights acquired from the corresponding row of M,-1.-- Then the node generates weights over output edges through accompanying the router and stores them into the column of M.;- In this way, the adjacency matrix is updated progressively and shared within the graph. For a batch with B samples, different matrices are concatenated in the dimension of batch and cached a defined buffer (denoted as M E BxNN where M = M), as shown in the right of Fig. 2. With the buffer mechanism, DG-Net can be trained as an ordinary network without introducing excessive computation or time-consuming burden.
3.5 OPTIMIZATION OF DG-NET
During training, the parameters of the network Wa. as well as the parameters of routers W,, are optimized simultancously using gradients back- propppprpagttion.. Given an input x and corresponding label y, the objective function can be represented as:
min W.
(x(T(x; W., w)
(6)
5

---

Under review as a conference paper at ICLR 2021
6.09
OM
In- edge
edge
Node-1
Node=2
Node-3
M4
M2
6.24
M2
OUT
Node-4
Node-5
Node-6
Node-7
Batch Matrices Buffer
The procedure of updating the adjacency matrix and the proposed buffer for storing. A node obtains the weights of input edges from the row (blue) and storing weights output edges saving in the column (green). The matrices are saved in a buffer that supports batch training efficiently.
Figure 2:
where (-(-,) -) denotes the loss function w.r.t specific tasks (e.g. cross-entropy loss for image classifi- cation and regression loss for object detection). This method has two benefits. First, simultaneous optimization can effectively reduce the time consumption of training. The time to obtain a trained dynamic network is the same as that static network. Second, different from DARTS (Liu et al., 2019) that selects operation with the maximum probability, our method learns the connectivity in continuous manner, which better preserves the consistency between training and testing.
Set aL, be the gradients that the network flows backwards to the convolutional weights of the j-th Owh node (f) Let 8Lt be the gradients to x x(i). Then the gradients update to the weights of node G
exl the weights of router !! the biases router 6,5) and threshold -() are of the form:
Oh.
w(() - ((() 17 Owh
OC,
0
Oip'
w((t.) + w((c)) -n Σ
x²)
100 (x')
(8)
0xj
Ozi
dC.
Oei
((2-5) - ((:c)) -11 Σ
0
Ox
Ox
Oz
UL,
Oul
(()) - ,(:ي - n Σ
x
(10)
dxi
Oxe
where w (1.)) Cx1 and ((t-)) E R are the weights and bias of the router that determine the output
edge of (i..)) And x²' is the aggregated features of Σ (t,f). (1) Eqn. (3), 2² is calculated by
w(.) o(x)) + b(.5) in Eqn. (4). And Bri is a aaaa 7²) (a -d) 10-1 1). And @ indicates
entrywise product. The gradients w.r.t (1-5) can be noted > 3x O ar
C x').
4 EXPERIMENTS
4.1 IMAGENET CLASSIFICATION
Dataset and evaluation metries. We evaluate our approach on the ImageNet 2012 classification dataset (Russakovsky et al., 2015). The ImageNet dataset consists of 1.28 million training images and 50,000 validation images from 1000 classes. We train all models on the entire training set and compare the single-crop 1 validation set accuracy with input image resolution x24. We measure performance as ImageNet top-1 accuracy relative to the number of parameters and computational cost in FLOPs.

---

Under review as a conference paper at ICLR 2021
Model selection and training setups. We validate approach number of widely used models including MobileNetV2-L10 (Sandler et al., 2018), BesNNet-18/50/00 (He et al., 2016a) and ResNeXt50-32xer (Xie et al., 2017). To further test the effectiveness of DG-Net, optimize recent NAS-based networks of RegNets (Radosavovic et al., 2020), which are the best models out of search space with 10 possible configurations. Our implementation is based PyTorch (Paszke et al., 2019) and all experiments are conducted using 16 NVIDIA Tesla V100 GPUs with total batch of 1024. All models are trained using SGD optimizer with 0.9 momentum. Detailed information about the training setting can be seen in appendix 6.1.
attempt to
Table 1:
ImageNet validation accuracy (%) and inference cost. DG-Nets improve the accuracy of all baseline architectures with small relative increase in the number of parameters and inference cost.
Baselines
DG-Net
Network
Top-1
Params(M)
FLOP((M)
Top-1
Params(M)
FLOPs(M)
Top-1
MohileNctV2-1.0
3.51
299
72.60
3.58
312
73.54
0.94
ResNet18
11.69
1813
70.30
11.71
1826
71.32
1.02
ResNet50
25.55
4087
76.70
25.62
4125
78.28
1.58
ResNet101
44.54
7799
78.29
44.90
7837
79.90
1.61
ResNeXt50-32x4d
25.02
4228
77.97
25.09
4305
79.39
1.42
RegNet-X-600M
6.19
599
74.03
6.22
600
74.68
0.65
RogNet-X-1660M
9.19
1602
77.26
9.22
1604
77.91
0.65
Analysis of experimental results. We verify that DG-Net improves performance on a wide range of architectures in Table 1. For fair comparison, retrain all of our baseline models with the same hyperparameters as the DG-Net models Compared with baselines, DG-Net gets considerable gains with a small relative increase in the number of parameters (< 2%) and inference cost of FLOPSs (< 1%). This includes architectures with mobile setting (Sandler et al., 2018), classical residual wirings (He al., 2016a; Xie et al.. 2017). multi-brance operation (Xie al., 2017) and architecture search (Radosavovic et al., 2020). We further find that DG-Net benefits from the large search space which can be seen in the improvements of ResNets. With the increase of the depth from 18 to 101, the formed complete graph includes more nodes, resulting in larger search space and more possible connectivity. And the gains raise from 1.02% to 1.61% in top-1 accuracy. Ablation study on different connectivity method are given in appendix 6.3.
4.2 COCO OBJECT DETECTION
We report the transferability results by fine-tuning the networks for COCO object detection (Lin et al., 2014). We use Faster R-CNN (Ren et al., 2015) with FPN (Lin et al., 2017b) the object detector. Our fine-tuning is based on the 1x setting of the publicly available Detectron2 (Girshick et al., 2018). We replace the backbone with those in Table
The object detection results are given in Table 2. And FLOPs of the backbone are computed with an input size 800x 1333. Compared with the static network. DG-Net improves AP by 1.70% with ResNet-50 backbone. When using a larger search space of ResNet101, our method significantly improves the performance by 2.73% AP. is worth noting that stable gains are obtained for objects of different scales varying from small to large. This further verifies that instance- aware connectivity can improve the representation capacity toward the dataset with large distribution variance.
"Our re- imiplemmntatinn the baseline models and our DG-Net models use the same hyperparameters. For reference, published results for baselines are: MobileNetV2- 1.0 (Sandler et al., 2018): 72.00%. ResNet- 18 (He et al.. 2016b), 69.57%, ResNet-50 (Goyal et al., 2017): 76.40%, ResNet-101 (Goyal et al., 2017): 77. 77.92%, ResNext50--2x44 (Xie et al., 2017): 77.80%, RegNetX-600M (Radosavovic et al., 2020): 74.10%, RegNetX-16000 1600M (Radosavovic et al., 2020): 77.00%.

---

Under review as a conference paper at ICLR 2021
Table 2: COCO object detection minival performance. APs (%) of bounding box detection are
reported. DG-Nets brings consistently improvement across multiple backbones on all scales.
Backbone
Method
GFLOPs
AP
AP.s
AP.75
APs
APM
APz
Baseline
174
36.42
58.54
39.11
21.93
40.02
46.58
ResNet50
DG-Net
176
18.12
60.53
41.00
23.61
41.52
48.39
Baseline DG-Net
333
38.59
60.56
41.63
22.45
43.08
49,46
ResNet101
335
4:.77
63.54
44.97
25.71
45.60
52.62
Baseline DG-Net
181
38.07
60.42
41.01
22.97
42.10
48.68
ResNeXt50-32x40
39.52+1.45
62.41
42.56
25.71
43.34
49.83
4.3 COMPARED WITH RELATED WORK
Compared with InstaNAS (Cheng al., 2020). InstaNAS generates data-dependent networks from designed meta-graph. During inference, it uses a controller sample possible architectures by a Bernoulli distribution. But it needs to carefully design the training process to avoid collapsing the controller. Differently, DG-Net builds continuous connections between nodes, which allowing more possible connectivity. And the proposed method is compatible with gradient descent, and be trained in differentiable way easily. MobileNetV2 is used as the backbone network in InstaNAS. It provides multiple searched architectures under different latencies. For a fair comparison, DG-Net adopts the same structure the backbone and reports the results of ImageNet. The latency is tested using the same hardware. The results in Table 3 demonstrate DG-Net can generate better instance-aware architectures in the dimension of connectivity.
Compared with WandWire (Xie et al., 2019). Randomly wired neural networks explore using flexible graphs generated by different graph generators as networks, losing the constraint on wiring patterns. But for the entire dataset, the network architecture it uses is still consistent. Furthermore, DG-Net allows instance-aware connectivity patterns learned from the complete graph. We compare three types of generators in their paper with best hyperparameters, including Erdös-Rényi (ER), Barabási-A (BA), and -ttts-Strogatz (WS). Since the original paper does not release codes, reproduce these graphs using Net Networkx', We follow the small computation regime to form networks. Experiments are performed in ImageNet using its original training setting except for the DropPath and DropOut. Comparison results are shown in Table DG-Net is superior three classical graph generators in similar computational cost. This proves that under the same search space, the optimized data-dependent connectivity is better than randomly wired static connectivity.
Table 3: Compared with InstaNAS under com- parable latency in ImageNet.
Table 4: Compared with RandWire under
small computation regime in ImageNet.
Model
Top-1
Latency (ms)
Wiring Type
Top-1
FLOPs(M)
InstaNAS-ImgNettA
71.9
0.23940.014
ER (P=0.2)
71.3440.40
602
InstaNAS-ImgNet-B
71.1
0.189+a.012
BA (M=5)
71.1640.34
582
InstaNAS- Im
69.9
0.171
WS (K=4, P=0.75)
72.26
572
DG-Ne:-MBv2-1.0
73.5
0.257
DG-Net
73.5220.05
611
Compared with state-of-the-art NAS-based methods. For completeness, compare with the most accurate NAS based networks under the mobile setting (~ 600M FLOPs) in ImageNet. is worth noting that this is not the focus of this paper. We select RegNet the basic architecture as shown Table For fair comparisons, here we train 250 epochs and other settings are the same with section 4.1. We note RegNet-X with the dynamic graph as DG-Net-A and RegNet-Y with a dynamic graph as DG-Net-B (with SE-module for comparison with particular searched architectures e.g. EfficientNet). The experimental results are given in Table 5, It shows that with single operation
Phtpps: //ttww/kkx......./ github. 'The original performance of 1-0gNe--X-600M1 is 75.03%, and RegNet-Y-600M is 76.10% under this training setting.
8

---

Under review as a conference paper at ICLR 2021
type (Regular Bottleneck), DG-Net can obtain considerable performance with other NAS methods with less search cost.
Table 5: Comparision with NAS methods under mobile setting. Here train for 250 epochs similar to (Zoph et al., 2018; Real et al., 2019; Xie et al., 2019; Liu et al., 2018; 2019), for fair comparisons.
Network
Params(M)
FLOPs(M)
Search Cost
Top-1
Top-5
NASNet-A (Zoph al., 2018)
5.3
564
2000
74.0
91.6
NASNet-B (Zoph al., 2018)
5.3
488
2000
72.8
91.3
NASNet-C (Zoph al., 2018)
4.9
2000
72.5
91.0
Amoeba-A (Real et al., 2019)
5.1
555
3150
74.5
92.0
Amoeba-B (Real et al., 2019)
5.3
555
3150
74.0
91.5
RandWire- WS (Xie et al., 2019)
5.6
583
74.7
92.2
PNAS (Liu al., 2018)
588
225
74.2
91.9
DARTS (Liu et al., 2019)
4.9
595
73.1
91.0
EfficientNet-BO (Tan & Le, 2019)
5.3
390
76,3
93.2
DG-Net-A DG-Net-B
6.2
601
1.5
75.8
92,6
6.3
601
1.5
77.0
93.4
5 CONCLUSION AND FUTURE WORK
In this paper, present the Dynamic Graph Network (noted as DG-Net), that allows learning instance-aware connectivity for neural networks. Without introducing much computation cost, the model capacity be increased diversity. We show that DG-Net superior to many static networks, including human-designn and automatically searched architectures. Besides, DG-Net demonstrates good generalization ability on ImageNet classification as well as COCO object detection. It also achieved SOTA results compared with related work. DG-Net explores the connectivity in an enlarged search space, which believe is a new research direction. In future work, we consider verifying DG-Net more NAS -AS-searched architectures. Moreover, will study learning dynamic operations beyond the connectivity as well as adjusting the computation cost based upon the difficulties of samples.
the difficulties of feature representation for data with high
REFERENCES
Karim Ahmed and Lorenzo Torresani. Maskconnect: Connectivity learning by gradient descent. In ECCV (5), volume 1209 of Lecture Notes in Computer Science, pp. 362-378. Springer, 2018.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.
Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu, Dynamic convolution: Attention convolution kemelss. In CVPR, pp. 11027-11036. IEEE, 2020.
Zhourong Chen, Yang Li, Samy Bengio, and Si Si. Gaternet: Dynamic filter selection in convolutional neural network via dedicated global gating network. CoRR, abs/18 1.1111111111 1 2018.
An-Chieh Cheng, Chieh Hubert Lin, Da-Cheng Juan, Wei Wei, and Min Sun. Instanas: Instance-aware neural architecture search. In AAAI, pp. 3577-3584. AAAI Press, 2020.
Felix A. Gers, Jürgen Schmidhuber, and Fred A. Cummins. Learning to forget: Continual prediction with LSTM. Neural Comput., 2451::44512271 2000.
Ross Girshick, Ilija Radosavovic, Georgia Gkioxari, Piotr Dollár, and Kaiming He. Detectron. 2018.
https: //ttt//gghhhb.//
Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch. Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: training imagenet in hour. CoRR, abs/1 706777 2017.

---

Under review as a conference paper at ICLR 2021
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770-778. IEEE Computer Society, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV (4), volume 9908 of Lecture Notes in Computer Science, pp. 530-645. Springer, 2016b.
Jie Hu, Li Shen, and Gang Sun. suuuezze-and-xxiittte networks. In CVPR, pp- 7132-7141, IEEE Computer Society, 2018.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In CVPR, pp. 2261-2269. IEEE Computer Society, 2017.
Yanping Huang. Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoe V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In NeurIPS, pp. 103-112, 2019.
Ji Lin, Yongming Rao, Jiwen Lu, and Jie Zhou. Runtime neural pruning. In NIPS, pp. 2181-2191, 2017a.
Tsung- Yi Lin, Michael Maire, Serge J. Belongie. James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. ECCV (5), volume 8693 of Lecture Notes in Computer Science, pp. 740-755. Springer, 2014.
Tsung-Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie. Feature pyramid networks for object detection. In CVPR, pp. 936-944. IEEE Computer Society, 2017b.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang. and Kevin Murphy. Progressive neural architecture search. In ECCV (I), volume 11205 of Lecture Notes in Computer Science, pp. 19-35. Springer, 2018.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. In ICLR (Poster). OpenReview.net, 2019.
Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention based neural machine translation. In EMNLP, pp. 1412-1421. The Association for Computational Linguistics, 2015.
Dhruv Mahajan, Ross B. Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV (2), volume 11206 of Lecture Notes in Computer Science, pp. 185-201. Springer, 2018
Ravi Teja Mullapudi, William R. Mark, Noam Shazeer, and Kayvon Fatahalian. Hydranets: Spe- cialized dynamic architectures for efficient inference. In CVPR, pp. 8080-8089. IEEE Computer Society, 2018
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zachary Vito,o Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, pp. 8024-8035, 2019.
Hicu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In ICML, volume 80 of Proceedings of Machine Learning Research, pp. 4092-4101 PMLR, 2018.
Ilija Radosavovic, Raj Prateck Kosaraju, Ross B. Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. CoRR, abs/2003. 1077700003...8.... 2020.
JP Rauschecker. Neuronal mechanisms of developmental plasticity in the cat's visual system. Human neurobiology, 3(2):109-114, 1984.
10

---

Under review as a conference paper at ICLR 2021
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V. Le. Regularized evolution for image classifier architecture search. AAAI, pp. 4780-4788 AAAI Press, 2019.
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. In NIPS, pp. 91-99, 2015.
Olga Russakovsky. Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Scan Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis., 115(3):211-255 2015.
Mark Sandler. Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Chinh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR. pp. 4510-4520. IEEE Computer Society, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1): 1958, 2014.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. CoRR, abs/1505.00380 2015.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, pp. 1-9. IEEE Computer Society, 2015.
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander A. Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In AAAI, pp. 4278 4284. AAAI Press, 2017.
Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, volume 97 of Proceedings of Machine Learning Research, pp. 6105-61 14. PMLR, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, pp. 5998-6008, 2017.
Andreas Veit, Michael J. Wilber, and Serge J. Belongie. Residual networks behave like ensembles of relatively shallow networks. In NIPS, pp. 550-558, 2016.
Xiaolong Wang. Ross B. Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, pp. 7794-7803. IEEE Computer Society, 2018a.
Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E. Gonzalez. Skipnet: Learning dynamic routing in convolutional networks. In ECCV (13), volume 11217 of Lecture Notes in Computer Science, pp. 430- Springer, 2018b.
Mitchell Wortsman, Ali Farhadi, and Mohammad Rastegari. Discovering neural wirings. In NeurIPS, pp. 2680-2690, 2019.
Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S. Davis, Kristen Grauman, and Rogério Schmidt Feris. Blockdrop: Dynamic inference paths in residual networks. In CVPR, pp. 8817-8826. IEBE Computer Society, 2018.
Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR, pp. 5987-5995. IEEE Computer Society, 2017.
Saining Xie, Alexander Kirillov, Ross B. Girshick, and Kaiming He. Exploring randomly wired neural networks for image recognition. In ICCV, pp. 1284-1293. IEEE, 2019.
Brandon Yang, Gabriel Bender, Quoc V. Le, and Jiquan Ngiam. Condconv: Conditionally parameter- ized convolutions for efficient inference. In NeurlPS, pp. 1305-1316, 2019.
11

---

Under review as a conference paper at ICLR 2021
Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, and Ping Wang. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. In NewrIPS, pp. 2130-2141, 2019.
Kun Yuan, Quanquan Li, Jing Shao, and Junjie Yan. Learning connectivity of neural networks from topological perspective. CoRR, abs/2008.08260 2020.
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV (1), volume 8689 of Lecture Notes in Computer Science, pp. 818-833. Springer, 2014.
Bolei Zhou, Aditya Khosla, Àgata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. CVPR, pp. 2921-2929. IEEE Computer Society, 2016.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V. Le. Learning transferable architectures for scalable image recognition. In CVPR, pp. 8697-8710. IEEE Computer Society, 2018.
12

---

Under review as a conference paper at ICLR 2021
6 APPENDIX
6.1 MODEL SETTINGS IN IMAGENET CLASSIFICATION
Setup for MobileNetV2. We train the networks for 200 epochs with a hall--period-cosine-happpp learning rate decay. The initial learning rate is 0.4 with warmup phase of 5 epochs. The weight decay is set to 4e-5. To prevent overfitting, we use label smoothing (Szegedy et al., 2017) with a coefficient of 0.1 and dropout (Srivastava et al., 2014) before the last layer with rate of 0.2. For building DG-Net, the Inverted Bottleneck block is represented as a node.
Setup for ResNets and ResNeXt. The networks are trained for 100 epochs with half- half-period- cosine- learning rate decay. The initial learning rate is 0.4 with warmup phase of epochs. The weight decay is set to e-4. We use label smoothing with a coefficient of 0.1. Other details of the the same as (Goyal et al., 2017). To form DG-Net, the BasicBlock of ResNet18 and Bottleneck block of ResNer50/101 is denoted as nodes. For ResNeXt, the Aggregated Boutlenck block set to be node.
training procedure
Setup for RegNets. The networks are trained for 100 epochs with learning rate decay. The initial learning rate is 0.8 with a warmup phase of epochs. The weight decay is set 5e-5. We use label smoothing with coefficient of 0.1. Other details are the same as (Radosavovic et al., 2020). For DG-Net, the Regular Bottleneck is transformed to be a node.
6.2 LOCATION OF ROUTERS
X5
X5
X3
Router
Rousor
Man
Convolution
Convolution
Router
Convolution
Rouber
0667
0,77
d67
(a)
(b)
(c)
Figure 3: Different routing methods with different locations of routers.
In this paper, the connectivity is determined by the weights of edges which are predicted using extra conditional module routers. Within node, there are two ways to obtain the weights, respectively predicting the weights of the output edges (as shown in Fig. 3 (a)) or predicting the weights of the input edges (as shown in Fig. 3 (b)). For the output type, as described in section 3.3, the router receives the transformed features as input and generates attention over output edges connected with posterior nodes. For the input type, the number of routers is related to the in-degree of the current node. Each router receives features from the connected preceding node and predicts weight for each input edge independently.
Although the form seems to be different, the two methods are equivalent under the routing function designed. The router module consists of global average pooling, fully-connected layer and
13

---

Under review as a conference paper at ICLR 2021
sigmoid activation. The router in Fig. 3 (a) can be split into the type of Fig. 3 (c). It can be noted as: (11) where g is the feature vector after global average pooling, w.j is the weight of independent fully- connected layer after splitting, and [-] indicates concatenation. The prediction for the output edge of the current node is equal to the prediction for the input edge of the next node. Therefore, the two
p(g) - oooooow""' g b) = [o a : bj)l
methods are equivalent. To simplify, we select the first type in implementation.
6.3 ABLATION STUDY ON CONNECTIVITY METHOD
We conduct an ablation study on different connectivity methods to reflect the effectiveness of the proposed DG-Net. The experiments are performed in ImageNet and follow the training setting in section 4.1. For fair comparison, denotes assigning learnable parameters to the edge directly, which learns fixed connectivity for all samples. The symbol a, denotes the type DG-Net, which learns instance-aware connectivity. The experimental results given in Table 6. In this way, ResNet-50 with still outperforms one with by 1.28% in top-1 accuracy. And ResNet-101 101 is the same. This demonstrates that due to the enlarged optimization space, dynamic connectivity is better than static connectivity in these networks.
select ResNet-50/101 as the backbone structure. The symbol
Table 6: Ablation study on different connectivity methods. Experiments are conducted in ImageNet and Top-1 accuracy are reported. Results show that the proposed dynamic graph outperforms static networks with/without learnable weights of edges large marigins.
Backbone
as Top-1
A Top-1
ResNet-18
70.30
70.51
0.21
71.32
ResNet-50
76.70
77.00
0.30
78.28
1.58
ResNet-101
78.29
78.64
+ 0.35
79.90
1.61
MobileNelV2-1.0
72.60
72.86
0.26
73.54
0.94
6.4 ABLATION STUDY ON INITIALIZATION SCHEMES FOR ROUTERS
80
blas ot bies
blas of Top1g78
70-
bies -3, Top1
Validation Accurscy
60
50
Trainnn
40-
30-
20000
50000
Rtertionn
Iteaaions
Figure 4: Different initialization schemes for routers. The positive bias initializes the connections as existence, obtains lower training loss in the early training procedure, and achieves higher validation accuracy than negative and zero biases.
For the routers, we use the routing transformation defined as p(x) = oowT ((xx + b), where w" the weight matrix and b the bias vector. This suggests simple initialization scheme that the bias
14

---

Under review as a conference paper at ICLR 2021
can be initialized with positive value (e.g. etc.) such that the network is initially biased towards existence connections behavior. This scheme is strongly inspired by the proposal of (Gers et al., 2000) to initially bias the gates in Long Short-Term Memory recurrent network to help bridge long-term temporal dependencies early in learning. And this initialization scheme also adpoted in Highway netowrks (Srivastava et al., 2015) and Non-local networks (Wang al., 2018 8a).
We conduct ablation study using DG-Net based on ResNet-50 in ImageNet. Details of the training procedure the same in section 4.1. The bias initialize with {3,0, 0, -3] respectively. These initialization methods correspond to existence connections, unbiased and non-existeml connections. Experimental results are given in Fig. 4. be seen that the positive initialization of bias achieves lower training loss in the early training procedure and obtains higher validation Top-1 accuracy of 78. 78.28%. This suggests that initializing connections to existing is better than unbiased initialization and non-existent.
6.5
STATISTICAL AND VISUALIZATION ANALYSES OF THE LEARNED CONNECTIVITY
Mean of weights in stage-1
Mean weights stage-2
0.5
4
0.188
0.46
0.422
0.318
0.395
0.299
0.514
0.49
0.437
0.4
0.4
3
0.5
0.307
0.307
0.0
0.3
0.3
0.5
0.382
0.373
0.0
0.0
2
0.5
0.368
0.0
0.0
0.2
0.2
0.5
0.5
0.0
0.0
0.0
0.1
0.1
1-
0.5
0.0
0.0
0.0
0.5
0.0
0.0
0.0
0.0
0.0
Mean of weights in stage-3
Mean weights stage-4
7
0.547
0.5
0.131
0.487
0.258
0.847
6
0.5
0.361
0.313
0.331
0.275
0.209
0.0
0.8
0.4
0.5
0.397
0.262
0.251
0.276
6.0
0.0
0.5
0.344
0.978
0.0
0.6
0.3
0.5
0.318
0.0
0.5
0.0
0.2
2
0.5
0.986
0.0
0.0
2
0.5
0.37
0.0
0.0
00
0.0
0.1
1-
0.5
0.0
0.0
0.0
0.5
6.0
0.0
0.0
0.0
6.0
0.0
0.0
5
The distribution of mean of weights of edges in different graphs/stages. Darker colors represent larger weights.
Figure 5:
To analyze the architecture representations, visualize the learned connectivity through the adja- cency matrices as noted in section 3.4. The validation dataset that contains 50000 images of ImageNet used for inference. We select the trained DG-Net with ResNet-50 that contains stages for analysis.
For statistical analysis, we show the distribution of the mean of the weights edges in Fig. and the distribution of standard deviation in Fig. 6. To visually analyze the connections corresponding
15

---

Under review as a conference paper at ICLR 2021
different samples, we give the learned connectivity in Fig. 7. Some observations and analysis can be made:
From the statistical analysis in Fig. 5, the weights of connections between different nodes have obvious differences. The difference is related to the topological orders of the nodes and the stages located.
Statistically, graph, the output edges of the nodes in the front of topological orders have larger weights. This can be explained that for node with the order of i, the generated X, can be received by node j (where j > i). This causes the features generated by the front nodes to participate in aggregation more, which can be used to reallocate calculation resources future work.
downstream input. It makes the front nodes contribute
From Fig. it can be seen that there exist discrepancies in weight changes for different edges with respect to input samples. The difference is also related to the topological orders of the nodes and the stages located.
(4) Interestingly, in stage, the edges of the nodes in the back of topological orders have a larger variance. Similarly, the weights of edges in deeper stages also have a larger variance. We speculate that it related to the level of semantic information of features. Specifically, features generated by the deep layers have high-level semantic information and the correlation of samples is stronger than features with low-level information generated by the shallow layers.
(5) In Fig. 7, it can be seen that for different samples, both the structure and the corresponding weights learned by routers are different. For some "easy" samples, part edges masked, resulting in a lighter model. This benefits computational efficiency and can be studied further in future work.
16

---

Under review as a conference paper at ICLR 2021
Standard Deviation of weights in stage-1
Standard Deviation of weights in stage-2
0.047
0.034
0.01
0.021
0.049
0.036
0.05
0.04
0.0
0.031
0.032
0.039
0.0
0.04
0.015
0.0
0.03
0.0
0.038
0.042
0.0
0.02
2
0.0
0.011
0.0
0.0
0.02
0.024
0.0
0.01
0..01
1
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
3
Standard Deviation weights stage-3
Standard Deviation weights stage-4
0.10
7
0.019
0.064
0.083
0.195
0.175
6-
0.043
0.093
0.104
0.150
5
0.0
0.048
0.039
0.055
0.073
0.0
0.0
0.0
0.106
0.052
0.0
0.125
0.06
4
0.0
0.043
0.039
0.076
0.0
0.0
0.0
0.100
0.04
0.075
0.050
2
0.025
1
0.0
0.0
0.0
0.00
0.000
4
5
Figure 6:
The distribution of standard deviation of weights of edges in different graphs/stages.
Darker colors represent larger weights.
17

---

Under review as a conference paper at ICLR 2021
XTXX
2.31
6.1
AMA
SALE
A.13
AM
5.11
3-111
111
CWLL
Figure Visualization of the learned connectivity for different input samples.
18


Reviews:
[{"review_id": "mLzmYfChPw", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a novel method, called Dynamic Graph Network (DG-Net), for optimizing the architecture of a neural network. Building on the previous work introduced by (Xie et al., 2019), the authors propose to consider the network as a complete directed acyclic graph (DAG). Then, the edge weights of the DAG are generated dynamically for each input of the network. At each node of the network, the authors introduce an extra-module, called router, to estimate the edge weights as function of the input features.\n\nThe proposed method addresses the problem of optimizing the connectivity of neural networks in an interesting way, where the architecture is not fixed but it depends on the input instances. Moreover, I think that a strong advantage of the proposed technique is that the optimization of the architecture comes with a negligible extra cost both in terms of parameters and computational complexity. Overall, the paper is well written and easy to follow.\n\nMy only serious concern is the degree of novelty with respect to (Yuan et al., 2020), which was published at ECCV 2020. The main difference seems to be that in the proposed method the graph is dynamic (i.e., it depends on the input instances), instead  in (Yuan et al., 2018) the graph is learned but fixed for all the input samples. In the experimental results, I would have expected a deeper ablation study on the importance of the dynamic graph, since this is the main contribution of the paper. Instead, there is only one experiment in the appendix (Table 6). Therefore, the impact of the dynamic graph in the performance of the proposed method is not clear and it is difficult to evaluate the importance of this contribution.\n\nOther comments:\n\n- In Sec. 3.1 the authors say that the ResNet architecture can be represented with a DAG where the set of edges is defined as E={(i,j)|j=i+1,i+2}. This is not true: if you unroll the definition of the ResNet architecture, as done in Eq. (4)-(6) in [1], and compare it with what you obtain using Eq. (1), it is easy to see that the two resulting functions are different.\n\n- The definition of the convolutional block is not clear, is it a ReLU-conv-BN triplet as in (Xie et al., 2019)? \n\n- The use of a DAG with edge weights for representing the architecture is not novel, it was already introduced in (Xie et al., 2019).\n\n- In Sec. 4.3, Table 5 shows a comparison with state-of-the-art NAS-based methods. DG-Net is implemented using RegNet-X and RegNet-Y as the basic architecture, however in Table 5 the performance of the basic architectures (without the dynamic graph optimization) is not reported, this would be useful to evaluate the gain provided by the optimization of the architecture.  \n\n\n[1] Veit et al., Residual Networks Behave like Ensembles of Relatively Shallow Networks, NIPS 2016\n\n###############################################################################################\n\nAfter the discussion period:\n\nI thank the authors for their responses and for updating the paper. The authors have added a deeper analysis on the impact of the dynamic graph, however I still believe that the novelty of the paper is a bit limited. I have slightly increased my score to 6.", "extra_scores": {}, "extra_text": {}}, {"review_id": "OEZKq5tY-e", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a novel approach (DG-Net) to “generate” a dynamic structure for the neural network, by learning to predict and select the edges between computational nodes in an end-to-end manner. The method is based on a gating mechanism, applied on top of a fully connected graph (similar to the connectivity in a DenseNet), designed to control the quantity of information received from each previous layer. The experiments show consistent improvement in image classification (ImageNet) and object detection (COCO).\n\nPro:\n- I enjoy the interpretation of the “weighted edges” as a dynamic architecture, able not only to address a more general class of models but also to adapt to each input accordingly (using a second-order approach).\n- The method is simple, yet effective with the results on both tasks showing improvement at a low computational cost.\n- The paper is clearly written and easy to follow and understand.\n\nCons:\n- In my opinion, two approaches are very related to this paper: the Highway network [1] and the ablations performed in the ResNet paper [2]. In both cases, the intuition is a little bit different: improve the expressiveness by replacing the residual connection with something more powerful - an input-dependent gate. While I agree that this paper offers a more general framework, working on a densely connected graph instead of a limited subset of residual edges, (plus some other minor technical differences: map gates vs scalar gates), the connection to those papers should be clearly discussed in the related work. Also, it is interesting to notice that in [2] this kind of gating decreases the performance compared to the simpler summation, contrary to what we see in DG-Net.\n- This idea suggests an additional ablation study: constant-gating instead of dynamic ones (by setting the alpha scalar equals to 1). This is related to the static experiment in the Supplementary material, but more closely to the DenseNet architecture and it would clarify that the improvements come indeed from the learned structure and not from the ability to combine features from different levels.\n- It is not clear from the paper how the ResNet architecture is adapted to the dynamic setup. I guess the graph is fully connected on each stage (as long as the spatial dimension is preserved), but since the paragraph about the “multi-stage” architectures comes only in section 3.1, that speaks about classical architectures, not the DG-Net, it is not very clear what densely connected means in the experiments.\n- Is any particular initialisation scheme used for the routers? Could it be important for the optimization to start with some parameters that lead to the original version of the model (initialize the parameters such that the residual connections start with alpha=1 and non-existent connections are close to 0)?  For example in [3] all the skip connections are initialised such that the extra module is ignored in the first iterations.\n\nMinors and observations:\n- Even if the focus of this work is the dynamic of the edges, it would be interesting to observe if the architecture learns to use all the computational nodes and only connect them differently than what we used to do, or if it rather prefers to drop nodes, thus using a lighter model in the end. (maybe a regularization term that encourages that would be useful for computational efficiency).  - A statistical analysis in this direction could reveal interesting ideas.\n- The formatting of Table 1 is hard to follow. At least a vertical line between Baselines and DG-Net would improve the readability.\n\n[1] Highway Networks, Srivastava et. al\n[2] Identity mappings in deep residual networks, He et. al, ECCV 2016\n[3] Non-local Neural Networks, Wang et. al, CVPR 2018\n\nMy main concern regards the connection to previous works that use gating to aggregate information from previous layers, which I see very related to the current work. However, the proposed method is more general and seeing these gates as a routing mechanism that allows the model to learn its own structure in a differentiable way is interesting and could lead to further, more advanced ways of doing this. So, with a clearer discussion of those previous methods and how the current approach differs from them, I lean towards acceptance.\n\n############# UPDATE #############\n\nI thank the authors for their response and for updating the manuscript according to our questions. I agree with the other reviewers that the novelty of this paper is quite limited. However, the idea of using a dynamic, learned graph from a general large search space of models is interesting, provides good empirical results on both image classification and object detection and the authors provide ablation for the new components that motivate the paper. I maintain my initial score: 6.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ls_Z0Ge92k", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n- Interesting extension of RandWire to learn better architectures\n- Good experimental results\n\nCons:\n- Idea could be seen as minor modification of RandWire\n- Doubts about memory requirements\n\nThe paper proposes an improve over the idea of randomly wired architectures [1] by exploiting a complete graph where edges are weighted by dynamically computed weights. Dynamically computing edge weights allows the network to optimize its topology. While this idea could be seen as a relatively small modification of [1], it is still very interesting and the results prove its effectiveness in commonly used tasks.\n\nIt would have been interesting to show an analysis of what the method is actually learning, e.g. if a particular topology emerges from the adjacency matrix, in order to provide more insights on why the method is effective.\n\nI also have a concern about memory requirements that I would like the authors to address. It is true that DG-Net does not require significantly more FLOPS or trainable parameters. However, you use a complete graph which means that it requires to store in memory O(L^2) activation tensors, i.e. the activation of each convolutional layer weighted by edge weight. This could become a limitation for larger networks. Note that this is also a problem for [1] but, in that case, the sparse connectivity mitigates the issue. \n\n\n[1] Saining Xie, Alexander Kirillov, Ross B. Girshick, and Kaiming He.  Exploring randomly wired neural networks for image recognition.", "extra_scores": {}, "extra_text": {}}, {"review_id": "0ZLdWwSnSdY", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper discusses a model for learning the architecure of a convolutional networks starting from a fully connected graph. The idea is to learn the adjacency graph of the model together with the weights of the networks.\n\nStrenghts:\n\n- The idea of thinking out-of-the-box by imagining new architectures is very attractive and interesting.\n\nWeaknesses:\n\n- The actual advantages in the model do not look apparent given the results in the evaluation section.\n\n- The theoretical foundation of this work is unclear. For example, it is unclear how the proposed solution will work in practice in terms of back-propagation.\n\n- The performance results show that the proposed method is characterized by performance close to those of existing methods.\n\n\nIn general, I really welcome this type of work: non-conventional, experimental and quite radical in terms of approach. However, unfortunately, the authors do not provide a convincing description of their approach. Unfortunately, it does not appear that the method is developed on a sufficiently strong theoretical basis. For example, it is unclear how back-propagation work in these circumstances when you don't have a stacked architecture.\n\nThe choice of the thresholds and the actual learning of the adjacency matrix is not described in sufficient detail.\n\nThe actual computation complexity and the trade-offs in terms of computational complexity/accuracy is unclear.\n\nIn the experimental results, the actual performance of the method appear very similar to the other methods. In some cases they might be the same since the confidence intervals are overlapping.\n\nQuestions:\n\n- What is the theoretical foundation of the method? Given the fully connected nature of the graph, how does back-propagation work in this case? In a sense, in fact, you have a DAG, how do you deal with cycles? When you do you stop the back-propagation if you do not have a stacked architecture?\n\n- Could you please provide the confidence intervals for all the results you presented? In fact, it seems that the values related to your approach are better than existing techniques in some cases but they look very close?\n\n- Is the computational complexity justified? Also note: you probably need a larger number of samples to learn the additional adjacency graph. What is the trade-off? Given the gain in terms of performance, the actual additional complexity might not be completely justified.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The idea presented in the paper is interesting and has caught the attention of the reviewers. However there seem to be only a tepid support for acceptance with a reviewer championing rejection. 
There is little novelty in the approach but empirical validation shows results that consistently improve over selected baselines. I am afraid that more evaluations would be needed at this stage to consider this work for acceptance.
Decision: Reject