iSparse: Output Informed Sparsification of Neural Networks
Authors: Yash Garg
K. Selcuk Candan
Venue:  (2020)
Keywords: dropout
dropconnect
sparsification
deep learning
neural network

Abstract: Deep neural networks have demonstrated unprecedented success in various knowledge management applications. However, the networks created are often very complex, with large numbers of trainable edges which require extensive computational resources. We note that many successful networks nevertheless often contain large numbers of redundant edges. Moreover, many of these edges may have negligible contributions towards the overall network performance. In this paper, we propose a novel iSparse framework and experimentally show, that we can sparsify the network, by 30-50%, without impacting the network performance. iSparse leverages a novel edge significance score, E, to determine the importance of an edge with respect to the final network output. Furthermore, iSparse can be applied both while training a model or on top of a pre-trained model, making it a  retraining-free approach - leading to a minimal computational overhead. Comparisons of iSparse against PFEC, NISP, DropConnect, and Retraining-Free on benchmark datasets show that iSparse leads to effective network sparsifications.

Under review as a conference paper at ICLR 2020
ISPARSE: OUTPUT INFORMED SPARSIFICATION OF NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Deep neural networks have demonstrated unprecedented success in various knowledge management applications. However, the networks created are often very complex, with large numbers of trainable edges which require extensive com- putational resources. We note that many successful networks nevertheless often contain large numbers of redundant edges. Moreover, many of these edges may have negligible contributions towards the overall network performance. In this pa- per, propose novel iSparse framework, and experimentally show, that we can sparsify the network, by 30-50%, without impacting the network performance. iSparse leverages novel edge significance score, E, to determine the importance an edge with respect to the final network output. Furthermore, iSparse can be applied both while training model a retraining-free approach leading to a minimal computational overhead. Com- parisons of iSparse against PFEC, NISP, DropConnect, and Retraining-Fiee benchmark datasets show that iSparse leads to effective network sparsifications.
top of pre-trained model, making it
INTRODUCTION
Deep neural networks (DNNs), particularly convolutional neural networks (CNN), have shown im- pressive success in many applications, such as facial recognition (Lawrence et al. 1997). time series analysis (Yang et al. 2015), speech recognition (Hinton al. 2012). object classification (Liang & Hu 2015), and video surveillance (Karpathy & et. at. 2014). As the term "deep" neural networks implies, this success often relies on large networks, with large number of trainable edges (weights)
(Huang et al. 2017 Zoph et al. 2018: He et al. 2016 Simonyan & Zisserman 2015).
While a large number of trainable edges help generalize the network for complex and diverse pat- terns in large-scale datasets, this often comes with enormous computation cost to account for the non-linearity of the deep networks (ReLJ,, sigmoid, tanh). In fact, DNNs owe their recent suc- cess to hardware level innovations that render the immense computational requirements practi- cal (Ovtcharov & et. al. 2015: Matthieu Courbariaux et al. 2015). However, the benefits of hard- ware solutions and optimizations that be applied to general purpose DNN or CNN limited and these solutions are fast reaching their limits. This has lead to significant interest network- specific optimization techniques, such as network compression (Choi & al.. (2088), pruning (Li let al.] 2016)] Yu et al. 2018), and regularization (Srivastava & et. al.[ Wan et al. 2013). aim to reduce the number of edges in the network. However, many of these techniques require retraining the pruned network, leading to the significant amount of computational waste.
1.1 NETWORK SPARSIFICATION
Many successful networks nevertheless often contain large numbers of redundant edges. Consider for example, the weights of sample network shown in Figure Ta] As we here, the weight distribu- tion is centered around zero and has significant number weights with insignificant contribution to the network output. Such edges may add noise -on-informative information leading to reduction
in the network performance. (Denil et al. 2013] Ashouri et al. 2018 eu al. 2018) has shown that it is possible predict 95% network parameters while only learning 5% parameters.
Sparsification techniques can generally be classified into neuron/kernel sparcification (Li et al. 2016 Yu al. 2018) and edge/weight sparcification techniques (Wan al. 2013) Ashouri al.

---

Under review as a conference paper at ICLR 2020
scom. Looe she wht aTe3
Accuric
1400
0.590
2003
Model Azzu
0.500
6.00
0.570
0.568
0.508
10
20
25
15
40
50
Spar sliicattionn Factor
(a) Weight Distribution for LcNet-5
(b) Model Accuracy vs Sparsification Factor
Figure 1: Overview of weight distribution and model accuracies for MNIST dataset (LeNet-5)
2016 proposed eliminate neurons that have low 12-norms of their weights, whereas (Yu et al. 2018) proposed a neuron importance score propagation (NISP) technique where neuron importance scores (using Roffo & al. (2015) See Equation
2018) (Li et al.
propagated from the output layer to the input layer in a back-propagation fashion. Drop-out (Srivastava & et. al.) 2014) technique instead deactivates
activations random. As edge sparsification tech- nique, DropConnect (Wan et al.) 2013) selects edges to be sparsified randomly. (Ashouri et al., 2018) showed that the network performance can be maintained by eliminating insignificant weights without modifying the network architecture.
1.2 OUR CONTRIBUTIONS: OUTPUT INFORMED EDGE SPARSIFICATION
Following these works, we argue network sparsification can be very effective tool for reducing sizes and complexities of DNNs and CNNs, without any significant loss in accuracy. However, also argue that edge weights cannot be used "as is" for pruning the network. Instead, one needs to consider the significance of each edge within the context of their place in the network (Figure 2 "Two edges a network with the same edge weight may have different degrees of contributions to the final network output" and in this paper, we show that is possible to quantify significance each edge in the network, relative to their contributions to the final network output and use these measures significance to minimize the redundancy in the network by sparsifying the weights that contributes insignificantly to network. We, therefore, propose novel iSparse framework, and experimentally show, that we can sparsify the network. by almost 50%, without impacting the network performance. The key contributions of our proposed work are as follows:
Output-informed quantification of the significance of network parameters: Informed by the final layer network output, iSparse computes and propagates edge significant scores that measure the importance of each edge with respect to the model output (Section 3)
Retraining- network sparsification (Sparsify-wiithe The proposed iSparse framework is robust to edge sparsification and can maintain network performance without having to retraining the network. This implies that one can apply iSparse on pre-trained networks, on-the-fly, to achieve the desired level of sparsification (Section 3.3
Sparsification during training (Train-withh: iSparse can also be used as regularizer during the model training allowing for learning of sparse networks from scratch (Section 4).
As the sample results Figure Tb shows, iSparse able to achieve 30-50% sparsification with minimal impact on model accuracy. More detailed experimental comparisons (See Section 5 of iSparse against PFEC, NISP, Retraining-Free and DropConnect on benchmark datasets illustrated that iSparse leads to more effective network sparsifications.
RELATED WORKS
A neural network is sequence of layers of neurons to help learn (and remember) complex linear patterns in a given dataset (Grossberg 1988). Recently, deep neural networks (DNNs), and particularly CNNs, which leverage recent hardware advances increase the number of layers in the
2

---

Under review as a conference paper at ICLR 2020
rt's contribution nb
Inpa
weight signilicance,
Output
ny's contribution to the overall output
Overview of iSparse sparsification, considering the ny's contribution to overall output rather than only between and nj neurons
Figure 2:
network to scales that were not practical until recently. (Lawrence al. 1997 Yang et al. 2015
Hinton et al. [012]] Liang & Hu 2015 Karpathy & et. at. 2014) have shown impressive success in several data analysis and machine learning applications.
A typical CNN consists of (1)feature extraction layers are responsible for learning complex patterns in the data and remember them through layer weights. An m layer, Li, maps an n-dimensional input (Y,-1) .) to an m-dimensional output (Y)) by training for a weight matrix Wick th X (see Section 3.1 for further details); (2) activation layers, which help capture non-linear patterns in the data through activation functions (σ) which maps the output from feature extraction layer to non- linear space ReLU and softmax are commonly used activation functions); and (3) pooling layers, (sampling) which up- down- sample the intermediate data in the network.
The training process a neural network often comprises of two key stages: (1) (Dfforwardprropagation (upstream) maps the input data, X, to an output variable, Y. At each layer, we have Y = L(Y)) = (W,Y++B)) where Y X. Intuitively, each layer learns extract new features from the features extracted by the prior layer. (2) taickwanl-roppagulle
(downstream) revises network weights, W, based on the training error, Err = Y Yl. where Y and Y are the ground truth and predictions of the model, respectively, such that update weights are defined as W W Errrr.
The number of trainable parameters in a deep network can range from as low as tens of thousands (LeCun et al. 1999) hundreds of millions (Simonyan & Zisserman 2015) (Table Section (5). The three order increase in the number trainable parameter may lead to parameters being redundant or may have negligible contribution to the overall network output. This redundancy and insignifi- cance of the network parameters has led to advancements in network regularization, by introducing dynamic or informed sparsification the network.
These advancements can be broadly classified into two main categories: parameter pruning and pa- rameter regularization. In particular, pruning focuses on compressing the network by eliminating the redundant insignificant parameters. (Han et al. 2015 Han & et. al. 2016) aims to prune the pa- rameters with zero weights inspired from 4 and lo regularization (Tibshirani 1996. Tikhonov] 1963). (Li et al. 2016) choose to filter convolutional kernel with minimum weight values in given layer. Recently, Yu et af.) 2018) minimizes the change in final network performance by elimi- nating the neuron that have minimal impact on the network output by leveraging neuron importance score (N2) (See Section (.3) computed using Inf-FS (Roffo & et. al. 2015).. More complex ap- proaches have been proposed to tackle the problem of redundancy in the network through weight quantization. (Rastegari & et. al. 2016) propose to the quantize the inputs and output activations of the layers CNN by using step function and also leveraging the binary operation by using the binary weights opposed to the real-values weights. (Chen & et. al. 2015) focuses on low-level mo- bile hardware with limited computational power, and proposed leverage the inherent redundancy in the network for using hashing functions to compress the weights in the network.
(Bahdanau et al., 2014 Woo et al. 2018) showed that the each input feature to a given layer in the network rarely have the same importance, therefore, learning there individual importance (attention) helps improve the performance of the network. More recently, (Garg & Candan 2019) has shown that input data informed deep networks provide high-performance network configurations. In this paper, we rely on output information for identifying and eliminating insignificant parameters from the network, without having to update the edge weights or retraining the network.
3

---

Under review as a conference paper at ICLR 2020
3 ISPARSE: OUTPUT INFORMED SPARSIFICATION OF NEURAL NETWORKS
As discussed in Section in order to tackle complex inputs, deep neural networks have gone in- creasingly deeper and wider. This design strategy, however, often results in large numbers of in- significant edges (weights), if not redundant. In this section, we describe the proposed iSparse, framework which quantifies the significance of each individual connection in the network with spect to the overall network output to determine the set of edges that can be sparsified to alleviate network redundancy and eliminate insignificant edges. iSparse aims to determine the significance of the edges in the network to make informed sparsification of the network.
3.1 MASK MATRIX
A typical neural network, N,, can be viewed as sequential arrangement of convolutional (C) and fully connected (F) layers: ((X) = f (Lu-1.... ... (Lo (x)))))) here, X is the input, 7. the total number of layers in the network and L (C,F) s.t., any given layer, Li s le L, can be generalized as
(1) where, Yi is the input to the layer (s.t. Y = Ýi- and for = L, = X) and or, Wi, and B₁ are the activation function, weight, and bias respectively. Note that, if the 1h layer has my neurons and the (1 1) th layer has ny neurons, then Y; [Rmxx1 Yi E Rm=x1, W rgmxx,, and B R
((()) 07 (W + B)) = Yt
Given this formulation, the problem of identifying insignificant edges can be formulated as the prob- lem of generating a sequence of binary mask matrices, Mi...., MM, that collectively represents whether any given edge in the network is sparsified (0) or not (1):
Mi = B Xm I {0, 1515L.
(2)
3.2 EDGE SIGNIFICANCE SCORE
Let Mj be mask matrix as defined in Equation 2 and Mj can be expanded as
M
MLL..:
M =
(3)
M
M MI
where each Mr E {0, 1} corresponds to an edge e1,i in the network. Our goal in the paper is to develop an edge significant score measure help set the binary value of MI, for each edge in the network. More specifically, we aim to associate a non-negative real valued number, Ér,i, 2 0, to each edge in the network, s.t.
{ E 2 (01)) E.a. < (0,)) Vi = 1... m,,j 1... 73g-
Mr
(4)
Here, (6:) represents the lowest significance of the 6 of the most significant edges in the layer Intuitively, given target sparsification rate, 6,, we rank all the edges based on their edge significance scores and keep only the highest scoring 0,% % of the edges by setting their mask values 1.
As we have seen in Figure Ta the (signed) weight distribution of the edges in a layer is often centered around zero, with large numbers of edges having weights very close to 0. As we also argued in the Introduction, such edges can work counter-intuitively and add noise non-informative information leading to reduction in the network performance. In fact, several existing works, such as (Ashouri et al.) 2018), relies on these weights for eliminating insignificant edges without having to retrain the network architecture. However, as we also commented in the Introduction, we argue that edge weights should not be used alone for sparsifying the network. Instead, edge within the context of their place in the network: Two edges in a network with the same edge weight may have different degrees of contributions to the final network output. Unlike existing works, iSparse takes this into account when selecting the edges to be sparsified (Figure
needs to consider each
An edge is defined direct connection (weighted) between neurons.
4

---

Under review as a conference paper at ICLR 2020
0.8
(3.2)
(4)
0.4
(a) Sample Network
(b) Retrain-free
(c) iSparse
Figure 3: A sample network architecture and its sparsification using Retraining-free (Ashouri et al., 2018) and iSparse; here node labels indicate input to the node; edge labels [0.1] indicate the edge weights; and edge labels between parentheses indicate edge contribution
More specifically, let W be the absolute positive of the weight matrix, WT, for edges in 1 layer. We compute the corresponding edge significance score matrix, E;, as
Wii1 Ni.1
W Lim, Ni.
E = W N
(5)
1mm,1 N
Lmm.n N1,
where, N₁ represents the neuron significance score N through Ni,,, and @ represents the scalar multiplication between edge weights and neuron scores. NI,i, denotes the significance of the tth input neuron the 1th connection layer of the network, which itself is defined recursively, based on the following layer in the network, using the conventional dot product:
Fl
W 11111 Nit1,1
W 141.1444
NN+1,
N₂ W N
(6)
1+1, X NN+1.
W 141m441004
N141.me
Note that Nj can be expanded as
N
(W 142
(W) NZ))),
Above. Nt denotes the neuron scores of the final output layer, and Nr is defined using infinite feature selection (Roffo & et. al. 2015 Yu et al. 2018)) Nı inffs(Y,, where RR (x is the number of input samples and is the number of output neurons) to determine neuron importance score with respect to the the final network output. Given the above, the edge score (Equation 5) can be rewritten as
E. W; (Wit (W (WE_(W)) NE))))
(8)
Note that the significance scores of edges in layer considers not only the weights of the edges, but also the weights of all downstream edges between these edges and the final output layer.
3.3 EDGE SPARSIFICATION
As noted in Section 3.1 the binary values in the masking matrix Mi depends on (6i),, which represents the lowest significance of the 9%% of the most significant edges in the layer therefore, given target sparsification rate, 0;, for layer I, we rank all the edges based their edge significance scores and keep only the highest scoring 0,% of the edges by setting their mask values to Note that, once edge is sparsified, change in its contribution is not propagated back to the layers carlier in the network relative to the sparsified edge. Having determined the insignificant edges with respect to the final layer output, represented in form of the mask matrix, M, (described in Section 3.1, the next step is to integrate this mask matrix in the layer itself. To achieve this, iSparse extends the layer 1 (Equation]
account for the corresponding mask matrix (Mi):
(X or ((W, Mi) Y: B:),
2 N, summarizes the edge and neuron importance in the subsequent layers i.c. Ci-1... LL.
3 In the experiments reported in Section 5 without loss of generality, assume that 0 has the same value for all connection layers in the network. This is not fundamental assumption and iSparse can casily accommodate different rates of sparsification across connection layers.

---

Under review as a conference paper at ICLR 2020
Table 1: Number of trainable parameters (weights) and statistics for various benchmark datasets
Network
LoNet
VGG
Datasets
MNIST 44,126
FMNIST
COIL20
COIL
100
NORB 105,801
CIFARIO 32.418.834
CIFARZ0
CIFAR100
SVHN
GTSRB 08,705,925
ImageNet 138,55
Weights
31,026
62,350
60,350
$2,028814
32,508
32.18.84
Resolution
28 28
32 x 92
82 92
96 x 56
32 x 32
32
82
x2
x2
221 x 224
Trai Set
60,000
90000
1300
6380
25,500
50,000
50000
50,000
-73,00
59,200
>11,281 21,281_1611 167
Test Set
10,000
10,000
140
720
26,300
10,000
100000
10,000
-26,000
12,030
~600000
Lahhls
where, * represents the element-wise multiplication between the matrices Wi and Mj. Intuitively, Mi facilitates introduction of informed sparsity in the layer by eliminating edges that do not con- tribute significantly to the final output layer.
INTEGRATION OF ISPARSE WITHIN MODEL TRAINING
In the previous section, we discussed the computation of edge significance scores on pre-trained network, such of pre-trained ImageNet models, and the use of these scores for network sparsifica- tion. In this section, we highlight that iSparse can also be integrated directly within the the training process.
To achieve this, the edge significance score computed for every trainable layer in the network using the strategy described in Section 3.2 and the mask matrix is updated using Equation[4] Furthermore, the hack-propagation rule, described Section
2 is updated to account for the mask matrices:
w Wi (M * Erri)
(10)
where, W are the updated weights, Wi original weights, 17 is the learning rate, and Err, is the error recorded by as the divergence in between ground truth (Y) and model predictions (Y) as Err, = Y Vi Note that, we argue that any edge that does not contribute towards the final model output, must not be included in the pack-propagation. Therefore, we mask the error as Err, * Mi.
EXPERIMENTAL EVALUATION
In this section, we experimentally evaluate of the proposed iSparse framework using LeNet and VGG architectures (See Section 5.2 and compare it against the approaches, such PFEC, NISP,
and DropConnect (see Section 5.3
5.1 SYSTEM CONFIGURATION
We implemented iSparse in Python environment (3.5.2) using Keras Deep Learning Library (2.2.4- tf) (Chollet et al. 2015 with TensorFlow Backend (1.14.0). All experiments were performed on an Intel Xeon E5-2670 2.3 GHz Quad-Core Processor with 32GB RAM equipped with Nvidia Tesla P100 GPU with 16 GiB GDDRS RAM with CUDA-10.0 and cuDNN v
5.2 BENCHMARK NETWORKS AND DATASETS
In this paper, without loss of generality, we leverage LeNet-5 (LeCun et al. 1999) and VGG-16 (Simonyan Zisserman 2015) as the baseline architectures evaluate sparsification performance on different benchmark image classification datasets and for varying degrees of edge sparsification. In this section, we present an overview of these architectures (See Table Ip.
LeNet-5: Designed for recognizing handwritten digits. LeNet-5 is simple network with 5 trainable (2 convolution and 3 dense) and 2 non-trainable layers using average pooling with tanh and soft- max as the hidden and output activation. LeNet's simplicity has made it common benchmark for datasets recorded constrained environments, such as MNIST (LeCun et al. 1998), FMNIST (Xiao)
al. 2017), COIL (Nene al. 1996a b). and NORB (LeCun al. 2004)
Results presented in this paper were obtained using NSF testhed: "Chameleon: A Large-Scale Re- configurable Experimental Environment for Cloud Research https: htmp::://wwt chamel eoncl oud.org/"

---

Under review as a conference paper at ICLR 2020
Doptornet
1.2
212
Node
2 21 9/
Sparsiiiatio
Figure 4: Top-1 and top-S accuracy for spar-Figure 5: Model classification time sparsi- sified VGG-16 for ImageNet (sparsify-wihh) fication factor for MNIST dataset (train-with)
83
68
Mhodl
- 2 -|
1
OFARD
CRUDO
IBARI0E
R/M
Sparifictioon Fortor
Dets Sees
(a) VGG network (VGG-16)
MBC
Dretieead
Model
40
0
6/
| 5/3
Sparifiction
(b) LeNet network (LeNet-5)
Figure 6: Top-1 classification accuracy results for sparsified pre-trained models (sparsify-with)
VGG-16: VGG (Simonyan & Zisserman 2015) ss 16 layer network with 13 convolution and 3 dense layers, with interleaved max-poot max-pooling layers. VGG leverages ReLU as the hidden activation to overcome the problem of vanishing gradient, as opposed to tanh. Given the ability of VGG network to learn the complex pattern in the real-world dataset, we use the network on benchmark datasets, such as CIFAR 10/20/100 (Krizhevsky) (009), SVHN (Netzer & et. al.) 2011), GTSRB Stallkamp & et. al.] 2012), and ImageNet (Deng et. al. 2009). Table [ reports the number of trainable parameters (or weights) for each model/data set pair considered in the experiments.
5.3 COMPETITORS
We compared iSparse against several state-of-the-ar network sparsification techniques: DropCon- nect (Wan al. 2013) a purely random approach, where edges are randomly selected for sparsifi- cation. Retraining-free (Ashouri al. 2018) considers each layer independently and sparsifies in- significant weights the layer, without accounting for the final network output contribution. PFEC (Li et al.) 2016) is kernel pruning strategy that aims to eliminate neurons that have low impact on the overall model accuracy. In order to determine the impact, PFEC computes the 22-norms of the weights of the neurons and ranks them, separately, for each layer. NISP (Yu al. 2018) pro- poses a neuron importance score propagation (NISP) technique where neuron importance scores are propagated from the output layer to the input layer in back-propagation fashion.

---

Under review as a conference paper at ICLR 2020
(a) PFEC
(b) NISP
(c) DropConnect
(d) Retrain-Free
(e) iSparse
Figure 7: Mask matrices for the LeNet network conv 22 layer for MNIST data (sparsification factor 50%): dark regions indicate the edges that have been marked for sparsification; in (c) iSparse, the arrows point those edges that are subject to different pruning decision from retrain-free in(d) (green arrows point to edges that are kept iSparse instead of being pruned and red arrows point to edges that are sparsified in iSparse instead of being kept)
5.4
ACCURACY RESULTS
5.4.1 SPARSIFICATION OF PRE-TRAINED MODELS (sparsify-wihh)
In Figure we first present top-1 and top-5 classification results for ImageNet dataset for VGG- 16 network. As we see in the Figure 4 iSparse provides the highest robustness to the degree of sparsification the network. In particular, with iSparse, the network can be sparsified by 50% with S 6% drop in accuracy for top-1 and < 2% drop in accuracy for top-5 classification, respectively. In contrast, the competitors, see larger drops in accuracy. The closest competitor, Retrain-free, suffers loss in accuracy of 16% and ~ 6% for top-1 and top-5 classification, respectively, The other competitors suffer significant accuracy drops after mere 10-20% sparsification.
Figures 6a and 6b show the top-1 classification accuracy results for other models and data sets. As we see here, the above pattern holds for configurations considered: iSparse provides the best robustness. It is interesting in accuracy for the VGG-16 network and especially on the CIFAR data. This is likely because, VGG-CIFAR already relatively sparse (20% > sparsity as opposed and < 1% for LeNet) and these three techniques are not able to introduce additional sparseness in robust manner. In contrast, iSparse is able introduce significant additional sparsification with minimal impact on accuracy.
note that DropConnect, NISP, and PFEC see especially drastic drops
7% for VGG-ImageNet
Figure provides the mask matrices created by the different algorithms to visual illustrate the key differences between the competitors. As we see in this figure, PFEC and NISP, both sparsify input neurons. Consequently, their effect mask out entire columns from the weight matrix and this prevents these algorithms to provide fine grain adaption during sparsification. DropConnect selects individual edges for sparsification, but only randomly and this prevents the algorithm to provide sufficiently high robustness. Retrain-free and iSparse both select edges in an fine-grained manner: retrain-fire free uses relies on adge-weights whereas iSparse complements edge-weight with an edge significance measure that accounts for each edges contribution to the final output within the overall network. As we see in Figure / (d) and (e), this results in some differences the corresponding mask matrices, and these differences sufficient provide significant boost in accuracy.
8

---

Under review as a conference paper at ICLR 2020
Table 2: Top-1 classification accuracy for sparsified different architectures and datasets (train-wirh)
Datusets
MINIST
FMNIST RT
COIL.0
COILI10
NORB
Factor
DC
RT
18
DC
18
DC
RT
18
DC
RT
18
DC
RT
18
Rase (0%)
98.79
98.79
98,79
88.02
88.02
88.02
95.52
95.56
95.56
90.00
90.00
90,001
84,42
84,42
84,42
5%
98.90
97.99
99.00
88,
88. 70
88.75
95.00
94. 16
94.72
89.83
90.12
90.61
84.51
84.75
85,03
98.45
98.36
87.60
87.89
54.72
95.35
90.05
90.33
35.35
85.98
86.25
15%
98.76
98.79
98.92
88.14
87,43
87.96
95.00
95.21
95.50
89,30
89.73
90.00
83.91
84.12
84.73
98.79
98.81
98,98
88.11
95.83
96.10
90.83
91.03
83.29
25%
90.85
08.85
58.20
87.62
87.54
88.19
925.7
94.66
95.80
90.11
89.58
90.22
83.33
85.01
85.61
301%
98.08
98.71
98.98
87.75
88.25
55.00
94.74
98.00
90.12
90.12
901.55
83.09
X5.91
86.088
35%
98.73
98.73
98.79
88.14
86.99
88.06
94.72
94.99
95.27
90.11
91.35
91.99
83,98
85.01
84.81
40%
98.85
98.97
98.99
87.81
87.25
88,30
95.00
95.31
95.56
88.22
90.01
90.44
83.14
84,12
85,08
45%
98.91
98.95
985.99
87,45
87.11
87.82
95.29
95.04
95.56
89.45
89.87
90.23
85:47
85.12
86.02
50P4
98.55
98.85
99.000
87.26
88.15
88.46
95.00
94. 16
94.72
90.06
90.21
90.83
84.44
X5.61
86.77
Datasets
CIFAR10
CIFAR20
CIFAR100
SVHN
GTSRB
Factor
DC
RT
18
DC
RT
RT
18
DC
RT
18
DC
18
Base (0%)
K4.14
84.14
84.14
66.96
66.96
66:96
55.09
53.09
55.00
91.35
91.33
91.35
97.08
97.68
97.68
3%
84.21
84,36
84.66
62.16
62.01
62.85
57.35
52.74
33.10
93.49
92.42
92.70
96.49
97.68
988.64
10%
$4.35
84,33
84,35
63.63
66.33
67.33
31.32
31.01
51.48
95.00
92.56
92.96
95.03
91.09
93.21
15%
83.43
83,51
83.69
60.95
59.87
62.49
52.18
52:53
52.97
91.96
93.11
93.40
96.83
95.91
97.29
82.83
85.04
85.81
61.79
51.67
50.01
50.52
92.65
93.74
95.02
94,41
95.76
25%
$4.07
84.32
84.56
61.85
60.19
62.52
56.47
53.25
53.81
82.56
87.14
87.63
95.11
96.56
97.76
81.17
54.50
91.95
92.26
98.56
35%
83.06
84.95
85.42
60.09
37.81
61.79
$2.05
48.52
49.05
92.38
92.38
92.65
97.18
97.61
98.17
40%
83.96
82.01
82.21
66.34
65.09
66.62
50.19
55.11
58.68
88.79
91.92
92.31
95.61
96.91
97.61
43%
82.75
82.81
82.83
65.65
66.93
68.72
54.18
53.47
53.63
91.38
91.23
91.62
96.71
93.31
94.02
30%
83.44
84.15
84.53
65.37
65.04
68.22
31.65
31.78
52.56
82.94
37.61
87,68
94.31
95.91
97.31
Table 3: Robustness analysis for edge-based strategies vs iSparse (train-wiil with)
lan-Adam RT
Janb-RMS
ReLU-Adam
DC
18
MNIST
(0%)
98.82
98.82
5%
98.90
98.23
98.94
98.90
97.99
98.00
98.66
98.91
98..5
10%
99.02
98.75
99.15
98.17
98.45
98.56
98.90
98.75
98.86
15%
98.94
98.01
98.92
99.23
20%
98.76
98.41
98.97
98.79
98.81
98.86
99.05
99.32
99.53
25%
98.91
98.25
98.92
98.85
98.78
98.80
98.89
98.85
98.96
99.01
98.04
98.92
98.71
98.88
35%
98.94
98.65
99.05
98.73
98.75
98.79
98.86
98.62
98.92
40%
99.12
98.
99.57
98.83
98.97
98.99
98.83
98.
98.95
99.04
98.91
98.95
99.07
99.01
50%
99.07
98.01
99.12
98.55
98.85
99.00)
98.98
98.16
98.89
CIFAR1O
Base (%)
87.66
87.66
87.66
81,95
81.95
81,95
84.14
84.14
84.14
5%
88.87
89.56
89.66
81.17
85.94
86.14
84.21
84.56
84.66
10%
84.75
88.21
88.31
87.01
87.42
84.33
84.35
15%
81.78
81.89
82.49
88.75
88.99
89.04
83.43
83.51
83.69
20%
85.64
85.95
86.21
87.10
89:10
89.23
82.83
85.04
85.81
87.68
87.86
84.81
87.31
87.74
84.07
84.32
30%
87.07
86.21
86.30
80.20
83.45
83,56
83,41
84.17
84,48
35%
82.38
86.43
86.48
85.91
86.75
86.93
83.06
84.95
85.42
40%
86.76
88.63
88.77
82.17
34.10
87.44
83.96
82.01
82.21
45%
83,90
52
85.61
85.79
86.40
89.42
82.75
82.81
82.83
50%
80.45
79.21
79.59
83.89
86.42
86.52
83.44
84.15
84.53
5.4.2 SPARSIFICATION DURING TRAINING (train-with)
Tables present accuracy results for the scenarios where iSparse (iS) is used to sparsify the model during the training process. The table also considers DropConnect (DC) and Retrain-Fres (RF), as alternatives. As we see in the table, for both network architectures, under most sparsification rates, the output informed sparsification approach underlying iSparse leads networks with the highest classification accuracies.
5.5 ROBUSTNESS TO THE VARIATIONS IN NETWORK ELEMENTS
In this section, we study the effect of the variations in network elements. In particular, we compare the performance of iSparse (iS) against DropConnect (DC) and Retraining-Free (RF) for different

---

Under review as a conference paper at ICLR 2020
ilpansi Spano ahaan
08
10/40/
[n 13
10 20/00000 90
23
30 40
8 19
OPARX
Sparillution Peroutt Seta
Figure 8: Robustness to the layer order while sparsifying the network with iSparse (train-with)
hidden activation functions and network optimizers. Table presents classification performances for networks that rely on different activation functions (tanh and ReLU) and for optimizers (Adam and RMSProp). As in these Two tables, iSparse remains the alternative which provides the best classification accuracy under different activation/optimizztion configurations.
5.6 ROBUSTNESS TO THE VARIATIONS IN SPARSIFICATION ORDER
We investigate the performance of iSparseunder different orders in which the network layers are sparsified. In particular, we consider/hree sparsification orders: (a) input-to-oulpul layer order: this is the most intuitive approach as it does not require edge significance scores be revised based on sparsified edges in layers closer to the input; (b) nuiput-to-input layer-order: in this case, edges in layers closer the network output are sparsified first but, this implies that edge significance scores are updated in the earlier layers in the network to account for changes in the overall edge contributions to the network; (c) random layer order: in this case, to order of the layers to be sparsified is selected randomly. Figure 8 presents the sparsification results for different orders, data sets, and sparsification rates. As we see in the figure, the performance iSparse is not sensitive to the sparsification order of the network layers.
5.7 IMPACT OF SPARSIFICATION ON CLASSIFICATION TIME
In Figure 5 we investigate the impact of edge sparsification on the classification time. As we see in this Figure, edge sparsification rate has a direct impact on the classification time of the resulting model. When we consider that iSparse allows for ~ 30 50% edge sparsification without any major impact on classification accuracies, this indicates that iSparse has the potential to provide significant performance gains. What is especially interesting note in Figure 5 is that, while all three sparsification methods, iSparse, DropConnect, and Retraining- of sparsified edges for given sparsification factor, the proposed iSparse approach leads to the least execution times among the three alternatives. We argue that this because the output informed sparsification provided by iSparse allows for more efficient computations in the sparsified space
all have the same number
CONCLUSIONS
In this paper, proposed iSparse, novel output-informedt framework for edge sparsification in deep neural networks (DNNs). In particular, we propose a novel edge significance score that quantifies the significance of each edge in the network relative its contribution to the final network output. iSparse leverages this edge significance score to minimize the redundancy in the network by sparsifying those edges that contribute least to the final network output. Experiments, with 11 benchmark datasets and using two well-know network architectures have shown that the proposed iSparse framework enables 30- 50% network sparsification with minimal impact on the model classification accuracy. Experiments have also shown that the iSparse is highly robust to variations network elements (activation and model optimization functions) and that iSparse provides a much better accuracy/olassificaion-iii trade-off against competitors.
Hardware configuration can be found in Section 5.1
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Amir H Ashouri, Tarek Abdelrahman, and Alwyn Dos Remedios. Fast on-the-fty retraining-free sparsification of convolutional neural networks. NIPS, 2018.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint urXiv:149.04477 2014.
Wenlin Chen and et. al. Compressing neural networks with the hashing trick. In ICML,
Yoojin Choi and et. al. Universal deep neural network compression. NIPS, 2018.
François Chollet et al. Keras, 2015.
Jia Deng et. al. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Misha Denil, Babak Shakibi, Laurent Dinh, Nando De Freitas, al. Predicting parameters in deep learning. NIPS, 2013.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. ICLR, 2019.
Yash Garg and Selçuk Candan. Racknet: Robust allocation of convolutional kernels in neural networks for image classification. ICMR, 2019.
Stephen Grossberg. Nonlinear neural networks: Principles, mechanisms, and architectures. Neural networks, 1988.
Song Han and et. al. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. 2016.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In NIPS, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Geoffrey Hinton, Li Deng, and al. Yu. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal processing magazine, 2012.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinherger. Densely connected convolutional networks. In CVPR, 2017.
Andrej Karpathy and et. at. Large-scale video classification with convolutional neural networks. In CVPR, 2014.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Steve Lawrence, C Lee Giles, Ah Chung Tsoi, and Andrew Back. Face recognition: A convolu- tional neural-network approach. TNN, 1997.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998.
Yann LeCun, Patrick Haffner, Léon Bottou, and Yoshua Bengio. Object recognition with gradient- based learning. Shape, Contour and Grouping in CV. 1999.
Yann LeCun, Fu Jie Huang, and Leon Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In CVPR, 2004.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. ICLR, 2016.
Ming Liang and Xiaolin Hu. Recurrent convolutional neural network for object recognition. In CVPR, 2015.
11

---

Under review as a conference paper at ICLR 2020
Matthieu Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Low precision arithmetic for deep learning. In ICLR, 2015.
Sameer A Nene, Shree K Nayar, Hiroshi Murase, et al. Columbia object image library (COIL-20). 1996a.
Sameer A Nene, Shree Nayar, Hiroshi Murase, et al. Columbia object image library (COIL-100). 1996b.
Yuval Netzer and et. al. Reading digits in natural images with unsupervised feature learning. In NIPS, 2011.
Kalin Ovtcharov and et. al. Accelerating deep convolutional neural networks using specialized hardware. Microsoft Research Whitepaper, 2015.
Karl Pearson. Liii. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 1901.
Mohammad Rastegari and et. al. Xnor-net: Imagenet classification using binary convolutional neural networks. ECCV, 2016.
Giorgio Roffo and et. al. Infinite feature selection. In ICCV, 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. ICLR, 2015.
Nitish Srivastava and et. al. Dropout: simple way to prevent neural networks from overfitting. JMLR, 2014.
Johannes Stallkamp and et. al. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. NN, 2012.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 1996.
Andrei Nikolaevich Tikhonov. On the regularization of ill- ill-posed problems. In Doklady Akademii Nauk, 1963.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In JCML, 2013.
Sanghyun Woo, Jongchan Park, Joon- Young Lee, and In So Kweon. Cham: Convolutional block attention module. In Proceedings the European Conference on Computer Vision (ECCV), pp. 3-19, 2018.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms, 2017.
Jianbo Yang. Minh Nhut Nguyen, Phyo Phyo San, Xiao Li Li, and Shonali Krishnaswamy. Deep convolutional neural networks on multichannel time series for human activity recognition. In IJCAI, 2015.
Ruichi Yu, Ang I.,, Chun-Fu Chen, Jui-Hsin Lai, Vlad Morariu, Xintong Han, Mingfei Gao, Ching- Yung Lin, and Larry S Davis. Nisp: Pruning networks using neuron importance score propagation. In CVPR, 2018
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In CVPR, 2018.
12

---

Under review as a conference paper at ICLR 2020
Table 4: Model accuracies under different strategies for initializing neuron importance score CI-
FAR10 dataset for different sparsification factors; Base = 0%
VGG
ResNet
Factor
IDN
PCA
InFFS
IDN
PCA
InfFSS
Base (0%)
84.14
84.14
84.14
73.61
73.61
73.61
5%
82.94
83.96
84.66
73.25
73.20
74.13
10%
81.59
83.85
84.35
73.46
73.86
74.28
15%
81.24
83.45
83.69
73.32
73.57
74.12
20%
82.67
84.65
85.81
74.52
74.21
75.01
25%
81.42
83.98
84.56
74.86
74.45
74.95
30%
80.64
82.96
84.48
73.72
73.80
74.36
35%
80.45
81.56
85.42
73.53
73.99
74.01
40%
78.48
79.54
82.21
74.94
75.09
75.27
45%
80.23
81.24
82.83
73.06
72.39
73.39
50%
80.85
82.68
84.53
74.19
74.19
74.21
A APPENDIX
A.1 ABLATION STUDIES FOR NEURON SCORE INITIALIZATION
In this section, evaluate the performance iSparse framework with different type of neuron score initialization strategies). We evaluate the following scoring strategies:
Identity score (IDN) (all scores are "1")
Principal Component Analysis (PCA) PPearson 1901
Infinite Feature Selection (InfFS) (Roffo & et. al. (2015)
In Table 4 we observed that the iSparse framework with InfFS as mechanism to determine the neuron scores for the final layers leads to a better performance opposed to the other strategics. Intu- itively, this is because InfFS effectively identifies the importance of individual features (neurons), while considering all possible subset features. We believe this is very highly desirable property: conventional feature selection strategies, such PCA, ranks features based their discriminatory power irrespective of how features relate with each other. Similarly, using same importance score (identity) treats each feature in mutually exclusive fashion. In contrast, the way InfFS approaches the "feature selection" problem is highly appropriate to NNs, where the neuron output for the true label will be high, but we will also see low/small output for false labels.
A.2 PERFORMANCE EVALUATION FOR THE RESNET ARCHITECTURE
In this section, we evaluate iSparse for the ResNet-18 architecture (He et al. (2016)) for both, sparsify-with (no retraining after sparsification) and train-with (retraining after sparsification) figurations Results are presented Tables andd
A.2.1 SPARSIFY-WITH FOR RESNET
As observed in Table iSparse for ResNet demonstrates significant robustness to edge sparsifi- cation. iSparse outperforms DropConnect, and Retrain-Free, thus highlighting the importance of accounting edge's ss significance subsequent network along with edge weight.
A.2.2 TRAIN-WITH FOR RESNET
As we can see in the Table 6 the iSparse framework outperforms the competitors, such as Dropout, L1, DropConnect, Retraining- Free, and Lottery Winning Hypothesis (Frankle & Carbin 2019)) iSparse ability to account for edge significance based on both, edge weights and edge contributions to the final network output, leads to more robust and superior results.
13

---

Under review as a conference paper at ICLR 2020
Table 5: Model accuracy vs sparsification factor (sparsify-with) for ResNet architecture CIFARIO
Factor
DropConnect
Retraining-
iSparse
Base (0%)
73.61
73.61
73.61
5%
69.19
73.61
73.61
10%
63.48
73.61
73.61
15%
52.89
73.52
73.58
20%
41.38
72.85
73.47
25%
32.60
71.45
72.98
30%
21.23
71.07
72.77
35%
16.38
70.53
72.36
40%
12.88
68.41
72.05
45%
10.45
67.98
71.85
50%
10.00
65.48
71.01
Table 6: Model accuracy vs sparsification factor (train-with) for ResNet architecture CIFARIO
Factor
Dropout
L1
DropConnect
Retraning-
Lottery
iSparse
Base (0%)
73.61
73.98
73.61
73.61
73.61
73.61
5%
73.83
N.A.
73.72
74.35
74.10
74.43
10%
74.52
N.A.
73.22
74.49
74.31
74.78
15%
73.18
N.A.
73.69
73.82
74.40
74.12
20%
75.42
N.A.
69.84
73.08
73.38
75.01
25%
75.97
N.A.
72.43
73.75
73.77
74.95
30%
74.69
N.A.
72.43
74.10
73.39
74.36
35%
73.5
N.A.
72.59
74.07
73.99
74.01
40%
67.89
N.A.
70.38
73.15
74.10
75.27
45%
71.52
N.A.
67.07
72.37
74.15
73.39
50%
66.2
N.A.
71.35
72.69
74.13
74.21
A.2.3
ADDITIONAL TRTH RESULTS FOR VGG
In Table 7 we present model accuracies for iSparse along with Dropout, L1, Retraining-Free and Lottery winning hypothesis sparsification strategies. iSparse's informed sparsification leads superior performance in this model architecture as well.
Table 7: Model accuracy vs sparsification factor (Irain-with) for VGG architecture CIFAR10
Factor
Dropout
L1
DropConnect
Retraining-Free
Lottery
iSparse
Base (0%)
84.14
84.51
84.14
84.14
84.14
84.14
5%
85.13
N.A.
84.21
84,56
84.29
84.66
10%
84.77
N.A.
84.35
84.33
84.23
84.35
15%
86.81
N.A.
83,43
83.51
84.88
83.69
20%
77.81
N.A.
82.83
85.04
84.68
85.81
25%
73.42
N.A.
84.07
84.32
84.87
84.56
30%
76.16
N.A.
83,41
84.17
83.48
84.48
35%
79.28
N.A.
83.06
84.95
84.79
85,42
40%
77.51
N.A.
83.96
82.01
81.37
82.21
45%
73.51
N.A.
83,96
82.01
81.37
82.21
50%
69.21
N.A.
83.44
84.15
81.59
84,53
14


Reviews:
[{"review_id": "rkxpvDD1cr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes iSparse framework, which aims to sparsify a neural network by removing redundant edges. Unlike previous works where the edges were removed based on their weight value, or based on the relationship between input and output neurons, iSparse selects the edges to remove by computing the contribution of each edge with respect to the final outcome. The experiments show that, compared to several baseline methods, iSparse perform favorably on multiple datasets. \n\nComment:\nAlthough I am not an expert in network pruning or network sparsification, I know that the Lottery Ticket Hypothesis (Frankle & Carbin, 2019) were able to remove at most 80% of the weights of neural networks (both fully-connected and ConvNets) and still retain the original performance level. Compared to that, iSparse's performance does not seem too impressive. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "SkxE4-vpKr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed a method for network sparsification based on the significance of each edge (weight). Unlike some of the existing works, edge significance in this work is explicitly defined based on their influence over the network output.\n\nThe algorithm can be summarized as follows:\n1. Compute the significance of each activation (neuron) using an existing method (infinite feature selection).\n2. Compute the significance of each edge as the product between the neuron significance and the absolute edge weight.\n3. Sort the edges according to their significance scores and keep the top portion.\n\nI'm mainly concerned about the limited technical novelty: the proposed technique is essentially a heuristic. Specifically, it is not clear why the significance of each edge should be defined according to equation (5) rather than some other forms. The intuition behind masking out the gradients of in-significant edges in (10), i.e., \"we argue that any edge that does not contribute towards the final model output, must not be included in the back-propagation\", is again a heuristic that lacks justification. If theoretical analysis is not possible, it might be necessary for the authors to conduct controlled experiments/ablation studies to show that some of the design choices made in the paper are indeed superior over other alternatives.\n\nAnother of my concern is that, if the goal is to make the sparsification decision aware of the network output (e.g., the value of the loss function), a simpler approach would be to enforce L1 regularization over the edges. This way, edges that do not lead to significant impact on the loss would be automatically pruned away. I wonder how would the proposed approach compare against this simple baseline.\n\nOther suggestions:\n* Empirical evaluation is conducted using LeNet and VGG16. It would be interesting to extend the analysis to some other seminal architectures, such as ResNet, Inception and MobileNets.\n* It would be informative to report the hardware configuration used to obtain the execution time in Figure 5. Note the relative inference cost of different models may differ substantially over different hardware platforms.\n* Writing of the paper can probably be polished further for better clarity. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJgXGnVe5B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a sparsification technique that seeks edges contributing negligible amounts to the performance of a network. \n\npros)\n(+) This paper is written well but needs more clarity.\n\ncons)\n(-) This paper did not cite modern pruning/sparsification methods published recently.\n(-) The proposed method has only compared with some outdated methods.\n(-) Only LeNet-5 and VGG-16 have been used to validate the proposed method.\n(-) This paper lacks any analysis of why the proposed methods would work well. Specifically, on the formulation of the sparsification method, it is hard to find sufficient backups why the authors did like that.\n\nComments)\n- How would you guarantee using the infinite feature selection method could give proper score in general?\n- How did you determine theta_l for each layer?\n- The major problem of this paper is the experimental section. This paper only compared with outdated methods, so it is hardly verifying the effectiveness of the proposed method compared to other methods. \n- It is necessary to involve ResNets as one of the baselines.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thank you very much for your feedback to the reviewers, which helped us a lot to better understand your paper.
However, the paper is still premature to be accepted to ICLR2020. We hope that the detailed reviewers' comments help you improve your paper for potential future submission.

Decision: Reject