Title: ISPARSE: OUTPUT INFORMED SPARSIFICATION OF NEURAL NETWORKS

Abstract: Deep neural networks have demonstrated unprecedented success in various knowledge management applications. However, the networks created are often very complex, with large numbers of trainable edges which require extensive computational resources. We note that many successful networks nevertheless often contain large numbers of redundant edges. Moreover, many of these edges may have negligible contributions towards the overall network performance. In this paper, we propose a novel iSparse framework, and experimentally show, that we can sparsify the network, by 30-50%, without impacting the network performance. iSparse leverages a novel edge significance score, E, to determine the importance of an edge with respect to the final network output. Furthermore, iSparse can be applied both while training a model or on top of a pre-trained model, making it a retraining-free approach -leading to a minimal computational overhead. Comparisons of iSparse against PFEC, NISP, DropConnect, and Retraining-Free on benchmark datasets show that iSparse leads to effective network sparsifications.

1. INTRODUCTION:

Deep neural networks (DNNs), particularly convolutional neural networks (CNN), have shown impressive success in many applications, such as facial recognition (Lawrence et al., 1997) , time series analysis (Yang et al., 2015) , speech recognition (Hinton et al., 2012) , object classification (Liang & Hu, 2015) , and video surveillance (Karpathy & et. at., 2014) . As the term "deep" neural networks implies, this success often relies on large networks, with large number of trainable edges (weights) (Huang et al., 2017; Zoph et al., 2018; He et al., 2016; Simonyan & Zisserman, 2015) .

While a large number of trainable edges help generalize the network for complex and diverse patterns in large-scale datasets, this often comes with enormous computation cost to account for the non-linearity of the deep networks (ReLU, sigmoid, tanh) . In fact, DNNs owe their recent success to hardware level innovations that render the immense computational requirements practical (Ovtcharov & et. al., 2015; Matthieu Courbariaux et al., 2015) . However, the benefits of hardware solutions and optimizations that can be applied to a general purpose DNN or CNN are limited and these solutions are fast reaching their limits. This has lead to significant interest in networkspecific optimization techniques, such as network compression (Choi & et. al., 2018) , pruning (Li et al., 2016; Yu et al., 2018) , and regularization (Srivastava & et. al., 2014; Wan et al., 2013) , aim to reduce the number of edges in the network. However, many of these techniques require retraining the pruned network, leading to the significant amount of computational waste.

1.1. NETWORK SPARSIFICATION:

Many successful networks nevertheless often contain large numbers of redundant edges. Consider for example, the weights of sample network shown in Figure 1a . As we see here, the weight distribution is centered around zero and has significant number of weights with insignificant contribution to the network output. Such edges may add noise or non-informative information leading to reduction in the network performance. (Denil et al., 2013; Ashouri et al., 2018; Yu et al., 2018) has shown that it is possible to predict 95% network parameters while only learning 5% parameters.

Sparsification techniques can generally be classified into neuron/kernel sparcification (Li et al., 2016; Yu et al., 2018) and edge/weight sparcification techniques (Wan et al., 2013; Ashouri et al., Figure 1 : Overview of weight distribution and model accuracies for MNIST dataset (LeNet-5) 2018): (Li et al., 2016) proposed to eliminate neurons that have low l2-norms of their weights, whereas (Yu et al., 2018) proposed a neuron importance score propagation (NISP) technique where neuron importance scores (using Roffo & et. al. (2015) -See Equation 5) are propagated from the output layer to the input layer in a back-propagation fashion. Drop-out (Srivastava & et. al., 2014) technique instead deactivates neuron activations at random. As an edge sparsification technique, DropConnect (Wan et al., 2013) selects edges to be sparsified randomly. (Ashouri et al., 2018) showed that the network performance can be maintained by eliminating insignificant weights without modifying the network architecture.

1.2. OUR CONTRIBUTIONS: OUTPUT INFORMED EDGE SPARSIFICATION:

Following these works, we argue network sparsification can be a very effective tool for reducing sizes and complexities of DNNs and CNNs, without any significant loss in accuracy. However, we also argue that edge weights cannot be used "as is" for pruning the network. Instead, one needs to consider the significance of each edge within the context of their place in the network (Figure 2 ): "Two edges in a network with the same edge weight may have different degrees of contributions to the final network output" and in this paper, we show that it is possible to quantify significance of each edge in the network, relative to their contributions to the final network output and use these measures significance to minimize the redundancy in the network by sparsifying the weights that contributes insignificantly to network. We, therefore, propose a novel iSparse framework, and experimentally show, that we can sparsify the network, by almost 50%, without impacting the network performance.

The key contributions of our proposed work are as follows:

• Output-informed quantification of the significance of network parameters: Informed by the final layer network output, iSparse computes and propagates edge significant scores that measure the importance of each edge with respect to the model output (Section 3). • Retraining-free network sparsification (Sparsify-with): The proposed iSparse framework is robust to edge sparsification and can maintain network performance without having to retraining the network. This implies that one can apply iSparse on pre-trained networks, on-the-fly, to achieve the desired level of sparsification (Section 3.3) • Sparsification during training (Train-with): iSparse can also be used as a regularizer during the model training allowing for learning of sparse networks from scratch (Section 4).

As the sample results in Figure 1b shows, iSparse is able to achieve 30-50% sparsification with minimal impact on model accuracy. More detailed experimental comparisons (See Section 5) of iSparse against PFEC, NISP, Retraining-Free and DropConnect on benchmark datasets illustrated that iSparse leads to more effective network sparsifications.

2. RELATED WORKS:

A neural network is a sequence of layers of neurons to help learn (and remember) complex nonlinear patterns in a given dataset (Grossberg, 1988) . Recently, deep neural networks (DNNs), and particularly CNNs, which leverage recent hardware advances to increase the number of layers in the

!!!""""" !!!# $%&'( )'(&'( !!!# # *+ ,-. (/"* 0,-%,1 ,23 %2 +/" 0 % 3 % 4 !"#$ % &' ()$*+,-.*,)$#*)#$ - '"#$ % &' ()$*+,-.*,)$#*)#*/0#)10+%22#).*3.*

Figure 2 : Overview of iSparse sparsification, considering the n i 's contribution to overall output rather than only between n i and n j neurons network to scales that were not practical until recently, (Lawrence et al., 1997; Yang et al., 2015; Hinton et al., 2012; Liang & Hu, 2015; Karpathy & et. at., 2014) have shown impressive success in several data analysis and machine learning applications.

A typical CNN consists of ( 1) feature extraction layers are responsible for learning complex patterns in the data and remember them through layer weights. An m × n layer, L l , maps an n-dimensional input (Y l-1 ) to an m-dimensional output (Y l ), by training for a weight matrix W l R m l ×n l (see Section 3.1 for further details); (2) activation layers, which help capture non-linear patterns in the data through activation functions (σ) which maps the output from a feature extraction layer to a nonlinear space ( ReLU and softmax are commonly used activation functions); and (3) pooling layers, (sampling) which up-or down-sample the intermediate data in the network.

The training process of a neural network often comprises of two key stages: (1) forward-propagation (upstream) maps the input data, X, to an output variable, Ŷ . At each layer, we have The number of trainable parameters in a deep network can range from as low as tens of thousands (LeCun et al., 1999) to hundreds of millions (Simonyan & Zisserman, 2015) (Table 1 in Section 5).

Ŷl = L(Y l ) = σ(W l Y l +B l ), where Y 1 = X.

The three order increase in the number trainable parameter may lead to parameters being redundant or may have negligible contribution to the overall network output. This redundancy and insignificance of the network parameters has led to advancements in network regularization, by introducing dynamic or informed sparsification in the network.

These advancements can be broadly classified into two main categories: parameter pruning and parameter regularization. In particular, pruning focuses on compressing the network by eliminating the redundant or insignificant parameters. (Han et al., 2015; Han & et. al., 2016) aims to prune the parameters with near-zero weights inspired from l 1 and l 2 regularization (Tibshirani, 1996; Tikhonov, 1963) . (Li et al., 2016) choose to filter out convolutional kernel with minimum weight values in given layer. Recently, (Yu et al., 2018) minimizes the change in final network performance by eliminating the neuron that have minimal impact on the network output by leveraging neuron importance score (N L ) (See Section 5.3) -computed using Inf-FS (Roffo & et. al., 2015) . More complex approaches have been proposed to tackle the problem of redundancy in the network through weight quantization. (Rastegari & et. al., 2016) propose to the quantize the inputs and output activations of the layers in a CNN by using step function and also leveraging the binary operation by using the binary weights opposed to the real-values weights. (Chen & et. al., 2015) focuses on low-level mobile hardware with limited computational power, and proposed to leverage the inherent redundancy in the network for using hashing functions to compress the weights in the network. (Bahdanau et al., 2014; Woo et al., 2018) showed that the each input feature to a given layer in the network rarely have the same importance, therefore, learning there individual importance (attention) helps improve the performance of the network. More recently, (Garg & Candan, 2019) has shown that input data informed deep networks can provide high-performance network configurations. In this paper, we rely on output information for identifying and eliminating insignificant parameters from the network, without having to update the edge weights or retraining the network.

3. ISPARSE: OUTPUT INFORMED SPARSIFICATION OF NEURAL NETWORKS:

As discussed in Section 1, in order to tackle complex inputs, deep neural networks have gone increasingly deeper and wider. This design strategy, however, often results in large numbers of insignificant edges1 (weights), if not redundant. In this section, we describe the proposed iSparse, framework which quantifies the significance of each individual connection in the network with respect to the overall network output to determine the set of edges that can be sparsified to alleviate network redundancy and eliminate insignificant edges. iSparse aims to determine the significance of the edges in the network to make informed sparsification of the network.

3.1. MASK MATRIX:

A typical neural network, N , can be viewed as a sequential arrangement of convolutional (C) and fully connected (F) layers:

N (X) = L L (L L-1 . . . (L 2 ( L 1 (X))))

, here, X is the input, L is the total number of layers in the network and L {C, F} , s.t., any given layer,

EQUATION

where, Y l is the input to the layer (s.t. Y l = Ŷl-1 and for l = 1, Y 1 = X) and σ l , W l , and B l are the activation function, weight, and bias respectively. Note that, if the l th layer has m l neurons and the (l -1)

th layer has n l neurons , then Ŷl R m l ×1 , Y l R n l ×1 , W l R m l ×n l and B R m l ×1 .

Given this formulation, the problem of identifying insignificant edges can be formulated as the problem of generating a sequence of binary mask matrices, M 1 , . . . , M L , that collectively represents whether any given edge in the network is sparsified (0) or not (1):

M l = B n l ×m l , B {0, 1}, 1 ≤ l ≤ L (2)

3.2. EDGE SIGNIFICANCE SCORE:

Let M l be a mask matrix as defined in Equation 2, and M l can be expanded as

EQUATION

where each M l,i,j ∈ {0, 1} corresponds to an edge e l,i,j in the network. Our goal in the paper is to develop an edge significant score measure to help set the binary value of M l,i,j for each edge in the network. More specifically, we aim to associate a non-negative real valued number, E l,i,j ≥ 0, to each edge in the network, s.t.

EQUATION

Here, τ l (θ l ) represents the lowest significance of the θ l % of the most significant edges in the layer l.

Intuitively, given a target sparsification rate, θ l , we rank all the edges based on their edge significance scores and keep only the highest scoring θ l % of the edges by setting their mask values to 1.

As we have seen in Figure 1a , the (signed) weight distribution of the edges in a layer is often centered around zero, with large numbers of edges having weights very close to 0. As we also argued in the Introduction, such edges can work counter-intuitively and add noise or non-informative information leading to reduction in the network performance. In fact, several existing works, such as (Ashouri et al., 2018) , relies on these weights for eliminating insignificant edges without having to retrain the network architecture. However, as we also commented in the Introduction, we argue that edge weights should not be used alone for sparsifying the network. Instead, one needs to consider each edge within the context of their place in the network: Two edges in a network with the same edge weight may have different degrees of contributions to the final network output. Unlike existing works, iSparse takes this into account when selecting the edges to be sparsified (Figure 3 ). More specifically, let W + l be the absolute positive of the weight matrix,W l , for edges in l th layer. We compute the corresponding edge significance score matrix, E l , as

! ! ! "# ! "$ ! "# ! "% & ' " ( ) * & % " ( ) #"# & # ) & # ) '! (a) Sample Network (b) Retrain-free (c) iSparse

EQUATION

where, N l represents the neuron significance scores2 , N l,1 through N l,n l , and " " represents the scalar multiplication between edge weights and neuron scores. N l,i , denotes the significance of the i th input neuron to the l th connection layer of the network, which itself is defined recursively, based on the following layer in the network, using the conventional dot product:

EQUATION

Note that N l can be expanded as

EQUATION

Above, N L denotes the neuron scores of the final output layer, and N L is defined using infinite feature selection (Roffo & et. al., 2015; Yu et al., 2018) as N L = inf f s( ŶL ) where ŶL R x×n (x is the number of input samples and n is the number of output neurons) to determine neuron importance score with respect to the the final network output. Given the above, the edge score (Equation 5) can be rewritten as

E l = W + l W + l+1 (W + l+2 . . . (W + L-1 (W + L N L ))) .

(8) Note that the significance scores of edges in layer l considers not only the weights of the edges, but also the weights of all downstream edges between these edges and the final output layer.

3.3. EDGE SPARSIFICATION:

As noted in Section 3.1, the binary values in the masking matrix M l depends on τ l (θ l ), which represents the lowest significance of the θ l % of the most significant edges in the layer3 : therefore, given a target sparsification rate, θ l , for layer l, we rank all the edges based on their edge significance scores and keep only the highest scoring θ l % of the edges by setting their mask values to 1. Note that, once an edge is sparsified, change in its contribution is not propagated back to the layers earlier in the network relative to the sparsified edge. Having determined the insignificant edges with respect to the final layer output, represented in form of the mask matrix, M l (described in Section 3.1), the next step is to integrate this mask matrix in the layer itself. To achieve this, iSparse extends the layer l (Equation 1) to account for the corresponding mask matrix (M l ): where, * represents the element-wise multiplication between the matrices W l and M l . Intuitively, M l facilitates introduction of informed sparsity in the layer by eliminating edges that do not contribute significantly to the final output layer.

EQUATION

4. INTEGRATION OF ISPARSE WITHIN MODEL TRAINING:

In the previous section, we discussed the computation of edge significance scores on a pre-trained network, such as of pre-trained ImageNet models, and the use of these scores for network sparsification. In this section, we highlight that iSparse can also be integrated directly within the the training process.

To achieve this, the edge significance score is computed for every trainable layer in the network using the strategy described in Section 3.2 and the mask matrix is updated using Equation 4. Furthermore, the back-propagation rule, described in Section 2 , is updated to account for the mask matrices:

EQUATION

where, W l are the updated weights, W l original weights, η is the learning rate, and Err l is the error recorded by as the divergence in between ground truth (Y l ) and model predictions ( Ŷl ) as

Err l = |Y l -Ŷl |.

Note that, we argue that any edge that does not contribute towards the final model output, must not be included in the back-propagation. Therefore, we mask the error as Err l * M l .

5. EXPERIMENTAL EVALUATION:

In this section, we experimentally evaluate of the proposed iSparse framework using LeNet and VGG architectures (See Section 5.2) and compare it against the approaches, such as PFEC, NISP, and DropConnect (see Section 5.3).

5.1. SYSTEM CONFIGURATION:

We implemented iSparse in Python environment (3.5.2) using Keras Deep Learning Library (2.2.4tf) (Chollet et al., 2015) with TensorFlow Backend (1.14.0). All experiments were performed on an Intel Xeon E5-2670 2.3 GHz Quad-Core Processor with 32GB RAM equipped with Nvidia Tesla P100 GPU with 16 GiB GDDR5 RAM with CUDA-10.0 and cuDNN v7.6.4.4 .

5.2. BENCHMARK NETWORKS AND DATASETS:

In this paper, without loss of generality, we leverage LeNet-5 (LeCun et al., 1999) and VGG-16 (Simonyan & Zisserman, 2015) as the baseline architectures to evaluate sparsification performance on different benchmark image classification datasets and for varying degrees of edge sparsification.

In this section, we present an overview of these architectures (See Table 1 ).

LeNet-5: Designed for recognizing handwritten digits, LeNet-5 is simple network with 5 trainable (2 convolution and 3 dense) and 2 non-trainable layers using average pooling with tanh and softmax as the hidden and output activation. LeNet's simplicity has made it a common benchmark for datasets recorded in constrained environments, such as MNIST (LeCun et al., 1998) , FMNIST (Xiao et al., 2017) (b) LeNet network (LeNet-5)

Figure 6 : Top-1 classification accuracy results for sparsified pre-trained models (sparsify-with)

VGG-16: VGG (Simonyan & Zisserman, 2015)'s, a 16 layer network with 13 convolution and 3 dense layers, with interleaved 5 max-pooling layers. VGG leverages ReLU as the hidden activation to overcome the problem of vanishing gradient, as opposed to tanh. Given the ability of VGG network to learn the complex pattern in the real-world dataset, we use the network on benchmark datasets, such as CIFAR10/20/100 (Krizhevsky, 2009) , SVHN (Netzer & et. al., 2011) , GTSRB (Stallkamp & et. al., 2012) , and ImageNet (Deng et. al., 2009) . Table 1 reports the number of trainable parameters (or weights) for each model/data set pair considered in the experiments.

5.3. COMPETITORS:

We compared iSparse against several state-of-the-art network sparsification techniques: DropConnect (Wan et al., 2013) is a purely random approach, where edges are randomly selected for sparsification. Retraining-free (Ashouri et al., 2018) considers each layer independently and sparsifies insignificant weights in the layer, without accounting for the final network output contribution. PFEC (Li et al., 2016) is a kernel pruning strategy that aims to eliminate neurons that have low impact on the overall model accuracy. In order to determine the impact, PFEC computes the l2-norms of the weights of the neurons and ranks them, separately, for each layer. NISP (Yu et al., 2018) proposes a neuron importance score propagation (NISP) technique where neuron importance scores are propagated from the output layer to the input layer in a back-propagation fashion. 

5.4.1. SPARSIFICATION OF PRE-TRAINED MODELS (sparsify-with):

In Figure 4 , we first present top-1 and top-5 classification results for ImageNet dataset for VGG-16 network. As we see in the Figure 4 , iSparse provides the highest robustness to the degree of sparsification in the network. In particular, with iSparse , the network can be sparsified by 50% with ≤ 6% drop in accuracy for top-1 and ≤ 2% drop in accuracy for top-5 classification, respectively. In contrast, the competitors, see larger drops in accuracy. The closest competitor, Retrain-free, suffers a loss in accuracy of ∼ 16% and ∼ 6% for top-1 and top-5 classification, respectively. The other competitors suffer significant accuracy drops after a mere 10-20% sparsification.

Figures 6a and 6b show the top-1 classification accuracy results for other models and data sets. As we see here, the above pattern holds for all configurations considered: iSparse provides the best robustness. It is interesting to note that DropConnect, NISP, and PFEC see especially drastic drops in accuracy for the VGG-16 network and especially on the CIFAR data. This is likely because, VGG-CIFAR is already relatively sparse (20% > sparsity as opposed to ∼ 7% for VGG-ImageNet and < 1% for LeNet) and these three techniques are not able to introduce additional sparseness in a robust manner. In contrast, iSparse is able to introduce significant additional sparsification with minimal impact on accuracy.

Figure 7 provides the mask matrices created by the different algorithms to visual illustrate the key differences between the competitors. As we see in this figure, PFEC and NISP, both sparsify input neurons. Consequently, their effect is to mask out entire columns from the weight matrix and this prevents these algorithms to provide fine grain adaption during sparsification. DropConnect selects individual edges for sparsification, but only randomly and this prevents the algorithm to provide sufficiently high robustness. Retrain-free and iSparse both select edges in an fine-grained manner: retrain-free uses relies on edge-weights, whereas iSparse complements edge-weight with an edge significance measure that accounts for each edges contribution to the final output within the overall network. As we see in Figure 7 (d) and (e), this results in some differences in the corresponding mask matrices, and these differences are sufficient to provide significant boost in accuracy. Tables 2 present accuracy results for the scenarios where iSparse (iS) is used to sparsify the model during the training process. The table also considers DropConnect (DC) and Retrain-Free (RF), as alternatives. As we see in the table, for both network architectures, under most sparsification rates, the output informed sparsification approach underlying iSparse leads to networks with the highest classification accuracies.

5.5. ROBUSTNESS TO THE VARIATIONS IN NETWORK ELEMENTS:

In this section, we study the effect of the variations in network elements. In particular, we compare the performance of iSparse (iS) against DropConnect (DC) and Retraining-Free (RF) for different hidden activation functions and network optimizers. Table 3 presents classification performances for networks that rely on different activation functions (tanh and ReLU) and for optimizers (Adam and RMSProp). As we see in these two tables, iSparse remains the alternative which provides the best classification accuracy under different activation/optimization configurations.

5.6. ROBUSTNESS TO THE VARIATIONS IN SPARSIFICATION ORDER:

We next investigate the performance of iSparseunder different orders in which the network layers are sparsified. In particular, we considerthree sparsification orders: (a) input-to-output layer order: this is the most intuitive approach as it does not require edge significance scores to be revised based on sparsified edges in layers closer to the input; (b) output-to-input layer-order: in this case, edges in layers closer to the network output are sparsified first -but, this implies that edge significance scores are updated in the earlier layers in the network to account for changes in the overall edge contributions to the network; (c) random layer order: in this case, to order of the layers to be sparsified is selected randomly. Figure 8 presents the sparsification results for different orders, data sets, and sparsification rates. As we see in the figure, the performance of iSparse is not sensitive to the sparsification order of the network layers.

5.7. IMPACT OF SPARSIFICATION ON CLASSIFICATION TIME:

In Figure 5 , we investigate the impact of edge sparsification on the classification time. As we see in this Figure, edge sparsification rate has a direct impact on the classification time of the resulting model. When we consider that iSparse allows for ∼ 30 -50% edge sparsification without any major impact on classification accuracies, this indicates that iSparse has the potential to provide significant performance gains. What is especially interesting to note in Figure 5 is that, while all three sparsification methods, iSparse, DropConnect, and Retraining-Free, all have the same number of sparsified edges for a given sparsification factor, the proposed iSparse approach leads to the least execution times among the three alternatives. We argue that this is because the output informed sparsification provided by iSparse allows for more efficient computations in the sparsified space5 .

6. CONCLUSIONS:

In this paper, we proposed iSparse, a novel output-informed, framework for edge sparsification in deep neural networks (DNNs). In particular, we propose a novel edge significance score that quantifies the significance of each edge in the network relative to its contribution to the final network output. iSparse leverages this edge significance score to minimize the redundancy in the network by sparsifying those edges that contribute least to the final network output. Experiments, with 11 benchmark datasets and using two well-know network architectures have shown that the proposed iSparse framework enables 30 -50% network sparsification with minimal impact on the model classification accuracy. Experiments have also shown that the iSparse is highly robust to variations in network elements (activation and model optimization functions) and that iSparse provides a much better accuracy/classification-time trade-off against competitors. In this section, we evaluate the performance on iSparse framework with different type of neuron score initialization strategies). We evaluate the following scoring strategies:

-Identity score (IDN) (all scores are "1") -Principal Component Analysis (PCA) (Pearson (1901) )

-Infinite Feature Selection (InfFS) (Roffo & et. al. (2015) )

In Table 4 , we observed that the iSparse framework with InfFS as a mechanism to determine the neuron scores for the final layers leads to a better performance opposed to the other strategies. Intuitively, this is because InfFS effectively identifies the importance of an individual features (neurons), while considering all possible subset of features. We believe this is a very highly desirable property: conventional feature selection strategies, such as PCA, ranks features based on their discriminatory power irrespective of how features relate with each other. Similarly, using same importance score (identity) treats each feature in a mutually exclusive fashion. In contrast, the way InfFS approaches the "feature selection" problem is highly appropriate to NNs, where the neuron output for the true label will be high, but we will also see low/small output for false labels.

A.2 PERFORMANCE EVALUATION FOR THE RESNET ARCHITECTURE:

In this section, we evaluate iSparse for the ResNet-18 architecture (He et al. (2016) ) for both, sparsify-with (no retraining after sparsification) and train-with (retraining after sparsification) configurations Results are presented in Tables 5 and 6 .

A.2.1 SPARSIFY-WITH FOR RESNET:

As observed in Table 5 , iSparse for ResNet demonstrates significant robustness to edge sparsification. iSparse outperforms DropConnect, and Retrain-Free, thus highlighting the importance of accounting edge's significance to subsequent network along with edge weight. .

A.2.2 TRAIN-WITH FOR RESNET:

As we can see in the Table 6 , the iSparse framework outperforms the competitors, such as Dropout, L1, DropConnect, Retraining-Free, and Lottery Winning Hypothesis (Frankle & Carbin (2019) ). iSparse ability to account for edge significance based on both, edge weights and edge contributions to the final network output, leads to more robust and superior results. 

:

An edge is defined as a direct connection (weighted) between two neurons.

N l summarizes the edge and neuron importance in the subsequent layers i.e. L l+1 . . . LL.

In the experiments reported in Section 5, without loss of generality, we assume that θ l has the same value for all connection layers in the network. This is not a fundamental assumption and iSparse can easily accommodate different rates of sparsification across connection layers.

Results presented in this paper were obtained using NSF testbed: "Chameleon: A Large-Scale Reconfigurable Experimental Environment for Cloud Research (https://www.chameleoncloud.org/)"

Hardware configuration can be found in Section 5.1

Figures and Tables:

Intuitively, each layer learns to extract new features from the features extracted by the prior layer. (2) backward-propagation (downstream) revises network weights, W , based on the training error, Err = |Y -Ŷ |, where Y and Ŷ are the ground truth and predictions of the model, respectively, such that update weights are defined as W = W -Err.

Figure 3: A sample network architecture and its sparsification using Retraining-free (Ashouri et al., 2018) and iSparse; here node labels indicate input to the node; edge labels [0,1] indicate the edge weights; and edge labels between parentheses indicate edge contribution

Figure 7: Mask matrices for the LeNet network conv 2 layer for MNIST data (sparsification factor = 50%): dark regions indicate the edges that have been marked for sparsification; in (e) iSparse, the arrows point to those edges that are subject to different pruning decision from retrain-free in(d) (green arrows point to edges that are kept in iSparse instead of being pruned and red arrows point to edges that are sparsified in iSparse instead of being kept)

Figure 8: Robustness to the layer order while sparsifying the network with iSparse (train-with)

Number of trainable parameters (weights) and statistics for various benchmark datasets

Table content: <table><tr><td>Network</td><td/><td/><td>LeNet</td><td/><td/><td/><td/><td/><td>VGG</td><td/><td/></tr><tr><td>Datasets</td><td colspan="5">MNIST FMNIST COIL20 COIL100 NORB</td><td>CIFAR10</td><td>CIFAR20</td><td colspan="2">CIFAR100 SVHN</td><td>GTSRB</td><td>ImageNet</td></tr><tr><td>Weights</td><td>44,426</td><td>44,426</td><td>62,556</td><td>69,356</td><td colspan="7">168,801 32,418,834 32,428,844 32,508,924 32,418,834 38,743,323 138,357,544</td></tr><tr><td/><td/><td/><td/><td/><td/><td>Statistics</td><td/><td/><td/><td/><td/></tr><tr><td>Resolution</td><td colspan="2">28 × 28</td><td colspan="2">32 × 32 32 × 32</td><td>96 × 96</td><td>32 × 32</td><td>32 × 32</td><td>32 × 32</td><td>32 × 32</td><td>32 × 32</td><td>224 × 224</td></tr><tr><td>Train Set</td><td>60,000</td><td>60,000</td><td>1300</td><td>6480</td><td>24,300</td><td>50,000</td><td>50,000</td><td>50,000</td><td>∼73,000</td><td>39,209</td><td>∼1,281,167</td></tr><tr><td>Test Set</td><td>10,000</td><td>10,000</td><td>140</td><td>720</td><td>24,300</td><td>10,000</td><td>10,000</td><td>10,000</td><td>∼26,000</td><td>12,630</td><td>∼50,000</td></tr><tr><td>Labels</td><td>10</td><td>14</td><td>20</td><td>100</td><td>6</td><td>10</td><td>20</td><td>100</td><td>10</td><td>43</td><td>1000</td></tr></table>

Top-1 classification accuracy for sparsified different architectures and datasets (train-with) 0%)98.79 98.79 98.79 88.02 88.02 88.02 95.52 95.56 95.56 90.00 90.00 90.00 84.42 84.42 84.42  5% 98.90 97.99 98.09 88.23 88.70 88.75 95.00 94.16 94.72 89.83 90.12 90.61 84.51 84.75 85.03  10% 98.17 98.45 98.56 87.60 87.02 87.89 94.72 95.01 95.55 90.05 89.81 90.33 85.35 85.98 86.25  15% 98.76 98.79 98.92 88.14 87.43 87.96  95.00 95.21 95.50 89.50 89.73 90.00 83.91 84.12 84.73 20% 98.79 98.81 98.98 88.02 88.00 88.11 95.83 95.95 96.10 90.83 88.54 91.03 83.04 82.56 83.29 25% 98.85 98.85 98.80 87.62 87.54 88.19 95.27 94.66 95.80 90.11 89.98 90.22 85.33 85.01 85.61 30% 98.78 98.71 98.98 88.00 87.75 88.25 95.00 94.74 95.00 90.12 90.12 90.55 83.69 85.91 86.08 35% 98.73 98.73 98.79 88.14 86.99 88.06 94.72 94.99 95.27 90.11 91.35 91.99 83.98 83.01 84.81 40% 98.85 98.97 98.99 87.81 87.25 88.30 95.00 95.31 95.56 88.22 90.01 90.44 83.14 84.12 85.08 45% 98.91 98.95 98.99 87.45 87.11 87.82 95.29 95.04 95.56 89.45 89.87 90.23 85.47 85.12 86.02 50% 98.55 98.85 99.00 87.76 88.15 88.46 95.00 94.16 94.72 90.66 90.21 90.85 84.49 85.61 86.77 0%) 84.14 84.14 84.14 66.96 66.96 66.96 55.09 55.09 55.09 91.33 91.33 91.33 97.68 97.68 97.68 5% 84.21 84.56 84.66 62.26 62.01 62.85 57.35 52.74 53.10 93.49 92.42 92.70 96.49 97.68 98.64 10% 84.35 84.33 84.35 63.63 66.53 67.33 51.32 51.01 51.48 95.00 92.56 92.96 95.03 91.09 93.24 15% 83.43 83.51 83.69 60.95 59.87 62.49 52.18 52.53 52.97 91.96 93.11 93.40 96.83 95.91 97.29 20% 82.83 85.04 85.81 61.79 63.45 65.03 51.67 50.01 50.52 92.63 93.34 93.74 93.02 94.41 95.76 25% 84.07 84.32 84.56 61.85 60.19 62.52 56.47 53.25 53.81 82.56 87.14 87.63 95.11 96.66 97.76 30% 83.41 84.17 84.48 62.14 58.39 62.14 51.19 54.05 54.50 91.95 91.97 92.26 97.18 97.43 98.56 35% 83.06 84.95 85.42 60.09 57.81 61.79 52.05 48.52 49.03 92.58 92.38 92.65 97.18 97.61 98.17 40% 83.96 82.01 82.21 66.34 65.09 66.62 50.19 55.11 55.68 88.79 91.92 92.31 95.61 96.91 97.6145% 82.75 82.81 82.83 65.65 66.93 68.72 54.18 53.47 53.63 91.38 91.23 91.62 96.71 93.31 94.02  50% 83.44 84.15 84.53 65.37 65.04 68.82 51.65 51.78 52.56 82.94 87.61 87.68 94.31 95.91 97.31

Table content: <table><tr><td>Datasets</td><td/><td>MNIST</td><td/><td/><td>FMNIST</td><td/><td colspan="2">COIL20</td><td/><td/><td>COIL100</td><td/><td/><td>NORB</td><td/></tr><tr><td>Factor</td><td>DC</td><td>RT</td><td>IS</td><td>DC</td><td>RT</td><td>iS</td><td>DC</td><td>RT</td><td>iS</td><td>DC</td><td>RT</td><td>iS</td><td>DC</td><td>RT</td><td>iS</td></tr><tr><td colspan="8">Base (VGG-16</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>Datasets</td><td/><td>CIFAR10</td><td/><td/><td>CIFAR20</td><td/><td colspan="2">CIFAR100</td><td/><td/><td>SVHN</td><td/><td/><td>GTSRB</td><td/></tr><tr><td>Factor</td><td>DC</td><td>RT</td><td>IS</td><td>DC</td><td>RT</td><td>iS</td><td>DC</td><td>RT</td><td>iS</td><td>DC</td><td>RT</td><td>iS</td><td>DC</td><td>RT</td><td>iS</td></tr><tr><td>Base (</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

Robustness analysis for edge-based strategies vs iSparse (train-with)

Table content: <table><tr><td/><td/><td>Tan-Adam</td><td/><td colspan="2">Tanh-RMS</td><td/><td/><td>ReLU-Adam</td><td/></tr><tr><td/><td>DC</td><td>RT</td><td>iS</td><td>DC</td><td>RT</td><td>iS</td><td>DC</td><td>RT</td><td>iS</td></tr><tr><td/><td/><td/><td/><td>MNIST</td><td/><td/><td/><td/><td/></tr><tr><td>Base (0%)</td><td>98.82</td><td>98.82</td><td>98.82</td><td>98.79</td><td>98.79</td><td>98.79</td><td>98.89</td><td>98.89</td><td>98.89</td></tr><tr><td>5%</td><td>98.90</td><td>98.23</td><td>98.94</td><td>98.90</td><td>97.99</td><td>98.09</td><td>98.66</td><td>98.91</td><td>98.95</td></tr><tr><td>10%</td><td>99.02</td><td>98.75</td><td>99.15</td><td>98.17</td><td>98.45</td><td>98.56</td><td>98.90</td><td>98.75</td><td>98.86</td></tr><tr><td>15%</td><td>98.94</td><td>98.01</td><td>98.83</td><td>98.76</td><td>98.79</td><td>98.92</td><td>98.94</td><td>99.01</td><td>99.23</td></tr><tr><td>20%</td><td>98.76</td><td>98.41</td><td>98.97</td><td>98.79</td><td>98.81</td><td>98.86</td><td>99.05</td><td>99.32</td><td>99.53</td></tr><tr><td>25%</td><td>98.91</td><td>98.25</td><td>98.92</td><td>98.85</td><td>98.78</td><td>98.80</td><td>98.89</td><td>98.85</td><td>98.96</td></tr><tr><td>30%</td><td>99.01</td><td>98.04</td><td>98.92</td><td>98.78</td><td>98.71</td><td>98.78</td><td>98.88</td><td>98.74</td><td>98.94</td></tr><tr><td>35%</td><td>98.94</td><td>98.65</td><td>99.05</td><td>98.73</td><td>98.75</td><td>98.79</td><td>98.86</td><td>98.62</td><td>98.92</td></tr><tr><td>40%</td><td>99.12</td><td>98.88</td><td>99.57</td><td>98.85</td><td>98.97</td><td>98.99</td><td>98.83</td><td>98.23</td><td>98.95</td></tr><tr><td>45%</td><td>98.84</td><td>98.42</td><td>99.04</td><td>98.91</td><td>98.95</td><td>98.99</td><td>99.07</td><td>99.01</td><td>99.48</td></tr><tr><td>50%</td><td>99.07</td><td>98.01</td><td>99.12</td><td>98.55</td><td>98.85</td><td>99.00</td><td>98.98</td><td>98.16</td><td>98.89</td></tr><tr><td/><td/><td/><td/><td>CIFAR10</td><td/><td/><td/><td/><td/></tr><tr><td>Base (%)</td><td>87.66</td><td>87.66</td><td>87.66</td><td>81.95</td><td>81.95</td><td>81.95</td><td>84.14</td><td>84.14</td><td>84.14</td></tr><tr><td>5%</td><td>88.87</td><td>89.56</td><td>89.66</td><td>81.17</td><td>85.94</td><td>86.14</td><td>84.21</td><td>84.56</td><td>84.66</td></tr><tr><td>10%</td><td>84.75</td><td>88.21</td><td>88.31</td><td>86.75</td><td>87.01</td><td>87.42</td><td>84.35</td><td>84.33</td><td>84.35</td></tr><tr><td>15%</td><td>81.78</td><td>81.89</td><td>82.49</td><td>88.75</td><td>88.99</td><td>89.04</td><td>83.43</td><td>83.51</td><td>83.69</td></tr><tr><td>20%</td><td>85.64</td><td>85.95</td><td>86.21</td><td>87.10</td><td>89.10</td><td>89.23</td><td>82.83</td><td>85.04</td><td>85.81</td></tr><tr><td>25%</td><td>86.10</td><td>87.68</td><td>87.86</td><td>84.81</td><td>87.31</td><td>87.74</td><td>84.07</td><td>84.32</td><td>84.56</td></tr><tr><td>30%</td><td>87.07</td><td>86.21</td><td>86.30</td><td>80.20</td><td>83.45</td><td>83.56</td><td>83.41</td><td>84.17</td><td>84.48</td></tr><tr><td>35%</td><td>82.38</td><td>86.43</td><td>86.48</td><td>85.91</td><td>86.75</td><td>86.93</td><td>83.06</td><td>84.95</td><td>85.42</td></tr><tr><td>40%</td><td>86.76</td><td>88.63</td><td>88.77</td><td>82.17</td><td>84.10</td><td>87.44</td><td>83.96</td><td>82.01</td><td>82.21</td></tr><tr><td>45%</td><td>83.90</td><td>85.52</td><td>85.61</td><td>85.79</td><td>86.40</td><td>89.42</td><td>82.75</td><td>82.81</td><td>82.83</td></tr><tr><td>50%</td><td>80.45</td><td>79.21</td><td>79.59</td><td>83.89</td><td>86.42</td><td>86.52</td><td>83.44</td><td>84.15</td><td>84.53</td></tr><tr><td colspan="6">5.4.2 SPARSIFICATION DURING TRAINING (train-with)</td><td/><td/><td/><td/></tr></table>

Model accuracies under different strategies for initializing neuron importance score -CI-FAR10 dataset -for different sparsification factors; Base = 0%

Table content: <table><tr><td/><td>VGG</td><td/><td>ResNet</td></tr><tr><td>Factor IDN</td><td colspan="2">PCA InfFS IDN</td><td>PCA InfFS</td></tr><tr><td colspan="2">Base (0%) 84.14 84.14 84.14</td><td colspan="2">73.61 73.61 73.61</td></tr><tr><td colspan="2">5% 82.94 83.96 84.66</td><td colspan="2">73.25 73.20 74.13</td></tr><tr><td colspan="2">10% 81.59 83.85 84.35</td><td colspan="2">73.46 73.86 74.28</td></tr><tr><td colspan="2">15% 81.24 83.45 83.69</td><td colspan="2">73.32 73.57 74.12</td></tr><tr><td colspan="2">20% 82.67 84.65 85.81</td><td colspan="2">74.52 74.21 75.01</td></tr><tr><td colspan="2">25% 81.42 83.98 84.56</td><td colspan="2">74.86 74.45 74.95</td></tr><tr><td colspan="2">30% 80.64 82.96 84.48</td><td colspan="2">73.72 73.80 74.36</td></tr><tr><td colspan="2">35% 80.45 81.56 85.42</td><td colspan="2">73.53 73.99 74.01</td></tr><tr><td colspan="2">40% 78.48 79.54 82.21</td><td colspan="2">74.94 75.09 75.27</td></tr><tr><td colspan="2">45% 80.23 81.24 82.83</td><td colspan="2">73.06 72.39 73.39</td></tr><tr><td colspan="2">50% 80.85 82.68 84.53</td><td colspan="2">74.19 74.19 74.21</td></tr><tr><td>A APPENDIX</td><td/><td/></tr><tr><td colspan="4">A.1 ABLATION STUDIES FOR NEURON SCORE INITIALIZATION</td></tr></table>

Model accuracy vs sparsification factor (train-with) for VGG architecture -CIFAR10

Table content: <table><tr><td colspan="2">Factor Dropout L1</td><td colspan="4">DropConnect Retraining-Free Lottery iSparse</td></tr><tr><td>Base (0%) 84.14</td><td colspan="2">84.51 84.14</td><td>84.14</td><td>84.14</td><td>84.14</td></tr><tr><td>5% 85.13</td><td>N.A.</td><td>84.21</td><td>84.56</td><td>84.29</td><td>84.66</td></tr><tr><td>10% 84.77</td><td>N.A.</td><td>84.35</td><td>84.33</td><td>84.23</td><td>84.35</td></tr><tr><td>15% 86.81</td><td>N.A.</td><td>83.43</td><td>83.51</td><td>84.88</td><td>83.69</td></tr><tr><td>20% 77.81</td><td>N.A.</td><td>82.83</td><td>85.04</td><td>84.68</td><td>85.81</td></tr><tr><td>25% 73.42</td><td>N.A.</td><td>84.07</td><td>84.32</td><td>84.87</td><td>84.56</td></tr><tr><td>30% 76.16</td><td>N.A.</td><td>83.41</td><td>84.17</td><td>83.48</td><td>84.48</td></tr><tr><td>35% 79.28</td><td>N.A.</td><td>83.06</td><td>84.95</td><td>84.79</td><td>85.42</td></tr><tr><td>40% 77.51</td><td>N.A.</td><td>83.96</td><td>82.01</td><td>81.37</td><td>82.21</td></tr><tr><td>45% 73.51</td><td>N.A.</td><td>83.96</td><td>82.01</td><td>81.37</td><td>82.21</td></tr><tr><td>50% 69.21</td><td>N.A.</td><td>83.44</td><td>84.15</td><td>81.59</td><td>84.53</td></tr></table>

Peer review (iclr2020):
The paper proposed a method for network sparsification based on the significance of each edge (weight). Unlike some of the existing works, edge significance in this work is explicitly defined based on their influence over the network output. The algorithm can be summarized as follows: 1. Compute the significance of each activation (neuron) using an existing method (infinite feature selection). 2. Compute the significance of each edge as the product between the neuron significance and the absolute edge weight. 3. Sort the edges according to their significance scores and keep the top portion. I*m mainly concerned about the limited technical novelty: the proposed technique is essentially a heuristic. Specifically, it is not clear why the significance of each edge should be defined according to equation (5) rather than some other forms. The intuition behind masking out the gradients of in-significant edges in (10), i.e., *we argue that any edge that does not contribute towards the final model output, must not be included in the back-propagation*, is again a heuristic that lacks justification. If theoretical analysis is not possible, it might be necessary for the authors to conduct controlled experiments/ablation studies to show that some of the design choices made in the paper are indeed superior over other alternatives. Another of my concern is that, if the goal is to make the sparsification decision aware of the network output (e.g., the value of the loss function), a simpler approach would be to enforce L1 regularization over the edges. This way, edges that do not lead to significant impact on the loss would be automatically pruned away. I wonder how would the proposed approach compare against this simple baseline. Other suggestions: * Empirical evaluation is conducted using LeNet and VGG16. It would be interesting to extend the analysis to some other seminal architectures, such as ResNet, Inception and MobileNets. * It would be informative to report the hardware configuration used to obtain the execution time in Figure 5. Note the relative inference cost of different models may differ substantially over different hardware platforms. * Writing of the paper can probably be polished further for better clarity.