Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a sparsification technique that seeks edges contributing negligible amounts to the performance of a network. 

pros)
(+) This paper is written well but needs more clarity.

cons)
(-) This paper did not cite modern pruning/sparsification methods published recently.
(-) The proposed method has only compared with some outdated methods.
(-) Only LeNet-5 and VGG-16 have been used to validate the proposed method.
(-) This paper lacks any analysis of why the proposed methods would work well. Specifically, on the formulation of the sparsification method, it is hard to find sufficient backups why the authors did like that.

Comments)
- How would you guarantee using the infinite feature selection method could give proper score in general?
- How did you determine theta_l for each layer?
- The major problem of this paper is the experimental section. This paper only compared with outdated methods, so it is hardly verifying the effectiveness of the proposed method compared to other methods. 
- It is necessary to involve ResNets as one of the baselines.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This work proposes iSparse framework, which aims to sparsify a neural network by removing redundant edges. Unlike previous works where the edges were removed based on their weight value, or based on the relationship between input and output neurons, iSparse selects the edges to remove by computing the contribution of each edge with respect to the final outcome. The experiments show that, compared to several baseline methods, iSparse perform favorably on multiple datasets. 

Comment:
Although I am not an expert in network pruning or network sparsification, I know that the Lottery Ticket Hypothesis (Frankle & Carbin, 2019) were able to remove at most 80% of the weights of neural networks (both fully-connected and ConvNets) and still retain the original performance level. Compared to that, iSparse's performance does not seem too impressive. 
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: The paper proposed a method for network sparsification based on the significance of each edge (weight). Unlike some of the existing works, edge significance in this work is explicitly defined based on their influence over the network output.

The algorithm can be summarized as follows:
1. Compute the significance of each activation (neuron) using an existing method (infinite feature selection).
2. Compute the significance of each edge as the product between the neuron significance and the absolute edge weight.
3. Sort the edges according to their significance scores and keep the top portion.

I'm mainly concerned about the limited technical novelty: the proposed technique is essentially a heuristic. Specifically, it is not clear why the significance of each edge should be defined according to equation (5) rather than some other forms. The intuition behind masking out the gradients of in-significant edges in (10), i.e., "we argue that any edge that does not contribute towards the final model output, must not be included in the back-propagation", is again a heuristic that lacks justification. If theoretical analysis is not possible, it might be necessary for the authors to conduct controlled experiments/ablation studies to show that some of the design choices made in the paper are indeed superior over other alternatives.

Another of my concern is that, if the goal is to make the sparsification decision aware of the network output (e.g., the value of the loss function), a simpler approach would be to enforce L1 regularization over the edges. This way, edges that do not lead to significant impact on the loss would be automatically pruned away. I wonder how would the proposed approach compare against this simple baseline.

Other suggestions:
* Empirical evaluation is conducted using LeNet and VGG16. It would be interesting to extend the analysis to some other seminal architectures, such as ResNet, Inception and MobileNets.
* It would be informative to report the hardware configuration used to obtain the execution time in Figure 5. Note the relative inference cost of different models may differ substantially over different hardware platforms.
* Writing of the paper can probably be polished further for better clarity. 
review_title: Official Blind Review #2

Meta-review: Thank you very much for your feedback to the reviewers, which helped us a lot to better understand your paper.
However, the paper is still premature to be accepted to ICLR2020. We hope that the detailed reviewers' comments help you improve your paper for potential future submission.

Decision: Reject