Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: [Summary]:
This paper interprets the underlying objective of magnitude pruning(MP) as minimizing the Frobenius distortion of a single layer. Then the authors provide a motivating example to show that MP may cause a large Frobenius distortion due to ignoring the inter-layer interactions. Based on this observation, the authors propose a simple modification to MP by explicitly enforcing to minimize the Frobenius distortion of an operator block consisting of multiple linear layers, and demonstrate better performance than MP on CIFAR10 and MNIST datasets.

[Pros]:
- The proposed algorithm is simple and easy to implement.
- The empirical results show that the proposed method beat MP consistently, and in particular for high sparsities.
- The ablation study about LAP, LFP and LBP is interesting.

[Cons & Questions]:
(1) Minimizing Frobenius distortion is not meaningful, and it only minimizes the change in the output to the first order. Moreover, I don’t think minimizing the change in the output is as meaningful as minimizing the increase in training error as is done in Hessian-based pruning methods, e.g., Optimal Brain Damage (OBD) ( LeCun et al. 1989). My reason is that it is possible that the output changes a lot, but the training error still remains low after pruning.

(2) Can the authors elaborate what are the advantages of MP/LAP over Hessian-based pruning, such as OBD? OBD only needs the diagonal Hessian matrix and is also tractable, and MP is only a special case of OBD when the Hessian is an identity matrix. I am not quite convinced MP can achieve state-of-the-art performance, and also Gale et al. (2019) did not include any Hessian-based pruning algorithm into comparisons. Therefore, it would be great if the authors can provide more justifications for why MP/LAP is advantageous to Hessian-based pruning methods, e.g., OBD. Besides, I would be happy to see the authors can include OBD as a baseline in the experiments.

(3) The interpretation of the objective of MP as minimizing the Frobenius distortion is well-known, and more general results are already presented in Dong et al. (2017). The authors should discuss it in the main paragraph and give the corresponding credits to Dong et al. (2017).

(4) Why do you need to specify the pruning ratio for each layer manually? It makes MP or LAP hard to use in practice, and it usually needs expert knowledge to specify the pruning ratios for different layers. For Hessian-based methods, it reflects the change in loss and thus can be used to automatically determine the pruning ratio at each layer. 

(5) In table 1 and table 2, LFP is better than LBP when pruning ratio is low, while LBP becomes better for high pruning ratios. Is there any explanations?

(6) My understanding of the LAP is that it tries to minimize the Frobenius distortion of the input-output Jacobian matrix of the operator block. In the paper, the operator block consists of 3 consecutive linear layers. I am curious about what is the performance if we treat the entire network as an operator block?

(7) The experiments are only conducted on MNIST and CIFAR-10, which are overly simple. Further experiments on larger datasets will make the paper stronger and the results more convincing. Anyway, this is not a big issue, but I encourage the authors can test the proposed method on more challenging datasets and make fair comparisons.

Overall, my rating is largely due to the concerns of (1)&(2).

[References]:
Y. LeCun, J. S. Denker, and S. A. Solla. Optimal brain damage. In Advances in Neural Information Processing Systems, 1989.
T. Gale, E. Elsen, and S. Hooker. The state of sparsity in deep neural networks. arXiv preprint 1902.09574, 2019.
Dong, Xin, Shangyu Chen, and Sinno Pan. "Learning to prune deep neural networks via layer-wise optimal brain surgeon." Advances in Neural Information Processing Systems. 2017.

review_title: Official Blind Review #5

final_score: rating: 6: Weak Accept
review_content: *Summary*
The paper proposes a multi-layer alternative to magnitude-based pruning. The operations entailed in the previous, current, and subsequent layers are treated as linear operations (by omitting any nonlinearities), weights are selected for pruning to minimize the "Frobenius distortion", the Frobenius norm of the difference between products of the (i-1, i, i+1)-layer Jacobians with and without the selected weight. This simplifies to a cost-effective pruning criterion. In spite of the simplistic linear setting assumed for the derivation, results show the criterion prunes better than weight-based methods at unstructured pruning of a variety of modern architectures with CIFAR-10, particularly excelling at higher sparsity.

*Rating*
The paper has some clear positives, particularly:
    + Clear writing and formatting
    + Simple method
    + Good structure for the experimental analysis (with 5x replications!)
However there are a few limitations, noted below; while none is fatal on its own, in total the limitations have led me to recommend "weak reject" currently.

Limitations of the method:
(1) Residual networks: The lack of an explicit strategy for handling residual connections (and the accompanying worsened relative performance) is a notable limitation since residual/skip connections are nearly universal in state-of-the-art large networks. The performance was shown to still be *slightly* better than with magnitude pruning.
(2) Global ranking: Since connections are pruned layerwise, rather than taking the best-k neurons across the entire network at once, I assume that the LAP pruning criterion doesn't scale reasonably across layers. This implies that the method cannot be used to learn network structure. Instead the user must decide the desired number of neurons at each layer.
(3) Structured pruning: There is no mention of pruning entire convolutional kernels or "neurons" at once, so I assume that only individual weights were pruned. Since structured pruning is the simplest way to achieve speedup in network inference (as opposed to merely reduction in model size), how does the LAP criterion perform when adapted for structured pruning, e.g. removing filters/neurons with the best average LAP score?

Limitations of the experiments:
(4) Baselines: While the paper is explicitly focused on an easy to compute replacement for magnitude-based pruning, there are a wide variety of alternative methods available. These vary in complexity, runtime, etc., but they deserve mention and either explicit comparison in the experiments or reasoning to justify the omission of such comparisons.
(5) ImageNet: (Insert obligatory statement about the ubiquity of ImageNet experiments, ...) While it is cliche to request ImageNet experiments and CIFAR-10 is a helpful stand-in, they would be really nice to have.
(6) Activations after non-linearities: While Fig. 3 and the remaining experiments present a reasonable case that the presence of non-linearities doesn't prevent LAP from improving upon magnitude-based pruning, it doesn't resolve the issue either. Whether considering negative values clipped by with ReLU or large magnitude values that are squashed by sigmoid and tanh, the linear-only model is a poor approximation for some unknown fraction of neurons for probably most inputs. Does this mean that LAP is underperforming in those cases? Are those cases sufficiently rare or randomly distributed that they are merely noise? Is there another mechanism at play? In practical terms, how much does the activation rate (positive for ReLU, linear/unsquashed for sigmoid/tanh) vary by neuron? This seems like a reasonably simple thing to compute and incorporate into pruning.

*Notes*
Eq. (4): Is (4) simply the one-step/greedy approximation to the optimization in (3)? If so, it may be helpful to state this explicitly. Also, is $w = W_i[j,k]$? If so, this is useful to explicitly state.
Sec 2.1: Consider noting that the linear-model setup is used to construct the method, but non-linearities are addressed subsequently
Sec 2.2: Is the activation probability p_j used in practice, or is it merely an explanatory device?
pg5: "gradually prune (p/5)%" and marked with a suffix '-seq'"
pg5: note that residual connections are discussed in the experiments?
Tables 3-6: note that these all use CIFAR-10
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper proposes a new magnitude-based pruning method (and a few variants) by extending the single-layer distortion minimization problem to multi-layer cases so that the correlation between layers is taken into account. Particularly, the authors take into account the weight tensors of neighboring layers in addition to the original layer. The proposed algorithm looks promising and interesting. Empirically, the authors show that the proposed method consistently outperforms the standard magnitude-based pruning method.

Overall, the paper is well-written and I think the algorithm is novel. Therefore, I've given the score of 6.

Comments:
(1) It seems obvious that the proposed method would increase the computation cost, but the authors didn't give any discussion or results on that. 
(2) Although the main focus of the paper is magnitude-based pruning, I think the authors should include one baseline of Hessian-based pruning methods for comparison. As I know, the computation overhead of Hessian-based methods (e.g., OBD) is relatively small for the networks used in this paper. In particular, Hessian-based pruning methods can also be interpreted as a distortion minimization problem but in a different space/metric. So I wonder if the authors can extend LAP to Hessian-based pruning methods.
(3) The authors introduced LAP with deep linear networks. However, the details of LAP in non-linear network are missing. I encourage the authors to fill in the details in section 2.2 in the next revision.
(4) Currently, all experiments are done on CIFAR-10 dataset. I wonder if the author can include one more dataset. For example, comparison between MP and LAP on Tiny-ImageNet or even ImageNet. As I know, experiments of ImageNet can fit into a 4-GPU server for magnitude-based methods.
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This paper proposed Lookahead Pruning (LAP), a new method for model weight pruning to generate neural networks with sparse weights. The authors interpret the conventional magnitude pruning (MP) as pruning each layer individually while the proposed lookahead method considers neighboring layers' weights. Specifically, the proposed methods prune weights which introduce small distortion to the Jacobian matrix of 3 consecutively connected (linear) layers; and the conventional magnitude pruning can be viewed as a degenerated/special case of the LAP. The primary contributions of the paper are: 1) the authors propose the LAP method for fully connected layers; 2) they present empirical applications/extensions to models with non-linear-activations and batchnorms (which are two important components of modern neural networks); 3) The authors empirically show that LAP (and its variants such LAP-forward and LAP-forward-seq) can generate sparse models with better test accuracy than MP, across fully-connected network (FCN), and Conv models (such as VGG and ResNet) on MNIST and CIFAR dataset.

I think the method in this paper is well motivated both mathematically (minimizing distortions of Jacobian) and intuitively (considering multiple layers holistically). The empirical benefits of the proposed method is properly validated against MP in various dataset and models. Also the paper is well written. Thus I give weak accept and I am willing to raise the score if convincing clarification on the following questions can be provided in author responses and in the future draft:

1. As the LAP method introduces higher computation overhead in pruning weights, I was wondering how it compares to MP in terms of run-time-efficiency. As LAP requires computing a score for each single weight value (though some of the computationally heavy terms can be reused), it is important to discuss how long does LAP pruning take, comparing to the run-time of retraining (after pruning). This will help further evaluate the empirical efficiency of the LAP method.

2. The experiment focus on CNNs while the authors advocating versatility of the methods. Thus I was wondering how LAP performs on models in other domains, such as transformer-based NLP models.

3. (Relatively minor). The experiment purely focused on comparing pruning methods. To demonstrate the empirical merits of LAP, I think it will be more convincing to also compare with naive baselines of using narrow / shallower networks such as in Sohoni et al.[1]. This will demonstrate that pruning itself and LAP as an instantiation of pruning should be considered over these naive narrow / shallower baselines with the same amount of weight parameters as pruned models.

4. The tables (table 1-5) are massive but the take-away message is not crystal clear in the text or captions of the table. Is the take-away message something like 1) one might want to use LAP-forward(backward) over LAP when you have very high sparsity, and 2) the sequential versions can further enhance the performance? 


Minor suggestions to improve the paper:

1. Line 5 and 6 in Algorithm 1 is confusing. I suppose the authors mean selecting the weights which trigger small value for L to zero-out. To me the current line 5,6 does not directly reflect this.

2. It might be good to provide a proof in appendix on equation 5), it take me quite a few minutes to verify it. Providing a proof can help readers to read more smoothly.

3. The order of the content can be slight reorganized, e.g. why talking about the adaptation of LAP on batchnorm after you discussed all the directional and sequential variants of LAP?

Reference
[1] Low-Memory Neural Network Training: A Technical Report. Sohoni et al. 


review_title: Official Blind Review #3

Meta-review: This paper introduces a pruning criterion which is similar to magnitude-based pruning, but which accounts for the interactions between layers. The reviewers have gone through the paper carefully, and after back-and-forth with the authors, they are all satisfied with the paper and support acceptance.
Decision: Accept (Poster)