Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a novel structured pruning method called DiscriminAtive Masking (DAM) for neuron-level sparsity. A relu-tanh gating function is proposed to apply on the neuron activations, and gradually zeroing out the neurons at lower order by introducing a L0 norm. The proposed method is a single stage method, and does not require further after-prune finetuning. Extensive experiments demonstrate the effectiveness of the proposed method
main_review: The proposed DAM method for learning compact representations is novel. Unlike previous method in applying L1 norm as sparsity-inducing regularization followed by further pruning and finetuning, the proposed method applies the L0 norm and does not need additional step of finetuning with the proposed gating function and the neuron ordering. And it does not require combinatorial search, and can be trained with simple gradient descent.

Extensive experiments have been conducted to demonstrate the effectiveness of the proposed method. Specifically, the authors conduct experiments for representation learning problems, including dimensionality reduction, recommendation, graph representation, etc., in addition to the structure network pruning. The results show that even better performance can be achieved with more compact representations with the proposed DAM for recommendation problem. For structure pruning, it is shown to perform better than the state-of-the-art ChipNet. Stability analysis and permutation invariance experiments have also been conducted.

Overall, I think the paper is of high quality, and contain significant novelty.
limitations_and_societal_impact: - It is understandable that lambda controls the compactness of tthe representations or the pruning sparsity. But it seems that what is the exact relationship between lambda and the resulting compactness is not clear and could depend on the specific dataset. For example, if we want to achieve 20% sparsity, in one dataset lambda needs to be 10.512, but in another dataset it could be 5.123, etc. Is it a right understanding? In practise, how should be find the right lambda if we have an sparsity in mind?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a single stage pruning method (DAM) that jointly prunes and refines weights during training. The method uses a monotonically increasing gate function for the neurons/channels in each layer with one trainable parameter. The gate function only discriminates neurons based on the position of them in the layer. The proposed method achieves reasonable results without any fine-tuning.
main_review: The proposed method is rather simple, in a positive way, and tries to address an important problem. It does not need fine-tuning which can result in lower training time. The performance of it is evaluated on several applications, like graph representation learning and structured network pruning for image classification, where it shows reasonable performance compared with the baselines. In addition, studies on sensitivity to hyperparameters are provided which are helpful, and detailed experimental setups are included that is a plus.

In the related work, [24] can be applied as a structured pruning through group sparsity, it also considers the L0 problem and can be single stage and versatile. So, the discussion there can be adjusted as the proposed method is not the first single-stage pruning method considering the L0 problem. [24] can also be used as a baseline.

In the contributions, it is stated that the proposed method can be applied to both training from scratch and pre-trained setups. Currently, no experiment is done to support the pre-trained setup. It appears that the success of the method is claimed to be based on neurons being symmetrically invariant to each other at initialization. For a pre-trained network, that symmetry may not hold so it is unclear whether the method would work in that case. 

In the recommendation system (where the results are very close) and graph representation examples, it will be helpful to tune the dropout rate or replace the layer with a learnable version like concrete dropout to have a better baseline for comparison to support the paper’s claims. 

---After author response---
I thank the authors for their responses. I still think that for the mentioned experiments having a more comparable and better baseline would be helpful. After reading other reviews and responses, I remain positive about the paper.


limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a gradual structured pruning method, DAM, that can achieve good performance with various applications. The paper conducts extensive experiments to support the effectiveness of the proposed methods. The method part is reasonable and makes sense.
main_review: I generally like the simplicity of the proposed method and the empirical results are promising. 

What I don't like is the method is proposed heuristically and suddenly. I want to read more reasoning parts of the proposed method. For instance, is this method stemmed from other disciplines or just heuristically? Can we use sigmoid function rather than tanh in equation 2? 

Moreover, I encourage the authors to add more unstructured sparsity methods in the related work. First,  gradual magnitude pruning [1,2] also gradually prunes the model to the target sparsity during training, which should be added to the related work. An important branch of unstructured sparsity methods, dynamic sparse training [3,4,5], is ignored. Even though it is an unstructured method, the sparse connectivity is allowed to be gradually optimized during training.  

[1] Zhu, Michael, and Suyog Gupta. "To prune, or not to prune: exploring the efficacy of pruning for model compression." arXiv preprint arXiv:1710.01878 (2017).
[2]Gale, Trevor, Erich Elsen, and Sara Hooker. "The state of sparsity in deep neural networks." arXiv preprint arXiv:1902.09574 (2019).
[3] Mocanu, Decebal Constantin, et al. "Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science." Nature communications 9.1 (2018): 1-12.
[4] Evci, Utku, et al. "Rigging the lottery: Making all tickets winners." International Conference on Machine Learning. PMLR, 2020.
[5] Liu, Shiwei, et al. "Do we actually need dense over-parameterization? in-time over-parameterization in sparse training." arXiv preprint arXiv:2102.02887 (2021).
limitations_and_societal_impact: No. I agree with the authors that the limitations can be good future work directions.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper presents a simple and novel in-training model pruning algorithm to learn sparse feature representation with reduced computational load w/o compromising accuracy. To do this, authors introduce a single learnable parameter to simplify the optimization involved in enforcing L0 sparsity. Authors further provide theoretic analysis on how reducing the single learnable parameter would enforce L0 sparsity. Finally, authors demonstrate the remarkable performance over various applications comparing with the established strong baselines.    
main_review: While authors have theoretically and empirically justified the using a single learnable parameter to achieve L0 sparsity, they only empirically demonstrate the attractive property of permutation invariance via sensitivity analysis.  In addition to the claim “all neurons are symmetrically invariant to each other at initialization and any neuron has an equally good chance of refining themselves to capture useful features later in the training process as any other neuron”, can authors provide analysis why a trivial choice of \mu_{ij} in line 187 would work? For empirical analysis reported in Table 2, what are the corresponding values of the learnable parameters? Any guidance on the “optimal” sparsity for a given task and data set that would balance the trade-off between accuracy and network size? 

In addition to reducing resource consumption, network pruning is also to improve generalization to new examples, either In-Distribution (ID) or Out-Of-Distribution (OOD), it seems that all the evaluations are based on ID validation sets in a relatively smaller scale (up to CIFAR-100), additional experiments using new larger data sets (e.g., mini-ImageNet or ImageNet) would further demonstrate the advantages of your network pruning approach.     

limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper proposes a single-stage structured pruning method, DiscriminAtive Masking (DAM), which learns a mask to discriminatively identify neurons that needs to be pruned out.
main_review: Overall, the contribution of the paper is not that clear to me. My first impression is that this paper proposes a structured pruning method (as indicated in the abstract) and expect that it will conduct extensive experiments on pruning/compressing various networks on various benchmark datasets and compare the performance (generalization/FLOPS/runtime/size) with other state-of-the-art methods. However, it spends 2-3 pages in describing its applications in dimensionality reduction, recommendation system and graph representation learning, which confused me a lot. Most importantly, due to the limited length of the paper and so many topics it intends to cover, for each of these applications as well as the structured pruning in Section 5, the demonstrated experiments are rather superficial: either only synthetic data is involved in the experiments or the algorithm is only evaluated on toy dataset (MNIST, CIFAR10...). In Section 5 of structured pruning, the competitors are only restricted to [15,16], which is not that convincing to me. 

I would recommend concentrating on one direction, namely the structured pruning, and conducting serious experiments on this particular part. For example, it would be better to test different backbones, e.g., VGG/ResNet/DenseNet, on ImageNet, and compare the FLOPS reduction, generalization performance, inference speed-up of the resulting compressed model with many other state-of-the-art methods. This would highlight the main contribution to a much larger extent than the current practice.

In the same time, I think the paper lacks a basic proofreading since there are so many typos or mistakes:

- line 73, function "g" should be bold to be consistent with the notation later?
- equation 1, the subscript g in "F_g" should also be bold; the "x" in "x\sim ..." should be bold
- line 152, \theta is not only the learnable parameters of the network, since it also optimizes over \beta. The current formulation is a little bit wired since there is no \theta appears in the objective function 
- Figure 2, in the caption "k=... \beta=..." should be in math mode when typesetting; in (c), (d), what is the setting for \alpha and \mu?
- line 171-172, it says the learnable \beta introduces "a tremendous amount of simplicity", which is a negative description, however, what it actually wants to convey is that this is a positive aspect of the proposed method
- line 207, after the bold phrase, there should be a period at the end
- line 268, after "Lasso)" there are two periods
- line 304, the "E[\beta]" should be in math mode; more importantly, I think think the plot is for the expectation of \beta, since it is impossible to collect infinitely many of running samples for calculating such expectation, what you actually do here is only the average of \beta?






limitations_and_societal_impact: Yes.
review_title: null

Meta-review: The submission proposes a method for learning a network with sparse parameters.  The key idea is to optimize an L0 penalized objective (1).  This is non-differentiable, so instead a ReLU is applied to a parametrized tanh scaling (2) which will selectively zero out some fraction of the weights, where the fraction is monotonic in the beta parameter.  This leads to the main result of the paper (3), which explicitly demonstrates this relationship.  I was initially confused by the comment of Reviewer VSsj who stated that an L0 constraint (non-differentiable) could be trained by gradient descent.  One clue is that the "equality" in (3) includes a ceiling function, implicitly showing that there is indeed a relaxation of the L0 constraint, through the tanh function, albeit a different relaxation than is given by e.g. the L1 relaxation.

Overall, a majority of the reviewers feel that this paper is ready to be accepted at NeurIPS.  It is an interesting and clearly presented contribution to learned network sparsity.  Comparison to baseline methods is on the weak side, but overall this paper is acceptable at NeurIPS.
Decision: Accept (Poster)