Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a component-wise network sparsification framework based on proximal gradient descent. In the proposed method, each component is associated with a learnable coefficient which is further regularized through a regularizer. The coefficients learning, based on proximal gradient methods, will be projected/thresholded to zero, indicating the component is dropped. Authors also adopt some tricks to relax the backward gradient flow to ensure the components are not dead always. The experiments show some analysis of the proposed method on channel pruning, discovering neural wirings, and graph structure learning.
main_review: originality:
The proposed method in the paper is a quite straightforward extension of previous works which use proximal gradient descent to perform network sparsification. The main difference is that the target variable to be regularized is changed to some component coefficients.

quality:
The proposed method should be technically sound. However, it seems that there are several potential concerns
- Although the authors mention that normal sparsification would lead to irregular structures, it seems that there are no experiments to validate that irregular structures could potentially cause some problems. Neural networks are normally randomly initialized with each weight potentially swappable. It's hard to see how this dropping component intuition would be very different/useful in network sparsification.
- The component size also needs to be specified in prior manually, which introduces another limitation in the proposed method.

clarity: The paper's writing is pretty clear. A small typo: line 133 "singed"

significance: The proposed method is a simple application of proximal gradient-based sparsification. Doesn't seem like this is a significant method for related fields.
limitations_and_societal_impact: No obvious potential negative societal impact
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This study proposes a method that enables differentiable sparsification for deep neural network and can be applied into model compression and architecture learning tasks. Experiments on small datasets are conducted to demonstrate the effectiveness of reducing network sizes and learning network architectures. 
main_review: The proposed method is simple. The paper has a clear demonstration. However, this study lacks novelty. The core contribution is an engineering practice composed of the three steps in Eq. 9 - Eq. 11, and the regularization term in Eq. 14. There are variant ways to achieve differentiable zero weights. What is and how to verify the superiority of the proposed method in Eq. 9 – Eq. 11? In experiments, only analytical experiments on small datasets are conducted. There is no comparison with other model compression or architecture search methods on challenging tasks. Therefore, the potential applicability of this study is not convincing. 
limitations_and_societal_impact: I have no concern with the limitations and societal impact of the study.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper presents a network sparsification method. It represents a network as a linear combination of different components, and the weights for different components are parameterized by some unconstrained parameters. By enforcing the Lp norm on the weights of different components, all parameters can be learnt in an end-to-end manner. Then the proposed method is applied to three applications: channel pruning in CNN, Discovering Neural Wirings, and Learning relationship between nodes in Graph. Experiments validate the effectiveness of the proposed solution. 
main_review: +The idea of representing the network as a linear combination of different components and sparsify the weights of different components is interesting. 
+The idea of parameterize the weights with unconstrained parameters is interesting. 
+The paper is readable and easy to follow. 


- The proposed method imposes an Lp norm on the weights. However, in the first application (Channel Pruning), p is set to 1. while in the third application, p is set to 0.5. As stated in the section 3.3, the line before equation 14, p should be less than one. Then why you use p=1 in the first application? Then how different p's affect the final performance?
- Regarding the baselines in Channel pruning, there are many state-of-the-art methods, however, only NS is compared and it is a relative out-of-date method. More recent methods should be compared.
- Regarding the first and second applications, only a small dataset is used. More challenging dataset, like ImageNet should be used to further validate the performance of different methods. As reported in previous work, the performance of different methods on different datasets sometimes is not consistent. 
- As  lambda in equation 13 is an important factor for evaluating the sparsity of the network. Then please show how different lambda's affect the performance in the first and second applications. 
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper proposes a new differentiable sparsification method to learn sparse structure in neural networks in an end-to-end manner. This objective is straightforward and well-motivated. However, the execution is not novel, comparisons to the SOTA is lacking, reasons for design choices were unsubstantiated with experiments and would thus require additional ablation analysis.

main_review: ## Strengths
This paper proposes differentiable sparsification for CNNs and GCNs. The method is straightforward. A potentially novel improvement when compared to DARTS [18] is the inclusion of the proximal operator into the architecture parameter as shown in equation (10) and the evaluation of the backward pass with non-zero surrogate gradients to avoid dead neurons. 

## Weaknesses
To begin, Section 2 related work is missing a large chunk of alternative methods that can also sparsity neural networks with end-to-end training. For instance, stochastic networks with policy gradients [b,c], other differentiable approaches to sparsity [d,g], and other more modern baselines for comparison, e.g. for pruning [e,f,h,i]. 
The majority of Section 2.1 “proximal gradient” bears low relevance to the method proposed in the paper, and should be simplified. How does your method compare to proximal-gradient-based pruning such as [a]?
Section 2.2 has only two references. This subsection require substantial rewrite to include relevant works such as the ones mentioned earlier.

The experiments on CNNs are lacking. The paper has many magical design choices and could necessitate additional ablation and sensitivity analysis. For instance, the choices of $\alpha$ and $\beta$ in Line 204 seem arbitrary. The GCN results are confusing (see Minor issues below), and the reviewer cannot judge its quality in its current presentation.

## Minor issues
* Line 38: “performed as a separate step separately” repetition.
* Line 133: “singed” -> “signed”.
* Line 197: “In network-slimming, λ in Eq. 13 is fixed to $10^{−5}$, but in our approach we vary its value to induce different level of sparsity” It is not clear how this varies the level of sparsity, as in NS the sparsity is mostly determined by $k$ as we keep the top-$k$% of all channels.
* Line 275: grammatical error.
* Table 3: The columns are unexplained. What do you mean by “#N.Z”, “MAPE” and “L.R.”? It is also not clear to the reviewer what models I, II and III mean or how they were defined even after reading the appendix.
* Line 311: Table 4a -> Table 3a.
* Line 317: Table 4b -> Table 3b.

## References:
```
[a]: Li, Yawei et al., Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network Compression, CVPR 20.
[b] Louizos et al., Learning sparse neural networks through L0 regularization, ICLR 18. 
[c] Guo, Shaopeng et al., DMCP: Differentiable Markov Channel Pruning for Neural Networks, CVPR 20.
[d] Kang, Minsoo et al., Operation-Aware Soft Channel Pruning using Differentiable Masks, ICML 20. 
[e] Zhuang et al., Neuron-level Structured Pruning using Polarization Regularizer, NeurIPS 20.
[f] Li et al.,  EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruning, ECCV 20.
[g] Ning et al., DSA: More Efficient Budgeted Pruning via Differentiable Sparsity Allocation, ECCV 20.
[h] Chin et al., Towards Efficient Model Compression via Learned Global Ranking, CVPR 20.
[i] Guo et al., Multi-Dimensional Pruning: A Unified Framework for Model Compression, CVPR 20.
```

limitations_and_societal_impact: The limitations of the paper is sufficiently discussed in the paper.
review_title: null

Meta-review: This paper proposes a method based on sub-differential to sparsifying a network parameters during training. Although this paper contains novel interesting ideas, the results are not convincing nor significant enough to convince the reviewers. The reviews unanimously suggest the quality and significance of this paper is below the standard of the conference. The AC agrees with their recommendations. 
Decision: Reject