Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a new approach for enforcing sparsity with the L0-regularization technique proposed by Louizos et al. that allows for explicit specification of the target sparsity at the end of the optimization process. They demonstrate that their approach produces high quality sparse models on a range of applications and datasets.

strengths_and_weaknesses: Strengths
1. The paper is very well written and easy to follow. The authors do a good job of highlighting the benefits of their technique, including the ability to target specific sparsity levels and the modularity and extensibility provided by this capability.
2. I think the results of this work are impactful. L0-regularization was a very promising technique but prior work (Gale et al.) was unable to scale it to large models. The authors addressed its limitations both in the difficulty of tuning the loss coefficient to achieve a desired level of sparsity and the previously reported issues with training stability
 
I did not identify any major weaknesses. In the next section I highlight a few things that could help to bring my score up further.

questions: 1. I wonder whether your fixes for the training instability reported by Gale et al. would generalize to unstructured sparsity as they explored. If you could comment on this or include results in the paper I think it would be useful to readers.
2. In Table 2 it would be nice to include some higher sparsity levels. I recognize that you’re inducing structured sparsity and that expected compression levels will be lower than with unstructured but I think it would be useful to readers to see up to 50% compression.
3. If you wanted to demonstrate another level of impact you could compare your improved L0-regularization technique to other state-of-the-art methods for neuron/channel pruning. I do not think this is necessary for this work to be published at NeurIPS, but it would be a way to increase the impact of the contributions further.

limitations: I am not aware of any limitations of the proposed approach that were not mentioned in the work. The authors did explicitly analyze potential issues - for example, the dynamics of their constraint during training.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper mainly focuses on a previous work l0 regularization and demonstrate its feasibility  when solving it with constraints. The work considers the problem of L0 regularization thoroughly by 1) learning models with controllable levels of sparsity 2) dual restart heuristic to avoid the excessive regularization 3) fixing the performance dropping problem of L0 regularization.
strengths_and_weaknesses: Strengths: 
1. This work presents valid improvements to previous L0 regularization, since I know L0 regularization cannot achieve stable results on ImageNet.

2. This work gives a nice modification to the original test-time model selection criterion for its mediean property.

Weaknesses: 
1. There is already work on dealing with the penality and unstable problem of L0 regularzation [1](Effective Sparsification of Neural Networks with Global Sparsity Constraint) and the final accuracy result is lower than [1] on ImageNet (73.44\% with 12% params vs. 74.68% with 10% params). Please discuss the difference and connections with [1] and better make more empirical comparisons.

2. The good property of test-time model selection should be further justified. Can you please give a table of statistics to demonstrate the changing dynamics of the test-time model selected during the training process? 

questions: 1. Can you present more comparisons with more sota methods and sparser region [2] ?

2. Can this method be applied to sparse training and improve performance?

[2] Winning the Lottery with Continuous Sparsification
limitations: I don't see any negative societal impact of this work.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper considers sparse neural network training with cardinality constraints instead of regularization. Based on the stochastic gates, they adopted a constrained formulation, which allows for simultaneously training and controlling exact sparsity in an end-to-end fashion. In terms of optimization algorithm, they considered the Lagrangian min-max problem and proposed gradient-descent-ascent algorithm with dual restarts. They conducted extensive and comprehensive numerical experiments on various network models, and demonstrated the effectiveness of their proposal.
strengths_and_weaknesses: The paper proposes an end-to-end training framework for neural network training with exact sparsity constraint, which avoids tuning the sparsity-inducing regularization coefficients to satisfy the desired sparsity. They extended the previous work on stochastic gates (Louizos et al., 2018) to the constrained version, and solve it by gradient descent ascent on its Lagrangian. The paper is well written and comprehensive. Their extensive numerical experiments are sound and demonstrate their method consistently help achieve the sparsity targets for various models. In general, the paper is solid and good.
Here are some comments/questions:

1. Line 91-94: the introduction of $\phi$ is a bit abrupt and may not be friendly to those who have not read the previous work Louizos et al. 2018. It is probably better to describe the idea of Louizos’ paper, and provide explicit expression of $\lambda_{\text{pen}}\mathbb{E}_{\bf{z}|\phi}[\|z\|_0]$ wrt $\phi$. 
2. The $L_0$-density throughout the paper (in experiment section, figures and tables): is it the “smoothed” version $g_{\text{const}}(\phi_g)$ (the LHS of the constraint) or the density based on test-time model? It is sort of confusing. At first, I thought it was the actual parameter density of the final purged model. So, it is probably better to clarify/define the term somewhere.

3. Question related to test-time and purging: each $\mathbb{E}_{\bf{z}|\phi}[1\{z_i\neq 0\}]$ is a number between $[0,1]$. This means even $g (\phi_g)\leq \epsilon_g$ holds, there are still chances that the $z$ in the test-time model has a larger density compared to $\epsilon_g$. But why the Params(%) is in general smaller than $L_0$-density in Table 2?
4. Question related to $L_0$-density: in the left panel of Figure 3, the curves of $L_0$-density do not meet the target sparsity level?  Any intuition why that happens?
5. As a following work of Louizos et al. 2018, Yutaro et al. 2020 [1] mentioned the logistic-based hard-concrete distribution (also used in this paper) “yields high-variance gradient estimates”, and they proposed to use Gaussian distribution to replace the HC distribution for $z$. It would be interesting to check if Gaussian distribution works better in your case.

[1] Yamada, Yutaro, Ofir Lindenbaum, Sahand Negahban, and Yuval Kluger. "Feature selection using stochastic gates." In International Conference on Machine Learning, pp. 10648-10659. PMLR, 2020.

questions: See "Strengths And Weaknesses"
limitations: See "Strengths And Weaknesses"
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors take the $L_0$ penalty method from Louizos et al. and present a constrained version of it, i.e., estimate the Lagrange multiplier to satisfy some desired level of parameter sparsity. They do this by specifying a desired parameter sparsity density, either for the entire model or per layer. The Lagrange factor appears to be estimated using (non-stochastic) gradient descent, with one additional trick: only apply the constraint when it isn’t satisfied. Experiments are performed using convolutional and fully-connected models on image datasets (CIFAR10, TinyImageNet, and ImageNet) in the supervised learning setting. They compare with the penalized version of the regularizer, as well as a pruning method based on weight magnitude.

strengths_and_weaknesses: **Strengths**

*Originality*:
The originality of the paper is somewhat unclear to me, as I am not especially familiar with the latest techniques for neural network compression. To the best of my knowledge, I am not aware of previous works that allow setting a desired number of used parameters (rather than setting a Lagrange hyperparameter). In this regard, the paper presents a somewhat original method.

The authors also present a number of additional aspects, which includes a ‘dual restart heuristic’ and techniques for applying regularization to residual networks. These are mildly original, but I do not see them as a core contribution to the paper’s novelty.

*Quality*: 
The paper is fairly high quality. The contributions of the paper are clearly defined. The diagrams clearly demonstrate the main benefit: being able to trace out the curve of parameters vs. performance in a more targeted manner. Experiments with multiple model architectures and datasets demonstrate the method, along with analysis plots in Figures 2 and 3.

*Clarity*:
The paper is very clear in its presentation. The writing is clear, containing emphases on important words and ideas. Mathematical concepts are generally presented and explained clearly. Experiments clearly demonstrate the settings described in the paper. The figures are all well-presented, with clear legends and labels.

*Significance*:
Assuming that the paper is at least moderately original, then I see the results as being at least somewhat significant.

In particular, in Table 2, the authors demonstrate that their method is able to train models that contain similar numbers of parameters as magnitude pruning while obtaining significantly improved validation accuracy (in some cases as 70% difference). In fairness, it would be reasonable to also compare with smaller models trained without weight regularization, as well as models that are optimized post-training. Thus, the significance may be somewhat diminished after these additional comparisons.

Again, assuming the technique has not been introduced previously, the notion of being able to pre-specify the size of the model could be useful, e.g., to ensure that the final model fits within memory on a smaller device.

The authors also claim that their method, along with several additional techniques discussed in Section 5.1, is able to train $L_0$-regularized residual networks, whereas, supposedly, standard $L_0$-regularization struggles in these settings. I am not familiar with this literature, so it’s somewhat unclear to me, but I see this as a further significant contribution. Many large-scale vision models contain residual components, and this set of techniques opens up the applicability of $L_0$-regularization to these networks.

**Weaknesses**

*Originality*:
The main contribution of the paper is to replace the $L_0$-regularized training objective with a constrained objective, adjusting the Lagrange factor to satisfy this constraint. While I’m generally in support of simple methods, this method is so simple as to make me question whether something similar hasn’t already been tried before. Replacing regularized objectives with their constrained counterparts is a common, classic technique within the machine learning literature. Some examples with modern deep networks can be found in variational autoencoders ([Rezende & Viola, 2018](https://arxiv.org/abs/1810.00597)) and policy regularization in reinforcement learning ([Haarnoja et al., 2018](https://arxiv.org/abs/1812.05905)). Given that this method is seemingly *so* simple, in my view, it should provide highly significant or surprising results to warrant publication.

*Quality*: 
The quality of the paper is generally great, however, I have the following concerns:

It’s not clear whether post-training sparsification would obviate the issues with penalty-based regularization raised by the authors. That is, one could distill a larger, trained model into a smaller model of pre-specified size. In the introduction, the authors point to the additional computational overhead required by these methods, however, I do not find this to be the most compelling point. Additional discussion or empirical evaluation around this area would help to strengthen the authors’ claims.

Experiments are performed entirely within the setting of supervised learning for image classification. While it’s understandable to demonstrate a method with practical applications in domains that are commonly used in-practice, I worry that this limits the scope of the paper. A more complete paper would demonstrate this method in a range of settings (e.g., within various networks for generative modeling or reinforcement learning). Given the seeming simplicity of the method, it’s unclear why the authors did not demonstrate this technique more broadly.

*Clarity*:
I do not see any major weaknesses in the clarity of the paper. The main ideas are presented well.

*Significance*:
The significance of this paper may be somewhat limited.

As noted above, it may be the case that post-training sparsification methods would already provide a solution for obtaining a sparse network with a particular number of parameters. If this is the case, then the fact that this method is capable of doing so during training seems less consequential.

Also as mentioned above, the paper currently only contains empirical evaluations with supervised learning on image datasets. Expanding this to include other data modalities and tasks would expand the appeal of the paper.

The main selling-point of the paper is the ability to pre-specify the number of active parameters in the final, trained model. That is, one no longer has to manually adjust regularization penalty hyper-parameters during training. While I somewhat believe the appeal of this idea, in practice, I’m not sure whether this would actually save one from having to perform hyper-parameter tuning. The notion of having to pre-specify the density of active parameters (particularly per layer) is not so conceptually distinct from specifying the size (depth and width) of the network itself. Given that we generally sweep over these parameters, I do not see replacing a regularization penalty with a constraint to be a huge win here. Admittedly, this method allows one to set a global constraint, which is ideal compared with setting per-layer constraints / sizes.

The experiments in Table 2 (training ResNet50 on ImageNet) are fairly significant in terms of improvements in error. However, given the additional proposed techniques, I wonder whether a penalty-based regularization approach would perform similarly. If this is the case, then the only remaining benefit is the ease of tuning.
questions: My main question is regarding the use of gradient descent for the Lagrange multiplier. Updating this factor once per epoch seems somewhat arbitrary, and I would guess that the issues with the slowly updating factor affecting training stem from this choice. Why not update the Lagrange multiplier online using stochastic gradients from each batch? Did you try this? The ‘dual restart heuristic’ may not be necessary in this case.
limitations: While the authors do not explicitly address the limitations of their work, they discuss the various tradeoffs of different sparsification algorithms to a reasonable extent.
review_title: null

Meta-review: metareview: Ratings: 7/8/5/7.
Confidence: 3/4/4/4.
Discussion among reviewers: No.

Summary: this paper introduces a method for learning neural networks with an exact sparsity % target instead of regularization constraints. The method builds on the smoothed L0 regularization objective from Louizos et al, which was shown to be difficult to scale. 

The reviewers generally agree that the paper is easy to follow, introduces ideas that are interesting to the NeurIPS community, and that the results look promising. 

The authors wrote detailed responses to the reviewers' concerns. During the rebuttal period, the authors performed additional experiments on ImageNet as suggested by reviewer TJdj, and updated their paper. The reviewer(s) did not respond to this update, but their reviews were already positive. My recommendation is to accept.
confidence: Certain
Decision: Accept