Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper drew a connection between soft threshold pruning and Iterative Shrinkage-Thresholding Algorithm (ISTA). Based on this interpretation of soft threshold pruning the authors derive novel threshold schedulers that appear to perform better than alternatives in terms of sparsity-accuracy trade-off.
strength_and_weaknesses: Strength:
- The authors aspire to come up with a unified theoretical backbone of a variety of soft thresholding pruning algorithm. This is an important problem because the community can benefit from a shared theoretical infrastructure for analyzing threshold pruning techniques, which are often empirical in nature.
- The authors propose a theoretically grounded threshold scheduler and demonstrate  some benefits of this threshold scheduler empirically.

Weakness:
- The paper is highly disorganized on at least two levels.
    - **Inadequate explanation of core concepts.** The authors made little to no attempt to introduce and define essential concepts such as "threshold scheduler" for the audience early in the paper. As a result I have to start with Sec.3 Preliminaries and work my way back to make sense of the abstract and introduction.
    - **Disorganized prsentation of technical details.** This disorganized style of writing manifests even when the author discusses specific technical details. For example, when the authors introduce Eq.2, the symbol \mu appears out of nowhere. Upon further examination, I realize that this variable \mu is implicitly defined in the L.3 of the pseudocode presented in Algorithm. 1 as the output of the learning rate scheduler function. The author must clearly define symbols before using them.

        While the authors may very well have much to share with and educate their audience, the disorganized writing style is a significant obstacle to achieving this potential.

- There are major red flags in experimental evaluation.
	- **Reproducibility and fairness of comparison**. I believe authors obtain individual data points in Fig. 2 comparing the proposed threshold scheduler with alternatives by "tuning final threshold D". But the author does not provide details for how this threshold is tuned. This raises questions on 1). whether the core empirical evaluation of this paper can be reproduced 2). whether comparison with alternative soft threshold pruning techniques is fair.
	- **Missing simple baseline.** For the left plot of Fig. 2, the authors instrumented a prior soft threshold pruning technique called STDS to use their proposed threshold scheduler and demonstrated superior performance to other soft threshold pruning techniques. However, it is unclear whether this superior performance is due to the STDS pruning algorithm or the proposed threshold scheduler. The author should report performance of STDS with the default threshold scheduler as an important baseline.
	- **No characterization of uncertainties.** The performance of pruning algorithms can have significant variance between different weight initialization/SGD data order. The results presented in this work lack characterization of uncertainties stemming from this variance.
clarity,_quality,_novelty_and_reproducibility: The clarity and quality of this paper is quite low for the reasons outlined above in the weakness section. I will list a few more instances where the clarity and quality needs significant work below. The paper is reasonably novel. To the best of my knowledge, such a theoretically grounded analysis of threshold scheduler constitutes original work. The paper is not reproducible for that important methodological details are missing. See my first point under Weakness #2. 

Here are a non-exhaustive list of instances in the paper where clarify and quality is clearly lacking:
- On page 1, what do the authors mean by this and what purpose does this sentence serve: "vanilla magnitude-based pruning is considered only as a baseline method."
- On page 2, when the authors describe the merits of ISTA algorithm, the authors wrote "[ISTA] has long been certified as an effective 'pruning' algorithm in all sorts of fields". The author provides neither details on what field is ISTA algorithm effective within nor any citations backing up this claim.
- On page 2, what do the authors mean by "the rationality of training threshold"?
- On page 2, what do the authors mean by "stable regularized term implies a coherent pace of the foregoing two schedulers"? I do not understand this even after reading the whole paper.
- On contribution #3, why should people care about early soft threshold pruning algorithm? From my understanding, soft threshold pruning algorithm does not improve training efficiency because the pruned weights are not actually removed. Then what is the point of pruning early when applying soft threshold pruning?
- On page 3, Sec 3.1, what is n?
- Again on page 3, Sec 3.1, GPO is only defined as one of 12 related works on sparsify-during-training that the authors cited (without any description/explaination). The authors should not expect this term to still exist in the working memory of the audience.
- On page 4, in Algorithm 1, what is sigma? It is never defined.
summary_of_the_review: While the authors investigated an important problem and conducted original research, the quality of presentation and empirical evaluation is significantly below that required for acceptance into ICLR.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a unified framework of iterative soft threshold pruning (ISTA). The framework reveals that previous studies with threshold pruning can be casted as adding regularization terms (i.e., L1-regularization) for training. The authors further connect the learning rate with pruning threshold, and propose the learning rate adapted optimal threshold scheduler. The framework also has the link to other pruning methods such as early pruning, pruning at initialization. Experimental results on various network architectures (ResNet-50, MobileNet-V1 and Spiking Neural Networks) demonstrate the effectiveness of the approach. 
strength_and_weaknesses: Strengths:

- The paper establishes nice connections between the learning rate and pruning threshold scheduler via Theorem 1. 

- The methodologies developed in this paper are well grounded in theory, e.g., the learning rate adapted optimal threshold scheduler for pruning, and PGH schedulers that connects to early pruning.

- The authors conduct extensive studies across various network architectures (ANNs & SNNs) as well as different pruning settings. 

Weakness: 

- While I appreciate the authors' effort in introducing the optimal threshold scheduler and sibling schedulers for pruning, it is not new to formulate pruning with local ISTA, which has been studied by plenty of previous works, e.g., [1,2,3].

  - [1] He Y, Zhang X, Sun J. Channel pruning for accelerating very deep neural networks. Proceedings of the IEEE international conference on computer vision. 2017: 1389-1397.

  - [2] Zhang D, Wang H, Figueiredo M, et al. Learning to share: Simultaneous parameter tying and sparsification in deep learning, International Conference on Learning Representations. 2018.

  - [3] Bai H, Wu J, King I, et al. Few shot network compression via cross distillation. Proceedings of the AAAI Conference on Artificial Intelligence. 2020, 34(04): 3203-3210.

- Only unstructured pruning is discussed in the paper, this however, has limited acceleration in practice. I wonder whether similar results can be established for structured pruning, e.g., via $\mathcal{L}_{2,1}$ regularization. 

Detailed comments: 

- Some parts of the paper are not self-contaiend. While it is claimed to be verified on Mobilenet-V1 and spiking neural networks, their results are in the appendix and not referenced in the main text. 

- Section 3.1 could have been organized better. There is some notation overlapping (e.g., $d$, $\phi(\cdot)$ and $g(\cdot)$ for the pruning threshold). It would be better to have a short overview of STR and STDS first.

- Provide more details on how Eq. 12 and Eq. 13 are derived.

clarity,_quality,_novelty_and_reproducibility: - Clarity: The paper is cleanly written in the general. Some issues are listed in detailed comments.

- Quality: The paper is technically sound.

- Novelty: As mentioned above, the framework with local ISTA is not really new for network pruning. More innovations come from the theory grounded pruning threshold. 

- Reproducibility: Code is provided for reproduction.




summary_of_the_review: The paper provides a comprehensive and theoretical study of soft threshold pruning. While it is not new to tackle network pruning with ISTA or proximal mapping of $\mathcal{L}_1$ regularizer, the theory developed in this paper further motivates the design of optimal pruning threshold scheduler. The experiments are also adequate and effective to demonstrate the proposed solutions. The authors also provide in-depth analysis on how the framework relates early pruning, spiking neural nets, and more in the appendix. One potential issue with the paper is the practicality, as unstructured pruning can hardly bring actual acceleration. Overall, the paper is quite informative and well presented.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper has reformulated soft threshold pruning as an implicit optimization problem solved using the Iterative Shrinkage-Thresholding Algorithm (ISTA), and proposed a unified theoretical framework for soft threshold pruning. Experiments have shown that the proposed method outperforms SOTA soft threshold pruning algorithms, such as STR and STDS.
strength_and_weaknesses: Strength:
1. The paper is well written and the presentation is clear.
2. The pruning problem is an important one.
3. The theoretical soundness is good.

Weaknesses:
1. The method is evaluated on relatively a small number of networks. Whether the performance improvement persists for other network architectures, such as recurrent neural networks?
2. How the performance is influenced by hyper-parameters? Whether the performance improvement is robust under varying hyper-parameters?
clarity,_quality,_novelty_and_reproducibility: This paper is clearly written and the quality is good. The proposed method is valid and the novelty of this paper is good. The experiments in this paper are reproducible.
summary_of_the_review: The considered problem is very important. The proposed framework is technically sound and achieves better performance than existing SOTA methods. It would be nice to test the method on more architectures and diverse hyper-parameter settings.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: This paper unifies the literature on various pruning methods that use soft thresholding. (Yes, this is a rehash of the paper's title, but the paper's title is very descriptive.) These pruning methods learn a score parameter alongside each weight, and only the weights with the highest sigmoid(score) parameter are kept. The authors generalize this framework in a way that subsumes several prior papers, provide theoretical justification for how to set the many hyperparameters of this class of methods (scheduling the threshold and regularizing the score parameters), and produce a pruning algorithm that appears to outperform prior work on standard benchmarks.

Strengths: This paper solves the technical problem convincingly.

Weaknesses:
* I agree with reviewer 3EDj that this paper is quite obtuse writing-wise in many places. It's not easy to parse unless you're familiar with both the pruning literature and the optimization literature. This paper is not self-contained and, even if the technical results are nice, the authors are limiting the potential influence of this work by requiring so much prior knowledge and skipping around to make sense of the paper (I read it and had the same challenges as reviewer 3EDj).
* This paper implicitly has a strong degree of "theoretical inevitable-ism" - the idea that, because formal math says something should work according to a simplified set of assumptions, it will work in practice in large-scale settings. That isn't true in much of deep learning (e.g., classical bias-variance tradeoff vs. double descent, NTK results having limited practical applicability), and I wish the authors were more careful about acknowledging that the derivations are a source of inspiration for good empirical results that are not inevitable merely because the authors did some proofs. This is a problem throughout the pruning literature, and the authors should not be held solely responsible for this, but they do have an opportunity to do better.

Overall: This is a probable accept. The technical results are solid, it brings order to a chaotic area of the literature, and the evaluation shows that the techniques in this paper improve the state-of-the-art in the process. The concerns of Reviewer 3EDj are acknowledged, and I hope the authors clean up this paper and make it accessible to a wider audience. Otherwise, in its current state, it isn't sufficiently accessible to the pruning community to have impact, and I worry it will collect dust as a contribution that isn't built on because it is too obtuse.
justification_for_why_not_higher_score: This paper is a nice contribution to a niche corner (soft threshold pruning) of a niche corner (unstructured neural network pruning) of a somewhat niche corner (neural network pruning) of the neural network literature. It tidies up this corner (which is great), but it is not of sufficiently broad interest to deserve a spotlight.
justification_for_why_not_lower_score: The paper makes a solid technical advance in a relatively popular space.
Decision: Accept: poster