Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a norm based metric (PQ-Index) to quantify weight sparsity of a neural network. The metric is designed to satisfy the six properties proposed by earlier research in economy similar to the Gini Index. PQ-Index is then used to decide how much a neural network can be pruned at a given time during the training. Authors compare their method to Lottery-ticket style iterative pruning on CIFAR and Fashion-MNIST.
strength_and_weaknesses: # Strengths
- Proposal of a grounded sparsity metric. As far as I know, this part is novel and the most exciting part of this work. I appreciate the utilization of cross-domain knowledge.

# Weakness
- Experimental results are not clear and convincing. Lottery ticket pruning does as good or better than proposed method in many plots. Do you do hyper-parameter search for different methods? A fair comparison should include different fraction values for lottery style pruning. Similarly when different p/q values are selected one probably needs to pick different hyper-parameters. Also, plots needs to be improved to highlight the differences between methods. Using log on the y axis (Figure-2/3) can help with this I think.
- "sota algorithms such as LT-based pruning" This statement is not correct. See [1], which show rewinding the weights is a bad choice and [2] which shows gradual magnitude pruning often achieves best results. It's fine not to have SOTA results/algorithms but this statement is not correct and needs to be changed. I recommend using another pruning algorithm like gradual magnitude pruning with or without learning rate recycling (see [2]).
- Authors say in the introduction "the sparsity will first decrease when a large model is being effectively regularized", however I haven't seen any evidence for this claim. Have I missed a plot? It would be nice to show this in a regular training run (like resnet in cifar-10 with learning rate decay and 200 epochs of training). Similarly one could think of experiment that show the usefulness of the PQ-Index at predicting pruning rates. Can PQ-Index predict maximum pruning rate over the course of the training? How well this compare to using a fix magnitude threshold? 

# Questions
- Looking at Theorem-2, n_r=0 seems to imply removing only zeros. Is that accurate? If so, would be nice to make this clear.
- Do you include pruned connections (zeros) in your PQI calculations? 
- Figure:5-a there is a peak on the PQ-Index at T=20. Why is this? With a high score like this I would expect performance not to drop after pruning. 

# Suggestions
- When p=0.5 and q=1, sparsity would be equal to PQ-metric if I understand correctly. d=100, one-hot, sparsity=PQ-index=0.99. It would be nice to do experiments on pretrained models at different widths and sparsities (this can be achieved with different regularization coefficients) and show how well PQ-index provide maximum pruning rate without significant performance drop. It is very difficult to make conclusion in the LT-rewinding setting since, pruning is repeated multiple times and it is not clear a decision was good or not.

# Minor
- (Page 8: "It aligns with the intuition..") [2] shows that early layers are often better kept dense.
- It would be nice to compare the final sparsity distributions found by PQI with the ones provided by the ERK distribution [3]. 
- "pruning is the most popular and effective approach" probably quantization had much more impact on this due to the int8 support on many recent hardware. 
- Remark 3 "... does not satisfy the six properties" -> which of the six? I think some of them are satisfied. It would be nice to be more specific here.
- Lottery ticket prunes p% of the remaining weights. So the #weights pruned at each iteration decreases over time.
- Figure:3, it is not clear which methods does better (like many others).

## After Rebuttal
I thank authors for their extensive response. After reading their response and checking the changes indicated with blue; I decided to keep my score as it is. That's being said, I wouldn't oppose acceptance. Overall I find the the metric quite interesting, however experimental evaluation for proposed pruning method needs to be improved. I recommend authors to focus on showing the utility of PQI directly using the same model but different metrics (e.g. measuring exactly how close it gets to the max-prune rate). Similarly I think the fixed-pruning rate needs to be searched-over for the LT baseline (like trying P=0.2,0.1,0.3) just to make sure gains are not due to better rate average rate (first point in weakness section). Also I like to re-iterate switching the log-scale for plots, as it is still very difficult to compare different curves at the moment. 

[1] https://arxiv.org/abs/2003.02389
[2] https://arxiv.org/abs/1902.09574
[3] https://arxiv.org/abs/1911.11134
clarity,_quality,_novelty_and_reproducibility: To my knowledge, the contribution of connecting sparsity metrics introduced in economics to weight sparsity in neural networks is novel. Also I found the explanation of the proposed pruning method clear. Reading the text one could implement the method easily. My main concerns are about the evaluation of the proposed method.  
summary_of_the_review: Overall quite interesting work, my concerns lie in experimental evaluations of the work. More specifically contributions 2 and 4 in introduction are not well supported:
- Relationship between PQI and compressibility is not measured directly. This can be done with one-shot pruning experiments (See Suggestion above).
- "SAP can compress more efficiently than SOTA algorithms" is also not well supported since, Lottery style iterative magnitude pruning is far from being a SOTA algorithm and most plots doesn't show improved performance (or hard to read). 

I recommend authors to focus on showing the utility of the metric they developed and understand its strengths and limitations. I think such contribution alone would make a good paper. If proposing a pruning algorithm, experiments/plots/validation needs to be improved/clarified.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The proposed work emphasizes the problem that during pruning there isn't a concrete way to estimate the compressability of a sub-network, which may lead to over- or under-pruning. The PQIndex is proposed to measure this concept and by extension is used to define the Sparsity-informed Adaptive Pruning (SAP) algorithm. Overall, the proposed work measures the trends in sparsity and how it fluctuates during different levels of pruning.
strength_and_weaknesses: Strengths
- A well rounded explanation of the various properties of a sparsity measure and how PQI satisfies them.
- The scope of ablation and results presented analyze the given method well.

Weaknesses
- I encourage the authors to put forward the general notion of sparsity that is assumed across the paper (as defined in $S(w)$) early on in the introduction so that reader can follow the ideas put forward in Fig. 1.
- One of the major issues in the context of pruning literatures' results is the use of MNIST, FashionMNIST and CIFAR10 to evaluate the performance of the proposed model. I encourage the authors to further expand the set of dataset-DNN pairs they experiment on in order to incorporate more real-world data and ensure their observations remain consistent.
- From a more practical perspective, could the authors discuss the absolute limit up to which they can push the sparsity limit of various networks? (Since that is the ultimate goal)
- By extension, could the authors discuss difference in performance values and PQI at the extreme end of sparsity (highlight in existing results)?

clarity,_quality,_novelty_and_reproducibility: Clarity
The context, explanation and technical content are well presented.

Quality and Originality
The PQI measure itself is well presented and seems novel, alongside its application to pruning.
summary_of_the_review: The novelty of the proposed PQI measure and the pruning algorithm are of keen interest. However, the results could be further improved to make it more comparable to state-of-the-art approaches.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a pruning quality index (PQI) that satisfies desired properties (according to previous work) which can be used to set sparsity ratios. They use PQI to obtain sparsity ratios in the lottery ticket algorithm and show improved results.
strength_and_weaknesses: ## Strengths
1. The problem of identifying sparsity ratios is important and it is valuable for many pruning algorithms. The introduced PQI has desirable properties and shows improved performance when used to adaptively set sparsity ratios while pruning.
2. The paper is well-written and the claims are backed by theory and experiments

## Weaknesses
1. The final algorithm introduces additional hyperparameters (eg., \eta_r, \gamma, \beta) and is not clear about the necessity and if they are necessary raises concerns about the introduced PQI. Specifically if \eta_r is a hyperparameter then how is it different to having r as a hyperparameter? I presume the algorithm is not sensitive to the value of \eta_r too much but it would be good to clarify this. 
2. The argument made in fig.1 is handwavy and not clear how it helps the narrative of the paper. To me, it was a distraction while reading the paper and it would be better to introduce PQI as early as possible.
clarity,_quality,_novelty_and_reproducibility: The introduced PQI is novel and the paper is clear and of high quality. I would encourage the authors to release the code to allow reproducibility.
summary_of_the_review: The introduced PQI index is novel and would be valuable to the community. The benefits are demonstrated in the experiments. However, the necessity of additional hyperparameters raises some questions about the practical value. 

## Post rebuttal
I thank the authors for the response. As mentioned by reviewer dVA5, the practical implementation of the method is cumbersome due to additional hyperparameters. Nevertheless, the introduced PQI has value and I'm keeping the marginal-accept score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this work, the authors proposed PQ Index (PQI) to measure the potential compressibility of deep neural networks and it is used to develop a Sparsity-informed Adaptive Pruning (SAP) algorithm.
strength_and_weaknesses: +The authors developed the PQ index, which satisfies the properties of an ideal sparsity measure.

+They further proposed the sparsity-informed adaptive pruning method based on the PQ index.

-It's unclear how SAP can reach a pre-defined pruning rate given some iterations since SAP is an adaptive pruning algorithm. Many hyperparameters are related to the pruning rate per iteration, like $\eta_r$, $\beta$, and $\gamma$.

-Authors mentioned that "pruning is based on the assumption that parameters with small magnitudes are removable." However, this is not always the case, many methods use other criteria to measure the importance of each weight besides its magnitude. 

-The paper didn't include large-scale datasets and models. As a result, the scalability of this method is not clear. Large models like ResNet-50 on ImageNet should also be added. The initial version of the lottery ticket can not work for large models, and the authors should show that SAP does not have the same problem. 
clarity,_quality,_novelty_and_reproducibility: Overall speaking, the paper is well-written and easy to follow. The idea of this paper is novel. The reproducibility should be good since no large datasets and models are included.
summary_of_the_review: This paper proposed a new sparsity measure: PQ index, and they developed SAP for pruning. However, some details of the SAP algorithm are missing, and there are no experiments for large-scale datasets and models.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces a norm based metric (PQ-Index) to quantify weight sparsity of a neural network. The metric is designed to satisfy the six properties proposed by earlier research in economy similar to the Gini Index. PQ-Index is then used to decide how much a neural network can be pruned at a given time during the training. Authors compare their method to Lottery-ticket style iterative pruning on CIFAR and Fashion-MNIST.

Strengths:
- Proposal of a grounded sparsity metric, which is novel and interesting
- The scope of ablation and results presented analyze the given method well.
- The authors proposed the sparsity-informed adaptive pruning method based on the PQ index.

Weakness:
- the practical implementation of the method is cumbersome due to additional hyperparameters.
- It's unclear how SAP can reach a pre-defined pruning rate given some iterations since SAP is an adaptive pruning algorithm. Many hyperparameters are related to the pruning rate per iteration-.
-Authors mentioned that "pruning is based on the assumption that parameters with small magnitudes are removable." However, this is not always the case, many methods use other criteria to measure the importance of each weight besides its magnitude.
-The paper didn't include large-scale datasets and models.


Requirements for the camera ready:
Experimental evaluation for proposed pruning method needs to be improved. We recommend authors to focus on showing the utility of PQI directly using the same model but different metrics (e.g. measuring exactly how close it gets to the max-prune rate). Similarly I think the fixed-pruning rate needs to be searched-over for the LT baseline (like trying P=0.2,0.1,0.3) just to make sure gains are not due to better rate average rate (first point in weakness section). Also please switch the log-scale for plots, as it is still very difficult to compare different curves at the moment.


justification_for_why_not_higher_score: Please see the weakness section of the meta review.
justification_for_why_not_lower_score: see strength section
Decision: Accept: poster