Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper highlights the fact that when doing unstructured pruning with extreme sparsity targets, we end up pruning entire neurons, which effectively detaches neurons from the input/output. The authors argue that the effective sparsity levels in these cases is higher than our target levels, and taking it into accounts exhibits differences between various pruning method that are otherwise not obvious. 
main_review: While indirect pruning is a reasonably known phenomenon, it is an interesting question to see whether it can give us any insights when comparing various pruning methods. Unfortunately, this paper doesn't do a thorough job doing that due to choice of baselines and poor motivation of extreme pruning regime:

- I am surprised to see LeNet-300-100 as baselines here. I do not want to be the "show me imagenet results" reviewer, but I feel the choice of MNIST, and LeNet architectures are not justified here. Indirect pruning happens in extreme pruning regimes and I am assuming we care about that for very, very large networks. LeNet-300-100 is tiny and does not have convolutions, so is this a good baseline to draw conclusions from?
- In the introduction, the authors do say that typical compression rates are 10x (0.9 pruning) and 100x (0.99 pruning) and motivate extreme pruning because of models with billions of parameters, yet their biggest models are ResNet-18 and VGG-19, where indirect pruning does not seem to be as big of a problem anyway.
- Overall, the authors should do a better job motivating extreme pruning regimes. Pruning is interesting because we think models are over-parameterized, and we can get "comparable performance" while pruning away a good portion of parameters. However, the performance in this extreme regime is very poor, and the architectures are simply "trainable". So why is this an interesting problem to look at?

**Structured pruning and CNNs**
- I think the authors should also clarify the link between indirect pruning and structured pruning. Isn't the phenomena described by the authors the intended behaviour in structured pruning? It would be great if authors could also compare with structured pruning methods but I think at least the link between the two should be discussed in the paper.
- The authors do not discuss what happens to the disconnected neurons as a result of indirect pruning. Can you set their value to zero (i.e. actually prune them)? Can their value always be fused into the next layer? or perhaps into the batch norm parameters? 
- The authors also don't discuss convolutional networks in details. I'm assuming indirect pruning is less of a problem there because you'd have to prune an entire channel to have indirect pruning. Given that most of our commonly used architectures and other pruning baselines use CNNs it seems crucial to have it analysed in detail and not rely on MLPs to make conclusions

**Layerwise Sparsity Quotas (LSQ) and allocation method IGQ**

I don't understand the Ideal Gas Law approach at all! What is the motivation for borrowing this from thermodynamics? What is the justification that neural networks behave like gasses? It is one thing to mention this in passing as an inspiration source, but when you go as far as mapping a neural network to a system in thermodynamic equilibrium, the height of a cylinder to the number of weights, and pruning to applying an external force to then you should provide better justifications. Moreover, the final derivation for prune ratio of each layer is $F | \Theta _l | + 1$. There are two things I don't understand about this: (1) How is $F$ related to the desired pruning target? The authors say $F$ can be found via a binary search from pruning target but do not elaborate. (2) No matter how F is found I'm assuming it's a global value and not a per-layer one. Doesn't this reduce the formula to simply applying pruning uniformly??

**Other things**
- The authors use VGG19. I have seen different variants of VGG-19 for non-imagenet datasets. In Some variants there's a single dense layer after convolutional layers, and in some other variants there are multiple dense layers. Could the authors confirm which architecture they've used? This makes a big difference in prunability of the network and I'm curious to know if the indirect pruning is mostly happening in the final dense layers or in convolutions as well.
- In the discussion section the authors argue that "effective compression" is the "correct" measure for pruning algorithm. I think that's a strong claim that's not well supported in the paper. You should at least clarify that it's aimed at extreme pruning regime (and probably non CNN based architectures?
- The caption in Figure 2 says "SynFlow has a better sparsity-accuracy tradeoff than SNIP" Isn't the opposite true? I guess it's a typo
- In figure 3, would things look too weird if you use the same units in the x axis and y axis so that we can compare the effect across architectures more easily?
- In figure 5 could you rename "Random" to "SNIP + RandomReshuffle" or something. I think random is misleading here given that you're still applying layer-wise pruning ratios.
summary_of_the_review: This paper draws too many conclusions based on very small networks, datasets, and for a pruning regime where the network is simply trainable and has nowhere near the performance of the unpruned model. Overall I'm not convinced that "effective pruning", and extreme pruning gives us a enough insight to justify accepting the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper raises a warning signal on an aspect which is typically ignored in sparse neural networks, i.e., the “unconnected” connections (the ones which do not belong to the paths connecting the input to the output neurons). Consequently, the paper studies the so-called “effective sparsity”, to formally understand the efficiency of various pruning methods to obtain sparse neural networks while considering the connections which contribute to the neural network inference (the ones which do belong to the paths connecting the input to the output).  Complementary, the paper proposes some (minor) contributions to cope with effective sparsity. Overall, the paper is well written and has well-designed experiments. 


main_review: **Strong points:**

- From my experience, I recognize this problem of “effective sparsity” and, up to my best knowledge, this is the first time when I see it so well formulated. Thus, the paper was a pleasure to read, providing useful insights in understanding sparse neural networks and pruning techniques.

- Few small innovations are introduced to cope with the concept of “effective sparsity” in order to improve the trade-off model size/accuracy

- The code is provided for easy reproducibility

**Weak points and suggestions for improvement:**

- The paper contributions are hard to derive from the actual text. I suggest adding somewhere in the Introduction section, few short and concise bullet points to present the paper contributions and to sharply discuss what is novel with respect to the literature.

- The Related Work section contains just a categorization of various pruning techniques. I suggest to add an extra paragraph which clearly discuss the related work with respect to the “effective sparsity” concept.

- The experiments are performed just on dense-to-sparse training methods. I suggest considering also sparse-to-sparse training methods with prune and grow strategies. I agree with the authors statement from the Discussion section that it would be more difficult to incorporate their proposed effective pruning strategy in sparse training. This incorporation can be indeed let for future work, but just studying effective sparsity in sparse training should not be so difficult and the results can be of interest (e.g., same as in Figure 3). 

- I understand the ERK baseline for the proposed IGQ method but decoupling ERK from sparse training (as it was originally designed) may lead to misleading results interpretation. Nothing to change in the experimental section, but I suggest clarifying (discussing better) this aspect.

- It is not very clear to me if the paper suggests that the connectivity pattern itself is also quite important in ensuring a good trade-off model size/ performance and not just the sparsity distribution. A discussion has been started on this topic in the last paragraph of page 2, but I would expect the Discussion section to come back to this topic and perhaps some empirical validation would be necessarily to support better the statement “…but find the truth to be more nuanced at higher compression rates…”. I wouldn’t expect a detailed study as it is outside of the scope of this paper, but rather some hints in order to avoid cutting from start future works on this topic. 

summary_of_the_review: Overall, I believe that this is a well-written paper, with a clear message, which also has some space for improvement.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper points out that in pruning, there is often a difference between the number of zero parameters and the number effective zero parameters due to disconnection during pruning. A thorough analysis is done comparing many pruning methods across MNIST, Cifar-10, and TinyImageNet using various architectures. The authors introduce a new physics inspired layerwise sparsity quota  baseline the "Ideal Gas quota" which sets a sparsity per layer. 
main_review: Strengths:
* The paper is clear and presents an interesting analysis about the discrepancy between pruning models and the resulting, underlying graph.
* The detail and analysis is thorough, and the methodology is clear and concise. 
* The presented layer-wise sparsity quota Ideal Gas Quotas (IGQ) are interesting and produce compression distributions very similar to other methods (such as in Fig. 9). 
Weaknesses:
* Many important points/figures in the paper are on very small datasets where models are extremely over-paramaterised. For example, Fig. 2 on MNIST would be much stronger, I think, on a slightly larger dataset (CIFAR-10, perhaps?)
* Overall, I found the results largely focused on models that do not reach competitive performance on larger datasets. ResNet18 on TinyImageNet is shown, but it'd be nice if more modern architectures were shown. 

Comments:
* Would it be possible to adjust for effective FLOPs in a correctly pruned graph?
* It would be cool to investigate various choices and their impact on sparsity/effective sparsity during pruning. For example, I imagine models with residual connections will have slightly different dynamics. 
* I'd be interested in a bit of a comparison of different model architectures-- for example, sparsity in convolutional filters will have a different impact in the effective sparsity metric than sparsity in MLPs (if I understand correctly). Therefore, there will be some differences between architectures that allocate parameters differently (and what is pruned). 
* Showing how some of these methods look in another domain such as NLP could be interesting. 

Minor comments: 
* I found the caption to Figure 3 slightly confusing. 
summary_of_the_review: The paper is well written, very clear, and well presented. The authors raise and clearly document the discrepancy between raw sparsity and the actual effective sparsity of a model. An interesting and compelling new baseline is presented.
However, the results are shown on small datasets without comparison with many modern architectures. 
Replacing the MNIST/LeNet/VGG results with more modern architectures would clearly increase the impact of the paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper highlights the difference between the explicit and effective (through disconnection) number of zeroed parameters in a model --- and argues it is the effective parameters that should be used as the benchmark for comparison.
Further comparing pruning methods and re-affirming their performance relative to random pruning equivalents. Finally introducing a layer allocation method which can serve as a better performing random baseline.

main_review: 
Overview:

The issue of effectively disconnected parameters [Tenaka et. al.], while known, beckons the proper methodological accounting of parameters --- to those 'effectively' contributing. This paper describes this discrepancy and quantifies it at small scale.
While disconnections are known, and to this reviewer at least.--- known to be quantified by practitioners --- such quantification is indeed poorly documented and this paper eloquently captures it. Beyond the methodological importance of such explicit clarification, the practical relevance and expectation of their impact could use further discussion and motivation.

Major Pros:
1. The issue of proper methodology in the context of counting of parameters is important and by extension so is accounting for effectively defunct parameters due to disconnection. 
2. The paper is well written and clear.

Major Cons:
1. This discrepancy (between effective and direct parameter count) is known (as the authors mention [e.g. Tenaka et al]) and thus the major contribution of this paper is of limited novelty.

2. While the actual methodological claim (one should account for disconnections via effective counting) is very much valid and orthogonal to experimental settings, the practical discrepancy (and thus implications) between direct/effective is very much dependent on the settings:

2.a. As mentioned by the authors (and demonstrated in figure 3), for the iterative pruning of Syntflow and SNIP-iter, no disconnection associated discrepancy appears. Tanaka et al single out the general role of the Iterative nature in upholding continuity even in methods which are not explicitly designed to preserve continuity.
In light of such an observation --- for SOTA, such as Iterative Magnitude Pruning (IMP) , it is not clear that in practice a large discrepancy would be present even up to higher than practically employed sparsity levels. 

2.b. Further, in terms of architectures, probability of disconnection in modern architectures --- and especially those containing residual connections --- is much reduced. Indeed, in the Resnet experiments (figure 4) there seems to be a relatively small difference between effective and direct. As such, for practical architectures --- the level of actual discrepancy in practice is not demonstrated to be (or as expected to be) meaningful. 

2.c. The paper demonstrated the results for small datasets (and weak networks) (mnist/lenet , VGG/cifar10-100, Resnet18/TinyImagenet). 
The lack of practical large scale (Imagenet) and SOTA architectures limits the conclusions to the applicability.


summary_of_the_review: The paper eloquently presents and quantifies --- in the small scale --- the known issue of disconnections as affecting parameter count and pruning methods [Tenaka et al]. The paper can further engage with practical implications. 
I agree that counting effective rather than direct parameters is methogologically good practice. However, the limited scale and demonstration  limits the insight applicability in practice --- especially, as SOTA methods and architectures have structural characteristics which call into question the the level of discrepancy (missing discussion/dimensions of exploration).
review_title: null

Meta-review: ### Summary

This work investigates effective sparsity: an assessment of the sparsity of pruned networks that accounts for the fact that unpruned neurons can still be completely disconnected through pruning.  Hence, the effective sparsity of a network may be much lower than otherwise reported.

### Discussion

#### Strengths

- The paper studies an important metric that deserves additional attention in the community, where a change in metric may guide either the theory or practice of pruning.

- The paper evaluates direct versus empirical sparsity for a healthy number of pruning techniques.


#### Weaknesess 

- While this paper appears to be the most direct study of effective sparsity at the moment, it is not the first. Appendix M of [1] defines effective sparsity and shows that direct and effective sparsity are similar for contemporary pruning at initialization techniques. However, that work does not evaluate random pruning. This work here will need to revise its novelty claims to account for these results as its characterization that [1] only considers direct sparsity is incorrect.


- "Computing effective sparsity:" the procedure in question is similar to that of Appendix M in [1], thus its relationship should be detailed.


- With the primary observations residing in the regime of extremely sparse neural networks, the elements of the response (and in the last paragraph of the paper) that claim this regime is productive for ensembling should make a more prominent appearance in the introduction of the work.

[1] Pruning Neural Networks at Initialization: Why Are We Missing the Mark? 
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, Michael Carbin. ICLR, 21


### Recommendation

I recommend Reject. Generally, the paper is well-written and the empirical characterization of direct versus effective sparsity is thorough (except for ResNet-50 results). However, the results and the language around these results need significant rescoping to account for novelty and the relation of the work to an area in which it is anticipated that these results will change theory, practice, or thinking (e.g., ensembling).

Though I cannot speak for future reviewers, IMO, an extension of the results here to ResNet-50+ImageNet should suffice to establish the extent of the discrepancy between direct and effective sparsity. However, to satisfy additional demands from reviewers for more practical relevance, I suggest an evaluation that demonstrates a consequential difference in behavior for a task that maps more closely to the anticipated area of impact (e.g., ensembling)
Decision: Reject