Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: *Summary*
This paper compares network pruning masks learned via different iterative pruning methods. Experiments on LeNet + MNIST show (a) different methods can achieve similar accuracy, (b) pruned sub-networks may differ significantly despite identical initialization, (c) weight reinitialization between pruning iterations yields more structured convolutional layer pruning than not reinitializing, and (d) pruning methods may differ in the stability of weights over pruning iterations.

*Rating*
There are interesting bits of data in this paper, but the overall story is somewhat muddled and some inferences seem to be insufficiently supported by data (1-2 below). In addition, the text would benefit from better organization and presentation (3-4 below) and replications on other datasets and architectures (5 below). As a result, my rating is currently weak reject.

(1) *Overlap in pruned sub-networks*: In the middle of Sec. 4, Fig 3-5 examine the similarity of pruning masks between methods. It seems clear from several of the plots that multiple methods produce identical layer-wise masks, e.g. Fig 3(a), while others show a wide variance. The overlap in lines makes this difficult to assess at times: perhaps a table would communicate it better? Also, are Fig 3-4 depicting the Jaccard distance between masks of unpruned or pruned weights? Is the ordering of training samples fixed in addition to network initialization? Is reinitialization used between iterations? Also, Fig 5 seems to contradict the conclusion that methods tend to learn different masks, since the structures are noticeably similar.

(2) *Weight stability during pruning*: It is difficult to discern a conclusion in Sec 5. First, a clarification on the figures: are lines for pruned weights terminated where they are pruned? If so, this would be helpful to state. The 4th paragraph claims, "we empirically find a correlation between weight stability and performance", but this is not at all obvious from Figures 6-7. I'm not sure what a more stable evolution looks like. Hybrid is shown to be accurate in Fig 1, but the conv. weights in 6(a) are a spaghetti tangle and the FC weights in 7(a) are constantly increasing in magnitude. Perhaps a mathematical formulation for stability (perhaps based on average standard deviation of each weight's values over training) with a table of values for each method/layer would help to clarify.

(3) *Organization*: Since the paper has many intertwined observations, a better organization would be helpful. Consider mirroring the structure of Sec 1.1 in a combined Sec. 4-5 with clear paragraph headers summarizing each conclusion.

(4) *Presentation*: Figure is too small throughout to read from a printed copy (or even on a screen without significant zooming). Several results could be presented with less ambiguity in tabular form, as noted above.

(5) *Replications*: The paper presents results only a single set of experiments using the MNIST dataset with the LeNet architecture. While this isn't a fatal issue, it is a significant weakness.

*Notes*
Fig 1 and 2: What spacing is used for the x- and y- axes?
Fig 8: Perhaps scale vertically by the standard deviation of the weights?
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper study the lottery ticket hypothesis by observing the properties of lottery tickets. In particular, the authors tested several different pruning techniques by varying evaluation criteria (L_1, L_2, L_-\infty and random) and pruning structures (structured, unstructured and hybrid). The authors perform experiments mainly on LeNet with the MNIST dataset and analyze the observations.

Overall, I think that the observations presented in the paper are not significant due to the following reasons.

First, the paper consists of the list of observations but how the observations extend to is not clearly described. There are no guidelines how to utilize the observations in future research (e.g., how they can be used for verifying the lottery ticket hypothesis or how they affect to existing pruning techniques) while some observations might be trivial or not very interesting (e.g., contribution 1 and contribution 2) for me.

Second, the observations are only presented for LeNet and MNIST and it is non-trivial whether they extend to large scale models. The authors present VGG11 and AlexNet results in Appendix but they are not large enough to verify their hypothesis for practice. The authors mentioned that larger models are not their subject, but this significantly reduces the confidence of the observations.

Other comments:
I think that Figure 5 is not well described. Explicitly noting the meaning of color in the figure would be better.

Texts in Figure 7 are too small to read.

review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: There are major problems with this paper. It is concerned with the examination of pruning experiments for a LeNet on the MNIST dataset.  I fail to see how anything useful can be derived from this, as MNIST is a completely trivial dataset and LeNet is a very old, small architecture which does not at all resemble the massive overparameterised models that we care about.

From a narrative perspective, I am not sure what the key point is, what should the reader take home? What should they take account of when performing network pruning?

In terms of presentation, some of the figures are unreadable (figure 4). Figure 15 looks like noise. The writing is good however, if a bit grandiloquent.

I dislike writing short reviews, but I fear this paper falls too far short of ICLR standard.

Pros:
- Well written

Cons:
- Experiments are weak
- Unclear narrative; what's the one key message?

I have to give this paper a reject as the experiments conducted are far too weak, and there is little evidence anything found here will, say, generalise to a ResNet/DenseNet on ImageNet.


review_title: Official Blind Review #1

Meta-review: This is an observational work with experiments for comparing iterative pruning methods.

I agree with the main concerns of all reviewers:

(a) Experimental setups are of too small-scale or with easy datasets, so hard to believe they would generalize for other settings, e.g., large-scale residual networks. This aspect is very important as this is an observational paper.
(b) The main take-home contribution/message is weak considering the high-standard of ICLR.

Hence, I recommend rejection. 

I would encourage the authors to consider the above concerns as it could yield a valuable contribution.
Decision: Reject