Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: The paper investigates methods to train neural networks so the final network has sparse weights, both in convolutional layers and in fully connected layers. In particular, the paper focuses on modifying the training so that the network is first trained without sparsification for a certain number of epochs, then trained to be increasingly sparse, and then fine-tuned with a fixed sparsity pattern at the end.

While I find the overall approach of the paper interesting, currently the experiments are not systematic enough to derive clear insights from the paper. Hence I unfortunately recommend rejecting the paper at this point. I hope the authors find time to conduct more systematic experiments for a future version of the paper.

Concretely, the following would be interesting experiments / questions:

- How effective is the proposed training method on architectures other than ResNets?

- What happens if the "pruning era" is made longer, started substantially earlier, or started substantially later? Currently it is not clear if the epoch 30 - 50 pruning era is (approximately) optimal and how much performance varies with begin and end of the pruning era.

- Due to the small variation between some of the methods, it would be good to investigate how robust the ordering is when the experiment is re-ran with different random seeds etc.


In addition, I have the following suggestions:

- The authors may want to remove or enhance the adversarial robustness evaluation. Currently the authors only evaluate robustness against FGSM, but it is well known that iterative attacks such as PGD are more effective.

- Instead of "intra-epoch pruning" or "intra", the name "combined" may be more clear for the combined method.

- In the description of the experimental setup, it could be good to specify what GPUs were used (since this lead to the smaller batch size).

- It could be helpful for the reader to discuss how predictive results on Tiny-ImageNet are for results on ImageNet.

- In Table 2, it would be good to add context by comparing to prior work with sparsity level 60% and some of the compression-focused methods from Table 4.

- In the comparison to Mao et al. (2017), it could be good to clarify that they also work with ResNet models on ImageNet.

review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: SUMMARY
-------

This paper explores a series of incremental variations of existing pruning techniques for compressing Resnet-50 for ImageNet. Specifically, it proposes concentrating all pruning during an early "era" of training (the first 20-50 epochs out of 100 total). It also explores hybrids between sparse pruning and structured pruning. Finally, it considers the adversarial robustness of the resulting networks to the FGSM attack. 

This paper makes no novel proposals and experiments are minimal. There are no clear takeaways from the results of these experiments. The goals of the paper are unclear, and it is difficult to compare this paper to existing work. 

This paper has no clear motivation and makes no tangible contributions to the literature and, therefore, I recommend a rejection. 

CONTRIBUTIONS
-------------

1) A study of the appropriate window ("pruning era") for pruning Resnet-50 on ImageNet and TinyImageNet
2) A study of the tradeoffs between various forms of structured and unstructured pruning.
3) An analysis of the adversarial robustness of the pruned networks.


DETAILED COMMENTS
------

PROBLEMS ADDRESSED

It was challenging to discern the specific problems that this paper sought to address and, relatedly, the goals that the paper sought to achieve. The introduction of the paper lists a wide variety of problems in the existing literature:

1) Paragraph 3: Structured sparsity introduces "regularization and computational overhead."
2) Paragraph 3: "Coarse-grained sparsity" cannot eliminate enough parameters to perform well on "edge devices."
3) Paragraph 4: Dynamic sparsity techniques require more training epochs.
4) Paragraph 4: Dynamic sparsity techniques do not preserve network accuracy (1-2 percentage point drop at 80% sparsity).
5) Paragraph 4: Dynamic sparsity requires reconfiguring the sparsity pattern frequently, which is computationally expensive.

The paper does not justify the fact that any of these are actually problems, nor does it make any attempt to quantify the extent of these problems. Moreover, the proposed techniques do not resolve any of these problems. Corresponding to the numbers above:

1) The paper never measures this overhead nor justifies that it is a problem in practice. Meanwhile, the techniques proposed in the paper introduce substantial overhad of their own, including training for an extra ten epochs. It is possible that the techniques proposed in this paper have worse overhead than the techniques that are criticized in the introduction. Since the paper provides on numbers either way, it is impossible to tell. In short, computational cost is a key part of the author's argument despite the fact that there is no empirical support for any of these claims.

2) I believe the paper means that, in order to get to sufficient levels of sparsity to work on "edge devices," accuracy drops unacceptably far. What does the paper mean by "edge devices," what are sufficient levels of sparsity, and what does it mean for accuracy to drop unacceptably far? The paper has numbers for the proposed methods, so it should be possible to make this comparison if such baselines are explicit.

3) The proposed techniques also require the same number of additional training epochs, so this complaint is unaddressed.

4) The proposed techniques show a 2-3 percentage point drop at 80% sparsity (Table 1), which is actually worse than the technique that the authors criticize.

5) The proposed techniques require pruning after every single training step during the "pruning era." This is likely to be more computationally expensive than any of the other gradual pruning and dynamic sparsity techniques listed, which prune at intervals of hundreds or thousands of iterations. In addition, the authors never justify why changing the sparsity pattern frequently throughout training will affect performance. On GPUs with modern frameworks, I see no reason why this should matter so long as the sparsity pattern does not change too frequently (although that is exactly what this paper proposes to do during the "pruning era").


GOALS

It was also challenging to discern the goals of the paper. Was it:

1) To produce the smallest possible trained networks with the highest possible accuracy?

2) To reduce the cost of obtaining a pruned network for inference-time? (Or to reduce the cost of obtaining a sufficiently efficient pruned network for inference-time?)

3) To reduce the cost of training neural networks in general by pruning them during training?

In the introduction and the related work section, these goals go unstated, making it difficult to determine how this paper compares to existing work. The comparisons provided in the paper focus on specific aspects of each related work rather than the entire picture. For example, in comparison to Mao et al., the authors claim better accuracy at one sparsity level, implying goal 1. However, for to Lym et al., the paper focuses on the computational costs of training the network, implying goal 3.


UNJUSTIFIED CLAIMS ABOUT NEURAL NETWORK COMPUTATION

Throughout the paper, there are a number of unjustified claims about which neural network configurations will perform better on contemporary hardware. Considering computational efficiency appears to be a key element of the paper's argument, these claims require citations or - particularly when various configurations are compared to one another - empirical support. Some examples:

* Section 1, Paragraph 3: "The regularization term [of structured sparsity] modifies the original training and can be expensive in hardware."
* Section 1, Paragraph 3: "The final network [from Lym et al. 2019] contains an insufficient degree of sparsity for deployment on edge devices."
* Section 1, Paragraph 4: "Continuous reconfiguration of the sparsity pattern is expensive as it does not allow for compression of weights during training"
* Section 1, Paragraph 5: "having a fixed sparse multiply-accumulate pattern allows weight compression during training and can save compute and energy in hardware"
* Section 5, Paragraph 2: "A strict parameter allows the hardware mapping to plan for a fixed number of multiply-accumulate operations."
* Section 5, Paragraph 2: "Regularization, although useful in forcing the network to learn prunable weights, adds more irregularity to computation flow."


PRUNING TECHNIQUES

* Recomputing the pruning mask at every training step seems gratuitously inefficient.
* Sorting the weights in the entire network shouldn't be particularly inefficient if it isn't done on every single iteration. (Section 2.1 paragraph 1)
* Why do you maintain the same number of weights in each convolutional filter with window pruning? (Presumably for performance reasons, but you never say that.)
* None of the pruning methods are novel. They're simply various permutations of structured and unstructured magnitude pruning as proposed by many others in the literature.


EXPERIMENTS

* Section 3.1 Paragraph 2: It appears that you are exploring the best "pruning era." If you are to do so, you will have to sweep over (1) the length of the pruning era (2) the starting epoch of the pruning era, and (3) the shape of the function used to determine sparsity. Instead, it sounds like you tried two arbitrary pruning eras (0-30 and 30-50). Likewise, in Paragraph 3, you test only a small number of possible scenarios.
* Section 3.1 is generally hard to parse. It is unclear what you are studying. The ideal pruning era? The relative performance of the pruning methods introduced in section 2? 
* How many times did you replicate each experiment? You should ideally include at least 3 (and preferrably 5) replicates with mean and stddev reported.
* What baselines are you including? You should include a random pruning baseline and you should ideally replicate any methods that you compare to.


RESULTS

* Section 4.1: The data you refer to is in an appendix even though it is crucial to the main body of the paper. The appendices should contain material that is nonessential for making sense of the paper.
* Section 4.2 Paragraph 1: Are these numbers good? A standard sparse pruning technique (Gale et al. 2019, https://arxiv.org/pdf/1902.09574.pdf) achieve 70% sparsity without any change in accuracy. Please include baselines comparing to other methods in the literature.
* Table 2: It is difficult to compare the results in these papers. PruneTrain aims to reduce the cost of training and measures cost reductions in FLOPS. If you intend to compare against this paper, you should quantify the cost of training using your method against that of PruneTrain. Merely presenting sparsity and accuracy numbers is insufficient. Likewise for the dynamic sparsity experiments. What is your goal in showing this comparison, and did Mostafa and Wang share that goal when they justified their technique?
* You do not describe the hyperparameters for Intraepoch pruning (the balance between window and CK - last paragraph of 2.1.1)


ADVERSARIAL ROBUSTNESS

Considering the fact that this paper focuses on proposing new variations of existing pruning techniques, any discussion of adversarial robustness seems to be (1) out of place and (2) an afterthought. If the authors delete a half-page of content (one phrase from the abstract, a paragraph and bullet from the introduction, and a paragraph each from sections 3 and 4), this content could be removed with minimal impact to the paper's main contributions. The content on adversarial robustness is cursory, uses a weak and out-of-date attack (FGSM), and does not compare to any other pruning methods. In fact, the one comparison is to the results in a paper (Wang et al, 2018) that looks at both FGSM and PGD (a stronger attack) on completely different networks and tasks (MNIST and CIFAR10). The paper would be stronger if content on adversarial robustness was removed entirely.


OTHER MINOR COMMENTS

* The title includes the word "starfire," but it never appears again in the paper. The paper proposes no specific technique, so there isn't anything to name.
* Use the \ begin{appendix} command before you create the appendices and the \ end{appendix} command when you are done. You can then use \section normally and each section so-created will appear with a letter rather than a number.
* Figure 4 is very hard to read.
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: This paper introduces a strategy to prune a convolutional neural network during training. To speed up training, the proposed method prunes the weights with the smallest magnitude during only a small number of epochs at the beginning of training, later on continuing training with a fixed sparsity pattern. Several granularity levels for convolutional and fully-connected layers are studied. Furthermore, the robustness of the resulting pruned networks to adversarial attacks is investigated.

Originality:
- As acknowledged at the beginning of section two, the general pruning strategy used here is very similar to that introduced by Narang et al., 2017. While the authors argued that the threshold is computed in a different manner, it also increases gradually during training, as in Narang et al., 2017.
- I acknowledge that Narang et al., 2017 focuses on RNNs, while here the focus is on CNNs. However, the originality of the different pruning strategies used here for convolutional and fully-connected layers is very limited. In essence, these strategies directly follow those studied by Mao et al., 2017.
- The study of robustness to adversarial attacks, while interesting, is also not novel per se, as the idea of performing such a study was proposed in Wang et al.,  2018. I acknowledge that the conclusions drawn here differ from those in Wang et al., 2018. However, there are no explanations for this different behavior.

Methodology:
- While the beginning of Section 2 states that the pruning threshold gradually increases during training, the specific way this is achieved is not clearly explained.
- The pruning strategies depicted by Fig. 2, whether for convolutional layers or for fully-connected ones, never aim to remove entire output channels. However, the only way to truly end up with a smaller network is to remove entire channels and/or layers, as argued in Wen et al., 2016 and in Alvarez & Salzmann, NIPS 2016, as well as studied in Mao et al., 2017 via the filter-level granularity. It is unclear to me how speed would be affected by having a network with the same number of channels and layers, but many parameters set to zero. 

Experiments:
- The experiments show the good behavior of the proposed algorithm in terms of sparsity vs accuracy tradeoff. However, while the introduction seems to focus on the benefits of the proposed method in terms of training speed, these benefits are not demonstrated in the experiments, where no timings (neither for training not for inference) are reported.
- As mentioned above, it is not clear to me that the speedup will be significant if the sparsity pattern does not remove entire channels, but I am willing to be proven wrong.

Summary:
My main concern about this paper is its novelty, as the method essentially uses the method of Narang et al., 2017, albeit with a different threshold, with the sparsity patterns of Mao et al., 2017. The experiments demonstrate that the method is effective at pruning, but do not provide any timings to evaluate the resulting speedups.
review_title: Official Blind Review #3

Meta-review: This paper concerns a training procedure for neural networks which results in sparse connectivity in the final resulting network, consisting of an "early era" of training in which pruning takes place, followed by fixed connectivity training thereafter, and a study of tradeoffs inherent in various approaches to structured and unstructured pruning, and an investigation of adversarial robustness of pruned networks.

While some reviewers found the general approach interesting, all reviewers were critical of the lack of novelty, clarity and empirical rigour. R2 in particular raised concerns about the motivation, evaluation of computational savings (that FLOPS should be measured directly), and felt that the discussion of adversarial robustness was out of place and "an afterthought".

Reviewers were unconvinced by rebuttals, and no attempts were made at improving the paper (additional experiments were promised, but not delivered). I therefore recommend rejection. 
Decision: Reject