Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a novel one-shot-pruning algorithm which improves the training of sparse networks by maximizing the norm of the gradient at initialization. The utility of training sparse neural networks and shortcomings of dense-to-sparse algorithms like Pruning, LotteryTicket are nicely motivated at introduction. The pruning criterion is motivated by the first order approximation of the change in the gradient norm when a single connection is removed, though the results show that removing many connections together with GraSP increases the total gradient norm therefore allowing the loss to decrease faster. Experiments suggest employing such pruning algorithm improves final performance over two baselines: random and SNIP. 

Though I find the proposed method intriguing and well motivated, experiments section of the paper misses some important sparse training baselines and needs some improvement. I am willing to increase my score given my concerns/questions below are addressed. 

(1) The paper doesn't mention some important prior work on the topic. Since the paper focuses on end-to-end sparse training, the following sparse training methods needs to be considered and compared with:
- Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science [Mocanu, 2018]
- Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization [Mostafa, 2019]
- Deep Rewiring: Training very sparse deep networks [Bellec, 2017] 
- There is also few recent work submitted to ICLR2020: https://openreview.net/forum?id=SJlbGJrtDB, https://openreview.net/forum?id=ryg7vA4tPB, https://openreview.net/forum?id=ByeSYa4KPS

(2) Pruning baselines can be improved. I am not convinced that they represent the best achievable sparse training results. I would recommend method proposed by `Prune or Not to Prune` as a strong baseline. You can also check `State of Sparsity` paper to obtain some competitive pruning results.

(3) It's great that the authors are aware of the importance of having experiments on larger datasets. Though, I found the results reported on Imagenet to be limited. Is there a reason why Imagenet-2012 results are missing pruning baselines? I think having other reported pruning results here along with performance of other sparse training methods (SET, DSR) would be useful. Most of these numbers should be readily available in the papers mentioned above, but I guess it is always better to run them using the same settings. 

(4) To demonstrate the usefulness of the pruning criteria proposed, it would be nice to do some simple ablations. Some suggestions: (1) Remove weights that would *decrease* the gradient norm most (2) Do random pruning while preserving exact per layer sparsity fractions. (3) sweep over batch size used to calculate the importance scores and evaluate final accuracies or the initial gradient norm. The second experiment would help identifying whether the gains are due to better allocation of sparsities across layers or due to increased gradient norm. Looking at Figure-4 and seeing the per layer sparsities are different, It is not clear to me which one is the underlying reason for improved performance.

(5) (Page 8 / Table 5) Do you aggregate all accuracies in Table 5 using different batch sizes and initialization methods? If, so I am not sure what the intended message here is, since it is difficult to infer how these hyper-parameters affect the result. Do you sweep different batch sizes for estimating importance of units, too? It would be nice to see whether the two batch sizes interact with each other and/or how increased batch size affects the quality of pruned networks. 

Some minor comments:
(a) (Page 1) I found the motivation very intriguing. Though the statement `Recently, F&C (2019) shed light on this...` seems a bit off, given that LT can't find solutions as well as the pruning solution in most practical (larger datasets and architectures) settings. Therefore I would be better to pose this as an `open problem`. 

(b) (end of page-1) `However, connection sensitivity is sub-optimal as a criterion because the gradient of each weight might change dramatically after pruning due to complicated interactions between weights`. I think this is still the case for GraSP. Since the criterion it uses assumes independence (i.e. what if we remove a single weight?). It would be nice to see some ablations on this. Does `K=number of weights removed` affect the norm of the sparsified networks?

(c) (Figure 1) I find the comparative illustration between SNIP and GraSP very useful. Though, the architecture presented seems a bit artificial (i.e. I am not aware of any architecture with single hidden layer and a single output unit). I think the same motivation can be made by removing the top unit (therefore having 6-4-1 units) and removing all incoming connections for the output unit until a single connection remains. Then SNIP would remove that single connection whereas GraSP would remove one of the connections in the previous layer.

(d) (Section 2.1) `In contrast, Hessian based algorithms...` Though it is a structured pruning algorithm It might be nice to include the following work, https://arxiv.org/abs/1611.06440. 

(e) (Section 2.1) Previous work needs following citations:  [Bellec, 2017], [Mocanu, 2018] and [Mostafa, 2019] 

(f) (Section 2.2) Why the initial dynamics affect the final performance? One explanation given in the paper is through recent work on NTK and this is great. Though training settings used at `Lee et al (2019a)` and in the paper are a bit different. Usage of MSE, small datasets, etc… So it might be nice to point out differences. 

(g) (Section 3) At $D = {(x_i, y_i)}_{i=1}^n$, `n`->`N` 

(h) (Page 4) `Preserving the loss value motivated several…` -> `motivated by several…`
I think it is better to use existing terminology whenever available.I think using `One-shot pruning` instead of `Foresight pruning` would be a better choice and would prevent confusion. 

(j) (Page 5) `However, it has been observed that different weights are highly coupled …` This has been observed much earlier, too: like in Hassibi, 1993. 

(k) (Page 7) Last sentence `and thus hopefully..`: needs to be fixed.

(l) (Page 8) The whole page needs some proof-reading. Some of them: (a) `SNIP and GraSP. We present...` probably connect with comma (b) `aims for preserving` -> `aims to preserve` (c) `In contrast, SNIP are more` `are`->`is` (d) `for ablation study` -> `as an ablation study`... 

(m) Is there a specific reason why VGG networks are preferred for experiments? I don't think they are relevant to any practical application anymore and they are massively over-parameterized for tasks in hand. Specifically for Cifar-10. I think focusing on more recent networks and larger datasets would increase the impact of the work. 
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This paper introduces a method to prune networks at initialization in a way that (mostly) preserves the gradient flow through the resulting pruned network. This is a direct improvement over previous methods (e.g. SNIP) which have no guarantees that pruned connections will break the gradient flow and thereby harm learning.

I quite like this paper, the motivation and results are convincing and it is well presented. The writing is excellent for most of the paper. From section 5 onwards the writing does need quite a bit of editing, as its quality is significantly reduced from what came before.

Some detailed comments:
- Figure 1 is very nice and really clarifies the idea!
- In paragraph below Equation (8): what does "can be computed by backward twice" mean?
- Please specify where the equalities in equation (9) are coming from.
- Table 3 & 4: Why are the pruning ratios different for each model?
- Table 3: Why are values missing for the baseline for 80% and 90%?
- Section 5.2: "We observed that, the main bottleneck or pruned... when deriving the pruning criteria": it's not clear where this conclusion is coming from.
- Table 5 has no batch size results, even though you're referencing them in the text.

And some minor comments to help with the writing:
- Intro: "As shown in Dey et al. (2019) that with pre-specified sparsity, they can achieve" would read better as "As shown by Dey et al. (2019), with pre-specified sparsity one can achieve"
- Equation (3): Clarify that this is a function of $t$
- Sentence below Equation (6): "of the pruned network, and thus our goal" remove the "and thus"
- Table 1: Specify that you're reporting accuracy.
- Section 4.1: "e.g. wide ResNet (Zagaruyko & Komodakis, 2016), and thus we can regard" remove the "and thus"
- Sentence below equation (9): "encouraging the eigenspace of \Theta align" add a "to" before "align"
- Sentence before section 5: "it will encourage the eigenspace of the NTK distributing large eigenvalues in the direction of Y, which will in turn accelerates the decrease of the loss (Arora et al., 2019) and benefits to the optimization in A" would read better as "it will encourage the eigenspace of the NTK to distribute large eigenvalues in the direction of Y, which in turn accelerates the decrease of the loss (Arora et al., 2019) and benefits the optimization in A"
- Throughout section 5, write it in present tense rather than past tense. e.g. "In this section, we conduct various experiments" instead of "In this section, we conducted various experiments"
- Sentence below table 2: you have "the the"
- Second paragraph of section 5.1: "We can observe GraSP outperform random pruning clearly" would read better as "We can observe GraSP clearly outperforms random pruning"
- Second paragraph of section 5.1: "In the next, we further compared" remove "In the next"
- Second paragraph of section 5.1: "Besides, we further experimented with the late resetting" remove "Besides"
- Paragraph above section 5.2: "GraSP surpassing SNIP" use "surpasses" instead
- Paragraph above section 5.2: "investigate the reasons behind in Section 5.2 for promoting better understanding" would read better as "investigate the reasons behind this in Section 5.2 for obtaining a better understanding"
- Section 5.2: "We observed that, the main bottleneck" -> "We observe that the main bottleneck"
- Section 5.2: "Besides, we also plotted the the gradient norm of the pruned", remove "Besides" and the extra "the"
- Section 5.2: "the average of the gradients of the entire dataset" use "over the entire dataset"
- Section 5.2: "hopefully more training progress can make as evidenced" would read better as "hopefully more training progress can be made as evidenced"
- Section 5.3 title would be better using "Visualizing" instead of "Visualize"
- Section 5.3: Join the first two sentences with a comma into a single sentence.
- Section 5.3: "In contrast, SNIP are more likely" -> In contrast, SNIP is more likely"
- Section 5.4: "for ablation study" would read better as "via ablations"
- Section 5.4: "we tested GraSP with three different initialization methods;" use a ":" instead of ";"
- Section 6: "Besides, readers may notice that", remove the "Besides"
- Section 6: "traditional pruning algorithms while still enjoy the cheaper training cost. As an evidence," would read better as "traditional pruning algorithms while still enjoying cheaper training costs. As evidence,"
- Your citation for Evci et al. (2019) is missing the publication venue/arxiv ID.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: 
The paper proposes a new prunning criterion that performs better than Single-shot Network Pruning (SNIP) in prunning a network at the initalization. This is an important and potentially very impactful research direction, The key idea is to optimize the mask for the loss decrease after an infinimitesal step, rather than for the preservation of loss after prunning. While with the benefit of hindsights it might seem simple, it is a clever innovation. However, I am not convinced by the theoretical explanation and some of the experimental results (see detailed comment below). Based on this I am leaning at the moment towards rejecting the paper. I will be happy to revisit my score if these concerns are addressed.

Detailed comments:

1. I am not sure that NTK based analysis helps explain the efficacy of the method. An increase of the (matrix) norm of the NTK kernel can be achieved by simply scaling up by a constant scalar the logits weights (see for instance https://arxiv.org/abs/1901.08244). Or equivalently (comparing the resulting learning dynamics in NTK, as also can be read from (3)), by just increasing the learning rate. In other words, I could just prune weights randomly, and then scale up logits' weights, and end up with the same effect on the NTK kernel. I think that for this argument to work, NTK kernel should change in a scale-invariant manner. This would correspond to a better conditioning of the loss surface (because Hessian has the same eigenspectrum as the NTK kernel under the NTK assumption), which is a scale invariant property.

2. From the Figure 2 it seems SNIP-prunned network underfits data severly. Could you add training accuracy to the Tables (maybe in the Supplement)? If in all cases when GraSP wins, it is due to underfitting, this should be commented on. Is it common for prunning algorithms to result in underfitting, or is achieving generalization a larger challenge? Could the bad performance at high prunning ratios of SNIP be due to a conflation of two effects: (1) "good" prunning, but (2) lowering the effective learning rate (given the gradient norm is low)? Would, for high prunning ratios, a tuned learning rate improve SNIP performance/reduce underfitting? 

3. In Table 5 is the batch-size used for training of the network, or only for the computation of the Hessian-vector product in the GraSP procedure? If for training, then the relatively small spread of results is a bit surprising given results by Keskar (https://arxiv.org/abs/1609.04836)

Edit

Thank you for the rebuttal. Raise my score. I agree with Reviewer #4 that increasing gradient norm at initialization is a promising direction on its own, which warrants acceptance.


review_title: Official Blind Review #2

Meta-review: This paper proposes a method to improve the training of sparse network by ensuring the gradient is preserved at initialization. The reviewers found that the approach was well motivated and well explained. The experimental evaluation considers challenging benchmarks such as Imagenet and includes strong baselines. 
Decision: Accept (Poster)