Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win

Abstract: Sparse Neural Networks (NNs) can match the generalization of dense NNs using a fraction of the compute/storage for inference, and also have the potential to enable efficient training. However, naively training unstructured sparse NNs from random initialization results in significantly worse generalization, with the notable exception of Lottery Tickets (LTs) and Dynamic Sparse Training (DST). In this work, we attempt to answer: (1) why training unstructured sparse networks from random initialization performs poorly and; and (2) what makes LTs and DST the exceptions? We show that sparse NNs have poor gradient flow at initialization and propose a modified initialization for unstructured connectivity. Furthermore, we find that DST methods significantly improve gradient flow during training over traditional sparse training methods. Finally, we show that LTs do not improve gradient flow, rather their success lies in re-learning the pruning solution they are derived from — however, this comes at the cost of learning novel solutions.

Reviews and author responses:

Overview:

Summary:
This paper tries to answer the following two questions: i) why training unstructured sparse networks from random initiation perform poorly? 2) what makes LTs and DST the exception? The authors show the following findings:
1. Sparse NNs have poor gradient flow at initialization. They show that existing methods for initializing sparse NNs are incorrect in not considering heterogeneous connectivity. Improved methods are sample initialization from a dynamic gaussian whose variance is related to the fan-in numbers. fan-in = fan-out rule plays an important role here and improves the gradient flow.
2. Sparse NNs have poor gradient flow during training. They show that DST based methods achieving the best generalization have improved gradient flow.
3. They find the LTs do not improve gradient flow, rather their success lies in re-learning the pruning solution they are derived from.


Strength bullets:
1. The idea is very interesting. I appreciate the novel analysis. The proposed methods are well-motivated.
2. The paper is well written and easy to understand.
3. The finding is surprising but the experiment design is poor which I will list more detailed limitations in the weakness sections. I like the idea, I will raise my score if the authors can completely address my confusion and concerns.


Weakness bullets:
1. For Table 1, a Strong baseline is missing. Why not compare with the performance of the lottery ticket setting? I think it is a more natural baseline than SET and RigL.
2. In my opinion, there is a must-do experiment: Lottery ticket mask + proposed initialization and compare it to LT and random tickets. Because the LT mask + random reinitialization = random tickets fail in the previous literature. According to the explanation in the paper, it can also be the problem of random reinitialization. Thus, strong supportive evidence is that show proposed modified random reinitialization + LT mask can surpass random ticket performance.
3. Missing details what is the pruning ratio of each stage in iterative magnitude pruning? The appendix only tells me the author using 95% and 80% sparsity, why pick these two sparsity? Because this sparsity gives the extreme matching subnetworks? And the author uses iterative magnitude pruning, if they follow the original LTH setting, pruning 20% for each time. Then the sparsity should be 1-0.8^i, how to achieve 95% and 80%?
4. What is the definition of "pruning solution"? Is it the obtained mask or initialization or subnetworks contains both mask and initialization? Super confused
5. Conflicted experiments results with Linear Mode Connectivity and the Lottery Ticket Hypothesis paper, ResNet 50 IMP LT on ImageNet without Early weight rewinding can not have good linear mode connectivity. However, the pruning solution and LT solution have good linear mode connectivity. It is wired, even for two LTs (ResNet 50 IMP LT on ImageNet) trained with the same initialization in different data orders, they do not have a linear path where interpolated training loss is flat, as evidenced in figure 5 in the paper "Linear Mode Connectivity and the Lottery Ticket Hypothesis". Early weight rewinding is needed for the presented results while I think the author did not use it. 
6. The comparison in Table 2 is unfair. Scratch settings are trained from five different random initialization, while LT settings are trained from the same initialization with different data orders. LT setting results should also be from different initialization, otherwise can not achieve the conclusion that "Lottery Tickets Learn Similar Functions to the Pruning Solution".

Minor:
1. The definition of LTH in 3.3 "perform as well as O^N(f,\theta)*M", why there is M? It should be the full dense model without the mask, right?

------ Post Rebuttal------

Thanks to the authors for the extra experiments and feedback!

[Lottery baseline for Table-1] Although RigL does not need dense network training, it cost more to find the mask (Table 2 of the RigL paper).

[Random tickets] Random Ticket = LT mask + random re-initialization rather than random pruning + random init. The front one will be much more interesting. "Because the LT mask + random reinitialization = random tickets fail in the previous literature. According to the explanation in the paper, it can also be the problem of random reinitialization. Thus, strong supportive evidence is that show proposed modified random reinitialization + LT mask can surpass random ticket performance." I personally do the experiment that performing proposed initialization on random tickets and the performance is unchanged. Of course, there may exist lots of reasons for the results. I will not degrade the paper according to my experiments.

Other concerns are will-addressed. Thanks!

Although I do like the idea of this paper, I think it might need to be revised and resubmitted, incorporating the extensive discussion presented by all the reviewers. I tend to keep my scores unchanged. But I don’t think this is 100% a clear reject and depending on the opinions of the other reviewers I would not feel that accepting this paper was completely out of bounds.


We have updated our supplementary material with code which includes an implementation of the initialisation and along with the code to calculate hessian of sparse networks. 

*Continued from Reviewer2 Response (1/2)...*

6. **Table-2 Comparisons:** We like to thank the reviewer for the suggestion. We updated the Scratch results such that they start from the same initialization making the comparison more fair. We renamed the existing Scratch runs (same mask-different init) as **Scratch (Diff. Init.)**. We also added  **Prune Restart** runs, in which we repeat the training starting from the pruning solution. We are also happy to run more experiments using different pruning masks; however the conclusions are likely to stay the same.
The goal of the comparisons in Table-2 as the reviewer mentioned is to check whether lottery solutions and pruning solutions are similar in the function space. All other methods are included as a baseline and we run them with multiple seeds to get a better estimate. Additionally from the ensemble perspective; different initializations bring more diverse results and thus better ensemble performance. This is practically free in the case of scratch training. However different LT initializations require costly dense training. We added this discussion to the relevant section and updated Table-2. Here are the results:

 | Initialization       | (Top-1) Test Acc. | Ensemble | Disagree.     | Disagree. w/ Pruned |
|----------------------|-------------------|----------|---------------|---------------------|
| Scratch (Diff Init.) | 97.19 ± 0.3        | 98.43    |	0.0352± 0.0037 | 0.0278 ± 0.0032    |
| Scratch | 97.04 ± 0.15        | 98.00    |	0.0316± 0.0023 | 0.0278 ± 0.0020    |
| LT                   | 98.52 ± 0.02		|98.58 	| 0.0043± 0.0006 | 0.0089 ± 0.0002
|Prune Restart | 98.60 ± 0.01        | 98.63    | 0.0027± 0.0003 | 0.0077 ± 0.0003    |

Conclusions: (a) 'Disagree. w/ Pruned' numbers are same/similar as expected when the same initialization is used and thus the conclusions remain the same (b) We observe more stable training for LT initialization (i.e. smaller 'Disagree.') compared to *Scratch* despite both of them having a fixed initialization at each run. This is further evidence for LT initializations being biased (stable) towards the pruning basin. (c) Scratch solutions found from the same initializations are less diverse therefore their ensemble performs worse compared to the scratch training with 5 different initializations. (d) Disagreement between LTs and pruning solutions (.0089) are similar to disagreement between PruneRestart and pruning solution (0.0077). In other words LT solutions are as diverse as the solutions found when the training is restarted from the original pruning solution.

We thank Review-2 for their detailed feedback, here we address the concerns. We updated the paper accordingly. We also commit to improving our work with the feedback and suggestions provided. 

1. **Lottery baseline for Table-1.** We agree LTs (and similarly pruning results) can be another meaningful baseline and therefore we added them to Table-1. We compare training sparse networks from scratch with DST baselines (SET, RigL) since these methods also train sparse networks from scratch without requiring expensive dense training. They are as efficient as training sparse networks without changing connectivity.
2. **Random tickets.** The proposed initialization when used with a LT mask improves performance over a "random ticket" (random mask+random init). In our MNIST experiments we use the same mask (obtained through pruning) for all methods in order to disentangle the effect of different masks from the initialization and optimization (mentioned in Section-4.3 and Appendix B.2, "*...networks share the same pruning mask*"). In the original LTH paper using random masks was shown to achieve worse performance compared to the ones found by pruning (in [1] Figure.14). We observe the same behaviour in our experiments. If we shuffle the masks (and thus create random tickets) the performance drops compared to when lottery masks are used. Our initialization still improves in the context of random tickets (Scratch+Shuffled). RigL's performance however is not affected by the choice of masks, since it automatically adjusts the mask during the training. We are happy to further clarify if we misunderstood your question.

	ScratchShuffled: 11.35$\pm$0.00 (vs 62.99$\pm$42.16 Scratch)\
	Scratch+Shuffled: 93.17$\pm$1.98 (vs 97.70$\pm$0.09)\
	RigL+Shuffled: 98.07$\pm$0.09 (vs 98.20$\pm$0.18 RigL+)

3. **Pruning method/sparsities.** We selected  95% sparsity for LeNet-5 since for smaller sparsities the difference between lottery and scratch performance were small and for larger sparsities scratch training was not learning at all. We chose 80% sparse for ResNet-50 because it is a popular benchmark for many other papers in the literature [2]. We are happy to add results with other sparsities but the results/conclusions would likely remain the same. We use iterative magnitude pruning (Zhu et al., 2018 Equation: 1) in our experiments (mentioned in Section 4.3 Experimental setup), which is a well studied and more efficient pruning schedule compared to the one used by Frankle et al. (2019a). Our pruning algorithm performs iterative pruning without rewinding the weights between intermediate steps and requires significantly fewer iterations. We expect our results would be even more pronounced with additional rewinding steps. 
4. **Pruning solution** = pruned mask + pruned weights, i.e. output of pruning. The text has been updated to make this clearer: "...lottery tickets and the solution they are derived from (i.e. pruning solutions)"
5. **ResNet-50 Lotteries:** We do use late-rewinding as mentioned in Section 4.3: "*..an 80% sparse ResNet-50 (Wu et al.,2018) on ImageNet-2012 (Russakovsky et al., 2015) (where K= 0 doesn’t work (Frankle et al., 2019b)), for which we use values from K= 2000 (≈ 6th epoch).*" So our results confirm the ones presented in [3]. We would also like to point out that the experiments in [3] investigate paths between different LT solutions. The main novelty/difference of our experiments comes from the fact we look at paths between pruning solutions and the lottery solutions.

[1] [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://arxiv.org/pdf/1803.03635.pdf)\
[2] [What is the State of Neural Network Pruning?](https://arxiv.org/abs/2003.03033)\
[3] [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://arxiv.org/abs/1912.05671)


*Continued from Reviewer1 Response (1/2)...*

* **(2) The results on gradient flow during training are only superficially commented.** We believe that results in Fig. 3 are not well known, the closest results we know being Figure-2 in the Grasp paper (Wang, 2019) which compares one-shot pruning algorithms only at initialization. None of the DST papers studied the effect of their methods on gradient flow which makes our contribution novel. Our results highlight an interesting correlation between improved gradient flow and test accuracy for sparse models; identifying new and promising research directions.
* **(3.1) Improvements on the terminology.** We changed the terminology as suggested.
* **(3.2) (Section 4.3) results are somehow expected.** Much of the recent work attempting to show that LTs generalize suggests that there are many in the community who would not expect these results, motivating our work. We discuss some of these works briefly in the *Implications* paragraph, many of which, we believe, would have been less interesting if the results presented in this work were known earlier. Regardless of whether the reviewer (or we ourselves) expected the results, it is important to have performed the experiments and observed the results to verify these expectations. Furthermore, we do believe this is novel - if not we would appreciate citations that we may refer to or compare with. 
* **(3a) Expected — by construction — to find that LT final networks are close to the IMP solution.** We agree that MDS embeddings are an illustration and have potential to be misleading. Therefore we share Euclidean distances in the high dimensional space and perform additional experiments (linear connectivity and function similarity). The closeness of LTs and the Pruning solution has not been measured in the past as far as we are aware.
* **(3b) The argument used to confirm that LT are in the basin of the IMP solution.** This is a very interesting point. First we want to highlight that the footnote is given for the path that shows the barrier between scratch and pruning solution. We don't know whether they (scratch&pruning) are in the same basin or not. A basin can be defined as a set of points each of which is linearly connected to at least one other point in the set. We added this definition to the text. An alternative definition of a basin is introduced at Neyshabur, 2020. Regardless, since the LT solution and the pruning solution are linearly connected, by definition they are in the same basin.
* **(3c) The connection to the empirical study on the “distance” between IMP solutions and LT final solutions.** Thanks for your question. We have added a sentence that clarifies the connection. Having shown LT solutions end-up in the same basin, next we want to understand whether 2 points from the same basin represent similar functions. It might be the case that LTs learn significantly different solutions each time in function space despite being linearly connected to the pruning solution. Results in 4.3 address this possibility and have important implications for the usability of lottery tickets (for example transfer); as discussed in the implications paragraph. 

(Wang, 2019) [Picking Winning Tickets Before Training by Preserving Gradient Flow](https://arxiv.org/abs/2002.07376)\
(Neyshabur, 2020) [What is being transferred in transfer learning?](https://arxiv.org/abs/2008.11687)


Thank you for your detailed review and comments. In our work we attempted to touch on a group of topics that we believe are important for understanding and improving training sparse neural networks. We identified Gradient Flow as an important tool for studying various phenomena from initialization to the success of dynamic training methods. We understand that there are some interesting questions that remain unanswered, some of them being exciting future research directions. 

* **(1.1) The proposed generalisation of He’s method is not sufficiently exploited.**  There is no instance where the Liu et. all initialization works substantially better than our proposed initialization. On the contrary, with the updated results (see the main response above for the details) we started to observe improvements due to our initialization on the MNIST setting as compared to Liu et. al. Additionally unlike the Liu et. al initialization, our proposed initialization is provably correct for all sparse masks (unlike the Liu initialization). While it is disappointing that we did not see a substantial difference in using the "correct" initialization on the particular models we trained v.s. Liu et. al, it is notable that even the authors of Liu et. al claimed their initialization had no effect - we are the first to show it does have an effect in networks without batchnorm and skip connections. We speculate that the lack of a difference is due to the fact that neurons do not have a much variance in the number of masked weights for the sparse masks we looked at.
We agree there is value in investigating the initialization for sparse networks further and understanding in which settings the correct initialization matters. However, given the insensitivity of modern sparse architectures (like ResNet-50) to the initialization; we chose to leave that investigation as a future work.
* **(1.2) The interaction of “sparse initialization” with batch normalisation and skip connections is not sufficiently studied.** We are not the first to observe the interaction between batch normalization/skip connections and initialization in general (Balduzzi et al, 2017; Zhang et al 2019; Yang et al, 2019), but we believe we are the first to observe this in sparse neural networks, and we believe it is an important observation. We agree this needs much more study, but even explaining this in the context of dense models has proven to be challenging for the research community, and we believe it is fair to say this is beyond the scope of this work.
* **(1.3) “small but dense” network achieves better results … and calls again for further study.** While deep neural networks do suffer from poor gradient flow (and thus this was addressed by batch norm/skip connections), small/shallow neural networks do not suffer gradient flow problems, and we believe this is well-known. Similarly training sparse networks from scratch are shown to be challenging (Frankle, 2019) compared to small-dense models (Evci, 2019). However, methods like pruning, pruning+LT and RigL seem to find better results in many interesting tasks (Kalchbrenner, 2018; Evci, 2019 and Li, 2020), however this is not a requirement.
With the updated results: RigL/SET/Lottery get 98.13 / 98.16 / 98.26 test accuracies respectively on MNIST while Small Dense gets 98.21% test accuracy. We also checked pruning results in the same setting and the mean accuracy was 98.64%, which is better than what a small dense model gets. We agree that it is an interesting question: Why can't RigL exceed the Small Dense performance while constantly having better gradient flow? Looking solely on gradients can be limiting and Hessians can provide us additional useful information. In Figure-11b we observe that the Hessian spectrum of small dense models have smaller positive outliers while having significantly larger negative eigenvalues; which is a good property for optimisation (in terms of conditioning) and can explain why small dense models perform well despite having smaller gradient flow. We added this discussion to section 4.2.
 

(Kalchbrenner, 2018) [Efficient Neural Audio Synthesis](https://arxiv.org/abs/1802.08435)\
(Frankle, 2019) [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://arxiv.org/pdf/1803.03635.pdf)\
(Evci, 2019) [Rigging the Lottery: Making All Tickets Winners](https://arxiv.org/abs/1911.11134)\
(Li, 2020) [Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers](https://arxiv.org/abs/2002.11794)\
(Zhang, 2019) [Fixup initialization: Residual learning without normalization](https://arxiv.org/abs/1901.09321)\
(Balduzzi, 2017) [The shattered gradients problem: If resnets are the answer, then what is the question?](https://arxiv.org/abs/1702.08591)\
(Yang 2019) [A mean field theory of batch normalization](https://arxiv.org/abs/1902.08129)

Thank you for your detailed review and comments. In particular, we appreciate your feedback about making our paper easier to understand. We would highly appreciate any other specific feedback about which paragraphs/sections you found difficult to comprehend, as you'll appreciate these are often hard to identify as the author!
* **Paragraph 7 hard to follow:** Indeed this paragraph was a bit convoluted, and we've revised it as follows: "Here we adopt the analysis of (Fort et al., 2020), but in comparing LT initializations and random initializations using fractional disagreement. The fractional disagreement with the pruning solution is the fraction of class predictions over which the LT and scratch models disagree with the pruning solution they were derived from. In Table 2 we show the mean fractional disagreement over all pairs of models. The results presented in Table 2 suggest that all 5 LTs models converge on a solution function almost identical to the pruning solution. ..."
* **New initialization proposal is unnecessary.** With the updated results we observe small but significant differences on MNIST training. The important distinction between our initialization and Liu et. al, is that their initialization is technically incorrect for many sparsity masks where neurons have different numbers of unmasked weights, while ours is provably (see Appendix A) correct for all sparse masks. We also must point out that Liu et. al claimed their initialization had no effect, so we are actually the first to show that in many contexts having a sparsity-aware initialization is important. We believe both of these points are critical to publish for the research community's benefit.
* **Only CNNs used for experiments.** See our feedback on a very similar point from Reviewer3. Copied below:
> We think this is a fair point, and indeed we suspect that sparse transformers would benefit from our initialization for example, but the existing literature on the subject of sparse neural networks is dominated by CNNs and vision evaluation, making it difficult to compare results outside of this domain, moreso page limitations make it difficult to fit more results into the main text. We added however Appendix D, which includes results using a fully-connected model showing the generalizability of our results to other architectures.
* **Third hypothesis' conclusion is very strong.** We do not wish to overclaim, and are willing to consider rewording our conclusion if there is a statement the reviewers feel is not well-supported by the results. Indeed the statement that lottery tickets learn similar functions to the pruning solution is our main point in this section. To support our claims and make the comparison more meaningful we ran an additional experiment which we call as *Prune-Restart*. In this experiment we restart our regular training starting from the solution found by the pruning algorithm using 5 different seeds. Each seed uses a different data order and therefore the solutions found are slightly different. The similarity of *Prune-Restart* solutions to the original pruning solution matches the similarity of LT solutions. We added these results to Table-2 and also share them here:
 | Initialization       | (Top-1) Test Acc. | Ensemble | Disagree.     | Disagree. w/ Pruned |
|----------------------|-------------------|----------|---------------|---------------------|
| Scratch | 97.04 ± 0.15        | 98.00    |	0.0316± 0.0023 | 0.0278 ± 0.0020    |
|Prune Restart | 98.60 ± 0.01        | 98.63    | 0.0027± 0.0003 | 0.0077 ± 0.0003    |
| LT                   | 98.52 ± 0.02		|98.58 	| 0.0043± 0.0006 | 0.0089 ± 0.0002


We thank all reviewers for their valuable comments and suggestions. We have responded to individual reviewer feedback  below. Here we give an overview of changes made to the paper during the rebuttal. 

* **Updated experiments:** During our rebuttal we found a minor mistake in our LeNet5 architecture's definition (using softmax activation twice at the output), and as a result we have re-run all LeNet5 experiments. All of our conclusions remain the same; except now we see some minor improvements on LeNet5 using our initialization compared to Liu. et al.'s scaled initialization. 
* We added LT results to Table-1 as another baseline following Reviewer2’s suggestion.
* We updated  scratch training to use  the same weight initialization in Table-2 as suggested by Reviewer2. These solutions are more diverse as compared to the lottery solutions found when trained with different data orders; giving us further evidence that LT initializations are extremely stable and biased towards a single pruning solution independent of the data order they are trained with.
* We added *Prune-restart* results (doing regular training starting from the pruning solution) to Table-2. The similarity of these solutions to the original pruning solution matches the similarity of lottery solutions; which supports our claim of "Lottery Tickets Learn Similar Functions to the Pruning Solution."
* We added Appendix D, which demonstrates our results/conclusions using fully connected networks show our results hold outside of CNNs, and address the points raised by Reviewer3 and Reviewer5.
We added Appendix C which includes additional gradient flow plots as suggested by Reviewer3.
* Finally, we commit to open-source the code for the paper by next week.


We thank the reviewer for this feedback, and are happy they found the paper interesting. Here we attempt to address the weak points/minor comments they have highlighted for discussion.
* **Only CNNs used for experiments.** We think this is a fair point, and indeed we suspect that sparse transformers would benefit from our initialization for example, but the existing literature on the subject of sparse neural networks is dominated by CNNs and vision evaluation, making it difficult to compare results outside of this domain, moreso page limitations make it difficult to fit more results into the main text. We added however Appendix D, which includes results using a fully-connected model showing the generalizability of our results to other architectures.
* **Sparse training methods cope much better with the ResNet architecture than with the VGG architecture.** It's an interesting point, we also assume this is due to skip connections (and batch-normalization), although we did not run experiments to determine this. Skip connections are well known to help optimization in the context of dense NNs  allowing higher learning rates. High learning rates are important for DST methods as they depend on new connections to grow (becoming non-zero) so that they are not removed again in the next update iteration. In other words, we believe that higher learning rates allow DST algorithms to search a larger space of connectivity patterns. We are happy to add a short discussion on this to the text. 
* **Can you add in figure 3 the “+” version of the training algorithms for ResNet-50 and the version without “+” for LeNet5?** Plots are shared and discussed in Appendix C. In Figure-8 you can see that the MNIST dense initialization (without +) starts slow as initially the gradient doesn't flow through the network (see "RigL" at Figure 7-left). When the learning starts we also start observing gradient flow improvements after connectivity updates. For Resnet-50 we pretty much see the same picture, providing further evidence on initialization insensitivity.


##########################################################################

Paper Summary:

This paper presents an empirical study of sparse deep nets, either obtained by sparsification methods such as “dynamic sparse training” or by pruning according to the lottery ticket hypothesis.
The main contribution of this work is to study gradient flow both at initialisation and during training, and to propose an extension of known initialisation methods that works for sparse networks.
In addition, this work also attempts at explaining why lottery tickets are successful, despite sharing similar problems related to the gradient flow, when compared to other sparsification methods.

##########################################################################

Reasons for score: 

Overall, I like this kind of empirical study, where authors set the stage for important questions and attempt at answering them with a thorough empirical study. My major concern for accepting this work relates to the depth of contributions. In my humble opinion, the proposed generalisation of “He’s initialisation” could have have been the main (only?) focus of this work, with additional experiments and considerations: relation to other sparsity inducing methods (not only DST), a better understanding of the interaction between initialisation, Batch normalisation and skip connections, …
Instead, the presentation strategy in this paper is to illustrate several findings but, due to space constraints, in a more shallow manner. This choice dilutes the contributions too much. 

##########################################################################

Positive points: 

1) Empirical work that addresses an important topic, that of sparse NN. Questions are well motivated, and sufficiently well described, although they have the slightly negative effect of diluting the overall take home message from reading this paper.

2) The proposed extension to a known initialisation method to cope with issues related to gradient flow during the early stages of training is reasonable, and effective as shown by the experiments.

3) The experiments on gradient flow during training and the ones on lottery tickets confirm either known results or intuition. They can be viewed as a reproducibility study, which is commendable. Some by products of the study indicate important properties of LT, which are of direct practical relevance, e.g. rewinding strategies.


##########################################################################

Negative points:

1) The proposed generalisation of He’s method is not sufficiently exploited. Focusing on the forward pass, the idea is to initialise weights on a per neuron basis, using the mask computed by the sparsification method. As the authors notice, the work by Lui et al. achieves similar performance (Fig 1.c) and in many cases it outperforms the proposed method (Tab.1, bold results). This calls for a better understanding of the advantages of the proposed method. Furthermore, the interaction of “sparse initialisation” with batch normalisation and skip connections is not sufficiently studied: in Fig.2 all methods appear similar. Finally, the fact that a “small but dense” network achieves better results (LeNet on MNIST, Tab1), and does not suffer from gradient flow problems (Fig.2 c) is interesting and calls again for further study.

2) The results on gradient flow during training are only superficially commented, although the authors hint at additional ideas based on second order approximations of the loss, i.e. considering the Hessian and its eigen-spectrum. Overall, the take home message from Fig.3 confirms the known behaviour of DST methods such as RigL. Albeit interesting, in my humble opinion this results seem to be given more “real estate” on the paper than it deserves, subtracting space for the main contribution on a new initialisation scheme.

3) The results on lottery ticket could enjoy some improvements on the terminology, which is a minor remark. Indeed, it would be easier to refer to: 1) IMP solution <-> pruning solution; 2) Random init/final <-> start/end; 3) LT init/final <-> start/end. My main concern with this set of results is that on the one hand, they are somehow expected, especially with respect to the large literature available on the topic. It is not bad per se to collect in one coherent piece of work previous observations and place them in a thorough experimental framework. However, I have problems with the following.
a) The notion of “closeness” as shown in Fig 5 a,d and reported in fig 5 b,e is the result of a dramatic dimensionality reduction. Similar techniques have been used in other contexts (e.g. Hao Li, et al. “Visualizing the Loss Landscape of Neural Nets.” NIPS, 2018) and the warnings are to take results with a grain of salt. That said, it is expected — by construction — to find that LT final networks are close to the IMP solution.
b) The argument used to confirm that LT are in the basin of the IMP solution is based on path connectivity and, as the author also note in a foot note in page 7, studying in detail this path is outside the scope of the paper. The geometry of the loss landscapes is in general very complex (especially when there is no Batch normalisation nor skip connections, which have the effect of smoothing it): I am not sure it is correct to claim that if two solutions (that is the params of a neural net) have the same loss and they are connected by a linear path, then it is necessary true that they lie in the same basin. Even if results in Tab.2 on the disagreement are compelling, it might still not be necessarily true that the two compared models are the same instance of function approximation.
c) As a minor remark, the implications of the results in Sec 4.3 are interesting and valuable, but I failed to understand properly the connection to the empirical study on the “distance” between IMP solutions and LT final solutions.
 

#########################################################################

Additional comments:

I found this paper well written in most parts (just a minor comment on the terminology used in one sub-section). I liked this work and I think it has plenty of potential. 
As an humble suggestion, would it make sense to attempt at focussing more the message, and insist on the main contribution of the paper as per a new initialisation scheme, or was this not seen as sufficient in light of the results from Liu et al. (2019).

This paper presents three key hypotheses for sparse NN training dynamics and provides empirical studies and observations to verify them. This review will first provide general comments and then specific ones on each hypothesis. 

Main comments:

Pros: Three messages that the authors try to convey are important and interesting to the audiences in pruning. It is an observational paper which provides insights on 1) what is a good initialization of the for sparse NN training 2) why DST can achieve good generalization, and 3) is LTH really different from pruning. 

Cons: The presentation of the paper needs work. The high-level structure is good and clear but for each paragraph, the logic flow is hard to follow. For example, in a very key paragraph on P7: “Lottery Tickets Learn Similar Functions to the Pruning Solution”, I have to read repeatedly and infer inner logics of each sentence to see the conclusion. 

Hypotheses:
I appreciate identifying the problem of naive initialization of sparse NN and connecting it with gradient flow. However, a new proposal for initialization here (as the major contribution) is unnecessary and actually negatively affects the credits of the true contribution. The results presented in table 1 are not very impressive. It is ok to just compare original and liu et al, and provide insights in an observational paper.
The observations provide one possible explanation on why DST might work. The authors could try to test this in different architectures (even beyond CNNs) to see if they are widely held. If so, it is potentially a good metric or analysis tool for sparse NN training.
I am not fully convinced by the third hypothesis. First, the models and datasets for ensemble and prediction disagreement are too limited while the conclusion is very strong. Also I think a more appropriate statement could be Lottery Tickets Learn Similar Functions to the Pruning Solution than random /scratch since that is the only thing you are comparing with.

Minor comment:

It would be interesting to see if all the conclusions hold in other models besides CNNs.

Summary:

The paper is clear and very well written. It makes important steps towards understanding if sparse convolutional neural networks can represent a substitute for their dense counterparts. Moreover, it unveils the relation between the performance of sparse neural networks and gradient flow. Based on this relation, it explains also why the dynamic sparse training approach has higher potential of improving sparse neural networks in the future, while lottery tickets are limited by the performance of the pruning solutions from which they are derived. The last but not the least, the paper introduces a simple and practical method specially designed to initialise sparse networks weights.

Strong points:

•	The paper brings novel basic knowledge and understanding of sparse neural networks.

•	The extensive set of experiments is well-designed, very informative, and support the paper claims.

•	The fundamental study performed in this paper is timely and has the potential of advancing seriously the field.  


Weak Points:

•	While the abstract and some other parts of the paper discuss about deep neural networks in general, the experiments are solely focused on convolutional neural networks. I believe that extending them also to other types of networks would improve the overall quality of the paper.

For the discussion phase, I suggest to the authors to consider the weak point and the following minor comments:

1) I find very interesting that any sparse training method cope much better with the ResNet architecture than with the VGG architecture. It is easy to observe this in Table 1. Do you have any idea why is this happening? Is this the effect of skip connections?

2) Can you add in figure 3 the “+” version of the training algorithms for ResNet-50 and the version without “+” for LeNet5?

3) The relation between Hessian, gradient flow, and sparse training raised my curiosity, but I agree with the authors that this investigation can be let for future work.


Meta-review: The paper shows empirically that training unstructured sparse networks from random initialization performs poorly as sparse NNs have poor gradient flow at initialization. Besides, the authors argue that sparse NNs have poor gradient flow during training. They show that DST based methods achieving the best generalization have improved gradient flow. Moreover, they find the LTs do not improve gradient flow, rather their success lies in re-learning the pruning solution they are derived from. I read the paper and the reviewers discussed the rebuttal. Although all the reviewers found the rebuttal helpful and they all agree that the paper is decently well written and has some clear value, the majority believes that further observations are required for making the paper and its hypothesis convincing. There are also some recent related work on initialization of pruned networks, e.g. by rescaling their weights at initialization. I believe, adding the discussion of such related techniques and making the connection to existing work will greatly strengthen the paper and provides more evidence to support its claims.




Decision: withdrawn-rejected-submissions