Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary:

The paper is clear and very well written. It makes important steps towards understanding if sparse convolutional neural networks can represent a substitute for their dense counterparts. Moreover, it unveils the relation between the performance of sparse neural networks and gradient flow. Based on this relation, it explains also why the dynamic sparse training approach has higher potential of improving sparse neural networks in the future, while lottery tickets are limited by the performance of the pruning solutions from which they are derived. The last but not the least, the paper introduces a simple and practical method specially designed to initialise sparse networks weights.

Strong points:

•	The paper brings novel basic knowledge and understanding of sparse neural networks.

•	The extensive set of experiments is well-designed, very informative, and support the paper claims.

•	The fundamental study performed in this paper is timely and has the potential of advancing seriously the field.  


Weak Points:

•	While the abstract and some other parts of the paper discuss about deep neural networks in general, the experiments are solely focused on convolutional neural networks. I believe that extending them also to other types of networks would improve the overall quality of the paper.

For the discussion phase, I suggest to the authors to consider the weak point and the following minor comments:

1) I find very interesting that any sparse training method cope much better with the ResNet architecture than with the VGG architecture. It is easy to observe this in Table 1. Do you have any idea why is this happening? Is this the effect of skip connections?

2) Can you add in figure 3 the “+” version of the training algorithms for ResNet-50 and the version without “+” for LeNet5?

3) The relation between Hessian, gradient flow, and sparse training raised my curiosity, but I agree with the authors that this investigation can be let for future work.

review_title: Sparse networks understanding

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents three key hypotheses for sparse NN training dynamics and provides empirical studies and observations to verify them. This review will first provide general comments and then specific ones on each hypothesis. 

Main comments:

Pros: Three messages that the authors try to convey are important and interesting to the audiences in pruning. It is an observational paper which provides insights on 1) what is a good initialization of the for sparse NN training 2) why DST can achieve good generalization, and 3) is LTH really different from pruning. 

Cons: The presentation of the paper needs work. The high-level structure is good and clear but for each paragraph, the logic flow is hard to follow. For example, in a very key paragraph on P7: “Lottery Tickets Learn Similar Functions to the Pruning Solution”, I have to read repeatedly and infer inner logics of each sentence to see the conclusion. 

Hypotheses:
I appreciate identifying the problem of naive initialization of sparse NN and connecting it with gradient flow. However, a new proposal for initialization here (as the major contribution) is unnecessary and actually negatively affects the credits of the true contribution. The results presented in table 1 are not very impressive. It is ok to just compare original and liu et al, and provide insights in an observational paper.
The observations provide one possible explanation on why DST might work. The authors could try to test this in different architectures (even beyond CNNs) to see if they are widely held. If so, it is potentially a good metric or analysis tool for sparse NN training.
I am not fully convinced by the third hypothesis. First, the models and datasets for ensemble and prediction disagreement are too limited while the conclusion is very strong. Also I think a more appropriate statement could be Lottery Tickets Learn Similar Functions to the Pruning Solution than random /scratch since that is the only thing you are comparing with.

Minor comment:

It would be interesting to see if all the conclusions hold in other models besides CNNs.
review_title: Official Blind Review #4

final_score: rating: 5: Marginally below acceptance threshold
review_content: ##########################################################################

Paper Summary:

This paper presents an empirical study of sparse deep nets, either obtained by sparsification methods such as “dynamic sparse training” or by pruning according to the lottery ticket hypothesis.
The main contribution of this work is to study gradient flow both at initialisation and during training, and to propose an extension of known initialisation methods that works for sparse networks.
In addition, this work also attempts at explaining why lottery tickets are successful, despite sharing similar problems related to the gradient flow, when compared to other sparsification methods.

##########################################################################

Reasons for score: 

Overall, I like this kind of empirical study, where authors set the stage for important questions and attempt at answering them with a thorough empirical study. My major concern for accepting this work relates to the depth of contributions. In my humble opinion, the proposed generalisation of “He’s initialisation” could have have been the main (only?) focus of this work, with additional experiments and considerations: relation to other sparsity inducing methods (not only DST), a better understanding of the interaction between initialisation, Batch normalisation and skip connections, …
Instead, the presentation strategy in this paper is to illustrate several findings but, due to space constraints, in a more shallow manner. This choice dilutes the contributions too much. 

##########################################################################

Positive points: 

1) Empirical work that addresses an important topic, that of sparse NN. Questions are well motivated, and sufficiently well described, although they have the slightly negative effect of diluting the overall take home message from reading this paper.

2) The proposed extension to a known initialisation method to cope with issues related to gradient flow during the early stages of training is reasonable, and effective as shown by the experiments.

3) The experiments on gradient flow during training and the ones on lottery tickets confirm either known results or intuition. They can be viewed as a reproducibility study, which is commendable. Some by products of the study indicate important properties of LT, which are of direct practical relevance, e.g. rewinding strategies.


##########################################################################

Negative points:

1) The proposed generalisation of He’s method is not sufficiently exploited. Focusing on the forward pass, the idea is to initialise weights on a per neuron basis, using the mask computed by the sparsification method. As the authors notice, the work by Lui et al. achieves similar performance (Fig 1.c) and in many cases it outperforms the proposed method (Tab.1, bold results). This calls for a better understanding of the advantages of the proposed method. Furthermore, the interaction of “sparse initialisation” with batch normalisation and skip connections is not sufficiently studied: in Fig.2 all methods appear similar. Finally, the fact that a “small but dense” network achieves better results (LeNet on MNIST, Tab1), and does not suffer from gradient flow problems (Fig.2 c) is interesting and calls again for further study.

2) The results on gradient flow during training are only superficially commented, although the authors hint at additional ideas based on second order approximations of the loss, i.e. considering the Hessian and its eigen-spectrum. Overall, the take home message from Fig.3 confirms the known behaviour of DST methods such as RigL. Albeit interesting, in my humble opinion this results seem to be given more “real estate” on the paper than it deserves, subtracting space for the main contribution on a new initialisation scheme.

3) The results on lottery ticket could enjoy some improvements on the terminology, which is a minor remark. Indeed, it would be easier to refer to: 1) IMP solution <-> pruning solution; 2) Random init/final <-> start/end; 3) LT init/final <-> start/end. My main concern with this set of results is that on the one hand, they are somehow expected, especially with respect to the large literature available on the topic. It is not bad per se to collect in one coherent piece of work previous observations and place them in a thorough experimental framework. However, I have problems with the following.
a) The notion of “closeness” as shown in Fig 5 a,d and reported in fig 5 b,e is the result of a dramatic dimensionality reduction. Similar techniques have been used in other contexts (e.g. Hao Li, et al. “Visualizing the Loss Landscape of Neural Nets.” NIPS, 2018) and the warnings are to take results with a grain of salt. That said, it is expected — by construction — to find that LT final networks are close to the IMP solution.
b) The argument used to confirm that LT are in the basin of the IMP solution is based on path connectivity and, as the author also note in a foot note in page 7, studying in detail this path is outside the scope of the paper. The geometry of the loss landscapes is in general very complex (especially when there is no Batch normalisation nor skip connections, which have the effect of smoothing it): I am not sure it is correct to claim that if two solutions (that is the params of a neural net) have the same loss and they are connected by a linear path, then it is necessary true that they lie in the same basin. Even if results in Tab.2 on the disagreement are compelling, it might still not be necessarily true that the two compared models are the same instance of function approximation.
c) As a minor remark, the implications of the results in Sec 4.3 are interesting and valuable, but I failed to understand properly the connection to the empirical study on the “distance” between IMP solutions and LT final solutions.
 

#########################################################################

Additional comments:

I found this paper well written in most parts (just a minor comment on the terminology used in one sub-section). I liked this work and I think it has plenty of potential. 
As an humble suggestion, would it make sense to attempt at focussing more the message, and insist on the main contribution of the paper as per a new initialisation scheme, or was this not seen as sufficient in light of the results from Liu et al. (2019).
review_title: Interesting work with some potential, but with some flaws too

final_score: rating: 5: Marginally below acceptance threshold
review_content: Overview:

Summary:
This paper tries to answer the following two questions: i) why training unstructured sparse networks from random initiation perform poorly? 2) what makes LTs and DST the exception? The authors show the following findings:
1. Sparse NNs have poor gradient flow at initialization. They show that existing methods for initializing sparse NNs are incorrect in not considering heterogeneous connectivity. Improved methods are sample initialization from a dynamic gaussian whose variance is related to the fan-in numbers. fan-in = fan-out rule plays an important role here and improves the gradient flow.
2. Sparse NNs have poor gradient flow during training. They show that DST based methods achieving the best generalization have improved gradient flow.
3. They find the LTs do not improve gradient flow, rather their success lies in re-learning the pruning solution they are derived from.


Strength bullets:
1. The idea is very interesting. I appreciate the novel analysis. The proposed methods are well-motivated.
2. The paper is well written and easy to understand.
3. The finding is surprising but the experiment design is poor which I will list more detailed limitations in the weakness sections. I like the idea, I will raise my score if the authors can completely address my confusion and concerns.


Weakness bullets:
1. For Table 1, a Strong baseline is missing. Why not compare with the performance of the lottery ticket setting? I think it is a more natural baseline than SET and RigL.
2. In my opinion, there is a must-do experiment: Lottery ticket mask + proposed initialization and compare it to LT and random tickets. Because the LT mask + random reinitialization = random tickets fail in the previous literature. According to the explanation in the paper, it can also be the problem of random reinitialization. Thus, strong supportive evidence is that show proposed modified random reinitialization + LT mask can surpass random ticket performance.
3. Missing details what is the pruning ratio of each stage in iterative magnitude pruning? The appendix only tells me the author using 95% and 80% sparsity, why pick these two sparsity? Because this sparsity gives the extreme matching subnetworks? And the author uses iterative magnitude pruning, if they follow the original LTH setting, pruning 20% for each time. Then the sparsity should be 1-0.8^i, how to achieve 95% and 80%?
4. What is the definition of "pruning solution"? Is it the obtained mask or initialization or subnetworks contains both mask and initialization? Super confused
5. Conflicted experiments results with Linear Mode Connectivity and the Lottery Ticket Hypothesis paper, ResNet 50 IMP LT on ImageNet without Early weight rewinding can not have good linear mode connectivity. However, the pruning solution and LT solution have good linear mode connectivity. It is wired, even for two LTs (ResNet 50 IMP LT on ImageNet) trained with the same initialization in different data orders, they do not have a linear path where interpolated training loss is flat, as evidenced in figure 5 in the paper "Linear Mode Connectivity and the Lottery Ticket Hypothesis". Early weight rewinding is needed for the presented results while I think the author did not use it. 
6. The comparison in Table 2 is unfair. Scratch settings are trained from five different random initialization, while LT settings are trained from the same initialization with different data orders. LT setting results should also be from different initialization, otherwise can not achieve the conclusion that "Lottery Tickets Learn Similar Functions to the Pruning Solution".

Minor:
1. The definition of LTH in 3.3 "perform as well as O^N(f,\theta)*M", why there is M? It should be the full dense model without the mask, right?

------ Post Rebuttal------

Thanks to the authors for the extra experiments and feedback!

[Lottery baseline for Table-1] Although RigL does not need dense network training, it cost more to find the mask (Table 2 of the RigL paper).

[Random tickets] Random Ticket = LT mask + random re-initialization rather than random pruning + random init. The front one will be much more interesting. "Because the LT mask + random reinitialization = random tickets fail in the previous literature. According to the explanation in the paper, it can also be the problem of random reinitialization. Thus, strong supportive evidence is that show proposed modified random reinitialization + LT mask can surpass random ticket performance." I personally do the experiment that performing proposed initialization on random tickets and the performance is unchanged. Of course, there may exist lots of reasons for the results. I will not degrade the paper according to my experiments.

Other concerns are will-addressed. Thanks!

Although I do like the idea of this paper, I think it might need to be revised and resubmitted, incorporating the extensive discussion presented by all the reviewers. I tend to keep my scores unchanged. But I don’t think this is 100% a clear reject and depending on the opinions of the other reviewers I would not feel that accepting this paper was completely out of bounds.

review_title: The idea is novel and interesting, while the experiments design is poor. I give borderline reject. I expect the response from the authors. If all concerns are addressed, I will raise my scores.

Meta-review: The paper shows empirically that training unstructured sparse networks from random initialization performs poorly as sparse NNs have poor gradient flow at initialization. Besides, the authors argue that sparse NNs have poor gradient flow during training. They show that DST based methods achieving the best generalization have improved gradient flow. Moreover, they find the LTs do not improve gradient flow, rather their success lies in re-learning the pruning solution they are derived from. I read the paper and the reviewers discussed the rebuttal. Although all the reviewers found the rebuttal helpful and they all agree that the paper is decently well written and has some clear value, the majority believes that further observations are required for making the paper and its hypothesis convincing. There are also some recent related work on initialization of pruned networks, e.g. by rescaling their weights at initialization. I believe, adding the discussion of such related techniques and making the connection to existing work will greatly strengthen the paper and provides more evidence to support its claims.



Decision: Reject