Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper studies two concepts, namely, layer freezing and data sieving, in conjunction with sparse training for DNN architectures. The layer freezing methodology is not new, but it has been studied for the first time in this context. The newly proposed data sieving methodology relies on first removing a % of the training instances and then at each epoch, a similar % of the easiest instances are removed and replaced with instances removed from the original set. Both ideas are studied in the context of sparse training, and extensive experimentation shows promising improvement in reducing computational and memory costs with good accuracy.
strengths_and_weaknesses: Strengths:

S1. Important and timely problem
S2. Well-written paper, nicely motivated 
S3. Extensive experimental results

Weaknesses:

W1. The technical depth is somewhat low
questions: I enjoyed reading the paper as it's well-written and the two concepts studied are properly introduced and motivated by experiments to demonstrate their flaws and importance.

The experimental results support the claims, and they are extensive.

The code is provided, which can help in the validation of the results as well as in the adaptation of the proposed solution.

My main concern is that the paper is lacking in terms of technical depth. The ideas explored are relatively simple, and the idea of sieving reminds me of ideas of perturbation analysis (we don't perturb but instead remove instances and replace the easily classified cases). That being said, sometimes we don't know over-complicated solutions, and this approach seems to work well.


limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper brings a new perspective in sparse neural networks training with dynamic sparsity (or, on short, DST) by studying layer freezing and data sieving. Moreover, it highlights and brings understanding on why DST is fundamentally different than dense training. Consequently, it proposes a new framework to improve DST performance by considering three dimensions, i.e., weight Sparsity, layer Freezing, and Dataset Efficiency (SpFDE). The empirical validation shows good performance for SpFDE.
strengths_and_weaknesses: **Strengths**

_ Originality. Up to my best knowledge, this paper is the first to study layer Freezing in Sparse Training. The study itself is novel, and the findings on structural and representational similarity bring a new perspective in DST.

_ Quality. The novel proposed methodology and the well-designed empirical validation seem sound. Good qualitative discussions, reflecting a good understanding of DST and its challenges. 

_ Clarity. The paper is well written.

_ Significance. The paper may have some impact. Particularly, the knowledge unveiled on structural and representational similarity can lay the ground for impactful future research.

**Weaknesses**

_ Clarity. A careful proofread is recommended. There are still typos (e.g., on line 305)

_ Statements. Some statements are too strong. Please consider revising. For instance, line 326, *reducing training costs by pushing sparsity towards extreme ratios is no longer a desirable methodology* prunes valid research directions.

_ (not really a weak point) The improvement in performance over the well-chosen baselines is relatively small, but quite constant.

questions: Structural similarity. Line 163 – 50% seems an arbitrary choice? How other values would affect the structural similarity findings? 

Representational similarity. Line 188 – Why you find this “surprisingly”? As long as connections are added and removed from the networks in DST during learning, the representation is somehow “forced” to change.

Training set. Lines 288-289 – Is there any relation with bagging?

limitations: n/a
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a framework for efficient sparse training. Instead of focusing on pushing the sparsity levels, the authors investigate the effect of two known methods, including layer freezing and data sieving, in the dynamic sparse training framework. The authors show why layer freezing can be applied to sparse training. Then, they described a framework that reduces both memory requirement and FLOPs count while achieving a comparable performance or outperforming state-of-the-art sparse training algorithms. The authors performed an extensive evaluation to support their claims.
strengths_and_weaknesses: Strength:

- The paper is very well-written and has decent organization, and it was a pleasant read.
-  This paper provides insights on other approaches to push further the efficiency of sparse training algorithms instead of only focusing on increasing the sparsity level.
- Extensive evaluation that supports the initial claims.
-  The results on the ImageNet dataset are significant in decreasing the training FLOPs while increasing the accuracy and having different trade-offs between efficiency and accuracy. 

Weakness:

- The effect of each component (layer freezing and data sieving) on the accuracy is not separately clear. It would be nice to add an ablation to analyze the effect of each component.
- It is unclear how the authors determine the value of T_{frz} (starting freezing). Is it difficult to choose a value for this hyperparameter for different sparsity levels/models/datasets? 
questions: - I suggest the authors add a brief explanation about the CKA metric and how it is calculated in the background information.
- If enough time and resources are available, it would be nice to see how much the layer freezing and data sieving can be increased without a significant drop in accuracy. E.g., to present in a 2d-plot where the x-axis shows the change in one of these components and the y-axis shows the accuracy.
-  Minor: typosà lines 288 (,), 305 (sparse) 
limitations: Yes, the authors have discussed the limitations and adequately addressed them.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The authors propose to modify an existing dynamic sparse training (DST) method, MEST, by (a) freezing the training of early layers in a CNN  sequentially early in training to remove the requirements to calculate/maintain backwards pass, and (b) use a “data sieving” method to train in a more data efficient manner. The freezing method is proposed under the intuition that early layers converge to both weights and sparse structure earlier than later layers, and the authors present experimental results claiming to demonstrate this. The authors compare their method (SpFDE) to existing sparse training methods, including SNIP, GrasP, SET and MEST on CIFAR-100 with ResNet-32.
strengths_and_weaknesses: Strengths:
* Questions the wisdom of using the same amount of data/training time to train different layers in a sparse DNN training given the common intuition that early layers converge on weights faster.
* Interesting and intuitive hypothesis that the early layers in a CNN may converge on sparse structure early in training with a dynamic sparse training method.
* Analysis of working memory footprint of model is interesting, and rare to find in papers.

Weaknesses:
* Only theoretical FLOPS shown, and the calculations of theoretical FLOPS for frozen layers is flawed in that it doesn’t consider the real-world scenarios of usage of these models. Forward passes are still required for training through the frozen layers, and this analysis seems to ignore this. Theoretical FLOPS are at best misleading and often completely unrelated to real-world efficiency (e.g. unstructured sparse masks of 99% have a theoretical FLOPS that is low, but hard to accelerate), and any paper claiming to demonstrate real-world speed up of training/inference must include real-world computational measurements of timings. Unfortunately, this fact alone means this paper cannot be accepted as-is. It is not explained what FLOPs these are (e.g. multiply-accumulate), or how they are calculated in general, or specifically for frozen layers.
* Results showing decreased FLOPS with good generalization only demonstrated on a single model (ResNet-32) and on a single dataset (CIFAR-100). Results on CIFAR-10 and ImageNet are in the appendix, but seem to show increased FLOPS for similar/better generalization.
* Missing substantial experimental details, making some of the experiments non-reproducible. In fact, some of the results of experiments presented contradict prior work/well-known issues in sparse training (see questions).
* The first few layers in a CNN are actually upwards of 90% of the compute of the whole model (while being a small fraction of the weights as the authors ). In this sense, the intuition presented by the authors on why they focus on the latter layers doesn’t make sense.
* By trying to present two very different approaches to improving sparse training as one method, the paper reads as a confusing story — these two methods are not related enough to roll into one paper/method IMHO.
* SpFDE incorporates both the frozen layers and data sieving methods, and results are only shown as such. To evaluate the individual contribution of each of these methods the authors need to provide an ablation analysis, however there is none. It is impossible to evaluate the claims of the authors unfortunately on each of the individual proposed methods without this. Furthermore, the results comparing the proposed method with other DST and sparse training methods are not directly comparable, since they are not using data sieving.
* Experiments only focus on one specific dynamic sparse training method (MEST), but conclusions are made over all dynamic sparse training methods.
questions: Questions:
* The experiment presented in 3.1/fig 1(a) comparing the masks of two models trained with a DST method seems to assume that the mask found by DST methods is the same for a DNN trained with different initializations - this is well known not to be the case (as also seen with lottery tickets)! How is your experiment finding masks that are substantially similar, up to 100% in some cases? Are the initializations the same? What is the saliency measure used when choosing the “50% most significant weights”?
* The experiment presented in 3.2/fig 1(b) claims that sparsity levels do not affect sparse training convergence time with MEST. Is this a known result? At least with other DST methods, such as RiGL, I believe this may not be the case.
How exactly do you calculate the theoretical FLOPSin general, and what are your assumptions on the FLOPS of a frozen layer? On a sparse layer?
* You state that you focus on sparse training from scratch rather than pre-training as it’s not feasible to use pre-training on edge devices. Why? Intuition suggests the exact opposite if anything.

Suggestions:
* I actually find the research direction(s) presented by this paper to be very interesting and intuitive, especially the freezing direction, however the current paper doesn’t have sufficient experimental analysis, results or focus to make for a convincing story. I would encourage the authors to take the feedback of the reviewers and attempt to correct this for a future submission.
* I would recommend focusing on each of these methods as it’s own paper (or only on the freezing), they are not related enough to write together as one method, and it weakens the story and clarity of your research motivation and findings. There are significant experimental details missing likely as a result.
* Your work is not the first work to explore “frozen” layers (e.g. fine-tuning, or freezing weights) & sparsity, I would remove this claim as it’s hard to justify. Also, FreezeNet in particular should be cited here as other work looking at freezing weights in pruning/sparsity.
* “front layers” -> “early layers” in all instances, to match the common terminology in the field.
limitations: The authors do not discuss any limitations or potential impacts of their work. I would note specifically that the data sieving method could affect the learning of biases from datasets since it is sub-sampling the dataset during training, and the authors do not identify this.
review_title: null

Meta-review: metareview: The paper provides methods to improve the task of sparse training. The reviews agree that the idea is well motivated, novel and that the paper brings insights to sparse training that would be of interest to the community. The experiments seem quite extensive and show that these methods allow to improve the Pareto curve of training process FLOPs vs obtained model quality. One of the reviews raised several issues about the paper, questioning the soundness of the experiments and method. After reviewing the discussion, the major issues seem to be not fundamental flaws but unclear details in the paper. Since these are clarified in the discussion, I view these as minor issues that can be fixed towards a camera ready version, and I urge the authors to carefully go over the reviews and fix the paper to be more clear.
Concluding, the consensus around novelty and overall positive feedback that remained positive through the discussion phase, lead me to believe the advantages of the paper outweigh its flaws.

confidence: Less certain
Decision: Accept