Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper seeks to explore and understand the improved performance of Neural Networks trained with additional parameters even though a majority of the weights can be pruned during inference. They analyze the magnitude, correlation, and movement of weights during training and propose a hypothesis that the reason is that the weights which can be pruned during inference provide degrees of freedom in the loss landscape which allows better optimization of the non-prunable weights. Based on this, they explore and recommend a methodology for sparse training as a method to efficiently approximate a higher dimensional weight space that maintains these good optimization properties. They ablate the effects of the different recommendations and explore the hyperparameter space to have a better understanding of the dynamics of the sparse training.
main_review: Strengths
- This paper is well written and easy to understand
- They make clear the positioning of the paper in relation to prior work and what they aim to demonstrate and compare against.
- The central idea about extra removable weights being extra degrees of freedom is well explained and may be somewhat novel
- They demonstrate significantly better results compared to prior work that does not use frequent "rewiring" with gradient backpropagation on non-contributing weights.
- They demonstrate the effects of the different methods and recommendations with their thorough ablation and exploration of the effects of different hyperparameter values.


Weaknesses
- The novelty of the paper (though admitted by the authors) could be more significant. The method itself does not seem novel and has been explored and used in depth in several other papers.
- These papers are cited, but not compared against. While it is understandable that the main focus of the paper is not on doing significantly better than these methods since they share a similar concept, the paper may be more novel if it can more clearly differentiate their contribution. It is difficult to judge the empirical results in that respect without those comparisons.
- While the related works are mentioned and cited, the central idea could be better compared to the variety of recent work analyzing Deep Neural networks from a loss landscape perspective. It is somewhat difficult to ascertain how novel this central idea is without this comparison. It is quite similar in effect and possibly concept with many other works on analyzing escaping local minima and overparameterized networks.

Questions:
It may be helpful to directly state this in the response especially if I am misunderstanding or underappreciating something. What is the main novel contribution of this paper that differentiates it from other works in this space?
Based on Figure 3, it seems like resetting weights to 0 may have a negative effect on model performance? Doe this trend continue if the length between resets in lengthened even further?

Can you baseline your analysis of the weight movement of active and inactive weights compared to random walks of the model weights? Some of the trends are difficult to differentiate from the numerical properties of the metrics. The trend for active and inactive weights must be monotonically increasing for all the weights due to the definition since at the end of training there is no more movement of the weights. For the correlation, larger weights with a random walk for a fixed step size would naturally require more time to uncorrelate. What is the measure for distance traversed in the second graph in figure 1?

The paper may be improved by exploring the benefits of this sparse training in terms of inference speedup and training speedup.



Minor comments:
"While LOTTERY (Frankle & Carbin, 2019) determines at initialization which weights participate in sparse models"
That cited paper determined based on a full training run based on weight magnitude and then reset the initialization.

Table 1 and 2 would be easier to understand if all the results were consistently error or accuracy or at least if + or - delta was consistently better or worse

summary_of_the_review: This paper is well written, and clearly demonstrates the advantages of sparse training with rewiring, gradient updates for non-participating weights, and an induced exploitation step. The paper also includes very thorough and useful empirical results on sparse training and through ablation experiments, but lack comparisons to similar methods. Unfortunately, I currently recommend weak rejection since I don't believe that there are quite enough significant novel contributions. While the experiments and ablations shed some further light into how to effectively use these methods and provides some better understanding, this space and the concepts have been thoroughly explored in previous works.


===============================================================
Post-rebuttal
I thank the authors for their clarifications in the comments and changes in the paper. I believe much of it is clearer now. Due to this, I will be raising my recommendation to marginally below the acceptance threshold. Still believe that as other reviewers have mentioned, the paper would be significantly stronger if they compared their recommendations more directly with other recent work in the space since they claim to be providing extensive hyperparameter recommendations and comprehensive recommendations for how to train sparse models.


There are also minor changes which would make the figures much clearer. Figure 3 should include their method with the combination of  Fix, Reset, and Regularization. The trend with rewiring and resetting steps in Figure 2 would also be clearer if the graphs were slightly extended to demonstrate at what frequency resetting and rewiring is required.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The domain is the optimization of deep and large neural networks. The author argues and presents information that supports (figure 1), a hypothesis that a large portion of neurons is pushed towards dormancy while optimizing a neural network and only a subset of the neurons are actively used. The authors, moreover, argue that these dormant weights are important during optimization as they can propose new solutions during training (figure 2). However, these dormant weights can also be an issue as they introduce gradient noise. The authors propose a new scheme to balance the role of the dormant weights in order to maximize new solutions while minimizing gradient noise. The authors do this by either setting them to zero at a certain iteration or regularizing their sizes. They only optimize a set amount of the weights at once (rewind). The authors show results on multiple benchmark datasets highlighting their performance and how new sparsity-oriented hardware works well with their algorithms. The dormitory vs active weights are picked by a threshold of their magnitude.
main_review: Strengths:
They provide a very intuitive explanation of the problem with good drawings and figures. This gave me a new intuition about how neural networks learn and optimize
There's a lot of comparisons with current regular neural networks.
The project details are well documented in the appendix
The paper is well written

Weaknesses:
I can't find any comparisons with other, similar, sparsity methods. I would like to see how this holds up against state-of-the-art in sparsity.
I would like to see confidence intervals on the different model performances (if you have the resources of course).
The code is not available (as far as I can see).
summary_of_the_review: I think this is interesting research and highlights a general and contemporary challenge when training deep neural networks. The problem is well motivated and the implementation well described. The experiments are extensive. however, as this is not compared against any other model in the field I cannot assess whether this approach is truly useful. I would be able to change my mind if the authors provide such results, or argue why they don't provide it. Also, please make your code available in an anonymous repository, and publicly available at the end of the review session. If you have the compute, please provide confidence intervals on your results. Also, please elaborate on what it means for weights to "decorrelate". E.g. why is the threshold important.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the training of sparse neural networks and the effect of extra parameters (dense connectivity) during training. They hypothesize that pruned connection are useful, because they help optimization to take some random steps in those direction which help optimization escaping the local minima. Authors support their claims with experiments and finally share a very long list of experimental results on training sparse networks.  
main_review: This work has two main contributions. First authors argue that over-parameterization is useful for `search`. Later authors provide extensive experimental results (maybe the largest I've seen in this field) on comparing various sparse training techniques. I thank authors for their extensive experimental results, which is quite useful to the community, as many researcher can't do such extensive experiments. This contribution alone could have a significant impact on the field. Do they plan open-sourcing their code? The idea of grouping weights into 3 subsets (inactive/active/search-space) is also interesting and novel (though this idea is mentioned in various work). Though this work has quite a bit potential to impact and inspire future research, I found few things confusing and have questions about its novelty.

## Concerns
0. Main method used in the paper reminds me previous work like Discovering Neural Wirings and TopKast, in which forward pass is made on top-magnitude weights and gradients are applied to all weights. It would be nice to make a clear comparison with previous work; i.e. what is the novelty? 
1. Naming in Figure-4 is a bit confusing, I recommend using terms like 'Small-Dense', 'Static' instead of no-explore/exploit (also NO EXPLORE becomes REDUCE in the next figure, which is not consistent) as it is used in previous work. Furthermore 'Fix' and 'Reset' have similar curves. So resetting doesn't help? Then, the text needs to be updated as it claims 'fix' brings better results. Maybe bringing results from Figure:12 that shows the gains of resetting weights might be useful.
2. rigl is known to take longer training steps to converge. Are your experiments made with default dense training steps? I recommend authors to do extended training comparisons (like 2x or 5x) for the Resnet50 classification experiments. Similarly do you use the ITOP recipe proposed by [4] in your experiments? It would be nice to make this clear in the text, as if using limited training steps, ITOP improves the performance of rigl. And finally the difference between 'rigl/set' and 'search' needs to be clear in the text. fully-sparse DST methods (set/rigl) don't need to store values of the inactive weights, which reduces the peak memory consumption significantly. I recommend authors to make this distinction early in the paper, motivate the setting they are focusing on and clearly indicate before each experiment.
3. I think part of Appendix C should be brought to the main text (like the algorithm), as it would help the reader to understand the experiments and contributions better. One way to enable that would be to make Section 2 more succinct, as few ideas are repeat in different subsections.
4. Section 2.4 seems very similar to the experiments/observation in [3] (and some other parts of Section 2). Authors should discuss similarities and differences. I am not sure what new insights are provided in Section 2 compared to [3]. Can you make this clear? 
5. Hypothesis and evidence provided in section 2.3 is not convincing to me. When weights approach to zero, they naturally become uncorrelated. Therefore the correlation tells more about which weights go to zero than anything else I think. Furthermore since low magnitude weights are selected at the end it becomes a bit circular. Finally, the relationship between correlation and exploration is not clear. 
6. In appendix it says "For smaller models, we reduce the width of neural layers by a factor of 1/d to match the number of weights used in sparse models." The factor should be 1/sqrt(d), as width=0.1, would have roughly 1% of the parameters. 
7. weight decay has an important effect on exploration and used successfully in literature [1,2]. Do you use weight decay in your experiments?  Are they included in the gradients when you scale the them for Figure:3?
8. I am not sure the illustration on Figure-2 is helpful giving an example where extra dimensions can be helpful. Since even with the vertical dimension, the optimization would stay at the bad local minima for the given example. Maybe "mexican-hat" example would be a better fit. 

## Minor
- First paragraph of Section 2.1, authors should cite the original work on each point made as much as possible. Happy to suggest early references if needed. 
- "since they contribute the most" -> the least
- "They are also are tolerant"
- "...use search spaces to describe reasons more weights are needed for training" - "to describe why more weights are needed..."
- "Task error" doesn't seem like the right term. I would recommend using something like 'Delta Accuracy' or 'Compression Error'
- In captions ResNet50 is mentioned without the dataset. I assume it is Imagenet?
- "by training sparse models delineated so far" Not clear what this refers to. So far when?
- RIGL (Mocanu et al., 2018) (typo)
- Figure 17: Clockwise is ambiguous. update titles with model names?
- Related Work/ 'which improved accuracy but not enough to match that of larger models' This statement is vague and possibly not correct. This depends on the sparsity. Maybe you mean dense-to-sparse training? Then, rigl and topkast do match pruning results. 
- 'these results open many question' 

[1] https://arxiv.org/abs/2110.00296
[2] TopKAST: https://proceedings.neurips.cc/paper/2020/file/ee76626ee11ada502d5dbf1fb5aae4d2-Paper.pdf
[3] https://arxiv.org/abs/1906.10732
[4] https://arxiv.org/abs/2102.02887

## After Rebuttal
I read author's response and looked the revised version quickly. A lot seems to be changed during rebuttal, which is probably a good thing; though makes it difficult to evaluate. Regardless, as I mentioned above, the experimental results (which includes basically everything except RL) is unique and thus important for informing future research, thus I raised my score to 6. 

If rejected, venues that accept longer papers (like JMLR) might be a good fit. It would be great to discuss experiments in different domains in detail and try to understand/quantify why different behaviours are observed in different domains. 
summary_of_the_review: As mentioned above the experimental results provided at the end of the paper (one of the most extensive set of experiments I have seen in a paper), would be a strong contribution alone. However, authors use half of the paper focusing on explaining why more connections are needed for better results, which I find not very novel and a bit confusing. Authors say that pruned connections are used in training for exploration and search very vaguely; which doesn't provide any new insights on the existing understanding of the training of sparse neural networks. I am willing to increase my score if my concerns are addressed and looking forward to read the next version of this work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors proposed a new point of view to explain why deep learning models often require more weights are needed to train models rather than to run inference for tasks. One example is that sparse models usually perform well in inference, but do not perform well in training. The main content of this viewpoint is that more weights during training can expand the search space, create extra degrees of freedom for search, and form new optimization paths when model optimization falls into critical points, thereby facilitating model training. Based on this point of view, this paper proposes a series of steps to augment search spaces of sparse models during training to approximate the behavior of larger models. The experimental results show that this series of steps can effectively reduce the model error.
main_review: Strengths:
1.	It is interesting to understanding the training of deep models from the perspective of search space.
2.	This paper proposed a series of steps to train the sparse models as larger models, and quantitative experimental results show that these steps can effectively reduce the training error of the sparse model

Weaknesses:

1. The third paragraph of Section 2.1 introduces the changes in the role played by model weights during the training process (shown in Fig. 1). And authors make an assumption that the weights which are eventually discarded (or added weights) play a crucial role in determining what will be used for inference. Nevertheless, it is hard to verify the role of these weights according to existing results. More results and explanations are required.

2. Since this paper focuses on training sparse models, it would be better to compare the proposed method with model compression methods (e.g., [a,b]). 

3. The authors propose several techniques to encourage the exploration and exploitation when training deep models. However, it is still unclear how to derive the final sparse model from a full network. More details should be provided.


Reference:

[a] Discrimination-aware network pruning for deep model compression. PAMI 2021.

[b] ResRep: Lossless CNN Pruning via Decoupling Remembering and Forgetting. ICCV 2021.

summary_of_the_review: It is an interesting paper and the experiments are strong for me.
review_title: null

Meta-review: The reviewers were generally split on this paper. On the one hand, reviewers generally appreciated the clear presentation, discussion, and explanations, and the experiments. On the other hand, most reviewers commented on the lack of comparative evaluation to other works, including works that are related conceptually. While the authors have a potentially reasonable argument for omitting such comparisons, in the balance I do not believe that the reviewers were actually convinced by this. Particularly when the novelty of the contribution is not crystal clear, such comparisons are important, so I am inclined to not recommend acceptance at this point (though I acknowledge that the paper is clear borderline and could be accepted).
Decision: Reject