Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: There are recent works questioning the value of inheriting weights in structured neural network pruning as it is empirically observed training from scratch can match or outperform finetuning a pruned model. This paper mainly includes three components: 1) the authors reinvestigated the problem and demonstrates that the conclusion is inaccurate because of improperly small finetuning learning rates. They show finetuning with pruned weights actually outperforms training from scratch, when larger learning rates and longer training epochs are adopted. 2)  the authors explored dynamical isometry (DI) to understand how finetuning LR affects the final performance. They show that weight pruning breaks dynamical isometry and finetuning can recover it and a larger LR can recover faster. 3) They proposed to fully recover dynamical isometry in fitler pruning before finetuning.

main_review: The paper is well written and organized. The effect of learning rate on fine-tuning stage of pruning is not thoroughly investigated and this paper provides an in-time and thorough study. My major concerns are as follows:

- Pruning as poor initialization: the concept of initialization needs to be cleared. Does the pruning in Fig 1 refer to weights pruning for both networks? It might be true if it is weights pruning as different pruning ratios correspond to the same weights dimensions with different values. However, filter pruning results in different architectures and different pruning results are not different initializations with respect to the same architecture. 

- The dynamic isometry might explain the easiness of optimization, however, the relationship with the generalization is not quite clear. There might be other metrics such as flatness might show a similar trend as JSV. A comparison with other metrics for generalization could be more persuasive.

- The analysis with MLP-7 and MNIST might be too simple for practical usage, and this method does not generalize to more practical non-linear convolutional neural networks. The authors propose strong regularization helps, however, no explanation on why this method helps and how it connects with previous analysis. Does this method work for MLP-7 and how is it connected to DI?
summary_of_the_review: The paper carefully studies recent thoughts on the meaning of pruning and made some rigorous investigations. Though some issues exists and DI may not be the ultimate solution to the question, the paper provides valuable thoughts for this filed, including fair experimental comparison and new explanations. Therefore I recommend to accept.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper argues pruning a pretrained network and finetuning is better than training a sparse network from scratch and tries to connect the fine-tuning phase properties to dynamical isometry (DI) literature. The empirical observation makes a case for inheriting the pretrained weights.
main_review: # Strengths

1. The empirical observation that inheriting pretrained weights is better than training a sparse network from scratch if the fine-tuning phase is carefully tuned.
2. The empirical analysis on fine-tuning hyperparameters (LR, #epochs) might be useful to practitioners.

# Weaknesses

1. lack of novelty: connection to DI and pruning breaks DI is known [Lee et al 2020, analysed at initialization] and analysing for a pretrained network is straightforward and claiming it as a contribution is weak.

2. The hypothesis on page 7 connecting larger LR in fine-tuning and DI is dubious and not backed by any theory or experiments. In fig.2 plots mean JSV increases and that does not mean improved DI. As DI theory suggests to have mean JSV around 1.

3. The comparison against scratch does not seem fair. Usually, fine-tuning phase is much shorter compared to training from scratch but in this case fine-tuning is done for almost the same no of epochs as the scratch version. Is it possible that the scratch version could be tuned to improve the performance?

4. In section 4, the L2 regularization of pruned weights is regarded as a method for DI recovery which in my opinion is unsubstantiated.
summary_of_the_review: There are many unsubstantiated claims (or weak statements) in the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper challenges an existing argument that “inheriting the weights of the pruned network is not necessary for fine-tuning”. This paper provides empirical evidence that the previous experiments are not carried out with proper learning rates and training epochs to ensure the network convergence. 

The authors conjecture that fine-tuning pruned network requires larger learning rates and longer training epochs because its dynamical isometry is broken. As a result, large learning rates (if training for shorter epochs) or long training epochs are required for the recovery of dynamic isometry. 

Then the authors propose OrthP to re-initialize the pruned weights to completely recover the dynamic isometry in simple MLP. With OrthP, the fine-tuning process is less sensitive to different learning rate and training epochs. 

main_review: Strength: 

1) This paper brings new insights for the role of the “weights” in the network pruning.
2) This paper re-examines the established arguments with careful designed experiments over wide range of pruning ratios.
3) The proposed technique is clearly motivated by empirical observations and is very effective on MLP networks under large pruning ratios.
4) This paper is easy to follow.

Weakness:

1) Table 1,2 shows that fine-tuning a pruned network requires much more training time and tuning efforts to achieve a satisfying result. Although the author claims that OrthP can “complete” resolve the dynamical isometry issue, the pruned network converges still only after prolonged training epochs (90 and 900 epochs). Does that means the pruned networks weights are more difficult to optimize than random initialized ones, which contradicts the "the value of weights" claim?
2) The application area of the proposed OrthP is limited to MLP (Table 6). Although the author proposed a Strong L2 Regularization for more complicated networks, this section seems to be “unfinished” — the connection with dynamical isometry seems a little bit weak to me, can the authors provides more elaborations? This seems to be more of practical interests.
3) I have some confusions regarding the experiment designs and presented results (see the Questions section below). 

Questions:

1)  For Table 1: In “Our rerun”,  are “Scratch” results trained under 120 epochs with a decayed LR schedule? For Table 2, what are the hyper-parameters for the “Scratch” results, are they carefully tuned?
2) Have the authors investigated the JSV of randomly initialized sparse networks? I am not sure if applying OrthP initialization or other orthogonal regularization methods on “Scratch” can also boost its performance. I believe this would be a fairer comparison to support "the value of weights" claim.
3) Table 5: Can the authors also provide the “Scratch” results and their hyper-parameters? I am asking because the reasons stated in Weakness 1) — if the “Scratch” result is comparable to OrthP without using prolonged epochs, then this means even when the dynamical isometry is recovered, the pruning selected weights need more training time to optimize than randomly initialized ones. Also, in Table 2, it looks like “Scratch” results work well under large PR, which is exactly the case in Table 5.
4) Related to 3), from Figure 2 and Table 5, it looks like OrthP takes the role of a large training epochs to recover dynamical isometry. It would be nice to see if OrthP works for non-prolonged epochs, e.g., < 90.
summary_of_the_review: I like this paper in that it thoroughly investigates existing arguments on the value of pruning weights. The perspective of OrthP is not that novel, but is simple and effective on MLP under large pruning ratios. 

My major concern is in the experiments design and results part. I would consider raising my score if my questions can be addressed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work takes a second look at the set of papers (Crowley et al 2018, Liu et al 2019) which claim that there is no generalization benefit offered by pruning, i.e., training a large network first and then pruning performs identically or worse than training the same pruned network architecture from scratch. This work claims that these previous works do not set learning rates during fine-tuning correctly while performing their experiments and hence these claims are invalid. The paper also offers an explanation for this phenomenon via dynamical isometery theory.
main_review: 1) **Nice analysis of previous papers**: It is commendable that this paper points out issues with two previous papers (Crowley et al 2018, Liu et al 2019) who make methodological errors by not tuning the learning rate for fine-tuning, and use the results of these erroneous experiments to make general statements about the efficacy of pruning (regardless of whether these statements are true or not). However from the perspective of these previous papers, it seems they had a fixed small fine-tuning budget of 20 epochs, and did not want to exceed this as we require number of fine-tuning epochs to be much smaller than the number of training epochs. While I do think that this requirement is arbitrary, it helps understand the claims of the previous papers.

2) **Unfair experimental comparison**: In Table 1, it seems that the small Resnet models (A,B) trained from scratch are trained for 120 epochs, while the best prune-finetuned models are trained for 90 epochs. However also accounting for the training of the original large Resnet-34 model, the total number of training epochs are 90+90=180 epochs. Hence it seems that the prune+finetuned models are trained overall for a larger number of epochs, which may be a confounding factor which explains their better results. I would *strongly* suggest that the experiments be run such that the number of training epochs for "training small model from scratch" and "large model training with pruning and finetuning" be equalized for a fair comparison.      

3) **Unclear why hypothesis must be true**: This paper rest on the hypothesis that training a large model followed by pruning and fine-tuning offers improved generalization benefits over training a small model from scratch. In principle it should be clear that both two methods can produce models with similar generalization given sufficiently strong optimizers, as the only difference between them is the initialization of the small model. Even in practice, we often are able to train models such that the train loss is close to zero (or sufficiently small), regardless of whether we train small models from scratch or fine-tune pruned models.  
Hence given that both methods solve the same overall optimization problem (i.e, minimizing loss of the small model), I am confused regarding why one would expect pruning + fine-tuning to generalize better than training from scratch, even in principle. For example, it is clear that given infinite epochs and a suitable (well-tuned) optimizer both methods should work equally well. Is the paper's hypothesis a claim about the speed of optimization, that pruning+fine-tuning is a fast way to obtain a small model? If so, how does this change after having accounted for the training of the dense model itself (which is usually slower to train)?

4) **Incorrect metric for measuring Dynamical Isometry**: In equation 1, the paper uses mean singular values as a metric for assessing dynamical isometry. However if we require that all singular values are equal to one, then the metric to use would be the sum of (squared) deviations from unity of each singular value (\sum_i (\Sigma_{ii} - 1)^2). In such a metric, zero corresponds to dynamical isometry being achieved. However the metric of mean singular values has no such meaning as the standard deviation can be large. I am hence confused regarding why the paper chose to use the mean JSV metric, and if there is no strong reason for doing so, I would *strongly* recommend re-running experiments with the deviations-from-unity metric.

5) **Relationship between mean JSVs and convergence unclear**: Hypotheses 1-4 and the boxed explanation talk about the "recovery" of mean JSVs and their relationship to network convergence. I am confused regarding why the behaviour of mean JSVs during or after training is deemed meaningful. I understand that too small or too large mean JSVs must be avoided at initialization as they can cause vanishing or exploding gradients, but can increasing mean JSVs from ~1 to ~5 (as in Fig 2) be deemed meaningful? It is thus not clear why mean JSV (or any measure of dynamical isometry) can be used to measure model convergence (ref. to boxed explanation in page 7). For example, if gradient-norm regularization is used, then the model would converge and yet have small JSVs. The only measure of model convergence is the gradient norm of the loss w.r.t. weights, and other measures are meaningless unless explicitly shown otherwise. 
summary_of_the_review: This paper does not make a solid case for the hypothesis that "pruning+finetuning produces models with better generalization than training small models from scratch", as there are issues with both the experimental and the analytical studies. In the experimental setup, the comparisons are not performed fairly, while in the analytical parts use flawed metric to measure model convergence and dynamical isometry. Even fundamentally, it is unclear why the proposed hypothesis must be true in the first place. For all these reasons, I would recommend rejection.

Post rebuttal update: The discussions did not change my opinion of the paper. However I appreciate the efforts taken by the authors to update the paper in light of reviewer suggestions, particularly to make the core argument more rigorous. Although I do not think the paper still succeeds at this, I am increasing my score slightly to reflect this change.
review_title: null

Meta-review: The paper's primary contributions are:
* Contrary to previous claims, the authors empirically show that inheriting the weights after pruning can be beneficial when using *larger* fine-tuning learning rates than previously done.
* As an explanation, the authors provide suggestive results showing that pruning breaks dynamical isometry, which they claim explains why larger learning rates are needed.
* They propose a regularization-based technique to recover dynamical isometry on modern residual CNNs.

Generally, reviewers were positive about the ideas in the paper, however, even after the rebuttal 3/4 reviewers did not find the arguments were clear or strongly supported yet. One issue that came up several times is a request for more investigation of StrongReg+pruning. At this time, I have to recommend rejection, but I encourage the authors to follow up on the reviewers suggestions and submit to a future venue.
Decision: Reject