Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a layer freezing method to reduce computational time for training of ML models. An attention-based layer freezing model takes sampled parameters from a layer as input, and predicts if it should be frozen or not. The layers get dynamically frozen over the training period. Results should significant improvement over hand-crafted and heuristic methods. 
strength_and_weaknesses: * Strengths:
- Paper is well written and easy to follow
- Experiments are extensive, with sensitivity studies, comparison with state of the art, and computation overhead measurements.
- Proposed method is simple but effective

* Weaknesses:
- Analysis limited to CNN vision models. Unclear how well it will translate to other architectures (e.g. Transformers) or domains (e.g. Language) or objectives (e.g. Regression, Object detection)
- A few clarifying questions: 
--- it is not clear how the layer freezing works dynamically given that all layers below the chosen layer needs to be frozen as well. This is mentioned in passing in Section 2, but not mentioned in methods.
--- It is not clear what the training and test datasets are. It is mentioned in passing that ImageNet + ResNet50 is used for training, is that all the training required?
clarity,_quality,_novelty_and_reproducibility: Clarity: Good, but can be improved. Please address the questions asked above.

Quality: Excellent. 

Novelty: Good idea to learn attention layers for freezing

Reproducibility: Excellent
summary_of_the_review: Overall, a good paper with substantial improvement over baselines.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a layer freezing method to improve training efficiency by introducing attention-guided layers. SmartFRZ aggregates the historical weights using the attention-based predictor to allow the dynamic decision of freezing layers leading to the efficient training framework. The authors support their methods through empirical comparisons of existing algorithms.
strength_and_weaknesses: Strengths
1. The author shows that SmartFRZ achieves performance more efficient training framework than existing algorithms, such as Linear Freezing and AutoFreeze. 
2. The paper is well written in general.

Weakness
1. SmartFRZ needs to train the predictor separately from the actual network training. 
2. Limited comparisons only on CIFAR and two CNN architectures.
3. Lack of explanations on the motivation for the suggested approaches.

clarity,_quality,_novelty_and_reproducibility: I list the questions and concerns below:

1. According to Figure 1, the attention-based predictor must access all previous timestamps' previous weights (historical weights). If $t$ is large, the SmartFRZ algorithm may struggle with the memory problem, unlike Linear Freezing and AutoFreeze.

2. Can the author provide more detailed motivations for using attention in the predictor architecture? Why would an attention module be a better choice than regular LSTM-based architecture? It is well-known attention module is notorious for quadratic complexity. 

3. In Section 3.2 Layer Taioloring paragraph, I am not aware Lottery Ticket Hypothesis (LTH) paper mentions "the over-parameterization not only exists in the inference process but also in the training process." The hypothesis says there exists an initialization that the sub-network matches the original network after, at most, training the same number of iterations. To my understanding, LTH is an entirely different statement from what the author is trying to convey. Furthermore, I do not buy the line "the gradient distribution is one of the indicators that best characterizes the features of parameters during the training process." Can the author refer to the literature supporting this claim?

4. The biggest weakness of the paper is testing their algorithms on a simple dataset, such as CIFAR. The motivation for freezing business is to reduce the overhead of training procedure, which the CIFAR dataset usually do not need. We cannot extrapolate the SmartFRZ empirical results collected on only the CIFAR and CNN architecture to the different classes of architectures and other tasks. In order to empirically show that SmartFRZ is better than AutoFreeze, can the author also make comparisons on NLP tasks listed in the AutoFreeze paper? 
summary_of_the_review: The authors provide empirical results regarding the efficient training framework on neural networks. However, their motivations for the methodology are not convincing to me, and their results are limited to a specific dataset (CIFAR) and the family of convolutional neural networks. Furthermore, the author's explanation of their approach needs to be clearer. Overall, it is a well-written paper, but I do not think the paper does not meet the ICLR bars.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes an interesting approach to predicting freezing layers during training by training a meta-predictor. The attention based meta-predictor takes in weight history and predicts if a layer should be frozen or not. The predictor is also pretrained on a dataset generated using the CKA similarities between a well trained and novel network. The authors show improvements over other layer freezing approaches that rely on heuristics for image classification on a variety of architectures.
strength_and_weaknesses: **Strengths**
1. The meta-predictor is cleverly designed to consider to be layer agnostic by subsampling weights to a fixed vector. 
2. The performance of SmartFrz is better than Linear or Autofreeze for the given experiments.
3. The authors perform a fairly complrehensive series of experiments showing the effect of sequence lengths.

**Weaknesses and Qeustions**
1. The main advantage of freezing sequentially is that gradients do not need to be calculated. How are the authors showing lower tFlops given that the gradients still need to be calculated for the frozen layers? I am curious to see if the implementation to see how exactly the savings occur.
2. The authors also do not mention the hyperparameters and the experimental setup for Linear and Autofreeze. This is extremely important as Autofreeze appears to be very susceptible to hyperaprameter values.
3. it appears that the attention based predictor is a Resnet50 trained on Imagenet. How transferable is this predictor to other datasets? While Cifar-10/100 are a good baseline, it would be a valuable addition to include results on a variety of datasets to confirm the performance.
4. Further, the authors claim that they use CKA to train the predictor. Do they use CKA with the subsampled weights or the original size? Also are there any thresholds that are used to generate the training dataset for the predictor? Fig. 4 shows that CKA is not a monotonic curve, and the choice of this threshold may affect the predictor adversely.
5. The authors also claim that they report the average results over 5 runs. However, there are no error bars or std deviation reported for any of the numbers. I suggest that the authors add these in order to determine the significance.
6. Weight histories are also dependent on the optimizer used.  What optimizers does SmartFrz work with?
clarity,_quality,_novelty_and_reproducibility: The writing, while clear, is missing several important factors like reporting hyperparameters for baselines and experiment setups. In terms of novelty, the idea is interesting and novel, and in fact reveals that weight histories may be transferable and predictive of performance. For reproducibility however, the paper currently lacks several important details (see weaknesses above.),
summary_of_the_review: Overall, the idea of using a meta predictor for layer freezing is inspired. However, the experiments are not documented well, and several details are missing. In addition, it is unclear why SmartFrz outperforms other freezing algorithms given that layers are still being frozen out of order. It would be also be of independent interest to consider more tasks such as language modelling and architectures like ViTs to see if the predictor is transferable. 

I am currently leaning towards a weak reject, but am open to changing my opinion if the authors can provide reasonable explanations for my concerns listed above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper propose a dynamic layer freezing technique during DNN training. Specifically, they adopt attention-based predictor to predict which layer should be freezed. The predictor is jointly applyed to each timestamp.
strength_and_weaknesses: Strength:

* A valid efficient training method towards cheaper DNN training.

Weakness or Questions:

* I doubt the necesarity of using attention-based model to predict the layers. Actually, attention-based model suffers from the quadractic complexity as compared to the local feature extractor, CNNs. So use Attention model to guide CNN training looks overkill to me. Instead, we should use small models to predict much larger models' freezing schedule.

* How is the overhead of such a attention based model? Since it will be repeatedly used for prediction, thus it must have to be negligible overhead otherwise I cannot see the point.

* The most direct baseline is to reduce the layer numbers and compare with your method, and randomly drop some layers during training process. I am not sure whether your predicted schedule beats the random one.

* Also, the exps are mostly conducted on CNNs, how about ViTs? We should natually reuse the attention based model as both predictors and main contributors to the model performance in the ViTs.
clarity,_quality,_novelty_and_reproducibility: It is well written.
summary_of_the_review: In summary, I think that this paper provide a decent point of view for efficient training method. But the idea is not very novel as there are also many other works working on freezing layers. But the proposed methods beat several baselines.

Also, it is not clear whether it is worth to leverage attention model to help CNN training. It should be reversed IMO.

I suggests that the author could consider adding more ViT-based exps, where their attention can be naturally server as predictors.

If that works, I would increase my score as it shows that your methods can be generalized to ViT models.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposed an interesting topic of adaptively freezing the layers during training to save training time and memory.  The proposed predictor output attention for each layer by integrating the history layer information during training. The training and freezing are alternatively performed on layers of models. 
strength_and_weaknesses: Strength:
The motivation is clear and the topic is useful. The proposed predictor can dynamically choose which layer to be frozen to enable training efficiency and memory saving. 

Weakness:
The predictor also consumes the memory during normal training. Seems that there are no numerical results about this aspect. Do all reductions of memories or time reported in the paper have taken the predictor part into accounting？ 
How do you choose the structure and config of the predictor according to different models that have very various parameters? 
Training from scratch experiments did not include the results on the ImageNet dataset. Are there any reasons for this?  

clarity,_quality,_novelty_and_reproducibility: Yes, the presentation is good. The analyses of the experiments and ablations are adequate and solid.
summary_of_the_review: Although there are some weaknesses in this work, I still think that this work is interesting and can be an easy-to-use tool in practice.  The authors are welcome to release the code and describe the additional memory cost brought by this predictor. And if possible, results on ImageNet are also convincing.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a new approach for improving neural network training efficiency via layer freezing. Previous such approaches have chosen heuristics to decide which layers to freeze; this paper instead proposes a simple "meta" network (which leverages self-attention) to predict which layers should be frozen (and when). This meta-network is lightweight and trained with randomly sampled "weight histories" along with centered kernel alignment scores which give an indication of whether the weights have stabilized. 

This is a simple but effective idea (and beats existing freezing based approaches), which the authors support via experiments on a range of model architectures and datasets. Since the overhead of training this meta-predictor is quite low, the method is likely to be widely applicable.

Originally, a few reviewers raised concerns. Primarily, the original paper limited its experimental results to CNN models and image classification on CIFAR-10. However, during the discussion phase the authors provided a large number of additional results on transformer architectures, recurrent architectures, language models, other vision datasets, etc, which significantly strengthened the paper. The authors were also able to satisfactorily address other concerns related to conceptual benefits of the proposed approach. 

Overall, this is a well-written paper that proposes a simple idea which works well in several settings. 
Recommendation: accept.
justification_for_why_not_higher_score: See summary for a description of strengths. The idea is nice, but in hindsight, fairly intuitive and the natural "next step" in this line of work. I commend the authors for doing a good job of executing the idea, and certainly the technique should be spotlighted. I don't think there is enough to say here that merits a full-length Oral  
justification_for_why_not_lower_score: See summary for details; I personally believe that this paper is worthy of acceptance, and that the technique is conceptually simple enough for a large subset of ICLR attendees to be interested in using it. Therefore, I recommend a spotlight.
Decision: Accept: notable-top-25%