SmartFRZ: An Efficient Training Framework using Attention-Based Layer Freezing
Authors: Sheng Li
Geng Yuan
Yue Dai
Youtao Zhang
Yanzhi Wang
Xulong Tang
Venue: ICLR 2023 notable top 25% (2023)

Abstract: There has been a proliferation of artificial intelligence applications, where model training is key to promising high-quality services for these applications. However, the model training process is both time-intensive and energy-intensive, inevitably affecting the user's demand for application efficiency. Layer freezing, an efficient model training technique, has been proposed to improve training efficiency. Although existing layer freezing methods demonstrate the great potential to reduce model training costs, they still remain shortcomings such as lacking generalizability and compromised accuracy. For instance, existing layer freezing methods either require the freeze configurations to be manually defined before training, which does not apply to different networks, or use heuristic freezing criteria that is hard to guarantee decent accuracy in different scenarios. Therefore, there lacks a generic and smart layer freezing method that can automatically perform ``in-situation'' layer freezing for different networks during training processes. To this end, we propose a generic and efficient training framework (SmartFRZ). The core proposed technique in SmartFRZ is attention-guided layer freezing, which can automatically select the appropriate layers to freeze without compromising accuracy. Experimental results show that SmartFRZ effectively reduces the amount of computation in training and achieves significant training acceleration, and outperforms the state-of-the-art layer freezing approaches.

Published as a conference paper at ICLR 2023
SMARTFRZ: AN EFFICIENT TRAINING FRAMEWORK USING INTENTION-BASER LAYER FREEZING
Sheng Li Geng Yuan"?, Yue Dai Youtao Zhang', Yanzhi Wang?, Xulong Tang'
Universitty of Pittsburgh NNotheesterrn University {sh1188, yud42, xulongtang]epii== edu [yuan. gyngn yanz. wang wnothnnnttttetteeeeeee
chaneytecs.pict.ett
ABSTRACT
There has been proliferation of artificial intelligence applications, where model training is key to promising high-quality services for these applications. However, inevitably
the model training process is both time-intensive and energy-intenns affecting the user's demand for application efficiency. Layer freezing, an efficient model training technique, has been proposed to improve training efficiency. Al- though existing layer freezing methods demonstrate the great potential to reduce model training costs, they still remain shortcomings such as lacking generaliz- ability and compromised accuracy. For instance, existing layer freezing methods either require the freeze configurations to be manually defined before training. which does not apply to different networks, or use heuristic freezing criteria that is hard to guarantee decent aceuracy in different scenarios. Therefore, there lacks a generic and smart layer freezing method that can automatically perform "in- situation" layer freezing for different networks during training processes. To this end, we propose generic and efficient training framework (SmartFRZ) The core proposed technique in SmartFRZ is attention- tomatically select the appropriate layers to freeze without compromising accuracy. Experimental results show that SmartFRZ effectively reduces the amount of com- putation in training and achieves significant training acceleration, and outperforms the state-of- -f-t----t-tttttt layer freezing approaches.
layer freezing, which can au-
INTRODUCTION
Deep neural networks (DNNs) have become the core enabler of wide spectrum of Al applica-
tions, such as natural language processing (Vaswani et al. 2017 Kenton & Toutanova| (019). visual
recognition (Li al. 2022, Faraki et al. 2021), automatic machine translation (Sun et 2020
Zheng et al. 2021). and also the emerging application domains such as robot-assisted eldercare (Do al. 2018 Bemelmans et al.,
al.] 2012). mobile diagnosis (Panindre et al.) [202T] Abdel-Basser al.
2020 , and wild surveillance (Akbari et al. 2021, Ke 2020). To satisfy the growing demand for
adaptability and training efficiency of DNN models in deployment, a surge of research efforts has
been devoted to designing efficient training paradigms (He et al. 2021 Yuan et al. 202 Wu al. 2020 2021). For example, sparse training (Evci et al.) 2020; Yuan et al. (021) and low- www-precssion training Yang et al., 2020; Zhao al.) 2021) )
active research areas for efficient training that can effectively reduce training costs, such as computing FLOPs and memory. However, there are still fundamental limitations in the generality of these methods when used in practice. Concretely, sparse training methods generally require the support of dedicated libraries or compiler optimiza- tions (Chn al., 2018, Niu al. 2020 to leverage the model sparsity and save computation costs. Similarly, www-precssion training (e.g., INT-8 or fewer bits) is hard to be supported by GPUs and requires specialized designs for edge devices such field- programoorea gate arrays (FPGAs) and application-spedde specific integrated circuits (ASICs). Therefore, it is desirable to have a generic efficient training method that can be easily and effectively adapted various application scenarios.
Recent studies (Brock et al. 2017 Liu et al. 2021) revealed that freezing some DNN layers at certain stage (i.e., training iteration) during the training process will not degrade the accuracy of
"These authors contributed equally.

---

Published as a conference paper at ICLR 2023
the final trained model and can effectively reduce training costs. Most importantly, layer freezing can be achieved using native training frameworks such as T without additional support, making more accessible to wide applications and users for training costs reduction and acceleration. Previous work has mainly used heuristic freezing strategies, such as empirically se- lecting which DNN layers freeze and when freeze (Lee et al. 2019 Yuan al., 2022). As such, these heuristic freezing strategies require a trial-and-eroor process to find appropriate freezing configurations for individual tasks/networks, resulting in inconvenience and inefficiency when de- ployed to various application scenarios. Some recent works attempt to freeze the DNN layers in an adaptive manner by using the gradient-norm (Liu et al. (02T) or SVCCA score (He et al.) (021). However, it is known that DNN models generally do not monotonically converge to their optimal position. As a result, these adaptive methods, which decide whether the layers are frozen or not us- ing the heuristic criteria, are less robust to the fluctuation of model training, leading to compromised accuracy. Therefore, we raisc a fundamental question that has seldom been asked:
Is there a layer freezing method that can overcome the above-mentioned short- comings while keeping the method efficient?
Inspired by the outstanding performance of attention mechanism solving sequence classification problems such as classification tasks (Long et al. 2018), dialog detection (Shen & Lee, 2016), and affect recognition (Gorrosticta et al., 2018). 3) it is possible that the attention mechanism could also be promising solution for layer freezing efficient training. but it has never been explored in prior lit- erature. In this paper, we innovatively introduce attention mechanism for layer freezing. In specific, design a lightweight attention-basee predictor collect and rank the DNN context information from multiple timestamps during training process. Based on the prediction, we adaptively freeze DNN layers save training computation costs and accelerate training process while maintaining high model accuracy. To train the attention-base predictor, we propose layer representational similarity-based method to generate special training dataset using publicly available dataset (e.g., ImageNet). Then, the predictor is trained offline once, and learns the generic convergence pattern along the training history, which can be generalized across different models and datasets. We sum- marize our contributions as:
We design a novel and lightweight predictor using attention mechanism for layer freezing in efficient training. The predictor automatically captures DNN context information from multiple timestamps and adaptively freezes the layers during the training process.
We propose to leverage the layer representational similarity to generate special dataset for training the attention-based predictor. The trained predictor can be used for different datasets and networks.
Combining the attention- iitention-based predictor design and its training method, we propose generic efficient training framework, namely SmartFRZ, which can effectively reduce the training costs and accelerate training time. Specifically, for fine-tuning scenarios, Smart- FRZ consistently achieves higher training acceleration while maintaining similar or higher accuracy compared to the prior works. For training from scratch scenarios, Smart- FRZ shows more significant advantages compared to the baseline methods including full training, linear freezing, and AutoFreeze, which achieves 0.16%, 2.65%, and 4.82% higher accuracy with 24%, 13%, and 15% less training time, respectively.
2 BACKGROUND AND RELATED WORK
Layer Freezing. Recent researches (Brock et al. 2017 Kornblith et al. 2019) have found that not all layers in deep nerual networks need to be trained equally. For example, the early layers in DNNs responsible for low-level features extraction and usually have fewer parameters than the later layers, making the early layers converge faster during training. Therefore, the layer freezing techniques are proposed. which stop updating certain layers during the training process to save the training costs (Lee et al. 2019 Zhang & He 2020).
The previous work mainly uses heuristic strategies determine which layers freeze and when to freeze them. For example, Brock et al.) (2017) uses a linear/cubic schedule to freeze the layers sequentially (one by from the first layers to the later layers). The recent work Yuan et al. (2022)
2

---

Published as a conference paper at ICLR 2023
Training Stage
Freezing Stage
Training Stage
Waights of Layer
Weights of Layor 2
Weights of Layer
Weights of Layer
Weights
Waights of Layer
Weights Layer
Weights
Layer
KII
MII 22
MM
Altention-Based Predictor
Frozen
Layer
U
Decision Three Layers
h+1
Figure 1: Overview of our attention-basee layer freezing framework SmartFRZ. SmartFRZ con- tinuously records the weights of actively being-trained layers during the training process. Pe- riodically, there will be a freezing stage, when SmartFRZ applies an attention-based predictor to determine whether each active layer can be frozen at this moment, based on the collected historical data. After the freezing stage, the model training process continues, and the frozen layer will not be updated and its weights will not be recorded anymore. this figure, we assume that the attention- based predictor predicts to freeze layer 1 only.
also freezes layer by layer in a constant interval, but uses target training FLOPs reduction to decide the time to start freezing. Another branch of works attempt to freeze DNN layers adaptively and use the criteria such as gradient-norm (Liu et al. (021) or SVCCA score (He et al. 2021) 1) to control the freezing. However, all of these heuristic-based based freezing strategies require manually designed freezing configurations
threshold, which be varied between different networks and datasets. In this paper, we intend to address these shortcomings and propose generic freezing method that can be adopted to different scenarios.
Training Costs Reduction through Layer Freezing. Layer freezing technique can save training costs including computation FLOPss and memory accesses. Generally, each iteration of DNN train- ing consists of forward and backward propagation. On the one hand, once a layer is frozen, it still needs to compute the forward propagation since the subsequent layers still need its output features as their input features. Therefore, layer freezing cannot save training costs in forward propagation. On the other hand, the backward propagation consists of two parts of computations: i) calculating the gradients of weights and ii) calculating the gradients of activations. A frozen layer can always eliminate computation cost and memory cost of the former part since the weights will not be updated anymore. However, the computation costs and memory costs of the latter part can only be eliminated if all the predecessor layers earlier than the current layer are frozen as well. This is because if predecessor layer is not frozen, it needs to calculate the gradients of activations of the current layer to maintain the data pass through the backward propagation. Compared to the significant advantage of layer
the layer
other efficient training techniques such as pruning or quantization, freezing is that it does not require dedicated supports such as compiler optimizations or specialized hardware. can be easily integrated into the general deep learning frameworks (e.g., PyTorch) and achieve almost linear acceleration according to the computation FLOPs reduction (Yuan et al., p.22). Meanwhile, focuses on layers as granularity which more efficient in reducing the training cost, and potentially, it can be combined with pruning and quantization to bring combined benefits.
3 SMARTFRZ FRAMEWORK DESIGN
3.1 FRAMEWORK OVERVIEW
In this section, we introduce our proposed layer freezing framework for efficient training. which can automatically freeze appropriate network layers to reduce unnecessary computation during the training process without compromising accuracy. Figure [ shows the overview of the SmartFRZ framework. The core component of our framework is a lightweight attention-based predictor, which can automatically decide which layers will be frozen and when to freeze them during the training process. However, the attention-basee predictor cannot be directly trained using conventional dataset for the classification tasks such as ImageNet. Therefore, we propose a novel method to generate the training dataset. We adopt offline training to train the predictor. Once the predictor is well-trained can be used for different datasets and networks since it learns generic converging patterns from collected training histories.
3

---

Published as a conference paper at ICLR 2023
Time fo
MLP.
MLP
MLP.
Training history
encading
Predictor
Attention-besed
training history aggregating
MLP
More attention @ Weights timestamp
Attention-based Predictor
Argmax
-+Less attention mi Nolsy weights history
Optimal weights
(b)
Figure 2: (a) An illustrated example of noisy weight history in which darker shade within the circle indicates a lower loss. The training loss decreases at iteration (1.2.5} 5]. yet increases at iterations (3.4). So the predictor needs pay more attention the weight history from iteration {1.2, and less attention to the weight history from iterations {3, 4] since the latter converge towards less predictor. At timestamp the pre- dictor decides whether to freeze a specific layer in three steps. First, it encodes the training history independently into feature vectors (i.e., KJ Q/ V. ). Second, it computes attention scores of and aggregates the historical state feature into context vector C₁. Third, it predicts the confidence scores of freezing not and selects the decision with higher confidence.
optimal directions. (b) Detailed workflow of the attention-
In the following parts of this section, we will first introduce the design principle and workflow of our attention- based predictor in Section [32 and Section 3.3 Then, we will introduce how to create the special training dataset for the predictor in Section
3.4
3.2 DESIGN PRINCIPLE OF THE ATTENTION-BASE PREDICTOR
The goal of the predictor is decide whether freeze layer or not at specific training iteration. While existing approaches measure the gradient norm change between two timestamps (Liu et al. 2021). the patterns of long-term converging history are missed. To this end, is desirable leverage the long-term weight history sequence for the prediction. Specifically, the predictor is designed to capture the generic converging pattern within the model training histories. We can define the task of the predictor as follows:
Freezing Prediction Task (Definition #1). For a layer I, whose weights at timestamp ý can be denoted as WV:: Given a sequence of its weight history (W7))- timestamp 4, yield positive decision to freeze the layer at the current iteration (i.e., di = 1) the layer is ready to be frozen, and yield negative decision (i.e., df 0) if the layer needs further training.
The task is non-trivial due to noisy weight history information within the input sequence. Since the training model does not monotonically converge the optimal solution, there exist scenarios such that: the history weight W from iteration n is less optimal than the history weight W7n from iteration m, even if the W updated from W (i.c., n m.) We call weights like WT" as noisy weight history. These noisy weight histories represent incorrect converging directions, thus introducing unnecessary noises. We illustrate an example of the scenario Figure a). To this end, the predictor should avoid leveraging these misleading weight histories and selectively focus on other weight histories that provide more accurate training information toward the optimal weights.
We apply an attention-based predictor to meet the requirement. The predictor ranks the information from each timestamp (i.e., sampled iteration) and adaptively aggregates weight history from the input sequence. The aggregation is input-dependent and can select the history from those preferred timestamps. The detailed workflow is discussed in Section 3.3
Layer Tailoring. To minimize the overheads introduced by the predictor, we need to make the predictor lightweight. Instead of assigning individual predictors for each layer, we intend to use only one predictor to different layers, thanks to the attention mechanism that makes the predictor can learn the generic convergence pattern. However, this raises another issue: how to fit weight information with different sizes from different layers into fixed-sized MLP predictor?
4

---

Published as a conference paper at ICLR 2023
0.500
4.000
3.200
3,000
2.500
2,000
50
[1.002 AE 60.00
(S.00) 4.440
(4.00) 3.84-)
(17 6.E
(4 6.:4)
(4E0) a.E-4
(3E0) 2 2E400
(3200) (32-0)-1 -1:440
2.5.0
12. 3.5-3
(X RA 4.00
(4E SA SE:U
JA8-55 6.5-11
(RE-45 9.5000
(BEE-A 1.541
[1.002 0E:00
(0.540) 46-00
(4.50) (184))
(4.50) 4E.0)
(4.20) +E.00
14 a
(9 a.:00
1. 1.440
11 2.500
RE:SA 1.543
A. 4 43)
4:400
1A:-45 15442
(.E
(7.5
I...
Gradient Value
Gradient Value
(a)
Figure 3: The histogram of the gradient value frequency distribution of a network layer at a certain point during the training. Figure (a) counts all the parameters of the layer, while Figure (b) counts the randomly selected 1024 parameters. These gradient values belong to the 30th CONV layer of ResNer50 and were sampled during the training of ResNet50 CIFAR-IO.
To solve the above issue, conduct experiment explore the relationship between layer's parameter subsets and all its parameters. We observe the gradient value frequency distribution since the gradients the most direct factor that determines the weight updates in the training process. As shown in Figure 3 we can observe that, for a network layer, the gradient distribution of the randomly selected weights subset is highly similar the gradient distribution of all the weights. Therefore, we make the assumption that the updating pattern of parameter subsets can represent that of the whole layer. This characteristic provides us with the feasibility to design the layer tailoring technique. In specific, we randomly sample the weights from each layer to tailor all the layers to a uniform size (e.g., 1024) and fit the sampled weights into the predictor model. With our layer tailoring, the predictor can be shared by the layers with different sizes.
3.3 WORKFLOW OF THE LIGHTWEIGHT ATTENTION-BASEE PREDICTOR
The overview of the detailed predictor workflow depicted in Figure 2t) For each historical
weight W?' (sampled parameter of layer at timestamp j E {0, 1, f)), the predictor first encodes
them into a query vector Q/ key vector K/. and value vector V/², following Equation | in which MLPp///o()) denotes trainable three-layer Malti-Layyr-Pereepttee
K/- MLP.(WW) of (W)) v/ MLP.. 10.1. ,t)
(I)
Next, computes the correlation score af between the current time (i.e., iteration f) and each pre- vious timestamp j. Specifically, it conducts dot-product between query vector at time I (i.e., QD with key vectors from timestamp (i.e., K/), then gets normalized attention score of by softmax across all previous timestamps, following Equation 2
af = Qi KK, E 10,1,-...1 .t)
(2)
exp (ap(
af Σ exp((tt)))* .j E 10,1, .t.
The value vector from each timestamp j (i.e.. o/) is weighted by the attention score a/ and accu- mulated to the final context vector Cf, following Equation 3 Hence, the aggregated context vector keeps more information from those with higher attention scores while paying less attention those with low scores.
C
V
In the end, the prediction module .MPP L.P(-) (i.c., a three-layer MLP followed by a softmax) com- putes confident scores of freezing or not as two-element vector. We choose decision with higher confidence, as shown in Equation continue its training at the moment. The predictor is lightweight in terms of both computing and memory overhead, Details of the model size and overheads are discussed in Appendix C
If di = 1, we will freeze layer at timestamp E; otherwise, we
df
5

---

Published as a conference paper at ICLR 2023
3.4 TRAINING THE PREDICTOR
In order to not seize the computational resources during the model training, we decide to train the predictor offline and then use it during the training process instead of jointly training the predictor. To train the attention-based predictor, we need to generate a dataset ourselves. As mentioned in Section 3.2 we use the layer weights as input data and then apply a layer representational similarity- based method to label the input data.
Layer Representational Similarity. As training proceeds, the model layers are continuously being updated, and the representational similarity between a layer of the model being trained and the corresponding layer of the well-trained model will increase. The representational similarity indicates the feature extraction capability of specific layer. We adopt a widely-used index Centered Kernel Alignment (CKA) to indicate the representational similarity of two layers (Korbblihh et al., 2019). CKA is obtained by comparing the output feature maps of two layers under the same input image batch. It is calculated as:
Y) = -|>=>>||>>]>>>>>>>>>>>>>>> / xIe |
(5)
where X and Y are the output feature maps from two layers, and || || represents the square of the Frobenius norm of a matrix. A higher CKA value indicates that the two layers will output more similar feature maps with the same inputs. Further, if the CKA value of a layer stabilizes during the training process, we consider this layer converged and ready to be frozen.
Figure A the CKA value curve of several CONV layers as training proceeds. As one can observe from the figure, after a period of fine-tuning, the layer's CKA stabilizes, which means that the feature extrac- tion capability of those layers hardly changes even if
Repreconational Simllarity
0.0
0.9
layert0
the training continues. At this moment, freezing those
0.7
kayr200
layer30
layers will have little impact accuracy. Morcover, the similarity with the well-trained model is not mono-
0.6
layo400
layor50
0.5
tonically increasing but fluctuating, which correlates
10
with our discussion in section 3.2 that it is not always the most recent model that has the most significant im- pact on the current model parameters. We can also observe from the figure that different layers require different training epochs to stabilize, and basically the layers in the front stabilize faster than the layers in the back (e.g., layers 10 and 20 stabilize faster than layers 40 and 50). But it is interesting to see that the layer in the back is possible to stabilize faster than the front layer (e.g., layer 30 stabilizes faster than layers 10 and 20). This is due to the presence of residual connections in the ResNet architecture, which allows some of the later layers to behave like the earlier layers (Veit et al. 2016). These observations demonstrate the need for an adaptive layer freezing method, which is exactly what the SmartFRZ does, rather than forcing sequential freezing from front to back. Note that we cannot di- rectly use this representational similarity- guided freezing because cannot obtain the well-trained reference model in advance in practice.
Training Epochs
Figure 4: CKA variation curve as training proceeds. Result is obtained while training ResNet50 on CIFAR-10 and the ResNet50 is pre-trained on ImageNet.
Generating the Training Dataset. We first train a model on certain dataset and use it as a well- trained reference model for CKA calculation. Then we train the same network on the same dataset again, in which we periodically calculate the CKA value between the well-trained model and the model under training for all the layers. With the CKA value, we are able to determine whether to freeze a layer or not. In this way, each piece of training data includes a sequence of historical weights of a tailored layer and a label indicating whether to freeze that layer or not at the moment of the latest record of this sequence.
We conduct an experiment to show the effectiveness of labeling the data through layer representa- tional similarity. Specifically, we first train and obtain well-trained ResNet50 (He et al., 2016) and VGG11 (Simonyan & Zisserman 2015) models on the CIFAR-100 (Krizhevsky & Hinton 2009) dataset. Then we train another ResNett0 and VGG11 model with CIFAR-100 100 dataset again and freeze the network layers under the guidance of layer representational similarity during the training process. In this experiment, the freezing is conducted at the end of each epoch. Table [ shows the experimental results of the training with and without layer freezing. As shown in the table, training

---

Published as a conference paper at ICLR 2023
Table 1: Comparison of model accuracy and total computation of the training process using different methods: full training and layer representational Limilarity-guidee freezing. Both models are pre- trained on the ImageNet dataset and then trained for 10 epochs CIFAR-100.
ResNet50
VGGII
Method
Accuracy
Comp. (TFLOPS)
Accuracy
Comp. (TFLOPS)
Full Training
81.68%
12,360
74.95%
22,950
Similarity-Guide Freezing
81.75%
6,454
74.89%
12,811
the networks using the representational amnilarity-guidee freezing method can significantly reduce the computation cost (c.g., it saves 48% computation for training ResNet50) while maintaining the model accuracy, demonstrating the effectiveness of our similarity-guided labeling method and the correctness of the generated dataset.
4 EVALUATION
4.1 EXPERIMENTAL SETUP
In this section, we evaluate the proposed SmartFRZ framework in both computer vision and lan- guage domains. In CV domain, we use three representative CNN models ResNet50, VGG11, and MobileNetV2 (Sandler et al. 2018), and a vision transformer model DeïT-T (Touvron et al. 2021).
And we use three widely-used datasets ImageNet (Deng al. 2009). CIFAR-10, and CIFAR-100
In NLP domain, we fine-tune the pre-trained BERT-base model (Kenton & Toutanoval 2019) using two datasets MRPC (Dolan Brockett (2055) and CoLA (Warstadt et al. 2019) in GLUE bench- mark (Wang et al. 2019). We conduct our experiments on two servers: i) server with 8x NVIDIA RTX 208011 GPUs is used for the experiments on ImageNet dataset and ii) an NVIDIA Tesla P100 GPU server is used in all other experiments. Both the predictor and the target networks are trained using the SGD optimizer with momentum. The training data is divided into batches with a size of 32. In CNN models, we freeze the BN layer together with its corresponding CONV layer to avoid unnecessary computation costs in back- back-propagation. The attention- trained once on ImageNet using ResNet50 and then used in different models and datasets. The at- tention window size for the predictor is 30. And we tailor all the layers into the size of 1024 by random sampling to fit into the generic predictor. All the accuracy results in this paper the aver- age of 5 runs using different random seeds. And the overhead introduced by predictor is included in the results. benchmark is defined as a model-dataset combination (e.g.. VGGI1+CIFAR-100)
lightweight predictor is
4.2 MAIN RESULTS
Fine-tuning. Table 2 shows the results of different freezing methods (Linear Freezing (Brock al., and
2017) and AutoFreeze (Liu et al., (021)). Linear freezing is a manually determined sequential freez- ing method. In our experiments, we set the number of frozen layers to increase according to the num- ber of training epochs and let its total training time similar to that of SmartFRZ for fair comparison as it is highly rely on manually predefined freezing configurations. AutoFreeze is also sequential freezing scheme and uses the variation of gradients norm as the metric to guide the freezing. More details of the hyperparameters of these two methods are provided in the Appendix SmartFRZ can significantly reduce computation cost and training time without compromising accuracy compared to full training. And it consistently outperforms Linear Freezing and AutoFreeze.
Compared to Linear Freezing, SmartFRZ provides up to 1.4% higher accuracy (VGG11 I CIFAR- 100), with 0.7% higher on average, while consuming similar time and energy. This is because Linear Freezing freezes layers in fixed order at pre-determined time, so some layers that are not well-trained might also be frozen. Moreover, linear freezing performs poorly on some benchmarks (i.e.. VGG11 CIFAR-10, 10, VGG11 CIFAR- and MobileNetV2 CIFAR-100). This is because linear freezing is highly dependent on a predefined freezing configuration, which does not adapt to different scenarios. For example, the training epochs required converge is definitely different when training different model on different datasets with different complexity (e.g.. ImageNet vs. MNIST), the freeze configuration should also be different.
Compared to AutoFreeze, SmartFRZ reduces up to 20.4% time (MobileNetV2 CIFAR-10 bench- mark) and 24.7% computation cost. (VGG11 CIFAR-10 benchmark) while providing similar aceu-
7

---

Published as a conference paper at ICLR 2023
Table 2: Comparison of different freezing methods. All the models are pre-trained on ImageNet dataset. Then the CNN models and ViT model are trained for 10 epochs and 100 epochs to converge, respectively, using cosine annealing learning rate scheduler according the training epochs.
CIFAR-10
CIFAR-100
Model
Method
Time Computation (TFLOPs)
Time
Computation
Accuracy
Accuracy
(Second)
(Second)
(TFLOPs)
Full Training
H0.1025
2,594
12,360
81.68%+0..1 10.5%
2,500
12,360
Linear Freezing
96.05%:1.225%
1,980
8.247
81.48%
1,844
7,085
ResNet50
AutoFreeze
96.24%:10.388%
2,424
8,716
81.30%10.48%%
1,963
7.716
SmartRR (Ours)
96.12%:10.11 1..%%
1,955
8,122
81.73%:10.22%%
1.787
6,398
Full Training
93.36%++..17%.
2,698
22,950
74.95%++0.119
2,703
22,950
Linear Freezing
92.35%:10.119%
1,587
10,291
73.26%
1,846
12,307
VGGI1
AutoFreeze
93.08%
1,895
13.350
74.74%.10.38
2,027
13,946
SmartRZ (Ours)
93.58%
1,554
10,059
74.66%.10.22
1,831
12,121
Full Training
94.13%110.10%
I,369
960
76.52% 10.21%
1,371
960
Linear Freezing
94.19%.10.26%%
977
567
75.59%110.27
1,000
540
MobileNerV2
AutoFreeze
94.26%
1,245
678
76.43% 10.42%
1,248
652
SmartFRZ (Ours)
94.03%++0.177
972
561
76.73%=0.20
986
532
Full Training
97.48%
14,603
32,400
85.03% 10.28%
14,628
32,400
Linear Freezing
97.06%%.10.23%%
8,760
16,290
83.89%%10.199
9,956
17,542
DeiT-T
AutoFreeze
97.35%::10.465.
10,368
17.786
84.59%
11,154
18,710
SmartFRZ (Ours) 97.65%
8,662
15,529
84.82%
9,599
16,636
racy, The reasons behind this are three-fold. First, SmartFRZ applies an attention-based predictor that can precisely determine the moment to freeze a layer. Second, AutoFreeze only freezes the layer where all the layers in front of have been frozen. However, as discussed in Section [3.4] one layer might stabilize earlier than some of the layers in front of it. Therefore, forcing sequential freezing may result in wasting computation. Third, AutoFreeze incurs considerable overhead as it needs to compute the 1.2 Norm for all the gradients. Specifically, this overhead consumes 20%, 12%, 11% and 3% the total training time while training ResNet50, MobileNetV2, DeiT-T, and VGGII, re- spectively. As result, AutoFreeze only shows marginal improvement in time consumption the MobileNetV2 benchmark. We also evaluate the SmartFRZ framework in NLP domains and the experimental results are provided in the Appendix A.1
Training From Scratch. addi- tion to fine-tuning, we also investi- gate the performance of our Smart- FRZ framework in training model from scratch. As shown in table SmartFRZ saves training time by 22.4% on average without compro- mising accuracy compared full training. SmartFRZ also consis- tently outperforms linear freezing and AutoFreeze in training from scratch. Compared to linear freez- ing, SmartFRZ provides 2% higher accuracy with 6.7% less time consumption on average. The performance of linear freezing degrades severely compared to fine-tuning. This is because the robustness of the model in the training from scratch experiment is much lower than that of the pre-trained model in the fine-tuning exper- iment. Linear freezing, which relies heavily on manually predefined freezing configuration, cannot handle this more complex training scenario. Compared to AutoFreeze, SmartFRZ provides a sig- nificantly 3.78% higher accuracy with 11.8% less time consumption. Besides the CNN model, we also evaluate the effectiveness of our SmartFRZ framework in training the vision transformer model De from scratch, and the experimental results are shown the Appendix A.2
Table 3: Comparison of different freezing methods in training model from scratch (160 epochs for ResNet50-CIFAR-ID and 100 epochs for DesNet550-ImageNon
CIFAR-100
ImageNet
Method
Time
Time
Accuracy
Accuracy
(Min.)
(Min.)
Full Training
77.48%
711 76.89%11..12%
780
Linear Freezing
74.99%=10.1998
615 75.45%
626
AutoFreeze 72.82%%-0..55%
630
74.06%
682
SmartFRZ (Ours) 77.66%8:10002288
538
76.80%:+0.26%
621
4.3 MEMORY COST REDUCTION AND FREEZING PATTERN
Our SmartFRZ framework not only reduces training computation costs by freezing layers, but also reduces memory costs due to the reduction intermediate data generated during back-propagation Figure ((a) shows the memory cost as training proceeds. We observe from the figure that
8

---

Published as a conference paper at ICLR 2023
11
Memory Cost Cost (GB)
VGG11
Freezing Moment (Epoch)
ResNet50
MobileNetV2
2
3
1
Training Epochs
Network Layer (b)
(a)
Figure 5: (a) Memory cost as training proceeds. Results are obtained by training VGGI1, ResNet50, and MobileNerV2 on CIFAR-100 dataset. (b) The frozen moment for each layer while training ResNet50 on CIFAR-100.
Table Comparison of the model accuracy when the attention window size for freezing pre- diction varies. The results are obtained using ResNer50 and VGGI1 on CIFAR-100. The tai- lored layer size set to 1024.
Table 5:
Comparison of the model accuracy when the uniform tailored layer size varies. The results are obtained using ResNet50 and VGGI1 on CIFAR-100. The attention window size is set to 30.
Window Size
Tailored Layer Size
Model 20
Model 512
30
40
50
1024
2048
4096
ResNet50
81.17%
81.73%
81.75%
82.11 %
ResNet50 VGG11 73.98%
81.28%
81.73%
81.33%
81.69%
VGG11
74.45%
74.66%
75.11%
74.67%
74.66%
74.48%
75.12%
SmartFRZ can significantly save more than 80% memory for ResNet50 and about 50% memory for VGG11 and MobileNerV2. Considering the limited GPU memory capacity, our approach enlarges the potential of accommodating multiple DNN tasks within single GPU. Figure 5))) shows the freezing moment for each layer while training ResNet50 on CIFAR-100. As shown in the figure, although there are still some layers in the back getting frozen earlier than some front layers, Smart- FRZ tends freeze the front layers earlier than the layers in the back. This is because the front layers mainly extract low-level features (e.g.. detecting edges), while the back layers extract high- level features (Yosinski et al., 2014; Zeiler Fergus) 2014). And the low-level feature extraction capability can be inherited from the pre-training process more easily.
4.4 SENSITIVITY STUDY AND OVERIEAD ANALYSIS
We investigate the effect of adopting different attention window sizes (i.e., the number of historical data used) to predict the freezing probability. Table shows the accuracy when the attention win- dow size varies. As one can observe, SmartFRZ consistently delivers a high model accuracy under different window sizes. demonstrates the robustness of SmartFRZ handling scenarios dif- ferent attention window sizes. This is because the attention mechanism adaptively selects valuable information within the long window ranges.
We also study how the uniform tailored layer size affects our SmartFRZ framework. As shown in table 5 our proposed SmartRRZ framework remains robust except for very slight accuracy drop in accuracy (i.e., less than 1%) when the layer size is 512. And the reason behind this is that if the tailored layer size too small, the sampled parameters might hard to represent the status of the whole layer's 's parameters. Therefore, we tailor the layer to the size of at least 1024, which samples richer layer information while yielding negligible overheads (Details discussed in Section B
CONCLUSION
This work considers an efficient training technique layer freezing on both fine-tuning and training from scratch scenarios. We propose an attention-based layer freezing framework SmartFRZ. The key idea to leverage the training histories to predict the freezing probability of a layer during the train- ing process. Our attention-based layer freezing predictor can learn comprehensive information about the network layer. The predictor is trained offline on dataset with layer weights as input data which is labeled by a layer representation similarity-base based method. Our extensive experiments demonstrate that SmartFRZ significantly saves training time and computation by automatically freezing layers without compromising accuracy.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
The authors would like thank the anonymous reviewers for their constructive feedback and sug- gestions. This work is supported in part by NSF grants 1146, #2154973, #1725657, and #1909172.
REFERENCES
Mohamed Abdel-Bassed Weiping Ding, and Laila Abdel-Fatah. The fusion of internet of intel- ligent things (ioit) in remote diagnosis of obstructive sleep apnea: A survey and new model. Information Fusion, 61:84-100, 2020.
Younes Akbari, Noor Almaadeed, Somaya Al-Maadeed, and Omar Elharrouss. Applications, databases and open computer vision research from drone videos and images: a survey. Artificial Intelligence Review, 54(5):388733938, 2021.
Roger Bemelmans, Gert Jan Gelderblom, Pieter Jonker, and Lue De Witte. Socially assistive robots in elderly care: a systematic review into effects and effectiveness. Journal of the American Medical Directors Association, 13(2):1 14-120, 2012.
Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Freezeout: Accelerate training by progressively freezing layers. arXiv preprint urXiv:1706.04988 2017.
Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy. Tvm: An automated end-to-end optimizing compiler for deep learning. In 13th USENIX Symposium 2018.
Operating Systems Design and Implementation (OSDI 18), pp.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. Ieee, 2009.
Ha Manh Do, Minh Pham, Weihua Sheng, Dan Yang, and Meiqin Liu. Rish: A robot-integrated smart home for elderly care. Robotics and Autonomous Systems, 101:74-92, 2018.
Bill Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Third International Workshop on Paraphrasing (IWP2005), 2005.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. International Conference on Machine Learning, pp. 2943-2952. PMLR, 2020.
Masoud Faraki, Xiang Yu, Yi-Hsuan Tsai, Yumin Suh, and Manmohan Chandraker. Cross-domain similarity learning for face recognition unseen domains. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15292-15300 2021.
Cristina Gorrostieta, Richard Brutti, Kye Taylor, Avi Shapiro, Joseph Moran, Ali Azarbayejani, and John Kane. Attention-basse based sequence classification for affect detection. Interspeech, pp- 506-510, 2018.
Chaoyang He, Shen Li, Mahdi Soltanolkotabi, and Salman Avestimehr. Pipetransformer: Automated elastic pipelining for distributed training of transformers. arXiv preprint arXiv:2102.031611 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Ruimin Ke, Yifan Zhuang, Ziyuan Pu, and Yinhai Wang- A smart, efficient, and reliable park- ing surveillance system with edge artificial intelligence on iot devices. IEEE Transactions on Intelligent Transportation Systems, 22(8):4962 4974, 2020.
Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of NAACL-HLT, pp. 4171- 4186, 2019.
10

---

Published as a conference paper at ICLR 2023
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pp. 3519- 3529. PMLR. 2019.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
Jaejun Lee, Raphael Tang, and Jimmy Lin. What would elsa do? freezing layers during transformer fine-tuning arXiv preprint 1rXiv: 1. 1.03090, 2019.
Yehao Li, Ting Yao, Yingwei Pan, and Tao Mei. Contextual transformer networks for visual recog- nition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Yuhan Liu, Saurabh Agarwal, and Shivaram Venkataraman. Autofreeze: Automatically freezing model blocks to accelerate fine-tuning, arXiv preprint arXiv:2102.01388 2021.
Xiang Long, Chuang Gan, Gerard De Melo. Jiajun Wu, Xiao Liu, and Shilei Wen. Attention clusters: Purely attention based local feature integration for video classification. In Proceedings of the IEFE conference computer vision and pattern recognition, pp. 1834-7843. 2018.
Wei Niu, Xiaolong Ma, Sheng Lin, Shihao Wang. Xuehai Qian, Xue Lin, Yanzhi Wang, and Bin Ren. Patdnn: Achieving real-time dnn execution on mobile devices with pattern- weight pruning. In Proceedings of the Twenty-Fifth International Conference Architectural Support for Programming Languages and Operating Systems, pp. 907-922, 2020.
Prabodh Panindre, Vijay Gandhi, and Sunil Kumar. Artificial intelligence- based remote diagnosis of sleep apnea using instantaneous heart rates. In 2021 11th International Conference Cloud Computing, Data Science & Engineering (Confluence), pp. 169-174. IEEE, 2021.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiae Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 45 4510-4520, 2018.
Sheng-syun Shen and Hung-yi Lee. Neural attention models for sequence classification: Anal- ysis and application key term extraction and dialogue act detection. arXiv preprint arXiv:1604.0007. 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
Zeyu Sun, Jie M Zhang, Mark Harman, Mike Papadakis, and Lu Zhang. Automatic testing and improvement of machine translation. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, pp. 974-985, 2020.
Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers amp; distillation through attention. In International Conference on Machine Learning, volume 139, pp. 10347-10357, July 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. Advances neural information processing systems, 29, 2016.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019. URL httph:t://tteeeiiiiiii
hett/Fozuu?1dd=xJJ442222.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641, 2019.
Yawen Wu, Zhepeng Wang. Yiyu Shi, and Jingtong Hu. Enabling on-device enn training by self- supervised instance filtering and error map pruning. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 39(11):3445 3:33455-3457, 2020.
11

---

Published as a conference paper at ICLR 2023
Yawen Wu, Zhepeng Wang, Dewen Zeng. Yiyu Shi, and Jingtong Hu. Enabling on-device self- supervised contrastive learning with selective data contrast. In Design Automation Conference, pp. 655-660 IEEE, 2021.
Yukuan Yang, Lei Deng, Shuang Wu, Tianyi Yan, Yuan Xie. and Guoqi Li. Training high- performance and large-scale deep neural networks with full 8-bit integers. Neural Networks, 125:70-82, 2020.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in neural information processing systems, 27, 2014.
Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, et al. Mest: Accurate and fast memory-economic sparse training framework on the edge. Advances in Neural Information Processing Systems, 14:20838-208555
2021.
Geng Yuan, Yanyu Li, Sheng Li, Zhenglun Kong, Sergey Tulyakov, Xulong Tang, Yanzhi Wang, and Jian Ren. Layer freezing & data sieving: Missing pieces of generic framework for sparse training. arXiv preprint 2rXi9:2009, 1204, 2022.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 118-833. Springer, 2014.
Minjia Zhang and Yuxiong He. Accelerating training of transformer-based language models with progressive layer dropping. Advances in Neural Information Processing Systems, 33:14011- 14023, 2020.
Kang Zhao. Sida Huang, Pan Pan, Yinghan Li, Yingya Zhang, Zhenyu Gu, and Yinghui Xu. Dis- tribution adaptive int8 quantization for training enns. In Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence, 2021.
Zaixiang Zheng, Xiang Yue, Shujian Huang, Jiajun Chen, and Alexandra Birch. Towards making the most of context in neural machine translation. In Proceedings of the Twenty-Ninth International Conference International Joint Conferences on Artificial Intelligence, pp. 3983-3989, 2021.
12

---

Published as a conference paper at ICLR 2023
APPENDIX
A EXTENDED EXPERIMENTAL RESULTS
A.1 NLP TASKS USING BERT-BASE
In this section, provide the experimental results of different freezing approaches in NLP tasks. As shown in Table 6 SmartFrz able to significantly reduce fine-tuning time without accuracy loss. Moreover, SmartFrz delivers average of 0.86% higher accuracy than linear freezing with similar training time. Compared to AutoFreeze, SmartFrz provides 0.68% higher accuracy while consuming 10.5% less time average. These results demonstrate the superiority of SmartFiz over linear freezing and AutoFreeze.
Table 6: Comparison of different freezing methods in NL.P domain. The results are obtained by fine-tuning 5 epochs of pretrained BERT-base two datasets MRPC and CoLA, respectively.
MRPC
CoLA
Method
Accuracy
Time (Second)
Accuracy
Time (Second)
Full Training
86.51%1:..244
316
56.65%=0.18%
723
Linear Freezing
85.80%10.221%
221
55.90%=0.29%
429
AutoFreeze
86.02%:10.366%
266
56.05%=0.21%
431
SmartFRZ (Ours)
86.76%::00.55%
215
56.66%:0.224%
423
A.2 TRAINING FROM SCRATCH EXPERIMENTS USING DETT-T MODEL
Table 7: Comparison of different freezing methods in training DeiT-T from scratch on ImageNet dataset (300 epochs).
Method Full Training
Accuracy
Time (Minute)
72.21%:10.12%%
2,190
Linear Freezing
71.19%-10.24%
1,817
AutoFreeze
71.55%
1,815
SmartFRZ (Ours)
72.06%-10.200
1,801
In this section, we provide the experimental results of training the vision transformer model DeiT-T from scratch ImageNet dataset. As one observe from Table 7 SmartFRZ delivers higher accuracy (0.69% on average) than linear freezing and AutoFreeze with the lowest training time.
B CONFIGURATIONS OF LINEAR FREEZING AND AUTOFREEZE BASELINES
To show the fairness of the experiments, in this section, we provide the details of the configurations of linear freezing (Brock al. 2017) and AutoFreeze (Liu et al. 2021).
Linear Freezing. Linear freezing employs a layer-wise cosine annealing learning rate schedule without restarts, where the first layer's learning rate is reduced
partway through training (at to), and each subsequent layer's learning rate is annealed to zero thereafter. Once a layer's 's learning rate reaches zero, frozen and will not be updated afterward. The learning rate a; layer 4 (whose learning rate anneals to zero at t.) at iteration t is calculated as:
0y (l) = 0.5 (0) + cosml//1)
(6)
There is one hyperparameter in linear freezing, to. which defines which epoch the learning of the first layer reaches zero. The value of f, is linearly spaced between to and the total number of epochs. We explore the impact of the value of to. Table & shows the results with different configurations of to Following the setting in the original paper,
vary the number of total training epochs for each configuration to make the total training time of each configuration almost the same. As shown in the table, with similar training time, setting lg 0.5 provides the highest accuracy, which is also
13

---

Published as a conference paper at ICLR 2023
Table 8: Hyperparameter sweep at different to values in linear freezing (Brock et al. 2017). The results are obtained by fine-tuning ResNet50 on CIFAR-10 dataset.
Accuracy
Time (Second)
to = 0.3
95.62%
1,989
to 0.4
95.81%
1,986
to 0.5
96.05%
1,980
tg 0.6
95.97%
2,015
recommended by the original paper. To ensure fair comparisons, we have selected the best results of linear freezing as baseline to compare against our method.
AutoFreeze. AutoFreeze periodically freezes layers whose gradients norm change rate is low. There are two hyperparameters in the AutoFreeze framework. The first one the number of evaluation intervals M, which indicates the frequency of freezing layers. For example, 4 intervals/epoch means that AutoFreeze evaluates the gradient norm change rate for each active layer and freezes the layers with low gradient norm change rate 4 times per epoch. The second hyperparameter is the threshold N to determine whether a layer is converged or not. If layer's 's gradient norm change rate is in the bottom N percentile of all the active layers, it is considered converged and ready to be frozen (there is another requirement that a layer can be frozen until all the layers in front of have been frozen).
Table 9: Hyperparameter sweep at different number of evaluation intervals per epoch in Aut- oFreeze (Liu et al. )02T). The results are obtained by fine-tuning ResNet50 CIFAR-IO dataset.
Accuracy
Time (Second)
=
96. 19%
2,536
M =
96.28%
2,487
=
96.24%
2,424
M =
95.69%
2,378
In our experiments, we follow the recommendation from original paper (Liu et al. 2021) and set the N 50%. For fair comparison, we further study how the number of evaluation intervals per epoch (i.e., .M) affects the accuracy and training time. We conduct the experiments by varying the intervals per epoch from
since (Liu et al.) (202I) finds the trade-off between accuracy and training time is balanced for range of values (2 to 5 intervals/epoch). Table[]shows the experimental results, and can be observed from the table that sets the M to 4 can maintain the accuracy with high training time reduction. To this end, in main paper, we set the N to 50% and M to 4 to get the best results of AutoFreeze.
C OVERHEAD ANALYSIS
In terms of predictor size, our lightweight predictor consists of only one MLPx() (+), one MLP (.),
one LP,(..) and one M LP2(). as shown in Figure 2 In specific, the MLPa(-), MLP, (-). and M-P.(-) in equation [ which are used for encoding weight histories, are 3-layer Multi-Layer- Perceptrons (input size as 1024, hidden sizes as 256, and output sizes as 64). We adopt one layer of attention (i.e,, without stacking multiple attention layers as BERT|Kenter & Toutanova (2019) with one single head (i.e., encoding one single group of key. query, and value vectors with one group of MTPP&()). MLP,(()), and M (.) for each timestamp) in our design. The (.).:()) in equation 4 which is used for computing confident scores of freezing or not, is 3-layer (input size as 64, hidden sizes as 32, and output sizes as 2). To this end, the whole predictor consists of four 3-layer MLPs. The predictor size is about 4 MB, which is negligible compared to the training overheads. Moreover, since we tailor the layer by subsampling layer weights to fixed-size vector (e.g., size of 1024), SmartFRZ only requires hundreds of KB to store the whole model once (e.g., about 200 KB for ResNet50). As such, SmartFRZ only requires about 6 MB memory to store the historical weights for freezing prediction even with large attention window of size 30.
14

---

Published as a conference paper at ICLR 2023
The computation cost for one inference is 0.12 GFLOPs if we set the attention window size to 30 and the tailored layer size to 1024. So, it only consumes negligible less than 0.1% of total train- ing time. Noticeably, the computation complexity of our predictor is lincar-scaled the sequence length instead of quadratic-scaled since it handles the sequence classification task (which requires all-to-one attention) instead of the sequence-to-sequee
translation task (which requires all-to-all attention). The encoding stage (i.c., equation ! encodes each time stamp, yielding (tt) complexity. The aggregating stage (i.e., equation 2 equation 3 calculates weights from the current timestamp to sequences of timestamps as one-to-all procedure, and it causes (t)) as well. Furthermore, the last MLP classification steps only cause (1)). summary, our predictor has linear complexity (~ O()) in terms of sequence length instead of quadratic complexity (~ (!)). Moreover, once a layer frozen, we not need predict the freezing decision for that layer anymore and can discard its historical weights. So the computation cost and memory cost introduced by the predictor
will decrease as training proceeds.
15


Reviews:
[{"review_id": "xCj3rbPtk2", "reviewer": "Reviewer_Spyk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a layer freezing method to reduce computational time for training of ML models. An attention-based layer freezing model takes sampled parameters from a layer as input, and predicts if it should be frozen or not. The layers get dynamically frozen over the training period. Results should significant improvement over hand-crafted and heuristic methods. ", "summary_of_the_review": "Overall, a good paper with substantial improvement over baselines. ", "strength_and_weaknesses": "* Strengths:\n- Paper is well written and easy to follow\n- Experiments are extensive, with sensitivity studies, comparison with state of the art, and computation overhead measurements.\n- Proposed method is simple but effective\n\n* Weaknesses:\n- Analysis limited to CNN vision models. Unclear how well it will translate to other architectures (e.g. Transformers) or domains (e.g. Language) or objectives (e.g. Regression, Object detection)\n- A few clarifying questions: \n--- it is not clear how the layer freezing works dynamically given that all layers below the chosen layer needs to be frozen as well. This is mentioned in passing in Section 2, but not mentioned in methods.\n--- It is not clear what the training and test datasets are. It is mentioned in passing that ImageNet + ResNet50 is used for training, is that all the training required?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "uFN7Dejkd5", "reviewer": "Reviewer_D3g6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a layer freezing method to improve training efficiency by introducing attention-guided layers. SmartFRZ aggregates the historical weights using the attention-based predictor to allow the dynamic decision of freezing layers leading to the efficient training framework. The authors support their methods through empirical comparisons of existing algorithms.", "summary_of_the_review": "The authors provide empirical results regarding the efficient training framework on neural networks. However, their motivations for the methodology are not convincing to me, and their results are limited to a specific dataset (CIFAR) and the family of convolutional neural networks. Furthermore, the author's explanation of their approach needs to be clearer. Overall, it is a well-written paper, but I do not think the paper does not meet the ICLR bars. ", "strength_and_weaknesses": "Strengths\n1. The author shows that SmartFRZ achieves performance more efficient training framework than existing algorithms, such as Linear Freezing and AutoFreeze. \n2. The paper is well written in general.\n\nWeakness\n1. SmartFRZ needs to train the predictor separately from the actual network training. \n2. Limited comparisons only on CIFAR and two CNN architectures.\n3. Lack of explanations on the motivation for the suggested approaches.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "V2chVBNkUts", "reviewer": "Reviewer_QAQ7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed an interesting topic of adaptively freezing the layers during training to save training time and memory.  The proposed predictor output attention for each layer by integrating the history layer information during training. The training and freezing are alternatively performed on layers of models. ", "summary_of_the_review": "Although there are some weaknesses in this work, I still think that this work is interesting and can be an easy-to-use tool in practice.  The authors are welcome to release the code and describe the additional memory cost brought by this predictor. And if possible, results on ImageNet are also convincing. ", "strength_and_weaknesses": "Strength:\nThe motivation is clear and the topic is useful. The proposed predictor can dynamically choose which layer to be frozen to enable training efficiency and memory saving. \n\nWeakness:\nThe predictor also consumes the memory during normal training. Seems that there are no numerical results about this aspect. Do all reductions of memories or time reported in the paper have taken the predictor part into accounting？ \nHow do you choose the structure and config of the predictor according to different models that have very various parameters? \nTraining from scratch experiments did not include the results on the ImageNet dataset. Are there any reasons for this?  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TF3Dl7hBY0", "reviewer": "Reviewer_VTwh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an interesting approach to predicting freezing layers during training by training a meta-predictor. The attention based meta-predictor takes in weight history and predicts if a layer should be frozen or not. The predictor is also pretrained on a dataset generated using the CKA similarities between a well trained and novel network. The authors show improvements over other layer freezing approaches that rely on heuristics for image classification on a variety of architectures.", "summary_of_the_review": "Overall, the idea of using a meta predictor for layer freezing is inspired. However, the experiments are not documented well, and several details are missing. In addition, it is unclear why SmartFrz outperforms other freezing algorithms given that layers are still being frozen out of order. It would be also be of independent interest to consider more tasks such as language modelling and architectures like ViTs to see if the predictor is transferable. \n\nI am currently leaning towards a weak reject, but am open to changing my opinion if the authors can provide reasonable explanations for my concerns listed above.", "strength_and_weaknesses": "**Strengths**\n1. The meta-predictor is cleverly designed to consider to be layer agnostic by subsampling weights to a fixed vector. \n2. The performance of SmartFrz is better than Linear or Autofreeze for the given experiments.\n3. The authors perform a fairly complrehensive series of experiments showing the effect of sequence lengths.\n\n**Weaknesses and Qeustions**\n1. The main advantage of freezing sequentially is that gradients do not need to be calculated. How are the authors showing lower tFlops given that the gradients still need to be calculated for the frozen layers? I am curious to see if the implementation to see how exactly the savings occur.\n2. The authors also do not mention the hyperparameters and the experimental setup for Linear and Autofreeze. This is extremely important as Autofreeze appears to be very susceptible to hyperaprameter values.\n3. it appears that the attention based predictor is a Resnet50 trained on Imagenet. How transferable is this predictor to other datasets? While Cifar-10/100 are a good baseline, it would be a valuable addition to include results on a variety of datasets to confirm the performance.\n4. Further, the authors claim that they use CKA to train the predictor. Do they use CKA with the subsampled weights or the original size? Also are there any thresholds that are used to generate the training dataset for the predictor? Fig. 4 shows that CKA is not a monotonic curve, and the choice of this threshold may affect the predictor adversely.\n5. The authors also claim that they report the average results over 5 runs. However, there are no error bars or std deviation reported for any of the numbers. I suggest that the authors add these in order to determine the significance.\n6. Weight histories are also dependent on the optimizer used.  What optimizers does SmartFrz work with?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0HLlD9q5VVE", "reviewer": "Reviewer_Uzai", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper propose a dynamic layer freezing technique during DNN training. Specifically, they adopt attention-based predictor to predict which layer should be freezed. The predictor is jointly applyed to each timestamp.", "summary_of_the_review": "In summary, I think that this paper provide a decent point of view for efficient training method. But the idea is not very novel as there are also many other works working on freezing layers. But the proposed methods beat several baselines.\n\nAlso, it is not clear whether it is worth to leverage attention model to help CNN training. It should be reversed IMO.\n\nI suggests that the author could consider adding more ViT-based exps, where their attention can be naturally server as predictors.\n\nIf that works, I would increase my score as it shows that your methods can be generalized to ViT models.", "strength_and_weaknesses": "Strength:\n\n* A valid efficient training method towards cheaper DNN training.\n\nWeakness or Questions:\n\n* I doubt the necesarity of using attention-based model to predict the layers. Actually, attention-based model suffers from the quadractic complexity as compared to the local feature extractor, CNNs. So use Attention model to guide CNN training looks overkill to me. Instead, we should use small models to predict much larger models' freezing schedule.\n\n* How is the overhead of such a attention based model? Since it will be repeatedly used for prediction, thus it must have to be negligible overhead otherwise I cannot see the point.\n\n* The most direct baseline is to reduce the layer numbers and compare with your method, and randomly drop some layers during training process. I am not sure whether your predicted schedule beats the random one.\n\n* Also, the exps are mostly conducted on CNNs, how about ViTs? We should natually reuse the attention based model as both predictors and main contributors to the model performance in the ViTs.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%