A Base Model Selection Methodology for Efficient Fine-Tuning
Authors: Yosuke Ueno
Masaaki Kondo
Venue:  (2020)
Keywords: transfer learning
fine-tuning
parameter transfer

Abstract: While the accuracy of image classification achieves significant improvement with deep Convolutional Neural Networks (CNN), training a deep CNN is a time-consuming task because it requires a large amount of labeled data and takes a long time to converge even with high performance computing resources.
Fine-tuning, one of the transfer learning methods, is effective in decreasing time and the amount of data necessary for CNN training. It is known that fine-tuning can be performed efficiently if the source and the target tasks have high relativity.
However, the technique to evaluate the relativity or transferability of trained models quantitatively from their parameters has not been established. In this paper, we propose and evaluate several metrics to estimate the transferability of pre-trained CNN models for a given target task by featuremaps of the last convolutional layer.
We found that some of the proposed metrics are good predictors of fine-tuned accuracy, but their effectiveness depends on the structure of the network. Therefore, we also propose to combine two metrics to get a generally applicable indicator. 
The experimental results reveal that one of the combined metrics is well correlated with fine-tuned accuracy in a variety of network structure and our method has a good potential to reduce the burden of CNN training.

Under review as a conference paper at ICLR 2020
A BASE MODEL SELECTION METHODOLOGY FOR EFFICIENT FINE-TUNING
Anonymous authors Paper under double-blind review
ABSTRACT
While the accuracy of image classification achieves significant improvement with deep Convolutional Neural Networks (CNN), training a deep CNN is a time- consuming task because it requires large amount of labeled data and takes a long time to converge even with high performance computing resources. Fine- tuning, one of the transfer learning methods, is effective in decreasing time and the amount of data necessary for CNN training. It is known that fine- fine-tuning can be performed efficiently if the source and the target tasks have high relativity. How- ever, the technique to evaluate the relativity or transferability of trained models quantitatively from their parameters has not been established. In this paper, we propose and evaluate several metries to estimate the transferability of pre-trained CNN models for a given target task by featuremaps of the last convolutional layer. We found that some of the proposed metrics are good predictors of fine-tuned ac- curacy, but their effectiveness depends on the structure of the network. Therefore, we also propose combine metrics get generally applicable indicator. The experimental results reveal that one of the combined metrics is well corre- lated with fine-tuned accuracy variety of network structures and our method has a good potential to reduce the burden of CNN training.
INTRODUCTION
As convolutional neural networks (CNNs) have been widely used in many applications, CNN model tends to become deeper and larger which necessitates a large amount of training data. This makes the number of arithmetic operations required for training huge (Canziani et al. (016)) re- sulting in long training time and large energy consumption. Moreover, there are large demands to process CNNs in edge devices where training a CNN model a challenging task due to there lim- ited computing resources. Having efficient CNN learning methodology for embedded systems is very important. Hence, techniques to reduce the computational complexity of CNN training with maintaining inference accuracy have been studied intensively far. These include, for example, quantization (Moons et al. 2016) and pruning (Molhaaovv et al.) (017))
Since obtaining enough training data is usually difficult and expensive in real situations, there is a great demand to create an efficient CNN training methodology. This is the motivation of using transfer learning which optimizes a trained model for target domain by transferring information from a related source domain (Caruanal 1997). Fine-tuning, one of the transfer learning techniques, a promising approach to reduce the burden of CNN training.
In fine-tuning, source CNN model trained with large labeled dataset (source task) is adapted to different application (target task) to create new model (target model). For example, ImageNet (Russakovsky et al., 2015) is becoming de facto dataset as source task for image recognition target tasks. In fine-tuning, is a common practice to truncate the last fully connected layer (FC layer) of the pre-trained network, replace it with new one that is relevant to the target task, and freeze the weights of the other layers of the pre-trained network.
It is known that fine-tuning can be performed efficiently if the source and the target tasks have high relativity (Caruana 1997). we choose a suitable source model for a given target task from multiple source task candidates, time and energy consumption for training are likely to be reduced. There has not been enough research effort to evaluate whether the transferred parameters are suitable for another task so far. In this paper, we investigate several metrics to evaluate "transferability" of source

---

Under review as a conference paper at ICLR 2020
CNN models for certain target tasks quantitatively without actually performing time consuunng fine-tuning process. The evaluation results reveal that some of the metrics are well correlated with fine-tuned accuracy and our technique will greatly reduce the burden of CNN training.
2 RELATED WORK
Kornblith et al. (0018) examined the fine-tuning effect on using ImageNet IK classification task as a souree task for various target tasks. They evaluated the correlation between the accuracy of the source task (ImageNet) and fine-tuned target tasks with 13 classification models on 12 image classification tasks. positive correlation was observed and the correlation coefficient was very high on average (r² 0.86). Based on the results, they claimed that ImageNet accuracy is good predictor of fine-tuning performance. their research, the focus is on the fine-tuned performance after target tasks are actually trained. Compare to that, we consider fine-tuned accuracy without actually performing a cime-conuuiing training phase. Moreover, they did not apply for small datasets such as Stanford Cars and FGVC aireraft which do not necessarily have high fine-tuned performance even using ImageNet as a source task, meaning a model pre- pre-trained with a large data set does not always give high fine-tuned accuracy. This implies that selecting an appropriate source task for a given target task is very important.
In Mahajan al. (2018), they proposed method of pre-training with large amounts of images on Instagram using its hashtag as a label. They found that some of the design choices that are made in current network architectures are too tailored ImageNet-Ik Ik classification. They also mentioned that pre- pre-training with a large amount of weakly labeled data improves classification task, but they did not apply it for detection or segmentation tasks.
Sato et al. (2018) raised a problem of negative transfer in the medical imaging area. They found that the results of transfer learning sometimes become worse rather than learning from scratch. This may also suggest that having appropriate pre-trained source tasks are necessary to train better models with transfer learning.
Transferring the data in source domain to target domain is called "instance transfer". In Liu et al. (2018), they proposed a new technique to transfer knowledge safely by measuring relativity between the source task and the target task.
Bau et al. (017)) quantified interpretability of latent representations of trained CNNs by evaluating their featuremaps with method similar to the segmentation. It is judged by whether featuremaps can identify a certain concept and the number of identified concepts indicates interpretability of the target model. In this research, the authors revealed that CNNs identify higher-level concepts such as objects at higher-level convolutional layers and lower-level concepts such as textures at lower-level convolutional layers.
Morcos et al. (2018) reported that a highly selective neuron or featuremap which responds to input data in certain class is not important for higher accuracy. Instead, neurons featuremaps acti- vated by inputs of a variety of classes are more important. They pointed out the grandmother cell hypothesis in the neural network may not be correct.
Hinton et al. 2015) proposed a knowledge distillation approach which compresses the knowledge of large computational expensive model (teacher model) to create smaller neural network model (student model). The idea of the knowledge distillation is to train student model with a transfer set provided by the teacher model. It is becoming a popular technique to compress neural net models. The base idea of transferring knowledge of larger models to smaller model to get a computationally efficient model is originally proposed by Buciluă al. (2006). Fuurlaneloo et al. (2018) has reported that repeating knowledge distillation leads to better generalization ability of trained models. In addition, Zhang et al. (2018) has proposed a new learning method with distilling the knowledge of multiple models mutually get higher performance models.
In many previous research efforts suggested that there exists an optimal model architecture for a target task and pre-trained model using source task greatly affects the fine-tuned accuracy of the network in transfer learning. To the best of our knowledge, this is the first attempt to evaluate transferability of pre-trained models without actually performing fine-tuning. This is one of the
2

---

Under review as a conference paper at ICLR 2020
novel points in this work. Besides it, technique be used to truncate CNN model as presented in Appendix A.2 The method could another approach to compress CNN models.
MODEL SELECTION METRICS
In this paper, propose the metrics evaluate the transferability of trained CNNS for certain target task without actually retraining the networks to accelcrate a fine-tuning process. We calculate the metrics based the outputs of the last convolutional layer (featuremaps) for each architecture when images of the target task are input to the CNN. We calculate metrics according to the following three hypotheses and in total six metrics considered.
H1. If models' featuremaps are easy be classified by fully connected layers, they are highly transferable for given task.
H2. If models' featuremaps have more divergent information, they are highly transferable for given task.
H3. If models' featuremaps are sparse, they are tolerant to over-fitting, thus they are preferable as source models for a given task.
Based on the first hypothesis, we propose three metries which are described in subsection 3.1 to Another metric described in subsection i.4 based on the second hypothesis and the other two metries shown in subsection 3.5 and 3.6 are based on the third hypothesis.
3.3
The notation used here is listed in Table [ For the explanation, we
model architecture of AlexNet as an example. In that case, "the last convolutional layer" means Conv5 layer and the dimension of it is F; (256, 13, 13). Only small number of images for class is enough to compute some metrics and we use randomly chosen ten images in this paper.
The summary of the metries described in the following subsections is presented in Table 2
Table 1: Notation used in this paper
Table 2: Summary of the proposed metrics
Notation
Description
Better
Hypothesis
Description
D
Target task dataset
for
based
x;
i-th image in dataset D
S1
Inter-class discreteness
Larger
HI.H2
C
j-th class in dataset 2)
S2
Intra-class discreteness
Smaller HI
N
Num. of all images dataset
S3
Pscudo value
Larger
HI,H2
K
Num. of class in D
Sum of absolute values of channel correlations
S4
Smaller
H2
Num. of images included
n
the j-th class
Ss
Featuremap sparsity
Smaller
H3
Tensor of featuremap for i-th
S6
Featuremap steepness
Smaller
H3
F.
image at the last conv. layer Average of all F. in dataset D N Σ F.
F
Average of all F, in class C ie.
J
i.c. F Felss. = ny Ex,co, F
3.1 METRIC BASED ON SUM VARIANCE OF CLASS AVERAGE
If featuremaps for different class images differ from each other, the images may be easily classified by the FC layers. Based on this consideration, we propose the following metric S₁ based on the sum of variance of class average of featuremaps.
S
Σ k=0
d
F Feluss
ny
This indicates the inter-class discreteness. The larger the S1, the better it is.
3

---

Under review as a conference paper at ICLR 2020
3.2 METRIC BASED ON SUM OF CLASS- LLASS-WISE FEATUREMAP VARIANCE
featuremaps for different images in class are similar, the images may be easily classified by the FC layers. Based on this consideration, propose the following metric S2 based the sum of class-wise variance of featuremap in a class.
K = Σ
S
Σ
d
class k- k- (Fetass
- -F:)²
(2)
R= mcECk
This indicates the intra-class discreteness and is usually used as an objective function of k-means method. The smaller the S2, the better is.
3.3 METRIC BASED ON PSEUDO-F VALUE
The Pseudo-F known as one of the metrics of clustering algorithns Caliski & Harabasz, 1974).
It is defined as the ratio of inter-class variance to intra-class variance of featuremaps defined by the following formula
SI/(K 1)
S3 S2/(N - K)
(3)
The larger the S3, the better it is.
3.4 METRIC BASED ON CORRELATIONS BETWEEN EACH FEATUREMAP CHANNEL OF AN IMAGE
Convolutional layers have multiple filters with different parameters and extract different features from imput images. For example, the Conv5 layer of AlexNet outputs featuremaps with x 13 256 dimension extracted by 256 different filters. Wu et al. (018) proposed method to estimate contents of featuremaps by using correlations between channels of the featuremaps. Based on the previous observation, we propose the metric S4 based on the absolute value of correlations between channels of featuremaps of each image. The S4 is mathematically defined as follows.
S₁ = -E ccor (F,, (i,;,:). F,, (i+::)))
Here, cor(A.B) indicates the Pearson's product moment correlation coefficient between the argu- ments of and B. The smaller the S4 value, the better it is.
3.5 METRIC BASED ON SPARSITY OF FEATUREMAPS
Yaguchi al. (018)) stated that using Adam<Kingma & Ba 2014
the optimizer for training
CNNs is an effective way to suppress over-fitting and observed that weights of convolutional layers become sparse as a result of Adam. They also mentioned that the sparsity of featuremaps after
ReLu layer become more sparse as training progresses. Based on this observation, we propose the
metric S3 which indicates the sparsity of featuremaps follows:
(Num. of zero elements in F:)
1= So == N
(5)
(Num. of elements in F;)
The smaller the S5 value, the better it is.
3.6 METRIC BASED ON STEEPNESS OF FEATUREMAPS
Based on the third hypothesis, it is expected that if there are large number of zero elements in featuremaps and only few elements have larger values, the network well captures the feature of the images. We then propose the metric which quantifies the steepness of the featuremaps as follows:
(Num. of high value elements in Fi)
S.
N
(Num. of zero elements in F:)
The number high value element F, means that the number elements in featuremaps whose difference from the maximum element value is less than The smaller the Sa value, the better it is.
4

---

Under review as a conference paper at ICLR 2020
Table 3: Configurations for pre-training and fine-tuning.
Parameter
Description
epoch
30
Loss function
Cross-entropy
Optimizer
Stochastic gradient descent (SGD)
Learning rate
0.01 (beginning) and 0.1x every 7-epoch
EXPERIMENTAL. SETUP
Throughout the experiments, evaluate the relation between the fine-tuning accuracy and the proposed metrics for each base model. The evaluation flow is as follow.
1. Prepare base models trained with various source tasks and several network architectures
2. Input the images of target training sets to each base model
3. Get the featuremap from the last convolutional layer of each base model
4. Perform fine-tuning for each target task using all the based models
5. Evaluate the correlation between fine-tuned accuracy and proposed metrics
On the first step of the flow, we use various ImageNet subsets for preparing the base models as described in subsection 4.1 The target tasks to evaluated is described in (.22 On the fifth step. use the Spearman's 's rank correlation coefficient (Spearman's 's p) criterion of the effectiveness of the metrics. We explain the details
Appendix A.I
4.1 PREPARING PRE-TRAINED CNN MODELS
We prepared CNN models trained with different source tasks for evaluating our method. We use various subsets of ImageNetIK 2012 training set as source tasks. A total of 60 subsets of ImageNet are created with two policies, "class-wise division" and "horizontal division".
In "class-wise division". the image classes the ImageNet dataset are divided into several sets each with 100 classes. For example, classes from class No. to No. 100 form subtask. Similarly, classes from class No. 101 to No. 200 form another subtask. We also create smaller subtasks with 50 classes
As for "horizontal division". the total number of classes each subtask the same as ImageNet but the images of each class divided to form subtasks. The original ImageNet dataset has 1000 classes each with around 1300 images. we divide into ten subtasks, each subtask has 1000 classes each of which has about 130 images. We two cases of 10 and 20 divisions in this policy.
As model architectures, AlexNet, VGG16, and 8 are used to prepare the base models. We pre-train these architectures using the subtasks mentioned above prepare the base model. The configurations of pre-training are summarized in Table 3 Note that this configuration is also used for the fine-tuning phase.
4.2 TARGET TASKS
Table shows the datasets used as the target tasks in the experiment. We evaluate fine-tuned ac- curacy with eight image classification datasets whose training set size differs ranging from 2,040 to 8. 8.144 images. The pre-trained base models are fine-tuned to each target task independently. We measure the top-1 accuracy for all the tasks.
5

---

Under review as a conference paper at ICLR 2020
Table 4: Datasets examined in this research
Dataset
Classes
Size(train/test)
Stanford Cars (Krause et al.
2013
196
8,144/8,041
FGVC Aireraft (Maji al. 2013
100
6,667/3,333
Describable Textures (DITD) (Cimpoi al. 2014
3,760/1,880
Oxford-IIIT Pots (Parkhi et al. 20122
37
3,680/3,369
Caltech- 101 (Fei-Fei al. 2004]
102
3.060/6,084
Oxford 102 Flowers (Nilsback Zisserman
2008
102
2,040/6,149
AlexNet
VGG16
ResNet18
0.7
Metric value [arbitrary unit]
0.7
Metric value [arbitrary unit]
0.7
Metric value (arbitrary unit]
0.6
0.6
Accuracy
0.5
2
Accuracy
0.5
Accuracy
0.5
0.4
0.3
0.3
0.3
0.2
0.2
0.1
0.1
0.1
0.0
0.0
0
5
10
15 Epoch
20
25
29
0
5
10
15
20
25
5
10
15 Epoch
20
25
Epoch
Accuracy
S2
S6
S1
5g
Ss
Feature norm
Figure Validation accuracy vs. proposed metries on normal ImageNet training.
5 RESULTS
5.1 THE RELATIONSHIP BETWEEN ACCURACY AND METRICS
We first investigate how the proposed metrics related to the inference accuracy within the normal training phase. We trained the ImageNet 1K classification task from scratch and recorded both the proposed metrics and the validation accuracy in each epoch. The metrics were calculated with featuremaps of the last convolutional layer for each model architecture.
Figure shows the validation accuracy (on the left Y-axis) and proposed metric values (on the right Y-axis) for each epoch (X-axis) in ImageNet training for three network architectures. All the metrics normalized so that the mean is 0 and the variance 1.
For the metrics of S1 and Sg. the higher the better. On the other hand, the smaller the better in the other metrics. Since the results for AlexNet and VGG16 have a similar trend and are easy to analyze, we discuss those results first. As training proceeds, the validate accuracy improves and all the metrics except for S₁ and S2 go into small value. S and S2 have irregular trends the beginning of training, but become high as validation accuracy improves. These indicate that the proposed metrics and the inference accuracy have some correlations. Though we expected that S₂ and S3 metrics would go into larger and smaller training proceeds, the results were opposite. It seems that the CNNs prefer diversified vectors as the featuremap results for higher classification accuracy when intra-class images are applied.
As for the case ResNet18, there seem have some trends but they are not very clear. This may from the effect of normalization layers. However, S4 and Sg have relatively clear and similar trends with the cases of AlexNet and VGG16. In general, these metrics are expected to be appropriate metrics for predicting the suitability of the convolutional layers for given tasks.
We also show the feature norm value of the last convolutional layer in Figure [ We found that and $2 do not have any correlation with the feature norm, but S6 does show negative correlation.

---

Under review as a conference paper at ICLR 2020
AlexNet
VGG16
ResNet18
1.0
1.0
1.0
Spearman's
Spearman's
Spearman's
0.0
0.0
0.0
-0.5
0.5
0.5
-1.0
Aircraft
Pets
Caltech101
DTD
Flowers
Cars
-1.0
Aircraft
Pets
Caltech1
DTD
Flowers
Cars
-1.0
Aircraft
Pets
Caltech101 DTD
Flowers
Cars
53
S4
Ss
Figure 2: Spearman's rank correlation between fine-tuning accuracy and the proposed metries.
5.2 MODEL SELECTION EFFICIENCY
We next evaluate the ability of proposed metrics to select a suitable model for a given target task from different pre-trained models. The results for several target tasks (in X-axis) are summarized with Spearman's 's (ranking correlations among 60 pre-trained models) in Figure 2 Higher absolute values of Spearman's 's p indicate good correlations, showing the effectiveness of the metries.
Overall, S4 and Sq have a high correlation in all the network architectures. If the architecture is the same, there in significant difference in given tasks, but the trend of each metric different for different architectures. Specially, the results in ResNet18 show different trends compared with the other two architectures. For example, S₁ shows a relatively high positive correlation for AlexNet and VGG16 6 while it has high negative correlation for ResNet18. We conjecture that this is because ResNet18 has unique neural network structure among the three, that is, batch normalization layers. Each model architecture extracts image features in different ways and the metrics reflect those dif- ference appropriately. Though most of the metries have high correlations, all of them cannot be used for selecting suitable base models in general way. However, S₂ shows consistently high positive correlation in many cases and could be one of the most suitable metrics.
5.3 COMBINING TWO METRICS
In the previous subsection, we show that not every single metric can be a generally good indicator of selecting suitable base models. In this subsection, combine any two metrics using the following weighted sum to create new indices.
Ciy(a) = (1 + aS,
where a E [0, 1], i =1, 1,..., and i 1,..... 6.
To combine two metrics fairly, we normalize all the metries considering all the network architectures and target tasks so that each of them is within the range of -1 to 1. In addition to that, we invert S2, S4, So. and Sg so that all the metrics exhibit better correlation for positive values.
Figure 3 presents the results of all the combinations of Ci.-- The horizontal and vertical axes of each graph show a and the Spearman's 's rank correlation between fine-tuning accuracy and values of (a), respectively. The value of is changed from 0 to I with the increment of 0.001 in all the experiments.
Again the trends for AlexNet (solid lines) are similar to VGG16 (dotted lines). In these cases, the combinations of S4 and Sg, S₁ and Sg, or S₃ and S₂ give relatively high correlation independent from the given In the of ResNet18 (dashed line), any combinations which include Sg or Se have high correlation we apply an appropriate a. For all the three architectures, Cs.s exhibits fairly high correlations with fine-tuned accuracy. Both Ss and Sg are based the sparsity of

---

Under review as a conference paper at ICLR 2020
(1-a)'S: atS)
(1-a/5: + dtS,
(1-a)'S: drSa
0
0.00
0.25
0,50
0.75
1.00
0.00
0.25
0.50
0.75
1.00 0.00
0.25
0.50
0.75
1.00
(1-a)'5;
atSs
(1-a)*S: + a"S.
(1-a)'5) d'Sa
0-
0.00
0.25
0,50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Spearman's
(1-a)*52 a*54
(1-a)*5, + a*ss
(1-a)*52
d'Sa
0
0.00
0.25
0,50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
(1-a)*5; 5*5
(1-a)*5; + a²5,
(1-a1*5) 5
1
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
(1-a1*5 54
at5;
(1-a)*5, + a55g
(1-a1*5; d 5g
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
AlexNet Aircraft
AlexNet Cars
ResNet18 Flowers
VGG16 Caltech101
AlexNet Pets
ResNet18 Aircraft
ResNet18 Cars
VGG16 DTD
AlexNet Caltech101
ResNet18 Pets
VGG16 Aircraft
VGG16 Flowers
AlexNet DTD
ResNet18 Caltech101
VGG16 Pets
VGG16
AlexNet Flowers
ResNet18 DTD
Figure 3: The rank correlations for the combinations of two metrics.
the featuremaps, hence it is suggested that the sparsity is specially important factor to evaluate transferability of pre-trained models.
Focusing on the combination of Ss and Ss. (0.57))) = 0.4265, 0.574Se gives the highest rank correlation for all combinations of architectures and target tasks on average. The average rank correlation between fine-tuning accuracy and Cr.e is 0.7514 and the variance 0.005993. The maximum and minimum are 0.8874 and 0.5854 obtained in Caltech101 with AlexNet and Pets with ResNet18. Since is known that high correlation is recognized if p larger than about 0.7, we can conclude that C5.6 is well correlated to the fine-tuning accuracy and can be used for selecting some suitable base models without actually performing time-consuming training. Note that S5 and Se metrics can be calculated without target task labels, hence C5.6 can be easily calculated which would reduce the burden of the fine-tuning process further.
CONCLUSION
In this paper, we proposed a method to evaluate the transferability of convolutional layers of trained CNN models for certain target task. Our experiments demonstrated that some of the proposed metrics can be good indicator of fine-tuned model accuracy. We also showed that combining two metrics, especially the sparsity and the steepness of featuremaps, gives us more efficient model selection metric. This result may suggest that the proposed method with some metrics is useful in selecting suitable base models among a number of candidate networks.
In this paper, we focus a base model selection methodology, but the proposed technique is easily applicable to an appropriate layer selection methodology. Using all the convolutional layers a base model does not always provide good classification accuracy after fine-tuning. We can shrink the network size by truncating the convolutional layers and the suitable final convolutional layer can be selected by using the proposed metric. The details described in Appendix [A.2]
One of the clear extensions of this work is to evaluate the proposed technique with other network model architectures and other image recognition tasks such as object detection and semantic seg-
8

---

Under review as a conference paper at ICLR 2020
mentation. It is reported that pre-training with a large number of images is not effective in object detection or semantic segmentation (Mahajan et al. 2018) and there is no clear relation between Ima- geNet classification accuracy and the imgge recommendation ability (del Rio et al. 2018 Studying transferability for those tasks is also our future work.
REFERENCES
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability deep visual representations. In Computer Vision and Pattern Recognition, 2017.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizzi Model compression. Proceed- ings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Min- ing. KDD '06, pp. 535-541 New York, NY, USA, 2006. ACM. ISBN 1-59593-339-5. doi: 10. 1145/1 150402. 150464.
acm. org 11t.:1111111111111111 0tt.://11150022111011.11 1150464
Caliski and J Harabasz. A dendrite method for cluster analysis. Communications Statistics, (1):1-27, 1974. doi: 10. 10.108000331009244888877110
Alfredo Canziani, Adam Paszke, and Eugenio Culurciello. An analysis of deep neural network models for practical applications, 2016.
Rich Caruana. Multitask learning. Machine Learning, 28(1):41-75, 1997. doi: 10.1023/A:
1007379606734. RtLLhhtps::/doii /ttp///.......0.....001111111113
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
Felipe del Rio, Pablo Messina, Vicente Domingue7, and Denis Parra. Do better imagenet models transfer better... for image recommendation?, 2018.
Li Fei-Fei, Rob Fergus, and Pietro Perona.
Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. 2004 Conference on Computer Vision and Pattern Recognition Workshop, pp. 178-178, 2004.
Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, and Anima Anand- kumar. Born again neural networks. In ICML, 2018.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. CoRR, abs/1 503.02331, 2015.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 12 2014.
Simon Korblith, Jonathon Shlens, and Quoe V. Le. Do better imagenet models transfer better?, 2018.
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. 3d object representations for fine- -rained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13) 13), Sydney, Australia, 2013.
Bo Liu, Yanshan Xiao, and Zhifeng Hao. selective multiple instance transfer learning method for text categorization problems. Strowledge-Basele Systems, 141:178 - 187, 2018. ISSN 0950-7051. doi: http://bbbioog///....oo/l/ssssss com/
URL http:/ /www://wwwww///.///////////wwww sciencedirect.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining, 2018.
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft, 2013.

---

Under review as a conference paper at ICLR 2020
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In ICLR 2017, 2017.
Bert Moons, Bert De Brabandere, Luc Van Gool, and Marian Verhelst. Energy-efficient convnets through approximate computing. 2016 IEEE Winter Conference Applications of Computer Vision ( (WACV), pp. 1-8, 2016.
Ari S. Morcos, David G. T. Barrett, Neil C. Rabinowitz, and Matthew Botvinick. On the importance of single directions for generalization. In International Conference on Learning Representations, 2018.
M-E. Nilshack and A. Zisserman. Automated flower classification over a large number of classes. In Proceedings of the Indian Conference Computer Vision, Graphics and Image Processing. Dec 2008.
O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar. Cats and dogs. In JEEE Conference on Computer Vision and Pattern Recognition, 2012.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vi- doi:
sion, 115(3):211 20:::221122222 252. 12 2015. ISSN 1573-1405.
URL
https://dos wrg/10....00. 1007/s11263 015 015-0816-y
Issei Sato, Yukihiro Nomura, Shohei Hanaoka, Soichiro Miki, Naoto Hayashi, Osamu Abe, and Yoshitaka Masutani. Managing camputer-assisted detection system based on transfer learning with negative transfer inhibition. In KDD, 2018.
Bichen Wu, Alvin Wan, Xiangyu Yue, Peter Jin, Sicheng Zhao, Noah Golmant, Amir Gholaminejad, Joseph Gonzalez, and Kurt Keutzer. Shift: A zero flop, zero parameter alternative to spatial convolutions. Computer Vision and Pattern Recognition, 2018
Atsushi Yaguchi, Taiji Suzuki, Wataru Asano, Shuhei Nitta, Yukinobu Sakata, and Akiyuki Tanizawa. Adam induces implicit weight sparsity in rectifier neural networks. 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 318-325, 2018.
Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. Deep mutual learning. CVPR, 2018.
A APPENDIX
A.1 RANK CORRELATIONS BETWEEN METRICS AND TRANSFER ACCURACY
Spearman's 's rank correlation coefficient or Spearman's p is non-parametric measure of statistical dependence between the rankings of two variables (rank correlation). is denoted as follows:
d
p
M (
where d; means the difference between the two ranks of cach value and M is the number of values. We evaluate this value between the rank of each proposed metric and one of fine-tuning accuracy.
In general, it is said that the order of two values have a higher correlation if |p| 2 0.7.
A.2 SELECTING SUITABLE LAYER FOR RUNCATING CNN MODELS
well known that the lower layers CNN usually recognize low-level local features such as edges, corners, spots in generic fashion. It progresses toward higher layers to identify higher-level features which are optimized for a certain task. There is a possibility that the lower convolutional layers extract features which is better suited for target task in fine-tuning compared with the last conv. layer. We next evaluate the effect of proposed metrics for selecting the most transferable layer in given pre-trained model.
10

---

Under review as a conference paper at ICLR 2020
A.2.1 CONCEPT
In order evaluate the layer selecting capability with the proposed metrics, fine-tuning method shown below. We use only lower part of the CNN model and prune the deeper part. Proper pooling and FC layer are added to the selected lower layers. We call this "truncate fine-tuning".
A.2.2 RESULTS
In the experiment, of truncate fine-tuning when varying the selected convolutional layer. The base model is trained for 30 epochs with ImageNet dataset. The horizontal axis indicates the index of the convolutional layer which is used for the last conv. layer and the vertical axis means tne-tunnn accuracy for each target task.
only R base model architecture. Figure 4 shows the accuracy
Overall, the results are better when using all the conv. layers, However, truncating one or two layers does not have much impact accuracy. For example, in case of the Flowers dataset, the accuracy is 0.92 when using 16th conv. layer, while it is 0.91 and 0.87 when using 15th and 14th conv. layer, respectively. Moreover, in case of Caltech101, the accuracy of the 7th layer is better than that of the 8th layer.
Trancate accuracy of ResNet18 8 model
Aircraft
Dtd
Flowers
0.8
Truncate accuracy
0.6
0.4
0.2
7
10 11 12 14 15 16
Conv
Figure 4: Truncate fine-tuning accuracy for each selected convolutional layer in ResNet 18
ResNet18 trained with ImageNet
5,
0.5
Speaman's
0.0
-0.5
Aircraft
Cars
Pets Caltech101 Did
Flowers
Target task
Figure 5: Rank correlations between the proposed metrics and truncate fine-tuning accuracy
We can use the proposed metrics to select the suitable convolutional layer for efficient truncate fine-tuning. Figure shows the Spearman's 's rank corrllation between truncate fine-tuning accuracy and the proposed metrics. For all the target tasks, Spearman's 's p between Sg metric and truncate fine-tuning accuracy is larger than 0.7. This result shows that the Sg metric is useful for selecting

---

Under review as a conference paper at ICLR 2020
an appropriate layer for truncate fine-tuning and the proposed method has potential to effectively compress the base CNN model in fine-tuning process.
12


Reviews:
[{"review_id": "r1lkxZb0FH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose several metrics to evaluate the transferability of pretrained CNN models for a target task without actually fine-tuning the networks to accelerate fine-tuning. \n\nThe paper has done some interesting empirical studies on how to predict the transferability of a neural network. The motivation of performing model selection without actual finetuning has many benefits for practical applications and I believe this is the right direction. In this perspective, the paper is novel. However, my major concern is a lack of in-depth analysis and thorough verification of the proposed metrics.\n\nFirst, there is no clear relationship between the six evaluation metrics and the fine-tuning performance. Figure 1 depicts some correlation, however, it is not clear enough. The difference in ResNet18 is not further investigated. Given the dominant usage of ResNets and its variations in practice, it is important to provide analysis for the reasons behind.\n\nSecond, some of the proposed metrics (S1/S2) are actually closely related with weight norms.  The definition of high value elements in S6 is also based on the absolute difference with the maximum feature map values. Does the author apply weight decay during training? Normally the weight norm will becomes smaller and so could the featuremap values. The S1/S2/S6 metrics could therefore be influenced. I would like to see figures showing the relationship between the weight/feature norms and the metric during training. \n\nThird, it turns out that the only useful metrics are the combination of feature sparsity and feature steepness (i.e., C_56). The authors should make it more specific and clear for their contributions. Why only two combination of the metrics is considered? Does the best selected coefficient generalize to other models and datasets?\n\nFinally, according to https://arxiv.org/abs/1805.08974, better pretrained ImageNet model also generalize better. I would expect a deeper ResNets should outperform AlexNet and VGG16.  What is the relationship between the proposed metrics and the ImageNet performance? \n\nMinor:\n\nThe S3 metric is actually “a ratio of inter-class variance to intra-class variance” rather than “ ratio of intra-class variance to inter-class variance”.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkgI8AGkYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors tried several metrics, from which they pinpoint one as the indicator to select a pretrained model without practically fine-tuning. I appreciate that the authors addressed an important problem, while the experimental setup and results are less convincing. Therefore, the proposed metric(s) and settings in the current shape are not that practical. \n\nPros:\n-\tIn this work, the authors focused on an important problem – selecting the best pretrained model from a zoo of models without finetuning all of them in a brute-force manner. \n-\tThe idea of applying the metric(s) to select a layer from which the consecutive layers are truncated is intriguing, for the sake of model compression.\n\nCons:\n-\tThe work is more alike a course project than a novel scientific contribution, with all the metrics mainly inherited from the evaluation of clustering and other existing literatures. \n-\tThe experiments are not comprehensive, so that the conclusions drawn are weak and untenable.  \n-\tThe writing with many grammatical errors and typos definitely needs polishing.\n\nDetailed comments:\n1.\tThe authors calculated all the metrics in terms of feature maps in the last layer. Therefore, it is not valid to conclude that other metrics are less effective than sparsity. Some metrics, like fisher discriminator, are likely effective only in the low-level features. It is better for the authors to investigate all metrics in terms of all layers.\n2.\tSince all the pretrained models are pretrained from ImageNet, it is possible that they lose some diversity, which tends to deactivate metrics other than S5 and S6. Unless the authors trained several models from scratch on different datasets and achieved similar results, the conclusion that S5 and S6 are strong is not that convincing.\n3.\tWhy during fine-tuning, do you use SGD instead of Adam?\n4.\tWhat kind of correlation metrics do you use in Eqn. (4)? And will the correlation metric influence the effectiveness of S4?\n5.\tThe most confusing part lies in that the valid metric suggested by the author, i.e., C5,6(0.574), is only an empirical observation on a very limited number of datasets, without any principled methodology. Actually, it is highly possible that given a highly different dataset, the best metric changes. Should the practitioners determine which metric to use first, and even the coefficient to combine metrics?  This is paradox, in my opinion. \n6.\tGrammatical errors and missing details:\no\tAbstract: a variety of network structure -> a variety of network structures\no\tSection 3: The sentences for the three hypotheses, H1/2/3, do not even have verbs in the if part. \no\tSection 3.4: to estimates -> to estimate\no\tSection 3.6: which quantify -> which quantifies\no\t…", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "H1xRBbkYtB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to speed up finetuning of pretrained deep image classification networks by predicting the success rate of the process without running it. The authors suggest running samples from the target task on the (trained) source model, and computing a few sensible measures from the final output layer which indicate how well the trained features are separating the target images. Many experiments are presented, and some of them seem promising. \n\nOverall the idea is, while simple, interesting and potentially promising: DL Training costs have been rising significantly in the past few years, and being able to make sensible predictions about which source dataset/task combination best fits a target task would be a great contribution. Nonetheless, there are some methodological concerns that cast doubt about the presented results, which would need to be addressed before making this paper ready for publication.\n\nComments:\n\n1. The authors present multiple experimental results, many of which indicate a somewhat noisy signal. The only method that works on most tasks, combining S5 and S6, is somewhat underreported. How many different \\alpha values did the authors consider? how were they selected? Was the same alpha values used for each all experiments? Given this large set of measures and combinations of measures experimented with in this paper, I am left wondering whether this approach would generalize to new target datasets.\n\n2. A major assumption the authors are making is that there is a single number that determines whether a given trained architecture would transfer well to a new target task. But the finetuning process is affected by many factors (e.g, hyperparameters, random seeds), and thus it might be that with a different hyperparameter selection, the observed correlations would look completely different. I would have liked to see at the very least an analysis of the correlation between multiple runs of finetuning that show that they are well correlated before computing correlations with external measures.\n\n3. The authors point out that transfer learning works poorly on medical imaging (Sato et al., 2018). It would be interesting to experiment with such datasets and observe whether the proposed method is able to make accurate predictions in this domain.\n\n4. The experiments regarding truncating CNNs seem interesting, and I was disappointed to find out they are not part of the main paper.\n\nOther comments:\n\n1. Moons et al. (2016) and Molchanov et al. (2017) speed up *inference* and not training.\n\n2. The authors argue that Mahajan et al. (2018) \"did not apply it (their method) for detection or segmentation tasks.\". However, neither did this paper.\n\n3. Writing: \n-- Several typos and grammatical errors across the paper. For instance:\n- \"*In* many previous research efforts suggested\" (\"in\" should be dropped)\n- Hypothesis H1 is ungrammatical \n\n-- Many of the citations were in the wrong format (intro: Canziani et al. 2016), repetitive (3.5: Yaguchi et al.), etc.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to speed up finetuning of pretrained deep image classification networks by predicting the success rate of a zoom of pre-trained  networks without completely running them on the test set. The idea is that a sensible measure from the output layer might well correlate with the performance of the network. All reviewers consider this is an important problem and a good direction to make the effort. However, various concerns are raised and all reviewers unanimously rate weak reject. The major concerns include the unclear relationship between the metrics and the fine-tuning performance, non- comprehensive experiments, poor writing quality. The authors respond to Reviewers’ concerns but did not change the major concerns. The ACs concur the concerns and the paper can not be accepted at its current state.
Decision: Reject