A Base Model Selection Methodology for Efficient Fine-Tuning

1 INTRODUCTION . As convolutional neural networks ( CNNs ) have been widely used in many applications , a CNN model tends to become deeper and larger which necessitates a large amount of training data . This makes the number of arithmetic operations required for training huge ( Canziani et al. , 2016 ) , resulting in long training time and large energy consumption . Moreover , there are large demands to process CNNs in edge devices where training a CNN model is a challenging task due to there limited computing resources . Having an efficient CNN learning methodology for embedded systems is very important . Hence , techniques to reduce the computational complexity of CNN training with maintaining inference accuracy have been studied intensively so far . These include , for example , quantization ( Moons et al. , 2016 ) and pruning ( Molchanov et al. , 2017 ) . Since obtaining enough training data is usually difficult and expensive in real situations , there is a great demand to create an efficient CNN training methodology . This is the motivation of using transfer learning which optimizes a trained model for a target domain by transferring information from a related source domain ( Caruana , 1997 ) . Fine-tuning , one of the transfer learning techniques , is a promising approach to reduce the burden of CNN training . In fine-tuning , a source CNN model trained with a large labeled dataset ( source task ) is adapted to a different application ( target task ) to create a new model ( target model ) . For example , ImageNet ( Russakovsky et al. , 2015 ) is becoming a de facto dataset as a source task for image recognition target tasks . In fine-tuning , it is a common practice to truncate the last fully connected layer ( FC layer ) of the pre-trained network , replace it with a new one that is relevant to the target task , and freeze the weights of the other layers of the pre-trained network . It is known that fine-tuning can be performed efficiently if the source and the target tasks have high relativity ( Caruana , 1997 ) . If we choose a suitable source model for a given target task from multiple source task candidates , time and energy consumption for training are likely to be reduced . There has not been enough research effort to evaluate whether the transferred parameters are suitable for another task so far . In this paper , we investigate several metrics to evaluate “ transferability ” of source CNN models for certain target tasks quantitatively without actually performing a time-consuming fine-tuning process . The evaluation results reveal that some of the metrics are well correlated with fine-tuned accuracy and our technique will greatly reduce the burden of CNN training . 2 RELATED WORK . Kornblith et al . ( 2018 ) examined the fine-tuning effect on using ImageNet 1K classification task as a source task for various target tasks . They evaluated the correlation between the accuracy of the source task ( ImageNet ) and fine-tuned target tasks with 13 classification models on 12 image classification tasks . A positive correlation was observed and the correlation coefficient was very high on average ( r2 = 0.86 ) . Based on the results , they claimed that ImageNet accuracy is a good predictor of fine-tuning performance . In their research , the focus is on the fine-tuned performance after target tasks are actually trained . Compare to that , we consider fine-tuned accuracy without actually performing a time-consuming training phase . Moreover , they did not apply it for small datasets such as Stanford Cars and FGVC aircraft which do not necessarily have high fine-tuned performance even using ImageNet as a source task , meaning a model pre-trained with a large data set does not always give high fine-tuned accuracy . This implies that selecting an appropriate source task for a given target task is very important . In Mahajan et al . ( 2018 ) , they proposed a method of pre-training with large amounts of images on Instagram using its hashtag as a label . They found that some of the design choices that are made in current network architectures are too tailored to ImageNet-1k classification . They also mentioned that pre-training with a large amount of weakly labeled data improves classification task , but they did not apply it for detection or segmentation tasks . Sato et al . ( 2018 ) raised a problem of negative transfer in the medical imaging area . They found that the results of transfer learning sometimes become worse rather than learning from scratch . This may also suggest that having appropriate pre-trained source tasks are necessary to train better models with transfer learning . Transferring the data in source domain to target domain is called “ instance transfer ” . In Liu et al . ( 2018 ) , they proposed a new technique to transfer knowledge safely by measuring relativity between the source task and the target task . Bau et al . ( 2017 ) quantified interpretability of latent representations of trained CNNs by evaluating their featuremaps with a method similar to the segmentation . It is judged by whether featuremaps can identify a certain concept and the number of identified concepts indicates interpretability of the target model . In this research , the authors revealed that CNNs identify higher-level concepts such as objects at higher-level convolutional layers and lower-level concepts such as textures at lower-level convolutional layers . Morcos et al . ( 2018 ) reported that a highly selective neuron or featuremap which responds to input data in a certain class is not important for higher accuracy . Instead , neurons or featuremaps activated by inputs of a variety of classes are more important . They pointed out the grandmother cell hypothesis in the neural network may not be correct . Hinton et al . ( 2015 ) proposed a knowledge distillation approach which compresses the knowledge of a large computational expensive model ( teacher model ) to create a smaller neural network model ( student model ) . The idea of the knowledge distillation is to train a student model with a transfer set provided by the teacher model . It is becoming a popular technique to compress neural net models . The base idea of transferring knowledge of larger models to a smaller model to get a computationally efficient model is originally proposed by Buciluǎ et al . ( 2006 ) . Furlanello et al . ( 2018 ) has reported that repeating knowledge distillation leads to better generalization ability of trained models . In addition , Zhang et al . ( 2018 ) has proposed a new learning method with distilling the knowledge of multiple models mutually to get higher performance models . In many previous research efforts suggested that there exists an optimal model architecture for a target task and a pre-trained model using a source task greatly affects the fine-tuned accuracy of the network in transfer learning . To the best of our knowledge , this is the first attempt to evaluate transferability of pre-trained models without actually performing fine-tuning . This is one of the novel points in this work . Besides it , our technique can be used to truncate a CNN model as presented in Appendix A.2 . The method could be another approach to compress CNN models . 3 MODEL SELECTION METRICS . In this paper , we propose the metrics to evaluate the transferability of trained CNNs for a certain target task without actually retraining the networks to accelerate a fine-tuning process . We calculate the metrics based on the outputs of the last convolutional layer ( featuremaps ) for each architecture when images of the target task are input to the CNN . We calculate metrics according to the following three hypotheses and in total six metrics are considered . H1 . If models ’ featuremaps are easy to be classified by fully connected layers , they are highly transferable for a given task . H2 . If models ’ featuremaps have more divergent information , they are highly transferable for a given task . H3 . If models ’ featuremaps are sparse , they are tolerant to over-fitting , thus they are preferable as source models for a given task . Based on the first hypothesis , we propose three metrics which are described in subsection 3.1 to 3.3 . Another metric described in subsection 3.4 is based on the second hypothesis and the other two metrics shown in subsection 3.5 and 3.6 are based on the third hypothesis . The notation used here is listed in Table 1 . For the explanation , we use a model architecture of AlexNet as an example . In that case , “ the last convolutional layer ” means Conv5 layer and the dimension of it is Fi = ( 256 , 13 , 13 ) . Only a small number of images for a class is enough to compute some metrics and we use randomly chosen ten images in this paper . The summary of the metrics described in the following subsections is presented in Table 2 . Table 1 : Notation used in this paper Notation Description D Target task dataset xi i-th image in dataset D Cj j-th class in dataset D N Num . of all images in dataset D K Num . of class in D nj Num . of images included in the j-th class Fi Tensor of featuremap for i-th image at the last conv . layer F̄ Average of all Fi in dataset Di.e . F̄ = 1 N ∑ Fi F̄class j Average of all Fi in class Cj i.e . F̄class j = 1nj ∑ xi∈Cj Fi Table 2 : Summary of the proposed metrics Description Betterfor Hypothesis based S1 Inter-class discreteness Larger H1 , H2 S2 Intra-class discreteness Smaller H1 S3 Pseudo F value Larger H1 , H2 S4 Sum of absolute values of channel correlations Smaller H2 S5 Featuremap sparsity Smaller H3 S6 Featuremap steepness Smaller H3 3.1 METRIC BASED ON SUM OF VARIANCE OF CLASS AVERAGE . If featuremaps for different class images differ from each other , the images may be easily classified by the FC layers . Based on this consideration , we propose the following metric S1 based on the sum of variance of class average of featuremaps . S1 = K−1∑ k=0 d ( F̄− F̄class k ) 2 nk ( 1 ) This indicates the inter-class discreteness . The larger the S1 , the better it is . 3.2 METRIC BASED ON SUM OF CLASS-WISE FEATUREMAP VARIANCE . If featuremaps for different images in a class are similar , the images may be easily classified by the FC layers . Based on this consideration , we propose the following metric S2 based on the sum of class-wise variance of featuremap in a class . S2 = K−1∑ k=0 ∑ xi∈Ck d ( F̄class k− − Fi ) 2 ( 2 ) This indicates the intra-class discreteness and is usually used as an objective function of k-means method . The smaller the S2 , the better it is . 3.3 METRIC BASED ON PSEUDO-F VALUE . The Pseudo-F is known as one of the metrics of clustering algorithms ( Caliski & Harabasz , 1974 ) . It is defined as the ratio of inter-class variance to intra-class variance of featuremaps defined by the following formula 3 : S3 = S1/ ( K − 1 ) S2/ ( N −K ) ( 3 ) The larger the S3 , the better it is . 3.4 METRIC BASED ON CORRELATIONS BETWEEN EACH FEATUREMAP CHANNEL OF AN IMAGE . Convolutional layers have multiple filters with different parameters and extract different features from input images . For example , the Conv5 layer of AlexNet outputs featuremaps with 13×13×256 dimension extracted by 256 different filters . Wu et al . ( 2018 ) proposed a method to estimate contents of featuremaps by using correlations between channels of the featuremaps . Based on the previous observation , we propose the metric S4 based on the absolute value of correlations between channels of featuremaps of each image . The S4 is mathematically defined as follows . S4 = ∑ n ∑ i , j |cor ( Fn ( i , : , : ) , Fn ( j , : , : ) ) | ( 4 ) Here , cor ( A , B ) indicates the Pearson ’ s product moment correlation coefficient between the arguments of A and B . The smaller the S4 value , the better it is .

Summary: This paper aims to speed up finetuning of pretrained deep image classification networks by predicting the success rate of the process without running it. The authors suggest running samples from the target task on the (trained) source model, and computing a few sensible measures from the final output layer which indicate how well the trained features are separating the target images. Many experiments are presented, and some of them seem promising.