Rethinking the Hyperparameters for Fine-tuning

1 INTRODUCTION . Many real-world applications often have a limited number of training instances , which makes directly training deep neural networks hard and prone to overfitting . Transfer learning with the knowledge of models learned on a similar task can help to avoid overfitting . Fine-tuning is a simple and effective approach of transfer learning and has become popular for solving new tasks in which pre-trained models are fine-tuned with the target dataset . Specifically , fine-tuning on pre-trained ImageNet classification models ( Simonyan & Zisserman , 2015 ; He et al. , 2016b ) has achieved impressive results for tasks such as object detection ( Ren et al. , 2015 ) and segmentation ( He et al. , 2017 ; Chen et al. , 2017 ) and is becoming the de-facto standard of solving computer vision problems . It is believed that the weights learned on the source dataset with a large number of instances provide better initialization for the target task than random initialization . Even when there is enough training data , fine-tuning is still preferred as it often reduces training time significantly ( He et al. , 2019 ) . The common practice of fine-tuning is to adopt the default hyperparameters for training large models while using smaller initial learning rate and shorter learning rate schedule . It is believed that adhering to the original hyperparameters for fine-tuning with small learning rate prevents destroying the originally learned knowledge or features . For instance , many studies conduct fine-tuning of ResNets ( He et al. , 2016b ) with these default hyperparameters : learning rate 0.01 , momentum 0.9 and weight decay 0.0001 . However , the default setting is not necessarily optimal for fine-tuning on other tasks . While few studies have performed extensive hyperparameter search for learning rate and weight decay ( Mahajan et al. , 2018 ; Kornblith et al. , 2019 ) , the momentum coefficient is rarely changed . Though the effectiveness of the hyperparameters has been studied extensively for training a model from scratch , how to set the hyperparameters for fine-tuning is not yet fully understood . ∗Work done while at Amazon Web Services In addition to using ad-hoc hyperparameters , commonly held beliefs for fine-tuning also include : • Fine-tuning pre-trained networks outperforms training from scratch ; recent work ( He et al. , 2019 ) has already revisited this . • Fine-tuning from similar domains and tasks works better ( Ge & Yu , 2017 ; Cui et al. , 2018 ; Achille et al. , 2019 ; Ngiam et al. , 2018 ) . • Explicit regularization with initial models matters for transfer learning performance ( Li et al. , 2018 ; 2019 ) . Are these practices or beliefs always valid ? From an optimization perspective , the difference between fine-tuning and training from scratch is all about the initialization . However , the loss landscape of the pre-trained model and the fine-tuned solution could be much different , so as their optimization strategies and hyperparameters . Would the hyperparameters for training from scratch still be useful for fine-tuning ? In addition , most of the hyperparameters ( e.g. , batch size , momentum , weight decay ) are frozen ; will the conclusion differ when some of them are changed ? With these questions in mind , we re-examined the common practices for fine-tuning . We conducted extensive hyperparameter search for fine-tuning on various transfer learning benchmarks with different source models . The goal of our work is not to obtain state-of-the-art performance on each fine-tuning task , but to understand the effectiveness of each hyperparameter for fine-tuning , avoiding unnecessary computation . We explain why certain hyperparameters work so well on certain datasets while fail on others , which can guide hyperparameter search for fine-tuning . Our main findings are as follows : • Optimal hyperparameters for fine-tuning are not only dataset dependent , but are also dependent on the similarity between the source and target domains , which is different from training from scratch . Therefore , the common practice of using optimization schedules derived from ImageNet training can not guarantee good performance . It explains why some tasks are not achieving satisfactory results after fine-tuning because of inappropriate hyperparameter selection . Specifically , as opposed to the common practice of rarely tuning the momentum value beyond 0.9 , we find that zero momentum sometimes work better for fine-tuning on tasks that are similar with the source domain , while nonzero momentum works better for target domains that are different from the source domain . • Hyperparameters are coupled together and it is the effective learning rate—which encap- sulates the learning rate and momentum—that matters for fine-tuning performance . While effective learning rate has been studied for training from scratch , to the best of our knowledge , no previous work investigates effective learning rate for fine-tuning and is less used in practice . Our observation of momentum can be explained as small momentum actually decreases the effective learning rate , which is more suitable for fine-tuning on similar tasks . We show that the optimal effective learning rate depends on the similarity between the source and target domains . • We find regularization methods that were designed to keep models close to the initial model does not necessarily work for “ dissimilar ” datasets , especially for nets with Batch Normalization . Simple weight decay can result in as good performance as the referencebased regularization methods for fine-tuning with better search space . 2 RELATED WORK . In transfer learning for image classification , the last layer of a pre-trained network is usually replaced with a randomly initialized fully connected layer with the same size as the number of classes in the target task ( Simonyan & Zisserman , 2015 ) . It has been shown that fine-tuning the whole network usually results in better performance than using the network as a static feature extractor ( Yosinski et al. , 2014 ; Donahue et al. , 2014 ; Huh et al. , 2016 ; Mormont et al. , 2018 ; Kornblith et al. , 2019 ) . Ge & Yu ( 2017 ) select images that have similar local features from source domain to jointly fine-tune pre-trained networks . Cui et al . ( 2018 ) estimate domain similarity with ImageNet and demonstrate that transfer learning benefits from pre-training on a similar source domain . Besides image classification , many object detection frameworks also rely on fine-tuning to improve over training from scratch ( Girshick et al. , 2014 ; Ren et al. , 2015 ) . Many researchers re-examined whether fine-tuning is a necessity for obtaining good performance . Ngiam et al . ( 2018 ) find that when domains are mismatched , the effectiveness of transfer learning is negative , even when domains are intuitively similar . Kornblith et al . ( 2019 ) examine the fine-tuning performance of various ImageNet models and find a strong correlation between ImageNet top-1 accuracy and the transfer accuracy . They also find that pre-training on ImageNet provides minimal benefits for some fine-grained object classification dataset . He et al . ( 2019 ) questioned whether ImageNet pre-training is necessary for training object detectors . They find the solution of training from scratch is no worse than the fine-tuning counterpart as long as the target dataset is large enough . Raghu et al . ( 2019 ) find that transfer learning has negligible performance boost on medical imaging applications , but speed up the convergence significantly . There are many literatures on hyperparameter selection for training neural networks from scratch , mostly on batch size , learning rate and weight decay ( Goyal et al. , 2017 ; Smith et al. , 2018 ; Smith & Topin , 2019 ) . There are few works on the selection of momentum ( Sutskever et al. , 2013 ) . Zhang & Mitliagkas ( 2017 ) proposed an automatic tuner for momentum and learning rate in SGD . There are also studies on the correlations of the hyperparameters , such as linear scaling rule between batch size and learning ( Goyal et al. , 2017 ; Smith et al. , 2018 ; Smith , 2017 ) . However , most of these advances on hyperparameter tuning are designed for training from scratch and have not examined on fine-tuning tasks for computer vision problems . Most work on fine-tuning simply choose fixed hyperparameters ( Cui et al. , 2018 ) or use dataset-dependent learning rates ( Li et al. , 2018 ) in their experiments . Due to the huge computational cost for hyperparameter search , only a few works ( Kornblith et al. , 2019 ; Mahajan et al. , 2018 ) performed large-scale grid search of learning rate and weight decay for obtaining the best performance . 3 TUNING HYPERPARAMETERS FOR FINE-TUNING . In this section , we first introduce the notations and experimental settings , and then present our observations on momentum , effective learning rate and regularization . The fine-tuning process is not different from learning from scratch except for the weights initialization . The goal of the process is still to minimize the objective function L = 1N ∑N i=1 ` ( f ( xi , θ ) , yi ) + λ 2 ‖θ‖ 2 2 , where ` is the loss function , N is the number of samples , xi is the input data , yi is its label , f is the neural network , θ is the model parameters and λ is the regularization hyperparameter or weight decay . Momentum is widely used for accelerating and smoothing the convergence of SGD by accumulating a velocity vector in the direction of persistent loss reduction ( Polyak , 1964 ; Sutskever et al. , 2013 ; Goh , 2017 ) . The commonly used Nesterov ’ s Accelerated Gradient ( Nesterov , 1983 ) is given by : vt+1 = mvt − ηt 1 n n∑ i=1 ∇ ` ( f ( xi , θt +mvt ) , yi ) ( 1 ) θt+1 = θt + vt+1 − ηλθt ( 2 ) where θt indicates the model parameters at iteration t. The hyperparameters include the learning rate ηt , batch size n , momentum coefficient m ∈ [ 0 , 1 ) , and the weight decay λ . 3.1 EXPERIMENTAL SETTINGS . We evaluate fine-tuning on seven widely used image classification datasets , which covers tasks for fine-grained object recognition , scene recognition and general object recognition . Detailed statistics of each dataset can be seen in Table 1 . We use ImageNet ( Russakovsky et al. , 2015 ) , Places-365 ( Zhou et al. , 2018 ) and iNaturalist ( Van Horn et al. , 2018 ) as source domains for pre-trained models . We resize the input images such that the aspect ratio is preserved and the shorter side is 256 pixels . The images are normalized with mean and std values calculated over ImageNet . For data augmentation , we adopt the common practices used for training ImageNet models ( Szegedy et al. , 2015 ) : random mirror , random scaled cropping with scale and aspect variations , and color jittering . The augmented images are resized to 224×224 . Note that state-of-the-art results could achieve even better performance by using higher resolution images ( Cui et al. , 2018 ) or better data augmentation ( Cubuk et al. , 2018 ) . We mainly use ResNet-101-V2 ( He et al. , 2016a ) as our base network , which is pre-trained on ImageNet ( Russakovsky et al. , 2015 ) . Similar observations are also made on DenseNets ( Huang et al. , 2017 ) and MobileNet ( Howard et al. , 2017 ) . The hyperparameters to be tuned ( and ranges ) are : learning rate ( 0.1 , 0.05 , 0.01 , 0.005 , 0.001 , 0.0001 ) , momentum ( 0.9 , 0.99 , 0.95 , 0.9 , 0.8 , 0.0 ) and weight decay ( 0.0 , 0.0001 , 0.0005 , 0.001 ) . We set the default hyperparameters to be batch size 2561 , learning rate 0.01 , momentum 0.9 and weight decay 0.0001 . To avoid insufficient training and observe the complete convergence behavior , we use 300 epochs for fine-tuning and 600 epochs for scratch-training , which is long enough for the training curves to converge . The learning rate is decayed by a factor of 0.1 at epoch 150 and 250 . We report the Top-1 validation ( test ) error at the end of training . The total computation time for the experiments is more than 10K GPU hours .

Summary: This submission studies the problem of transfer learning and fine tuning. This submission proposes four insights: Momentum hyperparameters are essential for fine-tuning; When the hyperparameters satisfy some certain relationships, the results of fine-tuning are optimal; The similarity between source and target datasets influences the optimal choice of the hyperparameters; Existing regularization methods for DNN is not effective when the datasets are dissimilar. This submission provides multiple experiments to support their opinion.