Rethinking the Hyperparameters for Fine-tuning

1 INTRODUCTION . Many real-world applications often have a limited number of training instances , which makes directly training deep neural networks hard and prone to overfitting . Transfer learning with the knowledge of models learned on a similar task can help to avoid overfitting . Fine-tuning is a simple and effective approach of transfer learning and has become popular for solving new tasks in which pre-trained models are fine-tuned with the target dataset . Specifically , fine-tuning on pre-trained ImageNet classification models ( Simonyan & Zisserman , 2015 ; He et al. , 2016b ) has achieved impressive results for tasks such as object detection ( Ren et al. , 2015 ) and segmentation ( He et al. , 2017 ; Chen et al. , 2017 ) and is becoming the de-facto standard of solving computer vision problems . It is believed that the weights learned on the source dataset with a large number of instances provide better initialization for the target task than random initialization . Even when there is enough training data , fine-tuning is still preferred as it often reduces training time significantly ( He et al. , 2019 ) . The common practice of fine-tuning is to adopt the default hyperparameters for training large models while using smaller initial learning rate and shorter learning rate schedule . It is believed that adhering to the original hyperparameters for fine-tuning with small learning rate prevents destroying the originally learned knowledge or features . For instance , many studies conduct fine-tuning of ResNets ( He et al. , 2016b ) with these default hyperparameters : learning rate 0.01 , momentum 0.9 and weight decay 0.0001 . However , the default setting is not necessarily optimal for fine-tuning on other tasks . While few studies have performed extensive hyperparameter search for learning rate and weight decay ( Mahajan et al. , 2018 ; Kornblith et al. , 2019 ) , the momentum coefficient is rarely changed . Though the effectiveness of the hyperparameters has been studied extensively for training a model from scratch , how to set the hyperparameters for fine-tuning is not yet fully understood . ∗Work done while at Amazon Web Services In addition to using ad-hoc hyperparameters , commonly held beliefs for fine-tuning also include : • Fine-tuning pre-trained networks outperforms training from scratch ; recent work ( He et al. , 2019 ) has already revisited this . • Fine-tuning from similar domains and tasks works better ( Ge & Yu , 2017 ; Cui et al. , 2018 ; Achille et al. , 2019 ; Ngiam et al. , 2018 ) . • Explicit regularization with initial models matters for transfer learning performance ( Li et al. , 2018 ; 2019 ) . Are these practices or beliefs always valid ? From an optimization perspective , the difference between fine-tuning and training from scratch is all about the initialization . However , the loss landscape of the pre-trained model and the fine-tuned solution could be much different , so as their optimization strategies and hyperparameters . Would the hyperparameters for training from scratch still be useful for fine-tuning ? In addition , most of the hyperparameters ( e.g. , batch size , momentum , weight decay ) are frozen ; will the conclusion differ when some of them are changed ? With these questions in mind , we re-examined the common practices for fine-tuning . We conducted extensive hyperparameter search for fine-tuning on various transfer learning benchmarks with different source models . The goal of our work is not to obtain state-of-the-art performance on each fine-tuning task , but to understand the effectiveness of each hyperparameter for fine-tuning , avoiding unnecessary computation . We explain why certain hyperparameters work so well on certain datasets while fail on others , which can guide hyperparameter search for fine-tuning . Our main findings are as follows : • Optimal hyperparameters for fine-tuning are not only dataset dependent , but are also dependent on the similarity between the source and target domains , which is different from training from scratch . Therefore , the common practice of using optimization schedules derived from ImageNet training can not guarantee good performance . It explains why some tasks are not achieving satisfactory results after fine-tuning because of inappropriate hyperparameter selection . Specifically , as opposed to the common practice of rarely tuning the momentum value beyond 0.9 , we find that zero momentum sometimes work better for fine-tuning on tasks that are similar with the source domain , while nonzero momentum works better for target domains that are different from the source domain . • Hyperparameters are coupled together and it is the effective learning rate—which encap- sulates the learning rate and momentum—that matters for fine-tuning performance . While effective learning rate has been studied for training from scratch , to the best of our knowledge , no previous work investigates effective learning rate for fine-tuning and is less used in practice . Our observation of momentum can be explained as small momentum actually decreases the effective learning rate , which is more suitable for fine-tuning on similar tasks . We show that the optimal effective learning rate depends on the similarity between the source and target domains . • We find regularization methods that were designed to keep models close to the initial model does not necessarily work for “ dissimilar ” datasets , especially for nets with Batch Normalization . Simple weight decay can result in as good performance as the referencebased regularization methods for fine-tuning with better search space . 2 RELATED WORK . In transfer learning for image classification , the last layer of a pre-trained network is usually replaced with a randomly initialized fully connected layer with the same size as the number of classes in the target task ( Simonyan & Zisserman , 2015 ) . It has been shown that fine-tuning the whole network usually results in better performance than using the network as a static feature extractor ( Yosinski et al. , 2014 ; Donahue et al. , 2014 ; Huh et al. , 2016 ; Mormont et al. , 2018 ; Kornblith et al. , 2019 ) . Ge & Yu ( 2017 ) select images that have similar local features from source domain to jointly fine-tune pre-trained networks . Cui et al . ( 2018 ) estimate domain similarity with ImageNet and demonstrate that transfer learning benefits from pre-training on a similar source domain . Besides image classification , many object detection frameworks also rely on fine-tuning to improve over training from scratch ( Girshick et al. , 2014 ; Ren et al. , 2015 ) . Many researchers re-examined whether fine-tuning is a necessity for obtaining good performance . Ngiam et al . ( 2018 ) find that when domains are mismatched , the effectiveness of transfer learning is negative , even when domains are intuitively similar . Kornblith et al . ( 2019 ) examine the fine-tuning performance of various ImageNet models and find a strong correlation between ImageNet top-1 accuracy and the transfer accuracy . They also find that pre-training on ImageNet provides minimal benefits for some fine-grained object classification dataset . He et al . ( 2019 ) questioned whether ImageNet pre-training is necessary for training object detectors . They find the solution of training from scratch is no worse than the fine-tuning counterpart as long as the target dataset is large enough . Raghu et al . ( 2019 ) find that transfer learning has negligible performance boost on medical imaging applications , but speed up the convergence significantly . There are many literatures on hyperparameter selection for training neural networks from scratch , mostly on batch size , learning rate and weight decay ( Goyal et al. , 2017 ; Smith et al. , 2018 ; Smith & Topin , 2019 ) . There are few works on the selection of momentum ( Sutskever et al. , 2013 ) . Zhang & Mitliagkas ( 2017 ) proposed an automatic tuner for momentum and learning rate in SGD . There are also studies on the correlations of the hyperparameters , such as linear scaling rule between batch size and learning ( Goyal et al. , 2017 ; Smith et al. , 2018 ; Smith , 2017 ) . However , most of these advances on hyperparameter tuning are designed for training from scratch and have not examined on fine-tuning tasks for computer vision problems . Most work on fine-tuning simply choose fixed hyperparameters ( Cui et al. , 2018 ) or use dataset-dependent learning rates ( Li et al. , 2018 ) in their experiments . Due to the huge computational cost for hyperparameter search , only a few works ( Kornblith et al. , 2019 ; Mahajan et al. , 2018 ) performed large-scale grid search of learning rate and weight decay for obtaining the best performance . 3 TUNING HYPERPARAMETERS FOR FINE-TUNING . In this section , we first introduce the notations and experimental settings , and then present our observations on momentum , effective learning rate and regularization . The fine-tuning process is not different from learning from scratch except for the weights initialization . The goal of the process is still to minimize the objective function L = 1N ∑N i=1 ` ( f ( xi , θ ) , yi ) + λ 2 ‖θ‖ 2 2 , where ` is the loss function , N is the number of samples , xi is the input data , yi is its label , f is the neural network , θ is the model parameters and λ is the regularization hyperparameter or weight decay . Momentum is widely used for accelerating and smoothing the convergence of SGD by accumulating a velocity vector in the direction of persistent loss reduction ( Polyak , 1964 ; Sutskever et al. , 2013 ; Goh , 2017 ) . The commonly used Nesterov ’ s Accelerated Gradient ( Nesterov , 1983 ) is given by : vt+1 = mvt − ηt 1 n n∑ i=1 ∇ ` ( f ( xi , θt +mvt ) , yi ) ( 1 ) θt+1 = θt + vt+1 − ηλθt ( 2 ) where θt indicates the model parameters at iteration t. The hyperparameters include the learning rate ηt , batch size n , momentum coefficient m ∈ [ 0 , 1 ) , and the weight decay λ . 3.1 EXPERIMENTAL SETTINGS . We evaluate fine-tuning on seven widely used image classification datasets , which covers tasks for fine-grained object recognition , scene recognition and general object recognition . Detailed statistics of each dataset can be seen in Table 1 . We use ImageNet ( Russakovsky et al. , 2015 ) , Places-365 ( Zhou et al. , 2018 ) and iNaturalist ( Van Horn et al. , 2018 ) as source domains for pre-trained models . We resize the input images such that the aspect ratio is preserved and the shorter side is 256 pixels . The images are normalized with mean and std values calculated over ImageNet . For data augmentation , we adopt the common practices used for training ImageNet models ( Szegedy et al. , 2015 ) : random mirror , random scaled cropping with scale and aspect variations , and color jittering . The augmented images are resized to 224×224 . Note that state-of-the-art results could achieve even better performance by using higher resolution images ( Cui et al. , 2018 ) or better data augmentation ( Cubuk et al. , 2018 ) . We mainly use ResNet-101-V2 ( He et al. , 2016a ) as our base network , which is pre-trained on ImageNet ( Russakovsky et al. , 2015 ) . Similar observations are also made on DenseNets ( Huang et al. , 2017 ) and MobileNet ( Howard et al. , 2017 ) . The hyperparameters to be tuned ( and ranges ) are : learning rate ( 0.1 , 0.05 , 0.01 , 0.005 , 0.001 , 0.0001 ) , momentum ( 0.9 , 0.99 , 0.95 , 0.9 , 0.8 , 0.0 ) and weight decay ( 0.0 , 0.0001 , 0.0005 , 0.001 ) . We set the default hyperparameters to be batch size 2561 , learning rate 0.01 , momentum 0.9 and weight decay 0.0001 . To avoid insufficient training and observe the complete convergence behavior , we use 300 epochs for fine-tuning and 600 epochs for scratch-training , which is long enough for the training curves to converge . The learning rate is decayed by a factor of 0.1 at epoch 150 and 250 . We report the Top-1 validation ( test ) error at the end of training . The total computation time for the experiments is more than 10K GPU hours .

Summary: This paper studies the role of different hyperparameters in finetuning image recognition models on new target tasks. The authors run a large set of experiments and show that, perhaps non-surprisingly, hyperparameters matter. In particular, they show that momentum, which is typically ignored in finetuning, is quite important, and that the momentum values  that work well depend on the similarity between the source and target datasets. They also show important correlations between momentum, learning rate, and weight decay.