user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Rethinking the Hyperparameters for Fine-tuning

Paper content:
Fine-tuning from pre-trained ImageNet models has become the de-facto standard for various computer vision tasks.
Current practices for fine-tuning typically involve selecting an ad-hoc choice of hyper-parameters and keeping them fixed to values normally used for training from scratch.
This paper re-examines several common practices of setting hyper-parameters for fine-tuning.
Our findings are based on extensive empirical evaluation for fine-tuning on various transfer learning benchmarks.
(1) While prior works have thoroughly investigated learning rate and batch size, momentum for fine-tuning is a relatively unexplored parameter.
We find that picking the right value for momentum is critical for fine-tuning performance and connect it with previous theoretical findings.
(2) Optimal hyper-parameters for fine-tuning in particular the effective learning rate are not only dataset dependent but also sensitive to the similarity between the source domain and target domain.
This is in contrast to hyper-parameters for training from scratch.
(3) Reference-based regularization that keeps models close to the initial model does not necessarily apply for "dissimilar" datasets.
Our findings challenge common practices of fine- tuning and encourages deep learning practitioners to rethink the hyper-parameters for fine-tuning.
Many real-world applications often have limited number of training instances, which makes directly training deep neural networks hard and prone to overfitting.
Transfer learning with the knowledge of models learned on a similar task can help to avoid overfitting.
Fine-tuning is a simple and effective approach of transfer learning and has become popular for solving new tasks in which pre-trained models are fine-tuned with the target dataset.
Specifically, fine-tuning on pre-trained ImageNet classification models (Simonyan & Zisserman, 2015; He et al., 2016b) has achieved impressive results for tasks such as object detection (Ren et al., 2015) and segmentation (He et al., 2017; Chen et al., 2017) and is becoming the de-facto standard of solving computer vision problems.
It is believed that the weights learned on the source dataset with a large number of instances provide better initialization for the target task than random initialization.
Even when there is enough training data, fine-tuning is still preferred as it often reduces training time significantly (He et al., 2019) .
The common practice of fine-tuning is to adopt the default hyperparameters for training large models while using smaller initial learning rate and shorter learning rate schedule.
It is believed that adhering to the original hyperparameters for fine-tuning with small learning rate prevents destroying the originally learned knowledge or features.
For instance, many studies conduct fine-tuning of ResNets (He et al., 2016b) with these default hyperparameters: learning rate 0.01, momentum 0.9 and weight decay 0.0001.
However, the default setting is not necessarily optimal for fine-tuning on other tasks.
While few studies have performed extensive hyperparameter search for learning rate and weight decay (Mahajan et al., 2018; Kornblith et al., 2018) , the momentum coefficient is rarely changed.
Though the effectiveness of the hyperparameters has been studied extensively for training a model from scratch, how to set the hyperparameters for fine-tuning is not yet fully understood.
In addition to using ad-hoc hyperparameters, commonly held beliefs for fine-tuning also include:
• Fine-tuning pre-trained networks outperforms training from scratch; recent work (He et al., 2019) has already revisited this.
• Fine-tuning from similar domains and tasks works better (Ge & Yu, 2017; Cui et al., 2018; Achille et al., 2019; Ngiam et al., 2018) .
• Explicit regularization with initial models matters for transfer learning performance (Li et al., 2018; 2019) .
Are these practices or beliefs always valid?
From an optimization perspective, the difference between fine-tuning and training from scratch is all about the initialization.
However, the loss landscape of the pre-trained model and the fine-tuned solution could be much different, so as their optimization strategies and hyperparameters.
Would the hyperparameters for training from scratch still be useful for fine-tuning?
In addition, most of the hyperparameters (e.g., batch size, momentum, weight decay) are frozen; will the conclusion differ when some of them are changed?
With these questions in mind, we re-examined the common practices for fine-tuning.
We conducted extensive hyperparameter search for fine-tuning on various transfer learning benchmarks with different source models.
The goal of our work is not to obtain state-of-the-art performance on each fine-tuning task, but to understand the effectiveness of each hyperparameter for fine-tuning, avoiding unnecessary computations.
We explain why certain hyperparameters work so well on certain datasets while fail on others, which can guide future hyperparameter search for fine-tuning.
Our main findings are as follows:
• Optimal hyperparameters for fine-tuning are not only dataset dependent, but also depend on the similarity between the source and target domains, which is different from training from scratch.
Therefore, the common practice of using optimization schedules derived from ImageNet training cannot guarantee good performance.
It explains why some tasks are not achieving satisfactory results after fine-tuning because of inappropriate hyperparameter selection.
Specifically, as opposed to the common practice of rarely tuning the momentum value beyond 0.9, we verified that zero momentum could work better for fine-tuning on tasks that are similar with the source domain, while nonzero momentum works better for target domains that are different from the source domain.
• Hyperparameters are coupled together and it is the effective learning rate-which encapsulates the learning rate, momentum and batch size-that matters for fine-tuning performance.
While effective learning rate has been studied for training from scratch, to the best of our knowledge, no previous work investigates effective learning rate for fine-tuning and is less used in practice.
Our observation of momentum can be explained as small momentum actually decreases the effective learning rate, which is more suitable for fine-tuning on similar tasks.
We show that the optimal effective learning rate actually depends on the similarity between the source and target domains.
• We find regularization methods that were designed to keep models close to the initial model does not apply for "dissimilar" datasets, especially for nets with Batch Normalization.
Simple weight decay can result in as good performance as the reference based regularization methods for fine-tuning with better search space.
The two extreme ways for selecting hyperparameters-performing exhaustive hyperparameter search or taking ad-hoc hyperparameters from scratch training-could be either too computationally expensive or yield inferior performance.
Different with training from scratch, the default hyperparameter setting may work well for random initialization, the choice of hyperparameters for fine-tuning is not only dataset dependent but is also influenced by the similarity between the target domain and the source domains.
The rarely tuned momentum value could impede the performance when the target domain and source domain are close.
These observations connect with previous theoretical works on decreasing momentum at the end of training and effective learning rate.
We further identify the optimal effective learning rate depends on the similarity of source domain and target domain.
With this understanding, one can significant reduce the hyperparameter search space.
We hope these findings could be one step towards better hyperparameter selection strategies for fine-tuning.

assistant: text: This paper re-examines several common practices of setting hyper-parameters for fine-tuning.