Rethinking the Hyperparameters for Fine-tuning
Authors: Hao Li
Pratik Chaudhari
Hao Yang
Michael Lam
Avinash Ravichandran
Rahul Bhotika
Stefano Soatto
Venue:  (2020)
Keywords: fine-tuning
hyperparameter search
transfer learning

Abstract: Fine-tuning from pre-trained ImageNet models has become the de-facto standard for various computer vision tasks. Current practices for fine-tuning typically involve selecting an ad-hoc choice of hyperparameters and keeping them fixed to values normally used for training from scratch. This paper re-examines several common practices of setting hyperparameters for fine-tuning.  Our findings are based on extensive empirical evaluation for fine-tuning on various transfer learning benchmarks. (1) While prior works have thoroughly investigated learning rate and batch size, momentum for fine-tuning is a relatively unexplored parameter. We find that the value of momentum also affects fine-tuning performance and connect it with previous theoretical findings.  (2) Optimal hyperparameters for fine-tuning, in particular, the effective learning rate, are not only dataset dependent but also sensitive to the similarity between the source domain and target domain. This is in contrast to hyperparameters for training from scratch. (3) Reference-based regularization that keeps models close to the initial model does not necessarily apply for "dissimilar" datasets. Our findings challenge common practices of fine-tuning and encourages deep learning practitioners to rethink the hyperparameters for fine-tuning.

Published as a conference paper at ICLR 2020
RETHINKING THE HYPERPARAMETERS FOR FINE-TUNING
Hao Li', Pratik Chaudhari"; Hao Yang' Michael Lam', Avinash Ravichandran', Rahul Bhotika', Stefano Soatto
1 Ammzon Web Services, 2 UUiiversity of Pennsylvania, UUniversity of California, Los Angeles (haolimax, haoyng, michlam, ravinash, bhotikar, soattos] @amazzo.com, pratikac @seas.upenn.ddu
ABSTRACT
Fine-tuning from pre-trained ImageNet models has become the de-facto standard for various computer vision tasks. Current practices for fine-tuning typically involve selecting an ad-hoc choice of hyperparameters and keeping them fixed to values normally used for training from scratch. This paper re-examines several common practices of setting hyperparameters for fine-tuning. Our findings are based extensive empirical evaluation for fine-tuning various transfer learning benchmarks. (1) While prior works have thoroughly investigated learning rate and batch size, momentum for fine-tuning is relatively unexplored parameter. We find that the value of momentum also affects fine-tuning performance and connect it with previous theoretical findings. (2) Optimal hyperparameters for fine-tuning, in particular, the effective learning rate, are not only dataset dependent but also sensitive to the similarity between the source domain and target domain. This is in contrast to hyperparameters for training from scratch. (3) Reference-based regularization that keeps models close to the initial model does not necessarily apply for "dissimilar" datasets. Our findings challenge common practices of fine- tuning and encourages deep learning practitioners to rethink the hyperparameters for fine-tuning.
INTRODUCTION
Many real-world applications often have a limited number of training instances, which makes directly training deep neural networks hard and prone to overfitting. Transfer learning with the knowledge of models learned on a similar task can help to avoid overfitting. Fine-tuning is a simple and effective approach of transfer learning and has become popular for solving new tasks in which pre-trained models are fine-tuned with the target dataset. Specifically, fine-tuning on pre-trained ImageNet classification models (Simonyan & Zisserman, 2015; He et al., 2016b) has achieved impressive results for tasks such as object detection (Ren et al., 2015) and segmentation (He et al., 2017: Chen et al., 2017) and is becoming the de-facto standard of solving computer vision problems. It is believed that the weights learned on the source dataset with large number of instances provide better initialization for the target task than random initialization. Even when there is enough training data, fine-tuning is still preferred as it often reduces training time significantly (He et al., 2019).
The common practice of fine-tuning is to adopt the default hyperparameters for training large models while using smaller initial learning rate and shorter learning rate schedule. is believed that adhering the original hyperparameters for fine-tuning with small learning rate prevents destroying the originally learned knowledge or features. For instance, many studies conduct fine-tuning of ResNets (He et al., 2016b) with these default hyperparameters: learning rate 0.01, momentum 0.9 and weight decay 0.0001. However, the default setting is not necessarily optimal for fine-tuning on other tasks, While few studies have performed extensive hyperparameter search for learning rate and weight decay (Mahajan et al., 201 8; Kornblith et al., 2019), the momentum coefficient is rarely changed. Though the effectiveness of the hyperparameters has been studied extensively for training a model from scratch, how to set the hyperparameters for fine-tuning is not yet fully understood.
Work done while Amazon Web Services

---

Published as a conference paper at ICLR 2020
In addition using ad-hoc hyperparameters, commonly held beliefs for fine-tuning also include:
Fine-tuning pre-trained networks outperforms training from scratch; recent work (He et al., 2019) has already revisited this.
Fine-tuning from similar domains and tasks works better (Ge & Yu, 2017; Cui et al., 2018; Achille et al., 2019; Ngiam et al., 2018). 8).
Explicit regularization with initial models matters for transfer learning performance (Li et al., 2018; 2019).
Are these practices or beliefs always valid? From an optimization perspective, the difference between fine-tuning and training from scratch is all about the initialization. However, the loss landscape of the pre-trained model and the fine-tuned solution could be much different, so as their optimization strategies and hyperparameters. Would the hyperparameters for training from scratch still be useful for fine-tuning? addition, most of the hyperparameters (e.g., batch size, momentum, weight decay) are frozen; will the conclusion differ when some of them are changed?
With these questions in mind, we re-examined the common practices for fine-tuning. We conducted extensive hyprrarrmeeter search for fine-ture fine-tuning various transfer learning benchmarks with different source models. The goal of our work is not to obtain state-of-the-ar performance on each fine-tuning task, but to understand the effectiveness of each hyperparameter for fine-tuning, avoiding unnecessary computation. We explain why certain hyperparameters work so well on certain datasets while fail on others, which can guide hyperparameter search for fine-tuning.
Our main findings are as follows:
Optimal hyperparameters for fine-tuning are not only dataset dependent, but are also depen- dent on the similarity between the source and target domains, which different from training from scratch. Therefore, the common practice of using optimization schedules derived from ImageNet training cannot guarantee good performance. explains why some tasks are not achieving satisfactory results after fine-tuning because of inappropriate hyperparameter selection. Specifically, as opposed to the common practice of rarely tuning the momentum value beyond 0.9, we find that zero momentum sometimes work better for fine-tuning on tasks that are similar with the source domain, while nonzero momentum works better for target domains that are different from the source domain.
Hyperparameters are coupled together and it is the effective learning wate-which encap- sulates the learning rate and momentum- that matters for fine-tuning performance. While effective learning rate has been studied for training from scratch, to the best of our knowl- edge, no previous work investigates effective learning rate for fine-tuning and is less used in practice. Our observation of momentum can be explained small momentum actually decreases the effective learning rate, which is more suitable for fine-tuning on similar tasks. We show that the optimal effective learning rate depends on the similarity between the source and target domains.
We find regularization methods that were designed keep models close to the initial model does not necessarily work for "dissimilar" datasets, especially for nets with Batch Normalization. Simple weight decay can result in as good performance the reference- based regularization methods for fine-tuning with better search space.
2 RELATED WORK
In transfer learning for image classification, the last layer of a pre-trained network is usually replaced with randomly initialized fully connected layer with the same size as the number of classes in the target task (Simonyan Zisserman, 2015). has been shown that fine-tuning the whole network usually results in better performance than using the network as static feature extractor (Yosinski et al., 2014; Donahue et al., 2014; Huh et al., 2016; Mormont et al., 2018; Kornblith et al., 2019). Ge & Yu (2017) select images that have similar local features from source domain to jointly fine-tune pre- pre-trained networks. Cui et al. (2018) estimate domain similarity with ImageNet and demonstrate that transfer learning benefits from pre- training on a similar source domain. Besides image classi- fication, many object detection frameworks also rely on fine-tuning to improve over training from scratch (Girshick al., 2014; Ren al., 2015).
2

---

Published as a conference paper at ICLR 2020
Many researchers re-examined whether fine-tuning is a necessity for obtaining good performance. Ngiam et al. (2018) find that when domains are mismatched, the effectiveness of transfer learning is negative, even when domains are intuitively similar. Kornblith et al. (2019) examine the fine-tuning performance of various ImageNet models and find a strong correlation between ImageNet top-1 accuracy and the transfer accuracy. They also find that pre-training benefits for some fine-grained object classification dataset. He et al. (2019) questioned whether ImageNet pre-training necessary for training object detectors. They find the solution of training from scratch is no worse than the fine-tuning counterpart as long as the target dataset is large enough. Raghu et al. (2019) find that transfer learning has negligible performance boost on medical imaging applications, but speed up the convergence significantly.
ImageNet provides minimal
There are many literatures on hyperparameter selection for training neural networks from scratch, mostly on batch size, learning rate and weight decay (Goyal et al., 2017; Smith et al., 201 8; Smith & Topin, 2019). There are few works on the selection of momentum (Sutskever et al., 2013). Zhang & Mitliagkas (2017) proposed automatic tuner for momentum and learning rate in SGD. There are also studies on the correlations of the hyperparameters, such as linear scaling rule between batch size and learning (Goyal et al., 2017; Smith et al., 2018; Smith, 2017). However, most of these advances on hyperparameter tuning are designed for training from scratch and have not examined on fine-tuning tasks for computer vision problems. Most work fine-tuning simply choose fixed hyperparameters (Cui et al., 2018) or use dataset-dependent learning rates (Lii et al., 2018) in their experiments. Due to the huge computational cost for hyperparameter search, only few works (Kornblith et al., 2019; Mahajan et al., 2018) performed large-scale grid search of learning rate and weight decay for obtaining the best performance.
TUNING HYPERPARAMETERS FOR FINE-TUNING
In this section, first introduce the notations and experimental settings, and then present observations on momentum, effective learning rate and regularization. The fine-tuning process is not different from learning from scratch except for the weights initialization. The goal of the process still to minimize the objective function I. 1 EN (ff f). yi) + A 11011, where is the loss 2
N
function, N is the number of samples, z; is the input data, Vi is its label, J is the neural network, O is the model parameters and À is the regularization hyperparameter or weight decay. Momentum is widely used for accelerating and smoothing the convergence of SGD by accumulating velocity vector in the direction of persistent loss reduction (Polyak, 1964; Sutskever et al., 2013; Goh, 2017). The commonly used Nesterov's 's Accelerated Gradient (Nesterov, 1983) is given by:
1 Σ Ve(f(z,, +m,)).
Ut41 mu4 1)t n
0,41 0, + 1/41 79
where 6, indicates the model parameters at iteration t. The hyperparameters include the learning rate mt. batch size n, momentum coefficient m [0, 1), and the weight decay X.
3.1 EXPERIMENTAL SETTINGS
We evaluate fine-tuning on seven widely used image classification datasets, which covers tasks for fine-grained object recognition, scene recognition and general object recognition. Detailed statisties of each dataset can be seen in Table 1. We use ImageNet (Russakovsky et al., 2015), Places-365 (Zhou et al., 2018) and iNaturalist (Van Horn et al., 2018) as source domains for pre-trained models. We resize the input images such that the aspect ratio is preserved and the shorter side is 256 pixels. The images are normalized with mean and std values calculated over ImageNet. For data augmentation, we adopt the common practices used for training ImageNet models (Szegedy et al., 2015): random mirror, random scaled cropping with scale and aspect variations, and color jittering. The augmented images resized to 224x 224. Note that state-of-the-an results could achieve even better performance by using higher resolution images (Cui et al., 2018) or better data augmentation (Cubuk et al., 2018).
We mainly use ResNet-101-V2 (He al., 2016a) our base network, which is pre-trained on ImageNet (Russakovsky et al., 2015). Similar observations are also made on DenseNets (Huang et al., 2017) and MobileNet (Howard al., 2017). The hyperparameters to be tuned (and ranges)
3

---

Published as a conference paper at ICLR 2020
Table 1: Datasets statistics. For the Caltech-256 dataset, we randomly sampled 60 images for each class
following the procedure used in (Li et al., 2018). For the Aircraft and Flower dataset, we combined the original training set and validation set and evaluated on the test set. For iNat 2017, we combined the original training set and 90% of the validation set following (Cui et al., 2018).
Datasets
Task Category
Classes
Training
Test
Oxford Flowers (Nilshack & Zisserman, 2008)
fine-grained ohject recog.
102
2,040
6,149
CUB-Bird 200-2011 (Wah et al., 2011)
fine-grained object recog,
200
5,994
5,794
FGVC Airerafts (Maji et al., 2013)
fine-grained object recog.
100
6,667
3.333
Stanford Cars (Krause et al., 2013)
fine-grained object recog.
196
8,144
8,041
Stanford Dogs (Khosla et al., 2011)
fine-grained object recog. scene classification general object recog.
120
12,000
8,580
MIT Indoor-67 (Sharif Razavian et al., 2014)
67
5,360
1,340
Caliech-256660 (Griffin al., 2007)
256
15,360
15,189
iNaturalist 2017 (Van Horn et al., 2018)
fine-grained object recog.
5,089
665,571
9,599
Place365 (Zhou al., 2018)
scene classification
365
1,803,460
36,500
are: learning rate (0.1, 0.05, 0.01. 0.005, 0.001, 0,0001), momentum (0.9, 0.99, 0.95, 0.9, 0.8, 0.0) and weight decay (0.0, 0.0001, 0.0005, 0.001). We set the default hyperparameters to be batch size 256', learning rate 0.01, momentum 0.9 and weight decay 0.0001. To avoid insufficient training and observe the complete convergence behavior, for ccratch-triining which is long enough for the training curves to converge. The learning rate is decayed by factor of 0.1 at epoch 150 and 250. We report the Top-1 validation (test) error at the end of training. The total computation time for the experiments is more than 10K GPU hours.
300 epochs for fine-tuning and 600 epochs
3.2 EFFECT OF MOMENTUM AND DOMAIN SIMILARITY
Momentum 0.9 is the most widely used value for training from seratch (Krizhevsky et al., 2012; Simonyan & Zisserman, 2015; He et al., 2016b) and is also widely adopted for fine-tuning (Kornblith al., 2019). To the best of our knowledge, it is rarely changed. regardless of the network architectures or target tasks. To check the influence of momentum on fine-tuning, we first search for the best momentum value for fine-tuning on the Birds dataset with different weight decay and learning rate. Figure ((a) shows the performance of fine-tuning with and without weight decays. Surprisingly, momentum zero actually outperforms the nonzero momentum. The optimal learning rate also increases when the momentum is disabled as shown in Figure 1(b).
imagenet, A=
imagenet.
0,0001, min top1 7.441 m= 0.0
a D.O. min top1-17.21, m- 0.0
Validation Error
Valiatioon
min top1 =17.24, n 0.005 0.0. min top1-16.98, o=
14
0.1 Momentum
bid
leamiig rate
Figure 1: (a) Searching for the optimal momentum on Birds dataset with fixed learning rate 0.01 and different weight decays. Detailed learning curves and results of other hyperparameters can be found in Appendix A. (b) Comparison of momentum 0.9 and 0.0 with different learning rates on the Birds dataset, À is fixed at 0.0001.
To verify this observation, we further compare momentum 0.9 and 0.0 on other datasets. Table shows the performance of 8 hyperparameter settings on 7 datasets. We observe a clear pattern that disabling momentum works better for Dogs, Caltech and Indoor, while momentum 0.9 works better for Cars, Airerafts and Flowers.
For each training job with ResNet- 101 and batch size 256, use 8 NVIDIA Tesla V100 GPUs for synchronous training, where each GPU uses batch of 32 and SyncBN is used.
4

---

Published as a conference paper at ICLR 2020
Table 2: Top-1 validation errors on seven datascts by fine-tuning ImageNet pre-trained ResNet-101 with different hyperparmeters. Each row represents a network fine-tuned by a set of hyperparameters (left four columns). The datasets are ranked by the relative improvement by disabling momentum. The lowest error rates with the same momentum marked as bold. Note that the performance difference for Birds is not very significant.
À
Dogs
Caltech
Indoor
Birds
Cars
Aircrafts
Flowers
0.9
0.01
0.0001
17.20
14.85
23.76
18.10
9.10
17.55
3.12
0.9
0.01
17.41
14.51
24.59
18.42
9.60
17.40
3.33
0.9
0.005
0.0001
14.14
13.42
24.59
17.24
9.08
18.21
3.50
0.9
0.005
14.80
13.67
22.79
17.54
9.31
17.82
3.53
0
0.01
0.0001
11.00
12.11
21.14
17.41
11.07
20.58
5.48
0
0.01
10.87
12.16
21.29
17.21
10.65
20.46
5.25
0
0.005
0.0001
10.21
11.86
21.96
18.24
13.22
24.39
7.03
0
0.005
10.12
11.61
20.76
18.40
13.11
23,91
6.78
Table 3: Verification of the effect of momentum other source domains rather than ImageNet. The hyperpa- rameters are 256, 7) 0.01, and 0.0001. Momentum works better transferring from iNat-2017 to Birds and transferring from Places-365 to Indoor comparing to momentum 0.9 counterparts.
Source domain
m
Indoor
Birds
Dogs
Caltech
Cars
Aircrafts
0.9
30.73
14.69
24.74
20.12
11.16
19.86
NN1122017
34.11
12.29
23.87
21.47
16.89
27.21
0.9
22.19
27.72
30.84
22.53
11.06
21.27
Places-365
20.16
32.17
32.47
22.60
14.67
25.29
Interestingly, datasets such as Dogs, Caltech, Indoor and Birds are known to have high overlap with ImageNet dataset?, while Cars and Aircrafts are identified to be difficult to benefit from fine-tuning from pre-trained ImageNet models (Kornblith et al., 2019). According to Cui et al. (2018), in which the Earth Mover's Distance (EMD) is used to calculate the similarity between ImageNet and other domains, the similarity to Dogs and Birds are 0.619 and 0.563, while the similarity to Cars, Aircrafts and Flowers are 0.560, 0.556 and 0.525 The relative order of similarities to ImageNet
Dogs, Birds, Cars, Aircrafts and Flowers
which aligns well with the transition of optimal momentum value from 0.0 to 0.9. Following the similarity calculation, we can also verified Caltech and Indoor are more close to ImageNet than Cars/Airerafts/llower (Table 3.3).
To verify the connection between momentum and domain similarity, we further fine-tune from different source domains such as Places-365 and iNaturalist, which known be better source domains than ImageNet for fine-tuning on Indoor and Birds dataset (Cui et al., 2018). We may expect that fine-tuning from iNaturalist works better for Birds with 0 and similarly, Places for Indoor. Indeed, as shown in Table 3, disabling momentum improves the performance when the source and target domain are similar, such as Places for Indoor and iNaturalist for Birds.
Small momentum works better for fine-tuning on domains that are close to the source domain One explanation for the above observations is that because the Dogs dataset is very close to ImageNet, the pre-trained ImageNet model is expected to be close to the fine-tuned solution on the Dogs dataset. In this case, momentum may not help much as the gradient direction around the minimum could be much random and accumulating the momentum direction could be meaningless. Whereas, for
Stanford Dogs (Khosla al., 2011) 1) built using images and annotation from ImageNet for the task of fine-graine image categorization. Caltech-256 has at least categories exist in ImageNet (Deng et al., 2010). Images in the CUB-Birds dataset overlap with images in ImageNct.
'The domain similarity calucation is discussed in Appendix and the exact value be found in Table 3.3.
5

---

Published as a conference paper at ICLR 2020
(a) dissimilar, m = 0.9 (b) dissimilar, / = (c) similar, m = 0.9
(d) similar, m = 0.0
Figure 2:
An illustration of the effect of momentum on different fine-tuning scenarios from the loss-landscape perspective. The red point is the pre-trained model and the blue point is the fine-tuned solution. The dashed lines loss contours. Assuming the step size is fixed, large momentum accelerates the convergence when initialization far from the minimum ((a) and (b)). On the contrary. large momentum may impede convergence as shown in (c) and (d) when initialization close to the minimum.
faraway target domains (e.g., Cars and Aircrafts) where the pre-trained ImageNet model could be much different with the fine-tuned solution, the fine-tuning process more similar with training from scratch, where large momentum stabilizes the decent directions towards the minimum. An illustration of the difference can be found in Figure 2.
Connections to early observations on decreasing momentum Early work (Sutskever et al., 2013) actually pointed out that reducing momentum during the final stage of training allows finer conver- gence while aggressive momentum would prevent this. They recommended reducing momentum from 0.99 to 0.9 in the last 1000 parameter updates but not disabling it completely. Recent work (Liu et al., 2018; Smith, 2018) showed that a large momentum helps escape from saddle points but can hurt the final convergence within the neighborhood of the optima, implying that momentum should be reduced at the end of training. Liu et al. (2018) find that larger momentum introduces higher variance noise and encourages more exploration at the beginning of optimization, and encourages more aggressive exploitation at the end of training. They suggest that at the final stage of the step size annealing, momentum SGD should use a much smaller step size than that vanilla SGD. When applied to fine-tuning,
can interpret that if the pre-trained model lies in the neighborhood of the optimal solution on the target dataset, the momentum should be small. Our work identifies the empirical evidence of disabling momentum helps final convergence, and fine-tuning close domains a good exemplar.
3.3 COUPLED HYPERPARAMETEES AND THE VIEW OF EFFECTIVE LEARNING RATE
Now that we had discovered the effect momentum by fixing other hyperparameters and only allowed momentum to change. But note that the two difficult scenarios shown in Figure (b)
Imagenet resnett01_a,
and (c) might also be mitigated by increasing or decreasing learning rate. That is, hyperparameters are coupled and vary- ing hyperparameter
Bust nater Emaor
40
change the optimal values of the
dee0.002
m=03
other hyperparameters that lead to the best performance. In addition, optimal values of certain hyperparameters depend the values of other hyperparameters in systematic ways. For example, learning rate entangled with batch size, momentum and weight decay. There is notion of effective learning rate (ELR) (Hertz et al., 1991; Smith al., 2018; Smith Le, 2018) for SGD with momentum: 1/ (1 m), which was shown be more closely related with training dynamics and final performance rather than 7). The effective learning rate with m = 0.9 10x higher than the one with m = if other hyperparameters are fixed, which is probably why we see an increase in optimal learning rate when momentum disabled in Figure 1(b) and Appendix A.
m-
drgs
ards
Figure 3: The effect of momentum w/ and w/o fixing ELR n/. When r' the same, momentum and 0.9 are almost equivalent. If is allowed to change, there is almost no difference between optimal performance obtained by different
the effective learning rate that matters for fine-tuning performance Because hyperparam- eters coupled, looking the performance with only one hyperparameter varied may give

---

Published as a conference paper at ICLR 2020
misleading understanding of the effect of hyperparameters. Therefore, to examine the effect of momentum, we should report the best result obtainable with and without momentum, as long as other hyperparameters explored are sufficiently explored. We re-examine previous experiments that demonstrated the importance of momentum tuning when the ELR r'' = nn/(1 - m) is held fixed instead of simply fixing learning rate 7/. Figure 3 shows that when of is constant, the best performance obtained by m = 0.9 and m = O are almost equivalent when other hyperparameters are allowed change. However, different ELR does result in different performance, which indicates its importance for the best performance. It explains why the common practice of changing only learning rate generally works, though changing momentum may results in the same result, they both change the ELR. In fact, as long as the initial learning rate is small enough, we can always search for the optimal momentum as is an amplifier, making the ELR larger by factor of 1/(1 - m). Therefore, momentum does determine the search ranges of learning rate.
Optimal ELR depends on the similarity between source domain and target domain Now that we have shown ELR is critical for fine-tuning performance, we are interested in the factors that determine the optimal ELR for a given task. Previous work (Smith & Le, 2018) found that there an optimum ELR which maximizes the test accuracy. However, the observations are only based on scratch training on small datasets (e.g. CIFAR- 10); the relationship between ELR and domain similarity, especially for fine-tuning, is still unexplored. To examine this, we search the best ELR on each fine-tuning task and reports in Fig. 4 the best validation error obtained by each ELR while allowing other hyperparameters to change. It shows the optimal ELR depends on both source domain and target domain. As shown in Fig. 4 (a-c), the optimal ELR for Dogs/Caltech//ndoor are much smaller than these for AircraftsFFlower/CCary when fine-tuned from ImageNet pre-trained model. Similar observations can be made DenseNets and MobileNet. Though the optimal ELR value is different, the relative order of domain similarity is consistent and architecture agnostic. We can also see a smaller ELR works better when source domain and target domain are similar, such as Dogs for ImageNet and Birds for iNar2017 (Fig. 4 (a, d-e)). Interestingly, the optimal ELR for training from scratch much larger and very similar across different target datasets, which indicates the distance from a random initialization is uniformly similar to different target dataset.
densener21, Imagenit, fineune recond
moMimet Imagenee freeann and indddd
dogs, of-
dogs, 21.03..'-0.00
11.61. qf-
00.00
Brsst Vatiostinnn Error
mi57, 20.54, 0.001
mite7, 22.10. 0.05
Bus Voloooonn Enror
mk57, 25.43, 6.00.5
Birds. 15.24, 0.00
hreds, 18.71, /f =0.00
birds, 20.59, of B.1
cars. 7.81, of 0.5
cars, 7.76, 0.5
cars, 8:34, of 10
aircrafts, 12:33, =1.0
sircrofts. 11.97, -03 0.1
alreratts, 13:92, 1'=
flowers, 2/91../= o/- 0.1
flowers, 2.53, 00.
flowers, 3.51. 6'= 0.1
28-
affective leaming rates gilz -ml
effectiv hamming tule 1ill (10)
affective lamiin rate: olt mm)
(a) ImageNet, ResNet-101
(b) ImageNet, DenseNet-122 121
(c) ImageNet, Mob1leNet1.0
frotune,
1amett311 .2 8aca335. fretuns rand resiadd crp. 234
ovp,
109
130
doge, 23.55, o' 0.05
dogs, 26.24, =/= 0.5
dogs. 29.32. 110
caltech, 18.82 g/=0.1
caltech. 22.14, o/ 2.05
caltech, 29.62, 110
Best Vostvattaatnn Emrr
min57, 28.11. 4'1 6.1 birds. 12.06, of 0.005
nite7. 19.42. of 70.05
02
ni267. 20.36, 2
Mrds, 22,90, n 0.5
binds, 2'=
9.58,
cars, 110
cars. 8.37. (/0
aircratts 15.45,
strccats. 15.48.
sirrreft, 14:34, (
2.70. p/1
offective (d) iNat2017, ResNet- 101
leaming oll
effoctive Isaming rote: (1) Scratch, ResNet- 101
(e) Places365, ResNet-101
Figure 4: The best validation errors obtained by different ELRs for different source-target domains. Note that the optimal ELR for each target dataset falls in the interior of search space. Each point in (a-e) is the lowest validation error obtained with different weight decay values while ELR fixed. The first row suggests that the connection between optimal ELR and domain similarity is architecture agnostic. The second row verifies that optimal ELR depends on the similarity between source domain and target domain.
7

---

Published as a conference paper at ICLR 2020
Table 4:
The connection between domain similarity and optimal ELR. The values in the second columa is provided by Cui et al. (2018). in which JFT pretrained 1esNet-101 was used as the feature extractor. Note that neither the pre-trained model or the datasct is released and we cannot calculate the metric for other datascts. In other columns, we calculate domain similarity using ImageNet pre-trained model as the feature extractor. The 1st, 2nd, 3rd and 4th highest scores are color coded. The optimal ELRs are also listed, which corresponds to the values in Fig 4.
JFT
ImageNet
iNat2017
Places365
ResNet-101
ResNet- 101
DenseNet- 121
MobileNet
ResNet- 101
ResNet- 101
sim
sim
sim
7
sim
sim
7
sim
"
Dogs
0.619
0.862
0.001
0.851
0.01
0.852
0.01
0.854
0.05
0.856
0.5
Caltech
0.892
0.005
0.881
0.01
0,878
0.01
0.871
0.1
0.888
0.05
Indoor
0,856
0.01
0.850
0.05
0.839
0.01
0.843
0.1
0.901
0.05
Birds
0.563
0,860
0,05
0.842
0.05
0.849
0.1
0.901
0.005
0.861
0.5
Cars 0.560
0.845
0.5
0.831
0.5
0.830
1.0
0.847
1.0
0.864
1.0
Aircrafts
0.556
0,840
1.0
0.817
0.1
0.831
1.0
0.846
0.5
0.853
0.5
Flowers
0.525
0.844
0.1
0.821
0.5
0.825
0.1
0.879
0.1
0.851
1.0
Optimal ELR selection based on domain similarity Now we have made qualitative observations about the relationship between domain similarity and optimal ELR. A quantitative characterization of the relationship could reduce the hyperparameter search ranges for HPO or even eliminate HPO by accurately predicting hyperparameters. We followed the domain similarity calculation in (Cui et al., 2018) and recalculate similarity scores for all source-tare similarity calculation in (Cui et al., 2018) use pre-trained JFT (Sun et al., 2017) models as feature extractor, which are not public available. We alternatively use Image pre-trained model or the source model as feature extractor. As shown in Table 4, there is good correlation between domain similarity score and the scale of optimal ELR. Generally, the more similar the two domains, the smaller the optimal ELR. Though is strictly corresponding the domain similarity score, the score provides reasonable prediction about the scale of optimal ELR, such as [0.001, 0.01], [0.01, 0.1], [0.1, 1.0] and therefore can reduce the search space for optimal ELR. Based on this correlation, simple strategy can be developed for optimal ELR selection given frequently used source model: one can calculate domain similarities and perform exhaustive hyperparameter searches for few reference datasets, including similar and dissimilar datasets. Then given new dataset to fine-tune, one can calculate the domain similarity and compare with the scores of reference datasets, and choose the range of ELRs with the closest domain similarity.
domain pairs. Note the original domain
Weight Decay and Learning Rate The relationship between weight decay and effective learning rate is recently well-studied (van Laarhoven, 2017: Zhang et al., 2018: Loshchilov Hutter, 2018). It was shown that the effect of weight decay models with BN layers is equivalent to increasing the ELR by shrinking the weights scales, i.e., n/' 0// And if the optimal effective learning rate exists, the optimal weight decay value À is inversely related with the optimal learning rate 1y. The "effective" weight decay X' A/n. We show in Figure that the optimal effective weight decay also correlated with domain similarity.
1marwi101 imageral, Finelume
1Mstie101
finelure
IMSIM6201 placa2e5,
dags, 9.83, -00.
100
dogs, 23.53, X
dogs, 26.24,
caftech, 11.61, 0.0
caltech. 18.82, A' 0.005
caltech, 22.14, -
Mino Vel dddcenn SToo
mit57. 20.54,2'=03
mit67, mit67.28.11, 0.005
mR57, 19-42, 0.002
hirds, 16 6.07
binds, 12.06, 0.02
binds, 22. 90, 0,000
<E
cars, 7.65, 0.000 0.005
cars, 3.58. 00.001 0.001
cars, 9.13, X 0,901
12.33, =
strorafts,
aircrafts. 15.45,
With
10-
decayt x1 ImageNet
effective weigtt dacay: (b) iNa12017
offective weight decay: a1-14 (c) Places-365
Figure 5: The relationship between optimal effective weight decay and source datasets. The optimal effective weight decay larger when the source domain similar with target domain.
8

---

Published as a conference paper at ICLR 2020
3.4 THE CHOICE OF REGULARIZATION
L2 regularization weight decay is widely used for constraining the model capacity (Hanson & Pratt, 1989; Krogh Hertz, 1992). Recently Li al. (2018; 2019) pointed out that standard L2 regularization, which drives the parameters towards the origin, is not adequate in transfer learning. To retain the knowledge learned by the pre-trained model, reference-based regularization was used to regularize the distance between fine-tuned weights and the pre-trained weights, so that the fine- tuned model is not too different from the initial model. Li et al. (2018) propose L2-SP norm, i.e. 2 11100 0o11 2 + A I1 where O' refers to the part of network that shared with the source network, and 6" refers to the novel part, e.g., the last layer with different number of neurons. While the motivation is intuitive, there are several issues for adopting reference based regularization for fine-tuning:
Many applications actually adopt fine-tuning target domains that quite different from source domain, such as fine-tuning ImageNet models for medical imaging (Mormont et al., 2018; Raghu al., 2019). The fine-tuned model does not necessarily have be close with the initial model.
The scale invariance introduced by Batch Normalization (BN) (loffe & Szegedy, 2015) layers enable models with different parameter scales to function the same, i.c., f(0) = f(ad). Therefore, when T2 regularization drives |00lß towards zeros, it could still have the same functionality as the initial model. On the contrary, model could still be different even when the L2-SP norm small.
L2-SP regularization would constrain 8" to be close to fg. so that ||0|] is relatively stable in comparison with L2 regularization. Given that ELR is approximately proportional to n/ /|@@|1 and smaller ELR is beneficial for fine-tuning from similar domains, it may explain why L2-SP provides better performance. If this is true, then by decreasing the initial ELR, L2-norm may function the same.
To examine these conjectures, we revisited the work of (Li ct al., 2018) with additional experiments. To show the effectiveness of L2-SP norm, the authors conducted experiments on datasets such as Dogs, Caltech and Indoor, which are all close to the source domain (ImageNet or Places-365), We extend their experiments by fine-tuning on both "similar" and "dissimilar" datasets, including Birds, Cars, Aircrafts and Flowers, with both L2 and L2-SP regularization (details in Appendix D). For fair comparison, we perform the same hyperparameter search for both methods. As expected, Table 5 shows that La regularization is very competitive with L2PSP on Birds, Cars, Aircrafts and Flowers, which indicates that reference based regularization may not generalize well for fine-tuning on dissimilar domains.
Table 5:
The average class error of et al., 2018) and the extension of their experiments of on "dissimilar" datasets. The italic datasets and numbers are our experimental results. Note that the original Indoor result fine-tuned from Places-365, while we fine-tune just from ImageNet pre-trained models.
Method
Dogs
Caltech
Indoor
Birds
Cars
Flowers
Aircrafts
T. (Li et al., 2018)
18.6
14.7
20.4
L2-SP (Li et al., 2018)
14.9
13.6
15.8
L2 with HPO
16.79
14.98
23.00
22.51
10.10
5.70
13.03
SP-SP with HPO
13.86
14.45
21.77
22.32
9.59
5.28
13.31
We also check the change of regularization terms during training for both methods as well as their best hyperparameters. As shown in Figure 6, the L2 regularization usually decrease the weights norm more aggressively, depending on the value of X, while L2-SP regularization keeps the norm less changed. We can see that the optimal learning rate of L2 regularization is mostly smaller than L2-SP, which may compensate for the decreased weight norm or increased ELR. Interestingly, for Dogs dataset, both regularization terms grow much larger after a few iterations and then become stable, which means constraining the weights to be close to initialization is not necessarily the reason for SP to work even for close domains. also seems contradicting to previous finding (Zhang al., 2018) that weight decay functions increasing ELR by decreasing weight norms. However,

---

Published as a conference paper at ICLR 2020
might be reasonable as large norm actually decreases the ELR, which could be helpful due to the close domain similarity between Dogs and ImageNet.
resnet1bl_y1 yl, Imagenet, 64
resnetio1y1. Imagenet n
10
36.73, 0,0001
caltech. 2.005, 0,0000
caltech. 14.45, 0.00055
Indoor 23,00. p--0.00.000 A 20000001
Indoor. 21.77 2-000... -11 brnh,, 22.32.
normal lzed Ly norm
birds, 22.52, Su005, D.000
gm
normalized U-
1.0
cars, 10 10.12, p..0.,nmm............. b.0m,, 5.0.01
cars, 9.59, 1.....000..2.... 0000000
fiowers; -.0000000000000 A- 0.001
307
nowers, 5,28, c. 0.02. 0.001
22-1
2000
4000
G600
Iterstions
normalized L2 norm
(b) normalized S2-SP norm
Figure 6: The normalized Le norm and L2-SP norm during training. The y-axis is the relative change of the regularization term in comparison the initial value, i.e., 2/ for L2 norm and (Aıll0; doll 2 + >2110" (A2l)(2))l)))) 9 l for L2-SP norm. Optimal hyperparamcters are also given in the legend. Note that experiment uses batch size 64 instead of 256, which results in smaller optimal learning rate comparing to previous result.
DISCUSSION
The two extreme ways for selecting pepppaaameterssppffrrmiii exhaustive hyperparameter search or taking ad-hoc hyperparameters from scratch training-cono could be either too computationally ex- pensive or yield inferior performance. Different from training from scratch, where the default hyperparameter setting may work well for random initialization, the choice of hyperparameters for fine-tuning is not only dataset dependent but is also influenced by the similarity between the target and source domains. The rarely tuned momentum value could also improve impede the performance when the target domain and source domain are close given insufficienlly searched learning rate. These observations connect with previous theoretical works on decreasing momentum at the end of training and effective learning rate. We further identify that the optimal effective learning rate correlates with the similarity between the source and target domains. With this understanding, one significantly reduce the hyperparameter search space. We hope these findings could be one step towards better and efficient hyperparameter selection for fine-tuning.
ACKNOWLEDGMENTS
The authors would like to thank all anonymous reviewers for their valuable feedback.
REFERENCES
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji. Charless Fowlkes, Stefano Soatto, and Pietro Perona, Task2vec: Task embedding for meta-learning. arXiv preprint 10XXv::902.0354::: 2019.
Liang- Ching-Chich Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE T-PAMI, 40(4))834 848, 2017.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.095: 2018.
Yin Cui, Yang Song, Chen Sun, Andrew Howard, and Serge Belongie. Large scale fine-grained categorization and domain-specific transfer learning, In CVPR, 2018.
Jia Deng, Alexander Berg, Kai Li, and Li Fei-Fei. What does classifying more than 10,000 image categories tell us? In ECCV, 2010.
10

---

Published as a conference paper at ICLR 2020
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: deep convolutional activation feature for generic visual recognition. In ICML, 2014.
Weifeng Ge and Yizhou Yu. Borrowing treasures from the wealthy: Deep transfer learning through selective joint fine-tuning. In CPVR, 2017.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.
Gabriel Goh. Why momentum really works. Distill, 2(4):e6, 2017.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv: (tfl::/7060067777 2017.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.
Stephen Jose Hanson and Lorien Y. Pratt. Comparing biases for minimal network construction with back-propagation. NIPS, 1989.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016b.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. ICCV, 2017.
Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In ICCV, 2019.
John Hertz, A Krogh, and Richard G Palmer. Introduction to the theory of neural computation. 1991.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.0448 2017.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
Minyoung Huh, Pulkit Agrawal, and Alexei A Efros. What makes imagenet good for transfer 2016.
learning? arXiv preprint
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In /CML, 2015.
Aditya Khosla, Nityananda Jayadevaprakash. Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine-grained image categorization. In First Workshop on Graiiedd Visual Categorization, CVPR, 2011.
Simon Kornblith, Jonathon Shlens, and Quoe Le. Do better imagenet models transfer better? In CVPR, 2019.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolu- tional neural networks. In NIPS, 2012.
Anders Krogh and John A Hertz. simple weight decay can improve generalization. In NIPS, 1992.
Xingjian Li, Haoyi Xiong, Hanchao Wang, Yuxuan Rao, Liping Liu, and Jun Huan. Delta: Deep learning transfer using feature map with attention for convolutional networks. In ICLR, 2019.

---

Published as a conference paper at ICLR 2020
Xuhong Li, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In JCML, 2018.
Tianyi Liu, Zhehui Chen, Enlu Zhou, and Tuo Zhao. Toward deeper understanding of noncon- vex stochastic optimization with momentum using diffusion approximations. arXiv preprint 1rX2v:/802.05/555 2018.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2018.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, 2018.
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013.
Romain Mormont, Pierre Geurts, and Raphaël Marée. Comparison deep transfer learning strategies for digital pathology. In CVPR Workshops, 2018.
Yurixi E Nesterov. A method for solving the convex programming problem with convergence rate (1/K" 2). In Dokl. akad. nauk Sssr, volume 269, pp. 543-547, 1983.
Jiquan Ngiam, Daiyi Peng, Vijay Vasudevan, Simon Kornblith, Quoc V Le, and Ruoming Pang. Domain adaptive transfer learning with specialist models. arXiv preprint 1rXiv::811.000000 2018.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics & Image Processing. IEEE, 2008.
Boris T Polyak. Some methods of speeding up the convergence iteration methods. USSR
Computational Mathematics and Mathematical Physics, 4(5):1-17, 1964.
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning with applications to medical imaging. In NIPS, 2019.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, (5(3):211-253 2015.
Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Cnn features off-the-shelf: an astounding baseline for recognition. In CVPR workshops, pp. 806-813, 2014.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. In ICLR, 2015.
Leslie N Smith. Cyclical learning rates for training neural networks. In WACV, 2017.
Leslie N Smith. A disciplined approach to neural network hyper-parameters: Part 1-learning rate, batch size, momentum, and weight decay. arXiv preprint arXiv: 1803,0980, 2018.
Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial Intelligence and Machine Learning for Multi- Domain Operations Applications, volume 1 1006, pp. 1100612. International Society for Optics and Photonics, 2019.
Samuel L Smith and Quoe V Le. A bayesian perspective generalization and stochastic gradient descent. In ICLR, 2018.
Samuel L Smith, Pieter-Jan Kindermans, Chris Ying, and Quoc V Le. Don't decay the learning rate, increase the batch size. In ICLR, 2018.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning In CVPR, 2017.
12

---

Published as a conference paper at ICLR 2020
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. JCML, 2013.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In CVPR, 2018.
Twan Laarhoven. L.2 regularization versus batch and weight normalization. In NIPS, 2017.
C. Wah, S. Branson, P. Welinder, Perona, and S. Belongie. The Caltech-UCSD Birds-200- Dataset. Technical Report ENS-TR-2011-001 California Institute of Technology, 2011.
Junyuan Xie, Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, and Mu Li. Bag of tricks for image classification with convolutional neural networks. arXiv preprint urXiv: 1812.01 187, 2018.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In NIPS, 2014.
Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms of weight decay regularization. arXiv preprint arXiv:1810.12280 2018.
Jian Zhang and loannis Mitliagkas. Yellowfin and the art of momentum tuning, arXiv preprint arXiv: (0ov:/766.34411 2017.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE T-PAMI, 40(6):1452-1468 2018.
13

---

Published as a conference paper at ICLR 2020
A THE EFFECTIVENES, OF MOMENTUM
Searching for Optimal Momentum To check the effectiveness of momentum on fine-tuning, we can search the best momentum values for fine-tuning with fixed learning rate but different weight decay and batch size. Taking Birds dataset as an example, Figure 7 provides the convergence curves for the results shown in Figure ((a), which shows the learning curves fine-tuning with 6 different batch sizes and weight decay combinations. Zero momentum outperforms the nonzero momentum in 5 out of 6 configurations.
1esve:1722.6 g- 2.0 2-200
12404300 v2, 0=f.h..-222.--11110
1esrec17L vt, 000 n=2.1-00
0 21.69
m- 23.58
m-5.55 25.53
20 0.35, 17.54
m- 0.95, 29.09
m= 19.30
16.88
m.S. 18:10
m= 18.42
Mh 0.8. 17:21
-- 0.8 17.59
0...8.17.59
0.0, 17.26
17:41
0.0, 17.21
20-
0
109
186
201
159
(a) 256, À= 0.0005
(b) 256, À 0.0001
(c) 256, X = 0.0
ruurat20L v2, nn 0.22 en 26, 6.0055
reave:121v3 1-0 n- 1000
naret 12. 1-30LL-28.4-00
0.99, 62.15
No 0.99 40.42
m -.99 43.67
0.95, 23.64
0.85 10.20
0.95 35.92
0.3. 21.18
-- 0.9 21.92
0.9. 0.3.23.777
O.S. 19.49
20.86
G.8. 22.70
18.59
04
280
150
Sp507s
(d)
16, X 0.0005
(e) 16,
0.0001
(1) 12 16, X 0.0
Figure 7: Searching for the optimal momentum on Birds dataset with fixed learning rate and weight decays. The solid lines are training errors and the dashed lines validation errors.
Effective learning rate increases after disabling momentum. Figure 8 compares the perfor- mance of with and without momentum for Dogs dataset with a range of different learning rates. Note that the learning rate with similar performance generally increases 10x after changing m from 0.9 to 0.0, which is coherent with the rule of effective learning rate n/ =n/(1 m). Same observations can be made on other datasets as shown in Figure 9.
n=0.1, 24.48
0.0.1. 17:63
0-0.55 23
n- 0.05, 14.37
0.03, 17.20
0.01, 11.00
0 0.005, 14.14
n= 0. 00, 1021
20.0011 10.96
O 0005. 10.30
0 0000005, 10.98
0.0001. 9.83
n- 0000000, 38.57
10
10-
150
260
100
203
(a) Dogs, 0.9
(b) Dogs, m 0.0
Figure 8: The effect of momentum when learning rate is allowed to change. The learning rate for the best performance increases 10x after changing m from 0.9 to 0.0, which is coherent with the rule of effective learning rate. Note that weight decay À is fixed at 0.0001.
14

---

Published as a conference paper at ICLR 2020
19.0
25.44
end 14.5
12A11
1-38L 2336
0=000 21
nnd 911, 32
EEEL 14.92
1331 22.44
p=00000 29.95
and 525--144 21.9
3
0-84 15.20
4-3225, 21.22
0m0.8ss 81.00
(a) Caltech, m = 0.9
(b) Caltech, m = 0.0
(c) Indoor, m 0.9
(d) Indoor, 172 0.0
23.0
25% 14:14
2--21 h-16
and 16.38
17AL
5.10
1422
and 30L-2733 33
E66L 21
2-2321 25
0.00, 52.92
1001, 33.25
060000 $7.00
0.000 96.2
(c) Birds, m = 0.9
Birds, 772 0.0
g Cars, 772 0.9
(h) Cars, m = 0.0
meerodt 2000
1-31 655
0.L 291
34.25 pnG
17.01
2013
3.00
521,
6:01
(i) Aircrafts, m 0.9
(i) Aircrafts, m = 0.0
(k) Flowers, m7 0.9
(l) Flowers, m 0.0
Figure 9: The effect of momentum when learning rate is allowed to change (Figure 8 continued). The learning rate for the best performance increases 10x after changing m from 0.9 to 0.0, which is coherent with the rule of effective learning rate.
B DOMAIN SIMILARITY
The domain similarity calculation based on Earth Mover Distance (EMD) is introduced in the section 4.1 of (Cui et al., 2018)*. Here we briefly introduce the steps. In (Cui et al., 2018), the authors first train ResNet-101 They extracted features from the penultimate layer of the model for each image of the training set of the source domain and target domain. For ResNet- 101, the length of the feature vector is 2048. The features of images belonging to the same category are averaged and g(*;) denotes the average feature vector of ith label in source domain S, similarly, glz)) denotes the average feature vector of jth label in target domain T. The distance between the averaged features of two labels dr, (l9(s;) g(tz)]l ][. Each label is associated with weight w E [0, 1] corresponding to the percentage of images with this label in the dataset. So the source domain S with m labels and the target domain T with labels can be represented as {(si, )m and {(tz. W/, wx,)) )i
the large scale JFT dataset (Sun et al., 2017) and use it as a feature extractor.
The EMD between the two domains is defined as
Eist,e= feydis
(S = EMD(S,T)
(3)
Ex fij
where the optimal flow Jay corresponds to the least amount of total work by solving the EMD) optimization problem. The domain similarity defined as
s T) e -d(,)))
(4)
where y is 0.01. Note that the domain similarity value is not ranging from 0 to 1.
Due to the unavailability of the large-scale JFT dataset (300x larger than ImageNet) and its pre-trained 10s model, we cannot use it for extracting features for new datasets, such as Caltech256 and
"The extracted features and code are available in https://gither thg.tttt..... com/ ri chardaecn/ cvpr18- 1on118-nnaturaaa ist-
15

---

Published as a conference paper at ICLR 2020
MIT67-Indoor. Instead of using the powerful feature representation, model (ResNet-101) as the feature extractor. Table 4 compares the domain similarities calculated by consistent patterns across different architectures:
pre-trained ImageNet
different pre- pre-trained models and we can see e.g. The 1st and 2nd highest similarity scores are Caltech and Dogs regardless of architectures; the 3rd and 4th highest similarity scores refers Birds and Indoor; the most dissimilar datasets Cars, Aircrafts and Flowers, though the relative orders for them are not exactly the same. Besides using fixed feature extractor, an alternative way is use the source domain model directly as the feature extractor for both source domain and target domain, which may captures the transfer learning process more precisely than uniform feature extractor.
C THE EFFECTIVENESS OF BN MOMENTUM
Kornblith et al. (2019) conducted extensive fine-tuning experiments with different hyperparameters. One observation they made is that the momentum parameter of BN layer is essential for fine- tuning. They found it usefuu to decrease the BN momentum parameter from its ImageNet value to max(1 - 1, 0.9) where 8 is the number of steps per epoch. This will change the default BN momentum value (0.9) when is larger than 100, but it only applies when the dataset size is larger than 25.6 25.6K with batch size 256. The maximum data size used in our experiments is Caltech-256, which is 15K, this strategy seems not applicable.
We further validate the effect of BN momentum by performing a similar study as to ELR. The goal to identify whether there is an optimal BN momentum for given task. For each dataset, we fine-tune the pre-trained model using previously obtained best hyperparameters and only vary BN momentum. In addition to the default value 0.9, we also set it to 0.0, 0.95 and 0.99. The rational is that if BN mommentum is a critical hyperparameter, we should expect significant performance differences when the value changed from the optimal value. As shown in Figure 10, we can see Via 0.99 slightly improves the performance for some datasets, however, there is no significant performance difference among values greater than 0.9. One hypothesis is that similar domains will share similar BN parameters and statistics, BN momentum may affect the parameter adaptation. More investigation is still needed to fully understand its effectiveness.
reanw:10) V2, Imagenel, fieetune rand res and crop, 2e-
20
dogs, 9.70, caltech. 11.59, mcc = 0.99 mit67. 20.54, 0.9
Best Valldation Error
birds, 16.34, mm = 0.9
7.47, 0.99
aircraft, 12.33, mm 0.9
22
flowors, 2.91, man - 0.9
30
BN momentum man
Figure 10: Performance of different BN momentum for each dataset with existing optimal hyperparameters.
D EXPERIMENTAL SETTINGS FOR COMPARISON OF L2 AND L2-SP
The experiments in Section 3.4 is based the code provided by (Li al., 2018). The base network ImageNet pretrained Res-ee--01--V The model fine-tuned with batch size 64 for 9000 iterations, and learning rate is decayed once at iteration 6000. Following the original setting, we use momentum 0.9. We performed grid search on learning rate and weight decay, with the range of n (0.02, 0.01, 0.005, 0.001, 0.0001} and À (0.1, 0.01, 0.001, 0.0001). and report the best average class error (1 average accuracy) for both methods. For LPPSP norm, we follow the authors' setting to use constant X = 0.01. Different with the original setting for L2 regularization, we set Ag = > to simulate normal Lg-norm.
assi caasificcatiin fication
16

---

Published as a conference paper at ICLR 2020
E DATA AUGMENTATION
Data augmentation is an important way of increasing data quantity and diversity to make models more robust. It is even critical for transfer learning with few instances. The effect of data augmentation can be viewed as a regularization and the choice of data augmentation can be also viewed as a hyperparameter. Most current widely used data augmentation methods have verified their effectiveness on training ImageNet models, such as random mirror flipping, random rescaled cropping". color jittering and etc (Szegedy et al., 2015; Xie et al., 2018).
Do these methods transfer for fine-tuning on other datasets? Here we compare three settings for data augmentation with different momentum settings: 1) random resized cropping: our default data augmentation: 2) random cropping: the same as standard data augmentation except that we use random cropping with fixed size; 3) random flip: simply random horizontal flipping. The training and validation errors of fine-tuning with different data augmentation strategies and hyperparameters are shown Figure and Figure 12.
1earat001 v2. 0.0. mw0.0..-------....
rosrat000 v2. 5 238.m=
rad resized rrop, 17.20
rand resized cmp, 17.55
rand restred crop. 3.12 rand crap. 2.0.5.0
rand crog. 11.99
rand crop. 21.72
sand flp. 12.7b
rand flp. 24.09
rend flip, 6.41
35
33-
50
350
200
259
X
254
3x
59
300
156
240
250
30
(a) Dogs, 172 0.9
(b) Aircrafts, m 0.9
(c) Flowers, m 0.9
resvit01 vB. g.0.0 6- 236, O. --200
(0 vt. e- 3. 250, 2 3
reseect11 yt, n- n=255 mi A. A-5.000 5000000
sand resized crop, 11.00
rand resioed crup, rand crop. 29.04
tend resiced crop. 5.45 rand crnpicrp.... 7.76
rand crop, 11.35
259
300
Epects
(d) Dogs, 0.0
(e) Aircrafts, mI 0.0
(l) Flowers, m 0.0
Figure 11: Fine-tuning with different data augmentation methods and hyperparameters. Dashed curves are the validation errors. Strong data augmentation is harder to train as it converge slowly and needs more number of epochs to observe the advanced performance on datasets such as Aircrafts. Simple data augmentation (red curves) converges much faster in training error. Strong data augmentation (blue curves) overfits the Dogs dataset with default hyperparameter but performs well with m 0.
The effect of data augmentation is dataset dependent and is also influenced by other hyperpa- rameters The first row in Figure 11 shows that advanced data augmentation with default hyperpa- rameters (m = 0.9 and 7 = 0.01) leads to overfitting for Dogs while generalize better on Aircrafts and Flowers. Similar observations can be made in Figure 12. However, when momentum is disabled, the overfitting disappears for Dogs and Caltech. This is explainable since random resized cropping adds more variance to the gradient direction, and disabling momentum will lead smaller ELR which will be helpful for fine-tuning from a similar domain. On the other hand, the performance of random cropping decreases when momentum disabled. As random cropping produces training samples with less variation than random resized cropping, disabling momentum or decreasing the ELR might lead to underfitting or stucking in poor local minima. This can mitigated as we increase the learning rate for random cropping, which adds variation to the gradients. As shown in Table 6,
'Randomly crop rectangular region with aspect ratio randomly sampled in [3/4, 4/3] and area randomly sampled in [8%, 100%] (Szegedy et al., 20155)
17

---

Published as a conference paper at ICLR 2020
12.1
(a) Caltech, m 0.9
(b) Caltech, 0.0
(c) Indoor, 0.9
(d) Indoor, 0.0
emeerit pa3at ma3t warh can
mosrth pwaiit wa
23.03 oed 20.3
ralued 17.43
20.13
$2.72
266
16
(e) Birds, 0.9
Birds, 0.0
(g) Cars, 0.9
(h) Cars, 0.0
Figure 12: Comparison of data augmentation methods with different momentum values (Figure 11 continued). The other hyperparameters are: 256, 17 0.01 and X = 0.0001.
when learning rate increased fro 0.01 to 0.05, disabling momentum shows better performance than nonzero momentum on datasets that are close, similar to previous findings with random resized cropping.
Table 6: Comparison of data augmentation methods with different momentum values. The rest of the hyperpa-
rameters are: n 256 and À = 0.0001.
Data Augmentation
"
Dogs
Caltech
Indoor
Birds
Cars
Flowers
Airerafts
0.9
0.01
17.20
14.85
23.76
18.10 18.10
9.10
3.12
17.55
Rand resized crop
0
0.01
11.00
12.11
21.14
17.41
11.06
5.48
20.58
0.9
0.01
11.99
12.42
23.39
20.31
17.77
5.63
21.72
0.01
11.35
12.89
25.19
22.11
23.87
7.76
29.04
Rand crop
0.9
0.05
16.85
14.80
23.46
18.81
13.70
4.85
17.64
0.05
11.79
12.52
23.24
20.69
20.00
7.06
23.43
F SOURCE DOMAINS
Pre-trained models For most of experiments, from the model zoo of MXNet GluonCV ?, To get the pre- pre-trained models for NN17 and Places- 365, fine-tune from the ImageNet pre-trained model with the default fine-tuning hyperparameters for 60 epochs, where learning rate is decayed at epoch 45 by factor of 10. Table 7 illustrates the Top-1 errors of each pre-traines traitre model on their validation sets.
the pre-trained ResNet- 100__011 v__ model
Table 7: The Top-1 error of ResNet-101 pre-trained on different source dataset.
Dataset
class
Top-1 error
ImageNet
1000
21.4
iNat2017
5,089
32.2
Places-365
365
31.5
"https://g1 uon-cv. coan-cy.mmee.....mmm wwnet. 1o/rodel a_teellooooooaaaaa asai ficati io.tion
18

---

Published as a conference paper at ICLR 2020
Training from Scratch with HPO The default hyperparameters for training from scratch = 0.1, = 0.0001, m = 0.9 and n = 256. We train 600 epochs, and decay the learning rate at epoch 400 and 550 by factor of 10. To perform Hyperparameter Optimization (HPO), search hyperparameters the following space: 1/ E [0.1, 0.2, 0.5] and À E [0.0001, 0.0005]. Figure 13 shows the training/validation errors of training from scratch each dataset with different learning rate and weight decay. We observe weight decay 0.0005 consistently performs better than 0.0001.
Insufficient hyperparameter search may lead to miss-leading conclusion show the impor- tance of hyperparameter tuning, Table 8 compares the performance with and without hyperparameter tuning for both fine-tuning and training from scratch tasks. With the default hyperparameters, some inappropriate conclusions might be made, c.g., "there is significant gap between fine-tuning and training from scratch", "fine-tuning always surpass training from scratch" or "fine-tuning from iNat cannot beat the performance of ImageNet". However, with HPO, those statements may not be valid. For example, training from scratch surpass the default fin-ttning result on Cars and Aircrafts and the gap between fine-tuning and training from scratch is much smaller. Previous studies (Kornblith et al., 2019; Cui et al., 2018) also identified that datasets like Cars and Aircrafts do not benefit too much from fine-tuning.
Table Comparison of default hyperparameters HPO both fine-tuning (FT) and training from scratch (ST) tasks. FT Default and ST Default use their defaulr hyperparameters, respectively. HPO refers to the finding the best hyperparameters with grid search.
Method
Source
Dogs
Caltech
Indoor
Birds
Cars
Aircrafts
Flowers
FT Default
ImageNet
17.20
13.42
23.76
18.10
9.10
17.55
3.12
FT Default
iNat2017
24.74
20.12
30.73
14.69
11.16
19.86
3.19
FT Default
Places-365
30.84
22.53
22.19
27.72
11.06
21.27
5.66
ST Default
38.26
36.21
45.28
43.72
16.73
26.49
22.88
FT HPO
ImageNet
9.83
11.61
20.54
16.34
7.61
12.33
2.91
FT HPO
iNat2017
23.51
18.82
28.11
12.06
9.58
15.45
2.70
FT HPO
Places-365
26.24
22.14
19.42
22.90
9.13
15.48
5,06
ST HPO
29.32
29.62
39.36
30.08
8.37
14.34
16.51
19

---

Published as a conference paper at ICLR 2020
resnet101 V2, 256, m= 0.9, 0.0001
resnet101 V2, 256,
0.0005
0.5, 37.14
n-0.5 32.20
0.2, 36.14
30-
n 30.21
0.1, 38.26
29.32
40
40-
Error
Error
30
30-
20
100
200
500
400
500
500
100
200
300
400
500
500
Epochs
Epochs
(a) Dogs, À 0.0001
(b) Dogs, X 0.0005
(c) Caltech, X 0.0001
(d) Caltech, À 0.0005
(e) Indoor, X 0.0001
n Indoor, À 0.0005
42-42.72
21.20
D 2633
Birds, 0.0001
(h) Birds, A 0.0005
Cars, 0.0001
Cars, A 0.0005
17.6
16.0
(k) Aircrafts, 0.0001 (l) Aircrafts, X 0.0005 (m) Flowers, X = 0.0001 (n) Flowers, À 0.0005
Figure 13: Training from scratch with various learning rate and weight decay. The batch size is 256 and the momentum 0.9. The solid curves are training error and the dashed lines are valdiation error.
20


Reviews:
[{"review_id": "SJev0nZKYB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the role of different hyperparameters in finetuning image recognition models on new target tasks. The authors run a large set of experiments and show that, perhaps non-surprisingly, hyperparameters matter. In particular, they show that momentum, which is typically ignored in finetuning, is quite important, and that the momentum values  that work well depend on the similarity between the source and target datasets. They also show important correlations between momentum, learning rate, and weight decay.\nOverall, despite some issues detailed below, the paper is clearly written, presents a coherent story, and its conclusions will be useful to the community. \n\nComments:\n\n1. My main concern about this paper relates to the importance of momentum. The authors argue that this hyperparameter is \"critical for fine-tuning performance\". However, they later show that in fact what matters is the ratio between the learning rate (LR) and the momentum. In this case, it might be justified to fix the momentum value and only modify the LR, as often done. \n\n2. The EMD values of Birds, Cars and Aircrafts are within 0.7 points of each other (while Dogs is much higher and Flowers is quite lower). Although I am not too familiar with this method, I find it somewhat hard to believe that these small differences explain the error differences on Table 2.\n\n3. While the paper is fairly clear in writing, the figures (e.g., fig. 3 and 4) are extremely hard to read on print, and thus hard to draw conclusions from. Figure 4 is confusing also on screen.\n\n4. To promote reproducibility, it would be better to report in this kind of research validation rather than test results. There is some confusion in Figure 4, the axes say validation error, while the caption says test error, but in the other figures test results are reported.\n\nMinor:\n\n1. The authors say in the intro \"Even when there is enough training data, fine-tuning is still preferred as it often reduces training time significantly (He et al., 2019).\", but later make a somewhat contradictory claim: \"He et al. (2019) questioned whether ImageNet pre-training is necessary for training object detectors. They find the solution of training from scratch is no worse than the fine-tuning counterpart as long as the target dataset is large enough.\".\n\n2. A couple of typos around the paper:\n- section 2: \"However, most of these advances on hyperparameter tuning are designed *from* training from scratch\" (should be \"for\")\n- The first sentence of 3.3 is ungrammatical\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BkxO9K7JcH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission studies the problem of transfer learning and fine tuning. This submission proposes four insights: Momentum hyperparameters are essential for fine-tuning; When the hyperparameters satisfy some certain relationships, the results of fine-tuning are optimal; The similarity between source and target datasets influences the optimal choice of the hyperparameters; Existing regularization methods for DNN is not effective when the datasets are dissimilar. This submission provides multiple experiments to support their opinion.\n\nPros:\n+  This submission provides interesting facts that are omitted in previous research works.\n+  This submission examines the previous theoretical results in empirical setting and finds some optimal hyperparameter selection strategies.\n+  This submission provides many experiment results of fine-tuning along with its choice of hyperparameters that could be taken as baselines in future researches.\n\nCons:\n-\tAll experiments results are based on same backbone, which makes all discoveries much less reliable. More experiments on other backbones are necessary. Furthermore, this submission claims that the regularization methods such as L2-SP may not work on networks with Batch Normalization module. But there is no comparison on networks without BN.\n-\tProviding a complete hyperparameter selecting strategy for fine-tuning could be an important contribution of this submission. I suggest authors to think about it.\n-\tThis submission claim that the choice of hyperparameters should depend on similarity of different domains. But this submission does not propose a proper method for measure the similarity or provide detailed experiments on previous measurements. \n-\tIt seems that the MITIndoors Dataset is not similar with ImageNet from the semantic view. This submission does not provide similarity measurement between these datasets. Why the optimal momentum is 0? \n-\tThe effective learning rate and ‘effective’ weight decay are not first given in this submission. This makes the novelty of this submission relatively weak. Authors only test these strategies in fine-tuning setting and find that they also work with a different initialization.\n-\tIt seems that merely searching for learning rate and weight decay hyperparameters (as Kornblith et al. (2018) did) on a fixed momentum is Ok if there is a most effective relationship between learning rate and momentum. So the discoveries in the first part that a 0 momentum can be better is based on a careless search of learning rates?\n-\tThis submission omits that Kornblith et al. (2018) also referred to the fact that the momentum parameter of BN is essential for fine-tuning and provided a strategy in section A.5. Discussion about this strategy will make this submission more complete.\n\nThis submission gives important discoveries about the hyperparameter choice in the fine-tuning setting. But there are several flaws in this submission. I vote for rejecting this submission now but I expect authors to improve the submission in the future version.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xkAOgitB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides extensive experimental results to investigate the influence of hyper-parameters on fine-tuning and challenges several commonly-held beliefs. The hyper-parameters of training from scratch does not always perform well when applied to fine-tuning. Furthermore, current L_2-SP regularization is not necessarily helpful when the domain discrepancy is large. \nThe authors discover that the optimal momentum value is closely related to domain similarity. For similar target datasets, 0 momentum is a better choice than 0.9, since it potentially allows better convergence. Similar to training from scratch, the actual effect at play is the effective learning rate and ‘effective’weight decay. This further involves the coupling of hyper-parameters.\nDifferent from the commonly-held belief, the L_2-SP regularization does not always perform better than L_2. When domain discrepancy is large, the regularization effect will be worsened. \nThis paper is well-written and makes several interesting discoveries. My question for the authors is as follows:\nIn the momentum section, the authors postulate that for more similar target datasets, smaller momentum performs better. Here, the similarity is quantified by EM distance defined in the feature space. However, for the five datasets provided, the similarity of them are really close, making this claim less convincing. The conclusion is reasonable, but the authors may need a more reliable method to compare the similarity between datasets.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a guide for setting hyperparameters when fine-tuning from one domain to another. This is an important problem as many practical deep learning applications repurpose an existing model to a new setting through fine-tuning.  All reviewers were positive saying that this work provides new experimental insights, especially related to setting momentum parameters. Though other works may have previously discussed the effect of momentum during fine-tuning, this work presented new experiments which contributes to the overall understanding. Reviewer 3 had some concern about the generalization of the findings to other backbone architectures, but this concern was resolved during the discussion phase. The authors have provided detailed clarifications during the rebuttal and we encourage them to incorporate any remaining discussion or any new clarifications into the final draft. 
Decision: Accept (Poster)