Title: DOMAIN ADAPTIVE TRANSFER LEARNING

Abstract: Transfer learning is a widely used method to build high performing computer vision models. In this paper, we study the efficacy of transfer learning by examining how the choice of data impacts performance. We find that more pre-training data does not always help, and transfer performance depends on a judicious choice of pre-training data. These findings are important given the continued increase in dataset sizes. We further propose domain adaptive transfer learning, a simple and effective pre-training method using importance weights computed based on the target dataset. Our methods achieve state-of-the-art results on multiple finegrained classification datasets and are well-suited for use in practice.

1. INTRODUCTION:

Transfer learning using pre-trained models is one of the most successfully applied methods in the field of computer vision. In practice, a model is first trained on a large labeled dataset such as Ima-geNet (Russakovsky et al., 2015) , and then fine-tuned on a target dataset. During fine-tuning, a new classification layer is learned from scratch, but the parameters for the rest of the network layers are initialized from the ImageNet pre-trained model. This method to initialize training of image models has proven to be highly successful and is now a central component of object recognition (Razavian et al., 2014) , detection (Girshick, 2015; Ren et al., 2015; Huang et al., 2017) , and segmentation (Shelhamer et al., 2017; Chen et al., 2018; He et al., 2017) .

By initializing the network with ImageNet pre-trained parameters, models train with higher accuracy and converge faster, requiring less training time. They have also achieved good performance when the target dataset is small. Most prior work have considered only ImageNet as the source of pretraining data due its large size and availability. In this work, we explore how the choice of pretraining data can impact the accuracy of the model when fine-tuned on a new dataset.

To motivate the problem, consider a target task where the goal is to classify images of different food items (e.g., 'hot dog' v.s. 'hamburger') for a mobile application (Anglade, 2017) . A straight-forward approach to applying transfer learning would be to employ an ImageNet pre-trained model finetuned on a food-specific dataset. However, we might wonder whether the pre-trained model, having learned to discriminate between irrelevant categories (e.g., 'dogs' vs. 'cats'), would be helpful in this case of food classification. More generally, if we have access to a large database of images, we might ask: is it more effective to pre-train a classifier on all the images, or just a subset that reflect food-like items? Furthermore, instead of making a hard decision when selecting pre-training images, we can consider a soft decision that weights each example based on their relevancy to the target task. This could be estimated by comparing the distributions of the source pre-training data and the target dataset. This approach has parallels to the covariate shift problem often encountered in survey and experimental design (Shimodaira, 2000) .

We study different choices of source pre-training data and show that a judicious choice can lead to better performance on all target datasets we studied. Furthermore, we propose domain adaptive transfer learning -a simple and effective pre-training method based on importance weights computed based on the target dataset.

1.1. SUMMARY OF FINDINGS:

More pre-training data does not always help. We find that using the largest pre-training dataset does not always result in the best performance. By comparing results of transfer learning on different subsets of pre-training data, we find that the best results are obtained when irrelevant examples are discounted. This effect is particularly pronounced with fine-grained classification datasets.

Matching to the target dataset distribution improves transfer learning. We demonstrate a simple and computationally-efficient method to determine relevant examples for pre-training. Our method computes importance weights for examples on a pre-training dataset and is competitive with hand-curated pre-training datasets. Using this method, we obtain state-of-the-art results on the fine-grained classification datasets we studied (e.g., Birdsnap, Oxford Pets, Food-101).

Fine-grained target tasks require fine-grained pre-training. We find that transfer learning performance is dependent on whether the pre-training data captures similar discriminative factors of variations to the target data. When features are learned on coarse grained classes, we do not observe significant benefits transferred to fine-grained datasets.

2. RELATED WORK:

The success of applying convolution neural networks to the ImageNet classification problem (Krizhevsky et al., 2012) led to the finding that the features learned by a convolutional neural network perform well on a variety of image classification problems (Razavian et al., 2014; Donahue et al., 2014) . Further fine-tuning of the entire model was found to improve performance (Agrawal et al., 2014) . Yosinski et al. (2014) conducted a study of how transferable ImageNet features are, finding that the higher layers of the network tend to specialize to the original task, and that the neurons in different layers in a network were highly co-adapted. They also showed that distance between tasks matters for transfer learning and examined two different subsets (man-made v.s. natural objects). Azizpour et al. (2016) also examined different factors of model design such as depth, width, data diversity and density. They compared data similarity to ImageNet based on the task type: whether it was classification, attribute detection, fine-grained classification, compositional, or instance retrieval.

Pre-training on weakly labeled or noisy data was also found to be effective for transfer learning. Krause et al. (2016) obtained additional noisy training examples by searching the web with the class labels. We note that our method does not use the class labels to collect additional data. Mahajan et al. (2018) were able to attain impressive ImageNet performance by pre-training on 3 billion images from Instagram. Notably, they found that it was important to appropriately select hash-tags (used as weak labels) for source pre-training.

Understanding the similarity between datasets based on their content was studied by Cui et al. (2018) , who suggest using the Earth Mover's Distance (EMD) as a distance measure between datasets. They constructed two pre-training datasets by selecting subsets of ImageNet and iNaturalist, and showed that selecting an appropriate pre-training subset was important for good performance. Ge & Yu (2017) used features from filter bank responses to select nearest neighbor source training examples and demonstrated better performance compared to using the entire source dataset. Zamir et al. (2018) define a method to compute transferability between tasks on the same input; our work focuses on computing relationships between different input datasets.

In a comprehensive comparison, Kornblith et al. (2018) studied fine-tuning a variety of models on multiple datasets, and showed that performance on ImageNet correlated well with fine-tuning performance. Notably, they found that transfer learning with ImageNet was ineffective for small, fine-grained datasets.

Our approach is related to domain adaptation which assumes that the training and test set have differing distributions (Shimodaira, 2000) . We adopt similar ideas of importance weighting examples (Sugiyama et al., 2007; Saerens et al., 2002; Zhang et al., 2013) and adapt them to the pre-training step instead, showing that this is an effective approach.

In this work, we show that transfer learning to fine-grained datasets is sensitive to the choice of pre-training data, and demonstrate how to select pre-training data to significantly improve transfer learning performance. We build on the work of (Cui et al., 2018; Ge & Yu, 2017) , demonstrating the effectiveness of constructing pre-training datasets. Furthermore, we present a simple, scalable, and computationally-efficient way to construct pre-training datasets.

3. TRANSFER LEARNING SETUP:

We use the ANON1 (Anonymous) and ImageNet (Russakovsky et al., 2015) datasets as our source pre-training data and consider a range of target datasets for fine-tuning (Section 3.2). For each target dataset, we consider different strategies for selecting pre-training data, and compare the finetuned accuracy. We do not perform any label alignment between the source and target datasets. During fine-tuning, the classification layer in the network is trained from random initialization. The following sections describe the datasets and experiments in further detail.

3.1. SOURCE PRE-TRAINING DATA:

The ANON dataset has 300 million images and 18,291 classes. Each image can have multiple labels and on average, each image has 1.26 labels. The large number of labels include many fine-grained categories, for example, there are 1,165 different categories for animals. While the labels are noisy and often missing, we do not find this to a be a problem for transfer learning in practice. The labels form a semantic hierarchy: for example, the label 'mode of transport' includes the label 'vehicle', which in turn includes 'car'.

The semantic hierarchy of the labels suggests a straight-forward approach to constructing different subsets of ANON as source pre-training data. Given a label, we can select all of its child labels in the hierarchy to form a label set, with the corresponding set of training examples. We created 7 subsets of ANON across a range of labels2 (Table 1 ). However, creating subsets using the label hierarchy can be limiting for several reasons: (a) the number of examples per label are pre-defined by the ANON dataset; (b) not all child labels may be relevant; (c) a union over different sub-trees of the hierarchy may be desired; and (d) not all source datasets have richly-defined label hierarchies. In section 3.3, we discuss a domain adaptive transfer learning approach that automatically selects and weights the relevant pre-training data.

3.2. TARGET TRAINING DATASET:

We evaluate the performance of transfer learning on a range of classification datasets ( computed empirically over a source dataset D s . L(f θ (x), y) is often the cross entropy loss between the predictions of the model f θ (x) and the ground-truth labels y. However, the distribution of source pre-training dataset D s may differ from the target dataset D t . This could be detrimental as the model may emphasize features which are not relevant to the target dataset. We will mitigate this by upweighting the examples that are most relevant to the target dataset. This is closely related 4 to prior probability shift (Saerens et al., 2002; Storkey, 2009 ) also known as target shift (Zhang et al., 2013) .

We start by considering optimizing the loss function over the target dataset, D t instead:

E x,y∼Dt L(f θ (x), y) = x,y P t (x, y)L(f θ (x), y)

where we use P s and P t to denote distributions over the source and target datasets respectively. We first reformulate the loss to include the source dataset D s : =

x,y P s (x, y) P t (x, y) P s (x, y) L(f θ (x), y) =

x,y P s (x, y) P t (y)P t (x|y) P s (y)P s (x|y) L(f θ (x), y)

Next, we make the assumption that P s (x|y) ≈ P t (x|y), that is the distribution of examples given a particular label in the source dataset is approximately the same as that of the target dataset. We find this assumption reasonable in practice: for example, the distribution of 'bulldog' images from a large natural image dataset can be expected to be similar to that of a smaller animal-only dataset.

This assumption also allows us to avoid having to directly model the data distribution P (x).

Cancelling out the terms, we obtain:

≈ x,y P s (x, y) P t (y) P s (y) L(f θ (x), y) = E x,y∼Ds P t (y) P s (y) L(f θ (x), y)

Intuitively, P t (y) describes the distribution of labels in the target dataset, and P t (y)/P s (y) reweights classes during source pre-training so that the class distribution statistics match P t (y). We refer to P t (y)/P s (y) as importance weights and call this approach of pre-training Domain Adaptive Transfer Learning.

For this approach to be applicable in practice, we need to relax the earlier assumption that the source and target datasets share the same label space. Our goal is to estimate P t (y)/P s (y) for each label in the source dataset. The challenge is that the source and target datasets have different sets of labels.

Our solution is to estimate both P t (y) and P s (y) for labels in the source domain. The denominator P s (y) is obtained by dividing the number of times a label appears by the total number of source dataset examples. To estimate P t (y), we use a classifier to compute the probabilities of labels from source dataset on examples from the target dataset.

Concretely, we first train an image classification model on the entire source dataset. Next, we feed only the images from the target dataset into this model to obtain a prediction for each target example.

The predictions are averaged across target examples, providing an estimate of P t (y), where y is specified over the source label space. We emphasize that this method does not use the target labels when computing importance weights.

Our approach is in contrast to Ge & Yu (2017) , which is computationally expensive as they compute a similarity metric between every pair of images in the source dataset and target dataset. It is also more adaptive than Cui et al. (2018) , which suggests selecting appropriate labels to pretrain on, without specifying a weight on each label.

4. EXPERIMENTS:

We used the Inception v3 (Szegedy et al., 2016) , and AmoebaNet-B (Real et al., 2018) models in our experiments.

For Inception v3 models, we pre-train from random initialization for 2,000,000 steps using Stochastic Gradient Descent (SGD) with Nesterov momentum. Each mini-batch contained 1,024 examples. The same weight regularization and learning rate parameters were used for all pre-trained models and were selected based on a separate hold-out dataset. We used a learning rate schedule that first starts with a linear ramp up for 20,000 steps, followed by cosine decay. AmoebaNet-B models followed a similar setup with pre-training from random initialization for 250,000 steps using SGD and Nesterov momentum. We used larger mini-batches of 2,048 examples to speed up training. The same weight regularization and learning rate parameters were used for all models, and matched the parameters that Real et al. (2018) used for ImageNet training. We chose to use AmoebaNet-B with settings (N=18, F=512), resulting in over 550 million parameters when trained on ImageNet, so as to evaluate our methods on a large model.

During fine-tuning, we used a randomly initialized classification layer in place of the pre-trained classification layer. Models were trained for 20,000 steps using SGD with momentum. Each minibatch contained 256 examples. The weight regularization and learning rate parameters were determined using a hold-out validation set. We used a similar learning rate schedule with a linear ramp for 2,000 steps, followed by cosine decay.

For domain adaptive transfer learning, we found that adding a smooth prior when computing P t (y) helped performance with ImageNet as a source pre-training data. Hence, we used a temperature5 of 2.0 when computing the softmax predictions for the computation of the importance weights.

4.1. PRE-TRAINING SETUP:

While it is possible to directly perform pre-training with importance weights, we found it challenging as the importance weights varied significantly. When pre-training on a large dataset, this means that it is possible to have batches of data that are skewed in their weights with many examples weighted lightly. This is also computationally inefficient as the examples with very small weights contribute little to the gradients during training.

Hence, we created pre-training datasets by sampling examples from the source dataset using the importance weights. We start by choosing a desired pre-training dataset size, often large. We then Domain adaptive transfer is effective. When pre-training with ANON and ImageNet, we find that the domain adaptive transfer models are better or competitive with manually selected labels from the hierarchy. For datasets that are composed of multiple categories such as CIFAR-10 which includes animals and vehicles, we find further improved results since the constructed dataset includes multiple different categories.

In Figure 1 , we observe that the distributions are much more concentrated with FGVC Aircraft and Stanford Cars: this arises from the fact that ImageNet has only coarse-grained labels for aircraft and cars. In effect, ImageNet captures less of the discriminative factors of variation that is captured in either FGVC Aircraft and Stanford Cars. Hence, we observe that transfer learning only improves the results slightly. The horizontal axis represents the top 100 ImageNet labels sorted by importance weight for each dataset; each dataset has a different order. The distributions vary widely between target datasets. FGVC Aircraft selects only a few labels that turn out to be coarse grained, whereas Oxford Pets selects a wider variety of fine-grained labels. This reflects the inherent bias in the ImageNet dataset.

4.3. COMPARING PRE-TRAINING SAMPLING MECHANISMS:

In section 4.1, we described a method to construct pre-training datasets from sampling the source dataset. This process also allows us to study the effect of different distributions. Rather than sampling with replacement, as we did earlier, we could also sample without replacement when constructing the pre-training dataset. When sampling without replacement, we deviate from the importance weights assigned, but gain more unique examples to train on. We compare these two methods of sampling: (a) sampling with replacement -'same distribution matcher', and (b) sampling without replacement -'elastic distribution matcher'. Details of the methods are elaborated in the appendix. We see that when dataset size increases, the performance of same distribution matcher increases and then saturates, while that of elastic distribution matcher drops after a peak. Notice that the elastic distribution matcher also has significantly more unique examples than same distribution matcher as the dataset size increases.

We find that the performance of the same distribution matcher increases, and then saturates. Conversely, the elastic distribution matcher performance first increases then decreases. Note that at the low end of the dataset sizes, both methods will generate similar datasets. Thus, the later decrease in performance from the elastic distribution matcher comes from diverging from the original desired distribution. This indicates that using the importance weights during pre-training is more important than having more unique examples to train on.

4.4. RESULTS ON LARGE MODELS:

We furthered studied our method on large models to understand if large models are better able to generalize because the increased capacity enables them to capture more factors of variation. We conducted the same experiments on AmoebaNet-B, with over 550 million parameters. We found that the general findings persisted with AmoebaNet-B: (a) using the entire ANON dataset was always worse compared to an appropriate subset and (b) our domain adaptive transfer method was better or competitive with the hand selected subsets.

Furthermore, we find that the large model was also able to narrow the performance gap between the more general subsets and specific subsets: for example, the performance on Birdsnap between ANON-Bird and ANON-Animal is smaller with AmoebaNet-B compared to Inception v3. We also observe better transfer learning between the transportation datasets compared to Inception v3.

Our results are state of the art compared to the best published results (Table 4 ). The performance of the AmoebaNet-B was also better in all cases than Inception v3, except for the FGVC Aircraft dataset. This is consistent with Kornblith et al. (2018) who also found that Inception v3 did slightly better than NasNet-A (Zoph et al., 2017) .

5. DISCUSSION:

Transfer learning appears most effective when the pre-trained model captures the discriminative factors of variation present in the target dataset. This is reflected in the significant overlap in the classes between ImageNet and other datasets such as Caltech101, CIFAR-10, etc. where transfer learning with ImageNet is successful. Our domain adaptive transfer method is also able to identify the relevant examples in the source pre-training dataset that capture these discriminative factors.

Conversely, the cases where transfer learning is less effective are when it fails to capture the discriminative factors. In the case of the "FGVC Aircraft" dataset (Maji et al., 2013) , the task is to discriminate between 100 classes over manufacturer and models of aircraft (e.g., Boeing 737-700). However, ImageNet only has coarse grained labels for aircraft (e.g., airliner, airship). In this case, ImageNet models tend to learn to "group" different makes of aircraft together rather than differentiate them. It turns out that the ANON dataset has fine-grained labels for aircraft and is thus able to demonstrate better transfer learning efficacy.

Our results using AmoebaNet-B show that even large models transfer better when pre-trained on a subset of classes, suggesting that they make capacity trade-offs between the fine-grained classes when training on the entire dataset. This finding posits new research directions for developing large models that do not make such a trade-off.

We have seen an increase in dataset sizes since ImageNet; for example, the YFCC100M dataset (Thomee et al., 2016) has 100M examples. We have also seen developments of more efficient methods to train deep neural networks. Recent benchmarks (Coleman et al., 2018) demonstrate that it is possible to train a ResNet-50 model in half an hour, under fifty US dollars. This combination of data and compute will enable more opportunities to employ better methods for transfer learning. For the elastic distribution matcher, we avoid selecting each example more than once. In order to keep the distribution as similar to the desired one, we consider a sequential approach: we start with the class with the highest importance weight, in this case A, and exhaust the 10 samples available. Next, we recursively consider sampling a dataset of the remaining desired examples (30 -10 = 20) from the rest of the classes. Thus, we obtain the following number of examples per class: (A : 10), (B : 12), (C : 8). In Table 5 , we show how the sampling distribution turns out to differ for the CIFAR-10 dataset when using ImageNet as source pre-training data.

6.2. UNDERSTANDING THE IMPORTANCE OF THE PRE-TRAINING DISTRIBUTION:

To further understand the importance of the distribution, we created 3 ANON subsets of the same size but with different distributions from top 4,000 matched labels on Oxford-IIIT Pets. The uniform distribution experiment tells us how important it is to select relevant images, and the reverse distribution experiment tells us the importance of choosing the weighted distribution that matches the target dataset.

We observed that their transfer performance aligns well with the degree that their distribution matches the distribution of target dataset (Table 6 ).

:

Dataset anonymized for ICLR submission.

The following parent-child relationships exists in the label hierarchy: bird ⊂ animal; car ⊂ vehicle ⊂ transport; aircraft ⊂ vehicle ⊂ transport. We note that Anonymous excluded classes with too few training examples during training, while we include all classes available.

We used a CNN-based duplicate detector and chose a conservative threshold for computing near-duplicates to err on the side of ensuring that duplicates were removed. We removed a total of

48k examples from ANON, corresponding to duplicates that were found in target datasets.4 Prior work on prior probability shift usually considered shifts between train and test set, while we instead consider differences between the pre-training and training datasets.

The logits are divided by the temperature before computing the softmax.

Figures and Tables:

Figure1: Distribution of importance weights for each target dataset when using ImageNet as a source pretraining dataset. The horizontal axis represents the top 100 ImageNet labels sorted by importance weight for each dataset; each dataset has a different order. The distributions vary widely between target datasets. FGVC Aircraft selects only a few labels that turn out to be coarse grained, whereas Oxford Pets selects a wider variety of fine-grained labels. This reflects the inherent bias in the ImageNet dataset.

Figure 2: Performance (top) and unique examples (bottom) of the same distribution matcher and elastic dis-tribution matcher at different sampled dataset sizes. We see that when dataset size increases, the performance of same distribution matcher increases and then saturates, while that of elastic distribution matcher drops after a peak. Notice that the elastic distribution matcher also has significantly more unique examples than same distribution matcher as the dataset size increases.

distribution matching methods in detail in this section. Let us start by assuming that we have a source dataset with 100 examples with three different classes: (A: 10 examples), (B: 40 examples), and (C: 50 examples). Next, consider a scenario where the target dataset has a predicted label distribution over the source label set such that (A: 50%), (B: 30%), and (C: 20%). From this we can examine how to construct a pre-training dataset, say of size 30 examples. With the same distribution matcher, we sample the examples at a rate proportional to the importance weight computed using the ratio of the two distributions. Hence, (A: 0.5/0.1 = 5), (B: 0.3/0.4 = (C: 0.2/0.5 = 0.4). We then adjust this based on the desired pre-training dataset size (30/100 = 0.3). Thus, in expectation, this results in the following number of examples per class: A: (0.3 × 5 × 10 = 15), (B: 0.3 × 0.75 × 40 = 9), (C: 0.3 × 0.4 × 50 = 6).

ANON subsets by hand-selecting labels. Top Ancestor Label # Examples # Classes

Table content: <table><tr><td>Entire Dataset</td><td>300M</td><td>18,291</td></tr><tr><td>Animal</td><td>33.5M</td><td>2,992</td></tr><tr><td>Bird</td><td>5.4M</td><td>403</td></tr><tr><td>Car</td><td>27.9M</td><td>2,959</td></tr><tr><td>Aircraft</td><td>3.1M</td><td>418</td></tr><tr><td>Vehicle</td><td>43.7M</td><td>3,969</td></tr><tr><td>Transport</td><td>45.0M</td><td>3,987</td></tr><tr><td>Food</td><td>18.4M</td><td>3,532</td></tr></table>

), we ensured that the source pre-training data did not contain any of the target training data by removing all near-duplicates of the target training and test data from the ANON dataset 3 .

Table content: <table/>

Target datasets for fine-tuning.

Table content: <table><tr><td>Target Dataset</td><td colspan="3"># Training Examples # Test Examples # Classes</td></tr><tr><td>CIFAR-10 (Krizhevsky &amp; Hinton, 2009)</td><td>50,000</td><td>10,000</td><td>10</td></tr><tr><td>Birdsnap (Berg et al., 2014)</td><td>47,386</td><td>2,443</td><td>78</td></tr><tr><td>Stanford Cars (Krause et al., 2013)</td><td>8,144</td><td>8,041</td><td>196</td></tr><tr><td>FGVC Aircraft (Maji et al., 2013)</td><td>6,667</td><td>3,333</td><td>100</td></tr><tr><td>Oxford-IIIT Pets (Parkhi et al., 2012)</td><td>3,680</td><td>3,369</td><td>37</td></tr><tr><td>Food-101 (Bossard et al., 2014)</td><td>75,750</td><td>25,250</td><td>101</td></tr><tr><td colspan="3">3.3 DOMAIN ADAPTIVE TRANSFER LEARNING BY IMPORTANCE WEIGHTING</td><td/></tr></table>

Transfer learning results with Inception v3. Each row corresponds to a pre-training method. Adaptive transfer refers to our proposed method described in section 3.3. Each column corresponds to one target dataset. Results reported are top-1 accuracy for all datasets except Oxford-IIIT Pets, where we report mean accuracy per class. All results are averaged over 5 fine-tuning runs. Adaptive transfer is better or competitive with the hand selected subsets.More data is not necessarily better. Remarkably, more data during pre-training can hurt transfer learning performance. In all cases, the model pre-trained on the entire ANON dataset did worse than models trained on more specific subsets. These results are surprising as common wisdom suggests that more pre-training data should improve transfer learning performance if generic features are learned. Instead, we find that it is important to determine how relevant additional data is.The ImageNet results with Domain Adaptive Transfer further emphasize this point. For ImageNet with Adaptive Transfer, each pre-training dataset only has around 450k unique examples. While this is less than half of the full ImageNet dataset of 1.2 million examples, the transfer learning results are slightly better than using the full ImageNet dataset for many of the target datasets.

Table content: <table><tr><td>Target Dataset</td></tr></table>

Transfer learning results with AmoebaNet-B. Wei et al. (2018) b Kornblith et al. (2018) c Yu et al. d Cui et al. (2018) e Cubuk et al. (2018) f Krause et al. (2016) achieve 83.9% on Birdsnap and 94.5% on FGVC Aircraft by adding additional bird and aircraft images during training of the source and target datasets; images were collected from Google image search using class names from the target datasets.

Table content: <table><tr><td>Target Fine-tuned Dataset</td></tr></table>

Peer review (iclr2019):
 This paper is of limited technical contribution. The proposed method in Section 3.3 is just too close to covariate shift where importance weighting has been widely used. I don’t know whether authors are aware of these works. The findings listed in Section 1.1 are obvious and intuitive. There is no interesting finding. It is better to move the introduction of the datasets to the experimental section. A typo: hand-curated -> handcrafted