Year: 2020

Abstract:
Transferring knowledge from large source datasets is an effective way to fine-tune the deep neural networks of the target task with a small sample size. A great number of algorithms have been proposed to facilitate deep transfer learning, and these techniques could be generally categorized into two groups – Regularized Learning of the target task using models that have been pre-trained from source datasets, and Multitask Learning with both source and target datasets to train a shared backbone neural network. In this work, we aim to improve the multitask paradigm for deep transfer learning via Cross-domain Mixup (XMixup). While the existing multitask learning algorithms need to run backpropagation over both the source and target datasets and usually consume a higher gradient complexity, XMixup transfers the knowledge from source to target tasks more efficiently: for every class of the target task, XMixup selects the auxiliary samples from the source dataset and augments training samples via the simple mixup strategy. We evaluate XMixup over six real world transfer learning datasets. Experiment results show that XMixup improves the accuracy by 1.9% on average. Compared with other state-of-the-art transfer learning approaches, XMixup costs much less training time while still obtains higher accuracy.

heading: 1 INTRODUCTION
text: Performance of deep learning algorithms in real-world applications is often limited by the size of training datasets. Training a deep neural network (DNN) model with a small number of training samples usually leads to the over-fitting issue with poor generalization performance. A common yet effective solution is to train DNN models under transfer learning Pan et al. (2010) settings using large source datasets. The knowledge transfer from the source domain helps DNNs learn better features and acquire higher generalization performance for the pattern recognition in the target domain Donahue et al. (2014); Yim et al. (2017).
Backgrounds. For example, the paradigm Donahue et al. (2014) proposes to first train a DNN model using the large (and possibly irrelevant) source dataset (e.g. ImageNet), then uses the weights of the pre-trained model as the starting point of optimization and fine-tunes the model using the target dataset. In this way, blessed by the power of large source datasets, the fine-tuned model is usually capable of handling the target task with better generalization performance. Furthermore, authors in Yim et al. (2017); Li et al. (2018; 2019) propose transfer learning algorithms that regularize the training procedure using the pre-trained models, so as to constrain the divergence of the weights and feature maps between the pre-trained and fine-tuned DNN models. Later, the work Chen et al. (2019); Wan et al. (2019) introduces new algorithms that prevent the regularization from the hurts to transfer learning, where Chen et al. (2019) proposes to truncate the tail spectrum of the batch of gradients while Wan et al. (2019) proposes to truncate the ill-posed direction of the aggregated gradients.
In addition to the aforementioned strategies, a great number of methods have been proposed to transfer knowledge from the multi-task learning perspectives, such as Ge & Yu (2017b); Cui et al. (2018). More specifically, Seq-Train Cui et al. (2018) proposes a two phase approach, where the algorithm first picks up auxiliary samples from the source datasets with respect to the target task, then pre-train a model with the auxiliary samples and fine-tune the model using the target dataset. Moreover, Co-Train Ge & Yu (2017b) adopts a multi-task co-training approach to simultaneously train a shared backbone network using both source and target datasets and their corresponding separate
Fully-Connected (FC) layers. While all above algorithms enable knowledge transfer from source datasets to target tasks, they unfortunately perform poorly, sometimes, due to the critical technical issues as follows.
• Catastrophic Forgetting and Negative Transfer. Most transfer learning algorithms Donahue et al. (2014); Yim et al. (2017); Li et al. (2018; 2019) consist of two steps – pre-training and fine-tuning. Given the features that have been learned in the pre-trained models, either forgetting some good features during the fine-tuning process (catastrophic forgetting) Chen et al. (2019) or preserving the inappropriate features/filters to reject the knowledge from the target domain (negative transfer) Li et al. (2019); Wan et al. (2019) would hurt the performance of transfer learning. In this way, there might need a way to make compromises between the features learned from both source/target domains during the fine-tuning process, where multi-task learning with Seq-Train Cui et al. (2018) and Co-Train Ge & Yu (2017b) might suggest feasible solutions to well-balance the knowledge learned from the source/target domains, through fine-tuning the model with a selected set of auxiliary samples (rather than the whole source dataset) Cui et al. (2018) or alternatively learning the features from both domains during fine-tuning Ge & Yu (2017b). • Gradient Complexity for Seq-Train and Co-Train. The deep transfer learning algorithms
based on multi-task learning are ineffective. Though the pre-trained models based on some key datasets, such as ImageNet, are ubiquitously available for free, multi-tasking algorithms usually need additional steps for knowledge transfer. Prior to the fine-tuning procedure based on the target dataset, Seq-Train requires an additional step to select auxiliary samples and “mid-tunes” the pre-trained model using the selected auxiliary samples Cui et al. (2018). Furthermore, Co-Train Ge & Yu (2017b) requests additional cost for backpropagation in-situ as the two dataset combined. In this way, there might need a deep transfer learning algorithm that does not require explicit “mid-tuning” procedure or additional backpropagation to learn from the source dataset.
Our Work. With both technical issues in mind, we aim to study efficient and effective deep transfer learning algorithms with low computational complexity from the multi-task learning perspectives. We propose XMixup, namely Cross-domain Mixup, which is a novel deep transfer learning algorithm enabling knowledge transfer from source to target domains through the low-cost Mixup Zhang et al. (2018b). More specifically, given the source and target datasets for image classification tasks, XMixup runs deep transfer learning in two steps – (1) Auxiliary sample selection: XMixup pairs every class from the target dataset to a dedicated class in the source dataset, where the samples in the source class are considered as the auxiliary samples for the target class; then (2) Mixup with auxiliary samples and Fine-tuning: XMixup combines the samples from the paired classes of the two domains randomly using mixup strategy Zhang et al. (2018a), and performs fine-tuning process over the mixup data. To the best of our knowledge, this work has made three sets of contributions as follows.
1. We study the problems of cross-domain deep transfer learning for DNN classifiers from the multitask learning perspective, where the knowledge transfer from the source to the target tasks is considered as a co-training procedure of the shared DNN layers using the target dataset and auxiliary samples Ge & Yu (2017b); Cui et al. (2018). We review the existing solutions Donahue et al. (2014); Yim et al. (2017); Li et al. (2018; 2019), summarize the technical limitations of these algorithms, and particularly take care of the issues in catastrophic forgetting Chen et al. (2019), negative transfer Wan et al. (2019), and computational complexity.
2. In terms of methodologies, we extend the use of Mixup Zhang et al. (2018b) to the applications of cross-domain knowledge transfer, where both source and target datasets own different sets of classes and the aim of transfer learning is to adapt classes in the target domain. While vanilla mixup augments the training data with rich features and regularizes the stochastic training beyond empirical risk minimization (ERM), the proposed algorithm XMixup in this paper uses mixup to fuse the samples from source and target domains. In this way, the catastrophic forgetting issue could be solved in part, as the model keeps learning from both domains, but with lower cost compared to Chen et al. (2019). To control the effects of knowledge transfer, XMixup also offers a tuning parameter to make trade-off between the two domains in the mixup of samples Zhang et al. (2018b).
3. We carry out extensive experiments using a wide range of source and target datasets, and compare the results of XMixup with a number of baseline algorithms, including fine-tuning with weight decay (L2) Donahue et al. (2014), fine-tuning with L2-regularization on the starting point (L2-SP ) Li et al. (2018), Batch Singular Shrinkage (BSS) Chen et al. (2019), Seq-Train Cui et al. (2018), and Co-Train Ge & Yu (2017b). The experiment results showed that XMixup can outperform all these algorithms with significant improvement in both efficiency and effectiveness.
Organizations of the Paper The rest of this paper is organized as follows. In Section 2, we review the relations between our work to the existing algorithms, where the most relevant studies are discussed. We later present the algorithm design in Section 3, and the experiments with overall comparison results in Section 4, respectively. We discuss the details about the algorithm with case studies and ablation studies in Section 5, then conclude the paper in Section 6.

heading: 2 RELATED WORK
text: The most relevant studies to our algorithm are Donahue et al. (2014); Chen et al. (2019); Cui et al. (2018); Ge & Yu (2017b); Zhang et al. (2018b); Xu et al. (2020). All these algorithms, as well as the proposed XMixup algorithm, start the transfer learning from a pre-trained model, which has been well-trained using the source dataset. However, XMixup makes unique technical contributions in comparisons to these works.
Compared to Donahue et al. (2014), which fine-tunes the pre-trained model using the target set only and might cause the so-called catastrophic forgetting effects, XMixup proposes to fine-tune the pre-trained model using the mixup data from both domains. Compared Chen et al. (2019), which uses the computationally expensive singular value decomposition (SVD) on the batch gradients to avoid catastrophic forgetting and negative transfer effects, XMixup employs the low-cost mixup strategies to achieve similar goals. Compared to Cui et al. (2018), the proposed algorithm XMixup also adopts a similar procedure (pairing the classes in source/target domains) to pick up auxiliary samples from the source domain for knowledge transfer. XMixup however further mixes up the target training set with auxiliary samples and fine-tunes the pre-trained model with the data in an end-to-end manner, rather than using a two-step approach for fine-tuning Cui et al. (2018). Compared to Ge & Yu (2017b), which combines source/target tasks together to fine-tune the shared DNN backbones, the proposed algorithm here mixes up data from the two domains and boosts the performance through a simple fine-tuning process over the mixup data with low computational cost.
Finally, we extend the usage of vanilla mixup strategy Zhang et al. (2018b) for the applications of transfer learning, where in terms of methodologies we propose to pair classes of the two domains and perform mixup over the selected auxiliary samples for improved performance. Actually, mixup strategies have been used in Xu et al. (2020) for unsupervised domain adaption. Since the target task is assumed to share the same set of classes as the source domain in Xu et al. (2020), selecting auxiliary samples or pairing the source classes to fit the classes in the target domain is not required.

heading: 3 XMIXUP: CROSS-DOMAIN MIXUP FOR DEEP TRANSFER LEARNING
text: Given the source and target datasets and a pre-trained model (that has been well-trained using the source dataset), XMixup performs deep transfer learning using two steps as follows.
Auxiliary Sample Selection Given a source dataset S with m classes and a target training dataset T with n classes, XMixup assumes the source domain is usually with more classes than the target one (i.e., m > n), and it intends to pair every class in the target training dataset with a unique and dedicated class in the source dataset (one-to-one pairing from the target to source classes). More specifically, given a pre-trained model, XMixup first passes every sample from the two datasets through the pre-trained model and obtains features extracted from the last layer of the feature extractor. Then, XMixup groups the features of the samples according to the ground truth classes in their datasets, and estimates the centroid of the features for every class in both datasets. Such that, for every class c in the source or target dataset, XMixup represents the class as the centroid of the
features using the pre-trained model Θpretrain for every sample xi in the class c, i.e.,
centroid(c) = 1 |c| ∑ ∀xi∈c Φ(xi,Θpretrain), for c ∈ S or c ∈ T. (1)
Given two classes cs and ct in the source and target domains respectively, we consider the similarity between the two classes as the potentials for knowledge transfer, while XMixup measures the similarity between the two classes using the cosine measures between the centroids of the two classes, such that dist(cs, ct) = cosine < centroid(cs), centroid(ct) >. In this way, the auxiliary sample selection could be reduced to search the optimal transport between the sets of classes of S and T respectively, via the pre-defined distance measure. Hereby XMixup intends to find a one-to-one mapping P∗ : T → S, such that
P∗ ← argmin ∀P⊂(S×T )∩O2O ∑ ∀ct∈T dist(ct, P (ct)), (2)
where S × T refers to the Cartesian product of the target and source class sets, O2O refers to the constraint of the one-to-one mapping, P(ct) maps the target class to a unique class from the source domain. Note that P∗ refers to the optimal mapping that potentially exists to minimize the overall distances, while XMixup solves the optimization problem using a simple Greedy search Cui et al. (2018) to pursue a robust solution denoted as Pgreedy in low complexity. Compared to XMixup, the Seq-Train algorithm Cui et al. (2018) uses Greedy algorithm to pair the source/target classes via the measure Earth Mover’s Distance (EMD), which might be inappropriate in our settings of transfer learning.
Cross-domain Mixup with Auxiliary Samples and Fine-tuning Given the one-to-one pairing Pgreedy from target to source classes, XMixup carries out the fine-tuning process over the two datasets. In every iteration of fine-tuning, XMixup first picks up a mini-batch of training samples B drawn from the target dataset T ; then for every sample xt in the batch B, the algorithm retrieves the class of xt as xt.class and randomly draws one sample xs from the paired class of xt.class, such that
xs i.i.d∼ Pgreedy(xt.class), ∀xi ∈ B. (3)
We consider xs as an auxiliary sample of xt in the current iteration of fine-tuning. XMixup then mixes up the two samples as well as their labels through linear combination with a trade-off parameter λ drawn from the Beta distribution Beta(α, β), such that
x = λxt + (1− λ)xs, y = λyt + (1− λ)ys, and λ i.i.d∼ Beta(α, β). (4)
In this way, XMixup augments the original training sample (xt, yt) from the target domain using the auxiliary sample (xs, ys) from the paired source class, for knowledge transfer purposes. XMixup fine-tunes the pre-trained model Θpretrain using the mixup samples accordingly.

heading: 4 EXPERIMENTS AND OVERALL COMPARISONS

heading: 4.1 DATASETS
text: Stanford Dogs. The Stanford Dogs Khosla et al. (2011) dataset contains images of 120 breeds of dogs worldwide, each of which containing 100 examples for training and 72 for testing.
CUB-200-2011. Caltech-UCSD Birds-200-2011 Wah et al. (2011) consists of 11,788 images of 200 bird species. Each species is associated with a Wikipedia article.
Food-101. Food-101 Bossard et al. (2014) is a large scale data set consisting of more than 100k food images divided into 101 different kinds.
Flower-102. Flower-102 Nilsback & Zisserman (2008) consists of 102 flower categories. 1,020 images are used for training and 6149 images for testing. Only 10 samples are provided for each category during training.
Stanford Cars. The Stanford Cars Krause et al. (2013) dataset contains 16,185 images of 196 classes of cars. The data is split into 8,144 training and 8,041 testing images.
FGVC-Aircraft. FGVC-Aircraft Maji et al. (2013) is a fine-grained visual classification dataset composing more than 10,000 images of aircraft across 102 different aircraft models.

heading: 4.2 TRAINING DETAILS
text: We evaluate the recent state-of-the-art transfer learning methods in addition to XMixup. They are divided into two categories, which are regularized learning and multitask learning. For the former, we evaluate fine-tuning with L2 regularization, L2 − SP regularization Li et al. (2018) and BSS regularization Chen et al. (2019). Note that the multitask learning is defined in a broad sense here, including traditional co-training Ge & Yu (2017a), sequential training Cui et al. (2018) and co-training with XMixup. Compared with regularized learning, the essential difference is to re-train labeled auxiliary examples from the source dataset instead of regularizing the source model. We use the strategy described in Section 3 to select auxiliary examples for all multitask learning experiments. According to the empirical study in Ge & Yu (2017a), a threshold value of the auxiliary dataset size is required to guarantee the effect. In our implementation, we repeat the one-to-one pairing procedure until the size of the selected subset reaches a specific threshold. The value is 100,000 for Stanford Dogs and 200,000 for other datasets.
All experiments are performed using modern deep neural architecture ResNet-50 He et al. (2016), pre-trained with ImageNet. Input images are resized with the shorter edge being 256 and center cropped to square. We perform standard data augmentation composed of random flip and random cropped to 224× 224. For optimization, we use SGD with the momentum of 0.9 and batch size of 48. We train 9,000 iterations for all datasets. The initial learning rate is set to 0.001 for Stanford Dogs and 0.01 for remaining datasets. It is divided by 10 after 6,000 iterations. Each experiment is repeated five times. The average top-1 classification accuracy and standard division are reported.

heading: 4.3 RESULTS
text: Accuracy. The top-1 classification accuracies are reported in Table 1. We observe that regularized learning methods obtain similar results, while L2 − SP performs obviously better on Stanford Dogs. BSS achieves similar improvements on average but is more stable, consistently outperforming L2
by a small margin on most datasets. As for multitask learning, pre-training with auxiliary examples before fine-tuning often hurts the performance surprisingly. This may be caused by the phenomenon of catastrophic forgetting Li & Hoiem (2017), that pre-training with a subset of the source dataset loses general knowledge stored in the source model, but this general knowledge is probably useful for the target task. Co-training with auxiliary examples performs much better because auxiliary examples from the source task are used to preserve the knowledge. However, Co-training still hurts the performance on some datasets which have larger distance of data distribution to the source dataset such as Stanford Cars and FGVC-Aircraft. XMixup obtains obvious higher average accuracy than all baseline methods. It is also very robust and stably outperforms naive fine-tuning with L2 regularization.
Complexity. As stated in Li et al. (2018); Chen et al. (2019), regularized learning methods are usually efficient because the computational complexity involved by the regularization is approximately proportional to the number of parameters or features. While previous multitask learning methods Ge & Yu (2017a); Cui et al. (2018) are time consuming to deal with auxiliary examples, costing additional time whose scale is at least the same as training only target examples. XMixup achieves the efficiency almost the same as naive fine-tuning, because extra computations for data mixing are negligible.

heading: 5 DISCUSSIONS AND ABLATION STUDIES
text: In this section, we provide more empirical studies to analyze the effectiveness and applicability of our algorithm. In subsection 5.1, we show that XMixup effectively alleviates catastrophic forgetting and negative transfer. In subsection 5.2 and 5.3, XMixup is proved to be not sensitive with the auxiliary dataset selection and hyperparameter setting, indicating that XMixup is easy to be applied in various real world tasks. Finally in subsection 5.4, we analyze two essential characteristics, crossing domain and supervised mixing, showing that they are both necessary.

heading: 5.1 ANALYSIS OF THE EFFECTIVENESS
text: Authors in Chen et al. (2019) figure out that deep transfer learning suffers from two kinds of problems, which are catastrophic forgetting and negative transfer. We do empirical analysis showing that XMixup explicitly or implicitly deals with both issues. Three different datasets are used for detailed analysis.
Catastrophic Forgetting. A widely adopted measurement about catastrophic forgetting is to evaluate the accuracy of a fine-tuned model on the previous task Li & Hoiem (2017); Kirkpatrick et al.

heading: 103 104 105 106
text: (2017). It is worth noting that although parameters are totally changed after fine-tuning, the general knowledge is still preserved for feature representation. We thus can use models fine-tuned on target tasks as feature extractors for images in source domains. A randomly initialized fully connected layer is trained to adapt to each specified source task. Results in Table 2 show that XMixup helps preserve the knowledge about auxiliary samples, although this may hurt the capacity of representing samples not in the auxiliary dataset.
Negative Transfer. Chen et al. (2019) finds that the distribution of tail singular values indicates the degree of negative transfer. Specifically, negative transfer can be reduced by suspending tail singular values. In Figure 2, we observe that XMixup shows similar trends with increasing number of training examples, which is the most meaningful approach to avoid negative transfer. Through sufficient utilization of auxiliary examples, XMixup further decreases smallest singular values.

heading: 5.2 SENSITIVITY OF THE AUXILIARY DATASET
text: Since the only external dependence is the auxiliary dataset, we analyze in detail how characteristics of the auxiliary dataset influences the effect of XMixup.
Size. We first discuss how XMixup performs as the size of the auxiliary dataset varies. We create auxiliary datasets with different scales based on the default dataset described in experiment settings. To increase the size, we continue adding the most similar category from the source dataset until the whole set is selected. To decrease the size, we perform random sampling from the default dataset. Note that these smaller auxiliary datasets are only different on the size but not the domain similarity, while larger auxiliary datasets are less similar with the target domain. As illustrated in Figure 3, CUB-200-2011 and FGVC-Aircraft show very good robustness to the size of the auxiliary dataset. Specifically, XMixup performs well enough when the number of auxiliary examples is more than 100,000. Surprisingly, we find that XMixup still significantly outperforms naive fine-tuning even simply using the whole source dataset without selection. The task of Stanford Dogs is a bit different that using too many dissimilar auxiliary examples hurts the performance, since it is closely related with a subset of ImageNet. However, all tasks can benefit from XMixup obviously with a wide range of the number of auxiliary examples starting from about 30,000.
Domain Similarity. In order to investigate how XMixup depends on the similarity of auxiliary examples, we keep the size of the auxiliary dataset the same, but only replace auxiliary examples by random sampling from the entire source dataset. The result is presented in Table 3. We observe that most datasets are not affected obviously by the similarity removing, indicating that XMixup is a robust approach to integrate the general knowledge during fine-tuning. In subsection 5.4, we further show through ablation study that, knowledge from the source domain plays an important role in XMixup. In other words, crossing domain mixing is more than a kind of perturbation on target examples.

heading: 5.3 SENSITIVITY OF HYPERPARAMETERS
text: There are two parameters α and β in Beta distribution. We use fixed β of 1 and only change α to control the mixing weight of the two domains. Larger α means larger sampling weights for examples from the target domain. We explore a broad range of values for α and investigate how well XMixup performs under different settings. We illustrate Beta distribution and the influence of varying α in Fig. 4 and Fig. 5 in Appendix.
General trends. We observe that, in all experiments conducted, XMixup demonstrates a relatively continuous and smooth change as α varies. Top-1 accuracy tends to be lower when α is small because the training assigns a too small weight to samples in the target dataset. It then rises to its maximum as α increases, followed by a gradual drop when α continues to increase as using a too large value for α degenerates to naive fine-tuning.
Suggested range for α depending on datasets. Our experimental results show that, for datasets that are very similar to the source datasets, such as Stanford Dogs which is a subset of ImageNet, a lower α (2−3 ∼ 2−1) usually performs better. Otherwise, α in a wide range between 20 and 26 is generally safe to improve the naive fine-tuning and even state-of-the-art baseline methods.

heading: 5.4 ABLATION STUDY
text: To further determine whether both components, namely crossing domain and supervised mixing, are essential for XMixup, we evaluate the two variants. First we remove the characteristic of crossing domain, implementing traditional Mixup Zhang et al. (2018b) using only samples in the target dataset (in-domain). Second, we keep crossing domain sample mixing in the data generation step but remove labels of auxiliary samples .
As reported in Table 4, Mixup has lower test accuracy than XMixup by an average of 2.7% in the three datasets we experimented. This consistent difference implies that crossing domain is an essential factor that contributes to the effectiveness of XMixup. Furthermore, Mixup has similar, or even slightly worse, performance compared to L2 models, whose results are shown in Table. 1. This phenomenon suggests that general data augmentation may not work well under transfer learning scenario where sample size is limited. Table 4 also shows that XMixup is more than data augmentation. We observe that model performance drops significantly (for an average of 5.2%) if labels of auxiliary examples are removed. This result indicates that knowledge of the source dataset is essential.

heading: 6 CONCLUSIONS
text: In this paper, we study the problem of knowledge transfer for DNN models from the perspectives of multi-task learning. We propose a novel deep transfer learning algorithm XMixup, namely Crossdomain Mixup, with superiority in both effectiveness and efficiency. Through a two-step approach with (1) auxiliary sample selection and (2) cross-domain mixup and fine-tuning, XMixup achieves significant performance improvements with 1.9% higher classification accuracy on average, when compared to the state-of-the-art algorithms Donahue et al. (2014); Li et al. (2018); Chen et al. (2019). XMixup is also robust to hyperparameter choices and ways of auxiliary sample selection. Finally, we conclude that knowledge transfer through multitask learning with a set of selected auxiliary samples is no doubt a promising direction with huge potentials, while this work suggests a solid yet easy-to-use baseline method.

heading: A APPENDIX
text: A.1 BETA DISTRIBUTION
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
alpha=0.1, beta=1
0.0
0.2
0.4
0.6
0.8
1.0
A.2 INFLUENCE OF THE MIXING WEIGHT

Review summaries:
This paper proposes XMixup, a strategy for improving transfer learning in neural networks. Specifically, XMixup consists of mixup applied between target samples and source samples from the class pre-determined to be closest to target sample’s class. Experiments conducting transfer learning from pre-trained ImageNet to 6 smaller image classification datasets demonstrate XMixup to outperform the baseline approaches.

This paper proposes a simple variant for the mixup training mechanism for transfer learning problems: cross-domain mixup (XMixup). The key idea is to mix up the training samples from both domains where the samples are generated by nearest-center assignment in each class. Experiments on several datasets have shown its effectiveness in transfer learning compared to some SOTA methods.