XMixup: Efficient Transfer Learning with Auxiliary Samples by Cross-Domain Mixup

1 INTRODUCTION . Performance of deep learning algorithms in real-world applications is often limited by the size of training datasets . Training a deep neural network ( DNN ) model with a small number of training samples usually leads to the over-fitting issue with poor generalization performance . A common yet effective solution is to train DNN models under transfer learning Pan et al . ( 2010 ) settings using large source datasets . The knowledge transfer from the source domain helps DNNs learn better features and acquire higher generalization performance for the pattern recognition in the target domain Donahue et al . ( 2014 ) ; Yim et al . ( 2017 ) . Backgrounds . For example , the paradigm Donahue et al . ( 2014 ) proposes to first train a DNN model using the large ( and possibly irrelevant ) source dataset ( e.g . ImageNet ) , then uses the weights of the pre-trained model as the starting point of optimization and fine-tunes the model using the target dataset . In this way , blessed by the power of large source datasets , the fine-tuned model is usually capable of handling the target task with better generalization performance . Furthermore , authors in Yim et al . ( 2017 ) ; Li et al . ( 2018 ; 2019 ) propose transfer learning algorithms that regularize the training procedure using the pre-trained models , so as to constrain the divergence of the weights and feature maps between the pre-trained and fine-tuned DNN models . Later , the work Chen et al . ( 2019 ) ; Wan et al . ( 2019 ) introduces new algorithms that prevent the regularization from the hurts to transfer learning , where Chen et al . ( 2019 ) proposes to truncate the tail spectrum of the batch of gradients while Wan et al . ( 2019 ) proposes to truncate the ill-posed direction of the aggregated gradients . In addition to the aforementioned strategies , a great number of methods have been proposed to transfer knowledge from the multi-task learning perspectives , such as Ge & Yu ( 2017b ) ; Cui et al . ( 2018 ) . More specifically , Seq-Train Cui et al . ( 2018 ) proposes a two phase approach , where the algorithm first picks up auxiliary samples from the source datasets with respect to the target task , then pre-train a model with the auxiliary samples and fine-tune the model using the target dataset . Moreover , Co-Train Ge & Yu ( 2017b ) adopts a multi-task co-training approach to simultaneously train a shared backbone network using both source and target datasets and their corresponding separate Fully-Connected ( FC ) layers . While all above algorithms enable knowledge transfer from source datasets to target tasks , they unfortunately perform poorly , sometimes , due to the critical technical issues as follows . • Catastrophic Forgetting and Negative Transfer . Most transfer learning algorithms Donahue et al . ( 2014 ) ; Yim et al . ( 2017 ) ; Li et al . ( 2018 ; 2019 ) consist of two steps – pre-training and fine-tuning . Given the features that have been learned in the pre-trained models , either forgetting some good features during the fine-tuning process ( catastrophic forgetting ) Chen et al . ( 2019 ) or preserving the inappropriate features/filters to reject the knowledge from the target domain ( negative transfer ) Li et al . ( 2019 ) ; Wan et al . ( 2019 ) would hurt the performance of transfer learning . In this way , there might need a way to make compromises between the features learned from both source/target domains during the fine-tuning process , where multi-task learning with Seq-Train Cui et al . ( 2018 ) and Co-Train Ge & Yu ( 2017b ) might suggest feasible solutions to well-balance the knowledge learned from the source/target domains , through fine-tuning the model with a selected set of auxiliary samples ( rather than the whole source dataset ) Cui et al . ( 2018 ) or alternatively learning the features from both domains during fine-tuning Ge & Yu ( 2017b ) . • Gradient Complexity for Seq-Train and Co-Train . The deep transfer learning algorithms based on multi-task learning are ineffective . Though the pre-trained models based on some key datasets , such as ImageNet , are ubiquitously available for free , multi-tasking algorithms usually need additional steps for knowledge transfer . Prior to the fine-tuning procedure based on the target dataset , Seq-Train requires an additional step to select auxiliary samples and “ mid-tunes ” the pre-trained model using the selected auxiliary samples Cui et al . ( 2018 ) . Furthermore , Co-Train Ge & Yu ( 2017b ) requests additional cost for backpropagation in-situ as the two dataset combined . In this way , there might need a deep transfer learning algorithm that does not require explicit “ mid-tuning ” procedure or additional backpropagation to learn from the source dataset . Our Work . With both technical issues in mind , we aim to study efficient and effective deep transfer learning algorithms with low computational complexity from the multi-task learning perspectives . We propose XMixup , namely Cross-domain Mixup , which is a novel deep transfer learning algorithm enabling knowledge transfer from source to target domains through the low-cost Mixup Zhang et al . ( 2018b ) . More specifically , given the source and target datasets for image classification tasks , XMixup runs deep transfer learning in two steps – ( 1 ) Auxiliary sample selection : XMixup pairs every class from the target dataset to a dedicated class in the source dataset , where the samples in the source class are considered as the auxiliary samples for the target class ; then ( 2 ) Mixup with auxiliary samples and Fine-tuning : XMixup combines the samples from the paired classes of the two domains randomly using mixup strategy Zhang et al . ( 2018a ) , and performs fine-tuning process over the mixup data . To the best of our knowledge , this work has made three sets of contributions as follows . 1 . We study the problems of cross-domain deep transfer learning for DNN classifiers from the multitask learning perspective , where the knowledge transfer from the source to the target tasks is considered as a co-training procedure of the shared DNN layers using the target dataset and auxiliary samples Ge & Yu ( 2017b ) ; Cui et al . ( 2018 ) . We review the existing solutions Donahue et al . ( 2014 ) ; Yim et al . ( 2017 ) ; Li et al . ( 2018 ; 2019 ) , summarize the technical limitations of these algorithms , and particularly take care of the issues in catastrophic forgetting Chen et al . ( 2019 ) , negative transfer Wan et al . ( 2019 ) , and computational complexity . 2 . In terms of methodologies , we extend the use of Mixup Zhang et al . ( 2018b ) to the applications of cross-domain knowledge transfer , where both source and target datasets own different sets of classes and the aim of transfer learning is to adapt classes in the target domain . While vanilla mixup augments the training data with rich features and regularizes the stochastic training beyond empirical risk minimization ( ERM ) , the proposed algorithm XMixup in this paper uses mixup to fuse the samples from source and target domains . In this way , the catastrophic forgetting issue could be solved in part , as the model keeps learning from both domains , but with lower cost compared to Chen et al . ( 2019 ) . To control the effects of knowledge transfer , XMixup also offers a tuning parameter to make trade-off between the two domains in the mixup of samples Zhang et al . ( 2018b ) . 3 . We carry out extensive experiments using a wide range of source and target datasets , and compare the results of XMixup with a number of baseline algorithms , including fine-tuning with weight decay ( L2 ) Donahue et al . ( 2014 ) , fine-tuning with L2-regularization on the starting point ( L2-SP ) Li et al . ( 2018 ) , Batch Singular Shrinkage ( BSS ) Chen et al . ( 2019 ) , Seq-Train Cui et al . ( 2018 ) , and Co-Train Ge & Yu ( 2017b ) . The experiment results showed that XMixup can outperform all these algorithms with significant improvement in both efficiency and effectiveness . Organizations of the Paper The rest of this paper is organized as follows . In Section 2 , we review the relations between our work to the existing algorithms , where the most relevant studies are discussed . We later present the algorithm design in Section 3 , and the experiments with overall comparison results in Section 4 , respectively . We discuss the details about the algorithm with case studies and ablation studies in Section 5 , then conclude the paper in Section 6 . 2 RELATED WORK . The most relevant studies to our algorithm are Donahue et al . ( 2014 ) ; Chen et al . ( 2019 ) ; Cui et al . ( 2018 ) ; Ge & Yu ( 2017b ) ; Zhang et al . ( 2018b ) ; Xu et al . ( 2020 ) . All these algorithms , as well as the proposed XMixup algorithm , start the transfer learning from a pre-trained model , which has been well-trained using the source dataset . However , XMixup makes unique technical contributions in comparisons to these works . Compared to Donahue et al . ( 2014 ) , which fine-tunes the pre-trained model using the target set only and might cause the so-called catastrophic forgetting effects , XMixup proposes to fine-tune the pre-trained model using the mixup data from both domains . Compared Chen et al . ( 2019 ) , which uses the computationally expensive singular value decomposition ( SVD ) on the batch gradients to avoid catastrophic forgetting and negative transfer effects , XMixup employs the low-cost mixup strategies to achieve similar goals . Compared to Cui et al . ( 2018 ) , the proposed algorithm XMixup also adopts a similar procedure ( pairing the classes in source/target domains ) to pick up auxiliary samples from the source domain for knowledge transfer . XMixup however further mixes up the target training set with auxiliary samples and fine-tunes the pre-trained model with the data in an end-to-end manner , rather than using a two-step approach for fine-tuning Cui et al . ( 2018 ) . Compared to Ge & Yu ( 2017b ) , which combines source/target tasks together to fine-tune the shared DNN backbones , the proposed algorithm here mixes up data from the two domains and boosts the performance through a simple fine-tuning process over the mixup data with low computational cost . Finally , we extend the usage of vanilla mixup strategy Zhang et al . ( 2018b ) for the applications of transfer learning , where in terms of methodologies we propose to pair classes of the two domains and perform mixup over the selected auxiliary samples for improved performance . Actually , mixup strategies have been used in Xu et al . ( 2020 ) for unsupervised domain adaption . Since the target task is assumed to share the same set of classes as the source domain in Xu et al . ( 2020 ) , selecting auxiliary samples or pairing the source classes to fit the classes in the target domain is not required . 3 XMIXUP : CROSS-DOMAIN MIXUP FOR DEEP TRANSFER LEARNING . Given the source and target datasets and a pre-trained model ( that has been well-trained using the source dataset ) , XMixup performs deep transfer learning using two steps as follows . Auxiliary Sample Selection Given a source dataset S with m classes and a target training dataset T with n classes , XMixup assumes the source domain is usually with more classes than the target one ( i.e. , m > n ) , and it intends to pair every class in the target training dataset with a unique and dedicated class in the source dataset ( one-to-one pairing from the target to source classes ) . More specifically , given a pre-trained model , XMixup first passes every sample from the two datasets through the pre-trained model and obtains features extracted from the last layer of the feature extractor . Then , XMixup groups the features of the samples according to the ground truth classes in their datasets , and estimates the centroid of the features for every class in both datasets . Such that , for every class c in the source or target dataset , XMixup represents the class as the centroid of the features using the pre-trained model Θpretrain for every sample xi in the class c , i.e. , centroid ( c ) = 1 |c| ∑ ∀xi∈c Φ ( xi , Θpretrain ) , for c ∈ S or c ∈ T. ( 1 ) Given two classes cs and ct in the source and target domains respectively , we consider the similarity between the two classes as the potentials for knowledge transfer , while XMixup measures the similarity between the two classes using the cosine measures between the centroids of the two classes , such that dist ( cs , ct ) = cosine < centroid ( cs ) , centroid ( ct ) > . In this way , the auxiliary sample selection could be reduced to search the optimal transport between the sets of classes of S and T respectively , via the pre-defined distance measure . Hereby XMixup intends to find a one-to-one mapping P∗ : T → S , such that P∗ ← argmin ∀P⊂ ( S×T ) ∩O2O ∑ ∀ct∈T dist ( ct , P ( ct ) ) , ( 2 ) where S × T refers to the Cartesian product of the target and source class sets , O2O refers to the constraint of the one-to-one mapping , P ( ct ) maps the target class to a unique class from the source domain . Note that P∗ refers to the optimal mapping that potentially exists to minimize the overall distances , while XMixup solves the optimization problem using a simple Greedy search Cui et al . ( 2018 ) to pursue a robust solution denoted as Pgreedy in low complexity . Compared to XMixup , the Seq-Train algorithm Cui et al . ( 2018 ) uses Greedy algorithm to pair the source/target classes via the measure Earth Mover ’ s Distance ( EMD ) , which might be inappropriate in our settings of transfer learning . Cross-domain Mixup with Auxiliary Samples and Fine-tuning Given the one-to-one pairing Pgreedy from target to source classes , XMixup carries out the fine-tuning process over the two datasets . In every iteration of fine-tuning , XMixup first picks up a mini-batch of training samples B drawn from the target dataset T ; then for every sample xt in the batch B , the algorithm retrieves the class of xt as xt.class and randomly draws one sample xs from the paired class of xt.class , such that xs i.i.d∼ Pgreedy ( xt.class ) , ∀xi ∈ B . ( 3 ) We consider xs as an auxiliary sample of xt in the current iteration of fine-tuning . XMixup then mixes up the two samples as well as their labels through linear combination with a trade-off parameter λ drawn from the Beta distribution Beta ( α , β ) , such that x = λxt + ( 1− λ ) xs , y = λyt + ( 1− λ ) ys , and λ i.i.d∼ Beta ( α , β ) . ( 4 ) In this way , XMixup augments the original training sample ( xt , yt ) from the target domain using the auxiliary sample ( xs , ys ) from the paired source class , for knowledge transfer purposes . XMixup fine-tunes the pre-trained model Θpretrain using the mixup samples accordingly .

Summary: This paper proposes XMixup, a strategy for improving transfer learning in neural networks. Specifically, XMixup consists of mixup applied between target samples and source samples from the class pre-determined to be closest to target sample’s class. Experiments conducting transfer learning from pre-trained ImageNet to 6 smaller image classification datasets demonstrate XMixup to outperform the baseline approaches.