XMixup: Efficient Transfer Learning with Auxiliary Samples by Cross-Domain Mixup
Authors: Xingjian Li
Haoyi Xiong
Haozhe An
Cheng-zhong Xu
Dejing Dou
Venue:  (2021)
Keywords: transfer learning
deep learning

Abstract: Transferring knowledge from large source datasets is an effective way to fine-tune the deep neural networks of the target task with a small sample size. A great number of algorithms have been proposed to facilitate deep transfer learning, and these techniques could be generally categorized into two groups – Regularized Learning of the target task using models that have been pre-trained from source datasets, and Multitask Learning with both source and target datasets to train a shared backbone neural network. In this work, we aim to improve the multitask paradigm for deep transfer learning via Cross-domain Mixup (XMixup). While the existing multitask learning algorithms need to run backpropagation over both the source and target datasets and usually consume a higher gradient complexity, XMixup transfers the knowledge from source to target tasks more efficiently: for every class of the target task, XMixup selects the auxiliary samples from the source dataset and augments training samples via the simple mixup strategy. We evaluate XMixup over six real world transfer learning datasets. Experiment results show that XMixup improves the accuracy by 1.9% on average. Compared with other state-of-the-art transfer learning approaches, XMixup costs much less training time while still obtains higher accuracy.

Under review as a conference paper at ICLR 2021
XMIXUP: EFFICIENT TRANSFER LEARNING WITH AUXILIARY SAMPLES BY CROSS-DOMAIN MIXUP
Anonymous authors Paper under double-blind review
ABSTRACT
Transferring knowledge from large source datasets is an effective way to fine-tune the deep neural networks of the target task with small sample size. A great number of algorithms have been proposed to facilitate deep transfer learning, and these techniques could be generally categorized into two groups Regularized Learning of the target task using models that have been pre-trained from source datasets, and Multitask Learning with both source and target datasets to train a shared backbone neural network. In this work, we aim to improve the multitask paradigm for deep transfer learning via Cross-domain Mixup (XMixup). While the existing multitask learning algorithms need to run backpropagation over both the source and target datasets and usually consume a higher gradient complexity, XMíxup transfers the knowledge from source to target tasks more efficiently: for every class of the target task, XMixup selects the auxiliary samples from the source dataset and augments training samples via the simple mixup strategy. We evaluate XMixup over six real world transfer learning datasets. Experiment results show that XMixup improves the accuracy by 1.9% on average. Compared with other state-of- teefthe-att art transfer learning approaches, XMixup costs much less training time while still obtains higher accuracy.
INTRODUCTION
Performance of deep learning algorithms in real-world applications is often limited by the size of training datasets. Training deep neural network (DNN) model with a small number of training samples usually leads to the over-fitting issue with poor generalization performance. common yet effective solution is to train DNN models under transfer learning Pan et al. (2010) settings using large source datasets. The knowledge transfer from the source domain helps DNNs learn better features and acquire higher generalization performance for the pattern recognition in the target domain Donahue jet al. (0144) Yim et al. (2017).
Backgrounds. For example, the paradigm Donahue al. (2014) proposes first train DNN model using the large (and possibly irrelevant) source dataset (c.g. ImageNet), then uses the weights of the pre-trained model the starting point of optimization and fine-tunes the model using the target dataset. In this way, blessed by the power of large source datasets, the fine-tuned model is usually capable of handling the target task with better generalization performance. Furthermore, authors in Yim et al. (201)); Li et al. (2018) 2019) propose transfer learning algorithms that regularize the training procedure using the pre-trained models, so as to constrain the divergence of the weights and feature maps between the pre-trained and fine-tuned DNN models. Later, the work Chen et al. (2019); Wan et al.) (2019) introduces new algorithms that prevent the regularization from the hurts transfer learning, where Chen et al. (2019) proposes to truncate the tail spectrum of the batch of gradients while Wan et (2099) proposes to truncate the ill-posed direction of the aggregated gradients.
In addition to the aforementioned strategies, great number of methods have been proposed to transfer knowledge from the multi-task learning perspectives, such as Ge & Yu (001b)); Cui et al. (2018). More specifically, Seq-Train Cui et al. (2018) proposes two phase approach, where the algorithm first picks up auxiliary samples from the source datasets with respect to the target task, then pre-train model with the auxiliary samples and fine-tune the model using the target dataset. Moreover, Co- To-Trrin&e Ge & (2077)) adopts a multi-takk co-trat co-training approach to simultancously train shared backbone network using both source and target datasets and their corresponding separate

---

Under review as a conference paper at ICLR 2021
Fully-Com Cooneeted (FC) layers. While all above algorithms enable knowledge transfer from source datasets to target tasks, they unfortunately perform poorly, sometimes, due to the critical technical issues as follows.
Catastrophic Forgetting and Negative Transfer. Most transfer learning algorithms Don-
ahue et al. 2014);; Yim et al.] (017); Li et al. (2018) 2019) consist of two steps - -pre-training and fine-tuning. Given the features that have been learned in the pre-trained models, either forgetting some good features during the fine-tuning process (catastrophic forgetring) Chen
et al.] (2019) or preserving the inappropriate features/filters to reject the knowledge from the target domain (negative transfer) [i et al. (2019): Wan al. (2019) would hurt the performance of transfer learning. In this way, there might need a way to make compro- mises between the features learned from both source/target domains during the fine-tuning process, where multi-task learning with Seq-Train Cui et al. (2018) and Co-Train Ge & Yu (2017b) might suggest feasible solutions to well ballncalance the knowledge learned from the source/target domains, through fine-tuning the model with selected set of auxiliary samples (rather than the whole source dataset) Cui et al. (2018) or alternatively learning the features from both domains during fine-tuning Ge & Yu (201bb))
Gradient Complexity for Seq Train and Co-Train. The deep transfer learning algorithms based on multi-task learning are ineffective. Though the pre-trained models based on some key datasets, such as ImageNet, are ubiquitously available for free, multi-tasking algorithms usually need additional steps for knowledge transfer. Prior to the fine-tuning procedure based on the target dataset, Seq-Trar Beq-Train requires an additional step to select auxiliary samples and "mid "mid-tunes" the pre-trained model using the selected auxiliary samples Cui et al. (2018). Furthermore, Co- Train Ge & Yu (2017b) requests additional cost for backpropagation in-situ as the two dataset combined. In this way, there might need a deep transfer learning algorithm that does not require explicit "mid-tuning" procedure additional backpropagation to learn from the source dataset.
Our Work. With both technical issues in mind, we aim to study efficient and effective deep transfer learning algorithms with low computational complexity from the multi-task learning perspectives. We propose XMixup, namely Cross-domain Mixup, which is a novel deep transfer learning algorithm enabling knowledge transfer from source to target domains through the low-cost Mixup Zhang (et al. (2018b). More specifically, given the source and target datasets for image classification tasks, XMixup runs deep transfer learning in two steps - (1) Auxiliary sample selection: XMixup pairs every class from the target dataset to a dedicated class in the source dataset, where the samples in the source class are considered as the auxiliary samples for the target class; then (2) Mixup with auxiliary samples and Fine-tuning: XMixup combines the samples from the paired classes of the two domains randomly using mixup strategy Zhhng al. (2018a), and performs fine-tuning process over the mixup data. To the best of our knowledge, this work has made three sets of contributions as follows.
1. We study the problems of cross-domain deep transfer learning for DNN classifiers from the multitask learning perspective, where the knowledge transfer from the source to the target tasks is considered as co-training procedure of the shared DNN layers using the target dataset and auxiliary samples Ge & Yu (2017b); Cui et al. (2018). We review the existing solutions Donahue et al. (2014); Yim et al. (207)) Li et al. (2018) 2019). summarize the technical limitations of these al algorithms, and partieularly take care of the issues in catastrophic forgetting Chen et al.) 2011), negative transfer Wan et al. (211). and computational complexity.
2. In terms of methodologies, we extend the use of Mixup Zhang et al. (2018b) to the appli- cations of cross cross-domain knowledge transfer, where both source and target datasets own different sets of classes and the aim of transfer learning to adapt classes in the target domain. While vanilla mixup augments the training data with rich features and regularizes the stochastic training beyond empirical risk minimization (ERM), the proposed algorithm XMixup in this paper uses mixup to fuse the samples from source and target domains. In this way, the catastrophic forgetting issue could be solved in part, as the model keeps learning from both domains, but with lower cost compared to Chen et al. (2019). To control the effects of knowledge transfer, XMixup also offers a tuning parameter to make trade-off between the two domains in the mixup of samples /Zhang al. (0018).
2

---

Under review as a conference paper at ICLR 2021
3. We carry out extensive experiments using wide range of source and target datasets, and compare the results of XMixup with a number of baseline algorithms, including fine-tuning with weight decay (L3) Donahuc et al. (2014), fine-tuning with L'-regularization
the
starting point (I.2-P)) P) Li et al. (0188) Batch Singular Shrinkage (BSS) Chen et al. (2019). Seq-Train Cui et al. 2018), and Co-TTrain [Ge & Yu (2017b). The experiment results showed that XMixup can outperform all these algorithms with significant improvement in both efficiency and effectiveness.
Organizations of the Paper The rest of this paper is organized as follows. In Section 2, we review the relations between our work to the existing algorithms, where the most relevant studies are discussed. We later present the algorithm design in Section 3, and the experiments with overall comparison results in Section 4, respectively. We discuss the details about the algorithm with case studies and ablation studies in Section 5, then conclude the paper in Section 6.
2 RELATED WORK
The most relevant studies to our algorithm are Donahue et al. (2014); Chen et al. 2 Cui et al. (2018); Ge & Yu (017b); Zhang et al. (2018b); Xu et (2020). All these algorithms, well as the proposed XMixup algorithm, start the transfer learning from a pre-trained model, which has been well-trained using the source dataset. However, XMixup makes unique technical contributions in comparisons to these works.
Compared Donahue et al. 2014), which fine-tunes the pre-trained model using the target set only and might cause the so-called catastrophic forgetting effects, XMixup proposes to fine-tune the pre-trained model using the mixup data from both domains. Compared Chen et al. (2019), which uses the computationally expensive singular value decomposition (SVD) on the batch gradients to avoid catastrophic forgetting and negative transfer effects, XMixup employs the low-cost mixup strategies to achieve similar goals. Compared to Cui et al. 20188) the proposed algorithm XMixup also adopts similar procedure (pairing the classes in cource/targe domains) pick up auxiliary samples from the source domain for knowledge transfer. XMixup however further mixes up the target training set with auxiliary samples and fine-tunes the pre-trained model with the data an end-to-end manner, rather than using two-step approach for fine-tuning Cui al. (2018). Compared to Ge & Yu (2017b), which combines source/target tasks together to fine-tune the shared DNN backbones, the proposed algorithm here mixes up data from the two domains and boosts the performance through a simple fine-tuning process over the mixup data with low computational cost.
Finally, we extend the usage of vanilla mixup strategy Zhang al. (2018b) for the applications of transfer learning, where in terms of methodologies we propose to pair classes the two domains and perform mixup over the selected auxiliary samples for improved performance. Actually, mixup strategies have been used in Xu et al.] (2020) for unsupervised domain adaption. Since the target task assumed to share the same set of classes as the source domain in Xu al. (2020), selecting auxiliary samples or pairing the source classes to fit the classes in the target domain is not required.
3 XMIXUP: CROSS-DOMAN MIXUP FOR DEEP TRANSFER LEARNING
Given the source and target datasets and a pre-trained model (that has been well-trained using the source dataset), XMixup performs deep transfer learning using steps as follows.
Auxiliary Sample Selection Given a source dataset S with m classes and a target training dataset T with n classes, XMixup assumes the source domain usually with more classes than the target one (i.e., m > n), and it intends to pair every class in the target training dataset with unique and dedicated class the source dataset (one-to-one pairing from the target to source classes). More specifically, given a pre-trained model, XMixup first passes every sample from the two datasets through the pre-trained model and obtains features extracted from the last layer the feature extractor. Then, XMixup groups the features of the samples according to the ground truth classes in their datasets, and estimates the centroid of the features for every class in both datasets. Such that, for every class in the source target dataset, XMixup represents the class as the centroid of the
3

---

Under review as a conference paper at ICLR 2021
(1) Auxiliary Sample Selection
(2) Cross-domain Mixup and Fine-tuning
Auxilary Datasat
Source Dalasel
Cless 2
Clea
Cass
Croesin Domatn
CNN
Auty Alea where Beta(os,
Target
Target Dataset
Figure I: Overall Design of XMixup Algorithm
features using the pre-trained model θ
ppeetrain for every sample z; in the class c, i...
centroid(c) 1
d(2,, pretrain)) for e S e
(1)
c Va,Ec
Given two classes e, and et in the source and target domains respectively, we consider the similarity between the two classes as the potentials for knowledge transfer, while XMixup measures the similarity between the two classes using the cosine measures between the centroids of the two classes, such that dist(c.,, ax) cosine < centroid(c.), centroid(ca) >. In this way, the auxiliary sample selection could be reduced to search the optimal transport between the sets of classes of and T respectively, via the pre-defined distance measure. Hereby XMixup intends to find one-to-one mapping T** + S, such that
Pa
argmin VPC(SXT)n020 vaTT
Σ ((et(c,
(2)
where S refers to the Cartesian product of the target and source class sets, 020 refers the constraint of the one-to-one mapping. P(++) maps the target class to a unique class from the source domain. Note that p refers to the optimal mapping that potentially exists to minimize the overall distances, while XMixup solves the optimization problem using simple Greedy search Cui et al. (2018) to pursue a robust solution denoted as pareedy in low complexity. Compared to XMixup, the Seq-Train algorithm Cui et al. (2018) uses Greedy algorithm to pair the source/target classes via the measure Earth Mover's ' Distance (EMD), which might be inappropriate our settings of transfer learning.
Cross-domain Mixup with Auxiliary Samples and Fine-tuning Given the one-to-one pairing pereedy from target to source classes, XMixup carries out the fine-tuning process datasets. In every iteration of fine-tuning, XMixup first picks up mini-batch of training samples B drawn from the target dataset T; then for every sample x, in the batch B, the algorithm retrieves the class of Te as Fz. class and randomly draws one sample S. from the paired class of class, such that
the two
Fs i.id pgrredy (xy.clas)), Va; E B.
(3) We consider vs as an auxiliary sample of xt in the current iteration of fine-tuning. XMixup then mixes up the two samples well as their labels through linear combination with trade-off parameter drawn from the Beta distribution Beta(a, B), such that
x = Ax + (1 A)x., Ate + (1 - A)ye, and X i.idd Beta 3).
(4)
In this way, XMixup augments the original training sample (zt, y:) from the target domain using the auxiliary sample (z., ya) from the paired source class, for knowledge transfer purposes. XMixup fine-tunes the pre-trained model prettaiin using the mixup samples accordingly.
EXPERIMENTS AND OVERALL COMPARISONS
4.1 DATASETS
Stanford Dogs. The Stanford Dogs Khosla et al. (201T) dataset contains images of 120 breeds of dogs worldwide, each of which containing 100 examples for training and 72 for testing.
4

---

Under review as a conference paper at ICLR 2021
Table 1: Comparison of top-1 accuracy (%) with different transfer learning algorithms. L refers to naive sine-tuning with weight decay. L SP refers regularizing around the pre-trained weight Li et al. (2018). BSS refers to Batch Spectral Shrinkage Chen et al. (2019). Seq- Beq-Train refers to pre- training with the auxiliary dataset and then fine-tuning the target dataset Cui et al. (2018) refers to selective joint training with the auxiliary dataset Ge & Yu (2017a).
Co-train
Regularized Learning
Multitask Learning
Dataset
L
L SP
BSS
Seq- Train
Co-Train
XMixup
CUB-200-211
80.37=0.12
80.04=0.19
80.76=0..35
78.8910..3
80.83:10.19
81.64:1.23
Stanford Cars
89.23=0.11
88.63=0.118
89.98:1.116
87.29::0115
88.16:10.09
90.58:10.17
Flower- 102
91.64110.2
91.98:10.24
91.59-00.2
90.27+0.31
90.66:10.33
93.45::0.14
Food-101
83.84+0.10
84.06++0.04
83.90=+0.06
83,49+00.99
84.04+0.114
84,33++0.05
Stanford Dogs
86.31=0.07
88.72-+0.12
86.41+0.04
87.2+0.17
90.13++0.07
91,24++0.08
FGVC-Aircraft
83.12=00.00
82.51
83.77+00.33
00.74
82.78+0.42
84.66+0.57
Average 85.75
85.99
86.07
84.65
86.10
87.65
CUB-200-2011. Caltech-UCSD Birds-200-2011 Wah et al. (201 consists of 11,788 images of 200 bird species. Each species is associated with a Wikipedia article.
Food-101. Food-101 [Bsssrdd et al. (2014) is a large scale data set consisting of more than 100k food images divided into 101 different kinds.
Flower- 102. Flower-102 Nilsback & Zisserman (2008) consists of 102 flower categories. 1,020 images are used for training and 6149 images for testing. Only 10 samples are provided for each category during training.
Stanford Cars.
The Stanford Cars Krause (2013) dataset contains 16,185 images of 196 classes of cars. The data is split into 8,144 training and 8,041 testing images.
FGVC-Aireraft. FGVC-Aireraft Maji et al. (2013) is fine-graines visual classification dataset composing more than 10,000 images of aircraft across 102 different aircraft models.
4.2 TRAINING DETAILS
We evaluate the recent state-of-the-arr transfer learning methods in addition to XMixup. They divided into two categories, which are regularized learning and multitask learning. For the former, evaluate fine-tuning with L regularization, - SP regularization Li et al. (2018) and BSS regularization Chen et al. 2011)) Note that the multitask learning is defined in a broad sense here, including traditional co-training Ge & Yu (017a), sequential training Cui al. (2018) and co-training with XMixup. Compared with regularized learning, the essential difference Is to re-train labeled auxiliary examples from the souree dataset instead of regularizing the source model. We use the strategy described in Section! 3 to select auxiliary examples for all multitask learning experiments. According to the empirical study in Ge Yu (2017a), a threshold value the auxiliary dataset size is required to guarantee the effect. In our implementation, until the size of the selected subset reaches a specific threshold. The value is 100,000 for Stanford Dogs and 200,000 for other datasets.
repeat the one-to-one pairing procedure
All experiments are performed using modern deep neural architecture ResNet-50 et al. (2016), pre-trained with ImageNet. Input images are resized with the shorter edge being 256 and center cropped to square. We perform standard data augmentation composed of random flip and random cropped to 224 x 224. For optimization, we use SGD with the momentum of 0.9 and batch size of 48. We train 9,000 iterations for all datasets. The initial learning rate is set to 0.001 for Stanford Dogs and 0.01 for remaining datasets. It divided by 10 after 6,000 iterations. Each experiment is repeated five times. The average top-1 classification accuracy and standard division are reported.
4.3 RESULTS
Accuracy. The top-1 classification accuracies reported in Table We observe that regularized learning methods obtain similar results, while L SP performs obviously better on Stanford Dogs. BSS achieves similar improvements average but is more stable, consistently outperforming 1.2
5

---

Under review as a conference paper at ICLR 2021
Table 2: Evaluating the accuracy on different subsets of the source dataset. They are samples in the auxiliary dataset (Auxiliary), samples not in the auxiliary dataset (ABA: All But Auxiliary Samples), and the whole source dataset (All).
CUB- 200-2010-2011
FGVC-Aircraft
Stanford Dogs
Subset
L
XMixup
L
XMixup
L
XMixup
Auxiliary
62.11
62.91
49.76
52.78
75.92
79.37
ABA
38.17
36.82
20.36
18.93
64.67
64.84
All
38.73
36.83
26.40
26.89
64.39
64.70
6.0
1.60
4.00
0.5
Figure 2: Singular values of feature matrices extracted by different transfer learning models. Singular values are divided by the corresponding largest one for scale normalization. Top 10 smallest values are presented. T.² models are trained with random sampling rates 10%, 30%, 50% and 100% respectively.
by a small margin on most datasets. As for multitask learning, pre-training with auxiliary examples before fine-tuning often hurts the performance surprisingly. This may be caused by the phenomenon of catastrophic forgetting Li Hoiem (2017), that pre-training with a subset of the source dataset loses general knowledge stored in the source model, but this general knowledge is probably useful for the target task. Co-training with auxiliary examples performs much better because auxiliary examples from the source task are used to preserve the knowledge. However, Co-training still hurts the performance on some datasets which have larger distance of data distribution to the source dataset such as Stanford Cars and FGVC-Aircraft. XMixup obtains obvious higher average accuracy than all baseline methods. It is also very robust and stably outperforms naive fine-tuning with L regularization.
Complexity. As stated in Li et al. (2018); Chen et al. (2019), regularized learning methods are usually efficient because the computational complexity involved by the regularization is approximately proportional to the number of parameters or features. While previous multitask learning methods Ge & (017 Cui al.) (2018) are time consuming deal with auxiliary examples, costing additional time whose scale is at least the same as training only target examples. XMixup achieves the efficiency almost the same as naive fine-tuning, because extra computations for data mixing are negligible.
5 DISCUSSIONS AND ABLATION STUDIES
In this section, we provide more empirical studies to analyze the effectiveness and applicability of our algorithm. In subsection 5.1 we show that XMixup effectively alleviates catastrophic forgetting and negative transfer. In subsection 5.2 and [.3 XMixup is proved to be not sensitive with the auxiliary dataset selection and hyperparameter setting, indicating that XMixup is easy to be applied in various real world tasks. Finally in subsection 5.4 we analyze two essential characteristics, crossing domain and supervised mixing, showing that they are both necessary.
5.1 ANALYSIS OF THE EPFECTIVENESS
Authors in Chen et al. 2019 figure out that deep transfer learning suffers from two kinds of problems, which are catastrophic forgetting and negative transfer. We do empirical analysis showing that XMixup explicitly or implicitly deals with both issues. Three different datasets are used for detailed analysis.
Catastrophic Forgetting. A widely adopted measurement about catastrophic forgetting is to evalu- ate the accuracy of fine-tuned model on the previous task Li & Hoiem (2017); Kirkpatrick et al.

---

Under review as a conference paper at ICLR 2021
Figure 3: Influence of the auxiliary dataset size on the performance of XMixup.
Table 3: Top-1 accuracy of XMixup using random auxiliary examples.
CUB-200-2011
Stanford Cars
Flower-102
Food-101
Stanford Dogs
FGVC-Aireraft
81.39(-0.25)
90.66(+0.08)
93.33(-0.12)
84.12(-0.21) 85.58(-5.66)
84.37(-0.29)
(017)). It worth noting that although parameters are totally changed after fine-tuning, the general knowledge is still preserved for feature representation. We thus can use models fine-tuned on target tasks as feature extractors for images in source domains. randomly initialized fully connected layer trained to adapt to each specified source task. Results in Table ssho that XMixup helps preserve the knowledge about auxiliary samples, although this may hurt the capacity of representing samples not in the auxiliary dataset.
Negative Transfer. Chen et al. 2019) finds that the distribution of tail singular values indicates the degree of negative transfer. Specifically, negative transfer be reduced by suspending tail singular values. In Figure 2 we observe that XMixup shows similar trends with increasing number of training examples, which is the most meaningful approach avoid negative transfer. Through sufficient utilization of auxiliary examples, XMixup further decreases smallest singular values.
5.2 SENSITIVITY OF THE AUXILIARY DATASET
Since the only external dependence is the auxiliary dataset, we analyze in detail how characteristics of the auxiliary dataset influences the effect of XMixup.
Size. We first discuss how XMixup performs as the size of the auxiliary dataset varies. We create auxiliary datasets with different scales based on the default dataset described in experiment settings. To increase the size, we continue adding the most similar category from the source dataset until the whole set is selected. To decrease the size, we perform random sampling from the default dataset. Note that these smaller auxiliary datasets are only different on the size but not the domain similarity, while larger auxiliary datasets are less similar with the target domain. As illustrated in Figure 3 CUB-200-2011 1 and PGVC-Aircrale show very good robustness to the size of the auxiliary dataset. Specifically, XMixup performs well enough when the number of auxiliary examples is more than 100,000. Surprisingly, we find that XMixup still significantly outperforms naive fine-tuning even simply using the whole source dataset without selection. The task of Stanford Dogs is bit different that using too many dissimilar auxiliary examples hurts the performance, since it is closely related with subset of ImageNet. However, all tasks can benefit from XMixup obviously with a wide range of the number of auxiliary examples starting from about 30,000.
Domain Similarity. In order to investigate how XMixup depends on the similarity of auxiliary examples, we keep the size of the auxiliary dataset the same, but only replace auxiliary examples by random sampling from the entire source dataset. The result is presented in Table We observe that most datasets are not affected obviously by the similarity removing, indicating that XMixup a robust approach to integrate the general knowledge during fine-tuning. In subsection [5.4, we further show through ablation study that, knowledge from the source domain plays an important role in XMixup. In other words, crossing domain mixing is more than a kind of perturbation on target examples.
7

---

Under review as a conference paper at ICLR 2021
Table XMixup without the characteristic crossing domain (Mixup) or supervised mixing (w/o Label) are evaluated respectively.
CUB-220-2011
FGVC-Aireraft
Stanford Dogs
XMixup
Mixup
w/o Label
XMixup
Mixup
w/o Label
XMixup
Mixup
w/o Label
81.64
80.21
77.89
84.66
82.79
80.40
91.24
86.43
83.65
5.3 SENSITIVITY OF HYPERPARAMETERS
There are two parameters a and 3 in Beta distribution. We use fixed of and only change to control the mixing weight the two domains. Larger a means larger sampling weights for examples from the target domain. We explore broad range of values for and investigate how well XMixup performs under different settings. We illustrate Beta distribution and the influence of varying a in Fig. 4 and Fig. 5 in Appendix.
General trends. We observe that, in all experiments conducted, XMixup demonstrates relatively continuous and smooth change as varies. Top-1 accuracy tends to be lower when a is small because the training assigns as a increases, followed by gradual drop when continues to increase as using too large value for a degenerates to naive fine-tuning.
small weight to samples in the target dataset. It then rises its maximum
Suggested range for a depending on datasets. Our experimental results show that, for datasets that are very similar to the source datasets, such Stanford Dogs which is subset of ImageNet, lower a (2 ~ 2-1) usually performs better. Otherwise, a in a wide range between 2 and 26 is generally safe to improve the naive fine-tuning and even state-of-the-arr baseline methods.
5.4 ABLATION STUDY
To further determine whether both components, namely crossing domain and supervised mixing, are essential for XMixup, we evaluate the variants. First remove the characteristic of crossing domain, implementing traditional Mixup Zhang et al. (2018b) using only samples in the target dataset (in-domain). Second, we keep crossing domain sample mixing the data generation step but remove labels of auxiliary samples.
As reported in Table Mixup has lower accuracy than XMixup by an average of 2.7% in the three datasets we experimented. This consistent difference implies that crossing domain is an essential factor that contributes to the effectiveness of XMixup. Furthermore, Mixup has similar, or even slightly worse, performance compared to T,2 models, whose results are shown in Table. [ This phenomenon suggests that general data augmentation may not work well under transfer learning scenario where sample size is limited. Tasle shows that XMixup is more than data augmentation. observe that model performance drops significantly (for an average of 5.2%) if labels of auxiliary examples removed. This result indicates that knowledge of the source dataset is essential.
CONCLUSIONS
In this paper, we study the problem of knowledge transfer for DNN models from the perspectives of multi-task learning. We propose novel deep transfer learning algorithm XMixup, namely Cross- domain Mixup, with superiority in both effectiveness and efficiency. Through a two-step approach with (1) auxiliary sample selection and (2) cross-domain mixup and fine-tuning XMixup achieves significant performance improvements with 1.9% higher classification accuracy on average, when compared to the state-of-the-ar algorithms Donahue et al. (2014); Li al. (2018): Chen et al. (2019). XMixup is also robust to hyperparameter choices and ways of auxiliary sample selection. Finally, we conclude that knowledge transfer through multitask learning with set of selected auxiliary samples is no doubt a promising direction with huge potentials, while this work suggests a solid yet easy-to-use baseline method.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Lukas Bossard, Matthicu Guillaumin, and Luc Van Gool. Food-101 - mining discriminative compo- nents with random forests. In European Conference on Computer Vision, 2014.
Xinyang Chen, Sinan Wang, Bo Fu, Mingsheng Long, and Jianmin Wang. Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer learning. In Advances in Neural Information Processing Systems, pp. 1906-1916, 2019.
Yin Cui, Yang Song, Chen Sun, Andrew Howard, and Serge Belongie. Large scale fine-grat fine-grained categorization and domain-specifis transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4109-4118, 2018.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: deep convolutional activation feature for generic visual recognition. In International conference on machine learning, pp. 647-655, 2014.
Weifeng Ge and Yizhou Yu. Borrowing treasures from the wealthy: Deep transfer learning through selective joint fine-tuning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 10066-199, 2017a.
Weifeng Ge and Yizhou Yu. Borrowing treasures from the wealthy: Deep transfer learning through selective joint fine-tuning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 10-19, 2017b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Aditya Khosla, Nityananda Jayadevaprakash. Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine-grained image categorization. In First Workshop FneeGrained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, June 2011.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Bard Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-33) 2017.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. object representations for fine graingri categorization. In 4th International IEEE Workshop 3D Representation and Recognition (3dRR-13) 13), Sydney, Australia, 2013.
Xingjian Li, Haoyi Xiong, Hanchao Wang, Yuxuan Rao, Liping Liu, and Jun Huan. Delta: Deep learning transfer using feature map with attention for convolutional networks. arXiv preprint (rXiv:1901.0922) 2019.
Xuhong Li, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. Thirty-ffith International Conference on Machine Learning, 2018
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. ArXiv, abs/1306.5155 2013.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing. pp. 722-729. IEEE, 2008
Sinno Jialin Pan, Qiang Yang, et al. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10): 2010.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.

---

Under review as a conference paper at ICLR 2021
R. Wan, H. Xiong, X. Li, Z. Zhu, and J. Huan. Towards making deep transfer learning hurt. In 2019 IEEE International Conference on Data Mining (ICDM), pp. 578-587, 2019.
Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang. Adversarial domain adaptation with domain mixup. In Proceedings the AAAI Conference on Artificial Intelligence, 2020.
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), volume 2, 2017.
Chunjie Zhang, Jian Cheng, and Qi Tian. Image-level classification by hierarchical structure learning with visual and semantic similarities. Information Sciences, 422:271-2811 2018 8a.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018b. URL https://openrevie net um rDdpp
A APPENDIX
A.1 BETA DISTRIBUTION
54
04-
24
20
32
Figure Examples of Beta Distribution.
A.2 INFLUENCE OF THE MIXING WEIGHT
12
960
Figure 5: Influences the choice of the hyperparameter a log scale. Black nodes refer to the default value used in previous experiments. Doted lines refer to the accuracy of state-of-the-art baselines.
10


Reviews:
[{"review_id": "xy_hv7w2eb", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposes a simple variant for the mixup training mechanism for transfer learning problems: cross-domain mixup (XMixup). The key idea is to mix up the training samples from both domains where the samples are generated by nearest-center assignment in each class. Experiments on several datasets have shown its effectiveness in transfer learning compared to some SOTA methods.\n\nPros:\n\n1. A simple but effective method for mixup in transfer learning.\n2. The method is easy to implement with improvements in experiments.\n3. The experiments are fair enough.\n\nCons:\n\n1. The main disadvantage is its limited novelty. Computing the class center of each class and then initiating an assignment between each source and target class is a common strategy and also has been used by several existing works (see references below). In addition, the experiments on arbitrary sample selection (table 3) also show that even using random samples to form the mixup training samples, the performance is also good and has a very small gap to that in table 1. Therefore, I think this method can only be seen as a moderate extension to mixup for transfer learning problems.\n2. The second disadvantage is that even if the experiments in table 1 show some positive results, the improvement is rather marginal, and all the datasets are rather small. For example, the largest dataset in this paper is Stanford Cars with 16,185 images where XMixup only achieves an accuracy improvement of 0.6%, which is not significant. Why not use other larger datasets such as Caltech-101, Caltech-256, Sun397, STL, and CIFAR-100? Since the method itself is simple, you should experiment on more larger datasets to show that it really works.\n3. This method is problematic in real applications. It is not easy to use as it requires to store and access the source domain data (e.g., ImageNet), which is almost not possible. Therefore, even if XMixup is simple and doable in the idea level, the efficiency should be considered by not only the running time, but also the data storage and access cost. To this end, the regularized learning methods in table 1 are more applicable since they do not have to access the original source data. In addition, only marginal improvements are achieved by storing and accessing the large volume of source data. Therefore, the method in its current form is not helpful in real applications.\n4. One of the claims in this paper is that XMixup can prevent catastrophic forgetting, i.e., \"remember\" both the source and target tasks. Why do we care about the forgetting phenomenon in transfer learning since its goal is to transfer the knowledge to the target domain to ensure that best performance can be achieved in the target domain? Do you have any evidence to show that preventing the catastrophic forgetting can somehow benefit the performance on the target?\n5. Continued from point four, the efficiency of this method is not validated in experiments to compare with both regularized learning and multitask learning methods. So, it is hard to evaluate its time efficiency.\n6. Finally, there lacks detailed analysis of why this simple mixup strategy would work apart from the accuracy results. For instance, what kind of representations did this method learn by mixing up that help it generalize better to target domain? Why it achieves comparable results with random selection? \n7. Application scenario: All images in this paper are natural images that should remain similar in general. Will this method work for distant domain images such as ImageNet->medical images? It remains unclear when to use this method. A doable application scenario is needed. \n\nTo sum up, this is a good paper, but not appropriate for ICLR in its current form due to limited novelty, poor analysis, and marginal improvements.\n\nReferences:\n\n[1] Snell J, Swersky K, Zemel R. Prototypical networks for few-shot learning[C]//Advances in neural information processing systems. 2017: 4077-4087.\n\n[2] Pan Y, Yao T, Li Y, et al. Transferrable prototypical networks for unsupervised domain adaptation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019: 2239-2247.\n\n[3] Wang J, Chen Y, Yu H, et al. Easy transfer learning by exploiting intra-domain structures[C]//2019 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2019: 1210-1215.\n\n[4] Hu D, Liang J, Hou Q, et al. PANDA: Prototypical Unsupervised Domain Adaptation[J]. arXiv preprint arXiv:2003.13274, 2020.\n\n[5] Sun Q, Liu Y, Chua T S, et al. Meta-transfer learning for few-shot learning[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2019: 403-412.\n\n[6] Choi J, Hwang S J, Sigal L, et al. Knowledge Transfer with Interactive Learning of Semantic Relationships[C]//AAAI. 2016: 1505-1511.", "extra_scores": {}, "extra_text": {}}, {"review_id": "nxdk47DhYQX", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposes a method to address the transfer learning in the multi-task learning setup. The proposed method has two main components: (1) finding a one-to-one mapping from source dataset classes to the target dataset classes (2) applying mixup between the samples from the source and target class pairs.\n\nReasons for score:\nOverall, I vote for rejecting the paper. \n\n1.The main idea of this paper is to compute the one-to-one pairing of classes in source and target domain ( i.e. for a given class in target dataset, find the closest class in the source dataset) I think this idea of having one-to-one mapping is flawed in principle. For example, let us consider that  the source dataset consists of 100 classes which represent different dog breeds and the target dataset has 10 classes representing different cars. In this scenario, one to one pairing between a dog class and the car class has no benefit for transfer learning. The authors should clearly state the underlying assumptions about the source dataset and the target datasets for the proposed method to work.\n\n2. The technical novelty of the paper is very limited. The closest related work to this paper is Cui et. al. which selects K classes from source domain which are closest to the target dataset classes. In this work, the authors propose instead to use one-to-one pairing, without explicit explanation of why this is a good idea. Applying mixup between the source and target dataset samples is not novel by itself, without making explicit why it helps to facilitate better transfer learning. \n\n3.The paper claims that “XMixup however further mixes up the target training set with\nauxiliary samples and fine-tunes the pre-trained model with the data in an end-to-end manner, rather than using a two-step approach for fine-tuning Cui et al. (2018).” As far as I understand this is the biggest contribution of this paper ( to make the learning efficient), yet the authors do not make an attempt to elaborate  on this point of “two-step approach of Cui et. al vs their approach) \n\n4.Overall, the tone of the paper is not scientific at many places ( examples below) \n\nPros:\nThe paper studies a very relevant problem of transfer learning.\nIt seems that the authors have tried to do robust experimental evaluation. \n\nOther minor comments:\n\n(A)The sentence formation is incorrect at many places:\n“Later, the work Chen et al. (2019);Wan et al. (2019) introduces new algorithms that prevent the regularization from the hurts to transfer learning,”\n“Given two classes c s and c t in the source and target domains respectively, we consider the similarity between the two classes as the potentials for knowledge transfer, while XMixup measures the similarity between the two classes using the cosine measures between the centroids of the two classes”\n“Note that P ∗ refers to the optimal mapping that potentially exists to minimize the overall distances, while XMixup solves the optimization problem using a simple Greedy search”\n\n(B)“While vanilla mixup augments the training data with rich features and regularizes\nthe stochastic training beyond empirical risk minimization (ERM),” saying “rich features” is not a technical description of the related work. \n\nC. ”According to the empirical study in Ge & Yu (2017a), a threshold value of the auxiliary dataset size is required to guarantee the effect. In our implementation, we repeat the one-to-one pairing procedure until the size of the selected subset reaches a specific threshold. The value is 100,000 for Stanford Dogs and 200,000 for other datasets. ” This seems like an important hyperparameter, yet it is not obvious where it is in the method section.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Mx3PuH0o6h", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a multi-task paradigm for deep transfer learning via cross-domain mixup. Specifically, authors develop a strategy to transfer the knowledge from source to target tasks more efficiently by selecting the auxiliary samples from the source\ndataset and augments training samples via the simple mixup strategy. The paper is well written and easy to follow. Experiments on six datasets show the efficacy of the proposed method over existing alternatives.\n\nOverall, I vote for rejecting the paper due to limited novelty and lack of convincing experiments. The proposed approach simply combines different existing well-known techniques such as sample selection and mixup without any significant changes. Mixup has been widely used as an augmentation technique to improve the performance of a model. Similarly, as mentioned by the authors, source sample selection is also not new in transfer learning and has shown to be effective in Seq-Train. The only difference with Seq-Train is that the proposed approach uses a multi-task learning set up instead of a two phase approach for transfer learning. Low-cost mixup has also been proposed in Zhang et. al, 2018. Thus, I fail to understand the major contributions of the paper beside combination of the two simple strategies for transfer learning.\n\nAuthors mention about reviewing exiting solutions as a contribution in the paper. However, apart from a couple of paragraphs, there is no analysis on the catastrophic forgetting, negative transfer and efficiency. How does the proposed method is superior in terms of catastrophic forgetting and negative transfer compared to prior multi-task learning approaches? How prior methods are affected by catastrophic forgetting is not clear? More experiments and analysis is essential to consider this as a major contribution of the paper. \n\nMixup across different domain samples has been used. in domain adaptation. E.g., Adversarial Domain Adaptation with Domain Mixup, Dual Mixup Regularized Learning for Adversarial Domain Adaptation. Besides the difference in application, how is XmixUp different from these cross-domain mixup strategies? Please explain on this.\n\nAuthors compare few existing method in experiments. However, there are few very related recent methods that should be compared in the experiments to verify the effectiveness of the proposed approach. E.g., Why comparison with Delta: Deep learning transfer using feature map with attention for convolutional networks, ICLR 2019 is missing in the paper? Although DELTA is based on regularization, it is one of the recent transfer learning approach that has much better results compared to L2-SP. Comparison with recent baselines are essential to show the advantage of the proposed approach in transfer learning.\n\nHow is the proposed method different from Learning to Transfer Learn: Reinforcement Learning-Based Selection for Adaptive Transfer Learning, ECCV 2020? L2TL reweighs source samples in a multi-task learning set up which is very close to the current approach. Authors should compare and discuss L2TL in the paper.\n\nHow is the proposed method comparable to Pay Attention to Features, Transfer Learn Faster CNNs, ICLR 2020? AFDS is much better than L2SP and should be compared in the experiments to verify the effectiveness of the proposed transfer learning approach.\n\nI would like the authors to do more works related to experiments to improve the quality of the paper. I think the paper needs significant changes including new experiments and discussions before being accepted to any major conference.", "extra_scores": {}, "extra_text": {}}, {"review_id": "FfjHw8XfP-x", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes XMixup, a strategy for improving transfer learning in neural networks. Specifically, XMixup consists of mixup applied between target samples and source samples from the class pre-determined to be closest to target sample’s class. Experiments conducting transfer learning from pre-trained ImageNet to 6 smaller image classification datasets demonstrate XMixup to outperform the baseline approaches.\n\nPros:\n1.\tThe proposed method is conceptually simple and easy to understand/implement. Since source and target information are combined into a single sample, only a single gradient needs to be backpropagated through the network. Additionally, the mixup operation is simple (draw from beta distribution, then a convex combination of the two images + labels), so overall XMixup is fairly cheap computationally, once class feature centroids have been computed (see Cons #2).\n2.\tThe experiments demonstrate that XMixUp is fairly effective for the target image classification datasets shown, moderately outperforming the baseline methods on the 6 datasets, which have varying degree of overlap/similarity with ImageNet (source dataset).\n3.\tThe authors made an effort to demonstrate why XMixup is effective, with many different kinds of experiments, including several I wouldn’t have thought of. See Cons #1, however.\n\nCons:\n1.\tThe novelty of XMixup is moderately low; for the most part, it’s a fairly standard application of mixup, applied to transfer learning with a simple pairing of source and target classes. This could be fine if the experiments were particularly thorough or convincing. However, all experiments are in image classification, transferring from ImageNet as the source dataset. I would have liked to see experiments on more settings (e.g. NLP) or problem types (e.g. source: classification -> target: object detection). Additionally, several ablation studies that would shed more light on how XMixup works (see 2. below) are missing.\n2.\tWhile training is fairly efficient, XMixup does require computing the centroids for all source and target samples first, which can be somewhat expensive (for example, if ImageNet pre-training is the source, then features for all 1M ImageNet samples must be computed). It’s not clear how important this class-matching step is though. An ablation study without class matching target samples with the closest source class would be helpful. It’s also unclear how essential that the mapping between source and target classes be one-to-one; again, an ablation would have been helpful.\n3.\tOne of the central arguments made is that XMixup prevents catastrophic forgetting, with experiments (Table 2) providing further evidence. Why does catastrophic forgetting matter for transfer learning though? “Forgotten” features are most likely to be those from the source pre-training task that have little relevance to the target task. For example, if fine-tuning ImageNet for aircraft classification, why should one care if the network can still distinguish dog breeds, or foods?\n4.\tThe writing quality is not up to publication standards; please give this draft another careful round of edits. A non-exhaustive list of corrections:\no\tPg 1: Backgrounds -> background\no\t“Later, the work Chen et al. (2019); Wan et al. (2019) introduces new algorithms that prevent the regularization from the hurts to transfer learning.” \no\t“two phase” -> “two-phase”\no\t“large scale” -> “large-scale”\no\tBoth “pre-train” and “pretrain” appear in the text\no\t“each of which containing 100 examples” -> “each containing 100 examples”\no\t“standard division” -> “standard deviation”\n\nQuestions:\n-\tUsing centroids of the features for each class (Eq 1) to determine class similarity makes several assumptions about the shapes of the class distributions in the feature space. Have you inspected the feature space of the pre-trained network to verify if these assumptions are reasonable?\n-\tWhat is the dimensionality of the model output? Assuming m source classes and n target classes, is the output dimensionality m, or m+n? Figure 1 implied the latter, but I didn’t see it stated anywhere in the text.\n-\tFigure 3: Why would there be a drop in performance (especially for Stanford dogs) when the auxiliary dataset gets large? Why is it surprising that XMixup would outperform fine-tuning when using the full source dataset?\n-\tPre-training is not only used just for classification. Object Detection, segmentation, VQA, and many other types of problems also rely on pre-trained networks. Can XMixup be used for other problem types? What about for NLP (e.g. BERT fine-tuning)?\n\nMiscellaneous:\n-\tUnless using it like a noun, citations should be parenthetical (\\citep).\n-\tx_t.class is awkward notation\n-\tTable 1: It would be useful to see plain naïve fine-tuning (no regularization, no XMixup, etc.) as well. Also, consider bolding the best values; makes the table easier to read.\n\nDecision:\nDespite mediocre novelty, XMixup seems to be an effective and simple technique. However, although the authors include several experiments aimed at understanding XMixup’s efficacy, several key ablations are missing, and there’s a lack of diversity in experimental settings (all ImageNet -> smaller image classification set). The writing is also not quite at publication standards and could use some work. As such, while I think there may be potential, I don’t think this work is ready at this time, and vote for rejection. I think this work can be significantly improved with increased experimental setting variety, better ablations, and a careful round of edits.\n\n\n=======\nPost-rebuttal\n=======\n\nI thank the authors for their comments. The current draft isn't a bad start, but in agreement with the other reviewers, it needs more work before it's ready for publication. Please carefully consider our recommendations for your next draft.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers recommend rejection due to limited novelty and insufficient experimental analysis. The author’s response has addressed several other questions raised by the reviewers, but it was not sufficient to eliminate the main concerns about novelty (as the method is a combination of existing techniques) and missing comparisons to justify the effectiveness of the proposed approach.
Decision: Reject