SMILE: Sample-to-feature Mixup for Efficient Transfer Learning
Authors: Xingjian Li
Haoyi Xiong
Cheng-zhong Xu
Dejing Dou
Venue: Accepted by TMLR ()

Abstract: To improve the performance of deep learning, mixup has been proposed to force the neural networks favoring simple linear behaviors in-between training samples. Performing mixup for transfer learning with pre-trained models however is not that simple,  a high capacity pre-trained model with a large fully-connected (FC) layer could easily overfit to the target dataset even with samples-to-labels mixed up. In this work, we propose SMILE — Sample-to-feature Mixup for Efficient Transfer Learning. With mixed images as inputs, SMILE regularizes the outputs of CNN feature extractors to learn from the mixed feature vectors of inputs, in addition to the mixed labels. SMILE incorporates a mean teacher to provide the surrogate "ground truth" for mixed feature vectors. The sample-to-feature mixup regularizer is imposed both on deep features for the target domain and classifier outputs for the source domain, bounding the linearity in-between samples for target tasks. Extensive experiments have been done to verify the performance improvement made by SMILE, in comparisons with a wide spectrum of transfer learning algorithms, including fine-tuning, L$^2$-SP, DELTA, BSS, RIFLE, Co-Tuning and RegSL, even with mixup strategies combined. Ablation studies show that the vanilla sample-to-label mixup strategies could marginally increase the linearity in-between training samples but lack of generalizability, while SMILE significantly improves the mixup effects in both label and feature spaces with both training and testing datasets. The empirical observations backup our design intuition and purposes.


Published in Transactions on Machine Learning Research (01/2023)
SMILE: Sample-to-feature Mlxup for Efficient Transfer LEarning
Xingjian Li
kringgänn@baddu.com com
Big Data Lob, Boidu Inc. State Key Lab of 10TSC, University of Mocoa
Haoyi Xiong" Big Data Lab, Boida Inc.
zionghavyi@@usddu.ccm com
Dejing Dou Big Data Lob, Boida Inc.
doudejing@baiitu.com
Chengzhong Xu
czrutuumm edu
State Key Lab of IOTSC, University of Macau
Reviewed on OpenReview: https: openreview. net/ forum? id- cZgNCpurDM
Abstract
To improve the performance deep learning, mixup has been proposed to force the neural networks lavoring simple linear behaviors in-between training samples. Performing mixup for transfer learning with pre-trained models however is not that simple, high capacity pre-trained model with large fully-connected (FC) layer could casily overfit to the target dataset even with tamples-to-labels mixed up. In this work, propose SMILE Sample- to-feature MIxup for Efficient Transfer LEarning. With mixed images as inputs, SMILE regularizes the outputs of CNN feature extractors to learn from the mixed feature vectors of imputs, in addition to the mixed labels. SMILE incorporates a mean teacher to provide the surrogate 'ground truth' for mixed feature vectors. The sample-to-featu imposed both on deep features for the target domain and classitier outputs for the source domain, bounding the lincarity in-between samples for target tasks. Extensive experiments have been done to verify the performance improvement made by SMILE, in comparisons with wide spectrum of transfer learning algorithms, including fine-tuning, T.--PP, DELTA, BSS, RIFLE, Co-Tuning and RegSL, even with mixup strategies combined. Ablation studies show that the vanilla sample-to-label mixup strategies could marginally increase the linearity in-between training samples but lack of generalizabilit, while SMILE significantly improves the mixup effects in both label and feature spaces with both training and testing datascts. The empirical observations backup our design intuition and purposes. Our code is available at https://github.. wit.ght.hh..ooooo com /lixingjaan/SMil /SMILE
mixup regularizer
1 Introduction
Performance of deep learning algorithms real-world applications is often limited by the size of training datasets. Training a deep neural network (DNN) model with a small number of training samples usually leads the over-fitting issue with poor generalization performance. common yet effective solution to train DNN models under transfer learning (aan et al. 2010) settings using large source datasets. The knowledge transfer
"Equal Contribution. Correspondence Hanyi Xiong.

---

Published in Transactions on Machine Learning Research (01/2023)
Fine-tuning Pre-trained
Training Scratch ResNet50
90
35
85
Mean Accuracy
Accuraty
25
70
20
65
QUB-200-2011-44
CUB-200-201144
60
CUB 200-2010-20000 Mixup
15
CUB LUB-200-2011 Mixup
55
Stanford-Cars-LF
Stanford-Cars44
Stanford Cars Mixup
Starfordd Cars--iispppp
400
600
600 800
1600
Labeled
Figure Performance comparison between the 1,2 regularization and mixup for fine-tuning an ImageNet. pre-trained model (left) and training from scratch (right). To simulate scenarios with limited target datasets, we randomly select 50 classes from two transfer learning benchmarks, which CUB-200-2011 and Stanford- Cars. As seen, Mixup brings remarkable improvements if training from scratch (right), but this does not apply to transfer learning (left).
from the source domair helps DNNs learn better features and acquire higher generalization performance for the pattern recognition in the target domain (Donahuc et al. 2014 Yim et al. 2017).
In addition to deep transfer learning, another effective strategy for improve generalization performance of DNN is mixup ZZhang al. 2018), where the objective is to have DNNS in the learning procedure favor the linear behaviors in-between training samples. To achieve the goal, the mixup strategy picks up multiple images from the training set, mixes the samples and labels proportionally to generale new pair of sample and label for data augmentation. The regularization effects brought by mixup could help control the complexity of DNN models (Hanin & Rolnick
2019 Vapnik 2013) while largely improving the robustness and genoralization
performance (Zhang et al. 2020
Research Motivation While existing studies about mixup mainly focus the general setting, to the best of our knowledge, using mixup in transfer learning has rarely been investigated. A straightforward conjecture is that, mixup should be also effective in transfer learning, where typical applications have only limited training examples. In such situations, regularizers aiming to control the model complexity should beneficial for better generalization. However, the facts are surprisingly just the opposite. We find that in deep transfer learning, mixup improves the performance with reduced margins or even downgrades the performance when the larget dataset is small. See Figure [or detailed results.
Considering that either transfer learning and mixup is widely proven beneficial, natural question is raised as follow,
Why mixup tends to negatively affect transfer learning when training samples are limited!
Our Analyses Two reasons caused the ineffectiveness of mixup in transfer learning. First of all, we find fine-tuning with high capacity pre-trained models CAN overfit to the mixup samples/lbells From mixup, simply derive a linear interpolation loss to measure the error of linear interpolation between pair of samples (I1.31 and (2) for the model f(-),
[| + (1 A)x2) - (AS(z)) + (1 (x2(1))
(1)
where a lower linear interpolation loss indicates stronger linear behaviors in-between the samples and usually better generalization performance [Zhngg et al. 2020). Our experiments however find that fine-tuning with
The term domain indicates the concept of features knowledge learned from task. Please that, this paper focuses transferring pre-trained model downstream dataset. (labels are available), rather that domain adaptation.
2

---

Published in Transactions on Machine Learning Research (01/2023)
mixup could obtain low interpolation loss in the training set while suffering high interpolation loss in the test. set (225% higher interpolation loss on the testing set than the one on training set, please see also in Section 5). This observation indicates that the linear behaviors gained by mixup regularization could not well generalize to the testing dataset and overfit to the mixup
from the training dataset. The second problem is particularly linked to a major challenge transfer learning called calastrophic lorgetting (Li et al. 2018 You et al., 2020). The additional interpolated images generated by mix-up drive the fine-tuned model farther from the starting point, which aggravates the loss of transferable knowledge in the pre-trained model.
Thus, our research intends to study way to make mizup strategies generalizable in deep transfer learning settings while alleviating the knowledgee loss during fine-tuning. To achieve the above goal, some non-trivial technical challenges should be tackled.
Cemple-to-Featurr Mizup. high-capacity pro-trained model, offering large quantity of well-trained features, would force Fully-Connected (FC) Layer memorize samples and labels mixed-up with trivial updates to weights of its CNN feature extractor. Though some randomized strategies, such RIFLE LL et al. 2020 could deepen back- prok-propngation in vanilla transfer learning settings, it is still challenging to reinforce the mixup effects in the CNN feature extractor.
Mired-up Feature Vectors. To ensure mixup effects in outputs of CNN feature extractors, a possible way is to let CNN feature extractors learn from the mixed-up samples and feature vectors, while the ground-truth feature vectors are usually not available. Thus, surrogate labels of the feature vectors need to be obtained for any sample in the target dataset before having the CNN trained.
Cross-Domain Generalizabilitt. A pre-trained DNN usually is capable of behaving linearly under interpolation of the source dataset. During the fine-tuning procedure, it is reasonable to doubt that such linear behaviors in source domain might be forgolten (Chen et al. 2019). To improve the generalization performance, there thus needs to preserve the linear behaviors in the source domain and transfer such ability to the target domain during fine-tuning.
Our Work To address above technical challenges, in this work, we propose SMILE Sample-to-feature Mixup strategies for Efficient Transfer Learning. Instead of regularizing mixup effects in target label spaces (i.c., sample-to-label mixup), SMILE enables the sample-to-feature mixup through regularizing the CNN feature extractor to learn from the surrogate of "fine-tuned" feature vectors mixed-up images even when the CNN has not yet been well-tuned on the target domain. To the best of our knowledge, this work has made three sets of contributions as follows.
We study the problem of regularizing DNNs to enjoy mixup effects under deep transfer learning settings, where the major concern is to avoid the overfitting to mixed-up samples and labels, using high-capacity pre-trained model but with small target training dalaset. We elaborate the technical issues, and propose to solve the problem through enabling sample-to-featur mixup, where obtaining the feature vectors for mixup and ensuring cross-domain generalizability of linear behaviors become the key challenges.
2. We propose SMILE Sample-to-featur mixup for efficient transfer learning, where sample-to-featur mixup, through either the target deep feature and the source label space, has been used as the core framework of the solution. Given two samples drawn from the target domain as the input, SMILE linearly combines two samples proportionally and sends the mixed-up sample to the target model. It constrains the Euclidean distance between the output of target model's CNN feature extractor and mixed-up feature vector (i.e., linear combination of mean teacher model's outputs for the two samples) via a sample-to-featur mirup. Morcover, to obtain cross-domain generalizability, SMILE trains an additional FC classifier for the larget network to adapt. the larget dataset but in the source domain. It regularizes the target network using cample-to-labed mixup to learn from the linear combination of classification results on source domain, whose labels are also features for the target domain. SMILE also optimizes the target model via vanilla samplo-to-labe mixup using the mixed-up label (i.e., linear combination of ground-truth labels).
3

---

Published in Transactions on Machine Learning Research (01/2023)
Feature Extractor
9*
Classifier (source)
[Source Model)
1-A
2
9
L F.
I FC
Classifier (Auxiaay))
Feature Extractor (Target Model)
Classifier
+ MMP
(Target)
21
mix operation
forward
Euclideean distonce
teacher
Figure 2: The Architecture of SMILE: Deep Transfer Learning with Sample-to-faatrre Mixup Regularization. To fully exploit the capacity of the source model, incorporate two components I,FE and I,FC beyond the vanilla mixup L MXP, L focuses on the deep features and L is imposed on the source labels through an auxiliary classifier z₁- We feed original images in the source model, and mixed images in the target model during fine-tuning Mean teacher is performed on the source model to provide more accurate pseudo ground truth for mixed features It and zz. After fine-tuning, we use only the feature extractor and classifier of the target model for prediction.
3. We carry out extensive experiments using a wide range of source and target datasets, and compare the results of SMILE with a number of baseline algorithms, including fine-tuning with weight decay
(I2) Donahue et al. 2014). fine-tuning with I'-regularization on the starting point (L²-SP) [Li ct al,
2018). DELTA (Li al. 2019), Batch Singular Shrinkage (BSS) (Chen et al. 2019). RIFLE (Li et al,
2020), Co-Tuming You al. 2020 and RegSL (Li & Zhang 2021) with withhutt mixup strategies. The experiment results showed that SMILE can outperform all these algorithms with significant improvement. The ablation studies show that (1) cample-to-feature mixup design is significantly better than vanilla sample-to-label mixup for deep transfer learning; and (2) the proposed sample-to- label mixup on the source domain can further improve the generalization performance.
2
Related Work
In this section, we first introduce the related works from deep transfer learning's perspectives, then we discuss the most relevant work to our study.
2.1 Deep Transfer Learning
To enable transfer learning for DNNs, fine-tuning (Donahue et al. 2014) has been proposed to first train a DNN model using the large (and possibly irrelevant) source dataset (e.g. ImageNet), then uses the weights of the pre-trained model the starting point of optimization and fine-tunes the model using the target dataset. In this way, by sharing the rich and diverse knowledge contained in large source datascts, the finc-tuned model is usually capable of handling the target lask with better generalization performance. Furthermore, authors in YYm et al. 2017 Li et al. 2018 2019) propose transfer learning algorithms that regularize the training procodure using the pro-trained models, so as constrain the divergence of the weights and feature maps between the pre-trained and fine-tuned DNN models. Later, the work (Chen al.) [019 Wan al. 2019) introduce new algorithms that prevent the regularization from hurting the adaptation to the target.
4

---

Published in Transactions on Machine Learning Research (01/2023)
domain in transfer learning, where (Chen et al. 2019) propose to truncate the tail spectrum of the batch of gradients while (Wan et al. 2019) propose to truncate the ill-posed direction of the aggregated gradients. In addition to the aforementioned strategies, algorithms based on the multi-task learning paradigm have been Cui et al. 2018).
used for deep transfer learning, such as (Ge & Yu 2017
While all above algorithms enable knowledge transfer from source datasets to target tasks, they unfortu- nately perform poorly due to the catastrophic forgetting and negative transfer. Most transfer learning algorithms (Donahue et al.) 2014 Yim et al.) 2017 Li et al. 2018 2019) consist of two steps pre-training and fine- fine-tuning. Given the features that have been learned in the pre-trained models, either forgetting some good features during the fine-tuning process (catastrophic forgetting) (Chen et al. 2019) or preserving the inappropriate features/ filters to reject the knowledge from the target domain (negative transfer) {Li et al.) 2019; Wan et al.) 2019) would hurt the performance of transfer learning. In this way, proper compromises should be made between the features learned from both source//arget domains during the fine-tuning process, where multi-task learning with Seq-Train (Cui et al., 2018) and Co-Train (Ge & Yul 2017) might suggest feasible solutions to well-balance the knowledge learned from the source/ Larget domains, through fine-tuning the model with selected set of auxiliary samples (rather than the whole source dataset) (Cui et al. 2018)
alternatively learning the features from both domains during fine-tuning (Ge & Yu 2017 A recent study in medical imaging (Wang et al. 2022 employs the Co-Train fashion with auxiliary attributes during both the pre-training and fine-tuning step.
Some other studies (Zhong et al. 2020 Wang et al. 2021: Abuduweill et al. 2021 | intend to improve the generalization of the target model by further exploiting the target data. For example, Bi- Bi-Tuning (Zhong et al. 2020) incorporates self-supervised learning on top of the standard supervised fine-tuning. Self-tuning (Wang et al. 2021 and Adaptive Consistency Regularization (Abuduweili et al. 2021) consider more practical scenario that set of unlabeled target data is available. They find that fine-tuning can be substantially promoted by reasonably utilizing the unlabeled data.
2.2 Connections to Our work
The most relevant studies to our algorithm are (Verma et al.
2019 Yun et al. 2019 Li et al.,
2019
Chen
et al. 2019 Li et al., 2020 You et al. 2020 Wang et al. 2022).
While the first two works (Verma et al. 2019
Yun et al. 2019) propose to improve mixup and its derivatives for data augmentation through interpolating
the fcature spaces, the following three works (Li ct al.
2019 Chen et al., 2019 Li ct al. 2020 You et al.
2020) focus on improving deep transfer learning through regularizing the feature or label spaces. Authors in (Wang et al., 2022) impose proximal regularizer to constrain the parameter distance from the pre-trained model, and an auxiliary task that predicts a set of pre-defined attributes such as age, race and so on. In
comparison, our method serves as a general algorithm that is free of additional domain knowledge.
The manifold mixup strategy (Verma et al. 2019) has been proposed to smooth the decision boundary of DNN classifiers using mixed-up feature maps and labels, in a feature-to-label mixup fashion. On the other hand, CutMix YYun et al. 2019) also propose sample-to-feature data augmentation strategy, where the algorithm fuses two images into one and forms a new feature map accordingly, with respect to the localizable visual features in two images. Compared to above works, the major technical difficulty of SMILE is that above algorithms use feature maps extracted from CNN models directly, while SMILE regularizes the output of CNN frature extractor when accurate estimates of feature vectors are not available (the CNN is under fine-tuning to adapt the target datasot).
While (Li et al. 2019 Chen et al. 2019 proposed to improve the feature-wise knowledge distillation spectral regularization for transfer learning, Li et al. 2020) studies way to regularize the pre-trained CNN feature extractor, during fine-tuning, by incorporating randomness from FC layers. Compared to above algorithms, SMILE is proposed to solve the problem of overfitting to mixup under deep transfer learning settings. Our ablation studies in Section 4 show that the simple combination of fine-tuning and mixup strategies does not perform well in transfer learning settings from both the preservation of lincar behaviors and generalization performance aspects. SMILE makes unique contributions in proposing novel sample-to- feature mirup strategies to improve performance in transfer learning with pre-trained models.
5

---

Published in Transactions on Machine Learning Research (01/2023)
3 SMILE: Sample-to-feature Mixup for Efficient Transfer Learning
In this section, we first introduce the overall framework of SMILE, where the architectures of deep transfer learning with Sample-to-feature mixup regularization is presented. Then, we specify the design of the proposed regularizer and discuss the mixup effects incorporated by SMILE.
3.1 Overall Framework
Given target training dataset D {(11, 31), (22, y2)... (Ia, Mn)3 and initial model us pre-trained with the source dataset, SMILE learns model to adapt the target dataset by fine-tuning procedure. Since the source model is involved during fine-tuning, we use subscripts t and to distinguish between the target and source model, superscripts to denote the training iteration. Specifically, SMILE initializes the target model wf with the pre-trained source model wi. For each training iteration, SMILE updates the target model wa through minimizing loss function as follow,
1 n min [(w)us)=
T1 Σ
L MXP (w, (4, + I Res Be Was 22)
(2)
where L MXP (w, Iz. y.) refers to the vanilla sample-to-label mixup loss based on target model on the target domain D and I, (w,,, w;) refers to the loss for Sample-to-feature mixup regularization based on both the source and target model. Note that the computation of L Reg adopts only the target dataset D as the input and does not rely on labels.
The regularizer Reg contains two components, FE and LFC, where FE refers to feature extractor and FC refers to fully-connected classifier. For better exploiting the capacity of the source model, FF and PCC leverage the fine-tuned target deep features and the auxiliary source labels, respectively. We present detailed implementations in Section 3.3
Figure 2
presents the architecture of SMILE.
3.2 Vanilla Mixup
We first introduce the formulation of vanilla mixup (Zhang et al. 2017 Given deep neural network f. we denote its classifier output as 2 and deep feature as Since both and depend on the input data and their corresponding parameters, they are formulated as functions for simpleness. Vanilla mixup aims to minimize the linear interpolation loss in the target label space, which can formulated as
2 (Mix (I,, - Mixx v;)1 MXP (w) AwBeta(a,a) T.
E
E
(3)
where the operator Mixx(u, v) = X) A refers to the linear combination of two inputs. We follow the vanilla mixup (Zhang et al. 2018 to sample the linear combination coefficient À from a symmetric Beta distribution X ~ Beta(a, a). The Beta distribution is usually used to sample a random proportion, c.g. in mixup, since it's the conjugate prior for the Bernoulli distribution. larger a tends to sample a balanced mixture, i.e. more likely near 0.5, and smaller a leads to near
3.3 Deep Transfer Learning with Regularization
Omitting the notation of input data Iı, we present the Sample-to-featury mixup regularizer as follow,
L RoR wx) -IFE L
FE (w, + TFC FC (w,
where "FE and 7FC refer to the weight of the two terms, the term L FE [ wa) refers to the sample-to-feature mixup regularizer which borrows general knowledge from the source domain over the target datasct D, and the term FF (w,,)) refers the sample-to-label mixup regularizer on the label space of the source domain (e.g., 1000 classes when the model pre-trained using ImageNet)..

---

Published in Transactions on Machine Learning Research (01/2023)
Specifically, given deep neural network f, denote its classifier output as and deep feature g. Since both and y depend on the input data and their corresponding parameters, they are formulated as functions for simpleness. Thus, the sample-to- feature mixup regularizer based on target and source models is defined
as
11. (Mix Zy): -M (xi 9. E L (FB(w, ArBeeta(..a) ma -
(5)
where ((x;iw) refers to the CNN feature extractor output based on weight w and the sample æ;. This term encourages DNN to learn lincar behaviors from samples to hidden features.
Further, the sample-to-label mixup regularizer based on target and source models on the label space of source domain is defined as
(w)()))))= A=Beta(a,a) Ba 1 (Ix -=xx
(6)
where (((w) w) refers the output of an auxiliary classifier [Fully-Connected of the target model with x; on and (x) refers to the classifier output of the source model with æ; on was Both classifiers zf and as are in the source domain (e.g., with softmax outputs in 1,000 dimensions when the model is pre-trained using ImageNet).. More specifically, the FC layer in z is also initialized with the weights of the FC layer in the pre-trained source model w..
The vanilla sample-to- torlt-tt--lbboo label mixup regularizer L MXP derived from the standard implementation of mixup strategy (Zhang et al. 2018) based on target model using the target dataset D. Algorithm presents the design of the overall training procodure of SMILE.
3.4 Incorporating with a Mean Teacher
Note that an advantage of our method is the compatibility with mean teacher, which widely used semi-supervised learning problems (Tarvainen Valpola 2017) to generate pesudo labels. So far as we know, it has hardly been utilized in deep transfer learning promote pesduo features. Specifically, we periodically update the source model w, by the fine-tuned model in moving average manner.
Algorithm 1: Deep Transfer Learning with SMILE
Input D: target training data, Ws: pre-trained source model, yy: learning rate, K: training iterations, a: hyperparameter for Beta distribution;
Output: wh final learned target model after K iterations;
begin
wPt + W/ for
K do
'*Data Sampling and Mixing*/
B + mini-batch sampling from D
A~ Beta(a, a)
Bm + X (1 A) Shuffle(B)
8
/*Calculate Vanilla Mixup and Sample-Feature Mixup Loss*/
Calculate MXP based on (Bmmww/-)
Calculate L FE based on 9e (Bm, w.) and gi gi (B, wi k-1)
10
Calculate FC based 2, (Bm, w.) and 2 (B. w k-1) /*Updating the Target Model with SGD*/
11
12
C(wk-1 ..as) "FE LF + TPC FC + MXP
13
ge + VL(w k-1 (wa)
14
wt + wk 11-66
15
16
return wi

---

Published in Transactions on Machine Learning Research (01/2023)
4
Experiments
We evaluate our method a wide range of tasks, covering different kinds of datasets, pre-trained models, data scales and model architectures. Through exhaustive experiments, SMILE is compared against multiple
state-of-the- fine-tuning algorithms including L (Donahue et al. 2014). L Li et al. 2018). DELTA LLi
itt al. 2019). BSS (Chen ct al. 2019), RIFLE (Li ct al. 2020), Co- Tuning (You et al. 2020 and RegSL
& Zhang 2021 To achieve a comprehensive evaluation, we also compare our method with relevant data-
augmentation strategies including Mixup (Zhang et al. 2018). Manifold Mixup (Verma al., 2019) and
CutMix Yun et al. 2019
Table 1: Characteristics of the target tasks.
target dataset
task category
source task
architocture
# training
# classes
CUB-200-2011
Object Recognition
ImageNet
Res.eet-50
5,994
200
Stanford-Cars
Object Recognition
ImageNet
ResNet-50
8,144
196
IGVC--Aircaal
Object Recognition
ImageNet
Res.ett-50
6,677
100
MIT-Indoor-67
Scene Classification
Places365
ResNett-50
5,356
76
Food-101
Object Recognition
ImageNet
Efficient
75,000
101
4.1
Image Classification
We first present the experiment results based on image classification tasks using wide range of transfer learning algorithms and datascts.
4.1.1 Datasets and Models
We conduct experiments on three popular object recognition datasets: CUB-200-2011 (Wah et al. 2011
Stanford Cars (Krause et al., 2013) and FGVC-Aircraft (Maji et al. 2013), which are intensively used in -fate-of-theeart transfer learning literatures (Chen al.] 2019 Li al. 2020: You al. 2020). Each of these datasets contains about 6k 8k training samples. We ImageNet (Deng et al. 2009) pre-trained ResNet-50 (He et al., 2016) as the source model. For each dataset, create four subsets with different number of calegories and training examples, divided into two experimental groups. For the first group, we first randomly select 25% of all the categories from each of these standard datasets. Then we randomly sample 400 and 800 training samples from the selected categorics. For the second group, we use all categorics, while evaluate with 15% or 100% training samples respectively, following the practice in existing baselines BSS (Chen al. 2019) and Co-tuning (You et al., 2020). Among these, the last setting uses the entire dataset, which reflects the performance when adaptation is relatively sufficient, while the remaining three rely more knowledge transfer from the pro-trained model. The first group simulates real world datasets with relatively fewer categories and more instances pre category, while the second group (15% data.) simulates the opposite.
To further confirm the performance improvement by SMILE is independent with the choice of pre-trained datasets and model architectures, we conduct additional experiments comparing our method with competitive basclines. Spocifically, we use the Places365 [Zhou et al. 2017) pre-trained ResNet-50 to perform fine-tuning MIT-Indoors-77 (Quattoni & Torralba, 2009), which is a scene classification task. We also evaluate our method more powerful model EfficientNet-B.. (Tan & Le 2019) designed by NAS over large scale dataset Food-101 (Bossard et al. 2014). The descriptions about the benchmarks used in image classification tasks are summarized in Table
4.1.2 Training Details
We apply standard data augmentation strategics for image pre-processing composod resizing to 256 x 256, random flipping and random cropping to 224 224 during training. For inference, the test image is resized 256 256 and then center cropped to 224 x 224. We do not use post-processing methods such ten-crop
8

---

Published in Transactions on Machine Learning Research (01/2023)
Table 2: Comparison of top-1 accuracy (%) on transfer learning benchmarks. The notation C:X/N:Y refers using examples from X selected categorics.
Dataset
Dataset
Method
C:25%//N:400
C:25%/N:800
C:AlI/N:15%
C:AMI/N:100%
L DDnahue al.
2014
55.59:+1002
71.85+0.12
11.70:10.17
80.61+0.30
Mixup (Zhang et a.
2017
52.39=t0.68
73.02#0.11
44.27=00.11
81.86+0.20
Manifod Mixup Verma al.
2019
55.38=+0.16
74.09::0.49
49.57:0.30
83.09
CUB-200-2011
CutMix Yun et al.
2019
28.08
59.89+0.94
29.73+0.26
81.52+0.255
LA-SP (Li et al.
2018
54.38:10.32
73.90.10.22
45.30110.23
81.5810. 10
DELTA Li et al. 2019
58.15:10.26
75.84110.08
17.88.0..15
02.2110.15
BSS Chen et
2019
54.99-+0.73
74.14+0.34
46.41++0.09
81.10+0.04
RIFLE Li al.
2020
53.68++0.89
73.05++1.09
44.13+-0.38
81.94+0.06
Co- Tuning You et a.
2020
37.98:10.08
75.11=0.447
49.98:+0.23
82.60+0.03
RegSL Li & Zhang
2021
57.62=10.88
75.51=0.44
46.92::0.28
80.20
SMILE
62.13:10.55
77.27+0.35
51.73:10.0
83 83.62=0.07
L.2 (Donahue et al.
2014)
61.17=0.36
82.73+0.59
43.01+0.53
90.14+0.12
Mixup Zhang al.
2017
60.25+00668
83.60+0.02
45.73+0.15
91.51
Manifod Mixup
Verma et al.
2019
64.38110.73
85.0110.18
50.53
91.8810.16
Stanford-Care
CutMix Yun et al.
2019
47.8910.77
76.7110.36
37.6210.14
92.5610.200
T.-SP Li et al.
2018
61.00++0.28
82.05+00.05
44.12+0..3
90.61+0.12
DELTA Li et al. 2019
62.05-+00.3
82.1+0.44
43.27+00.27
90.86++0.08
BSS Chen et a.l 2019
61.97-00.69
83.81+0.399
47.45+00.23
91.14+0.01
RIFLE [Li et al.
2020
62.85+0..22
83.5740.43
43.61+0.07
91.08
Co Tuning You al.
2020
66.05+0.41
01.05+0.399
44.29:2..42
91.19 11
RegSL Li & Zhang
2021
60.
82.9110.08
42.52.10.37
91.02.10.05
SMILE
15.1711.111
85.90.10.16
50.93
92.2110.005
L Donaluue et al.
2014)
19.6
79.5711.118
51.1310.45
88.2710.51
Mixup Zhang et al.
2017
65.20110.80
84.53.10.62
54.4211.55
89 19.3331..1 0.17
Manifod Mixup Verma et al.
2019
61.1010.56
80.40:11.90
57.97
89.5310.24
FGVC-Aircraft
CutMix
Yun et al. 2019
53.50++0.80
77.60++0.73
11.10+0.78
08.28
I.2-SP [Li et al. 2018
51.70--00733
76.13-10.82
48.85+00.70
87.97+0.66
DELTA Li al. 2019
53.47=0.24
71.73+1.02
51.05
88,92=0.25
BSS Chen et al. 2019
61.40+1.18
81.47+0.24
52.61+0.111
88.47
RIFLE E
(Li al. 2020
60.97-10.49
79.87110.38
52.1311..31
89.45.10.41
Co- Toming You et al.
2020
62.98.10.72
80.03-10.04
52.05110.43
88.19:00.33
RegST. [Li & Zhang
2021
61.87+0.37
09.40+
51.6440.43
88.87+0.26
SMILE
68.40-+0.33
84.57+0.29
60.04+0.33
90.16++0.15
ensemble (Liang et al., 2020). We train all models using SGD with the momentum of 0.9, weight decay of 10-1 and batch size of 48. We train 15,000 iterations for Food-101 considering its large scale and 9,000 iterations for the remaining datasets. The initial learning rate is set to 0.001 for MIT-Indoor-67 due to its high similarity with the pre-trained dataset Places365 and 0.01 for the remaining. The learning rate is divided by 10 after two-thirds of total iterations. Each experiment is repeated five times and we report the average top-1 classification accuracy and standard derivations for uncertainty quantification.
For hyper-parameles search, we use a simple three-fold cross validation on the original training set from "FE E [0.01, 0.1] and 7FC [0.01, 0.1]. The selected best configurations are used in all experiments. As for bascline methods, use the recommended choices of hyper-paramcters reported in corresponding papers.
We notice that CutMix performs surprisingly worse in low-data regime (e.g., less than 1000 training examples in total), However, when fine-tuning with full data, CutMix always outperforms vauilla fine-tuning. This phenomenon is consisteut with our preliminary experiments Introduction. conjecture that, when training data insufficient, operation cutting and replacing patches might cause much severer over-fitting risks. The CutMix results can be reproduced by our published code.
9

---

Published in Transactions on Machine Learning Research (01/2023)
Table 3: Comparison of top-1 accuracy (%) with different transfer learning algorithms on more task types and architectures.
Sampling Rates
Datasel
Method
30%
50%
100%
L Donahue et al.
2014
78.68-1020
80,80=0.11
82.00:10.21
Mixup Zhang
2017
77.44:0.44
80.2810.28 228
82.87=0.50
MIT-Indoor-67
DELTA (Li et al.
2019
80.80++0.22
82.80=10.25
83.67-00.18
BSS Chen
2019
78.23:10.50
80.35:10.28
82.15=0022
RIFLE E [Li et al.
2020
76.76:10.08
78.71-10.33
81.78:10.07
SMILE
82.00+0.144
83.54+0.200
85.37+0.16
L DDnnhuue et al. 2014
80.25:10.28
83.43-10.15
86.77-10.03
Mixup Zhang et 201
82.63+0.11
84.93+0.06
87.82+0.06
Food-101
DELTA [Li et al. 2019
0..8
84.07+0.06
87.34+0.07
BSS Chen et al. 2019
81.13:0.04
83.9610.09
87.3310.03
RIFLE [Li et al.
2020
81.13:0.004
83.8240.02
87.2910.11
SMILE
82.84+0.16
85.25-0.09
88.2011.110
4.1.3 Results
As observed in Table 2 our proposed SMILE achieves remarkable improvements to vanilla fine-tuning on three standard benchmarks, and outperforms all state-of-the-art methods. As the size of training data becomes smaller, our method yields more significant benefits, e-g. SMILE outperforms vanilla fine-tuning by more than 8% on FGVC-Aireraft when only 15% training samples are used.
Our method is shown scalable to more challenging datasets. For MIT-Indoor-67, vanilla fine-tuning with small learning rate is quite competitive as the pre-trained model is highly adaptable for the target task. While for large-scale dataset Food-101, the benefit from all fine-tuning algorithms becomes less. As shown in Table SMILE still delivers decent performance on these datasets.
As for the time complexity, although SMILE requires an extra forward pass, the actual running time increases less than 20% against vanilla fine-tuning.
4.2 Natural Language Processing
We also evaluate SMILE the text classification task using powerful transformer-based architecture, showing that our method can be applied to NLP tasks.
4.2.1 Datasets and Models
To carry out experiments of transfer learning for NLP tasks, use the fine- fine-grained sentiment classification task SST-5, which offers the Stanford Sentiment Treebank datascts with five categorics. The pre-trained model in this experiment is base model of BERT (Devlin et al. 2018 with 12 transformer blocks and 12 attention heads.
4.2.2 Training Details
We fine-tune the pre-trained BERT model with the batch size to 24 for 3 epochs, using Adam optimizer with a learning rate of 2 x 10-5 while incorporating with deep transfer learning algorithms including L and
BSS (Chen al. 2019) and vanilla Mixup.
4.2.3 Results
Results are shown Table 4 where we include performance based on LSTM Tai 2015). CNN Kim 2014), and the vanilla. BERT base reported in (Munikar al.) 2019) for comparisons. We also find that both
10

---

Published in Transactions on Machine Learning Research (01/2023)
mixup and SMILE outperform standard fine-tuning and SMILE achieves more improvements. Regularizers L2-SP and BSS without mixup are not superior to standard fine-tuning in this task.
5
Analysis
5.1
Ablation Study
We here present ablation study to exhibit the unique contribution corresponding to cach component in our framework. Specifically, we evaluate the performances of SMILE by removing F and L respectively. As observed in Table 5 while they both make non-trivial contributions, the influence of the sample-to-feature mixup regularizer L is more significant than the feature-to-label regularizer LPC.
Furthermore, we consider simple combinations of mixup strategies and state-of ahe transfer learning algorithms. Specifically, we employ DELTA (Li et al. 2019) to improve knowledge distillation and RIFLE (Li et al. 2020 BSS (Chen et al., 2019) to alleviate negative transfer, both on top of the vanilla mixup. As shown in Figure 3 though such straightforward combination with vanilla mixup sometimes improves either transfer learning and mixup, SMILE is still significantly superior.
Performance on CUB-200-2011 for DELTA
Performance on CUB-200-2011 for BSS
Performance on LUB-200-2011 for RIFLE
85
85
75
75
Moan Accurazy
65
65
65
60-
60
L
Mixup
Mixup
Mixup
50
SMILE
50
SMILE
50
SMILE
DELTA+
US54Moxup
MILLD
15%
30%
50%
100%
15%
50% Ratio Labeled Data
30%
100%
15%
50% Ratio Labeled Data
30%
100%
Ratio of Labeled Data
Figure 3: Comparison of top-1 accuracy (%) with various SOTA transfer learning baselines combined with Mixup.
5.2 Discussions Linear Interpolation Effects
One major assumption of our work is that fine-tuning with pre-trained models would overfit to mized-up samples and labels. In order to verify this assumption and interpret the performance improvement of SMILE, we now investigate the linear interpolation effects in fine-tuning. It worth noting that, the terminology overfit and generalize this section are particularly w.r.t. the linear interpolation, rather than the model accuracy in general sense. For example, model overfits mized-up samples and labels means, the model remembers all interpolated labels for the mixed training samples, but mixture of two test samples fails to get the prediction that lies on the linear interpolation of their respective predictions.
Table 4: Experimental results on NLP task SST-5.
Table 5: Ablation Study on the CUB-200- 20B-200-2011 dataset.
Methods
Accuracy
LSTM (Tai et al.
2015
46.4
Sampling Rates
CNN (Kim
2014
48.0
Methods
30%
100%
BERT hase
Dev li et al.
2018
53.2
BERT bass
w. Mixup
53.7
SMILE
70.42
83.62
BERT base
W. LP-SP (Li et al.
2018
53.2
SMILE w/o. LF
69.68-10.12
83.1110.19
BERT base
W. BSS (Chen
2019
53.4
SMILE w/o. FE
82.92+0.20 10.127..27
BERT base
SMILE
54.6
11

---

Published in Transactions on Machine Learning Research (01/2023)
5.2.1 Measuring Linear Interpolation Effects
To quantify the linear interpolation effects, we define metric to quantify the effect of linear interpolation. Derived from standard mixup (Zhang et al. 2018), we introduce a generalized form interpolation loss (TI.) w.r.t a function f employing its own outputs as labels, eliminating the influence of the faithfulness of the approximation, i.c. how the learned function fits the ground truth f*, as follows:
IL(
BAn ( (0 f(Mixs, (2, (Mixx, x1)),
where D refers to the Euclidean distance between the output w.r.t the interpolated inputs and the proportionally mixed outputs. À conforms to the Beta distribution as described in Section 3.2 and 61, ó2 are sampled from a uniform distribution between and 1. Note that Equation 7 is a metric for evaluating to what degree does a model favor linear behaviors, rather than an optimization objective.
Compared with original form of linear interpolation loss used in standard mixup (Zhang et al. 2018 the metric defined in Equation has the following two merits.
Equation is feasible for measuring linear interpolation effects for both the feature layer (noted as Feature-II, IL when considering the CNN fcature extractor as the function f) and the label outputs (noted Label-IL. when considering the classifier's outputs f).
Equation 7 relics on the network's own outputs rather than external labels, and thus the influence of label fitting (e.g. model accuracy) is disentangled from the evaluation of linear interpolation.
5.2.2 Sample-to-Feature Mixup: Linear Interpolation Effects and Generalization
We use Eq to measure Label-IL using the classifier outputs and Feature-IL using the last hidden layer of ResNet-50 for different transfer learning methods, with CUB-200-2011 (with 30% sampling rates) as the training set for all methods. Several arguments can be deduced from results in Table
More Data, Better Generalization, and Lower Label-IL and Feature-IL. There no doubt to assume that, in practice, a model trained with more data should enjoy better generalization performance. addition to improve the testing accuracy we find that, when we involve additional training samples, both Label-IL and Feature-IL would be lower on the testing sets, compared vanilla fine-tuning.
Fine-tuming with vanilla mirup is NOT generalizable even in the label space, due to the lack of linear interpolation in feature spaces. As shown in Table 6 although Label-IL of the vanilla mixup is significantly lower on the training set than other methods, its Label-IL is high on the testing set (not generalizable). Furthermore, compared to other methods on both training/ training/testing testing sets (even Fine-tuning on the testing set), Feature-II. of the vanilla mixup high, i.c., poor linear interpolation in feature spaces.
Cempleetoo-eattree Mirup could ensure the generalizability of micup effects in the label space, as SMILE is with low Feature-Il. IL and Label-IL on both training and testing sets. While SMILE achieves the lowest Feature-IL on both training and testing datasets, it also achieves the lowest testing Label-IL. The comparisons with vanilla mixup suggest that doing mixup in the label space is just not enough for fino-tuning. Besides the quantitative results, we in Appendix B present some visualized cases of interpolation behaviors the feature space with different fine-tuning methods.
These arguments solidify our motivation of sample-toffetuure mixup for fine-tuning
Note that the over-fitting in the label space may not be directly calihrated with training /test accuracy there exists other factors influence the accuracy, e.g. mixup also benefits from the effect of label smoothing (Singh Bay 2019).
12

---

Published in Transactions on Machine Learning Research (01/2023)
Table Feature-TI, and Label-IL. for different fine-tuning methods over the training (sampling the CUR-200- 2011 training set by 30%) and testing dataset. Lower better. Add. Data refers involving the remaining
70% training examples for fine-tuning. However, the interpolation loss for the training set still calculated the original 30%.
Label-IL
Feature-IL II
Method
Train
Test
Train
Test
Finetune
1.80
1.92
1.92
1.93
Finetune + Add. Data
1.85
1.88
1.58
1.63
Finetune + MXP
1.65
2.00
1.98
2.02
SMILE
1.75
1.82
1.48
1.53
5.2.3 Feature-to-Label Mixup: déjà vu can help.
To enforee lincar behaviors on features, SMILE inherits the Feature-to-Labe classifier (i.c., the FC layer) from the source model as the initialization of fine-tuning. It is because we assume that the label space of the source task is partially overlapped with the target task. Thus, the FC layer with a considerable number of parameters contains useful information for the target task. This has been investigated by correlating the label spaces between these two tasks in recent studies (You al. 2020). Furthermore, although it is impossible for SMILE to mix the feature vectors extracted from the fine-tuned CNN during the fine-tuning procedure, Feature-to-Labee Label Mixup still works well the FC classifier has been well-trained on the source datasets, which provides rich semantic information.
5.3 Discussions on Catastrophic Forgetting
Here we present discussions to confirm our hypothesis stated in the introduction, iee. the vanilla mixup aggravates the risk of catastrophic forgetting, while our approach alleviates this problem by reusing rich source knowledge. As suggested by existing literature (Li al.) 2018 Gouk al.] 2021 Li & Zhang 2021), use the parameter distance between the fine-tuned and pre-trained model to measure the degree of catastrophic forgetting. We use the same experimental setting as that used in Section 5.2 As shown in Table 7 both mixup and SMILE get the parameter distance even larger than vanilla fine-tuning with double training examples, while SMILE alleviates the deviation caused by mixup.
Table 7: Parameter distance between the fine-tuned and pre-trained model with different methods. The distance is calculated as the summation of the distance w.r.t. each layer, measured by the Euclidean distance between two tensors. The FC layers are not included when calculating the distance.
Method
Finetunc Finctune (2x data) Finetuno+mixup Finetume+ SMILE 32.3
Distance
37.3
90.1
60.7
5.4 Role of Source Model
Our method is effective particularly transfer learning, and a reliable teacher model is vital to the proposed framework. In the setting of fine-tuning, the source model acts as good starting point to provide supervision for both cross-domain mixed labels and in-domain mixed features. Thereby, our method is not directly feasible for general-purpose supervised learning. The reason two folds.
In general supervised learning, i.e., learning a single task from scratch, the cross-domain sample-to-label regularier FFFC is no longer applicable, since no auxiliary tasks are available. This makes our framework incompatible with general supervised learning.
The source model also plays an essential role for supervision of mixed features. If the teacher not trustworthy enough; the effects of fampte-to-feature mixup cannot. be guaranteed. We design two groups of experiments to
13

---

Published in Transactions on Machine Learning Research (01/2023)
verify this. In the first group; evaluate SMILE with degenerated teachers, including without knowledge preserving from the source model, and the other without adaptation to the target data. They are denoted by "w/o Source' and 'w/o Target' respectively. The second group simulates general supervised learning, where only original mixup and sample-to-feature mixup are involved in SMILE. Specifically, results in Table g show that. preserving the source weights contributes more than adapling to the targel data, terms sample-to-fetture mixup. Experiments on the second group further backup the analysis. As shown in Table when training from scratch, SMILEis inferior to vanilla Mixup, indicating that the sample-to-feature component. has negative influence if using low-quality teacher.
Table 8: Evaluation of fine-tuning CUB-200-201 C:25%/N:800..
Mixup 73.02
SMILE SMILE w/o Target
SMILE w/o Source
77.27
75.72
73.45
Table 9: Evaluation of training from scratch on CUB-200-2011 C:25%/N:800.
T
Mixup
SMILE
16.97
25.36
21.86
6 Conclusion
In this work, we figure out the difficulty of applying mixup to transfer learning, and introduce SMILE-Sample to-feature Mixup strategies for Efficient. Transfer Learning. Beyond direct combination of fine-tuning and mixup, SMILE pursues generalizable linear behaviors through incorporating both features of the target domain and the label space of the source domain. We conduct extensive experiments using a wide spectrum of target datasets. Results show that SMILE can significantly promote the effectiveness of fine-tuning and outperform various competitive fine-tuning algorithms. Ablation studies and empirical discussions further backup our design intuition and purposes.
References
Abulikemu Abuduweili, Xingjian Li, Humphrey Shi, Cheng-Zhong Xu, and Dejing Dou. Adaptive consistency regularization for semi-supervised transfer learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Paltern Recognition, pp. 6923 6932, 2021.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 mining discriminative components with random forests. In European Conference on Compuler Vision, 2014.
Xinyang Chen, Sinan Wang, Bo Fu, Mingshong Long, and Jianmin Wang. Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer learning. In Advances in Neural Information Processing Systems, pp. 1906-1916, 2019.
Yin Cui, Yang Song, Chen Sun, Andrew Howard, and Serge Belongie. Large scale fine-grained categorization and domain-specific transfer learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4109-4118, 2018.
Jun Deng, Wei Dong, Richard Socher, Li-Jia Li, Kuntai Li, and Li Fei-Fei. Imagenet: A largo-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248 255, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 2018.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: deep convolutional activation feature for generic visual recognition. In International conference machine learning, pp. 647-655, 2014.
14

---

Published in Transactions on Machine Learning Research (01/2023)
Weifeng Ge and Yizhou Yu. Borrowing treasures from the wealthy: Deep transfer learning through selective joint fine-tuning. In Proceedings of the IEEE conference on computer vision and paltern. recognition, pp- 10 19, 2017.
Henry Gouk, Timothy Hospedales, et al. Distanco-based rogularisation of deep networks for fino-tuning. International Conference on Learning Representations, 2021.
Boris Hanin and David Rolnick. Complexity linear regions deep networks. In International Conference on Machine Learning, pp. 2596 2604, PMLR, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Yoon Kim. Convolutional neural networks for sentence classification. corr abs/1408.5882 (2014). arXiv preprint Xr.ie:1408..588 5882, 2014.
Jonathan Krause, Michael Stark, Jia. Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13),
Sydney, Australia, 2013.
Dongyue Li and Hongyang Zhang. Improved regularization and robustness for fine-tuning in neural networks. Advances in Neural Information Processing Systems, (127249-27262 2021.
Xingjian Li, Haoyi Xiong, Hanchao Wang, Yuxuan Rao, Liping Liu, and Jun Huan. Delta: Deep learning transfer using feature map with attention for convolutional networks. arXiv preprint or wrXi::1901.0999 2019.
Xingjian Li, Haoyi Xiong, Haozhe An, Cheng-Zhong Xu, and Dejing Dou. Rifle: Backpropagation depth for deep transfer learning through re-initializing the fully-connected layer. In International Conference on Machine Learning, pp. 6010 6019. PMLR, 2020.
Xuhong Li, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. Thirty-fifth International Conference Machine Learning, 2018.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference Machine Learning, pp. 6028-6039. PMLR, 2020.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. ArXiv, abs/1306.5152 2013.
Manish Munikar, Sushil Shakya, and Aakash Shrestha. Fine-grained sentiment classification using bert. 2019 Artificial Intelligence for Transforming Business and Society (AITB), volume pp. 1-5. IEEE, 2019.
Sinno Jialin Pan, Qiang Yang, al. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 131003:334--,35, 2010.
Ariadna Quattoni and Antonio Torralba. Recognizing indoor scenes. In 2009 IEEE Conference Computer Vision and Pottern Recognition, pp. 413-420. IEEE, 2009.
Aditya Singh and Alessandro Bay. On mixup training: Improved calibration and predictive uncertainty for deep neural networks neurips reprodueibility challenge 2019. 2019.
Kai Sheng Tai, Richard Socher, and Christopher D Manning. Improved semantic representations from troo-structured long short-term memory networks. arXiv preprint UrXiv:1503.00075 2015.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pp. 6105-6114, PMLR, 2019.
15

---

Published in Transactions on Machine Learning Research (01/2023)
Antti Tarvainen and Harri Valpola. Mean teachers better role models: Weightt-aeraged consistency targets improve semi-supervised deep learning results. In Proceedings of the Advances in newral information processing systems, pp. 1195 1204, 2017.
Vladimir Vapnik. The nature of statistical learning theory. Springer science business media, 2013.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, loannis Mitliagkas, David Lopez-Pad and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, pp. 6438-6445 PMLR, 2019.
Catherine Wah; Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd hirds-200-2011 dataset. 2011.
R. Wan, H. Xiong, X. Li, 7. Zhu, and J. Huan. Towards making deep transfer learning never hurt. Tn 2019 IEEE International Conference on Data Mining (ICDM), pp. 578-587, 2019.
Rongguang Wang, Pratik Chaudhari, and Christos Davatzikos. Embracing the disharmony in medical imaging:
A simple and effective framework for domain adaptation. Medical Image Analysis, 76:102309, 2022.
Ximei Wang, Jinghan Gao, Mingsheng Long, and Jianmin Wang. Self-tuning for data-elficienl deep learning. In International Conference on Machine Learning, pp. 10738-10749, PMLR, 2021.
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volune 2, 2017.
Kaichao You, Zhi Kou, Mingsheng Long, and Jianmin Wang. Co-tuning for transfer learning. Advances in Neural Information Processing Systems, 33, 2020.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 0123-6032, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 7710.094411 2017.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz Paz. mixup: Beyond empirical risk minimization. In International Conference Learning Representations, 2018. URL https://openreeiew
net/forum'id==1
Linjun Zhang, Zhun Dong, Kenji Kawaguchi, Amirata Ghorbani, and James Zou. How does mixup help with robustness and generalization? arXiv preprint arXiv:2010.0488 04819, 2020.
Jincheng Zhong, Ximei Wang, Zhi Kou, Jianmin Wang, and Mingsheng Long, Bi-tuning of pre-trained representations. arXiv preprint orXio:2011.06180 2020.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE tronsactions on pattern analysis and machine intelligence, 40(6): 1452 1464, 2017.
A Detailed Results for Figure 1
Here we provide detailed results for Figure 2 with the mean accuracy and standard deviation among five random trials for each experiment. We additionally report performance for 200 training samples for clearer comparison. Results in Table 10 clcarly show that, in transfer learning, Mixup performs worse (compared against the baseline) as fewer training samples are available. However, when training from scratch, Mixup consistently improves the performance with large margin.
16

---

Published in Transactions on Machine Learning Research (01/2023)
Table 10: Accuracies with standard deviations corresponding Figure 2 where fine-tuning ImageNet pro-trained checkpoint to adapt the new task. ST refers to standard training, Le. training from random initialization.
TL refers to transfer learning,
Size of Training Sct
Dataset
Method
200
400
600
800
1000
Baseline
32.98:10.13
55.59:11.02
68.24.10.35
74.85.10.12
78.83.10.42
CUB(TL)
Mixup
37.25=t0118--57777
(..39-t0.68(-3.200
65.72=00.08
73.02+0.11(-1888)
77. (7.44=0.14(11339)
Baseline
38.13+00.00
61.17+0.36
76.22=0.44
82.73+0.59
87.01=0.07
Cars(TI)
Mixup
33.85+0.222(4.28)
60.25+:0688(-00
(7.45
83.60+0.02(+0.887
88.34+0.22(
CUB(ST) Baseline
9.37=0
16.45:1.211
19.06::066
26.11=3.24
Mixup
16.07:1 (.20(++))0)
24.87 04.87
32.49:1..22 1.2((113333333
3..53:10.96( (69442)2
Baseline
9.96
9.89112.80
15.61.1..15
19.53.10.68
Cars(ST)
Mixup
16. 29.29.2.222 7723))
20.36.11.92( 10.47)
24.77.10.41(1 9.16)
35.24 15.71)
B
Demonstrated Effects of Feature Interpolation
Here we present show cases for comparison in feature interpolation among different algorithms. To obtain the interpolation points, we first randomly select a pair of images and then generate five mixed inputs with the interpolation coefficients of [0.6, 0.7, 0.8, 0.9, 1] respectively. Forward computation is performed given these mixed inputs and then, their corresponding deep features are extracted and projected to the 2-D space using PCA. We plot the interval of X 0.5 for better demonstration of projection the same sample dominates the interpolated result when all À lie in either (0, 0.5) (0.5, 1). Results of four random pairs are illustrated in Figure
17

---

Published in Transactions on Machine Learning Research (01/2023)
Feature Extractor
42
240
43
Standard line-tumi tuning
2.0
Line lininnging w/ Mixup
80
SMILE
05
Figure 4: Visualizations of leature interpolation behaviors for different fine-tuning methods. We extract the representations from the last hidden layer which are 2048 dimensional feature vectors and then project them into the 2-D space using PCA. Each column corresponds to the projected deep features generated by putting forward the interpolation of random pair of images. The number marked next to the point refers to the interpolation coefficient X.
18


Reviews:
[{"review_id": "WIqDLD2s2E", "reviewer": "Reviewer_AUGU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n+ Proposes an empirically sound methodology for leveraging Mixup in the context of deep transfer learning. \n+ Reasonable gains in terms of empirical performance.\n\nWeaknesses: \n- The motivation for why deep transfer learning would benefit from Mixup is not clearly discussed. In particular, the introduction section of the paper is a bit confusing and doesn’t set the proper context.\n- For a more comprehensive comparison of existing methods, it would be interesting to see the results when the proposed Mixup methodology is added to more recent transfer learning approaches like Co-Tuning, RegSL considered in the paper.\n\nQuestions: \n* Comparison with existing methods: Are all the existing methods re-trained on the data splits generated for the experiments, or was a standard transfer learning benchmark used?  Using benchmarks makes the comparison fair. For example, Guo et al., 2018, SpotTune [1], uses the Visual Decathlon benchmark. There are more recent benchmarks like Jiang et al. 2020 [2]\n\n* “As the size of training data becomes smaller, our method yields more significant benefits,” -- As shown in the results, as the data size increases, the gains decrease, does this mean as the architecture is relying more on the pre-trained features instead of learning features specific to the target domain?\n\n[1] Guo, Yunhui, Shi, Honghui, Kumar, Abhishek, Grauman, Kristen, Rosing, Tajana, and Rogerio Feris. \"SpotTune: Transfer Learning through Adaptive Fine-tuning.\" arXiv, (2018). https://doi.org/10.48550/arXiv.1811.08737.\n[2] Junguang Jiang, Baixu Chen, Bo Fu, Mingsheng Long, Transfer-Learning-library, Github repository, 2020, https://github.com/thuml/Transfer-Learning-Library\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work primarily focuses on making Mixup strategies work for deep transfer learning. While the motivation why Mixup should be used for transfer learning is not very clear, the results seem promising.  The proposed method efficiently leverages existing techniques like Mixup, Mean Teacher to improve performance in transfer learning. While the novelty is limited, it’s a simple solution that works in practice.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Experiments and Writing:\n* The introduction section is confusing and doesn’t motivate “Why deep transfer learning would benefit from Mixup?”. I understand there are marginal gains, but what would be the primary advantage of using Mixup in Transfer Learning context? This needs to be motivated better.\n* Experiments to show the effectiveness of the proposed Mixup strategy when used in combination with existing SoTA transfer learning methods like Co-Tuning and RegSL\n* Details on the experimental setting like data splits, was a transfer learning benchmark used? If not, why? \n\nMinor Issues: \n* Error in Figure-1 legend: while the plot has dotted lines, the legend doesn’t have any dotted lines.\n* Use of terms like source domain and target domain can be misleading and lead to questions and comparisons with domain adaptation; It might be better to use source dataset/target dataset instead of source domain and target domain. \n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4bJgy65kJL", "reviewer": "Reviewer_eYuc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n* The proposed technique is simple, yet empirical results seem to support its effectiveness for transfer learning compared to some other recent methods. (Unfortunately, I do not know additional state-of-the-art techniques comparison with which could be important here.)\n* Authors consider both the image and natural language domains (though with only one task).\n\n## Weaknesses\n\n* The overall level of manuscript preparedness is very low as if it was still a draft and not a finished paper. There are numerous errors and inaccuracies in the text from the language to mathematical notation and even the text structure (clearly wrong order of sections).\n* Many explanations in the text are poorly articulated and together with clear mistakes in mathematical notation, they make reading and understanding this manuscript exceptionally difficult.\n* The interpolation loss ${\\rm IL}(f)$ in Equation 6 is not clearly explained.\n   1. Firstly, when we speak of generalization, we typically consider one metric and analyze it for both training and test data distributions. For example, we can look at the model accuracy or the loss and compare their values on the training set and a hold-out test set to see whether overfitting occurs. Similar analysis could be carried out with $L^{\\rm FE}$ and $L^{\\rm FC}$. The difference of these losses on the training and test datasets would already characterize the degree of \"overfitting\" occurring on the training set and it is not entirely clear why we need another loss characterizing the linear structure on the test set.\n   2. Secondly, while it is clear that ${\\rm IL}(f)$ vanishes for linear functions $f$, one can also come up with other similar measures be that with only two samples (increasing the order / number of $\\delta$ terms), or more than two samples. Was this form of ${\\rm IL}(f)$ chosen because of its simplicity compared to other alternatives, or does it possess any other advantageous properties?\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a simple, yet seemingly effective technique for transfer learning based on mixup. The core idea is applying mixup to both sample-to-label (on both the source and target domains) and sample-to-feature maps. The authors validate the effectiveness of this technique in image classification (primarily for ResNet-50 models pretrained with ImageNet) and NLP benchmarks (fine-tuning BERT for the sentiment classification task SST-5). The paper includes a small ablation study showing the importance of both additional loss terms.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "In my opinion, the manuscript still needs a major editing effort in order for it to published. Authors might need to improve readability and fix the most obvious mistakes for the paper to be easy enough to comprehend.\n\n## Language\n\nIn the following, I list some examples of sentences that seem incorrect:\n* \"Thus, **there needs to estimate the feature vector** for any sample in the target dataset before having the CNN trained.\"\n* \"Later, the work ... introduces new algorithms that prevent the regularization **from the hurts to transfer learning**, where ... **proposes** to truncate the tail spectrum of the batch of gradients while .. proposes to truncate the ill-posed ..\"\n* \"multi-tasking algorithms\" -- not sure if the authors meant \"multi-task\".\n* \"In this way, **there might need a way** to make compromises between the features learned from both ...\"\n* \"They find that, fine-tuning can be substantially\" -- not sure about punctuation here.\n* \"Our ablation studies in Section 4 show that the simple combination of fine-tuning and mixup strategies **does not well in** transfer learning settings from both **linear behaviors preservation** and generalization performance aspects.\"\n* \"ten-crop ensemble\" -- not sure what this is and there was no reference provided. Is this a misprint?\n\nExamples of stylistic changes that would benefit the paper:\n* Sentence \"The selected best configurations are used for all configurations\" contains the word \"configurations\" twice.\n* \"As for baseline methods, we use the recommended choices of hyper-parameters reported in **their papers**.\" -- perhaps replace \"their papers\" with \"corresponding papers\"?\n* \"offering **tons** of well-trained features\", \"**blessed by the power** of large source datasets\" -- word \"tons\" and expression \"blessed by the power\" are in my opinion incompatible with the style of a scientific publication.\n* \"consider a more actual scenario\" -- perhaps \"relevant\" or \"practical\", or another word instead of \"actual\"?\n\n## Mathematical Notation\n\nHere we outline some mistakes in mathematical notation found on page 5 and additional questions that need to be clarified in the text:\n  * Authors introduce $\\omega^k_s$ for the $k$-th iteration of the model. But $\\omega_s$ is the source (presumably fixed) model. Should not it be $\\omega^k_t$, the target model instead?\n  * Underneath Equation (2) authors write $L^{\\rm Reg} (\\omega,\\omega_t,\\xi)$. Should it be $L^{\\rm Reg}(\\omega,\\omega_s,\\xi)$ instead?\n  * $L^{\\rm MXP}$ is mentioned, but is never defined explicitly. In my opinion, the expression for the final loss optimized by the authors has to be provided with _all_ of its terms.\n  * The spacing around minus sign in Equation (4) is confusing.\n  * Authors mention the fact that $\\lambda$ is sampled from the beta distribution, it would be interesting to know a little bit more about the reasons for this choice without referring to another publication.\n  * In Equation (4) we see both $g_t(\\cdot;\\omega_s)$ and $g_s(\\cdot;\\omega)$? Why are the $s$ and $t$ subscripts used on both $g$ and $\\omega$? Why does $g_t$ (presumably target features) depend on $\\omega_s$ (source weights) and $g_s$ (presumably source features) depends on $\\omega$ (target weights)? This is either a mistake or needs a clarification. Right now this is simply confusing.\n* Equation (6) misses a lot of closing brackets.\n* $P_\\delta$ is mentioned, but is not defined (is it the same distribution as $P_\\lambda$?).\n* \"1e-4\" in the text should be replaced with $10^{-4}$.\n\nPlease review the text thoroughly and fix all similar issues.\n\n## Structure\n\nThere are Sections 4.0.1 and 4.0.2 that seemingly belong to Section 4.1 and should thus be Sections 4.1.1 and 4.1.2.\n", "broader_impact_concerns": "I did not identify any concerns on the ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UA5n2lFlXa", "reviewer": "Reviewer_Dv9f", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper shows the effectiveness of the proposed SMILE in a wide range of datasets. It is convincing to me that SMILE is useful in fine-tuning.\nWeaknesses:\n1. Though good experimental results are shown in a wide range of datasets, it is still not clear why SMILE is a good solution. The whole story is based on the assumption that mix-up can benefit transfer learning. However, no explorations about the relationship between mix-up and transfer learning are shown. To further improve the paper, I suggest the authors explore what is the key factors in transfer learning and why mix-up can benefit the transfer learning process.\n2. This paper lacks insights. I think it should be some correspondences between research questions and experimental results. A good paper should solve the targeted problem and tell readers why the proposed method can solve the principal difficulty. There should be more clarification about why SMILE can solve the essential problem of transfer learning.\n3. Vanilla mix-up overfits in transfer learning, this is why SMILE is proposed. Insights about why mix-up overfits should be explored.\n4. Considering the above three weaknesses, it should be the case that SMILE is a combination of existing technologies and no extra insights are provided.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on the problem of transfer learning, the authors try to answer how to fine-tune pre-trained models to downstream tasks. Specifically, they find that mix-up overfits when fine-tuning deep neural networks. To solve this problem, they propose a mix-up based fine-tuning method, SMILE, to enable a better generalization in the downstream tasks.\nSpecific technical details: SMILE applies mix-up to both the input space and the output space of the training model (feature vectors and classifier outputs), and performs regularization to solve the problem of catastrophic forgetting. In the experiments, the paper compares SMILE with SOTA methods in the fine-tuning literature using image classification and NLP tasks, and shows that SMILE is effective in enhancing the generalization ability of deep models.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Though good experimental results are provided, this paper should add more insights to make the proposed method convincing:\n1. Why mix-up can benefit transfer learning?\n2. Why SMILE can solve the essential problem of transfer learning?\n3. Why mix-up overfits in transfer learning?\n4. For further improving the paper, The authors should reorganize the research question and give some correspondences between research questions and experimental results.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KOFJIkkVaP", "reviewer": "Reviewer_aFiM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The authors performed a lot of experiments on many datasets and tasks;\n- The proposed method shows substantial improvement compared to other baselines;\n\nWeaknesses:\n- The study is motivated by the observation that mixup doesn't help when transferring models to a small target dataset compared to training from scratch. But please note that the performance of a transferred model and the scratch model is at a very different scale. The authors also warn that mixup in transfer learning might be harmful. However, the difference of the performance between a pain transferred model and a mixup transferred model might be not statistically significant;\n- The proposed strategy is very similar to the proxy regularizer introduced in [1] but in a mixup setting. Perhaps the authors want to discuss the connection;\n- The methodology section is very hard to follow in general. The authors need to fix the logic and flow. For example, $L^{FE}$ and $L^{FC}$ are not defined in section 3.1;\n- Is there any theoretical justification of the proposed regularization method?\n- The caption of Figure 2 is not self-contained. Please elaborate on the design of the proposed architecture;\n- Where are the cross-domain (out-of-sample) generalization experiments mentioned in the introduction?\n\n[1] Wang, R., Chaudhari, P. and Davatzikos, C., \"Embracing the disharmony in medical imaging: a simple and effective framework for domain adaptation.\" Medical Image Analysis 76 (2022): 102309.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors found that mixup data augmentation technique doesn’t help much in a transfer learning setup especially when the target dataset is small. They therefore propose to regularize both sample-to-feature mixup and sample-to-label mixup by leveraging a mean teacher feature extractor and a classifier to boost the model’s performance on the target task. They perform experiments on several datasets for image classification and a natural language processing task. The proposed method outperformed a list of baseline methods.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- Justify on the motivation of the study;\n- Discuss the relationship of a similar work in the literature;\n- Improve the language and flow of the methodology section;\n- Theoretical analysis of the proposed method;\n- Add cross-domain generalization experiments;\n", "broader_impact_concerns": "No ethical concerns observed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies Sample-to-feature Mixup, an adapted Mixup technique for transfer learning, targeting on improving the data-efficiency of low target data regime. Four expert reviewers provided detailed and constructive comments for this paper; In correspondence, the authors provided a revised version with a large ratio of revised materials, which significantly strengthened the paper. The AE ensured that all reviewers took the revised version as well as the authors' response into consideration. In the final recommendation phase, two reviewers voted for acceptance but the other two reviewers voted for rejection. Two of them provided further information to justify their recommendations, which were highly appreciated. 

AE justified all the material as well as the recommendation as a neutral referee, and reached the conclusion of "Revision". The main concern is that, this paper is an empirical study of a new transfer learning technique, but the current experimentation lacks deeper insights and background mechanism of mixup for transfer learning. In particular, why such a variant of Mixup with sample-to-feature strategy is useful for transfer learning--is it also useful for general-purpose supervised learning? While a theoretical understanding is definitely hard since Mixup itself is a data augmentation technique, a thorough empirical investigation is needed, as suggested by the reviewers. Also, since the mixup-related work is fruitful in our community, fine-tuning + some Mixup variants (e.g. Manifold Mixup, MixCut) shall be studied to make the evaluation more complete. 

After confirmation with the EiC, AE would offer the authors with an opportunity for a revision. Authors shall be careful in addressing the major concerns.
Decision: Accept (poster)