Meta-learning Transferable Representations with a Single Target Domain
Authors: Hong Liu
Jeff Z. HaoChen
Colin Wei
Tengyu Ma
Venue:  (2021)
Keywords: transfer learning
fine-tuning
supervised transfer learning

Abstract: Recent works found that fine-tuning and joint training---two popular approaches for transfer learning---do not always improve accuracy on downstream tasks. First, we aim to understand more about when and why fine-tuning and joint training can be suboptimal or even harmful for transfer learning. We design semi-synthetic datasets where the source task can be solved by either source-specific features or transferable features. We observe that (1) pre-training may not have incentive to learn transferable features and (2) joint training may simultaneously learn source-specific features and  overfit to the target. Second, to improve over fine-tuning and joint training, we propose Meta Representation Learning MeRLin to learn transferable features. MeRLin meta-learns representations by ensuring that a head fit on top of the representations with target training data also performs well on target validation data. We also prove that MeRLin recovers the target ground-truth model with a quadratic neural net parameterization and a source distribution that contains both transferable and source-specific features. On the same distribution,  pre-training and joint training provably fail to learn transferable features. MeRLin empirically outperforms previous state-of-the-art transfer learning algorithms  on various real-world vision and NLP transfer learning benchmarks. 

Under review as a conference paper at ICLR 2021
META-LEARNING TRANSFERABLE REPRESENTATIONS WITH A SINGLE TARGET DOMAIN
Anonymous authors Paper under double-blind review
ABSTRACT
Recent works found that fine-tuning and joint training two popular approaches for transfer learning- do not always improve accuracy on downstream tasks. First, we aim to understand more about when and why fine-tuning and joint training can be suboptimal or even harmful for transfer learning. We design semi-synthetic datasets where the source task can be solved by either source-specific features transferable features. We observe that (1) pre-training may not have incentive to learn transferable features and (2) joint training may simultaneously learn source- specific features and overfit to the target. Second, to improve over fine-tuning and joint training, we propose Meta Representation Learning (MeRLin) to learn transferable features. MeRL.n meta-learns representations by ensuring that a head fit top of the representations with target training data also performs well on target validation data. We also prove that McRLin recovers the target ground-truth model with quadratic neural net parameterization and source distribution that contains both transferable and source-specific features. On the same distribution, pre-training and joint training provably fail to learn transferable features. MeRLin empirically outperforms previous state-of-the- art transfer learning algorithms on various real-world vision and NLP transfer learning benchmarks.
INTRODUCTION
Transfer learning transferring knowledge learned from large-scale source dataset to a small target dataset- an important paradigm in machine learning Yosiinkki et al., (2014) with wide applications in vision (Donahue et al. 2014) and natural language processing (NLP) (Howard & Ruder 2018 Devlin et 20T9). Because the source and target tasks are often related, we expect to be able to learn features that transferable to the target task from the source data. These features may help learn the target task with fewer examples (Long et al. 2013 Tamkin et al. 2020).
Mainstream approaches for transfer learning fine-tuning and joint training. Fine-tuning initializes from a model pre-trained on a large-scale source task (e.g., ImageNet) and continues training on the target task with a potentially different set of labels (e.g., object recognition (Wang al. [017]] Yang et al. 2018 Kolesnikov et al. 2019), object detection (Girshick et al.) 2014). and segmentation (Long al. 2015 He al. 2017 ). Another enormously successful example of fine-tuning is in
NLP: pre- pre-training transformers and fine-tuning on downstream tasks leads to state-of-the art results 2019). In contrast the two-stage optimization process of fine-tuning, joint training optimizes linear combination of the objectives of the source and the target tasks (Kokkinos) 2017 Kendall et al.
for many NLP tasks (Devlin et 2019 Yang al.
2017 Liu al. 2019b).
Despite the pervasiveness of fine-tuning and joint training. recent works uncover that they are not always panaceas for transfer learning. Geirhos al. (2019) found that the pre-trained models learn the texture of ImageNet, which is biased and not transferable to target tasks. ImageNet pre-training does not necessarily improve accuracy on COCO (He et al. 2018), fine-grained classification (Kornblith) et al. 2019), and medical imaging tasks (Raghu et al. 2019). Wu et al. (2020) observed that large model capacity and discrepancy between the source and target domain eclipse the effect of joint training. Nonetheless, we do not yet have systematic understanding of what makes the successes of fine-tuning and joint training inconsistent.
The goal of this paper is two-fold: (1) to understand more about when and why fine-tuning and joint training can be suboptimal or even harmful for transfer learning; (2) to design algorithms that overcome the drawbacks of fine-tuning and joint training and consistently outperform them.

---

Under review as a conference paper at ICLR 2021
To address the first question, we hypothesize that tinettunig and joint training do not have incentives to prefer learning transferable features over source-specific features, and thus their capability of learning transferable features is rather accidental depending on the property of the datasets. To empirically analyze the hypothesis, we design a semi-synthetic dataset that contains artificially- amplified transferable features and source-specific features simultaneously in the source data. Both the transferable and source-specific features can solve the source task, but only transferable features are useful for the target. We analyze what features fine-tuning and joint training will learn. See Figure I for an illustration of the semi- semi-synthetic experiments. We observed following failure patterns of tineingning and joint training on the semi-synthetic dataset.
Pre- Pre-training may learn non-transferable features that don't help the target when both transferable and source-specific features can solve the source task, since it's oblivious to the target data. When the dataset contains source-specific features that are more convenient for neural nets use. pre- training learns them; as result, fine-tuning starting from the source- source-specific features does not lead to improvement.
Joint training learns source- -sprcccc features and overfits on the target. A priori, it may appear that the joint training should prefer transferable features because the target data is present in the training loss. However, joint training easily overfits to the target especially when the target dataset is small. When the source-specifie features the most convenient for the source, joint training simultancoossy learns the source-specific features and memorizes the target dataset.
Toward overcoming the drawbacks of fine- tne-tuning and joint training, we first note that any proposed algorithm, unlike fine-tuning, should use the source and the target simultaneously to encourage extracting shared structures. Second and more importantly. we recall that good representations should enable generalization: we should not only be able to fit a target head with the representations (as joint training does), but the learned head should also generalize well to a held-out target dataset. With this intuition, we propose Meta Representation Learning MeRLin) encourage learning transferable and generalizable features: we meta-learn a feature extractor such that the head fit to a target training set performs well on a target validation set. In contrast to the standard model-agnostic meta-learning (MAML) (Finn et al. 2017), which aims to learn prediction models that are adaptable to multiple target tasks from multiple source tasks, our method meta-learns transferable representations with only one source and one target domain.
Empirically, we first verify that McRLin learns transferable features on the cemi-synthetic dataset. We then show that McRLin outperforms statc-of-the-arr transfer learning baselines in real-world vision and NLP tasks such as ImageNet to fine- ame-grained classification and language modeling to GLUE
Theoretically, we analyze the mechanism of the improvement brought by MeRLin. In a simple two-layer quadratic neural network setting, we prove that MeRLin recovers the target ground truth with only limited target examples whereas both fine-tuning and joint training fail to learn transferable features that can perform well on the target.
In summary, our contributions are as follows. (1) Using semi-synthetic dataset, we analyze and diagnose when and why fine-tuning and joint training fail to learn transferable representations. (2) We design meta representation learning algorithm (MeRLin) which outperforms state-of-the-ar transfer learning baselines. (3) We rigorously analyze the behavior of fine-tuning, joint training, and McRLin a special two-layer neural net setting.
2 SETUP AND PRELIMINARIES
In this paper, we study supervised transfer learning. Consider an input-label pair (x, y) E Bdd x R. We are provided with a source distributions D. and a target distribution D. over R2 x R. The source dataset D. = {=1, of uf) } and the target dataset D. = {at, y consist of n, i.i.d. samples from D. and n, i.i.d. samples from Dj respectively. Typically n. >> Re. We view a predictor as composition of a feature extractor his : Rd + I!' parametrized by Φ E Φ, which is often a deep neural net, and head classifier go: IR"
IR parametrized by 0 E θ, which is often linear. That is, the final prediction is (x) = go (h.(x)) Suppose the loss function is É(; -), such as cross entropy loss for classification tasks. Our goal is to learn an accurate model on the target domain D,.
Since the label sets of the source and target tasks can be different, we usually learn two heads for the source task and the target task separately, denoted by O, and 0., with a shared feature extractor
2

---

Under review as a conference paper at ICLR 2021
Sourcee
Tearures
SoSurce source-
spedific features
Source mowd festures
AB Training rusing
40 Taiger
Pre-tran
Pre-triit
Pre traie
Thrget
barst
70
Source = Fine turting
Transferble
Han-frundbrask features
specific festures
Torgen
Torget
Tarpet
fine-tuning
Inint
McRLin
Tarpetionty
Good performance
Gooo performance
sourre
the
Fine-ture
Fiie tune
Fine June
Good performance on the targes
Evcn worle than bargebirlty
Similor
Overlitding
Good performance on the target
bargebarnty
target
Figure 1: Comparison of fine-tuning, joint training, and MeRLin on the semi-synthetic dataset. Left: The semi-synthetic dataset and the qualitative observations the representations learned by three algorithms. Right: Quantitative results on the target test accuracy. See more interpretations, analysis, and results in Section 3
6. Let (0,0,) Φ) be the empirical loss of model ge (x)() on the empirical distribution D, that is, ( 6) :=
y) where (ar y) D means sampling uniformly from the dataset P. Using this notation, the standard supervised loss on the source (with the source head 0,) and loss on the target (with the target head 0₂) can be written LD, (0., o) and L., (0,, Φ) respectively.
E ==
We next review mainstream transfer learning baselines and describe them in our notations.
Target-only is the trivial algorithm that only trains on the target data D. with the objective L.g, (0,, o) starting from random initialization. With insufficient target data, target-only is prone overfitting. Pre-training starts with random initialization and pre-trains on the source dataset with objective function LB, (0., +) obtain the pre-trained feature extractor Opre and head 0s.
Fine-tuning initializes the target head 0g randomly and initializes the feature extractor e by Ópe obtained in pre-training, and fine-tunes à and 6, on the target by optimizing I.D, (0c, x) over both 0, and Φ Note that in this paper, fine-tuning refers to fine-tuning all layers by default.
Joint training starts with random initialization, and trains on the source and target dataset jointly by optimizing linear combination of their objectives over the heads 0., Og and the shared feature extractor Φ: ming, .0..0 join (0., 6r, 6) == (1 a) Lo, (0... o) + aاكة (6., o). The paper--rrameter is used to balance source training and target training. We use cross-validation to select optimal a.
LIMITATIONS OF FINE-TUNING AND JOINT TRAINING: ANALYSIS ON SEMI-SYNTHETIC DATA
Previous works (He al., 2018: Wu al., 2020) have observed cases when fine-tuning and joint training fail to improve over target-only. Our hypothesis is that both pre-training and joint training do not have incentives to prefer learning transferable features over source-specific features, and thus the performance of fine-tuning and joint training rely whether the transferable features happen to be the best features for predicting the source labels. Validating this hypothesis on real datasets is challenging, if not intractable-inc it's unclear what's the precise definition characterization of transferable features and source-specific features. Instead, we create a semi-synthetic dataset where transferable features and source-specifie features are prominent and well defined.
A semi-synthetic dataset. The target training dataset we use is a uniformly-sampled subset of the CIFAR-10 training set of size 500. The target test dataset is the original CIFAR-I0 test set. The source dataset of size 49500, denoted by AB, is created as follows. The upper halves of the examples are the upper halves of the CIFAR- 10 images (excluding the 500 example used in target). The lower halves contain signature pattern that strongly correlates with the class label: for class c, the pixels of the lower half are drawn i.i.d. from gaussian distribution MCc/10, 0.22). Therefore, averaging the pixels in the lower half of the image can reveal the label because the noise will get averaged out. The benefit of this dataset that any features related to the top half of the images can be defined as transferable features, whereas the features related to the bottom half are source-specific. Moreover, we can easily tell which features are used by a model by testing the performance on images with masked top or bottom half. For analysis and comparison, we define A to be the dataset that contains the top half of dataset AB and zeros out the bottom half, and B vice versa. See Figure (left) for an illustration of the datasets. Further details are deferred Section A.
3

---

Under review as a conference paper at ICLR 2021
Joint Training
Pre-training
Fine-tuning
MeRLin
Source AB training
100
Train
Test
20
Eve Evalustion held A
he d out
(a) T-SNE embeddings of features on the target dataset.
(b) Ablation.
Figure 2: (a) T-SNE visualizations of features on the target train and test set. The representations of pre-training work poorly on both larget train and test set, indicating that transferable features are not learned. Both joint training and fine-tuning work well on the target train set but poorly on the test set, indicating overfitting. MeRLin works well on the target test set. (b) Evaluation of different methods on A and B. Joint-trainnng and pre-training rely heavily the source-specific feature B and learn the transferable feature A poorly compared to McRLin. See more details in Section 3
In Figure I (right), we evaluate various algorithms' performance on target test data. In Figure a (left), we run algorithms with AB being the source dataset and visualize the learned features on the target training dataset and target test dataset to examine the generalizability
the features. In Figure 2(a) (right), we evaluate the algorithms on the held-out version of dataset A and B to examine what features the algorithms learn. ResNet-32 (He et al. 2016 is used for all settings.
Analysis: First of all, target-only has low accuracy (38%) because the target training set is small. Except when explicitly mentioned, all the discussions below are about algorithms on the source AB.
Fine-tuning fails because pre-training does not prefer learn transferable features and fine-tuning overfits. Figure 2(b] (pre-training) shows that the pre-trained model has near-trivial accuracy on held- out A but near-perfect accuracy on held-out B. indicating that it solely relies the source-specific feature (bottom half) and does not learn transferable features. Figure 2(a) (pre- training) shows that indeed pre-trained features do not have even correlation with target training and test sets. Figure 2(a) (fine-tuning) shows that fine-tuning improves the features' correlation with the training target labels but it does not generalize to the target test because of overfitting. The performance of fine-tuning (with source =AB) in Figure [ (right) also corroborates the lack of generalization.
Joint training fails because it simultareously learns mostly source-specific features and features that overfit to the target. Figure 2(b) (joint training) shows that the joint training model performs much better on held-out B (with 92% accuracy) than on the held-out (with 46% accuracy), indicating learns the source-specific feature very well but not the transferable features. The next question is what features joint training relies on to fit the target training labels. Figure [(a) shows strong correlation between joint training model's ''s features and labels on the target training set, but much less correlation on the target test set, suggesting that the joint training model's feature extractor, applied on the target data (which doesn't 't have source-specific features), overfits to the target training set. This corroborates the poor accuracy of joint training on the target test set (Figure 1). which similar to target-only 's
In Section 5]
we rigorously analyze the behavior of these algorithms on more simplified settings and show that the phenomena above can theoretically occur.
4 MERLIN: META REPRESENTATION LEARNING
In this section, we design a meta representation learning algorithm that encourages the discovery of transferable features. As shown in the semi-synthetic experiments, fine-tuning does not have any incentive to learn transferable features if they are not the most convenient for predicting the source labels because it is oblivious target data. Thus have use the source and target together to
As sanity checks, when the source contains only transferable features (Figure right, source A), fine- tuning works well, and when no transferable features (Figure right, source B), does not.
4

---

Under review as a conference paper at ICLR 2021
learn transferable representations. A natural attempt would have been joint training, but it overfits to the target when the target data is scarce as shown in the t-SNE visualizations in Figure ((a)
To fix the drawbacks of the joint training, recall that good representations should not only work well for the target training set but also generalize to the target distribution. More concretely, a good representation hg should enable the generalization of the linear head learned on top of it linear head 0 that is learned by fixing the feature (z) as the inputs should generalize well to a held-out dataset. We design a bi-level optimization objective learn such features, inspired by meta- learning for fast adaptation (Finn et al. 2017) and saarningtto-leare for automatic hype hyperparameter optimization (Maclaurin et al. 2015 Thrun& Pratt 2012) (more discussions below.)
We first split the target training set D. randomly into D. and Diall Given feature extractor ø, let O. (() be the linear classifier learned by using (xx) as the inputs on the dataset DY.
(Φ) = arg min T. ())
Note that (o) depends on the choice by ø (and is almost uniquely decided by it because the objective is convex in 0.) As alluded before, our final objective involves the generalizability of (() to the held-out dataset Dyal:
L menaa (0) - Lz (Dc(l)) (a ( (ha())),
The final objective is a linear combination of L (o) with the source loss
minimize L 0s) = L,D, (0., () + p Limeta (+)) @
To optimize the objective, we can use standard bi-level optimization technique as in learning-to- -taminggto-leann approaches. We also design sped-up version of MeRLin by changing the loss to squared loss so that the (()) has analytical solution. More details are provided in Section A.3 (Algorithm
Comparison to other meta-learning work. The key distinction of our approach from MAML (Finn eet al. 2017) and other meta-learning algorithms (e.g., (Nichol et al. 2018a| Bertinetto al. is that we only have a single source task and a single target task. Recent work (Raghu et al. 2020)
2019)
argues that feature reuse is the dominating factor of the effectiveness of MAML. In our case, the training target task is exactly the same as the test task, and thus the only possible contributing factor better-learned representation instead of fast adaptation. Our algorithm is in fact closer to the
work on hyperparameter optimization (Maclaurin et al.] 2015 Zoph & Le| 2016)- if we view the parameters of the head 0, as m hyperparameters and view é and f. as the ordinary parameters, then our algorithm is tuning hyperparameters on the validation set using gradient descent.
4.1 MERLIN LEARNS TRANSFERABLE FEATURES ON SEMI-S SYNTITIT DATASET
We verify that MeRLin learns transferable features in the semi-synthetic setting of Section 3 where fine-tuning and joint training fail. Figure | (right) shows that MeRLin outperforms fine-tuning and joint training by a large margin and is close to fine-tuning from the source A, which can be almost viewed as an upper bound of any algorithm's 's performance with AB as the source. Figure 2(b) shows that MeRLin (trained with source AB) performs well on A, indicating it learns the transferable features. Figure (igure(((a(( (MeRLin, train& test) corroborates the conclusion.
5 THEORETICAL ANALYSIS WITH TWO-LAYER QUADRATIC NEURAL NETS
The experiments in Section 3 demonstrate the weakness of fine-tuning and joint training. On the other hand, MeRLin is able to learn the transferable features from the source datasets. In this section, we instantiate transfer learning in quadratic neural network where the algorithms can be rigorously studied. For a specific data distribution, we prove that (1) fine-tuning and joint training fail to learn transferable features, and (2) MeRLin recovers target ground truth with limited target examples.
Models. Consider a two-layer neural network (x) = go (he (z)) with go(2) = A 2 and ha = ( z), where á = [61 Φ2. ... ...
@ml E Rdxm head, and (+)) is element-wiss quadratic. We consider squared loss ((f,,(x)) v) (f) y)².
m is the weight of the first layer, f E Rm is the linear
5

---

Under review as a conference paper at ICLR 2021
Source distribution. Let k E = such that 2 2 k S d. We consider the following source distribution which can be solved by multiple feature extractors. Let 21g denotes the i-th entry of E IR.. Let y = happens with prob. 1/a, and conditioned on y = 0, we have F[J = 0 for s k, and "[]] ~ {=1, 0} uniformly randomly and independently for i > k. With prob. 3/3 we have y = 1, and
conditioned on y 1, we have =i[] ~ { uniformly randomly and independently for i S k, and =[(] 111 0} uniformly randomly and independently for i k.
The design choice here is that 2.1))..... Tik] are the useful entries for predicting the source label, because = _[(] for any S k. In other words, features (( for i S k are useful features to learn, and any linear mixture of them works. All other entries of x are independent with the label y-
Target distribution. The target distribution is exactly k 1 version of the source distribution. Therefore, y = Ti and a [e z) is the correct feature extractor for the target. All other 211 for i >> 1 independent with the label.
Source-specific features and transferable features. As mentioned before, o (e of (eTT))))
all good features for the source, whereas only ((T is transferable the target.
Since usually the source dataset much larger than target, we assume access to infinite source data for simplicity, so D_ = .D.- We assume access to ny target data Dr.
Because the limited target data, the optimal solutions with unregularized objective are often not unique. Therefore, we study & fe-regularized version of the baselines and MeRLin. The regularization strength À > 0 is selected to achieve optimal LD. The regularized MeRLin objective is m Φ) [mm 02) x(11e,1 + I1 The regularized joint training objective is T.A.
Regularization:
"jii (0,, Oc. +)= == "yoim (0., Or, () + X A[|0, - ||0, 1 +1+1)²) We also regularize the two objectives in the pre-training and fine-tuning. We pre-train with L.,, (0., LD, (0.; () + 1(1 Il011 and then only fine-tune the head by minimizing the target loss "b, (0c) = T.D. (ft, P.ee) + A ||0,1] 1|2
The following theorem shows that neither joint training nor fine-tuning is capable of recovering the target ground truth given limited number of target data.
Theorem 1. There exists universal constants c E (0, 1) and > O, such that so long as n4 S cd, for any X > 0, the following statements true: the solution (0z, Ô,, @joim)) of the joint training satisfies
With prob. at least 1 - 4
LD. (0r, @yain) 2
(4)
With prob. at least 1 - & (over the randomness of pre-training), the solution (ō., Spre) of the head-only fine-tuning satisfies
Lo, (0,, 2
(5)
As will be shown in the proof, not surprisingly, fine-tuning fails because it learns a random feature o (eTTz)) (where i E [A:) for the source during pre-training which does not transfer to the target when i A I. Joint training fails because it uses one neuron to learn a generalized linear model to overfit the target n, training data exactly, and then use another neuron to learn a random feature (+ (where i [k)) for the source. The proof of Theorem is deferred to Section
In contrast, the following theorem shows that MeRLin can recover the ground truth of the target task: For any X < Ao where do is some universal constant and any failure rate & > O. if the target set size Tu > O(log' )), with probability at least 1 E, the feature extractor Pmea found by MeRLin and the head 0, (, (Fmea) trained on Dy recovers the ground truth of the target task:
Theorem 2.
LD, (0.(am)))) &mecaa) = 0.
(6)
Intuitively, MeRLin learns the transferable feature o (eiT .) because it simultaneously fits the source and enables the generalization the head on the target. The proof can be found in Section
For theoretical analysis consider only fine-tuning 0g- It is worth noting that fine-tuning both 0 and Φ converges the solution target-only training, which also has large generalization gap due to overfitting.

---

Under review as a conference paper at ICLR 2021
Table 1: Accuracy (%) on computer vision tasks.
Source
Fashion
SVHN
ImageNet
Food-101
Backbone
LeNet
ResNet-18
Target
USPS (600)
CUB-200
Caltech-256 [Stanford Cars]
CUB-200
Target-only
991.07 = 3.5
= 0.67
2.222 + 1.02
23.22
32.13 + 0.64
Joint training
89.59 + 0.56 91.54 55.81
7.36 0.50
78.20
63.25 + 0.72
42.08 + 0.59
Fine-tuning
90.80 = 0.20 92.12 92.12
72.52
81.12 = 0.27
81.59 = 0.49
52.30 + 0.51
L2-sp
89.74 + 0.41 91.86 +: 0.27
7 0.38
82.31 = 0.22
81.26
0.27
53.84 + 0.37
MeRLin
93.34 + 0.41 93.10 0.38//5.422 0.47
82.45 = 0.26
83.68 + 0.57
58.68 + 0.43
Table 2: Accuracy (%) of BERT-base on GLUE sub-tasks dev set.
Target
MRPC
RTE
QNLI
Fine-tuning
83.74 + 0.93
68.35 + 0.86
91.54 + 0.25
1.2-sp
84.31 + 0.37
67.50 0.62
91.29 + 0.36
MeRLin-ft
86.03 + 0.25
70.22 + 0.86
92.10 + 0.27
EXPERIMENTS
We evaluate MeRLin on several vision and NLP datasets. We show that (1) MeRLin consistently improves over baseline transfer learning algorithms including tinettunig and joint training in both vision and NLP (Section 6.2). and (2) as indicated by our theory, MeRLin succeeds because it learns features that are more transferable than fine-tuning and joint training (Section 6.3
6.1 SETUP: TASKS, MODELS, BASELINES, AND OUR ALGORITHMS
The evaluation metric for all tasks is the top-1 accuracy. We run all tasks for 3 times and report their means and standard deviations. Further experimental details are deferred to Section
Datasets and models. We consider the following four settings. The first three object recognition problems (with different label sets). The fourth problem is the prominent NLP benchmark where the source is language modeling task and the targets are classification problems.
SVHN or Fashion-MNIST" > USPS. We use either SVHN (Netzer et al., 2011) (73K street view house numbers) Pashion-MNIST Xiao al. (017)) (50K clothes) as the source dataset. The target dataset is a random subset of 600 examples of USPS Huull 1994), hand-written digit dataset. We down-sampled USPS simulate the setting where the target dataset is much smaller than the source. We use LeNer (LeCun et al. 1998), a three-layer ReLU network in this experiment.
ImageNer - CUB-200, Stanford Cars, or Caltech-256. We use ImageNet (Russakovsky al. 2015) as the source dataset. The target dataset is Caltech-256, CUB-200 (Wah et al. 22011), or Stanford Cars (Krause al., (2133). These datasets have 25468, 5994, 8144 labeled examples respectively, much smaller than ImageNet with 1.2M labeled examples. Caltech is a general image classification dataset of 256 classes. Stanford Cars and CUB are fine-grained classification datasets with 196 categories of cars and 200 categories of birds, respectively. We use ResNet-18 (He et al. 2016).
Food-10 101 + CUB-200. Food (Bossard et al. 2014) is fine-grained classification dataset 101 classes of food. Here we validate MeRLin when the gap between the source and target is large. We also use ResNet-18.
ImageNet + Stanford Dogs, MIT-indoors, Aircraft. We further test the proposed method with ResNet-50 (He et al.) 2016) as baselines. We still use ImageNet (Russakovsky et al. 2015) as the source dataset. The target dataset is Stanford Dogs (Khosla al. 2011), MIT-indoors (Quattoni & Torralba 2009). or Aircraft (Maji et al. 2013) consisting of 12000, 5360, 8000 labeled examples, respectively.
Language modeling + GLUE. Pre-training on language modeling tasks and fine- sanettnnng on labeled dataset such as GLUE (Wang et al. 2019) is dominant following the success of BERT (Devlin et al.
7

---

Under review as a conference paper at ICLR 2021
Table 3: Accuracy (%) of computer vision tasks with ResNet-50 backbone and ImageNet.
Target
MIT-indoors
Stanford Cars
Aircraft
Fine-tuning
83.21 + 0.21
84.77 0.32
81.13 + 0.21
12-sp
83.98 + 0.29
86.42 = 0.20
80.98 + 0.29
DELTA
83.66 + 0.20
85,01 = 0.22
80.44 0.20 81.48 + 0.18
BSS
83.73 + 0.18
86.69 = 0.19
McRLin
84.50 + 0.26
87.05 = 0.33
82.57 + 0.27
2019).
We fine-tune BERT with McRLin and evaluate it the three tasks of GLUE with the smallest number of labeled examples which standard fine-tuning likely overfits.
Baselines. (1) larget-only, (2) fine-tuning, and (3) joint-training have been defined in Section 2 Following standard practice, the initial learning rate of fine-tuning is 0.1 x the initial learning rate of pre-tramiig to avoid overfitting For joint training, the overall objective can be formulated as: (1-a) a. We tune to achieve optimal performance. The fourth baseline is (4) L2-sp eL al. 2018), which fine-tunes the models with regularization penalizing the parameter distance the pre-trained feature extractor. (5) DELTA (L et 2019) constrains the layer outputs selected with attention. (6) BSS (Chen et al. 2019) regularized the singular value of feature matrix. We also tuned the strength the L2-sp. Delta and BSS regularizer.
MeRLin. We perform standard training with cross entropy loss on the source domain while meta- learning the representation in the target domain as described in Section
MeRLin-ft. In BERT experiments, training on the source masked language modeling task is prohibitively time-consuming, so we opt light-weight variant instead: start from pre-trained BERT, and only meta-learn the representation in the target domain.
6.2 RESULTS
Results of digits classification and object recognition are provided in Table McRLin consistently outperforms all baselines. Note that the discrepancy between Fashion-MNIST and USPS is very large, where fine-tuning and joint training perform even worse than target-only. Nonetheless, MeRLin is still capable of harnessing the knowledge from the source domain. On MeRLin improves over fine-tuning by 6.58%, indicating that MeRLin helps learn transferable features when the gap between the source and target tasks is huge. In Table our method on GLUE tasks. McRLin-ft outperforms standard BERT fine-tuning and L2-sp. Since McRLin-ft only changes the training objective fine-tuning, it can be easily applied to NLP models.
validate
6.3 ANALYSIS
2.5
Training
We empirically analyze the representations and verify that MeRLin indeed learns more transferable features than fine- tuning and joint training.
Fine-tuning
2.0
MeRLin
1.5
Intra-class inter-class variance ratio. Suppose the rep- resentation of the j-th example of the i-th class is di.j.
1.0
H4 N: Σ di.j and μ 1 E Mi. Then the intra-class inter-class variance ratio can be calculated as Cuntr En, - Lnnnr N |M-aa|= = Low values this ratio cor-
Intra to
0.5
respond representations where classes well-separated. Results on ImageNet - CUB-200 and Stanford Cars task shown in Figure
Stanford-Cars
Figure 3: Comparison of intra-class to inter-class variance ratio, This quantity is lowest for McRLin, indicat- ing that separates classes hest.
MeRLin reaches much smaller ratio
than baselines.
ADDITIONAL RELATED WORK
Transfer learning is prevalent in deep learning applications. In computer vision, ImageNet pre-training
is common practice for nearly all target tasks. Early works (Oquab al. 2014 Donahue et al. 2014) directly apply ImageNet features to target tasks. Fine-tuning from ImageNet pre pre-trained models
has become dominant (Long al. 2015 He et al. 2017 Kolesnikov al.) 2019). On the other
8

---

Under review as a conference paper at ICLR 2021
hand, transfer learning is also crucial to the success of NLP. Pre-training transformers large-scale language tasks boosts performance on downstream tasks (Devlin et al. 2019 Yang et al. 2019).
A recent line of literature casts doubt the consistency of transfer learning's ' success (Raghu et al. 2019) He et al. 2018; Kornblith et al. 2019). Huh ct al. (2016) observed that some set of examples in ImageNet are more transferable than the others. Geeihhs al. (2019) found out that the texture of ImageNet is not transferable to some target tasks. Training on the source dataset may also need early
stopping find optimal transferability Liu al. (2019a): Neyshabur et al. (2020).
Meta-learning, originated from the learning to learn idea (Hochreiter et al. 2001 Vilalta & Drissi
2002; Maclaurin al. 2013 Zoph & Le| 2016), learns from multiple training tasks models that can be swiftly adapted to new tasks (Finn et al.) 2017 Rajeswaran et al.) 2019; Nichol et al. 2018b). Raghu et al. (2020); Goldblum et al. (2020) empirically studied the mechanism of MAML's success. Computationally, our method uses bi-level optimization techniques similar to meta-learning work. E.g., Beg.inettnettto et al. (2019) speeds up the implementation of MAML|Fiin et al. (2017) with closed- form solution of the inner loop, which is technique that we also use. However, the key difference between our paper from the meta-learning approach that we only learn from a single target task and evaluate on it. Therefore, conceptually, our algorithm is closer to the ceaningg-t-leaar approach for hyperparameter optimization (Maclaurin et al. 2015 Zoph & Le 2016), where there is a single distribution that generates the training and validation dataset.
8 CONCLUSION
We study the limitations of fine-tuning and joint training. To overcome their drawbacks, propose meta representation learning to learn transferable features. Both theoretical and empirical evidence verify our findings. Results vision and NLP tasks validate our method real-world datasets. Our work raises many intriguing questions for further study. Could we apply meta-learning to heterogeneous target tasks? What's more, future work can pay attention disentangling transferable features from non-transferable features explicitly for better transfer learning.
REFERENCES
Sanjeev Arora, Simon Du Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine- Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 322-332, 2019a.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances Neural Information Processing Systems 32, pp. 1141-8150 2019b.
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural Information Processing Systems 32, pp. 690-701. 2019.
Luca Bertinetto, Joao F. Henriques, Philipp Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. In International Conference Learning Representations, 2019.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative compo- nents with random forests. In European Conference on Computer Vision, 2014.
Yuan Cao and Quanquan Gu. Generalization bounds of stochastic gradient descent for wide and deep neural networks. In Advances in Neural Information Processing Systems 32, pp. 10836-10846. 2019.
Xinyang Chen, Sinan Wang, Bo Fu, Mingsheng Long, and Jianmin Wang. Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer learning. In Advances in Neural Information Processing Systems, volume 32, pp. 1908-1918. Curran Associates, Inc., 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186, 2019.

---

Under review as a conference paper at ICLR 2021
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In Proceedings of the 31st International Conference on Machine Learning, volume 32, pp. 647- 655, 2014.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1126-1135. PMLR, 2017.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In inPrro Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2014.
Micah Goldblum, Steven Reich, Liam Fowl, Renkun Ni, Valeriia Cherepanova, and Tom Goldstein. Unraveling meta-leamnig; Understanding feature representations for few-shot tasks. volume 119 of Proceedings of Machine Learning Research, 2020.
G. Griffin, A. Holub, and P. Perona. Caltech-256 object category dataset. Technical report, California Institute of Technology, 2007.
Kaiming. He, Xiangyu. Zhang, Shaoqing. Ren, and Jian. Sun. Deep residual learning for image recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Kaiming He, Ross B. Girshick, and Piotr Dollár. Rethinking imagenet pre-training. arxiv, abs/1811.08883, 2018
Sepp Hochreiter, A. Steven Younger, and Peter R. Conwell. Learning learn using gradient descent. In International Conference on Artificial Neural Networks, 2001.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. 2018.
Mi- Young Huh, Pulkit Agrawal, and Alexei A. Efros. What makes imagenet good for transfer learning? arxiv, abs/1608. 086 14, 2016.
Jonathan J. Hull A database for handwritten text recognition research. IEEE Transactions on pattern analysis and machine intelligence, 16(5):550-555 1994.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, 2017.
Aditya Khosla, Nityananda Jayadevaprakash. Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine-grained image categorization: Stanford dogs. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011.
lasonas Kokkinos. Ubernet: Training universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning, 2019.
Simon Kornblith, Jonathon Shlens, and Quoe V. Le. Do better imagenet models transfer better? In The IEEE Conference Computer Vision and Patern Recognition (CVPR), pp. 2661-2671, 2019.
10

---

Under review as a conference paper at ICLR 2021
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (GdRR-13), Sydney, Australia, 2013.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to document recognition. Proceedings of the IEFE, 86(11):2278 1998.
Xingjian Li, Haoyi Xiong, Hanchao Wang, Yuxuan Rao, Liping Liu, and Jun Huan. DELTA: deep learning transfer using feature map with attention for convolutional networks. International Conference on Learning Representations, 2019.
Xuhong Li, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pp. 2825-2834, 2018.
Zichuan Lin, Garrett Thomas, Guangwen Yang, and Tengyu Ma. Model-based adversarial meta- reinforcement learning, 2020.
Hong Liu, Mingsheng Long, Jianmin Wang, and Michael I. Jordan. Towards understanding the transferability of deep representations, 2019a.
Xiaodong Liu, Pengcheng He. Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4487-4496, 2019b.
Long, E. Shelhamer, and Darrell. Fully convolutional networks for semantic segmentation. In 2015 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 3431-3440, 2015.
M. Long, Y. Cao, J. Wang, and M. I. Jordan. Learning transferable features with deep adaptation networks. In Proceedings of the 32nd International Conference on Machine Learning (ICML), pp. 97-105, 2015.
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-basee hyperparameter optimization through reversible learning. In International Conference on Machine Learning, pp. 2113-2122, 2015.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft, 2013.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. 20111..
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning?, 2020.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint 1rXivv1803.00990 2018a.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-leaming algorithms, 2018b.
Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Learning and transferring mid-level image representations using convolutional neural networks. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2014.
A. Quattoni and A. Torralba. Recognizing indoor scenes. In IEEE Conference Computer Vision and Pattern Recognition, 2009.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning feature reuse? towards understanding the effectiveness maml. In International Conference on Learning Representations, 2020.
11

---

Under review as a conference paper at ICLR 2021
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging. In Advances in Neural Information Processing Systems 32, pp. 3342-3352. 2019.
Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. In Advances in Neural Information Processing Systems 32, pp. 113-124. 2019.
Mark Rudelson and Roman Vershynin. Non-asymptotic theory of random matrices: extreme singular values. In Proceedings of the International Congress of Mathematicians 2010 (ICM 2010) (In Volumes) Vol. I: Plenary Lectures and Ceremonies Vols. II-IV: Invited Lectures, pp. 1576-1602. World Scientific, 2010.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 11((3): 211-252, 2015.
Alex Tamkin, Trisha Singh, Davide Giovanardi, and Noah Goodman. Investigating transferability in pretrained language models, 2020.
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 2012.
Laurens J.P. van der Maaten and Geoffrey E. Hinton. Visualizing high -igh-dimeniional data using t-sne. Journal of Machine Learning Research, 9(2):2579-2605, 2008
Ricardo Vilalta and Youssef Drissi, A perspective view and survey of meta-learning. Artificial Intelligence Review, 18(2):77-95, 2002.
C. Wah, S. Branson, P. Welinder, Perona, and S. Belongie. The Caltech-UCSD 2irds-200-2011 Dataset. Technical Report ANS-TR-2001-001 California Institute of Technology, 2011.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference Learning Representations, 2019.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu. Mohammadhadi Bagheri, and Ronald M. Summers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervise classifi- cation and localization of common thorax diseases. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Sen Wu, Hongyang R. Zhang, and Christopher Ré. Understanding and improving information transfer in multi-task learning. In International Conference on Learning Representations, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arxiv, 0tbs/170..000 708.07747, 2017.
Ze Yang, Tiange Luo, Dong Wang, Zhiqiang Hu, and Liwei Wang. Learning to navigate for fine- grained classification. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ Salakhutdinov, and Quoe Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems 32, pp. 3753-5763. 2019.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems 27, pp. 3320-3328. 2014.
Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. arXiv preprint 1rX.v.161..0157. 2016.
12

---

Under review as a conference paper at ICLR 2021
A ADDITIONAL DETAILS OF EXPERIMENTS
A.1 THE SEMI-SYNTHETIC EXPERIMENT
The original CIFAR images are of resolution 32 x 32. For the transferable dataset A, we reserve the upper 16 x 32 and fill the lower half with [0.485, 0.456, 0.406] for the three channels (the mean of CIFAR-I1 10 images). For the non- non-trans an-transfferalle dataset B, the lower part 16 x 32 pixels are generated with i.i.d. gaussian distribution with the upper half filled with [0.485, 0.456, 0.406] similarly. To make the non-transferable part related to the labels, we set the mean of the gaussian distribution to 0.1 x e, where c is the class index of the image. The variance of the gaussian noise is set to 0.2. We always clamp the images to [0, 1] to make the generated images valid. For the source dataset, we use 49500 CIFAR-10 images, while for the target, we use the other 500 to avoid memorizing target examples.
implementation provided in github. com/ http://aaaaaatteeeeeee resnet s_f_cifarllo We set the initial learning rate to 0.1, and decay the learning rate by 0.1 after every 50 epochs. We use t-SNE (van der Maaten & Hinton 2008) visualizations provided in sklearn. The perplexity is set to 80.
We use ResNet-32
A.2 IMPLEMENTATION ON REAL DATASETS
We implement all models on PyTorch with 2080Ti GPUs. All models are optimized by SGD with 0.9 momentum. For object recognition tasks, the initial learning rate is set to 0.01, with 5 x 10-4 weight decay. The batch-size is set to 64. We run each model for 100 epochs. ImageNet pre-trained models can be found in torchvision We use a batch size of 128 on the source dataset and 512 on the target dataset. The initial learning rate is set to 0.1 for training from scratch and 0.01 for ImageNet initialization. We decay the learning rate by 0.1 every 50 epochs until 150 epochs. The weight decay is set to 5 x 10-4. For GLUE tasks, we follow the standard practice of Devlin et al. (2019). The BERT model is provided in i thub www.///w//ee........ooooo fttpttorchyttppppppppp Benncchpreerraa ned BERT For each model, we set the head (classifier) to the top one linear layer. We use a batch size of 32. The learning rate is set to 5 x 10-5 with 0.1 warmup proportion. During fine-tuning, the initial learning rate is 10 times smaller than training from scratch following standard practice. The hyper-pas syper-parameter is set to 2, and À is found with cross validation. We randomly split the training set into 80% and 20%. We train on the 80% part and use the rest 20% as the validation set. We also provide the results of varying and in Section A.7 In object recognition however, we also provides a pre-trained version: starting from ImageNet pre-trained solution.
A.3 IMPLEMENTING THE SPEED-UP VERSION
Practical implementation: speeding up with MSE loss. Training the head ge in the inner loop of meta learning can be time-consuming. Even using implicit function theorem or implicit gradients as proposed in (Bai et al. 2019 Rajeswaran et al. 2019 Lin et al. 2020). we have to approximate the inverse of Hessian. To solve the optimization issues, we propose analytically calculate the prediction of the linear head O, and directly back-prop to the feature extractor hg. Thus, we only need compute the gradient once in single step. Concretely, suppose we use MSE-loss. Denote by H 2 xm the feature matrix of the 7 target samples in the target meta-training set Dy. Then 0, in equation I can be analytically computed as 0, (HH" AI)-'y, where À is a hyper- parameer for regularization. The objective of the outer loop can be directly computed as
2 p- Th
minimize 0(0,0.) L5, 6
(0,, +
((G +AD) by o ho (.
We implement the kernel speed-up version on classification tasks following Arora et al. (2019b). We treat the classification problems as multi-variate ridge regression. Suppose we have label c E {1, 2.... 10). Then the target encoding for regression is -0.1
e., For example, if the -0.1). Then the parameters of the target head in the inner loop be computed as 0, (HHT -II- We then compute the MSE loss
label is 3, then the encoding will be (-0.1, --.1,.0
13

---

Under review as a conference paper at ICLR 2021
on the target validation set: (Hu vaullé [ in the outer loop. We summarize the details of the vanilla version and the speed-up version in Algorithm and Algorithm
Algorithm Meta Representation Learning (MeRLin).
1: Input: the source dataset D. and the evolving target dataset Dr.
2: Output: learned representations Φ.
3: for iter = 0 to MaxIter do
4: Initialize the target head Oz.
5: Randomly sample target train set DI and target validation set Dial from D..
6: for t 0 to n do
7: Train the target head on DT:
0g - 02 17 Vo,,, の).
8: end for
9: In the outer loop, update the representation ó and the source head 0,: (Ø, (,) - (0. 0.) (V,.))))) [TD. (0,, + pl. (0,,
10: end for
Algorithm 2 Meta Representation Learning (MeRLin): speed-up implementation.
1: Input: the source dataset D, and the evolving target dataset D,..
2: Output: learned representations d.
3: for t= t=00 0 to MaxIter do
4: Randomly sample target train set Dj and target validation set Dral from D..
5: Analytically calculate the solution target head (c))) in the inner loop
(Ø) = (HH + AT) -Y.
6: In the outer loop, update the representation di and the source head 0,:
2
(g) + (0, 0.) V (0.0.) Ls, (6.. Φ)
H Hyalhh Y vall
n
7: end for
A.4 DATASETS
CUB-200 (Wah 2011 is fine-grae tne-grained dataset of 200 bird species. The training dataset consists of 5994 images and the test set consists of 5794 images. http: www. vissin.cclteen edu/ vis pedia a/ CU/CCUB--2 200 2011 html
Stanford Cars Krause et al. 2013) dataset contains 16, 185 images of 196 classes of cars. The data split into 8, 144 training images and 8,041 testing images. httt:::/aa.sttaffood.eedd stanford.
jkrause/cas caaesssseerrrrreeeeeeeeeeeee httpccarraaaatttttiaaaaaa
html
Food-101 (Bossard et 2014 a fine-grained classification dataset of 101 kinds of food, with 750 training images and 250 test images for each kind. http::////:tt viss on ee. ethz. ch/
datasets Batbbets__ettt______ee_e ext ext.raffood- ftd./food-11111
Caltech-256 (Griffin et al. 2007) is an object recognition dataset of 256 categories. In our experiments, the training set consists of 25468 images, and the test set consists of 5139 im- ages/http://www vis wwvisissnne. www.ecaltccc..... edu/ wttp:///mmm.e.attttttaaaaaeeeeee
MIT-indoors (Quattoni & Torralba 2009 is a scene recognition dataset of 67 classes. The training set contains 80 examples for each class, and the test set contains 20 examples for each class.
http://web.nnt edu. oor wtw.u///oooraaaaaaaaaaaaaaaeaeae
wwtpe///o.....aaa////ea.o.aeaaae hwt.hbhwwwwwwnnddd
htmi
Stanford-Cars (Khosla et al. 2011) is fine-grained classification dataset of 120 kinds of dogs. edu
hett::/vision.sta.e wtt.:odddddddddddd.//////// http://ddyyye/mmameeeettegog/ caoogg//mmiinn htt.ii.iinimllllm....m.mmmm....m
Aireraft (Maji al. 2013) is object recognition dataset of 100 aircraft variants. Each class has 80 images for training and 20 images for testing. http: /tt:://wwww... www. robots. ox. ac. uk/ vvgg data/ fgvc-av aircfft/
14

---

We further compare with two variants of MeRLin as Under review as a conference paper at ICLR 2021
MNIST (LeCun al. 1998) is dataset of hand- Kand-written digits. It has training set of 60,000
examples, and a test set 10,000 examples. We randomly sample 600 samples as the training set for the target domain. http: /tt::/yynnn. /e///////////c//////////////// exdb/mnist/
SVHN
image dataset of street view house numbers. It has 73257 digits for training. 26032 digits for testing. In Figure 3 of the main text, randomly sample 600 digits for training, and all the 26032 digits are used testing. http: http:///ufldi http//fffl..saaffod....ee
(Ntzerr et al. 2011) a
housenumbers/
USPS (Hull 1994) a database for handwritten text recognition. It has 7291 train and 2007 test images. We randomly sample 600 samples the training set for the target domain. https
77 www. //www.ke www.wkaggle.cm haaggeee.cco//tstttuuuavvs/sssse
A.5 FURTHER ABLATION STUDY.
We extend the last column of Table 2 ablation study:
Table
MeRLin (pre-trained). We first pre-train the model on the source dataset and then optimize the MeRLin objective starting from the pre-trained solution.
McRLin tatLiintargt--only MeRLin-target-only only meta-learns representations on the target domain starting from random initialization. We test whether the meta-learning objective itself has regulariza- tion effect.
Table 4: Accuracy on Food -> CUB.
Algorithm
Target-only
Fine-tuning Joint Training MeRLinn-targe--nil MeRLin (pre-trained)
MeRLin
Accuracy 32. 10 +
0.64| 52.30 = 0.51 42.08 0.59
40.17 + 0.44
55.26 + 0.43 58.68 + 0.43
MeRLin (pre-trained) performs worse than MeRLin, but it still improves over fine-tuning and joint training. Note-thar MeRLin (pre-trained) only need to train on ImageNet for epochs, much shorter than joint training. MeRLin-targeet-n improves target-only by 8%, indicating that meta meta-learning helps avoid overfitting even without the source dataset.
A.6 FEATURE-LABEL CORRELATION
Feature label Correlation
Joint Training
Fine tuning MeRLin
Dor
CUB-200
Stanford-Cars
(a) Comparison feature-label correlation.
(b) Sensitivity to hyper-parameters
Figure 4: (a) Analysis of Feature Quality, Comparison of feature-label correlation. A lower quantity is better, and MeRLin has the lowest value. (b) Sensitivity of the proposed method to hyper-parameters. We test the accuracy on Food-101-$CURB with varying / and and provide the visualization.
Suppose the feature matrix H, and the label vector is y, then the correlation between feature and label can be defined as y (HTH) H) y. As is shown by Arora et al. 2019a): Cao & Gu (2019), this term is closely related the generalization error of neural networks, with a smaller quantity indicating better generalization. We calculate y (H" H) y on ImageNet - CUB-200 and Stanford Cars. As shown in Figure 4(a) the features learned by MeRLin are more closely related
15

---

Under review as a conference paper at ICLR 2021
to labels than fine-tuning and joint training, indicating MeRLin is indeed learning more transferable features compared with baselines.
A.7 SENSITIVITY OF THE PROPOSED METHOD TO HYPER- PAPPERPARAMETERS.
with varying hyper-parameters p and A. Results in Figure that the model is not sensitive to varying and A. Intuitively, larger indicates more emphasis on the target meta-task. When P approaches fine-tuning, À exerts regularization to the classifier in the inner loop training. It is also note worthy that À can avoid the problem that HH is occasionally invertible. Without À the model can fail to converge sometimes.
We test the model on
the performance of MRL is approaching
A.8
BOOTSTRAPPED CONFIDENCE INTERVAL OF OBJECTION RECOGNITION RESULTS
We run 10 seeds of each experiment on 1.2-sp and MeRLin in Table and calculate 95% bootstrapped confidence interval of each results in Figure
95
L2 sp merlin
90
Accuracy
85
ImageNet->CUB
ImageNet->Cars
Fashion->USPS
SVHN->USPS
Dataset
Figure 5: Comparison of L2-sp and MeRLin. The results are averaged over 10 runs, with error bars representing 95% confidence interval drawn by 1,000 bootstraps.
16

---

Under review as a conference paper at ICLR 2021
B MISSING DETAILS IN SECTION 5
B. PROOF OF THEOREM 1
Lemma 1. Suppose S c 3/. For each solution 0z. & satisfying B..yDDD (2).,..()).)) Sc.
the joint training objective (0, fr, 6) is lower bounded:
jine (0.. Φ) = (1 LD. (02, Φ) + a.. (0,, () + A (10-112 + +10&112 + 10
2/3
31 24/3
a) 1 2/3 + 5 (1 -a)(a - 11²)
31
3e
min
1 1
V
24/3
Proof of Lemma !
Define d x d matrix
A 0200
(8)
Define || and 1:d be the first & and last d dimensions of x. AR, Ag.ū and Ag.ū be k k k x (d - k) and (d (d-AA k) x (d - k) matrices that correspond to the upper left, upper right and lower right part of A. For a random vector z where the first & dimensions are uniformly independently from {+1}. the last d - I dimensions
uniformly indepdently from {0, +1), define random variables A a1 [u.) A2 = x I++ I:d] Ag [R-+1--- (Note that x is defined on a different distribution than D..)
We have bound
(1 Ex (1)),,.(0))))) (10.11²+ + 1001( )
(9)
=(1 a) Le (TTTAAA + T T 1:d] + 2To AR -
-y) 27
+1 110/112 + 1101
(10)
(o + TT ALr - ")² + A [10b11p + 1
2(1 - a) Ex E..DD.
||
(11)
=(1 a)
2
(Ai - -1)² + + B [(AI 1) + E [AA]
(10.112= + 1
|
(12)
3
=(1 0))
(2RR
2
(A1 + A2 - 1)9
+ [48]
+ + (10..12
+ 1 1001
(13)
3
2 (1 a) (E [A| A2) 11)
31
EE [A
+ A₂
(14)
24/3
The first inequality is because
Ex.yyD. (=Tra) AB 0,
(15)
Ex.yy,, Al (IT O,
(16)
8) 0.
ExyyD.
(17)
17

---

Under review as a conference paper at ICLR 2021
The second inequality is because
(0.i) 2 + 2
1105112)
2
24/3
3
Σ
1ssl 11
12)
12)
2/3
(18)
1=1
2/3
3
Σ
00el
||0bal
(19)
24/3
2/3
3
(20)
24/3
where the first inequality is AM-GM inequality, the second inequality is by concavity of (.)2/3 The third inequality is because for diagonal matrix D) that has at (i, if Au. 2 0, -1 at (f, Ap.a] have
m Σ
m
m Σ
Ap.il = tr(Dt) = Σ 0 DΦ: =
0sll 1|
(21)
On the other hand, for the target, we define d matrix
B = 0.
(22)
Define 2[1] and Z22.d be the first 1 and last d dimensions of x. B1.1. B1,i and B1,1 be x 1, 1 x (d - 1) and (d - 1) x (d - matrices that correspond to the upper left, upper right and lower right part of B. For random vector where the first dimension uniformly independently from [= the last d - 1 dimensions are uniformly indepdently from {0, =1}, define random variables B1 = "1 B₂ 2/2.qq B (Note that is defined on different distribution than D,.)
Using similar argument as above, we have
2 (E|B,
Em,yDD
B2] 1)².
(23)
However, know that Ey,,DD, (((o (x), y)] S e, there has to be
3x
E B1 + B2
(24)
therefore we have
2/3
A [10.)12 + |101
31
3e
2
1
-
V
(25)
24/3
2
Summing up Equation 9 and Equation 25 finishes the proof.
Lemma 2. Assume &r is vector such that (fr, = 2111 for all xt Dr, then there exists some
solution (0., Or, σ) such that
J (0,, Ozs ()) S min
22/3
3A
1u
2/3
+
/
3
2
- (1-a)(u- 1)2) +
22/3
31
4/3
(26)
Proof of Lemma 2 Assume 4³ arg min,,
( (2733
| 2/3
+ [1 a) 1)²). then obviously μ²
[0, 1]. Let Φ1 (V2 dr
2176
3r. Φi 0 for i
2, 02 (u* (4*/2)// and
| 173
0, ||0.12/3 62. Now we prove that this model satisfies the Equation 26
18

---

Under review as a conference paper at ICLR 2021
First of all, notice that for any of E D., there is
of Σ
Eandrot)
(27)
x 01202222 1r
(28)
x4)
(29)
(30)
Therefore we have
(0,)0) =0
(31)
On the other hand, we have
TTD (0.. + x(10.11² (100)1++1<bb)) -11bb117)
(32)
(1 -a)(n" - -1)²+ 3)
22/3 1="12/8
(33)
min 33
22233 1
3/ (1 a) 1)8
(34)
Plugging Equation 31 and Equation 32 into the formula of I (0.. t2 6) finishes the proof.
Lemma 3. Let X lddx be a random matrix where each entry is uniformly random and inde- d pendently sample from +1, < 2- Let Pxe1 be the projection of ej to the column space of
X. Then, there exists absolute constants co > and O, such that with probability at least
1 - (exx(=CCd)) there is
Pxeall2 S 5 co /m
(35)
Proof of Lemma 3
Let 8min (X) and Smaz (X) be the minimal and maximal singular values of X
respectively. Then we have
Pxxill2 = |X(X x) 1x Teillz
(36)
- l| ||(x x) - "lop ||
X)
(37)
S Smax (X) (x)) -2Vn.
(38)
By Theorem 3.3 Rudelson & Vershynin 2010). there exists constants C1, C2 > such that P(smin (X) S cv(v vn)) s 2cxp(-c2d)
(39)
By Proposition 2.4 in Rudelson & Vershynin (010). there exists constants C3 C4 0, such that
Plsmaz (X) 2 cs(Vd+ vn)) s 2cxp(-cad).
(40)
Let C min(ee, ), then with probability at least 4exp(Cd), there is
Smaz (X) (X)) 2
(41)
C3 Vd Vn
27 Vd Vn)² Vn
(42)
(2 - 22 n
(V2 1)2c3 V
d'
(43)
which completes the proof.
19

---

Under review as a conference paper at ICLR 2021
Proof of Theorem 7 We prove the joint training part of Theorem following this intuition: (1) the total loss of each solution with target loss Ex,,,,, [x(f y)] S is lower bounded as indicated by Lemma [ and (2) there exists solution with loss smaller than the aforementioned lower bound as indicated by Lemma 2
By Lemma 1 for any 0s 0s,0 & satisfying Br,YD, (2). S C, the joint training loss Lo (0,, dc. ④) is lower bounded,
2/3
31 24/3
2 (11-))(m-
31
3c
Ljuinl (0 O2, の)
min 1
2/33
3
+
24/3
1
2
(44)
Let Pxex be the projection of vector e1 to the subspace spanned by the target data. According to there exists some solution (0s, de, Φ) such that
Lemma 2
Iiyoin (0s, 0c, Φ) S
3A
2/3
+ (1 a) -1)²)
2
3A
4/3
(45)
22/3
22/3
2/3
Let co > be a constant such that 21/3
Jen
According Lemma 3 there
2/3
2473.
exists absolute constants E (0, C 0, such that long as nu S cd,, there is with probability least 1 - exp(
2/3
3co
4/3
1
0t
(46)
24/3
2
22/3
24/3
22/3
Now we prove the upper bound in Equation because
45
is smaller than the lower bound in Equation
This
min P
3\
|m|
2/3
+
31 a)(a - 1)2)
2
+
Q
4/3
(47)
22/3
3
22/3
3\
2
22/3
3λ
4/3
min
22/3
|1]
2/3
+
3
(1 (1- - a)( 1)2)
-
3A
24/3
+
22/3
6
+
3X(
22/3
24/3
(48)
3\
2
31
1//
4/3
1
<min
21/3 (A)
2/3
+
(1 - a)(H - -1)²)
+
+ 3X(
(49)
3
22/3
22/3
21/3
2/3
<min 1
3A
2/3
+ 3
(1 (M 1)2)
2
+
31
1
3e0
(50)
24/3
24/3
2
where the first inequality uses that fact that |M|
for the optimal μr, the second inequality is by Equation 46 This completes the proof for joint training.
Then, we prove the result about fine fine-tuning. According to Lemma any minimizer (6., âprc) of L (0., o) either satisfies ôpe or only one di is non-zero but looks like (up scaling) ej for
j E [k]. When âpmc- = 0, there is
Eny-D. [e im (z). = 2 10
(51)
27
When only one &; is non-zero but looks like e, for j E [k). since all the first dimensions are equivalent for the source task, with probability 1 L. this dimension j * 1. The target funciton fine-tuned on this ó.me looks like f (I) = ? for some 7 E IR, so there is
Ex.gDD, (o)
E By,DnD [(>0) - =
(52)
= 72 E
[x]]]
F. [ + E EEDDD [21]
(53)
2
2 10
(54)
3
3 27
Combining these two possibilities finishes the proof for fine-tuning, Finnaly, setting e = min(f. finishes the proof of Theorem 1
10
27
20

---

Under review as a conference paper at ICLR 2021
B.2 PROOF OF THEOREM 2
Lemma 4. Define the source loss as
D (O,, 1) = (0,, (0s, () + (110/12 + 11b1
Then, for any x > O, any minimizer of L (0., .) is one of the following cases:
(i) 0, = 0 and Φ
(ii) for one E [m]. Oz: > 0, Φ. = =(V20.) eg for some j S k; for all other i E [m]]
18 |
Furthermore, when 0 < X < 0.1, all the minimizers look like (ii).
Proof of lemma 4 Define d matrix
A =
(55)
Define [ss and W[k+1:d] be the first and last d - A dimensions of x. Ak As,E and Ag,8 be k x k, x (d - k) and (d - k) x (d - k) matrices that correspond to the upper left, upper right and lower right part of A. For random vector where the first & dimensions are uniformly independently from {= the last d - k dimensions are uniformly indepdently from {0, =1}, define random variables A1 2 Bl1ay (us)) A2 (i++1. I.d] Ag. 1sd]- (Note that x is defined on different distribution than ...)
The loss part of 1 (0., Φ) can be lower bounded by:
E, ((fe),,(z)
(56)
- v) ")²
= Ex E.ADD,
(RT + 207T.: Ag, A
(57)
L,,, A1 + 1 As [+ ")
(58)
E (Ai - 1)91 + _I [[A1 - 1)A2) + E [A]]
(59)
3
3 (A1 + A2 - 1)²] + [Aß].
A2
(60)
The inequality is because
Ex ( (ST (MTuN) Ae, )] 0,
(61)
E Ex,yD. A (=T AR O,
(62)
ER.yD. ( (*T A 11. u] O.
(63)
The inequality is equality if and only if Ar, = O.
The regularizer part of L3, (0., Φ)) can be lower bounded by:
Σ
m
(0nz)² + 110.11²)
3
(10zl (10:112) 2/3 1
(64)
22/3
1=1
2/3
3
10 I
(65)
22/3
2/3
3
App.yl
(66)
22/3
21

---

Under review as a conference paper at ICLR 2021
where the first inequality is AM-GM inequality, the second inequality is by concavity of (-) The
third inequality is because for diagonal matrix D that has at (i, if A 2 0, -1 at (i, Ab.i) < O, we have
m
m
m
A = tr(D) = Σ i=1
02e DØ: 5
Ossl || 11
(67)
i=1
All the inequalities are equality if and only if (0.:)² = |10b1 > 0 for at most one E [m]] and for all other E [m]] there is 18,1 || = O.
Combining the two parts gives a lower bound for L (0s, 6):
2/3
L3, (0., 6) 2
3A
Σ April
m
2 E
A2 1) [
(AI
[A]]
(68)
22/3
31
2
2/3
2
[Ir(AR.
[B + +A2] 1)²
(69)
22/3
where both incqualitites are equality if and only if AF,E = 0 (therefore A = 0) and Var [Ai] = (therefore AR, is diagonal by Lemma 5).
Notice that E[A, + A2] tr(AA, + Ag the above lower bound is further minimized when (1x)) + 1)? 22/3
B[A]] μ¹¹ where fs" is the minimizer of function L(p)
3A
To see when this lower bound is achieved, we combine all the conditions for the inequalities to be equality. When μ = 0, this lower bound is achieved if and only if 0, = 0 and 6 = O. When μ > 0, this lower bound is only achieved when the solution look like this: for one i E [m/] O.. 1/3,
Φ; = (V20..) eg for some j S k; for all other i [m], 10.dl = || = 0. Obviously, there is either f4 = 0 or n² > O. Also, when À < 0.1, the minimizer μ² of L(j) is strictly larger than (since [(1 < L(O).. So this completes the proof.
Lemma 5. Let M E Rkxk be symmetric matrix, R* is a random vector where each dimension is indepedently uniformly from {+1}. Then, Var(z Mx = 0 if and only if M is a diagonal matrix.
Proof of Lemma 5 In one direction, when M is diagonal matrix, obviously Varix T Mx] = 0. In the other direction, when Var[x: Mæ] 0, there has to be x: Mr be the same for all x E ( {+1]* For any i + j. let = 1 2e, 20g, (2) 1, .:)
2es, xx(4) 1 2eg- Then the (i,
element of M is (1(11) Mx - (2) Mx 233 Mz x44) Mb(1). which is 0. So M
has to be diagonal matrix.
Proof of Theorem Define the source loss as in Lemma 4 then we have
Livera (0., v) = Ltd, (0., Φ) + E (0)p)
(70)
By Lemma the source loss L.B, (0z, Φ) is minimized by a set of solutions that look like this: for one E [m]. 0s: > 0, Φ; = + 20.)) Ej for some j < k; for all other i E [m]] |0..| = ||<b]|| = 0.
When j = 1, the only feature in Φ is e1. When 2 18 log ? according to Chernoff bound, with probability at least
there is strictly less than half of the data satisfy # = 0. Therefore, any Dj contains data with Φ1 + O, and the only target head that fits D. has recover the ground truth. Hence there is E (o)))
o) 0.
When j * 1, the only feature is ej. This feature can be used to fit the target data if and only if either = = 2²11 for all target data 2,, or Z2p1] = 0 for all z,, Since there are at most - 1 possible j. by union bound we know the probability of any of these happens for any j # 1 is at most A(3)".
22

---

Under review as a conference paper at ICLR 2021
Hence, when ny 2 3 log
2k the probability of any ey fits the target data is smaller than 5 Therefore, {.
with probabiltiy 5. E م. (o.()), o) > 0 for any j # 1.
So with probability at least 1 - &. the only minimizer of mgtu (0x, @) is the subset of minimizers of L (0x: Φ) with feature e1, and with this Φ and any random Dj, the only 0. (Dy, Φ) that fits the target recovers the ground truth, i.e., Bx (f,, v) 0.
23


Reviews:
[{"review_id": "dPjF2FfGR6h", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary of the Work\nThe work analyzes the failure modes of fine-tuning and joint training for multi-task learning in a computer vision setting, then proposes a meta-learning algorithm dubbed \"MeRLin\" to address these failure modes using bi-level optimization. The outer optimization minimizes the loss of inner-trained network when it is used to learn a new classification head for a validation set from the target domain. \n\nThe authors first motivate their work by discussing the success and failures of joint training and fine-tuning, then they use principled experiments to verify hypotheses about the failure modes of these two popular transfer methods. Thereafter, they introduce their proposed method (MeRLin), provide a theoretical analysis which compares its behavior to fine-tuning and joint training in a limited problem class, and then present empirical results showing the effectiveness of MeRLin on several popular CV and NLP benchmarks. \n\n## Pros and Cons\n\n### Pros\n* Principled approach to analyzing the failure modes of fine-tuning and joint training\n\n\n### Cons\n* Requiring data from the target domain during training negates many of the sought-after benefits of transfer learning (e.g. re-using pre-trained models quickly without lengthy pre-training, adapting to unforseen new tasks, etc.)\n\n## Evaluation\n### Quality\n3/5\nThe overall quality of execution of this work is high, but not flawless.   The work's laudable careful theoretical and synthetic-empirical evaluations give way to less-careful statistical comparisons of the empirical-nonsynthetic results. I believe works making specific performance claims based on empirical experiments without exception need to provide specific evidence in the form of statistical tests of significance, especially in cases like these when the performance gaps between the proposed method and baselines are so small. \n\n### Clarity\n5/5\nThe text and diagrams themselves are clean, lucid, well-written, and well-presented. I don't personally find the analysis in Sec. 5 to be particularly helpful to understanding the method or the failure modes of fine-tuning and joint training, and would be happy to see it moved to the appendix in favor of more careful empirical evaluation, but perhaps some readers find it easier to understand concepts with theory.\n\n### Originality\n4/5\nThis work has clear relationships to many prior works on parameter-space meta-learning and automatic hyperparameter tuning, and its own spin on the problem, which the authors acknowledge and provide context for. I have erred on the side of a higher rating in this category, thought don't have encyclopedic experience with all of the background literature. While the proposal in the work is not eye-wateringly novel considering previous work on regularization and meta-learning, novelty in ML research is, in general, overrated, and I believe this work is sufficiently novel to be a contribution.\n\n### Significance\n2/5\nThe analysis and synthetic experiments in this work are, I believe, a significant-enough contribution to the field irrespective of the proposed algorithm. If the paper itself contained only extended versions of the theoretical and synthetic-empirical experiment portions, I think it would be significant enough.\n\nUnfortunately, I have concerns about the significance of this work--inclusive of the proposed algorithm, which readers will focus on--for 2 reasons:\n(1) The proposed transfer solution negates many perceived and desired benefits of transfer and meta-learning, because it requires access to samples from the target distribution at train-time. Specifically, most researchers and practitioners pursue transfer learning specifically because they anticipate *not* having access to target samples at train time, e.g. in the case of adapting CV classifiers to unforseen target samples, or in the case of re-using a very large pre-trained CV or NLP model which--might be impractical for the end-user to train--for some unforseen application by an end user. The work does not address or acknowledge at all this slight compromise of the problem setting, or make efforts to justify its practicality in spite of this.\n(2) While the synthetic experiments show great performance benefits to the proposed method in a fabricated setting, the empirical-non-synthetic experiments show very little improvement over the baselines, especially L2-sp (Li, et al. 2018), and it's not clear from the provided analysis that the performance differences presented are even statistically-significantly different from L2-sp. This is especially notable given that fact that L2-sp *does not* require access to target samples at train-time (i.e. it conforms to the more-general transfer learning setting), while MeRLin compromises the setting to achieve apparently-comparable but not substantially-improved results. \n\n### Misc Editorial Comments and Reviewer's Notes\n\n#### Claims\n* The reader will learn about \"when and why fine-tuning and joint training can be suboptimal or even harmful for transfer learning\"\n* Proposed method MERLIN empirically outperforms SOTA transfer learning algorithms on vision and NLP benchmarks\n* Proves that MERLIN recovers the target ground truth model under some assumptions on NN architecture and source distributions, and that pre-training and join training fail\n\n\n#### Mechanisms\n* Ensuring that a head fit on top of the learned representations with (target) training data also performs well on (target) validation data\n\n\n#### 1. Introduction\n* Does the assertion \"joint training easily overfits to the target especially when the target dataset is small\" hold if the target dataset is intentionally oversampled so that its examples appear as often as the source dataset? If we don't have the answer to this question, it could be that we are simply observing the effect of the data distribution being skewed.\n* \"any proposed algorithm [...] should use the source and target simultaneously...\" -- I think this statement is overly-strong considering the results in the paper. Certainly this is *one way* to ensure that we \"extract shared structures,\" but it also presupposes we have the target up front, which is opposed to many sought-after benefits of a transfer learning approach. Using the same evidence, we can argue instead that transfer learning methods should make use of as much data as possible, or search for data sources which are more diverse and general than both the source and the target.\n\n#### 5. Theoretical Analysis\n* I find this lucid but perhaps unnecessary for the reader's understanding of the concept. It seems fairly intuitive that a learning algorithm with no access, or unbalanced access to to the target distribution will fail to learn features which generalize to that target distribution, and a learning algorithm which is intentionally supervised, using a loss computed on samples from the target distribution, will learn features which generalize to both the source and the target. *The important question then is \"do we give up more than we gain by requiring access to the target distribution?\"* I recommend moving this portion to the Appendix to make more room for justifying the method and better in-text coverage of empirical results.\n\n#### 6. Experiments\n* Many of these results would seem to be quite close, especially when using only 3 trials and simple mean/stddev. How do the results compare with a real confidence interval? e.g. a 95% bootstrapped interval (note the seaborn package will easily calculate these for you)? Are the differences observed statistically significant? \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Fmi7POqmE20", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThe paper investigates failure cases for transfer learning (fine-tuning and joint training), specifically in the context where training on the source data may highlight features that are irrelevant for the target data. This is done through semi-synthetic data. Based on the insights, the authors present an approach called Meta Representation Learning (MeRLin) inspired by Meta Learning and Learning-to-learn approaches. This approach is evaluated on several real-world transfer-learning tasks from vision and NLP. The authors also derive theoretical results on constructed data distributions for which superiority of Merlin can be shown analytically.\n\n### Quality, clarity, originality and significance\n\nThe work is well-motivated, and the paper is well-written and clear. \n\nThe \"failure modes\" and the behavior on the semi-synthetic data are not entirely surprising in my opinion (and I would have expected more analysis after reading the introduction), but they highlight the targeted problem well.\n\nThe suggested algorithm is well-motivated and linked to existing work. However in many practical settings, it may be infeasible to train including the whole source data for reasons of compute and data availability, which is why many transfer-learning approaches do not assume the availability of the source data during transfer. This scenario is encountered in the case of BERT, where the authors then \"only meta-learn the representation\". In that setting, Merlin seems to reduce to reduce to a changed objective inspired by meta-learning and learning-to-learn. This is not a problem in itself, but it makes it less clear, what exactly the authors think that the main contribution of the paper is.\n\nThe experimental results look good at first glance, but overall I found it hard to evaluate how convincing they are because of several potential problems:\n* The experimental results are not compared to any results from the existing literature, all baselines and comparisons are entirely from this paper. This makes it hard to know how much careful tuning went into the proposed algorithm versus the baselines. I think at least *some* comparison to results from the literature should be possible or it should be carefully explained why such an overfitting to the proposed algorithm clearly did not happen. \n* There are a lot of seemingly random choices in the selection of the datasets, e.g. why USPS and not MNIST, why CUB/Caltech/Cars and not e.g. CIFAR, Pets, Flowers, or Birds? I'm not saying any of these choices are inherently better, but the authors should explain *why* these datasets are chosen, to avoid the impression that the datasets may have been chosen because the proposed method works particularly well on these dataset combinations. One way to make the results stronger would therefore be to use combinations of datasets that other, previous work has already used, which would also enable a direct comparison (see above).\n* Similarly, it is unclear why only a subset of GLUE was used and not the full GLUE benchmark. Again, it would be better in my opinion to use all of GLUE to avoid the impression that the subset was hand-picked. Also, I tried to find results on GLUE for direct comparison, because they should exist, but was not successful immediately, because the presented results are on the dev-set (Table 2) while usually results for comparison come from the official test set as far as I understand? E.g. the BERT paper seems to have tuned on dev and then reported numbers on test, but I did not find dev-set numbers in a quick search. For something fairly standard as BERT-base and GLUE I think it would be good to be able to find numbers in the literature that are *directly comparable*. \n* Again in a similar fashion, Sec.6.3. evaluates only two target data sets, why only these? And A.5 looks only at Food->Cub, why only this pair?\n\n### Pros and cons\n* Pros: interesting approach, well-motivated, well-written, interesting theoretical analysis\n* Cons: experimental results are not entirely convincing, the main message is not 100% clear to me\n\n### Minor details and comments\n* The paper is fairly \"squeezed\": E.g. lack of vertical space after captions of Figures 1 and 2; also the main suggested algorithm is not explained in detail in the main text, but deferred to the Appendix. It might be more readable to try to shorten the content.\n* \"Pre-training only\" is not a complete transfer-learning algorithm because of potential class number mismatch, yet it is used for comparison (e.g. Fig. 2b). This only works in this very specially constructed scenario, it seems. Maybe this should be mentioned?\n* \"Target-only\" is discussed in the context of Fig. 2 but it is not contained in the Figure, why? Also, \"fine-tuning\" results are not shown in Fig 2b. \n* Towards the end of Section 3, when discussing the synthetic data, \"overfitting\" is mentioned twice as a problem. There exist several approaches that are often used to try to overcome overfitting, and the authors even use L2-sp as a baseline in Section 6. It remains a bit unclear how much a careful combination of fine-tuning and such methods could help here. I don't think this is a major shortcoming, though, because the artificial setup is constructed exactly in such a way that the pre-trained classifier will focus on the non-transferable features.\n* Table 1: For ImageNet->C256 L2-sp seems to be not clearly worse than MeRLin given the standard deviations of three runs. Maybe both should be bolded?\n* A.7 \"the model is not sensitive to varying ρ and λ.\" - it was not clear to me how much this statement can be derived from the figure.\n\n* Minor comments / typos:\n  * Caption of Fig.1 - I did not see how this is an illustration of the \"features learned\"\n  * End of Sec.3: \"on a much simplified settings\"\n  * page 5: \"Merlin to by changing\" - \"to\" too much\n  * page 5: \"with AB as the source .\" - space before \".\"\n  * page 6: \"with their own best regularization strength\" - I did not understand this phrase here.\n  * before Eq.(4) - what is $\\Omega$ here?\n  * In Theorems 1 and 2, what exactly is meant by \"universal constant\"?\n  * after Eq.(6): \"because its simultaneously fits\" -> it\n  * Food-101->CUB: here, the model is not mentioned. I guess it was also ResNet-18?\n  * page 7 \"join training\" -> joint\n  * page 8 \"the our method\"\n  * References: some letters in the reference titles are lowercased when they shouldn't be, e.g. \"cnns\", \"Mask r-cnn\", \"reuse? towards\", \"t-sne\", \"mnist\", \"Xlnet\". Also, several references do not contain a \"publishing venue\", e.g. Kolesnikov 2019, Lin 2002, Liu 2019a, Neyshabur 2020 - in the age of search engines this is not a real problem, but it might be nice to give at least an arXiv number if available or some of these have probably appeared in conferences or journals.\n  * page 12: \"images is of resolution\"\n  * A.2 \"lambda is found with cross validation\" - how exactly?\n  * A.4 for completeness maybe include ImageNet and USPS?\n  * Figure 4 caption: \"textbfSensitivity\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "AcdBd-k61XP", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper first investigates when and why fine-tuning and joint training are not the best methods for transfer learning. The authors generate a toy dataset, in which the source-specific and transferable features are thus clearly distinguishable, for the transfer learning tasks. As both fine-tuning and joint training are not optimal solutions for the experimental dataset, they further design MeRLin, which is based on the meta-learning mechanism, to learn a generalizable model weight.\n\n*Strengths*:\n1. In order to investigate how a model (including a feature extractor and a classifier) interacts with the source-specific feature and transferable feature within a data, the authors generate a toy dataset and examine the effectiveness of fine-tuning and joint-training methods. This provides clear and insightful experimental observations for the further understanding of transfer feature learning.\n\n2. The observation that the fine-tuning and the joint training tends to learn the \"convenient\" feature (i.e., both methods tend to learn easy source-specific feature rather than a transferable feature) provides a reasonable argument why both models cannot work well.\n\n3. The designed meta-learning-based method is convincing empirically, as the objective function implies the model has to learn a generalizable feature extractor for loss minimization.  \n\n*Weakness/comments*\n1. Although the proposed demonstrates the above observations, the connection to existing domain adaptation/domain generation works seems lacking. Some of these works also aim to learn the feature, which can be generalized to the target domain. And the source-specific feature can be also regarded as the domain shift between source and target domains (so that it cannot be generalized by using fine-tuning). I understand the problem settings of domain adaptation/domain generation are a bit different from the setting used in the paper, while such discussion can further improve the completeness of this paper. \n\n2. Another weakness is the baseline methods, as these baseline methods are too simple. I am wondering about the comparison to existing works on the relevant areas (e.g., other general feature learning or domain-relevant works). \n\n3. One reason why people use pretraining method is neglected: one does not need to access the source dataset when the target task is trained. In some practical scenarios, one cannot access the large scale of source dataset (i.e., imagenet), and this limits the use of the proposed model, which requires access to source data. But this point is just a comment, not a weakness as the authors have clarified the problem scope at the beginning of the paper. \n\nOverall, the paper presents an insightful study and a reasonable new method. I am inclined to the score \"Marginally above acceptance threshold\".", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper compares transfer learning with fine-tuning and joint training and then proposes a new approach (Merlin). Reviewers have pointed to the fact that Merlin works in a setting that is different from normal transfer learning settings (it assumes some target domain data is available during training). The authors acknowledge this and think it can still be a reasonable setting, but of course it makes comparisons more difficult. Overall, while there are interesting analysis and results, the paper remains borderline and more work should be done to make it a good contribution, including significantly improving the presentation to make clear the distinction in settings. I therefore recommend to reject the paper.
Decision: Reject