Year: 2020

Abstract:
Recent works found that fine-tuning and joint training—two popular approaches for transfer learning—do not always improve accuracy on downstream tasks. First, we aim to understand more about when and why fine-tuning and joint training can be suboptimal or even harmful for transfer learning. We design semi-synthetic datasets where the source task can be solved by either source-specific features or transferable features. We observe that (1) pre-training may not have incentive to learn transferable features and (2) joint training may simultaneously learn sourcespecific features and overfit to the target. Second, to improve over fine-tuning and joint training, we propose Meta Representation Learning (MeRLin) to learn transferable features. MeRLin meta-learns representations by ensuring that a head fit on top of the representations with target training data also performs well on target validation data. We also prove that MeRLin recovers the target ground-truth model with a quadratic neural net parameterization and a source distribution that contains both transferable and source-specific features. On the same distribution, pre-training and joint training provably fail to learn transferable features. MeRLin empirically outperforms previous state-of-the-art transfer learning algorithms on various real-world vision and NLP transfer learning benchmarks.

heading: 1 INTRODUCTION
text: Transfer learning—transferring knowledge learned from a large-scale source dataset to a small target dataset—is an important paradigm in machine learning (Yosinski et al., 2014) with wide applications in vision (Donahue et al., 2014) and natural language processing (NLP) (Howard & Ruder, 2018; Devlin et al., 2019). Because the source and target tasks are often related, we expect to be able to learn features that are transferable to the target task from the source data. These features may help learn the target task with fewer examples (Long et al., 2015; Tamkin et al., 2020).
Mainstream approaches for transfer learning are fine-tuning and joint training. Fine-tuning initializes from a model pre-trained on a large-scale source task (e.g., ImageNet) and continues training on the target task with a potentially different set of labels (e.g., object recognition (Wang et al., 2017; Yang et al., 2018; Kolesnikov et al., 2019), object detection (Girshick et al., 2014), and segmentation (Long et al., 2015; He et al., 2017)). Another enormously successful example of fine-tuning is in NLP: pre-training transformers and fine-tuning on downstream tasks leads to state-of-the-art results for many NLP tasks (Devlin et al., 2019; Yang et al., 2019). In contrast to the two-stage optimization process of fine-tuning, joint training optimizes a linear combination of the objectives of the source and the target tasks (Kokkinos, 2017; Kendall et al., 2017; Liu et al., 2019b).
Despite the pervasiveness of fine-tuning and joint training, recent works uncover that they are not always panaceas for transfer learning. Geirhos et al. (2019) found that the pre-trained models learn the texture of ImageNet, which is biased and not transferable to target tasks. ImageNet pre-training does not necessarily improve accuracy on COCO (He et al., 2018), fine-grained classification (Kornblith et al., 2019), and medical imaging tasks (Raghu et al., 2019). Wu et al. (2020) observed that large model capacity and discrepancy between the source and target domain eclipse the effect of joint training. Nonetheless, we do not yet have a systematic understanding of what makes the successes of fine-tuning and joint training inconsistent.
The goal of this paper is two-fold: (1) to understand more about when and why fine-tuning and joint training can be suboptimal or even harmful for transfer learning; (2) to design algorithms that overcome the drawbacks of fine-tuning and joint training and consistently outperform them.
To address the first question, we hypothesize that fine-tuning and joint training do not have incentives to prefer learning transferable features over source-specific features, and thus their capability of learning transferable features is rather accidental depending on the property of the datasets. To empirically analyze the hypothesis, we design a semi-synthetic dataset that contains artificiallyamplified transferable features and source-specific features simultaneously in the source data. Both the transferable and source-specific features can solve the source task, but only transferable features are useful for the target. We analyze what features fine-tuning and joint training will learn. See Figure 1 for an illustration of the semi-synthetic experiments. We observed following failure patterns of fine-tuning and joint training on the semi-synthetic dataset.
• Pre-training may learn non-transferable features that don’t help the target when both transferable and source-specific features can solve the source task, since it’s oblivious to the target data. When the dataset contains source-specific features that are more convenient for neural nets to use, pretraining learns them; as a result, fine-tuning starting from the source-specific features does not lead to improvement.
• Joint training learns source-specific features and overfits on the target. A priori, it may appear that the joint training should prefer transferable features because the target data is present in the training loss. However, joint training easily overfits to the target especially when the target dataset is small. When the source-specific features are the most convenient for the source, joint training simultaneously learns the source-specific features and memorizes the target dataset.
Toward overcoming the drawbacks of fine-tuning and joint training, we first note that any proposed algorithm, unlike fine-tuning, should use the source and the target simultaneously to encourage extracting shared structures. Second and more importantly, we recall that good representations should enable generalization: we should not only be able to fit a target head with the representations (as joint training does), but the learned head should also generalize well to a held-out target dataset. With this intuition, we propose Meta Representation Learning (MeRLin) to encourage learning transferable and generalizable features: we meta-learn a feature extractor such that the head fit to a target training set performs well on a target validation set. In contrast to the standard model-agnostic meta-learning (MAML) (Finn et al., 2017), which aims to learn prediction models that are adaptable to multiple target tasks from multiple source tasks, our method meta-learns transferable representations with only one source and one target domain.
Empirically, we first verify that MeRLin learns transferable features on the semi-synthetic dataset. We then show that MeRLin outperforms state-of-the-art transfer learning baselines in real-world vision and NLP tasks such as ImageNet to fine-grained classification and language modeling to GLUE.
Theoretically, we analyze the mechanism of the improvement brought by MeRLin. In a simple two-layer quadratic neural network setting, we prove that MeRLin recovers the target ground truth with only limited target examples whereas both fine-tuning and joint training fail to learn transferable features that can perform well on the target.
In summary, our contributions are as follows. (1) Using a semi-synthetic dataset, we analyze and diagnose when and why fine-tuning and joint training fail to learn transferable representations. (2) We design a meta representation learning algorithm (MeRLin) which outperforms state-of-the-art transfer learning baselines. (3) We rigorously analyze the behavior of fine-tuning, joint training, and MeRLin on a special two-layer neural net setting.

heading: 2 SETUP AND PRELIMINARIES
text: In this paper, we study supervised transfer learning. Consider an input-label pair (x, y) ∈ Rd × R. We are provided with a source distributions Ds and a target distribution Dt over Rd × R. The source dataset D̂s = {xsi , ysi } ns i=1 and the target dataset D̂t = {xti, yti} nt i=1 consist of ns i.i.d. samples from Ds and nt i.i.d. samples from Dt respectively. Typically ns nt. We view a predictor as a composition of a feature extractor hφ : Rd → Rm parametrized by φ ∈ Φ, which is often a deep neural net, and a head classifier gθ : Rm → R parametrized by θ ∈ Θ, which is often linear. That is, the final prediction is fθ,φ(x) = gθ(hφ(x)). Suppose the loss function is `(·, ·), such as cross entropy loss for classification tasks. Our goal is to learn an accurate model on the target domain Dt. Since the label sets of the source and target tasks can be different, we usually learn two heads for the source task and the target task separately, denoted by θs and θt, with a shared feature extractor
φ. Let LD̂(θ, φ) be the empirical loss of model gθ(hφ(x)) on the empirical distribution D̂, that is, LD̂(θ, φ) := E(x,y)∈D̂`(gθ(hφ(x)), y) where (x, y) ∈ D̂ means sampling uniformly from the dataset D̂. Using this notation, the standard supervised loss on the source (with the source head θs) and loss on the target (with the target head θt) can be written as LD̂s(θs, φ) and LD̂t(θt, φ) respectively.
We next review mainstream transfer learning baselines and describe them in our notations. Target-only is the trivial algorithm that only trains on the target data D̂t with the objective LD̂t(θt, φ) starting from random initialization. With insufficient target data, target-only is prone to overfitting. Pre-training starts with random initialization and pre-trains on the source dataset with objective function LD̂s(θs, φ) to obtain the pre-trained feature extractor φ̂pre and head θ̂s. Fine-tuning initializes the target head θt randomly and initializes the feature extractor φ by φ̂pre obtained in pre-training, and fine-tunes φ and θt on the target by optimizing LD̂t(θt, φ) over both θt and φ. Note that in this paper, fine-tuning refers to fine-tuning all layers by default. Joint training starts with random initialization, and trains on the source and target dataset jointly by optimizing a linear combination of their objectives over the heads θs, θt and the shared feature extractor φ: minθs,θt,φ Ljoint(θs, θt, φ) := (1− α)LD̂s(θs, φ) + αLD̂t(θt, φ). The hyper-parameter α is used to balance source training and target training. We use cross-validation to select optimal α.

heading: 3 LIMITATIONS OF FINE-TUNING AND JOINT TRAINING: ANALYSIS ON SEMI-SYNTHETIC DATA
text: Previous works (He et al., 2018; Wu et al., 2020) have observed cases when fine-tuning and joint training fail to improve over target-only. Our hypothesis is that both pre-training and joint training do not have incentives to prefer learning transferable features over source-specific features, and thus the performance of fine-tuning and joint training rely on whether the transferable features happen to be the best features for predicting the source labels. Validating this hypothesis on real datasets is challenging, if not intractable—it’s unclear what’s the precise definition or characterization of transferable features and source-specific features. Instead, we create a semi-synthetic dataset where transferable features and source-specific features are prominent and well defined.
A semi-synthetic dataset. The target training dataset we use is a uniformly-sampled subset of the CIFAR-10 training set of size 500. The target test dataset is the original CIFAR-10 test set. The source dataset of size 49500, denoted by AB, is created as follows. The upper halves of the examples are the upper halves of the CIFAR-10 images (excluding the 500 example used in target). The lower halves contain a signature pattern that strongly correlates with the class label: for class c, the pixels of the lower half are drawn i.i.d. from gaussian distribution N (c/10, 0.22). Therefore, averaging the pixels in the lower half of the image can reveal the label because the noise will get averaged out. The benefit of this dataset is that any features related to the top half of the images can be defined as transferable features, whereas the features related to the bottom half are source-specific. Moreover, we can easily tell which features are used by a model by testing the performance on images with masked top or bottom half. For analysis and comparison, we define A to be the dataset that contains the top half of dataset AB and zeros out the bottom half, and B vice versa. See Figure 1 (left) for an illustration of the datasets. Further details are deferred to Section A.1
In Figure 1 (right), we evaluate various algorithms’ performance on target test data. In Figure 2(a) (left), we run algorithms with AB being the source dataset and visualize the learned features on the target training dataset and target test dataset to examine the generalizability of the features. In Figure 2(a) (right), we evaluate the algorithms on the held-out version of dataset A and B to examine what features the algorithms learn. ResNet-32 (He et al., 2016) is used for all settings.
Analysis: First of all, target-only has low accuracy (38%) because the target training set is small. Except when explicitly mentioned, all the discussions below are about algorithms on the source AB.
Fine-tuning fails because pre-training does not prefer to learn transferable features and fine-tuning overfits. Figure 2(b) (pre-training) shows that the pre-trained model has near-trivial accuracy on heldout A but near-perfect accuracy on held-out B, indicating that it solely relies on the source-specific feature (bottom half) and does not learn transferable features. Figure 2(a) (pre-training) shows that indeed pre-trained features do not have even correlation with target training and test sets. Figure 2(a) (fine-tuning) shows that fine-tuning improves the features’ correlation with the training target labels but it does not generalize to the target test because of overfitting. The performance of fine-tuning (with source =AB) in Figure 1 (right) also corroborates the lack of generalization.
Joint training fails because it simultaneously learns mostly source-specific features and features that overfit to the target. Figure 2(b) (joint training) shows that the joint training model performs much better on held-out B (with 92% accuracy) than on the held-out A (with 46% accuracy), indicating it learns the source-specific feature very well but not the transferable features. The next question is what features joint training relies on to fit the target training labels. Figure 2(a) shows strong correlation between joint training model’s features and labels on the target training set, but much less correlation on the target test set, suggesting that the joint training model’s feature extractor, applied on the target data (which doesn’t have source-specific features), overfits to the target training set. This corroborates the poor accuracy of joint training on the target test set (Figure 1), which is similar to target-only’s.1
In Section 5, we rigorously analyze the behavior of these algorithms on a more simplified settings and show that the phenomena above can theoretically occur.

heading: 4 MERLIN: META REPRESENTATION LEARNING
text: In this section, we design a meta representation learning algorithm that encourages the discovery of transferable features. As shown in the semi-synthetic experiments, fine-tuning does not have any incentive to learn transferable features if they are not the most convenient for predicting the source labels because it is oblivious to target data. Thus we have to use the source and target together to
1As sanity checks, when the source contains only transferable features (Figure 1, right, source = A), finetuning works well, and when no transferable features (Figure 1, right, source = B), it does not.
learn transferable representations. A natural attempt would have been joint training, but it overfits to the target when the target data is scarce as shown in the t-SNE visualizations in Figure 2(a).
To fix the drawbacks of the joint training, we recall that good representations should not only work well for the target training set but also generalize to the target distribution. More concretely, a good representation hφ should enable the generalization of the linear head learned on top of it—a linear head θ that is learned by fixing the feature hφ(x) as the inputs should generalize well to a held-out dataset. We design a bi-level optimization objective to learn such features, inspired by metalearning for fast adaptation (Finn et al., 2017) and learning-to-learn for automatic hyperparameter optimization (Maclaurin et al., 2015; Thrun & Pratt, 2012) (more discussions below.)
We first split the target training set D̂t randomly into D̂trt and D̂valt . Given a feature extractor φ, let θ̂t(φ) be the linear classifier learned by using hφ(x) as the inputs on the dataset D̂trt .
θ̂t(φ) = arg min θ LD̂trt (θ, φ) (1)
Note that θ̂t(φ) depends on the choice by φ (and is almost uniquely decided by it because the objective is convex in θ.) As alluded before, our final objective involves the generalizability of θ̂t(φ) to the held-out dataset D̂valt :
Lmeta,t(φ) = LD̂valt (θ̂t(φ), φ) = E(x,y)∈D̂valt `(gθ̂t(φ)(hφ(x)), y) (2)
The final objective is a linear combination of Lmeta,t(φ) with the source loss
minimize φ∈Φ,θs∈Θ Lmeta(φ, θs) := LD̂s(θs, φ) + ρ · Lmeta,t(φ) (3)
To optimize the objective, we can use standard bi-level optimization technique as in learning-to-learn approaches. We also design a sped-up version of MeRLin by changing the loss to squared loss so that the θ̂t(φ) has an analytical solution. More details are provided in Section A.3 (Algorithm 2).
Comparison to other meta-learning work. The key distinction of our approach from MAML (Finn et al., 2017) and other meta-learning algorithms (e.g., (Nichol et al., 2018a; Bertinetto et al., 2019)) is that we only have a single source task and a single target task. Recent work (Raghu et al., 2020) argues that feature reuse is the dominating factor of the effectiveness of MAML. In our case, the training target task is exactly the same as the test task, and thus the only possible contributing factor is a better-learned representation instead of fast adaptation. Our algorithm is in fact closer to the work on hyperparameter optimization (Maclaurin et al., 2015; Zoph & Le, 2016)—if we view the parameters of the head θt as m hyperparameters and view φ and θs as the ordinary parameters, then our algorithm is tuning hyperparameters on the validation set using gradient descent.

heading: 4.1 MERLIN LEARNS TRANSFERABLE FEATURES ON SEMI-SYNTHETIC DATASET
text: We verify that MeRLin learns transferable features in the semi-synthetic setting of Section 3 where fine-tuning and joint training fail. Figure 1 (right) shows that MeRLin outperforms fine-tuning and joint training by a large margin and is close to fine-tuning from the source A, which can be almost viewed as an upper bound of any algorithm’s performance with AB as the source. Figure 2(b) shows that MeRLin (trained with source = AB) performs well on A, indicating it learns the transferable features. Figure 2(a) (MeRLin, train& test) corroborates the conclusion.

heading: 5 THEORETICAL ANALYSIS WITH TWO-LAYER QUADRATIC NEURAL NETS
text: The experiments in Section 3 demonstrate the weakness of fine-tuning and joint training. On the other hand, MeRLin is able to learn the transferable features from the source datasets. In this section, we instantiate transfer learning in a quadratic neural network where the algorithms can be rigorously studied. For a specific data distribution, we prove that (1) fine-tuning and joint training fail to learn transferable features, and (2) MeRLin recovers target ground truth with limited target examples.
Models. Consider a two-layer neural network fθ,φ(x) = gθ(hφ(x)) with gθ(z) = θ>z and hφ = σ(φ>x), where φ = [φ1, φ2, · · · , φm] ∈ Rd×m is the weight of the first layer, θ ∈ Rm is the linear head, and σ(·) is element-wise quadratic. We consider squared loss `(fθ,φ(x), y) = (fθ,φ(x)− y)2.
Source distribution. Let k ∈ Z+ such that 2 ≤ k ≤ d. We consider the following source distribution which can be solved by multiple possible feature extractors. Let x[i] denotes the i-th entry of x ∈ Rd. Let y = 0 happens with prob. 1/3, and conditioned on y = 0, we have x[i] = 0 for i ≤ k, and x[i] ∼ {±1, 0} uniformly randomly and independently for i > k. With prob. 2/3 we have y = 1, and conditioned on y = 1, we have x[i] ∼ {±1} uniformly randomly and independently for i ≤ k, and x[i] ∼ {±1, 0} uniformly randomly and independently for i > k. The design choice here is that x[1], . . . , x[k] are the useful entries for predicting the source label, because y = x2[i] for any i ≤ k. In other words, features σ(e > [i]x) for i ≤ k are useful features to learn, and any linear mixture of them works. All other entries of x are independent with the label y.
Target distribution. The target distribution is exactly k = 1 version of the source distribution. Therefore, y = x2[1], and σ(e > [1]x) is the correct feature extractor for the target. All other x[i] for i > 1 are independent with the label.
Source-specific features and transferable features. As mentioned before, σ(e>[1]x), · · · , σ(e > [k]x) are all good features for the source, whereas only σ(e>[1]x) is transferable to the target.
Since usually the source dataset is much larger than target, we assume access to infinite source data for simplicity, so D̂s = Ds. We assume access to nt target data D̂t. Regularization: Because the limited target data, the optimal solutions with unregularized objective are often not unique. Therefore, we study `2-regularized version of the baselines and MeRLin. The regularization strength λ > 0 is selected to achieve optimal LDt . The regularized MeRLin objective is Lλmeta(θs, φ) := Lmeta(φ, θs) + λ(‖θs‖ 2 + ‖φ‖2F ). The regularized joint training objective is Lλjoint(θs, θt, φ) := Ljoint(θs, θt, φ)+λ(‖θs‖ 2 +‖θt‖2 +‖φ‖2F ).We also regularize the two objectives in the pre-training and fine-tuning. We pre-train with LλDs(θs, φ) := LDs(θs, φ) +λ(‖θs‖ 2
+ ‖φ‖2F ), and then only fine-tune the head2 by minimizing the target loss Lλ,φ̂pre
D̂t (θt) := LD̂t(θt, φ̂pre)+λ ‖θt‖ 2.
The following theorem shows that neither joint training nor fine-tuning is capable of recovering the target ground truth given limited number of target data.
Theorem 1. There exists universal constants c ∈ (0, 1) and > 0, such that so long as nt ≤ cd, for any λ > 0, the following statements are true: • With prob. at least 1− 4 exp(−Ω(d)), the solution (θ̂s, θ̂t, φ̂joint) of the joint training satisfies
LDt(θ̂t, φ̂joint) ≥ . (4) • With prob. at least 1 − 1k (over the randomness of pre-training), the solution (θ̂t, φ̂pre) of the
head-only fine-tuning satisfies LDt(θ̂t, φ̂pre) ≥ . (5)
As will be shown in the proof, not surprisingly, fine-tuning fails because it learns a random feature σ(e>[i]x) (where i ∈ [k]) for the source during pre-training which does not transfer to the target when i 6= 1. Joint training fails because it uses one neuron to learn a generalized linear model to overfit the target nt training data exactly, and then use another neuron to learn a random feature σ(e>[i]x) (where i ∈ [k]) for the source. The proof of Theorem 1 is deferred to Section B. In contrast, the following theorem shows that MeRLin can recover the ground truth of the target task:
Theorem 2. For any λ < λ0 where λ0 is some universal constant and any failure rate ξ > 0, if the target set size nt > Θ(log kξ ), with probability at least 1 − ξ, the feature extractor φ̂meta found by MeRLin and the head θ̂t(φ̂meta) trained on D̂trt recovers the ground truth of the target task:
LDt(θ̂t(φ̂meta), φ̂meta) = 0. (6)
Intuitively, MeRLin learns the transferable feature σ(e>[1]x) because it simultaneously fits the source and enables the generalization of the head on the target. The proof can be found in Section B.
2For theoretical analysis we consider only fine-tuning θt. It is worth noting that fine-tuning both θt and φ converges to the same solution as target-only training, which also has large generalization gap due to overfitting.

heading: 6 EXPERIMENTS
text: We evaluate MeRLin on several vision and NLP datasets. We show that (1) MeRLin consistently improves over baseline transfer learning algorithms including fine-tuning and joint training in both vision and NLP (Section 6.2), and (2) as indicated by our theory, MeRLin succeeds because it learns features that are more transferable than fine-tuning and joint training (Section 6.3).

heading: 6.1 SETUP: TASKS, MODELS, BASELINES, AND OUR ALGORITHMS
text: The evaluation metric for all tasks is the top-1 accuracy. We run all tasks for 3 times and report their means and standard deviations. Further experimental details are deferred to Section A.
Datasets and models. We consider the following four settings. The first three are object recognition problems (with different label sets). The fourth problem is the prominent NLP benchmark where the source is a language modeling task and the targets are classification problems.
SVHN or Fashion-MNIST → USPS. We use either SVHN (Netzer et al., 2011) (73K street view house numbers) or Fashion-MNIST Xiao et al. (2017) (50K clothes) as the source dataset. The target dataset is a random subset of 600 examples of USPS (Hull, 1994), a hand-written digit dataset. We down-sampled USPS to simulate the setting where the target dataset is much smaller than the source. We use LeNet (LeCun et al., 1998), a three-layer ReLU network in this experiment.
ImageNet→ CUB-200, Stanford Cars, or Caltech-256. We use ImageNet (Russakovsky et al., 2015) as the source dataset. The target dataset is Caltech-256, CUB-200 (Wah et al., 2011), or Stanford Cars (Krause et al., 2013). These datasets have 25468, 5994, 8144 labeled examples respectively, much smaller than ImageNet with 1.2M labeled examples. Caltech is a general image classification dataset of 256 classes. Stanford Cars and CUB are fine-grained classification datasets with 196 categories of cars and 200 categories of birds, respectively. We use ResNet-18 (He et al., 2016).
Food-101→ CUB-200. Food (Bossard et al., 2014) is a fine-grained classification dataset of 101 classes of food. Here we validate MeRLin when the gap between the source and target is large. We also use ResNet-18.
ImageNet→ Stanford Dogs, MIT-indoors, or Aircraft. We further test the proposed method with ResNet-50 (He et al., 2016) as baselines. We still use ImageNet (Russakovsky et al., 2015) as the source dataset. The target dataset is Stanford Dogs (Khosla et al., 2011), MIT-indoors (Quattoni & Torralba, 2009), or Aircraft (Maji et al., 2013) consisting of 12000, 5360, 8000 labeled examples, respectively.
Language modeling→ GLUE. Pre-training on language modeling tasks and fine-tuning on labeled dataset such as GLUE (Wang et al., 2019) is dominant following the success of BERT (Devlin et al.,
2019). We fine-tune BERT with MeRLin and evaluate it on the three tasks of GLUE with the smallest number of labeled examples which standard fine-tuning likely overfits.
Baselines. (1) target-only, (2) fine-tuning, and (3) joint-training have been defined in Section 2. Following standard practice, the initial learning rate of fine-tuning is 0.1× the initial learning rate of pre-training to avoid overfitting. For joint training, the overall objective can be formulated as: (1− α)LD̂s + αLD̂t . We tune α to achieve optimal performance. The fourth baseline is (4) L2-sp (Li et al., 2018), which fine-tunes the models with a regularization penalizing the parameter distance to the pre-trained feature extractor. (5) DELTA (Li et al., 2019) constrains the layer outputs selected with attention. (6) BSS (Chen et al., 2019) regularized the singular value of feature matrix. We also tuned the strength the L2-sp, Delta and BSS regularizer.
MeRLin. We perform standard training with cross entropy loss on the source domain while metalearning the representation in the target domain as described in Section 4.
MeRLin-ft. In BERT experiments, training on the source masked language modeling task is prohibitively time-consuming, so we opt to a light-weight variant instead: start from pre-trained BERT, and only meta-learn the representation in the target domain.

heading: 6.2 RESULTS
text: Results of digits classification and object recognition are provided in Table 1. MeRLin consistently outperforms all baselines. Note that the discrepancy between Fashion-MNIST and USPS is very large, where fine-tuning and joint training perform even worse than target-only. Nonetheless, MeRLin is still capable of harnessing the knowledge from the source domain. On Food-101→CUB-200, MeRLin improves over fine-tuning by 6.58%, indicating that MeRLin helps learn transferable features even when the gap between the source and target tasks is huge. In Table 2, we validate our method on GLUE tasks. MeRLin-ft outperforms standard BERT fine-tuning and L2-sp. Since MeRLin-ft only changes the training objective of fine-tuning, it can be easily applied to NLP models.
6.3 ANALYSIS
We empirically analyze the representations and verify that MeRLin indeed learns more transferable features than finetuning and joint training.
Intra-class to inter-class variance ratio. Suppose the representation of the j-th example of the i-th class is φi,j . µi =
1 Ni ∑Ni j=1 φi,j , and µ = 1 C ∑C i=1 µi. Then the
intra-class to inter-class variance ratio can be calculated as σ 2 intra
σ2inner = CN
∑ i,j ‖φi,j−µi‖
2∑ i ‖µi−µ‖2
. Low values of this ratio correspond to representations where classes are well-separated. Results on ImageNet→ CUB-200 and Stanford Cars task are shown in Figure 3. MeRLin reaches much smaller ratio than baselines.

heading: 7 ADDITIONAL RELATED WORK
text: Transfer learning is prevalent in deep learning applications. In computer vision, ImageNet pre-training is a common practice for nearly all target tasks. Early works (Oquab et al., 2014; Donahue et al., 2014) directly apply ImageNet features to target tasks. Fine-tuning from ImageNet pre-trained models has become dominant (Long et al., 2015; He et al., 2017; Kolesnikov et al., 2019). On the other
hand, transfer learning is also crucial to the success of NLP. Pre-training transformers on large-scale language tasks boosts performance on downstream tasks (Devlin et al., 2019; Yang et al., 2019).
A recent line of literature casts doubt on the consistency of transfer learning’s success (Raghu et al., 2019; He et al., 2018; Kornblith et al., 2019). Huh et al. (2016) observed that some set of examples in ImageNet are more transferable than the others. Geirhos et al. (2019) found out that the texture of ImageNet is not transferable to some target tasks. Training on the source dataset may also need early stopping to find optimal transferability Liu et al. (2019a); Neyshabur et al. (2020).
Meta-learning, originated from the learning to learn idea (Hochreiter et al., 2001; Vilalta & Drissi, 2002; Maclaurin et al., 2015; Zoph & Le, 2016), learns from multiple training tasks models that can be swiftly adapted to new tasks (Finn et al., 2017; Rajeswaran et al., 2019; Nichol et al., 2018b). Raghu et al. (2020); Goldblum et al. (2020) empirically studied the mechanism of MAML’s success. Computationally, our method uses bi-level optimization techniques similar to meta-learning work. E.g., Bertinetto et al. (2019) speeds up the implementation of MAML Finn et al. (2017) with closedform solution of the inner loop, which is a technique that we also use. However, the key difference between our paper from the meta-learning approach is that we only learn from a single target task and evaluate on it. Therefore, conceptually, our algorithm is closer to the learning-to-learn approach for hyperparameter optimization (Maclaurin et al., 2015; Zoph & Le, 2016), where there is a single distribution that generates the training and validation dataset.

heading: 8 CONCLUSION
text: We study the limitations of fine-tuning and joint training. To overcome their drawbacks, we propose meta representation learning to learn transferable features. Both theoretical and empirical evidence verify our findings. Results on vision and NLP tasks validate our method on real-world datasets. Our work raises many intriguing questions for further study. Could we apply meta-learning to heterogeneous target tasks? What’s more, future work can pay attention to disentangling transferable features from non-transferable features explicitly for better transfer learning.

heading: A ADDITIONAL DETAILS OF EXPERIMENTS

heading: A.1 THE SEMI-SYNTHETIC EXPERIMENT
text: The original CIFAR images are of resolution 32× 32. For the transferable dataset A, we reserve the upper 16× 32 and fill the lower half with [0.485, 0.456, 0.406] for the three channels (the mean of CIFAR-10 images). For the non-transferable dataset B, the lower part 16× 32 pixels are generated with i.i.d. gaussian distribution with the upper half filled with [0.485, 0.456, 0.406] similarly. To make the non-transferable part related to the labels, we set the mean of the gaussian distribution to 0.1× c, where c is the class index of the image. The variance of the gaussian noise is set to 0.2. We always clamp the images to [0, 1] to make the generated images valid. For the source dataset, we use 49500 CIFAR-10 images, while for the target, we use the other 500 to avoid memorizing target examples.
We use ResNet-32 implementation provided in github.com/akamaster/pytorch_ resnet_cifar10. We set the initial learning rate to 0.1, and decay the learning rate by 0.1 after every 50 epochs. We use t-SNE (van der Maaten & Hinton, 2008) visualizations provided in sklearn. The perplexity is set to 80.
A.2 IMPLEMENTATION ON REAL DATASETS
We implement all models on PyTorch with 2080Ti GPUs. All models are optimized by SGD with 0.9 momentum. For object recognition tasks, the initial learning rate is set to 0.01, with 5× 10−4 weight decay. The batch-size is set to 64. We run each model for 100 epochs. ImageNet pre-trained models can be found in torchvision. We use a batch size of 128 on the source dataset and 512 on the target dataset. The initial learning rate is set to 0.1 for training from scratch and 0.01 for ImageNet initialization. We decay the learning rate by 0.1 every 50 epochs until 150 epochs. The weight decay is set to 5 × 10−4. For GLUE tasks, we follow the standard practice of Devlin et al. (2019). The BERT model is provided in github.com/Meelfy/pytorch_pretrained_BERT. For each model, we set the head (classifier) to the top one linear layer. We use a batch size of 32. The learning rate is set to 5× 10−5 with 0.1 warmup proportion. During fine-tuning, the initial learning rate is 10 times smaller than training from scratch following standard practice. The hyper-parameter ρ is set to 2, and λ is found with cross validation. We randomly split the training set into 80% and 20%. We train on the 80% part and use the rest 20% as the validation set. We also provide the results of varying ρ and λ in Section A.7. In object recognition however, we also provides a pre-trained version: starting from ImageNet pre-trained solution.
A.3 IMPLEMENTING THE SPEED-UP VERSION
Practical implementation: speeding up with MSE loss. Training the head gθ in the inner loop of meta learning can be time-consuming. Even using implicit function theorem or implicit gradients as proposed in (Bai et al., 2019; Rajeswaran et al., 2019; Lin et al., 2020), we have to approximate the inverse of Hessian. To solve the optimization issues, we propose to analytically calculate the prediction of the linear head θt and directly back-prop to the feature extractor hφ. Thus, we only need to compute the gradient once in a single step. Concretely, suppose we use MSE-loss. Denote by H ∈ R
nt 2 ×m the feature matrix of the nt2 target samples in the target meta-training set D̂ tr t . Then θ̂t
in equation 1 can be analytically computed as θ̂t = (HH> + λI)−1y, where λ is a hyper-parameter for regularization. The objective of the outer loop can be directly computed as
minimize φ∈Φ,θs∈Θ
J(φ, θs) = LD̂s(θs, φ) + ρ 2
nt nt 2∑ i=1 `(g(HH>+λI)−1y ◦ hφ(xt ′ i ), y t′ i ). (7)
We implement the kernel speed-up version on classification tasks following Arora et al. (2019b). We treat the classification problems as multi-variate ridge regression. Suppose we have label c ∈ {1, 2, · · · , 10}. Then the target encoding for regression is −0.1× 1 + ec. For example, if the label is 3, then the encoding will be (−0.1,−0.1, 0.9, · · · ,−0.1). Then the parameters of the target head in the inner loop can be computed as θ̂t = (HH> + λI)−1Y. We then compute the MSE loss
on the target validation set: ‖Hvalθ̂t −Yval‖22 in the outer loop. We summarize the details of the vanilla version and the speed-up version in Algorithm 1 and Algorithm 2.
Algorithm 1 Meta Representation Learning (MeRLin).
1: Input: the source dataset D̂s and the evolving target dataset D̂t. 2: Output: learned representations φ. 3: for iter = 0 to MaxIter do 4: Initialize the target head θt. 5: Randomly sample target train set D̂trt and target validation set D̂valt from D̂t. 6: for t = 0 to n do 7: Train the target head on D̂trt : θt ← θt − η∇θtLD̂trt (θt, φ).8: end for 9: In the outer loop, update the representation φ and the source head θs:
(φ, θs)← (φ, θs)− η∇(φ,θs) [ LD̂s(θs, φ) + ρLD̂valt (θt, φ) ] .
10: end for
Algorithm 2 Meta Representation Learning (MeRLin): speed-up implementation.
1: Input: the source dataset D̂s and the evolving target dataset D̂t. 2: Output: learned representations φ. 3: for t = 0 to MaxIter do 4: Randomly sample target train set D̂trt and target validation set D̂valt from D̂t. 5: Analytically calculate the solution of target head θ̂t(φ) in the inner loop
θ̂t(φ) = (HH > + λI)−1Y.
6: In the outer loop, update the representation φ and the source head θs: (φ, θs)← (φ, θs)− η∇(φ,θs) [ LD̂s(θs, φ) + ρ 2
nt ‖Hvalθ̂t −Yval‖22
] .
7: end for

heading: A.4 DATASETS
text: CUB-200 (Wah et al., 2011) is a fine-grained dataset of 200 bird species. The training dataset consists of 5994 images and the test set consists of 5794 images. http://www.vision.caltech.edu/ visipedia/CUB-200-2011.html
Stanford Cars (Krause et al., 2013) dataset contains 16,185 images of 196 classes of cars. The data is split into 8,144 training images and 8,041 testing images. http://ai.stanford.edu/ ˜jkrause/cars/car_dataset.html
Food-101 (Bossard et al., 2014) is a fine-grained classification dataset of 101 kinds of food, with 750 training images and 250 test images for each kind. http://www.vision.ee.ethz.ch/ datasets_extra/food-101/
Caltech-256 (Griffin et al., 2007) is an object recognition dataset of 256 categories. In our experiments, the training set consists of 25468 images, and the test set consists of 5139 images.http://www.vision.caltech.edu/Image_Datasets/Caltech256/
MIT-indoors (Quattoni & Torralba, 2009) is a scene recognition dataset of 67 classes. The training set contains 80 examples for each class, and the test set contains 20 examples for each class. http://web.mit.edu/torralba/www/indoor.html
Stanford-Cars (Khosla et al., 2011) is a fine-grained classification dataset of 120 kinds of dogs. http://vision.stanford.edu/aditya86/ImageNetDogs/main.html
Aircraft (Maji et al., 2013) is an object recognition dataset of 100 aircraft variants. Each class has 80 images for training and 20 images for testing. http://www.robots.ox.ac.uk/˜vgg/ data/fgvc-aircraft/
MNIST (LeCun et al., 1998) is a dataset of hand-written digits. It has a training set of 60,000 examples, and a test set of 10,000 examples. We randomly sample 600 samples as the training set for the target domain. http://yann.lecun.com/exdb/mnist/
SVHN (Netzer et al., 2011) is a real-world image dataset of street view house numbers. It has 73257 digits for training, 26032 digits for testing. In Figure 3 of the main text, we randomly sample 600 digits for training, and all the 26032 digits are used in testing. http://ufldl.stanford.edu/ housenumbers/
USPS (Hull, 1994) is a database for handwritten text recognition. It has 7291 train and 2007 test images. We randomly sample 600 samples as the training set for the target domain. https: //www.kaggle.com/bistaumanga/usps-dataset

heading: A.5 FURTHER ABLATION STUDY.
text: We extend the last column of Table 2 in Table 4. We further compare with two variants of MeRLin as ablation study:
MeRLin (pre-trained). We first pre-train the model on the source dataset and then optimize the MeRLin objective starting from the pre-trained solution.
MeRLin-target-only. MeRLin-target-only only meta-learns representations on the target domain starting from random initialization. We test whether the meta-learning objective itself has regularization effect.
MeRLin (pre-trained) performs worse than MeRLin, but it still improves over fine-tuning and joint training. Note-that MeRLin (pre-trained) only need to train on ImageNet for 2 epochs, much shorter than joint training. MeRLin-target-only improves target-only by 8%, indicating that meta-learning helps avoid overfitting even without the source dataset.

heading: A.6 FEATURE-LABEL CORRELATION
text: Suppose the feature matrix is H, and the label vector is y, then the correlation between feature and label can be defined as y> ( H>H )−1 y. As is shown by Arora et al. (2019a); Cao & Gu (2019), this term is closely related to the generalization error of neural networks, with a smaller quantity indicating better generalization. We calculate y> ( H>H )−1 y on ImageNet→ CUB-200 and Stanford Cars. As shown in Figure 4(a), the features learned by MeRLin are more closely related
to labels than fine-tuning and joint training, indicating MeRLin is indeed learning more transferable features compared with baselines.

heading: A.7 SENSITIVITY OF THE PROPOSED METHOD TO HYPER-PARAMETERS.
text: We test the model on Food-101→CUB-200 with varying hyper-parameters ρ and λ. Results in Figure 4(b) indicate that the model is not sensitive to varying ρ and λ. Intuitively, larger ρ indicates more emphasis on the target meta-task. When ρ approaches 0, the performance of MRL is approaching fine-tuning. λ exerts regularization to the classifier in the inner loop training. It is also note worthy that λ can avoid the problem that HH> is occasionally invertible. Without λ the model can fail to converge sometimes.

heading: A.8 BOOTSTRAPPED CONFIDENCE INTERVAL OF OBJECTION RECOGNITION RESULTS
text: We run 10 seeds of each experiment on L2-sp and MeRLin in Table 1 and calculate 95% bootstrapped confidence interval of each results in Figure 5.

heading: B MISSING DETAILS IN SECTION 5

heading: B.1 PROOF OF THEOREM 1
text: Lemma 1. Suppose 0 ≤ ≤ 23 . For each solution θs, θt, φ satisfying Ex,y∼Dt [`(fθt,φ(x), y)] ≤ , the joint training objective Lλjoint(θs, θt, φ) is lower bounded:
Lλjoint(θs, θt, φ) = (1− α)LDs(θs, φ) + αLD̂t(θt, φ) + λ ( ‖θs‖2 + ‖θt‖2 + ‖φ‖2F ) ≥ min
µ
( 3λ
24/3 |µ|2/3 + 2 3 (1− α)(µ− 1)2
) + 3λ
24/3
( 1− √ 3
2
)2/3 .
Proof of Lemma 1. Define d× d matrix
A = m∑ i=1 θsiφiφ > i . (8)
Define x[1:k] and x[k+1:d] be the first k and last d− k dimensions of x. Ak,k, Ak,k̄ and Ak̄,k̄ be k× k, k × (d− k) and (d− k)× (d− k) matrices that correspond to the upper left, upper right and lower right part of A. For a random vector x where the first k dimensions are uniformly independently from {±1}, the last d− k dimensions are uniformly indepdently from {0,±1}, define random variables A1 = x > [1:k]Ak,kx[1:k], A2 = x > [k+1:d]Ak̄,k̄x[k+1:d]. (Note that x is defined on a different distribution than Ds.) We have bound
(1− α)Ex,y∼Ds [`(fθs,φ(x), y)] + λ ( ‖θs‖2 + 1
2 ‖φ‖2F
) (9)
=(1− α)Ex,y∼Ds [( x>[1:k]Ak,kx[1:k] + 2x > [1:k]Ak,k̄x[k+1:d] + x > [k+1:d]Ak̄,k̄x[k+1:d] − y )2] + λ ( ‖θs‖2 + 1
2 ‖φ‖2F
) (10)
≥(1− α)Ex,y∼Ds [( x>[1:k]Ak,kx[1:k] + x > [k+1:d]Ak̄,k̄x[k+1:d] − y )2] + λ ( ‖θs‖2 + 1
2 ‖φ‖2F ) (11)
=(1− α) ( 2 3 E [ (A1 − 1)2 ] + 4 3 E [(A1 − 1)A2] + E [ A22 ]) + λ ( ‖θs‖2 + 1 2 ‖φ‖2F ) (12)
=(1− α) ( 2 3 E [ (A1 +A2 − 1)2 ] + 1 3 E [ A22 ]) + λ ( ‖θs‖2 + 1 2 ‖φ‖2F ) (13)
≥2 3 (1− α) (E [A1 +A2]− 1)2 + 3λ 24/3 (|E [A1 +A2] |)2/3 (14)
The first inequality is because Ex,y∼Ds [( x>[1:k]Ak,kx[1:k] )( x>[1:k]Ak,k̄x[k+1:d] )] = 0, (15)
Ex,y∼Ds [( x>[k+1:d]Ak̄,k̄x[k+1:d] )( x>[1:k]Ak,k̄x[k+1:d] )] = 0, (16)
Ex,y∼Ds [( x>[1:k]Ak,k̄x[k+1:d] ) y ] = 0. (17)
The second inequality is because m∑ i=1 ( (θsi) 2 + 1 2 ‖φi‖2 ) ≥ 3 24/3 m∑ i=1 ( |θsi| · ‖φi‖ 2 )2/3
(18)
≥ 3 24/3 ( m∑ i=1 |θsi| · ‖φi‖ 2 )2/3 (19)
≥ 3 24/3 ( m∑ i=1 |A[i,i]| )2/3 , (20)
where the first inequality is AM-GM inequality, the second inequality is by concavity of (·)2/3. The third inequality is because for diagonal matrix D that has 1 at (i, i) if A[i,i] ≥ 0, −1 at (i, i) if A[i,i] < 0, we have
m∑ i=1 |A[i,i]| = tr(AD) = m∑ i=1 θsiφ > i Dφi ≤ m∑ i=1 |θsi| · ‖φi‖ 2 . (21)
On the other hand, for the target, we define d× d matrix
B = m∑ i=1 θtiφiφ > i . (22)
Define x[1] and x[2:d] be the first 1 and last d − 1 dimensions of x. B1,1, B1,1̄ and B1̄,1̄ be 1 × 1, 1× (d− 1) and (d− 1)× (d− 1) matrices that correspond to the upper left, upper right and lower right part of B. For a random vector x where the first dimension is uniformly independently from {±1}, the last d− 1 dimensions are uniformly indepdently from {0,±1}, define random variables B1 = x > [1]B1,1x[1], B2 = x > [2:d]B1̄,1̄x[2:d]. (Note that x is defined on a different distribution than Dt.)
Using similar argument as above, we have
Ex,y∼Dt [`(fθt,φ(x), y)] ≥ 2
3 (E [B1 +B2]− 1)2 . (23)
However, we know that Ex,y∼Dt [`(fθt,φ(x), y)] ≤ , so there has to be E [B1 +B2] ≥ 1− √ 3
2 , (24)
therefore we have
λ ( ‖θt‖2 + 1
2 ‖φ‖2F
) ≥ 3λ
24/3
( 1− √ 3
2
)2/3 . (25)
Summing up Equation 9 and Equation 25 finishes the proof.
Lemma 2. Assume φ̂t is a vector such that 〈φ̂t, xti〉 = xti[1] for all x t i ∈ D̂t, then there exists some solution (θs, θt, φ) such that
Lλjoint(θs, θt, φ) ≤ min µ
( 3λ
22/3 |µ|2/3 + 2 3 (1− α)(µ− 1)2
) + 3λ
22/3 ∥∥∥φ̂t∥∥∥4/3 2 . (26)
Proof of Lemma 2. Assume µ∗ ∈ arg minµ ( 3λ 22/3 |µ|2/3 + 23 (1− α)(µ− 1) 2 ) , then obviously µ∗ ∈ [0, 1]. Let φ1 = ( √ 2µ∗)1/3e1, φ2 = 2 1/6
‖φ̂t‖1/3 2
φ̂t, φi = 0 for i > 2, θs = (µ∗/2)1/3e1 and
θt = ‖φ̂t‖2/3
21/3 e2. Now we prove that this model satisfies the Equation 26.
First of all, we notice that for any xti ∈ D̂t, there is xti > ( m∑ i=1 θtiφiφ > i ) xti (27)
=xti > θt2φ2φ > 2 x t i (28)
=〈φ̂t, xti〉2 (29) =yti . (30)
Therefore we have
LD̂t(θt, φ) = 0 (31)
.
On the other hand, we have (1− α)LDs(θs, φ) + λ ( ‖θs‖2 + ‖φ1‖2 ) (32)
= 2 3 (1− α)(µ∗ − 1)2 + 3λ 22/3 |µ∗|2/3 (33)
= min µ
( 3λ
22/3 |µ|2/3 + 2 3 (1− α)(µ− 1)2
) . (34)
Plugging Equation 31 and Equation 32 into the formula of Lλjoint(θs, θt, φ) finishes the proof.
Lemma 3. Let X ∈ Rd×n be a random matrix where each entry is uniformly random and independently sample from {0,±1}, n < d2 . Let PXe1 be the projection of e1 to the column space of X . Then, there exists absolute constants c0 > 0 and C > 0, such that with probability at least 1− 4 exp(−Cd), there is
‖PXe1‖2 ≤ c0 √ n
d . (35)
Proof of Lemma 3. Let smin(X) and smax(X) be the minimal and maximal singular values of X respectively. Then we have
‖PXe1‖2 = ∥∥X(X>X)−1X>e1∥∥2 (36)
≤ ‖X‖op ∥∥(X>X)−1∥∥
op ∥∥X>e1∥∥2 (37) ≤ smax(X)(smin(X))−2 √ n. (38)
By Theorem 3.3 in Rudelson & Vershynin (2010), there exists constants c1, c2 > 0, such that
P (smin(X) ≤ c1( √ d− √ n)) ≤ 2 exp(−c2d). (39)
By Proposition 2.4 in Rudelson & Vershynin (2010), there exists constants c3, c4 > 0, such that
P (smax(X) ≥ c3( √ d+ √ n)) ≤ 2 exp(−c4d). (40)
Let C = min{c2, c4}, then with probability at least 1− 4 exp(Cd), there is
smax(X)(smin(X)) −2√n (41)
≤c3 c21
√ d+ √ n
( √ d− √ n)2
√ n (42)
≤ (2 + √ 2)c3
( √
2− 1)2c21
√ n
d , (43)
which completes the proof.
Proof of Theorem 1. We prove the joint training part of Theorem 1 following this intuition: (1) the total loss of each solution with target loss Ex,y∼Dt [`(fθt,φ(x), y)] ≤ is lower bounded as indicated by Lemma 1, and (2) there exists a solution with loss smaller than the aforementioned lower bound as indicated by Lemma 2.
By Lemma 1, for any θs, θt, φ satisfying Ex,y∼Dt [`(fθt,φ(x), y)] ≤ , the joint training loss Lλjoint(θs, θt, φ) is lower bounded,
Lλjoint(θs, θt, φ) ≥ min µ
( 3λ
24/3 |µ|2/3 + 2 3 (1− α)(µ− 1)2
) + 3λ
24/3
( 1− √ 3
2
)2/3 . (44)
Let PXe1 be the projection of vector e1 to the subspace spanned by the target data. According to Lemma 2, there exists some solution (θs, θt, φ) such that
Lλjoint(θs, θt, φ) ≤ µ ( 3λ
22/3 |µ|2/3 + 2 3 (1− α)(µ− 1)2
) + 3λ
22/3 ∥∥∥φ̂t∥∥∥4/3 2 . (45)
Let 0 > 0 be a constant such that 124/3 ( 1− √ 3 0 2 )2/3 > 1 22/3 − 1 24/3 . According to Lemma 3, there exists absolute constants c ∈ (0, 1), C > 0, such that so long as nt ≤ cd, there is with probability at least 1− 4 exp(−Cd),
1
24/3
( 1− √ 3 0 2 )2/3 > 1 22/3 − 1 24/3 + 1 22/3 ∥∥∥φ̂t∥∥∥4/3 2 . (46)
Now we prove the upper bound in Equation 45 is smaller than the lower bound in Equation 44. This is because
min µ
( 3λ
22/3 |µ|2/3 + 2 3 (1− α)(µ− 1)2
) + 3λ
22/3 ∥∥∥φ̂t∥∥∥4/3 2
(47)
= min µ
( 3λ
22/3 |µ|2/3 + 2 3 (1− α)(µ− 1)2
) − 3λ(2
2/3 − 1 24/3 ) + 3λ 22/3 ∥∥∥φ̂t∥∥∥4/3 2 + 3λ( 1 22/3 − 1 24/3 )
(48)
≤min µ
( 3λ
24/3 |µ|2/3 + 2 3 (1− α)(µ− 1)2
) + 3λ
22/3 ∥∥∥φ̂t∥∥∥4/3 2 + 3λ( 1 22/3 − 1 24/3 ) (49)
≤min µ
( 3λ
24/3 |µ|2/3 + 2 3 (1− α)(µ− 1)2
) + 3λ
24/3
( 1− √ 3 0 2 )2/3 , (50)
where the first inequality uses that fact that |µ| < 1 for the optimal µ, the second inequality is by Equation 46. This completes the proof for joint training.
Then, we prove the result about fine-tuning. According to Lemma 4, any minimizer (θ̂s, φ̂pre) of LλDs(θs, φ) either satisfies φ̂pre = 0, or only one φi is non-zero but looks like (up to scaling) ej for j ∈ [k]. When φ̂pre = 0, there is
Ex,y∼Dt [ `(fθ̂t,φ̂pre(x), y) ] = 2
3 >
10 27 . (51)
When only one φi is non-zero but looks like ej for j ∈ [k], since all the first k dimensions are equivalent for the source task, with probability 1− 1k , this dimension is j 6= 1. The target funciton fine-tuned on this φ̂pre looks like fθ̂t,φ̂pre(x) = γx 2 j for some γ ∈ R, so there is
Ex,y∼Dt [ `(fθ̂t,φ̂pre(x), y) ] = Ex,y∼Dt [ (γx2t − x21)2 ] (52)
= γ2Ex,y∼Dt [ x4t ] − 2γEx,y∼Dt [ x2tx 2 1 ] + Ex,y∼Dt [ x41 ]
(53)
= 2 3 γ2 − 8 9 γ + 2 3 ≥ 10 27 . (54)
Combining these two possibilities finishes the proof for fine-tuning. Finnaly, setting = min{ 0, 1027} finishes the proof of Theorem 1.

heading: B.2 PROOF OF THEOREM 2
text: Lemma 4. Define the source loss as LλDs(θs, φ) = LDs(θs, φ) + λ ( ‖θs‖2 + ‖φ‖2F ) .
Then, for any λ > 0, any minimizer of LλDs(θs, φ) is one of the following cases:
(i) θs = 0 and φ = 0.
(ii) for one i ∈ [m], θsi > 0, φi = ±( √
2θsi) · ej for some j ≤ k; for all other i ∈ [m], |θsi| = ‖φi‖ = 0.
Furthermore, when 0 < λ < 0.1, all the minimizers look like (ii).
Proof of lemma 4. Define d× d matrix
A = m∑ i=1 θsiφiφ > i . (55)
Define x[1:k] and x[k+1:d] be the first k and last d− k dimensions of x. Ak,k, Ak,k̄ and Ak̄,k̄ be k× k, k × (d− k) and (d− k)× (d− k) matrices that correspond to the upper left, upper right and lower right part of A. For a random vector x where the first k dimensions are uniformly independently from {±1}, the last d− k dimensions are uniformly indepdently from {0,±1}, define random variables A1 = x > [1:k]Ak,kx[1:k], A2 = x > [k+1:d]Ak̄,k̄x[k+1:d]. (Note that x is defined on a different distribution than Ds.) The loss part of LλDs(θs, φ) can be lower bounded by:
Ex,y∼Ds [`(fθs,φ(x), y)] (56) =Ex,y∼Ds [( x>[1:k]Ak,kx[1:k] + 2x > [1:k]Ak,k̄x[k+1:d] + x > [k+1:d]Ak̄,k̄x[k+1:d] − y )2] (57)
≥Ex,y∼Ds [( x>[1:k]Ak,kx[1:k] + x > [k+1:d]Ak̄,k̄x[k+1:d] − y )2] (58)
= 2 3 E [ (A1 − 1)2 ] + 4 3 E [(A1 − 1)A2] + E [ A22 ]
(59)
= 2 3 E [ (A1 +A2 − 1)2 ] + 1 3 E [ A22 ] . (60)
The inequality is because Ex,y∼Ds [( x>[1:k]Ak,kx[1:k] )( x>[1:k]Ak,k̄x[k+1:d] )] = 0, (61)
Ex,y∼Ds [( x>[k+1:d]Ak̄,k̄x[k+1:d] )( x>[1:k]Ak,k̄x[k+1:d] )] = 0, (62)
Ex,y∼Ds [( x>[1:k]Ak,k̄x[k+1:d] ) y ] = 0. (63)
The inequality is equality if and only if Ak,k̄ = 0.
The regularizer part of LλDs(θs, φ) can be lower bounded by: m∑ i=1 ( (θsi) 2 + ‖φi‖2 ) ≥ 3 22/3 m∑ i=1 ( |θsi| · ‖φi‖ 2 )2/3
(64)
≥ 3 22/3 ( m∑ i=1 |θsi| · ‖φi‖ 2 )2/3 (65)
≥ 3 22/3 ( m∑ i=1 |A[i,i]| )2/3 , (66)
where the first inequality is AM-GM inequality, the second inequality is by concavity of (·)2/3. The third inequality is because for diagonal matrix D that has 1 at (i, i) if A[i,i] ≥ 0, −1 at (i, i) if A[i,i] < 0, we have
m∑ i=1 |A[i,i]| = tr(AD) = m∑ i=1 θsiφ > i Dφi ≤ m∑ i=1 |θsi| · ‖φi‖ 2 . (67)
All the inequalities are equality if and only if (θsi) 2 = 12 ‖φi‖ 2 > 0 for at most one i ∈ [m], and for all other i ∈ [m] there is |θsi| = ‖φi‖ = 0. Combining the two parts gives a lower bound for LλDs(θs, φ):
LλDs(θs, φ) ≥ 3λ
22/3 ( m∑ i=1 |A[i,i]| )2/3 + 2 3 E [ (A1 +A2 − 1)2 ] + 1 3 E [ A22 ]
(68)
≥ 3λ 22/3
( |tr(Ak,k + 2
3 Ak̄,k̄)|
)2/3 + 2
3 (E [A1 +A2]− 1)2 , (69)
where both inequalitites are equality if and only if Ak̄,k̄ = 0 (therefore A2 = 0) and V ar [A1] = 0 (therefore Ak,k is diagonal by Lemma 5).
Notice that E[A1 + A2] = tr(Ak,k + 23Ak̄,k̄), the above lower bound is further minimized when E[A1] = µ∗ where µ∗ is the minimizer of function L(µ) = 3λ22/3 (|µ|) 2/3 + 23 (µ− 1) 2.
To see when this lower bound is achieved, we combine all the conditions for the inequalities to be equality. When µ∗ = 0, this lower bound is achieved if and only if θs = 0 and φ = 0. When µ∗ > 0, this lower bound is only achieved when the solution look like this: for one i ∈ [m], θsi = (µ ∗ 2 ) 1/3,
φi = ±( √
2θsi) · ej for some j ≤ k; for all other i ∈ [m], |θsi| = ‖φi‖ = 0. Obviously, there is either µ∗ = 0 or µ∗ > 0. Also, when λ < 0.1, the minimizer µ∗ of L(µ) is strictly larger than 0 (since L(1) < L(0)). So this completes the proof.
Lemma 5. Let M ∈ Rk×k be a symmetric matrix, x ∈ Rk is a random vector where each dimension is indepedently uniformly from {±1}. Then, Var[x>Mx] = 0 if and only if M is a diagonal matrix.
Proof of Lemma 5. In one direction, when M is diagonal matrix, obviously Var[x>Mx] = 0. In the other direction, when Var[x>Mx] = 0, there has to be x>Mx be the same for all x ∈ {±1}k. For any i 6= j, let x(1) = 1− 2ei − 2ej , x(2) = 1, x(3) = 1− 2ei, x(4) = 1− 2ej . Then the (i, j) element ofM is 18 ( x(1) > Mx(1) + x(2) > Mx(2) − x(3)>Mx(3) − x(4)>Mx(4) ) , which is 0. SoM has to be a diagonal matrix.
Proof of Theorem 2. Define the source loss as in Lemma 4, then we have
Lλmeta(θs, φ) = L λ Ds(θs, φ) + E [ LD̂valt (θ̂t(φ), φ) ] . (70)
By Lemma 4, the source loss LλDs(θs, φ) is minimized by a set of solutions that look like this: for one i ∈ [m], θsi > 0, φi = ±( √ 2θsi) · ej for some j ≤ k; for all other i ∈ [m], |θsi| = ‖φi‖ = 0.
When j = 1, the only feature in φ is e1. When nt ≥ 18 log 2ξ , according to Chernoff bound, with probability at least 1 − ξ2 there is strictly less than half of the data satisfy x1 = 0. Therefore, any D̂trt contains data with x1 6= 0, and the only target head that fits D̂trt has to recover the ground truth. Hence there is E [ LD̂valt (θ̂t(φ), φ) ] = 0.
When j 6= 1, the only feature is ej . This feature can be used to fit the target data if and only if either x2i[j] = x 2 i[1] for all target data xi, or xi[1] = 0 for all xi. Since there are at most k − 1 possible j, by union bound we know the probability of any of these happens for any j 6= 1 is at most k( 23 ) nt .
Hence, when nt ≥ 3 log 2kξ , the probability of any ej fits the target data is smaller than ξ 2 . Therefore, with probabiltiy 1− ξ2 , E [ LD̂valt (θ̂t(φ), φ) ] > 0 for any j 6= 1.
So with probability at least 1− ξ, the only minimizer of Lλmeta(θs, φ) is the subset of minimizers of LλDs(θs, φ) with feature e1, and with this φ and any random D̂ tr t , the only θt(D̂trt , φ) that fits the target
recovers the ground truth, i.e., Ex,y∼Dt [ `θt(D̂trt ,φ),φ (x, y) ] = 0.

Review summaries:
The paper investigates failure cases for transfer learning (fine-tuning and joint training), specifically in the context where training on the source data may highlight features that are irrelevant for the target data. This is done through semi-synthetic data. Based on the insights, the authors present an approach called Meta Representation Learning (MeRLin) inspired by Meta Learning and Learning-to-learn approaches. This approach is evaluated on several real-world transfer-learning tasks from vision and NLP. The authors also derive theoretical results on constructed data distributions for which superiority of Merlin can be shown analytically.

This paper first investigates when and why fine-tuning and joint training are not the best methods for transfer learning. The authors generate a toy dataset, in which the source-specific and transferable features are thus clearly distinguishable, for the transfer learning tasks. As both fine-tuning and joint training are not optimal solutions for the experimental dataset, they further design MeRLin, which is based on the meta-learning mechanism, to learn a generalizable model weight.