Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: ## Summary of the Work
The work analyzes the failure modes of fine-tuning and joint training for multi-task learning in a computer vision setting, then proposes a meta-learning algorithm dubbed "MeRLin" to address these failure modes using bi-level optimization. The outer optimization minimizes the loss of inner-trained network when it is used to learn a new classification head for a validation set from the target domain. 

The authors first motivate their work by discussing the success and failures of joint training and fine-tuning, then they use principled experiments to verify hypotheses about the failure modes of these two popular transfer methods. Thereafter, they introduce their proposed method (MeRLin), provide a theoretical analysis which compares its behavior to fine-tuning and joint training in a limited problem class, and then present empirical results showing the effectiveness of MeRLin on several popular CV and NLP benchmarks. 

## Pros and Cons

### Pros
* Principled approach to analyzing the failure modes of fine-tuning and joint training


### Cons
* Requiring data from the target domain during training negates many of the sought-after benefits of transfer learning (e.g. re-using pre-trained models quickly without lengthy pre-training, adapting to unforseen new tasks, etc.)

## Evaluation
### Quality
3/5
The overall quality of execution of this work is high, but not flawless.   The work's laudable careful theoretical and synthetic-empirical evaluations give way to less-careful statistical comparisons of the empirical-nonsynthetic results. I believe works making specific performance claims based on empirical experiments without exception need to provide specific evidence in the form of statistical tests of significance, especially in cases like these when the performance gaps between the proposed method and baselines are so small. 

### Clarity
5/5
The text and diagrams themselves are clean, lucid, well-written, and well-presented. I don't personally find the analysis in Sec. 5 to be particularly helpful to understanding the method or the failure modes of fine-tuning and joint training, and would be happy to see it moved to the appendix in favor of more careful empirical evaluation, but perhaps some readers find it easier to understand concepts with theory.

### Originality
4/5
This work has clear relationships to many prior works on parameter-space meta-learning and automatic hyperparameter tuning, and its own spin on the problem, which the authors acknowledge and provide context for. I have erred on the side of a higher rating in this category, thought don't have encyclopedic experience with all of the background literature. While the proposal in the work is not eye-wateringly novel considering previous work on regularization and meta-learning, novelty in ML research is, in general, overrated, and I believe this work is sufficiently novel to be a contribution.

### Significance
2/5
The analysis and synthetic experiments in this work are, I believe, a significant-enough contribution to the field irrespective of the proposed algorithm. If the paper itself contained only extended versions of the theoretical and synthetic-empirical experiment portions, I think it would be significant enough.

Unfortunately, I have concerns about the significance of this work--inclusive of the proposed algorithm, which readers will focus on--for 2 reasons:
(1) The proposed transfer solution negates many perceived and desired benefits of transfer and meta-learning, because it requires access to samples from the target distribution at train-time. Specifically, most researchers and practitioners pursue transfer learning specifically because they anticipate *not* having access to target samples at train time, e.g. in the case of adapting CV classifiers to unforseen target samples, or in the case of re-using a very large pre-trained CV or NLP model which--might be impractical for the end-user to train--for some unforseen application by an end user. The work does not address or acknowledge at all this slight compromise of the problem setting, or make efforts to justify its practicality in spite of this.
(2) While the synthetic experiments show great performance benefits to the proposed method in a fabricated setting, the empirical-non-synthetic experiments show very little improvement over the baselines, especially L2-sp (Li, et al. 2018), and it's not clear from the provided analysis that the performance differences presented are even statistically-significantly different from L2-sp. This is especially notable given that fact that L2-sp *does not* require access to target samples at train-time (i.e. it conforms to the more-general transfer learning setting), while MeRLin compromises the setting to achieve apparently-comparable but not substantially-improved results. 

### Misc Editorial Comments and Reviewer's Notes

#### Claims
* The reader will learn about "when and why fine-tuning and joint training can be suboptimal or even harmful for transfer learning"
* Proposed method MERLIN empirically outperforms SOTA transfer learning algorithms on vision and NLP benchmarks
* Proves that MERLIN recovers the target ground truth model under some assumptions on NN architecture and source distributions, and that pre-training and join training fail


#### Mechanisms
* Ensuring that a head fit on top of the learned representations with (target) training data also performs well on (target) validation data


#### 1. Introduction
* Does the assertion "joint training easily overfits to the target especially when the target dataset is small" hold if the target dataset is intentionally oversampled so that its examples appear as often as the source dataset? If we don't have the answer to this question, it could be that we are simply observing the effect of the data distribution being skewed.
* "any proposed algorithm [...] should use the source and target simultaneously..." -- I think this statement is overly-strong considering the results in the paper. Certainly this is *one way* to ensure that we "extract shared structures," but it also presupposes we have the target up front, which is opposed to many sought-after benefits of a transfer learning approach. Using the same evidence, we can argue instead that transfer learning methods should make use of as much data as possible, or search for data sources which are more diverse and general than both the source and the target.

#### 5. Theoretical Analysis
* I find this lucid but perhaps unnecessary for the reader's understanding of the concept. It seems fairly intuitive that a learning algorithm with no access, or unbalanced access to to the target distribution will fail to learn features which generalize to that target distribution, and a learning algorithm which is intentionally supervised, using a loss computed on samples from the target distribution, will learn features which generalize to both the source and the target. *The important question then is "do we give up more than we gain by requiring access to the target distribution?"* I recommend moving this portion to the Appendix to make more room for justifying the method and better in-text coverage of empirical results.

#### 6. Experiments
* Many of these results would seem to be quite close, especially when using only 3 trials and simple mean/stddev. How do the results compare with a real confidence interval? e.g. a 95% bootstrapped interval (note the seaborn package will easily calculate these for you)? Are the differences observed statistically significant? 


review_title: A strong piece of theory and and principled analysis which gives way to an algorithm with questionable benefits

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper first investigates when and why fine-tuning and joint training are not the best methods for transfer learning. The authors generate a toy dataset, in which the source-specific and transferable features are thus clearly distinguishable, for the transfer learning tasks. As both fine-tuning and joint training are not optimal solutions for the experimental dataset, they further design MeRLin, which is based on the meta-learning mechanism, to learn a generalizable model weight.

*Strengths*:
1. In order to investigate how a model (including a feature extractor and a classifier) interacts with the source-specific feature and transferable feature within a data, the authors generate a toy dataset and examine the effectiveness of fine-tuning and joint-training methods. This provides clear and insightful experimental observations for the further understanding of transfer feature learning.

2. The observation that the fine-tuning and the joint training tends to learn the "convenient" feature (i.e., both methods tend to learn easy source-specific feature rather than a transferable feature) provides a reasonable argument why both models cannot work well.

3. The designed meta-learning-based method is convincing empirically, as the objective function implies the model has to learn a generalizable feature extractor for loss minimization.  

*Weakness/comments*
1. Although the proposed demonstrates the above observations, the connection to existing domain adaptation/domain generation works seems lacking. Some of these works also aim to learn the feature, which can be generalized to the target domain. And the source-specific feature can be also regarded as the domain shift between source and target domains (so that it cannot be generalized by using fine-tuning). I understand the problem settings of domain adaptation/domain generation are a bit different from the setting used in the paper, while such discussion can further improve the completeness of this paper. 

2. Another weakness is the baseline methods, as these baseline methods are too simple. I am wondering about the comparison to existing works on the relevant areas (e.g., other general feature learning or domain-relevant works). 

3. One reason why people use pretraining method is neglected: one does not need to access the source dataset when the target task is trained. In some practical scenarios, one cannot access the large scale of source dataset (i.e., imagenet), and this limits the use of the proposed model, which requires access to source data. But this point is just a comment, not a weakness as the authors have clarified the problem scope at the beginning of the paper. 

Overall, the paper presents an insightful study and a reasonable new method. I am inclined to the score "Marginally above acceptance threshold".
review_title: Insightful study and reasonable method; while experiment and related works can be further polished

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary
The paper investigates failure cases for transfer learning (fine-tuning and joint training), specifically in the context where training on the source data may highlight features that are irrelevant for the target data. This is done through semi-synthetic data. Based on the insights, the authors present an approach called Meta Representation Learning (MeRLin) inspired by Meta Learning and Learning-to-learn approaches. This approach is evaluated on several real-world transfer-learning tasks from vision and NLP. The authors also derive theoretical results on constructed data distributions for which superiority of Merlin can be shown analytically.

### Quality, clarity, originality and significance

The work is well-motivated, and the paper is well-written and clear. 

The "failure modes" and the behavior on the semi-synthetic data are not entirely surprising in my opinion (and I would have expected more analysis after reading the introduction), but they highlight the targeted problem well.

The suggested algorithm is well-motivated and linked to existing work. However in many practical settings, it may be infeasible to train including the whole source data for reasons of compute and data availability, which is why many transfer-learning approaches do not assume the availability of the source data during transfer. This scenario is encountered in the case of BERT, where the authors then "only meta-learn the representation". In that setting, Merlin seems to reduce to reduce to a changed objective inspired by meta-learning and learning-to-learn. This is not a problem in itself, but it makes it less clear, what exactly the authors think that the main contribution of the paper is.

The experimental results look good at first glance, but overall I found it hard to evaluate how convincing they are because of several potential problems:
* The experimental results are not compared to any results from the existing literature, all baselines and comparisons are entirely from this paper. This makes it hard to know how much careful tuning went into the proposed algorithm versus the baselines. I think at least *some* comparison to results from the literature should be possible or it should be carefully explained why such an overfitting to the proposed algorithm clearly did not happen. 
* There are a lot of seemingly random choices in the selection of the datasets, e.g. why USPS and not MNIST, why CUB/Caltech/Cars and not e.g. CIFAR, Pets, Flowers, or Birds? I'm not saying any of these choices are inherently better, but the authors should explain *why* these datasets are chosen, to avoid the impression that the datasets may have been chosen because the proposed method works particularly well on these dataset combinations. One way to make the results stronger would therefore be to use combinations of datasets that other, previous work has already used, which would also enable a direct comparison (see above).
* Similarly, it is unclear why only a subset of GLUE was used and not the full GLUE benchmark. Again, it would be better in my opinion to use all of GLUE to avoid the impression that the subset was hand-picked. Also, I tried to find results on GLUE for direct comparison, because they should exist, but was not successful immediately, because the presented results are on the dev-set (Table 2) while usually results for comparison come from the official test set as far as I understand? E.g. the BERT paper seems to have tuned on dev and then reported numbers on test, but I did not find dev-set numbers in a quick search. For something fairly standard as BERT-base and GLUE I think it would be good to be able to find numbers in the literature that are *directly comparable*. 
* Again in a similar fashion, Sec.6.3. evaluates only two target data sets, why only these? And A.5 looks only at Food->Cub, why only this pair?

### Pros and cons
* Pros: interesting approach, well-motivated, well-written, interesting theoretical analysis
* Cons: experimental results are not entirely convincing, the main message is not 100% clear to me

### Minor details and comments
* The paper is fairly "squeezed": E.g. lack of vertical space after captions of Figures 1 and 2; also the main suggested algorithm is not explained in detail in the main text, but deferred to the Appendix. It might be more readable to try to shorten the content.
* "Pre-training only" is not a complete transfer-learning algorithm because of potential class number mismatch, yet it is used for comparison (e.g. Fig. 2b). This only works in this very specially constructed scenario, it seems. Maybe this should be mentioned?
* "Target-only" is discussed in the context of Fig. 2 but it is not contained in the Figure, why? Also, "fine-tuning" results are not shown in Fig 2b. 
* Towards the end of Section 3, when discussing the synthetic data, "overfitting" is mentioned twice as a problem. There exist several approaches that are often used to try to overcome overfitting, and the authors even use L2-sp as a baseline in Section 6. It remains a bit unclear how much a careful combination of fine-tuning and such methods could help here. I don't think this is a major shortcoming, though, because the artificial setup is constructed exactly in such a way that the pre-trained classifier will focus on the non-transferable features.
* Table 1: For ImageNet->C256 L2-sp seems to be not clearly worse than MeRLin given the standard deviations of three runs. Maybe both should be bolded?
* A.7 "the model is not sensitive to varying ρ and λ." - it was not clear to me how much this statement can be derived from the figure.

* Minor comments / typos:
  * Caption of Fig.1 - I did not see how this is an illustration of the "features learned"
  * End of Sec.3: "on a much simplified settings"
  * page 5: "Merlin to by changing" - "to" too much
  * page 5: "with AB as the source ." - space before "."
  * page 6: "with their own best regularization strength" - I did not understand this phrase here.
  * before Eq.(4) - what is $\Omega$ here?
  * In Theorems 1 and 2, what exactly is meant by "universal constant"?
  * after Eq.(6): "because its simultaneously fits" -> it
  * Food-101->CUB: here, the model is not mentioned. I guess it was also ResNet-18?
  * page 7 "join training" -> joint
  * page 8 "the our method"
  * References: some letters in the reference titles are lowercased when they shouldn't be, e.g. "cnns", "Mask r-cnn", "reuse? towards", "t-sne", "mnist", "Xlnet". Also, several references do not contain a "publishing venue", e.g. Kolesnikov 2019, Lin 2002, Liu 2019a, Neyshabur 2020 - in the age of search engines this is not a real problem, but it might be nice to give at least an arXiv number if available or some of these have probably appeared in conferences or journals.
  * page 12: "images is of resolution"
  * A.2 "lambda is found with cross validation" - how exactly?
  * A.4 for completeness maybe include ImageNet and USPS?
  * Figure 4 caption: "textbfSensitivity"
review_title: Interesting meta-learning ideas for transfer-learning

Meta-review: The paper compares transfer learning with fine-tuning and joint training and then proposes a new approach (Merlin). Reviewers have pointed to the fact that Merlin works in a setting that is different from normal transfer learning settings (it assumes some target domain data is available during training). The authors acknowledge this and think it can still be a reasonable setting, but of course it makes comparisons more difficult. Overall, while there are interesting analysis and results, the paper remains borderline and more work should be done to make it a good contribution, including significantly improving the presentation to make clear the distinction in settings. I therefore recommend to reject the paper.
Decision: Reject