Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper revisits the agreement of recent results of transfer learning methods outperforming meta-learning algorithms in few-shot learning domains, and claims that they are in fact not much different in performance particularly for a dataset with low diversity. To quantify the diversity, a new metric, called diversity coefficient, is proposed by leveraging the vectorized representation of each task obtained by Task2Vec. In empirical study, the authors' claim seems to be valid as the accuracy are pretty much similar to each other in between MAML and USL (i.e., a transfer learning method) using Cifar-FS and Mini-ImageNet, both of which have low diversity.
strengths_and_weaknesses: (Strengths)
1. The main claim of this paper sounds very important as it tries to break the common knowledge agreed by many recent works in few-shot learning.
2. Experiments are well designed in a way that the corresponding results properly justify the main claims and questions.

(Weaknesses)
1. Although the paper starts with ambitious insights, it does not properly give a meaningful conclusion in the end. There is no explanation on why low diversity leads to the similar performance of MAML and USL. Furthermore, when the diversity gets higher, it seems that the common knowledge is still correct as USL starts to outperform MAML. Thus, it would be much more interesting if the paper deeply investigates the hidden relationship between diversity and performance of both methodologies in few-shot learning. At the present form, the paper looks kind of an intermediate work on the way to its finalized version.

2. The proposed diversity metric is adequate, but its novelty is mostly coming from the corresponding preliminary work, i.e., Task2Vec. What is newly proposed in this work is to obtain the expected distance between tasks via their vectorized representations.

3. In terms of presentation quality, this write-up should be more improved and finalized. There are some typos and missing references throughout the paper, and most of figures are in low quality and hard to recognize.
questions: 1. The experimental results of Figure 2 is somewhat obvious. Are they intended to see models trained by MAML and USL are not the same in their weights?

2. Could you provide any insights on why diversity matters in the performance of MAML and USL? 

3. Given a high diversity dataset, USL seems to be still better than MAML. Doesn't this imply that USL deals with a more challenging learning problem setting better than MAML?

4. How can we trust the proposed metric indeed well reflects the true diversity? Any experimental justification on that?
limitations: Potential negative societal impact is neither discussed nor applicable to this work.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper presents a novel metric which quantifies the diversity of few-shot learning benchmarks. The authors compare MAML with transfer learning w.r.t. multiple aspects on two few-shot learning benchmarks. Several empirical discoveries are presented including that transfer learning fails to outperform MAML when the intrinsic diversity is low for the evaluation benchmark.
strengths_and_weaknesses: Strengths:
- The paper is quite easy to follow.
- The paper compares MAML and transfer learning from an interesting perspective, the so-called "problem centric" approach.
- The idea of rethinking the diversity of datasets is important for analyzing models, and this paper makes a good case for it by conducting empirical evaluations.
- The diversity coefficient is intuitive.

Weakness:
- The empirical evaluation needs to be stronger. 
- Some parts are not clear.
- Minor typos.
Please see questions for details.
questions: 1. In the empirical evaluation, it seems that only classification task and ConvNet-based models are used. For making a concrete conclusion, more tasks and diverse architectures (e.g. transformer-based backbone) will be more persuasive. Otherwise, the scope of the conclusion needs to be narrowed down.
2. The definition of tasks is not clear in the caption of Table 1. Are the 500 few-shot learning tasks subsets from the original dataset? Do they have disjoint classes? Does changing the definition of these tasks results in different diversity coefficient?
3. Line 292, 396, the section number is missing.
limitations: The limitation of the work is discussed in Section 6. The potential negative societal impact is not discussed.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The aim of the paper is to empirically compare the performance of transfer-learning (fine-tuning last fully connected layer in a pre-trained neural network) and meta-learning (and in particular, MAML - an instance of meta-learning). The two learning approaches of interest are investigated in different points of views: intrinsic diversity of the dataset used, similarity between their features at some imtermediate layers of the neural networks used and different model sizes. The empirical results show that MAML and fine-tuning a pre-trained model are equivalent in terms of performance.
strengths_and_weaknesses: **Strengths**

The major strength of the paper is to study the difference between meta-learning and transfer-learning. This is an interesting research direction to understand which method performs better in which setting. Also, as stated at the end of the introduction section, it provides another point of view in terms of evaluation, not just simply making larger and larger datasets.

Another strength of the paper is to quantify the characteristic of tasks sampled according to the episode setting. The newly-proposed metric, *diversity coefficient* , is based on the representation of tasks, known as Task2Vec, to calculate the cosine similarity between tasks belonging to the same dataset. This quantifies the "information" of the dataset, providing insights into the performances of the model of interest.

One more strength is that the paper is gone through a thorough review of the literature with an extended section in the literature.

---

**Weaknesses**

Firstly, the paper lacks of a coherence to connect all the contributions together. As I understand, the main goal is to show that meta-learning and transfer learning share similar performances even when the dataset used is diversity enough or not. However, the presentation and writing leads to a hard time for me to get to the point.

Secondly, the newly-proposed metric, known as *diversity coefficient*, is a straight-forward extension from Task2Vec, which heavily relies on the *probe* network used. Thus, I believe that using a single network is not enough, but we might need an ensemble of networks to marginalize out the bias of *probe* networks. In addition, the proposed metric might not work well in the present of outlier tasks since it is based on the expected (or average) similarity distance. Furthermore, as shown in (Dhillon et. al 2020, Figure 1) and (Nguyen et. al 2021, Figure 1), the accuracy of different tasks evaluated by the same model varies significantly. Thus, simply picking a small number of tasks, e.g. 500 tasks as mentioned in Table 1, to calculate the metric is inadequate. And since the number of tasks formed from a dataset is very large, e.g. there are $n = \binom{64}{5} = 7,624,512$ 5-way classification tasks from the training set of mini-ImageNet, the number of distances calculate would be $\mathcal{O}(n^{2})$, making this metric intractable when there are more classes. Of course, we might not need to take all tasks into account, but only consider the set of *core* tasks. However, finding that set is another problem.

Finally, the evaluation to compare meta-learning and transfer-learning is based on classification accuracy on two datasets only. I believe that this might be insufficient to have a clear conclusion.

**References**
- Guneet S Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. "A baseline for few-shot image classification". In *International Conference on Learning Representations*, 2020.
- Cuong C. Nguyen, Thanh-Toan Do, and Gustavo Carneiro. "Probabilistic task modelling for meta-learning". In *Uncertainty in Artificial Intelligence*, 2021.
questions: The proposed *diversity coefficient* based on an average distance is not very convincing. I wonder if the diversity of a dataset can be calculated informatically. What I mean is that given a set of vectors representing tasks (obtained from Task2Vec), could we estimate the task distribution and explicitly calculate the entropy of that distribution, or implicitly calculate the entropy from those vectors? The reason for entropy is that the higher the entropy, the more informative that task distribution, which, to me, means the higher diversity.

I have a concern about comparison between meta-learning and transfer-learning. Since both are using the blackbox approach, calculating the feature similarity of hidden layers might not be a good idea. The reason is that if two things are similar at the output, it does not mean that their components must be the same. Since their classification accuracy is similar, I suggest to look at the model complexity and we can base on the Occam's razor principle to see which method is better. Or, could we use different evaluation metric, such as calibration error, to see how difference the two methods are?
limitations: The current form of the paper is quite limited since it is only applicable for one instance of meta-learning (MAML).
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper analyses some of the earlier claims about a simple transfer learning baseline outperforming SOTA meta-learning methods through the lens of task diversity. The paper posits, that with a fair comparison (e.g. same architecture, optimizer etc. ) MAML and transfer learning methods are statistically similar when the meta-training task diversity is low. 
strengths_and_weaknesses: Strengths:
	- The paper tackles an important question on the differences b/w transfer learning and meta-learning methods. It's an important question for the few-shot community whose answer is still not clear except for certain empirical insights. Hence, it's a good problem to study.


While the problem being studied is important, the paper lacks in execution and does not have solid insights or takeaways. 

- The task diversity metric is not novel, but that is fine if more analysis is shown on why such a metric captures task diversity well. The current diversity metric is computed only on miniImagenet and CIFAR which are toy datasets for few-shot learning currently. It would be a good idea to take a wide range of datasets (e.g. MetaDataset, ORBIT, VTAB) and show how the task diversity metric plays out. I feel the current analysis and subsequent insights are incomplete.

-  The empirical results state that under a fair condition, when task diversity is low: MAML and Supervised learning methods are equivalent. What is only MAML chosen for the analysis? It is quite far from the state-of-the-art meta-learning methods. It would be a good idea to choose more meta-learning methods to provide a complete picture. While there exists plethora of methods, some technically simple, but well performing methods could be chosen (e.g., R2D2, MetaOptNet, FEAT).

- There are no empirical results when the task diversity is high. It would be good to run such analysis on more real-world datasets.
I couldn't understand the clear takeaway or message from the paper except for certain empirical insights. From a practical standpoint, it would be good to link the effectiveness of task diversity to how a method might be chosen. For e.g., if the task diversity has a direct correlation with certain methods, just computing the task diversity can help choose from a variety of meta-learners.

questions: The questions are stated in Strengths and Weaknesses.
limitations: While the paper tackles an interesting and important question on when to use transfer learning vs. meta-learning, the paper falls short of execution in its current form. I would suggest to authors to revisit the paper, look into the comments and make the draft stronger with a) more experiments on the task diversity metric; (b) insights and experiments on how to link task diversity with few-shot methods; c) extending beyond MAML and testing more meta-learners.
review_title: null

Meta-review: metareview: The paper performs some empirical study between transfer learning and MAML (as a meta-learning method) through the lens of task diversity. When the task diversity is low, the authors claim that the performance of MAML and transfer learning methods are similar under a fair comparison (e.g. same architecture, optimizer etc). All reviewers are on a negative side for this paper due to weak experimental supports, poor write-up, weak novelty, etc, and the authors also fail to convince the reviewers through their rebuttal responses. Hence, AC cannot recommend acceptance at the current form. In particular, AC agrees that "the paper looks kind of an intermediate work on the way to its finalized version" and "I couldn't understand the clear takeaway or message from the paper except for certain empirical insights" pointed out by reviewers. AC thinks that this paper becomes much stronger if the authors can propose new better meta-learning benchmarks using the insights obtained through the authors' analysis.
confidence: Certain
Decision: Reject