The Curse of Low Task Diversity: On the Failure of Transfer Learning to Outperform MAML and their Empirical Equivalence
Authors: Brando Miranda
Patrick Yu
Yu-Xiong Wang
Oluwasanmi O Koyejo
Venue: NeurIPS 2022 Submitted (2022)
Keywords: meta-learning
few-shot learning
machine learning
deep learning

Abstract: Recently, it has been observed that a transfer learning solution might be all we need to solve many few-shot learning benchmarks -- thus raising important questions about when and how meta-learning algorithms should be deployed. 
In this paper, we seek to clarify these questions by 
1. proposing a novel metric -- the {\it diversity coefficient} -- to measure the diversity of tasks in a few-shot learning benchmark and 
2. by comparing MAML and transfer learning under fair conditions (same architecture, same optimizer and all models trained to convergence).
Using the diversity coefficient, we show that the popular MiniImagenet and Cifar-fs few-shot learning benchmarks have low diversity. 
This novel insight contextualizes claims that transfer learning solutions are better than meta-learned solutions in the regime of low diversity under a fair comparison. 
Specifically, we empirically find that a low diversity coefficient correlates with a high similarity between transfer learning and Model-Agnostic Meta-Learning (MAML) learned solutions in terms of accuracy at meta-test time and classification layer similarity (using feature based distance metrics like SVCCA, PWCCA, CKA, and OPD). 
To further support our claim, we find this meta-test accuracy holds even as the model size changes. 
Therefore, we conclude that in the low diversity regime, MAML and transfer learning have equivalent meta-test performance when both are compared fairly.
We also hope our work inspires more thoughtful constructions and quantitative evaluations of meta-learning benchmarks in the future.

The Curse of Low Task Diversity: On the Failure of Transfer Learning to Outperform MAML and their Empirical Equivalence
Anonymous Author(s)
Affiliation Address email
Abstract
Recently, it has been observed that a transfer learning solution might be all we need to solve many few-shot learning benchmarks thus raising important questions about when and how meta-learning algorithms should be deployed. In this paper, we seek clarify these questions by proposing novel metric the diversity coefficient to measure the diversity of tasks in few-shot learning benchmark and 2. by comparing MAML and transfer learning under fair conditions (same architecture, same optimizer and all models trained to convergence). Using the diversity coefficient, we show that the popular Minilmagenet and Cifar-fs fs few-shot learning benchmarks have low diversity. This novel insight contextualizes claims that transfer learning solutions are better than meta-learned solutions in the regime of low diversity under fair comparison. Specifically, diversity coefficient correlates with a high similarity between transfer learning and Model-Agnostic Meta-Learning (MAML) learned solutions in terms of accuracy at meta-test time and classification layer similarity (using feature based distance metrics like SVCCA, PWCCA, CKA, and OPD). To further support our claim, we find this meta-test accuracy holds even as the model size changes. Therefore, we conclude that in the low diversity regime, MAML and transfer learning have equivalent meta-test performance when both are compared fairly. We also hope our work inspires more thoughtful constructions and quantitative evaluations of meta-learning benchmarks in the future.
10
empirically find that a low
12
14
15
16
17
18
19
20
Introduction
21
22
The success of deep learning in computer vision @1 2), natural language processing (3) 4b game playing (5) 6 ? and more, keeps motivating a growing body of applications of deep learning on an increasingly wide variety of domains. particular, deep learning is now routinely applied to few-shot learning research challenge that assesses model's ability to learn to adapt to new tasks,
23
24
26 new distributions, or new environments. This has been the main research area where meta-learning algorithms have been applied since such strategy seems promising in small data regime due to
28 its potential to learn to learn or learn to adapt. However, it was recently shown ( that a transfer learning model with fixed embedding learning algorithms on numerous few-shot learning benchmarks [C [2) This growing body of evidence coupled with these surprising results in meta-learning raise the question if researchers
match and outperform many modern sophisticated meta-
30
32 applying meta-learning with the right inductive biases (13) [4] and designing appropriate benchmarks
for meta-learnin. Our evidence suggests this is not the case.
34 In this work, we show that when the task diversity - a novel measure variability across tasks low, then MAML (Model Agnostic Meta-Learniigg) (15) learned solutions have the same accuracy
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

36
as transfer learning (i.e., a supervised learned model with a fine-tuned final linear layer). We want to emphasize the importance of doing such an analysis fairly: with the same architecture, same optimizer and all models trained to convergence. This empirical equivalence remained true even as the model size changed - thus further suggesting this equivalence more property of the data than of the model. Therefore, we suggest taking a problem-centric approach to meta-learning and suggest applying Marr's level of analysis 16 17 to few-shot learning - to identify the family of problems suitable for meta-learning. Marr emphasized the importance of understanding the computational problem being solved and not only analyzing the algorithms or hardware that attempts to solve them. An example given by Marr is marveling at the rich structure of bird feathers without also understanding the problem they solve is flight. Similarly, there has been analysis of MAML solutions and transfer learning without putting the problem such solutions should solve into perspective (9). Therefore, in this work, we hope to clarify some of these results by partially placing the current state of affairs in meta-learning from a problem-centrie view. In addition, the novelty of our analysis compared to previous work is that we make analysis intrinsic of the data as a first class citizen.
37
38
39
40
41
43
45
46
47
49
Our contributions summarized as follows:
51
1.
We propose a novel metric that quantifies the intrinsic diversity of the data of a sew- learning benchmark. We call it the diversity coefficient. It enables analysis of meta-learning algorithms through
52
58
cobblem-centric framework. It also goes beyond counting the number of classes or number of data points or counting the number concatenated data sets - and instead quantifies the expected diversity/variabiiiyy of tasks in a few-shot learning benchmark.
54
55
56
2. We analyze the two most prominent few-shot learning benchmarks Minilmagenet and Cifar-fs-and and show that their diversity is low. These results are robust across different ways to measure the diversity coefficient, suggesting that our approach is robust.
57
58
59
3. With this context, we partially clarify the surprising results from 19 by comparing their transfer learning method against models trained with MAML (5). In particular, when making a fair comparison, transfer learning method with a feature extractor fails to outperform MAML. We define a fair comparison when the two methods are compared using the same architecture (backbone), same optimizer and all models trained to convergence. We also show that their final layer makes similar predictions according to neural network distance techniques like distance based Singular Value Canonical Correlation Analysis (SVC)A), Projection Weighted (PWCCA), Linear Centered Kernel Analysis (LINCKA) and Orthogonal Procrustes Distance (OPD). This equivalence holds even as the model size increases.
80
61
62
63
64
85
66
67
68
89
4. Interestingly, we also find that even in the regime where task diversity is low (in Minilma- genet and Cifar-fs), the features extracted by supervised learning and MAML are different implying that the mechanism by which they function is different despite the similarity of their final predictions.
70
71
72
73
5.
As an actionable conclusion, we provide a metric that can be used to analyze the intrinsic diversity of the data in a few-shot learning benchmarks and therefore build more thoughtful environments to drive research in meta-learning.
74
75
addition, our evidence suggests the following test to predict the empirical equivalence of MAML and transfer learning: if the task diversity is low, then transfer learned solutions might fail to outperform meta-learned solutions. This test is easy to run because our diversity coefficient can be done using the Task2Vec method (20) using pre-trained neural network. We also found that random networks were consistent with the results of pre-trained networks on Imagenet.
76
77
78
79
80
81
We hope that this line of work inspires a problem-centric first approach meta-learning- which 82 appears to be especially sensitive to the properties of the problem in question. Therefore, we hope future work takes a more thoughtful and quantitative approach to benchmark creation instead of
83
84 focusing only on making huge data sets.
2 Background
85
In this section, we provide a summary of the background needed to understand our main results. 86
87 Model-Agnostic Meta-Learning (MAML)): The MAMI. algorithm (15) attempts to meta-learn
88 an initialization of parameters for a neural network so that it is primed for fast gradient descent
2

---

89
adaptation. It consists of two main optimization loops: 1) an outer loop used to prime the parameters 90 for fast adaptation, and 2) an inner loop that does the fast adaptation. During meta- meetattestnng, only the inner loop is used to adapt the representation learned by the outer loop.
91
Transfer Learning with Union Supervised Learning (USL): Previous work 19 shows that an initialization trained with supervised learning, on union of all tasks, can outperform many sophisticated methods in meta-learning. In particular, their method consists of two stages: 1) first they use union of all the labels in the few-shot learning benchmark during meta-training and train with standard supervised learning (SL), then 2) during the meta- meta-testing, they use an inference method common in transfer learning: extract a fixed feature from the neural network and fully fine-tune the final classification layer (i.e., the head). Note that our experiments only consider when the final layer is regularized Logistic Regression trained with LBGFS.
93
94
95
96
97
96
99
100
Distances for Deep Neural Network Feature Analysis: To compute the distance between neural
101
networks we use the distance versions of Singular Value Canonical Correlation Analysis (SVCCA) Linear Centered Kernel Analysis
102
2D. Projection Weighted Canonical Correlation (PWCCA)
103
(LINCKA) (3) and Orthogonal Procrustes Distance (OPD) (24 These distances are in the interval
104
0,, 1] and are not necessarily formal distance metric but are guaranteed to be zero when their
105
inputs are equal and nonzero otherwise. This is true because SVCCA, PWCCA, LINCKA are based on similarity metrics and OPD is already a distance. Note that we use the formula d(x,
106
107
1 - simm X, Y) for our distance metrics where sim is one either SVCCA, PWCCA, LINCKA similarity metric and X, Y are matrices of activations (called layer matrices). The distance between two models is computed by choosing a layer and then comparing the features/activations after adaptation for that layer given a batch of tasks represented as a support and query set. more thorough overview of these metrics for the analysis of internal representations for convolutional neural networks (CNNS) can be found in the appendix, section G
108
109
110
111
112
113
The diversity coefficient we propose is the expectation of distance between tasks (explain in more detail in section 5}. Therefore, it is essential to define the distance between different pairs of tasks. We choose the cosine distance between Task2Vec (vectorial) embeddings as in 20). Therefore, we provide a summary of the Tesk2Vec method to compute task embeddings. The vectorial representation of tasks provided by Vec (20 is the vector of diagonal entries of the Fisher Information Matrix (FIM) given fix neural network as a feature extractor also called a probe network after fine-tuning the final classification layer to the task. The authors explain this is good vectorial representation of tasks because 1. It approximately indicates the most informative weights for solving the current task (up to a second order approximation) 2. For rich probe networks like CNNs, the diagonal is more computationally tractable. We choose Vec because the original authors provide extensive evidence that their embeddings correlate with semantic and taxonomic relations between different visual classes- making it convincing embedding for tasks (20) The Task2Vee embedding of task is the diagonal of the following matrix:
Task2 Embeddings for Distances between Tasks:
114
115
116
117
118
119
120
122
124
126
PD..So = F(D, = E.. Bw..y-Moolrhhaull..... V logp(y x.) V pl x. fw)
(I)
127
where fur is the neural networks used a feature extractor with architecture / and weights w,
pf r) is the empirical distribution defined by the training data D) = {(0;, y: } for task r, and ply x. fww) is a deep neural network trained to approximate the (empirical) posterior ply xx r).
128
129
130
We'd like to emphasize that the there is a dependence on target label since Task2Vec fixes the
131
feature extractor (using fw) and then fits the final layer (or "head") to approximate the task posterior
132
distribution p(y x, T).
133
3
Definition of the Diversity Coefficient
134
The diversity coefficient aims to measure the intrinsic diversity (or variability) of tasks in a few-shot learning benchmark. At a high level, the diversity coefficient is the expected distance between a pair of different tasks given a fixed probe network. In this work, we choose the distance to be the cosine distance between vectorial representations (i.e. embeddings) of tasks according to Task2 Vec as described in section 2 Using a fixed probe networks is essential because: 1. Using a fixed probe network means that the distances between different tasks are comparable, as discussed in the original Task2Vee (0 and 2. Since we are computing the distance between different tasks, we need to make
135
136
137
138
140
sure the difference comes from intrinsic properties of the data and not from different source, e.g. if
3

---

142
one uses different models then this might confound the source of variability in our metric. We define the diversity coefficient of a few-shot learning benchmark B follows:
143
div(B) = En 1B ED, mpl
(all) [d(FD..e FD2.tu))
(2)
144
where fw is the neural networks used as a feature extractor with architecture f and weights w, P( | 7) is the empirical distribution defined by the training data D, {(0, V:) for task
146
T1, 72 are tasks sampled from the empirical distribution of tasks p(T B) for the current benchmark
B (i.e. batch of tasks with their data sets D = (Ti, Dr.) )=))),)) a task is the probability distribution
147
148
p(r, y r) of the data, d is a distance metric (for us cosine), fu is the neural networks used as feature extractor with architecture f and weights w. and p(x 7) is the empirical distribution defined by the training data D- = { Vi P= for task T. We'd also like to recall the reader that the definition of task in this setting is of a n-way, k-shot few-shot learning task. Therefore, each task has n classes sampled with k examples used for the adaptation. We'd like to emphasize that the adaptation here is only to fine-tune the final layer according to the Tesk2Vec method for the correct computation of the FIM. Therefore, in this setting we combine the support and query set as the split is not relevant for the computation of the task embedding using Tesk2 Note that the above formulation can be casily adapted to any distance function between tasks, and is not necessarily specific to using the FIM or cosine distance. For example, given the true distributions for tasks one can real distances between probability distributions e.g. Hellinger distance. In addition, it is obvious one can use a distance function besides the cosine distance but choose it in accordance to the original work of
149
150
152
154
156
158
160
Task2 Vec (20).
Experiments
161
This section explains the experiments backing up main results outlined in our list of contributions. Experimental details are provided in the supplementary section and the learning curves displaying the convergence for fair comparison are in supplementary section
163
185
4.1
The Diversity Coefficient of Minilmagenet and Cifar-fss
166
To put our analysis into problem-centric framework, first analyze the problem they are trying to solve through the diversity coefficient. Recall that the diversity coefficient aims to quantify the
187
188
intrinsic variation tasks in a few-shot learning benchmark. We show that the diversity coefficient
189
of the popular Minilmagenet and Cifar-fs benchmarks are low with good confidence intervals using
170
four different probe networks in table
Probe Network
Diversity on MI
Diversity on Cifar-fs
Resnet18 (pt)
0.117 2.098e-5
0.100 + 2.18
Resnet18 (rand)
0.0955 = 1.29e-5
0.103 + 1.05e-5
Resnet34 (pt)
0.0999 = 1.95e-5
0.0847 = 3.06e-5
Resnet34 (rand)
0.0620 = 8.12e-6
0.0643 = 9.64c-6
Table The diversity coefficient of Minilmagenet (MI) and Cifar-fs is low. The diversity coeffi- cient was computed using the cosine distance between different standard n-way, k-shot classification tasks from the few-shot learning benchmark using the Task2Vec method described in section B We used n=5 (number of classes) and k=20 (number of examples per class) since we can use the whole task data to compute the diversity coefficient (no splitting of support and query set required). We used Resnet18 and Resnet34 networks probe networks both pre- pre-trained on ImageNet (indicated as "pt" on table) and randomly initialized (indicated as "rand" on table). We observe that both type of networks and weights give similar diversity results. All confidence intervals were at 95%. To compute results, we used 500 few-shot learning tasks and only compare pairs of different tasks. This results in (5002 500)/2 /2- 124, 750 pair-wise distances used to compute the diversity coefficient.
4.2 Low Diversity Correlates with Equivalence of MAML and Transfer Learning
172 Now that we have placed ourselves in problem-centri framework and shown the diversity coefficient of the popular Minilmagenet and Cifar-fs benchmarks are low proceed to show the failure of

---

174
transfer learning (with USL) to outperform MAML. Crucially, the analysis was done using fair comparison: using the same model architecture, optimizer, and training all models to convergence
175
176
details in section A We used the five-layer CNN used in 15 25) and Resnet12 as in (1)). We provide evidence that in the setting of low diversity:
177
1. The accuracy of an adapted MAML meta-leanas vs. adapted USL pre-trained model are similar and statistically significant, except for one result where transfer learning with USL is worse. This is shown in table 2 and
179
2. The distance for the classification layer decreases sharply according to four cestance-baseer metrics SVCCA, PWCCA, LINCKA, and OPD as shown figure 2 predictions of the two are similar.
182
This implies the
184
For the first point, we emphasize that tables and table taken taken together support our central hypothesis: that models trained with meta-learning not inferior transfer learning models (using USL) when the diversity coefficient is low. Careful inspection reveals that the methods have the same meta-test accuracy with intersecting confidence intervals making the results statistically significant across few-shot benchmarks and architectures. The one exception is the third set of bar plots, where transfer learning with USL is in fact worse.
186
188
189
190
For the second point, refer to figure 2 and observe that as the depth of the network increases, the distance between the activation layers of model trained with MAML vs USL increases until it reaches the final classification layer - where all four metrics display noticeable dip. In particular, PWCCA considers the two prediction layers identical (approximately zero distance). This final point is particularly interesting because PWCCA is weighted according to the CCA weights that stabilize with the final predictions of the network. This means that the PWCCA distance value is reflective of what the networked actually learned and gives a more reliable distance metric (for details, refer to the appendix section G.5). This is important because this supports our main hypothesis: that at prediction time there is an equivalence between transfer learning and MAML when the diversity coefficient is low.
192
194
196
198
189
Porformarce ce Compar son MANL vs Transfer Leaming (US.)
NovE 1o
USL
No MecaTallccorr
60-
PanMe:22 Datonct, Arch teclure
Clast, 5CME Crlani, hoNee12
Figure 1: MAML trained models and union supervised trained (USL) models have statistically equivalent meta-test accuracy for Minilmagenet and Cifar--s with Resnet12 and five layer CNNs. This holds for both the Resnet12 architecture used in 19 and the 5 layer CNN (indicated as "SCNN") in (25). Results used a (meta) batch-size of 100 tasks and 95% confidence intervals. All MAMI. models were trained with 5 inner steps during meta-training. "MAML55 and "MAML10" in the bar plot indicates the adaptation method used at test time i.e. we used 5 inner steps and 10 inner steps at test time. Minilmagenet is abbreviated as "MI" in the figure.
200
4.3
Is the Equivalence of MAML and Transfer Learning related to Model Size or Low Diversity?
201
202
An alternative hypothesis to explain the equivalence of transfer learning (with USL) and MAML could
203
be due to the capabilities of large neural networks to be better meta-learners in general. Inspired by the impressive ability of large language models to be few-shot (or even zero-shot) learners 4 27 28 3
205
we hypothesized that perhaps the meta-learning capabilities of deep learning models is a function of the model size. If this were true, then we expected to see the difference in meta-test accuracy
5

---

Meta-train Initialization
Adaptation Inference
Meta-test Accuracy
Random
no adaptation no adaptation
19.3 + 0.80
MAMLO
20.0 + 0.00
USL
no adaptation
15.0 + 0.26
Random
MAML5 adaptation
34.2 + 1,16
MAML5
MAML5 adaptation
62.4 + 1.64
USL
MAML5 adaptation
25.1 + 0.98
Random
MAMLI0 adaptation
34.1 + 1.23
MAML5
MAML10 adaptation
62.3 + 1.50
USL
MAML10 adaptation
25.1 + 0.97
Random
Adapt Head only (with LR)
40.2 + 1.30
MAML5
Adapt Head only (with LR)
59.7 + 1.37
USL
Adapt Head only (with LR)
60.1 + 1.37
Table 2: MAML trained representations and supervised trained representation have statisti- cally equivalent meta-test accuracy on Minilmagenet which has low diversity. The transfer model's adaptation is labeled "'Aapted Head only (with LR)" which stands for "Logistic Regres- sion (LR)" used in (I9). More precisely, we used Logistic Regression (LR) with LBFGS with the default value for the 12 regularization parameter given by Python's 's Sklearn. Note that increase in inner steps from 5 to 10 with the MAML5 trained model does not provide an additional meta-test accuracy boost, consistent with previous work (26). Note that the fact that the MAML representation matches the USL. representation when both use the same adaptation method is not surprising given that: 1) previous work has shown that the distance between the body of adapted MAML model is minimal compared to the unadapted MAML (which we reproduce in S in the green line) and 2) the fact that a MAML5 adaptation is only 5 steps of MAML while LR fully converges the prediction layer. We want to highlight that only the MAML5 model achieved the maximum meta-test performance of 0.6 with the MAML5 adaptation suggesting that the USL and MAML5 meta-learing algorithms might learn different representations. For USL to have fair comparison during meta-test time when using the MAML adaptation, we provide the MAML final layer learned initialization parameters to the USL. model (but any is fine due to convexity when using a fixed feature extractor). This is needed since during meta-training USL is trained with a union of all the labels (64) - so does not even have the right output size of 5 for sho-shot prediction. Meta-testing was done in the standard 5-way, 5-shot regime.
of MAML and USL to be larger for smaller models and the difference to decrease the model size increased, Once the two models were, of the same size but large enough, we hypothesized that the meta-test accuracy would be the same. We tested this to rule out that observations were consequence of the model size. The results were negative and surprisingly the equivalence between hypothesis that the low task diversity might be a bigger factor explaining our observations. We show this in figure 3 and want draw attention to the fact this statistical equivalence holds even when using only four filters - the case where we expected the biggest difference.
208
210
MAML and USL seems to hold even as the model increased strengthening
212
213
214
215
4.4
MAML learns a different base model compared to Union Supervised Learned models even in the presence of low task diversity
217
The first four layers of figure 2 shows how large the distance is of a MAML representation compared to SL representation. In particular, it is much larger than the distance value in the range [0. 0.1] from previous work that compared MAML vs. adapted MAML (8). We reproduced that and indeed
218
219
MAML vs. adapted MAML has small difference (smaller for us) supporting our observations that a MAML vs. a USL learned representations are different at the feature extractor layer even when the diversity is low, Results are statistically significant.
221

---

Recrosetiaioon dolance with metrs SVCDA
Reprosetiation dillance metric PWOCA
13-
Hadp Adluted
Adppped wead Adapted HAML
14
0.5
0.4
04
Cano Mayer
Laper
Reproser Lution dolance matri
Resrisemtation distune with matri
2.3
1.
3.8
0.5
0.2
Figure 2: The classification layer of transfer learning and MAML5 model decrease in distance -implying similar predictions. More precisely, an initialization trained with 5 inner steps (MAML5) has an increasingly similar head (classifier) after adaptation with MAML5 compared the classifier layer of the union supervise learned (USL) model that has been adapted only at the final layer. In particular, the USL model has been adapted with Logistic Regression (LR) with LBFGS with the default value for the 12 regularization parameter given by Python's ' Sklearn (as in (I)). We showed this trend with four different distance metries SVCCA, PWCCA, LICKA, and OPD referenced in section 2 Observe that according to PWCCA the distance between the predictions is zero. This true because the distance of classification layer (indicated as "head" the figure) zero. The architecture used here is a five layer CNN as in (15 25 with their same setup. The benchmark used for this analysis is Minilmagenet.
Modol Size Effect Performance Diffrrnnce USL MAML
Model Offect Pe-formance Di Difheence
0.575
0.330
0:50
0.525
0.225
0.303
0.500
0.475
0:475
0.450
0.43
The meta-test accuracy of MAML and transfer learning using USL is similar in statistically significant way regardless of the model size. In this experiment, we used the Minilmagenet benchmark, the five layer CNN used in (5 25), and only increased the filter size using sizes 4, 8, 16, and 32. We made sure the comparison was fair by using the same architecture, optimizer, and trained all models to convergence. During meta-traine training. the MAML model was trained using 5 inner steps. The legends indicating MAM15 and MAML10 refer to the number of inner steps used at test time. We used a (meta) batch size of 100 tasks.
Figure 3:

---

223
4.5
Synthetic Experiments showing closeness of MAML and Transfer Learning as Diversity Changes
224
225
In this section, we show the closeness of MAML and transfer learning (with USL) for synthetic experiments for low and high diversity regimes in Figure I In the low regime, the two methods are equivalent in a statistically significant way which supports the main claims of our paper. As the diversity increases, however, the difference between USL and MAML increases (in favor of USL).
226
227
228
This will be explored further in future work.
230
The task is the usual n-way, k-shot tasks, but the data comes from Gaussian and the meta-learners are tasked with classifying from which Gaussian the data points came from in a few-shot learning manner. Benchmarks are created by sampling a Gaussian distribution with means moving away from
231
232
233
the origin as the benchmark changes. Therefore, the Gaussian benchmark with the highest diversity coefficient has Gaussians that are the furthest from the origin. We computed the diversity coefficient using a proper distance between distributions using the Hellinger distance eluded in section E instead of the FIM distance. We can do this because we know the ground truth distribution our synthetic experiments, and Gaussians have a closed form Hellinger distance. Details on the n-way Gaussian
234
236
238
benchmark and diversity coefficient using the Hellinger distance can be found in supplementary
section D and F
Divesity effect Performance Difference USL os HANI
Diversily Offict Pas formara nue Difference of USL vs H4ML
1.0
1.0
USL
0.8
0.-
0.5
0.7
Hellingr Conficitnt
Figure 4: The meta-test accuracy of MAML and transfer learning using USL is similar in a statistically equivalent way in the low diversity regime in the 5-way, 10-shot Gaussian Benchmarks. MAMI models were trained with inner steps. MAML55 and MAML.1 indicate the adaptation procedure at test time. Results used a (meta) batch-size of 500 tasks and 95% confidence intervals. As the diversity of the benchmark increases, the Gaussian tasks are sampled further away from the origin. Note, as the diversity increases, the difference between USL and MAML increases (in favor of USL).
5
Related Work
240
241
Our work proposes a problem- centric framework for the analysis of meta-learning algorithms inspired from previous puzzling results (1)). We propose to use a pair-wise distance between tasks and
242
243
analyze how this metric might correlate with meta-learning. The closest line of work for this the long line of work by (20) where they suggest methods to analyze the complexity of a task, propose unsymmetrical distance metrics for data sets, reachability of tasks with SGD, ways to embed entire
245
30 ET). We believe this line of work to be very fruitful and hope that more people adopt tools like the ones they suggest and we propose in this paper before researching or deploying meeaallerning algorithms. We hope this helps meta-learning methods succeed in practice since cognitive science suggests meta-learning is a powerful method humans use to learn (32). In the future, we hope to compare 200 's distance metrics between tasks with ours to provide a further unified understanding of meta- learning and transfer learning. A contrast between their work and ours
data sets and more 20
2y
247
249
251
is that we focus our analysis from meta- perspective applied to few-shot learning while their focus is understanding transfer learning methods between data sets.
253
The use of distance metric in our definition of the diversity coefficient is inspired by the analysis done by (18). They showed that MAMI. functions mainly via feature re-use than by rapid learning i.e. that model trained with MAML changes very little after the MAML adaptation. The main difference
255
8

---

257
of their work with our is: 1) that we compare MAML trained models against union supervised learned models (USL) instead of only comparing MAML against adapted MAML, and 2) that we explicitly analyzed properties of the data sets. In addition, we use large set of distance metrics for our analysis including: SVCCA, PWCCA, LINCKA and OPD as proposed by 21 22 23 24).
258
260
281
Our work is most influenced by previous work suggesting modern meta-learning requires rethinking The main difference of our work with theirs that we analyzed the internal representation of the meta-Icarning algorithms and contextualize these with quantifiable metrics of the problem being solved. Unlike their work, we focused on a fair comparison between meta-learning methods by ensuring the same neural network backbone was used. Another difference is that they gained further accuracy gains by using distillation a method we did not analyze and leave for future work.
262
263
264
285
266
267 A related line of work (33 26 first showed that there exist synthetic data sets that are capable of exhibiting higher degrees of adaptation as compared to the original work by (I8). The difference is that they did not compare MAML. models against transfer learning methods like we did here. Instead, they focused on comparing adapted MAML models vs, unadapted MAML models.
268
269
270
271
Another related line of work is the predictability of adversarial transferability and transfer learning. They show this both theoretically and with extensive experiments (34) The main difference between their work and ours is that they focus their analysis mainly on transfer learning, while we concentrated on meta- meta-learning for few-shot learning. In addition, we did not consider adversarial transferability while that was a central piece of their analysis. Further, related work is outlined in the supplementary section MI
273
275
276
6
Discussion and Future Work
278
In this work, we presented a problem-centric framework when comparing transfer learning methods with meta-learning algorithms- using USL and MAML as the canonical representatives of transfer and meta-learning methods respectively. We showed the diversity coefficient of the popular Minilma- genet and Cifar-fs benchmark is low and that under fair comparison MAML is very similar to transfer learning with USL. This was also true even when decreasing the model size removing the alternative hypothesis that the equivalence of MAML and transfer learning with USL held due to large models. Instead, this suggests strengthens our hypothesis that the diversity of the data might be the driving factor. The equivalence of MAML and USL also replicated in our synthetic experiments. Therefore, we challenge the suggestions from previous work (19) that only a good embedding can beat more effective than sophisticated meta-learning especially in the low diversity regime. In addition, our synthetic experiments show promising scenario where we can systematically differ- entiate meta-learning algorithms from transfer learning algorithms- which supports our actionable suggestion to use the diversity coefficient to effectively study meta-learning and transfer learning algorithms, We hope to study this in more depth in the future with real and synthetic data.
280
282
284
286
288
289
290
291
292
We also have theoretical results from a statistical decision perspective in the supplementary section ?? that inspired this work and suggest that when the distance between tasks is zero then the predictions of transfer learning, meta- learning and even a fixed model with no adaptation are all equivalent (with the 12 loss). The results are theoretically limited because we can only reason when the diversity is exactly zero, but regardless provided an interesting perspective to study and inspire empirical work.
294
296
297
We hope this work inspires the community in meta-learning and machine learning to construct benchmarks from a problem-centric perspective - that go beyond large scale data sets - using have quantitative metrics.
298
299
References
301 [1] A. Krizhevsky, I. Sutskever, and G. E, Hinton, "ImageNet Classification with Deep Convolu- tional Neural Networks,"
302
303
[2] K. He, X. Zhang, S. Ren, and J. Sun, "Deep Residual Learning for Image Recognition," " Proceed- ings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2016-December, pp. 770-778, dec 2015.
304
305
[3] Devlin, M. W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of Deep Bidirectional 306
Transformers for Language Understanding." NAACL HLT 2019 2019 Conference of the

---

308
North American Chapter of the Association for Computational Linguisties: Human Language Technologies Proceedings of the Conference, vol. 1, pp. 4171-4186, oct 2018.
309
310 [4] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, Clark, C. Berner, S. Mccandlish, A. Radford, Sutskever, and D. A. Openai, "Language Models are Few-Shot Learners," tech. rep., 2020.
312
313
314
315
[5] D. Silver, A. Huang, C. J. Maddison, A. Guez, L., Sifre, G. Van Den Driessche, Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalch- brenner, I. Sutskever, "Mastering the game of Go with deep neural networks and tree search," Nature 2016 529:7587, vol. 529, pp. 484-489, jan 2016.
316
317
Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis.
318
319
320
[6] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, PPayynn Atari with Deep Reinforcement Learning."
322
77 W. Ye, S. Liu, T. Kurutach, P. Abbeel, Y. Gao, T. University, U. C. Berkeley, S. Qi, and Z. Institute, "Mastering Atari Games with Limited Data," oct 2021.
324
88 Y. Tian, Y. Wang, D. Krishnan, J. B. Tenenbaum, and P. Isola, "Rethinking Few-Shot Image Classification: Good Embedding Is All You Need?," 2020.
325
326
9 W.- Chen, Y.-C. Liu, Z. Kira, Y.-C. F. Wang, and J.-B. Huang. "A Closer Look Few-shot Classification," " 7th International Conference on Learning Representations, ICLR 2019, 2019.
327
328
[10] Y. Chen, X. Wang, Z. Liu, H. Xu, and Darrell, "A New Meta-Baseline for Few-Shot Learning, tech. rep.
329
[11] G. S. Dhillon, P. Chaudhari, A. Ravichandran, and S. Soatto, "A Baseline for Few-Shot Image Classification," 2019.
331
[12] S. Huang and D. Tao, "All you need is a good representation: A multi-level and classifier-centric representation for few-shot learning." " 2019.
333
334
[13] T. M. Mitchell, "The Need for Biases in Learning Generalizations by The Need for Biases in Learning Generalizations," 1980.
335
336
[14] S. B.-D. Shai Shalev-Shwartz, "Understanding Machine Learning: From Theory to Algorithms," " Cambridge University Press, 2014.
337
338
[15] C. Finn, P. Abbeel, and S. Levine, -Mooel-Agnostic Meta-Learning for Fast Adaptation of Deep Networks," 2017.
340
[16] J. B. Hamrick Deepmind and S. M. Deepmind, "LEVELS OF ANALYSIS FOR MACHINE LEARNING," 2020.
342
[17] D. Marr, "Vision: Computational Investigation into the Human Representation and Processing of Visual Information," Phenomenology and the Cognitive Sciences, vol. 8, no. 4, p. 397, 1982.
343
3/4
[18] A. Raghu, M. Raghu, S. Bengio, and G. Brain, "Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML," tech. rep., 2020.
345
[19] Y. Tian, Y. Wang, D. Krishnan, J. B. Tenenbaum, and P. Isola, "Rethinking Few-Shot Image Classification: a Good Embedding Is All You 2020.
347
[20] A. Achille UCLA, M. Lam AWS, R. Tewari AWS, A. Ravichandran AWS, S. Maji UMass,
349
S. Soatto UCLA, and P. Perona Caltech, "TASK2VEC: Task Embedding for Meta- Metta-Ln Charless Fowlkes UCI and AWS," tech. rep., 2019.
350
351
[21] M. Raghu, J. Gilmer, Yosinski, and Sohl-Dickstein, "SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretabilityy" tech. rep., 2017.
10

---

353 [22] A. S. Morcos, Deepmind. M. Raghu, S. Bengio, and G. Brain, "Insights on representational similarity in neural networks with canonical correlation." tech. rep., 2018
354
355
[23] S. Kornblith, M. Norouzi, H. Lee, and G. Hinton, "Similarity of Neural Network Representations Revisited," tech. rep., may 2019.
357
[24] Ding, J.-S. Denain, and Steinhardt, "Grounding Representation Similarity with Statistical Testing," " 2021.
358
[25] S. Ravi and H. Larochelle, "Optimization as a model for few-shot learning," tech. rep., 2017.
360
[26] B. Miranda, "An empirical study of the properties of meta-learning presentation," Illinois Digital Environment for Access to Learning and Scholarship (IDEALS), dec 2020.
361
362
[27] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg. A. Bosselut, E. Brunskill, E. Brynjolfsson, S. Buch, D. Card, R. Castellon, N. Chatterji. A. Chen, K. Creel, J. Q. Davis, D. Demszky, C. Donahue, M. Doumbouya, E. Durmus, S. Ermon, J. Etchemendy, K. Ethayarajh, I., Fei-Fei, C. Finn, Gale, L. Gillespie, K. Goel, N. Goodman, S. Grossman, N. Guha, T. Hashimoto, P. Henderson, J. Hewitt, D. E. Ho, J. Hong, K. Hsu, Khani,
363
365
366
367
J. Huang, T. Icard, Jain, D. Jurafsky, P. Kalluri, S. Karamcheti, G. Keeling. O. Khattab, P. W. Kohd, M. Krass, R. Krishna, R. Kuditipudi, A. Kumar, F. Ladhak, M. Lee, T. Lee, J. Leskovec, I. Levent, X. L. Li, X. Li, T. Ma, A. Malik, C. D. Manning, S. Mirchandani, E. Mitchell, Z. Munyikwa, S. Nair, A. Narayan, D. Narayanan, B. Newman, A. Nie, J. C. Niebles, H. Nilforoshan, J. Nyarko, G. Ogut, L. Orr, I. Papadimitriou, J. S. Park, C. Piech, E. Portelance, C. Potts, A. Raghunathan, R. Reich, H. Ren, F. Rong, Y. Roohani, C. Ruiz, J. Ryan, C. Ré, D. Sadigh, S. Sagawa, K. Santhanam, A. Shih, K. Srinivasan, A. Tamkin, R. Taori, A. W. Thomas, F. Tramèr, R. F. Wang, and W. Wang, "On the Opportunities and Risks of Foundation Models," aug 2021.
369
371
373
375
378
[28] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, Clark, G. Krueger, and I. Sutskever, "Learning Transferable Visual Models From Natural Language Supervision," feb 2021.
377
378
[29] "The Dynamic Distance Between Learning Tasks: From Kolmogorov Complexity to Transfer Learning via Quantum Physics and the Information Bottleneck of the Weights of Deep Networks," 2018.
380
382
[30] A. Achille, G. B. Mbeng, and S. Soatto, "Dynamics and Reachability of Learning Tasks," 2019.
383
[31] A. Achille, G. Paolini, G. Mbeng, and S. Soatto, "The Information Complexity of Learning Tasks, their Structure and their Distance," 2020..
[32] B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. Gershman, "Building Machines That Learn and Think Like People," Behavioral and Brain Sciences, vol. 40, 2016.
386
387
[33] B. Miranda, "An empirical study of the properties of meta-learning presentation," Illinois Digital Environment for Access to Learning and Scholarship (IDEALS), 2020.
388
389
[34] K. Liang, J. Y. Zhang, B. Wang, Z. Yang, O. Koyejo, and B. Li, "Uncovering the Connections Between Adversarial Transferability and Knowledge Transferability. 2021.
Checklist
392
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO]] to [Yes], [No] or [N/A]. . You are strongly encouraged to include a justification to your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
394
396
Did you include the license to the code and datasets? [Yes] See Section ??.
397
Did you include the license to the code and datasets? [No] Code and data will be released accepted.
11

---

399
Did you include the license to the code and datasets? [N/A]
400
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
402
403
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
405
406
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
408
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
409
410
2. If you are including theoretical results...
411
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
412
(b) Did you include complete proofs of all theoretical results? [Yes]
413
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [No]
415
416
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
417
418
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
420
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
421
422
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
423
(a) your work uses existing assets, did you cite the creators? [Yes]
424
(b) Did you mention the license of the assets? [No]
425
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
427
428
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
430
5.
If you used crowdsourcing conducted research with human subjects...
431
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [TODO]
432
435
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [TODO]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [TODO]
436
12


Reviews:
[{"review_id": "uhVB3MINJj", "reviewer": "Reviewer_3tgb", "summary": "This paper presents a novel metric which quantifies the diversity of few-shot learning benchmarks. The authors compare MAML with transfer learning w.r.t. multiple aspects on two few-shot learning benchmarks. Several empirical discoveries are presented including that transfer learning fails to outperform MAML when the intrinsic diversity is low for the evaluation benchmark.", "questions": "1. In the empirical evaluation, it seems that only classification task and ConvNet-based models are used. For making a concrete conclusion, more tasks and diverse architectures (e.g. transformer-based backbone) will be more persuasive. Otherwise, the scope of the conclusion needs to be narrowed down.\n2. The definition of tasks is not clear in the caption of Table 1. Are the 500 few-shot learning tasks subsets from the original dataset? Do they have disjoint classes? Does changing the definition of these tasks results in different diversity coefficient?\n3. Line 292, 396, the section number is missing.", "limitations": "The limitation of the work is discussed in Section 6. The potential negative societal impact is not discussed.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is quite easy to follow.\n- The paper compares MAML and transfer learning from an interesting perspective, the so-called \"problem centric\" approach.\n- The idea of rethinking the diversity of datasets is important for analyzing models, and this paper makes a good case for it by conducting empirical evaluations.\n- The diversity coefficient is intuitive.\n\nWeakness:\n- The empirical evaluation needs to be stronger. \n- Some parts are not clear.\n- Minor typos.\nPlease see questions for details.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "jema0OvfYf", "reviewer": "Reviewer_JSFh", "summary": "The paper analyses some of the earlier claims about a simple transfer learning baseline outperforming SOTA meta-learning methods through the lens of task diversity. The paper posits, that with a fair comparison (e.g. same architecture, optimizer etc. ) MAML and transfer learning methods are statistically similar when the meta-training task diversity is low. ", "questions": "The questions are stated in Strengths and Weaknesses.", "limitations": "While the paper tackles an interesting and important question on when to use transfer learning vs. meta-learning, the paper falls short of execution in its current form. I would suggest to authors to revisit the paper, look into the comments and make the draft stronger with a) more experiments on the task diversity metric; (b) insights and experiments on how to link task diversity with few-shot methods; c) extending beyond MAML and testing more meta-learners. ", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\t- The paper tackles an important question on the differences b/w transfer learning and meta-learning methods. It's an important question for the few-shot community whose answer is still not clear except for certain empirical insights. Hence, it's a good problem to study.\n\n\nWhile the problem being studied is important, the paper lacks in execution and does not have solid insights or takeaways. \n\n- The task diversity metric is not novel, but that is fine if more analysis is shown on why such a metric captures task diversity well. The current diversity metric is computed only on miniImagenet and CIFAR which are toy datasets for few-shot learning currently. It would be a good idea to take a wide range of datasets (e.g. MetaDataset, ORBIT, VTAB) and show how the task diversity metric plays out. I feel the current analysis and subsequent insights are incomplete.\n\n-  The empirical results state that under a fair condition, when task diversity is low: MAML and Supervised learning methods are equivalent. What is only MAML chosen for the analysis? It is quite far from the state-of-the-art meta-learning methods. It would be a good idea to choose more meta-learning methods to provide a complete picture. While there exists plethora of methods, some technically simple, but well performing methods could be chosen (e.g., R2D2, MetaOptNet, FEAT).\n\n- There are no empirical results when the task diversity is high. It would be good to run such analysis on more real-world datasets.\nI couldn't understand the clear takeaway or message from the paper except for certain empirical insights. From a practical standpoint, it would be good to link the effectiveness of task diversity to how a method might be chosen. For e.g., if the task diversity has a direct correlation with certain methods, just computing the task diversity can help choose from a variety of meta-learners.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "W96vJbpx-Si", "reviewer": "Reviewer_PmXX", "summary": "The aim of the paper is to empirically compare the performance of transfer-learning (fine-tuning last fully connected layer in a pre-trained neural network) and meta-learning (and in particular, MAML - an instance of meta-learning). The two learning approaches of interest are investigated in different points of views: intrinsic diversity of the dataset used, similarity between their features at some imtermediate layers of the neural networks used and different model sizes. The empirical results show that MAML and fine-tuning a pre-trained model are equivalent in terms of performance.", "questions": "The proposed *diversity coefficient* based on an average distance is not very convincing. I wonder if the diversity of a dataset can be calculated informatically. What I mean is that given a set of vectors representing tasks (obtained from Task2Vec), could we estimate the task distribution and explicitly calculate the entropy of that distribution, or implicitly calculate the entropy from those vectors? The reason for entropy is that the higher the entropy, the more informative that task distribution, which, to me, means the higher diversity.\n\nI have a concern about comparison between meta-learning and transfer-learning. Since both are using the blackbox approach, calculating the feature similarity of hidden layers might not be a good idea. The reason is that if two things are similar at the output, it does not mean that their components must be the same. Since their classification accuracy is similar, I suggest to look at the model complexity and we can base on the Occam's razor principle to see which method is better. Or, could we use different evaluation metric, such as calibration error, to see how difference the two methods are?", "limitations": "The current form of the paper is quite limited since it is only applicable for one instance of meta-learning (MAML).", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe major strength of the paper is to study the difference between meta-learning and transfer-learning. This is an interesting research direction to understand which method performs better in which setting. Also, as stated at the end of the introduction section, it provides another point of view in terms of evaluation, not just simply making larger and larger datasets.\n\nAnother strength of the paper is to quantify the characteristic of tasks sampled according to the episode setting. The newly-proposed metric, *diversity coefficient* , is based on the representation of tasks, known as Task2Vec, to calculate the cosine similarity between tasks belonging to the same dataset. This quantifies the \"information\" of the dataset, providing insights into the performances of the model of interest.\n\nOne more strength is that the paper is gone through a thorough review of the literature with an extended section in the literature.\n\n---\n\n**Weaknesses**\n\nFirstly, the paper lacks of a coherence to connect all the contributions together. As I understand, the main goal is to show that meta-learning and transfer learning share similar performances even when the dataset used is diversity enough or not. However, the presentation and writing leads to a hard time for me to get to the point.\n\nSecondly, the newly-proposed metric, known as *diversity coefficient*, is a straight-forward extension from Task2Vec, which heavily relies on the *probe* network used. Thus, I believe that using a single network is not enough, but we might need an ensemble of networks to marginalize out the bias of *probe* networks. In addition, the proposed metric might not work well in the present of outlier tasks since it is based on the expected (or average) similarity distance. Furthermore, as shown in (Dhillon et. al 2020, Figure 1) and (Nguyen et. al 2021, Figure 1), the accuracy of different tasks evaluated by the same model varies significantly. Thus, simply picking a small number of tasks, e.g. 500 tasks as mentioned in Table 1, to calculate the metric is inadequate. And since the number of tasks formed from a dataset is very large, e.g. there are $n = \\binom{64}{5} = 7,624,512$ 5-way classification tasks from the training set of mini-ImageNet, the number of distances calculate would be $\\mathcal{O}(n^{2})$, making this metric intractable when there are more classes. Of course, we might not need to take all tasks into account, but only consider the set of *core* tasks. However, finding that set is another problem.\n\nFinally, the evaluation to compare meta-learning and transfer-learning is based on classification accuracy on two datasets only. I believe that this might be insufficient to have a clear conclusion.\n\n**References**\n- Guneet S Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. \"A baseline for few-shot image classification\". In *International Conference on Learning Representations*, 2020.\n- Cuong C. Nguyen, Thanh-Toan Do, and Gustavo Carneiro. \"Probabilistic task modelling for meta-learning\". In *Uncertainty in Artificial Intelligence*, 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4-O5PLN94Fb", "reviewer": "Reviewer_vKuj", "summary": "This paper revisits the agreement of recent results of transfer learning methods outperforming meta-learning algorithms in few-shot learning domains, and claims that they are in fact not much different in performance particularly for a dataset with low diversity. To quantify the diversity, a new metric, called diversity coefficient, is proposed by leveraging the vectorized representation of each task obtained by Task2Vec. In empirical study, the authors' claim seems to be valid as the accuracy are pretty much similar to each other in between MAML and USL (i.e., a transfer learning method) using Cifar-FS and Mini-ImageNet, both of which have low diversity.", "questions": "1. The experimental results of Figure 2 is somewhat obvious. Are they intended to see models trained by MAML and USL are not the same in their weights?\n\n2. Could you provide any insights on why diversity matters in the performance of MAML and USL? \n\n3. Given a high diversity dataset, USL seems to be still better than MAML. Doesn't this imply that USL deals with a more challenging learning problem setting better than MAML?\n\n4. How can we trust the proposed metric indeed well reflects the true diversity? Any experimental justification on that?", "limitations": "Potential negative societal impact is neither discussed nor applicable to this work.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "(Strengths)\n1. The main claim of this paper sounds very important as it tries to break the common knowledge agreed by many recent works in few-shot learning.\n2. Experiments are well designed in a way that the corresponding results properly justify the main claims and questions.\n\n(Weaknesses)\n1. Although the paper starts with ambitious insights, it does not properly give a meaningful conclusion in the end. There is no explanation on why low diversity leads to the similar performance of MAML and USL. Furthermore, when the diversity gets higher, it seems that the common knowledge is still correct as USL starts to outperform MAML. Thus, it would be much more interesting if the paper deeply investigates the hidden relationship between diversity and performance of both methodologies in few-shot learning. At the present form, the paper looks kind of an intermediate work on the way to its finalized version.\n\n2. The proposed diversity metric is adequate, but its novelty is mostly coming from the corresponding preliminary work, i.e., Task2Vec. What is newly proposed in this work is to obtain the expected distance between tasks via their vectorized representations.\n\n3. In terms of presentation quality, this write-up should be more improved and finalized. There are some typos and missing references throughout the paper, and most of figures are in low quality and hard to recognize.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject